Login
[x]
Log in using an account from:
Fedora Account System
Red Hat Associate
Red Hat Customer
Or login using a Red Hat Bugzilla account
Forgot Password
Login:
Hide Forgot
Create an Account
Red Hat Bugzilla – Attachment 1458119 Details for
Bug 1600138
[Bluestore]: one of the osds flapped multiple times with 1525: FAILED assert(0 == "bluefs enospc")
[?]
New
Simple Search
Advanced Search
My Links
Browse
Requests
Reports
Current State
Search
Tabular reports
Graphical reports
Duplicates
Other Reports
User Changes
Plotly Reports
Bug Status
Bug Severity
Non-Defaults
|
Product Dashboard
Help
Page Help!
Bug Writing Guidelines
What's new
Browser Support Policy
5.0.4.rh83 Release notes
FAQ
Guides index
User guide
Web Services
Contact
Legal
This site requires JavaScript to be enabled to function correctly, please enable it.
osd.10.log
ceph-osd.10_assert.log (text/plain), 17.54 MB, created by
Parikshith
on 2018-07-11 13:57:13 UTC
(
hide
)
Description:
osd.10.log
Filename:
MIME Type:
Creator:
Parikshith
Created:
2018-07-11 13:57:13 UTC
Size:
17.54 MB
patch
obsolete
>2018-07-11 03:42:01.402783 7f41651ed700 -1 received signal: Hangup from PID: 11025 task name: killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw UID: 0 >2018-07-11 03:42:03.575697 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2523] Generated table #35242: 2638241 keys, 74546976 bytes >2018-07-11 03:42:03.575747 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280523575719, "cf_name": "default", "job": 2523, "event": "table_file_creation", "file_number": 35242, "file_size": 74546976, "table_properties": {"data_size": 67112103, "index_size": 690700, "filter_size": 6743183, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:42:05.532201 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2523] Generated table #35243: 1628385 keys, 46137746 bytes >2018-07-11 03:42:05.532257 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280525532222, "cf_name": "default", "job": 2523, "event": "table_file_creation", "file_number": 35243, "file_size": 46137746, "table_properties": {"data_size": 41547691, "index_size": 426728, "filter_size": 4162339, "raw_key_size": 47057989, "raw_average_key_size": 28, "raw_value_size": 19465004, "raw_average_value_size": 11, "num_data_blocks": 10210, "num_entries": 1628385, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "13", "kMergeOperands": "0"}} >2018-07-11 03:42:05.532299 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2523] Compacted 4@1 + 5@2 files to L2 => 642507683 bytes >2018-07-11 03:42:05.550725 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:42:05.550441) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 3 48 446 3639 671 0] max score 1.09, MB/sec: 23.8 rd, 23.8 wr, level 2, files in(4, 5) out(9) MB in(269.5, 343.3) out(612.7), read-write-amplify(4.5) write-amplify(2.3) OK, records in: 22734307, records dropped: 0 > >2018-07-11 03:42:05.550735 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:42:05.550468) EVENT_LOG_v1 {"time_micros": 1531280525550456, "job": 2523, "event": "compaction_finished", "compaction_time_micros": 26962798, "output_level": 2, "num_output_files": 9, "total_output_size": 642507683, "num_input_records": 22734307, "num_output_records": 22734307, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 3, 48, 446, 3639, 671, 0]} >2018-07-11 03:42:05.551652 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280525551648, "job": 2523, "event": "table_file_deletion", "file_number": 35232} >2018-07-11 03:42:05.551677 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280525551675, "job": 2523, "event": "table_file_deletion", "file_number": 35231} >2018-07-11 03:42:05.551690 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280525551687, "job": 2523, "event": "table_file_deletion", "file_number": 35230} >2018-07-11 03:42:05.551701 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280525551698, "job": 2523, "event": "table_file_deletion", "file_number": 35229} >2018-07-11 03:42:05.551715 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280525551713, "job": 2523, "event": "table_file_deletion", "file_number": 35208} >2018-07-11 03:42:05.551726 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280525551724, "job": 2523, "event": "table_file_deletion", "file_number": 35204} >2018-07-11 03:42:05.551740 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280525551738, "job": 2523, "event": "table_file_deletion", "file_number": 35203} >2018-07-11 03:42:05.551754 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280525551752, "job": 2523, "event": "table_file_deletion", "file_number": 35201} >2018-07-11 03:42:05.551766 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280525551764, "job": 2523, "event": "table_file_deletion", "file_number": 35200} >2018-07-11 03:42:05.551967 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_picker.cc:461] [default] Expanding@2 1+1(74546976+46965085 bytes) to 3+1 (223639116+46965085bytes) > >2018-07-11 03:42:05.552062 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2524] Compacting 3@2 + 1@3 files to L3, score 1.09 >2018-07-11 03:42:05.552073 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2523 Base level 2, inputs: [35240(71MB) 35241(71MB) 35242(71MB)], [35211(44MB)] > >2018-07-11 03:42:05.552086 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280525552076, "job": 2524, "event": "compaction_started", "files_L2": [35240, 35241, 35242], "files_L3": [35211], "score": 1.09112, "input_data_size": 270604201} >2018-07-11 03:42:08.851823 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2524] Generated table #35244: 2638242 keys, 74546169 bytes >2018-07-11 03:42:08.851876 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280528851848, "cf_name": "default", "job": 2524, "event": "table_file_creation", "file_number": 35244, "file_size": 74546169, "table_properties": {"data_size": 67112193, "index_size": 689800, "filter_size": 6743186, "raw_key_size": 76215887, "raw_average_key_size": 28, "raw_value_size": 31365773, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638242, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:42:11.855558 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2524] Generated table #35245: 2638241 keys, 74547041 bytes >2018-07-11 03:42:11.855602 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280531855576, "cf_name": "default", "job": 2524, "event": "table_file_creation", "file_number": 35245, "file_size": 74547041, "table_properties": {"data_size": 67112149, "index_size": 690719, "filter_size": 6743183, "raw_key_size": 76215857, "raw_average_key_size": 28, "raw_value_size": 31365760, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:42:14.864016 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2524] Generated table #35246: 2638241 keys, 74546092 bytes >2018-07-11 03:42:14.864062 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280534864034, "cf_name": "default", "job": 2524, "event": "table_file_creation", "file_number": 35246, "file_size": 74546092, "table_properties": {"data_size": 67112093, "index_size": 689826, "filter_size": 6743183, "raw_key_size": 76215852, "raw_average_key_size": 28, "raw_value_size": 31365755, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:42:16.685253 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2524] Generated table #35247: 1519916 keys, 46965117 bytes >2018-07-11 03:42:16.685296 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280536685268, "cf_name": "default", "job": 2524, "event": "table_file_creation", "file_number": 35247, "file_size": 46965117, "table_properties": {"data_size": 42636652, "index_size": 433326, "filter_size": 3894151, "raw_key_size": 44348934, "raw_average_key_size": 29, "raw_value_size": 21522463, "raw_average_value_size": 14, "num_data_blocks": 10538, "num_entries": 1519916, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:42:16.685329 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2524] Compacted 3@2 + 1@3 files to L3 => 270604419 bytes >2018-07-11 03:42:16.708976 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:42:16.708760) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 3 45 449 3639 671 0] max score 1.01, MB/sec: 24.3 rd, 24.3 wr, level 3, files in(3, 1) out(4) MB in(213.3, 44.8) out(258.1), read-write-amplify(2.4) write-amplify(1.2) OK, records in: 9434640, records dropped: 0 > >2018-07-11 03:42:16.708987 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:42:16.708785) EVENT_LOG_v1 {"time_micros": 1531280536708774, "job": 2524, "event": "compaction_finished", "compaction_time_micros": 11133227, "output_level": 3, "num_output_files": 4, "total_output_size": 270604419, "num_input_records": 9434640, "num_output_records": 9434640, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 3, 45, 449, 3639, 671, 0]} >2018-07-11 03:42:16.709899 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280536709895, "job": 2524, "event": "table_file_deletion", "file_number": 35242} >2018-07-11 03:42:16.709925 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280536709923, "job": 2524, "event": "table_file_deletion", "file_number": 35241} >2018-07-11 03:42:16.709937 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280536709935, "job": 2524, "event": "table_file_deletion", "file_number": 35240} >2018-07-11 03:42:16.709949 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280536709947, "job": 2524, "event": "table_file_deletion", "file_number": 35211} >2018-07-11 03:42:16.710234 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2525] Compacting 1@2 + 2@3 files to L3, score 1.01 >2018-07-11 03:42:16.710251 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2524 Base level 2, inputs: [35239(71MB)], [35212(71MB) 35244(71MB)] > >2018-07-11 03:42:16.710265 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280536710255, "job": 2525, "event": "compaction_started", "files_L2": [35239], "files_L3": [35212, 35244], "score": 1.00781, "input_data_size": 223638532} >2018-07-11 03:42:19.900503 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2525] Generated table #35248: 2638241 keys, 74546614 bytes >2018-07-11 03:42:19.900549 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280539900521, "cf_name": "default", "job": 2525, "event": "table_file_creation", "file_number": 35248, "file_size": 74546614, "table_properties": {"data_size": 67112169, "index_size": 690272, "filter_size": 6743183, "raw_key_size": 76215852, "raw_average_key_size": 28, "raw_value_size": 31365755, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:42:22.946800 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2525] Generated table #35249: 2638240 keys, 74546266 bytes >2018-07-11 03:42:22.946845 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280542946818, "cf_name": "default", "job": 2525, "event": "table_file_creation", "file_number": 35249, "file_size": 74546266, "table_properties": {"data_size": 67112110, "index_size": 689985, "filter_size": 6743181, "raw_key_size": 76215825, "raw_average_key_size": 28, "raw_value_size": 31365745, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:42:25.990013 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2525] Generated table #35250: 2638242 keys, 74545961 bytes >2018-07-11 03:42:25.990056 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280545990039, "cf_name": "default", "job": 2525, "event": "table_file_creation", "file_number": 35250, "file_size": 74545961, "table_properties": {"data_size": 67112152, "index_size": 689633, "filter_size": 6743186, "raw_key_size": 76215881, "raw_average_key_size": 28, "raw_value_size": 31365767, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638242, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:42:25.990094 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2525] Compacted 1@2 + 2@3 files to L3 => 223638841 bytes >2018-07-11 03:42:26.008744 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:42:26.008477) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 3 44 450 3639 671 0] max score 1.01, MB/sec: 24.1 rd, 24.1 wr, level 3, files in(1, 2) out(3) MB in(71.1, 142.2) out(213.3), read-write-amplify(6.0) write-amplify(3.0) OK, records in: 7914723, records dropped: 0 > >2018-07-11 03:42:26.008754 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:42:26.008503) EVENT_LOG_v1 {"time_micros": 1531280546008492, "job": 2525, "event": "compaction_finished", "compaction_time_micros": 9279815, "output_level": 3, "num_output_files": 3, "total_output_size": 223638841, "num_input_records": 7914723, "num_output_records": 7914723, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 3, 44, 450, 3639, 671, 0]} >2018-07-11 03:42:26.009718 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280546009714, "job": 2525, "event": "table_file_deletion", "file_number": 35244} >2018-07-11 03:42:26.009748 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280546009746, "job": 2525, "event": "table_file_deletion", "file_number": 35239} >2018-07-11 03:42:26.009765 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280546009762, "job": 2525, "event": "table_file_deletion", "file_number": 35212} >2018-07-11 03:42:26.033662 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:42:26.010071) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #35245 to level-4 74547041 bytes > >2018-07-11 03:42:26.033672 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:42:26.033544) EVENT_LOG_v1 {"time_micros": 1531280546033536, "job": 2526, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 74547041} >2018-07-11 03:42:26.033675 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:42:26.033555) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-4 74547041 bytes OK: base level 1 max bytes base 268435456 files[0 3 44 449 3640 671 0] max score 1.01 > >2018-07-11 03:42:26.034865 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2527] Compacting 1@3 + 3@4 files to L4, score 1.01 >2018-07-11 03:42:26.034878 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2526 Base level 3, inputs: [35248(71MB)], [35183(71MB) 35184(71MB) 35185(71MB)] > >2018-07-11 03:42:26.034894 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280546034881, "job": 2527, "event": "compaction_started", "files_L3": [35248], "files_L4": [35183, 35184, 35185], "score": 1.00568, "input_data_size": 298185372} >2018-07-11 03:42:29.221082 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2527] Generated table #35251: 2638241 keys, 74546250 bytes >2018-07-11 03:42:29.221129 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280549221101, "cf_name": "default", "job": 2527, "event": "table_file_creation", "file_number": 35251, "file_size": 74546250, "table_properties": {"data_size": 67112124, "index_size": 689953, "filter_size": 6743183, "raw_key_size": 76215842, "raw_average_key_size": 28, "raw_value_size": 31365745, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:42:32.309220 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2527] Generated table #35252: 2638241 keys, 74546320 bytes >2018-07-11 03:42:32.309270 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280552309238, "cf_name": "default", "job": 2527, "event": "table_file_creation", "file_number": 35252, "file_size": 74546320, "table_properties": {"data_size": 67112186, "index_size": 689961, "filter_size": 6743183, "raw_key_size": 76215864, "raw_average_key_size": 28, "raw_value_size": 31365767, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:42:35.366894 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2527] Generated table #35253: 2638241 keys, 74545784 bytes >2018-07-11 03:42:35.366944 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280555366915, "cf_name": "default", "job": 2527, "event": "table_file_creation", "file_number": 35253, "file_size": 74545784, "table_properties": {"data_size": 67112110, "index_size": 689501, "filter_size": 6743183, "raw_key_size": 76215845, "raw_average_key_size": 28, "raw_value_size": 31365748, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:42:38.353313 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2527] Generated table #35254: 2638241 keys, 74546604 bytes >2018-07-11 03:42:38.353351 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280558353335, "cf_name": "default", "job": 2527, "event": "table_file_creation", "file_number": 35254, "file_size": 74546604, "table_properties": {"data_size": 67112159, "index_size": 690272, "filter_size": 6743183, "raw_key_size": 76215855, "raw_average_key_size": 28, "raw_value_size": 31365758, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:42:38.353390 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2527] Compacted 1@3 + 3@4 files to L4 => 298184958 bytes >2018-07-11 03:42:38.375240 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:42:38.374946) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 3 44 448 3641 671 0] max score 1.00, MB/sec: 24.2 rd, 24.2 wr, level 4, files in(1, 3) out(4) MB in(71.1, 213.3) out(284.4), read-write-amplify(8.0) write-amplify(4.0) OK, records in: 10552964, records dropped: 0 > >2018-07-11 03:42:38.375264 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:42:38.374972) EVENT_LOG_v1 {"time_micros": 1531280558374961, "job": 2527, "event": "compaction_finished", "compaction_time_micros": 12318481, "output_level": 4, "num_output_files": 4, "total_output_size": 298184958, "num_input_records": 10552964, "num_output_records": 10552964, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 3, 44, 448, 3641, 671, 0]} >2018-07-11 03:42:38.376154 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280558376149, "job": 2527, "event": "table_file_deletion", "file_number": 35248} >2018-07-11 03:42:38.376170 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280558376167, "job": 2527, "event": "table_file_deletion", "file_number": 35185} >2018-07-11 03:42:38.376182 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280558376180, "job": 2527, "event": "table_file_deletion", "file_number": 35184} >2018-07-11 03:42:38.376194 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280558376191, "job": 2527, "event": "table_file_deletion", "file_number": 35183} >2018-07-11 03:42:38.376499 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2528] Compacting 1@3 + 2@4 files to L4, score 1.00 >2018-07-11 03:42:38.376511 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2527 Base level 3, inputs: [35249(71MB)], [35213(71MB) 35210(71MB)] > >2018-07-11 03:42:38.376525 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280558376514, "job": 2528, "event": "compaction_started", "files_L3": [35249], "files_L4": [35213, 35210], "score": 1.00291, "input_data_size": 223639599} >2018-07-11 03:42:41.679976 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2528] Generated table #35255: 2638241 keys, 74546849 bytes >2018-07-11 03:42:41.680021 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280561679994, "cf_name": "default", "job": 2528, "event": "table_file_creation", "file_number": 35255, "file_size": 74546849, "table_properties": {"data_size": 67112115, "index_size": 690561, "filter_size": 6743183, "raw_key_size": 76215852, "raw_average_key_size": 28, "raw_value_size": 31365755, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:42:44.813343 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2528] Generated table #35256: 2638241 keys, 74546302 bytes >2018-07-11 03:42:44.813377 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280564813362, "cf_name": "default", "job": 2528, "event": "table_file_creation", "file_number": 35256, "file_size": 74546302, "table_properties": {"data_size": 67112153, "index_size": 689976, "filter_size": 6743183, "raw_key_size": 76215859, "raw_average_key_size": 28, "raw_value_size": 31365762, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:42:47.785666 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2528] Generated table #35257: 2638240 keys, 74545919 bytes >2018-07-11 03:42:47.785713 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280567785685, "cf_name": "default", "job": 2528, "event": "table_file_creation", "file_number": 35257, "file_size": 74545919, "table_properties": {"data_size": 67112087, "index_size": 689671, "filter_size": 6743171, "raw_key_size": 76215826, "raw_average_key_size": 28, "raw_value_size": 31365746, "raw_average_value_size": 11, "num_data_blocks": 16489, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:42:47.785746 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2528] Compacted 1@3 + 2@4 files to L4 => 223639070 bytes >2018-07-11 03:42:47.808565 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:42:47.808282) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 3 44 447 3642 671 0] max score 1.00, MB/sec: 23.8 rd, 23.8 wr, level 4, files in(1, 2) out(3) MB in(71.1, 142.2) out(213.3), read-write-amplify(6.0) write-amplify(3.0) OK, records in: 7914722, records dropped: 0 > >2018-07-11 03:42:47.808576 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:42:47.808308) EVENT_LOG_v1 {"time_micros": 1531280567808297, "job": 2528, "event": "compaction_finished", "compaction_time_micros": 9409208, "output_level": 4, "num_output_files": 3, "total_output_size": 223639070, "num_input_records": 7914722, "num_output_records": 7914722, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 3, 44, 447, 3642, 671, 0]} >2018-07-11 03:42:47.809509 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280567809505, "job": 2528, "event": "table_file_deletion", "file_number": 35249} >2018-07-11 03:42:47.809528 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280567809526, "job": 2528, "event": "table_file_deletion", "file_number": 35213} >2018-07-11 03:42:47.809541 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280567809538, "job": 2528, "event": "table_file_deletion", "file_number": 35210} >2018-07-11 03:42:47.858481 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:42:47.809844) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #35245 to level-5 74547041 bytes > >2018-07-11 03:42:47.858491 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:42:47.858285) EVENT_LOG_v1 {"time_micros": 1531280567858279, "job": 2529, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74547041} >2018-07-11 03:42:47.858494 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:42:47.858293) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74547041 bytes OK: base level 1 max bytes base 268435456 files[0 3 44 447 3641 672 0] max score 1.00 > >2018-07-11 03:42:47.883396 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:42:47.859311) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #3136 to level-5 74546868 bytes > >2018-07-11 03:42:47.883406 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:42:47.883307) EVENT_LOG_v1 {"time_micros": 1531280567883302, "job": 2530, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546868} >2018-07-11 03:42:47.883409 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:42:47.883315) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546868 bytes OK: base level 1 max bytes base 268435456 files[0 3 44 447 3640 673 0] max score 1.00 > >2018-07-11 03:42:47.900077 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:42:47.884201) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #16056 to level-5 74546868 bytes > >2018-07-11 03:42:47.900088 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:42:47.899991) EVENT_LOG_v1 {"time_micros": 1531280567899985, "job": 2531, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546868} >2018-07-11 03:42:47.900090 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:42:47.899998) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546868 bytes OK: base level 1 max bytes base 268435456 files[0 3 44 447 3639 674 0] max score 1.00 > >2018-07-11 03:42:47.925111 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:42:47.900924) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #35246 to level-4 74546092 bytes > >2018-07-11 03:42:47.925121 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:42:47.925024) EVENT_LOG_v1 {"time_micros": 1531280567925019, "job": 2532, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 74546092} >2018-07-11 03:42:47.925124 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:42:47.925032) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-4 74546092 bytes OK: base level 1 max bytes base 268435456 files[0 3 44 446 3640 674 0] max score 1.00 > >2018-07-11 03:42:47.950128 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:42:47.925956) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #19332 to level-5 74546868 bytes > >2018-07-11 03:42:47.950138 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:42:47.950041) EVENT_LOG_v1 {"time_micros": 1531280567950035, "job": 2533, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546868} >2018-07-11 03:42:47.950141 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:42:47.950049) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546868 bytes OK: base level 1 max bytes base 268435456 files[0 3 44 446 3639 675 0] max score 1.00 > >2018-07-11 03:43:10.160968 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:684] reusing log 35224 from recycle list > >2018-07-11 03:43:10.161096 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #35258. Immutable memtables: 0. > >2018-07-11 03:43:10.289127 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-03:43:10.161131) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1 >2018-07-11 03:43:10.289141 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 2534] Flushing memtable with next log file: 35258 > >2018-07-11 03:43:10.289157 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280590289149, "job": 2534, "event": "flush_started", "num_memtables": 1, "num_entries": 5000015, "num_deletes": 0, "memory_usage": 292933600} >2018-07-11 03:43:10.289160 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 2534] Level-0 flush table #35259: started >2018-07-11 03:43:15.253798 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280595253778, "cf_name": "default", "job": 2534, "event": "table_file_creation", "file_number": 35259, "file_size": 141286888, "table_properties": {"data_size": 127197776, "index_size": 1308397, "filter_size": 12779725, "raw_key_size": 144445101, "raw_average_key_size": 28, "raw_value_size": 59450671, "raw_average_value_size": 11, "num_data_blocks": 31252, "num_entries": 5000014, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:43:15.253834 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 2534] Level-0 flush table #35259: 141286888 bytes OK >2018-07-11 03:43:15.274365 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:242] adding log 35226 to recycle list > >2018-07-11 03:43:15.274376 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-03:43:15.253847) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #35259 started >2018-07-11 03:43:15.274393 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-03:43:15.274231) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #35259: memtable #1 done >2018-07-11 03:43:15.274396 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-03:43:15.274256) EVENT_LOG_v1 {"time_micros": 1531280595274240, "job": 2534, "event": "flush_finished", "lsm_state": [1, 3, 44, 446, 3639, 675, 0], "immutable_memtables": 0} >2018-07-11 03:43:15.274398 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-03:43:15.274316) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 3 44 446 3639 675 0] max score 1.00 > >2018-07-11 03:43:15.275257 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 2534] Try to delete WAL files size 178896755, prev total WAL file size 178896767, number of live WAL files 2. > >2018-07-11 03:43:55.819468 7f41821e8700 0 -- 10.8.128.107:6801/1595 >> 10.8.128.108:6801/1383 conn(0x562b9f827000 :6801 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg: challenging authorizer >2018-07-11 03:43:55.819669 7f41821e8700 0 -- 10.8.128.107:6801/1595 >> 10.8.128.108:6801/1383 conn(0x562b9f827000 :6801 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg accept connect_seq 3 vs existing csq=3 existing_state=STATE_STANDBY >2018-07-11 03:43:55.819859 7f41821e8700 0 -- 10.8.128.107:6801/1595 >> 10.8.128.108:6801/1383 conn(0x562b9f827000 :6801 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg accept connect_seq 4 vs existing csq=3 existing_state=STATE_STANDBY >2018-07-11 03:43:55.820078 7f41821e8700 0 -- 10.8.128.107:6801/1595 >> 10.8.128.108:6801/1383 conn(0x562b9f86c800 :-1 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=25 cs=3 l=0).handle_connect_msg: challenging authorizer >2018-07-11 03:44:22.189036 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:684] reusing log 35226 from recycle list > >2018-07-11 03:44:22.189207 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #35260. Immutable memtables: 0. > >2018-07-11 03:44:22.234204 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-03:44:22.189261) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1 >2018-07-11 03:44:22.234218 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 2535] Flushing memtable with next log file: 35260 > >2018-07-11 03:44:22.234242 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280662234230, "job": 2535, "event": "flush_started", "num_memtables": 1, "num_entries": 5000018, "num_deletes": 0, "memory_usage": 292942088} >2018-07-11 03:44:22.234257 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 2535] Level-0 flush table #35261: started >2018-07-11 03:44:26.235479 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280666235459, "cf_name": "default", "job": 2535, "event": "table_file_creation", "file_number": 35261, "file_size": 141289030, "table_properties": {"data_size": 127198897, "index_size": 1309411, "filter_size": 12779732, "raw_key_size": 144445177, "raw_average_key_size": 28, "raw_value_size": 59451623, "raw_average_value_size": 11, "num_data_blocks": 31252, "num_entries": 5000017, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:44:26.235495 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 2535] Level-0 flush table #35261: 141289030 bytes OK >2018-07-11 03:44:26.255971 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:242] adding log 35258 to recycle list > >2018-07-11 03:44:26.255984 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-03:44:26.235506) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #35261 started >2018-07-11 03:44:26.256000 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-03:44:26.255709) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #35261: memtable #1 done >2018-07-11 03:44:26.256002 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-03:44:26.255740) EVENT_LOG_v1 {"time_micros": 1531280666255720, "job": 2535, "event": "flush_finished", "lsm_state": [2, 3, 44, 446, 3639, 675, 0], "immutable_memtables": 0} >2018-07-11 03:44:26.256004 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-03:44:26.255908) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[2 3 44 446 3639 675 0] max score 1.05 > >2018-07-11 03:44:26.256127 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2536] Compacting 2@0 + 2@1 files to L1, score 1.05 >2018-07-11 03:44:26.256144 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2535 Base level 0, inputs: [35261(134MB) 35259(134MB)], [35228(71MB) 35233(65MB)] > >2018-07-11 03:44:26.256162 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280666256148, "job": 2536, "event": "compaction_started", "files_L0": [35261, 35259], "files_L1": [35228, 35233], "score": 1.05268, "input_data_size": 425333858} >2018-07-11 03:44:26.256862 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 2535] Try to delete WAL files size 178897793, prev total WAL file size 178897805, number of live WAL files 2. > >2018-07-11 03:44:29.268785 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2536] Generated table #35262: 2624965 keys, 74513320 bytes >2018-07-11 03:44:29.268816 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280669268801, "cf_name": "default", "job": 2536, "event": "table_file_creation", "file_number": 35262, "file_size": 74513320, "table_properties": {"data_size": 67111158, "index_size": 691165, "filter_size": 6710007, "raw_key_size": 75861278, "raw_average_key_size": 28, "raw_value_size": 31537666, "raw_average_value_size": 12, "num_data_blocks": 16475, "num_entries": 2624965, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:44:32.081442 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2536] Generated table #35263: 2638242 keys, 74546423 bytes >2018-07-11 03:44:32.081476 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280672081461, "cf_name": "default", "job": 2536, "event": "table_file_creation", "file_number": 35263, "file_size": 74546423, "table_properties": {"data_size": 67112133, "index_size": 690114, "filter_size": 6743186, "raw_key_size": 76215874, "raw_average_key_size": 28, "raw_value_size": 31365760, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638242, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:44:34.792954 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2536] Generated table #35264: 2638241 keys, 74546257 bytes >2018-07-11 03:44:34.793002 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280674792985, "cf_name": "default", "job": 2536, "event": "table_file_creation", "file_number": 35264, "file_size": 74546257, "table_properties": {"data_size": 67112146, "index_size": 689938, "filter_size": 6743183, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:44:37.498506 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2536] Generated table #35265: 2638241 keys, 74545806 bytes >2018-07-11 03:44:37.498539 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280677498524, "cf_name": "default", "job": 2536, "event": "table_file_creation", "file_number": 35265, "file_size": 74545806, "table_properties": {"data_size": 67112087, "index_size": 689546, "filter_size": 6743183, "raw_key_size": 76215853, "raw_average_key_size": 28, "raw_value_size": 31365756, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:44:39.610239 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2536] Generated table #35266: 2085356 keys, 58924120 bytes >2018-07-11 03:44:39.610286 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280679610259, "cf_name": "default", "job": 2536, "event": "table_file_creation", "file_number": 35266, "file_size": 58924120, "table_properties": {"data_size": 53047745, "index_size": 545350, "filter_size": 5330037, "raw_key_size": 60243630, "raw_average_key_size": 28, "raw_value_size": 24792578, "raw_average_value_size": 11, "num_data_blocks": 13034, "num_entries": 2085356, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:44:40.772597 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2536] Generated table #35267: 166080 keys, 68245014 bytes >2018-07-11 03:44:40.772637 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280680772621, "cf_name": "default", "job": 2536, "event": "table_file_creation", "file_number": 35267, "file_size": 68245014, "table_properties": {"data_size": 67110235, "index_size": 565016, "filter_size": 568775, "raw_key_size": 15371854, "raw_average_key_size": 92, "raw_value_size": 53116794, "raw_average_value_size": 319, "num_data_blocks": 15211, "num_entries": 166080, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "80898", "kMergeOperands": "0"}} >2018-07-11 03:44:40.806471 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2536] Generated table #35268: 4 keys, 5089 bytes >2018-07-11 03:44:40.806497 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280680806484, "cf_name": "default", "job": 2536, "event": "table_file_creation", "file_number": 35268, "file_size": 5089, "table_properties": {"data_size": 4074, "index_size": 28, "filter_size": 20, "raw_key_size": 333, "raw_average_key_size": 83, "raw_value_size": 3749, "raw_average_value_size": 937, "num_data_blocks": 1, "num_entries": 4, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:44:40.806566 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2536] Compacted 2@0 + 2@1 files to L1 => 425326029 bytes >2018-07-11 03:44:40.822538 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:44:40.822210) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 8 44 446 3639 675 0] max score 1.80, MB/sec: 29.2 rd, 29.2 wr, level 1, files in(2, 2) out(7) MB in(269.5, 136.1) out(405.6), read-write-amplify(3.0) write-amplify(1.5) OK, records in: 12791137, records dropped: 8 > >2018-07-11 03:44:40.822549 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:44:40.822237) EVENT_LOG_v1 {"time_micros": 1531280680822225, "job": 2536, "event": "compaction_finished", "compaction_time_micros": 14550386, "output_level": 1, "num_output_files": 7, "total_output_size": 425326029, "num_input_records": 12791137, "num_output_records": 12791129, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 8, 44, 446, 3639, 675, 0]} >2018-07-11 03:44:40.823438 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280680823434, "job": 2536, "event": "table_file_deletion", "file_number": 35261} >2018-07-11 03:44:40.823451 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280680823449, "job": 2536, "event": "table_file_deletion", "file_number": 35259} >2018-07-11 03:44:40.823467 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280680823465, "job": 2536, "event": "table_file_deletion", "file_number": 35233} >2018-07-11 03:44:40.823482 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280680823480, "job": 2536, "event": "table_file_deletion", "file_number": 35228} >2018-07-11 03:44:40.823748 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_picker.cc:461] [default] Expanding@1 1+5(74546423+344321946 bytes) to 4+5 (282562606+344321946bytes) > >2018-07-11 03:44:40.823848 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2537] Compacting 4@1 + 5@2 files to L2, score 1.80 >2018-07-11 03:44:40.823861 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2536 Base level 1, inputs: [35263(71MB) 35264(71MB) 35265(71MB) 35266(56MB)], [35235(71MB) 35236(71MB) 35237(71MB) 35238(71MB) 35243(44MB)] > >2018-07-11 03:44:40.823876 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280680823865, "job": 2537, "event": "compaction_started", "files_L1": [35263, 35264, 35265, 35266], "files_L2": [35235, 35236, 35237, 35238, 35243], "score": 1.79568, "input_data_size": 626884552} >2018-07-11 03:44:44.079325 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2537] Generated table #35269: 2638238 keys, 74545698 bytes >2018-07-11 03:44:44.079371 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280684079343, "cf_name": "default", "job": 2537, "event": "table_file_creation", "file_number": 35269, "file_size": 74545698, "table_properties": {"data_size": 67112104, "index_size": 689427, "filter_size": 6743177, "raw_key_size": 76215765, "raw_average_key_size": 28, "raw_value_size": 31365719, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638238, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:44:47.163640 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2537] Generated table #35270: 2638241 keys, 74546216 bytes >2018-07-11 03:44:47.163685 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280687163668, "cf_name": "default", "job": 2537, "event": "table_file_creation", "file_number": 35270, "file_size": 74546216, "table_properties": {"data_size": 67112113, "index_size": 689930, "filter_size": 6743183, "raw_key_size": 76215857, "raw_average_key_size": 28, "raw_value_size": 31365760, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:44:50.014989 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2537] Generated table #35271: 2638239 keys, 74546041 bytes >2018-07-11 03:44:50.015035 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280690015007, "cf_name": "default", "job": 2537, "event": "table_file_creation", "file_number": 35271, "file_size": 74546041, "table_properties": {"data_size": 67112151, "index_size": 689721, "filter_size": 6743179, "raw_key_size": 76215806, "raw_average_key_size": 28, "raw_value_size": 31365743, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638239, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:44:53.001754 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2537] Generated table #35272: 2638239 keys, 74545932 bytes >2018-07-11 03:44:53.001797 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280693001780, "cf_name": "default", "job": 2537, "event": "table_file_creation", "file_number": 35272, "file_size": 74545932, "table_properties": {"data_size": 67112151, "index_size": 689612, "filter_size": 6743179, "raw_key_size": 76215803, "raw_average_key_size": 28, "raw_value_size": 31365740, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638239, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:44:56.066845 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2537] Generated table #35273: 2638241 keys, 74546248 bytes >2018-07-11 03:44:56.066892 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280696066863, "cf_name": "default", "job": 2537, "event": "table_file_creation", "file_number": 35273, "file_size": 74546248, "table_properties": {"data_size": 67112154, "index_size": 689921, "filter_size": 6743183, "raw_key_size": 76215849, "raw_average_key_size": 28, "raw_value_size": 31365752, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:44:59.201408 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2537] Generated table #35274: 2638241 keys, 74546470 bytes >2018-07-11 03:44:59.201451 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280699201433, "cf_name": "default", "job": 2537, "event": "table_file_creation", "file_number": 35274, "file_size": 74546470, "table_properties": {"data_size": 67112123, "index_size": 690174, "filter_size": 6743183, "raw_key_size": 76215861, "raw_average_key_size": 28, "raw_value_size": 31365764, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:45:02.557400 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2537] Generated table #35275: 2638241 keys, 74545920 bytes >2018-07-11 03:45:02.557444 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280702557418, "cf_name": "default", "job": 2537, "event": "table_file_creation", "file_number": 35275, "file_size": 74545920, "table_properties": {"data_size": 67112160, "index_size": 689587, "filter_size": 6743183, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:45:05.713182 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2537] Generated table #35276: 2638241 keys, 74546932 bytes >2018-07-11 03:45:05.713241 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280705713223, "cf_name": "default", "job": 2537, "event": "table_file_creation", "file_number": 35276, "file_size": 74546932, "table_properties": {"data_size": 67112148, "index_size": 690611, "filter_size": 6743183, "raw_key_size": 76215852, "raw_average_key_size": 28, "raw_value_size": 31365755, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:45:07.163666 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2537] Generated table #35277: 1075501 keys, 30515653 bytes >2018-07-11 03:45:07.163696 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280707163681, "cf_name": "default", "job": 2537, "event": "table_file_creation", "file_number": 35277, "file_size": 30515653, "table_properties": {"data_size": 27483317, "index_size": 282133, "filter_size": 2749215, "raw_key_size": 31085784, "raw_average_key_size": 28, "raw_value_size": 12891827, "raw_average_value_size": 11, "num_data_blocks": 6755, "num_entries": 1075501, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "13", "kMergeOperands": "0"}} >2018-07-11 03:45:07.163730 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2537] Compacted 4@1 + 5@2 files to L2 => 626885110 bytes >2018-07-11 03:45:07.180557 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:45:07.180155) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 4 48 446 3639 675 0] max score 1.09, MB/sec: 23.8 rd, 23.8 wr, level 2, files in(4, 5) out(9) MB in(269.5, 328.4) out(597.8), read-write-amplify(4.4) write-amplify(2.2) OK, records in: 22181422, records dropped: 0 > >2018-07-11 03:45:07.180568 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:45:07.180180) EVENT_LOG_v1 {"time_micros": 1531280707180168, "job": 2537, "event": "compaction_finished", "compaction_time_micros": 26339835, "output_level": 2, "num_output_files": 9, "total_output_size": 626885110, "num_input_records": 22181422, "num_output_records": 22181422, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 4, 48, 446, 3639, 675, 0]} >2018-07-11 03:45:07.181458 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280707181454, "job": 2537, "event": "table_file_deletion", "file_number": 35266} >2018-07-11 03:45:07.181473 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280707181471, "job": 2537, "event": "table_file_deletion", "file_number": 35265} >2018-07-11 03:45:07.181487 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280707181485, "job": 2537, "event": "table_file_deletion", "file_number": 35264} >2018-07-11 03:45:07.181501 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280707181498, "job": 2537, "event": "table_file_deletion", "file_number": 35263} >2018-07-11 03:45:07.181519 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280707181517, "job": 2537, "event": "table_file_deletion", "file_number": 35243} >2018-07-11 03:45:07.181532 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280707181530, "job": 2537, "event": "table_file_deletion", "file_number": 35238} >2018-07-11 03:45:07.181545 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280707181543, "job": 2537, "event": "table_file_deletion", "file_number": 35237} >2018-07-11 03:45:07.181559 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280707181556, "job": 2537, "event": "table_file_deletion", "file_number": 35236} >2018-07-11 03:45:07.181573 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280707181571, "job": 2537, "event": "table_file_deletion", "file_number": 35235} >2018-07-11 03:45:07.181772 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_picker.cc:461] [default] Expanding@2 1+1(74546932+46965117 bytes) to 3+1 (223639322+46965117bytes) > >2018-07-11 03:45:07.181867 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2538] Compacting 3@2 + 1@3 files to L3, score 1.09 >2018-07-11 03:45:07.181878 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2537 Base level 2, inputs: [35274(71MB) 35275(71MB) 35276(71MB)], [35247(44MB)] > >2018-07-11 03:45:07.181891 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280707181881, "job": 2538, "event": "compaction_started", "files_L2": [35274, 35275, 35276], "files_L3": [35247], "score": 1.0853, "input_data_size": 270604439} >2018-07-11 03:45:10.320333 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2538] Generated table #35278: 2638241 keys, 74546081 bytes >2018-07-11 03:45:10.320367 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280710320352, "cf_name": "default", "job": 2538, "event": "table_file_creation", "file_number": 35278, "file_size": 74546081, "table_properties": {"data_size": 67112162, "index_size": 689746, "filter_size": 6743183, "raw_key_size": 76215856, "raw_average_key_size": 28, "raw_value_size": 31365759, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:45:13.696191 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2538] Generated table #35279: 2638241 keys, 74546683 bytes >2018-07-11 03:45:13.696252 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280713696229, "cf_name": "default", "job": 2538, "event": "table_file_creation", "file_number": 35279, "file_size": 74546683, "table_properties": {"data_size": 67112150, "index_size": 690360, "filter_size": 6743183, "raw_key_size": 76215859, "raw_average_key_size": 28, "raw_value_size": 31365762, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:45:16.798568 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2538] Generated table #35280: 2638241 keys, 74546741 bytes >2018-07-11 03:45:16.798610 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280716798585, "cf_name": "default", "job": 2538, "event": "table_file_creation", "file_number": 35280, "file_size": 74546741, "table_properties": {"data_size": 67112148, "index_size": 690420, "filter_size": 6743183, "raw_key_size": 76215851, "raw_average_key_size": 28, "raw_value_size": 31365754, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:45:18.647884 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2538] Generated table #35281: 1519916 keys, 46965136 bytes >2018-07-11 03:45:18.647926 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280718647909, "cf_name": "default", "job": 2538, "event": "table_file_creation", "file_number": 35281, "file_size": 46965136, "table_properties": {"data_size": 42636614, "index_size": 433383, "filter_size": 3894151, "raw_key_size": 44348935, "raw_average_key_size": 29, "raw_value_size": 21522464, "raw_average_value_size": 14, "num_data_blocks": 10538, "num_entries": 1519916, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:45:18.647964 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2538] Compacted 3@2 + 1@3 files to L3 => 270604641 bytes >2018-07-11 03:45:18.663536 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:45:18.663230) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 4 45 449 3639 675 0] max score 1.01, MB/sec: 23.6 rd, 23.6 wr, level 3, files in(3, 1) out(4) MB in(213.3, 44.8) out(258.1), read-write-amplify(2.4) write-amplify(1.2) OK, records in: 9434639, records dropped: 0 > >2018-07-11 03:45:18.663547 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:45:18.663263) EVENT_LOG_v1 {"time_micros": 1531280718663244, "job": 2538, "event": "compaction_finished", "compaction_time_micros": 11466057, "output_level": 3, "num_output_files": 4, "total_output_size": 270604641, "num_input_records": 9434639, "num_output_records": 9434639, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 4, 45, 449, 3639, 675, 0]} >2018-07-11 03:45:18.664452 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280718664448, "job": 2538, "event": "table_file_deletion", "file_number": 35276} >2018-07-11 03:45:18.664469 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280718664466, "job": 2538, "event": "table_file_deletion", "file_number": 35275} >2018-07-11 03:45:18.664480 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280718664478, "job": 2538, "event": "table_file_deletion", "file_number": 35274} >2018-07-11 03:45:18.664495 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280718664493, "job": 2538, "event": "table_file_deletion", "file_number": 35247} >2018-07-11 03:45:18.679988 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:45:18.664784) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #35280 to level-4 74546741 bytes > >2018-07-11 03:45:18.679999 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:45:18.679899) EVENT_LOG_v1 {"time_micros": 1531280718679893, "job": 2539, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 74546741} >2018-07-11 03:45:18.680001 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:45:18.679907) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-4 74546741 bytes OK: base level 1 max bytes base 268435456 files[0 4 45 448 3640 675 0] max score 1.00 > >2018-07-11 03:45:18.696647 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:45:18.680818) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #35279 to level-4 74546683 bytes > >2018-07-11 03:45:18.696655 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:45:18.696561) EVENT_LOG_v1 {"time_micros": 1531280718696556, "job": 2540, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 74546683} >2018-07-11 03:45:18.696658 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:45:18.696568) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-4 74546683 bytes OK: base level 1 max bytes base 268435456 files[0 4 45 447 3641 675 0] max score 1.00 > >2018-07-11 03:45:18.697443 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2541] Compacting 1@2 + 2@3 files to L3, score 1.00 >2018-07-11 03:45:18.697455 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2540 Base level 2, inputs: [35273(71MB)], [35250(71MB) 35278(71MB)] > >2018-07-11 03:45:18.697468 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280718697458, "job": 2541, "event": "compaction_started", "files_L2": [35273], "files_L3": [35250, 35278], "score": 1.00199, "input_data_size": 223638290} >2018-07-11 03:45:19.043316 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:684] reusing log 35258 from recycle list > >2018-07-11 03:45:19.043433 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #35283. Immutable memtables: 0. > >2018-07-11 03:45:19.118910 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-03:45:19.043490) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 0 >2018-07-11 03:45:19.118924 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 2542] Flushing memtable with next log file: 35283 > >2018-07-11 03:45:19.118941 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280719118932, "job": 2542, "event": "flush_started", "num_memtables": 1, "num_entries": 5000018, "num_deletes": 0, "memory_usage": 292933168} >2018-07-11 03:45:19.118944 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 2542] Level-0 flush table #35284: started >2018-07-11 03:45:21.850716 7f41649ec700 1 bluefs _allocate failed to allocate 0x8700000 on bdev 1, free 0x5afe000; fallback to bdev 2 >2018-07-11 03:45:21.854454 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2541] Generated table #35282: 2638242 keys, 74545962 bytes >2018-07-11 03:45:21.854492 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280721854477, "cf_name": "default", "job": 2541, "event": "table_file_creation", "file_number": 35282, "file_size": 74545962, "table_properties": {"data_size": 67112148, "index_size": 689638, "filter_size": 6743186, "raw_key_size": 76215879, "raw_average_key_size": 28, "raw_value_size": 31365765, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638242, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:45:23.810367 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280723810346, "cf_name": "default", "job": 2542, "event": "table_file_creation", "file_number": 35284, "file_size": 141289031, "table_properties": {"data_size": 127198898, "index_size": 1309411, "filter_size": 12779732, "raw_key_size": 144445177, "raw_average_key_size": 28, "raw_value_size": 59451623, "raw_average_value_size": 11, "num_data_blocks": 31252, "num_entries": 5000017, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:45:23.810389 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 2542] Level-0 flush table #35284: 141289031 bytes OK >2018-07-11 03:45:23.821665 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:242] adding log 35260 to recycle list > >2018-07-11 03:45:23.821675 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-03:45:23.810413) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #35284 started >2018-07-11 03:45:23.821678 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-03:45:23.821548) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #35284: memtable #1 done >2018-07-11 03:45:23.821680 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-03:45:23.821563) EVENT_LOG_v1 {"time_micros": 1531280723821557, "job": 2542, "event": "flush_finished", "lsm_state": [1, 4, 45, 447, 3641, 675, 0], "immutable_memtables": 0} >2018-07-11 03:45:23.821682 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-03:45:23.821581) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 4 45 447 3641 675 0] max score 1.00 > >2018-07-11 03:45:23.822752 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 2542] Try to delete WAL files size 178897793, prev total WAL file size 178897805, number of live WAL files 2. > >2018-07-11 03:45:26.805806 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2541] Generated table #35285: 2638241 keys, 74546305 bytes >2018-07-11 03:45:26.805855 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280726805824, "cf_name": "default", "job": 2541, "event": "table_file_creation", "file_number": 35285, "file_size": 74546305, "table_properties": {"data_size": 67112178, "index_size": 689954, "filter_size": 6743183, "raw_key_size": 76215857, "raw_average_key_size": 28, "raw_value_size": 31365760, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:45:29.704346 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2541] Generated table #35286: 2638241 keys, 74545724 bytes >2018-07-11 03:45:29.704391 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280729704363, "cf_name": "default", "job": 2541, "event": "table_file_creation", "file_number": 35286, "file_size": 74545724, "table_properties": {"data_size": 67112132, "index_size": 689419, "filter_size": 6743183, "raw_key_size": 76215850, "raw_average_key_size": 28, "raw_value_size": 31365753, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:45:29.704428 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2541] Compacted 1@2 + 2@3 files to L3 => 223637991 bytes >2018-07-11 03:45:29.721832 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:45:29.721581) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[1 4 44 448 3641 675 0] max score 1.00, MB/sec: 20.3 rd, 20.3 wr, level 3, files in(1, 2) out(3) MB in(71.1, 142.2) out(213.3), read-write-amplify(6.0) write-amplify(3.0) OK, records in: 7914724, records dropped: 0 > >2018-07-11 03:45:29.721841 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:45:29.721603) EVENT_LOG_v1 {"time_micros": 1531280729721593, "job": 2541, "event": "compaction_finished", "compaction_time_micros": 11006947, "output_level": 3, "num_output_files": 3, "total_output_size": 223637991, "num_input_records": 7914724, "num_output_records": 7914724, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 4, 44, 448, 3641, 675, 0]} >2018-07-11 03:45:29.722758 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280729722755, "job": 2541, "event": "table_file_deletion", "file_number": 35278} >2018-07-11 03:45:29.722786 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280729722783, "job": 2541, "event": "table_file_deletion", "file_number": 35273} >2018-07-11 03:45:29.722800 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280729722797, "job": 2541, "event": "table_file_deletion", "file_number": 35250} >2018-07-11 03:45:29.723087 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2543] Compacting 1@3 + 1@4 files to L4, score 1.00 >2018-07-11 03:45:29.723110 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2542 Base level 3, inputs: [35285(71MB)], [35246(71MB)] > >2018-07-11 03:45:29.723123 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280729723113, "job": 2543, "event": "compaction_started", "files_L3": [35285], "files_L4": [35246], "score": 1.00291, "input_data_size": 149092397} >2018-07-11 03:45:32.999537 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2543] Generated table #35287: 2638241 keys, 74546054 bytes >2018-07-11 03:45:32.999583 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280732999556, "cf_name": "default", "job": 2543, "event": "table_file_creation", "file_number": 35287, "file_size": 74546054, "table_properties": {"data_size": 67112095, "index_size": 689786, "filter_size": 6743183, "raw_key_size": 76215855, "raw_average_key_size": 28, "raw_value_size": 31365758, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:45:35.987990 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2543] Generated table #35288: 2638241 keys, 74546302 bytes >2018-07-11 03:45:35.988035 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280735988008, "cf_name": "default", "job": 2543, "event": "table_file_creation", "file_number": 35288, "file_size": 74546302, "table_properties": {"data_size": 67112175, "index_size": 689954, "filter_size": 6743183, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:45:35.988070 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2543] Compacted 1@3 + 1@4 files to L4 => 149092356 bytes >2018-07-11 03:45:36.004986 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:45:36.004823) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[1 4 44 447 3642 675 0] max score 1.00, MB/sec: 23.8 rd, 23.8 wr, level 4, files in(1, 1) out(2) MB in(71.1, 71.1) out(142.2), read-write-amplify(4.0) write-amplify(2.0) OK, records in: 5276482, records dropped: 0 > >2018-07-11 03:45:36.004995 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:45:36.004845) EVENT_LOG_v1 {"time_micros": 1531280736004835, "job": 2543, "event": "compaction_finished", "compaction_time_micros": 6264936, "output_level": 4, "num_output_files": 2, "total_output_size": 149092356, "num_input_records": 5276482, "num_output_records": 5276482, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 4, 44, 447, 3642, 675, 0]} >2018-07-11 03:45:36.005885 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280736005881, "job": 2543, "event": "table_file_deletion", "file_number": 35285} >2018-07-11 03:45:36.005914 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280736005912, "job": 2543, "event": "table_file_deletion", "file_number": 35246} >2018-07-11 03:45:36.021770 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:45:36.006201) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #23693 to level-5 74546867 bytes > >2018-07-11 03:45:36.021780 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:45:36.021621) EVENT_LOG_v1 {"time_micros": 1531280736021614, "job": 2544, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546867} >2018-07-11 03:45:36.021782 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:45:36.021629) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546867 bytes OK: base level 1 max bytes base 268435456 files[1 4 44 447 3641 676 0] max score 1.00 > >2018-07-11 03:45:36.038316 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:45:36.022649) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #15251 to level-5 74546867 bytes > >2018-07-11 03:45:36.038326 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:45:36.038219) EVENT_LOG_v1 {"time_micros": 1531280736038213, "job": 2545, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546867} >2018-07-11 03:45:36.038329 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:45:36.038226) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546867 bytes OK: base level 1 max bytes base 268435456 files[1 4 44 447 3640 677 0] max score 1.00 > >2018-07-11 03:45:36.054975 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:45:36.039125) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #10578 to level-5 74546867 bytes > >2018-07-11 03:45:36.054984 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:45:36.054893) EVENT_LOG_v1 {"time_micros": 1531280736054887, "job": 2546, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546867} >2018-07-11 03:45:36.054987 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:45:36.054900) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546867 bytes OK: base level 1 max bytes base 268435456 files[1 4 44 447 3639 678 0] max score 1.00 > >2018-07-11 03:45:36.055719 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_picker.cc:461] [default] Expanding@3 1+6(74546052+447278702 bytes) to 2+6 (74547222+447278702bytes) > >2018-07-11 03:45:36.055824 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2547] Compacting 2@3 + 6@4 files to L4, score 1.00 >2018-07-11 03:45:36.055837 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2546 Base level 3, inputs: [34955(71MB) 34956(1170B)], [34835(71MB) 34836(71MB) 34831(71MB) 34832(71MB) 34833(71MB) 34903(71MB)] > >2018-07-11 03:45:36.055852 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280736055841, "job": 2547, "event": "compaction_started", "files_L3": [34955, 34956], "files_L4": [34835, 34836, 34831, 34832, 34833, 34903], "score": 1.00013, "input_data_size": 521825924} >2018-07-11 03:45:39.365034 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2547] Generated table #35289: 2638241 keys, 74546161 bytes >2018-07-11 03:45:39.365068 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280739365053, "cf_name": "default", "job": 2547, "event": "table_file_creation", "file_number": 35289, "file_size": 74546161, "table_properties": {"data_size": 67112132, "index_size": 689856, "filter_size": 6743183, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:45:42.526544 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2547] Generated table #35290: 2638241 keys, 74545938 bytes >2018-07-11 03:45:42.526589 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280742526561, "cf_name": "default", "job": 2547, "event": "table_file_creation", "file_number": 35290, "file_size": 74545938, "table_properties": {"data_size": 67112150, "index_size": 689615, "filter_size": 6743183, "raw_key_size": 76215850, "raw_average_key_size": 28, "raw_value_size": 31365753, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:45:45.630907 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2547] Generated table #35291: 2638241 keys, 74547041 bytes >2018-07-11 03:45:45.630953 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280745630925, "cf_name": "default", "job": 2547, "event": "table_file_creation", "file_number": 35291, "file_size": 74547041, "table_properties": {"data_size": 67112105, "index_size": 690763, "filter_size": 6743183, "raw_key_size": 76215861, "raw_average_key_size": 28, "raw_value_size": 31365764, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:45:48.708379 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2547] Generated table #35292: 2638241 keys, 74546364 bytes >2018-07-11 03:45:48.708413 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280748708398, "cf_name": "default", "job": 2547, "event": "table_file_creation", "file_number": 35292, "file_size": 74546364, "table_properties": {"data_size": 67112159, "index_size": 690032, "filter_size": 6743183, "raw_key_size": 76215851, "raw_average_key_size": 28, "raw_value_size": 31365754, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:45:51.803251 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2547] Generated table #35293: 2638241 keys, 74546447 bytes >2018-07-11 03:45:51.803294 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280751803267, "cf_name": "default", "job": 2547, "event": "table_file_creation", "file_number": 35293, "file_size": 74546447, "table_properties": {"data_size": 67112115, "index_size": 690159, "filter_size": 6743183, "raw_key_size": 76215849, "raw_average_key_size": 28, "raw_value_size": 31365752, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:45:55.107257 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2547] Generated table #35294: 2638241 keys, 74545890 bytes >2018-07-11 03:45:55.107299 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280755107274, "cf_name": "default", "job": 2547, "event": "table_file_creation", "file_number": 35294, "file_size": 74545890, "table_properties": {"data_size": 67112160, "index_size": 689557, "filter_size": 6743183, "raw_key_size": 76215855, "raw_average_key_size": 28, "raw_value_size": 31365758, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:45:58.409642 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2547] Generated table #35295: 2638241 keys, 74546702 bytes >2018-07-11 03:45:58.409686 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280758409660, "cf_name": "default", "job": 2547, "event": "table_file_creation", "file_number": 35295, "file_size": 74546702, "table_properties": {"data_size": 67112149, "index_size": 690380, "filter_size": 6743183, "raw_key_size": 76215859, "raw_average_key_size": 28, "raw_value_size": 31365762, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:45:58.455461 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2547] Generated table #35296: 4 keys, 1141 bytes >2018-07-11 03:45:58.455486 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280758455474, "cf_name": "default", "job": 2547, "event": "table_file_creation", "file_number": 35296, "file_size": 1141, "table_properties": {"data_size": 129, "index_size": 27, "filter_size": 20, "raw_key_size": 116, "raw_average_key_size": 29, "raw_value_size": 48, "raw_average_value_size": 12, "num_data_blocks": 1, "num_entries": 4, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:45:58.455516 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2547] Compacted 2@3 + 6@4 files to L4 => 521825684 bytes >2018-07-11 03:45:58.471368 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:45:58.471113) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[1 4 44 445 3641 678 0] max score 1.00, MB/sec: 23.3 rd, 23.3 wr, level 4, files in(2, 6) out(8) MB in(71.1, 426.6) out(497.7), read-write-amplify(14.0) write-amplify(7.0) OK, records in: 18467691, records dropped: 0 > >2018-07-11 03:45:58.471378 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:45:58.471137) EVENT_LOG_v1 {"time_micros": 1531280758471127, "job": 2547, "event": "compaction_finished", "compaction_time_micros": 22399647, "output_level": 4, "num_output_files": 8, "total_output_size": 521825684, "num_input_records": 18467691, "num_output_records": 18467691, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 4, 44, 445, 3641, 678, 0]} >2018-07-11 03:45:58.472287 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280758472283, "job": 2547, "event": "table_file_deletion", "file_number": 34956} >2018-07-11 03:45:58.472301 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280758472298, "job": 2547, "event": "table_file_deletion", "file_number": 34955} >2018-07-11 03:45:58.472314 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280758472312, "job": 2547, "event": "table_file_deletion", "file_number": 34903} >2018-07-11 03:45:58.472331 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280758472328, "job": 2547, "event": "table_file_deletion", "file_number": 34836} >2018-07-11 03:45:58.472342 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280758472340, "job": 2547, "event": "table_file_deletion", "file_number": 34835} >2018-07-11 03:45:58.472354 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280758472352, "job": 2547, "event": "table_file_deletion", "file_number": 34833} >2018-07-11 03:45:58.472368 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280758472365, "job": 2547, "event": "table_file_deletion", "file_number": 34832} >2018-07-11 03:45:58.472385 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280758472382, "job": 2547, "event": "table_file_deletion", "file_number": 34831} >2018-07-11 03:45:58.488058 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:45:58.472684) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #35291 to level-5 74547041 bytes > >2018-07-11 03:45:58.488069 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:45:58.487878) EVENT_LOG_v1 {"time_micros": 1531280758487871, "job": 2548, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74547041} >2018-07-11 03:45:58.488071 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:45:58.487887) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74547041 bytes OK: base level 1 max bytes base 268435456 files[1 4 44 445 3640 679 0] max score 1.00 > >2018-07-11 03:48:14.634270 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:684] reusing log 35260 from recycle list > >2018-07-11 03:48:14.634429 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #35297. Immutable memtables: 0. > >2018-07-11 03:48:14.686207 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-03:48:14.634464) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1 >2018-07-11 03:48:14.686221 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 2549] Flushing memtable with next log file: 35297 > >2018-07-11 03:48:14.686239 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280894686230, "job": 2549, "event": "flush_started", "num_memtables": 1, "num_entries": 5000015, "num_deletes": 0, "memory_usage": 292950032} >2018-07-11 03:48:14.686243 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 2549] Level-0 flush table #35298: started >2018-07-11 03:48:17.424134 7f41649ec700 1 bluefs _allocate failed to allocate 0x8700000 on bdev 1, free 0x5afe000; fallback to bdev 2 >2018-07-11 03:48:19.251109 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280899251087, "cf_name": "default", "job": 2549, "event": "table_file_creation", "file_number": 35298, "file_size": 141286889, "table_properties": {"data_size": 127197777, "index_size": 1308397, "filter_size": 12779725, "raw_key_size": 144445101, "raw_average_key_size": 28, "raw_value_size": 59450671, "raw_average_value_size": 11, "num_data_blocks": 31252, "num_entries": 5000014, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:48:19.251134 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 2549] Level-0 flush table #35298: 141286889 bytes OK >2018-07-11 03:48:19.267415 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:242] adding log 35283 to recycle list > >2018-07-11 03:48:19.267429 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-03:48:19.251148) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #35298 started >2018-07-11 03:48:19.267433 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-03:48:19.267276) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #35298: memtable #1 done >2018-07-11 03:48:19.267435 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-03:48:19.267292) EVENT_LOG_v1 {"time_micros": 1531280899267285, "job": 2549, "event": "flush_finished", "lsm_state": [2, 4, 44, 445, 3640, 679, 0], "immutable_memtables": 0} >2018-07-11 03:48:19.267437 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-03:48:19.267362) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[2 4 44 445 3640 679 0] max score 1.05 > >2018-07-11 03:48:19.267582 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2550] Compacting 2@0 + 2@1 files to L1, score 1.05 >2018-07-11 03:48:19.267599 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2549 Base level 0, inputs: [35298(134MB) 35284(134MB)], [35262(71MB) 35267(65MB)] > >2018-07-11 03:48:19.267617 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280899267603, "job": 2550, "event": "compaction_started", "files_L0": [35298, 35284], "files_L1": [35262, 35267], "score": 1.05268, "input_data_size": 425334254} >2018-07-11 03:48:19.268289 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 2549] Try to delete WAL files size 178896755, prev total WAL file size 178896767, number of live WAL files 2. > >2018-07-11 03:48:22.454641 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2550] Generated table #35299: 2624858 keys, 74513080 bytes >2018-07-11 03:48:22.454687 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280902454671, "cf_name": "default", "job": 2550, "event": "table_file_creation", "file_number": 35299, "file_size": 74513080, "table_properties": {"data_size": 67111147, "index_size": 691205, "filter_size": 6709738, "raw_key_size": 75858387, "raw_average_key_size": 28, "raw_value_size": 31538811, "raw_average_value_size": 12, "num_data_blocks": 16475, "num_entries": 2624858, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:48:24.470817 7f41641eb700 1 bluefs _allocate failed to allocate 0x4800000 on bdev 1, free 0x12fe000; fallback to bdev 2 >2018-07-11 03:48:25.422277 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2550] Generated table #35300: 2638241 keys, 74546868 bytes >2018-07-11 03:48:25.422328 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280905422299, "cf_name": "default", "job": 2550, "event": "table_file_creation", "file_number": 35300, "file_size": 74546868, "table_properties": {"data_size": 67112119, "index_size": 690576, "filter_size": 6743183, "raw_key_size": 76215845, "raw_average_key_size": 28, "raw_value_size": 31365748, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:48:27.494843 7f41641eb700 1 bluefs _allocate failed to allocate 0x4800000 on bdev 1, free 0x12fe000; fallback to bdev 2 >2018-07-11 03:48:28.518448 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2550] Generated table #35301: 2638241 keys, 74546503 bytes >2018-07-11 03:48:28.518493 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280908518466, "cf_name": "default", "job": 2550, "event": "table_file_creation", "file_number": 35301, "file_size": 74546503, "table_properties": {"data_size": 67112153, "index_size": 690177, "filter_size": 6743183, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:48:30.580053 7f41641eb700 1 bluefs _allocate failed to allocate 0x4800000 on bdev 1, free 0x12fe000; fallback to bdev 2 >2018-07-11 03:48:31.501221 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2550] Generated table #35302: 2638241 keys, 74546405 bytes >2018-07-11 03:48:31.501271 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280911501238, "cf_name": "default", "job": 2550, "event": "table_file_creation", "file_number": 35302, "file_size": 74546405, "table_properties": {"data_size": 67112087, "index_size": 690145, "filter_size": 6743183, "raw_key_size": 76215853, "raw_average_key_size": 28, "raw_value_size": 31365756, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:48:33.267499 7f41641eb700 1 bluefs _allocate failed to allocate 0x3900000 on bdev 1, free 0x12fe000; fallback to bdev 2 >2018-07-11 03:48:34.010967 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2550] Generated table #35303: 2085397 keys, 58924958 bytes >2018-07-11 03:48:34.011011 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280914010985, "cf_name": "default", "job": 2550, "event": "table_file_creation", "file_number": 35303, "file_size": 58924958, "table_properties": {"data_size": 53048796, "index_size": 545034, "filter_size": 5330140, "raw_key_size": 60244815, "raw_average_key_size": 28, "raw_value_size": 24793066, "raw_average_value_size": 11, "num_data_blocks": 13034, "num_entries": 2085397, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:48:35.156016 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2550] Generated table #35304: 166086 keys, 68245155 bytes >2018-07-11 03:48:35.156058 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280915156043, "cf_name": "default", "job": 2550, "event": "table_file_creation", "file_number": 35304, "file_size": 68245155, "table_properties": {"data_size": 67110354, "index_size": 565023, "filter_size": 568790, "raw_key_size": 15372132, "raw_average_key_size": 92, "raw_value_size": 53116676, "raw_average_value_size": 319, "num_data_blocks": 15211, "num_entries": 166086, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "80898", "kMergeOperands": "0"}} >2018-07-11 03:48:35.193059 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2550] Generated table #35305: 4 keys, 5087 bytes >2018-07-11 03:48:35.193097 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280915193072, "cf_name": "default", "job": 2550, "event": "table_file_creation", "file_number": 35305, "file_size": 5087, "table_properties": {"data_size": 4072, "index_size": 28, "filter_size": 20, "raw_key_size": 332, "raw_average_key_size": 83, "raw_value_size": 3748, "raw_average_value_size": 937, "num_data_blocks": 1, "num_entries": 4, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:48:35.193160 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2550] Compacted 2@0 + 2@1 files to L1 => 425328056 bytes >2018-07-11 03:48:35.209027 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:48:35.208799) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 9 44 445 3640 679 0] max score 1.80, MB/sec: 26.7 rd, 26.7 wr, level 1, files in(2, 2) out(7) MB in(269.5, 136.1) out(405.6), read-write-amplify(3.0) write-amplify(1.5) OK, records in: 12791076, records dropped: 8 > >2018-07-11 03:48:35.209039 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:48:35.208825) EVENT_LOG_v1 {"time_micros": 1531280915208814, "job": 2550, "event": "compaction_finished", "compaction_time_micros": 15925524, "output_level": 1, "num_output_files": 7, "total_output_size": 425328056, "num_input_records": 12791076, "num_output_records": 12791068, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 9, 44, 445, 3640, 679, 0]} >2018-07-11 03:48:35.209942 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280915209938, "job": 2550, "event": "table_file_deletion", "file_number": 35298} >2018-07-11 03:48:35.209958 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280915209955, "job": 2550, "event": "table_file_deletion", "file_number": 35284} >2018-07-11 03:48:35.209973 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280915209971, "job": 2550, "event": "table_file_deletion", "file_number": 35267} >2018-07-11 03:48:35.209986 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280915209984, "job": 2550, "event": "table_file_deletion", "file_number": 35262} >2018-07-11 03:48:35.210211 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_picker.cc:461] [default] Expanding@1 1+5(74546868+328699540 bytes) to 4+5 (282564734+328699540bytes) > >2018-07-11 03:48:35.210317 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2551] Compacting 4@1 + 5@2 files to L2, score 1.80 >2018-07-11 03:48:35.210330 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2550 Base level 1, inputs: [35300(71MB) 35301(71MB) 35302(71MB) 35303(56MB)], [35269(71MB) 35270(71MB) 35271(71MB) 35272(71MB) 35277(29MB)] > >2018-07-11 03:48:35.210344 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280915210334, "job": 2551, "event": "compaction_started", "files_L1": [35300, 35301, 35302, 35303], "files_L2": [35269, 35270, 35271, 35272, 35277], "score": 1.7957, "input_data_size": 611264274} >2018-07-11 03:48:38.492789 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2551] Generated table #35306: 2638240 keys, 74546432 bytes >2018-07-11 03:48:38.492834 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280918492806, "cf_name": "default", "job": 2551, "event": "table_file_creation", "file_number": 35306, "file_size": 74546432, "table_properties": {"data_size": 67112120, "index_size": 690140, "filter_size": 6743182, "raw_key_size": 76215822, "raw_average_key_size": 28, "raw_value_size": 31365742, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:48:41.691555 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2551] Generated table #35307: 2638240 keys, 74546993 bytes >2018-07-11 03:48:41.691598 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280921691572, "cf_name": "default", "job": 2551, "event": "table_file_creation", "file_number": 35307, "file_size": 74546993, "table_properties": {"data_size": 67112098, "index_size": 690724, "filter_size": 6743181, "raw_key_size": 76215829, "raw_average_key_size": 28, "raw_value_size": 31365749, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:48:44.933770 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2551] Generated table #35308: 2638238 keys, 74545845 bytes >2018-07-11 03:48:44.933815 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280924933789, "cf_name": "default", "job": 2551, "event": "table_file_creation", "file_number": 35308, "file_size": 74545845, "table_properties": {"data_size": 67112071, "index_size": 689607, "filter_size": 6743177, "raw_key_size": 76215777, "raw_average_key_size": 28, "raw_value_size": 31365731, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638238, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:48:48.148238 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2551] Generated table #35309: 2638240 keys, 74547092 bytes >2018-07-11 03:48:48.148287 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280928148260, "cf_name": "default", "job": 2551, "event": "table_file_creation", "file_number": 35309, "file_size": 74547092, "table_properties": {"data_size": 67112164, "index_size": 690757, "filter_size": 6743181, "raw_key_size": 76215832, "raw_average_key_size": 28, "raw_value_size": 31365752, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:48:51.154578 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2551] Generated table #35310: 2638241 keys, 74546347 bytes >2018-07-11 03:48:51.154622 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280931154594, "cf_name": "default", "job": 2551, "event": "table_file_creation", "file_number": 35310, "file_size": 74546347, "table_properties": {"data_size": 67112142, "index_size": 690031, "filter_size": 6743184, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:48:54.380283 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2551] Generated table #35311: 2638241 keys, 74546107 bytes >2018-07-11 03:48:54.380340 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280934380322, "cf_name": "default", "job": 2551, "event": "table_file_creation", "file_number": 35311, "file_size": 74546107, "table_properties": {"data_size": 67112145, "index_size": 689789, "filter_size": 6743183, "raw_key_size": 76215851, "raw_average_key_size": 28, "raw_value_size": 31365754, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:48:57.694008 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2551] Generated table #35312: 2638241 keys, 74546224 bytes >2018-07-11 03:48:57.694055 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280937694039, "cf_name": "default", "job": 2551, "event": "table_file_creation", "file_number": 35312, "file_size": 74546224, "table_properties": {"data_size": 67112102, "index_size": 689949, "filter_size": 6743183, "raw_key_size": 76215859, "raw_average_key_size": 28, "raw_value_size": 31365762, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:49:01.033135 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2551] Generated table #35313: 2638241 keys, 74547006 bytes >2018-07-11 03:49:01.033185 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280941033157, "cf_name": "default", "job": 2551, "event": "table_file_creation", "file_number": 35313, "file_size": 74547006, "table_properties": {"data_size": 67112175, "index_size": 690658, "filter_size": 6743183, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:49:01.676329 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2551] Generated table #35314: 522656 keys, 14894414 bytes >2018-07-11 03:49:01.676379 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280941676363, "cf_name": "default", "job": 2551, "event": "table_file_creation", "file_number": 35314, "file_size": 14894414, "table_properties": {"data_size": 13419927, "index_size": 137332, "filter_size": 1336169, "raw_key_size": 15114704, "raw_average_key_size": 28, "raw_value_size": 6319112, "raw_average_value_size": 12, "num_data_blocks": 3299, "num_entries": 522656, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "13", "kMergeOperands": "0"}} >2018-07-11 03:49:01.676415 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2551] Compacted 4@1 + 5@2 files to L2 => 611266460 bytes >2018-07-11 03:49:01.691878 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:49:01.691595) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 5 48 445 3640 679 0] max score 1.08, MB/sec: 23.1 rd, 23.1 wr, level 2, files in(4, 5) out(9) MB in(269.5, 313.5) out(582.9), read-write-amplify(4.3) write-amplify(2.2) OK, records in: 21628578, records dropped: 0 > >2018-07-11 03:49:01.691890 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:49:01.691620) EVENT_LOG_v1 {"time_micros": 1531280941691609, "job": 2551, "event": "compaction_finished", "compaction_time_micros": 26466051, "output_level": 2, "num_output_files": 9, "total_output_size": 611266460, "num_input_records": 21628578, "num_output_records": 21628578, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 5, 48, 445, 3640, 679, 0]} >2018-07-11 03:49:01.692817 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280941692812, "job": 2551, "event": "table_file_deletion", "file_number": 35303} >2018-07-11 03:49:01.692840 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280941692836, "job": 2551, "event": "table_file_deletion", "file_number": 35302} >2018-07-11 03:49:01.692864 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280941692860, "job": 2551, "event": "table_file_deletion", "file_number": 35301} >2018-07-11 03:49:01.692887 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280941692884, "job": 2551, "event": "table_file_deletion", "file_number": 35300} >2018-07-11 03:49:01.692915 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280941692911, "job": 2551, "event": "table_file_deletion", "file_number": 35277} >2018-07-11 03:49:01.692942 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280941692938, "job": 2551, "event": "table_file_deletion", "file_number": 35272} >2018-07-11 03:49:01.692968 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280941692964, "job": 2551, "event": "table_file_deletion", "file_number": 35271} >2018-07-11 03:49:01.692994 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280941692990, "job": 2551, "event": "table_file_deletion", "file_number": 35270} >2018-07-11 03:49:01.693020 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280941693016, "job": 2551, "event": "table_file_deletion", "file_number": 35269} >2018-07-11 03:49:01.693565 7f41641eb700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS ------- >2018-07-11 03:49:01.693572 7f41641eb700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] >** DB Stats ** >Uptime(secs): 45683.3 total, 620.0 interval >Cumulative writes: 196K writes, 1678M keys, 196K commit groups, 1.0 writes per commit group, ingest: 56.43 GB, 1.26 MB/s >Cumulative WAL: 196K writes, 56K syncs, 3.49 writes per sync, written: 56.43 GB, 1.26 MB/s >Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent >Interval writes: 62 writes, 29M keys, 62 commit groups, 1.0 writes per commit group, ingest: 989.54 MB, 1.60 MB/s >Interval WAL: 62 writes, 31 syncs, 1.94 writes per sync, written: 0.97 MB, 1.60 MB/s >Interval stall: 00:00:0.000 H:M:S, 0.0 percent > >** Compaction Stats [default] ** >Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop >---------------------------------------------------------------------------------------------------------------------------------------------------------- > L0 0/0 0.00 KB 0.0 0.0 0.0 0.0 44.1 44.1 0.0 1.0 0.0 32.4 1396 337 4.142 0 0 > L1 5/0 190.23 MB 0.7 73.3 44.2 29.0 73.2 44.2 0.0 1.7 25.8 25.8 2911 166 17.534 2316M 894K > L2 48/0 2.70 GB 1.1 84.6 44.2 40.4 84.6 44.2 0.0 1.9 21.7 21.7 3993 533 7.491 3151M 1358K > L3 445/0 24.93 GB 1.0 77.2 43.3 33.9 77.1 43.3 0.6 1.8 20.8 20.8 3795 261 14.540 2917M 4000K > L4 3640/0 250.00 GB 1.0 66.9 16.0 50.9 66.6 15.7 27.9 4.2 21.3 21.2 3213 215 14.944 2547M 15M > L5 679/0 47.12 GB 0.0 1.4 0.7 0.7 1.4 0.7 42.8 2.0 19.0 18.7 75 10 7.486 53M 1097K > Sum 4817/0 324.94 GB 0.0 303.4 148.5 154.9 347.0 192.1 71.4 7.9 20.2 23.1 15382 1522 10.107 10G 23M > Int 0/0 0.00 KB 0.0 6.0 2.8 3.2 6.8 3.5 1.3 8.6 21.2 23.9 289 24 12.043 217M 23 >Uptime(secs): 45683.3 total, 45683.3 interval >Flush(GB): cumulative 44.129, interval 0.790 >AddFile(GB): cumulative 0.000, interval 0.000 >AddFile(Total Files): cumulative 0, interval 0 >AddFile(L0 Files): cumulative 0, interval 0 >AddFile(Keys): cumulative 0, interval 0 >Cumulative compaction: 347.02 GB write, 7.78 MB/s write, 303.42 GB read, 6.80 MB/s read, 15382.4 seconds >Interval compaction: 6.76 GB write, 0.15 MB/s write, 5.97 GB read, 0.13 MB/s read, 289.0 seconds >Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count > >** File Read Latency Histogram By Level [default] ** > >** Compaction Stats [default] ** >Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop >---------------------------------------------------------------------------------------------------------------------------------------------------------- > L0 0/0 0.00 KB 0.0 0.0 0.0 0.0 44.1 44.1 0.0 1.0 0.0 32.4 1396 337 4.142 0 0 > L1 5/0 190.23 MB 0.7 73.3 44.2 29.0 73.2 44.2 0.0 1.7 25.8 25.8 2911 166 17.534 2316M 894K > L2 48/0 2.70 GB 1.1 84.6 44.2 40.4 84.6 44.2 0.0 1.9 21.7 21.7 3993 533 7.491 3151M 1358K > L3 445/0 24.93 GB 1.0 77.2 43.3 33.9 77.1 43.3 0.6 1.8 20.8 20.8 3795 261 14.540 2917M 4000K > L4 3640/0 250.00 GB 1.0 66.9 16.0 50.9 66.6 15.7 27.9 4.2 21.3 21.2 3213 215 14.944 2547M 15M > L5 679/0 47.12 GB 0.0 1.4 0.7 0.7 1.4 0.7 42.8 2.0 19.0 18.7 75 10 7.486 53M 1097K > Sum 4817/0 324.94 GB 0.0 303.4 148.5 154.9 347.0 192.1 71.4 7.9 20.2 23.1 15382 1522 10.107 10G 23M > Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 >Uptime(secs): 45683.3 total, 45683.3 interval >Flush(GB): cumulative 44.129, interval 0.000 >AddFile(GB): cumulative 0.000, interval 0.000 >AddFile(Total Files): cumulative 0, interval 0 >AddFile(L0 Files): cumulative 0, interval 0 >AddFile(Keys): cumulative 0, interval 0 >Cumulative compaction: 347.02 GB write, 7.78 MB/s write, 303.42 GB read, 6.80 MB/s read, 15382.4 seconds >Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds >Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count > >** File Read Latency Histogram By Level [default] ** > >2018-07-11 03:49:01.693695 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2552] Compacting 1@2 + 2@3 files to L3, score 1.08 >2018-07-11 03:49:01.693708 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2551 Base level 2, inputs: [35309(71MB)], [35181(71MB) 35182(1064B)] > >2018-07-11 03:49:01.693728 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280941693713, "job": 2552, "event": "compaction_started", "files_L2": [35309], "files_L3": [35181, 35182], "score": 1.07949, "input_data_size": 149094111} >2018-07-11 03:49:04.953916 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2552] Generated table #35315: 2638239 keys, 74545772 bytes >2018-07-11 03:49:04.953957 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280944953940, "cf_name": "default", "job": 2552, "event": "table_file_creation", "file_number": 35315, "file_size": 74545772, "table_properties": {"data_size": 67112107, "index_size": 689496, "filter_size": 6743179, "raw_key_size": 76215801, "raw_average_key_size": 28, "raw_value_size": 31365738, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638239, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:49:08.063148 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2552] Generated table #35316: 2638240 keys, 74546277 bytes >2018-07-11 03:49:08.063203 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280948063187, "cf_name": "default", "job": 2552, "event": "table_file_creation", "file_number": 35316, "file_size": 74546277, "table_properties": {"data_size": 67112117, "index_size": 689989, "filter_size": 6743181, "raw_key_size": 76215828, "raw_average_key_size": 28, "raw_value_size": 31365748, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:49:08.084129 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2552] Generated table #35317: 3 keys, 1113 bytes >2018-07-11 03:49:08.084168 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280948084155, "cf_name": "default", "job": 2552, "event": "table_file_creation", "file_number": 35317, "file_size": 1113, "table_properties": {"data_size": 105, "index_size": 27, "filter_size": 18, "raw_key_size": 87, "raw_average_key_size": 29, "raw_value_size": 36, "raw_average_value_size": 12, "num_data_blocks": 1, "num_entries": 3, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:49:08.084193 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2552] Compacted 1@2 + 2@3 files to L3 => 149093162 bytes >2018-07-11 03:49:08.100056 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:49:08.099853) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 5 47 446 3640 679 0] max score 1.05, MB/sec: 23.3 rd, 23.3 wr, level 3, files in(1, 2) out(3) MB in(71.1, 71.1) out(142.2), read-write-amplify(4.0) write-amplify(2.0) OK, records in: 5276482, records dropped: 0 > >2018-07-11 03:49:08.100068 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:49:08.099877) EVENT_LOG_v1 {"time_micros": 1531280948099866, "job": 2552, "event": "compaction_finished", "compaction_time_micros": 6390450, "output_level": 3, "num_output_files": 3, "total_output_size": 149093162, "num_input_records": 5276482, "num_output_records": 5276482, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 5, 47, 446, 3640, 679, 0]} >2018-07-11 03:49:08.100994 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280948100977, "job": 2552, "event": "table_file_deletion", "file_number": 35309} >2018-07-11 03:49:08.101011 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280948101008, "job": 2552, "event": "table_file_deletion", "file_number": 35182} >2018-07-11 03:49:08.101022 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280948101020, "job": 2552, "event": "table_file_deletion", "file_number": 35181} >2018-07-11 03:49:08.101237 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_picker.cc:461] [default] Expanding@2 1+1(74547006+46965136 bytes) to 3+1 (223639337+46965136bytes) > >2018-07-11 03:49:08.101341 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2553] Compacting 3@2 + 1@3 files to L3, score 1.05 >2018-07-11 03:49:08.101352 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2552 Base level 2, inputs: [35311(71MB) 35312(71MB) 35313(71MB)], [35281(44MB)] > >2018-07-11 03:49:08.101366 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280948101355, "job": 2553, "event": "compaction_started", "files_L2": [35311, 35312, 35313], "files_L3": [35281], "score": 1.05171, "input_data_size": 270604473} >2018-07-11 03:49:11.424087 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2553] Generated table #35318: 2638241 keys, 74546226 bytes >2018-07-11 03:49:11.424139 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280951424110, "cf_name": "default", "job": 2553, "event": "table_file_creation", "file_number": 35318, "file_size": 74546226, "table_properties": {"data_size": 67112135, "index_size": 689918, "filter_size": 6743183, "raw_key_size": 76215852, "raw_average_key_size": 28, "raw_value_size": 31365755, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:49:14.545473 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2553] Generated table #35319: 2638241 keys, 74546986 bytes >2018-07-11 03:49:14.545521 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280954545505, "cf_name": "default", "job": 2553, "event": "table_file_creation", "file_number": 35319, "file_size": 74546986, "table_properties": {"data_size": 67112107, "index_size": 690706, "filter_size": 6743183, "raw_key_size": 76215859, "raw_average_key_size": 28, "raw_value_size": 31365762, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:49:17.599988 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2553] Generated table #35320: 2638241 keys, 74546251 bytes >2018-07-11 03:49:17.600031 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280957600005, "cf_name": "default", "job": 2553, "event": "table_file_creation", "file_number": 35320, "file_size": 74546251, "table_properties": {"data_size": 67112162, "index_size": 689916, "filter_size": 6743183, "raw_key_size": 76215857, "raw_average_key_size": 28, "raw_value_size": 31365760, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:49:19.491577 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2553] Generated table #35321: 1519916 keys, 46965193 bytes >2018-07-11 03:49:19.491618 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280959491598, "cf_name": "default", "job": 2553, "event": "table_file_creation", "file_number": 35321, "file_size": 46965193, "table_properties": {"data_size": 42636621, "index_size": 433433, "filter_size": 3894151, "raw_key_size": 44348931, "raw_average_key_size": 29, "raw_value_size": 21522460, "raw_average_value_size": 14, "num_data_blocks": 10538, "num_entries": 1519916, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:49:19.491660 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2553] Compacted 3@2 + 1@3 files to L3 => 270604656 bytes >2018-07-11 03:49:19.508225 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:49:19.508011) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 5 44 449 3640 679 0] max score 1.01, MB/sec: 23.8 rd, 23.8 wr, level 3, files in(3, 1) out(4) MB in(213.3, 44.8) out(258.1), read-write-amplify(2.4) write-amplify(1.2) OK, records in: 9434639, records dropped: 0 > >2018-07-11 03:49:19.508237 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:49:19.508036) EVENT_LOG_v1 {"time_micros": 1531280959508025, "job": 2553, "event": "compaction_finished", "compaction_time_micros": 11390274, "output_level": 3, "num_output_files": 4, "total_output_size": 270604656, "num_input_records": 9434639, "num_output_records": 9434639, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 5, 44, 449, 3640, 679, 0]} >2018-07-11 03:49:19.509173 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280959509168, "job": 2553, "event": "table_file_deletion", "file_number": 35313} >2018-07-11 03:49:19.509200 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280959509196, "job": 2553, "event": "table_file_deletion", "file_number": 35312} >2018-07-11 03:49:19.509228 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280959509224, "job": 2553, "event": "table_file_deletion", "file_number": 35311} >2018-07-11 03:49:19.509261 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280959509257, "job": 2553, "event": "table_file_deletion", "file_number": 35281} >2018-07-11 03:49:19.524763 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:49:19.509547) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #35319 to level-4 74546986 bytes > >2018-07-11 03:49:19.524773 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:49:19.524673) EVENT_LOG_v1 {"time_micros": 1531280959524667, "job": 2554, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 74546986} >2018-07-11 03:49:19.524778 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:49:19.524680) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-4 74546986 bytes OK: base level 1 max bytes base 268435456 files[0 5 44 448 3641 679 0] max score 1.01 > >2018-07-11 03:49:19.525564 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_picker.cc:461] [default] Expanding@3 1+3(74546277+223638622 bytes) to 2+3 (74547390+223638622bytes) > >2018-07-11 03:49:19.525670 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2555] Compacting 2@3 + 3@4 files to L4, score 1.01 >2018-07-11 03:49:19.525685 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2554 Base level 3, inputs: [35316(71MB) 35317(1113B)], [35155(71MB) 35157(71MB) 35251(71MB)] > >2018-07-11 03:49:19.525706 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280959525690, "job": 2555, "event": "compaction_started", "files_L3": [35316, 35317], "files_L4": [35155, 35157, 35251], "score": 1.00568, "input_data_size": 298186012} >2018-07-11 03:49:22.968119 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2555] Generated table #35322: 2638241 keys, 74546609 bytes >2018-07-11 03:49:22.968172 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280962968156, "cf_name": "default", "job": 2555, "event": "table_file_creation", "file_number": 35322, "file_size": 74546609, "table_properties": {"data_size": 67112141, "index_size": 690295, "filter_size": 6743183, "raw_key_size": 76215850, "raw_average_key_size": 28, "raw_value_size": 31365753, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:49:26.357665 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2555] Generated table #35323: 2638240 keys, 74546039 bytes >2018-07-11 03:49:26.357712 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280966357692, "cf_name": "default", "job": 2555, "event": "table_file_creation", "file_number": 35323, "file_size": 74546039, "table_properties": {"data_size": 67112120, "index_size": 689748, "filter_size": 6743181, "raw_key_size": 76215828, "raw_average_key_size": 28, "raw_value_size": 31365748, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:49:29.744932 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2555] Generated table #35324: 2638241 keys, 74546253 bytes >2018-07-11 03:49:29.744986 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280969744969, "cf_name": "default", "job": 2555, "event": "table_file_creation", "file_number": 35324, "file_size": 74546253, "table_properties": {"data_size": 67112143, "index_size": 689937, "filter_size": 6743183, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:49:32.799070 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2555] Generated table #35325: 2638242 keys, 74546410 bytes >2018-07-11 03:49:32.799105 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280972799090, "cf_name": "default", "job": 2555, "event": "table_file_creation", "file_number": 35325, "file_size": 74546410, "table_properties": {"data_size": 67112182, "index_size": 690052, "filter_size": 6743186, "raw_key_size": 76215877, "raw_average_key_size": 28, "raw_value_size": 31365763, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638242, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:49:32.850433 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2555] Generated table #35326: 2 keys, 1088 bytes >2018-07-11 03:49:32.850465 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280972850448, "cf_name": "default", "job": 2555, "event": "table_file_creation", "file_number": 35326, "file_size": 1088, "table_properties": {"data_size": 81, "index_size": 27, "filter_size": 18, "raw_key_size": 58, "raw_average_key_size": 29, "raw_value_size": 24, "raw_average_value_size": 12, "num_data_blocks": 1, "num_entries": 2, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:49:32.850499 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2555] Compacted 2@3 + 3@4 files to L4 => 298186399 bytes >2018-07-11 03:49:32.866389 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:49:32.866155) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 5 44 446 3643 679 0] max score 1.00, MB/sec: 22.4 rd, 22.4 wr, level 4, files in(2, 3) out(5) MB in(71.1, 213.3) out(284.4), read-write-amplify(8.0) write-amplify(4.0) OK, records in: 10552966, records dropped: 0 > >2018-07-11 03:49:32.866401 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:49:32.866179) EVENT_LOG_v1 {"time_micros": 1531280972866169, "job": 2555, "event": "compaction_finished", "compaction_time_micros": 13324772, "output_level": 4, "num_output_files": 5, "total_output_size": 298186399, "num_input_records": 10552966, "num_output_records": 10552966, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 5, 44, 446, 3643, 679, 0]} >2018-07-11 03:49:32.867307 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280972867303, "job": 2555, "event": "table_file_deletion", "file_number": 35317} >2018-07-11 03:49:32.867323 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280972867321, "job": 2555, "event": "table_file_deletion", "file_number": 35316} >2018-07-11 03:49:32.867339 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280972867336, "job": 2555, "event": "table_file_deletion", "file_number": 35251} >2018-07-11 03:49:32.867356 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280972867353, "job": 2555, "event": "table_file_deletion", "file_number": 35157} >2018-07-11 03:49:32.867371 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531280972867368, "job": 2555, "event": "table_file_deletion", "file_number": 35155} >2018-07-11 03:49:32.882939 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:49:32.867671) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #35320 to level-4 74546251 bytes > >2018-07-11 03:49:32.882950 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:49:32.882846) EVENT_LOG_v1 {"time_micros": 1531280972882840, "job": 2556, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 74546251} >2018-07-11 03:49:32.882953 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:49:32.882854) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-4 74546251 bytes OK: base level 1 max bytes base 268435456 files[0 5 44 445 3644 679 0] max score 1.00 > >2018-07-11 03:49:32.916328 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:49:32.883777) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #35319 to level-5 74546986 bytes > >2018-07-11 03:49:32.916339 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:49:32.916210) EVENT_LOG_v1 {"time_micros": 1531280972916204, "job": 2557, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546986} >2018-07-11 03:49:32.916344 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:49:32.916218) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546986 bytes OK: base level 1 max bytes base 268435456 files[0 5 44 445 3643 680 0] max score 1.00 > >2018-07-11 03:49:32.932991 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:49:32.917160) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #11882 to level-5 74546867 bytes > >2018-07-11 03:49:32.933002 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:49:32.932901) EVENT_LOG_v1 {"time_micros": 1531280972932895, "job": 2558, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546867} >2018-07-11 03:49:32.933007 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:49:32.932909) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546867 bytes OK: base level 1 max bytes base 268435456 files[0 5 44 445 3642 681 0] max score 1.00 > >2018-07-11 03:49:32.949792 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:49:32.933868) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #4561 to level-5 74546866 bytes > >2018-07-11 03:49:32.949802 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:49:32.949705) EVENT_LOG_v1 {"time_micros": 1531280972949699, "job": 2559, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546866} >2018-07-11 03:49:32.949807 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:49:32.949713) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546866 bytes OK: base level 1 max bytes base 268435456 files[0 5 44 445 3641 682 0] max score 1.00 > >2018-07-11 03:49:32.966377 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:49:32.950619) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #35318 to level-4 74546226 bytes > >2018-07-11 03:49:32.966388 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:49:32.966287) EVENT_LOG_v1 {"time_micros": 1531280972966281, "job": 2560, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 74546226} >2018-07-11 03:49:32.966393 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:49:32.966295) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-4 74546226 bytes OK: base level 1 max bytes base 268435456 files[0 5 44 444 3642 682 0] max score 1.00 > >2018-07-11 03:49:32.983063 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:49:32.967219) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #3658 to level-5 74546866 bytes > >2018-07-11 03:49:32.983074 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:49:32.982971) EVENT_LOG_v1 {"time_micros": 1531280972982966, "job": 2561, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546866} >2018-07-11 03:49:32.983079 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:49:32.982980) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546866 bytes OK: base level 1 max bytes base 268435456 files[0 5 44 444 3641 683 0] max score 1.00 > >2018-07-11 03:50:22.597085 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:684] reusing log 35283 from recycle list > >2018-07-11 03:50:22.597359 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #35327. Immutable memtables: 0. > >2018-07-11 03:50:22.685489 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-03:50:22.597397) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1 >2018-07-11 03:50:22.685502 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 2562] Flushing memtable with next log file: 35327 > >2018-07-11 03:50:22.685524 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281022685513, "job": 2562, "event": "flush_started", "num_memtables": 1, "num_entries": 5000015, "num_deletes": 0, "memory_usage": 292938256} >2018-07-11 03:50:22.685529 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 2562] Level-0 flush table #35328: started >2018-07-11 03:50:26.575259 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281026575235, "cf_name": "default", "job": 2562, "event": "table_file_creation", "file_number": 35328, "file_size": 141286221, "table_properties": {"data_size": 127196936, "index_size": 1308576, "filter_size": 12779719, "raw_key_size": 144445077, "raw_average_key_size": 28, "raw_value_size": 59449772, "raw_average_value_size": 11, "num_data_blocks": 31252, "num_entries": 5000013, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:50:26.575290 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 2562] Level-0 flush table #35328: 141286221 bytes OK >2018-07-11 03:50:26.590023 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:242] adding log 35297 to recycle list > >2018-07-11 03:50:26.590034 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-03:50:26.575303) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #35328 started >2018-07-11 03:50:26.590038 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-03:50:26.589893) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #35328: memtable #1 done >2018-07-11 03:50:26.590044 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-03:50:26.589909) EVENT_LOG_v1 {"time_micros": 1531281026589902, "job": 2562, "event": "flush_finished", "lsm_state": [1, 5, 44, 444, 3641, 683, 0], "immutable_memtables": 0} >2018-07-11 03:50:26.590046 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-03:50:26.589970) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 5 44 444 3641 683 0] max score 1.00 > >2018-07-11 03:50:26.590904 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 2562] Try to delete WAL files size 178896755, prev total WAL file size 178896767, number of live WAL files 2. > >2018-07-11 03:50:53.149028 7f416b9fa700 0 log_channel(cluster) log [DBG] : 10.7 scrub starts >2018-07-11 03:50:53.836990 7f416b9fa700 0 log_channel(cluster) log [DBG] : 10.7 scrub ok >2018-07-11 03:50:55.138985 7f416a9f8700 0 log_channel(cluster) log [DBG] : 5.9 deep-scrub starts >2018-07-11 03:51:38.709794 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:684] reusing log 35297 from recycle list > >2018-07-11 03:51:38.709912 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #35329. Immutable memtables: 0. > >2018-07-11 03:51:38.755372 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-03:51:38.709971) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1 >2018-07-11 03:51:38.755387 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 2563] Flushing memtable with next log file: 35329 > >2018-07-11 03:51:38.755405 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281098755396, "job": 2563, "event": "flush_started", "num_memtables": 1, "num_entries": 5000019, "num_deletes": 0, "memory_usage": 292942472} >2018-07-11 03:51:38.755409 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 2563] Level-0 flush table #35330: started >2018-07-11 03:51:42.622653 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281102622633, "cf_name": "default", "job": 2563, "event": "table_file_creation", "file_number": 35330, "file_size": 141288024, "table_properties": {"data_size": 127198964, "index_size": 1308325, "filter_size": 12779745, "raw_key_size": 144445255, "raw_average_key_size": 28, "raw_value_size": 59451636, "raw_average_value_size": 11, "num_data_blocks": 31253, "num_entries": 5000018, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:51:42.622671 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 2563] Level-0 flush table #35330: 141288024 bytes OK >2018-07-11 03:51:42.638322 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:242] adding log 35327 to recycle list > >2018-07-11 03:51:42.638334 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-03:51:42.622695) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #35330 started >2018-07-11 03:51:42.638338 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-03:51:42.638060) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #35330: memtable #1 done >2018-07-11 03:51:42.638340 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-03:51:42.638078) EVENT_LOG_v1 {"time_micros": 1531281102638071, "job": 2563, "event": "flush_finished", "lsm_state": [2, 5, 44, 444, 3641, 683, 0], "immutable_memtables": 0} >2018-07-11 03:51:42.638342 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-03:51:42.638267) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[2 5 44 444 3641 683 0] max score 1.05 > >2018-07-11 03:51:42.638467 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2564] Compacting 2@0 + 2@1 files to L1, score 1.05 >2018-07-11 03:51:42.638484 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2563 Base level 0, inputs: [35330(134MB) 35328(134MB)], [35299(71MB) 35304(65MB)] > >2018-07-11 03:51:42.638508 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281102638490, "job": 2564, "event": "compaction_started", "files_L0": [35330, 35328], "files_L1": [35299, 35304], "score": 1.05267, "input_data_size": 425332480} >2018-07-11 03:51:42.639178 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 2563] Try to delete WAL files size 178897927, prev total WAL file size 178897939, number of live WAL files 2. > >2018-07-11 03:51:45.578762 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2564] Generated table #35331: 2624750 keys, 74512614 bytes >2018-07-11 03:51:45.578805 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281105578778, "cf_name": "default", "job": 2564, "event": "table_file_creation", "file_number": 35331, "file_size": 74512614, "table_properties": {"data_size": 67110829, "index_size": 691329, "filter_size": 6709466, "raw_key_size": 75855468, "raw_average_key_size": 28, "raw_value_size": 31539906, "raw_average_value_size": 12, "num_data_blocks": 16475, "num_entries": 2624750, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:51:48.198122 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2564] Generated table #35332: 2638241 keys, 74546869 bytes >2018-07-11 03:51:48.198165 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281108198138, "cf_name": "default", "job": 2564, "event": "table_file_creation", "file_number": 35332, "file_size": 74546869, "table_properties": {"data_size": 67112120, "index_size": 690576, "filter_size": 6743183, "raw_key_size": 76215846, "raw_average_key_size": 28, "raw_value_size": 31365749, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:51:51.155098 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2564] Generated table #35333: 2638241 keys, 74546503 bytes >2018-07-11 03:51:51.155130 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281111155115, "cf_name": "default", "job": 2564, "event": "table_file_creation", "file_number": 35333, "file_size": 74546503, "table_properties": {"data_size": 67112153, "index_size": 690177, "filter_size": 6743183, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:51:53.781875 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2564] Generated table #35334: 2638241 keys, 74546405 bytes >2018-07-11 03:51:53.781919 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281113781893, "cf_name": "default", "job": 2564, "event": "table_file_creation", "file_number": 35334, "file_size": 74546405, "table_properties": {"data_size": 67112087, "index_size": 690145, "filter_size": 6743183, "raw_key_size": 76215853, "raw_average_key_size": 28, "raw_value_size": 31365756, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:51:55.950365 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2564] Generated table #35335: 2085399 keys, 58925126 bytes >2018-07-11 03:51:55.950408 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281115950381, "cf_name": "default", "job": 2564, "event": "table_file_creation", "file_number": 35335, "file_size": 58925126, "table_properties": {"data_size": 53048959, "index_size": 545034, "filter_size": 5330145, "raw_key_size": 60244920, "raw_average_key_size": 28, "raw_value_size": 24793128, "raw_average_value_size": 11, "num_data_blocks": 13034, "num_entries": 2085399, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:51:57.100636 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2564] Generated table #35336: 166092 keys, 68245274 bytes >2018-07-11 03:51:57.100665 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281117100651, "cf_name": "default", "job": 2564, "event": "table_file_creation", "file_number": 35336, "file_size": 68245274, "table_properties": {"data_size": 67110458, "index_size": 565023, "filter_size": 568805, "raw_key_size": 15372406, "raw_average_key_size": 92, "raw_value_size": 53116554, "raw_average_value_size": 319, "num_data_blocks": 15211, "num_entries": 166092, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "80898", "kMergeOperands": "0"}} >2018-07-11 03:51:57.138647 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2564] Generated table #35337: 4 keys, 5095 bytes >2018-07-11 03:51:57.138673 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281117138661, "cf_name": "default", "job": 2564, "event": "table_file_creation", "file_number": 35337, "file_size": 5095, "table_properties": {"data_size": 4080, "index_size": 28, "filter_size": 20, "raw_key_size": 336, "raw_average_key_size": 84, "raw_value_size": 3752, "raw_average_value_size": 938, "num_data_blocks": 1, "num_entries": 4, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:51:57.138729 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2564] Compacted 2@0 + 2@1 files to L1 => 425327886 bytes >2018-07-11 03:51:57.154671 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:51:57.154392) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 10 44 444 3641 683 0] max score 1.80, MB/sec: 29.3 rd, 29.3 wr, level 1, files in(2, 2) out(7) MB in(269.5, 136.1) out(405.6), read-write-amplify(3.0) write-amplify(1.5) OK, records in: 12790975, records dropped: 7 > >2018-07-11 03:51:57.154683 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:51:57.154417) EVENT_LOG_v1 {"time_micros": 1531281117154406, "job": 2564, "event": "compaction_finished", "compaction_time_micros": 14500201, "output_level": 1, "num_output_files": 7, "total_output_size": 425327886, "num_input_records": 12790975, "num_output_records": 12790968, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 10, 44, 444, 3641, 683, 0]} >2018-07-11 03:51:57.155583 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281117155579, "job": 2564, "event": "table_file_deletion", "file_number": 35330} >2018-07-11 03:51:57.155598 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281117155596, "job": 2564, "event": "table_file_deletion", "file_number": 35328} >2018-07-11 03:51:57.155612 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281117155610, "job": 2564, "event": "table_file_deletion", "file_number": 35304} >2018-07-11 03:51:57.155627 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281117155625, "job": 2564, "event": "table_file_deletion", "file_number": 35299} >2018-07-11 03:51:57.155851 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_picker.cc:461] [default] Expanding@1 1+5(74546869+313080031 bytes) to 4+5 (282564903+313080031bytes) > >2018-07-11 03:51:57.155948 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2565] Compacting 4@1 + 5@2 files to L2, score 1.80 >2018-07-11 03:51:57.155960 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2564 Base level 1, inputs: [35332(71MB) 35333(71MB) 35334(71MB) 35335(56MB)], [35306(71MB) 35307(71MB) 35308(71MB) 35310(71MB) 35314(14MB)] > >2018-07-11 03:51:57.155974 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281117155964, "job": 2565, "event": "compaction_started", "files_L1": [35332, 35333, 35334, 35335], "files_L2": [35306, 35307, 35308, 35310, 35314], "score": 1.79572, "input_data_size": 595644934} >2018-07-11 03:52:01.725145 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2565] Generated table #35338: 2638238 keys, 74545728 bytes >2018-07-11 03:52:01.725177 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281121725162, "cf_name": "default", "job": 2565, "event": "table_file_creation", "file_number": 35338, "file_size": 74545728, "table_properties": {"data_size": 67112104, "index_size": 689457, "filter_size": 6743177, "raw_key_size": 76215765, "raw_average_key_size": 28, "raw_value_size": 31365719, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638238, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:52:06.821583 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2565] Generated table #35339: 2638240 keys, 74546147 bytes >2018-07-11 03:52:06.821628 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281126821601, "cf_name": "default", "job": 2565, "event": "table_file_creation", "file_number": 35339, "file_size": 74546147, "table_properties": {"data_size": 67112156, "index_size": 689820, "filter_size": 6743181, "raw_key_size": 76215829, "raw_average_key_size": 28, "raw_value_size": 31365749, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:52:12.059777 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2565] Generated table #35340: 2638239 keys, 74546334 bytes >2018-07-11 03:52:12.059820 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281132059795, "cf_name": "default", "job": 2565, "event": "table_file_creation", "file_number": 35340, "file_size": 74546334, "table_properties": {"data_size": 67112087, "index_size": 690078, "filter_size": 6743179, "raw_key_size": 76215806, "raw_average_key_size": 28, "raw_value_size": 31365743, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638239, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:52:17.115804 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2565] Generated table #35341: 2638239 keys, 74545869 bytes >2018-07-11 03:52:17.115847 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281137115820, "cf_name": "default", "job": 2565, "event": "table_file_creation", "file_number": 35341, "file_size": 74545869, "table_properties": {"data_size": 67112127, "index_size": 689573, "filter_size": 6743179, "raw_key_size": 76215796, "raw_average_key_size": 28, "raw_value_size": 31365733, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638239, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:52:20.841516 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2565] Generated table #35342: 2638240 keys, 74546238 bytes >2018-07-11 03:52:20.841561 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281140841534, "cf_name": "default", "job": 2565, "event": "table_file_creation", "file_number": 35342, "file_size": 74546238, "table_properties": {"data_size": 67112142, "index_size": 689925, "filter_size": 6743181, "raw_key_size": 76215819, "raw_average_key_size": 28, "raw_value_size": 31365739, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:52:23.868281 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2565] Generated table #35343: 2638241 keys, 74545976 bytes >2018-07-11 03:52:23.868326 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281143868311, "cf_name": "default", "job": 2565, "event": "table_file_creation", "file_number": 35343, "file_size": 74545976, "table_properties": {"data_size": 67112110, "index_size": 689693, "filter_size": 6743183, "raw_key_size": 76215864, "raw_average_key_size": 28, "raw_value_size": 31365767, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:52:27.267931 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2565] Generated table #35344: 2638241 keys, 74546189 bytes >2018-07-11 03:52:27.267972 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281147267946, "cf_name": "default", "job": 2565, "event": "table_file_creation", "file_number": 35344, "file_size": 74546189, "table_properties": {"data_size": 67112152, "index_size": 689864, "filter_size": 6743183, "raw_key_size": 76215847, "raw_average_key_size": 28, "raw_value_size": 31365750, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:52:30.786883 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2565] Generated table #35345: 2608057 keys, 73819966 bytes >2018-07-11 03:52:30.786915 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281150786900, "cf_name": "default", "job": 2565, "event": "table_file_creation", "file_number": 35345, "file_size": 73819966, "table_properties": {"data_size": 66468827, "index_size": 683833, "filter_size": 6666317, "raw_key_size": 75359628, "raw_average_key_size": 28, "raw_value_size": 31112281, "raw_average_value_size": 11, "num_data_blocks": 16333, "num_entries": 2608057, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "11", "kMergeOperands": "0"}} >2018-07-11 03:52:30.786952 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2565] Compacted 4@1 + 5@2 files to L2 => 595642447 bytes >2018-07-11 03:52:30.804178 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:52:30.803813) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 6 47 444 3641 683 0] max score 1.07, MB/sec: 17.7 rd, 17.7 wr, level 2, files in(4, 5) out(8) MB in(269.5, 298.6) out(568.0), read-write-amplify(4.2) write-amplify(2.1) OK, records in: 21075737, records dropped: 2 > >2018-07-11 03:52:30.804190 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:52:30.803839) EVENT_LOG_v1 {"time_micros": 1531281150803827, "job": 2565, "event": "compaction_finished", "compaction_time_micros": 33630959, "output_level": 2, "num_output_files": 8, "total_output_size": 595642447, "num_input_records": 21075737, "num_output_records": 21075735, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 6, 47, 444, 3641, 683, 0]} >2018-07-11 03:52:30.805067 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281150805064, "job": 2565, "event": "table_file_deletion", "file_number": 35335} >2018-07-11 03:52:30.805093 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281150805091, "job": 2565, "event": "table_file_deletion", "file_number": 35334} >2018-07-11 03:52:30.805105 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281150805103, "job": 2565, "event": "table_file_deletion", "file_number": 35333} >2018-07-11 03:52:30.805116 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281150805114, "job": 2565, "event": "table_file_deletion", "file_number": 35332} >2018-07-11 03:52:30.805131 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281150805129, "job": 2565, "event": "table_file_deletion", "file_number": 35314} >2018-07-11 03:52:30.805141 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281150805139, "job": 2565, "event": "table_file_deletion", "file_number": 35310} >2018-07-11 03:52:30.805156 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281150805154, "job": 2565, "event": "table_file_deletion", "file_number": 35308} >2018-07-11 03:52:30.805168 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281150805166, "job": 2565, "event": "table_file_deletion", "file_number": 35307} >2018-07-11 03:52:30.805181 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281150805179, "job": 2565, "event": "table_file_deletion", "file_number": 35306} >2018-07-11 03:52:30.805485 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2566] Compacting 1@2 + 2@3 files to L3, score 1.07 >2018-07-11 03:52:30.805496 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2565 Base level 2, inputs: [35340(71MB)], [35214(71MB) 35015(71MB)] > >2018-07-11 03:52:30.805509 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281150805499, "job": 2566, "event": "compaction_started", "files_L2": [35340], "files_L3": [35214, 35015], "score": 1.07366, "input_data_size": 223637979} >2018-07-11 03:52:36.550705 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2566] Generated table #35346: 2638240 keys, 74546448 bytes >2018-07-11 03:52:36.550753 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281156550737, "cf_name": "default", "job": 2566, "event": "table_file_creation", "file_number": 35346, "file_size": 74546448, "table_properties": {"data_size": 67112098, "index_size": 690179, "filter_size": 6743181, "raw_key_size": 76215828, "raw_average_key_size": 28, "raw_value_size": 31365748, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:52:42.764047 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2566] Generated table #35347: 2638238 keys, 74546405 bytes >2018-07-11 03:52:42.764092 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281162764066, "cf_name": "default", "job": 2566, "event": "table_file_creation", "file_number": 35347, "file_size": 74546405, "table_properties": {"data_size": 67112093, "index_size": 690145, "filter_size": 6743177, "raw_key_size": 76215770, "raw_average_key_size": 28, "raw_value_size": 31365724, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638238, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:52:47.919312 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2566] Generated table #35348: 2638240 keys, 74546887 bytes >2018-07-11 03:52:47.919355 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281167919329, "cf_name": "default", "job": 2566, "event": "table_file_creation", "file_number": 35348, "file_size": 74546887, "table_properties": {"data_size": 67112117, "index_size": 690599, "filter_size": 6743181, "raw_key_size": 76215828, "raw_average_key_size": 28, "raw_value_size": 31365748, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:52:47.970745 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2566] Generated table #35349: 2 keys, 1088 bytes >2018-07-11 03:52:47.970768 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281167970757, "cf_name": "default", "job": 2566, "event": "table_file_creation", "file_number": 35349, "file_size": 1088, "table_properties": {"data_size": 81, "index_size": 27, "filter_size": 18, "raw_key_size": 58, "raw_average_key_size": 29, "raw_value_size": 24, "raw_average_value_size": 12, "num_data_blocks": 1, "num_entries": 2, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:52:47.970804 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2566] Compacted 1@2 + 2@3 files to L3 => 223640828 bytes >2018-07-11 03:52:47.988381 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:52:47.988099) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 6 46 446 3641 683 0] max score 1.05, MB/sec: 13.0 rd, 13.0 wr, level 3, files in(1, 2) out(4) MB in(71.1, 142.2) out(213.3), read-write-amplify(6.0) write-amplify(3.0) OK, records in: 7914720, records dropped: 0 > >2018-07-11 03:52:47.988392 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:52:47.988125) EVENT_LOG_v1 {"time_micros": 1531281167988114, "job": 2566, "event": "compaction_finished", "compaction_time_micros": 17165267, "output_level": 3, "num_output_files": 4, "total_output_size": 223640828, "num_input_records": 7914720, "num_output_records": 7914720, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 6, 46, 446, 3641, 683, 0]} >2018-07-11 03:52:47.989302 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281167989298, "job": 2566, "event": "table_file_deletion", "file_number": 35340} >2018-07-11 03:52:47.989318 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281167989316, "job": 2566, "event": "table_file_deletion", "file_number": 35214} >2018-07-11 03:52:47.989331 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281167989329, "job": 2566, "event": "table_file_deletion", "file_number": 35015} >2018-07-11 03:52:47.989545 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_picker.cc:461] [default] Expanding@2 1+1(74546238+46965193 bytes) to 3+1 (223638403+46965193bytes) > >2018-07-11 03:52:47.989648 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2567] Compacting 3@2 + 1@3 files to L3, score 1.05 >2018-07-11 03:52:47.989659 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2566 Base level 2, inputs: [35342(71MB) 35343(71MB) 35344(71MB)], [35321(44MB)] > >2018-07-11 03:52:47.989672 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281167989662, "job": 2567, "event": "compaction_started", "files_L2": [35342, 35343, 35344], "files_L3": [35321], "score": 1.04589, "input_data_size": 270603596} >2018-07-11 03:52:53.572770 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2567] Generated table #35350: 2638240 keys, 74546253 bytes >2018-07-11 03:52:53.572809 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281173572794, "cf_name": "default", "job": 2567, "event": "table_file_creation", "file_number": 35350, "file_size": 74546253, "table_properties": {"data_size": 67112134, "index_size": 689948, "filter_size": 6743181, "raw_key_size": 76215821, "raw_average_key_size": 28, "raw_value_size": 31365741, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:52:59.109407 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2567] Generated table #35351: 2638241 keys, 74546126 bytes >2018-07-11 03:52:59.109450 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281179109433, "cf_name": "default", "job": 2567, "event": "table_file_creation", "file_number": 35351, "file_size": 74546126, "table_properties": {"data_size": 67112092, "index_size": 689861, "filter_size": 6743183, "raw_key_size": 76215850, "raw_average_key_size": 28, "raw_value_size": 31365753, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:53:04.183947 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2567] Generated table #35352: 2638241 keys, 74547072 bytes >2018-07-11 03:53:04.184001 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281184183984, "cf_name": "default", "job": 2567, "event": "table_file_creation", "file_number": 35352, "file_size": 74547072, "table_properties": {"data_size": 67112173, "index_size": 690726, "filter_size": 6743183, "raw_key_size": 76215860, "raw_average_key_size": 28, "raw_value_size": 31365763, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:53:07.413552 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2567] Generated table #35353: 1519916 keys, 46965006 bytes >2018-07-11 03:53:07.413597 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281187413582, "cf_name": "default", "job": 2567, "event": "table_file_creation", "file_number": 35353, "file_size": 46965006, "table_properties": {"data_size": 42636618, "index_size": 433249, "filter_size": 3894151, "raw_key_size": 44348930, "raw_average_key_size": 29, "raw_value_size": 21522459, "raw_average_value_size": 14, "num_data_blocks": 10538, "num_entries": 1519916, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:53:07.413637 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2567] Compacted 3@2 + 1@3 files to L3 => 270604457 bytes >2018-07-11 03:53:07.429551 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:53:07.429229) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 6 43 449 3641 683 0] max score 1.01, MB/sec: 13.9 rd, 13.9 wr, level 3, files in(3, 1) out(4) MB in(213.3, 44.8) out(258.1), read-write-amplify(2.4) write-amplify(1.2) OK, records in: 9434638, records dropped: 0 > >2018-07-11 03:53:07.429562 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:53:07.429268) EVENT_LOG_v1 {"time_micros": 1531281187429243, "job": 2567, "event": "compaction_finished", "compaction_time_micros": 19423949, "output_level": 3, "num_output_files": 4, "total_output_size": 270604457, "num_input_records": 9434638, "num_output_records": 9434638, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 6, 43, 449, 3641, 683, 0]} >2018-07-11 03:53:07.430492 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281187430488, "job": 2567, "event": "table_file_deletion", "file_number": 35344} >2018-07-11 03:53:07.430509 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281187430507, "job": 2567, "event": "table_file_deletion", "file_number": 35343} >2018-07-11 03:53:07.430523 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281187430521, "job": 2567, "event": "table_file_deletion", "file_number": 35342} >2018-07-11 03:53:07.430540 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281187430537, "job": 2567, "event": "table_file_deletion", "file_number": 35321} >2018-07-11 03:53:07.445999 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:53:07.430837) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #35352 to level-4 74547072 bytes > >2018-07-11 03:53:07.446009 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:53:07.445907) EVENT_LOG_v1 {"time_micros": 1531281187445901, "job": 2568, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 74547072} >2018-07-11 03:53:07.446012 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:53:07.445915) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-4 74547072 bytes OK: base level 1 max bytes base 268435456 files[0 6 43 448 3642 683 0] max score 1.01 > >2018-07-11 03:53:07.471145 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:53:07.446838) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #35348 to level-4 74546887 bytes > >2018-07-11 03:53:07.471157 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:53:07.470946) EVENT_LOG_v1 {"time_micros": 1531281187470940, "job": 2569, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 74546887} >2018-07-11 03:53:07.471159 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:53:07.470954) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-4 74546887 bytes OK: base level 1 max bytes base 268435456 files[0 6 43 447 3643 683 0] max score 1.00 > >2018-07-11 03:53:07.472000 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2570] Compacting 1@3 + 3@4 files to L4, score 1.00 >2018-07-11 03:53:07.472012 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2569 Base level 3, inputs: [35346(71MB)], [34948(71MB) 34949(71MB) 34950(71MB)] > >2018-07-11 03:53:07.472041 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281187472029, "job": 2570, "event": "compaction_started", "files_L3": [35346], "files_L4": [34948, 34949, 34950], "score": 1.00291, "input_data_size": 298185432} >2018-07-11 03:53:13.643577 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2570] Generated table #35354: 2638241 keys, 74546295 bytes >2018-07-11 03:53:13.643628 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281193643612, "cf_name": "default", "job": 2570, "event": "table_file_creation", "file_number": 35354, "file_size": 74546295, "table_properties": {"data_size": 67112130, "index_size": 689992, "filter_size": 6743183, "raw_key_size": 76215855, "raw_average_key_size": 28, "raw_value_size": 31365758, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:53:17.750893 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2570] Generated table #35355: 2638241 keys, 74546307 bytes >2018-07-11 03:53:17.750929 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281197750914, "cf_name": "default", "job": 2570, "event": "table_file_creation", "file_number": 35355, "file_size": 74546307, "table_properties": {"data_size": 67112177, "index_size": 689957, "filter_size": 6743183, "raw_key_size": 76215851, "raw_average_key_size": 28, "raw_value_size": 31365754, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:53:18.760277 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:684] reusing log 35327 from recycle list > >2018-07-11 03:53:18.760382 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #35357. Immutable memtables: 0. > >2018-07-11 03:53:18.830562 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-03:53:18.760438) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 0 >2018-07-11 03:53:18.830576 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 2571] Flushing memtable with next log file: 35357 > >2018-07-11 03:53:18.830594 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281198830586, "job": 2571, "event": "flush_started", "num_memtables": 1, "num_entries": 5000015, "num_deletes": 0, "memory_usage": 292943600} >2018-07-11 03:53:18.830598 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 2571] Level-0 flush table #35358: started >2018-07-11 03:53:22.685914 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2570] Generated table #35356: 2638241 keys, 74546428 bytes >2018-07-11 03:53:22.685961 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281202685934, "cf_name": "default", "job": 2570, "event": "table_file_creation", "file_number": 35356, "file_size": 74546428, "table_properties": {"data_size": 67112127, "index_size": 690128, "filter_size": 6743183, "raw_key_size": 76215850, "raw_average_key_size": 28, "raw_value_size": 31365753, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:53:22.753201 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281202753181, "cf_name": "default", "job": 2571, "event": "table_file_creation", "file_number": 35358, "file_size": 141284983, "table_properties": {"data_size": 127195853, "index_size": 1308422, "filter_size": 12779718, "raw_key_size": 144445053, "raw_average_key_size": 28, "raw_value_size": 59448873, "raw_average_value_size": 11, "num_data_blocks": 31252, "num_entries": 5000012, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:53:22.753234 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 2571] Level-0 flush table #35358: 141284983 bytes OK >2018-07-11 03:53:22.779113 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:242] adding log 35329 to recycle list > >2018-07-11 03:53:22.779124 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-03:53:22.762242) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #35358 started >2018-07-11 03:53:22.779130 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-03:53:22.778993) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #35358: memtable #1 done >2018-07-11 03:53:22.779134 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-03:53:22.779009) EVENT_LOG_v1 {"time_micros": 1531281202779002, "job": 2571, "event": "flush_finished", "lsm_state": [1, 6, 43, 447, 3643, 683, 0], "immutable_memtables": 0} >2018-07-11 03:53:22.779137 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-03:53:22.779026) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 6 43 447 3643 683 0] max score 1.00 > >2018-07-11 03:53:22.780188 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 2571] Try to delete WAL files size 178896755, prev total WAL file size 178896767, number of live WAL files 2. > >2018-07-11 03:53:26.336824 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2570] Generated table #35359: 2638240 keys, 74545736 bytes >2018-07-11 03:53:26.336863 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281206336847, "cf_name": "default", "job": 2570, "event": "table_file_creation", "file_number": 35359, "file_size": 74545736, "table_properties": {"data_size": 67112093, "index_size": 689483, "filter_size": 6743170, "raw_key_size": 76215828, "raw_average_key_size": 28, "raw_value_size": 31365748, "raw_average_value_size": 11, "num_data_blocks": 16489, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:53:26.336897 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2570] Compacted 1@3 + 3@4 files to L4 => 298184766 bytes >2018-07-11 03:53:26.354217 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:53:26.353953) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[1 6 43 446 3644 683 0] max score 1.00, MB/sec: 15.8 rd, 15.8 wr, level 4, files in(1, 3) out(4) MB in(71.1, 213.3) out(284.4), read-write-amplify(8.0) write-amplify(4.0) OK, records in: 10552963, records dropped: 0 > >2018-07-11 03:53:26.354241 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:53:26.353978) EVENT_LOG_v1 {"time_micros": 1531281206353967, "job": 2570, "event": "compaction_finished", "compaction_time_micros": 18864843, "output_level": 4, "num_output_files": 4, "total_output_size": 298184766, "num_input_records": 10552963, "num_output_records": 10552963, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 6, 43, 446, 3644, 683, 0]} >2018-07-11 03:53:26.355164 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281206355160, "job": 2570, "event": "table_file_deletion", "file_number": 35346} >2018-07-11 03:53:26.355180 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281206355178, "job": 2570, "event": "table_file_deletion", "file_number": 34950} >2018-07-11 03:53:26.355195 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281206355193, "job": 2570, "event": "table_file_deletion", "file_number": 34949} >2018-07-11 03:53:26.355206 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281206355204, "job": 2570, "event": "table_file_deletion", "file_number": 34948} >2018-07-11 03:53:26.412405 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:53:26.355503) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #35352 to level-5 74547072 bytes > >2018-07-11 03:53:26.412416 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:53:26.412297) EVENT_LOG_v1 {"time_micros": 1531281206412290, "job": 2572, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74547072} >2018-07-11 03:53:26.412418 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:53:26.412305) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74547072 bytes OK: base level 1 max bytes base 268435456 files[1 6 43 446 3643 684 0] max score 1.00 > >2018-07-11 03:53:26.413160 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_picker.cc:461] [default] Expanding@4 1+1(74546887+74546988 bytes) to 2+1 (149092693+74546988bytes) > >2018-07-11 03:53:26.413274 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2573] Compacting 2@4 + 1@5 files to L5, score 1.00 >2018-07-11 03:53:26.413286 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2572 Base level 4, inputs: [35348(71MB) 35216(71MB)], [35101(71MB)] > >2018-07-11 03:53:26.413300 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281206413289, "job": 2573, "event": "compaction_started", "files_L4": [35348, 35216], "files_L5": [35101], "score": 1.00055, "input_data_size": 223639681} >2018-07-11 03:53:31.121871 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2573] Generated table #35360: 2638241 keys, 74546214 bytes >2018-07-11 03:53:31.121914 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281211121897, "cf_name": "default", "job": 2573, "event": "table_file_creation", "file_number": 35360, "file_size": 74546214, "table_properties": {"data_size": 67112168, "index_size": 689873, "filter_size": 6743183, "raw_key_size": 76215856, "raw_average_key_size": 28, "raw_value_size": 31365759, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:53:36.177966 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2573] Generated table #35361: 2638241 keys, 74546393 bytes >2018-07-11 03:53:36.178020 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281216178003, "cf_name": "default", "job": 2573, "event": "table_file_creation", "file_number": 35361, "file_size": 74546393, "table_properties": {"data_size": 67112112, "index_size": 690108, "filter_size": 6743183, "raw_key_size": 76215855, "raw_average_key_size": 28, "raw_value_size": 31365758, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:53:40.083048 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2573] Generated table #35362: 2638239 keys, 74546342 bytes >2018-07-11 03:53:40.083099 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281220083070, "cf_name": "default", "job": 2573, "event": "table_file_creation", "file_number": 35362, "file_size": 74546342, "table_properties": {"data_size": 67112077, "index_size": 690107, "filter_size": 6743168, "raw_key_size": 76215793, "raw_average_key_size": 28, "raw_value_size": 31365730, "raw_average_value_size": 11, "num_data_blocks": 16489, "num_entries": 2638239, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:53:40.083130 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2573] Compacted 2@4 + 1@5 files to L5 => 223638949 bytes >2018-07-11 03:53:40.104157 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:53:40.103883) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[1 6 43 446 3641 686 0] max score 1.00, MB/sec: 16.4 rd, 16.4 wr, level 5, files in(2, 1) out(3) MB in(142.2, 71.1) out(213.3), read-write-amplify(3.0) write-amplify(1.5) OK, records in: 7914721, records dropped: 0 > >2018-07-11 03:53:40.104168 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:53:40.103908) EVENT_LOG_v1 {"time_micros": 1531281220103897, "job": 2573, "event": "compaction_finished", "compaction_time_micros": 13669815, "output_level": 5, "num_output_files": 3, "total_output_size": 223638949, "num_input_records": 7914721, "num_output_records": 7914721, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 6, 43, 446, 3641, 686, 0]} >2018-07-11 03:53:40.105076 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281220105072, "job": 2573, "event": "table_file_deletion", "file_number": 35348} >2018-07-11 03:53:40.105105 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281220105102, "job": 2573, "event": "table_file_deletion", "file_number": 35216} >2018-07-11 03:53:40.105118 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281220105116, "job": 2573, "event": "table_file_deletion", "file_number": 35101} >2018-07-11 03:53:40.105417 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2574] Compacting 1@3 + 5@4 files to L4, score 1.00 >2018-07-11 03:53:40.105429 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2573 Base level 3, inputs: [35347(71MB)], [35215(71MB) 35094(71MB) 35095(71MB) 35096(71MB) 35014(71MB)] > >2018-07-11 03:53:40.105443 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281220105432, "job": 2574, "event": "compaction_started", "files_L3": [35347], "files_L4": [35215, 35094, 35095, 35096, 35014], "score": 1.00013, "input_data_size": 447278633} >2018-07-11 03:53:45.399549 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2574] Generated table #35363: 2638241 keys, 74545910 bytes >2018-07-11 03:53:45.399602 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281225399585, "cf_name": "default", "job": 2574, "event": "table_file_creation", "file_number": 35363, "file_size": 74545910, "table_properties": {"data_size": 67112160, "index_size": 689577, "filter_size": 6743183, "raw_key_size": 76215853, "raw_average_key_size": 28, "raw_value_size": 31365756, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:53:52.786684 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2574] Generated table #35364: 2638240 keys, 74546845 bytes >2018-07-11 03:53:52.786726 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281232786701, "cf_name": "default", "job": 2574, "event": "table_file_creation", "file_number": 35364, "file_size": 74546845, "table_properties": {"data_size": 67112129, "index_size": 690545, "filter_size": 6743181, "raw_key_size": 76215824, "raw_average_key_size": 28, "raw_value_size": 31365744, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:53:58.453137 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2574] Generated table #35365: 2638241 keys, 74546456 bytes >2018-07-11 03:53:58.453181 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281238453154, "cf_name": "default", "job": 2574, "event": "table_file_creation", "file_number": 35365, "file_size": 74546456, "table_properties": {"data_size": 67112126, "index_size": 690157, "filter_size": 6743183, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:54:03.645261 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2574] Generated table #35366: 2638241 keys, 74546230 bytes >2018-07-11 03:54:03.645306 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281243645278, "cf_name": "default", "job": 2574, "event": "table_file_creation", "file_number": 35366, "file_size": 74546230, "table_properties": {"data_size": 67112168, "index_size": 689889, "filter_size": 6743183, "raw_key_size": 76215857, "raw_average_key_size": 28, "raw_value_size": 31365760, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:54:09.443419 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2574] Generated table #35367: 2638241 keys, 74545960 bytes >2018-07-11 03:54:09.443465 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281249443438, "cf_name": "default", "job": 2574, "event": "table_file_creation", "file_number": 35367, "file_size": 74545960, "table_properties": {"data_size": 67112102, "index_size": 689685, "filter_size": 6743183, "raw_key_size": 76215855, "raw_average_key_size": 28, "raw_value_size": 31365758, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:54:15.575688 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2574] Generated table #35368: 2638238 keys, 74546046 bytes >2018-07-11 03:54:15.575731 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281255575704, "cf_name": "default", "job": 2574, "event": "table_file_creation", "file_number": 35368, "file_size": 74546046, "table_properties": {"data_size": 67112048, "index_size": 689843, "filter_size": 6743165, "raw_key_size": 76215762, "raw_average_key_size": 28, "raw_value_size": 31365716, "raw_average_value_size": 11, "num_data_blocks": 16489, "num_entries": 2638238, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:54:15.575768 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2574] Compacted 1@3 + 5@4 files to L4 => 447277447 bytes >2018-07-11 03:54:15.594990 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:54:15.594731) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[1 6 43 445 3642 686 0] max score 1.00, MB/sec: 12.6 rd, 12.6 wr, level 4, files in(1, 5) out(6) MB in(71.1, 355.5) out(426.6), read-write-amplify(12.0) write-amplify(6.0) OK, records in: 15829442, records dropped: 0 > >2018-07-11 03:54:15.595000 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:54:15.594756) EVENT_LOG_v1 {"time_micros": 1531281255594745, "job": 2574, "event": "compaction_finished", "compaction_time_micros": 35470307, "output_level": 4, "num_output_files": 6, "total_output_size": 447277447, "num_input_records": 15829442, "num_output_records": 15829442, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 6, 43, 445, 3642, 686, 0]} >2018-07-11 03:54:15.595912 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281255595908, "job": 2574, "event": "table_file_deletion", "file_number": 35347} >2018-07-11 03:54:15.595942 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281255595939, "job": 2574, "event": "table_file_deletion", "file_number": 35215} >2018-07-11 03:54:15.595954 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281255595952, "job": 2574, "event": "table_file_deletion", "file_number": 35096} >2018-07-11 03:54:15.595965 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281255595963, "job": 2574, "event": "table_file_deletion", "file_number": 35095} >2018-07-11 03:54:15.595977 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281255595975, "job": 2574, "event": "table_file_deletion", "file_number": 35094} >2018-07-11 03:54:15.595990 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281255595988, "job": 2574, "event": "table_file_deletion", "file_number": 35014} >2018-07-11 03:54:15.611577 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:54:15.596295) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #6816 to level-5 74546865 bytes > >2018-07-11 03:54:15.611588 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:54:15.611404) EVENT_LOG_v1 {"time_micros": 1531281255611397, "job": 2575, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546865} >2018-07-11 03:54:15.611590 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:54:15.611413) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546865 bytes OK: base level 1 max bytes base 268435456 files[1 6 43 445 3641 687 0] max score 1.00 > >2018-07-11 03:55:44.487663 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:684] reusing log 35329 from recycle list > >2018-07-11 03:55:44.487766 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #35369. Immutable memtables: 0. > >2018-07-11 03:55:44.532073 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-03:55:44.487802) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1 >2018-07-11 03:55:44.532087 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 2576] Flushing memtable with next log file: 35369 > >2018-07-11 03:55:44.532110 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281344532098, "job": 2576, "event": "flush_started", "num_memtables": 1, "num_entries": 5000015, "num_deletes": 0, "memory_usage": 292946920} >2018-07-11 03:55:44.532115 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 2576] Level-0 flush table #35370: started >2018-07-11 03:55:48.440506 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281348440486, "cf_name": "default", "job": 2576, "event": "table_file_creation", "file_number": 35370, "file_size": 141286890, "table_properties": {"data_size": 127197778, "index_size": 1308397, "filter_size": 12779725, "raw_key_size": 144445101, "raw_average_key_size": 28, "raw_value_size": 59450671, "raw_average_value_size": 11, "num_data_blocks": 31252, "num_entries": 5000014, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:55:48.440529 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 2576] Level-0 flush table #35370: 141286890 bytes OK >2018-07-11 03:55:48.459131 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:242] adding log 35357 to recycle list > >2018-07-11 03:55:48.459144 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-03:55:48.440554) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #35370 started >2018-07-11 03:55:48.459160 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-03:55:48.458908) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #35370: memtable #1 done >2018-07-11 03:55:48.459162 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-03:55:48.458926) EVENT_LOG_v1 {"time_micros": 1531281348458919, "job": 2576, "event": "flush_finished", "lsm_state": [2, 6, 43, 445, 3641, 687, 0], "immutable_memtables": 0} >2018-07-11 03:55:48.459164 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-03:55:48.459067) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[2 6 43 445 3641 687 0] max score 1.05 > >2018-07-11 03:55:48.459295 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2577] Compacting 2@0 + 2@1 files to L1, score 1.05 >2018-07-11 03:55:48.459312 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2576 Base level 0, inputs: [35370(134MB) 35358(134MB)], [35331(71MB) 35336(65MB)] > >2018-07-11 03:55:48.459329 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281348459317, "job": 2577, "event": "compaction_started", "files_L0": [35370, 35358], "files_L1": [35331, 35336], "score": 1.05266, "input_data_size": 425329761} >2018-07-11 03:55:48.460023 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 2576] Try to delete WAL files size 178896755, prev total WAL file size 178896767, number of live WAL files 2. > >2018-07-11 03:55:51.283107 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2577] Generated table #35371: 2624680 keys, 74512219 bytes >2018-07-11 03:55:51.283141 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281351283126, "cf_name": "default", "job": 2577, "event": "table_file_creation", "file_number": 35371, "file_size": 74512219, "table_properties": {"data_size": 67110617, "index_size": 691321, "filter_size": 6709291, "raw_key_size": 75853658, "raw_average_key_size": 28, "raw_value_size": 31540576, "raw_average_value_size": 12, "num_data_blocks": 16475, "num_entries": 2624680, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:55:53.891314 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2577] Generated table #35372: 2638242 keys, 74546423 bytes >2018-07-11 03:55:53.891357 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281353891331, "cf_name": "default", "job": 2577, "event": "table_file_creation", "file_number": 35372, "file_size": 74546423, "table_properties": {"data_size": 67112133, "index_size": 690114, "filter_size": 6743186, "raw_key_size": 76215873, "raw_average_key_size": 28, "raw_value_size": 31365759, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638242, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:55:56.531575 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2577] Generated table #35373: 2638241 keys, 74546257 bytes >2018-07-11 03:55:56.531617 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281356531591, "cf_name": "default", "job": 2577, "event": "table_file_creation", "file_number": 35373, "file_size": 74546257, "table_properties": {"data_size": 67112146, "index_size": 689938, "filter_size": 6743183, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:55:59.133972 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2577] Generated table #35374: 2638241 keys, 74545807 bytes >2018-07-11 03:55:59.134017 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281359133989, "cf_name": "default", "job": 2577, "event": "table_file_creation", "file_number": 35374, "file_size": 74545807, "table_properties": {"data_size": 67112088, "index_size": 689546, "filter_size": 6743183, "raw_key_size": 76215853, "raw_average_key_size": 28, "raw_value_size": 31365756, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:56:01.276269 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2577] Generated table #35375: 2089247 keys, 61206787 bytes >2018-07-11 03:56:01.276311 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281361276285, "cf_name": "default", "job": 2577, "event": "table_file_creation", "file_number": 35375, "file_size": 61206787, "table_properties": {"data_size": 55296128, "index_size": 564678, "filter_size": 5344993, "raw_key_size": 60632730, "raw_average_key_size": 29, "raw_value_size": 26671622, "raw_average_value_size": 12, "num_data_blocks": 13590, "num_entries": 2089247, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:56:02.413403 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2577] Generated table #35376: 162211 keys, 65967010 bytes >2018-07-11 03:56:02.413443 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281362413418, "cf_name": "default", "job": 2577, "event": "table_file_creation", "file_number": 35376, "file_size": 65967010, "table_properties": {"data_size": 64866276, "index_size": 545862, "filter_size": 553884, "raw_key_size": 14983916, "raw_average_key_size": 92, "raw_value_size": 51241140, "raw_average_value_size": 315, "num_data_blocks": 14656, "num_entries": 162211, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "80898", "kMergeOperands": "0"}} >2018-07-11 03:56:02.413505 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2577] Compacted 2@0 + 2@1 files to L1 => 425324503 bytes >2018-07-11 03:56:02.433987 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:56:02.433735) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 10 43 445 3641 687 0] max score 1.80, MB/sec: 30.5 rd, 30.5 wr, level 1, files in(2, 2) out(6) MB in(269.5, 136.1) out(405.6), read-write-amplify(3.0) write-amplify(1.5) OK, records in: 12790868, records dropped: 6 > >2018-07-11 03:56:02.433998 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:56:02.433760) EVENT_LOG_v1 {"time_micros": 1531281362433749, "job": 2577, "event": "compaction_finished", "compaction_time_micros": 13954158, "output_level": 1, "num_output_files": 6, "total_output_size": 425324503, "num_input_records": 12790868, "num_output_records": 12790862, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 10, 43, 445, 3641, 687, 0]} >2018-07-11 03:56:02.434901 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281362434897, "job": 2577, "event": "table_file_deletion", "file_number": 35370} >2018-07-11 03:56:02.434926 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281362434924, "job": 2577, "event": "table_file_deletion", "file_number": 35358} >2018-07-11 03:56:02.434941 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281362434938, "job": 2577, "event": "table_file_deletion", "file_number": 35336} >2018-07-11 03:56:02.434952 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281362434950, "job": 2577, "event": "table_file_deletion", "file_number": 35331} >2018-07-11 03:56:02.435167 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_picker.cc:461] [default] Expanding@1 1+4(74546423+297457710 bytes) to 3+4 (223638487+297457710bytes) > >2018-07-11 03:56:02.435268 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2578] Compacting 3@1 + 4@2 files to L2, score 1.80 >2018-07-11 03:56:02.435280 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2577 Base level 1, inputs: [35372(71MB) 35373(71MB) 35374(71MB)], [35338(71MB) 35339(71MB) 35341(71MB) 35345(70MB)] > >2018-07-11 03:56:02.435294 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281362435284, "job": 2578, "event": "compaction_started", "files_L1": [35372, 35373, 35374], "files_L2": [35338, 35339, 35341, 35345], "score": 1.79573, "input_data_size": 521096197} >2018-07-11 03:56:07.071845 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2578] Generated table #35377: 2638238 keys, 74545700 bytes >2018-07-11 03:56:07.071891 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281367071864, "cf_name": "default", "job": 2578, "event": "table_file_creation", "file_number": 35377, "file_size": 74545700, "table_properties": {"data_size": 67112106, "index_size": 689427, "filter_size": 6743177, "raw_key_size": 76215767, "raw_average_key_size": 28, "raw_value_size": 31365721, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638238, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:56:11.515760 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2578] Generated table #35378: 2638241 keys, 74546213 bytes >2018-07-11 03:56:11.515793 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281371515779, "cf_name": "default", "job": 2578, "event": "table_file_creation", "file_number": 35378, "file_size": 74546213, "table_properties": {"data_size": 67112110, "index_size": 689930, "filter_size": 6743183, "raw_key_size": 76215853, "raw_average_key_size": 28, "raw_value_size": 31365756, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:56:15.980725 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2578] Generated table #35379: 2638238 keys, 74545856 bytes >2018-07-11 03:56:15.980783 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281375980766, "cf_name": "default", "job": 2578, "event": "table_file_creation", "file_number": 35379, "file_size": 74545856, "table_properties": {"data_size": 67112114, "index_size": 689575, "filter_size": 6743177, "raw_key_size": 76215768, "raw_average_key_size": 28, "raw_value_size": 31365722, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638238, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:56:20.316580 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2578] Generated table #35380: 2638241 keys, 74546942 bytes >2018-07-11 03:56:20.316624 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281380316597, "cf_name": "default", "job": 2578, "event": "table_file_creation", "file_number": 35380, "file_size": 74546942, "table_properties": {"data_size": 67112120, "index_size": 690649, "filter_size": 6743183, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:56:23.855433 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2578] Generated table #35381: 2638241 keys, 74546252 bytes >2018-07-11 03:56:23.855487 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281383855471, "cf_name": "default", "job": 2578, "event": "table_file_creation", "file_number": 35381, "file_size": 74546252, "table_properties": {"data_size": 67112162, "index_size": 689917, "filter_size": 6743183, "raw_key_size": 76215855, "raw_average_key_size": 28, "raw_value_size": 31365758, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:56:27.165180 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2578] Generated table #35382: 2638241 keys, 74546433 bytes >2018-07-11 03:56:27.165233 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281387165213, "cf_name": "default", "job": 2578, "event": "table_file_creation", "file_number": 35382, "file_size": 74546433, "table_properties": {"data_size": 67112148, "index_size": 690112, "filter_size": 6743183, "raw_key_size": 76215856, "raw_average_key_size": 28, "raw_value_size": 31365759, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:56:31.382599 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2578] Generated table #35383: 2608058 keys, 73818914 bytes >2018-07-11 03:56:31.382647 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281391382632, "cf_name": "default", "job": 2578, "event": "table_file_creation", "file_number": 35383, "file_size": 73818914, "table_properties": {"data_size": 66468861, "index_size": 682744, "filter_size": 6666320, "raw_key_size": 75359645, "raw_average_key_size": 28, "raw_value_size": 31112281, "raw_average_value_size": 11, "num_data_blocks": 16333, "num_entries": 2608058, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "11", "kMergeOperands": "0"}} >2018-07-11 03:56:31.382685 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2578] Compacted 3@1 + 4@2 files to L2 => 521096310 bytes >2018-07-11 03:56:31.400117 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:56:31.399770) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 7 46 445 3641 687 0] max score 1.05, MB/sec: 18.0 rd, 18.0 wr, level 2, files in(3, 4) out(7) MB in(213.3, 283.7) out(497.0), read-write-amplify(4.7) write-amplify(2.3) OK, records in: 18437498, records dropped: 0 > >2018-07-11 03:56:31.400129 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:56:31.399796) EVENT_LOG_v1 {"time_micros": 1531281391399785, "job": 2578, "event": "compaction_finished", "compaction_time_micros": 28947373, "output_level": 2, "num_output_files": 7, "total_output_size": 521096310, "num_input_records": 18437498, "num_output_records": 18437498, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 7, 46, 445, 3641, 687, 0]} >2018-07-11 03:56:31.401047 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281391401030, "job": 2578, "event": "table_file_deletion", "file_number": 35374} >2018-07-11 03:56:31.401061 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281391401059, "job": 2578, "event": "table_file_deletion", "file_number": 35373} >2018-07-11 03:56:31.401073 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281391401071, "job": 2578, "event": "table_file_deletion", "file_number": 35372} >2018-07-11 03:56:31.401088 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281391401086, "job": 2578, "event": "table_file_deletion", "file_number": 35345} >2018-07-11 03:56:31.401100 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281391401098, "job": 2578, "event": "table_file_deletion", "file_number": 35341} >2018-07-11 03:56:31.401111 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281391401109, "job": 2578, "event": "table_file_deletion", "file_number": 35339} >2018-07-11 03:56:31.401122 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281391401120, "job": 2578, "event": "table_file_deletion", "file_number": 35338} >2018-07-11 03:56:31.401350 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_picker.cc:461] [default] Expanding@2 1+3(74546942+196057385 bytes) to 3+3 (223639627+196057385bytes) > >2018-07-11 03:56:31.401449 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2579] Compacting 3@2 + 3@3 files to L3, score 1.05 >2018-07-11 03:56:31.401462 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2578 Base level 2, inputs: [35380(71MB) 35381(71MB) 35382(71MB)], [35350(71MB) 35351(71MB) 35353(44MB)] > >2018-07-11 03:56:31.401476 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281391401465, "job": 2579, "event": "compaction_started", "files_L2": [35380, 35381, 35382], "files_L3": [35350, 35351, 35353], "score": 1.04589, "input_data_size": 419697012} >2018-07-11 03:56:36.091370 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2579] Generated table #35384: 2638240 keys, 74546254 bytes >2018-07-11 03:56:36.091416 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281396091398, "cf_name": "default", "job": 2579, "event": "table_file_creation", "file_number": 35384, "file_size": 74546254, "table_properties": {"data_size": 67112133, "index_size": 689950, "filter_size": 6743181, "raw_key_size": 76215820, "raw_average_key_size": 28, "raw_value_size": 31365740, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:56:40.971121 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2579] Generated table #35385: 2638241 keys, 74546121 bytes >2018-07-11 03:56:40.971173 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281400971157, "cf_name": "default", "job": 2579, "event": "table_file_creation", "file_number": 35385, "file_size": 74546121, "table_properties": {"data_size": 67112094, "index_size": 689854, "filter_size": 6743183, "raw_key_size": 76215852, "raw_average_key_size": 28, "raw_value_size": 31365755, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:56:45.220845 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2579] Generated table #35386: 2638241 keys, 74546114 bytes >2018-07-11 03:56:45.220895 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281405220880, "cf_name": "default", "job": 2579, "event": "table_file_creation", "file_number": 35386, "file_size": 74546114, "table_properties": {"data_size": 67112120, "index_size": 689821, "filter_size": 6743183, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:56:49.347348 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2579] Generated table #35387: 2638241 keys, 74546317 bytes >2018-07-11 03:56:49.347396 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281409347381, "cf_name": "default", "job": 2579, "event": "table_file_creation", "file_number": 35387, "file_size": 74546317, "table_properties": {"data_size": 67112155, "index_size": 689989, "filter_size": 6743183, "raw_key_size": 76215848, "raw_average_key_size": 28, "raw_value_size": 31365751, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:56:54.691450 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2579] Generated table #35388: 2638241 keys, 74545711 bytes >2018-07-11 03:56:54.691498 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281414691483, "cf_name": "default", "job": 2579, "event": "table_file_creation", "file_number": 35388, "file_size": 74545711, "table_properties": {"data_size": 67112130, "index_size": 689408, "filter_size": 6743183, "raw_key_size": 76215857, "raw_average_key_size": 28, "raw_value_size": 31365760, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:56:59.406306 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2579] Generated table #35389: 1519916 keys, 46965395 bytes >2018-07-11 03:56:59.406356 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281419406329, "cf_name": "default", "job": 2579, "event": "table_file_creation", "file_number": 35389, "file_size": 46965395, "table_properties": {"data_size": 42636637, "index_size": 433619, "filter_size": 3894151, "raw_key_size": 44348935, "raw_average_key_size": 29, "raw_value_size": 21522464, "raw_average_value_size": 14, "num_data_blocks": 10538, "num_entries": 1519916, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:56:59.406391 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2579] Compacted 3@2 + 3@3 files to L3 => 419695912 bytes >2018-07-11 03:56:59.424532 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:56:59.424186) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 7 43 448 3641 687 0] max score 1.01, MB/sec: 15.0 rd, 15.0 wr, level 3, files in(3, 3) out(6) MB in(213.3, 187.0) out(400.3), read-write-amplify(3.8) write-amplify(1.9) OK, records in: 14711120, records dropped: 0 > >2018-07-11 03:56:59.424543 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:56:59.424211) EVENT_LOG_v1 {"time_micros": 1531281419424200, "job": 2579, "event": "compaction_finished", "compaction_time_micros": 28004898, "output_level": 3, "num_output_files": 6, "total_output_size": 419695912, "num_input_records": 14711120, "num_output_records": 14711120, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 7, 43, 448, 3641, 687, 0]} >2018-07-11 03:56:59.425446 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281419425442, "job": 2579, "event": "table_file_deletion", "file_number": 35382} >2018-07-11 03:56:59.425464 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281419425462, "job": 2579, "event": "table_file_deletion", "file_number": 35381} >2018-07-11 03:56:59.425476 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281419425473, "job": 2579, "event": "table_file_deletion", "file_number": 35380} >2018-07-11 03:56:59.425492 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281419425490, "job": 2579, "event": "table_file_deletion", "file_number": 35353} >2018-07-11 03:56:59.425506 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281419425504, "job": 2579, "event": "table_file_deletion", "file_number": 35351} >2018-07-11 03:56:59.425519 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281419425516, "job": 2579, "event": "table_file_deletion", "file_number": 35350} >2018-07-11 03:56:59.441015 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:56:59.425821) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #35387 to level-4 74546317 bytes > >2018-07-11 03:56:59.441026 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:56:59.440924) EVENT_LOG_v1 {"time_micros": 1531281419440918, "job": 2580, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 74546317} >2018-07-11 03:56:59.441028 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:56:59.440931) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-4 74546317 bytes OK: base level 1 max bytes base 268435456 files[0 7 43 447 3642 687 0] max score 1.00 > >2018-07-11 03:56:59.441867 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2581] Compacting 1@3 + 2@4 files to L4, score 1.00 >2018-07-11 03:56:59.441878 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2580 Base level 3, inputs: [35384(71MB)], [35318(71MB) 35320(71MB)] > >2018-07-11 03:56:59.441904 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281419441881, "job": 2581, "event": "compaction_started", "files_L3": [35384], "files_L4": [35318, 35320], "score": 1.00291, "input_data_size": 223638731} >2018-07-11 03:57:07.108763 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2581] Generated table #35390: 2638241 keys, 74546496 bytes >2018-07-11 03:57:07.108818 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281427108799, "cf_name": "default", "job": 2581, "event": "table_file_creation", "file_number": 35390, "file_size": 74546496, "table_properties": {"data_size": 67112129, "index_size": 690194, "filter_size": 6743183, "raw_key_size": 76215853, "raw_average_key_size": 28, "raw_value_size": 31365756, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:57:14.284427 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2581] Generated table #35391: 2638240 keys, 74546315 bytes >2018-07-11 03:57:14.284479 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281434284464, "cf_name": "default", "job": 2581, "event": "table_file_creation", "file_number": 35391, "file_size": 74546315, "table_properties": {"data_size": 67112103, "index_size": 690041, "filter_size": 6743181, "raw_key_size": 76215826, "raw_average_key_size": 28, "raw_value_size": 31365746, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:57:20.736617 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2581] Generated table #35392: 2638241 keys, 74546270 bytes >2018-07-11 03:57:20.736663 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281440736635, "cf_name": "default", "job": 2581, "event": "table_file_creation", "file_number": 35392, "file_size": 74546270, "table_properties": {"data_size": 67112147, "index_size": 689950, "filter_size": 6743183, "raw_key_size": 76215850, "raw_average_key_size": 28, "raw_value_size": 31365753, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:57:20.736696 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2581] Compacted 1@3 + 2@4 files to L4 => 223639081 bytes >2018-07-11 03:57:20.757406 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:57:20.757122) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 7 43 446 3643 687 0] max score 1.00, MB/sec: 10.5 rd, 10.5 wr, level 4, files in(1, 2) out(3) MB in(71.1, 142.2) out(213.3), read-write-amplify(6.0) write-amplify(3.0) OK, records in: 7914722, records dropped: 0 > >2018-07-11 03:57:20.757417 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:57:20.757147) EVENT_LOG_v1 {"time_micros": 1531281440757136, "job": 2581, "event": "compaction_finished", "compaction_time_micros": 21294778, "output_level": 4, "num_output_files": 3, "total_output_size": 223639081, "num_input_records": 7914722, "num_output_records": 7914722, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 7, 43, 446, 3643, 687, 0]} >2018-07-11 03:57:20.758378 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281440758374, "job": 2581, "event": "table_file_deletion", "file_number": 35384} >2018-07-11 03:57:20.758396 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281440758393, "job": 2581, "event": "table_file_deletion", "file_number": 35320} >2018-07-11 03:57:20.758408 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281440758406, "job": 2581, "event": "table_file_deletion", "file_number": 35318} >2018-07-11 03:57:20.773913 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:57:20.758728) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #1040 to level-5 74546865 bytes > >2018-07-11 03:57:20.773924 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:57:20.773820) EVENT_LOG_v1 {"time_micros": 1531281440773813, "job": 2582, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546865} >2018-07-11 03:57:20.773927 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:57:20.773827) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546865 bytes OK: base level 1 max bytes base 268435456 files[0 7 43 446 3642 688 0] max score 1.00 > >2018-07-11 03:57:20.790688 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:57:20.775183) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #8918 to level-5 74546864 bytes > >2018-07-11 03:57:20.790699 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:57:20.790527) EVENT_LOG_v1 {"time_micros": 1531281440790521, "job": 2583, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546864} >2018-07-11 03:57:20.790701 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:57:20.790535) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546864 bytes OK: base level 1 max bytes base 268435456 files[0 7 43 446 3641 689 0] max score 1.00 > >2018-07-11 03:57:20.807290 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:57:20.791549) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #35385 to level-4 74546121 bytes > >2018-07-11 03:57:20.807301 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:57:20.807193) EVENT_LOG_v1 {"time_micros": 1531281440807187, "job": 2584, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 74546121} >2018-07-11 03:57:20.807303 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:57:20.807201) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-4 74546121 bytes OK: base level 1 max bytes base 268435456 files[0 7 43 445 3642 689 0] max score 1.00 > >2018-07-11 03:57:20.823987 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:57:20.808107) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #22550 to level-5 74546864 bytes > >2018-07-11 03:57:20.823997 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:57:20.823898) EVENT_LOG_v1 {"time_micros": 1531281440823892, "job": 2585, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546864} >2018-07-11 03:57:20.824000 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-03:57:20.823906) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546864 bytes OK: base level 1 max bytes base 268435456 files[0 7 43 445 3641 690 0] max score 1.00 > >2018-07-11 03:58:02.614801 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:684] reusing log 35357 from recycle list > >2018-07-11 03:58:02.614948 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #35393. Immutable memtables: 0. > >2018-07-11 03:58:02.689615 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-03:58:02.615012) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1 >2018-07-11 03:58:02.689629 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 2586] Flushing memtable with next log file: 35393 > >2018-07-11 03:58:02.689644 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281482689636, "job": 2586, "event": "flush_started", "num_memtables": 1, "num_entries": 5000015, "num_deletes": 0, "memory_usage": 292930144} >2018-07-11 03:58:02.689647 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 2586] Level-0 flush table #35394: started >2018-07-11 03:58:06.487307 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281486487287, "cf_name": "default", "job": 2586, "event": "table_file_creation", "file_number": 35394, "file_size": 141286889, "table_properties": {"data_size": 127197777, "index_size": 1308397, "filter_size": 12779725, "raw_key_size": 144445101, "raw_average_key_size": 28, "raw_value_size": 59450671, "raw_average_value_size": 11, "num_data_blocks": 31252, "num_entries": 5000014, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:58:06.487329 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 2586] Level-0 flush table #35394: 141286889 bytes OK >2018-07-11 03:58:06.506306 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:242] adding log 35369 to recycle list > >2018-07-11 03:58:06.506318 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-03:58:06.487352) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #35394 started >2018-07-11 03:58:06.506322 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-03:58:06.506069) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #35394: memtable #1 done >2018-07-11 03:58:06.506324 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-03:58:06.506087) EVENT_LOG_v1 {"time_micros": 1531281486506080, "job": 2586, "event": "flush_finished", "lsm_state": [1, 7, 43, 445, 3641, 690, 0], "immutable_memtables": 0} >2018-07-11 03:58:06.506326 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-03:58:06.506244) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 7 43 445 3641 690 0] max score 1.00 > >2018-07-11 03:58:06.507165 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 2586] Try to delete WAL files size 178896755, prev total WAL file size 178896767, number of live WAL files 2. > >2018-07-11 03:59:48.858524 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:684] reusing log 35369 from recycle list > >2018-07-11 03:59:48.858629 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #35395. Immutable memtables: 0. > >2018-07-11 03:59:48.900140 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-03:59:48.858667) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1 >2018-07-11 03:59:48.900154 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 2587] Flushing memtable with next log file: 35395 > >2018-07-11 03:59:48.900173 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281588900163, "job": 2587, "event": "flush_started", "num_memtables": 1, "num_entries": 5000015, "num_deletes": 0, "memory_usage": 292957992} >2018-07-11 03:59:48.900177 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 2587] Level-0 flush table #35396: started >2018-07-11 03:59:53.167652 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281593167604, "cf_name": "default", "job": 2587, "event": "table_file_creation", "file_number": 35396, "file_size": 141286220, "table_properties": {"data_size": 127196935, "index_size": 1308576, "filter_size": 12779719, "raw_key_size": 144445077, "raw_average_key_size": 28, "raw_value_size": 59449772, "raw_average_value_size": 11, "num_data_blocks": 31252, "num_entries": 5000013, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:59:53.167677 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 2587] Level-0 flush table #35396: 141286220 bytes OK >2018-07-11 03:59:53.186655 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:242] adding log 35393 to recycle list > >2018-07-11 03:59:53.186668 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-03:59:53.167689) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #35396 started >2018-07-11 03:59:53.186672 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-03:59:53.186410) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #35396: memtable #1 done >2018-07-11 03:59:53.186674 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-03:59:53.186428) EVENT_LOG_v1 {"time_micros": 1531281593186422, "job": 2587, "event": "flush_finished", "lsm_state": [2, 7, 43, 445, 3641, 690, 0], "immutable_memtables": 0} >2018-07-11 03:59:53.186676 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-03:59:53.186589) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[2 7 43 445 3641 690 0] max score 1.05 > >2018-07-11 03:59:53.187081 7f41641eb700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS ------- >2018-07-11 03:59:53.187092 7f41641eb700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] >** DB Stats ** >Uptime(secs): 46334.8 total, 651.5 interval >Cumulative writes: 196K writes, 1707M keys, 196K commit groups, 1.0 writes per commit group, ingest: 57.40 GB, 1.27 MB/s >Cumulative WAL: 196K writes, 56K syncs, 3.49 writes per sync, written: 57.40 GB, 1.27 MB/s >Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent >Interval writes: 64 writes, 29M keys, 64 commit groups, 1.0 writes per commit group, ingest: 989.53 MB, 1.52 MB/s >Interval WAL: 64 writes, 32 syncs, 1.94 writes per sync, written: 0.97 MB, 1.52 MB/s >Interval stall: 00:00:0.000 H:M:S, 0.0 percent > >** Compaction Stats [default] ** >Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop >---------------------------------------------------------------------------------------------------------------------------------------------------------- > L0 2/0 269.48 MB 1.1 0.0 0.0 0.0 44.9 44.9 0.0 1.0 0.0 32.4 1420 343 4.139 0 0 > L1 7/0 246.43 MB 1.0 74.1 44.8 29.3 74.0 44.7 0.0 1.7 25.8 25.8 2939 168 17.495 2342M 894K > L2 43/0 2.41 GB 1.0 85.7 44.7 41.0 85.6 44.7 0.0 1.9 21.6 21.6 4055 535 7.580 3191M 1358K > L3 445/0 24.93 GB 1.0 78.4 44.1 34.3 78.4 44.0 0.6 1.8 20.7 20.7 3877 266 14.577 2963M 4000K > L4 3641/0 250.00 GB 1.0 68.1 16.3 51.8 67.8 16.0 28.4 4.2 21.1 21.0 3302 219 15.078 2592M 15M > L5 690/0 47.88 GB 0.0 1.6 0.8 0.8 1.6 0.8 43.5 1.9 18.5 18.2 89 11 8.048 60M 1097K > Sum 4828/0 325.72 GB 0.0 307.9 150.7 157.2 352.3 195.1 72.5 7.8 20.1 23.0 15682 1542 10.170 11G 23M > Int 0/0 0.00 KB 0.0 4.5 2.2 2.3 5.3 3.0 1.1 6.7 15.2 17.9 300 20 14.985 164M 15 >Uptime(secs): 46334.8 total, 46334.8 interval >Flush(GB): cumulative 44.918, interval 0.790 >AddFile(GB): cumulative 0.000, interval 0.000 >AddFile(Total Files): cumulative 0, interval 0 >AddFile(L0 Files): cumulative 0, interval 0 >AddFile(Keys): cumulative 0, interval 0 >Cumulative compaction: 352.27 GB write, 7.79 MB/s write, 307.89 GB read, 6.80 MB/s read, 15682.1 seconds >Interval compaction: 5.25 GB write, 0.12 MB/s write, 4.46 GB read, 0.10 MB/s read, 299.7 seconds >Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count > >** File Read Latency Histogram By Level [default] ** > >** Compaction Stats [default] ** >Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop >---------------------------------------------------------------------------------------------------------------------------------------------------------- > L0 2/0 269.48 MB 1.1 0.0 0.0 0.0 44.9 44.9 0.0 1.0 0.0 32.4 1420 343 4.139 0 0 > L1 7/0 246.43 MB 1.0 74.1 44.8 29.3 74.0 44.7 0.0 1.7 25.8 25.8 2939 168 17.495 2342M 894K > L2 43/0 2.41 GB 1.0 85.7 44.7 41.0 85.6 44.7 0.0 1.9 21.6 21.6 4055 535 7.580 3191M 1358K > L3 445/0 24.93 GB 1.0 78.4 44.1 34.3 78.4 44.0 0.6 1.8 20.7 20.7 3877 266 14.577 2963M 4000K > L4 3641/0 250.00 GB 1.0 68.1 16.3 51.8 67.8 16.0 28.4 4.2 21.1 21.0 3302 219 15.078 2592M 15M > L5 690/0 47.88 GB 0.0 1.6 0.8 0.8 1.6 0.8 43.5 1.9 18.5 18.2 89 11 8.048 60M 1097K > Sum 4828/0 325.72 GB 0.0 307.9 150.7 157.2 352.3 195.1 72.5 7.8 20.1 23.0 15682 1542 10.170 11G 23M > Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 >Uptime(secs): 46334.8 total, 46334.8 interval >Flush(GB): cumulative 44.918, interval 0.000 >AddFile(GB): cumulative 0.000, interval 0.000 >AddFile(Total Files): cumulative 0, interval 0 >AddFile(L0 Files): cumulative 0, interval 0 >AddFile(Keys): cumulative 0, interval 0 >Cumulative compaction: 352.27 GB write, 7.79 MB/s write, 307.89 GB read, 6.80 MB/s read, 15682.1 seconds >Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds >Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count > >** File Read Latency Histogram By Level [default] ** > >2018-07-11 03:59:53.187227 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2588] Compacting 2@0 + 3@1 files to L1, score 1.05 >2018-07-11 03:59:53.187242 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2587 Base level 0, inputs: [35396(134MB) 35394(134MB)], [35371(71MB) 35375(58MB) 35376(62MB)] > >2018-07-11 03:59:53.187279 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281593187263, "job": 2588, "event": "compaction_started", "files_L0": [35396, 35394], "files_L1": [35371, 35375, 35376], "score": 1.05267, "input_data_size": 484259125} >2018-07-11 03:59:53.187549 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 2587] Try to delete WAL files size 178896755, prev total WAL file size 178896767, number of live WAL files 2. > >2018-07-11 03:59:56.387268 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2588] Generated table #35397: 2624610 keys, 74511717 bytes >2018-07-11 03:59:56.387319 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281596387285, "cf_name": "default", "job": 2588, "event": "table_file_creation", "file_number": 35397, "file_size": 74511717, "table_properties": {"data_size": 67110315, "index_size": 691296, "filter_size": 6709116, "raw_key_size": 75851845, "raw_average_key_size": 28, "raw_value_size": 31541243, "raw_average_value_size": 12, "num_data_blocks": 16475, "num_entries": 2624610, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 03:59:59.260227 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2588] Generated table #35398: 2638241 keys, 74546333 bytes >2018-07-11 03:59:59.260280 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281599260252, "cf_name": "default", "job": 2588, "event": "table_file_creation", "file_number": 35398, "file_size": 74546333, "table_properties": {"data_size": 67112184, "index_size": 689976, "filter_size": 6743183, "raw_key_size": 76215859, "raw_average_key_size": 28, "raw_value_size": 31365762, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:00:02.315341 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2588] Generated table #35399: 2638241 keys, 74546879 bytes >2018-07-11 04:00:02.315390 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281602315374, "cf_name": "default", "job": 2588, "event": "table_file_creation", "file_number": 35399, "file_size": 74546879, "table_properties": {"data_size": 67112125, "index_size": 690581, "filter_size": 6743183, "raw_key_size": 76215851, "raw_average_key_size": 28, "raw_value_size": 31365754, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:00:04.382681 7f41641eb700 1 bluefs _allocate failed to allocate 0x4800000 on bdev 1, free 0xbfe000; fallback to bdev 2 >2018-07-11 04:00:05.581360 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2588] Generated table #35400: 2638241 keys, 74546245 bytes >2018-07-11 04:00:05.581406 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281605581391, "cf_name": "default", "job": 2588, "event": "table_file_creation", "file_number": 35400, "file_size": 74546245, "table_properties": {"data_size": 67112151, "index_size": 689921, "filter_size": 6743183, "raw_key_size": 76215856, "raw_average_key_size": 28, "raw_value_size": 31365759, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:00:07.732793 7f41641eb700 1 bluefs _allocate failed to allocate 0x4800000 on bdev 1, free 0xbfe000; fallback to bdev 2 >2018-07-11 04:00:08.787609 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2588] Generated table #35401: 2638241 keys, 74546379 bytes >2018-07-11 04:00:08.787661 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281608787645, "cf_name": "default", "job": 2588, "event": "table_file_creation", "file_number": 35401, "file_size": 74546379, "table_properties": {"data_size": 67112138, "index_size": 690068, "filter_size": 6743183, "raw_key_size": 76215850, "raw_average_key_size": 28, "raw_value_size": 31365753, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:00:10.077946 7f41641eb700 1 bluefs _allocate failed to allocate 0x2c00000 on bdev 1, free 0xbfe000; fallback to bdev 2 >2018-07-11 04:00:10.751021 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2588] Generated table #35402: 1536363 keys, 45584232 bytes >2018-07-11 04:00:10.751063 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281610751037, "cf_name": "default", "job": 2588, "event": "table_file_creation", "file_number": 35402, "file_size": 45584232, "table_properties": {"data_size": 41231716, "index_size": 419676, "filter_size": 3931852, "raw_key_size": 44660527, "raw_average_key_size": 29, "raw_value_size": 20098447, "raw_average_value_size": 13, "num_data_blocks": 10134, "num_entries": 1536363, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:00:11.474875 7f41641eb700 1 bluefs _allocate failed to allocate 0x3f00000 on bdev 1, free 0xbfe000; fallback to bdev 2 >2018-07-11 04:00:12.394304 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2588] Generated table #35403: 162221 keys, 65971237 bytes >2018-07-11 04:00:12.394335 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281612394320, "cf_name": "default", "job": 2588, "event": "table_file_creation", "file_number": 35403, "file_size": 65971237, "table_properties": {"data_size": 64870447, "index_size": 545884, "filter_size": 553918, "raw_key_size": 14984526, "raw_average_key_size": 92, "raw_value_size": 51244770, "raw_average_value_size": 315, "num_data_blocks": 14657, "num_entries": 162221, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "80898", "kMergeOperands": "0"}} >2018-07-11 04:00:12.394411 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2588] Compacted 2@0 + 3@1 files to L1 => 484253022 bytes >2018-07-11 04:00:12.411295 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:00:12.411034) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 11 43 445 3641 690 0] max score 2.02, MB/sec: 25.2 rd, 25.2 wr, level 1, files in(2, 3) out(7) MB in(269.5, 192.3) out(461.8), read-write-amplify(3.4) write-amplify(1.7) OK, records in: 14876165, records dropped: 7 > >2018-07-11 04:00:12.411308 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:00:12.411059) EVENT_LOG_v1 {"time_micros": 1531281612411048, "job": 2588, "event": "compaction_finished", "compaction_time_micros": 19207115, "output_level": 1, "num_output_files": 7, "total_output_size": 484253022, "num_input_records": 14876165, "num_output_records": 14876158, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 11, 43, 445, 3641, 690, 0]} >2018-07-11 04:00:12.412208 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281612412192, "job": 2588, "event": "table_file_deletion", "file_number": 35396} >2018-07-11 04:00:12.412223 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281612412221, "job": 2588, "event": "table_file_deletion", "file_number": 35394} >2018-07-11 04:00:12.412235 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281612412233, "job": 2588, "event": "table_file_deletion", "file_number": 35376} >2018-07-11 04:00:12.412250 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281612412243, "job": 2588, "event": "table_file_deletion", "file_number": 35375} >2018-07-11 04:00:12.412262 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281612412260, "job": 2588, "event": "table_file_deletion", "file_number": 35371} >2018-07-11 04:00:12.412501 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_picker.cc:461] [default] Expanding@1 1+1(74546879+73818914 bytes) to 3+1 (223639503+73818914bytes) > >2018-07-11 04:00:12.412599 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2589] Compacting 3@1 + 1@2 files to L2, score 2.02 >2018-07-11 04:00:12.412610 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2588 Base level 1, inputs: [35399(71MB) 35400(71MB) 35401(71MB)], [35383(70MB)] > >2018-07-11 04:00:12.412624 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281612412613, "job": 2589, "event": "compaction_started", "files_L1": [35399, 35400, 35401], "files_L2": [35383], "score": 2.01525, "input_data_size": 297458417} >2018-07-11 04:00:19.584125 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2589] Generated table #35404: 2638240 keys, 74545837 bytes >2018-07-11 04:00:19.584167 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281619584148, "cf_name": "default", "job": 2589, "event": "table_file_creation", "file_number": 35404, "file_size": 74545837, "table_properties": {"data_size": 67112117, "index_size": 689549, "filter_size": 6743181, "raw_key_size": 76215819, "raw_average_key_size": 28, "raw_value_size": 31365739, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:00:23.239094 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2589] Generated table #35405: 2638242 keys, 74546542 bytes >2018-07-11 04:00:23.239137 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281623239111, "cf_name": "default", "job": 2589, "event": "table_file_creation", "file_number": 35405, "file_size": 74546542, "table_properties": {"data_size": 67112172, "index_size": 690194, "filter_size": 6743186, "raw_key_size": 76215889, "raw_average_key_size": 28, "raw_value_size": 31365775, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638242, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:00:28.458989 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2589] Generated table #35406: 2638241 keys, 74546267 bytes >2018-07-11 04:00:28.459044 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281628459014, "cf_name": "default", "job": 2589, "event": "table_file_creation", "file_number": 35406, "file_size": 74546267, "table_properties": {"data_size": 67112167, "index_size": 689927, "filter_size": 6743183, "raw_key_size": 76215855, "raw_average_key_size": 28, "raw_value_size": 31365758, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:00:32.926664 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2589] Generated table #35407: 2608058 keys, 73818919 bytes >2018-07-11 04:00:32.926707 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281632926689, "cf_name": "default", "job": 2589, "event": "table_file_creation", "file_number": 35407, "file_size": 73818919, "table_properties": {"data_size": 66468834, "index_size": 682776, "filter_size": 6666320, "raw_key_size": 75359639, "raw_average_key_size": 28, "raw_value_size": 31112275, "raw_average_value_size": 11, "num_data_blocks": 16333, "num_entries": 2608058, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "11", "kMergeOperands": "0"}} >2018-07-11 04:00:32.926744 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2589] Compacted 3@1 + 1@2 files to L2 => 297457565 bytes >2018-07-11 04:00:33.127740 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:00:33.127491) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 8 46 445 3641 690 0] max score 1.18, MB/sec: 14.5 rd, 14.5 wr, level 2, files in(3, 1) out(4) MB in(213.3, 70.4) out(283.7), read-write-amplify(2.7) write-amplify(1.3) OK, records in: 10522781, records dropped: 0 > >2018-07-11 04:00:33.127752 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:00:33.127518) EVENT_LOG_v1 {"time_micros": 1531281633127506, "job": 2589, "event": "compaction_finished", "compaction_time_micros": 20514103, "output_level": 2, "num_output_files": 4, "total_output_size": 297457565, "num_input_records": 10522781, "num_output_records": 10522781, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 8, 46, 445, 3641, 690, 0]} >2018-07-11 04:00:33.128720 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281633128714, "job": 2589, "event": "table_file_deletion", "file_number": 35401} >2018-07-11 04:00:33.128747 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281633128745, "job": 2589, "event": "table_file_deletion", "file_number": 35400} >2018-07-11 04:00:33.128759 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281633128756, "job": 2589, "event": "table_file_deletion", "file_number": 35399} >2018-07-11 04:00:33.128775 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281633128773, "job": 2589, "event": "table_file_deletion", "file_number": 35383} >2018-07-11 04:00:33.129095 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2590] Compacting 1@1 + 4@2 files to L2, score 1.18 >2018-07-11 04:00:33.129107 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2589 Base level 1, inputs: [35398(71MB)], [35377(71MB) 35378(71MB) 35379(71MB) 35404(71MB)] > >2018-07-11 04:00:33.129120 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281633129110, "job": 2590, "event": "compaction_started", "files_L1": [35398], "files_L2": [35377, 35378, 35379, 35404], "score": 1.18213, "input_data_size": 372729939} >2018-07-11 04:00:38.029960 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2590] Generated table #35408: 2638238 keys, 74545730 bytes >2018-07-11 04:00:38.030004 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281638029978, "cf_name": "default", "job": 2590, "event": "table_file_creation", "file_number": 35408, "file_size": 74545730, "table_properties": {"data_size": 67112107, "index_size": 689456, "filter_size": 6743177, "raw_key_size": 76215769, "raw_average_key_size": 28, "raw_value_size": 31365723, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638238, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:00:42.908652 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2590] Generated table #35409: 2638240 keys, 74546145 bytes >2018-07-11 04:00:42.908697 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281642908669, "cf_name": "default", "job": 2590, "event": "table_file_creation", "file_number": 35409, "file_size": 74546145, "table_properties": {"data_size": 67112154, "index_size": 689820, "filter_size": 6743181, "raw_key_size": 76215827, "raw_average_key_size": 28, "raw_value_size": 31365747, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:00:47.501203 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2590] Generated table #35410: 2638239 keys, 74545858 bytes >2018-07-11 04:00:47.501253 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281647501220, "cf_name": "default", "job": 2590, "event": "table_file_creation", "file_number": 35410, "file_size": 74545858, "table_properties": {"data_size": 67112126, "index_size": 689563, "filter_size": 6743179, "raw_key_size": 76215794, "raw_average_key_size": 28, "raw_value_size": 31365731, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638239, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:00:51.674676 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2590] Generated table #35411: 2638241 keys, 74545852 bytes >2018-07-11 04:00:51.674720 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281651674703, "cf_name": "default", "job": 2590, "event": "table_file_creation", "file_number": 35411, "file_size": 74545852, "table_properties": {"data_size": 67112126, "index_size": 689553, "filter_size": 6743183, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:00:54.861871 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2590] Generated table #35412: 2638240 keys, 74546274 bytes >2018-07-11 04:00:54.861917 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281654861889, "cf_name": "default", "job": 2590, "event": "table_file_creation", "file_number": 35412, "file_size": 74546274, "table_properties": {"data_size": 67112068, "index_size": 690045, "filter_size": 6743171, "raw_key_size": 76215822, "raw_average_key_size": 28, "raw_value_size": 31365742, "raw_average_value_size": 11, "num_data_blocks": 16489, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:00:54.861950 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2590] Compacted 1@1 + 4@2 files to L2 => 372729859 bytes >2018-07-11 04:00:54.877223 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:00:54.876917) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 7 47 445 3641 690 0] max score 1.07, MB/sec: 17.2 rd, 17.2 wr, level 2, files in(1, 4) out(5) MB in(71.1, 284.4) out(355.5), read-write-amplify(10.0) write-amplify(5.0) OK, records in: 13191198, records dropped: 0 > >2018-07-11 04:00:54.877234 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:00:54.876943) EVENT_LOG_v1 {"time_micros": 1531281654876932, "job": 2590, "event": "compaction_finished", "compaction_time_micros": 21732815, "output_level": 2, "num_output_files": 5, "total_output_size": 372729859, "num_input_records": 13191198, "num_output_records": 13191198, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 7, 47, 445, 3641, 690, 0]} >2018-07-11 04:00:54.878143 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281654878139, "job": 2590, "event": "table_file_deletion", "file_number": 35404} >2018-07-11 04:00:54.878171 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281654878169, "job": 2590, "event": "table_file_deletion", "file_number": 35398} >2018-07-11 04:00:54.878184 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281654878182, "job": 2590, "event": "table_file_deletion", "file_number": 35379} >2018-07-11 04:00:54.878195 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281654878193, "job": 2590, "event": "table_file_deletion", "file_number": 35378} >2018-07-11 04:00:54.878206 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281654878204, "job": 2590, "event": "table_file_deletion", "file_number": 35377} >2018-07-11 04:00:54.878423 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_picker.cc:461] [default] Expanding@2 1+1(74546542+46965395 bytes) to 3+1 (223639083+46965395bytes) > >2018-07-11 04:00:54.878522 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2591] Compacting 3@2 + 1@3 files to L3, score 1.07 >2018-07-11 04:00:54.878533 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2590 Base level 2, inputs: [35412(71MB) 35405(71MB) 35406(71MB)], [35389(44MB)] > >2018-07-11 04:00:54.878546 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281654878536, "job": 2591, "event": "compaction_started", "files_L2": [35412, 35405, 35406], "files_L3": [35389], "score": 1.07366, "input_data_size": 270604478} >2018-07-11 04:00:59.657928 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2591] Generated table #35413: 2638242 keys, 74546641 bytes >2018-07-11 04:00:59.657971 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281659657945, "cf_name": "default", "job": 2591, "event": "table_file_creation", "file_number": 35413, "file_size": 74546641, "table_properties": {"data_size": 67112183, "index_size": 690282, "filter_size": 6743186, "raw_key_size": 76215882, "raw_average_key_size": 28, "raw_value_size": 31365768, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638242, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:01:03.992962 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2591] Generated table #35414: 2638241 keys, 74546171 bytes >2018-07-11 04:01:03.993011 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281663992982, "cf_name": "default", "job": 2591, "event": "table_file_creation", "file_number": 35414, "file_size": 74546171, "table_properties": {"data_size": 67112167, "index_size": 689831, "filter_size": 6743183, "raw_key_size": 76215859, "raw_average_key_size": 28, "raw_value_size": 31365762, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:01:07.940059 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2591] Generated table #35415: 2638241 keys, 74546233 bytes >2018-07-11 04:01:07.940103 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281667940076, "cf_name": "default", "job": 2591, "event": "table_file_creation", "file_number": 35415, "file_size": 74546233, "table_properties": {"data_size": 67112096, "index_size": 689964, "filter_size": 6743183, "raw_key_size": 76215845, "raw_average_key_size": 28, "raw_value_size": 31365748, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:01:10.745938 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2591] Generated table #35416: 1519915 keys, 46965515 bytes >2018-07-11 04:01:10.745991 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281670745975, "cf_name": "default", "job": 2591, "event": "table_file_creation", "file_number": 35416, "file_size": 46965515, "table_properties": {"data_size": 42636637, "index_size": 433742, "filter_size": 3894148, "raw_key_size": 44348915, "raw_average_key_size": 29, "raw_value_size": 21522461, "raw_average_value_size": 14, "num_data_blocks": 10538, "num_entries": 1519915, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:01:10.746029 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2591] Compacted 3@2 + 1@3 files to L3 => 270604560 bytes >2018-07-11 04:01:10.768597 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:01:10.768283) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 7 44 448 3641 690 0] max score 1.01, MB/sec: 17.1 rd, 17.1 wr, level 3, files in(3, 1) out(4) MB in(213.3, 44.8) out(258.1), read-write-amplify(2.4) write-amplify(1.2) OK, records in: 9434639, records dropped: 0 > >2018-07-11 04:01:10.768608 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:01:10.768309) EVENT_LOG_v1 {"time_micros": 1531281670768298, "job": 2591, "event": "compaction_finished", "compaction_time_micros": 15867467, "output_level": 3, "num_output_files": 4, "total_output_size": 270604560, "num_input_records": 9434639, "num_output_records": 9434639, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 7, 44, 448, 3641, 690, 0]} >2018-07-11 04:01:10.769556 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281670769552, "job": 2591, "event": "table_file_deletion", "file_number": 35412} >2018-07-11 04:01:10.769571 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281670769569, "job": 2591, "event": "table_file_deletion", "file_number": 35406} >2018-07-11 04:01:10.769597 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281670769595, "job": 2591, "event": "table_file_deletion", "file_number": 35405} >2018-07-11 04:01:10.769614 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281670769608, "job": 2591, "event": "table_file_deletion", "file_number": 35389} >2018-07-11 04:01:10.785039 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:01:10.769914) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #35413 to level-4 74546641 bytes > >2018-07-11 04:01:10.785049 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:01:10.784945) EVENT_LOG_v1 {"time_micros": 1531281670784939, "job": 2592, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 74546641} >2018-07-11 04:01:10.785053 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:01:10.784953) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-4 74546641 bytes OK: base level 1 max bytes base 268435456 files[0 7 44 447 3642 690 0] max score 1.00 > >2018-07-11 04:01:10.801832 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:01:10.785876) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #35415 to level-4 74546233 bytes > >2018-07-11 04:01:10.801846 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:01:10.801701) EVENT_LOG_v1 {"time_micros": 1531281670801691, "job": 2593, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 74546233} >2018-07-11 04:01:10.801848 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:01:10.801715) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-4 74546233 bytes OK: base level 1 max bytes base 268435456 files[0 7 44 446 3643 690 0] max score 1.00 > >2018-07-11 04:01:10.818442 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:01:10.803061) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #21408 to level-5 74546864 bytes > >2018-07-11 04:01:10.818453 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:01:10.818343) EVENT_LOG_v1 {"time_micros": 1531281670818336, "job": 2594, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546864} >2018-07-11 04:01:10.818456 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:01:10.818352) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546864 bytes OK: base level 1 max bytes base 268435456 files[0 7 44 446 3642 691 0] max score 1.00 > >2018-07-11 04:01:10.835159 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:01:10.819304) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #1636 to level-5 74546864 bytes > >2018-07-11 04:01:10.835173 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:01:10.835043) EVENT_LOG_v1 {"time_micros": 1531281670835032, "job": 2595, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546864} >2018-07-11 04:01:10.835176 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:01:10.835059) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546864 bytes OK: base level 1 max bytes base 268435456 files[0 7 44 446 3641 692 0] max score 1.00 > >2018-07-11 04:01:11.168243 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:01:10.836645) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #35414 to level-4 74546171 bytes > >2018-07-11 04:01:11.168263 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:01:11.168127) EVENT_LOG_v1 {"time_micros": 1531281671168120, "job": 2596, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 74546171} >2018-07-11 04:01:11.168266 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:01:11.168139) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-4 74546171 bytes OK: base level 1 max bytes base 268435456 files[0 7 44 445 3642 692 0] max score 1.00 > >2018-07-11 04:01:11.185102 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:01:11.169447) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #14491 to level-5 74546863 bytes > >2018-07-11 04:01:11.185112 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:01:11.185003) EVENT_LOG_v1 {"time_micros": 1531281671184997, "job": 2597, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546863} >2018-07-11 04:01:11.185115 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:01:11.185012) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546863 bytes OK: base level 1 max bytes base 268435456 files[0 7 44 445 3641 693 0] max score 1.00 > >2018-07-11 04:02:21.461473 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:684] reusing log 35393 from recycle list > >2018-07-11 04:02:21.461643 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #35417. Immutable memtables: 0. > >2018-07-11 04:02:21.540713 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:02:21.461706) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1 >2018-07-11 04:02:21.540726 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 2598] Flushing memtable with next log file: 35417 > >2018-07-11 04:02:21.540741 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281741540733, "job": 2598, "event": "flush_started", "num_memtables": 1, "num_entries": 5000015, "num_deletes": 0, "memory_usage": 292936984} >2018-07-11 04:02:21.540744 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 2598] Level-0 flush table #35418: started >2018-07-11 04:02:24.215343 7f41649ec700 1 bluefs _allocate failed to allocate 0x8700000 on bdev 1, free 0xbfe000; fallback to bdev 2 >2018-07-11 04:02:26.387187 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281746387167, "cf_name": "default", "job": 2598, "event": "table_file_creation", "file_number": 35418, "file_size": 141286889, "table_properties": {"data_size": 127197777, "index_size": 1308397, "filter_size": 12779725, "raw_key_size": 144445101, "raw_average_key_size": 28, "raw_value_size": 59450671, "raw_average_value_size": 11, "num_data_blocks": 31252, "num_entries": 5000014, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:02:26.387209 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 2598] Level-0 flush table #35418: 141286889 bytes OK >2018-07-11 04:02:26.408352 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:242] adding log 35395 to recycle list > >2018-07-11 04:02:26.408363 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:02:26.387234) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #35418 started >2018-07-11 04:02:26.408379 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:02:26.408116) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #35418: memtable #1 done >2018-07-11 04:02:26.408382 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:02:26.408133) EVENT_LOG_v1 {"time_micros": 1531281746408127, "job": 2598, "event": "flush_finished", "lsm_state": [1, 7, 44, 445, 3641, 693, 0], "immutable_memtables": 0} >2018-07-11 04:02:26.408384 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:02:26.408301) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 7 44 445 3641 693 0] max score 1.00 > >2018-07-11 04:02:26.409250 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 2598] Try to delete WAL files size 178896755, prev total WAL file size 178896767, number of live WAL files 2. > >2018-07-11 04:04:14.035922 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:684] reusing log 35395 from recycle list > >2018-07-11 04:04:14.036036 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #35419. Immutable memtables: 0. > >2018-07-11 04:04:14.709622 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:04:14.036101) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1 >2018-07-11 04:04:14.709635 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 2599] Flushing memtable with next log file: 35419 > >2018-07-11 04:04:14.709651 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281854709643, "job": 2599, "event": "flush_started", "num_memtables": 1, "num_entries": 5000015, "num_deletes": 0, "memory_usage": 292946872} >2018-07-11 04:04:14.709655 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 2599] Level-0 flush table #35420: started >2018-07-11 04:04:17.416091 7f41649ec700 1 bluefs _allocate failed to allocate 0x8700000 on bdev 1, free 0xbfe000; fallback to bdev 2 >2018-07-11 04:04:19.537046 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281859537005, "cf_name": "default", "job": 2599, "event": "table_file_creation", "file_number": 35420, "file_size": 141288967, "table_properties": {"data_size": 127198839, "index_size": 1309411, "filter_size": 12779727, "raw_key_size": 144445125, "raw_average_key_size": 28, "raw_value_size": 59451570, "raw_average_value_size": 11, "num_data_blocks": 31252, "num_entries": 5000015, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:04:19.537079 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 2599] Level-0 flush table #35420: 141288967 bytes OK >2018-07-11 04:04:20.105237 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:242] adding log 35417 to recycle list > >2018-07-11 04:04:20.105256 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:04:19.537092) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #35420 started >2018-07-11 04:04:20.105260 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:04:20.105076) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #35420: memtable #1 done >2018-07-11 04:04:20.105262 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:04:20.105092) EVENT_LOG_v1 {"time_micros": 1531281860105085, "job": 2599, "event": "flush_finished", "lsm_state": [2, 7, 44, 445, 3641, 693, 0], "immutable_memtables": 0} >2018-07-11 04:04:20.105264 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:04:20.105183) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[2 7 44 445 3641 693 0] max score 1.05 > >2018-07-11 04:04:20.105387 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2600] Compacting 2@0 + 3@1 files to L1, score 1.05 >2018-07-11 04:04:20.105404 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2599 Base level 0, inputs: [35420(134MB) 35418(134MB)], [35397(71MB) 35402(43MB) 35403(62MB)] > >2018-07-11 04:04:20.105421 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281860105408, "job": 2600, "event": "compaction_started", "files_L0": [35420, 35418], "files_L1": [35397, 35402, 35403], "score": 1.05268, "input_data_size": 468643042} >2018-07-11 04:04:20.106141 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 2599] Try to delete WAL files size 178896755, prev total WAL file size 178896767, number of live WAL files 2. > >2018-07-11 04:04:23.127758 7f41641eb700 1 bluefs _allocate failed to allocate 0x4800000 on bdev 1, free 0xbfe000; fallback to bdev 2 >2018-07-11 04:04:24.111683 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2600] Generated table #35421: 2624540 keys, 74511229 bytes >2018-07-11 04:04:24.111737 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281864111721, "cf_name": "default", "job": 2600, "event": "table_file_creation", "file_number": 35421, "file_size": 74511229, "table_properties": {"data_size": 67110048, "index_size": 691249, "filter_size": 6708942, "raw_key_size": 75850034, "raw_average_key_size": 28, "raw_value_size": 31541912, "raw_average_value_size": 12, "num_data_blocks": 16475, "num_entries": 2624540, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:04:27.339784 7f41641eb700 1 bluefs _allocate failed to allocate 0x4800000 on bdev 1, free 0xbfe000; fallback to bdev 2 >2018-07-11 04:04:28.530570 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2600] Generated table #35422: 2638241 keys, 74545960 bytes >2018-07-11 04:04:28.530619 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281868530602, "cf_name": "default", "job": 2600, "event": "table_file_creation", "file_number": 35422, "file_size": 74545960, "table_properties": {"data_size": 67112133, "index_size": 689654, "filter_size": 6743183, "raw_key_size": 76215855, "raw_average_key_size": 28, "raw_value_size": 31365758, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:04:32.226434 7f41641eb700 1 bluefs _allocate failed to allocate 0x4800000 on bdev 1, free 0xbfe000; fallback to bdev 2 >2018-07-11 04:04:33.513719 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2600] Generated table #35423: 2638241 keys, 74547055 bytes >2018-07-11 04:04:33.513754 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281873513738, "cf_name": "default", "job": 2600, "event": "table_file_creation", "file_number": 35423, "file_size": 74547055, "table_properties": {"data_size": 67112158, "index_size": 690724, "filter_size": 6743183, "raw_key_size": 76215859, "raw_average_key_size": 28, "raw_value_size": 31365762, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:04:36.753380 7f41641eb700 1 bluefs _allocate failed to allocate 0x4800000 on bdev 1, free 0xbfe000; fallback to bdev 2 >2018-07-11 04:04:37.639287 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2600] Generated table #35424: 2638241 keys, 74546210 bytes >2018-07-11 04:04:37.639328 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281877639310, "cf_name": "default", "job": 2600, "event": "table_file_creation", "file_number": 35424, "file_size": 74546210, "table_properties": {"data_size": 67112097, "index_size": 689940, "filter_size": 6743183, "raw_key_size": 76215850, "raw_average_key_size": 28, "raw_value_size": 31365753, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:04:39.831361 7f417420b700 1 bluestore(/var/lib/ceph/osd/ceph-10) _balance_bluefs_freespace gifting 0xcf59e00000~80000000 to bluefs >2018-07-11 04:04:39.912782 7f417420b700 1 bluefs add_block_extent bdev 2 0xcf59e00000~80000000 >2018-07-11 04:04:41.658275 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2600] Generated table #35425: 2638241 keys, 74546314 bytes >2018-07-11 04:04:41.658316 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281881658290, "cf_name": "default", "job": 2600, "event": "table_file_creation", "file_number": 35425, "file_size": 74546314, "table_properties": {"data_size": 67112160, "index_size": 689981, "filter_size": 6743183, "raw_key_size": 76215852, "raw_average_key_size": 28, "raw_value_size": 31365755, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:04:43.644613 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2600] Generated table #35426: 979588 keys, 27679399 bytes >2018-07-11 04:04:43.644664 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281883644648, "cf_name": "default", "job": 2600, "event": "table_file_creation", "file_number": 35426, "file_size": 27679399, "table_properties": {"data_size": 24918976, "index_size": 255669, "filter_size": 2503766, "raw_key_size": 28299222, "raw_average_key_size": 28, "raw_value_size": 11646226, "raw_average_value_size": 11, "num_data_blocks": 6123, "num_entries": 979588, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:04:45.886891 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2600] Generated table #35427: 166111 keys, 68248473 bytes >2018-07-11 04:04:45.886934 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281885886919, "cf_name": "default", "job": 2600, "event": "table_file_creation", "file_number": 35427, "file_size": 68248473, "table_properties": {"data_size": 67113646, "index_size": 564984, "filter_size": 568855, "raw_key_size": 15373293, "raw_average_key_size": 92, "raw_value_size": 53118732, "raw_average_value_size": 319, "num_data_blocks": 15212, "num_entries": 166111, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "80898", "kMergeOperands": "0"}} >2018-07-11 04:04:45.922108 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2600] Generated table #35428: 11 keys, 10568 bytes >2018-07-11 04:04:45.922146 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281885922121, "cf_name": "default", "job": 2600, "event": "table_file_creation", "file_number": 35428, "file_size": 10568, "table_properties": {"data_size": 9381, "index_size": 157, "filter_size": 62, "raw_key_size": 943, "raw_average_key_size": 85, "raw_value_size": 8712, "raw_average_value_size": 792, "num_data_blocks": 3, "num_entries": 11, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:04:45.922211 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2600] Compacted 2@0 + 3@1 files to L1 => 468635208 bytes >2018-07-11 04:04:45.938035 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:04:45.937811) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 12 44 445 3641 693 0] max score 1.96, MB/sec: 18.2 rd, 18.2 wr, level 1, files in(2, 3) out(8) MB in(269.5, 177.4) out(446.9), read-write-amplify(3.3) write-amplify(1.7) OK, records in: 14323223, records dropped: 9 > >2018-07-11 04:04:45.938046 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:04:45.937836) EVENT_LOG_v1 {"time_micros": 1531281885937825, "job": 2600, "event": "compaction_finished", "compaction_time_micros": 25816772, "output_level": 1, "num_output_files": 8, "total_output_size": 468635208, "num_input_records": 14323223, "num_output_records": 14323214, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 12, 44, 445, 3641, 693, 0]} >2018-07-11 04:04:45.938964 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281885938947, "job": 2600, "event": "table_file_deletion", "file_number": 35420} >2018-07-11 04:04:45.938980 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281885938977, "job": 2600, "event": "table_file_deletion", "file_number": 35418} >2018-07-11 04:04:45.938995 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281885938993, "job": 2600, "event": "table_file_deletion", "file_number": 35403} >2018-07-11 04:04:45.939006 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281885939004, "job": 2600, "event": "table_file_deletion", "file_number": 35402} >2018-07-11 04:04:45.939017 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281885939015, "job": 2600, "event": "table_file_deletion", "file_number": 35397} >2018-07-11 04:04:45.939255 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_picker.cc:461] [default] Expanding@1 1+1(74547055+73818919 bytes) to 4+1 (251318978+73818919bytes) > >2018-07-11 04:04:45.939353 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2601] Compacting 4@1 + 1@2 files to L2, score 1.96 >2018-07-11 04:04:45.939365 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2600 Base level 1, inputs: [35423(71MB) 35424(71MB) 35425(71MB) 35426(26MB)], [35407(70MB)] > >2018-07-11 04:04:45.939379 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281885939369, "job": 2601, "event": "compaction_started", "files_L1": [35423, 35424, 35425, 35426], "files_L2": [35407], "score": 1.95707, "input_data_size": 325137897} >2018-07-11 04:04:51.292564 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2601] Generated table #35429: 2638240 keys, 74545844 bytes >2018-07-11 04:04:51.292617 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281891292600, "cf_name": "default", "job": 2601, "event": "table_file_creation", "file_number": 35429, "file_size": 74545844, "table_properties": {"data_size": 67112088, "index_size": 689585, "filter_size": 6743181, "raw_key_size": 76215816, "raw_average_key_size": 28, "raw_value_size": 31365736, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:04:56.474189 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2601] Generated table #35430: 2638241 keys, 74546383 bytes >2018-07-11 04:04:56.474237 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281896474221, "cf_name": "default", "job": 2601, "event": "table_file_creation", "file_number": 35430, "file_size": 74546383, "table_properties": {"data_size": 67112177, "index_size": 690033, "filter_size": 6743183, "raw_key_size": 76215861, "raw_average_key_size": 28, "raw_value_size": 31365764, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:05:00.688978 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2601] Generated table #35431: 2638241 keys, 74546921 bytes >2018-07-11 04:05:00.689025 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281900689006, "cf_name": "default", "job": 2601, "event": "table_file_creation", "file_number": 35431, "file_size": 74546921, "table_properties": {"data_size": 67112129, "index_size": 690619, "filter_size": 6743183, "raw_key_size": 76215850, "raw_average_key_size": 28, "raw_value_size": 31365753, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:05:03.946140 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2601] Generated table #35432: 2638241 keys, 74546268 bytes >2018-07-11 04:05:03.946185 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281903946157, "cf_name": "default", "job": 2601, "event": "table_file_creation", "file_number": 35432, "file_size": 74546268, "table_properties": {"data_size": 67112147, "index_size": 689948, "filter_size": 6743183, "raw_key_size": 76215855, "raw_average_key_size": 28, "raw_value_size": 31365758, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:05:05.116425 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2601] Generated table #35433: 949406 keys, 26952796 bytes >2018-07-11 04:05:05.116467 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281905116453, "cf_name": "default", "job": 2601, "event": "table_file_creation", "file_number": 35433, "file_size": 26952796, "table_properties": {"data_size": 24275765, "index_size": 249118, "filter_size": 2426925, "raw_key_size": 27443040, "raw_average_key_size": 28, "raw_value_size": 11392760, "raw_average_value_size": 11, "num_data_blocks": 5967, "num_entries": 949406, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "11", "kMergeOperands": "0"}} >2018-07-11 04:05:05.116502 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2601] Compacted 4@1 + 1@2 files to L2 => 325138212 bytes >2018-07-11 04:05:05.137807 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:05:05.137488) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 8 48 445 3641 693 0] max score 1.08, MB/sec: 17.0 rd, 17.0 wr, level 2, files in(4, 1) out(5) MB in(239.7, 70.4) out(310.1), read-write-amplify(2.6) write-amplify(1.3) OK, records in: 11502369, records dropped: 0 > >2018-07-11 04:05:05.137819 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:05:05.137514) EVENT_LOG_v1 {"time_micros": 1531281905137503, "job": 2601, "event": "compaction_finished", "compaction_time_micros": 19177108, "output_level": 2, "num_output_files": 5, "total_output_size": 325138212, "num_input_records": 11502369, "num_output_records": 11502369, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 8, 48, 445, 3641, 693, 0]} >2018-07-11 04:05:05.138758 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281905138741, "job": 2601, "event": "table_file_deletion", "file_number": 35426} >2018-07-11 04:05:05.138772 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281905138770, "job": 2601, "event": "table_file_deletion", "file_number": 35425} >2018-07-11 04:05:05.138785 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281905138783, "job": 2601, "event": "table_file_deletion", "file_number": 35424} >2018-07-11 04:05:05.138797 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281905138795, "job": 2601, "event": "table_file_deletion", "file_number": 35423} >2018-07-11 04:05:05.138812 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281905138810, "job": 2601, "event": "table_file_deletion", "file_number": 35407} >2018-07-11 04:05:05.139034 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_picker.cc:461] [default] Expanding@2 1+1(74546921+46965515 bytes) to 4+1 (298185416+46965515bytes) > >2018-07-11 04:05:05.139133 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2602] Compacting 4@2 + 1@3 files to L3, score 1.08 >2018-07-11 04:05:05.139145 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2601 Base level 2, inputs: [35429(71MB) 35430(71MB) 35431(71MB) 35432(71MB)], [35416(44MB)] > >2018-07-11 04:05:05.139160 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281905139149, "job": 2602, "event": "compaction_started", "files_L2": [35429, 35430, 35431, 35432], "files_L3": [35416], "score": 1.08398, "input_data_size": 345150931} >2018-07-11 04:05:10.035168 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2602] Generated table #35434: 2638241 keys, 74546980 bytes >2018-07-11 04:05:10.035214 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281910035186, "cf_name": "default", "job": 2602, "event": "table_file_creation", "file_number": 35434, "file_size": 74546980, "table_properties": {"data_size": 67112168, "index_size": 690639, "filter_size": 6743183, "raw_key_size": 76215865, "raw_average_key_size": 28, "raw_value_size": 31365768, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:05:15.168691 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2602] Generated table #35435: 2638240 keys, 74546057 bytes >2018-07-11 04:05:15.168741 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281915168712, "cf_name": "default", "job": 2602, "event": "table_file_creation", "file_number": 35435, "file_size": 74546057, "table_properties": {"data_size": 67112111, "index_size": 689775, "filter_size": 6743181, "raw_key_size": 76215818, "raw_average_key_size": 28, "raw_value_size": 31365738, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:05:20.473519 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2602] Generated table #35436: 2638241 keys, 74546276 bytes >2018-07-11 04:05:20.473559 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281920473541, "cf_name": "default", "job": 2602, "event": "table_file_creation", "file_number": 35436, "file_size": 74546276, "table_properties": {"data_size": 67112144, "index_size": 689959, "filter_size": 6743183, "raw_key_size": 76215855, "raw_average_key_size": 28, "raw_value_size": 31365758, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:05:25.147239 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2602] Generated table #35437: 2638241 keys, 74545737 bytes >2018-07-11 04:05:25.147287 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281925147260, "cf_name": "default", "job": 2602, "event": "table_file_creation", "file_number": 35437, "file_size": 74545737, "table_properties": {"data_size": 67112141, "index_size": 689423, "filter_size": 6743183, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:05:27.827064 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2602] Generated table #35438: 1519915 keys, 46965472 bytes >2018-07-11 04:05:27.827105 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281927827078, "cf_name": "default", "job": 2602, "event": "table_file_creation", "file_number": 35438, "file_size": 46965472, "table_properties": {"data_size": 42636615, "index_size": 433721, "filter_size": 3894148, "raw_key_size": 44348905, "raw_average_key_size": 29, "raw_value_size": 21522451, "raw_average_value_size": 14, "num_data_blocks": 10538, "num_entries": 1519915, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:05:27.827135 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2602] Compacted 4@2 + 1@3 files to L3 => 345150522 bytes >2018-07-11 04:05:27.845725 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:05:27.845458) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 8 44 449 3641 693 0] max score 1.02, MB/sec: 15.2 rd, 15.2 wr, level 3, files in(4, 1) out(5) MB in(284.4, 44.8) out(329.2), read-write-amplify(2.3) write-amplify(1.2) OK, records in: 12072878, records dropped: 0 > >2018-07-11 04:05:27.845734 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:05:27.845483) EVENT_LOG_v1 {"time_micros": 1531281927845472, "job": 2602, "event": "compaction_finished", "compaction_time_micros": 22687959, "output_level": 3, "num_output_files": 5, "total_output_size": 345150522, "num_input_records": 12072878, "num_output_records": 12072878, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 8, 44, 449, 3641, 693, 0]} >2018-07-11 04:05:27.846668 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281927846664, "job": 2602, "event": "table_file_deletion", "file_number": 35432} >2018-07-11 04:05:27.846694 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281927846691, "job": 2602, "event": "table_file_deletion", "file_number": 35431} >2018-07-11 04:05:27.846705 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281927846703, "job": 2602, "event": "table_file_deletion", "file_number": 35430} >2018-07-11 04:05:27.846716 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281927846714, "job": 2602, "event": "table_file_deletion", "file_number": 35429} >2018-07-11 04:05:27.846731 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281927846726, "job": 2602, "event": "table_file_deletion", "file_number": 35416} >2018-07-11 04:05:27.847023 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2603] Compacting 1@1 + 4@2 files to L2, score 1.02 >2018-07-11 04:05:27.847046 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2602 Base level 1, inputs: [35422(71MB)], [35408(71MB) 35409(71MB) 35410(71MB) 35411(71MB)] > >2018-07-11 04:05:27.847060 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281927847051, "job": 2603, "event": "compaction_started", "files_L1": [35422], "files_L2": [35408, 35409, 35410, 35411], "score": 1.02084, "input_data_size": 372729545} >2018-07-11 04:05:33.064663 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2603] Generated table #35439: 2638238 keys, 74545701 bytes >2018-07-11 04:05:33.064708 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281933064681, "cf_name": "default", "job": 2603, "event": "table_file_creation", "file_number": 35439, "file_size": 74545701, "table_properties": {"data_size": 67112107, "index_size": 689427, "filter_size": 6743177, "raw_key_size": 76215768, "raw_average_key_size": 28, "raw_value_size": 31365722, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638238, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:05:35.763942 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:684] reusing log 35417 from recycle list > >2018-07-11 04:05:35.764053 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #35441. Immutable memtables: 0. > >2018-07-11 04:05:35.868750 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:05:35.764120) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 0 >2018-07-11 04:05:35.868763 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 2604] Flushing memtable with next log file: 35441 > >2018-07-11 04:05:35.868779 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281935868770, "job": 2604, "event": "flush_started", "num_memtables": 1, "num_entries": 5000016, "num_deletes": 0, "memory_usage": 292950424} >2018-07-11 04:05:35.868782 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 2604] Level-0 flush table #35442: started >2018-07-11 04:05:38.503353 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2603] Generated table #35440: 2638241 keys, 74546217 bytes >2018-07-11 04:05:38.503407 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281938503390, "cf_name": "default", "job": 2603, "event": "table_file_creation", "file_number": 35440, "file_size": 74546217, "table_properties": {"data_size": 67112114, "index_size": 689930, "filter_size": 6743183, "raw_key_size": 76215857, "raw_average_key_size": 28, "raw_value_size": 31365760, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:05:39.848350 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281939848331, "cf_name": "default", "job": 2604, "event": "table_file_creation", "file_number": 35442, "file_size": 141289677, "table_properties": {"data_size": 127199543, "index_size": 1309411, "filter_size": 12779733, "raw_key_size": 144445149, "raw_average_key_size": 28, "raw_value_size": 59452246, "raw_average_value_size": 11, "num_data_blocks": 31252, "num_entries": 5000016, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:05:39.848370 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 2604] Level-0 flush table #35442: 141289677 bytes OK >2018-07-11 04:05:39.870279 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:242] adding log 35419 to recycle list > >2018-07-11 04:05:39.870291 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:05:39.848382) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #35442 started >2018-07-11 04:05:39.870308 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:05:39.870148) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #35442: memtable #1 done >2018-07-11 04:05:39.870311 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:05:39.870165) EVENT_LOG_v1 {"time_micros": 1531281939870158, "job": 2604, "event": "flush_finished", "lsm_state": [1, 8, 44, 449, 3641, 693, 0], "immutable_memtables": 0} >2018-07-11 04:05:39.870313 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:05:39.870182) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 8 44 449 3641 693 0] max score 1.01 > >2018-07-11 04:05:39.871394 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 2604] Try to delete WAL files size 178897451, prev total WAL file size 178897463, number of live WAL files 2. > >2018-07-11 04:05:42.808569 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2603] Generated table #35443: 2638238 keys, 74545853 bytes >2018-07-11 04:05:42.808614 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281942808586, "cf_name": "default", "job": 2603, "event": "table_file_creation", "file_number": 35443, "file_size": 74545853, "table_properties": {"data_size": 67112111, "index_size": 689575, "filter_size": 6743177, "raw_key_size": 76215765, "raw_average_key_size": 28, "raw_value_size": 31365719, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638238, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:05:47.354568 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2603] Generated table #35444: 2638241 keys, 74545835 bytes >2018-07-11 04:05:47.354611 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281947354585, "cf_name": "default", "job": 2603, "event": "table_file_creation", "file_number": 35444, "file_size": 74545835, "table_properties": {"data_size": 67112125, "index_size": 689537, "filter_size": 6743183, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:05:52.247634 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2603] Generated table #35445: 2638241 keys, 74545960 bytes >2018-07-11 04:05:52.247678 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281952247650, "cf_name": "default", "job": 2603, "event": "table_file_creation", "file_number": 35445, "file_size": 74545960, "table_properties": {"data_size": 67112133, "index_size": 689654, "filter_size": 6743183, "raw_key_size": 76215855, "raw_average_key_size": 28, "raw_value_size": 31365758, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:05:52.247712 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2603] Compacted 1@1 + 4@2 files to L2 => 372729566 bytes >2018-07-11 04:05:52.270431 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:05:52.270041) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[1 7 45 449 3641 693 0] max score 1.01, MB/sec: 15.3 rd, 15.3 wr, level 2, files in(1, 4) out(5) MB in(71.1, 284.4) out(355.5), read-write-amplify(10.0) write-amplify(5.0) OK, records in: 13191199, records dropped: 0 > >2018-07-11 04:05:52.270440 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:05:52.270066) EVENT_LOG_v1 {"time_micros": 1531281952270055, "job": 2603, "event": "compaction_finished", "compaction_time_micros": 24400638, "output_level": 2, "num_output_files": 5, "total_output_size": 372729566, "num_input_records": 13191199, "num_output_records": 13191199, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 7, 45, 449, 3641, 693, 0]} >2018-07-11 04:05:52.271360 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281952271355, "job": 2603, "event": "table_file_deletion", "file_number": 35422} >2018-07-11 04:05:52.271382 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281952271379, "job": 2603, "event": "table_file_deletion", "file_number": 35411} >2018-07-11 04:05:52.271397 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281952271395, "job": 2603, "event": "table_file_deletion", "file_number": 35410} >2018-07-11 04:05:52.271411 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281952271409, "job": 2603, "event": "table_file_deletion", "file_number": 35409} >2018-07-11 04:05:52.271425 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281952271422, "job": 2603, "event": "table_file_deletion", "file_number": 35408} >2018-07-11 04:05:52.295190 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:05:52.271742) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #35434 to level-4 74546980 bytes > >2018-07-11 04:05:52.295201 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:05:52.295100) EVENT_LOG_v1 {"time_micros": 1531281952295094, "job": 2605, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 74546980} >2018-07-11 04:05:52.295216 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:05:52.295108) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-4 74546980 bytes OK: base level 1 max bytes base 268435456 files[1 7 45 448 3642 693 0] max score 1.01 > >2018-07-11 04:05:52.320187 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:05:52.296050) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #35436 to level-4 74546276 bytes > >2018-07-11 04:05:52.320197 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:05:52.320098) EVENT_LOG_v1 {"time_micros": 1531281952320092, "job": 2606, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 74546276} >2018-07-11 04:05:52.320200 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:05:52.320106) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-4 74546276 bytes OK: base level 1 max bytes base 268435456 files[1 7 45 447 3643 693 0] max score 1.00 > >2018-07-11 04:05:52.345274 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:05:52.321034) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #35386 to level-4 74546114 bytes > >2018-07-11 04:05:52.345284 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:05:52.345168) EVENT_LOG_v1 {"time_micros": 1531281952345161, "job": 2607, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 74546114} >2018-07-11 04:05:52.345287 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:05:52.345176) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-4 74546114 bytes OK: base level 1 max bytes base 268435456 files[1 7 45 446 3644 693 0] max score 1.00 > >2018-07-11 04:05:52.487028 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:05:52.346081) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #35434 to level-5 74546980 bytes > >2018-07-11 04:05:52.487038 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:05:52.486855) EVENT_LOG_v1 {"time_micros": 1531281952486849, "job": 2608, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546980} >2018-07-11 04:05:52.487041 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:05:52.486863) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546980 bytes OK: base level 1 max bytes base 268435456 files[1 7 45 446 3643 694 0] max score 1.00 > >2018-07-11 04:05:52.487923 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2609] Compacting 1@2 + 1@3 files to L3, score 1.00 >2018-07-11 04:05:52.487934 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2608 Base level 2, inputs: [35440(71MB)], [34906(71MB)] > >2018-07-11 04:05:52.487960 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281952487937, "job": 2609, "event": "compaction_started", "files_L2": [35440], "files_L3": [34906], "score": 1.00066, "input_data_size": 149092268} >2018-07-11 04:05:57.063533 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2609] Generated table #35446: 2638241 keys, 74546111 bytes >2018-07-11 04:05:57.063577 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281957063550, "cf_name": "default", "job": 2609, "event": "table_file_creation", "file_number": 35446, "file_size": 74546111, "table_properties": {"data_size": 67112153, "index_size": 689784, "filter_size": 6743184, "raw_key_size": 76215852, "raw_average_key_size": 28, "raw_value_size": 31365755, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:06:01.193796 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2609] Generated table #35447: 2638240 keys, 74546145 bytes >2018-07-11 04:06:01.193839 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281961193812, "cf_name": "default", "job": 2609, "event": "table_file_creation", "file_number": 35447, "file_size": 74546145, "table_properties": {"data_size": 67112158, "index_size": 689816, "filter_size": 6743181, "raw_key_size": 76215831, "raw_average_key_size": 28, "raw_value_size": 31365751, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:06:01.247631 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2609] Generated table #35448: 1 keys, 1064 bytes >2018-07-11 04:06:01.247655 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281961247643, "cf_name": "default", "job": 2609, "event": "table_file_creation", "file_number": 35448, "file_size": 1064, "table_properties": {"data_size": 57, "index_size": 27, "filter_size": 18, "raw_key_size": 29, "raw_average_key_size": 29, "raw_value_size": 12, "raw_average_value_size": 12, "num_data_blocks": 1, "num_entries": 1, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:06:01.247689 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2609] Compacted 1@2 + 1@3 files to L3 => 149093320 bytes >2018-07-11 04:06:01.270324 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:06:01.270057) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[1 7 44 448 3643 694 0] max score 1.00, MB/sec: 17.0 rd, 17.0 wr, level 3, files in(1, 1) out(3) MB in(71.1, 71.1) out(142.2), read-write-amplify(4.0) write-amplify(2.0) OK, records in: 5276482, records dropped: 0 > >2018-07-11 04:06:01.270335 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:06:01.270081) EVENT_LOG_v1 {"time_micros": 1531281961270071, "job": 2609, "event": "compaction_finished", "compaction_time_micros": 8759704, "output_level": 3, "num_output_files": 3, "total_output_size": 149093320, "num_input_records": 5276482, "num_output_records": 5276482, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 7, 44, 448, 3643, 694, 0]} >2018-07-11 04:06:01.271253 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281961271243, "job": 2609, "event": "table_file_deletion", "file_number": 35440} >2018-07-11 04:06:01.271270 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281961271267, "job": 2609, "event": "table_file_deletion", "file_number": 34906} >2018-07-11 04:06:01.271493 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_picker.cc:461] [default] Expanding@3 1+2(74546145+149092654 bytes) to 2+2 (74547209+149092654bytes) > >2018-07-11 04:06:01.271591 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2610] Compacting 2@3 + 2@4 files to L4, score 1.00 >2018-07-11 04:06:01.271603 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2609 Base level 3, inputs: [35447(71MB) 35448(1064B)], [34907(71MB) 35354(71MB)] > >2018-07-11 04:06:01.271617 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281961271607, "job": 2610, "event": "compaction_started", "files_L3": [35447, 35448], "files_L4": [34907, 35354], "score": 1.00291, "input_data_size": 223639863} >2018-07-11 04:06:05.558153 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2610] Generated table #35449: 2638241 keys, 74545963 bytes >2018-07-11 04:06:05.558182 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281965558168, "cf_name": "default", "job": 2610, "event": "table_file_creation", "file_number": 35449, "file_size": 74545963, "table_properties": {"data_size": 67112149, "index_size": 689641, "filter_size": 6743183, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:06:10.278094 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2610] Generated table #35450: 2638241 keys, 74546963 bytes >2018-07-11 04:06:10.278139 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281970278124, "cf_name": "default", "job": 2610, "event": "table_file_creation", "file_number": 35450, "file_size": 74546963, "table_properties": {"data_size": 67112095, "index_size": 690695, "filter_size": 6743183, "raw_key_size": 76215846, "raw_average_key_size": 28, "raw_value_size": 31365749, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:06:14.869662 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2610] Generated table #35451: 2638241 keys, 74546562 bytes >2018-07-11 04:06:14.869707 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281974869681, "cf_name": "default", "job": 2610, "event": "table_file_creation", "file_number": 35451, "file_size": 74546562, "table_properties": {"data_size": 67112147, "index_size": 690242, "filter_size": 6743183, "raw_key_size": 76215867, "raw_average_key_size": 28, "raw_value_size": 31365770, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:06:14.869741 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2610] Compacted 2@3 + 2@4 files to L4 => 223639488 bytes >2018-07-11 04:06:14.886671 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:06:14.886449) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[1 7 44 446 3644 694 0] max score 1.00, MB/sec: 16.4 rd, 16.4 wr, level 4, files in(2, 2) out(3) MB in(71.1, 142.2) out(213.3), read-write-amplify(6.0) write-amplify(3.0) OK, records in: 7914723, records dropped: 0 > >2018-07-11 04:06:14.886682 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:06:14.886474) EVENT_LOG_v1 {"time_micros": 1531281974886463, "job": 2610, "event": "compaction_finished", "compaction_time_micros": 13598110, "output_level": 4, "num_output_files": 3, "total_output_size": 223639488, "num_input_records": 7914723, "num_output_records": 7914723, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 7, 44, 446, 3644, 694, 0]} >2018-07-11 04:06:14.887632 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281974887628, "job": 2610, "event": "table_file_deletion", "file_number": 35448} >2018-07-11 04:06:14.887647 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281974887644, "job": 2610, "event": "table_file_deletion", "file_number": 35447} >2018-07-11 04:06:14.887660 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281974887658, "job": 2610, "event": "table_file_deletion", "file_number": 35354} >2018-07-11 04:06:14.887673 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281974887671, "job": 2610, "event": "table_file_deletion", "file_number": 34907} >2018-07-11 04:06:14.911491 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:06:14.887971) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #35450 to level-5 74546963 bytes > >2018-07-11 04:06:14.911502 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:06:14.911400) EVENT_LOG_v1 {"time_micros": 1531281974911394, "job": 2611, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546963} >2018-07-11 04:06:14.911504 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:06:14.911408) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546963 bytes OK: base level 1 max bytes base 268435456 files[1 7 44 446 3643 695 0] max score 1.00 > >2018-07-11 04:06:14.936570 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:06:14.912348) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #12335 to level-5 74546862 bytes > >2018-07-11 04:06:14.936580 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:06:14.936481) EVENT_LOG_v1 {"time_micros": 1531281974936475, "job": 2612, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546862} >2018-07-11 04:06:14.936583 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:06:14.936489) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546862 bytes OK: base level 1 max bytes base 268435456 files[1 7 44 446 3642 696 0] max score 1.00 > >2018-07-11 04:06:14.961596 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:06:14.937437) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #16740 to level-5 74546862 bytes > >2018-07-11 04:06:14.961607 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:06:14.961509) EVENT_LOG_v1 {"time_micros": 1531281974961503, "job": 2613, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546862} >2018-07-11 04:06:14.961609 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:06:14.961516) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546862 bytes OK: base level 1 max bytes base 268435456 files[1 7 44 446 3641 697 0] max score 1.00 > >2018-07-11 04:06:14.962442 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2614] Compacting 1@3 + 1@4 files to L4, score 1.00 >2018-07-11 04:06:14.962454 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2613 Base level 3, inputs: [35446(71MB)], [34905(71MB)] > >2018-07-11 04:06:14.962467 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281974962457, "job": 2614, "event": "compaction_started", "files_L3": [35446], "files_L4": [34905], "score": 1.00013, "input_data_size": 149092565} >2018-07-11 04:06:19.603946 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2614] Generated table #35452: 2638241 keys, 74546316 bytes >2018-07-11 04:06:19.603978 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281979603964, "cf_name": "default", "job": 2614, "event": "table_file_creation", "file_number": 35452, "file_size": 74546316, "table_properties": {"data_size": 67112196, "index_size": 689946, "filter_size": 6743184, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:06:24.130146 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2614] Generated table #35453: 2638240 keys, 74545970 bytes >2018-07-11 04:06:24.130193 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281984130164, "cf_name": "default", "job": 2614, "event": "table_file_creation", "file_number": 35453, "file_size": 74545970, "table_properties": {"data_size": 67112110, "index_size": 689700, "filter_size": 6743170, "raw_key_size": 76215826, "raw_average_key_size": 28, "raw_value_size": 31365746, "raw_average_value_size": 11, "num_data_blocks": 16489, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:06:24.130228 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2614] Compacted 1@3 + 1@4 files to L4 => 149092286 bytes >2018-07-11 04:06:24.153192 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:06:24.152929) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[1 7 44 445 3642 697 0] max score 1.00, MB/sec: 16.3 rd, 16.3 wr, level 4, files in(1, 1) out(2) MB in(71.1, 71.1) out(142.2), read-write-amplify(4.0) write-amplify(2.0) OK, records in: 5276481, records dropped: 0 > >2018-07-11 04:06:24.153203 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:06:24.152953) EVENT_LOG_v1 {"time_micros": 1531281984152943, "job": 2614, "event": "compaction_finished", "compaction_time_micros": 9167748, "output_level": 4, "num_output_files": 2, "total_output_size": 149092286, "num_input_records": 5276481, "num_output_records": 5276481, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 7, 44, 445, 3642, 697, 0]} >2018-07-11 04:06:24.154114 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281984154110, "job": 2614, "event": "table_file_deletion", "file_number": 35446} >2018-07-11 04:06:24.154135 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531281984154133, "job": 2614, "event": "table_file_deletion", "file_number": 34905} >2018-07-11 04:06:24.178001 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:06:24.154437) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #24536 to level-5 74546862 bytes > >2018-07-11 04:06:24.178012 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:06:24.177905) EVENT_LOG_v1 {"time_micros": 1531281984177899, "job": 2615, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546862} >2018-07-11 04:06:24.178017 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:06:24.177914) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546862 bytes OK: base level 1 max bytes base 268435456 files[1 7 44 445 3641 698 0] max score 1.00 > >2018-07-11 04:07:33.346684 7f416a9f8700 0 log_channel(cluster) log [DBG] : 5.9 deep-scrub ok >2018-07-11 04:08:46.504582 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:684] reusing log 35419 from recycle list > >2018-07-11 04:08:46.504744 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #35454. Immutable memtables: 0. > >2018-07-11 04:08:46.552125 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:08:46.504810) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1 >2018-07-11 04:08:46.552139 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 2616] Flushing memtable with next log file: 35454 > >2018-07-11 04:08:46.552157 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282126552147, "job": 2616, "event": "flush_started", "num_memtables": 1, "num_entries": 5000017, "num_deletes": 0, "memory_usage": 292945848} >2018-07-11 04:08:46.552160 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 2616] Level-0 flush table #35455: started >2018-07-11 04:08:50.459041 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282130459019, "cf_name": "default", "job": 2616, "event": "table_file_creation", "file_number": 35455, "file_size": 141286884, "table_properties": {"data_size": 127197769, "index_size": 1308397, "filter_size": 12779728, "raw_key_size": 144445126, "raw_average_key_size": 28, "raw_value_size": 59450675, "raw_average_value_size": 11, "num_data_blocks": 31252, "num_entries": 5000015, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:08:50.459064 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 2616] Level-0 flush table #35455: 141286884 bytes OK >2018-07-11 04:08:50.482631 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:242] adding log 35441 to recycle list > >2018-07-11 04:08:50.482655 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:08:50.459089) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #35455 started >2018-07-11 04:08:50.482660 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:08:50.482413) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #35455: memtable #1 done >2018-07-11 04:08:50.482662 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:08:50.482431) EVENT_LOG_v1 {"time_micros": 1531282130482424, "job": 2616, "event": "flush_finished", "lsm_state": [2, 7, 44, 445, 3641, 698, 0], "immutable_memtables": 0} >2018-07-11 04:08:50.482663 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:08:50.482579) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[2 7 44 445 3641 698 0] max score 1.05 > >2018-07-11 04:08:50.482785 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2617] Compacting 2@0 + 7@1 files to L1, score 1.05 >2018-07-11 04:08:50.482803 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2616 Base level 0, inputs: [35455(134MB) 35442(134MB)], [35421(71MB) 35427(65MB) 35428(10KB) 35337(5095B) 35305(5087B) 35268(5089B) 35234(54MB)] > >2018-07-11 04:08:50.482821 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282130482808, "job": 2617, "event": "compaction_started", "files_L0": [35455, 35442], "files_L1": [35421, 35427, 35428, 35337, 35305, 35268, 35234], "score": 1.05268, "input_data_size": 482059449} >2018-07-11 04:08:50.483556 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 2616] Try to delete WAL files size 178897722, prev total WAL file size 178897734, number of live WAL files 2. > >2018-07-11 04:08:54.611438 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2617] Generated table #35456: 2624472 keys, 74511958 bytes >2018-07-11 04:08:54.611481 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282134611455, "cf_name": "default", "job": 2617, "event": "table_file_creation", "file_number": 35456, "file_size": 74511958, "table_properties": {"data_size": 67110847, "index_size": 691351, "filter_size": 6708770, "raw_key_size": 75848272, "raw_average_key_size": 28, "raw_value_size": 31543484, "raw_average_value_size": 12, "num_data_blocks": 16475, "num_entries": 2624472, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:08:57.292902 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2617] Generated table #35457: 2638242 keys, 74546423 bytes >2018-07-11 04:08:57.292947 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282137292920, "cf_name": "default", "job": 2617, "event": "table_file_creation", "file_number": 35457, "file_size": 74546423, "table_properties": {"data_size": 67112133, "index_size": 690114, "filter_size": 6743186, "raw_key_size": 76215874, "raw_average_key_size": 28, "raw_value_size": 31365760, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638242, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:09:00.232465 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2617] Generated table #35458: 2638241 keys, 74546257 bytes >2018-07-11 04:09:00.232511 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282140232483, "cf_name": "default", "job": 2617, "event": "table_file_creation", "file_number": 35458, "file_size": 74546257, "table_properties": {"data_size": 67112146, "index_size": 689938, "filter_size": 6743183, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:09:02.899712 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2617] Generated table #35459: 2638241 keys, 74545806 bytes >2018-07-11 04:09:02.899756 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282142899729, "cf_name": "default", "job": 2617, "event": "table_file_creation", "file_number": 35459, "file_size": 74545806, "table_properties": {"data_size": 67112087, "index_size": 689546, "filter_size": 6743183, "raw_key_size": 76215853, "raw_average_key_size": 28, "raw_value_size": 31365756, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:09:05.034241 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2617] Generated table #35460: 2085356 keys, 58924120 bytes >2018-07-11 04:09:05.034289 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282145034262, "cf_name": "default", "job": 2617, "event": "table_file_creation", "file_number": 35460, "file_size": 58924120, "table_properties": {"data_size": 53047745, "index_size": 545350, "filter_size": 5330037, "raw_key_size": 60243630, "raw_average_key_size": 28, "raw_value_size": 24792578, "raw_average_value_size": 11, "num_data_blocks": 13034, "num_entries": 2085356, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:09:06.133344 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2617] Generated table #35461: 166117 keys, 68248638 bytes >2018-07-11 04:09:06.133385 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282146133358, "cf_name": "default", "job": 2617, "event": "table_file_creation", "file_number": 35461, "file_size": 68248638, "table_properties": {"data_size": 67113765, "index_size": 565010, "filter_size": 568875, "raw_key_size": 15373567, "raw_average_key_size": 92, "raw_value_size": 53118610, "raw_average_value_size": 319, "num_data_blocks": 15213, "num_entries": 166117, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "80898", "kMergeOperands": "0"}} >2018-07-11 04:09:07.182441 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2617] Generated table #35462: 138226 keys, 56723395 bytes >2018-07-11 04:09:07.182491 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282147182475, "cf_name": "default", "job": 2617, "event": "table_file_creation", "file_number": 35462, "file_size": 56723395, "table_properties": {"data_size": 55773627, "index_size": 474533, "filter_size": 474247, "raw_key_size": 6753180, "raw_average_key_size": 48, "raw_value_size": 49524023, "raw_average_value_size": 358, "num_data_blocks": 13421, "num_entries": 138226, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "58265"}} >2018-07-11 04:09:07.182557 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2617] Compacted 2@0 + 7@1 files to L1 => 482046597 bytes >2018-07-11 04:09:07.199144 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:09:07.198870) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 7 44 445 3641 698 0] max score 1.80, MB/sec: 28.9 rd, 28.9 wr, level 1, files in(2, 7) out(7) MB in(269.5, 190.2) out(459.7), read-write-amplify(3.4) write-amplify(1.7) OK, records in: 12928904, records dropped: 9 > >2018-07-11 04:09:07.199155 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:09:07.198895) EVENT_LOG_v1 {"time_micros": 1531282147198884, "job": 2617, "event": "compaction_finished", "compaction_time_micros": 16699717, "output_level": 1, "num_output_files": 7, "total_output_size": 482046597, "num_input_records": 12928904, "num_output_records": 12928895, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 7, 44, 445, 3641, 698, 0]} >2018-07-11 04:09:07.200070 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282147200066, "job": 2617, "event": "table_file_deletion", "file_number": 35455} >2018-07-11 04:09:07.200096 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282147200093, "job": 2617, "event": "table_file_deletion", "file_number": 35442} >2018-07-11 04:09:07.200107 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282147200105, "job": 2617, "event": "table_file_deletion", "file_number": 35428} >2018-07-11 04:09:07.200117 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282147200115, "job": 2617, "event": "table_file_deletion", "file_number": 35427} >2018-07-11 04:09:07.200128 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282147200126, "job": 2617, "event": "table_file_deletion", "file_number": 35421} >2018-07-11 04:09:07.200139 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282147200137, "job": 2617, "event": "table_file_deletion", "file_number": 35337} >2018-07-11 04:09:07.200150 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282147200148, "job": 2617, "event": "table_file_deletion", "file_number": 35305} >2018-07-11 04:09:07.200161 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282147200159, "job": 2617, "event": "table_file_deletion", "file_number": 35268} >2018-07-11 04:09:07.200171 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282147200169, "job": 2617, "event": "table_file_deletion", "file_number": 35234} >2018-07-11 04:09:07.200394 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_picker.cc:461] [default] Expanding@1 1+5(74546423+325136145 bytes) to 4+5 (282562606+325136145bytes) > >2018-07-11 04:09:07.200490 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2618] Compacting 4@1 + 5@2 files to L2, score 1.80 >2018-07-11 04:09:07.200503 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2617 Base level 1, inputs: [35457(71MB) 35458(71MB) 35459(71MB) 35460(56MB)], [35439(71MB) 35443(71MB) 35444(71MB) 35445(71MB) 35433(25MB)] > >2018-07-11 04:09:07.200517 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282147200506, "job": 2618, "event": "compaction_started", "files_L1": [35457, 35458, 35459, 35460], "files_L2": [35439, 35443, 35444, 35445, 35433], "score": 1.79576, "input_data_size": 607698751} >2018-07-11 04:09:12.127836 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2618] Generated table #35463: 2638238 keys, 74545730 bytes >2018-07-11 04:09:12.127884 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282152127869, "cf_name": "default", "job": 2618, "event": "table_file_creation", "file_number": 35463, "file_size": 74545730, "table_properties": {"data_size": 67112107, "index_size": 689456, "filter_size": 6743177, "raw_key_size": 76215768, "raw_average_key_size": 28, "raw_value_size": 31365722, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638238, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:09:16.820204 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2618] Generated table #35464: 2638240 keys, 74546538 bytes >2018-07-11 04:09:16.820255 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282156820223, "cf_name": "default", "job": 2618, "event": "table_file_creation", "file_number": 35464, "file_size": 74546538, "table_properties": {"data_size": 67112126, "index_size": 690241, "filter_size": 6743181, "raw_key_size": 76215823, "raw_average_key_size": 28, "raw_value_size": 31365743, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:09:21.397374 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2618] Generated table #35465: 2638241 keys, 74546559 bytes >2018-07-11 04:09:21.397433 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282161397415, "cf_name": "default", "job": 2618, "event": "table_file_creation", "file_number": 35465, "file_size": 74546559, "table_properties": {"data_size": 67112152, "index_size": 690233, "filter_size": 6743184, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:09:25.839695 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2618] Generated table #35466: 2638240 keys, 74547049 bytes >2018-07-11 04:09:25.839742 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282165839726, "cf_name": "default", "job": 2618, "event": "table_file_creation", "file_number": 35466, "file_size": 74547049, "table_properties": {"data_size": 67112154, "index_size": 690724, "filter_size": 6743181, "raw_key_size": 76215826, "raw_average_key_size": 28, "raw_value_size": 31365746, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:09:29.895055 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2618] Generated table #35467: 2638240 keys, 74546399 bytes >2018-07-11 04:09:29.895099 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282169895071, "cf_name": "default", "job": 2618, "event": "table_file_creation", "file_number": 35467, "file_size": 74546399, "table_properties": {"data_size": 67112120, "index_size": 690108, "filter_size": 6743181, "raw_key_size": 76215821, "raw_average_key_size": 28, "raw_value_size": 31365741, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:09:33.065392 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2618] Generated table #35468: 2638241 keys, 74545938 bytes >2018-07-11 04:09:33.065437 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282173065410, "cf_name": "default", "job": 2618, "event": "table_file_creation", "file_number": 35468, "file_size": 74545938, "table_properties": {"data_size": 67112133, "index_size": 689632, "filter_size": 6743183, "raw_key_size": 76215860, "raw_average_key_size": 28, "raw_value_size": 31365763, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:09:36.665242 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2618] Generated table #35469: 2638241 keys, 74547094 bytes >2018-07-11 04:09:36.665281 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282176665267, "cf_name": "default", "job": 2618, "event": "table_file_creation", "file_number": 35469, "file_size": 74547094, "table_properties": {"data_size": 67112169, "index_size": 690752, "filter_size": 6743183, "raw_key_size": 76215856, "raw_average_key_size": 28, "raw_value_size": 31365759, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:09:40.070706 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2618] Generated table #35470: 2638241 keys, 74546263 bytes >2018-07-11 04:09:40.070753 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282180070724, "cf_name": "default", "job": 2618, "event": "table_file_creation", "file_number": 35470, "file_size": 74546263, "table_properties": {"data_size": 67112096, "index_size": 689994, "filter_size": 6743183, "raw_key_size": 76215850, "raw_average_key_size": 28, "raw_value_size": 31365753, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:09:40.801090 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2618] Generated table #35471: 396522 keys, 11330547 bytes >2018-07-11 04:09:40.801121 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282180801106, "cf_name": "default", "job": 2618, "event": "table_file_creation", "file_number": 35471, "file_size": 11330547, "table_properties": {"data_size": 10211391, "index_size": 104388, "filter_size": 1013782, "raw_key_size": 11470835, "raw_average_key_size": 28, "raw_value_size": 4819583, "raw_average_value_size": 12, "num_data_blocks": 2511, "num_entries": 396522, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "11", "kMergeOperands": "0"}} >2018-07-11 04:09:40.801151 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2618] Compacted 4@1 + 5@2 files to L2 => 607702117 bytes >2018-07-11 04:09:40.815075 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:09:40.814793) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 3 48 445 3641 698 0] max score 1.08, MB/sec: 18.1 rd, 18.1 wr, level 2, files in(4, 5) out(9) MB in(269.5, 310.1) out(579.5), read-write-amplify(4.3) write-amplify(2.2) OK, records in: 21502444, records dropped: 0 > >2018-07-11 04:09:40.815086 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:09:40.814818) EVENT_LOG_v1 {"time_micros": 1531282180814807, "job": 2618, "event": "compaction_finished", "compaction_time_micros": 33600616, "output_level": 2, "num_output_files": 9, "total_output_size": 607702117, "num_input_records": 21502444, "num_output_records": 21502444, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 3, 48, 445, 3641, 698, 0]} >2018-07-11 04:09:40.815999 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282180815996, "job": 2618, "event": "table_file_deletion", "file_number": 35460} >2018-07-11 04:09:40.816027 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282180816024, "job": 2618, "event": "table_file_deletion", "file_number": 35459} >2018-07-11 04:09:40.816038 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282180816036, "job": 2618, "event": "table_file_deletion", "file_number": 35458} >2018-07-11 04:09:40.816049 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282180816047, "job": 2618, "event": "table_file_deletion", "file_number": 35457} >2018-07-11 04:09:40.816063 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282180816061, "job": 2618, "event": "table_file_deletion", "file_number": 35445} >2018-07-11 04:09:40.816079 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282180816077, "job": 2618, "event": "table_file_deletion", "file_number": 35444} >2018-07-11 04:09:40.816095 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282180816090, "job": 2618, "event": "table_file_deletion", "file_number": 35443} >2018-07-11 04:09:40.816109 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282180816107, "job": 2618, "event": "table_file_deletion", "file_number": 35439} >2018-07-11 04:09:40.816124 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282180816122, "job": 2618, "event": "table_file_deletion", "file_number": 35433} >2018-07-11 04:09:40.816351 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_picker.cc:461] [default] Expanding@2 1+1(74547094+46965472 bytes) to 3+1 (223639295+46965472bytes) > >2018-07-11 04:09:40.816449 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2619] Compacting 3@2 + 1@3 files to L3, score 1.08 >2018-07-11 04:09:40.816462 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2618 Base level 2, inputs: [35468(71MB) 35469(71MB) 35470(71MB)], [35438(44MB)] > >2018-07-11 04:09:40.816475 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282180816465, "job": 2619, "event": "compaction_started", "files_L2": [35468, 35469, 35470], "files_L3": [35438], "score": 1.07816, "input_data_size": 270604767} >2018-07-11 04:09:44.782826 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2619] Generated table #35472: 2638240 keys, 74546943 bytes >2018-07-11 04:09:44.782871 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282184782844, "cf_name": "default", "job": 2619, "event": "table_file_creation", "file_number": 35472, "file_size": 74546943, "table_properties": {"data_size": 67112117, "index_size": 690655, "filter_size": 6743181, "raw_key_size": 76215828, "raw_average_key_size": 28, "raw_value_size": 31365748, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:09:49.199096 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2619] Generated table #35473: 2638241 keys, 74546507 bytes >2018-07-11 04:09:49.199141 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282189199114, "cf_name": "default", "job": 2619, "event": "table_file_creation", "file_number": 35473, "file_size": 74546507, "table_properties": {"data_size": 67112150, "index_size": 690184, "filter_size": 6743183, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:09:53.177986 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2619] Generated table #35474: 2638241 keys, 74546385 bytes >2018-07-11 04:09:53.178032 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282193178003, "cf_name": "default", "job": 2619, "event": "table_file_creation", "file_number": 35474, "file_size": 74546385, "table_properties": {"data_size": 67112097, "index_size": 690115, "filter_size": 6743183, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:09:55.101620 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2619] Generated table #35475: 1519916 keys, 46965004 bytes >2018-07-11 04:09:55.101662 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282195101636, "cf_name": "default", "job": 2619, "event": "table_file_creation", "file_number": 35475, "file_size": 46965004, "table_properties": {"data_size": 42636653, "index_size": 433212, "filter_size": 3894151, "raw_key_size": 44348935, "raw_average_key_size": 29, "raw_value_size": 21522464, "raw_average_value_size": 14, "num_data_blocks": 10538, "num_entries": 1519916, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:09:55.101693 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2619] Compacted 3@2 + 1@3 files to L3 => 270604839 bytes >2018-07-11 04:09:55.114661 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:09:55.114534) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 3 45 448 3641 698 0] max score 1.01, MB/sec: 18.9 rd, 18.9 wr, level 3, files in(3, 1) out(4) MB in(213.3, 44.8) out(258.1), read-write-amplify(2.4) write-amplify(1.2) OK, records in: 9434638, records dropped: 0 > >2018-07-11 04:09:55.114671 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:09:55.114556) EVENT_LOG_v1 {"time_micros": 1531282195114546, "job": 2619, "event": "compaction_finished", "compaction_time_micros": 14285205, "output_level": 3, "num_output_files": 4, "total_output_size": 270604839, "num_input_records": 9434638, "num_output_records": 9434638, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 3, 45, 448, 3641, 698, 0]} >2018-07-11 04:09:55.116292 7f41641eb700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS ------- >2018-07-11 04:09:55.116298 7f41641eb700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] >** DB Stats ** >Uptime(secs): 46936.8 total, 601.9 interval >Cumulative writes: 196K writes, 1731M keys, 196K commit groups, 1.0 writes per commit group, ingest: 58.20 GB, 1.27 MB/s >Cumulative WAL: 196K writes, 56K syncs, 3.49 writes per sync, written: 58.20 GB, 1.27 MB/s >Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent >Interval writes: 52 writes, 24M keys, 52 commit groups, 1.0 writes per commit group, ingest: 818.93 MB, 1.36 MB/s >Interval WAL: 52 writes, 26 syncs, 1.93 writes per sync, written: 0.80 MB, 1.36 MB/s >Interval stall: 00:00:0.000 H:M:S, 0.0 percent > >** Compaction Stats [default] ** >Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop >---------------------------------------------------------------------------------------------------------------------------------------------------------- > L0 0/0 0.00 KB 0.0 0.0 0.0 0.0 45.4 45.4 0.0 1.0 0.0 32.4 1437 347 4.142 0 0 > L1 3/0 190.24 MB 0.7 75.4 45.6 29.8 75.3 45.5 0.0 1.7 25.7 25.7 3001 171 17.549 2384M 894K > L2 45/0 2.49 GB 1.0 87.5 45.6 42.0 87.5 45.5 0.0 1.9 21.5 21.5 4175 540 7.731 3261M 1358K > L3 448/0 25.14 GB 1.0 79.4 44.9 34.5 79.3 44.8 0.6 1.8 20.6 20.6 3939 270 14.589 3000M 4000K > L4 3641/0 250.00 GB 1.0 68.5 16.4 52.0 68.1 16.1 28.8 4.1 21.1 21.0 3325 221 15.044 2605M 15M > L5 698/0 48.44 GB 0.0 1.6 0.8 0.8 1.6 0.8 44.0 1.9 18.5 18.2 89 11 8.048 60M 1097K > Sum 4835/0 326.25 GB 0.0 312.4 153.2 159.1 357.3 198.2 73.5 7.9 20.0 22.9 15965 1560 10.234 11G 23M > Int 0/0 0.00 KB 0.0 4.5 2.5 2.0 5.0 3.1 1.0 9.5 16.2 18.1 283 18 15.726 161M 25 >Uptime(secs): 46936.8 total, 46936.8 interval >Flush(GB): cumulative 45.444, interval 0.526 >AddFile(GB): cumulative 0.000, interval 0.000 >AddFile(Total Files): cumulative 0, interval 0 >AddFile(L0 Files): cumulative 0, interval 0 >AddFile(Keys): cumulative 0, interval 0 >Cumulative compaction: 357.29 GB write, 7.79 MB/s write, 312.37 GB read, 6.81 MB/s read, 15965.1 seconds >Interval compaction: 5.01 GB write, 0.11 MB/s write, 4.49 GB read, 0.10 MB/s read, 283.1 seconds >Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count > >** File Read Latency Histogram By Level [default] ** > >** Compaction Stats [default] ** >Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop >---------------------------------------------------------------------------------------------------------------------------------------------------------- > L0 0/0 0.00 KB 0.0 0.0 0.0 0.0 45.4 45.4 0.0 1.0 0.0 32.4 1437 347 4.142 0 0 > L1 3/0 190.24 MB 0.7 75.4 45.6 29.8 75.3 45.5 0.0 1.7 25.7 25.7 3001 171 17.549 2384M 894K > L2 45/0 2.49 GB 1.0 87.5 45.6 42.0 87.5 45.5 0.0 1.9 21.5 21.5 4175 540 7.731 3261M 1358K > L3 448/0 25.14 GB 1.0 79.4 44.9 34.5 79.3 44.8 0.6 1.8 20.6 20.6 3939 270 14.589 3000M 4000K > L4 3641/0 250.00 GB 1.0 68.5 16.4 52.0 68.1 16.1 28.8 4.1 21.1 21.0 3325 221 15.044 2605M 15M > L5 698/0 48.44 GB 0.0 1.6 0.8 0.8 1.6 0.8 44.0 1.9 18.5 18.2 89 11 8.048 60M 1097K > Sum 4835/0 326.25 GB 0.0 312.4 153.2 159.1 357.3 198.2 73.5 7.9 20.0 22.9 15965 1560 10.234 11G 23M > Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 >Uptime(secs): 46936.8 total, 46936.8 interval >Flush(GB): cumulative 45.444, interval 0.000 >AddFile(GB): cumulative 0.000, interval 0.000 >AddFile(Total Files): cumulative 0, interval 0 >AddFile(L0 Files): cumulative 0, interval 0 >AddFile(Keys): cumulative 0, interval 0 >Cumulative compaction: 357.29 GB write, 7.79 MB/s write, 312.37 GB read, 6.81 MB/s read, 15965.1 seconds >Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds >Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count > >** File Read Latency Histogram By Level [default] ** > >2018-07-11 04:09:55.131298 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:09:55.116412) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #35472 to level-4 74546943 bytes > >2018-07-11 04:09:55.131312 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:09:55.131173) EVENT_LOG_v1 {"time_micros": 1531282195131167, "job": 2620, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 74546943} >2018-07-11 04:09:55.131317 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:09:55.131182) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-4 74546943 bytes OK: base level 1 max bytes base 268435456 files[0 3 45 447 3642 698 0] max score 1.00 > >2018-07-11 04:09:55.132080 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282195132062, "job": 2620, "event": "table_file_deletion", "file_number": 35470} >2018-07-11 04:09:55.132097 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282195132095, "job": 2620, "event": "table_file_deletion", "file_number": 35469} >2018-07-11 04:09:55.132109 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282195132107, "job": 2620, "event": "table_file_deletion", "file_number": 35468} >2018-07-11 04:09:55.132121 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282195132119, "job": 2620, "event": "table_file_deletion", "file_number": 35438} >2018-07-11 04:09:55.148003 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:09:55.132401) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #35473 to level-4 74546507 bytes > >2018-07-11 04:09:55.148013 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:09:55.147919) EVENT_LOG_v1 {"time_micros": 1531282195147913, "job": 2621, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 74546507} >2018-07-11 04:09:55.148015 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:09:55.147926) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-4 74546507 bytes OK: base level 1 max bytes base 268435456 files[0 3 45 446 3643 698 0] max score 1.00 > >2018-07-11 04:09:55.189724 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:09:55.148844) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #35472 to level-5 74546943 bytes > >2018-07-11 04:09:55.189735 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:09:55.189610) EVENT_LOG_v1 {"time_micros": 1531282195189604, "job": 2622, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546943} >2018-07-11 04:09:55.189741 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:09:55.189618) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546943 bytes OK: base level 1 max bytes base 268435456 files[0 3 45 446 3642 699 0] max score 1.00 > >2018-07-11 04:09:55.206513 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:09:55.190815) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #12730 to level-5 74546862 bytes > >2018-07-11 04:09:55.206535 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:09:55.206322) EVENT_LOG_v1 {"time_micros": 1531282195206316, "job": 2623, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546862} >2018-07-11 04:09:55.206538 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:09:55.206330) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546862 bytes OK: base level 1 max bytes base 268435456 files[0 3 45 446 3641 700 0] max score 1.00 > >2018-07-11 04:09:55.223135 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:09:55.207351) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #35474 to level-4 74546385 bytes > >2018-07-11 04:09:55.223145 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:09:55.222944) EVENT_LOG_v1 {"time_micros": 1531282195222938, "job": 2624, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 74546385} >2018-07-11 04:09:55.223160 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:09:55.222952) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-4 74546385 bytes OK: base level 1 max bytes base 268435456 files[0 3 45 445 3642 700 0] max score 1.00 > >2018-07-11 04:09:55.239823 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:09:55.223975) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #23741 to level-5 74546861 bytes > >2018-07-11 04:09:55.239834 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:09:55.239629) EVENT_LOG_v1 {"time_micros": 1531282195239623, "job": 2625, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546861} >2018-07-11 04:09:55.239837 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:09:55.239638) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546861 bytes OK: base level 1 max bytes base 268435456 files[0 3 45 445 3641 701 0] max score 1.00 > >2018-07-11 04:10:45.823139 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:684] reusing log 35441 from recycle list > >2018-07-11 04:10:45.823263 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #35476. Immutable memtables: 0. > >2018-07-11 04:10:45.893610 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:10:45.823326) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1 >2018-07-11 04:10:45.893624 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 2626] Flushing memtable with next log file: 35476 > >2018-07-11 04:10:45.893642 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282245893633, "job": 2626, "event": "flush_started", "num_memtables": 1, "num_entries": 5000021, "num_deletes": 0, "memory_usage": 292960920} >2018-07-11 04:10:45.893645 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 2626] Level-0 flush table #35477: started >2018-07-11 04:10:48.666749 7f41649ec700 1 bluefs _allocate failed to allocate 0x8700000 on bdev 1, free 0x68fe000; fallback to bdev 2 >2018-07-11 04:10:50.686030 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282250686009, "cf_name": "default", "job": 2626, "event": "table_file_creation", "file_number": 35477, "file_size": 141288324, "table_properties": {"data_size": 127199949, "index_size": 1307640, "filter_size": 12779745, "raw_key_size": 144445253, "raw_average_key_size": 28, "raw_value_size": 59452591, "raw_average_value_size": 11, "num_data_blocks": 31253, "num_entries": 5000020, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:10:50.686055 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 2626] Level-0 flush table #35477: 141288324 bytes OK >2018-07-11 04:10:50.696690 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:242] adding log 35454 to recycle list > >2018-07-11 04:10:50.696702 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:10:50.686069) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #35477 started >2018-07-11 04:10:50.696720 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:10:50.696555) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #35477: memtable #1 done >2018-07-11 04:10:50.696724 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:10:50.696571) EVENT_LOG_v1 {"time_micros": 1531282250696564, "job": 2626, "event": "flush_finished", "lsm_state": [1, 3, 45, 445, 3641, 701, 0], "immutable_memtables": 0} >2018-07-11 04:10:50.696727 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:10:50.696638) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 3 45 445 3641 701 0] max score 1.00 > >2018-07-11 04:10:50.697602 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 2626] Try to delete WAL files size 178898847, prev total WAL file size 178898859, number of live WAL files 2. > >2018-07-11 04:12:02.884570 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:684] reusing log 35454 from recycle list > >2018-07-11 04:12:02.884734 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #35478. Immutable memtables: 0. > >2018-07-11 04:12:02.938476 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:12:02.884804) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1 >2018-07-11 04:12:02.938491 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 2627] Flushing memtable with next log file: 35478 > >2018-07-11 04:12:02.938508 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282322938500, "job": 2627, "event": "flush_started", "num_memtables": 1, "num_entries": 5000015, "num_deletes": 0, "memory_usage": 292962832} >2018-07-11 04:12:02.938511 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 2627] Level-0 flush table #35479: started >2018-07-11 04:12:05.570042 7f41649ec700 1 bluefs _allocate failed to allocate 0x8700000 on bdev 1, free 0x68fe000; fallback to bdev 2 >2018-07-11 04:12:07.462042 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282327462022, "cf_name": "default", "job": 2627, "event": "table_file_creation", "file_number": 35479, "file_size": 141286889, "table_properties": {"data_size": 127197777, "index_size": 1308397, "filter_size": 12779725, "raw_key_size": 144445101, "raw_average_key_size": 28, "raw_value_size": 59450671, "raw_average_value_size": 11, "num_data_blocks": 31252, "num_entries": 5000014, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:12:07.462060 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 2627] Level-0 flush table #35479: 141286889 bytes OK >2018-07-11 04:12:07.478018 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:242] adding log 35476 to recycle list > >2018-07-11 04:12:07.478030 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:12:07.462084) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #35479 started >2018-07-11 04:12:07.478047 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:12:07.477875) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #35479: memtable #1 done >2018-07-11 04:12:07.478050 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:12:07.477890) EVENT_LOG_v1 {"time_micros": 1531282327477884, "job": 2627, "event": "flush_finished", "lsm_state": [2, 3, 45, 445, 3641, 701, 0], "immutable_memtables": 0} >2018-07-11 04:12:07.478052 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:12:07.477956) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[2 3 45 445 3641 701 0] max score 1.05 > >2018-07-11 04:12:07.478174 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2628] Compacting 2@0 + 2@1 files to L1, score 1.05 >2018-07-11 04:12:07.478191 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2627 Base level 0, inputs: [35479(134MB) 35477(134MB)], [35456(71MB) 35461(65MB)] > >2018-07-11 04:12:07.478209 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282327478196, "job": 2628, "event": "compaction_started", "files_L0": [35479, 35477], "files_L1": [35456, 35461], "score": 1.05267, "input_data_size": 425335809} >2018-07-11 04:12:07.478911 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 2627] Try to delete WAL files size 178896755, prev total WAL file size 178896767, number of live WAL files 2. > >2018-07-11 04:12:10.321932 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2628] Generated table #35480: 2624328 keys, 74511332 bytes >2018-07-11 04:12:10.321966 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282330321950, "cf_name": "default", "job": 2628, "event": "table_file_creation", "file_number": 35480, "file_size": 74511332, "table_properties": {"data_size": 67110558, "index_size": 691374, "filter_size": 6708410, "raw_key_size": 75844301, "raw_average_key_size": 28, "raw_value_size": 31545121, "raw_average_value_size": 12, "num_data_blocks": 16475, "num_entries": 2624328, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:12:12.295121 7f41641eb700 1 bluefs _allocate failed to allocate 0x4800000 on bdev 1, free 0x20fe000; fallback to bdev 2 >2018-07-11 04:12:13.256576 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2628] Generated table #35481: 2638242 keys, 74546376 bytes >2018-07-11 04:12:13.256620 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282333256592, "cf_name": "default", "job": 2628, "event": "table_file_creation", "file_number": 35481, "file_size": 74546376, "table_properties": {"data_size": 67112136, "index_size": 690064, "filter_size": 6743186, "raw_key_size": 76215875, "raw_average_key_size": 28, "raw_value_size": 31365761, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638242, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:12:15.242693 7f41641eb700 1 bluefs _allocate failed to allocate 0x4800000 on bdev 1, free 0x20fe000; fallback to bdev 2 >2018-07-11 04:12:16.191602 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2628] Generated table #35482: 2638241 keys, 74546216 bytes >2018-07-11 04:12:16.191642 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282336191616, "cf_name": "default", "job": 2628, "event": "table_file_creation", "file_number": 35482, "file_size": 74546216, "table_properties": {"data_size": 67112147, "index_size": 689896, "filter_size": 6743183, "raw_key_size": 76215855, "raw_average_key_size": 28, "raw_value_size": 31365758, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:12:18.157441 7f41641eb700 1 bluefs _allocate failed to allocate 0x4800000 on bdev 1, free 0x20fe000; fallback to bdev 2 >2018-07-11 04:12:19.168429 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2628] Generated table #35483: 2638241 keys, 74545765 bytes >2018-07-11 04:12:19.168461 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282339168447, "cf_name": "default", "job": 2628, "event": "table_file_creation", "file_number": 35483, "file_size": 74545765, "table_properties": {"data_size": 67112084, "index_size": 689508, "filter_size": 6743183, "raw_key_size": 76215850, "raw_average_key_size": 28, "raw_value_size": 31365753, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:12:20.797079 7f41641eb700 1 bluefs _allocate failed to allocate 0x3900000 on bdev 1, free 0x20fe000; fallback to bdev 2 >2018-07-11 04:12:21.657877 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2628] Generated table #35484: 2085436 keys, 58926338 bytes >2018-07-11 04:12:21.657931 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282341657914, "cf_name": "default", "job": 2628, "event": "table_file_creation", "file_number": 35484, "file_size": 58926338, "table_properties": {"data_size": 53049778, "index_size": 545331, "filter_size": 5330241, "raw_key_size": 60245943, "raw_average_key_size": 28, "raw_value_size": 24793531, "raw_average_value_size": 11, "num_data_blocks": 13034, "num_entries": 2085436, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:12:22.189483 7f41641eb700 1 bluefs _allocate failed to allocate 0x4200000 on bdev 1, free 0x20fe000; fallback to bdev 2 >2018-07-11 04:12:23.116941 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2628] Generated table #35485: 166112 keys, 68244108 bytes >2018-07-11 04:12:23.116991 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282343116975, "cf_name": "default", "job": 2628, "event": "table_file_creation", "file_number": 35485, "file_size": 68244108, "table_properties": {"data_size": 67109286, "index_size": 564977, "filter_size": 568857, "raw_key_size": 15372889, "raw_average_key_size": 92, "raw_value_size": 53114336, "raw_average_value_size": 319, "num_data_blocks": 15212, "num_entries": 166112, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "80898", "kMergeOperands": "0"}} >2018-07-11 04:12:23.136982 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2628] Generated table #35486: 15 keys, 9818 bytes >2018-07-11 04:12:23.137021 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282343137007, "cf_name": "default", "job": 2628, "event": "table_file_creation", "file_number": 35486, "file_size": 9818, "table_properties": {"data_size": 8730, "index_size": 60, "filter_size": 61, "raw_key_size": 1288, "raw_average_key_size": 85, "raw_value_size": 7904, "raw_average_value_size": 526, "num_data_blocks": 2, "num_entries": 15, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:12:23.137088 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2628] Compacted 2@0 + 2@1 files to L1 => 425329953 bytes >2018-07-11 04:12:23.152937 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:12:23.152707) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 8 45 445 3641 701 0] max score 1.80, MB/sec: 27.2 rd, 27.2 wr, level 1, files in(2, 2) out(7) MB in(269.5, 136.1) out(405.6), read-write-amplify(3.0) write-amplify(1.5) OK, records in: 12790623, records dropped: 8 > >2018-07-11 04:12:23.152949 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:12:23.152734) EVENT_LOG_v1 {"time_micros": 1531282343152723, "job": 2628, "event": "compaction_finished", "compaction_time_micros": 15658862, "output_level": 1, "num_output_files": 7, "total_output_size": 425329953, "num_input_records": 12790623, "num_output_records": 12790615, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 8, 45, 445, 3641, 701, 0]} >2018-07-11 04:12:23.153891 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282343153874, "job": 2628, "event": "table_file_deletion", "file_number": 35479} >2018-07-11 04:12:23.153905 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282343153903, "job": 2628, "event": "table_file_deletion", "file_number": 35477} >2018-07-11 04:12:23.153917 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282343153915, "job": 2628, "event": "table_file_deletion", "file_number": 35461} >2018-07-11 04:12:23.153928 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282343153926, "job": 2628, "event": "table_file_deletion", "file_number": 35456} >2018-07-11 04:12:23.154167 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_picker.cc:461] [default] Expanding@1 1+6(74546376+384062822 bytes) to 4+6 (282564695+384062822bytes) > >2018-07-11 04:12:23.154273 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2629] Compacting 4@1 + 6@2 files to L2, score 1.80 >2018-07-11 04:12:23.154286 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2628 Base level 1, inputs: [35481(71MB) 35482(71MB) 35483(71MB) 35484(56MB)], [35463(71MB) 35464(71MB) 35465(71MB) 35466(71MB) 35467(71MB) 35471(10MB)] > >2018-07-11 04:12:23.154305 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282343154290, "job": 2629, "event": "compaction_started", "files_L1": [35481, 35482, 35483, 35484], "files_L2": [35463, 35464, 35465, 35466, 35467, 35471], "score": 1.79579, "input_data_size": 666627517} >2018-07-11 04:12:26.498629 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2629] Generated table #35487: 2638238 keys, 74545730 bytes >2018-07-11 04:12:26.498672 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282346498655, "cf_name": "default", "job": 2629, "event": "table_file_creation", "file_number": 35487, "file_size": 74545730, "table_properties": {"data_size": 67112107, "index_size": 689456, "filter_size": 6743177, "raw_key_size": 76215767, "raw_average_key_size": 28, "raw_value_size": 31365721, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638238, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:12:29.586178 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2629] Generated table #35488: 2638240 keys, 74546538 bytes >2018-07-11 04:12:29.586226 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282349586197, "cf_name": "default", "job": 2629, "event": "table_file_creation", "file_number": 35488, "file_size": 74546538, "table_properties": {"data_size": 67112126, "index_size": 690241, "filter_size": 6743181, "raw_key_size": 76215823, "raw_average_key_size": 28, "raw_value_size": 31365743, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:12:32.630674 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2629] Generated table #35489: 2638241 keys, 74546559 bytes >2018-07-11 04:12:32.630718 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282352630701, "cf_name": "default", "job": 2629, "event": "table_file_creation", "file_number": 35489, "file_size": 74546559, "table_properties": {"data_size": 67112152, "index_size": 690233, "filter_size": 6743184, "raw_key_size": 76215855, "raw_average_key_size": 28, "raw_value_size": 31365758, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:12:35.742523 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2629] Generated table #35490: 2638240 keys, 74547051 bytes >2018-07-11 04:12:35.742569 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282355742542, "cf_name": "default", "job": 2629, "event": "table_file_creation", "file_number": 35490, "file_size": 74547051, "table_properties": {"data_size": 67112156, "index_size": 690724, "filter_size": 6743181, "raw_key_size": 76215828, "raw_average_key_size": 28, "raw_value_size": 31365748, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:12:38.886022 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2629] Generated table #35491: 2638240 keys, 74546400 bytes >2018-07-11 04:12:38.886056 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282358886041, "cf_name": "default", "job": 2629, "event": "table_file_creation", "file_number": 35491, "file_size": 74546400, "table_properties": {"data_size": 67112120, "index_size": 690109, "filter_size": 6743181, "raw_key_size": 76215821, "raw_average_key_size": 28, "raw_value_size": 31365741, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:12:41.943194 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2629] Generated table #35492: 2638241 keys, 74546304 bytes >2018-07-11 04:12:41.943237 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282361943211, "cf_name": "default", "job": 2629, "event": "table_file_creation", "file_number": 35492, "file_size": 74546304, "table_properties": {"data_size": 67112157, "index_size": 689974, "filter_size": 6743183, "raw_key_size": 76215852, "raw_average_key_size": 28, "raw_value_size": 31365755, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:12:45.020063 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2629] Generated table #35493: 2638241 keys, 74545756 bytes >2018-07-11 04:12:45.020108 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282365020081, "cf_name": "default", "job": 2629, "event": "table_file_creation", "file_number": 35493, "file_size": 74545756, "table_properties": {"data_size": 67112108, "index_size": 689475, "filter_size": 6743183, "raw_key_size": 76215857, "raw_average_key_size": 28, "raw_value_size": 31365760, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:12:48.048851 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2629] Generated table #35494: 2638241 keys, 74546513 bytes >2018-07-11 04:12:48.048886 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282368048871, "cf_name": "default", "job": 2629, "event": "table_file_creation", "file_number": 35494, "file_size": 74546513, "table_properties": {"data_size": 67112177, "index_size": 690163, "filter_size": 6743183, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:12:51.161981 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2629] Generated table #35495: 2481959 keys, 70256770 bytes >2018-07-11 04:12:51.162028 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282371162000, "cf_name": "default", "job": 2629, "event": "table_file_creation", "file_number": 35495, "file_size": 70256770, "table_properties": {"data_size": 63261170, "index_size": 650591, "filter_size": 6344020, "raw_key_size": 71716793, "raw_average_key_size": 28, "raw_value_size": 29613112, "raw_average_value_size": 11, "num_data_blocks": 15545, "num_entries": 2481959, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "11", "kMergeOperands": "0"}} >2018-07-11 04:12:51.162067 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2629] Compacted 4@1 + 6@2 files to L2 => 666627621 bytes >2018-07-11 04:12:51.177596 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:12:51.177189) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 4 48 445 3641 701 0] max score 1.10, MB/sec: 23.8 rd, 23.8 wr, level 2, files in(4, 6) out(9) MB in(269.5, 366.3) out(635.7), read-write-amplify(4.7) write-amplify(2.4) OK, records in: 23587881, records dropped: 0 > >2018-07-11 04:12:51.177607 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:12:51.177215) EVENT_LOG_v1 {"time_micros": 1531282371177203, "job": 2629, "event": "compaction_finished", "compaction_time_micros": 28007742, "output_level": 2, "num_output_files": 9, "total_output_size": 666627621, "num_input_records": 23587881, "num_output_records": 23587881, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 4, 48, 445, 3641, 701, 0]} >2018-07-11 04:12:51.178542 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282371178538, "job": 2629, "event": "table_file_deletion", "file_number": 35484} >2018-07-11 04:12:51.178555 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282371178553, "job": 2629, "event": "table_file_deletion", "file_number": 35483} >2018-07-11 04:12:51.178566 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282371178564, "job": 2629, "event": "table_file_deletion", "file_number": 35482} >2018-07-11 04:12:51.178576 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282371178574, "job": 2629, "event": "table_file_deletion", "file_number": 35481} >2018-07-11 04:12:51.178603 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282371178601, "job": 2629, "event": "table_file_deletion", "file_number": 35471} >2018-07-11 04:12:51.178616 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282371178613, "job": 2629, "event": "table_file_deletion", "file_number": 35467} >2018-07-11 04:12:51.178627 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282371178625, "job": 2629, "event": "table_file_deletion", "file_number": 35466} >2018-07-11 04:12:51.178638 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282371178636, "job": 2629, "event": "table_file_deletion", "file_number": 35465} >2018-07-11 04:12:51.178649 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282371178647, "job": 2629, "event": "table_file_deletion", "file_number": 35464} >2018-07-11 04:12:51.178659 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282371178657, "job": 2629, "event": "table_file_deletion", "file_number": 35463} >2018-07-11 04:12:51.178962 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2630] Compacting 1@2 + 1@3 files to L3, score 1.10 >2018-07-11 04:12:51.178986 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2629 Base level 2, inputs: [35490(71MB)], [35435(71MB)] > >2018-07-11 04:12:51.179000 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282371178990, "job": 2630, "event": "compaction_started", "files_L2": [35490], "files_L3": [35435], "score": 1.10011, "input_data_size": 149093108} >2018-07-11 04:12:54.325461 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2630] Generated table #35496: 2638240 keys, 74546325 bytes >2018-07-11 04:12:54.325507 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282374325491, "cf_name": "default", "job": 2630, "event": "table_file_creation", "file_number": 35496, "file_size": 74546325, "table_properties": {"data_size": 67112146, "index_size": 690008, "filter_size": 6743181, "raw_key_size": 76215821, "raw_average_key_size": 28, "raw_value_size": 31365741, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:12:57.371364 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2630] Generated table #35497: 2638240 keys, 74545939 bytes >2018-07-11 04:12:57.371406 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282377371380, "cf_name": "default", "job": 2630, "event": "table_file_creation", "file_number": 35497, "file_size": 74545939, "table_properties": {"data_size": 67112115, "index_size": 689664, "filter_size": 6743170, "raw_key_size": 76215825, "raw_average_key_size": 28, "raw_value_size": 31365745, "raw_average_value_size": 11, "num_data_blocks": 16489, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:12:57.371438 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2630] Compacted 1@2 + 1@3 files to L3 => 149092264 bytes >2018-07-11 04:12:57.385698 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:12:57.385455) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 4 47 446 3641 701 0] max score 1.07, MB/sec: 24.1 rd, 24.1 wr, level 3, files in(1, 1) out(2) MB in(71.1, 71.1) out(142.2), read-write-amplify(4.0) write-amplify(2.0) OK, records in: 5276480, records dropped: 0 > >2018-07-11 04:12:57.385709 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:12:57.385480) EVENT_LOG_v1 {"time_micros": 1531282377385470, "job": 2630, "event": "compaction_finished", "compaction_time_micros": 6192427, "output_level": 3, "num_output_files": 2, "total_output_size": 149092264, "num_input_records": 5276480, "num_output_records": 5276480, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 4, 47, 446, 3641, 701, 0]} >2018-07-11 04:12:57.386651 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282377386648, "job": 2630, "event": "table_file_deletion", "file_number": 35490} >2018-07-11 04:12:57.386679 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282377386677, "job": 2630, "event": "table_file_deletion", "file_number": 35435} >2018-07-11 04:12:57.386967 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2631] Compacting 1@2 + 1@3 files to L3, score 1.07 >2018-07-11 04:12:57.386978 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2630 Base level 2, inputs: [35489(71MB)], [35388(71MB)] > >2018-07-11 04:12:57.387004 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282377386981, "job": 2631, "event": "compaction_started", "files_L2": [35489], "files_L3": [35388], "score": 1.07234, "input_data_size": 149092270} >2018-07-11 04:13:00.531574 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2631] Generated table #35498: 2638240 keys, 74546811 bytes >2018-07-11 04:13:00.531623 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282380531592, "cf_name": "default", "job": 2631, "event": "table_file_creation", "file_number": 35498, "file_size": 74546811, "table_properties": {"data_size": 67112125, "index_size": 690515, "filter_size": 6743181, "raw_key_size": 76215829, "raw_average_key_size": 28, "raw_value_size": 31365749, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:13:03.448366 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2631] Generated table #35499: 2638240 keys, 74545838 bytes >2018-07-11 04:13:03.448397 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282383448383, "cf_name": "default", "job": 2631, "event": "table_file_creation", "file_number": 35499, "file_size": 74545838, "table_properties": {"data_size": 67112114, "index_size": 689553, "filter_size": 6743181, "raw_key_size": 76215825, "raw_average_key_size": 28, "raw_value_size": 31365745, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:13:03.494762 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2631] Generated table #35500: 2 keys, 1088 bytes >2018-07-11 04:13:03.494786 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282383494774, "cf_name": "default", "job": 2631, "event": "table_file_creation", "file_number": 35500, "file_size": 1088, "table_properties": {"data_size": 81, "index_size": 27, "filter_size": 18, "raw_key_size": 58, "raw_average_key_size": 29, "raw_value_size": 24, "raw_average_value_size": 12, "num_data_blocks": 1, "num_entries": 2, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:13:03.494820 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2631] Compacted 1@2 + 1@3 files to L3 => 149093737 bytes >2018-07-11 04:13:03.510645 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:13:03.510455) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 4 46 448 3641 701 0] max score 1.04, MB/sec: 24.4 rd, 24.4 wr, level 3, files in(1, 1) out(3) MB in(71.1, 71.1) out(142.2), read-write-amplify(4.0) write-amplify(2.0) OK, records in: 5276482, records dropped: 0 > >2018-07-11 04:13:03.510656 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:13:03.510480) EVENT_LOG_v1 {"time_micros": 1531282383510469, "job": 2631, "event": "compaction_finished", "compaction_time_micros": 6107793, "output_level": 3, "num_output_files": 3, "total_output_size": 149093737, "num_input_records": 5276482, "num_output_records": 5276482, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 4, 46, 448, 3641, 701, 0]} >2018-07-11 04:13:03.511632 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282383511629, "job": 2631, "event": "table_file_deletion", "file_number": 35489} >2018-07-11 04:13:03.511660 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282383511658, "job": 2631, "event": "table_file_deletion", "file_number": 35388} >2018-07-11 04:13:03.511947 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2632] Compacting 1@2 + 4@3 files to L3, score 1.04 >2018-07-11 04:13:03.511958 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2631 Base level 2, inputs: [35488(71MB)], [35349(1088B) 35315(71MB) 35282(71MB) 35286(71MB)] > >2018-07-11 04:13:03.511972 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282383511962, "job": 2632, "event": "compaction_started", "files_L2": [35488], "files_L3": [35349, 35315, 35282, 35286], "score": 1.04457, "input_data_size": 298185084} >2018-07-11 04:13:06.797849 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2632] Generated table #35501: 2638239 keys, 74546405 bytes >2018-07-11 04:13:06.797894 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282386797878, "cf_name": "default", "job": 2632, "event": "table_file_creation", "file_number": 35501, "file_size": 74546405, "table_properties": {"data_size": 67112136, "index_size": 690100, "filter_size": 6743179, "raw_key_size": 76215799, "raw_average_key_size": 28, "raw_value_size": 31365736, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638239, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:13:06.952978 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2632] Generated table #35502: 801 keys, 23681 bytes >2018-07-11 04:13:06.953018 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282386952993, "cf_name": "default", "job": 2632, "event": "table_file_creation", "file_number": 35502, "file_size": 23681, "table_properties": {"data_size": 20421, "index_size": 228, "filter_size": 2057, "raw_key_size": 23143, "raw_average_key_size": 28, "raw_value_size": 9526, "raw_average_value_size": 11, "num_data_blocks": 6, "num_entries": 801, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:13:10.093991 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2632] Generated table #35503: 2638241 keys, 74545944 bytes >2018-07-11 04:13:10.094035 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282390094008, "cf_name": "default", "job": 2632, "event": "table_file_creation", "file_number": 35503, "file_size": 74545944, "table_properties": {"data_size": 67112137, "index_size": 689633, "filter_size": 6743184, "raw_key_size": 76215850, "raw_average_key_size": 28, "raw_value_size": 31365753, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:13:13.144891 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2632] Generated table #35504: 2638240 keys, 74546324 bytes >2018-07-11 04:13:13.144937 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282393144921, "cf_name": "default", "job": 2632, "event": "table_file_creation", "file_number": 35504, "file_size": 74546324, "table_properties": {"data_size": 67112147, "index_size": 690006, "filter_size": 6743181, "raw_key_size": 76215821, "raw_average_key_size": 28, "raw_value_size": 31365741, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:13:16.063554 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2632] Generated table #35505: 2637443 keys, 74523339 bytes >2018-07-11 04:13:16.063599 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282396063571, "cf_name": "default", "job": 2632, "event": "table_file_creation", "file_number": 35505, "file_size": 74523339, "table_properties": {"data_size": 67091847, "index_size": 689364, "filter_size": 6741138, "raw_key_size": 76192798, "raw_average_key_size": 28, "raw_value_size": 31356267, "raw_average_value_size": 11, "num_data_blocks": 16485, "num_entries": 2637443, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:13:16.063634 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2632] Compacted 1@2 + 4@3 files to L3 => 298185693 bytes >2018-07-11 04:13:16.077167 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:13:16.076929) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 4 45 449 3641 701 0] max score 1.02, MB/sec: 23.8 rd, 23.8 wr, level 3, files in(1, 4) out(5) MB in(71.1, 213.3) out(284.4), read-write-amplify(8.0) write-amplify(4.0) OK, records in: 10552964, records dropped: 0 > >2018-07-11 04:13:16.077177 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:13:16.076954) EVENT_LOG_v1 {"time_micros": 1531282396076943, "job": 2632, "event": "compaction_finished", "compaction_time_micros": 12551647, "output_level": 3, "num_output_files": 5, "total_output_size": 298185693, "num_input_records": 10552964, "num_output_records": 10552964, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 4, 45, 449, 3641, 701, 0]} >2018-07-11 04:13:16.078102 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282396078098, "job": 2632, "event": "table_file_deletion", "file_number": 35488} >2018-07-11 04:13:16.078130 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282396078128, "job": 2632, "event": "table_file_deletion", "file_number": 35349} >2018-07-11 04:13:16.078144 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282396078142, "job": 2632, "event": "table_file_deletion", "file_number": 35315} >2018-07-11 04:13:16.078156 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282396078154, "job": 2632, "event": "table_file_deletion", "file_number": 35286} >2018-07-11 04:13:16.078171 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282396078169, "job": 2632, "event": "table_file_deletion", "file_number": 35282} >2018-07-11 04:13:16.078388 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_picker.cc:461] [default] Expanding@2 1+1(74546513+46965004 bytes) to 2+1 (149092269+46965004bytes) > >2018-07-11 04:13:16.078484 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2633] Compacting 2@2 + 1@3 files to L3, score 1.02 >2018-07-11 04:13:16.078494 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2632 Base level 2, inputs: [35493(71MB) 35494(71MB)], [35475(44MB)] > >2018-07-11 04:13:16.078508 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282396078498, "job": 2633, "event": "compaction_started", "files_L2": [35493, 35494], "files_L3": [35475], "score": 1.0168, "input_data_size": 196057273} >2018-07-11 04:13:19.297601 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2633] Generated table #35506: 2638241 keys, 74546217 bytes >2018-07-11 04:13:19.297646 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282399297619, "cf_name": "default", "job": 2633, "event": "table_file_creation", "file_number": 35506, "file_size": 74546217, "table_properties": {"data_size": 67112152, "index_size": 689892, "filter_size": 6743183, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:13:22.509174 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2633] Generated table #35507: 2638241 keys, 74546829 bytes >2018-07-11 04:13:22.509221 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282402509193, "cf_name": "default", "job": 2633, "event": "table_file_creation", "file_number": 35507, "file_size": 74546829, "table_properties": {"data_size": 67112154, "index_size": 690502, "filter_size": 6743183, "raw_key_size": 76215849, "raw_average_key_size": 28, "raw_value_size": 31365752, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:13:24.491984 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2633] Generated table #35508: 1519916 keys, 46964939 bytes >2018-07-11 04:13:24.492040 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282404492023, "cf_name": "default", "job": 2633, "event": "table_file_creation", "file_number": 35508, "file_size": 46964939, "table_properties": {"data_size": 42636625, "index_size": 433175, "filter_size": 3894151, "raw_key_size": 44348943, "raw_average_key_size": 29, "raw_value_size": 21522472, "raw_average_value_size": 14, "num_data_blocks": 10538, "num_entries": 1519916, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:13:24.492078 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2633] Compacted 2@2 + 1@3 files to L3 => 196057985 bytes >2018-07-11 04:13:24.510189 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:13:24.509988) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 4 43 451 3641 701 0] max score 1.01, MB/sec: 23.3 rd, 23.3 wr, level 3, files in(2, 1) out(3) MB in(142.2, 44.8) out(187.0), read-write-amplify(2.6) write-amplify(1.3) OK, records in: 6796398, records dropped: 0 > >2018-07-11 04:13:24.510200 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:13:24.510013) EVENT_LOG_v1 {"time_micros": 1531282404510002, "job": 2633, "event": "compaction_finished", "compaction_time_micros": 8413554, "output_level": 3, "num_output_files": 3, "total_output_size": 196057985, "num_input_records": 6796398, "num_output_records": 6796398, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 4, 43, 451, 3641, 701, 0]} >2018-07-11 04:13:24.511138 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282404511120, "job": 2633, "event": "table_file_deletion", "file_number": 35494} >2018-07-11 04:13:24.511154 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282404511151, "job": 2633, "event": "table_file_deletion", "file_number": 35493} >2018-07-11 04:13:24.511167 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282404511165, "job": 2633, "event": "table_file_deletion", "file_number": 35475} >2018-07-11 04:13:24.526815 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:13:24.511471) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #35507 to level-4 74546829 bytes > >2018-07-11 04:13:24.526826 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:13:24.526717) EVENT_LOG_v1 {"time_micros": 1531282404526710, "job": 2634, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 74546829} >2018-07-11 04:13:24.526829 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:13:24.526726) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-4 74546829 bytes OK: base level 1 max bytes base 268435456 files[0 4 43 450 3642 701 0] max score 1.01 > >2018-07-11 04:13:24.527687 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_picker.cc:461] [default] Expanding@3 1+6(74546811+447277633 bytes) to 2+6 (149070150+447277633bytes) > >2018-07-11 04:13:24.527798 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2635] Compacting 2@3 + 6@4 files to L4, score 1.01 >2018-07-11 04:13:24.527812 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2634 Base level 3, inputs: [35505(71MB) 35498(71MB)], [35390(71MB) 35391(71MB) 35392(71MB) 35385(71MB) 35386(71MB) 35387(71MB)] > >2018-07-11 04:13:24.527827 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282404527815, "job": 2635, "event": "compaction_started", "files_L3": [35505, 35498], "files_L4": [35390, 35391, 35392, 35385, 35386, 35387], "score": 1.00846, "input_data_size": 596347783} >2018-07-11 04:13:27.931916 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2635] Generated table #35509: 2638241 keys, 74546487 bytes >2018-07-11 04:13:27.931961 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282407931934, "cf_name": "default", "job": 2635, "event": "table_file_creation", "file_number": 35509, "file_size": 74546487, "table_properties": {"data_size": 67112128, "index_size": 690186, "filter_size": 6743183, "raw_key_size": 76215851, "raw_average_key_size": 28, "raw_value_size": 31365754, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:13:31.215148 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2635] Generated table #35510: 2638241 keys, 74546755 bytes >2018-07-11 04:13:31.215191 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282411215164, "cf_name": "default", "job": 2635, "event": "table_file_creation", "file_number": 35510, "file_size": 74546755, "table_properties": {"data_size": 67112147, "index_size": 690435, "filter_size": 6743183, "raw_key_size": 76215853, "raw_average_key_size": 28, "raw_value_size": 31365756, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:13:34.325892 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2635] Generated table #35511: 2638241 keys, 74546205 bytes >2018-07-11 04:13:34.325925 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282414325910, "cf_name": "default", "job": 2635, "event": "table_file_creation", "file_number": 35511, "file_size": 74546205, "table_properties": {"data_size": 67112172, "index_size": 689860, "filter_size": 6743183, "raw_key_size": 76215859, "raw_average_key_size": 28, "raw_value_size": 31365762, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:13:37.509908 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2635] Generated table #35512: 2638241 keys, 74546034 bytes >2018-07-11 04:13:37.509955 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282417509940, "cf_name": "default", "job": 2635, "event": "table_file_creation", "file_number": 35512, "file_size": 74546034, "table_properties": {"data_size": 67112096, "index_size": 689765, "filter_size": 6743183, "raw_key_size": 76215850, "raw_average_key_size": 28, "raw_value_size": 31365753, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:13:40.665344 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2635] Generated table #35513: 2638241 keys, 74546984 bytes >2018-07-11 04:13:40.665387 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282420665360, "cf_name": "default", "job": 2635, "event": "table_file_creation", "file_number": 35513, "file_size": 74546984, "table_properties": {"data_size": 67112154, "index_size": 690657, "filter_size": 6743183, "raw_key_size": 76215850, "raw_average_key_size": 28, "raw_value_size": 31365753, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:13:43.849344 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2635] Generated table #35514: 2638241 keys, 74546318 bytes >2018-07-11 04:13:43.849390 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282423849363, "cf_name": "default", "job": 2635, "event": "table_file_creation", "file_number": 35514, "file_size": 74546318, "table_properties": {"data_size": 67112174, "index_size": 689971, "filter_size": 6743183, "raw_key_size": 76215851, "raw_average_key_size": 28, "raw_value_size": 31365754, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:13:46.910767 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2635] Generated table #35515: 2638241 keys, 74545741 bytes >2018-07-11 04:13:46.910812 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282426910785, "cf_name": "default", "job": 2635, "event": "table_file_creation", "file_number": 35515, "file_size": 74545741, "table_properties": {"data_size": 67112143, "index_size": 689425, "filter_size": 6743183, "raw_key_size": 76215851, "raw_average_key_size": 28, "raw_value_size": 31365754, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:13:49.840630 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2635] Generated table #35516: 2637441 keys, 74524219 bytes >2018-07-11 04:13:49.840662 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282429840647, "cf_name": "default", "job": 2635, "event": "table_file_creation", "file_number": 35516, "file_size": 74524219, "table_properties": {"data_size": 67091785, "index_size": 690306, "filter_size": 6741138, "raw_key_size": 76192745, "raw_average_key_size": 28, "raw_value_size": 31356248, "raw_average_value_size": 11, "num_data_blocks": 16485, "num_entries": 2637441, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:13:49.840701 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2635] Compacted 2@3 + 6@4 files to L4 => 596348743 bytes >2018-07-11 04:13:49.859962 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:13:49.859647) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 4 43 448 3644 701 0] max score 1.00, MB/sec: 23.6 rd, 23.6 wr, level 4, files in(2, 6) out(8) MB in(142.2, 426.6) out(568.7), read-write-amplify(8.0) write-amplify(4.0) OK, records in: 21105128, records dropped: 0 > >2018-07-11 04:13:49.859973 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:13:49.859673) EVENT_LOG_v1 {"time_micros": 1531282429859661, "job": 2635, "event": "compaction_finished", "compaction_time_micros": 25312854, "output_level": 4, "num_output_files": 8, "total_output_size": 596348743, "num_input_records": 21105128, "num_output_records": 21105128, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 4, 43, 448, 3644, 701, 0]} >2018-07-11 04:13:49.860913 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282429860896, "job": 2635, "event": "table_file_deletion", "file_number": 35505} >2018-07-11 04:13:49.860929 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282429860926, "job": 2635, "event": "table_file_deletion", "file_number": 35498} >2018-07-11 04:13:49.860943 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282429860940, "job": 2635, "event": "table_file_deletion", "file_number": 35392} >2018-07-11 04:13:49.860955 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282429860953, "job": 2635, "event": "table_file_deletion", "file_number": 35391} >2018-07-11 04:13:49.860966 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282429860964, "job": 2635, "event": "table_file_deletion", "file_number": 35390} >2018-07-11 04:13:49.860980 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282429860978, "job": 2635, "event": "table_file_deletion", "file_number": 35387} >2018-07-11 04:13:49.860993 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282429860991, "job": 2635, "event": "table_file_deletion", "file_number": 35386} >2018-07-11 04:13:49.861006 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282429861004, "job": 2635, "event": "table_file_deletion", "file_number": 35385} >2018-07-11 04:13:49.861330 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2636] Compacting 1@3 + 21@4 files to L4, score 1.00 >2018-07-11 04:13:49.861349 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2635 Base level 3, inputs: [35501(71MB)], [35452(71MB) 35453(71MB) 35449(71MB) 35451(71MB) 35355(71MB) 35356(71MB) 35359(71MB) 35363(71MB) 35364(71MB) 35365(71MB) 35366(71MB) 35367(71MB) 35368(71MB) 35218(71MB) 35219(71MB) 35220(71MB) 35221(71MB) 35222(71MB) 35223(1064B) 35124(71MB) 35322(71MB)] > >2018-07-11 04:13:49.861368 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282429861356, "job": 2636, "event": "compaction_started", "files_L3": [35501], "files_L4": [35452, 35453, 35449, 35451, 35355, 35356, 35359, 35363, 35364, 35365, 35366, 35367, 35368, 35218, 35219, 35220, 35221, 35222, 35223, 35124, 35322], "score": 1.00291, "input_data_size": 1565473294} >2018-07-11 04:13:53.074137 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2636] Generated table #35517: 2638241 keys, 74546329 bytes >2018-07-11 04:13:53.074183 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282433074168, "cf_name": "default", "job": 2636, "event": "table_file_creation", "file_number": 35517, "file_size": 74546329, "table_properties": {"data_size": 67112201, "index_size": 689954, "filter_size": 6743184, "raw_key_size": 76215857, "raw_average_key_size": 28, "raw_value_size": 31365760, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:13:56.253334 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2636] Generated table #35518: 2638241 keys, 74546111 bytes >2018-07-11 04:13:56.253368 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282436253353, "cf_name": "default", "job": 2636, "event": "table_file_creation", "file_number": 35518, "file_size": 74546111, "table_properties": {"data_size": 67112163, "index_size": 689775, "filter_size": 6743183, "raw_key_size": 76215853, "raw_average_key_size": 28, "raw_value_size": 31365756, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:13:59.459519 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2636] Generated table #35519: 2638241 keys, 74547018 bytes >2018-07-11 04:13:59.459565 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282439459538, "cf_name": "default", "job": 2636, "event": "table_file_creation", "file_number": 35519, "file_size": 74547018, "table_properties": {"data_size": 67112102, "index_size": 690743, "filter_size": 6743183, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:14:02.555256 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2636] Generated table #35520: 2638240 keys, 74546241 bytes >2018-07-11 04:14:02.555299 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282442555272, "cf_name": "default", "job": 2636, "event": "table_file_creation", "file_number": 35520, "file_size": 74546241, "table_properties": {"data_size": 67112158, "index_size": 689912, "filter_size": 6743181, "raw_key_size": 76215835, "raw_average_key_size": 28, "raw_value_size": 31365755, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:14:05.632013 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2636] Generated table #35521: 2638241 keys, 74546454 bytes >2018-07-11 04:14:05.632058 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282445632031, "cf_name": "default", "job": 2636, "event": "table_file_creation", "file_number": 35521, "file_size": 74546454, "table_properties": {"data_size": 67112131, "index_size": 690150, "filter_size": 6743183, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:14:08.664599 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2636] Generated table #35522: 2638241 keys, 74545891 bytes >2018-07-11 04:14:08.664645 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282448664629, "cf_name": "default", "job": 2636, "event": "table_file_creation", "file_number": 35522, "file_size": 74545891, "table_properties": {"data_size": 67112140, "index_size": 689578, "filter_size": 6743183, "raw_key_size": 76215847, "raw_average_key_size": 28, "raw_value_size": 31365750, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:14:11.770135 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2636] Generated table #35523: 2638241 keys, 74546897 bytes >2018-07-11 04:14:11.770180 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282451770153, "cf_name": "default", "job": 2636, "event": "table_file_creation", "file_number": 35523, "file_size": 74546897, "table_properties": {"data_size": 67112148, "index_size": 690576, "filter_size": 6743183, "raw_key_size": 76215860, "raw_average_key_size": 28, "raw_value_size": 31365763, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:14:14.958870 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2636] Generated table #35524: 2638241 keys, 74546470 bytes >2018-07-11 04:14:14.958915 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282454958888, "cf_name": "default", "job": 2636, "event": "table_file_creation", "file_number": 35524, "file_size": 74546470, "table_properties": {"data_size": 67112122, "index_size": 690175, "filter_size": 6743183, "raw_key_size": 76215852, "raw_average_key_size": 28, "raw_value_size": 31365755, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:14:18.142257 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2636] Generated table #35525: 2638240 keys, 74546143 bytes >2018-07-11 04:14:18.142302 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282458142275, "cf_name": "default", "job": 2636, "event": "table_file_creation", "file_number": 35525, "file_size": 74546143, "table_properties": {"data_size": 67112137, "index_size": 689835, "filter_size": 6743181, "raw_key_size": 76215822, "raw_average_key_size": 28, "raw_value_size": 31365742, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:14:21.253381 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2636] Generated table #35526: 2638241 keys, 74545916 bytes >2018-07-11 04:14:21.253426 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282461253399, "cf_name": "default", "job": 2636, "event": "table_file_creation", "file_number": 35526, "file_size": 74545916, "table_properties": {"data_size": 67112102, "index_size": 689641, "filter_size": 6743183, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:14:24.332062 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2636] Generated table #35527: 2638241 keys, 74546268 bytes >2018-07-11 04:14:24.332111 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282464332081, "cf_name": "default", "job": 2636, "event": "table_file_creation", "file_number": 35527, "file_size": 74546268, "table_properties": {"data_size": 67112170, "index_size": 689925, "filter_size": 6743183, "raw_key_size": 76215864, "raw_average_key_size": 28, "raw_value_size": 31365767, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:14:27.392186 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2636] Generated table #35528: 2638241 keys, 74547011 bytes >2018-07-11 04:14:27.392228 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282467392202, "cf_name": "default", "job": 2636, "event": "table_file_creation", "file_number": 35528, "file_size": 74547011, "table_properties": {"data_size": 67112113, "index_size": 690725, "filter_size": 6743183, "raw_key_size": 76215848, "raw_average_key_size": 28, "raw_value_size": 31365751, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:14:30.684645 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2636] Generated table #35529: 2638241 keys, 74546300 bytes >2018-07-11 04:14:30.684690 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282470684663, "cf_name": "default", "job": 2636, "event": "table_file_creation", "file_number": 35529, "file_size": 74546300, "table_properties": {"data_size": 67112167, "index_size": 689960, "filter_size": 6743183, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:14:33.922048 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2636] Generated table #35530: 2638240 keys, 74545891 bytes >2018-07-11 04:14:33.922103 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282473922087, "cf_name": "default", "job": 2636, "event": "table_file_creation", "file_number": 35530, "file_size": 74545891, "table_properties": {"data_size": 67112133, "index_size": 689587, "filter_size": 6743181, "raw_key_size": 76215820, "raw_average_key_size": 28, "raw_value_size": 31365740, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:14:37.094950 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2636] Generated table #35531: 2638242 keys, 74546612 bytes >2018-07-11 04:14:37.094994 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282477094968, "cf_name": "default", "job": 2636, "event": "table_file_creation", "file_number": 35531, "file_size": 74546612, "table_properties": {"data_size": 67112177, "index_size": 690259, "filter_size": 6743186, "raw_key_size": 76215885, "raw_average_key_size": 28, "raw_value_size": 31365771, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638242, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:14:40.230085 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2636] Generated table #35532: 2638241 keys, 74546278 bytes >2018-07-11 04:14:40.230135 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282480230119, "cf_name": "default", "job": 2636, "event": "table_file_creation", "file_number": 35532, "file_size": 74546278, "table_properties": {"data_size": 67112165, "index_size": 689940, "filter_size": 6743183, "raw_key_size": 76215859, "raw_average_key_size": 28, "raw_value_size": 31365762, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:14:43.417100 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2636] Generated table #35533: 2638241 keys, 74545870 bytes >2018-07-11 04:14:43.417139 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282483417122, "cf_name": "default", "job": 2636, "event": "table_file_creation", "file_number": 35533, "file_size": 74545870, "table_properties": {"data_size": 67112097, "index_size": 689600, "filter_size": 6743183, "raw_key_size": 76215846, "raw_average_key_size": 28, "raw_value_size": 31365749, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:14:46.618584 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2636] Generated table #35534: 2638241 keys, 74546470 bytes >2018-07-11 04:14:46.618633 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282486618617, "cf_name": "default", "job": 2636, "event": "table_file_creation", "file_number": 35534, "file_size": 74546470, "table_properties": {"data_size": 67112161, "index_size": 690136, "filter_size": 6743183, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:14:49.912811 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2636] Generated table #35535: 2638241 keys, 74546277 bytes >2018-07-11 04:14:49.912860 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282489912840, "cf_name": "default", "job": 2636, "event": "table_file_creation", "file_number": 35535, "file_size": 74546277, "table_properties": {"data_size": 67112158, "index_size": 689946, "filter_size": 6743183, "raw_key_size": 76215857, "raw_average_key_size": 28, "raw_value_size": 31365760, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:14:53.115318 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2636] Generated table #35536: 2638241 keys, 74546388 bytes >2018-07-11 04:14:53.115361 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282493115345, "cf_name": "default", "job": 2636, "event": "table_file_creation", "file_number": 35536, "file_size": 74546388, "table_properties": {"data_size": 67112150, "index_size": 690065, "filter_size": 6743183, "raw_key_size": 76215855, "raw_average_key_size": 28, "raw_value_size": 31365758, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:14:56.227991 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2636] Generated table #35537: 2638236 keys, 74545897 bytes >2018-07-11 04:14:56.228032 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282496228016, "cf_name": "default", "job": 2636, "event": "table_file_creation", "file_number": 35537, "file_size": 74545897, "table_properties": {"data_size": 67111987, "index_size": 689760, "filter_size": 6743160, "raw_key_size": 76215704, "raw_average_key_size": 28, "raw_value_size": 31365692, "raw_average_value_size": 11, "num_data_blocks": 16489, "num_entries": 2638236, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:14:56.228086 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2636] Compacted 1@3 + 21@4 files to L4 => 1565472732 bytes >2018-07-11 04:14:56.242066 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:14:56.241625) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 4 43 447 3644 701 0] max score 1.00, MB/sec: 23.6 rd, 23.6 wr, level 4, files in(1, 21) out(21) MB in(71.1, 1421.9) out(1493.0), read-write-amplify(42.0) write-amplify(21.0) OK, records in: 55403054, records dropped: 0 > >2018-07-11 04:14:56.242077 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:14:56.241651) EVENT_LOG_v1 {"time_micros": 1531282496241640, "job": 2636, "event": "compaction_finished", "compaction_time_micros": 66366683, "output_level": 4, "num_output_files": 21, "total_output_size": 1565472732, "num_input_records": 55403054, "num_output_records": 55403054, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 4, 43, 447, 3644, 701, 0]} >2018-07-11 04:14:56.242974 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282496242970, "job": 2636, "event": "table_file_deletion", "file_number": 35501} >2018-07-11 04:14:56.242989 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282496242987, "job": 2636, "event": "table_file_deletion", "file_number": 35453} >2018-07-11 04:14:56.243002 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282496243000, "job": 2636, "event": "table_file_deletion", "file_number": 35452} >2018-07-11 04:14:56.243013 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282496243011, "job": 2636, "event": "table_file_deletion", "file_number": 35451} >2018-07-11 04:14:56.243024 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282496243022, "job": 2636, "event": "table_file_deletion", "file_number": 35449} >2018-07-11 04:14:56.243036 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282496243034, "job": 2636, "event": "table_file_deletion", "file_number": 35368} >2018-07-11 04:14:56.243047 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282496243045, "job": 2636, "event": "table_file_deletion", "file_number": 35367} >2018-07-11 04:14:56.243058 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282496243056, "job": 2636, "event": "table_file_deletion", "file_number": 35366} >2018-07-11 04:14:56.243068 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282496243066, "job": 2636, "event": "table_file_deletion", "file_number": 35365} >2018-07-11 04:14:56.243083 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282496243081, "job": 2636, "event": "table_file_deletion", "file_number": 35364} >2018-07-11 04:14:56.243097 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282496243095, "job": 2636, "event": "table_file_deletion", "file_number": 35363} >2018-07-11 04:14:56.243112 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282496243110, "job": 2636, "event": "table_file_deletion", "file_number": 35359} >2018-07-11 04:14:56.243126 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282496243123, "job": 2636, "event": "table_file_deletion", "file_number": 35356} >2018-07-11 04:14:56.243139 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282496243137, "job": 2636, "event": "table_file_deletion", "file_number": 35355} >2018-07-11 04:14:56.243155 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282496243153, "job": 2636, "event": "table_file_deletion", "file_number": 35322} >2018-07-11 04:14:56.243172 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282496243170, "job": 2636, "event": "table_file_deletion", "file_number": 35223} >2018-07-11 04:14:56.243185 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282496243183, "job": 2636, "event": "table_file_deletion", "file_number": 35222} >2018-07-11 04:14:56.243206 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282496243204, "job": 2636, "event": "table_file_deletion", "file_number": 35221} >2018-07-11 04:14:56.243222 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282496243220, "job": 2636, "event": "table_file_deletion", "file_number": 35220} >2018-07-11 04:14:56.243236 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282496243234, "job": 2636, "event": "table_file_deletion", "file_number": 35219} >2018-07-11 04:14:56.243254 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282496243252, "job": 2636, "event": "table_file_deletion", "file_number": 35218} >2018-07-11 04:14:56.243270 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282496243268, "job": 2636, "event": "table_file_deletion", "file_number": 35124} >2018-07-11 04:14:56.258411 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:14:56.243570) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #35519 to level-5 74547018 bytes > >2018-07-11 04:14:56.258422 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:14:56.258319) EVENT_LOG_v1 {"time_micros": 1531282496258313, "job": 2637, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74547018} >2018-07-11 04:14:56.258425 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:14:56.258327) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74547018 bytes OK: base level 1 max bytes base 268435456 files[0 4 43 447 3643 702 0] max score 1.00 > >2018-07-11 04:14:56.275075 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:14:56.259237) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #35528 to level-5 74547011 bytes > >2018-07-11 04:14:56.275084 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:14:56.274984) EVENT_LOG_v1 {"time_micros": 1531282496274978, "job": 2638, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74547011} >2018-07-11 04:14:56.275087 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:14:56.274992) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74547011 bytes OK: base level 1 max bytes base 268435456 files[0 4 43 447 3642 703 0] max score 1.00 > >2018-07-11 04:14:56.275937 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2639] Compacting 1@4 + 1@5 files to L5, score 1.00 >2018-07-11 04:14:56.275949 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2638 Base level 4, inputs: [35513(71MB)], [35352(71MB)] > >2018-07-11 04:14:56.275963 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282496275952, "job": 2639, "event": "compaction_started", "files_L4": [35513], "files_L5": [35352], "score": 1.00055, "input_data_size": 149094056} >2018-07-11 04:14:59.578999 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2639] Generated table #35538: 2638241 keys, 74546998 bytes >2018-07-11 04:14:59.579051 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282499579022, "cf_name": "default", "job": 2639, "event": "table_file_creation", "file_number": 35538, "file_size": 74546998, "table_properties": {"data_size": 67112157, "index_size": 690668, "filter_size": 6743183, "raw_key_size": 76215852, "raw_average_key_size": 28, "raw_value_size": 31365755, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:15:02.687474 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2639] Generated table #35539: 2638241 keys, 74546002 bytes >2018-07-11 04:15:02.687520 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282502687492, "cf_name": "default", "job": 2639, "event": "table_file_creation", "file_number": 35539, "file_size": 74546002, "table_properties": {"data_size": 67112107, "index_size": 689722, "filter_size": 6743183, "raw_key_size": 76215858, "raw_average_key_size": 28, "raw_value_size": 31365761, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:15:02.687552 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2639] Compacted 1@4 + 1@5 files to L5 => 149093000 bytes >2018-07-11 04:15:02.700276 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:15:02.700001) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 4 43 447 3641 704 0] max score 1.00, MB/sec: 23.3 rd, 23.3 wr, level 5, files in(1, 1) out(2) MB in(71.1, 71.1) out(142.2), read-write-amplify(4.0) write-amplify(2.0) OK, records in: 5276482, records dropped: 0 > >2018-07-11 04:15:02.700287 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:15:02.700025) EVENT_LOG_v1 {"time_micros": 1531282502700015, "job": 2639, "event": "compaction_finished", "compaction_time_micros": 6411578, "output_level": 5, "num_output_files": 2, "total_output_size": 149093000, "num_input_records": 5276482, "num_output_records": 5276482, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 4, 43, 447, 3641, 704, 0]} >2018-07-11 04:15:02.701197 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282502701193, "job": 2639, "event": "table_file_deletion", "file_number": 35513} >2018-07-11 04:15:02.701227 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282502701225, "job": 2639, "event": "table_file_deletion", "file_number": 35352} >2018-07-11 04:15:02.716835 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:15:02.701513) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #35523 to level-5 74546897 bytes > >2018-07-11 04:15:02.716846 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:15:02.716745) EVENT_LOG_v1 {"time_micros": 1531282502716739, "job": 2640, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546897} >2018-07-11 04:15:02.716849 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:15:02.716752) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546897 bytes OK: base level 1 max bytes base 268435456 files[0 4 43 447 3640 705 0] max score 1.00 > >2018-07-11 04:15:02.717638 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_picker.cc:461] [default] Expanding@3 1+3(74546325+223639045 bytes) to 3+3 (149093251+223639045bytes) > >2018-07-11 04:15:02.717744 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2641] Compacting 3@3 + 3@4 files to L4, score 1.00 >2018-07-11 04:15:02.717756 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2640 Base level 3, inputs: [35499(71MB) 35500(1088B) 35496(71MB)], [35413(71MB) 35414(71MB) 35415(71MB)] > >2018-07-11 04:15:02.717770 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282502717760, "job": 2641, "event": "compaction_started", "files_L3": [35499, 35500, 35496], "files_L4": [35413, 35414, 35415], "score": 1.00013, "input_data_size": 372732296} >2018-07-11 04:15:06.096323 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2641] Generated table #35540: 2638242 keys, 74546354 bytes >2018-07-11 04:15:06.096374 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282506096356, "cf_name": "default", "job": 2641, "event": "table_file_creation", "file_number": 35540, "file_size": 74546354, "table_properties": {"data_size": 67112161, "index_size": 690017, "filter_size": 6743186, "raw_key_size": 76215888, "raw_average_key_size": 28, "raw_value_size": 31365774, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638242, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:15:09.245027 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2641] Generated table #35541: 2638241 keys, 74546092 bytes >2018-07-11 04:15:09.245080 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282509245064, "cf_name": "default", "job": 2641, "event": "table_file_creation", "file_number": 35541, "file_size": 74546092, "table_properties": {"data_size": 67112148, "index_size": 689771, "filter_size": 6743183, "raw_key_size": 76215847, "raw_average_key_size": 28, "raw_value_size": 31365750, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:15:12.477570 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2641] Generated table #35542: 2638241 keys, 74546169 bytes >2018-07-11 04:15:12.477628 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282512477610, "cf_name": "default", "job": 2641, "event": "table_file_creation", "file_number": 35542, "file_size": 74546169, "table_properties": {"data_size": 67112099, "index_size": 689897, "filter_size": 6743183, "raw_key_size": 76215860, "raw_average_key_size": 28, "raw_value_size": 31365763, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:15:15.683717 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2641] Generated table #35543: 2638241 keys, 74546966 bytes >2018-07-11 04:15:15.683766 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282515683750, "cf_name": "default", "job": 2641, "event": "table_file_creation", "file_number": 35543, "file_size": 74546966, "table_properties": {"data_size": 67112160, "index_size": 690633, "filter_size": 6743183, "raw_key_size": 76215845, "raw_average_key_size": 28, "raw_value_size": 31365748, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:15:18.546912 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2641] Generated table #35544: 2638241 keys, 74546351 bytes >2018-07-11 04:15:18.546957 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282518546930, "cf_name": "default", "job": 2641, "event": "table_file_creation", "file_number": 35544, "file_size": 74546351, "table_properties": {"data_size": 67112170, "index_size": 690008, "filter_size": 6743183, "raw_key_size": 76215850, "raw_average_key_size": 28, "raw_value_size": 31365753, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:15:18.546996 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2641] Compacted 3@3 + 3@4 files to L4 => 372731932 bytes >2018-07-11 04:15:18.566731 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:15:18.566486) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 4 43 444 3642 705 0] max score 1.00, MB/sec: 23.5 rd, 23.5 wr, level 4, files in(3, 3) out(5) MB in(142.2, 213.3) out(355.5), read-write-amplify(5.0) write-amplify(2.5) OK, records in: 13191206, records dropped: 0 > >2018-07-11 04:15:18.566741 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:15:18.566511) EVENT_LOG_v1 {"time_micros": 1531282518566500, "job": 2641, "event": "compaction_finished", "compaction_time_micros": 15829210, "output_level": 4, "num_output_files": 5, "total_output_size": 372731932, "num_input_records": 13191206, "num_output_records": 13191206, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 4, 43, 444, 3642, 705, 0]} >2018-07-11 04:15:18.567689 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282518567685, "job": 2641, "event": "table_file_deletion", "file_number": 35500} >2018-07-11 04:15:18.567717 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282518567714, "job": 2641, "event": "table_file_deletion", "file_number": 35499} >2018-07-11 04:15:18.567729 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282518567727, "job": 2641, "event": "table_file_deletion", "file_number": 35496} >2018-07-11 04:15:18.567740 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282518567738, "job": 2641, "event": "table_file_deletion", "file_number": 35415} >2018-07-11 04:15:18.567752 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282518567750, "job": 2641, "event": "table_file_deletion", "file_number": 35414} >2018-07-11 04:15:18.567765 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282518567763, "job": 2641, "event": "table_file_deletion", "file_number": 35413} >2018-07-11 04:15:18.583239 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:15:18.568050) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #35543 to level-5 74546966 bytes > >2018-07-11 04:15:18.583259 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:15:18.583135) EVENT_LOG_v1 {"time_micros": 1531282518583129, "job": 2642, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546966} >2018-07-11 04:15:18.583262 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:15:18.583143) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546966 bytes OK: base level 1 max bytes base 268435456 files[0 4 43 444 3641 706 0] max score 1.00 > >2018-07-11 04:15:18.599986 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:15:18.584060) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #33257 to level-5 74546861 bytes > >2018-07-11 04:15:18.599997 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:15:18.599897) EVENT_LOG_v1 {"time_micros": 1531282518599891, "job": 2643, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546861} >2018-07-11 04:15:18.599999 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:15:18.599905) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546861 bytes OK: base level 1 max bytes base 268435456 files[0 4 43 444 3640 707 0] max score 1.00 > >2018-07-11 04:16:21.416980 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:684] reusing log 35476 from recycle list > >2018-07-11 04:16:21.417144 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #35545. Immutable memtables: 0. > >2018-07-11 04:16:21.500856 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:16:21.417213) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1 >2018-07-11 04:16:21.500868 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 2644] Flushing memtable with next log file: 35545 > >2018-07-11 04:16:21.500886 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282581500878, "job": 2644, "event": "flush_started", "num_memtables": 1, "num_entries": 5000015, "num_deletes": 0, "memory_usage": 292950920} >2018-07-11 04:16:21.500889 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 2644] Level-0 flush table #35546: started >2018-07-11 04:16:25.454490 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282585454469, "cf_name": "default", "job": 2644, "event": "table_file_creation", "file_number": 35546, "file_size": 141286889, "table_properties": {"data_size": 127197777, "index_size": 1308397, "filter_size": 12779725, "raw_key_size": 144445101, "raw_average_key_size": 28, "raw_value_size": 59450671, "raw_average_value_size": 11, "num_data_blocks": 31252, "num_entries": 5000014, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:16:25.454511 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 2644] Level-0 flush table #35546: 141286889 bytes OK >2018-07-11 04:16:25.473198 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:242] adding log 35478 to recycle list > >2018-07-11 04:16:25.473210 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:16:25.454524) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #35546 started >2018-07-11 04:16:25.473214 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:16:25.472983) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #35546: memtable #1 done >2018-07-11 04:16:25.473216 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:16:25.473000) EVENT_LOG_v1 {"time_micros": 1531282585472993, "job": 2644, "event": "flush_finished", "lsm_state": [1, 4, 43, 444, 3640, 707, 0], "immutable_memtables": 0} >2018-07-11 04:16:25.473218 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:16:25.473145) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 4 43 444 3640 707 0] max score 1.00 > >2018-07-11 04:16:25.474082 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 2644] Try to delete WAL files size 178896755, prev total WAL file size 178896767, number of live WAL files 2. > >2018-07-11 04:17:53.428649 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:684] reusing log 35478 from recycle list > >2018-07-11 04:17:53.428760 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #35547. Immutable memtables: 0. > >2018-07-11 04:17:53.495224 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:17:53.428798) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1 >2018-07-11 04:17:53.495234 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 2645] Flushing memtable with next log file: 35547 > >2018-07-11 04:17:53.495258 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282673495242, "job": 2645, "event": "flush_started", "num_memtables": 1, "num_entries": 5000015, "num_deletes": 0, "memory_usage": 292936792} >2018-07-11 04:17:53.495261 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 2645] Level-0 flush table #35548: started >2018-07-11 04:17:57.459997 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282677459978, "cf_name": "default", "job": 2645, "event": "table_file_creation", "file_number": 35548, "file_size": 141288967, "table_properties": {"data_size": 127198839, "index_size": 1309411, "filter_size": 12779727, "raw_key_size": 144445125, "raw_average_key_size": 28, "raw_value_size": 59451570, "raw_average_value_size": 11, "num_data_blocks": 31252, "num_entries": 5000015, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:17:57.460019 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 2645] Level-0 flush table #35548: 141288967 bytes OK >2018-07-11 04:17:57.495728 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:242] adding log 35545 to recycle list > >2018-07-11 04:17:57.495742 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:17:57.460044) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #35548 started >2018-07-11 04:17:57.495748 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:17:57.495462) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #35548: memtable #1 done >2018-07-11 04:17:57.495751 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:17:57.495480) EVENT_LOG_v1 {"time_micros": 1531282677495472, "job": 2645, "event": "flush_finished", "lsm_state": [2, 4, 43, 444, 3640, 707, 0], "immutable_memtables": 0} >2018-07-11 04:17:57.495755 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:17:57.495663) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[2 4 43 444 3640 707 0] max score 1.05 > >2018-07-11 04:17:57.495862 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2646] Compacting 2@0 + 2@1 files to L1, score 1.05 >2018-07-11 04:17:57.495878 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2645 Base level 0, inputs: [35548(134MB) 35546(134MB)], [35480(71MB) 35485(65MB)] > >2018-07-11 04:17:57.495896 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282677495882, "job": 2646, "event": "compaction_started", "files_L0": [35548, 35546], "files_L1": [35480, 35485], "score": 1.05268, "input_data_size": 425331296} >2018-07-11 04:17:57.496668 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 2645] Try to delete WAL files size 178896755, prev total WAL file size 178896767, number of live WAL files 2. > >2018-07-11 04:18:02.402171 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2646] Generated table #35549: 2624258 keys, 74510883 bytes >2018-07-11 04:18:02.402207 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282682402192, "cf_name": "default", "job": 2646, "event": "table_file_creation", "file_number": 35549, "file_size": 74510883, "table_properties": {"data_size": 67110315, "index_size": 691343, "filter_size": 6708235, "raw_key_size": 75842490, "raw_average_key_size": 28, "raw_value_size": 31545790, "raw_average_value_size": 12, "num_data_blocks": 16475, "num_entries": 2624258, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:18:05.850605 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2646] Generated table #35550: 2638242 keys, 74546423 bytes >2018-07-11 04:18:05.850639 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282685850624, "cf_name": "default", "job": 2646, "event": "table_file_creation", "file_number": 35550, "file_size": 74546423, "table_properties": {"data_size": 67112133, "index_size": 690114, "filter_size": 6743186, "raw_key_size": 76215874, "raw_average_key_size": 28, "raw_value_size": 31365760, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638242, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:18:09.702569 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2646] Generated table #35551: 2638241 keys, 74546257 bytes >2018-07-11 04:18:09.702600 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282689702586, "cf_name": "default", "job": 2646, "event": "table_file_creation", "file_number": 35551, "file_size": 74546257, "table_properties": {"data_size": 67112146, "index_size": 689938, "filter_size": 6743183, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:18:12.934288 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2646] Generated table #35552: 2638241 keys, 74545806 bytes >2018-07-11 04:18:12.934325 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282692934310, "cf_name": "default", "job": 2646, "event": "table_file_creation", "file_number": 35552, "file_size": 74545806, "table_properties": {"data_size": 67112087, "index_size": 689546, "filter_size": 6743183, "raw_key_size": 76215853, "raw_average_key_size": 28, "raw_value_size": 31365756, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:18:15.103565 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2646] Generated table #35553: 2089247 keys, 61206787 bytes >2018-07-11 04:18:15.103611 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282695103595, "cf_name": "default", "job": 2646, "event": "table_file_creation", "file_number": 35553, "file_size": 61206787, "table_properties": {"data_size": 55296128, "index_size": 564678, "filter_size": 5344993, "raw_key_size": 60632730, "raw_average_key_size": 29, "raw_value_size": 26671622, "raw_average_value_size": 12, "num_data_blocks": 13590, "num_entries": 2089247, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:18:16.457815 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2646] Generated table #35554: 162231 keys, 65965881 bytes >2018-07-11 04:18:16.457854 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282696457835, "cf_name": "default", "job": 2646, "event": "table_file_creation", "file_number": 35554, "file_size": 65965881, "table_properties": {"data_size": 64865131, "index_size": 545825, "filter_size": 553937, "raw_key_size": 14984399, "raw_average_key_size": 92, "raw_value_size": 51238922, "raw_average_value_size": 315, "num_data_blocks": 14657, "num_entries": 162231, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "80898", "kMergeOperands": "0"}} >2018-07-11 04:18:16.457929 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2646] Compacted 2@0 + 2@1 files to L1 => 425322037 bytes >2018-07-11 04:18:16.470367 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:18:16.470145) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 8 43 444 3640 707 0] max score 1.80, MB/sec: 22.4 rd, 22.4 wr, level 1, files in(2, 2) out(6) MB in(269.5, 136.1) out(405.6), read-write-amplify(3.0) write-amplify(1.5) OK, records in: 12790469, records dropped: 9 > >2018-07-11 04:18:16.470380 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:18:16.470169) EVENT_LOG_v1 {"time_micros": 1531282696470159, "job": 2646, "event": "compaction_finished", "compaction_time_micros": 18962016, "output_level": 1, "num_output_files": 6, "total_output_size": 425322037, "num_input_records": 12790469, "num_output_records": 12790460, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 8, 43, 444, 3640, 707, 0]} >2018-07-11 04:18:16.471294 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282696471288, "job": 2646, "event": "table_file_deletion", "file_number": 35548} >2018-07-11 04:18:16.471319 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282696471315, "job": 2646, "event": "table_file_deletion", "file_number": 35546} >2018-07-11 04:18:16.471343 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282696471339, "job": 2646, "event": "table_file_deletion", "file_number": 35485} >2018-07-11 04:18:16.471366 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282696471362, "job": 2646, "event": "table_file_deletion", "file_number": 35480} >2018-07-11 04:18:16.471594 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_picker.cc:461] [default] Expanding@1 1+4(74546423+293895204 bytes) to 3+4 (223638486+293895204bytes) > >2018-07-11 04:18:16.471699 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2647] Compacting 3@1 + 4@2 files to L2, score 1.80 >2018-07-11 04:18:16.471715 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2646 Base level 1, inputs: [35550(71MB) 35551(71MB) 35552(71MB)], [35487(71MB) 35491(71MB) 35492(71MB) 35495(67MB)] > >2018-07-11 04:18:16.471735 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282696471720, "job": 2647, "event": "compaction_started", "files_L1": [35550, 35551, 35552], "files_L2": [35487, 35491, 35492, 35495], "score": 1.7958, "input_data_size": 517533690} >2018-07-11 04:18:19.888661 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2647] Generated table #35555: 2638238 keys, 74545700 bytes >2018-07-11 04:18:19.888701 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282699888685, "cf_name": "default", "job": 2647, "event": "table_file_creation", "file_number": 35555, "file_size": 74545700, "table_properties": {"data_size": 67112107, "index_size": 689426, "filter_size": 6743177, "raw_key_size": 76215768, "raw_average_key_size": 28, "raw_value_size": 31365722, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638238, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:18:23.067893 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2647] Generated table #35556: 2638241 keys, 74545883 bytes >2018-07-11 04:18:23.067936 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282703067919, "cf_name": "default", "job": 2647, "event": "table_file_creation", "file_number": 35556, "file_size": 74545883, "table_properties": {"data_size": 67112135, "index_size": 689575, "filter_size": 6743183, "raw_key_size": 76215856, "raw_average_key_size": 28, "raw_value_size": 31365759, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:18:26.209304 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2647] Generated table #35557: 2638240 keys, 74546254 bytes >2018-07-11 04:18:26.209351 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282706209324, "cf_name": "default", "job": 2647, "event": "table_file_creation", "file_number": 35557, "file_size": 74546254, "table_properties": {"data_size": 67112136, "index_size": 689947, "filter_size": 6743181, "raw_key_size": 76215816, "raw_average_key_size": 28, "raw_value_size": 31365736, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:18:29.431372 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2647] Generated table #35558: 2638241 keys, 74546782 bytes >2018-07-11 04:18:29.431420 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282709431405, "cf_name": "default", "job": 2647, "event": "table_file_creation", "file_number": 35558, "file_size": 74546782, "table_properties": {"data_size": 67112140, "index_size": 690469, "filter_size": 6743183, "raw_key_size": 76215849, "raw_average_key_size": 28, "raw_value_size": 31365752, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:18:32.943955 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2647] Generated table #35559: 2638241 keys, 74546264 bytes >2018-07-11 04:18:32.944005 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282712943989, "cf_name": "default", "job": 2647, "event": "table_file_creation", "file_number": 35559, "file_size": 74546264, "table_properties": {"data_size": 67112136, "index_size": 689955, "filter_size": 6743183, "raw_key_size": 76215856, "raw_average_key_size": 28, "raw_value_size": 31365759, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:18:36.157799 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2647] Generated table #35560: 2638242 keys, 74545975 bytes >2018-07-11 04:18:36.157844 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282716157824, "cf_name": "default", "job": 2647, "event": "table_file_creation", "file_number": 35560, "file_size": 74545975, "table_properties": {"data_size": 67112160, "index_size": 689639, "filter_size": 6743186, "raw_key_size": 76215888, "raw_average_key_size": 28, "raw_value_size": 31365774, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638242, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:18:39.301730 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2647] Generated table #35561: 2481959 keys, 70257073 bytes >2018-07-11 04:18:39.301784 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282719301767, "cf_name": "default", "job": 2647, "event": "table_file_creation", "file_number": 35561, "file_size": 70257073, "table_properties": {"data_size": 63261174, "index_size": 650890, "filter_size": 6344020, "raw_key_size": 71716781, "raw_average_key_size": 28, "raw_value_size": 29613100, "raw_average_value_size": 11, "num_data_blocks": 15545, "num_entries": 2481959, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "11", "kMergeOperands": "0"}} >2018-07-11 04:18:39.301823 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2647] Compacted 3@1 + 4@2 files to L2 => 517533931 bytes >2018-07-11 04:18:39.320024 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:18:39.319747) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 5 46 444 3640 707 0] max score 1.04, MB/sec: 22.7 rd, 22.7 wr, level 2, files in(3, 4) out(7) MB in(213.3, 280.3) out(493.6), read-write-amplify(4.6) write-amplify(2.3) OK, records in: 18311402, records dropped: 0 > >2018-07-11 04:18:39.320035 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:18:39.319774) EVENT_LOG_v1 {"time_micros": 1531282719319762, "job": 2647, "event": "compaction_finished", "compaction_time_micros": 22830066, "output_level": 2, "num_output_files": 7, "total_output_size": 517533931, "num_input_records": 18311402, "num_output_records": 18311402, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 5, 46, 444, 3640, 707, 0]} >2018-07-11 04:18:39.320970 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282719320964, "job": 2647, "event": "table_file_deletion", "file_number": 35552} >2018-07-11 04:18:39.320996 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282719320992, "job": 2647, "event": "table_file_deletion", "file_number": 35551} >2018-07-11 04:18:39.321021 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282719321017, "job": 2647, "event": "table_file_deletion", "file_number": 35550} >2018-07-11 04:18:39.321052 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282719321048, "job": 2647, "event": "table_file_deletion", "file_number": 35495} >2018-07-11 04:18:39.321079 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282719321075, "job": 2647, "event": "table_file_deletion", "file_number": 35492} >2018-07-11 04:18:39.321105 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282719321101, "job": 2647, "event": "table_file_deletion", "file_number": 35491} >2018-07-11 04:18:39.321131 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282719321127, "job": 2647, "event": "table_file_deletion", "file_number": 35487} >2018-07-11 04:18:39.321350 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_picker.cc:461] [default] Expanding@2 1+2(74546782+121511156 bytes) to 3+2 (223639021+121511156bytes) > >2018-07-11 04:18:39.321454 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2648] Compacting 3@2 + 2@3 files to L3, score 1.04 >2018-07-11 04:18:39.321468 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2647 Base level 2, inputs: [35558(71MB) 35559(71MB) 35560(71MB)], [35506(71MB) 35508(44MB)] > >2018-07-11 04:18:39.321488 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282719321473, "job": 2648, "event": "compaction_started", "files_L2": [35558, 35559, 35560], "files_L3": [35506, 35508], "score": 1.04457, "input_data_size": 345150177} >2018-07-11 04:18:42.493067 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2648] Generated table #35562: 2638241 keys, 74546206 bytes >2018-07-11 04:18:42.493112 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282722493097, "cf_name": "default", "job": 2648, "event": "table_file_creation", "file_number": 35562, "file_size": 74546206, "table_properties": {"data_size": 67112150, "index_size": 689883, "filter_size": 6743183, "raw_key_size": 76215852, "raw_average_key_size": 28, "raw_value_size": 31365755, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:18:45.572650 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2648] Generated table #35563: 2638241 keys, 74546046 bytes >2018-07-11 04:18:45.572695 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282725572668, "cf_name": "default", "job": 2648, "event": "table_file_creation", "file_number": 35563, "file_size": 74546046, "table_properties": {"data_size": 67112128, "index_size": 689745, "filter_size": 6743183, "raw_key_size": 76215863, "raw_average_key_size": 28, "raw_value_size": 31365766, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:18:48.633284 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2648] Generated table #35564: 2638241 keys, 74546237 bytes >2018-07-11 04:18:48.633330 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282728633315, "cf_name": "default", "job": 2648, "event": "table_file_creation", "file_number": 35564, "file_size": 74546237, "table_properties": {"data_size": 67112149, "index_size": 689915, "filter_size": 6743183, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:18:51.891201 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2648] Generated table #35565: 2638242 keys, 74546745 bytes >2018-07-11 04:18:51.891252 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282731891219, "cf_name": "default", "job": 2648, "event": "table_file_creation", "file_number": 35565, "file_size": 74546745, "table_properties": {"data_size": 67112168, "index_size": 690401, "filter_size": 6743186, "raw_key_size": 76215885, "raw_average_key_size": 28, "raw_value_size": 31365771, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638242, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:18:54.213512 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2648] Generated table #35566: 1519916 keys, 46965202 bytes >2018-07-11 04:18:54.213559 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282734213544, "cf_name": "default", "job": 2648, "event": "table_file_creation", "file_number": 35566, "file_size": 46965202, "table_properties": {"data_size": 42636635, "index_size": 433428, "filter_size": 3894151, "raw_key_size": 44348936, "raw_average_key_size": 29, "raw_value_size": 21522465, "raw_average_value_size": 14, "num_data_blocks": 10538, "num_entries": 1519916, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:18:54.213594 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2648] Compacted 3@2 + 2@3 files to L3 => 345150436 bytes >2018-07-11 04:18:54.228032 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:18:54.227818) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 5 43 447 3640 707 0] max score 1.00, MB/sec: 23.2 rd, 23.2 wr, level 3, files in(3, 2) out(5) MB in(213.3, 115.9) out(329.2), read-write-amplify(3.1) write-amplify(1.5) OK, records in: 12072881, records dropped: 0 > >2018-07-11 04:18:54.228043 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:18:54.227842) EVENT_LOG_v1 {"time_micros": 1531282734227832, "job": 2648, "event": "compaction_finished", "compaction_time_micros": 14892090, "output_level": 3, "num_output_files": 5, "total_output_size": 345150436, "num_input_records": 12072881, "num_output_records": 12072881, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 5, 43, 447, 3640, 707, 0]} >2018-07-11 04:18:54.228955 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282734228951, "job": 2648, "event": "table_file_deletion", "file_number": 35560} >2018-07-11 04:18:54.228982 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282734228979, "job": 2648, "event": "table_file_deletion", "file_number": 35559} >2018-07-11 04:18:54.228994 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282734228992, "job": 2648, "event": "table_file_deletion", "file_number": 35558} >2018-07-11 04:18:54.229007 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282734229005, "job": 2648, "event": "table_file_deletion", "file_number": 35508} >2018-07-11 04:18:54.229018 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282734229016, "job": 2648, "event": "table_file_deletion", "file_number": 35506} >2018-07-11 04:18:54.244599 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:18:54.229320) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #35565 to level-4 74546745 bytes > >2018-07-11 04:18:54.244610 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:18:54.244507) EVENT_LOG_v1 {"time_micros": 1531282734244500, "job": 2649, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 74546745} >2018-07-11 04:18:54.244612 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:18:54.244514) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-4 74546745 bytes OK: base level 1 max bytes base 268435456 files[0 5 43 446 3641 707 0] max score 1.00 > >2018-07-11 04:18:54.261295 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:18:54.245451) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #10496 to level-5 74546861 bytes > >2018-07-11 04:18:54.261305 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:18:54.261194) EVENT_LOG_v1 {"time_micros": 1531282734261188, "job": 2650, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546861} >2018-07-11 04:18:54.261308 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:18:54.261202) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546861 bytes OK: base level 1 max bytes base 268435456 files[0 5 43 446 3640 708 0] max score 1.00 > >2018-07-11 04:18:54.262129 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2651] Compacting 1@3 + 3@4 files to L4, score 1.00 >2018-07-11 04:18:54.262153 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2650 Base level 3, inputs: [35504(71MB)], [35279(71MB) 35280(71MB) 35509(71MB)] > >2018-07-11 04:18:54.262169 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282734262158, "job": 2651, "event": "compaction_started", "files_L3": [35504], "files_L4": [35279, 35280, 35509], "score": 1.00013, "input_data_size": 298186235} >2018-07-11 04:18:57.592036 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2651] Generated table #35567: 2638241 keys, 74546336 bytes >2018-07-11 04:18:57.592076 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282737592057, "cf_name": "default", "job": 2651, "event": "table_file_creation", "file_number": 35567, "file_size": 74546336, "table_properties": {"data_size": 67112157, "index_size": 690006, "filter_size": 6743183, "raw_key_size": 76215847, "raw_average_key_size": 28, "raw_value_size": 31365750, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:19:00.757953 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2651] Generated table #35568: 2638241 keys, 74545732 bytes >2018-07-11 04:19:00.757999 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282740757971, "cf_name": "default", "job": 2651, "event": "table_file_creation", "file_number": 35568, "file_size": 74545732, "table_properties": {"data_size": 67112123, "index_size": 689436, "filter_size": 6743183, "raw_key_size": 76215864, "raw_average_key_size": 28, "raw_value_size": 31365767, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:19:04.279494 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2651] Generated table #35569: 2638241 keys, 74546679 bytes >2018-07-11 04:19:04.279529 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282744279514, "cf_name": "default", "job": 2651, "event": "table_file_creation", "file_number": 35569, "file_size": 74546679, "table_properties": {"data_size": 67112162, "index_size": 690344, "filter_size": 6743183, "raw_key_size": 76215851, "raw_average_key_size": 28, "raw_value_size": 31365754, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:19:07.726295 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2651] Generated table #35570: 2638240 keys, 74546555 bytes >2018-07-11 04:19:07.726349 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282747726332, "cf_name": "default", "job": 2651, "event": "table_file_creation", "file_number": 35570, "file_size": 74546555, "table_properties": {"data_size": 67112096, "index_size": 690299, "filter_size": 6743170, "raw_key_size": 76215820, "raw_average_key_size": 28, "raw_value_size": 31365740, "raw_average_value_size": 11, "num_data_blocks": 16489, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:19:07.726384 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2651] Compacted 1@3 + 3@4 files to L4 => 298185302 bytes >2018-07-11 04:19:07.744629 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:19:07.744321) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 5 43 445 3641 708 0] max score 1.00, MB/sec: 22.1 rd, 22.1 wr, level 4, files in(1, 3) out(4) MB in(71.1, 213.3) out(284.4), read-write-amplify(8.0) write-amplify(4.0) OK, records in: 10552963, records dropped: 0 > >2018-07-11 04:19:07.744641 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:19:07.744347) EVENT_LOG_v1 {"time_micros": 1531282747744336, "job": 2651, "event": "compaction_finished", "compaction_time_micros": 13464201, "output_level": 4, "num_output_files": 4, "total_output_size": 298185302, "num_input_records": 10552963, "num_output_records": 10552963, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 5, 43, 445, 3641, 708, 0]} >2018-07-11 04:19:07.745553 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282747745547, "job": 2651, "event": "table_file_deletion", "file_number": 35509} >2018-07-11 04:19:07.745579 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282747745575, "job": 2651, "event": "table_file_deletion", "file_number": 35504} >2018-07-11 04:19:07.745604 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282747745601, "job": 2651, "event": "table_file_deletion", "file_number": 35280} >2018-07-11 04:19:07.745627 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282747745624, "job": 2651, "event": "table_file_deletion", "file_number": 35279} >2018-07-11 04:19:07.761080 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:19:07.745923) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #12258 to level-5 74546860 bytes > >2018-07-11 04:19:07.761091 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:19:07.760985) EVENT_LOG_v1 {"time_micros": 1531282747760979, "job": 2652, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546860} >2018-07-11 04:19:07.761096 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:19:07.760993) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546860 bytes OK: base level 1 max bytes base 268435456 files[0 5 43 445 3640 709 0] max score 1.00 > >2018-07-11 04:21:04.883994 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:684] reusing log 35545 from recycle list > >2018-07-11 04:21:04.884148 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #35571. Immutable memtables: 0. > >2018-07-11 04:21:04.959659 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:21:04.884216) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1 >2018-07-11 04:21:04.959674 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 2653] Flushing memtable with next log file: 35571 > >2018-07-11 04:21:04.959691 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282864959682, "job": 2653, "event": "flush_started", "num_memtables": 1, "num_entries": 5000015, "num_deletes": 0, "memory_usage": 292931872} >2018-07-11 04:21:04.959694 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 2653] Level-0 flush table #35572: started >2018-07-11 04:21:08.933936 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282868933914, "cf_name": "default", "job": 2653, "event": "table_file_creation", "file_number": 35572, "file_size": 141284983, "table_properties": {"data_size": 127195853, "index_size": 1308422, "filter_size": 12779718, "raw_key_size": 144445053, "raw_average_key_size": 28, "raw_value_size": 59448873, "raw_average_value_size": 11, "num_data_blocks": 31252, "num_entries": 5000012, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:21:08.933959 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 2653] Level-0 flush table #35572: 141284983 bytes OK >2018-07-11 04:21:08.950976 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:242] adding log 35547 to recycle list > >2018-07-11 04:21:08.950987 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:21:08.933984) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #35572 started >2018-07-11 04:21:08.951005 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:21:08.950753) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #35572: memtable #1 done >2018-07-11 04:21:08.951007 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:21:08.950771) EVENT_LOG_v1 {"time_micros": 1531282868950764, "job": 2653, "event": "flush_finished", "lsm_state": [1, 5, 43, 445, 3640, 709, 0], "immutable_memtables": 0} >2018-07-11 04:21:08.951009 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:21:08.950924) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 5 43 445 3640 709 0] max score 1.00 > >2018-07-11 04:21:08.951869 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 2653] Try to delete WAL files size 178896755, prev total WAL file size 178896767, number of live WAL files 2. > >2018-07-11 04:21:41.364107 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:684] reusing log 35547 from recycle list > >2018-07-11 04:21:41.364226 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #35573. Immutable memtables: 0. > >2018-07-11 04:21:41.405626 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:21:41.364295) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1 >2018-07-11 04:21:41.405640 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 2654] Flushing memtable with next log file: 35573 > >2018-07-11 04:21:41.405657 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282901405648, "job": 2654, "event": "flush_started", "num_memtables": 1, "num_entries": 5000015, "num_deletes": 0, "memory_usage": 292937992} >2018-07-11 04:21:41.405661 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 2654] Level-0 flush table #35574: started >2018-07-11 04:21:44.141096 7f41649ec700 1 bluefs _allocate failed to allocate 0x8700000 on bdev 1, free 0x16fe000; fallback to bdev 2 >2018-07-11 04:21:45.922616 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282905922595, "cf_name": "default", "job": 2654, "event": "table_file_creation", "file_number": 35574, "file_size": 141286889, "table_properties": {"data_size": 127197777, "index_size": 1308397, "filter_size": 12779725, "raw_key_size": 144445101, "raw_average_key_size": 28, "raw_value_size": 59450671, "raw_average_value_size": 11, "num_data_blocks": 31252, "num_entries": 5000014, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:21:45.922639 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 2654] Level-0 flush table #35574: 141286889 bytes OK >2018-07-11 04:21:45.941659 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:242] adding log 35571 to recycle list > >2018-07-11 04:21:45.941672 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:21:45.922652) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #35574 started >2018-07-11 04:21:45.941675 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:21:45.941403) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #35574: memtable #1 done >2018-07-11 04:21:45.941677 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:21:45.941421) EVENT_LOG_v1 {"time_micros": 1531282905941414, "job": 2654, "event": "flush_finished", "lsm_state": [2, 5, 43, 445, 3640, 709, 0], "immutable_memtables": 0} >2018-07-11 04:21:45.941679 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:21:45.941596) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[2 5 43 445 3640 709 0] max score 1.05 > >2018-07-11 04:21:45.942096 7f41641eb700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS ------- >2018-07-11 04:21:45.942106 7f41641eb700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] >** DB Stats ** >Uptime(secs): 47647.6 total, 710.8 interval >Cumulative writes: 196K writes, 1757M keys, 196K commit groups, 1.0 writes per commit group, ingest: 59.07 GB, 1.27 MB/s >Cumulative WAL: 196K writes, 56K syncs, 3.49 writes per sync, written: 59.07 GB, 1.27 MB/s >Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent >Interval writes: 54 writes, 26M keys, 54 commit groups, 1.0 writes per commit group, ingest: 887.17 MB, 1.25 MB/s >Interval WAL: 54 writes, 27 syncs, 1.93 writes per sync, written: 0.87 MB, 1.25 MB/s >Interval stall: 00:00:0.000 H:M:S, 0.0 percent > >** Compaction Stats [default] ** >Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop >---------------------------------------------------------------------------------------------------------------------------------------------------------- > L0 2/0 269.48 MB 1.1 0.0 0.0 0.0 46.2 46.2 0.0 1.0 0.0 32.4 1463 353 4.144 0 0 > L1 5/0 246.45 MB 1.0 76.2 46.1 30.1 76.1 46.0 0.0 1.7 25.7 25.7 3036 173 17.546 2409M 894K > L2 43/0 2.40 GB 1.0 88.6 46.0 42.6 88.6 46.0 0.0 1.9 21.5 21.5 4226 542 7.796 3303M 1358K > L3 445/0 24.93 GB 1.0 80.5 45.4 35.0 80.4 45.3 0.6 1.8 20.7 20.6 3987 275 14.499 3040M 4000K > L4 3640/0 250.00 GB 1.0 71.1 16.8 54.3 70.8 16.5 29.2 4.2 21.1 21.0 3446 225 15.314 2705M 15M > L5 709/0 49.20 GB 0.0 1.7 0.9 0.8 1.7 0.9 44.7 1.9 18.7 18.5 95 12 7.912 66M 1097K > Sum 4844/0 327.04 GB 0.0 318.1 155.3 162.8 363.8 201.0 74.5 7.9 20.0 22.9 16252 1580 10.286 11G 23M > Int 0/0 0.00 KB 0.0 5.7 2.0 3.7 6.5 2.8 1.0 8.3 20.5 23.3 287 20 14.336 212M 17 >Uptime(secs): 47647.6 total, 47647.6 interval >Flush(GB): cumulative 46.234, interval 0.790 >AddFile(GB): cumulative 0.000, interval 0.000 >AddFile(Total Files): cumulative 0, interval 0 >AddFile(L0 Files): cumulative 0, interval 0 >AddFile(Keys): cumulative 0, interval 0 >Cumulative compaction: 363.81 GB write, 7.82 MB/s write, 318.10 GB read, 6.84 MB/s read, 16251.9 seconds >Interval compaction: 6.52 GB write, 0.14 MB/s write, 5.73 GB read, 0.12 MB/s read, 286.7 seconds >Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count > >** File Read Latency Histogram By Level [default] ** > >** Compaction Stats [default] ** >Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop >---------------------------------------------------------------------------------------------------------------------------------------------------------- > L0 2/0 269.48 MB 1.1 0.0 0.0 0.0 46.2 46.2 0.0 1.0 0.0 32.4 1463 353 4.144 0 0 > L1 5/0 246.45 MB 1.0 76.2 46.1 30.1 76.1 46.0 0.0 1.7 25.7 25.7 3036 173 17.546 2409M 894K > L2 43/0 2.40 GB 1.0 88.6 46.0 42.6 88.6 46.0 0.0 1.9 21.5 21.5 4226 542 7.796 3303M 1358K > L3 445/0 24.93 GB 1.0 80.5 45.4 35.0 80.4 45.3 0.6 1.8 20.7 20.6 3987 275 14.499 3040M 4000K > L4 3640/0 250.00 GB 1.0 71.1 16.8 54.3 70.8 16.5 29.2 4.2 21.1 21.0 3446 225 15.314 2705M 15M > L5 709/0 49.20 GB 0.0 1.7 0.9 0.8 1.7 0.9 44.7 1.9 18.7 18.5 95 12 7.912 66M 1097K > Sum 4844/0 327.04 GB 0.0 318.1 155.3 162.8 363.8 201.0 74.5 7.9 20.0 22.9 16252 1580 10.286 11G 23M > Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 >Uptime(secs): 47647.6 total, 47647.6 interval >Flush(GB): cumulative 46.234, interval 0.000 >AddFile(GB): cumulative 0.000, interval 0.000 >AddFile(Total Files): cumulative 0, interval 0 >AddFile(L0 Files): cumulative 0, interval 0 >AddFile(Keys): cumulative 0, interval 0 >Cumulative compaction: 363.81 GB write, 7.82 MB/s write, 318.10 GB read, 6.84 MB/s read, 16251.9 seconds >Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds >Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count > >** File Read Latency Histogram By Level [default] ** > >2018-07-11 04:21:45.942242 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2655] Compacting 2@0 + 3@1 files to L1, score 1.05 >2018-07-11 04:21:45.942264 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2654 Base level 0, inputs: [35574(134MB) 35572(134MB)], [35549(71MB) 35553(58MB) 35554(62MB)] > >2018-07-11 04:21:45.942280 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282905942268, "job": 2655, "event": "compaction_started", "files_L0": [35574, 35572], "files_L1": [35549, 35553, 35554], "score": 1.05266, "input_data_size": 484255423} >2018-07-11 04:21:45.942600 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 2654] Try to delete WAL files size 178896755, prev total WAL file size 178896767, number of live WAL files 2. > >2018-07-11 04:21:48.117464 7f41641eb700 1 bluefs _allocate failed to allocate 0x4800000 on bdev 1, free 0x16fe000; fallback to bdev 2 >2018-07-11 04:21:49.155557 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2655] Generated table #35575: 2624188 keys, 74510504 bytes >2018-07-11 04:21:49.155591 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282909155576, "cf_name": "default", "job": 2655, "event": "table_file_creation", "file_number": 35575, "file_size": 74510504, "table_properties": {"data_size": 67110078, "index_size": 691375, "filter_size": 6708061, "raw_key_size": 75840679, "raw_average_key_size": 28, "raw_value_size": 31546459, "raw_average_value_size": 12, "num_data_blocks": 16475, "num_entries": 2624188, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:21:51.199145 7f41641eb700 1 bluefs _allocate failed to allocate 0x4800000 on bdev 1, free 0x16fe000; fallback to bdev 2 >2018-07-11 04:21:52.281651 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2655] Generated table #35576: 2638241 keys, 74546332 bytes >2018-07-11 04:21:52.281696 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282912281669, "cf_name": "default", "job": 2655, "event": "table_file_creation", "file_number": 35576, "file_size": 74546332, "table_properties": {"data_size": 67112183, "index_size": 689976, "filter_size": 6743183, "raw_key_size": 76215857, "raw_average_key_size": 28, "raw_value_size": 31365760, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:21:54.291806 7f41641eb700 1 bluefs _allocate failed to allocate 0x4800000 on bdev 1, free 0x16fe000; fallback to bdev 2 >2018-07-11 04:21:55.267401 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2655] Generated table #35577: 2638241 keys, 74546879 bytes >2018-07-11 04:21:55.267446 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282915267418, "cf_name": "default", "job": 2655, "event": "table_file_creation", "file_number": 35577, "file_size": 74546879, "table_properties": {"data_size": 67112125, "index_size": 690581, "filter_size": 6743183, "raw_key_size": 76215851, "raw_average_key_size": 28, "raw_value_size": 31365754, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:21:57.271734 7f41641eb700 1 bluefs _allocate failed to allocate 0x4800000 on bdev 1, free 0x16fe000; fallback to bdev 2 >2018-07-11 04:21:58.309744 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2655] Generated table #35578: 2638241 keys, 74546245 bytes >2018-07-11 04:21:58.309788 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282918309761, "cf_name": "default", "job": 2655, "event": "table_file_creation", "file_number": 35578, "file_size": 74546245, "table_properties": {"data_size": 67112151, "index_size": 689921, "filter_size": 6743183, "raw_key_size": 76215856, "raw_average_key_size": 28, "raw_value_size": 31365759, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:22:00.324635 7f41641eb700 1 bluefs _allocate failed to allocate 0x4800000 on bdev 1, free 0x16fe000; fallback to bdev 2 >2018-07-11 04:22:01.342658 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2655] Generated table #35579: 2638241 keys, 74546379 bytes >2018-07-11 04:22:01.342702 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282921342675, "cf_name": "default", "job": 2655, "event": "table_file_creation", "file_number": 35579, "file_size": 74546379, "table_properties": {"data_size": 67112138, "index_size": 690068, "filter_size": 6743183, "raw_key_size": 76215850, "raw_average_key_size": 28, "raw_value_size": 31365753, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:22:02.557975 7f41641eb700 1 bluefs _allocate failed to allocate 0x2c00000 on bdev 1, free 0x16fe000; fallback to bdev 2 >2018-07-11 04:22:03.171616 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2655] Generated table #35580: 1536363 keys, 45584232 bytes >2018-07-11 04:22:03.171658 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282923171631, "cf_name": "default", "job": 2655, "event": "table_file_creation", "file_number": 35580, "file_size": 45584232, "table_properties": {"data_size": 41231716, "index_size": 419676, "filter_size": 3931852, "raw_key_size": 44660527, "raw_average_key_size": 29, "raw_value_size": 20098447, "raw_average_value_size": 13, "num_data_blocks": 10134, "num_entries": 1536363, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:22:03.661880 7f41641eb700 1 bluefs _allocate failed to allocate 0x3f00000 on bdev 1, free 0x16fe000; fallback to bdev 2 >2018-07-11 04:22:04.460056 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2655] Generated table #35581: 162241 keys, 65970097 bytes >2018-07-11 04:22:04.460091 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282924460076, "cf_name": "default", "job": 2655, "event": "table_file_creation", "file_number": 35581, "file_size": 65970097, "table_properties": {"data_size": 64869289, "index_size": 545851, "filter_size": 553969, "raw_key_size": 14985009, "raw_average_key_size": 92, "raw_value_size": 51242552, "raw_average_value_size": 315, "num_data_blocks": 14658, "num_entries": 162241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "80898", "kMergeOperands": "0"}} >2018-07-11 04:22:04.460163 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2655] Compacted 2@0 + 3@1 files to L1 => 484250668 bytes >2018-07-11 04:22:04.474615 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:22:04.474347) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 9 43 445 3640 709 0] max score 2.02, MB/sec: 26.2 rd, 26.2 wr, level 1, files in(2, 3) out(7) MB in(269.5, 192.3) out(461.8), read-write-amplify(3.4) write-amplify(1.7) OK, records in: 14875762, records dropped: 6 > >2018-07-11 04:22:04.474625 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:22:04.474373) EVENT_LOG_v1 {"time_micros": 1531282924474362, "job": 2655, "event": "compaction_finished", "compaction_time_micros": 18517863, "output_level": 1, "num_output_files": 7, "total_output_size": 484250668, "num_input_records": 14875762, "num_output_records": 14875756, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 9, 43, 445, 3640, 709, 0]} >2018-07-11 04:22:04.475529 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282924475525, "job": 2655, "event": "table_file_deletion", "file_number": 35574} >2018-07-11 04:22:04.475547 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282924475545, "job": 2655, "event": "table_file_deletion", "file_number": 35572} >2018-07-11 04:22:04.475563 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282924475560, "job": 2655, "event": "table_file_deletion", "file_number": 35554} >2018-07-11 04:22:04.475578 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282924475576, "job": 2655, "event": "table_file_deletion", "file_number": 35553} >2018-07-11 04:22:04.475591 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282924475589, "job": 2655, "event": "table_file_deletion", "file_number": 35549} >2018-07-11 04:22:04.475809 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_picker.cc:461] [default] Expanding@1 1+1(74546879+70257073 bytes) to 3+1 (223639503+70257073bytes) > >2018-07-11 04:22:04.475909 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2656] Compacting 3@1 + 1@2 files to L2, score 2.02 >2018-07-11 04:22:04.475921 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2655 Base level 1, inputs: [35577(71MB) 35578(71MB) 35579(71MB)], [35561(67MB)] > >2018-07-11 04:22:04.475935 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282924475924, "job": 2656, "event": "compaction_started", "files_L1": [35577, 35578, 35579], "files_L2": [35561], "score": 2.01532, "input_data_size": 293896576} >2018-07-11 04:22:07.576007 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2656] Generated table #35582: 2638240 keys, 74547082 bytes >2018-07-11 04:22:07.576052 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282927576025, "cf_name": "default", "job": 2656, "event": "table_file_creation", "file_number": 35582, "file_size": 74547082, "table_properties": {"data_size": 67112145, "index_size": 690766, "filter_size": 6743181, "raw_key_size": 76215819, "raw_average_key_size": 28, "raw_value_size": 31365739, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:22:10.606810 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2656] Generated table #35583: 2638241 keys, 74546118 bytes >2018-07-11 04:22:10.606854 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282930606827, "cf_name": "default", "job": 2656, "event": "table_file_creation", "file_number": 35583, "file_size": 74546118, "table_properties": {"data_size": 67112103, "index_size": 689842, "filter_size": 6743183, "raw_key_size": 76215851, "raw_average_key_size": 28, "raw_value_size": 31365754, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:22:13.715719 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2656] Generated table #35584: 2638241 keys, 74546331 bytes >2018-07-11 04:22:13.715763 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282933715736, "cf_name": "default", "job": 2656, "event": "table_file_creation", "file_number": 35584, "file_size": 74546331, "table_properties": {"data_size": 67112160, "index_size": 689998, "filter_size": 6743183, "raw_key_size": 76215861, "raw_average_key_size": 28, "raw_value_size": 31365764, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:22:16.605124 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2656] Generated table #35585: 2481960 keys, 70255848 bytes >2018-07-11 04:22:16.605164 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282936605147, "cf_name": "default", "job": 2656, "event": "table_file_creation", "file_number": 35585, "file_size": 70255848, "table_properties": {"data_size": 63261167, "index_size": 649669, "filter_size": 6344023, "raw_key_size": 71716807, "raw_average_key_size": 28, "raw_value_size": 29613109, "raw_average_value_size": 11, "num_data_blocks": 15545, "num_entries": 2481960, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "11", "kMergeOperands": "0"}} >2018-07-11 04:22:16.605201 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2656] Compacted 3@1 + 1@2 files to L2 => 293895379 bytes >2018-07-11 04:22:16.624488 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:22:16.624178) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 6 46 445 3640 709 0] max score 1.18, MB/sec: 24.2 rd, 24.2 wr, level 2, files in(3, 1) out(4) MB in(213.3, 67.0) out(280.3), read-write-amplify(2.6) write-amplify(1.3) OK, records in: 10396682, records dropped: 0 > >2018-07-11 04:22:16.624500 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:22:16.624204) EVENT_LOG_v1 {"time_micros": 1531282936624193, "job": 2656, "event": "compaction_finished", "compaction_time_micros": 12129249, "output_level": 2, "num_output_files": 4, "total_output_size": 293895379, "num_input_records": 10396682, "num_output_records": 10396682, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 6, 46, 445, 3640, 709, 0]} >2018-07-11 04:22:16.625411 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282936625407, "job": 2656, "event": "table_file_deletion", "file_number": 35579} >2018-07-11 04:22:16.625427 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282936625425, "job": 2656, "event": "table_file_deletion", "file_number": 35578} >2018-07-11 04:22:16.625440 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282936625437, "job": 2656, "event": "table_file_deletion", "file_number": 35577} >2018-07-11 04:22:16.625460 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282936625457, "job": 2656, "event": "table_file_deletion", "file_number": 35561} >2018-07-11 04:22:16.625764 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2657] Compacting 1@1 + 4@2 files to L2, score 1.18 >2018-07-11 04:22:16.625776 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2656 Base level 1, inputs: [35576(71MB)], [35555(71MB) 35556(71MB) 35557(71MB) 35582(71MB)] > >2018-07-11 04:22:16.625790 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282936625780, "job": 2657, "event": "compaction_started", "files_L1": [35576], "files_L2": [35555, 35556, 35557, 35582], "score": 1.1822, "input_data_size": 372731251} >2018-07-11 04:22:19.900104 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2657] Generated table #35586: 2638238 keys, 74545732 bytes >2018-07-11 04:22:19.900138 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282939900123, "cf_name": "default", "job": 2657, "event": "table_file_creation", "file_number": 35586, "file_size": 74545732, "table_properties": {"data_size": 67112108, "index_size": 689457, "filter_size": 6743177, "raw_key_size": 76215768, "raw_average_key_size": 28, "raw_value_size": 31365722, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638238, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:22:22.956861 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2657] Generated table #35587: 2638240 keys, 74546391 bytes >2018-07-11 04:22:22.956904 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282942956877, "cf_name": "default", "job": 2657, "event": "table_file_creation", "file_number": 35587, "file_size": 74546391, "table_properties": {"data_size": 67112123, "index_size": 690097, "filter_size": 6743181, "raw_key_size": 76215825, "raw_average_key_size": 28, "raw_value_size": 31365745, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:22:26.116380 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2657] Generated table #35588: 2638241 keys, 74546298 bytes >2018-07-11 04:22:26.116425 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282946116398, "cf_name": "default", "job": 2657, "event": "table_file_creation", "file_number": 35588, "file_size": 74546298, "table_properties": {"data_size": 67112151, "index_size": 689974, "filter_size": 6743183, "raw_key_size": 76215847, "raw_average_key_size": 28, "raw_value_size": 31365750, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:22:29.173500 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2657] Generated table #35589: 2638241 keys, 74547030 bytes >2018-07-11 04:22:29.173545 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282949173518, "cf_name": "default", "job": 2657, "event": "table_file_creation", "file_number": 35589, "file_size": 74547030, "table_properties": {"data_size": 67112149, "index_size": 690708, "filter_size": 6743183, "raw_key_size": 76215848, "raw_average_key_size": 28, "raw_value_size": 31365751, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:22:32.193372 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2657] Generated table #35590: 2638240 keys, 74545972 bytes >2018-07-11 04:22:32.193419 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282952193391, "cf_name": "default", "job": 2657, "event": "table_file_creation", "file_number": 35590, "file_size": 74545972, "table_properties": {"data_size": 67112040, "index_size": 689772, "filter_size": 6743170, "raw_key_size": 76215828, "raw_average_key_size": 28, "raw_value_size": 31365748, "raw_average_value_size": 11, "num_data_blocks": 16489, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:22:32.193455 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2657] Compacted 1@1 + 4@2 files to L2 => 372731423 bytes >2018-07-11 04:22:32.215865 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:22:32.215636) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 5 47 445 3640 709 0] max score 1.07, MB/sec: 23.9 rd, 23.9 wr, level 2, files in(1, 4) out(5) MB in(71.1, 284.4) out(355.5), read-write-amplify(10.0) write-amplify(5.0) OK, records in: 13191200, records dropped: 0 > >2018-07-11 04:22:32.215877 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:22:32.215662) EVENT_LOG_v1 {"time_micros": 1531282952215651, "job": 2657, "event": "compaction_finished", "compaction_time_micros": 15567649, "output_level": 2, "num_output_files": 5, "total_output_size": 372731423, "num_input_records": 13191200, "num_output_records": 13191200, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 5, 47, 445, 3640, 709, 0]} >2018-07-11 04:22:32.216830 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282952216827, "job": 2657, "event": "table_file_deletion", "file_number": 35582} >2018-07-11 04:22:32.216858 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282952216856, "job": 2657, "event": "table_file_deletion", "file_number": 35576} >2018-07-11 04:22:32.216872 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282952216870, "job": 2657, "event": "table_file_deletion", "file_number": 35557} >2018-07-11 04:22:32.216883 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282952216881, "job": 2657, "event": "table_file_deletion", "file_number": 35556} >2018-07-11 04:22:32.216894 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282952216892, "job": 2657, "event": "table_file_deletion", "file_number": 35555} >2018-07-11 04:22:32.217108 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_picker.cc:461] [default] Expanding@2 1+4(74547030+270603691 bytes) to 4+4 (298185451+270603691bytes) > >2018-07-11 04:22:32.217208 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2658] Compacting 4@2 + 4@3 files to L3, score 1.07 >2018-07-11 04:22:32.217221 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2657 Base level 2, inputs: [35589(71MB) 35590(71MB) 35583(71MB) 35584(71MB)], [35562(71MB) 35563(71MB) 35564(71MB) 35566(44MB)] > >2018-07-11 04:22:32.217235 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282952217224, "job": 2658, "event": "compaction_started", "files_L2": [35589, 35590, 35583, 35584], "files_L3": [35562, 35563, 35564, 35566], "score": 1.07234, "input_data_size": 568789142} >2018-07-11 04:22:35.349753 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2658] Generated table #35591: 2638241 keys, 74546215 bytes >2018-07-11 04:22:35.349798 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282955349771, "cf_name": "default", "job": 2658, "event": "table_file_creation", "file_number": 35591, "file_size": 74546215, "table_properties": {"data_size": 67112150, "index_size": 689892, "filter_size": 6743183, "raw_key_size": 76215852, "raw_average_key_size": 28, "raw_value_size": 31365755, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:22:38.350970 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2658] Generated table #35592: 2638241 keys, 74545995 bytes >2018-07-11 04:22:38.351014 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282958350987, "cf_name": "default", "job": 2658, "event": "table_file_creation", "file_number": 35592, "file_size": 74545995, "table_properties": {"data_size": 67112129, "index_size": 689693, "filter_size": 6743183, "raw_key_size": 76215863, "raw_average_key_size": 28, "raw_value_size": 31365766, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:22:41.577983 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2658] Generated table #35593: 2638241 keys, 74546196 bytes >2018-07-11 04:22:41.578028 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282961578000, "cf_name": "default", "job": 2658, "event": "table_file_creation", "file_number": 35593, "file_size": 74546196, "table_properties": {"data_size": 67112147, "index_size": 689876, "filter_size": 6743183, "raw_key_size": 76215852, "raw_average_key_size": 28, "raw_value_size": 31365755, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:22:44.678779 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2658] Generated table #35594: 2638241 keys, 74546436 bytes >2018-07-11 04:22:44.678823 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282964678795, "cf_name": "default", "job": 2658, "event": "table_file_creation", "file_number": 35594, "file_size": 74546436, "table_properties": {"data_size": 67112154, "index_size": 690109, "filter_size": 6743183, "raw_key_size": 76215851, "raw_average_key_size": 28, "raw_value_size": 31365754, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:22:47.697886 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2658] Generated table #35595: 2638241 keys, 74546427 bytes >2018-07-11 04:22:47.697930 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282967697903, "cf_name": "default", "job": 2658, "event": "table_file_creation", "file_number": 35595, "file_size": 74546427, "table_properties": {"data_size": 67112104, "index_size": 690150, "filter_size": 6743183, "raw_key_size": 76215859, "raw_average_key_size": 28, "raw_value_size": 31365762, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:22:50.808775 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2658] Generated table #35596: 2638241 keys, 74545943 bytes >2018-07-11 04:22:50.808820 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282970808792, "cf_name": "default", "job": 2658, "event": "table_file_creation", "file_number": 35596, "file_size": 74545943, "table_properties": {"data_size": 67112166, "index_size": 689604, "filter_size": 6743183, "raw_key_size": 76215852, "raw_average_key_size": 28, "raw_value_size": 31365755, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:22:53.853342 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2658] Generated table #35597: 2638241 keys, 74546530 bytes >2018-07-11 04:22:53.853374 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282973853359, "cf_name": "default", "job": 2658, "event": "table_file_creation", "file_number": 35597, "file_size": 74546530, "table_properties": {"data_size": 67112132, "index_size": 690225, "filter_size": 6743183, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:22:55.743395 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2658] Generated table #35598: 1519915 keys, 46965300 bytes >2018-07-11 04:22:55.743438 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282975743423, "cf_name": "default", "job": 2658, "event": "table_file_creation", "file_number": 35598, "file_size": 46965300, "table_properties": {"data_size": 42636616, "index_size": 433548, "filter_size": 3894148, "raw_key_size": 44348910, "raw_average_key_size": 29, "raw_value_size": 21522456, "raw_average_value_size": 14, "num_data_blocks": 10538, "num_entries": 1519915, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:22:55.743477 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2658] Compacted 4@2 + 4@3 files to L3 => 568789042 bytes >2018-07-11 04:22:55.765577 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:22:55.765210) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 5 43 449 3640 709 0] max score 1.01, MB/sec: 24.2 rd, 24.2 wr, level 3, files in(4, 4) out(8) MB in(284.4, 258.1) out(542.4), read-write-amplify(3.8) write-amplify(1.9) OK, records in: 19987602, records dropped: 0 > >2018-07-11 04:22:55.765589 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:22:55.765236) EVENT_LOG_v1 {"time_micros": 1531282975765224, "job": 2658, "event": "compaction_finished", "compaction_time_micros": 23526223, "output_level": 3, "num_output_files": 8, "total_output_size": 568789042, "num_input_records": 19987602, "num_output_records": 19987602, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 5, 43, 449, 3640, 709, 0]} >2018-07-11 04:22:55.766537 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282975766532, "job": 2658, "event": "table_file_deletion", "file_number": 35590} >2018-07-11 04:22:55.766551 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282975766548, "job": 2658, "event": "table_file_deletion", "file_number": 35589} >2018-07-11 04:22:55.766563 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282975766560, "job": 2658, "event": "table_file_deletion", "file_number": 35584} >2018-07-11 04:22:55.766573 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282975766571, "job": 2658, "event": "table_file_deletion", "file_number": 35583} >2018-07-11 04:22:55.766598 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282975766596, "job": 2658, "event": "table_file_deletion", "file_number": 35566} >2018-07-11 04:22:55.766610 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282975766608, "job": 2658, "event": "table_file_deletion", "file_number": 35564} >2018-07-11 04:22:55.766622 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282975766620, "job": 2658, "event": "table_file_deletion", "file_number": 35563} >2018-07-11 04:22:55.766633 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282975766631, "job": 2658, "event": "table_file_deletion", "file_number": 35562} >2018-07-11 04:22:55.781976 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:22:55.766927) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #35597 to level-4 74546530 bytes > >2018-07-11 04:22:55.781986 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:22:55.781883) EVENT_LOG_v1 {"time_micros": 1531282975781876, "job": 2659, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 74546530} >2018-07-11 04:22:55.781989 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:22:55.781890) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-4 74546530 bytes OK: base level 1 max bytes base 268435456 files[0 5 43 448 3641 709 0] max score 1.01 > >2018-07-11 04:22:55.782828 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2660] Compacting 1@3 + 1@4 files to L4, score 1.01 >2018-07-11 04:22:55.782840 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2659 Base level 3, inputs: [35594(71MB)], [35565(71MB)] > >2018-07-11 04:22:55.782867 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282975782843, "job": 2660, "event": "compaction_started", "files_L3": [35594], "files_L4": [35565], "score": 1.00568, "input_data_size": 149093181} >2018-07-11 04:22:58.863191 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2660] Generated table #35599: 2638242 keys, 74546730 bytes >2018-07-11 04:22:58.863235 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282978863209, "cf_name": "default", "job": 2660, "event": "table_file_creation", "file_number": 35599, "file_size": 74546730, "table_properties": {"data_size": 67112170, "index_size": 690384, "filter_size": 6743186, "raw_key_size": 76215885, "raw_average_key_size": 28, "raw_value_size": 31365771, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638242, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:23:01.799415 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2660] Generated table #35600: 2638241 keys, 74546436 bytes >2018-07-11 04:23:01.799462 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282981799434, "cf_name": "default", "job": 2660, "event": "table_file_creation", "file_number": 35600, "file_size": 74546436, "table_properties": {"data_size": 67112154, "index_size": 690109, "filter_size": 6743183, "raw_key_size": 76215851, "raw_average_key_size": 28, "raw_value_size": 31365754, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:23:01.799495 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2660] Compacted 1@3 + 1@4 files to L4 => 149093166 bytes >2018-07-11 04:23:01.815488 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:23:01.815222) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 5 43 447 3642 709 0] max score 1.00, MB/sec: 24.8 rd, 24.8 wr, level 4, files in(1, 1) out(2) MB in(71.1, 71.1) out(142.2), read-write-amplify(4.0) write-amplify(2.0) OK, records in: 5276483, records dropped: 0 > >2018-07-11 04:23:01.815499 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:23:01.815257) EVENT_LOG_v1 {"time_micros": 1531282981815236, "job": 2660, "event": "compaction_finished", "compaction_time_micros": 6016616, "output_level": 4, "num_output_files": 2, "total_output_size": 149093166, "num_input_records": 5276483, "num_output_records": 5276483, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 5, 43, 447, 3642, 709, 0]} >2018-07-11 04:23:01.816433 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282981816429, "job": 2660, "event": "table_file_deletion", "file_number": 35594} >2018-07-11 04:23:01.816448 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531282981816445, "job": 2660, "event": "table_file_deletion", "file_number": 35565} >2018-07-11 04:23:01.831982 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:23:01.816758) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #35595 to level-4 74546427 bytes > >2018-07-11 04:23:01.831993 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:23:01.831891) EVENT_LOG_v1 {"time_micros": 1531282981831885, "job": 2661, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 74546427} >2018-07-11 04:23:01.831995 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:23:01.831899) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-4 74546427 bytes OK: base level 1 max bytes base 268435456 files[0 5 43 446 3643 709 0] max score 1.00 > >2018-07-11 04:23:01.848685 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:23:01.832822) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #10757 to level-5 74546860 bytes > >2018-07-11 04:23:01.848695 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:23:01.848598) EVENT_LOG_v1 {"time_micros": 1531282981848592, "job": 2662, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546860} >2018-07-11 04:23:01.848698 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:23:01.848605) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546860 bytes OK: base level 1 max bytes base 268435456 files[0 5 43 446 3642 710 0] max score 1.00 > >2018-07-11 04:23:01.865385 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:23:01.849527) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #24554 to level-5 74546859 bytes > >2018-07-11 04:23:01.865395 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:23:01.865297) EVENT_LOG_v1 {"time_micros": 1531282981865291, "job": 2663, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546859} >2018-07-11 04:23:01.865398 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:23:01.865304) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546859 bytes OK: base level 1 max bytes base 268435456 files[0 5 43 446 3641 711 0] max score 1.00 > >2018-07-11 04:23:01.882001 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:23:01.866243) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #23405 to level-5 74546859 bytes > >2018-07-11 04:23:01.882011 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:23:01.881913) EVENT_LOG_v1 {"time_micros": 1531282981881907, "job": 2664, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546859} >2018-07-11 04:23:01.882014 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:23:01.881921) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546859 bytes OK: base level 1 max bytes base 268435456 files[0 5 43 446 3640 712 0] max score 1.00 > >2018-07-11 04:23:01.898688 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:23:01.882865) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #35591 to level-4 74546215 bytes > >2018-07-11 04:23:01.898699 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:23:01.898596) EVENT_LOG_v1 {"time_micros": 1531282981898590, "job": 2665, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 74546215} >2018-07-11 04:23:01.898701 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:23:01.898604) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-4 74546215 bytes OK: base level 1 max bytes base 268435456 files[0 5 43 445 3641 712 0] max score 1.00 > >2018-07-11 04:23:01.915387 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:23:01.899563) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #5338 to level-5 74546859 bytes > >2018-07-11 04:23:01.915397 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:23:01.915295) EVENT_LOG_v1 {"time_micros": 1531282981915289, "job": 2666, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546859} >2018-07-11 04:23:01.915400 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:23:01.915303) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546859 bytes OK: base level 1 max bytes base 268435456 files[0 5 43 445 3640 713 0] max score 1.00 > >2018-07-11 04:25:56.066283 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:684] reusing log 35571 from recycle list > >2018-07-11 04:25:56.066409 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #35601. Immutable memtables: 0. > >2018-07-11 04:25:56.735060 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:25:56.066446) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1 >2018-07-11 04:25:56.735074 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 2667] Flushing memtable with next log file: 35601 > >2018-07-11 04:25:56.735097 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283156735087, "job": 2667, "event": "flush_started", "num_memtables": 1, "num_entries": 5000015, "num_deletes": 0, "memory_usage": 292943888} >2018-07-11 04:25:56.735102 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 2667] Level-0 flush table #35602: started >2018-07-11 04:26:00.817999 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283160817978, "cf_name": "default", "job": 2667, "event": "table_file_creation", "file_number": 35602, "file_size": 141286889, "table_properties": {"data_size": 127197777, "index_size": 1308397, "filter_size": 12779725, "raw_key_size": 144445101, "raw_average_key_size": 28, "raw_value_size": 59450671, "raw_average_value_size": 11, "num_data_blocks": 31252, "num_entries": 5000014, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:26:00.818021 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 2667] Level-0 flush table #35602: 141286889 bytes OK >2018-07-11 04:26:00.852286 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:242] adding log 35573 to recycle list > >2018-07-11 04:26:00.852299 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:26:00.818034) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #35602 started >2018-07-11 04:26:00.852303 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:26:00.852022) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #35602: memtable #1 done >2018-07-11 04:26:00.852306 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:26:00.852040) EVENT_LOG_v1 {"time_micros": 1531283160852033, "job": 2667, "event": "flush_finished", "lsm_state": [1, 5, 43, 445, 3640, 713, 0], "immutable_memtables": 0} >2018-07-11 04:26:00.852308 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:26:00.852212) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 5 43 445 3640 713 0] max score 1.00 > >2018-07-11 04:26:00.853163 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 2667] Try to delete WAL files size 178896755, prev total WAL file size 178896767, number of live WAL files 2. > >2018-07-11 04:27:17.158122 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:684] reusing log 35573 from recycle list > >2018-07-11 04:27:17.158294 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #35603. Immutable memtables: 0. > >2018-07-11 04:27:17.213208 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:27:17.158364) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1 >2018-07-11 04:27:17.213222 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 2668] Flushing memtable with next log file: 35603 > >2018-07-11 04:27:17.213241 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283237213231, "job": 2668, "event": "flush_started", "num_memtables": 1, "num_entries": 5000015, "num_deletes": 0, "memory_usage": 292933336} >2018-07-11 04:27:17.213244 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 2668] Level-0 flush table #35604: started >2018-07-11 04:27:21.046790 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283241046769, "cf_name": "default", "job": 2668, "event": "table_file_creation", "file_number": 35604, "file_size": 141288967, "table_properties": {"data_size": 127198839, "index_size": 1309411, "filter_size": 12779727, "raw_key_size": 144445125, "raw_average_key_size": 28, "raw_value_size": 59451570, "raw_average_value_size": 11, "num_data_blocks": 31252, "num_entries": 5000015, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:27:21.046811 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 2668] Level-0 flush table #35604: 141288967 bytes OK >2018-07-11 04:27:21.066897 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:242] adding log 35601 to recycle list > >2018-07-11 04:27:21.066922 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:27:21.046835) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #35604 started >2018-07-11 04:27:21.066926 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:27:21.066676) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #35604: memtable #1 done >2018-07-11 04:27:21.066928 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:27:21.066694) EVENT_LOG_v1 {"time_micros": 1531283241066687, "job": 2668, "event": "flush_finished", "lsm_state": [2, 5, 43, 445, 3640, 713, 0], "immutable_memtables": 0} >2018-07-11 04:27:21.066930 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:27:21.066834) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[2 5 43 445 3640 713 0] max score 1.05 > >2018-07-11 04:27:21.067056 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2669] Compacting 2@0 + 3@1 files to L1, score 1.05 >2018-07-11 04:27:21.067073 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2668 Base level 0, inputs: [35604(134MB) 35602(134MB)], [35575(71MB) 35580(43MB) 35581(62MB)] > >2018-07-11 04:27:21.067091 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283241067078, "job": 2669, "event": "compaction_started", "files_L0": [35604, 35602], "files_L1": [35575, 35580, 35581], "score": 1.05268, "input_data_size": 468640689} >2018-07-11 04:27:21.067807 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 2668] Try to delete WAL files size 178896755, prev total WAL file size 178896767, number of live WAL files 2. > >2018-07-11 04:27:24.010131 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2669] Generated table #35605: 2624118 keys, 74510204 bytes >2018-07-11 04:27:24.010176 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283244010148, "cf_name": "default", "job": 2669, "event": "table_file_creation", "file_number": 35605, "file_size": 74510204, "table_properties": {"data_size": 67109879, "index_size": 691450, "filter_size": 6707885, "raw_key_size": 75838866, "raw_average_key_size": 28, "raw_value_size": 31547126, "raw_average_value_size": 12, "num_data_blocks": 16475, "num_entries": 2624118, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:27:26.768709 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2669] Generated table #35606: 2638241 keys, 74545961 bytes >2018-07-11 04:27:26.768753 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283246768727, "cf_name": "default", "job": 2669, "event": "table_file_creation", "file_number": 35606, "file_size": 74545961, "table_properties": {"data_size": 67112134, "index_size": 689654, "filter_size": 6743183, "raw_key_size": 76215857, "raw_average_key_size": 28, "raw_value_size": 31365760, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:27:29.416078 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2669] Generated table #35607: 2638241 keys, 74547055 bytes >2018-07-11 04:27:29.416124 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283249416097, "cf_name": "default", "job": 2669, "event": "table_file_creation", "file_number": 35607, "file_size": 74547055, "table_properties": {"data_size": 67112158, "index_size": 690724, "filter_size": 6743183, "raw_key_size": 76215859, "raw_average_key_size": 28, "raw_value_size": 31365762, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:27:32.138969 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2669] Generated table #35608: 2638241 keys, 74546210 bytes >2018-07-11 04:27:32.139014 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283252138986, "cf_name": "default", "job": 2669, "event": "table_file_creation", "file_number": 35608, "file_size": 74546210, "table_properties": {"data_size": 67112097, "index_size": 689940, "filter_size": 6743183, "raw_key_size": 76215850, "raw_average_key_size": 28, "raw_value_size": 31365753, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:27:34.768377 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2669] Generated table #35609: 2638241 keys, 74546314 bytes >2018-07-11 04:27:34.768420 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283254768403, "cf_name": "default", "job": 2669, "event": "table_file_creation", "file_number": 35609, "file_size": 74546314, "table_properties": {"data_size": 67112160, "index_size": 689981, "filter_size": 6743183, "raw_key_size": 76215852, "raw_average_key_size": 28, "raw_value_size": 31365755, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:27:36.088847 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2669] Generated table #35610: 983479 keys, 29962066 bytes >2018-07-11 04:27:36.088883 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283256088868, "cf_name": "default", "job": 2669, "event": "table_file_creation", "file_number": 35610, "file_size": 29962066, "table_properties": {"data_size": 27167359, "index_size": 274997, "filter_size": 2518722, "raw_key_size": 28688322, "raw_average_key_size": 29, "raw_value_size": 13525270, "raw_average_value_size": 13, "num_data_blocks": 6679, "num_entries": 983479, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:27:37.325144 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2669] Generated table #35611: 162251 keys, 65974354 bytes >2018-07-11 04:27:37.325174 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283257325159, "cf_name": "default", "job": 2669, "event": "table_file_creation", "file_number": 35611, "file_size": 65974354, "table_properties": {"data_size": 64873476, "index_size": 545887, "filter_size": 554003, "raw_key_size": 14985619, "raw_average_key_size": 92, "raw_value_size": 51246182, "raw_average_value_size": 315, "num_data_blocks": 14659, "num_entries": 162251, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "80898", "kMergeOperands": "0"}} >2018-07-11 04:27:37.325254 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2669] Compacted 2@0 + 3@1 files to L1 => 468632164 bytes >2018-07-11 04:27:37.341639 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:27:37.341390) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 9 43 445 3640 713 0] max score 1.96, MB/sec: 28.8 rd, 28.8 wr, level 1, files in(2, 3) out(7) MB in(269.5, 177.4) out(446.9), read-write-amplify(3.3) write-amplify(1.7) OK, records in: 14322821, records dropped: 9 > >2018-07-11 04:27:37.341651 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:27:37.341415) EVENT_LOG_v1 {"time_micros": 1531283257341404, "job": 2669, "event": "compaction_finished", "compaction_time_micros": 16258137, "output_level": 1, "num_output_files": 7, "total_output_size": 468632164, "num_input_records": 14322821, "num_output_records": 14322812, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 9, 43, 445, 3640, 713, 0]} >2018-07-11 04:27:37.342627 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283257342611, "job": 2669, "event": "table_file_deletion", "file_number": 35604} >2018-07-11 04:27:37.342641 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283257342639, "job": 2669, "event": "table_file_deletion", "file_number": 35602} >2018-07-11 04:27:37.342653 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283257342651, "job": 2669, "event": "table_file_deletion", "file_number": 35581} >2018-07-11 04:27:37.342665 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283257342662, "job": 2669, "event": "table_file_deletion", "file_number": 35580} >2018-07-11 04:27:37.342675 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283257342673, "job": 2669, "event": "table_file_deletion", "file_number": 35575} >2018-07-11 04:27:37.342891 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_picker.cc:461] [default] Expanding@1 1+1(74547055+70255848 bytes) to 3+1 (223639579+70255848bytes) > >2018-07-11 04:27:37.342991 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2670] Compacting 3@1 + 1@2 files to L2, score 1.96 >2018-07-11 04:27:37.343003 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2669 Base level 1, inputs: [35607(71MB) 35608(71MB) 35609(71MB)], [35585(67MB)] > >2018-07-11 04:27:37.343016 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283257343006, "job": 2670, "event": "compaction_started", "files_L1": [35607, 35608, 35609], "files_L2": [35585], "score": 1.95714, "input_data_size": 293895427} >2018-07-11 04:27:40.770049 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2670] Generated table #35612: 2638240 keys, 74545757 bytes >2018-07-11 04:27:40.770082 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283260770067, "cf_name": "default", "job": 2670, "event": "table_file_creation", "file_number": 35612, "file_size": 74545757, "table_properties": {"data_size": 67112108, "index_size": 689478, "filter_size": 6743181, "raw_key_size": 76215816, "raw_average_key_size": 28, "raw_value_size": 31365736, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:27:44.166678 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2670] Generated table #35613: 2638241 keys, 74546752 bytes >2018-07-11 04:27:44.166723 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283264166696, "cf_name": "default", "job": 2670, "event": "table_file_creation", "file_number": 35613, "file_size": 74546752, "table_properties": {"data_size": 67112144, "index_size": 690435, "filter_size": 6743183, "raw_key_size": 76215855, "raw_average_key_size": 28, "raw_value_size": 31365758, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:27:47.256693 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2670] Generated table #35614: 2638241 keys, 74546753 bytes >2018-07-11 04:27:47.256738 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283267256711, "cf_name": "default", "job": 2670, "event": "table_file_creation", "file_number": 35614, "file_size": 74546753, "table_properties": {"data_size": 67112148, "index_size": 690432, "filter_size": 6743183, "raw_key_size": 76215860, "raw_average_key_size": 28, "raw_value_size": 31365763, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:27:50.118561 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2670] Generated table #35615: 2481961 keys, 70256423 bytes >2018-07-11 04:27:50.118606 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283270118579, "cf_name": "default", "job": 2670, "event": "table_file_creation", "file_number": 35615, "file_size": 70256423, "table_properties": {"data_size": 63261193, "index_size": 650216, "filter_size": 6344025, "raw_key_size": 71716837, "raw_average_key_size": 28, "raw_value_size": 29613122, "raw_average_value_size": 11, "num_data_blocks": 15545, "num_entries": 2481961, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "11", "kMergeOperands": "0"}} >2018-07-11 04:27:50.118641 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2670] Compacted 3@1 + 1@2 files to L2 => 293895685 bytes >2018-07-11 04:27:50.141588 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:27:50.141289) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 6 46 445 3640 713 0] max score 1.12, MB/sec: 23.0 rd, 23.0 wr, level 2, files in(3, 1) out(4) MB in(213.3, 67.0) out(280.3), read-write-amplify(2.6) write-amplify(1.3) OK, records in: 10396683, records dropped: 0 > >2018-07-11 04:27:50.141599 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:27:50.141313) EVENT_LOG_v1 {"time_micros": 1531283270141303, "job": 2670, "event": "compaction_finished", "compaction_time_micros": 12775610, "output_level": 2, "num_output_files": 4, "total_output_size": 293895685, "num_input_records": 10396683, "num_output_records": 10396683, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 6, 46, 445, 3640, 713, 0]} >2018-07-11 04:27:50.142518 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283270142515, "job": 2670, "event": "table_file_deletion", "file_number": 35609} >2018-07-11 04:27:50.142543 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283270142541, "job": 2670, "event": "table_file_deletion", "file_number": 35608} >2018-07-11 04:27:50.142554 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283270142552, "job": 2670, "event": "table_file_deletion", "file_number": 35607} >2018-07-11 04:27:50.142567 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283270142565, "job": 2670, "event": "table_file_deletion", "file_number": 35585} >2018-07-11 04:27:50.142877 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2671] Compacting 1@1 + 4@2 files to L2, score 1.12 >2018-07-11 04:27:50.142889 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2670 Base level 1, inputs: [35606(71MB)], [35586(71MB) 35587(71MB) 35588(71MB) 35612(71MB)] > >2018-07-11 04:27:50.142903 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283270142892, "job": 2671, "event": "compaction_started", "files_L1": [35606], "files_L2": [35586, 35587, 35588, 35612], "score": 1.12402, "input_data_size": 372730139} >2018-07-11 04:27:53.253788 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2671] Generated table #35616: 2638238 keys, 74545704 bytes >2018-07-11 04:27:53.253833 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283273253806, "cf_name": "default", "job": 2671, "event": "table_file_creation", "file_number": 35616, "file_size": 74545704, "table_properties": {"data_size": 67112111, "index_size": 689426, "filter_size": 6743177, "raw_key_size": 76215770, "raw_average_key_size": 28, "raw_value_size": 31365724, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638238, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:27:56.363022 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2671] Generated table #35617: 2638241 keys, 74545881 bytes >2018-07-11 04:27:56.363068 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283276363041, "cf_name": "default", "job": 2671, "event": "table_file_creation", "file_number": 35617, "file_size": 74545881, "table_properties": {"data_size": 67112133, "index_size": 689575, "filter_size": 6743183, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:27:59.534822 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2671] Generated table #35618: 2638240 keys, 74546255 bytes >2018-07-11 04:27:59.534870 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283279534854, "cf_name": "default", "job": 2671, "event": "table_file_creation", "file_number": 35618, "file_size": 74546255, "table_properties": {"data_size": 67112137, "index_size": 689947, "filter_size": 6743181, "raw_key_size": 76215817, "raw_average_key_size": 28, "raw_value_size": 31365737, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:28:02.802227 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2671] Generated table #35619: 2638241 keys, 74545774 bytes >2018-07-11 04:28:02.802276 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283282802244, "cf_name": "default", "job": 2671, "event": "table_file_creation", "file_number": 35619, "file_size": 74545774, "table_properties": {"data_size": 67112121, "index_size": 689480, "filter_size": 6743183, "raw_key_size": 76215852, "raw_average_key_size": 28, "raw_value_size": 31365755, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:28:05.969072 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2671] Generated table #35620: 2638240 keys, 74546593 bytes >2018-07-11 04:28:05.969119 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283285969090, "cf_name": "default", "job": 2671, "event": "table_file_creation", "file_number": 35620, "file_size": 74546593, "table_properties": {"data_size": 67112092, "index_size": 690341, "filter_size": 6743170, "raw_key_size": 76215820, "raw_average_key_size": 28, "raw_value_size": 31365740, "raw_average_value_size": 11, "num_data_blocks": 16489, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:28:05.969158 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2671] Compacted 1@1 + 4@2 files to L2 => 372730207 bytes >2018-07-11 04:28:05.991150 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:28:05.990923) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 5 47 445 3640 713 0] max score 1.07, MB/sec: 23.6 rd, 23.6 wr, level 2, files in(1, 4) out(5) MB in(71.1, 284.4) out(355.5), read-write-amplify(10.0) write-amplify(5.0) OK, records in: 13191200, records dropped: 0 > >2018-07-11 04:28:05.991161 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:28:05.990949) EVENT_LOG_v1 {"time_micros": 1531283285990938, "job": 2671, "event": "compaction_finished", "compaction_time_micros": 15826238, "output_level": 2, "num_output_files": 5, "total_output_size": 372730207, "num_input_records": 13191200, "num_output_records": 13191200, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 5, 47, 445, 3640, 713, 0]} >2018-07-11 04:28:05.992094 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283285992078, "job": 2671, "event": "table_file_deletion", "file_number": 35612} >2018-07-11 04:28:05.992110 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283285992108, "job": 2671, "event": "table_file_deletion", "file_number": 35606} >2018-07-11 04:28:05.992127 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283285992125, "job": 2671, "event": "table_file_deletion", "file_number": 35588} >2018-07-11 04:28:05.992138 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283285992136, "job": 2671, "event": "table_file_deletion", "file_number": 35587} >2018-07-11 04:28:05.992149 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283285992147, "job": 2671, "event": "table_file_deletion", "file_number": 35586} >2018-07-11 04:28:05.992373 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_picker.cc:461] [default] Expanding@2 1+1(74546753+46965300 bytes) to 3+1 (223640098+46965300bytes) > >2018-07-11 04:28:05.992473 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2672] Compacting 3@2 + 1@3 files to L3, score 1.07 >2018-07-11 04:28:05.992484 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2671 Base level 2, inputs: [35620(71MB) 35613(71MB) 35614(71MB)], [35598(44MB)] > >2018-07-11 04:28:05.992498 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283285992487, "job": 2672, "event": "compaction_started", "files_L2": [35620, 35613, 35614], "files_L3": [35598], "score": 1.07234, "input_data_size": 270605398} >2018-07-11 04:28:09.133743 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2672] Generated table #35621: 2638240 keys, 74546660 bytes >2018-07-11 04:28:09.133789 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283289133773, "cf_name": "default", "job": 2672, "event": "table_file_creation", "file_number": 35621, "file_size": 74546660, "table_properties": {"data_size": 67112126, "index_size": 690363, "filter_size": 6743181, "raw_key_size": 76215830, "raw_average_key_size": 28, "raw_value_size": 31365750, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:28:12.139750 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2672] Generated table #35622: 2638241 keys, 74546084 bytes >2018-07-11 04:28:12.139792 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283292139765, "cf_name": "default", "job": 2672, "event": "table_file_creation", "file_number": 35622, "file_size": 74546084, "table_properties": {"data_size": 67112151, "index_size": 689760, "filter_size": 6743183, "raw_key_size": 76215855, "raw_average_key_size": 28, "raw_value_size": 31365758, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:28:15.164635 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2672] Generated table #35623: 2638241 keys, 74546067 bytes >2018-07-11 04:28:15.164680 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283295164653, "cf_name": "default", "job": 2672, "event": "table_file_creation", "file_number": 35623, "file_size": 74546067, "table_properties": {"data_size": 67112120, "index_size": 689774, "filter_size": 6743183, "raw_key_size": 76215847, "raw_average_key_size": 28, "raw_value_size": 31365750, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:28:17.104603 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2672] Generated table #35624: 1519915 keys, 46964805 bytes >2018-07-11 04:28:17.104633 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283297104619, "cf_name": "default", "job": 2672, "event": "table_file_creation", "file_number": 35624, "file_size": 46964805, "table_properties": {"data_size": 42636627, "index_size": 433042, "filter_size": 3894148, "raw_key_size": 44348913, "raw_average_key_size": 29, "raw_value_size": 21522459, "raw_average_value_size": 14, "num_data_blocks": 10538, "num_entries": 1519915, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:28:17.104665 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2672] Compacted 3@2 + 1@3 files to L3 => 270603616 bytes >2018-07-11 04:28:17.124339 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:28:17.124027) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 5 44 448 3640 713 0] max score 1.01, MB/sec: 24.4 rd, 24.4 wr, level 3, files in(3, 1) out(4) MB in(213.3, 44.8) out(258.1), read-write-amplify(2.4) write-amplify(1.2) OK, records in: 9434637, records dropped: 0 > >2018-07-11 04:28:17.124352 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:28:17.124059) EVENT_LOG_v1 {"time_micros": 1531283297124044, "job": 2672, "event": "compaction_finished", "compaction_time_micros": 11112154, "output_level": 3, "num_output_files": 4, "total_output_size": 270603616, "num_input_records": 9434637, "num_output_records": 9434637, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 5, 44, 448, 3640, 713, 0]} >2018-07-11 04:28:17.125486 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283297125481, "job": 2672, "event": "table_file_deletion", "file_number": 35620} >2018-07-11 04:28:17.125509 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283297125506, "job": 2672, "event": "table_file_deletion", "file_number": 35614} >2018-07-11 04:28:17.125530 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283297125527, "job": 2672, "event": "table_file_deletion", "file_number": 35613} >2018-07-11 04:28:17.125553 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283297125550, "job": 2672, "event": "table_file_deletion", "file_number": 35598} >2018-07-11 04:28:17.149148 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:28:17.125865) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #35621 to level-4 74546660 bytes > >2018-07-11 04:28:17.149159 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:28:17.149041) EVENT_LOG_v1 {"time_micros": 1531283297149034, "job": 2673, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 74546660} >2018-07-11 04:28:17.149174 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:28:17.149049) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-4 74546660 bytes OK: base level 1 max bytes base 268435456 files[0 5 44 447 3641 713 0] max score 1.00 > >2018-07-11 04:28:17.174173 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:28:17.150164) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #35593 to level-4 74546196 bytes > >2018-07-11 04:28:17.174183 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:28:17.174068) EVENT_LOG_v1 {"time_micros": 1531283297174062, "job": 2674, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 74546196} >2018-07-11 04:28:17.174187 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:28:17.174077) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-4 74546196 bytes OK: base level 1 max bytes base 268435456 files[0 5 44 446 3642 713 0] max score 1.00 > >2018-07-11 04:28:17.199189 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:28:17.175159) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #7459 to level-5 74546858 bytes > >2018-07-11 04:28:17.199200 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:28:17.199086) EVENT_LOG_v1 {"time_micros": 1531283297199079, "job": 2675, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546858} >2018-07-11 04:28:17.199216 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:28:17.199094) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546858 bytes OK: base level 1 max bytes base 268435456 files[0 5 44 446 3641 714 0] max score 1.00 > >2018-07-11 04:28:17.224220 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:28:17.200170) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #23366 to level-5 74546858 bytes > >2018-07-11 04:28:17.224231 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:28:17.224116) EVENT_LOG_v1 {"time_micros": 1531283297224110, "job": 2676, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546858} >2018-07-11 04:28:17.224235 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:28:17.224125) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546858 bytes OK: base level 1 max bytes base 268435456 files[0 5 44 446 3640 715 0] max score 1.00 > >2018-07-11 04:28:17.249242 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:28:17.225197) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #35622 to level-4 74546084 bytes > >2018-07-11 04:28:17.249259 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:28:17.249138) EVENT_LOG_v1 {"time_micros": 1531283297249132, "job": 2677, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 74546084} >2018-07-11 04:28:17.249263 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:28:17.249146) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-4 74546084 bytes OK: base level 1 max bytes base 268435456 files[0 5 44 445 3641 715 0] max score 1.00 > >2018-07-11 04:28:17.274277 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:28:17.250217) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #1164 to level-5 74546858 bytes > >2018-07-11 04:28:17.274287 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:28:17.274166) EVENT_LOG_v1 {"time_micros": 1531283297274159, "job": 2678, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546858} >2018-07-11 04:28:17.274291 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:28:17.274174) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546858 bytes OK: base level 1 max bytes base 268435456 files[0 5 44 445 3640 716 0] max score 1.00 > >2018-07-11 04:30:24.911095 7f416c1fb700 0 log_channel(cluster) log [DBG] : 11.1 scrub starts >2018-07-11 04:30:48.949147 7f416c1fb700 0 log_channel(cluster) log [DBG] : 11.1 scrub ok >2018-07-11 04:31:30.271988 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:684] reusing log 35601 from recycle list > >2018-07-11 04:31:30.272107 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #35625. Immutable memtables: 0. > >2018-07-11 04:31:30.367512 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:31:30.272187) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1 >2018-07-11 04:31:30.367526 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 2679] Flushing memtable with next log file: 35625 > >2018-07-11 04:31:30.367544 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283490367534, "job": 2679, "event": "flush_started", "num_memtables": 1, "num_entries": 5000018, "num_deletes": 0, "memory_usage": 292959520} >2018-07-11 04:31:30.367547 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 2679] Level-0 flush table #35626: started >2018-07-11 04:31:34.258634 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283494258609, "cf_name": "default", "job": 2679, "event": "table_file_creation", "file_number": 35626, "file_size": 141288725, "table_properties": {"data_size": 127199914, "index_size": 1308074, "filter_size": 12779747, "raw_key_size": 144445229, "raw_average_key_size": 28, "raw_value_size": 59452528, "raw_average_value_size": 11, "num_data_blocks": 31253, "num_entries": 5000018, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:31:34.258672 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 2679] Level-0 flush table #35626: 141288725 bytes OK >2018-07-11 04:31:34.277477 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:242] adding log 35603 to recycle list > >2018-07-11 04:31:34.277488 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:31:34.258686) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #35626 started >2018-07-11 04:31:34.277505 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:31:34.277232) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #35626: memtable #1 done >2018-07-11 04:31:34.277507 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:31:34.277262) EVENT_LOG_v1 {"time_micros": 1531283494277243, "job": 2679, "event": "flush_finished", "lsm_state": [1, 5, 44, 445, 3640, 716, 0], "immutable_memtables": 0} >2018-07-11 04:31:34.277509 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:31:34.277425) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 5 44 445 3640 716 0] max score 1.00 > >2018-07-11 04:31:34.278387 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 2679] Try to delete WAL files size 178897851, prev total WAL file size 178897863, number of live WAL files 2. > >2018-07-11 04:34:25.327797 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:684] reusing log 35603 from recycle list > >2018-07-11 04:34:25.327943 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #35627. Immutable memtables: 0. > >2018-07-11 04:34:25.384674 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:34:25.328002) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1 >2018-07-11 04:34:25.384688 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 2680] Flushing memtable with next log file: 35627 > >2018-07-11 04:34:25.384704 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283665384696, "job": 2680, "event": "flush_started", "num_memtables": 1, "num_entries": 5000015, "num_deletes": 0, "memory_usage": 292943592} >2018-07-11 04:34:25.384707 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 2680] Level-0 flush table #35628: started >2018-07-11 04:34:29.218560 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283669218539, "cf_name": "default", "job": 2680, "event": "table_file_creation", "file_number": 35628, "file_size": 141286221, "table_properties": {"data_size": 127196936, "index_size": 1308576, "filter_size": 12779719, "raw_key_size": 144445077, "raw_average_key_size": 28, "raw_value_size": 59449772, "raw_average_value_size": 11, "num_data_blocks": 31252, "num_entries": 5000013, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:34:29.218583 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 2680] Level-0 flush table #35628: 141286221 bytes OK >2018-07-11 04:34:29.239279 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:242] adding log 35625 to recycle list > >2018-07-11 04:34:29.239292 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:34:29.218607) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #35628 started >2018-07-11 04:34:29.239295 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:34:29.239122) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #35628: memtable #1 done >2018-07-11 04:34:29.239304 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:34:29.239138) EVENT_LOG_v1 {"time_micros": 1531283669239131, "job": 2680, "event": "flush_finished", "lsm_state": [2, 5, 44, 445, 3640, 716, 0], "immutable_memtables": 0} >2018-07-11 04:34:29.239306 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:34:29.239216) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[2 5 44 445 3640 716 0] max score 1.05 > >2018-07-11 04:34:29.239701 7f41641eb700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS ------- >2018-07-11 04:34:29.239711 7f41641eb700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] >** DB Stats ** >Uptime(secs): 48410.9 total, 763.3 interval >Cumulative writes: 196K writes, 1777M keys, 196K commit groups, 1.0 writes per commit group, ingest: 59.73 GB, 1.26 MB/s >Cumulative WAL: 196K writes, 56K syncs, 3.49 writes per sync, written: 59.73 GB, 1.26 MB/s >Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent >Interval writes: 44 writes, 20M keys, 44 commit groups, 1.0 writes per commit group, ingest: 682.44 MB, 0.89 MB/s >Interval WAL: 44 writes, 22 syncs, 1.91 writes per sync, written: 0.67 MB, 0.89 MB/s >Interval stall: 00:00:0.000 H:M:S, 0.0 percent > >** Compaction Stats [default] ** >Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop >---------------------------------------------------------------------------------------------------------------------------------------------------------- > L0 2/0 269.48 MB 1.1 0.0 0.0 0.0 46.8 46.8 0.0 1.0 0.0 32.4 1479 357 4.141 0 0 > L1 5/0 216.66 MB 0.8 77.1 46.6 30.5 77.0 46.6 0.0 1.7 25.7 25.7 3070 175 17.544 2438M 894K > L2 44/0 2.47 GB 1.0 89.9 46.6 43.3 89.8 46.5 0.0 1.9 21.5 21.5 4282 546 7.842 3350M 1358K > L3 445/0 24.93 GB 1.0 81.3 45.9 35.3 81.2 45.8 0.6 1.8 20.7 20.7 4022 277 14.519 3069M 4000K > L4 3640/0 250.00 GB 1.0 71.2 16.9 54.3 70.9 16.6 29.6 4.2 21.1 21.0 3452 226 15.273 2710M 15M > L5 716/0 49.69 GB 0.0 1.7 0.9 0.8 1.7 0.9 45.2 1.9 18.7 18.5 95 12 7.912 66M 1097K > Sum 4852/0 327.57 GB 0.0 321.2 156.9 164.2 367.4 203.1 75.4 7.9 20.1 22.9 16399 1593 10.295 11G 23M > Int 0/0 0.00 KB 0.0 3.0 1.6 1.4 3.6 2.2 0.9 6.8 21.2 24.8 147 13 11.336 111M 15 >Uptime(secs): 48410.9 total, 48410.9 interval >Flush(GB): cumulative 46.760, interval 0.526 >AddFile(GB): cumulative 0.000, interval 0.000 >AddFile(Total Files): cumulative 0, interval 0 >AddFile(L0 Files): cumulative 0, interval 0 >AddFile(Keys): cumulative 0, interval 0 >Cumulative compaction: 367.38 GB write, 7.77 MB/s write, 321.15 GB read, 6.79 MB/s read, 16399.2 seconds >Interval compaction: 3.58 GB write, 0.08 MB/s write, 3.05 GB read, 0.06 MB/s read, 147.4 seconds >Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count > >** File Read Latency Histogram By Level [default] ** > >** Compaction Stats [default] ** >Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop >---------------------------------------------------------------------------------------------------------------------------------------------------------- > L0 2/0 269.48 MB 1.1 0.0 0.0 0.0 46.8 46.8 0.0 1.0 0.0 32.4 1479 357 4.141 0 0 > L1 5/0 216.66 MB 0.8 77.1 46.6 30.5 77.0 46.6 0.0 1.7 25.7 25.7 3070 175 17.544 2438M 894K > L2 44/0 2.47 GB 1.0 89.9 46.6 43.3 89.8 46.5 0.0 1.9 21.5 21.5 4282 546 7.842 3350M 1358K > L3 445/0 24.93 GB 1.0 81.3 45.9 35.3 81.2 45.8 0.6 1.8 20.7 20.7 4022 277 14.519 3069M 4000K > L4 3640/0 250.00 GB 1.0 71.2 16.9 54.3 70.9 16.6 29.6 4.2 21.1 21.0 3452 226 15.273 2710M 15M > L5 716/0 49.69 GB 0.0 1.7 0.9 0.8 1.7 0.9 45.2 1.9 18.7 18.5 95 12 7.912 66M 1097K > Sum 4852/0 327.57 GB 0.0 321.2 156.9 164.2 367.4 203.1 75.4 7.9 20.1 22.9 16399 1593 10.295 11G 23M > Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 >Uptime(secs): 48410.9 total, 48410.9 interval >Flush(GB): cumulative 46.760, interval 0.000 >AddFile(GB): cumulative 0.000, interval 0.000 >AddFile(Total Files): cumulative 0, interval 0 >AddFile(L0 Files): cumulative 0, interval 0 >AddFile(Keys): cumulative 0, interval 0 >Cumulative compaction: 367.38 GB write, 7.77 MB/s write, 321.15 GB read, 6.79 MB/s read, 16399.2 seconds >Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds >Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count > >** File Read Latency Histogram By Level [default] ** > >2018-07-11 04:34:29.239832 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2681] Compacting 2@0 + 3@1 files to L1, score 1.05 >2018-07-11 04:34:29.239845 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2680 Base level 0, inputs: [35628(134MB) 35626(134MB)], [35605(71MB) 35610(28MB) 35611(62MB)] > >2018-07-11 04:34:29.239861 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283669239849, "job": 2681, "event": "compaction_started", "files_L0": [35628, 35626], "files_L1": [35605, 35610, 35611], "score": 1.05267, "input_data_size": 453021570} >2018-07-11 04:34:29.240166 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 2680] Try to delete WAL files size 178896755, prev total WAL file size 178896767, number of live WAL files 2. > >2018-07-11 04:34:32.089931 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2681] Generated table #35629: 2624010 keys, 74509474 bytes >2018-07-11 04:34:32.089978 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283672089962, "cf_name": "default", "job": 2681, "event": "table_file_creation", "file_number": 35629, "file_size": 74509474, "table_properties": {"data_size": 67109539, "index_size": 691330, "filter_size": 6707615, "raw_key_size": 75835953, "raw_average_key_size": 28, "raw_value_size": 31548251, "raw_average_value_size": 12, "num_data_blocks": 16475, "num_entries": 2624010, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:34:34.793702 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2681] Generated table #35630: 2638241 keys, 74546008 bytes >2018-07-11 04:34:34.793748 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283674793721, "cf_name": "default", "job": 2681, "event": "table_file_creation", "file_number": 35630, "file_size": 74546008, "table_properties": {"data_size": 67112190, "index_size": 689645, "filter_size": 6743183, "raw_key_size": 76215855, "raw_average_key_size": 28, "raw_value_size": 31365758, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:34:36.764914 7f41641eb700 1 bluefs _allocate failed to allocate 0x4800000 on bdev 1, free 0x2bfe000; fallback to bdev 2 >2018-07-11 04:34:37.754544 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2681] Generated table #35631: 2638241 keys, 74546562 bytes >2018-07-11 04:34:37.754588 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283677754561, "cf_name": "default", "job": 2681, "event": "table_file_creation", "file_number": 35631, "file_size": 74546562, "table_properties": {"data_size": 67112130, "index_size": 690259, "filter_size": 6743183, "raw_key_size": 76215851, "raw_average_key_size": 28, "raw_value_size": 31365754, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:34:39.725294 7f41641eb700 1 bluefs _allocate failed to allocate 0x4800000 on bdev 1, free 0x2bfe000; fallback to bdev 2 >2018-07-11 04:34:40.823579 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2681] Generated table #35632: 2638241 keys, 74546739 bytes >2018-07-11 04:34:40.823623 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283680823596, "cf_name": "default", "job": 2681, "event": "table_file_creation", "file_number": 35632, "file_size": 74546739, "table_properties": {"data_size": 67112141, "index_size": 690425, "filter_size": 6743183, "raw_key_size": 76215853, "raw_average_key_size": 28, "raw_value_size": 31365756, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:34:42.795772 7f41641eb700 1 bluefs _allocate failed to allocate 0x4800000 on bdev 1, free 0x2bfe000; fallback to bdev 2 >2018-07-11 04:34:43.713118 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2681] Generated table #35633: 2638241 keys, 74546100 bytes >2018-07-11 04:34:43.713160 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283683713143, "cf_name": "default", "job": 2681, "event": "table_file_creation", "file_number": 35633, "file_size": 74546100, "table_properties": {"data_size": 67112145, "index_size": 689782, "filter_size": 6743183, "raw_key_size": 76215855, "raw_average_key_size": 28, "raw_value_size": 31365758, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:34:44.448941 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2681] Generated table #35634: 426744 keys, 12058470 bytes >2018-07-11 04:34:44.448981 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283684448955, "cf_name": "default", "job": 2681, "event": "table_file_creation", "file_number": 35634, "file_size": 12058470, "table_properties": {"data_size": 10855623, "index_size": 111124, "filter_size": 1090737, "raw_key_size": 12328170, "raw_average_key_size": 28, "raw_value_size": 5073522, "raw_average_value_size": 11, "num_data_blocks": 2668, "num_entries": 426744, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:34:44.990392 7f41641eb700 1 bluefs _allocate failed to allocate 0x4200000 on bdev 1, free 0x1ffe000; fallback to bdev 2 >2018-07-11 04:34:45.858268 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2681] Generated table #35635: 166136 keys, 68246471 bytes >2018-07-11 04:34:45.858309 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283685858283, "cf_name": "default", "job": 2681, "event": "table_file_creation", "file_number": 35635, "file_size": 68246471, "table_properties": {"data_size": 67111660, "index_size": 564904, "filter_size": 568919, "raw_key_size": 15373916, "raw_average_key_size": 92, "raw_value_size": 53115264, "raw_average_value_size": 319, "num_data_blocks": 15213, "num_entries": 166136, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "80898", "kMergeOperands": "0"}} >2018-07-11 04:34:45.873287 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2681] Generated table #35636: 17 keys, 15845 bytes >2018-07-11 04:34:45.873313 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283685873300, "cf_name": "default", "job": 2681, "event": "table_file_creation", "file_number": 35636, "file_size": 15845, "table_properties": {"data_size": 14534, "index_size": 250, "filter_size": 93, "raw_key_size": 1468, "raw_average_key_size": 86, "raw_value_size": 13623, "raw_average_value_size": 801, "num_data_blocks": 4, "num_entries": 17, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:34:45.873375 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2681] Compacted 2@0 + 3@1 files to L1 => 453015669 bytes >2018-07-11 04:34:45.897614 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:34:45.897331) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 10 44 445 3640 716 0] max score 1.90, MB/sec: 27.2 rd, 27.2 wr, level 1, files in(2, 3) out(8) MB in(269.5, 162.6) out(432.0), read-write-amplify(3.2) write-amplify(1.6) OK, records in: 13769879, records dropped: 8 > >2018-07-11 04:34:45.897625 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:34:45.897356) EVENT_LOG_v1 {"time_micros": 1531283685897345, "job": 2681, "event": "compaction_finished", "compaction_time_micros": 16633495, "output_level": 1, "num_output_files": 8, "total_output_size": 453015669, "num_input_records": 13769879, "num_output_records": 13769871, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 10, 44, 445, 3640, 716, 0]} >2018-07-11 04:34:45.898547 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283685898543, "job": 2681, "event": "table_file_deletion", "file_number": 35628} >2018-07-11 04:34:45.898560 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283685898558, "job": 2681, "event": "table_file_deletion", "file_number": 35626} >2018-07-11 04:34:45.898570 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283685898568, "job": 2681, "event": "table_file_deletion", "file_number": 35611} >2018-07-11 04:34:45.898579 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283685898577, "job": 2681, "event": "table_file_deletion", "file_number": 35610} >2018-07-11 04:34:45.898602 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283685898600, "job": 2681, "event": "table_file_deletion", "file_number": 35605} >2018-07-11 04:34:45.898833 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_picker.cc:461] [default] Expanding@1 1+1(74546739+70256423 bytes) to 4+1 (235697871+70256423bytes) > >2018-07-11 04:34:45.898929 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2682] Compacting 4@1 + 1@2 files to L2, score 1.90 >2018-07-11 04:34:45.898940 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2681 Base level 1, inputs: [35631(71MB) 35632(71MB) 35633(71MB) 35634(11MB)], [35615(67MB)] > >2018-07-11 04:34:45.898953 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283685898943, "job": 2682, "event": "compaction_started", "files_L1": [35631, 35632, 35633, 35634], "files_L2": [35615], "score": 1.89896, "input_data_size": 305954294} >2018-07-11 04:34:49.046979 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2682] Generated table #35637: 2638240 keys, 74546317 bytes >2018-07-11 04:34:49.047024 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283689046996, "cf_name": "default", "job": 2682, "event": "table_file_creation", "file_number": 35637, "file_size": 74546317, "table_properties": {"data_size": 67112115, "index_size": 690031, "filter_size": 6743181, "raw_key_size": 76215817, "raw_average_key_size": 28, "raw_value_size": 31365737, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:34:52.098178 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2682] Generated table #35638: 2638241 keys, 74546016 bytes >2018-07-11 04:34:52.098222 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283692098195, "cf_name": "default", "job": 2682, "event": "table_file_creation", "file_number": 35638, "file_size": 74546016, "table_properties": {"data_size": 67112100, "index_size": 689743, "filter_size": 6743183, "raw_key_size": 76215851, "raw_average_key_size": 28, "raw_value_size": 31365754, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:34:55.122788 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2682] Generated table #35639: 2638241 keys, 74546084 bytes >2018-07-11 04:34:55.122833 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283695122807, "cf_name": "default", "job": 2682, "event": "table_file_creation", "file_number": 35639, "file_size": 74546084, "table_properties": {"data_size": 67112157, "index_size": 689754, "filter_size": 6743183, "raw_key_size": 76215856, "raw_average_key_size": 28, "raw_value_size": 31365759, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:34:58.231186 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2682] Generated table #35640: 2638241 keys, 74547037 bytes >2018-07-11 04:34:58.231219 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283698231205, "cf_name": "default", "job": 2682, "event": "table_file_creation", "file_number": 35640, "file_size": 74547037, "table_properties": {"data_size": 67112099, "index_size": 690765, "filter_size": 6743183, "raw_key_size": 76215857, "raw_average_key_size": 28, "raw_value_size": 31365760, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:34:58.565874 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2682] Generated table #35641: 270465 keys, 7768639 bytes >2018-07-11 04:34:58.565913 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283698565900, "cf_name": "default", "job": 2682, "event": "table_file_creation", "file_number": 35641, "file_size": 7768639, "table_properties": {"data_size": 7004686, "index_size": 71380, "filter_size": 691587, "raw_key_size": 7829185, "raw_average_key_size": 28, "raw_value_size": 3320902, "raw_average_value_size": 12, "num_data_blocks": 1723, "num_entries": 270465, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "11", "kMergeOperands": "0"}} >2018-07-11 04:34:58.565949 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2682] Compacted 4@1 + 1@2 files to L2 => 305954093 bytes >2018-07-11 04:34:58.589048 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:34:58.588733) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 6 48 445 3640 716 0] max score 1.08, MB/sec: 24.2 rd, 24.2 wr, level 2, files in(4, 1) out(5) MB in(224.8, 67.0) out(291.8), read-write-amplify(2.6) write-amplify(1.3) OK, records in: 10823428, records dropped: 0 > >2018-07-11 04:34:58.589059 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:34:58.588758) EVENT_LOG_v1 {"time_micros": 1531283698588747, "job": 2682, "event": "compaction_finished", "compaction_time_micros": 12666972, "output_level": 2, "num_output_files": 5, "total_output_size": 305954093, "num_input_records": 10823428, "num_output_records": 10823428, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 6, 48, 445, 3640, 716, 0]} >2018-07-11 04:34:58.589973 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283698589969, "job": 2682, "event": "table_file_deletion", "file_number": 35634} >2018-07-11 04:34:58.589986 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283698589984, "job": 2682, "event": "table_file_deletion", "file_number": 35633} >2018-07-11 04:34:58.589998 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283698589995, "job": 2682, "event": "table_file_deletion", "file_number": 35632} >2018-07-11 04:34:58.590008 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283698590006, "job": 2682, "event": "table_file_deletion", "file_number": 35631} >2018-07-11 04:34:58.590022 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283698590020, "job": 2682, "event": "table_file_deletion", "file_number": 35615} >2018-07-11 04:34:58.590233 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_picker.cc:461] [default] Expanding@2 1+1(74547037+46964805 bytes) to 4+1 (298185454+46964805bytes) > >2018-07-11 04:34:58.590340 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2683] Compacting 4@2 + 1@3 files to L3, score 1.08 >2018-07-11 04:34:58.590352 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2682 Base level 2, inputs: [35637(71MB) 35638(71MB) 35639(71MB) 35640(71MB)], [35624(44MB)] > >2018-07-11 04:34:58.590365 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283698590355, "job": 2683, "event": "compaction_started", "files_L2": [35637, 35638, 35639, 35640], "files_L3": [35624], "score": 1.07683, "input_data_size": 345150259} >2018-07-11 04:35:01.724208 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2683] Generated table #35642: 2638241 keys, 74545849 bytes >2018-07-11 04:35:01.724253 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283701724232, "cf_name": "default", "job": 2683, "event": "table_file_creation", "file_number": 35642, "file_size": 74545849, "table_properties": {"data_size": 67112160, "index_size": 689516, "filter_size": 6743183, "raw_key_size": 76215862, "raw_average_key_size": 28, "raw_value_size": 31365765, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:35:04.900002 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2683] Generated table #35643: 2638240 keys, 74546999 bytes >2018-07-11 04:35:04.900064 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283704900041, "cf_name": "default", "job": 2683, "event": "table_file_creation", "file_number": 35643, "file_size": 74546999, "table_properties": {"data_size": 67112106, "index_size": 690722, "filter_size": 6743181, "raw_key_size": 76215818, "raw_average_key_size": 28, "raw_value_size": 31365738, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:35:08.011474 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2683] Generated table #35644: 2638241 keys, 74546169 bytes >2018-07-11 04:35:08.011510 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283708011495, "cf_name": "default", "job": 2683, "event": "table_file_creation", "file_number": 35644, "file_size": 74546169, "table_properties": {"data_size": 67112101, "index_size": 689895, "filter_size": 6743183, "raw_key_size": 76215850, "raw_average_key_size": 28, "raw_value_size": 31365753, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:35:11.158137 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2683] Generated table #35645: 2638241 keys, 74546344 bytes >2018-07-11 04:35:11.158171 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283711158156, "cf_name": "default", "job": 2683, "event": "table_file_creation", "file_number": 35645, "file_size": 74546344, "table_properties": {"data_size": 67112167, "index_size": 690004, "filter_size": 6743183, "raw_key_size": 76215856, "raw_average_key_size": 28, "raw_value_size": 31365759, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:35:13.021053 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2683] Generated table #35646: 1519915 keys, 46964759 bytes >2018-07-11 04:35:13.021082 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283713021068, "cf_name": "default", "job": 2683, "event": "table_file_creation", "file_number": 35646, "file_size": 46964759, "table_properties": {"data_size": 42636601, "index_size": 433022, "filter_size": 3894148, "raw_key_size": 44348908, "raw_average_key_size": 29, "raw_value_size": 21522454, "raw_average_value_size": 14, "num_data_blocks": 10538, "num_entries": 1519915, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:35:13.021114 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2683] Compacted 4@2 + 1@3 files to L3 => 345150120 bytes >2018-07-11 04:35:13.038963 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:35:13.038664) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 6 44 449 3640 716 0] max score 1.02, MB/sec: 23.9 rd, 23.9 wr, level 3, files in(4, 1) out(5) MB in(284.4, 44.8) out(329.2), read-write-amplify(2.3) write-amplify(1.2) OK, records in: 12072878, records dropped: 0 > >2018-07-11 04:35:13.038974 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:35:13.038689) EVENT_LOG_v1 {"time_micros": 1531283713038678, "job": 2683, "event": "compaction_finished", "compaction_time_micros": 14430734, "output_level": 3, "num_output_files": 5, "total_output_size": 345150120, "num_input_records": 12072878, "num_output_records": 12072878, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 6, 44, 449, 3640, 716, 0]} >2018-07-11 04:35:13.039902 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283713039898, "job": 2683, "event": "table_file_deletion", "file_number": 35640} >2018-07-11 04:35:13.039928 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283713039926, "job": 2683, "event": "table_file_deletion", "file_number": 35639} >2018-07-11 04:35:13.039939 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283713039937, "job": 2683, "event": "table_file_deletion", "file_number": 35638} >2018-07-11 04:35:13.039950 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283713039948, "job": 2683, "event": "table_file_deletion", "file_number": 35637} >2018-07-11 04:35:13.039961 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283713039959, "job": 2683, "event": "table_file_deletion", "file_number": 35624} >2018-07-11 04:35:13.040256 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2684] Compacting 1@1 + 4@2 files to L2, score 1.02 >2018-07-11 04:35:13.040268 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2683 Base level 1, inputs: [35630(71MB)], [35616(71MB) 35617(71MB) 35618(71MB) 35619(71MB)] > >2018-07-11 04:35:13.040281 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283713040271, "job": 2684, "event": "compaction_started", "files_L1": [35630], "files_L2": [35616, 35617, 35618, 35619], "score": 1.02092, "input_data_size": 372729622} >2018-07-11 04:35:16.155672 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2684] Generated table #35647: 2638240 keys, 74546437 bytes >2018-07-11 04:35:16.155718 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283716155691, "cf_name": "default", "job": 2684, "event": "table_file_creation", "file_number": 35647, "file_size": 74546437, "table_properties": {"data_size": 67112123, "index_size": 690142, "filter_size": 6743182, "raw_key_size": 76215823, "raw_average_key_size": 28, "raw_value_size": 31365743, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:35:19.198940 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2684] Generated table #35648: 2638240 keys, 74546973 bytes >2018-07-11 04:35:19.198986 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283719198958, "cf_name": "default", "job": 2684, "event": "table_file_creation", "file_number": 35648, "file_size": 74546973, "table_properties": {"data_size": 67112101, "index_size": 690701, "filter_size": 6743181, "raw_key_size": 76215825, "raw_average_key_size": 28, "raw_value_size": 31365745, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:35:22.518694 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2684] Generated table #35649: 2638240 keys, 74545944 bytes >2018-07-11 04:35:22.518727 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283722518713, "cf_name": "default", "job": 2684, "event": "table_file_creation", "file_number": 35649, "file_size": 74545944, "table_properties": {"data_size": 67112146, "index_size": 689627, "filter_size": 6743181, "raw_key_size": 76215816, "raw_average_key_size": 28, "raw_value_size": 31365736, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:35:25.617033 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2684] Generated table #35650: 2638240 keys, 74545936 bytes >2018-07-11 04:35:25.617063 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283725617049, "cf_name": "default", "job": 2684, "event": "table_file_creation", "file_number": 35650, "file_size": 74545936, "table_properties": {"data_size": 67112164, "index_size": 689601, "filter_size": 6743181, "raw_key_size": 76215823, "raw_average_key_size": 28, "raw_value_size": 31365743, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:35:28.634769 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2684] Generated table #35651: 2638240 keys, 74546411 bytes >2018-07-11 04:35:28.634819 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283728634788, "cf_name": "default", "job": 2684, "event": "table_file_creation", "file_number": 35651, "file_size": 74546411, "table_properties": {"data_size": 67112094, "index_size": 690146, "filter_size": 6743181, "raw_key_size": 76215832, "raw_average_key_size": 28, "raw_value_size": 31365752, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:35:28.655968 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2684] Generated table #35652: 1 keys, 1064 bytes >2018-07-11 04:35:28.655992 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283728655980, "cf_name": "default", "job": 2684, "event": "table_file_creation", "file_number": 35652, "file_size": 1064, "table_properties": {"data_size": 57, "index_size": 27, "filter_size": 18, "raw_key_size": 29, "raw_average_key_size": 29, "raw_value_size": 12, "raw_average_value_size": 12, "num_data_blocks": 1, "num_entries": 1, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:35:28.656029 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2684] Compacted 1@1 + 4@2 files to L2 => 372732765 bytes >2018-07-11 04:35:28.680389 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:35:28.680074) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 5 46 449 3640 716 0] max score 1.01, MB/sec: 23.9 rd, 23.9 wr, level 2, files in(1, 4) out(6) MB in(71.1, 284.4) out(355.5), read-write-amplify(10.0) write-amplify(5.0) OK, records in: 13191201, records dropped: 0 > >2018-07-11 04:35:28.680401 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:35:28.680100) EVENT_LOG_v1 {"time_micros": 1531283728680089, "job": 2684, "event": "compaction_finished", "compaction_time_micros": 15615720, "output_level": 2, "num_output_files": 6, "total_output_size": 372732765, "num_input_records": 13191201, "num_output_records": 13191201, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 5, 46, 449, 3640, 716, 0]} >2018-07-11 04:35:28.681329 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283728681325, "job": 2684, "event": "table_file_deletion", "file_number": 35630} >2018-07-11 04:35:28.681345 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283728681342, "job": 2684, "event": "table_file_deletion", "file_number": 35619} >2018-07-11 04:35:28.681356 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283728681354, "job": 2684, "event": "table_file_deletion", "file_number": 35618} >2018-07-11 04:35:28.681369 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283728681367, "job": 2684, "event": "table_file_deletion", "file_number": 35617} >2018-07-11 04:35:28.681383 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283728681380, "job": 2684, "event": "table_file_deletion", "file_number": 35616} >2018-07-11 04:35:28.705191 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:35:28.681679) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #35643 to level-4 74546999 bytes > >2018-07-11 04:35:28.705202 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:35:28.705102) EVENT_LOG_v1 {"time_micros": 1531283728705096, "job": 2685, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 74546999} >2018-07-11 04:35:28.705204 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:35:28.705110) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-4 74546999 bytes OK: base level 1 max bytes base 268435456 files[0 5 46 448 3641 716 0] max score 1.01 > >2018-07-11 04:35:28.730147 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:35:28.706041) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #35645 to level-4 74546344 bytes > >2018-07-11 04:35:28.730158 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:35:28.730059) EVENT_LOG_v1 {"time_micros": 1531283728730054, "job": 2686, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 74546344} >2018-07-11 04:35:28.730161 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:35:28.730067) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-4 74546344 bytes OK: base level 1 max bytes base 268435456 files[0 5 46 447 3642 716 0] max score 1.00 > >2018-07-11 04:35:28.755231 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:35:28.730987) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #35644 to level-4 74546169 bytes > >2018-07-11 04:35:28.755241 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:35:28.755144) EVENT_LOG_v1 {"time_micros": 1531283728755138, "job": 2687, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 74546169} >2018-07-11 04:35:28.755244 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:35:28.755151) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-4 74546169 bytes OK: base level 1 max bytes base 268435456 files[0 5 46 446 3643 716 0] max score 1.00 > >2018-07-11 04:35:28.813693 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:35:28.756104) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #35643 to level-5 74546999 bytes > >2018-07-11 04:35:28.813704 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:35:28.813520) EVENT_LOG_v1 {"time_micros": 1531283728813514, "job": 2688, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546999} >2018-07-11 04:35:28.813706 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:35:28.813527) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546999 bytes OK: base level 1 max bytes base 268435456 files[0 5 46 446 3642 717 0] max score 1.00 > >2018-07-11 04:35:28.838618 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:35:28.814598) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #15800 to level-5 74546858 bytes > >2018-07-11 04:35:28.838628 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:35:28.838527) EVENT_LOG_v1 {"time_micros": 1531283728838522, "job": 2689, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546858} >2018-07-11 04:35:28.838631 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:35:28.838535) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546858 bytes OK: base level 1 max bytes base 268435456 files[0 5 46 446 3641 718 0] max score 1.00 > >2018-07-11 04:35:28.863784 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:35:28.839469) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #31635 to level-5 74546857 bytes > >2018-07-11 04:35:28.863808 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:35:28.863579) EVENT_LOG_v1 {"time_micros": 1531283728863572, "job": 2690, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546857} >2018-07-11 04:35:28.863811 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:35:28.863587) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546857 bytes OK: base level 1 max bytes base 268435456 files[0 5 46 446 3640 719 0] max score 1.00 > >2018-07-11 04:35:28.888671 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:35:28.864692) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #35623 to level-4 74546067 bytes > >2018-07-11 04:35:28.888682 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:35:28.888584) EVENT_LOG_v1 {"time_micros": 1531283728888578, "job": 2691, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 74546067} >2018-07-11 04:35:28.888684 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:35:28.888591) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-4 74546067 bytes OK: base level 1 max bytes base 268435456 files[0 5 46 445 3641 719 0] max score 1.00 > >2018-07-11 04:35:28.905265 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:35:28.889516) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #25960 to level-5 74546857 bytes > >2018-07-11 04:35:28.905275 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:35:28.905172) EVENT_LOG_v1 {"time_micros": 1531283728905166, "job": 2692, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546857} >2018-07-11 04:35:28.905278 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:35:28.905180) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546857 bytes OK: base level 1 max bytes base 268435456 files[0 5 46 445 3640 720 0] max score 1.00 > >2018-07-11 04:35:34.505986 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:684] reusing log 35625 from recycle list > >2018-07-11 04:35:34.506094 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #35653. Immutable memtables: 0. > >2018-07-11 04:35:34.594243 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:35:34.506161) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1 >2018-07-11 04:35:34.594263 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 2693] Flushing memtable with next log file: 35653 > >2018-07-11 04:35:34.594282 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283734594273, "job": 2693, "event": "flush_started", "num_memtables": 1, "num_entries": 5000015, "num_deletes": 0, "memory_usage": 292946288} >2018-07-11 04:35:34.594285 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 2693] Level-0 flush table #35654: started >2018-07-11 04:35:38.471848 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283738471828, "cf_name": "default", "job": 2693, "event": "table_file_creation", "file_number": 35654, "file_size": 141288967, "table_properties": {"data_size": 127198839, "index_size": 1309411, "filter_size": 12779727, "raw_key_size": 144445125, "raw_average_key_size": 28, "raw_value_size": 59451570, "raw_average_value_size": 11, "num_data_blocks": 31252, "num_entries": 5000015, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:35:38.471866 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 2693] Level-0 flush table #35654: 141288967 bytes OK >2018-07-11 04:35:38.480270 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:242] adding log 35627 to recycle list > >2018-07-11 04:35:38.480282 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:35:38.471891) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #35654 started >2018-07-11 04:35:38.480299 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:35:38.480037) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #35654: memtable #1 done >2018-07-11 04:35:38.480301 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:35:38.480054) EVENT_LOG_v1 {"time_micros": 1531283738480047, "job": 2693, "event": "flush_finished", "lsm_state": [1, 5, 46, 445, 3640, 720, 0], "immutable_memtables": 0} >2018-07-11 04:35:38.480303 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:35:38.480207) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 5 46 445 3640 720 0] max score 1.00 > >2018-07-11 04:35:38.481152 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 2693] Try to delete WAL files size 178896755, prev total WAL file size 178896767, number of live WAL files 2. > >2018-07-11 04:36:18.454404 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:684] reusing log 35627 from recycle list > >2018-07-11 04:36:18.454559 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #35655. Immutable memtables: 0. > >2018-07-11 04:36:18.506646 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:36:18.454597) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1 >2018-07-11 04:36:18.506674 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 2694] Flushing memtable with next log file: 35655 > >2018-07-11 04:36:18.506691 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283778506683, "job": 2694, "event": "flush_started", "num_memtables": 1, "num_entries": 5000015, "num_deletes": 0, "memory_usage": 292938560} >2018-07-11 04:36:18.506694 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 2694] Level-0 flush table #35656: started >2018-07-11 04:36:22.767514 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283782767494, "cf_name": "default", "job": 2694, "event": "table_file_creation", "file_number": 35656, "file_size": 141286221, "table_properties": {"data_size": 127196936, "index_size": 1308576, "filter_size": 12779719, "raw_key_size": 144445077, "raw_average_key_size": 28, "raw_value_size": 59449772, "raw_average_value_size": 11, "num_data_blocks": 31252, "num_entries": 5000013, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:36:22.767533 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 2694] Level-0 flush table #35656: 141286221 bytes OK >2018-07-11 04:36:22.779168 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:242] adding log 35653 to recycle list > >2018-07-11 04:36:22.779181 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:36:22.767556) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #35656 started >2018-07-11 04:36:22.779198 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:36:22.778950) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #35656: memtable #1 done >2018-07-11 04:36:22.779200 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:36:22.778967) EVENT_LOG_v1 {"time_micros": 1531283782778960, "job": 2694, "event": "flush_finished", "lsm_state": [2, 5, 46, 445, 3640, 720, 0], "immutable_memtables": 0} >2018-07-11 04:36:22.779202 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:36:22.779105) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[2 5 46 445 3640 720 0] max score 1.05 > >2018-07-11 04:36:22.779328 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2695] Compacting 2@0 + 2@1 files to L1, score 1.05 >2018-07-11 04:36:22.779344 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2694 Base level 0, inputs: [35656(134MB) 35654(134MB)], [35629(71MB) 35635(65MB)] > >2018-07-11 04:36:22.779362 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283782779349, "job": 2695, "event": "compaction_started", "files_L0": [35656, 35654], "files_L1": [35629, 35635], "score": 1.05267, "input_data_size": 425331133} >2018-07-11 04:36:22.780068 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 2694] Try to delete WAL files size 178896755, prev total WAL file size 178896767, number of live WAL files 2. > >2018-07-11 04:36:25.830149 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2695] Generated table #35657: 2623940 keys, 74508934 bytes >2018-07-11 04:36:25.830193 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283785830166, "cf_name": "default", "job": 2695, "event": "table_file_creation", "file_number": 35657, "file_size": 74508934, "table_properties": {"data_size": 67109280, "index_size": 691226, "filter_size": 6707438, "raw_key_size": 75834140, "raw_average_key_size": 28, "raw_value_size": 31548918, "raw_average_value_size": 12, "num_data_blocks": 16475, "num_entries": 2623940, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:36:28.545052 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2695] Generated table #35658: 2638242 keys, 74546426 bytes >2018-07-11 04:36:28.545096 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283788545069, "cf_name": "default", "job": 2695, "event": "table_file_creation", "file_number": 35658, "file_size": 74546426, "table_properties": {"data_size": 67112136, "index_size": 690114, "filter_size": 6743186, "raw_key_size": 76215876, "raw_average_key_size": 28, "raw_value_size": 31365762, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638242, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:36:31.462219 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2695] Generated table #35659: 2638241 keys, 74546257 bytes >2018-07-11 04:36:31.462267 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283791462236, "cf_name": "default", "job": 2695, "event": "table_file_creation", "file_number": 35659, "file_size": 74546257, "table_properties": {"data_size": 67112146, "index_size": 689938, "filter_size": 6743183, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:36:34.076283 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2695] Generated table #35660: 2638241 keys, 74545806 bytes >2018-07-11 04:36:34.076327 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283794076300, "cf_name": "default", "job": 2695, "event": "table_file_creation", "file_number": 35660, "file_size": 74545806, "table_properties": {"data_size": 67112087, "index_size": 689546, "filter_size": 6743183, "raw_key_size": 76215853, "raw_average_key_size": 28, "raw_value_size": 31365756, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:36:36.183879 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2695] Generated table #35661: 2085356 keys, 58924120 bytes >2018-07-11 04:36:36.183918 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283796183902, "cf_name": "default", "job": 2695, "event": "table_file_creation", "file_number": 35661, "file_size": 58924120, "table_properties": {"data_size": 53047745, "index_size": 545350, "filter_size": 5330037, "raw_key_size": 60243630, "raw_average_key_size": 28, "raw_value_size": 24792578, "raw_average_value_size": 11, "num_data_blocks": 13034, "num_entries": 2085356, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:36:37.763238 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2695] Generated table #35662: 166146 keys, 68250753 bytes >2018-07-11 04:36:37.763274 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283797763260, "cf_name": "default", "job": 2695, "event": "table_file_creation", "file_number": 35662, "file_size": 68250753, "table_properties": {"data_size": 67115867, "index_size": 564945, "filter_size": 568953, "raw_key_size": 15374526, "raw_average_key_size": 92, "raw_value_size": 53118894, "raw_average_value_size": 319, "num_data_blocks": 15214, "num_entries": 166146, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "80898", "kMergeOperands": "0"}} >2018-07-11 04:36:37.763343 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2695] Compacted 2@0 + 2@1 files to L1 => 425322296 bytes >2018-07-11 04:36:37.778735 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:36:37.778535) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 9 46 445 3640 720 0] max score 1.80, MB/sec: 28.4 rd, 28.4 wr, level 1, files in(2, 2) out(6) MB in(269.5, 136.1) out(405.6), read-write-amplify(3.0) write-amplify(1.5) OK, records in: 12790174, records dropped: 8 > >2018-07-11 04:36:37.778745 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:36:37.778559) EVENT_LOG_v1 {"time_micros": 1531283797778549, "job": 2695, "event": "compaction_finished", "compaction_time_micros": 14983964, "output_level": 1, "num_output_files": 6, "total_output_size": 425322296, "num_input_records": 12790174, "num_output_records": 12790166, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 9, 46, 445, 3640, 720, 0]} >2018-07-11 04:36:37.779668 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283797779664, "job": 2695, "event": "table_file_deletion", "file_number": 35656} >2018-07-11 04:36:37.779682 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283797779680, "job": 2695, "event": "table_file_deletion", "file_number": 35654} >2018-07-11 04:36:37.779696 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283797779693, "job": 2695, "event": "table_file_deletion", "file_number": 35635} >2018-07-11 04:36:37.779709 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283797779707, "job": 2695, "event": "table_file_deletion", "file_number": 35629} >2018-07-11 04:36:37.779919 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_picker.cc:461] [default] Expanding@1 1+7(74546426+380501404 bytes) to 4+7 (282562609+380501404bytes) > >2018-07-11 04:36:37.780016 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2696] Compacting 4@1 + 7@2 files to L2, score 1.80 >2018-07-11 04:36:37.780030 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2695 Base level 1, inputs: [35658(71MB) 35659(71MB) 35660(71MB) 35661(56MB)], [35647(71MB) 35648(71MB) 35649(71MB) 35650(71MB) 35651(71MB) 35652(1064B) 35641(7586KB)] > >2018-07-11 04:36:37.780044 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283797780033, "job": 2696, "event": "compaction_started", "files_L1": [35658, 35659, 35660, 35661], "files_L2": [35647, 35648, 35649, 35650, 35651, 35652, 35641], "score": 1.79586, "input_data_size": 663064013} >2018-07-11 04:36:40.967797 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2696] Generated table #35663: 2638240 keys, 74546319 bytes >2018-07-11 04:36:40.967845 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283800967829, "cf_name": "default", "job": 2696, "event": "table_file_creation", "file_number": 35663, "file_size": 74546319, "table_properties": {"data_size": 67112102, "index_size": 690046, "filter_size": 6743181, "raw_key_size": 76215823, "raw_average_key_size": 28, "raw_value_size": 31365743, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:36:44.069411 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2696] Generated table #35664: 2638240 keys, 74546574 bytes >2018-07-11 04:36:44.069456 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283804069428, "cf_name": "default", "job": 2696, "event": "table_file_creation", "file_number": 35664, "file_size": 74546574, "table_properties": {"data_size": 67112144, "index_size": 690258, "filter_size": 6743182, "raw_key_size": 76215827, "raw_average_key_size": 28, "raw_value_size": 31365747, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:36:47.246164 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2696] Generated table #35665: 2638240 keys, 74545949 bytes >2018-07-11 04:36:47.246209 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283807246181, "cf_name": "default", "job": 2696, "event": "table_file_creation", "file_number": 35665, "file_size": 74545949, "table_properties": {"data_size": 67112145, "index_size": 689633, "filter_size": 6743181, "raw_key_size": 76215816, "raw_average_key_size": 28, "raw_value_size": 31365736, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:36:50.252278 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2696] Generated table #35666: 2638240 keys, 74545933 bytes >2018-07-11 04:36:50.252323 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283810252295, "cf_name": "default", "job": 2696, "event": "table_file_creation", "file_number": 35666, "file_size": 74545933, "table_properties": {"data_size": 67112162, "index_size": 689600, "filter_size": 6743181, "raw_key_size": 76215822, "raw_average_key_size": 28, "raw_value_size": 31365742, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:36:53.327935 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2696] Generated table #35667: 2638240 keys, 74546395 bytes >2018-07-11 04:36:53.327980 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283813327952, "cf_name": "default", "job": 2696, "event": "table_file_creation", "file_number": 35667, "file_size": 74546395, "table_properties": {"data_size": 67112093, "index_size": 690131, "filter_size": 6743181, "raw_key_size": 76215831, "raw_average_key_size": 28, "raw_value_size": 31365751, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:36:56.471602 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2696] Generated table #35668: 2638240 keys, 74546046 bytes >2018-07-11 04:36:56.471658 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283816471641, "cf_name": "default", "job": 2696, "event": "table_file_creation", "file_number": 35668, "file_size": 74546046, "table_properties": {"data_size": 67112132, "index_size": 689743, "filter_size": 6743181, "raw_key_size": 76215821, "raw_average_key_size": 28, "raw_value_size": 31365741, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:37:00.362260 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2696] Generated table #35669: 2638241 keys, 74546980 bytes >2018-07-11 04:37:00.362298 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283820362283, "cf_name": "default", "job": 2696, "event": "table_file_creation", "file_number": 35669, "file_size": 74546980, "table_properties": {"data_size": 67112092, "index_size": 690715, "filter_size": 6743183, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:37:03.853797 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2696] Generated table #35670: 2638241 keys, 74546296 bytes >2018-07-11 04:37:03.853835 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283823853819, "cf_name": "default", "job": 2696, "event": "table_file_creation", "file_number": 35670, "file_size": 74546296, "table_properties": {"data_size": 67112182, "index_size": 689941, "filter_size": 6743183, "raw_key_size": 76215857, "raw_average_key_size": 28, "raw_value_size": 31365760, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:37:07.161364 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2696] Generated table #35671: 2355824 keys, 66692422 bytes >2018-07-11 04:37:07.161414 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283827161385, "cf_name": "default", "job": 2696, "event": "table_file_creation", "file_number": 35671, "file_size": 66692422, "table_properties": {"data_size": 60052501, "index_size": 617302, "filter_size": 6021630, "raw_key_size": 68072895, "raw_average_key_size": 28, "raw_value_size": 28113509, "raw_average_value_size": 11, "num_data_blocks": 14757, "num_entries": 2355824, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "11", "kMergeOperands": "0"}} >2018-07-11 04:37:07.161456 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2696] Compacted 4@1 + 7@2 files to L2 => 663062914 bytes >2018-07-11 04:37:07.178395 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:37:07.178095) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 5 48 445 3640 720 0] max score 1.10, MB/sec: 22.6 rd, 22.6 wr, level 2, files in(4, 7) out(9) MB in(269.5, 362.9) out(632.3), read-write-amplify(4.7) write-amplify(2.3) OK, records in: 23461746, records dropped: 0 > >2018-07-11 04:37:07.178405 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:37:07.178119) EVENT_LOG_v1 {"time_micros": 1531283827178108, "job": 2696, "event": "compaction_finished", "compaction_time_micros": 29381391, "output_level": 2, "num_output_files": 9, "total_output_size": 663062914, "num_input_records": 23461746, "num_output_records": 23461746, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 5, 48, 445, 3640, 720, 0]} >2018-07-11 04:37:07.179317 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283827179314, "job": 2696, "event": "table_file_deletion", "file_number": 35661} >2018-07-11 04:37:07.179333 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283827179331, "job": 2696, "event": "table_file_deletion", "file_number": 35660} >2018-07-11 04:37:07.179347 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283827179345, "job": 2696, "event": "table_file_deletion", "file_number": 35659} >2018-07-11 04:37:07.179359 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283827179357, "job": 2696, "event": "table_file_deletion", "file_number": 35658} >2018-07-11 04:37:07.179378 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283827179375, "job": 2696, "event": "table_file_deletion", "file_number": 35652} >2018-07-11 04:37:07.179392 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283827179390, "job": 2696, "event": "table_file_deletion", "file_number": 35651} >2018-07-11 04:37:07.179407 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283827179404, "job": 2696, "event": "table_file_deletion", "file_number": 35650} >2018-07-11 04:37:07.179420 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283827179418, "job": 2696, "event": "table_file_deletion", "file_number": 35649} >2018-07-11 04:37:07.179433 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283827179431, "job": 2696, "event": "table_file_deletion", "file_number": 35648} >2018-07-11 04:37:07.179444 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283827179442, "job": 2696, "event": "table_file_deletion", "file_number": 35647} >2018-07-11 04:37:07.179457 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283827179455, "job": 2696, "event": "table_file_deletion", "file_number": 35641} >2018-07-11 04:37:07.179670 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_picker.cc:461] [default] Expanding@2 1+1(74546980+46964759 bytes) to 2+1 (149093276+46964759bytes) > >2018-07-11 04:37:07.179766 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2697] Compacting 2@2 + 1@3 files to L3, score 1.10 >2018-07-11 04:37:07.179777 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2696 Base level 2, inputs: [35669(71MB) 35670(71MB)], [35646(44MB)] > >2018-07-11 04:37:07.179790 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283827179780, "job": 2697, "event": "compaction_started", "files_L2": [35669, 35670], "files_L3": [35646], "score": 1.09878, "input_data_size": 196058035} >2018-07-11 04:37:10.796689 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2697] Generated table #35672: 2638240 keys, 74545846 bytes >2018-07-11 04:37:10.796733 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283830796706, "cf_name": "default", "job": 2697, "event": "table_file_creation", "file_number": 35672, "file_size": 74545846, "table_properties": {"data_size": 67112090, "index_size": 689585, "filter_size": 6743181, "raw_key_size": 76215829, "raw_average_key_size": 28, "raw_value_size": 31365749, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:37:14.393195 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2697] Generated table #35673: 2638241 keys, 74546312 bytes >2018-07-11 04:37:14.393228 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283834393213, "cf_name": "default", "job": 2697, "event": "table_file_creation", "file_number": 35673, "file_size": 74546312, "table_properties": {"data_size": 67112162, "index_size": 689977, "filter_size": 6743183, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:37:16.509514 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2697] Generated table #35674: 1519916 keys, 46964761 bytes >2018-07-11 04:37:16.509555 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283836509543, "cf_name": "default", "job": 2697, "event": "table_file_creation", "file_number": 35674, "file_size": 46964761, "table_properties": {"data_size": 42636613, "index_size": 433009, "filter_size": 3894151, "raw_key_size": 44348936, "raw_average_key_size": 29, "raw_value_size": 21522465, "raw_average_value_size": 14, "num_data_blocks": 10538, "num_entries": 1519916, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:37:16.509600 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2697] Compacted 2@2 + 1@3 files to L3 => 196056919 bytes >2018-07-11 04:37:16.561543 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:37:16.561244) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 5 46 447 3640 720 0] max score 1.04, MB/sec: 21.0 rd, 21.0 wr, level 3, files in(2, 1) out(3) MB in(142.2, 44.8) out(187.0), read-write-amplify(2.6) write-amplify(1.3) OK, records in: 6796397, records dropped: 0 > >2018-07-11 04:37:16.561555 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:37:16.561278) EVENT_LOG_v1 {"time_micros": 1531283836561267, "job": 2697, "event": "compaction_finished", "compaction_time_micros": 9329782, "output_level": 3, "num_output_files": 3, "total_output_size": 196056919, "num_input_records": 6796397, "num_output_records": 6796397, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 5, 46, 447, 3640, 720, 0]} >2018-07-11 04:37:16.562450 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283836562445, "job": 2697, "event": "table_file_deletion", "file_number": 35670} >2018-07-11 04:37:16.562476 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283836562472, "job": 2697, "event": "table_file_deletion", "file_number": 35669} >2018-07-11 04:37:16.562500 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283836562497, "job": 2697, "event": "table_file_deletion", "file_number": 35646} >2018-07-11 04:37:16.562797 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2698] Compacting 1@2 + 4@3 files to L3, score 1.04 >2018-07-11 04:37:16.562811 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2697 Base level 2, inputs: [35664(71MB)], [35502(23KB) 35503(71MB) 35497(71MB) 35437(71MB)] > >2018-07-11 04:37:16.562830 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283836562815, "job": 2698, "event": "compaction_started", "files_L2": [35664], "files_L3": [35502, 35503, 35497, 35437], "score": 1.04324, "input_data_size": 298207875} >2018-07-11 04:37:16.787386 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2698] Generated table #35675: 1239 keys, 36020 bytes >2018-07-11 04:37:16.787419 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283836787404, "cf_name": "default", "job": 2698, "event": "table_file_creation", "file_number": 35675, "file_size": 36020, "table_properties": {"data_size": 31562, "index_size": 311, "filter_size": 3172, "raw_key_size": 35797, "raw_average_key_size": 28, "raw_value_size": 14734, "raw_average_value_size": 11, "num_data_blocks": 8, "num_entries": 1239, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:37:20.105262 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2698] Generated table #35676: 2638241 keys, 74545944 bytes >2018-07-11 04:37:20.105307 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283840105280, "cf_name": "default", "job": 2698, "event": "table_file_creation", "file_number": 35676, "file_size": 74545944, "table_properties": {"data_size": 67112137, "index_size": 689633, "filter_size": 6743184, "raw_key_size": 76215850, "raw_average_key_size": 28, "raw_value_size": 31365753, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:37:23.082271 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2698] Generated table #35677: 2638241 keys, 74545821 bytes >2018-07-11 04:37:23.082307 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283843082291, "cf_name": "default", "job": 2698, "event": "table_file_creation", "file_number": 35677, "file_size": 74545821, "table_properties": {"data_size": 67112155, "index_size": 689493, "filter_size": 6743183, "raw_key_size": 76215851, "raw_average_key_size": 28, "raw_value_size": 31365754, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:37:26.359082 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2698] Generated table #35678: 2638240 keys, 74546880 bytes >2018-07-11 04:37:26.359136 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283846359120, "cf_name": "default", "job": 2698, "event": "table_file_creation", "file_number": 35678, "file_size": 74546880, "table_properties": {"data_size": 67112115, "index_size": 690594, "filter_size": 6743181, "raw_key_size": 76215828, "raw_average_key_size": 28, "raw_value_size": 31365748, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:37:29.344131 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2698] Generated table #35679: 2637802 keys, 74533993 bytes >2018-07-11 04:37:29.344174 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283849344155, "cf_name": "default", "job": 2698, "event": "table_file_creation", "file_number": 35679, "file_size": 74533993, "table_properties": {"data_size": 67100964, "index_size": 689985, "filter_size": 6742054, "raw_key_size": 76203173, "raw_average_key_size": 28, "raw_value_size": 31360539, "raw_average_value_size": 11, "num_data_blocks": 16487, "num_entries": 2637802, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:37:29.344218 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2698] Compacted 1@2 + 4@3 files to L3 => 298208658 bytes >2018-07-11 04:37:29.361239 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:37:29.361017) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 5 45 448 3640 720 0] max score 1.02, MB/sec: 23.3 rd, 23.3 wr, level 3, files in(1, 4) out(5) MB in(71.1, 213.3) out(284.4), read-write-amplify(8.0) write-amplify(4.0) OK, records in: 10553763, records dropped: 0 > >2018-07-11 04:37:29.361258 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:37:29.361041) EVENT_LOG_v1 {"time_micros": 1531283849361031, "job": 2698, "event": "compaction_finished", "compaction_time_micros": 12781365, "output_level": 3, "num_output_files": 5, "total_output_size": 298208658, "num_input_records": 10553763, "num_output_records": 10553763, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 5, 45, 448, 3640, 720, 0]} >2018-07-11 04:37:29.362179 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283849362161, "job": 2698, "event": "table_file_deletion", "file_number": 35664} >2018-07-11 04:37:29.362197 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283849362194, "job": 2698, "event": "table_file_deletion", "file_number": 35503} >2018-07-11 04:37:29.362209 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283849362207, "job": 2698, "event": "table_file_deletion", "file_number": 35502} >2018-07-11 04:37:29.362222 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283849362220, "job": 2698, "event": "table_file_deletion", "file_number": 35497} >2018-07-11 04:37:29.362239 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283849362237, "job": 2698, "event": "table_file_deletion", "file_number": 35437} >2018-07-11 04:37:29.362557 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2699] Compacting 1@2 + 1@3 files to L3, score 1.02 >2018-07-11 04:37:29.362568 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2698 Base level 2, inputs: [35667(71MB)], [35642(71MB)] > >2018-07-11 04:37:29.362582 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283849362572, "job": 2699, "event": "compaction_started", "files_L2": [35667], "files_L3": [35642], "score": 1.01547, "input_data_size": 149092244} >2018-07-11 04:37:32.322202 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2699] Generated table #35680: 2638241 keys, 74546096 bytes >2018-07-11 04:37:32.322260 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283852322236, "cf_name": "default", "job": 2699, "event": "table_file_creation", "file_number": 35680, "file_size": 74546096, "table_properties": {"data_size": 67112161, "index_size": 689761, "filter_size": 6743184, "raw_key_size": 76215864, "raw_average_key_size": 28, "raw_value_size": 31365767, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:37:35.166258 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2699] Generated table #35681: 2638240 keys, 74546394 bytes >2018-07-11 04:37:35.166308 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283855166292, "cf_name": "default", "job": 2699, "event": "table_file_creation", "file_number": 35681, "file_size": 74546394, "table_properties": {"data_size": 67112092, "index_size": 690131, "filter_size": 6743181, "raw_key_size": 76215829, "raw_average_key_size": 28, "raw_value_size": 31365749, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:37:35.166343 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2699] Compacted 1@2 + 1@3 files to L3 => 149092490 bytes >2018-07-11 04:37:35.177956 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:37:35.177691) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 5 44 449 3640 720 0] max score 1.01, MB/sec: 25.7 rd, 25.7 wr, level 3, files in(1, 1) out(2) MB in(71.1, 71.1) out(142.2), read-write-amplify(4.0) write-amplify(2.0) OK, records in: 5276481, records dropped: 0 > >2018-07-11 04:37:35.177967 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:37:35.177716) EVENT_LOG_v1 {"time_micros": 1531283855177705, "job": 2699, "event": "compaction_finished", "compaction_time_micros": 5803747, "output_level": 3, "num_output_files": 2, "total_output_size": 149092490, "num_input_records": 5276481, "num_output_records": 5276481, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 5, 44, 449, 3640, 720, 0]} >2018-07-11 04:37:35.178910 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283855178906, "job": 2699, "event": "table_file_deletion", "file_number": 35667} >2018-07-11 04:37:35.178938 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283855178935, "job": 2699, "event": "table_file_deletion", "file_number": 35642} >2018-07-11 04:37:35.179234 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2700] Compacting 1@3 + 1@4 files to L4, score 1.01 >2018-07-11 04:37:35.179251 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2699 Base level 3, inputs: [35678(71MB)], [35436(71MB)] > >2018-07-11 04:37:35.179264 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283855179254, "job": 2700, "event": "compaction_started", "files_L3": [35678], "files_L4": [35436], "score": 1.00846, "input_data_size": 149093156} >2018-07-11 04:37:38.259951 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2700] Generated table #35682: 2638241 keys, 74546980 bytes >2018-07-11 04:37:38.259986 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283858259971, "cf_name": "default", "job": 2700, "event": "table_file_creation", "file_number": 35682, "file_size": 74546980, "table_properties": {"data_size": 67112123, "index_size": 690684, "filter_size": 6743183, "raw_key_size": 76215855, "raw_average_key_size": 28, "raw_value_size": 31365758, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:37:41.328595 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2700] Generated table #35683: 2638240 keys, 74546409 bytes >2018-07-11 04:37:41.328645 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283861328629, "cf_name": "default", "job": 2700, "event": "table_file_creation", "file_number": 35683, "file_size": 74546409, "table_properties": {"data_size": 67112095, "index_size": 690154, "filter_size": 6743170, "raw_key_size": 76215828, "raw_average_key_size": 28, "raw_value_size": 31365748, "raw_average_value_size": 11, "num_data_blocks": 16489, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:37:41.328680 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2700] Compacted 1@3 + 1@4 files to L4 => 149093389 bytes >2018-07-11 04:37:41.344363 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:37:41.344187) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 5 44 448 3641 720 0] max score 1.01, MB/sec: 24.2 rd, 24.2 wr, level 4, files in(1, 1) out(2) MB in(71.1, 71.1) out(142.2), read-write-amplify(4.0) write-amplify(2.0) OK, records in: 5276481, records dropped: 0 > >2018-07-11 04:37:41.344374 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:37:41.344210) EVENT_LOG_v1 {"time_micros": 1531283861344200, "job": 2700, "event": "compaction_finished", "compaction_time_micros": 6149404, "output_level": 4, "num_output_files": 2, "total_output_size": 149093389, "num_input_records": 5276481, "num_output_records": 5276481, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 5, 44, 448, 3641, 720, 0]} >2018-07-11 04:37:41.345352 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283861345348, "job": 2700, "event": "table_file_deletion", "file_number": 35678} >2018-07-11 04:37:41.345368 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283861345366, "job": 2700, "event": "table_file_deletion", "file_number": 35436} >2018-07-11 04:37:41.361038 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:37:41.345680) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #35681 to level-4 74546394 bytes > >2018-07-11 04:37:41.361049 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:37:41.360949) EVENT_LOG_v1 {"time_micros": 1531283861360943, "job": 2701, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 74546394} >2018-07-11 04:37:41.361051 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:37:41.360956) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-4 74546394 bytes OK: base level 1 max bytes base 268435456 files[0 5 44 447 3642 720 0] max score 1.00 > >2018-07-11 04:37:41.377801 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:37:41.361884) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #35673 to level-4 74546312 bytes > >2018-07-11 04:37:41.377811 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:37:41.377712) EVENT_LOG_v1 {"time_micros": 1531283861377707, "job": 2702, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 74546312} >2018-07-11 04:37:41.377814 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:37:41.377720) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-4 74546312 bytes OK: base level 1 max bytes base 268435456 files[0 5 44 446 3643 720 0] max score 1.00 > >2018-07-11 04:37:41.394344 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:37:41.378684) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #35682 to level-5 74546980 bytes > >2018-07-11 04:37:41.394353 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:37:41.394259) EVENT_LOG_v1 {"time_micros": 1531283861394253, "job": 2703, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546980} >2018-07-11 04:37:41.394355 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:37:41.394266) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546980 bytes OK: base level 1 max bytes base 268435456 files[0 5 44 446 3642 721 0] max score 1.00 > >2018-07-11 04:37:41.411194 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:37:41.395155) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #18610 to level-5 74546857 bytes > >2018-07-11 04:37:41.411205 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:37:41.410981) EVENT_LOG_v1 {"time_micros": 1531283861410974, "job": 2704, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546857} >2018-07-11 04:37:41.411208 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:37:41.410989) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546857 bytes OK: base level 1 max bytes base 268435456 files[0 5 44 446 3641 722 0] max score 1.00 > >2018-07-11 04:37:41.427866 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:37:41.412075) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #11953 to level-5 74546856 bytes > >2018-07-11 04:37:41.427878 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:37:41.427665) EVENT_LOG_v1 {"time_micros": 1531283861427659, "job": 2705, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546856} >2018-07-11 04:37:41.427880 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:37:41.427673) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546856 bytes OK: base level 1 max bytes base 268435456 files[0 5 44 446 3640 723 0] max score 1.00 > >2018-07-11 04:37:41.428763 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2706] Compacting 1@3 + 3@4 files to L4, score 1.00 >2018-07-11 04:37:41.428776 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2705 Base level 3, inputs: [35680(71MB)], [35621(71MB) 35622(71MB) 35623(71MB)] > >2018-07-11 04:37:41.428790 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283861428779, "job": 2706, "event": "compaction_started", "files_L3": [35680], "files_L4": [35621, 35622, 35623], "score": 1.00013, "input_data_size": 298184907} >2018-07-11 04:37:44.624746 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2706] Generated table #35684: 2638240 keys, 74546693 bytes >2018-07-11 04:37:44.624792 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283864624764, "cf_name": "default", "job": 2706, "event": "table_file_creation", "file_number": 35684, "file_size": 74546693, "table_properties": {"data_size": 67112124, "index_size": 690398, "filter_size": 6743181, "raw_key_size": 76215832, "raw_average_key_size": 28, "raw_value_size": 31365752, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:37:47.927697 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2706] Generated table #35685: 2638241 keys, 74546285 bytes >2018-07-11 04:37:47.927743 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283867927715, "cf_name": "default", "job": 2706, "event": "table_file_creation", "file_number": 35685, "file_size": 74546285, "table_properties": {"data_size": 67112149, "index_size": 689963, "filter_size": 6743183, "raw_key_size": 76215855, "raw_average_key_size": 28, "raw_value_size": 31365758, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:37:51.162389 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2706] Generated table #35686: 2638242 keys, 74546071 bytes >2018-07-11 04:37:51.162432 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283871162416, "cf_name": "default", "job": 2706, "event": "table_file_creation", "file_number": 35686, "file_size": 74546071, "table_properties": {"data_size": 67112160, "index_size": 689735, "filter_size": 6743186, "raw_key_size": 76215874, "raw_average_key_size": 28, "raw_value_size": 31365760, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638242, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:37:51.409001 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:684] reusing log 35653 from recycle list > >2018-07-11 04:37:51.409140 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #35688. Immutable memtables: 0. > >2018-07-11 04:37:51.490302 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:37:51.409199) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 0 >2018-07-11 04:37:51.490315 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 2707] Flushing memtable with next log file: 35688 > >2018-07-11 04:37:51.490332 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283871490323, "job": 2707, "event": "flush_started", "num_memtables": 1, "num_entries": 5000015, "num_deletes": 0, "memory_usage": 292951112} >2018-07-11 04:37:51.490335 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 2707] Level-0 flush table #35689: started >2018-07-11 04:37:54.249929 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2706] Generated table #35687: 2638240 keys, 74546984 bytes >2018-07-11 04:37:54.249983 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283874249954, "cf_name": "default", "job": 2706, "event": "table_file_creation", "file_number": 35687, "file_size": 74546984, "table_properties": {"data_size": 67112114, "index_size": 690710, "filter_size": 6743170, "raw_key_size": 76215835, "raw_average_key_size": 28, "raw_value_size": 31365755, "raw_average_value_size": 11, "num_data_blocks": 16489, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:37:54.697729 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2706] Compacted 1@3 + 3@4 files to L4 => 298186033 bytes >2018-07-11 04:37:55.394602 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:37:55.394437) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 5 44 445 3641 723 0] max score 1.00, MB/sec: 22.5 rd, 22.5 wr, level 4, files in(1, 3) out(4) MB in(71.1, 213.3) out(284.4), read-write-amplify(8.0) write-amplify(4.0) OK, records in: 10552963, records dropped: 0 > >2018-07-11 04:37:55.394612 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:37:55.394465) EVENT_LOG_v1 {"time_micros": 1531283875394453, "job": 2706, "event": "compaction_finished", "compaction_time_micros": 13268917, "output_level": 4, "num_output_files": 4, "total_output_size": 298186033, "num_input_records": 10552963, "num_output_records": 10552963, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 5, 44, 445, 3641, 723, 0]} >2018-07-11 04:37:55.541212 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283875541191, "cf_name": "default", "job": 2707, "event": "table_file_creation", "file_number": 35689, "file_size": 141286221, "table_properties": {"data_size": 127196936, "index_size": 1308576, "filter_size": 12779719, "raw_key_size": 144445077, "raw_average_key_size": 28, "raw_value_size": 59449772, "raw_average_value_size": 11, "num_data_blocks": 31252, "num_entries": 5000013, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:37:55.541232 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 2707] Level-0 flush table #35689: 141286221 bytes OK >2018-07-11 04:37:55.544297 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:37:55.396070) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #35687 to level-5 74546984 bytes > >2018-07-11 04:37:55.544305 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:37:55.544166) EVENT_LOG_v1 {"time_micros": 1531283875544160, "job": 2708, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546984} >2018-07-11 04:37:55.544307 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:37:55.544178) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546984 bytes OK: base level 1 max bytes base 268435456 files[0 5 44 445 3640 724 0] max score 1.00 > >2018-07-11 04:37:55.545262 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283875545258, "job": 2708, "event": "table_file_deletion", "file_number": 35680} >2018-07-11 04:37:55.545278 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283875545276, "job": 2708, "event": "table_file_deletion", "file_number": 35623} >2018-07-11 04:37:55.545291 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283875545289, "job": 2708, "event": "table_file_deletion", "file_number": 35622} >2018-07-11 04:37:55.545303 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283875545301, "job": 2708, "event": "table_file_deletion", "file_number": 35621} >2018-07-11 04:37:55.561082 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:242] adding log 35655 to recycle list > >2018-07-11 04:37:55.561093 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:37:55.541420) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #35689 started >2018-07-11 04:37:55.561109 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:37:55.560868) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #35689: memtable #1 done >2018-07-11 04:37:55.561112 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:37:55.560884) EVENT_LOG_v1 {"time_micros": 1531283875560877, "job": 2707, "event": "flush_finished", "lsm_state": [1, 5, 44, 445, 3640, 724, 0], "immutable_memtables": 0} >2018-07-11 04:37:55.561114 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:37:55.561033) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 5 44 445 3640 724 0] max score 1.00 > >2018-07-11 04:37:55.561899 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 2707] Try to delete WAL files size 178896755, prev total WAL file size 178896767, number of live WAL files 2. > >2018-07-11 04:39:57.813393 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:684] reusing log 35655 from recycle list > >2018-07-11 04:39:57.813471 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #35690. Immutable memtables: 0. > >2018-07-11 04:39:57.867362 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:39:57.813506) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1 >2018-07-11 04:39:57.867375 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 2709] Flushing memtable with next log file: 35690 > >2018-07-11 04:39:57.867392 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531283997867383, "job": 2709, "event": "flush_started", "num_memtables": 1, "num_entries": 5000015, "num_deletes": 0, "memory_usage": 292934080} >2018-07-11 04:39:57.867395 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 2709] Level-0 flush table #35691: started >2018-07-11 04:40:00.510911 7f41649ec700 1 bluefs _allocate failed to allocate 0x8700000 on bdev 1, free 0x14fe000; fallback to bdev 2 >2018-07-11 04:40:02.341284 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284002341259, "cf_name": "default", "job": 2709, "event": "table_file_creation", "file_number": 35691, "file_size": 141288967, "table_properties": {"data_size": 127198839, "index_size": 1309411, "filter_size": 12779727, "raw_key_size": 144445125, "raw_average_key_size": 28, "raw_value_size": 59451570, "raw_average_value_size": 11, "num_data_blocks": 31252, "num_entries": 5000015, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:40:02.341307 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 2709] Level-0 flush table #35691: 141288967 bytes OK >2018-07-11 04:40:02.357725 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:242] adding log 35688 to recycle list > >2018-07-11 04:40:02.357736 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:40:02.341331) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #35691 started >2018-07-11 04:40:02.357753 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:40:02.357504) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #35691: memtable #1 done >2018-07-11 04:40:02.357755 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:40:02.357522) EVENT_LOG_v1 {"time_micros": 1531284002357515, "job": 2709, "event": "flush_finished", "lsm_state": [2, 5, 44, 445, 3640, 724, 0], "immutable_memtables": 0} >2018-07-11 04:40:02.357757 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:40:02.357660) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[2 5 44 445 3640 724 0] max score 1.05 > >2018-07-11 04:40:02.357880 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2710] Compacting 2@0 + 2@1 files to L1, score 1.05 >2018-07-11 04:40:02.357898 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2709 Base level 0, inputs: [35691(134MB) 35689(134MB)], [35657(71MB) 35662(65MB)] > >2018-07-11 04:40:02.357916 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284002357902, "job": 2710, "event": "compaction_started", "files_L0": [35691, 35689], "files_L1": [35657, 35662], "score": 1.05267, "input_data_size": 425334875} >2018-07-11 04:40:02.358629 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 2709] Try to delete WAL files size 178896755, prev total WAL file size 178896767, number of live WAL files 2. > >2018-07-11 04:40:04.826154 7f41641eb700 1 bluefs _allocate failed to allocate 0x4800000 on bdev 1, free 0x14fe000; fallback to bdev 2 >2018-07-11 04:40:05.842012 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2710] Generated table #35692: 2623910 keys, 74509808 bytes >2018-07-11 04:40:05.842056 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284005842030, "cf_name": "default", "job": 2710, "event": "table_file_creation", "file_number": 35692, "file_size": 74509808, "table_properties": {"data_size": 67110110, "index_size": 691344, "filter_size": 6707364, "raw_key_size": 75833486, "raw_average_key_size": 28, "raw_value_size": 31550064, "raw_average_value_size": 12, "num_data_blocks": 16475, "num_entries": 2623910, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:40:07.940614 7f41641eb700 1 bluefs _allocate failed to allocate 0x4800000 on bdev 1, free 0x14fe000; fallback to bdev 2 >2018-07-11 04:40:08.900379 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2710] Generated table #35693: 2638241 keys, 74546890 bytes >2018-07-11 04:40:08.900414 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284008900399, "cf_name": "default", "job": 2710, "event": "table_file_creation", "file_number": 35693, "file_size": 74546890, "table_properties": {"data_size": 67112119, "index_size": 690598, "filter_size": 6743183, "raw_key_size": 76215845, "raw_average_key_size": 28, "raw_value_size": 31365748, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:40:11.157218 7f41641eb700 1 bluefs _allocate failed to allocate 0x4800000 on bdev 1, free 0x14fe000; fallback to bdev 2 >2018-07-11 04:40:12.264327 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2710] Generated table #35694: 2638241 keys, 74546491 bytes >2018-07-11 04:40:12.264360 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284012264344, "cf_name": "default", "job": 2710, "event": "table_file_creation", "file_number": 35694, "file_size": 74546491, "table_properties": {"data_size": 67112154, "index_size": 690164, "filter_size": 6743183, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:40:14.330553 7f41641eb700 1 bluefs _allocate failed to allocate 0x4800000 on bdev 1, free 0x14fe000; fallback to bdev 2 >2018-07-11 04:40:15.378361 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2710] Generated table #35695: 2638241 keys, 74546381 bytes >2018-07-11 04:40:15.378404 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284015378387, "cf_name": "default", "job": 2710, "event": "table_file_creation", "file_number": 35695, "file_size": 74546381, "table_properties": {"data_size": 67112084, "index_size": 690124, "filter_size": 6743183, "raw_key_size": 76215850, "raw_average_key_size": 28, "raw_value_size": 31365753, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:40:17.035479 7f41641eb700 1 bluefs _allocate failed to allocate 0x3900000 on bdev 1, free 0x14fe000; fallback to bdev 2 >2018-07-11 04:40:18.040467 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2710] Generated table #35696: 2085317 keys, 58922727 bytes >2018-07-11 04:40:18.040511 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284018040485, "cf_name": "default", "job": 2710, "event": "table_file_creation", "file_number": 35696, "file_size": 58922727, "table_properties": {"data_size": 53046767, "index_size": 545032, "filter_size": 5329940, "raw_key_size": 60242505, "raw_average_key_size": 28, "raw_value_size": 24792116, "raw_average_value_size": 11, "num_data_blocks": 13034, "num_entries": 2085317, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:40:18.796826 7f41641eb700 1 bluefs _allocate failed to allocate 0x4200000 on bdev 1, free 0x14fe000; fallback to bdev 2 >2018-07-11 04:40:19.511520 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2710] Generated table #35697: 166152 keys, 68248783 bytes >2018-07-11 04:40:19.511555 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284019511539, "cf_name": "default", "job": 2710, "event": "table_file_creation", "file_number": 35697, "file_size": 68248783, "table_properties": {"data_size": 67113967, "index_size": 564868, "filter_size": 568960, "raw_key_size": 15374797, "raw_average_key_size": 92, "raw_value_size": 53116787, "raw_average_value_size": 319, "num_data_blocks": 15214, "num_entries": 166152, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "80898", "kMergeOperands": "0"}} >2018-07-11 04:40:19.532999 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2710] Generated table #35698: 4 keys, 7218 bytes >2018-07-11 04:40:19.533037 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284019533011, "cf_name": "default", "job": 2710, "event": "table_file_creation", "file_number": 35698, "file_size": 7218, "table_properties": {"data_size": 6094, "index_size": 122, "filter_size": 35, "raw_key_size": 339, "raw_average_key_size": 84, "raw_value_size": 5737, "raw_average_value_size": 1434, "num_data_blocks": 2, "num_entries": 4, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:40:19.533096 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2710] Compacted 2@0 + 2@1 files to L1 => 425328298 bytes >2018-07-11 04:40:19.549001 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:40:19.548763) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 10 44 445 3640 724 0] max score 1.80, MB/sec: 24.8 rd, 24.8 wr, level 1, files in(2, 2) out(7) MB in(269.5, 136.1) out(405.6), read-write-amplify(3.0) write-amplify(1.5) OK, records in: 12790114, records dropped: 8 > >2018-07-11 04:40:19.549012 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:40:19.548787) EVENT_LOG_v1 {"time_micros": 1531284019548776, "job": 2710, "event": "compaction_finished", "compaction_time_micros": 17175163, "output_level": 1, "num_output_files": 7, "total_output_size": 425328298, "num_input_records": 12790114, "num_output_records": 12790106, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 10, 44, 445, 3640, 724, 0]} >2018-07-11 04:40:19.549949 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284019549931, "job": 2710, "event": "table_file_deletion", "file_number": 35691} >2018-07-11 04:40:19.549964 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284019549962, "job": 2710, "event": "table_file_deletion", "file_number": 35689} >2018-07-11 04:40:19.549977 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284019549974, "job": 2710, "event": "table_file_deletion", "file_number": 35662} >2018-07-11 04:40:19.549988 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284019549986, "job": 2710, "event": "table_file_deletion", "file_number": 35657} >2018-07-11 04:40:19.550204 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_picker.cc:461] [default] Expanding@1 1+5(74546890+364876669 bytes) to 4+5 (282562489+364876669bytes) > >2018-07-11 04:40:19.550311 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2711] Compacting 4@1 + 5@2 files to L2, score 1.80 >2018-07-11 04:40:19.550325 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2710 Base level 1, inputs: [35693(71MB) 35694(71MB) 35695(71MB) 35696(56MB)], [35663(71MB) 35665(71MB) 35666(71MB) 35668(71MB) 35671(63MB)] > >2018-07-11 04:40:19.550339 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284019550328, "job": 2711, "event": "compaction_started", "files_L1": [35693, 35694, 35695, 35696], "files_L2": [35663, 35665, 35666, 35668, 35671], "score": 1.79588, "input_data_size": 647439158} >2018-07-11 04:40:22.754945 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2711] Generated table #35699: 2638238 keys, 74545728 bytes >2018-07-11 04:40:22.754981 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284022754965, "cf_name": "default", "job": 2711, "event": "table_file_creation", "file_number": 35699, "file_size": 74545728, "table_properties": {"data_size": 67112105, "index_size": 689456, "filter_size": 6743177, "raw_key_size": 76215765, "raw_average_key_size": 28, "raw_value_size": 31365719, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638238, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:40:25.948943 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2711] Generated table #35700: 2638239 keys, 74546980 bytes >2018-07-11 04:40:25.948991 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284025948963, "cf_name": "default", "job": 2711, "event": "table_file_creation", "file_number": 35700, "file_size": 74546980, "table_properties": {"data_size": 67112117, "index_size": 690694, "filter_size": 6743179, "raw_key_size": 76215785, "raw_average_key_size": 28, "raw_value_size": 31365722, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638239, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:40:28.394390 7f417420b700 1 bluestore(/var/lib/ceph/osd/ceph-10) _balance_bluefs_freespace gifting 0xcfd9e00000~80000000 to bluefs >2018-07-11 04:40:28.823604 7f417420b700 1 bluefs add_block_extent bdev 2 0xcfd9e00000~80000000 >2018-07-11 04:40:29.154080 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2711] Generated table #35701: 2638240 keys, 74547039 bytes >2018-07-11 04:40:29.154115 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284029154100, "cf_name": "default", "job": 2711, "event": "table_file_creation", "file_number": 35701, "file_size": 74547039, "table_properties": {"data_size": 67112139, "index_size": 690729, "filter_size": 6743181, "raw_key_size": 76215823, "raw_average_key_size": 28, "raw_value_size": 31365743, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:40:32.252855 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2711] Generated table #35702: 2638240 keys, 74546712 bytes >2018-07-11 04:40:32.252898 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284032252871, "cf_name": "default", "job": 2711, "event": "table_file_creation", "file_number": 35702, "file_size": 74546712, "table_properties": {"data_size": 67112140, "index_size": 690401, "filter_size": 6743181, "raw_key_size": 76215820, "raw_average_key_size": 28, "raw_value_size": 31365740, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:40:35.342268 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2711] Generated table #35703: 2638240 keys, 74546208 bytes >2018-07-11 04:40:35.342317 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284035342289, "cf_name": "default", "job": 2711, "event": "table_file_creation", "file_number": 35703, "file_size": 74546208, "table_properties": {"data_size": 67112140, "index_size": 689897, "filter_size": 6743181, "raw_key_size": 76215826, "raw_average_key_size": 28, "raw_value_size": 31365746, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:40:38.525279 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2711] Generated table #35704: 2638241 keys, 74546098 bytes >2018-07-11 04:40:38.525321 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284038525297, "cf_name": "default", "job": 2711, "event": "table_file_creation", "file_number": 35704, "file_size": 74546098, "table_properties": {"data_size": 67112091, "index_size": 689834, "filter_size": 6743183, "raw_key_size": 76215859, "raw_average_key_size": 28, "raw_value_size": 31365762, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:40:41.722633 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2711] Generated table #35705: 2638241 keys, 74547051 bytes >2018-07-11 04:40:41.722668 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284041722653, "cf_name": "default", "job": 2711, "event": "table_file_creation", "file_number": 35705, "file_size": 74547051, "table_properties": {"data_size": 67112164, "index_size": 690714, "filter_size": 6743183, "raw_key_size": 76215845, "raw_average_key_size": 28, "raw_value_size": 31365748, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:40:44.867047 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2711] Generated table #35706: 2638241 keys, 74546100 bytes >2018-07-11 04:40:44.867095 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284044867068, "cf_name": "default", "job": 2711, "event": "table_file_creation", "file_number": 35706, "file_size": 74546100, "table_properties": {"data_size": 67112120, "index_size": 689807, "filter_size": 6743183, "raw_key_size": 76215860, "raw_average_key_size": 28, "raw_value_size": 31365763, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:40:47.051285 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2711] Generated table #35707: 1802904 keys, 51069059 bytes >2018-07-11 04:40:47.051320 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284047051305, "cf_name": "default", "job": 2711, "event": "table_file_creation", "file_number": 35707, "file_size": 51069059, "table_properties": {"data_size": 45987199, "index_size": 472472, "filter_size": 4608400, "raw_key_size": 52099648, "raw_average_key_size": 28, "raw_value_size": 21539902, "raw_average_value_size": 11, "num_data_blocks": 11301, "num_entries": 1802904, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "11", "kMergeOperands": "0"}} >2018-07-11 04:40:47.051359 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2711] Compacted 4@1 + 5@2 files to L2 => 647440975 bytes >2018-07-11 04:40:47.065278 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:40:47.064964) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 6 48 445 3640 724 0] max score 1.09, MB/sec: 23.5 rd, 23.5 wr, level 2, files in(4, 5) out(9) MB in(269.5, 348.0) out(617.4), read-write-amplify(4.6) write-amplify(2.3) OK, records in: 22908824, records dropped: 0 > >2018-07-11 04:40:47.065289 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:40:47.064989) EVENT_LOG_v1 {"time_micros": 1531284047064978, "job": 2711, "event": "compaction_finished", "compaction_time_micros": 27501000, "output_level": 2, "num_output_files": 9, "total_output_size": 647440975, "num_input_records": 22908824, "num_output_records": 22908824, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 6, 48, 445, 3640, 724, 0]} >2018-07-11 04:40:47.066207 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284047066202, "job": 2711, "event": "table_file_deletion", "file_number": 35696} >2018-07-11 04:40:47.066257 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284047066253, "job": 2711, "event": "table_file_deletion", "file_number": 35695} >2018-07-11 04:40:47.066286 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284047066282, "job": 2711, "event": "table_file_deletion", "file_number": 35694} >2018-07-11 04:40:47.066303 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284047066300, "job": 2711, "event": "table_file_deletion", "file_number": 35693} >2018-07-11 04:40:47.066318 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284047066316, "job": 2711, "event": "table_file_deletion", "file_number": 35671} >2018-07-11 04:40:47.066331 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284047066328, "job": 2711, "event": "table_file_deletion", "file_number": 35668} >2018-07-11 04:40:47.066342 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284047066340, "job": 2711, "event": "table_file_deletion", "file_number": 35666} >2018-07-11 04:40:47.066355 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284047066352, "job": 2711, "event": "table_file_deletion", "file_number": 35665} >2018-07-11 04:40:47.066366 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284047066363, "job": 2711, "event": "table_file_deletion", "file_number": 35663} >2018-07-11 04:40:47.066592 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_picker.cc:461] [default] Expanding@2 1+1(74547051+46964761 bytes) to 3+1 (223639249+46964761bytes) > >2018-07-11 04:40:47.066694 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2712] Compacting 3@2 + 1@3 files to L3, score 1.09 >2018-07-11 04:40:47.066705 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2711 Base level 2, inputs: [35704(71MB) 35705(71MB) 35706(71MB)], [35674(44MB)] > >2018-07-11 04:40:47.066719 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284047066709, "job": 2712, "event": "compaction_started", "files_L2": [35704, 35705, 35706], "files_L3": [35674], "score": 1.09296, "input_data_size": 270604010} >2018-07-11 04:40:50.264559 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2712] Generated table #35708: 2638241 keys, 74546252 bytes >2018-07-11 04:40:50.264605 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284050264578, "cf_name": "default", "job": 2712, "event": "table_file_creation", "file_number": 35708, "file_size": 74546252, "table_properties": {"data_size": 67112100, "index_size": 689979, "filter_size": 6743183, "raw_key_size": 76215856, "raw_average_key_size": 28, "raw_value_size": 31365759, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:40:53.416409 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2712] Generated table #35709: 2638241 keys, 74546308 bytes >2018-07-11 04:40:53.416445 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284053416430, "cf_name": "default", "job": 2712, "event": "table_file_creation", "file_number": 35709, "file_size": 74546308, "table_properties": {"data_size": 67112161, "index_size": 689974, "filter_size": 6743183, "raw_key_size": 76215851, "raw_average_key_size": 28, "raw_value_size": 31365754, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:40:56.549395 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2712] Generated table #35710: 2638241 keys, 74546445 bytes >2018-07-11 04:40:56.549443 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284056549415, "cf_name": "default", "job": 2712, "event": "table_file_creation", "file_number": 35710, "file_size": 74546445, "table_properties": {"data_size": 67112107, "index_size": 690165, "filter_size": 6743183, "raw_key_size": 76215856, "raw_average_key_size": 28, "raw_value_size": 31365759, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:40:58.456191 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2712] Generated table #35711: 1519916 keys, 46964918 bytes >2018-07-11 04:40:58.456240 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284058456213, "cf_name": "default", "job": 2712, "event": "table_file_creation", "file_number": 35711, "file_size": 46964918, "table_properties": {"data_size": 42636650, "index_size": 433129, "filter_size": 3894151, "raw_key_size": 44348937, "raw_average_key_size": 29, "raw_value_size": 21522466, "raw_average_value_size": 14, "num_data_blocks": 10538, "num_entries": 1519916, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:40:58.456280 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2712] Compacted 3@2 + 1@3 files to L3 => 270603923 bytes >2018-07-11 04:40:58.473550 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:40:58.473236) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 6 45 448 3640 724 0] max score 1.01, MB/sec: 23.8 rd, 23.8 wr, level 3, files in(3, 1) out(4) MB in(213.3, 44.8) out(258.1), read-write-amplify(2.4) write-amplify(1.2) OK, records in: 9434639, records dropped: 0 > >2018-07-11 04:40:58.473561 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:40:58.473269) EVENT_LOG_v1 {"time_micros": 1531284058473258, "job": 2712, "event": "compaction_finished", "compaction_time_micros": 11389546, "output_level": 3, "num_output_files": 4, "total_output_size": 270603923, "num_input_records": 9434639, "num_output_records": 9434639, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 6, 45, 448, 3640, 724, 0]} >2018-07-11 04:40:58.474458 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284058474453, "job": 2712, "event": "table_file_deletion", "file_number": 35706} >2018-07-11 04:40:58.474484 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284058474480, "job": 2712, "event": "table_file_deletion", "file_number": 35705} >2018-07-11 04:40:58.474507 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284058474503, "job": 2712, "event": "table_file_deletion", "file_number": 35704} >2018-07-11 04:40:58.474530 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284058474527, "job": 2712, "event": "table_file_deletion", "file_number": 35674} >2018-07-11 04:40:58.474838 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2713] Compacting 1@2 + 2@3 files to L3, score 1.01 >2018-07-11 04:40:58.474851 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2712 Base level 2, inputs: [35701(71MB)], [35592(71MB) 35596(71MB)] > >2018-07-11 04:40:58.474871 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284058474856, "job": 2713, "event": "compaction_started", "files_L2": [35701], "files_L3": [35592, 35596], "score": 1.00965, "input_data_size": 223638977} >2018-07-11 04:41:01.649381 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2713] Generated table #35712: 2638239 keys, 74546335 bytes >2018-07-11 04:41:01.649431 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284061649402, "cf_name": "default", "job": 2713, "event": "table_file_creation", "file_number": 35712, "file_size": 74546335, "table_properties": {"data_size": 67112090, "index_size": 690076, "filter_size": 6743179, "raw_key_size": 76215814, "raw_average_key_size": 28, "raw_value_size": 31365751, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638239, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:41:04.905101 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2713] Generated table #35713: 2638240 keys, 74546187 bytes >2018-07-11 04:41:04.905139 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284064905123, "cf_name": "default", "job": 2713, "event": "table_file_creation", "file_number": 35713, "file_size": 74546187, "table_properties": {"data_size": 67112103, "index_size": 689913, "filter_size": 6743181, "raw_key_size": 76215816, "raw_average_key_size": 28, "raw_value_size": 31365736, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:41:08.101406 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2713] Generated table #35714: 2638240 keys, 74546233 bytes >2018-07-11 04:41:08.101451 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284068101424, "cf_name": "default", "job": 2713, "event": "table_file_creation", "file_number": 35714, "file_size": 74546233, "table_properties": {"data_size": 67112131, "index_size": 689931, "filter_size": 6743181, "raw_key_size": 76215822, "raw_average_key_size": 28, "raw_value_size": 31365742, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:41:08.115662 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2713] Generated table #35715: 3 keys, 1111 bytes >2018-07-11 04:41:08.115685 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284068115673, "cf_name": "default", "job": 2713, "event": "table_file_creation", "file_number": 35715, "file_size": 1111, "table_properties": {"data_size": 104, "index_size": 27, "filter_size": 18, "raw_key_size": 86, "raw_average_key_size": 28, "raw_value_size": 35, "raw_average_value_size": 11, "num_data_blocks": 1, "num_entries": 3, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:41:08.115720 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2713] Compacted 1@2 + 2@3 files to L3 => 223639866 bytes >2018-07-11 04:41:08.131732 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:41:08.131460) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 6 44 450 3640 724 0] max score 1.01, MB/sec: 23.2 rd, 23.2 wr, level 3, files in(1, 2) out(4) MB in(71.1, 142.2) out(213.3), read-write-amplify(6.0) write-amplify(3.0) OK, records in: 7914722, records dropped: 0 > >2018-07-11 04:41:08.131743 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:41:08.131485) EVENT_LOG_v1 {"time_micros": 1531284068131474, "job": 2713, "event": "compaction_finished", "compaction_time_micros": 9640820, "output_level": 3, "num_output_files": 4, "total_output_size": 223639866, "num_input_records": 7914722, "num_output_records": 7914722, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 6, 44, 450, 3640, 724, 0]} >2018-07-11 04:41:08.132724 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284068132707, "job": 2713, "event": "table_file_deletion", "file_number": 35701} >2018-07-11 04:41:08.132741 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284068132739, "job": 2713, "event": "table_file_deletion", "file_number": 35596} >2018-07-11 04:41:08.132754 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284068132752, "job": 2713, "event": "table_file_deletion", "file_number": 35592} >2018-07-11 04:41:08.148233 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:41:08.133063) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #35710 to level-4 74546445 bytes > >2018-07-11 04:41:08.148244 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:41:08.148130) EVENT_LOG_v1 {"time_micros": 1531284068148124, "job": 2714, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 74546445} >2018-07-11 04:41:08.148255 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:41:08.148137) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-4 74546445 bytes OK: base level 1 max bytes base 268435456 files[0 6 44 449 3641 724 0] max score 1.01 > >2018-07-11 04:41:08.149077 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2715] Compacting 1@3 + 2@4 files to L4, score 1.01 >2018-07-11 04:41:08.149088 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2714 Base level 3, inputs: [35712(71MB)], [35591(71MB) 35507(71MB)] > >2018-07-11 04:41:08.149113 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284068149091, "job": 2715, "event": "compaction_started", "files_L3": [35712], "files_L4": [35591, 35507], "score": 1.00568, "input_data_size": 223639379} >2018-07-11 04:41:11.415480 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2715] Generated table #35716: 2638241 keys, 74546215 bytes >2018-07-11 04:41:11.415523 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284071415497, "cf_name": "default", "job": 2715, "event": "table_file_creation", "file_number": 35716, "file_size": 74546215, "table_properties": {"data_size": 67112149, "index_size": 689893, "filter_size": 6743183, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:41:14.592146 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2715] Generated table #35717: 2638241 keys, 74545989 bytes >2018-07-11 04:41:14.592192 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284074592165, "cf_name": "default", "job": 2715, "event": "table_file_creation", "file_number": 35717, "file_size": 74545989, "table_properties": {"data_size": 67112162, "index_size": 689654, "filter_size": 6743183, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:41:17.043797 7f41821e8700 0 -- 10.8.128.107:6801/1595 >> 10.8.128.106:6801/1595 conn(0x562bd8388800 :6801 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg: challenging authorizer >2018-07-11 04:41:17.044106 7f41821e8700 0 -- 10.8.128.107:6801/1595 >> 10.8.128.106:6801/1595 conn(0x562bd8388800 :6801 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg accept connect_seq 9 vs existing csq=9 existing_state=STATE_STANDBY >2018-07-11 04:41:17.044303 7f41821e8700 0 -- 10.8.128.107:6801/1595 >> 10.8.128.106:6801/1595 conn(0x562bd8388800 :6801 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg accept connect_seq 10 vs existing csq=9 existing_state=STATE_STANDBY >2018-07-11 04:41:17.044600 7f41821e8700 0 -- 10.8.128.107:6801/1595 >> 10.8.128.106:6801/1595 conn(0x562b9f86b000 :-1 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=58 cs=9 l=0).handle_connect_msg: challenging authorizer >2018-07-11 04:41:17.762395 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2715] Generated table #35718: 2638239 keys, 74546208 bytes >2018-07-11 04:41:17.762440 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284077762413, "cf_name": "default", "job": 2715, "event": "table_file_creation", "file_number": 35718, "file_size": 74546208, "table_properties": {"data_size": 67112042, "index_size": 690008, "filter_size": 6743168, "raw_key_size": 76215807, "raw_average_key_size": 28, "raw_value_size": 31365744, "raw_average_value_size": 11, "num_data_blocks": 16489, "num_entries": 2638239, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:41:17.762472 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2715] Compacted 1@3 + 2@4 files to L4 => 223638412 bytes >2018-07-11 04:41:17.781482 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:41:17.781194) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 6 44 448 3642 724 0] max score 1.00, MB/sec: 23.3 rd, 23.3 wr, level 4, files in(1, 2) out(3) MB in(71.1, 142.2) out(213.3), read-write-amplify(6.0) write-amplify(3.0) OK, records in: 7914721, records dropped: 0 > >2018-07-11 04:41:17.781493 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:41:17.781221) EVENT_LOG_v1 {"time_micros": 1531284077781209, "job": 2715, "event": "compaction_finished", "compaction_time_micros": 9613346, "output_level": 4, "num_output_files": 3, "total_output_size": 223638412, "num_input_records": 7914721, "num_output_records": 7914721, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 6, 44, 448, 3642, 724, 0]} >2018-07-11 04:41:17.782445 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284077782439, "job": 2715, "event": "table_file_deletion", "file_number": 35712} >2018-07-11 04:41:17.782478 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284077782474, "job": 2715, "event": "table_file_deletion", "file_number": 35591} >2018-07-11 04:41:17.782507 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284077782503, "job": 2715, "event": "table_file_deletion", "file_number": 35507} >2018-07-11 04:41:17.798080 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:41:17.782816) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #35709 to level-4 74546308 bytes > >2018-07-11 04:41:17.798091 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:41:17.797881) EVENT_LOG_v1 {"time_micros": 1531284077797874, "job": 2716, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 74546308} >2018-07-11 04:41:17.798094 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:41:17.797891) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-4 74546308 bytes OK: base level 1 max bytes base 268435456 files[0 6 44 447 3643 724 0] max score 1.00 > >2018-07-11 04:41:17.814673 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:41:17.799169) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #8701 to level-5 74546856 bytes > >2018-07-11 04:41:17.814686 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:41:17.814563) EVENT_LOG_v1 {"time_micros": 1531284077814555, "job": 2717, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546856} >2018-07-11 04:41:17.814689 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:41:17.814574) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546856 bytes OK: base level 1 max bytes base 268435456 files[0 6 44 447 3642 725 0] max score 1.00 > >2018-07-11 04:41:17.831356 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:41:17.815796) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #25247 to level-5 74546856 bytes > >2018-07-11 04:41:17.831367 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:41:17.831255) EVENT_LOG_v1 {"time_micros": 1531284077831240, "job": 2718, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546856} >2018-07-11 04:41:17.831370 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:41:17.831263) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546856 bytes OK: base level 1 max bytes base 268435456 files[0 6 44 447 3641 726 0] max score 1.00 > >2018-07-11 04:41:17.848035 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:41:17.832212) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #10133 to level-5 74546856 bytes > >2018-07-11 04:41:17.848046 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:41:17.847941) EVENT_LOG_v1 {"time_micros": 1531284077847935, "job": 2719, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546856} >2018-07-11 04:41:17.848048 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:41:17.847949) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546856 bytes OK: base level 1 max bytes base 268435456 files[0 6 44 447 3640 727 0] max score 1.00 > >2018-07-11 04:41:17.864822 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:41:17.848905) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #35708 to level-4 74546252 bytes > >2018-07-11 04:41:17.864833 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:41:17.864641) EVENT_LOG_v1 {"time_micros": 1531284077864635, "job": 2720, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 74546252} >2018-07-11 04:41:17.864835 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:41:17.864651) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-4 74546252 bytes OK: base level 1 max bytes base 268435456 files[0 6 44 446 3641 727 0] max score 1.00 > >2018-07-11 04:41:17.881436 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:41:17.865945) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #26982 to level-5 74546855 bytes > >2018-07-11 04:41:17.881447 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:41:17.881336) EVENT_LOG_v1 {"time_micros": 1531284077881330, "job": 2721, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546855} >2018-07-11 04:41:17.881450 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:41:17.881345) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546855 bytes OK: base level 1 max bytes base 268435456 files[0 6 44 446 3640 728 0] max score 1.00 > >2018-07-11 04:42:08.887641 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:684] reusing log 35688 from recycle list > >2018-07-11 04:42:08.887781 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #35719. Immutable memtables: 0. > >2018-07-11 04:42:08.966562 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:42:08.887814) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1 >2018-07-11 04:42:08.966586 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 2722] Flushing memtable with next log file: 35719 > >2018-07-11 04:42:08.966602 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284128966593, "job": 2722, "event": "flush_started", "num_memtables": 1, "num_entries": 5000019, "num_deletes": 0, "memory_usage": 292923296} >2018-07-11 04:42:08.966605 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 2722] Level-0 flush table #35720: started >2018-07-11 04:42:12.835914 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284132835893, "cf_name": "default", "job": 2722, "event": "table_file_creation", "file_number": 35720, "file_size": 141289699, "table_properties": {"data_size": 127200476, "index_size": 1308484, "filter_size": 12779749, "raw_key_size": 144445225, "raw_average_key_size": 28, "raw_value_size": 59453198, "raw_average_value_size": 11, "num_data_blocks": 31253, "num_entries": 5000019, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:42:12.835939 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 2722] Level-0 flush table #35720: 141289699 bytes OK >2018-07-11 04:42:12.846864 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:242] adding log 35690 to recycle list > >2018-07-11 04:42:12.846877 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:42:12.835964) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #35720 started >2018-07-11 04:42:12.846880 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:42:12.846584) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #35720: memtable #1 done >2018-07-11 04:42:12.846882 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:42:12.846601) EVENT_LOG_v1 {"time_micros": 1531284132846594, "job": 2722, "event": "flush_finished", "lsm_state": [1, 6, 44, 446, 3640, 728, 0], "immutable_memtables": 0} >2018-07-11 04:42:12.846884 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:42:12.846799) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 6 44 446 3640 728 0] max score 1.00 > >2018-07-11 04:42:12.847787 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 2722] Try to delete WAL files size 178898489, prev total WAL file size 178898501, number of live WAL files 2. > >2018-07-11 04:44:51.489232 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:684] reusing log 35690 from recycle list > >2018-07-11 04:44:51.489343 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #35721. Immutable memtables: 0. > >2018-07-11 04:44:51.550951 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:44:51.489412) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1 >2018-07-11 04:44:51.550964 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 2723] Flushing memtable with next log file: 35721 > >2018-07-11 04:44:51.550981 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284291550972, "job": 2723, "event": "flush_started", "num_memtables": 1, "num_entries": 5000015, "num_deletes": 0, "memory_usage": 292950104} >2018-07-11 04:44:51.550985 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 2723] Level-0 flush table #35722: started >2018-07-11 04:44:55.415699 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284295415676, "cf_name": "default", "job": 2723, "event": "table_file_creation", "file_number": 35722, "file_size": 141286889, "table_properties": {"data_size": 127197777, "index_size": 1308397, "filter_size": 12779725, "raw_key_size": 144445101, "raw_average_key_size": 28, "raw_value_size": 59450671, "raw_average_value_size": 11, "num_data_blocks": 31252, "num_entries": 5000014, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:44:55.415724 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 2723] Level-0 flush table #35722: 141286889 bytes OK >2018-07-11 04:44:55.434122 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:242] adding log 35719 to recycle list > >2018-07-11 04:44:55.434134 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:44:55.415737) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #35722 started >2018-07-11 04:44:55.434138 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:44:55.433875) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #35722: memtable #1 done >2018-07-11 04:44:55.434140 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:44:55.433893) EVENT_LOG_v1 {"time_micros": 1531284295433886, "job": 2723, "event": "flush_finished", "lsm_state": [2, 6, 44, 446, 3640, 728, 0], "immutable_memtables": 0} >2018-07-11 04:44:55.434142 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:44:55.434068) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[2 6 44 446 3640 728 0] max score 1.05 > >2018-07-11 04:44:55.434555 7f41641eb700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS ------- >2018-07-11 04:44:55.434563 7f41641eb700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] >** DB Stats ** >Uptime(secs): 49037.1 total, 626.2 interval >Cumulative writes: 196K writes, 1807M keys, 196K commit groups, 1.0 writes per commit group, ingest: 60.73 GB, 1.27 MB/s >Cumulative WAL: 196K writes, 56K syncs, 3.49 writes per sync, written: 60.73 GB, 1.27 MB/s >Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent >Interval writes: 62 writes, 30M keys, 62 commit groups, 1.0 writes per commit group, ingest: 1023.66 MB, 1.63 MB/s >Interval WAL: 62 writes, 31 syncs, 1.94 writes per sync, written: 1.00 MB, 1.63 MB/s >Interval stall: 00:00:0.000 H:M:S, 0.0 percent > >** Compaction Stats [default] ** >Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop >---------------------------------------------------------------------------------------------------------------------------------------------------------- > L0 2/0 269.49 MB 1.1 0.0 0.0 0.0 47.5 47.5 0.0 1.0 0.0 32.4 1503 363 4.140 0 0 > L1 6/0 190.27 MB 0.7 78.3 47.4 30.9 78.2 47.4 0.0 1.7 25.7 25.7 3119 178 17.523 2478M 894K > L2 44/0 2.45 GB 1.0 91.7 47.4 44.3 91.7 47.3 0.0 1.9 21.5 21.5 4367 550 7.940 3420M 1358K > L3 446/0 24.93 GB 1.0 82.6 46.8 35.9 82.5 46.7 0.6 1.8 20.7 20.7 4085 283 14.435 3121M 4000K > L4 3640/0 250.00 GB 1.0 71.9 17.1 54.7 71.5 16.8 30.2 4.2 21.1 21.0 3481 229 15.200 2734M 15M > L5 728/0 50.52 GB 0.0 1.7 0.9 0.8 1.7 0.9 46.0 1.9 18.7 18.5 95 12 7.912 66M 1097K > Sum 4866/0 328.36 GB 0.0 326.2 159.6 166.7 373.2 206.6 76.9 7.8 20.1 23.0 16650 1615 10.310 11G 23M > Int 0/0 0.00 KB 0.0 5.1 2.6 2.4 5.9 3.4 1.5 7.4 20.7 23.9 251 22 11.398 185M 24 >Uptime(secs): 49037.1 total, 49037.1 interval >Flush(GB): cumulative 47.550, interval 0.790 >AddFile(GB): cumulative 0.000, interval 0.000 >AddFile(Total Files): cumulative 0, interval 0 >AddFile(L0 Files): cumulative 0, interval 0 >AddFile(Keys): cumulative 0, interval 0 >Cumulative compaction: 373.24 GB write, 7.79 MB/s write, 326.23 GB read, 6.81 MB/s read, 16650.0 seconds >Interval compaction: 5.86 GB write, 0.12 MB/s write, 5.07 GB read, 0.11 MB/s read, 250.8 seconds >Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count > >** File Read Latency Histogram By Level [default] ** > >** Compaction Stats [default] ** >Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop >---------------------------------------------------------------------------------------------------------------------------------------------------------- > L0 2/0 269.49 MB 1.1 0.0 0.0 0.0 47.5 47.5 0.0 1.0 0.0 32.4 1503 363 4.140 0 0 > L1 6/0 190.27 MB 0.7 78.3 47.4 30.9 78.2 47.4 0.0 1.7 25.7 25.7 3119 178 17.523 2478M 894K > L2 44/0 2.45 GB 1.0 91.7 47.4 44.3 91.7 47.3 0.0 1.9 21.5 21.5 4367 550 7.940 3420M 1358K > L3 446/0 24.93 GB 1.0 82.6 46.8 35.9 82.5 46.7 0.6 1.8 20.7 20.7 4085 283 14.435 3121M 4000K > L4 3640/0 250.00 GB 1.0 71.9 17.1 54.7 71.5 16.8 30.2 4.2 21.1 21.0 3481 229 15.200 2734M 15M > L5 728/0 50.52 GB 0.0 1.7 0.9 0.8 1.7 0.9 46.0 1.9 18.7 18.5 95 12 7.912 66M 1097K > Sum 4866/0 328.36 GB 0.0 326.2 159.6 166.7 373.2 206.6 76.9 7.8 20.1 23.0 16650 1615 10.310 11G 23M > Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 >Uptime(secs): 49037.1 total, 49037.1 interval >Flush(GB): cumulative 47.550, interval 0.000 >AddFile(GB): cumulative 0.000, interval 0.000 >AddFile(Total Files): cumulative 0, interval 0 >AddFile(L0 Files): cumulative 0, interval 0 >AddFile(Keys): cumulative 0, interval 0 >Cumulative compaction: 373.24 GB write, 7.79 MB/s write, 326.23 GB read, 6.81 MB/s read, 16650.0 seconds >Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds >Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count > >** File Read Latency Histogram By Level [default] ** > >2018-07-11 04:44:55.434696 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2724] Compacting 2@0 + 6@1 files to L1, score 1.05 >2018-07-11 04:44:55.434710 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2723 Base level 0, inputs: [35722(134MB) 35720(134MB)], [35692(71MB) 35697(65MB) 35698(7218B) 35636(15KB) 35486(9818B) 35462(54MB)] > >2018-07-11 04:44:55.434727 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284295434715, "job": 2724, "event": "compaction_started", "files_L0": [35722, 35720], "files_L1": [35692, 35697, 35698, 35636, 35486, 35462], "score": 1.05268, "input_data_size": 482091455} >2018-07-11 04:44:55.435039 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 2723] Try to delete WAL files size 178896755, prev total WAL file size 178896767, number of live WAL files 2. > >2018-07-11 04:44:58.381123 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2724] Generated table #35723: 2623843 keys, 74510284 bytes >2018-07-11 04:44:58.381165 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284298381139, "cf_name": "default", "job": 2724, "event": "table_file_creation", "file_number": 35723, "file_size": 74510284, "table_properties": {"data_size": 67110779, "index_size": 691319, "filter_size": 6707196, "raw_key_size": 75831749, "raw_average_key_size": 28, "raw_value_size": 31551683, "raw_average_value_size": 12, "num_data_blocks": 16475, "num_entries": 2623843, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:45:01.067244 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2724] Generated table #35724: 2638242 keys, 74546426 bytes >2018-07-11 04:45:01.067296 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284301067268, "cf_name": "default", "job": 2724, "event": "table_file_creation", "file_number": 35724, "file_size": 74546426, "table_properties": {"data_size": 67112136, "index_size": 690114, "filter_size": 6743186, "raw_key_size": 76215876, "raw_average_key_size": 28, "raw_value_size": 31365762, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638242, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:45:03.682102 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2724] Generated table #35725: 2638241 keys, 74546257 bytes >2018-07-11 04:45:03.682145 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284303682120, "cf_name": "default", "job": 2724, "event": "table_file_creation", "file_number": 35725, "file_size": 74546257, "table_properties": {"data_size": 67112146, "index_size": 689938, "filter_size": 6743183, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:45:06.338848 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2724] Generated table #35726: 2638241 keys, 74545806 bytes >2018-07-11 04:45:06.338893 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284306338866, "cf_name": "default", "job": 2724, "event": "table_file_creation", "file_number": 35726, "file_size": 74545806, "table_properties": {"data_size": 67112087, "index_size": 689546, "filter_size": 6743183, "raw_key_size": 76215853, "raw_average_key_size": 28, "raw_value_size": 31365756, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:45:08.465960 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2724] Generated table #35727: 2085356 keys, 58924120 bytes >2018-07-11 04:45:08.466003 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284308465976, "cf_name": "default", "job": 2724, "event": "table_file_creation", "file_number": 35727, "file_size": 58924120, "table_properties": {"data_size": 53047745, "index_size": 545350, "filter_size": 5330037, "raw_key_size": 60243630, "raw_average_key_size": 28, "raw_value_size": 24792578, "raw_average_value_size": 11, "num_data_blocks": 13034, "num_entries": 2085356, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:45:09.716427 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2724] Generated table #35728: 166154 keys, 68244781 bytes >2018-07-11 04:45:09.716468 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284309716441, "cf_name": "default", "job": 2724, "event": "table_file_creation", "file_number": 35728, "file_size": 68244781, "table_properties": {"data_size": 67109997, "index_size": 564835, "filter_size": 568961, "raw_key_size": 15374737, "raw_average_key_size": 92, "raw_value_size": 53112915, "raw_average_value_size": 319, "num_data_blocks": 15213, "num_entries": 166154, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "80898", "kMergeOperands": "0"}} >2018-07-11 04:45:10.749769 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2724] Generated table #35729: 138270 keys, 56761389 bytes >2018-07-11 04:45:10.749802 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284310749787, "cf_name": "default", "job": 2724, "event": "table_file_creation", "file_number": 35729, "file_size": 56761389, "table_properties": {"data_size": 55810962, "index_size": 474991, "filter_size": 474448, "raw_key_size": 6756945, "raw_average_key_size": 48, "raw_value_size": 49558789, "raw_average_value_size": 358, "num_data_blocks": 13429, "num_entries": 138270, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "58265"}} >2018-07-11 04:45:10.749878 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2724] Compacted 2@0 + 6@1 files to L1 => 482079063 bytes >2018-07-11 04:45:10.767334 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:45:10.766966) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 7 44 446 3640 728 0] max score 1.80, MB/sec: 31.5 rd, 31.5 wr, level 1, files in(2, 6) out(7) MB in(269.5, 190.3) out(459.7), read-write-amplify(3.4) write-amplify(1.7) OK, records in: 12928357, records dropped: 10 > >2018-07-11 04:45:10.767346 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:45:10.766992) EVENT_LOG_v1 {"time_micros": 1531284310766981, "job": 2724, "event": "compaction_finished", "compaction_time_micros": 15315132, "output_level": 1, "num_output_files": 7, "total_output_size": 482079063, "num_input_records": 12928357, "num_output_records": 12928347, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 7, 44, 446, 3640, 728, 0]} >2018-07-11 04:45:10.768265 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284310768261, "job": 2724, "event": "table_file_deletion", "file_number": 35722} >2018-07-11 04:45:10.768279 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284310768276, "job": 2724, "event": "table_file_deletion", "file_number": 35720} >2018-07-11 04:45:10.768290 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284310768288, "job": 2724, "event": "table_file_deletion", "file_number": 35698} >2018-07-11 04:45:10.768305 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284310768302, "job": 2724, "event": "table_file_deletion", "file_number": 35697} >2018-07-11 04:45:10.768320 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284310768318, "job": 2724, "event": "table_file_deletion", "file_number": 35692} >2018-07-11 04:45:10.768333 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284310768331, "job": 2724, "event": "table_file_deletion", "file_number": 35636} >2018-07-11 04:45:10.768345 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284310768342, "job": 2724, "event": "table_file_deletion", "file_number": 35486} >2018-07-11 04:45:10.768358 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284310768356, "job": 2724, "event": "table_file_deletion", "file_number": 35462} >2018-07-11 04:45:10.768595 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_picker.cc:461] [default] Expanding@1 1+5(74546426+349254687 bytes) to 4+5 (282562609+349254687bytes) > >2018-07-11 04:45:10.768696 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2725] Compacting 4@1 + 5@2 files to L2, score 1.80 >2018-07-11 04:45:10.768709 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2724 Base level 1, inputs: [35724(71MB) 35725(71MB) 35726(71MB) 35727(56MB)], [35699(71MB) 35700(71MB) 35702(71MB) 35703(71MB) 35707(48MB)] > >2018-07-11 04:45:10.768723 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284310768712, "job": 2725, "event": "compaction_started", "files_L1": [35724, 35725, 35726, 35727], "files_L2": [35699, 35700, 35702, 35703, 35707], "score": 1.79588, "input_data_size": 631817296} >2018-07-11 04:45:14.072242 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2725] Generated table #35730: 2638238 keys, 74545702 bytes >2018-07-11 04:45:14.072291 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284314072264, "cf_name": "default", "job": 2725, "event": "table_file_creation", "file_number": 35730, "file_size": 74545702, "table_properties": {"data_size": 67112108, "index_size": 689427, "filter_size": 6743177, "raw_key_size": 76215767, "raw_average_key_size": 28, "raw_value_size": 31365721, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638238, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:45:17.298602 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2725] Generated table #35731: 2638239 keys, 74546968 bytes >2018-07-11 04:45:17.298644 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284317298618, "cf_name": "default", "job": 2725, "event": "table_file_creation", "file_number": 35731, "file_size": 74546968, "table_properties": {"data_size": 67112118, "index_size": 690681, "filter_size": 6743179, "raw_key_size": 76215786, "raw_average_key_size": 28, "raw_value_size": 31365723, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638239, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:45:20.495819 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2725] Generated table #35732: 2638239 keys, 74546647 bytes >2018-07-11 04:45:20.495851 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284320495837, "cf_name": "default", "job": 2725, "event": "table_file_creation", "file_number": 35732, "file_size": 74546647, "table_properties": {"data_size": 67112121, "index_size": 690357, "filter_size": 6743179, "raw_key_size": 76215790, "raw_average_key_size": 28, "raw_value_size": 31365727, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638239, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:45:23.572797 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2725] Generated table #35733: 2638240 keys, 74546301 bytes >2018-07-11 04:45:23.572843 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284323572816, "cf_name": "default", "job": 2725, "event": "table_file_creation", "file_number": 35733, "file_size": 74546301, "table_properties": {"data_size": 67112123, "index_size": 690007, "filter_size": 6743181, "raw_key_size": 76215826, "raw_average_key_size": 28, "raw_value_size": 31365746, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:45:26.610612 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2725] Generated table #35734: 2638240 keys, 74547065 bytes >2018-07-11 04:45:26.610657 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284326610630, "cf_name": "default", "job": 2725, "event": "table_file_creation", "file_number": 35734, "file_size": 74547065, "table_properties": {"data_size": 67112156, "index_size": 690738, "filter_size": 6743181, "raw_key_size": 76215823, "raw_average_key_size": 28, "raw_value_size": 31365743, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:45:29.594391 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2725] Generated table #35735: 2638241 keys, 74546086 bytes >2018-07-11 04:45:29.594434 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284329594417, "cf_name": "default", "job": 2725, "event": "table_file_creation", "file_number": 35735, "file_size": 74546086, "table_properties": {"data_size": 67112123, "index_size": 689790, "filter_size": 6743183, "raw_key_size": 76215855, "raw_average_key_size": 28, "raw_value_size": 31365758, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:45:32.842860 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2725] Generated table #35736: 2638241 keys, 74546299 bytes >2018-07-11 04:45:32.842905 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284332842877, "cf_name": "default", "job": 2725, "event": "table_file_creation", "file_number": 35736, "file_size": 74546299, "table_properties": {"data_size": 67112157, "index_size": 689969, "filter_size": 6743183, "raw_key_size": 76215850, "raw_average_key_size": 28, "raw_value_size": 31365753, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:45:35.911288 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2725] Generated table #35737: 2638241 keys, 74545679 bytes >2018-07-11 04:45:35.911345 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284335911327, "cf_name": "default", "job": 2725, "event": "table_file_creation", "file_number": 35737, "file_size": 74545679, "table_properties": {"data_size": 67112126, "index_size": 689380, "filter_size": 6743183, "raw_key_size": 76215859, "raw_average_key_size": 28, "raw_value_size": 31365762, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:45:37.620648 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2725] Generated table #35738: 1250022 keys, 35447153 bytes >2018-07-11 04:45:37.620690 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284337620663, "cf_name": "default", "job": 2725, "event": "table_file_creation", "file_number": 35738, "file_size": 35447153, "table_properties": {"data_size": 31922874, "index_size": 328021, "filter_size": 3195270, "raw_key_size": 36127501, "raw_average_key_size": 28, "raw_value_size": 14966749, "raw_average_value_size": 11, "num_data_blocks": 7845, "num_entries": 1250022, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "11", "kMergeOperands": "0"}} >2018-07-11 04:45:37.620725 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2725] Compacted 4@1 + 5@2 files to L2 => 631817900 bytes >2018-07-11 04:45:37.633632 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:45:37.633298) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 3 48 446 3640 728 0] max score 1.09, MB/sec: 23.5 rd, 23.5 wr, level 2, files in(4, 5) out(9) MB in(269.5, 333.1) out(602.5), read-write-amplify(4.5) write-amplify(2.2) OK, records in: 22355941, records dropped: 0 > >2018-07-11 04:45:37.633643 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:45:37.633324) EVENT_LOG_v1 {"time_micros": 1531284337633312, "job": 2725, "event": "compaction_finished", "compaction_time_micros": 26851982, "output_level": 2, "num_output_files": 9, "total_output_size": 631817900, "num_input_records": 22355941, "num_output_records": 22355941, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 3, 48, 446, 3640, 728, 0]} >2018-07-11 04:45:37.634572 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284337634568, "job": 2725, "event": "table_file_deletion", "file_number": 35727} >2018-07-11 04:45:37.634584 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284337634582, "job": 2725, "event": "table_file_deletion", "file_number": 35726} >2018-07-11 04:45:37.634608 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284337634606, "job": 2725, "event": "table_file_deletion", "file_number": 35725} >2018-07-11 04:45:37.634618 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284337634616, "job": 2725, "event": "table_file_deletion", "file_number": 35724} >2018-07-11 04:45:37.634632 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284337634630, "job": 2725, "event": "table_file_deletion", "file_number": 35707} >2018-07-11 04:45:37.634643 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284337634641, "job": 2725, "event": "table_file_deletion", "file_number": 35703} >2018-07-11 04:45:37.634654 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284337634652, "job": 2725, "event": "table_file_deletion", "file_number": 35702} >2018-07-11 04:45:37.634665 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284337634663, "job": 2725, "event": "table_file_deletion", "file_number": 35700} >2018-07-11 04:45:37.634678 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284337634676, "job": 2725, "event": "table_file_deletion", "file_number": 35699} >2018-07-11 04:45:37.634885 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_picker.cc:461] [default] Expanding@2 1+1(74547065+46964918 bytes) to 4+1 (298185129+46964918bytes) > >2018-07-11 04:45:37.634982 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2726] Compacting 4@2 + 1@3 files to L3, score 1.09 >2018-07-11 04:45:37.634993 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2725 Base level 2, inputs: [35734(71MB) 35735(71MB) 35736(71MB) 35737(71MB)], [35711(44MB)] > >2018-07-11 04:45:37.635007 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284337634997, "job": 2726, "event": "compaction_started", "files_L2": [35734, 35735, 35736, 35737], "files_L3": [35711], "score": 1.08714, "input_data_size": 345150047} >2018-07-11 04:45:40.784506 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2726] Generated table #35739: 2638240 keys, 74546333 bytes >2018-07-11 04:45:40.784538 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284340784524, "cf_name": "default", "job": 2726, "event": "table_file_creation", "file_number": 35739, "file_size": 74546333, "table_properties": {"data_size": 67112147, "index_size": 690015, "filter_size": 6743181, "raw_key_size": 76215825, "raw_average_key_size": 28, "raw_value_size": 31365745, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:45:43.862949 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2726] Generated table #35740: 2638241 keys, 74546475 bytes >2018-07-11 04:45:43.862994 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284343862967, "cf_name": "default", "job": 2726, "event": "table_file_creation", "file_number": 35740, "file_size": 74546475, "table_properties": {"data_size": 67112113, "index_size": 690189, "filter_size": 6743183, "raw_key_size": 76215859, "raw_average_key_size": 28, "raw_value_size": 31365762, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:45:46.858421 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2726] Generated table #35741: 2638241 keys, 74545947 bytes >2018-07-11 04:45:46.858466 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284346858439, "cf_name": "default", "job": 2726, "event": "table_file_creation", "file_number": 35741, "file_size": 74545947, "table_properties": {"data_size": 67112168, "index_size": 689606, "filter_size": 6743183, "raw_key_size": 76215847, "raw_average_key_size": 28, "raw_value_size": 31365750, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:45:49.857710 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2726] Generated table #35742: 2638241 keys, 74546750 bytes >2018-07-11 04:45:49.857754 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284349857727, "cf_name": "default", "job": 2726, "event": "table_file_creation", "file_number": 35742, "file_size": 74546750, "table_properties": {"data_size": 67112149, "index_size": 690428, "filter_size": 6743183, "raw_key_size": 76215855, "raw_average_key_size": 28, "raw_value_size": 31365758, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:45:51.689541 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2726] Generated table #35743: 1519916 keys, 46965305 bytes >2018-07-11 04:45:51.689568 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284351689555, "cf_name": "default", "job": 2726, "event": "table_file_creation", "file_number": 35743, "file_size": 46965305, "table_properties": {"data_size": 42636623, "index_size": 433543, "filter_size": 3894151, "raw_key_size": 44348938, "raw_average_key_size": 29, "raw_value_size": 21522467, "raw_average_value_size": 14, "num_data_blocks": 10538, "num_entries": 1519916, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:45:51.689613 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2726] Compacted 4@2 + 1@3 files to L3 => 345150810 bytes >2018-07-11 04:45:51.708304 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:45:51.707983) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 3 44 450 3640 728 0] max score 1.01, MB/sec: 24.6 rd, 24.6 wr, level 3, files in(4, 1) out(5) MB in(284.4, 44.8) out(329.2), read-write-amplify(2.3) write-amplify(1.2) OK, records in: 12072879, records dropped: 0 > >2018-07-11 04:45:51.708316 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:45:51.708008) EVENT_LOG_v1 {"time_micros": 1531284351707997, "job": 2726, "event": "compaction_finished", "compaction_time_micros": 14054591, "output_level": 3, "num_output_files": 5, "total_output_size": 345150810, "num_input_records": 12072879, "num_output_records": 12072879, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 3, 44, 450, 3640, 728, 0]} >2018-07-11 04:45:51.709232 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284351709228, "job": 2726, "event": "table_file_deletion", "file_number": 35737} >2018-07-11 04:45:51.709266 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284351709264, "job": 2726, "event": "table_file_deletion", "file_number": 35736} >2018-07-11 04:45:51.709278 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284351709276, "job": 2726, "event": "table_file_deletion", "file_number": 35735} >2018-07-11 04:45:51.709289 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284351709287, "job": 2726, "event": "table_file_deletion", "file_number": 35734} >2018-07-11 04:45:51.709301 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284351709299, "job": 2726, "event": "table_file_deletion", "file_number": 35711} >2018-07-11 04:45:51.724805 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:45:51.709609) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #35742 to level-4 74546750 bytes > >2018-07-11 04:45:51.724816 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:45:51.724714) EVENT_LOG_v1 {"time_micros": 1531284351724708, "job": 2727, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 74546750} >2018-07-11 04:45:51.724818 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:45:51.724721) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-4 74546750 bytes OK: base level 1 max bytes base 268435456 files[0 3 44 449 3641 728 0] max score 1.01 > >2018-07-11 04:45:51.741512 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:45:51.725698) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #35740 to level-4 74546475 bytes > >2018-07-11 04:45:51.741522 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:45:51.741422) EVENT_LOG_v1 {"time_micros": 1531284351741416, "job": 2728, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 74546475} >2018-07-11 04:45:51.741525 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:45:51.741430) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-4 74546475 bytes OK: base level 1 max bytes base 268435456 files[0 3 44 448 3642 728 0] max score 1.00 > >2018-07-11 04:45:51.742363 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2729] Compacting 1@3 + 3@4 files to L4, score 1.00 >2018-07-11 04:45:51.742376 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2728 Base level 3, inputs: [35739(71MB)], [35708(71MB) 35709(71MB) 35710(71MB)] > >2018-07-11 04:45:51.742391 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284351742380, "job": 2729, "event": "compaction_started", "files_L3": [35739], "files_L4": [35708, 35709, 35710], "score": 1.00291, "input_data_size": 298185338} >2018-07-11 04:45:54.904877 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2729] Generated table #35744: 2638241 keys, 74545763 bytes >2018-07-11 04:45:54.904915 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284354904900, "cf_name": "default", "job": 2729, "event": "table_file_creation", "file_number": 35744, "file_size": 74545763, "table_properties": {"data_size": 67112132, "index_size": 689458, "filter_size": 6743183, "raw_key_size": 76215857, "raw_average_key_size": 28, "raw_value_size": 31365760, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:45:55.735485 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:684] reusing log 35719 from recycle list > >2018-07-11 04:45:55.735635 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #35746. Immutable memtables: 0. > >2018-07-11 04:45:55.810461 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:45:55.735703) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 0 >2018-07-11 04:45:55.810475 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 2730] Flushing memtable with next log file: 35746 > >2018-07-11 04:45:55.810492 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284355810483, "job": 2730, "event": "flush_started", "num_memtables": 1, "num_entries": 5000015, "num_deletes": 0, "memory_usage": 292927064} >2018-07-11 04:45:55.810494 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 2730] Level-0 flush table #35747: started >2018-07-11 04:45:58.170368 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2729] Generated table #35745: 2638241 keys, 74546990 bytes >2018-07-11 04:45:58.170423 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284358170406, "cf_name": "default", "job": 2729, "event": "table_file_creation", "file_number": 35745, "file_size": 74546990, "table_properties": {"data_size": 67112095, "index_size": 690722, "filter_size": 6743183, "raw_key_size": 76215850, "raw_average_key_size": 28, "raw_value_size": 31365753, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:45:59.817161 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284359817140, "cf_name": "default", "job": 2730, "event": "table_file_creation", "file_number": 35747, "file_size": 141286889, "table_properties": {"data_size": 127197777, "index_size": 1308397, "filter_size": 12779725, "raw_key_size": 144445101, "raw_average_key_size": 28, "raw_value_size": 59450671, "raw_average_value_size": 11, "num_data_blocks": 31252, "num_entries": 5000014, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:45:59.817185 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 2730] Level-0 flush table #35747: 141286889 bytes OK >2018-07-11 04:45:59.832978 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:242] adding log 35721 to recycle list > >2018-07-11 04:45:59.832990 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:45:59.817199) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #35747 started >2018-07-11 04:45:59.832996 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:45:59.832850) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #35747: memtable #1 done >2018-07-11 04:45:59.832999 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:45:59.832867) EVENT_LOG_v1 {"time_micros": 1531284359832860, "job": 2730, "event": "flush_finished", "lsm_state": [1, 3, 44, 448, 3642, 728, 0], "immutable_memtables": 0} >2018-07-11 04:45:59.833003 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:45:59.832885) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 3 44 448 3642 728 0] max score 1.00 > >2018-07-11 04:45:59.834116 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 2730] Try to delete WAL files size 178896755, prev total WAL file size 178896767, number of live WAL files 2. > >2018-07-11 04:46:01.355204 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2729] Generated table #35748: 2638241 keys, 74546344 bytes >2018-07-11 04:46:01.355256 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284361355232, "cf_name": "default", "job": 2729, "event": "table_file_creation", "file_number": 35748, "file_size": 74546344, "table_properties": {"data_size": 67112175, "index_size": 689996, "filter_size": 6743183, "raw_key_size": 76215856, "raw_average_key_size": 28, "raw_value_size": 31365759, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:46:04.367024 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2729] Generated table #35749: 2638240 keys, 74546334 bytes >2018-07-11 04:46:04.367071 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284364367043, "cf_name": "default", "job": 2729, "event": "table_file_creation", "file_number": 35749, "file_size": 74546334, "table_properties": {"data_size": 67112065, "index_size": 690109, "filter_size": 6743170, "raw_key_size": 76215825, "raw_average_key_size": 28, "raw_value_size": 31365745, "raw_average_value_size": 11, "num_data_blocks": 16489, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:46:04.367105 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2729] Compacted 1@3 + 3@4 files to L4 => 298185431 bytes >2018-07-11 04:46:04.383176 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:46:04.382821) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[1 3 44 447 3643 728 0] max score 1.00, MB/sec: 23.6 rd, 23.6 wr, level 4, files in(1, 3) out(4) MB in(71.1, 213.3) out(284.4), read-write-amplify(8.0) write-amplify(4.0) OK, records in: 10552963, records dropped: 0 > >2018-07-11 04:46:04.383188 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:46:04.382847) EVENT_LOG_v1 {"time_micros": 1531284364382837, "job": 2729, "event": "compaction_finished", "compaction_time_micros": 12624701, "output_level": 4, "num_output_files": 4, "total_output_size": 298185431, "num_input_records": 10552963, "num_output_records": 10552963, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 3, 44, 447, 3643, 728, 0]} >2018-07-11 04:46:04.384145 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284364384129, "job": 2729, "event": "table_file_deletion", "file_number": 35739} >2018-07-11 04:46:04.384162 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284364384159, "job": 2729, "event": "table_file_deletion", "file_number": 35710} >2018-07-11 04:46:04.384174 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284364384171, "job": 2729, "event": "table_file_deletion", "file_number": 35709} >2018-07-11 04:46:04.384185 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284364384183, "job": 2729, "event": "table_file_deletion", "file_number": 35708} >2018-07-11 04:46:04.399734 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:46:04.384491) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #35745 to level-5 74546990 bytes > >2018-07-11 04:46:04.399745 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:46:04.399643) EVENT_LOG_v1 {"time_micros": 1531284364399637, "job": 2731, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546990} >2018-07-11 04:46:04.399747 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:46:04.399651) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546990 bytes OK: base level 1 max bytes base 268435456 files[1 3 44 447 3642 729 0] max score 1.00 > >2018-07-11 04:46:04.420646 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:46:04.400624) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #8393 to level-5 74546854 bytes > >2018-07-11 04:46:04.420656 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:46:04.420557) EVENT_LOG_v1 {"time_micros": 1531284364420551, "job": 2732, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546854} >2018-07-11 04:46:04.420659 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:46:04.420565) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546854 bytes OK: base level 1 max bytes base 268435456 files[1 3 44 447 3641 730 0] max score 1.00 > >2018-07-11 04:46:04.437496 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:46:04.421521) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #23803 to level-5 74546853 bytes > >2018-07-11 04:46:04.437507 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:46:04.437407) EVENT_LOG_v1 {"time_micros": 1531284364437401, "job": 2733, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546853} >2018-07-11 04:46:04.437510 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:46:04.437415) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546853 bytes OK: base level 1 max bytes base 268435456 files[1 3 44 447 3640 731 0] max score 1.00 > >2018-07-11 04:46:04.438231 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_picker.cc:461] [default] Expanding@3 1+2(74546233+149093223 bytes) to 2+2 (74547344+149093223bytes) > >2018-07-11 04:46:04.438361 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2734] Compacting 2@3 + 2@4 files to L4, score 1.00 >2018-07-11 04:46:04.438374 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2733 Base level 3, inputs: [35714(71MB) 35715(1111B)], [35597(71MB) 35684(71MB)] > >2018-07-11 04:46:04.438388 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284364438377, "job": 2734, "event": "compaction_started", "files_L3": [35714, 35715], "files_L4": [35597, 35684], "score": 1.00013, "input_data_size": 223640567} >2018-07-11 04:46:07.580629 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2734] Generated table #35750: 2638242 keys, 74546955 bytes >2018-07-11 04:46:07.580690 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284367580671, "cf_name": "default", "job": 2734, "event": "table_file_creation", "file_number": 35750, "file_size": 74546955, "table_properties": {"data_size": 67112159, "index_size": 690620, "filter_size": 6743186, "raw_key_size": 76215884, "raw_average_key_size": 28, "raw_value_size": 31365770, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638242, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:46:11.837999 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2734] Generated table #35751: 2638241 keys, 74546362 bytes >2018-07-11 04:46:11.838045 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284371838017, "cf_name": "default", "job": 2734, "event": "table_file_creation", "file_number": 35751, "file_size": 74546362, "table_properties": {"data_size": 67112145, "index_size": 690044, "filter_size": 6743183, "raw_key_size": 76215848, "raw_average_key_size": 28, "raw_value_size": 31365751, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:46:14.924258 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2734] Generated table #35752: 2638241 keys, 74546583 bytes >2018-07-11 04:46:14.924308 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284374924280, "cf_name": "default", "job": 2734, "event": "table_file_creation", "file_number": 35752, "file_size": 74546583, "table_properties": {"data_size": 67112124, "index_size": 690286, "filter_size": 6743183, "raw_key_size": 76215862, "raw_average_key_size": 28, "raw_value_size": 31365765, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:46:14.924344 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2734] Compacted 2@3 + 2@4 files to L4 => 223639900 bytes >2018-07-11 04:46:14.945730 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:46:14.945490) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[1 3 44 445 3641 731 0] max score 1.00, MB/sec: 21.3 rd, 21.3 wr, level 4, files in(2, 2) out(3) MB in(71.1, 142.2) out(213.3), read-write-amplify(6.0) write-amplify(3.0) OK, records in: 7914724, records dropped: 0 > >2018-07-11 04:46:14.945741 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:46:14.945517) EVENT_LOG_v1 {"time_micros": 1531284374945506, "job": 2734, "event": "compaction_finished", "compaction_time_micros": 10485939, "output_level": 4, "num_output_files": 3, "total_output_size": 223639900, "num_input_records": 7914724, "num_output_records": 7914724, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 3, 44, 445, 3641, 731, 0]} >2018-07-11 04:46:14.946709 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284374946704, "job": 2734, "event": "table_file_deletion", "file_number": 35715} >2018-07-11 04:46:14.946734 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284374946731, "job": 2734, "event": "table_file_deletion", "file_number": 35714} >2018-07-11 04:46:14.946765 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284374946762, "job": 2734, "event": "table_file_deletion", "file_number": 35684} >2018-07-11 04:46:14.946792 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284374946789, "job": 2734, "event": "table_file_deletion", "file_number": 35597} >2018-07-11 04:46:14.970591 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:46:14.947085) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #35750 to level-5 74546955 bytes > >2018-07-11 04:46:14.970602 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:46:14.970497) EVENT_LOG_v1 {"time_micros": 1531284374970490, "job": 2735, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546955} >2018-07-11 04:46:14.970607 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:46:14.970505) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546955 bytes OK: base level 1 max bytes base 268435456 files[1 3 44 445 3640 732 0] max score 1.00 > >2018-07-11 04:47:53.727106 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:684] reusing log 35721 from recycle list > >2018-07-11 04:47:53.727269 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #35753. Immutable memtables: 0. > >2018-07-11 04:47:53.779221 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:47:53.727335) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1 >2018-07-11 04:47:53.779236 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 2736] Flushing memtable with next log file: 35753 > >2018-07-11 04:47:53.779265 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284473779256, "job": 2736, "event": "flush_started", "num_memtables": 1, "num_entries": 5000015, "num_deletes": 0, "memory_usage": 292954296} >2018-07-11 04:47:53.779269 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 2736] Level-0 flush table #35754: started >2018-07-11 04:47:56.445873 7f41649ec700 1 bluefs _allocate failed to allocate 0x8700000 on bdev 1, free 0x24fe000; fallback to bdev 2 >2018-07-11 04:47:58.477680 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284478477660, "cf_name": "default", "job": 2736, "event": "table_file_creation", "file_number": 35754, "file_size": 141288967, "table_properties": {"data_size": 127198839, "index_size": 1309411, "filter_size": 12779727, "raw_key_size": 144445125, "raw_average_key_size": 28, "raw_value_size": 59451570, "raw_average_value_size": 11, "num_data_blocks": 31252, "num_entries": 5000015, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:47:58.477700 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 2736] Level-0 flush table #35754: 141288967 bytes OK >2018-07-11 04:47:58.501214 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:242] adding log 35746 to recycle list > >2018-07-11 04:47:58.501227 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:47:58.477724) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #35754 started >2018-07-11 04:47:58.501231 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:47:58.500967) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #35754: memtable #1 done >2018-07-11 04:47:58.501233 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:47:58.500985) EVENT_LOG_v1 {"time_micros": 1531284478500978, "job": 2736, "event": "flush_finished", "lsm_state": [2, 3, 44, 445, 3640, 732, 0], "immutable_memtables": 0} >2018-07-11 04:47:58.501235 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:47:58.501159) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[2 3 44 445 3640 732 0] max score 1.05 > >2018-07-11 04:47:58.501377 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2737] Compacting 2@0 + 2@1 files to L1, score 1.05 >2018-07-11 04:47:58.501394 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2736 Base level 0, inputs: [35754(134MB) 35747(134MB)], [35723(71MB) 35728(65MB)] > >2018-07-11 04:47:58.501411 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284478501398, "job": 2737, "event": "compaction_started", "files_L0": [35754, 35747], "files_L1": [35723, 35728], "score": 1.05268, "input_data_size": 425330921} >2018-07-11 04:47:58.502103 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 2736] Try to delete WAL files size 178896755, prev total WAL file size 178896767, number of live WAL files 2. > >2018-07-11 04:48:01.554143 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2737] Generated table #35755: 2623733 keys, 74508945 bytes >2018-07-11 04:48:01.554189 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284481554162, "cf_name": "default", "job": 2737, "event": "table_file_creation", "file_number": 35755, "file_size": 74508945, "table_properties": {"data_size": 67109629, "index_size": 691404, "filter_size": 6706922, "raw_key_size": 75828782, "raw_average_key_size": 28, "raw_value_size": 31551876, "raw_average_value_size": 12, "num_data_blocks": 16475, "num_entries": 2623733, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:48:04.211172 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2737] Generated table #35756: 2638241 keys, 74546868 bytes >2018-07-11 04:48:04.211219 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284484211191, "cf_name": "default", "job": 2737, "event": "table_file_creation", "file_number": 35756, "file_size": 74546868, "table_properties": {"data_size": 67112119, "index_size": 690576, "filter_size": 6743183, "raw_key_size": 76215845, "raw_average_key_size": 28, "raw_value_size": 31365748, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:48:07.176456 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2737] Generated table #35757: 2638241 keys, 74546503 bytes >2018-07-11 04:48:07.176500 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284487176483, "cf_name": "default", "job": 2737, "event": "table_file_creation", "file_number": 35757, "file_size": 74546503, "table_properties": {"data_size": 67112153, "index_size": 690177, "filter_size": 6743183, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:48:09.909235 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2737] Generated table #35758: 2638241 keys, 74546405 bytes >2018-07-11 04:48:09.909284 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284489909257, "cf_name": "default", "job": 2737, "event": "table_file_creation", "file_number": 35758, "file_size": 74546405, "table_properties": {"data_size": 67112087, "index_size": 690145, "filter_size": 6743183, "raw_key_size": 76215853, "raw_average_key_size": 28, "raw_value_size": 31365756, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:48:12.015704 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2737] Generated table #35759: 2085397 keys, 58924958 bytes >2018-07-11 04:48:12.015750 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284492015722, "cf_name": "default", "job": 2737, "event": "table_file_creation", "file_number": 35759, "file_size": 58924958, "table_properties": {"data_size": 53048796, "index_size": 545034, "filter_size": 5330140, "raw_key_size": 60244815, "raw_average_key_size": 28, "raw_value_size": 24793066, "raw_average_value_size": 11, "num_data_blocks": 13034, "num_entries": 2085397, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:48:13.153655 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2737] Generated table #35760: 166160 keys, 68244918 bytes >2018-07-11 04:48:13.153697 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284493153670, "cf_name": "default", "job": 2737, "event": "table_file_creation", "file_number": 35760, "file_size": 68244918, "table_properties": {"data_size": 67110094, "index_size": 564855, "filter_size": 568981, "raw_key_size": 15375011, "raw_average_key_size": 92, "raw_value_size": 53112793, "raw_average_value_size": 319, "num_data_blocks": 15214, "num_entries": 166160, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "80898", "kMergeOperands": "0"}} >2018-07-11 04:48:13.193469 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2737] Generated table #35761: 4 keys, 5094 bytes >2018-07-11 04:48:13.193495 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284493193482, "cf_name": "default", "job": 2737, "event": "table_file_creation", "file_number": 35761, "file_size": 5094, "table_properties": {"data_size": 4079, "index_size": 28, "filter_size": 20, "raw_key_size": 336, "raw_average_key_size": 84, "raw_value_size": 3752, "raw_average_value_size": 938, "num_data_blocks": 1, "num_entries": 4, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:48:13.193552 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2737] Compacted 2@0 + 2@1 files to L1 => 425323691 bytes >2018-07-11 04:48:13.209316 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:48:13.209118) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 8 44 445 3640 732 0] max score 1.80, MB/sec: 28.9 rd, 28.9 wr, level 1, files in(2, 2) out(7) MB in(269.5, 136.1) out(405.6), read-write-amplify(3.0) write-amplify(1.5) OK, records in: 12790026, records dropped: 9 > >2018-07-11 04:48:13.209326 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:48:13.209141) EVENT_LOG_v1 {"time_micros": 1531284493209131, "job": 2737, "event": "compaction_finished", "compaction_time_micros": 14692123, "output_level": 1, "num_output_files": 7, "total_output_size": 425323691, "num_input_records": 12790026, "num_output_records": 12790017, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 8, 44, 445, 3640, 732, 0]} >2018-07-11 04:48:13.210251 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284493210242, "job": 2737, "event": "table_file_deletion", "file_number": 35754} >2018-07-11 04:48:13.210269 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284493210267, "job": 2737, "event": "table_file_deletion", "file_number": 35747} >2018-07-11 04:48:13.210280 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284493210278, "job": 2737, "event": "table_file_deletion", "file_number": 35728} >2018-07-11 04:48:13.210294 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284493210292, "job": 2737, "event": "table_file_deletion", "file_number": 35723} >2018-07-11 04:48:13.210515 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_picker.cc:461] [default] Expanding@1 1+5(74546868+333632771 bytes) to 4+5 (282564734+333632771bytes) > >2018-07-11 04:48:13.210616 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2738] Compacting 4@1 + 5@2 files to L2, score 1.80 >2018-07-11 04:48:13.210629 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2737 Base level 1, inputs: [35756(71MB) 35757(71MB) 35758(71MB) 35759(56MB)], [35730(71MB) 35731(71MB) 35732(71MB) 35733(71MB) 35738(33MB)] > >2018-07-11 04:48:13.210643 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284493210633, "job": 2738, "event": "compaction_started", "files_L1": [35756, 35757, 35758, 35759], "files_L2": [35730, 35731, 35732, 35733, 35738], "score": 1.79591, "input_data_size": 616197505} >2018-07-11 04:48:16.433705 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2738] Generated table #35762: 2638240 keys, 74546440 bytes >2018-07-11 04:48:16.433749 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284496433722, "cf_name": "default", "job": 2738, "event": "table_file_creation", "file_number": 35762, "file_size": 74546440, "table_properties": {"data_size": 67112127, "index_size": 690141, "filter_size": 6743182, "raw_key_size": 76215827, "raw_average_key_size": 28, "raw_value_size": 31365747, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:48:19.496367 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2738] Generated table #35763: 2638240 keys, 74545939 bytes >2018-07-11 04:48:19.496411 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284499496385, "cf_name": "default", "job": 2738, "event": "table_file_creation", "file_number": 35763, "file_size": 74545939, "table_properties": {"data_size": 67112141, "index_size": 689627, "filter_size": 6743181, "raw_key_size": 76215812, "raw_average_key_size": 28, "raw_value_size": 31365732, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:48:22.587627 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2738] Generated table #35764: 2638239 keys, 74545974 bytes >2018-07-11 04:48:22.587672 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284502587645, "cf_name": "default", "job": 2738, "event": "table_file_creation", "file_number": 35764, "file_size": 74545974, "table_properties": {"data_size": 67112075, "index_size": 689730, "filter_size": 6743179, "raw_key_size": 76215791, "raw_average_key_size": 28, "raw_value_size": 31365728, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638239, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:48:25.686921 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2738] Generated table #35765: 2638240 keys, 74546955 bytes >2018-07-11 04:48:25.686956 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284505686940, "cf_name": "default", "job": 2738, "event": "table_file_creation", "file_number": 35765, "file_size": 74546955, "table_properties": {"data_size": 67112087, "index_size": 690697, "filter_size": 6743181, "raw_key_size": 76215825, "raw_average_key_size": 28, "raw_value_size": 31365745, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:48:28.765707 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2738] Generated table #35766: 2638240 keys, 74546195 bytes >2018-07-11 04:48:28.765752 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284508765726, "cf_name": "default", "job": 2738, "event": "table_file_creation", "file_number": 35766, "file_size": 74546195, "table_properties": {"data_size": 67112140, "index_size": 689884, "filter_size": 6743181, "raw_key_size": 76215826, "raw_average_key_size": 28, "raw_value_size": 31365746, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:48:31.866129 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2738] Generated table #35767: 2638241 keys, 74545880 bytes >2018-07-11 04:48:31.866174 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284511866146, "cf_name": "default", "job": 2738, "event": "table_file_creation", "file_number": 35767, "file_size": 74545880, "table_properties": {"data_size": 67112085, "index_size": 689622, "filter_size": 6743183, "raw_key_size": 76215848, "raw_average_key_size": 28, "raw_value_size": 31365751, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:48:35.059023 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2738] Generated table #35768: 2638241 keys, 74546173 bytes >2018-07-11 04:48:35.059068 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284515059040, "cf_name": "default", "job": 2738, "event": "table_file_creation", "file_number": 35768, "file_size": 74546173, "table_properties": {"data_size": 67112184, "index_size": 689816, "filter_size": 6743183, "raw_key_size": 76215864, "raw_average_key_size": 28, "raw_value_size": 31365767, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:48:38.130878 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2738] Generated table #35769: 2638241 keys, 74546954 bytes >2018-07-11 04:48:38.130923 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284518130895, "cf_name": "default", "job": 2738, "event": "table_file_creation", "file_number": 35769, "file_size": 74546954, "table_properties": {"data_size": 67112112, "index_size": 690669, "filter_size": 6743183, "raw_key_size": 76215850, "raw_average_key_size": 28, "raw_value_size": 31365753, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:48:38.983278 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2738] Generated table #35770: 697176 keys, 19825723 bytes >2018-07-11 04:48:38.983314 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284518983294, "cf_name": "default", "job": 2738, "event": "table_file_creation", "file_number": 35770, "file_size": 19825723, "table_properties": {"data_size": 17859473, "index_size": 183032, "filter_size": 1782232, "raw_key_size": 20156394, "raw_average_key_size": 28, "raw_value_size": 8394024, "raw_average_value_size": 12, "num_data_blocks": 4390, "num_entries": 697176, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "11", "kMergeOperands": "0"}} >2018-07-11 04:48:38.983354 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2738] Compacted 4@1 + 5@2 files to L2 => 616196233 bytes >2018-07-11 04:48:38.992374 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:48:38.992060) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 4 48 445 3640 732 0] max score 1.08, MB/sec: 23.9 rd, 23.9 wr, level 2, files in(4, 5) out(9) MB in(269.5, 318.2) out(587.7), read-write-amplify(4.4) write-amplify(2.2) OK, records in: 21803098, records dropped: 0 > >2018-07-11 04:48:38.992386 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:48:38.992086) EVENT_LOG_v1 {"time_micros": 1531284518992074, "job": 2738, "event": "compaction_finished", "compaction_time_micros": 25772690, "output_level": 2, "num_output_files": 9, "total_output_size": 616196233, "num_input_records": 21803098, "num_output_records": 21803098, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 4, 48, 445, 3640, 732, 0]} >2018-07-11 04:48:38.993298 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284518993294, "job": 2738, "event": "table_file_deletion", "file_number": 35759} >2018-07-11 04:48:38.993314 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284518993312, "job": 2738, "event": "table_file_deletion", "file_number": 35758} >2018-07-11 04:48:38.993327 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284518993324, "job": 2738, "event": "table_file_deletion", "file_number": 35757} >2018-07-11 04:48:38.993339 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284518993337, "job": 2738, "event": "table_file_deletion", "file_number": 35756} >2018-07-11 04:48:38.993356 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284518993353, "job": 2738, "event": "table_file_deletion", "file_number": 35738} >2018-07-11 04:48:38.993369 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284518993367, "job": 2738, "event": "table_file_deletion", "file_number": 35733} >2018-07-11 04:48:38.993383 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284518993381, "job": 2738, "event": "table_file_deletion", "file_number": 35732} >2018-07-11 04:48:38.993396 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284518993394, "job": 2738, "event": "table_file_deletion", "file_number": 35731} >2018-07-11 04:48:38.993409 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284518993407, "job": 2738, "event": "table_file_deletion", "file_number": 35730} >2018-07-11 04:48:38.993714 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2739] Compacting 1@2 + 1@3 files to L3, score 1.08 >2018-07-11 04:48:38.993725 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2738 Base level 2, inputs: [35765(71MB)], [35672(71MB)] > >2018-07-11 04:48:38.993737 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284518993728, "job": 2739, "event": "compaction_started", "files_L2": [35765], "files_L3": [35672], "score": 1.08132, "input_data_size": 149092801} >2018-07-11 04:48:42.178982 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2739] Generated table #35771: 2638240 keys, 74546288 bytes >2018-07-11 04:48:42.179027 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284522179000, "cf_name": "default", "job": 2739, "event": "table_file_creation", "file_number": 35771, "file_size": 74546288, "table_properties": {"data_size": 67112121, "index_size": 689996, "filter_size": 6743181, "raw_key_size": 76215831, "raw_average_key_size": 28, "raw_value_size": 31365751, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:48:45.176871 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2739] Generated table #35772: 2638240 keys, 74546953 bytes >2018-07-11 04:48:45.176906 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284525176891, "cf_name": "default", "job": 2739, "event": "table_file_creation", "file_number": 35772, "file_size": 74546953, "table_properties": {"data_size": 67112085, "index_size": 690697, "filter_size": 6743181, "raw_key_size": 76215823, "raw_average_key_size": 28, "raw_value_size": 31365743, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:48:45.176939 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2739] Compacted 1@2 + 1@3 files to L3 => 149093241 bytes >2018-07-11 04:48:45.192231 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:48:45.191993) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 4 47 446 3640 732 0] max score 1.05, MB/sec: 24.1 rd, 24.1 wr, level 3, files in(1, 1) out(2) MB in(71.1, 71.1) out(142.2), read-write-amplify(4.0) write-amplify(2.0) OK, records in: 5276480, records dropped: 0 > >2018-07-11 04:48:45.192263 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:48:45.192017) EVENT_LOG_v1 {"time_micros": 1531284525192006, "job": 2739, "event": "compaction_finished", "compaction_time_micros": 6183188, "output_level": 3, "num_output_files": 2, "total_output_size": 149093241, "num_input_records": 5276480, "num_output_records": 5276480, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 4, 47, 446, 3640, 732, 0]} >2018-07-11 04:48:45.193187 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284525193183, "job": 2739, "event": "table_file_deletion", "file_number": 35765} >2018-07-11 04:48:45.193217 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284525193214, "job": 2739, "event": "table_file_deletion", "file_number": 35672} >2018-07-11 04:48:45.193435 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_picker.cc:461] [default] Expanding@2 1+1(74546954+46965305 bytes) to 3+1 (223639007+46965305bytes) > >2018-07-11 04:48:45.193532 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2740] Compacting 3@2 + 1@3 files to L3, score 1.05 >2018-07-11 04:48:45.193542 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2739 Base level 2, inputs: [35767(71MB) 35768(71MB) 35769(71MB)], [35743(44MB)] > >2018-07-11 04:48:45.193556 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284525193546, "job": 2740, "event": "compaction_started", "files_L2": [35767, 35768, 35769], "files_L3": [35743], "score": 1.05355, "input_data_size": 270604312} >2018-07-11 04:48:48.258576 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2740] Generated table #35773: 2638241 keys, 74546385 bytes >2018-07-11 04:48:48.258632 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284528258603, "cf_name": "default", "job": 2740, "event": "table_file_creation", "file_number": 35773, "file_size": 74546385, "table_properties": {"data_size": 67112109, "index_size": 690103, "filter_size": 6743183, "raw_key_size": 76215857, "raw_average_key_size": 28, "raw_value_size": 31365760, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:48:51.332690 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2740] Generated table #35774: 2638241 keys, 74546996 bytes >2018-07-11 04:48:51.332734 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284531332707, "cf_name": "default", "job": 2740, "event": "table_file_creation", "file_number": 35774, "file_size": 74546996, "table_properties": {"data_size": 67112158, "index_size": 690665, "filter_size": 6743183, "raw_key_size": 76215852, "raw_average_key_size": 28, "raw_value_size": 31365755, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:48:54.338292 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2740] Generated table #35775: 2638241 keys, 74546001 bytes >2018-07-11 04:48:54.338339 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284534338323, "cf_name": "default", "job": 2740, "event": "table_file_creation", "file_number": 35775, "file_size": 74546001, "table_properties": {"data_size": 67112099, "index_size": 689729, "filter_size": 6743183, "raw_key_size": 76215853, "raw_average_key_size": 28, "raw_value_size": 31365756, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:48:56.184948 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2740] Generated table #35776: 1519916 keys, 46965117 bytes >2018-07-11 04:48:56.184989 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284536184963, "cf_name": "default", "job": 2740, "event": "table_file_creation", "file_number": 35776, "file_size": 46965117, "table_properties": {"data_size": 42636656, "index_size": 433322, "filter_size": 3894151, "raw_key_size": 44348938, "raw_average_key_size": 29, "raw_value_size": 21522467, "raw_average_value_size": 14, "num_data_blocks": 10538, "num_entries": 1519916, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:48:56.185021 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2740] Compacted 3@2 + 1@3 files to L3 => 270604499 bytes >2018-07-11 04:48:56.200440 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:48:56.200145) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 4 44 449 3640 732 0] max score 1.01, MB/sec: 24.6 rd, 24.6 wr, level 3, files in(3, 1) out(4) MB in(213.3, 44.8) out(258.1), read-write-amplify(2.4) write-amplify(1.2) OK, records in: 9434639, records dropped: 0 > >2018-07-11 04:48:56.200451 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:48:56.200169) EVENT_LOG_v1 {"time_micros": 1531284536200159, "job": 2740, "event": "compaction_finished", "compaction_time_micros": 10991450, "output_level": 3, "num_output_files": 4, "total_output_size": 270604499, "num_input_records": 9434639, "num_output_records": 9434639, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 4, 44, 449, 3640, 732, 0]} >2018-07-11 04:48:56.201387 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284536201383, "job": 2740, "event": "table_file_deletion", "file_number": 35769} >2018-07-11 04:48:56.201402 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284536201400, "job": 2740, "event": "table_file_deletion", "file_number": 35768} >2018-07-11 04:48:56.201414 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284536201411, "job": 2740, "event": "table_file_deletion", "file_number": 35767} >2018-07-11 04:48:56.201426 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284536201424, "job": 2740, "event": "table_file_deletion", "file_number": 35743} >2018-07-11 04:48:56.216915 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:48:56.201731) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #35774 to level-4 74546996 bytes > >2018-07-11 04:48:56.216926 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:48:56.216824) EVENT_LOG_v1 {"time_micros": 1531284536216818, "job": 2741, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 74546996} >2018-07-11 04:48:56.216928 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:48:56.216832) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-4 74546996 bytes OK: base level 1 max bytes base 268435456 files[0 4 44 448 3641 732 0] max score 1.01 > >2018-07-11 04:48:56.217802 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2742] Compacting 1@3 + 2@4 files to L4, score 1.01 >2018-07-11 04:48:56.217813 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2741 Base level 3, inputs: [35772(71MB)], [35673(71MB) 35744(71MB)] > >2018-07-11 04:48:56.217839 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284536217816, "job": 2742, "event": "compaction_started", "files_L3": [35772], "files_L4": [35673, 35744], "score": 1.00568, "input_data_size": 223639028} >2018-07-11 04:48:59.411713 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2742] Generated table #35777: 2638241 keys, 74546113 bytes >2018-07-11 04:48:59.411759 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284539411743, "cf_name": "default", "job": 2742, "event": "table_file_creation", "file_number": 35777, "file_size": 74546113, "table_properties": {"data_size": 67112096, "index_size": 689844, "filter_size": 6743183, "raw_key_size": 76215852, "raw_average_key_size": 28, "raw_value_size": 31365755, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:49:02.495649 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2742] Generated table #35778: 2638241 keys, 74546378 bytes >2018-07-11 04:49:02.495692 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284542495666, "cf_name": "default", "job": 2742, "event": "table_file_creation", "file_number": 35778, "file_size": 74546378, "table_properties": {"data_size": 67112175, "index_size": 690030, "filter_size": 6743183, "raw_key_size": 76215860, "raw_average_key_size": 28, "raw_value_size": 31365763, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:49:05.598901 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2742] Generated table #35779: 2638240 keys, 74546225 bytes >2018-07-11 04:49:05.598936 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284545598921, "cf_name": "default", "job": 2742, "event": "table_file_creation", "file_number": 35779, "file_size": 74546225, "table_properties": {"data_size": 67112059, "index_size": 690006, "filter_size": 6743170, "raw_key_size": 76215822, "raw_average_key_size": 28, "raw_value_size": 31365742, "raw_average_value_size": 11, "num_data_blocks": 16489, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:49:05.598970 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2742] Compacted 1@3 + 2@4 files to L4 => 223638716 bytes >2018-07-11 04:49:05.608607 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:49:05.608324) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 4 44 447 3642 732 0] max score 1.00, MB/sec: 23.8 rd, 23.8 wr, level 4, files in(1, 2) out(3) MB in(71.1, 142.2) out(213.3), read-write-amplify(6.0) write-amplify(3.0) OK, records in: 7914722, records dropped: 0 > >2018-07-11 04:49:05.608618 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:49:05.608349) EVENT_LOG_v1 {"time_micros": 1531284545608338, "job": 2742, "event": "compaction_finished", "compaction_time_micros": 9381118, "output_level": 4, "num_output_files": 3, "total_output_size": 223638716, "num_input_records": 7914722, "num_output_records": 7914722, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 4, 44, 447, 3642, 732, 0]} >2018-07-11 04:49:05.609567 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284545609563, "job": 2742, "event": "table_file_deletion", "file_number": 35772} >2018-07-11 04:49:05.609595 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284545609593, "job": 2742, "event": "table_file_deletion", "file_number": 35744} >2018-07-11 04:49:05.609608 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284545609606, "job": 2742, "event": "table_file_deletion", "file_number": 35673} >2018-07-11 04:49:05.625088 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:49:05.609914) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #35773 to level-4 74546385 bytes > >2018-07-11 04:49:05.625099 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:49:05.624994) EVENT_LOG_v1 {"time_micros": 1531284545624987, "job": 2743, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 74546385} >2018-07-11 04:49:05.625102 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:49:05.625001) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-4 74546385 bytes OK: base level 1 max bytes base 268435456 files[0 4 44 446 3643 732 0] max score 1.00 > >2018-07-11 04:49:05.658574 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:49:05.625957) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #35774 to level-5 74546996 bytes > >2018-07-11 04:49:05.658586 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:49:05.658365) EVENT_LOG_v1 {"time_micros": 1531284545658359, "job": 2744, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546996} >2018-07-11 04:49:05.658588 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:49:05.658373) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546996 bytes OK: base level 1 max bytes base 268435456 files[0 4 44 446 3642 733 0] max score 1.00 > >2018-07-11 04:49:05.675135 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:49:05.659458) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #3528 to level-5 74546852 bytes > >2018-07-11 04:49:05.675145 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:49:05.675042) EVENT_LOG_v1 {"time_micros": 1531284545675036, "job": 2745, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546852} >2018-07-11 04:49:05.675148 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:49:05.675050) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546852 bytes OK: base level 1 max bytes base 268435456 files[0 4 44 446 3641 734 0] max score 1.00 > >2018-07-11 04:49:05.691774 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:49:05.675980) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #8774 to level-5 74546852 bytes > >2018-07-11 04:49:05.691784 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:49:05.691687) EVENT_LOG_v1 {"time_micros": 1531284545691681, "job": 2746, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546852} >2018-07-11 04:49:05.691786 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:49:05.691694) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546852 bytes OK: base level 1 max bytes base 268435456 files[0 4 44 446 3640 735 0] max score 1.00 > >2018-07-11 04:49:05.708621 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:49:05.692668) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #35771 to level-4 74546288 bytes > >2018-07-11 04:49:05.708632 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:49:05.708443) EVENT_LOG_v1 {"time_micros": 1531284545708436, "job": 2747, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 74546288} >2018-07-11 04:49:05.708635 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:49:05.708451) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-4 74546288 bytes OK: base level 1 max bytes base 268435456 files[0 4 44 445 3641 735 0] max score 1.00 > >2018-07-11 04:49:05.725217 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:49:05.709529) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #12778 to level-5 74546852 bytes > >2018-07-11 04:49:05.725227 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:49:05.725122) EVENT_LOG_v1 {"time_micros": 1531284545725116, "job": 2748, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546852} >2018-07-11 04:49:05.725230 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:49:05.725130) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546852 bytes OK: base level 1 max bytes base 268435456 files[0 4 44 445 3640 736 0] max score 1.00 > >2018-07-11 04:50:10.702397 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:684] reusing log 35746 from recycle list > >2018-07-11 04:50:10.702569 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #35780. Immutable memtables: 0. > >2018-07-11 04:50:10.786739 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:50:10.702638) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1 >2018-07-11 04:50:10.786753 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 2749] Flushing memtable with next log file: 35780 > >2018-07-11 04:50:10.786769 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284610786761, "job": 2749, "event": "flush_started", "num_memtables": 1, "num_entries": 5000015, "num_deletes": 0, "memory_usage": 292940568} >2018-07-11 04:50:10.786772 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 2749] Level-0 flush table #35781: started >2018-07-11 04:50:14.613130 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284614613110, "cf_name": "default", "job": 2749, "event": "table_file_creation", "file_number": 35781, "file_size": 141286890, "table_properties": {"data_size": 127197778, "index_size": 1308397, "filter_size": 12779725, "raw_key_size": 144445101, "raw_average_key_size": 28, "raw_value_size": 59450671, "raw_average_value_size": 11, "num_data_blocks": 31252, "num_entries": 5000014, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:50:14.613155 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 2749] Level-0 flush table #35781: 141286890 bytes OK >2018-07-11 04:50:14.623606 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:242] adding log 35753 to recycle list > >2018-07-11 04:50:14.623617 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:50:14.613180) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #35781 started >2018-07-11 04:50:14.623620 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:50:14.623373) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #35781: memtable #1 done >2018-07-11 04:50:14.623622 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:50:14.623390) EVENT_LOG_v1 {"time_micros": 1531284614623383, "job": 2749, "event": "flush_finished", "lsm_state": [1, 4, 44, 445, 3640, 736, 0], "immutable_memtables": 0} >2018-07-11 04:50:14.623624 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:50:14.623554) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 4 44 445 3640 736 0] max score 1.00 > >2018-07-11 04:50:14.624511 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 2749] Try to delete WAL files size 178896755, prev total WAL file size 178896767, number of live WAL files 2. > >2018-07-11 04:51:27.443946 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:684] reusing log 35753 from recycle list > >2018-07-11 04:51:27.444089 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #35782. Immutable memtables: 0. > >2018-07-11 04:51:27.498286 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:51:27.444123) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1 >2018-07-11 04:51:27.498300 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 2750] Flushing memtable with next log file: 35782 > >2018-07-11 04:51:27.498318 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284687498310, "job": 2750, "event": "flush_started", "num_memtables": 1, "num_entries": 5000015, "num_deletes": 0, "memory_usage": 292972664} >2018-07-11 04:51:27.498322 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 2750] Level-0 flush table #35783: started >2018-07-11 04:51:30.220117 7f41649ec700 1 bluefs _allocate failed to allocate 0x8700000 on bdev 1, free 0x5afe000; fallback to bdev 2 >2018-07-11 04:51:32.006934 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284692006913, "cf_name": "default", "job": 2750, "event": "table_file_creation", "file_number": 35783, "file_size": 141286889, "table_properties": {"data_size": 127197777, "index_size": 1308397, "filter_size": 12779725, "raw_key_size": 144445101, "raw_average_key_size": 28, "raw_value_size": 59450671, "raw_average_value_size": 11, "num_data_blocks": 31252, "num_entries": 5000014, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:51:32.006956 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 2750] Level-0 flush table #35783: 141286889 bytes OK >2018-07-11 04:51:32.021454 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:242] adding log 35780 to recycle list > >2018-07-11 04:51:32.021466 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:51:32.006981) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #35783 started >2018-07-11 04:51:32.021470 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:51:32.021204) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #35783: memtable #1 done >2018-07-11 04:51:32.021472 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:51:32.021222) EVENT_LOG_v1 {"time_micros": 1531284692021215, "job": 2750, "event": "flush_finished", "lsm_state": [2, 4, 44, 445, 3640, 736, 0], "immutable_memtables": 0} >2018-07-11 04:51:32.021474 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:51:32.021400) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[2 4 44 445 3640 736 0] max score 1.05 > >2018-07-11 04:51:32.021621 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2751] Compacting 2@0 + 2@1 files to L1, score 1.05 >2018-07-11 04:51:32.021639 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2750 Base level 0, inputs: [35783(134MB) 35781(134MB)], [35755(71MB) 35760(65MB)] > >2018-07-11 04:51:32.021663 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284692021645, "job": 2751, "event": "compaction_started", "files_L0": [35783, 35781], "files_L1": [35755, 35760], "score": 1.05267, "input_data_size": 425327642} >2018-07-11 04:51:32.022333 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 2750] Try to delete WAL files size 178896755, prev total WAL file size 178896767, number of live WAL files 2. > >2018-07-11 04:51:35.279456 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2751] Generated table #35784: 2623703 keys, 74509601 bytes >2018-07-11 04:51:35.279503 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284695279487, "cf_name": "default", "job": 2751, "event": "table_file_creation", "file_number": 35784, "file_size": 74509601, "table_properties": {"data_size": 67110352, "index_size": 691413, "filter_size": 6706846, "raw_key_size": 75828127, "raw_average_key_size": 28, "raw_value_size": 31553021, "raw_average_value_size": 12, "num_data_blocks": 16475, "num_entries": 2623703, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:51:37.369806 7f41641eb700 1 bluefs _allocate failed to allocate 0x4800000 on bdev 1, free 0x12fe000; fallback to bdev 2 >2018-07-11 04:51:38.383142 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2751] Generated table #35785: 2638241 keys, 74546889 bytes >2018-07-11 04:51:38.383187 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284698383173, "cf_name": "default", "job": 2751, "event": "table_file_creation", "file_number": 35785, "file_size": 74546889, "table_properties": {"data_size": 67112118, "index_size": 690598, "filter_size": 6743183, "raw_key_size": 76215846, "raw_average_key_size": 28, "raw_value_size": 31365749, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:51:40.640615 7f41641eb700 1 bluefs _allocate failed to allocate 0x4800000 on bdev 1, free 0x12fe000; fallback to bdev 2 >2018-07-11 04:51:41.727024 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2751] Generated table #35786: 2638241 keys, 74546491 bytes >2018-07-11 04:51:41.727070 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284701727050, "cf_name": "default", "job": 2751, "event": "table_file_creation", "file_number": 35786, "file_size": 74546491, "table_properties": {"data_size": 67112154, "index_size": 690164, "filter_size": 6743183, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:51:43.744668 7f41641eb700 1 bluefs _allocate failed to allocate 0x4800000 on bdev 1, free 0x12fe000; fallback to bdev 2 >2018-07-11 04:51:44.670454 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2751] Generated table #35787: 2638241 keys, 74546381 bytes >2018-07-11 04:51:44.670496 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284704670471, "cf_name": "default", "job": 2751, "event": "table_file_creation", "file_number": 35787, "file_size": 74546381, "table_properties": {"data_size": 67112084, "index_size": 690124, "filter_size": 6743183, "raw_key_size": 76215850, "raw_average_key_size": 28, "raw_value_size": 31365753, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:51:46.246331 7f41641eb700 1 bluefs _allocate failed to allocate 0x3900000 on bdev 1, free 0x12fe000; fallback to bdev 2 >2018-07-11 04:51:47.001317 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2751] Generated table #35788: 2085317 keys, 58922727 bytes >2018-07-11 04:51:47.001361 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284707001334, "cf_name": "default", "job": 2751, "event": "table_file_creation", "file_number": 35788, "file_size": 58922727, "table_properties": {"data_size": 53046767, "index_size": 545032, "filter_size": 5329940, "raw_key_size": 60242505, "raw_average_key_size": 28, "raw_value_size": 24792116, "raw_average_value_size": 11, "num_data_blocks": 13034, "num_entries": 2085317, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:51:47.520676 7f41641eb700 1 bluefs _allocate failed to allocate 0x4200000 on bdev 1, free 0x12fe000; fallback to bdev 2 >2018-07-11 04:51:48.334343 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2751] Generated table #35789: 166166 keys, 68245107 bytes >2018-07-11 04:51:48.334372 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284708334358, "cf_name": "default", "job": 2751, "event": "table_file_creation", "file_number": 35789, "file_size": 68245107, "table_properties": {"data_size": 67110245, "index_size": 564878, "filter_size": 568996, "raw_key_size": 15375288, "raw_average_key_size": 92, "raw_value_size": 53112674, "raw_average_value_size": 319, "num_data_blocks": 15214, "num_entries": 166166, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "80898", "kMergeOperands": "0"}} >2018-07-11 04:51:48.346818 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2751] Generated table #35790: 4 keys, 5090 bytes >2018-07-11 04:51:48.346855 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284708346830, "cf_name": "default", "job": 2751, "event": "table_file_creation", "file_number": 35790, "file_size": 5090, "table_properties": {"data_size": 4075, "index_size": 28, "filter_size": 20, "raw_key_size": 333, "raw_average_key_size": 83, "raw_value_size": 3749, "raw_average_value_size": 937, "num_data_blocks": 1, "num_entries": 4, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:51:48.346913 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2751] Compacted 2@0 + 2@1 files to L1 => 425322286 bytes >2018-07-11 04:51:48.362838 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:51:48.362563) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 9 44 445 3640 736 0] max score 1.80, MB/sec: 26.1 rd, 26.1 wr, level 1, files in(2, 2) out(7) MB in(269.5, 136.1) out(405.6), read-write-amplify(3.0) write-amplify(1.5) OK, records in: 12789921, records dropped: 8 > >2018-07-11 04:51:48.362848 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:51:48.362587) EVENT_LOG_v1 {"time_micros": 1531284708362576, "job": 2751, "event": "compaction_finished", "compaction_time_micros": 16325230, "output_level": 1, "num_output_files": 7, "total_output_size": 425322286, "num_input_records": 12789921, "num_output_records": 12789913, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 9, 44, 445, 3640, 736, 0]} >2018-07-11 04:51:48.363801 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284708363797, "job": 2751, "event": "table_file_deletion", "file_number": 35783} >2018-07-11 04:51:48.363827 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284708363825, "job": 2751, "event": "table_file_deletion", "file_number": 35781} >2018-07-11 04:51:48.363838 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284708363836, "job": 2751, "event": "table_file_deletion", "file_number": 35760} >2018-07-11 04:51:48.363849 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284708363847, "job": 2751, "event": "table_file_deletion", "file_number": 35755} >2018-07-11 04:51:48.364059 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_picker.cc:461] [default] Expanding@1 1+5(74546889+318010271 bytes) to 4+5 (282562488+318010271bytes) > >2018-07-11 04:51:48.364156 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2752] Compacting 4@1 + 5@2 files to L2, score 1.80 >2018-07-11 04:51:48.364169 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2751 Base level 1, inputs: [35785(71MB) 35786(71MB) 35787(71MB) 35788(56MB)], [35762(71MB) 35763(71MB) 35764(71MB) 35766(71MB) 35770(18MB)] > >2018-07-11 04:51:48.364183 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284708364173, "job": 2752, "event": "compaction_started", "files_L1": [35785, 35786, 35787, 35788], "files_L2": [35762, 35763, 35764, 35766, 35770], "score": 1.79592, "input_data_size": 600572759} >2018-07-11 04:51:51.423574 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2752] Generated table #35791: 2638238 keys, 74545701 bytes >2018-07-11 04:51:51.423620 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284711423604, "cf_name": "default", "job": 2752, "event": "table_file_creation", "file_number": 35791, "file_size": 74545701, "table_properties": {"data_size": 67112107, "index_size": 689427, "filter_size": 6743177, "raw_key_size": 76215768, "raw_average_key_size": 28, "raw_value_size": 31365722, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638238, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:51:54.403551 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2752] Generated table #35792: 2638239 keys, 74546969 bytes >2018-07-11 04:51:54.403597 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284714403569, "cf_name": "default", "job": 2752, "event": "table_file_creation", "file_number": 35792, "file_size": 74546969, "table_properties": {"data_size": 67112119, "index_size": 690681, "filter_size": 6743179, "raw_key_size": 76215785, "raw_average_key_size": 28, "raw_value_size": 31365722, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638239, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:51:57.517686 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2752] Generated table #35793: 2638239 keys, 74546645 bytes >2018-07-11 04:51:57.517722 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284717517707, "cf_name": "default", "job": 2752, "event": "table_file_creation", "file_number": 35793, "file_size": 74546645, "table_properties": {"data_size": 67112119, "index_size": 690357, "filter_size": 6743179, "raw_key_size": 76215788, "raw_average_key_size": 28, "raw_value_size": 31365725, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638239, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:52:00.686640 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2752] Generated table #35794: 2638239 keys, 74546269 bytes >2018-07-11 04:52:00.686684 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284720686657, "cf_name": "default", "job": 2752, "event": "table_file_creation", "file_number": 35794, "file_size": 74546269, "table_properties": {"data_size": 67112148, "index_size": 689952, "filter_size": 6743179, "raw_key_size": 76215800, "raw_average_key_size": 28, "raw_value_size": 31365737, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638239, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:52:03.829650 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2752] Generated table #35795: 2638241 keys, 74546390 bytes >2018-07-11 04:52:03.829684 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284723829669, "cf_name": "default", "job": 2752, "event": "table_file_creation", "file_number": 35795, "file_size": 74546390, "table_properties": {"data_size": 67112152, "index_size": 690064, "filter_size": 6743184, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:52:06.980625 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2752] Generated table #35796: 2638241 keys, 74546173 bytes >2018-07-11 04:52:06.980659 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284726980644, "cf_name": "default", "job": 2752, "event": "table_file_creation", "file_number": 35796, "file_size": 74546173, "table_properties": {"data_size": 67112152, "index_size": 689848, "filter_size": 6743183, "raw_key_size": 76215851, "raw_average_key_size": 28, "raw_value_size": 31365754, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:52:10.012054 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2752] Generated table #35797: 2638241 keys, 74546259 bytes >2018-07-11 04:52:10.012100 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284730012072, "cf_name": "default", "job": 2752, "event": "table_file_creation", "file_number": 35797, "file_size": 74546259, "table_properties": {"data_size": 67112099, "index_size": 689987, "filter_size": 6743183, "raw_key_size": 76215856, "raw_average_key_size": 28, "raw_value_size": 31365759, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:52:13.075814 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2752] Generated table #35798: 2638241 keys, 74547017 bytes >2018-07-11 04:52:13.075859 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284733075832, "cf_name": "default", "job": 2752, "event": "table_file_creation", "file_number": 35798, "file_size": 74547017, "table_properties": {"data_size": 67112165, "index_size": 690679, "filter_size": 6743183, "raw_key_size": 76215853, "raw_average_key_size": 28, "raw_value_size": 31365756, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:52:13.300921 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2752] Generated table #35799: 144256 keys, 4202518 bytes >2018-07-11 04:52:13.300959 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284733300934, "cf_name": "default", "job": 2752, "event": "table_file_creation", "file_number": 35799, "file_size": 4202518, "table_properties": {"data_size": 3794185, "index_size": 38344, "filter_size": 369004, "raw_key_size": 4183150, "raw_average_key_size": 28, "raw_value_size": 1820420, "raw_average_value_size": 12, "num_data_blocks": 934, "num_entries": 144256, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "11", "kMergeOperands": "0"}} >2018-07-11 04:52:13.300988 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2752] Compacted 4@1 + 5@2 files to L2 => 600573941 bytes >2018-07-11 04:52:13.312463 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:52:13.312205) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 5 48 445 3640 736 0] max score 1.08, MB/sec: 24.1 rd, 24.1 wr, level 2, files in(4, 5) out(9) MB in(269.5, 303.3) out(572.8), read-write-amplify(4.3) write-amplify(2.1) OK, records in: 21250175, records dropped: 0 > >2018-07-11 04:52:13.312473 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:52:13.312228) EVENT_LOG_v1 {"time_micros": 1531284733312218, "job": 2752, "event": "compaction_finished", "compaction_time_micros": 24936787, "output_level": 2, "num_output_files": 9, "total_output_size": 600573941, "num_input_records": 21250175, "num_output_records": 21250175, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 5, 48, 445, 3640, 736, 0]} >2018-07-11 04:52:13.313389 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284733313385, "job": 2752, "event": "table_file_deletion", "file_number": 35788} >2018-07-11 04:52:13.313402 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284733313399, "job": 2752, "event": "table_file_deletion", "file_number": 35787} >2018-07-11 04:52:13.313413 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284733313411, "job": 2752, "event": "table_file_deletion", "file_number": 35786} >2018-07-11 04:52:13.313427 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284733313425, "job": 2752, "event": "table_file_deletion", "file_number": 35785} >2018-07-11 04:52:13.313443 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284733313441, "job": 2752, "event": "table_file_deletion", "file_number": 35770} >2018-07-11 04:52:13.313456 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284733313454, "job": 2752, "event": "table_file_deletion", "file_number": 35766} >2018-07-11 04:52:13.313470 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284733313467, "job": 2752, "event": "table_file_deletion", "file_number": 35764} >2018-07-11 04:52:13.313483 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284733313480, "job": 2752, "event": "table_file_deletion", "file_number": 35763} >2018-07-11 04:52:13.313497 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284733313494, "job": 2752, "event": "table_file_deletion", "file_number": 35762} >2018-07-11 04:52:13.313711 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_picker.cc:461] [default] Expanding@2 1+1(74547017+46965117 bytes) to 3+1 (223639449+46965117bytes) > >2018-07-11 04:52:13.313806 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2753] Compacting 3@2 + 1@3 files to L3, score 1.08 >2018-07-11 04:52:13.313818 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2752 Base level 2, inputs: [35796(71MB) 35797(71MB) 35798(71MB)], [35776(44MB)] > >2018-07-11 04:52:13.313830 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284733313821, "job": 2753, "event": "compaction_started", "files_L2": [35796, 35797, 35798], "files_L3": [35776], "score": 1.0755, "input_data_size": 270604566} >2018-07-11 04:52:16.438480 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2753] Generated table #35800: 2638241 keys, 74546208 bytes >2018-07-11 04:52:16.438533 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284736438516, "cf_name": "default", "job": 2753, "event": "table_file_creation", "file_number": 35800, "file_size": 74546208, "table_properties": {"data_size": 67112161, "index_size": 689874, "filter_size": 6743183, "raw_key_size": 76215850, "raw_average_key_size": 28, "raw_value_size": 31365753, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:52:19.530978 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2753] Generated table #35801: 2638241 keys, 74547036 bytes >2018-07-11 04:52:19.531022 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284739530995, "cf_name": "default", "job": 2753, "event": "table_file_creation", "file_number": 35801, "file_size": 74547036, "table_properties": {"data_size": 67112096, "index_size": 690767, "filter_size": 6743183, "raw_key_size": 76215859, "raw_average_key_size": 28, "raw_value_size": 31365762, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:52:22.573881 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2753] Generated table #35802: 2638241 keys, 74546304 bytes >2018-07-11 04:52:22.573931 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284742573902, "cf_name": "default", "job": 2753, "event": "table_file_creation", "file_number": 35802, "file_size": 74546304, "table_properties": {"data_size": 67112169, "index_size": 689962, "filter_size": 6743183, "raw_key_size": 76215855, "raw_average_key_size": 28, "raw_value_size": 31365758, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:52:24.652721 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2753] Generated table #35803: 1519916 keys, 46965206 bytes >2018-07-11 04:52:24.652763 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284744652736, "cf_name": "default", "job": 2753, "event": "table_file_creation", "file_number": 35803, "file_size": 46965206, "table_properties": {"data_size": 42636608, "index_size": 433459, "filter_size": 3894151, "raw_key_size": 44348934, "raw_average_key_size": 29, "raw_value_size": 21522463, "raw_average_value_size": 14, "num_data_blocks": 10538, "num_entries": 1519916, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:52:24.652794 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2753] Compacted 3@2 + 1@3 files to L3 => 270604754 bytes >2018-07-11 04:52:24.670540 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:52:24.670353) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 5 45 448 3640 736 0] max score 1.01, MB/sec: 23.9 rd, 23.9 wr, level 3, files in(3, 1) out(4) MB in(213.3, 44.8) out(258.1), read-write-amplify(2.4) write-amplify(1.2) OK, records in: 9434639, records dropped: 0 > >2018-07-11 04:52:24.670550 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:52:24.670376) EVENT_LOG_v1 {"time_micros": 1531284744670366, "job": 2753, "event": "compaction_finished", "compaction_time_micros": 11338952, "output_level": 3, "num_output_files": 4, "total_output_size": 270604754, "num_input_records": 9434639, "num_output_records": 9434639, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 5, 45, 448, 3640, 736, 0]} >2018-07-11 04:52:24.671455 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284744671451, "job": 2753, "event": "table_file_deletion", "file_number": 35798} >2018-07-11 04:52:24.671471 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284744671469, "job": 2753, "event": "table_file_deletion", "file_number": 35797} >2018-07-11 04:52:24.671487 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284744671485, "job": 2753, "event": "table_file_deletion", "file_number": 35796} >2018-07-11 04:52:24.671501 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284744671499, "job": 2753, "event": "table_file_deletion", "file_number": 35776} >2018-07-11 04:52:24.687285 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:52:24.671803) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #35801 to level-4 74547036 bytes > >2018-07-11 04:52:24.687296 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:52:24.687062) EVENT_LOG_v1 {"time_micros": 1531284744687056, "job": 2754, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 74547036} >2018-07-11 04:52:24.687299 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:52:24.687070) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-4 74547036 bytes OK: base level 1 max bytes base 268435456 files[0 5 45 447 3641 736 0] max score 1.00 > >2018-07-11 04:52:24.703898 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:52:24.688140) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #35802 to level-4 74546304 bytes > >2018-07-11 04:52:24.703908 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:52:24.703808) EVENT_LOG_v1 {"time_micros": 1531284744703802, "job": 2755, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 74546304} >2018-07-11 04:52:24.703911 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:52:24.703815) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-4 74546304 bytes OK: base level 1 max bytes base 268435456 files[0 5 45 446 3642 736 0] max score 1.00 > >2018-07-11 04:52:24.745465 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:52:24.704750) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #35801 to level-5 74547036 bytes > >2018-07-11 04:52:24.745474 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:52:24.745345) EVENT_LOG_v1 {"time_micros": 1531284744745339, "job": 2756, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74547036} >2018-07-11 04:52:24.745480 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:52:24.745353) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74547036 bytes OK: base level 1 max bytes base 268435456 files[0 5 45 446 3641 737 0] max score 1.00 > >2018-07-11 04:52:24.762412 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:52:24.746316) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #34926 to level-5 74546851 bytes > >2018-07-11 04:52:24.762423 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:52:24.762210) EVENT_LOG_v1 {"time_micros": 1531284744762204, "job": 2757, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546851} >2018-07-11 04:52:24.762425 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:52:24.762218) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546851 bytes OK: base level 1 max bytes base 268435456 files[0 5 45 446 3640 738 0] max score 1.00 > >2018-07-11 04:52:24.778970 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:52:24.763273) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #35800 to level-4 74546208 bytes > >2018-07-11 04:52:24.778980 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:52:24.778880) EVENT_LOG_v1 {"time_micros": 1531284744778874, "job": 2758, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 74546208} >2018-07-11 04:52:24.778983 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:52:24.778888) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-4 74546208 bytes OK: base level 1 max bytes base 268435456 files[0 5 45 445 3641 738 0] max score 1.00 > >2018-07-11 04:52:24.795575 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:52:24.779817) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #705 to level-5 74546851 bytes > >2018-07-11 04:52:24.795585 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:52:24.795486) EVENT_LOG_v1 {"time_micros": 1531284744795480, "job": 2759, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546851} >2018-07-11 04:52:24.795588 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:52:24.795494) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546851 bytes OK: base level 1 max bytes base 268435456 files[0 5 45 445 3640 739 0] max score 1.00 > >2018-07-11 04:52:48.470381 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:684] reusing log 35780 from recycle list > >2018-07-11 04:52:48.470547 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #35804. Immutable memtables: 0. > >2018-07-11 04:52:48.548937 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:52:48.470607) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1 >2018-07-11 04:52:48.548951 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 2760] Flushing memtable with next log file: 35804 > >2018-07-11 04:52:48.548969 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284768548960, "job": 2760, "event": "flush_started", "num_memtables": 1, "num_entries": 5000015, "num_deletes": 0, "memory_usage": 292941432} >2018-07-11 04:52:48.548973 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 2760] Level-0 flush table #35805: started >2018-07-11 04:52:52.408772 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284772408751, "cf_name": "default", "job": 2760, "event": "table_file_creation", "file_number": 35805, "file_size": 141286888, "table_properties": {"data_size": 127197776, "index_size": 1308397, "filter_size": 12779725, "raw_key_size": 144445101, "raw_average_key_size": 28, "raw_value_size": 59450671, "raw_average_value_size": 11, "num_data_blocks": 31252, "num_entries": 5000014, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:52:52.408797 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 2760] Level-0 flush table #35805: 141286888 bytes OK >2018-07-11 04:52:52.420059 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:242] adding log 35782 to recycle list > >2018-07-11 04:52:52.420070 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:52:52.408827) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #35805 started >2018-07-11 04:52:52.420074 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:52:52.419863) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #35805: memtable #1 done >2018-07-11 04:52:52.420076 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:52:52.419881) EVENT_LOG_v1 {"time_micros": 1531284772419873, "job": 2760, "event": "flush_finished", "lsm_state": [1, 5, 45, 445, 3640, 739, 0], "immutable_memtables": 0} >2018-07-11 04:52:52.420077 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:52:52.420008) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 5 45 445 3640 739 0] max score 1.00 > >2018-07-11 04:52:52.420963 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 2760] Try to delete WAL files size 178896755, prev total WAL file size 178896767, number of live WAL files 2. > >2018-07-11 04:55:00.765450 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:684] reusing log 35782 from recycle list > >2018-07-11 04:55:00.765626 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #35806. Immutable memtables: 0. > >2018-07-11 04:55:00.817421 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:55:00.765694) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1 >2018-07-11 04:55:00.817436 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 2761] Flushing memtable with next log file: 35806 > >2018-07-11 04:55:00.817454 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284900817445, "job": 2761, "event": "flush_started", "num_memtables": 1, "num_entries": 5000015, "num_deletes": 0, "memory_usage": 292956192} >2018-07-11 04:55:00.817457 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 2761] Level-0 flush table #35807: started >2018-07-11 04:55:04.703182 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284904703162, "cf_name": "default", "job": 2761, "event": "table_file_creation", "file_number": 35807, "file_size": 141286220, "table_properties": {"data_size": 127196935, "index_size": 1308576, "filter_size": 12779719, "raw_key_size": 144445077, "raw_average_key_size": 28, "raw_value_size": 59449772, "raw_average_value_size": 11, "num_data_blocks": 31252, "num_entries": 5000013, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:55:04.703206 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 2761] Level-0 flush table #35807: 141286220 bytes OK >2018-07-11 04:55:04.716541 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:242] adding log 35804 to recycle list > >2018-07-11 04:55:04.716554 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:55:04.703231) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #35807 started >2018-07-11 04:55:04.716557 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:55:04.716281) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #35807: memtable #1 done >2018-07-11 04:55:04.716559 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:55:04.716299) EVENT_LOG_v1 {"time_micros": 1531284904716292, "job": 2761, "event": "flush_finished", "lsm_state": [2, 5, 45, 445, 3640, 739, 0], "immutable_memtables": 0} >2018-07-11 04:55:04.716561 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:55:04.716474) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[2 5 45 445 3640 739 0] max score 1.05 > >2018-07-11 04:55:04.716977 7f41641eb700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS ------- >2018-07-11 04:55:04.716987 7f41641eb700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] >** DB Stats ** >Uptime(secs): 49646.4 total, 609.3 interval >Cumulative writes: 196K writes, 1837M keys, 196K commit groups, 1.0 writes per commit group, ingest: 61.73 GB, 1.27 MB/s >Cumulative WAL: 196K writes, 56K syncs, 3.48 writes per sync, written: 61.73 GB, 1.27 MB/s >Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent >Interval writes: 60 writes, 30M keys, 60 commit groups, 1.0 writes per commit group, ingest: 1023.66 MB, 1.68 MB/s >Interval WAL: 60 writes, 30 syncs, 1.94 writes per sync, written: 1.00 MB, 1.68 MB/s >Interval stall: 00:00:0.000 H:M:S, 0.0 percent > >** Compaction Stats [default] ** >Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop >---------------------------------------------------------------------------------------------------------------------------------------------------------- > L0 2/0 269.48 MB 1.1 0.0 0.0 0.0 48.3 48.3 0.0 1.0 0.0 32.4 1528 369 4.140 0 0 > L1 5/0 190.28 MB 0.7 79.5 48.2 31.3 79.5 48.1 0.0 1.6 25.7 25.7 3165 181 17.488 2516M 894K > L2 45/0 2.48 GB 1.0 93.4 48.2 45.2 93.4 48.1 0.0 1.9 21.5 21.5 4445 553 8.037 3486M 1358K > L3 445/0 24.93 GB 1.0 83.6 47.5 36.1 83.5 47.4 0.6 1.8 20.7 20.7 4128 287 14.382 3157M 4000K > L4 3640/0 250.00 GB 1.0 72.6 17.3 55.2 72.2 17.0 30.8 4.2 21.1 21.1 3513 232 15.143 2761M 15M > L5 739/0 51.28 GB 0.0 1.7 0.9 0.8 1.7 0.9 46.8 1.9 18.7 18.5 95 12 7.912 66M 1097K > Sum 4876/0 329.15 GB 0.0 330.8 162.1 168.7 378.7 209.9 78.2 7.8 20.1 23.0 16874 1634 10.327 11G 23M > Int 0/0 0.00 KB 0.0 4.6 2.6 2.1 5.4 3.3 1.3 6.9 21.2 24.8 224 19 11.776 166M 27 >Uptime(secs): 49646.4 total, 49646.4 interval >Flush(GB): cumulative 48.339, interval 0.790 >AddFile(GB): cumulative 0.000, interval 0.000 >AddFile(Total Files): cumulative 0, interval 0 >AddFile(L0 Files): cumulative 0, interval 0 >AddFile(Keys): cumulative 0, interval 0 >Cumulative compaction: 378.66 GB write, 7.81 MB/s write, 330.85 GB read, 6.82 MB/s read, 16873.7 seconds >Interval compaction: 5.41 GB write, 0.11 MB/s write, 4.62 GB read, 0.10 MB/s read, 223.7 seconds >Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count > >** File Read Latency Histogram By Level [default] ** > >** Compaction Stats [default] ** >Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop >---------------------------------------------------------------------------------------------------------------------------------------------------------- > L0 2/0 269.48 MB 1.1 0.0 0.0 0.0 48.3 48.3 0.0 1.0 0.0 32.4 1528 369 4.140 0 0 > L1 5/0 190.28 MB 0.7 79.5 48.2 31.3 79.5 48.1 0.0 1.6 25.7 25.7 3165 181 17.488 2516M 894K > L2 45/0 2.48 GB 1.0 93.4 48.2 45.2 93.4 48.1 0.0 1.9 21.5 21.5 4445 553 8.037 3486M 1358K > L3 445/0 24.93 GB 1.0 83.6 47.5 36.1 83.5 47.4 0.6 1.8 20.7 20.7 4128 287 14.382 3157M 4000K > L4 3640/0 250.00 GB 1.0 72.6 17.3 55.2 72.2 17.0 30.8 4.2 21.1 21.1 3513 232 15.143 2761M 15M > L5 739/0 51.28 GB 0.0 1.7 0.9 0.8 1.7 0.9 46.8 1.9 18.7 18.5 95 12 7.912 66M 1097K > Sum 4876/0 329.15 GB 0.0 330.8 162.1 168.7 378.7 209.9 78.2 7.8 20.1 23.0 16874 1634 10.327 11G 23M > Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 >Uptime(secs): 49646.4 total, 49646.4 interval >Flush(GB): cumulative 48.339, interval 0.000 >AddFile(GB): cumulative 0.000, interval 0.000 >AddFile(Total Files): cumulative 0, interval 0 >AddFile(L0 Files): cumulative 0, interval 0 >AddFile(Keys): cumulative 0, interval 0 >Cumulative compaction: 378.66 GB write, 7.81 MB/s write, 330.85 GB read, 6.82 MB/s read, 16873.7 seconds >Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds >Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count > >** File Read Latency Histogram By Level [default] ** > >2018-07-11 04:55:04.717120 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2762] Compacting 2@0 + 2@1 files to L1, score 1.05 >2018-07-11 04:55:04.717132 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2761 Base level 0, inputs: [35807(134MB) 35805(134MB)], [35784(71MB) 35789(65MB)] > >2018-07-11 04:55:04.717149 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284904717137, "job": 2762, "event": "compaction_started", "files_L0": [35807, 35805], "files_L1": [35784, 35789], "score": 1.05267, "input_data_size": 425327816} >2018-07-11 04:55:04.717468 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 2761] Try to delete WAL files size 178896755, prev total WAL file size 178896767, number of live WAL files 2. > >2018-07-11 04:55:07.815918 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2762] Generated table #35808: 2623673 keys, 74510224 bytes >2018-07-11 04:55:07.815964 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284907815936, "cf_name": "default", "job": 2762, "event": "table_file_creation", "file_number": 35808, "file_size": 74510224, "table_properties": {"data_size": 67111129, "index_size": 691331, "filter_size": 6706774, "raw_key_size": 75827471, "raw_average_key_size": 28, "raw_value_size": 31554165, "raw_average_value_size": 12, "num_data_blocks": 16475, "num_entries": 2623673, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:55:10.539494 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2762] Generated table #35809: 2638241 keys, 74546892 bytes >2018-07-11 04:55:10.539529 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284910539514, "cf_name": "default", "job": 2762, "event": "table_file_creation", "file_number": 35809, "file_size": 74546892, "table_properties": {"data_size": 67112121, "index_size": 690598, "filter_size": 6743183, "raw_key_size": 76215847, "raw_average_key_size": 28, "raw_value_size": 31365750, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:55:13.240240 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2762] Generated table #35810: 2638241 keys, 74546491 bytes >2018-07-11 04:55:13.240294 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284913240278, "cf_name": "default", "job": 2762, "event": "table_file_creation", "file_number": 35810, "file_size": 74546491, "table_properties": {"data_size": 67112154, "index_size": 690164, "filter_size": 6743183, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:55:16.113804 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2762] Generated table #35811: 2638241 keys, 74546381 bytes >2018-07-11 04:55:16.113849 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284916113822, "cf_name": "default", "job": 2762, "event": "table_file_creation", "file_number": 35811, "file_size": 74546381, "table_properties": {"data_size": 67112084, "index_size": 690124, "filter_size": 6743183, "raw_key_size": 76215850, "raw_average_key_size": 28, "raw_value_size": 31365753, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:55:18.212210 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2762] Generated table #35812: 2085317 keys, 58922727 bytes >2018-07-11 04:55:18.212260 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284918212227, "cf_name": "default", "job": 2762, "event": "table_file_creation", "file_number": 35812, "file_size": 58922727, "table_properties": {"data_size": 53046767, "index_size": 545032, "filter_size": 5329940, "raw_key_size": 60242505, "raw_average_key_size": 28, "raw_value_size": 24792116, "raw_average_value_size": 11, "num_data_blocks": 13034, "num_entries": 2085317, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:55:19.500211 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2762] Generated table #35813: 166172 keys, 68245221 bytes >2018-07-11 04:55:19.500258 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284919500237, "cf_name": "default", "job": 2762, "event": "table_file_creation", "file_number": 35813, "file_size": 68245221, "table_properties": {"data_size": 67110351, "index_size": 564870, "filter_size": 569012, "raw_key_size": 15375564, "raw_average_key_size": 92, "raw_value_size": 53112554, "raw_average_value_size": 319, "num_data_blocks": 15214, "num_entries": 166172, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "80898", "kMergeOperands": "0"}} >2018-07-11 04:55:19.516893 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2762] Generated table #35814: 4 keys, 5092 bytes >2018-07-11 04:55:19.516928 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284919516916, "cf_name": "default", "job": 2762, "event": "table_file_creation", "file_number": 35814, "file_size": 5092, "table_properties": {"data_size": 4077, "index_size": 28, "filter_size": 20, "raw_key_size": 334, "raw_average_key_size": 83, "raw_value_size": 3750, "raw_average_value_size": 937, "num_data_blocks": 1, "num_entries": 4, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:55:19.516986 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2762] Compacted 2@0 + 2@1 files to L1 => 425323028 bytes >2018-07-11 04:55:19.532936 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:55:19.532610) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 10 45 445 3640 739 0] max score 1.80, MB/sec: 28.7 rd, 28.7 wr, level 1, files in(2, 2) out(7) MB in(269.5, 136.1) out(405.6), read-write-amplify(3.0) write-amplify(1.5) OK, records in: 12789896, records dropped: 7 > >2018-07-11 04:55:19.532948 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:55:19.532641) EVENT_LOG_v1 {"time_micros": 1531284919532627, "job": 2762, "event": "compaction_finished", "compaction_time_micros": 14799820, "output_level": 1, "num_output_files": 7, "total_output_size": 425323028, "num_input_records": 12789896, "num_output_records": 12789889, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 10, 45, 445, 3640, 739, 0]} >2018-07-11 04:55:19.534092 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284919534075, "job": 2762, "event": "table_file_deletion", "file_number": 35807} >2018-07-11 04:55:19.534110 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284919534107, "job": 2762, "event": "table_file_deletion", "file_number": 35805} >2018-07-11 04:55:19.534128 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284919534124, "job": 2762, "event": "table_file_deletion", "file_number": 35789} >2018-07-11 04:55:19.534147 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284919534144, "job": 2762, "event": "table_file_deletion", "file_number": 35784} >2018-07-11 04:55:19.534379 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_picker.cc:461] [default] Expanding@1 1+6(74546892+376934492 bytes) to 4+6 (282562491+376934492bytes) > >2018-07-11 04:55:19.534490 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2763] Compacting 4@1 + 6@2 files to L2, score 1.80 >2018-07-11 04:55:19.534509 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2762 Base level 1, inputs: [35809(71MB) 35810(71MB) 35811(71MB) 35812(56MB)], [35791(71MB) 35792(71MB) 35793(71MB) 35794(71MB) 35795(71MB) 35799(4104KB)] > >2018-07-11 04:55:19.534530 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284919534516, "job": 2763, "event": "compaction_started", "files_L1": [35809, 35810, 35811, 35812], "files_L2": [35791, 35792, 35793, 35794, 35795, 35799], "score": 1.79594, "input_data_size": 659496983} >2018-07-11 04:55:22.834898 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2763] Generated table #35815: 2638240 keys, 74546321 bytes >2018-07-11 04:55:22.834942 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284922834915, "cf_name": "default", "job": 2763, "event": "table_file_creation", "file_number": 35815, "file_size": 74546321, "table_properties": {"data_size": 67112104, "index_size": 690046, "filter_size": 6743181, "raw_key_size": 76215826, "raw_average_key_size": 28, "raw_value_size": 31365746, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:55:25.888965 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2763] Generated table #35816: 2638240 keys, 74545933 bytes >2018-07-11 04:55:25.889009 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284925888982, "cf_name": "default", "job": 2763, "event": "table_file_creation", "file_number": 35816, "file_size": 74545933, "table_properties": {"data_size": 67112141, "index_size": 689621, "filter_size": 6743181, "raw_key_size": 76215813, "raw_average_key_size": 28, "raw_value_size": 31365733, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:55:28.901506 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2763] Generated table #35817: 2638239 keys, 74546033 bytes >2018-07-11 04:55:28.901551 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284928901525, "cf_name": "default", "job": 2763, "event": "table_file_creation", "file_number": 35817, "file_size": 74546033, "table_properties": {"data_size": 67112088, "index_size": 689776, "filter_size": 6743179, "raw_key_size": 76215794, "raw_average_key_size": 28, "raw_value_size": 31365731, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638239, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:55:31.935531 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2763] Generated table #35818: 2638240 keys, 74546143 bytes >2018-07-11 04:55:31.935575 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284931935549, "cf_name": "default", "job": 2763, "event": "table_file_creation", "file_number": 35818, "file_size": 74546143, "table_properties": {"data_size": 67112136, "index_size": 689836, "filter_size": 6743181, "raw_key_size": 76215823, "raw_average_key_size": 28, "raw_value_size": 31365743, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:55:35.044408 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2763] Generated table #35819: 2638241 keys, 74546247 bytes >2018-07-11 04:55:35.044452 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284935044425, "cf_name": "default", "job": 2763, "event": "table_file_creation", "file_number": 35819, "file_size": 74546247, "table_properties": {"data_size": 67112159, "index_size": 689915, "filter_size": 6743183, "raw_key_size": 76215855, "raw_average_key_size": 28, "raw_value_size": 31365758, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:55:38.144417 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2763] Generated table #35820: 2638241 keys, 74546489 bytes >2018-07-11 04:55:38.144460 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284938144433, "cf_name": "default", "job": 2763, "event": "table_file_creation", "file_number": 35820, "file_size": 74546489, "table_properties": {"data_size": 67112146, "index_size": 690169, "filter_size": 6743184, "raw_key_size": 76215856, "raw_average_key_size": 28, "raw_value_size": 31365759, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:55:41.314088 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2763] Generated table #35821: 2638241 keys, 74546279 bytes >2018-07-11 04:55:41.314133 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284941314106, "cf_name": "default", "job": 2763, "event": "table_file_creation", "file_number": 35821, "file_size": 74546279, "table_properties": {"data_size": 67112143, "index_size": 689963, "filter_size": 6743183, "raw_key_size": 76215849, "raw_average_key_size": 28, "raw_value_size": 31365752, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:55:44.381856 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2763] Generated table #35822: 2638241 keys, 74545863 bytes >2018-07-11 04:55:44.381902 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284944381875, "cf_name": "default", "job": 2763, "event": "table_file_creation", "file_number": 35822, "file_size": 74545863, "table_properties": {"data_size": 67112107, "index_size": 689583, "filter_size": 6743183, "raw_key_size": 76215860, "raw_average_key_size": 28, "raw_value_size": 31365763, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:55:47.043039 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2763] Generated table #35823: 2229569 keys, 63124993 bytes >2018-07-11 04:55:47.043082 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284947043055, "cf_name": "default", "job": 2763, "event": "table_file_creation", "file_number": 35823, "file_size": 63124993, "table_properties": {"data_size": 56840822, "index_size": 584244, "filter_size": 5698938, "raw_key_size": 64425525, "raw_average_key_size": 28, "raw_value_size": 26612474, "raw_average_value_size": 11, "num_data_blocks": 13968, "num_entries": 2229569, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "11", "kMergeOperands": "0"}} >2018-07-11 04:55:47.043120 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2763] Compacted 4@1 + 6@2 files to L2 => 659494301 bytes >2018-07-11 04:55:47.057597 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:55:47.057207) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 6 48 445 3640 739 0] max score 1.10, MB/sec: 24.0 rd, 24.0 wr, level 2, files in(4, 6) out(9) MB in(269.5, 359.5) out(628.9), read-write-amplify(4.7) write-amplify(2.3) OK, records in: 23335492, records dropped: 0 > >2018-07-11 04:55:47.057609 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:55:47.057233) EVENT_LOG_v1 {"time_micros": 1531284947057222, "job": 2763, "event": "compaction_finished", "compaction_time_micros": 27508568, "output_level": 2, "num_output_files": 9, "total_output_size": 659494301, "num_input_records": 23335492, "num_output_records": 23335492, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 6, 48, 445, 3640, 739, 0]} >2018-07-11 04:55:47.058490 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284947058484, "job": 2763, "event": "table_file_deletion", "file_number": 35812} >2018-07-11 04:55:47.058509 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284947058505, "job": 2763, "event": "table_file_deletion", "file_number": 35811} >2018-07-11 04:55:47.058532 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284947058528, "job": 2763, "event": "table_file_deletion", "file_number": 35810} >2018-07-11 04:55:47.058553 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284947058550, "job": 2763, "event": "table_file_deletion", "file_number": 35809} >2018-07-11 04:55:47.058582 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284947058579, "job": 2763, "event": "table_file_deletion", "file_number": 35799} >2018-07-11 04:55:47.058609 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284947058605, "job": 2763, "event": "table_file_deletion", "file_number": 35795} >2018-07-11 04:55:47.058630 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284947058626, "job": 2763, "event": "table_file_deletion", "file_number": 35794} >2018-07-11 04:55:47.058656 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284947058652, "job": 2763, "event": "table_file_deletion", "file_number": 35793} >2018-07-11 04:55:47.058687 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284947058683, "job": 2763, "event": "table_file_deletion", "file_number": 35792} >2018-07-11 04:55:47.058709 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284947058705, "job": 2763, "event": "table_file_deletion", "file_number": 35791} >2018-07-11 04:55:47.058927 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_picker.cc:461] [default] Expanding@2 1+1(74546489+46965206 bytes) to 3+1 (223638631+46965206bytes) > >2018-07-11 04:55:47.059028 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2764] Compacting 3@2 + 1@3 files to L3, score 1.10 >2018-07-11 04:55:47.059041 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2763 Base level 2, inputs: [35820(71MB) 35821(71MB) 35822(71MB)], [35803(44MB)] > >2018-07-11 04:55:47.059060 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284947059046, "job": 2764, "event": "compaction_started", "files_L2": [35820, 35821, 35822], "files_L3": [35803], "score": 1.09745, "input_data_size": 270603837} >2018-07-11 04:55:50.053619 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2764] Generated table #35824: 2638241 keys, 74546368 bytes >2018-07-11 04:55:50.053677 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284950053660, "cf_name": "default", "job": 2764, "event": "table_file_creation", "file_number": 35824, "file_size": 74546368, "table_properties": {"data_size": 67112114, "index_size": 690081, "filter_size": 6743183, "raw_key_size": 76215856, "raw_average_key_size": 28, "raw_value_size": 31365759, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:55:54.316056 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2764] Generated table #35825: 2638241 keys, 74546179 bytes >2018-07-11 04:55:54.316106 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284954316078, "cf_name": "default", "job": 2764, "event": "table_file_creation", "file_number": 35825, "file_size": 74546179, "table_properties": {"data_size": 67112155, "index_size": 689851, "filter_size": 6743183, "raw_key_size": 76215855, "raw_average_key_size": 28, "raw_value_size": 31365758, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:55:57.616778 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2764] Generated table #35826: 2638241 keys, 74546260 bytes >2018-07-11 04:55:57.616809 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284957616795, "cf_name": "default", "job": 2764, "event": "table_file_creation", "file_number": 35826, "file_size": 74546260, "table_properties": {"data_size": 67112097, "index_size": 689990, "filter_size": 6743183, "raw_key_size": 76215847, "raw_average_key_size": 28, "raw_value_size": 31365750, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:55:59.497767 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2764] Generated table #35827: 1519916 keys, 46965534 bytes >2018-07-11 04:55:59.497810 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284959497783, "cf_name": "default", "job": 2764, "event": "table_file_creation", "file_number": 35827, "file_size": 46965534, "table_properties": {"data_size": 42636651, "index_size": 433744, "filter_size": 3894151, "raw_key_size": 44348941, "raw_average_key_size": 29, "raw_value_size": 21522470, "raw_average_value_size": 14, "num_data_blocks": 10538, "num_entries": 1519916, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:55:59.497840 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2764] Compacted 3@2 + 1@3 files to L3 => 270604341 bytes >2018-07-11 04:55:59.515619 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:55:59.515377) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 6 45 448 3640 739 0] max score 1.01, MB/sec: 21.8 rd, 21.8 wr, level 3, files in(3, 1) out(4) MB in(213.3, 44.8) out(258.1), read-write-amplify(2.4) write-amplify(1.2) OK, records in: 9434639, records dropped: 0 > >2018-07-11 04:55:59.515630 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:55:59.515402) EVENT_LOG_v1 {"time_micros": 1531284959515391, "job": 2764, "event": "compaction_finished", "compaction_time_micros": 12438764, "output_level": 3, "num_output_files": 4, "total_output_size": 270604341, "num_input_records": 9434639, "num_output_records": 9434639, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 6, 45, 448, 3640, 739, 0]} >2018-07-11 04:55:59.516522 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284959516517, "job": 2764, "event": "table_file_deletion", "file_number": 35822} >2018-07-11 04:55:59.516547 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284959516543, "job": 2764, "event": "table_file_deletion", "file_number": 35821} >2018-07-11 04:55:59.516573 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284959516569, "job": 2764, "event": "table_file_deletion", "file_number": 35820} >2018-07-11 04:55:59.516598 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284959516595, "job": 2764, "event": "table_file_deletion", "file_number": 35803} >2018-07-11 04:55:59.516896 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2765] Compacting 1@2 + 17@3 files to L3, score 1.01 >2018-07-11 04:55:59.516918 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2764 Base level 2, inputs: [35815(71MB)], [35017(332KB) 35018(71MB) 35021(71MB) 35022(71MB) 35023(71MB) 35024(7106B) 35025(71MB) 35026(71MB) 35027(71MB) 35028(1064B) 35029(71MB) 35030(71MB) 35031(38KB) 35032(71MB) 35033(1064B) 35035(71MB) 35037(71MB)] > >2018-07-11 04:55:59.516940 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284959516924, "job": 2765, "event": "compaction_started", "files_L2": [35815], "files_L3": [35017, 35018, 35021, 35022, 35023, 35024, 35025, 35026, 35027, 35028, 35029, 35030, 35031, 35032, 35033, 35035, 35037], "score": 1.01414, "input_data_size": 969381443} >2018-07-11 04:55:59.741553 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2765] Generated table #35828: 14910 keys, 422222 bytes >2018-07-11 04:55:59.741583 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284959741569, "cf_name": "default", "job": 2765, "event": "table_file_creation", "file_number": 35828, "file_size": 422222, "table_properties": {"data_size": 379304, "index_size": 3825, "filter_size": 38115, "raw_key_size": 430729, "raw_average_key_size": 28, "raw_value_size": 177259, "raw_average_value_size": 11, "num_data_blocks": 94, "num_entries": 14910, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:56:03.001937 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2765] Generated table #35829: 2638241 keys, 74545810 bytes >2018-07-11 04:56:03.001981 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284963001964, "cf_name": "default", "job": 2765, "event": "table_file_creation", "file_number": 35829, "file_size": 74545810, "table_properties": {"data_size": 67112102, "index_size": 689535, "filter_size": 6743183, "raw_key_size": 76215860, "raw_average_key_size": 28, "raw_value_size": 31365763, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:56:06.113473 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2765] Generated table #35830: 2638240 keys, 74545852 bytes >2018-07-11 04:56:06.113523 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284966113490, "cf_name": "default", "job": 2765, "event": "table_file_creation", "file_number": 35830, "file_size": 74545852, "table_properties": {"data_size": 67112113, "index_size": 689568, "filter_size": 6743181, "raw_key_size": 76215828, "raw_average_key_size": 28, "raw_value_size": 31365748, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:56:09.342652 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2765] Generated table #35831: 2635616 keys, 74471603 bytes >2018-07-11 04:56:09.342705 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284969342688, "cf_name": "default", "job": 2765, "event": "table_file_creation", "file_number": 35831, "file_size": 74471603, "table_properties": {"data_size": 67045311, "index_size": 688840, "filter_size": 6736462, "raw_key_size": 76140017, "raw_average_key_size": 28, "raw_value_size": 31334545, "raw_average_value_size": 11, "num_data_blocks": 16473, "num_entries": 2635616, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:56:12.481781 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2765] Generated table #35832: 2638239 keys, 74545883 bytes >2018-07-11 04:56:12.481826 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284972481799, "cf_name": "default", "job": 2765, "event": "table_file_creation", "file_number": 35832, "file_size": 74545883, "table_properties": {"data_size": 67112147, "index_size": 689567, "filter_size": 6743179, "raw_key_size": 76215795, "raw_average_key_size": 28, "raw_value_size": 31365732, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638239, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:56:12.632737 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2765] Generated table #35833: 216 keys, 7106 bytes >2018-07-11 04:56:12.632762 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284972632750, "cf_name": "default", "job": 2765, "event": "table_file_creation", "file_number": 35833, "file_size": 7106, "table_properties": {"data_size": 5513, "index_size": 68, "filter_size": 555, "raw_key_size": 6241, "raw_average_key_size": 28, "raw_value_size": 2569, "raw_average_value_size": 11, "num_data_blocks": 2, "num_entries": 216, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:56:15.866807 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2765] Generated table #35834: 2638241 keys, 74545739 bytes >2018-07-11 04:56:15.866851 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284975866824, "cf_name": "default", "job": 2765, "event": "table_file_creation", "file_number": 35834, "file_size": 74545739, "table_properties": {"data_size": 67112145, "index_size": 689421, "filter_size": 6743183, "raw_key_size": 76215856, "raw_average_key_size": 28, "raw_value_size": 31365759, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:56:18.999757 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2765] Generated table #35835: 2638241 keys, 74545991 bytes >2018-07-11 04:56:18.999789 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284978999774, "cf_name": "default", "job": 2765, "event": "table_file_creation", "file_number": 35835, "file_size": 74545991, "table_properties": {"data_size": 67112163, "index_size": 689655, "filter_size": 6743183, "raw_key_size": 76215859, "raw_average_key_size": 28, "raw_value_size": 31365762, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:56:21.905798 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2765] Generated table #35836: 2638241 keys, 74546015 bytes >2018-07-11 04:56:21.905844 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284981905816, "cf_name": "default", "job": 2765, "event": "table_file_creation", "file_number": 35836, "file_size": 74546015, "table_properties": {"data_size": 67112188, "index_size": 689654, "filter_size": 6743183, "raw_key_size": 76215856, "raw_average_key_size": 28, "raw_value_size": 31365759, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:56:22.065998 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2765] Generated table #35837: 1 keys, 1064 bytes >2018-07-11 04:56:22.066025 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284982066013, "cf_name": "default", "job": 2765, "event": "table_file_creation", "file_number": 35837, "file_size": 1064, "table_properties": {"data_size": 57, "index_size": 27, "filter_size": 18, "raw_key_size": 29, "raw_average_key_size": 29, "raw_value_size": 12, "raw_average_value_size": 12, "num_data_blocks": 1, "num_entries": 1, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:56:25.106062 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2765] Generated table #35838: 2638238 keys, 74545661 bytes >2018-07-11 04:56:25.106111 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284985106095, "cf_name": "default", "job": 2765, "event": "table_file_creation", "file_number": 35838, "file_size": 74545661, "table_properties": {"data_size": 67112106, "index_size": 689388, "filter_size": 6743177, "raw_key_size": 76215775, "raw_average_key_size": 28, "raw_value_size": 31365729, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638238, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:56:28.066426 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2765] Generated table #35839: 2638241 keys, 74545823 bytes >2018-07-11 04:56:28.066471 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284988066444, "cf_name": "default", "job": 2765, "event": "table_file_creation", "file_number": 35839, "file_size": 74545823, "table_properties": {"data_size": 67112094, "index_size": 689556, "filter_size": 6743183, "raw_key_size": 76215853, "raw_average_key_size": 28, "raw_value_size": 31365756, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:56:28.224242 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2765] Generated table #35840: 1360 keys, 39415 bytes >2018-07-11 04:56:28.224274 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284988224262, "cf_name": "default", "job": 2765, "event": "table_file_creation", "file_number": 35840, "file_size": 39415, "table_properties": {"data_size": 34610, "index_size": 352, "filter_size": 3478, "raw_key_size": 39288, "raw_average_key_size": 28, "raw_value_size": 16168, "raw_average_value_size": 11, "num_data_blocks": 9, "num_entries": 1360, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:56:31.544129 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2765] Generated table #35841: 2638238 keys, 74545750 bytes >2018-07-11 04:56:31.544177 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284991544148, "cf_name": "default", "job": 2765, "event": "table_file_creation", "file_number": 35841, "file_size": 74545750, "table_properties": {"data_size": 67112088, "index_size": 689495, "filter_size": 6743177, "raw_key_size": 76215767, "raw_average_key_size": 28, "raw_value_size": 31365721, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638238, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:56:31.690895 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2765] Generated table #35842: 1 keys, 1064 bytes >2018-07-11 04:56:31.690918 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284991690907, "cf_name": "default", "job": 2765, "event": "table_file_creation", "file_number": 35842, "file_size": 1064, "table_properties": {"data_size": 57, "index_size": 27, "filter_size": 18, "raw_key_size": 29, "raw_average_key_size": 29, "raw_value_size": 12, "raw_average_value_size": 12, "num_data_blocks": 1, "num_entries": 1, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:56:34.845893 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2765] Generated table #35843: 2638239 keys, 74546019 bytes >2018-07-11 04:56:34.845940 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284994845910, "cf_name": "default", "job": 2765, "event": "table_file_creation", "file_number": 35843, "file_size": 74546019, "table_properties": {"data_size": 67112105, "index_size": 689745, "filter_size": 6743179, "raw_key_size": 76215804, "raw_average_key_size": 28, "raw_value_size": 31365741, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638239, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:56:37.855678 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2765] Generated table #35844: 2637761 keys, 74532474 bytes >2018-07-11 04:56:37.855724 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531284997855697, "cf_name": "default", "job": 2765, "event": "table_file_creation", "file_number": 35844, "file_size": 74532474, "table_properties": {"data_size": 67099940, "index_size": 689587, "filter_size": 6741957, "raw_key_size": 76201980, "raw_average_key_size": 28, "raw_value_size": 31360043, "raw_average_value_size": 11, "num_data_blocks": 16487, "num_entries": 2637761, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:56:40.790508 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2765] Generated table #35845: 2634759 keys, 74447661 bytes >2018-07-11 04:56:40.790556 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285000790541, "cf_name": "default", "job": 2765, "event": "table_file_creation", "file_number": 35845, "file_size": 74447661, "table_properties": {"data_size": 67023588, "index_size": 688808, "filter_size": 6734275, "raw_key_size": 76115266, "raw_average_key_size": 28, "raw_value_size": 31324363, "raw_average_value_size": 11, "num_data_blocks": 16468, "num_entries": 2634759, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:56:40.790618 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2765] Compacted 1@2 + 17@3 files to L3 => 969381152 bytes >2018-07-11 04:56:40.806719 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:56:40.806318) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 6 44 449 3640 739 0] max score 1.01, MB/sec: 23.5 rd, 23.5 wr, level 3, files in(1, 17) out(18) MB in(71.1, 853.4) out(924.5), read-write-amplify(26.0) write-amplify(13.0) OK, records in: 34307023, records dropped: 0 > >2018-07-11 04:56:40.806731 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:56:40.806346) EVENT_LOG_v1 {"time_micros": 1531285000806333, "job": 2765, "event": "compaction_finished", "compaction_time_micros": 41273633, "output_level": 3, "num_output_files": 18, "total_output_size": 969381152, "num_input_records": 34307023, "num_output_records": 34307023, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 6, 44, 449, 3640, 739, 0]} >2018-07-11 04:56:40.807696 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285000807679, "job": 2765, "event": "table_file_deletion", "file_number": 35815} >2018-07-11 04:56:40.807713 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285000807710, "job": 2765, "event": "table_file_deletion", "file_number": 35037} >2018-07-11 04:56:40.807727 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285000807724, "job": 2765, "event": "table_file_deletion", "file_number": 35035} >2018-07-11 04:56:40.807738 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285000807736, "job": 2765, "event": "table_file_deletion", "file_number": 35033} >2018-07-11 04:56:40.807749 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285000807747, "job": 2765, "event": "table_file_deletion", "file_number": 35032} >2018-07-11 04:56:40.807764 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285000807761, "job": 2765, "event": "table_file_deletion", "file_number": 35031} >2018-07-11 04:56:40.807777 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285000807775, "job": 2765, "event": "table_file_deletion", "file_number": 35030} >2018-07-11 04:56:40.807792 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285000807789, "job": 2765, "event": "table_file_deletion", "file_number": 35029} >2018-07-11 04:56:40.807805 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285000807803, "job": 2765, "event": "table_file_deletion", "file_number": 35028} >2018-07-11 04:56:40.807819 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285000807816, "job": 2765, "event": "table_file_deletion", "file_number": 35027} >2018-07-11 04:56:40.807833 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285000807830, "job": 2765, "event": "table_file_deletion", "file_number": 35026} >2018-07-11 04:56:40.807843 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285000807841, "job": 2765, "event": "table_file_deletion", "file_number": 35025} >2018-07-11 04:56:40.807857 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285000807854, "job": 2765, "event": "table_file_deletion", "file_number": 35024} >2018-07-11 04:56:40.807870 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285000807868, "job": 2765, "event": "table_file_deletion", "file_number": 35023} >2018-07-11 04:56:40.807883 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285000807881, "job": 2765, "event": "table_file_deletion", "file_number": 35022} >2018-07-11 04:56:40.807896 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285000807894, "job": 2765, "event": "table_file_deletion", "file_number": 35021} >2018-07-11 04:56:40.807909 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285000807907, "job": 2765, "event": "table_file_deletion", "file_number": 35018} >2018-07-11 04:56:40.807922 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285000807920, "job": 2765, "event": "table_file_deletion", "file_number": 35017} >2018-07-11 04:56:40.808228 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2766] Compacting 1@3 + 2@4 files to L4, score 1.01 >2018-07-11 04:56:40.808240 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2765 Base level 3, inputs: [35824(71MB)], [35800(71MB) 35802(71MB)] > >2018-07-11 04:56:40.808259 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285000808243, "job": 2766, "event": "compaction_started", "files_L3": [35824], "files_L4": [35800, 35802], "score": 1.00846, "input_data_size": 223638880} >2018-07-11 04:56:44.074440 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2766] Generated table #35846: 2638241 keys, 74546096 bytes >2018-07-11 04:56:44.074485 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285004074458, "cf_name": "default", "job": 2766, "event": "table_file_creation", "file_number": 35846, "file_size": 74546096, "table_properties": {"data_size": 67112161, "index_size": 689762, "filter_size": 6743183, "raw_key_size": 76215851, "raw_average_key_size": 28, "raw_value_size": 31365754, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:56:47.111400 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2766] Generated table #35847: 2638240 keys, 74546807 bytes >2018-07-11 04:56:47.111433 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285007111418, "cf_name": "default", "job": 2766, "event": "table_file_creation", "file_number": 35847, "file_size": 74546807, "table_properties": {"data_size": 67112128, "index_size": 690508, "filter_size": 6743181, "raw_key_size": 76215824, "raw_average_key_size": 28, "raw_value_size": 31365744, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:56:50.037708 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2766] Generated table #35848: 2638242 keys, 74546400 bytes >2018-07-11 04:56:50.037752 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285010037725, "cf_name": "default", "job": 2766, "event": "table_file_creation", "file_number": 35848, "file_size": 74546400, "table_properties": {"data_size": 67112172, "index_size": 690052, "filter_size": 6743186, "raw_key_size": 76215886, "raw_average_key_size": 28, "raw_value_size": 31365772, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638242, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:56:50.037787 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2766] Compacted 1@3 + 2@4 files to L4 => 223639303 bytes >2018-07-11 04:56:50.056502 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:56:50.056218) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 6 44 448 3641 739 0] max score 1.01, MB/sec: 24.2 rd, 24.2 wr, level 4, files in(1, 2) out(3) MB in(71.1, 142.2) out(213.3), read-write-amplify(6.0) write-amplify(3.0) OK, records in: 7914723, records dropped: 0 > >2018-07-11 04:56:50.056526 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:56:50.056244) EVENT_LOG_v1 {"time_micros": 1531285010056233, "job": 2766, "event": "compaction_finished", "compaction_time_micros": 9229513, "output_level": 4, "num_output_files": 3, "total_output_size": 223639303, "num_input_records": 7914723, "num_output_records": 7914723, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 6, 44, 448, 3641, 739, 0]} >2018-07-11 04:56:50.057451 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285010057447, "job": 2766, "event": "table_file_deletion", "file_number": 35824} >2018-07-11 04:56:50.057470 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285010057468, "job": 2766, "event": "table_file_deletion", "file_number": 35802} >2018-07-11 04:56:50.057484 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285010057482, "job": 2766, "event": "table_file_deletion", "file_number": 35800} >2018-07-11 04:56:50.072968 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:56:50.057787) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #35826 to level-4 74546260 bytes > >2018-07-11 04:56:50.072978 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:56:50.072877) EVENT_LOG_v1 {"time_micros": 1531285010072871, "job": 2767, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 74546260} >2018-07-11 04:56:50.072981 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:56:50.072884) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-4 74546260 bytes OK: base level 1 max bytes base 268435456 files[0 6 44 447 3642 739 0] max score 1.00 > >2018-07-11 04:56:50.073833 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2768] Compacting 1@3 + 4@4 files to L4, score 1.00 >2018-07-11 04:56:50.073845 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2767 Base level 3, inputs: [35713(71MB)], [35593(71MB) 35599(71MB) 35600(71MB) 35595(71MB)] > >2018-07-11 04:56:50.073870 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285010073848, "job": 2768, "event": "compaction_started", "files_L3": [35713], "files_L4": [35593, 35599, 35600, 35595], "score": 1.00291, "input_data_size": 372731976} >2018-07-11 04:56:53.276917 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2768] Generated table #35849: 2638241 keys, 74546037 bytes >2018-07-11 04:56:53.276965 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285013276936, "cf_name": "default", "job": 2768, "event": "table_file_creation", "file_number": 35849, "file_size": 74546037, "table_properties": {"data_size": 67112149, "index_size": 689715, "filter_size": 6743183, "raw_key_size": 76215845, "raw_average_key_size": 28, "raw_value_size": 31365748, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:56:54.036952 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:684] reusing log 35804 from recycle list > >2018-07-11 04:56:54.037108 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #35851. Immutable memtables: 0. > >2018-07-11 04:56:54.108878 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:56:54.037141) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 0 >2018-07-11 04:56:54.108893 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 2769] Flushing memtable with next log file: 35851 > >2018-07-11 04:56:54.108911 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285014108901, "job": 2769, "event": "flush_started", "num_memtables": 1, "num_entries": 5000015, "num_deletes": 0, "memory_usage": 292944824} >2018-07-11 04:56:54.108914 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 2769] Level-0 flush table #35852: started >2018-07-11 04:56:56.435831 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2768] Generated table #35850: 2638241 keys, 74546447 bytes >2018-07-11 04:56:56.435883 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285016435866, "cf_name": "default", "job": 2768, "event": "table_file_creation", "file_number": 35850, "file_size": 74546447, "table_properties": {"data_size": 67112109, "index_size": 690165, "filter_size": 6743183, "raw_key_size": 76215859, "raw_average_key_size": 28, "raw_value_size": 31365762, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:56:58.109713 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285018109693, "cf_name": "default", "job": 2769, "event": "table_file_creation", "file_number": 35852, "file_size": 141286889, "table_properties": {"data_size": 127197777, "index_size": 1308397, "filter_size": 12779725, "raw_key_size": 144445101, "raw_average_key_size": 28, "raw_value_size": 59450671, "raw_average_value_size": 11, "num_data_blocks": 31252, "num_entries": 5000014, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:56:58.109733 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 2769] Level-0 flush table #35852: 141286889 bytes OK >2018-07-11 04:56:58.122880 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:242] adding log 35806 to recycle list > >2018-07-11 04:56:58.122893 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:56:58.109758) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #35852 started >2018-07-11 04:56:58.122897 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:56:58.122737) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #35852: memtable #1 done >2018-07-11 04:56:58.122899 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:56:58.122755) EVENT_LOG_v1 {"time_micros": 1531285018122748, "job": 2769, "event": "flush_finished", "lsm_state": [1, 6, 44, 447, 3642, 739, 0], "immutable_memtables": 0} >2018-07-11 04:56:58.122901 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:56:58.122773) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 6 44 447 3642 739 0] max score 1.00 > >2018-07-11 04:56:58.124006 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 2769] Try to delete WAL files size 178896755, prev total WAL file size 178896767, number of live WAL files 2. > >2018-07-11 04:56:59.550222 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2768] Generated table #35853: 2638241 keys, 74546919 bytes >2018-07-11 04:56:59.550265 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285019550244, "cf_name": "default", "job": 2768, "event": "table_file_creation", "file_number": 35853, "file_size": 74546919, "table_properties": {"data_size": 67112166, "index_size": 690580, "filter_size": 6743183, "raw_key_size": 76215848, "raw_average_key_size": 28, "raw_value_size": 31365751, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:57:02.660549 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2768] Generated table #35854: 2638241 keys, 74546074 bytes >2018-07-11 04:57:02.660592 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285022660565, "cf_name": "default", "job": 2768, "event": "table_file_creation", "file_number": 35854, "file_size": 74546074, "table_properties": {"data_size": 67112152, "index_size": 689749, "filter_size": 6743183, "raw_key_size": 76215859, "raw_average_key_size": 28, "raw_value_size": 31365762, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:57:05.673941 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2768] Generated table #35855: 2638241 keys, 74546212 bytes >2018-07-11 04:57:05.673984 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285025673958, "cf_name": "default", "job": 2768, "event": "table_file_creation", "file_number": 35855, "file_size": 74546212, "table_properties": {"data_size": 67112126, "index_size": 689913, "filter_size": 6743183, "raw_key_size": 76215852, "raw_average_key_size": 28, "raw_value_size": 31365755, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:57:05.674021 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2768] Compacted 1@3 + 4@4 files to L4 => 372731689 bytes >2018-07-11 04:57:05.689729 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:57:05.689411) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[1 6 44 446 3643 739 0] max score 1.00, MB/sec: 23.9 rd, 23.9 wr, level 4, files in(1, 4) out(5) MB in(71.1, 284.4) out(355.5), read-write-amplify(10.0) write-amplify(5.0) OK, records in: 13191205, records dropped: 0 > >2018-07-11 04:57:05.689740 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:57:05.689436) EVENT_LOG_v1 {"time_micros": 1531285025689426, "job": 2768, "event": "compaction_finished", "compaction_time_micros": 15600137, "output_level": 4, "num_output_files": 5, "total_output_size": 372731689, "num_input_records": 13191205, "num_output_records": 13191205, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 6, 44, 446, 3643, 739, 0]} >2018-07-11 04:57:05.690690 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285025690687, "job": 2768, "event": "table_file_deletion", "file_number": 35713} >2018-07-11 04:57:05.690717 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285025690715, "job": 2768, "event": "table_file_deletion", "file_number": 35600} >2018-07-11 04:57:05.690731 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285025690729, "job": 2768, "event": "table_file_deletion", "file_number": 35599} >2018-07-11 04:57:05.690743 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285025690741, "job": 2768, "event": "table_file_deletion", "file_number": 35595} >2018-07-11 04:57:05.690755 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285025690752, "job": 2768, "event": "table_file_deletion", "file_number": 35593} >2018-07-11 04:57:05.706179 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:57:05.691057) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #35853 to level-5 74546919 bytes > >2018-07-11 04:57:05.706189 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:57:05.706089) EVENT_LOG_v1 {"time_micros": 1531285025706083, "job": 2770, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546919} >2018-07-11 04:57:05.706192 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:57:05.706096) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546919 bytes OK: base level 1 max bytes base 268435456 files[1 6 44 446 3642 740 0] max score 1.00 > >2018-07-11 04:57:05.722859 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:57:05.707033) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #27881 to level-5 74546851 bytes > >2018-07-11 04:57:05.722869 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:57:05.722770) EVENT_LOG_v1 {"time_micros": 1531285025722764, "job": 2771, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546851} >2018-07-11 04:57:05.722872 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:57:05.722777) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546851 bytes OK: base level 1 max bytes base 268435456 files[1 6 44 446 3641 741 0] max score 1.00 > >2018-07-11 04:57:05.739552 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:57:05.723722) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #7408 to level-5 74546851 bytes > >2018-07-11 04:57:05.739562 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:57:05.739463) EVENT_LOG_v1 {"time_micros": 1531285025739458, "job": 2772, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546851} >2018-07-11 04:57:05.739565 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:57:05.739471) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546851 bytes OK: base level 1 max bytes base 268435456 files[1 6 44 446 3640 742 0] max score 1.00 > >2018-07-11 04:57:05.756240 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:57:05.740390) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #35825 to level-4 74546179 bytes > >2018-07-11 04:57:05.756258 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:57:05.756152) EVENT_LOG_v1 {"time_micros": 1531285025756147, "job": 2773, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 74546179} >2018-07-11 04:57:05.756261 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:57:05.756160) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-4 74546179 bytes OK: base level 1 max bytes base 268435456 files[1 6 44 445 3641 742 0] max score 1.00 > >2018-07-11 04:57:05.772942 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:57:05.757087) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #3328 to level-5 74546851 bytes > >2018-07-11 04:57:05.772952 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:57:05.772852) EVENT_LOG_v1 {"time_micros": 1531285025772847, "job": 2774, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546851} >2018-07-11 04:57:05.772954 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:57:05.772860) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546851 bytes OK: base level 1 max bytes base 268435456 files[1 6 44 445 3640 743 0] max score 1.00 > >2018-07-11 04:58:58.098169 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:684] reusing log 35806 from recycle list > >2018-07-11 04:58:58.098295 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #35856. Immutable memtables: 0. > >2018-07-11 04:58:58.185685 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:58:58.098354) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1 >2018-07-11 04:58:58.185700 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 2775] Flushing memtable with next log file: 35856 > >2018-07-11 04:58:58.185718 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285138185709, "job": 2775, "event": "flush_started", "num_memtables": 1, "num_entries": 5000015, "num_deletes": 0, "memory_usage": 292945944} >2018-07-11 04:58:58.185721 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 2775] Level-0 flush table #35857: started >2018-07-11 04:59:02.198766 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285142198746, "cf_name": "default", "job": 2775, "event": "table_file_creation", "file_number": 35857, "file_size": 141286222, "table_properties": {"data_size": 127196937, "index_size": 1308576, "filter_size": 12779719, "raw_key_size": 144445077, "raw_average_key_size": 28, "raw_value_size": 59449772, "raw_average_value_size": 11, "num_data_blocks": 31252, "num_entries": 5000013, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:59:02.198789 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 2775] Level-0 flush table #35857: 141286222 bytes OK >2018-07-11 04:59:02.228020 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:242] adding log 35851 to recycle list > >2018-07-11 04:59:02.228035 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:59:02.198813) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #35857 started >2018-07-11 04:59:02.228053 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:59:02.227851) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #35857: memtable #1 done >2018-07-11 04:59:02.228058 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:59:02.227870) EVENT_LOG_v1 {"time_micros": 1531285142227861, "job": 2775, "event": "flush_finished", "lsm_state": [2, 6, 44, 445, 3640, 743, 0], "immutable_memtables": 0} >2018-07-11 04:59:02.228062 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-04:59:02.227955) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[2 6 44 445 3640 743 0] max score 1.05 > >2018-07-11 04:59:02.228201 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2776] Compacting 2@0 + 2@1 files to L1, score 1.05 >2018-07-11 04:59:02.228218 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2775 Base level 0, inputs: [35857(134MB) 35852(134MB)], [35808(71MB) 35813(65MB)] > >2018-07-11 04:59:02.228236 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285142228223, "job": 2776, "event": "compaction_started", "files_L0": [35857, 35852], "files_L1": [35808, 35813], "score": 1.05267, "input_data_size": 425328556} >2018-07-11 04:59:02.229223 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 2775] Try to delete WAL files size 178896755, prev total WAL file size 178896767, number of live WAL files 2. > >2018-07-11 04:59:05.974551 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2776] Generated table #35858: 2623563 keys, 74508710 bytes >2018-07-11 04:59:05.974599 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285145974572, "cf_name": "default", "job": 2776, "event": "table_file_creation", "file_number": 35858, "file_size": 74508710, "table_properties": {"data_size": 67109841, "index_size": 691382, "filter_size": 6706497, "raw_key_size": 75824504, "raw_average_key_size": 28, "raw_value_size": 31554358, "raw_average_value_size": 12, "num_data_blocks": 16475, "num_entries": 2623563, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:59:09.591207 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2776] Generated table #35859: 2638241 keys, 74546868 bytes >2018-07-11 04:59:09.591256 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285149591224, "cf_name": "default", "job": 2776, "event": "table_file_creation", "file_number": 35859, "file_size": 74546868, "table_properties": {"data_size": 67112119, "index_size": 690576, "filter_size": 6743183, "raw_key_size": 76215845, "raw_average_key_size": 28, "raw_value_size": 31365748, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:59:13.475278 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2776] Generated table #35860: 2638241 keys, 74546503 bytes >2018-07-11 04:59:13.475322 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285153475295, "cf_name": "default", "job": 2776, "event": "table_file_creation", "file_number": 35860, "file_size": 74546503, "table_properties": {"data_size": 67112153, "index_size": 690177, "filter_size": 6743183, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:59:16.523869 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2776] Generated table #35861: 2638241 keys, 74546405 bytes >2018-07-11 04:59:16.523914 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285156523886, "cf_name": "default", "job": 2776, "event": "table_file_creation", "file_number": 35861, "file_size": 74546405, "table_properties": {"data_size": 67112087, "index_size": 690145, "filter_size": 6743183, "raw_key_size": 76215853, "raw_average_key_size": 28, "raw_value_size": 31365756, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:59:18.672326 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2776] Generated table #35862: 2085397 keys, 58924958 bytes >2018-07-11 04:59:18.672372 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285158672357, "cf_name": "default", "job": 2776, "event": "table_file_creation", "file_number": 35862, "file_size": 58924958, "table_properties": {"data_size": 53048796, "index_size": 545034, "filter_size": 5330140, "raw_key_size": 60244815, "raw_average_key_size": 28, "raw_value_size": 24793066, "raw_average_value_size": 11, "num_data_blocks": 13034, "num_entries": 2085397, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:59:19.762530 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2776] Generated table #35863: 166172 keys, 68244399 bytes >2018-07-11 04:59:19.762571 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285159762544, "cf_name": "default", "job": 2776, "event": "table_file_creation", "file_number": 35863, "file_size": 68244399, "table_properties": {"data_size": 67109534, "index_size": 564867, "filter_size": 569010, "raw_key_size": 15375312, "raw_average_key_size": 92, "raw_value_size": 53111612, "raw_average_value_size": 319, "num_data_blocks": 15214, "num_entries": 166172, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "80898", "kMergeOperands": "0"}} >2018-07-11 04:59:19.795222 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2776] Generated table #35864: 10 keys, 6152 bytes >2018-07-11 04:59:19.795263 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285159795233, "cf_name": "default", "job": 2776, "event": "table_file_creation", "file_number": 35864, "file_size": 6152, "table_properties": {"data_size": 5079, "index_size": 60, "filter_size": 46, "raw_key_size": 862, "raw_average_key_size": 86, "raw_value_size": 4572, "raw_average_value_size": 457, "num_data_blocks": 2, "num_entries": 10, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:59:19.795319 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2776] Compacted 2@0 + 2@1 files to L1 => 425323995 bytes >2018-07-11 04:59:19.811263 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:59:19.810954) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 11 44 445 3640 743 0] max score 1.80, MB/sec: 24.2 rd, 24.2 wr, level 1, files in(2, 2) out(7) MB in(269.5, 136.1) out(405.6), read-write-amplify(3.0) write-amplify(1.5) OK, records in: 12789872, records dropped: 7 > >2018-07-11 04:59:19.811273 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:59:19.810979) EVENT_LOG_v1 {"time_micros": 1531285159810968, "job": 2776, "event": "compaction_finished", "compaction_time_micros": 17567065, "output_level": 1, "num_output_files": 7, "total_output_size": 425323995, "num_input_records": 12789872, "num_output_records": 12789865, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 11, 44, 445, 3640, 743, 0]} >2018-07-11 04:59:19.812180 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285159812176, "job": 2776, "event": "table_file_deletion", "file_number": 35857} >2018-07-11 04:59:19.812206 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285159812204, "job": 2776, "event": "table_file_deletion", "file_number": 35852} >2018-07-11 04:59:19.812218 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285159812216, "job": 2776, "event": "table_file_deletion", "file_number": 35813} >2018-07-11 04:59:19.812229 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285159812227, "job": 2776, "event": "table_file_deletion", "file_number": 35808} >2018-07-11 04:59:19.812446 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_picker.cc:461] [default] Expanding@1 1+5(74546868+361309349 bytes) to 4+5 (282564734+361309349bytes) > >2018-07-11 04:59:19.812544 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2777] Compacting 4@1 + 5@2 files to L2, score 1.80 >2018-07-11 04:59:19.812557 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2776 Base level 1, inputs: [35859(71MB) 35860(71MB) 35861(71MB) 35862(56MB)], [35816(71MB) 35817(71MB) 35818(71MB) 35819(71MB) 35823(60MB)] > >2018-07-11 04:59:19.812571 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285159812560, "job": 2777, "event": "compaction_started", "files_L1": [35859, 35860, 35861, 35862], "files_L2": [35816, 35817, 35818, 35819, 35823], "score": 1.79596, "input_data_size": 643874083} >2018-07-11 04:59:23.024461 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2777] Generated table #35865: 2638239 keys, 74546240 bytes >2018-07-11 04:59:23.024510 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285163024488, "cf_name": "default", "job": 2777, "event": "table_file_creation", "file_number": 35865, "file_size": 74546240, "table_properties": {"data_size": 67112122, "index_size": 689949, "filter_size": 6743179, "raw_key_size": 76215787, "raw_average_key_size": 28, "raw_value_size": 31365724, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638239, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:59:26.200526 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2777] Generated table #35866: 2638239 keys, 74546281 bytes >2018-07-11 04:59:26.200581 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285166200564, "cf_name": "default", "job": 2777, "event": "table_file_creation", "file_number": 35866, "file_size": 74546281, "table_properties": {"data_size": 67112108, "index_size": 690004, "filter_size": 6743179, "raw_key_size": 76215791, "raw_average_key_size": 28, "raw_value_size": 31365728, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638239, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:59:29.424121 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2777] Generated table #35867: 2638241 keys, 74546345 bytes >2018-07-11 04:59:29.424161 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285169424141, "cf_name": "default", "job": 2777, "event": "table_file_creation", "file_number": 35867, "file_size": 74546345, "table_properties": {"data_size": 67112157, "index_size": 690014, "filter_size": 6743184, "raw_key_size": 76215852, "raw_average_key_size": 28, "raw_value_size": 31365755, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:59:32.615555 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2777] Generated table #35868: 2638240 keys, 74546979 bytes >2018-07-11 04:59:32.615601 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285172615574, "cf_name": "default", "job": 2777, "event": "table_file_creation", "file_number": 35868, "file_size": 74546979, "table_properties": {"data_size": 67112151, "index_size": 690657, "filter_size": 6743181, "raw_key_size": 76215825, "raw_average_key_size": 28, "raw_value_size": 31365745, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:59:36.184743 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2777] Generated table #35869: 2638240 keys, 74546329 bytes >2018-07-11 04:59:36.184796 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285176184779, "cf_name": "default", "job": 2777, "event": "table_file_creation", "file_number": 35869, "file_size": 74546329, "table_properties": {"data_size": 67112082, "index_size": 690076, "filter_size": 6743181, "raw_key_size": 76215824, "raw_average_key_size": 28, "raw_value_size": 31365744, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:59:39.311672 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2777] Generated table #35870: 2638241 keys, 74545997 bytes >2018-07-11 04:59:39.311723 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285179311707, "cf_name": "default", "job": 2777, "event": "table_file_creation", "file_number": 35870, "file_size": 74545997, "table_properties": {"data_size": 67112160, "index_size": 689664, "filter_size": 6743183, "raw_key_size": 76215850, "raw_average_key_size": 28, "raw_value_size": 31365753, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:59:42.536139 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2777] Generated table #35871: 2638241 keys, 74546349 bytes >2018-07-11 04:59:42.536191 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285182536163, "cf_name": "default", "job": 2777, "event": "table_file_creation", "file_number": 35871, "file_size": 74546349, "table_properties": {"data_size": 67112109, "index_size": 690067, "filter_size": 6743183, "raw_key_size": 76215859, "raw_average_key_size": 28, "raw_value_size": 31365762, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:59:45.661089 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2777] Generated table #35872: 2638241 keys, 74546886 bytes >2018-07-11 04:59:45.661137 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285185661121, "cf_name": "default", "job": 2777, "event": "table_file_creation", "file_number": 35872, "file_size": 74546886, "table_properties": {"data_size": 67112171, "index_size": 690542, "filter_size": 6743183, "raw_key_size": 76215853, "raw_average_key_size": 28, "raw_value_size": 31365756, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:59:47.874275 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2777] Generated table #35873: 1676727 keys, 47503643 bytes >2018-07-11 04:59:47.874333 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285187874301, "cf_name": "default", "job": 2777, "event": "table_file_creation", "file_number": 35873, "file_size": 47503643, "table_properties": {"data_size": 42777493, "index_size": 439263, "filter_size": 4285899, "raw_key_size": 48454536, "raw_average_key_size": 28, "raw_value_size": 20039799, "raw_average_value_size": 11, "num_data_blocks": 10512, "num_entries": 1676727, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "11", "kMergeOperands": "0"}} >2018-07-11 04:59:47.874381 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2777] Compacted 4@1 + 5@2 files to L2 => 643875049 bytes >2018-07-11 04:59:47.894072 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:59:47.893745) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 7 48 445 3640 743 0] max score 1.09, MB/sec: 22.9 rd, 22.9 wr, level 2, files in(4, 5) out(9) MB in(269.5, 344.6) out(614.0), read-write-amplify(4.6) write-amplify(2.3) OK, records in: 22782649, records dropped: 0 > >2018-07-11 04:59:47.894084 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:59:47.893776) EVENT_LOG_v1 {"time_micros": 1531285187893762, "job": 2777, "event": "compaction_finished", "compaction_time_micros": 28061784, "output_level": 2, "num_output_files": 9, "total_output_size": 643875049, "num_input_records": 22782649, "num_output_records": 22782649, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 7, 48, 445, 3640, 743, 0]} >2018-07-11 04:59:47.895296 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285187895291, "job": 2777, "event": "table_file_deletion", "file_number": 35862} >2018-07-11 04:59:47.895327 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285187895324, "job": 2777, "event": "table_file_deletion", "file_number": 35861} >2018-07-11 04:59:47.895344 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285187895341, "job": 2777, "event": "table_file_deletion", "file_number": 35860} >2018-07-11 04:59:47.895362 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285187895358, "job": 2777, "event": "table_file_deletion", "file_number": 35859} >2018-07-11 04:59:47.895391 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285187895388, "job": 2777, "event": "table_file_deletion", "file_number": 35823} >2018-07-11 04:59:47.895416 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285187895413, "job": 2777, "event": "table_file_deletion", "file_number": 35819} >2018-07-11 04:59:47.895439 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285187895435, "job": 2777, "event": "table_file_deletion", "file_number": 35818} >2018-07-11 04:59:47.895461 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285187895458, "job": 2777, "event": "table_file_deletion", "file_number": 35817} >2018-07-11 04:59:47.895482 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285187895479, "job": 2777, "event": "table_file_deletion", "file_number": 35816} >2018-07-11 04:59:47.895823 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2778] Compacting 1@2 + 1@3 files to L3, score 1.09 >2018-07-11 04:59:47.895837 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2777 Base level 2, inputs: [35868(71MB)], [35775(71MB)] > >2018-07-11 04:59:47.895857 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285187895843, "job": 2778, "event": "compaction_started", "files_L2": [35868], "files_L3": [35775], "score": 1.09163, "input_data_size": 149092980} >2018-07-11 04:59:51.108476 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2778] Generated table #35874: 2638241 keys, 74546265 bytes >2018-07-11 04:59:51.108511 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285191108495, "cf_name": "default", "job": 2778, "event": "table_file_creation", "file_number": 35874, "file_size": 74546265, "table_properties": {"data_size": 67112175, "index_size": 689917, "filter_size": 6743183, "raw_key_size": 76215853, "raw_average_key_size": 28, "raw_value_size": 31365756, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:59:54.137552 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2778] Generated table #35875: 2638240 keys, 74546979 bytes >2018-07-11 04:59:54.137605 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285194137589, "cf_name": "default", "job": 2778, "event": "table_file_creation", "file_number": 35875, "file_size": 74546979, "table_properties": {"data_size": 67112151, "index_size": 690657, "filter_size": 6743181, "raw_key_size": 76215825, "raw_average_key_size": 28, "raw_value_size": 31365745, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 04:59:54.137640 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2778] Compacted 1@2 + 1@3 files to L3 => 149093244 bytes >2018-07-11 04:59:54.152286 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:59:54.152012) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 7 47 446 3640 743 0] max score 1.06, MB/sec: 23.9 rd, 23.9 wr, level 3, files in(1, 1) out(2) MB in(71.1, 71.1) out(142.2), read-write-amplify(4.0) write-amplify(2.0) OK, records in: 5276481, records dropped: 0 > >2018-07-11 04:59:54.152298 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-04:59:54.152038) EVENT_LOG_v1 {"time_micros": 1531285194152027, "job": 2778, "event": "compaction_finished", "compaction_time_micros": 6241766, "output_level": 3, "num_output_files": 2, "total_output_size": 149093244, "num_input_records": 5276481, "num_output_records": 5276481, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 7, 47, 446, 3640, 743, 0]} >2018-07-11 04:59:54.153231 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285194153225, "job": 2778, "event": "table_file_deletion", "file_number": 35868} >2018-07-11 04:59:54.153262 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285194153258, "job": 2778, "event": "table_file_deletion", "file_number": 35775} >2018-07-11 04:59:54.153510 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_picker.cc:461] [default] Expanding@2 1+1(74546886+46965534 bytes) to 3+1 (223639232+46965534bytes) > >2018-07-11 04:59:54.153621 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2779] Compacting 3@2 + 1@3 files to L3, score 1.06 >2018-07-11 04:59:54.153635 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2778 Base level 2, inputs: [35870(71MB) 35871(71MB) 35872(71MB)], [35827(44MB)] > >2018-07-11 04:59:54.153655 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285194153640, "job": 2779, "event": "compaction_started", "files_L2": [35870, 35871, 35872], "files_L3": [35827], "score": 1.06386, "input_data_size": 270604766} >2018-07-11 04:59:57.302584 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2779] Generated table #35876: 2638241 keys, 74546679 bytes >2018-07-11 04:59:57.302630 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285197302602, "cf_name": "default", "job": 2779, "event": "table_file_creation", "file_number": 35876, "file_size": 74546679, "table_properties": {"data_size": 67112177, "index_size": 690329, "filter_size": 6743183, "raw_key_size": 76215862, "raw_average_key_size": 28, "raw_value_size": 31365765, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:00:00.459445 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2779] Generated table #35877: 2638241 keys, 74546939 bytes >2018-07-11 05:00:00.459477 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285200459463, "cf_name": "default", "job": 2779, "event": "table_file_creation", "file_number": 35877, "file_size": 74546939, "table_properties": {"data_size": 67112116, "index_size": 690650, "filter_size": 6743183, "raw_key_size": 76215850, "raw_average_key_size": 28, "raw_value_size": 31365753, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:00:03.630238 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2779] Generated table #35878: 2638241 keys, 74546242 bytes >2018-07-11 05:00:03.630287 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285203630260, "cf_name": "default", "job": 2779, "event": "table_file_creation", "file_number": 35878, "file_size": 74546242, "table_properties": {"data_size": 67112162, "index_size": 689907, "filter_size": 6743183, "raw_key_size": 76215855, "raw_average_key_size": 28, "raw_value_size": 31365758, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:00:05.486668 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2779] Generated table #35879: 1519916 keys, 46965185 bytes >2018-07-11 05:00:05.486721 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285205486704, "cf_name": "default", "job": 2779, "event": "table_file_creation", "file_number": 35879, "file_size": 46965185, "table_properties": {"data_size": 42636636, "index_size": 433410, "filter_size": 3894151, "raw_key_size": 44348936, "raw_average_key_size": 29, "raw_value_size": 21522465, "raw_average_value_size": 14, "num_data_blocks": 10538, "num_entries": 1519916, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:00:05.486756 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2779] Compacted 3@2 + 1@3 files to L3 => 270605045 bytes >2018-07-11 05:00:05.502018 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:00:05.501810) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 7 44 449 3640 743 0] max score 1.01, MB/sec: 23.9 rd, 23.9 wr, level 3, files in(3, 1) out(4) MB in(213.3, 44.8) out(258.1), read-write-amplify(2.4) write-amplify(1.2) OK, records in: 9434639, records dropped: 0 > >2018-07-11 05:00:05.502030 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:00:05.501834) EVENT_LOG_v1 {"time_micros": 1531285205501824, "job": 2779, "event": "compaction_finished", "compaction_time_micros": 11333085, "output_level": 3, "num_output_files": 4, "total_output_size": 270605045, "num_input_records": 9434639, "num_output_records": 9434639, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 7, 44, 449, 3640, 743, 0]} >2018-07-11 05:00:05.502981 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285205502965, "job": 2779, "event": "table_file_deletion", "file_number": 35872} >2018-07-11 05:00:05.502997 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285205502994, "job": 2779, "event": "table_file_deletion", "file_number": 35871} >2018-07-11 05:00:05.503010 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285205503008, "job": 2779, "event": "table_file_deletion", "file_number": 35870} >2018-07-11 05:00:05.503023 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285205503021, "job": 2779, "event": "table_file_deletion", "file_number": 35827} >2018-07-11 05:00:05.503339 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2780] Compacting 1@3 + 1@4 files to L4, score 1.01 >2018-07-11 05:00:05.503351 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2779 Base level 3, inputs: [35875(71MB)], [35846(71MB)] > >2018-07-11 05:00:05.503364 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285205503354, "job": 2780, "event": "compaction_started", "files_L3": [35875], "files_L4": [35846], "score": 1.00846, "input_data_size": 149093075} >2018-07-11 05:00:08.618219 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2780] Generated table #35880: 2638241 keys, 74546459 bytes >2018-07-11 05:00:08.618272 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285208618255, "cf_name": "default", "job": 2780, "event": "table_file_creation", "file_number": 35880, "file_size": 74546459, "table_properties": {"data_size": 67112140, "index_size": 690146, "filter_size": 6743183, "raw_key_size": 76215859, "raw_average_key_size": 28, "raw_value_size": 31365762, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:00:11.801706 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2780] Generated table #35881: 2638240 keys, 74546116 bytes >2018-07-11 05:00:11.801757 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285211801729, "cf_name": "default", "job": 2780, "event": "table_file_creation", "file_number": 35881, "file_size": 74546116, "table_properties": {"data_size": 67112076, "index_size": 689880, "filter_size": 6743170, "raw_key_size": 76215817, "raw_average_key_size": 28, "raw_value_size": 31365737, "raw_average_value_size": 11, "num_data_blocks": 16489, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:00:11.801791 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2780] Compacted 1@3 + 1@4 files to L4 => 149092575 bytes >2018-07-11 05:00:11.818721 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:00:11.818438) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 7 44 448 3641 743 0] max score 1.01, MB/sec: 23.7 rd, 23.7 wr, level 4, files in(1, 1) out(2) MB in(71.1, 71.1) out(142.2), read-write-amplify(4.0) write-amplify(2.0) OK, records in: 5276481, records dropped: 0 > >2018-07-11 05:00:11.818732 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:00:11.818464) EVENT_LOG_v1 {"time_micros": 1531285211818454, "job": 2780, "event": "compaction_finished", "compaction_time_micros": 6298412, "output_level": 4, "num_output_files": 2, "total_output_size": 149092575, "num_input_records": 5276481, "num_output_records": 5276481, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 7, 44, 448, 3641, 743, 0]} >2018-07-11 05:00:11.819706 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285211819700, "job": 2780, "event": "table_file_deletion", "file_number": 35875} >2018-07-11 05:00:11.819733 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285211819729, "job": 2780, "event": "table_file_deletion", "file_number": 35846} >2018-07-11 05:00:11.835290 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:00:11.820038) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #35877 to level-4 74546939 bytes > >2018-07-11 05:00:11.835301 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:00:11.835082) EVENT_LOG_v1 {"time_micros": 1531285211835076, "job": 2781, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 74546939} >2018-07-11 05:00:11.835304 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:00:11.835090) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-4 74546939 bytes OK: base level 1 max bytes base 268435456 files[0 7 44 447 3642 743 0] max score 1.00 > >2018-07-11 05:00:11.851862 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:00:11.836154) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #35876 to level-4 74546679 bytes > >2018-07-11 05:00:11.851873 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:00:11.851768) EVENT_LOG_v1 {"time_micros": 1531285211851762, "job": 2782, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 74546679} >2018-07-11 05:00:11.851876 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:00:11.851776) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-4 74546679 bytes OK: base level 1 max bytes base 268435456 files[0 7 44 446 3643 743 0] max score 1.00 > >2018-07-11 05:00:11.894950 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:00:11.852750) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #35877 to level-5 74546939 bytes > >2018-07-11 05:00:11.894965 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:00:11.894696) EVENT_LOG_v1 {"time_micros": 1531285211894688, "job": 2783, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546939} >2018-07-11 05:00:11.894970 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:00:11.894705) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546939 bytes OK: base level 1 max bytes base 268435456 files[0 7 44 446 3642 744 0] max score 1.00 > >2018-07-11 05:00:11.911465 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:00:11.896026) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #20794 to level-5 74546850 bytes > >2018-07-11 05:00:11.911476 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:00:11.911371) EVENT_LOG_v1 {"time_micros": 1531285211911365, "job": 2784, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546850} >2018-07-11 05:00:11.911480 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:00:11.911379) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546850 bytes OK: base level 1 max bytes base 268435456 files[0 7 44 446 3641 745 0] max score 1.00 > >2018-07-11 05:00:11.928158 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:00:11.912343) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #8818 to level-5 74546850 bytes > >2018-07-11 05:00:11.928169 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:00:11.928066) EVENT_LOG_v1 {"time_micros": 1531285211928060, "job": 2785, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546850} >2018-07-11 05:00:11.928172 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:00:11.928074) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546850 bytes OK: base level 1 max bytes base 268435456 files[0 7 44 446 3640 746 0] max score 1.00 > >2018-07-11 05:00:11.929017 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2786] Compacting 1@3 + 1@4 files to L4, score 1.00 >2018-07-11 05:00:11.929028 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2785 Base level 3, inputs: [35874(71MB)], [35773(71MB)] > >2018-07-11 05:00:11.929056 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285211929044, "job": 2786, "event": "compaction_started", "files_L3": [35874], "files_L4": [35773], "score": 1.00013, "input_data_size": 149092650} >2018-07-11 05:00:15.214985 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2786] Generated table #35882: 2638241 keys, 74546374 bytes >2018-07-11 05:00:15.215029 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285215215002, "cf_name": "default", "job": 2786, "event": "table_file_creation", "file_number": 35882, "file_size": 74546374, "table_properties": {"data_size": 67112138, "index_size": 690063, "filter_size": 6743183, "raw_key_size": 76215855, "raw_average_key_size": 28, "raw_value_size": 31365758, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:00:18.166163 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2786] Generated table #35883: 2638240 keys, 74547011 bytes >2018-07-11 05:00:18.166208 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285218166191, "cf_name": "default", "job": 2786, "event": "table_file_creation", "file_number": 35883, "file_size": 74547011, "table_properties": {"data_size": 67112078, "index_size": 690762, "filter_size": 6743181, "raw_key_size": 76215826, "raw_average_key_size": 28, "raw_value_size": 31365746, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:00:18.416618 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2786] Generated table #35884: 1 keys, 1064 bytes >2018-07-11 05:00:18.416676 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285218416661, "cf_name": "default", "job": 2786, "event": "table_file_creation", "file_number": 35884, "file_size": 1064, "table_properties": {"data_size": 57, "index_size": 27, "filter_size": 18, "raw_key_size": 29, "raw_average_key_size": 29, "raw_value_size": 12, "raw_average_value_size": 12, "num_data_blocks": 1, "num_entries": 1, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:00:18.416720 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2786] Compacted 1@3 + 1@4 files to L4 => 149094449 bytes >2018-07-11 05:00:18.436660 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:00:18.436407) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 7 44 445 3642 746 0] max score 1.00, MB/sec: 23.0 rd, 23.0 wr, level 4, files in(1, 1) out(3) MB in(71.1, 71.1) out(142.2), read-write-amplify(4.0) write-amplify(2.0) OK, records in: 5276482, records dropped: 0 > >2018-07-11 05:00:18.436671 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:00:18.436434) EVENT_LOG_v1 {"time_micros": 1531285218436423, "job": 2786, "event": "compaction_finished", "compaction_time_micros": 6487639, "output_level": 4, "num_output_files": 3, "total_output_size": 149094449, "num_input_records": 5276482, "num_output_records": 5276482, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 7, 44, 445, 3642, 746, 0]} >2018-07-11 05:00:18.437672 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285218437668, "job": 2786, "event": "table_file_deletion", "file_number": 35874} >2018-07-11 05:00:18.437687 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285218437685, "job": 2786, "event": "table_file_deletion", "file_number": 35773} >2018-07-11 05:00:18.437999 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2787] Compacting 1@4 + 1@5 files to L5, score 1.00 >2018-07-11 05:00:18.438011 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2786 Base level 4, inputs: [35883(71MB)], [35774(71MB)] > >2018-07-11 05:00:18.438026 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285218438016, "job": 2787, "event": "compaction_started", "files_L4": [35883], "files_L5": [35774], "score": 1.00027, "input_data_size": 149094007} >2018-07-11 05:00:21.677013 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2787] Generated table #35885: 2638241 keys, 74546967 bytes >2018-07-11 05:00:21.677056 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285221677030, "cf_name": "default", "job": 2787, "event": "table_file_creation", "file_number": 35885, "file_size": 74546967, "table_properties": {"data_size": 67112101, "index_size": 690693, "filter_size": 6743183, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:00:24.658412 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2787] Generated table #35886: 2638240 keys, 74546144 bytes >2018-07-11 05:00:24.658456 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285224658440, "cf_name": "default", "job": 2787, "event": "table_file_creation", "file_number": 35886, "file_size": 74546144, "table_properties": {"data_size": 67112108, "index_size": 689876, "filter_size": 6743170, "raw_key_size": 76215824, "raw_average_key_size": 28, "raw_value_size": 31365744, "raw_average_value_size": 11, "num_data_blocks": 16489, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:00:24.658494 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2787] Compacted 1@4 + 1@5 files to L5 => 149093111 bytes >2018-07-11 05:00:24.869945 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:00:24.869753) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 7 44 445 3641 747 0] max score 1.00, MB/sec: 24.0 rd, 24.0 wr, level 5, files in(1, 1) out(2) MB in(71.1, 71.1) out(142.2), read-write-amplify(4.0) write-amplify(2.0) OK, records in: 5276481, records dropped: 0 > >2018-07-11 05:00:24.869956 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:00:24.869781) EVENT_LOG_v1 {"time_micros": 1531285224869770, "job": 2787, "event": "compaction_finished", "compaction_time_micros": 6220454, "output_level": 5, "num_output_files": 2, "total_output_size": 149093111, "num_input_records": 5276481, "num_output_records": 5276481, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 7, 44, 445, 3641, 747, 0]} >2018-07-11 05:00:24.870927 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285224870910, "job": 2787, "event": "table_file_deletion", "file_number": 35883} >2018-07-11 05:00:24.870944 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285224870941, "job": 2787, "event": "table_file_deletion", "file_number": 35774} >2018-07-11 05:01:01.892953 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:684] reusing log 35851 from recycle list > >2018-07-11 05:01:01.893085 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #35887. Immutable memtables: 0. > >2018-07-11 05:01:01.968704 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:01:01.893128) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1 >2018-07-11 05:01:01.968725 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 2788] Flushing memtable with next log file: 35887 > >2018-07-11 05:01:01.968746 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285261968735, "job": 2788, "event": "flush_started", "num_memtables": 1, "num_entries": 5000015, "num_deletes": 0, "memory_usage": 292908256} >2018-07-11 05:01:01.968749 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 2788] Level-0 flush table #35888: started >2018-07-11 05:01:06.562100 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285266562080, "cf_name": "default", "job": 2788, "event": "table_file_creation", "file_number": 35888, "file_size": 141286889, "table_properties": {"data_size": 127197777, "index_size": 1308397, "filter_size": 12779725, "raw_key_size": 144445101, "raw_average_key_size": 28, "raw_value_size": 59450671, "raw_average_value_size": 11, "num_data_blocks": 31252, "num_entries": 5000014, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:01:06.562121 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 2788] Level-0 flush table #35888: 141286889 bytes OK >2018-07-11 05:01:06.577135 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:242] adding log 35856 to recycle list > >2018-07-11 05:01:06.577147 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:01:06.562146) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #35888 started >2018-07-11 05:01:06.577150 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:01:06.576936) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #35888: memtable #1 done >2018-07-11 05:01:06.577152 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:01:06.576954) EVENT_LOG_v1 {"time_micros": 1531285266576947, "job": 2788, "event": "flush_finished", "lsm_state": [1, 7, 44, 445, 3641, 747, 0], "immutable_memtables": 0} >2018-07-11 05:01:06.577154 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:01:06.577083) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 7 44 445 3641 747 0] max score 1.00 > >2018-07-11 05:01:06.578055 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 2788] Try to delete WAL files size 178896755, prev total WAL file size 178896767, number of live WAL files 2. > >2018-07-11 05:04:32.577104 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:684] reusing log 35856 from recycle list > >2018-07-11 05:04:32.577201 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #35889. Immutable memtables: 0. > >2018-07-11 05:04:32.634928 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:04:32.577237) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1 >2018-07-11 05:04:32.634943 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 2789] Flushing memtable with next log file: 35889 > >2018-07-11 05:04:32.634964 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285472634953, "job": 2789, "event": "flush_started", "num_memtables": 1, "num_entries": 5000015, "num_deletes": 0, "memory_usage": 292941464} >2018-07-11 05:04:32.634969 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 2789] Level-0 flush table #35890: started >2018-07-11 05:04:36.492187 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285476492166, "cf_name": "default", "job": 2789, "event": "table_file_creation", "file_number": 35890, "file_size": 141286889, "table_properties": {"data_size": 127197777, "index_size": 1308397, "filter_size": 12779725, "raw_key_size": 144445101, "raw_average_key_size": 28, "raw_value_size": 59450671, "raw_average_value_size": 11, "num_data_blocks": 31252, "num_entries": 5000014, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:04:36.492209 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 2789] Level-0 flush table #35890: 141286889 bytes OK >2018-07-11 05:04:36.513105 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:242] adding log 35887 to recycle list > >2018-07-11 05:04:36.513129 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:04:36.492234) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #35890 started >2018-07-11 05:04:36.513134 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:04:36.512885) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #35890: memtable #1 done >2018-07-11 05:04:36.513136 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:04:36.512902) EVENT_LOG_v1 {"time_micros": 1531285476512895, "job": 2789, "event": "flush_finished", "lsm_state": [2, 7, 44, 445, 3641, 747, 0], "immutable_memtables": 0} >2018-07-11 05:04:36.513138 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:04:36.513040) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[2 7 44 445 3641 747 0] max score 1.05 > >2018-07-11 05:04:36.513265 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2790] Compacting 2@0 + 2@1 files to L1, score 1.05 >2018-07-11 05:04:36.513281 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2789 Base level 0, inputs: [35890(134MB) 35888(134MB)], [35858(71MB) 35863(65MB)] > >2018-07-11 05:04:36.513299 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285476513285, "job": 2790, "event": "compaction_started", "files_L0": [35890, 35888], "files_L1": [35858, 35863], "score": 1.05267, "input_data_size": 425326887} >2018-07-11 05:04:36.514015 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 2789] Try to delete WAL files size 178896755, prev total WAL file size 178896767, number of live WAL files 2. > >2018-07-11 05:04:39.521525 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2790] Generated table #35891: 2623533 keys, 74509512 bytes >2018-07-11 05:04:39.521572 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285479521557, "cf_name": "default", "job": 2790, "event": "table_file_creation", "file_number": 35891, "file_size": 74509512, "table_properties": {"data_size": 67110704, "index_size": 691397, "filter_size": 6706421, "raw_key_size": 75823849, "raw_average_key_size": 28, "raw_value_size": 31555503, "raw_average_value_size": 12, "num_data_blocks": 16475, "num_entries": 2623533, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:04:42.240059 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2790] Generated table #35892: 2638241 keys, 74546891 bytes >2018-07-11 05:04:42.240103 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285482240076, "cf_name": "default", "job": 2790, "event": "table_file_creation", "file_number": 35892, "file_size": 74546891, "table_properties": {"data_size": 67112120, "index_size": 690598, "filter_size": 6743183, "raw_key_size": 76215846, "raw_average_key_size": 28, "raw_value_size": 31365749, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:04:44.906471 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2790] Generated table #35893: 2638241 keys, 74546491 bytes >2018-07-11 05:04:44.906518 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285484906490, "cf_name": "default", "job": 2790, "event": "table_file_creation", "file_number": 35893, "file_size": 74546491, "table_properties": {"data_size": 67112154, "index_size": 690164, "filter_size": 6743183, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:04:47.594117 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2790] Generated table #35894: 2638241 keys, 74546381 bytes >2018-07-11 05:04:47.594162 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285487594136, "cf_name": "default", "job": 2790, "event": "table_file_creation", "file_number": 35894, "file_size": 74546381, "table_properties": {"data_size": 67112084, "index_size": 690124, "filter_size": 6743183, "raw_key_size": 76215850, "raw_average_key_size": 28, "raw_value_size": 31365753, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:04:49.767017 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2790] Generated table #35895: 2085317 keys, 58922727 bytes >2018-07-11 05:04:49.767060 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285489767034, "cf_name": "default", "job": 2790, "event": "table_file_creation", "file_number": 35895, "file_size": 58922727, "table_properties": {"data_size": 53046767, "index_size": 545032, "filter_size": 5329940, "raw_key_size": 60242505, "raw_average_key_size": 28, "raw_value_size": 24792116, "raw_average_value_size": 11, "num_data_blocks": 13034, "num_entries": 2085317, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:04:50.916947 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2790] Generated table #35896: 166182 keys, 68248648 bytes >2018-07-11 05:04:50.916999 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285490916982, "cf_name": "default", "job": 2790, "event": "table_file_creation", "file_number": 35896, "file_size": 68248648, "table_properties": {"data_size": 67113716, "index_size": 564899, "filter_size": 569045, "raw_key_size": 15375922, "raw_average_key_size": 92, "raw_value_size": 53115242, "raw_average_value_size": 319, "num_data_blocks": 15215, "num_entries": 166182, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "80898", "kMergeOperands": "0"}} >2018-07-11 05:04:50.917079 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2790] Compacted 2@0 + 2@1 files to L1 => 425320650 bytes >2018-07-11 05:04:50.937964 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:04:50.937637) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 11 44 445 3641 747 0] max score 1.80, MB/sec: 29.5 rd, 29.5 wr, level 1, files in(2, 2) out(6) MB in(269.5, 136.1) out(405.6), read-write-amplify(3.0) write-amplify(1.5) OK, records in: 12789763, records dropped: 8 > >2018-07-11 05:04:50.937978 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:04:50.937672) EVENT_LOG_v1 {"time_micros": 1531285490937659, "job": 2790, "event": "compaction_finished", "compaction_time_micros": 14403762, "output_level": 1, "num_output_files": 6, "total_output_size": 425320650, "num_input_records": 12789763, "num_output_records": 12789755, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 11, 44, 445, 3641, 747, 0]} >2018-07-11 05:04:50.938931 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285490938914, "job": 2790, "event": "table_file_deletion", "file_number": 35890} >2018-07-11 05:04:50.938946 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285490938943, "job": 2790, "event": "table_file_deletion", "file_number": 35888} >2018-07-11 05:04:50.938958 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285490938956, "job": 2790, "event": "table_file_deletion", "file_number": 35863} >2018-07-11 05:04:50.938970 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285490938968, "job": 2790, "event": "table_file_deletion", "file_number": 35858} >2018-07-11 05:04:50.939194 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_picker.cc:461] [default] Expanding@1 1+5(74546891+345688838 bytes) to 4+5 (282562490+345688838bytes) > >2018-07-11 05:04:50.939303 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2791] Compacting 4@1 + 5@2 files to L2, score 1.80 >2018-07-11 05:04:50.939317 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2790 Base level 1, inputs: [35892(71MB) 35893(71MB) 35894(71MB) 35895(56MB)], [35865(71MB) 35866(71MB) 35867(71MB) 35869(71MB) 35873(45MB)] > >2018-07-11 05:04:50.939333 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285490939321, "job": 2791, "event": "compaction_started", "files_L1": [35892, 35893, 35894, 35895], "files_L2": [35865, 35866, 35867, 35869, 35873], "score": 1.79598, "input_data_size": 628251328} >2018-07-11 05:04:54.329792 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2791] Generated table #35897: 2638240 keys, 74545922 bytes >2018-07-11 05:04:54.329839 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285494329811, "cf_name": "default", "job": 2791, "event": "table_file_creation", "file_number": 35897, "file_size": 74545922, "table_properties": {"data_size": 67112142, "index_size": 689609, "filter_size": 6743181, "raw_key_size": 76215815, "raw_average_key_size": 28, "raw_value_size": 31365735, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:04:57.623388 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2791] Generated table #35898: 2638239 keys, 74546030 bytes >2018-07-11 05:04:57.623434 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285497623407, "cf_name": "default", "job": 2791, "event": "table_file_creation", "file_number": 35898, "file_size": 74546030, "table_properties": {"data_size": 67112085, "index_size": 689776, "filter_size": 6743179, "raw_key_size": 76215790, "raw_average_key_size": 28, "raw_value_size": 31365727, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638239, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:05:00.668275 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2791] Generated table #35899: 2638240 keys, 74546144 bytes >2018-07-11 05:05:00.668321 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285500668294, "cf_name": "default", "job": 2791, "event": "table_file_creation", "file_number": 35899, "file_size": 74546144, "table_properties": {"data_size": 67112137, "index_size": 689836, "filter_size": 6743181, "raw_key_size": 76215824, "raw_average_key_size": 28, "raw_value_size": 31365744, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:05:03.932613 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2791] Generated table #35900: 2638239 keys, 74546349 bytes >2018-07-11 05:05:03.932658 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285503932631, "cf_name": "default", "job": 2791, "event": "table_file_creation", "file_number": 35900, "file_size": 74546349, "table_properties": {"data_size": 67112083, "index_size": 690097, "filter_size": 6743179, "raw_key_size": 76215799, "raw_average_key_size": 28, "raw_value_size": 31365736, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638239, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:05:06.964028 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2791] Generated table #35901: 2638241 keys, 74546885 bytes >2018-07-11 05:05:06.964072 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285506964045, "cf_name": "default", "job": 2791, "event": "table_file_creation", "file_number": 35901, "file_size": 74546885, "table_properties": {"data_size": 67112128, "index_size": 690584, "filter_size": 6743183, "raw_key_size": 76215855, "raw_average_key_size": 28, "raw_value_size": 31365758, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:05:09.944597 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2791] Generated table #35902: 2638241 keys, 74546242 bytes >2018-07-11 05:05:09.944653 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285509944636, "cf_name": "default", "job": 2791, "event": "table_file_creation", "file_number": 35902, "file_size": 74546242, "table_properties": {"data_size": 67112153, "index_size": 689916, "filter_size": 6743183, "raw_key_size": 76215847, "raw_average_key_size": 28, "raw_value_size": 31365750, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:05:13.311886 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2791] Generated table #35903: 2638241 keys, 74546373 bytes >2018-07-11 05:05:13.311932 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285513311906, "cf_name": "default", "job": 2791, "event": "table_file_creation", "file_number": 35903, "file_size": 74546373, "table_properties": {"data_size": 67112160, "index_size": 690040, "filter_size": 6743183, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:05:16.392507 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2791] Generated table #35904: 2638241 keys, 74545905 bytes >2018-07-11 05:05:16.392553 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285516392526, "cf_name": "default", "job": 2791, "event": "table_file_creation", "file_number": 35904, "file_size": 74545905, "table_properties": {"data_size": 67112128, "index_size": 689604, "filter_size": 6743183, "raw_key_size": 76215862, "raw_average_key_size": 28, "raw_value_size": 31365765, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:05:17.746107 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2791] Generated table #35905: 1123804 keys, 31880813 bytes >2018-07-11 05:05:17.746148 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285517746122, "cf_name": "default", "job": 2791, "event": "table_file_creation", "file_number": 35905, "file_size": 31880813, "table_properties": {"data_size": 28712123, "index_size": 295028, "filter_size": 2872674, "raw_key_size": 32481199, "raw_average_key_size": 28, "raw_value_size": 13466153, "raw_average_value_size": 11, "num_data_blocks": 7057, "num_entries": 1123804, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "11", "kMergeOperands": "0"}} >2018-07-11 05:05:17.746184 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2791] Compacted 4@1 + 5@2 files to L2 => 628250663 bytes >2018-07-11 05:05:17.762580 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:05:17.762213) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 7 48 445 3641 747 0] max score 1.09, MB/sec: 23.4 rd, 23.4 wr, level 2, files in(4, 5) out(9) MB in(269.5, 329.7) out(599.1), read-write-amplify(4.4) write-amplify(2.2) OK, records in: 22229726, records dropped: 0 > >2018-07-11 05:05:17.762591 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:05:17.762238) EVENT_LOG_v1 {"time_micros": 1531285517762227, "job": 2791, "event": "compaction_finished", "compaction_time_micros": 26806831, "output_level": 2, "num_output_files": 9, "total_output_size": 628250663, "num_input_records": 22229726, "num_output_records": 22229726, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 7, 48, 445, 3641, 747, 0]} >2018-07-11 05:05:17.763513 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285517763509, "job": 2791, "event": "table_file_deletion", "file_number": 35895} >2018-07-11 05:05:17.763526 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285517763524, "job": 2791, "event": "table_file_deletion", "file_number": 35894} >2018-07-11 05:05:17.763537 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285517763535, "job": 2791, "event": "table_file_deletion", "file_number": 35893} >2018-07-11 05:05:17.763548 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285517763546, "job": 2791, "event": "table_file_deletion", "file_number": 35892} >2018-07-11 05:05:17.763566 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285517763564, "job": 2791, "event": "table_file_deletion", "file_number": 35873} >2018-07-11 05:05:17.763579 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285517763577, "job": 2791, "event": "table_file_deletion", "file_number": 35869} >2018-07-11 05:05:17.763592 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285517763590, "job": 2791, "event": "table_file_deletion", "file_number": 35867} >2018-07-11 05:05:17.763617 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285517763615, "job": 2791, "event": "table_file_deletion", "file_number": 35866} >2018-07-11 05:05:17.763638 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285517763635, "job": 2791, "event": "table_file_deletion", "file_number": 35865} >2018-07-11 05:05:17.764197 7f41641eb700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS ------- >2018-07-11 05:05:17.764203 7f41641eb700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] >** DB Stats ** >Uptime(secs): 50259.4 total, 613.0 interval >Cumulative writes: 196K writes, 1860M keys, 196K commit groups, 1.0 writes per commit group, ingest: 62.50 GB, 1.27 MB/s >Cumulative WAL: 196K writes, 56K syncs, 3.48 writes per sync, written: 62.50 GB, 1.27 MB/s >Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent >Interval writes: 46 writes, 23M keys, 46 commit groups, 1.0 writes per commit group, ingest: 784.80 MB, 1.28 MB/s >Interval WAL: 46 writes, 23 syncs, 1.92 writes per sync, written: 0.77 MB, 1.28 MB/s >Interval stall: 00:00:0.000 H:M:S, 0.0 percent > >** Compaction Stats [default] ** >Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop >---------------------------------------------------------------------------------------------------------------------------------------------------------- > L0 0/0 0.00 KB 0.0 0.0 0.0 0.0 48.9 48.9 0.0 1.0 0.0 32.4 1544 373 4.140 0 0 > L1 7/0 190.30 MB 0.7 80.7 49.0 31.7 80.7 48.9 0.0 1.6 25.7 25.7 3212 184 17.457 2555M 894K > L2 48/0 2.71 GB 1.1 95.2 49.0 46.3 95.2 48.9 0.0 1.9 21.5 21.5 4527 556 8.142 3554M 1358K > L3 445/0 24.93 GB 1.0 85.1 48.1 37.1 85.1 48.0 0.6 1.8 20.8 20.7 4199 291 14.430 3216M 4000K > L4 3641/0 250.00 GB 1.0 73.4 17.6 55.8 73.1 17.3 31.0 4.2 21.2 21.1 3551 236 15.046 2792M 15M > L5 747/0 51.84 GB 0.0 1.9 1.0 0.9 1.9 0.9 47.3 1.9 19.0 18.8 101 13 7.782 71M 1097K > Sum 4888/0 329.67 GB 0.0 336.4 164.6 171.8 384.7 212.9 78.9 7.9 20.1 23.0 17134 1653 10.366 12G 23M > Int 0/0 0.00 KB 0.0 5.5 2.5 3.0 6.0 3.0 0.8 11.5 21.6 23.7 261 19 13.723 202M 22 >Uptime(secs): 50259.4 total, 50259.4 interval >Flush(GB): cumulative 48.866, interval 0.526 >AddFile(GB): cumulative 0.000, interval 0.000 >AddFile(Total Files): cumulative 0, interval 0 >AddFile(L0 Files): cumulative 0, interval 0 >AddFile(Keys): cumulative 0, interval 0 >Cumulative compaction: 384.69 GB write, 7.84 MB/s write, 336.35 GB read, 6.85 MB/s read, 17134.5 seconds >Interval compaction: 6.03 GB write, 0.12 MB/s write, 5.50 GB read, 0.11 MB/s read, 260.7 seconds >Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count > >** File Read Latency Histogram By Level [default] ** > >** Compaction Stats [default] ** >Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop >---------------------------------------------------------------------------------------------------------------------------------------------------------- > L0 0/0 0.00 KB 0.0 0.0 0.0 0.0 48.9 48.9 0.0 1.0 0.0 32.4 1544 373 4.140 0 0 > L1 7/0 190.30 MB 0.7 80.7 49.0 31.7 80.7 48.9 0.0 1.6 25.7 25.7 3212 184 17.457 2555M 894K > L2 48/0 2.71 GB 1.1 95.2 49.0 46.3 95.2 48.9 0.0 1.9 21.5 21.5 4527 556 8.142 3554M 1358K > L3 445/0 24.93 GB 1.0 85.1 48.1 37.1 85.1 48.0 0.6 1.8 20.8 20.7 4199 291 14.430 3216M 4000K > L4 3641/0 250.00 GB 1.0 73.4 17.6 55.8 73.1 17.3 31.0 4.2 21.2 21.1 3551 236 15.046 2792M 15M > L5 747/0 51.84 GB 0.0 1.9 1.0 0.9 1.9 0.9 47.3 1.9 19.0 18.8 101 13 7.782 71M 1097K > Sum 4888/0 329.67 GB 0.0 336.4 164.6 171.8 384.7 212.9 78.9 7.9 20.1 23.0 17134 1653 10.366 12G 23M > Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 >Uptime(secs): 50259.4 total, 50259.4 interval >Flush(GB): cumulative 48.866, interval 0.000 >AddFile(GB): cumulative 0.000, interval 0.000 >AddFile(Total Files): cumulative 0, interval 0 >AddFile(L0 Files): cumulative 0, interval 0 >AddFile(Keys): cumulative 0, interval 0 >Cumulative compaction: 384.69 GB write, 7.84 MB/s write, 336.35 GB read, 6.85 MB/s read, 17134.5 seconds >Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds >Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count > >** File Read Latency Histogram By Level [default] ** > >2018-07-11 05:05:17.764240 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_picker.cc:461] [default] Expanding@2 1+2(74546885+121511427 bytes) to 4+2 (298185405+121511427bytes) > >2018-07-11 05:05:17.764353 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2792] Compacting 4@2 + 2@3 files to L3, score 1.09 >2018-07-11 05:05:17.764366 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2791 Base level 2, inputs: [35901(71MB) 35902(71MB) 35903(71MB) 35904(71MB)], [35878(71MB) 35879(44MB)] > >2018-07-11 05:05:17.764380 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285517764369, "job": 2792, "event": "compaction_started", "files_L2": [35901, 35902, 35903, 35904], "files_L3": [35878, 35879], "score": 1.08581, "input_data_size": 419696832} >2018-07-11 05:05:21.003891 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2792] Generated table #35906: 2638241 keys, 74547024 bytes >2018-07-11 05:05:21.003937 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285521003909, "cf_name": "default", "job": 2792, "event": "table_file_creation", "file_number": 35906, "file_size": 74547024, "table_properties": {"data_size": 67112175, "index_size": 690676, "filter_size": 6743183, "raw_key_size": 76215853, "raw_average_key_size": 28, "raw_value_size": 31365756, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:05:23.983089 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2792] Generated table #35907: 2638241 keys, 74546053 bytes >2018-07-11 05:05:23.983135 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285523983108, "cf_name": "default", "job": 2792, "event": "table_file_creation", "file_number": 35907, "file_size": 74546053, "table_properties": {"data_size": 67112130, "index_size": 689750, "filter_size": 6743183, "raw_key_size": 76215849, "raw_average_key_size": 28, "raw_value_size": 31365752, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:05:26.964319 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2792] Generated table #35908: 2638241 keys, 74546253 bytes >2018-07-11 05:05:26.964353 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285526964337, "cf_name": "default", "job": 2792, "event": "table_file_creation", "file_number": 35908, "file_size": 74546253, "table_properties": {"data_size": 67112153, "index_size": 689927, "filter_size": 6743183, "raw_key_size": 76215859, "raw_average_key_size": 28, "raw_value_size": 31365762, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:05:29.996933 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2792] Generated table #35909: 2638241 keys, 74545702 bytes >2018-07-11 05:05:29.996976 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285529996949, "cf_name": "default", "job": 2792, "event": "table_file_creation", "file_number": 35909, "file_size": 74545702, "table_properties": {"data_size": 67112140, "index_size": 689389, "filter_size": 6743183, "raw_key_size": 76215856, "raw_average_key_size": 28, "raw_value_size": 31365759, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:05:33.025008 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2792] Generated table #35910: 2638241 keys, 74546874 bytes >2018-07-11 05:05:33.025053 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285533025026, "cf_name": "default", "job": 2792, "event": "table_file_creation", "file_number": 35910, "file_size": 74546874, "table_properties": {"data_size": 67112125, "index_size": 690576, "filter_size": 6743183, "raw_key_size": 76215851, "raw_average_key_size": 28, "raw_value_size": 31365754, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:05:34.907153 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2792] Generated table #35911: 1519916 keys, 46965201 bytes >2018-07-11 05:05:34.907197 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285534907170, "cf_name": "default", "job": 2792, "event": "table_file_creation", "file_number": 35911, "file_size": 46965201, "table_properties": {"data_size": 42636659, "index_size": 433403, "filter_size": 3894151, "raw_key_size": 44348941, "raw_average_key_size": 29, "raw_value_size": 21522470, "raw_average_value_size": 14, "num_data_blocks": 10538, "num_entries": 1519916, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:05:34.907233 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2792] Compacted 4@2 + 2@3 files to L3 => 419697107 bytes >2018-07-11 05:05:34.928778 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:05:34.928442) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 7 44 449 3641 747 0] max score 1.01, MB/sec: 24.5 rd, 24.5 wr, level 3, files in(4, 2) out(6) MB in(284.4, 115.9) out(400.3), read-write-amplify(2.8) write-amplify(1.4) OK, records in: 14711121, records dropped: 0 > >2018-07-11 05:05:34.928790 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:05:34.928469) EVENT_LOG_v1 {"time_micros": 1531285534928457, "job": 2792, "event": "compaction_finished", "compaction_time_micros": 17142836, "output_level": 3, "num_output_files": 6, "total_output_size": 419697107, "num_input_records": 14711121, "num_output_records": 14711121, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 7, 44, 449, 3641, 747, 0]} >2018-07-11 05:05:34.929719 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285534929714, "job": 2792, "event": "table_file_deletion", "file_number": 35904} >2018-07-11 05:05:34.929733 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285534929730, "job": 2792, "event": "table_file_deletion", "file_number": 35903} >2018-07-11 05:05:34.929745 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285534929743, "job": 2792, "event": "table_file_deletion", "file_number": 35902} >2018-07-11 05:05:34.929756 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285534929754, "job": 2792, "event": "table_file_deletion", "file_number": 35901} >2018-07-11 05:05:34.929770 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285534929767, "job": 2792, "event": "table_file_deletion", "file_number": 35879} >2018-07-11 05:05:34.929781 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285534929779, "job": 2792, "event": "table_file_deletion", "file_number": 35878} >2018-07-11 05:05:34.930088 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2793] Compacting 1@3 + 1@4 files to L4, score 1.01 >2018-07-11 05:05:34.930099 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2792 Base level 3, inputs: [35906(71MB)], [35876(71MB)] > >2018-07-11 05:05:34.930113 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285534930103, "job": 2793, "event": "compaction_started", "files_L3": [35906], "files_L4": [35876], "score": 1.00846, "input_data_size": 149093703} >2018-07-11 05:05:38.102309 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2793] Generated table #35912: 2638241 keys, 74546645 bytes >2018-07-11 05:05:38.102357 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285538102341, "cf_name": "default", "job": 2793, "event": "table_file_creation", "file_number": 35912, "file_size": 74546645, "table_properties": {"data_size": 67112169, "index_size": 690303, "filter_size": 6743183, "raw_key_size": 76215858, "raw_average_key_size": 28, "raw_value_size": 31365761, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:05:40.995638 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2793] Generated table #35913: 2638240 keys, 74546200 bytes >2018-07-11 05:05:40.995683 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285540995656, "cf_name": "default", "job": 2793, "event": "table_file_creation", "file_number": 35913, "file_size": 74546200, "table_properties": {"data_size": 67112078, "index_size": 689951, "filter_size": 6743181, "raw_key_size": 76215828, "raw_average_key_size": 28, "raw_value_size": 31365748, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:05:41.046033 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2793] Generated table #35914: 1 keys, 1064 bytes >2018-07-11 05:05:41.046057 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285541046046, "cf_name": "default", "job": 2793, "event": "table_file_creation", "file_number": 35914, "file_size": 1064, "table_properties": {"data_size": 57, "index_size": 27, "filter_size": 18, "raw_key_size": 29, "raw_average_key_size": 29, "raw_value_size": 12, "raw_average_value_size": 12, "num_data_blocks": 1, "num_entries": 1, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:05:41.046093 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2793] Compacted 1@3 + 1@4 files to L4 => 149093909 bytes >2018-07-11 05:05:41.062042 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:05:41.061845) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 7 44 448 3643 747 0] max score 1.01, MB/sec: 24.4 rd, 24.4 wr, level 4, files in(1, 1) out(3) MB in(71.1, 71.1) out(142.2), read-write-amplify(4.0) write-amplify(2.0) OK, records in: 5276482, records dropped: 0 > >2018-07-11 05:05:41.062053 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:05:41.061869) EVENT_LOG_v1 {"time_micros": 1531285541061859, "job": 2793, "event": "compaction_finished", "compaction_time_micros": 6115955, "output_level": 4, "num_output_files": 3, "total_output_size": 149093909, "num_input_records": 5276482, "num_output_records": 5276482, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 7, 44, 448, 3643, 747, 0]} >2018-07-11 05:05:41.062995 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285541062991, "job": 2793, "event": "table_file_deletion", "file_number": 35906} >2018-07-11 05:05:41.063023 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285541063021, "job": 2793, "event": "table_file_deletion", "file_number": 35876} >2018-07-11 05:05:41.078606 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:05:41.063322) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #35910 to level-4 74546874 bytes > >2018-07-11 05:05:41.078616 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:05:41.078515) EVENT_LOG_v1 {"time_micros": 1531285541078509, "job": 2794, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 74546874} >2018-07-11 05:05:41.078619 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:05:41.078522) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-4 74546874 bytes OK: base level 1 max bytes base 268435456 files[0 7 44 447 3644 747 0] max score 1.00 > >2018-07-11 05:05:41.095186 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:05:41.079454) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #35908 to level-4 74546253 bytes > >2018-07-11 05:05:41.095197 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:05:41.095098) EVENT_LOG_v1 {"time_micros": 1531285541095092, "job": 2795, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 74546253} >2018-07-11 05:05:41.095199 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:05:41.095106) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-4 74546253 bytes OK: base level 1 max bytes base 268435456 files[0 7 44 446 3645 747 0] max score 1.00 > >2018-07-11 05:05:41.145466 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:05:41.096057) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #35910 to level-5 74546874 bytes > >2018-07-11 05:05:41.145477 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:05:41.145269) EVENT_LOG_v1 {"time_micros": 1531285541145263, "job": 2796, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546874} >2018-07-11 05:05:41.145492 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:05:41.145277) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546874 bytes OK: base level 1 max bytes base 268435456 files[0 7 44 446 3644 748 0] max score 1.00 > >2018-07-11 05:05:41.161971 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:05:41.146329) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #35255 to level-5 74546849 bytes > >2018-07-11 05:05:41.161981 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:05:41.161880) EVENT_LOG_v1 {"time_micros": 1531285541161874, "job": 2797, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546849} >2018-07-11 05:05:41.161984 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:05:41.161887) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546849 bytes OK: base level 1 max bytes base 268435456 files[0 7 44 446 3643 749 0] max score 1.00 > >2018-07-11 05:05:41.178559 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:05:41.162853) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #29732 to level-5 74546848 bytes > >2018-07-11 05:05:41.178569 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:05:41.178461) EVENT_LOG_v1 {"time_micros": 1531285541178454, "job": 2798, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546848} >2018-07-11 05:05:41.178572 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:05:41.178470) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546848 bytes OK: base level 1 max bytes base 268435456 files[0 7 44 446 3642 750 0] max score 1.00 > >2018-07-11 05:05:41.195400 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:05:41.179417) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #35907 to level-4 74546053 bytes > >2018-07-11 05:05:41.195411 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:05:41.195312) EVENT_LOG_v1 {"time_micros": 1531285541195306, "job": 2799, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 74546053} >2018-07-11 05:05:41.195413 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:05:41.195320) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-4 74546053 bytes OK: base level 1 max bytes base 268435456 files[0 7 44 445 3643 750 0] max score 1.00 > >2018-07-11 05:05:41.211988 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:05:41.196252) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #21312 to level-5 74546848 bytes > >2018-07-11 05:05:41.211998 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:05:41.211895) EVENT_LOG_v1 {"time_micros": 1531285541211889, "job": 2800, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546848} >2018-07-11 05:05:41.212001 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:05:41.211903) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546848 bytes OK: base level 1 max bytes base 268435456 files[0 7 44 445 3642 751 0] max score 1.00 > >2018-07-11 05:06:20.855613 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:684] reusing log 35887 from recycle list > >2018-07-11 05:06:20.855778 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #35915. Immutable memtables: 0. > >2018-07-11 05:06:20.934872 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:06:20.855816) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1 >2018-07-11 05:06:20.934886 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 2801] Flushing memtable with next log file: 35915 > >2018-07-11 05:06:20.934910 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285580934898, "job": 2801, "event": "flush_started", "num_memtables": 1, "num_entries": 5000015, "num_deletes": 0, "memory_usage": 292936704} >2018-07-11 05:06:20.934916 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 2801] Level-0 flush table #35916: started >2018-07-11 05:06:23.646602 7f41649ec700 1 bluefs _allocate failed to allocate 0x8700000 on bdev 1, free 0x57fe000; fallback to bdev 2 >2018-07-11 05:06:25.635858 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285585635838, "cf_name": "default", "job": 2801, "event": "table_file_creation", "file_number": 35916, "file_size": 141286889, "table_properties": {"data_size": 127197777, "index_size": 1308397, "filter_size": 12779725, "raw_key_size": 144445101, "raw_average_key_size": 28, "raw_value_size": 59450671, "raw_average_value_size": 11, "num_data_blocks": 31252, "num_entries": 5000014, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:06:25.635885 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 2801] Level-0 flush table #35916: 141286889 bytes OK >2018-07-11 05:06:25.652520 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:242] adding log 35889 to recycle list > >2018-07-11 05:06:25.652532 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:06:25.635899) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #35916 started >2018-07-11 05:06:25.652539 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:06:25.652388) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #35916: memtable #1 done >2018-07-11 05:06:25.652543 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:06:25.652404) EVENT_LOG_v1 {"time_micros": 1531285585652397, "job": 2801, "event": "flush_finished", "lsm_state": [1, 7, 44, 445, 3642, 751, 0], "immutable_memtables": 0} >2018-07-11 05:06:25.652547 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:06:25.652466) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 7 44 445 3642 751 0] max score 1.00 > >2018-07-11 05:06:25.653441 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 2801] Try to delete WAL files size 178896755, prev total WAL file size 178896767, number of live WAL files 2. > >2018-07-11 05:08:06.279864 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:684] reusing log 35889 from recycle list > >2018-07-11 05:08:06.280015 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #35917. Immutable memtables: 0. > >2018-07-11 05:08:06.320613 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:08:06.280053) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1 >2018-07-11 05:08:06.320627 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 2802] Flushing memtable with next log file: 35917 > >2018-07-11 05:08:06.320657 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285686320644, "job": 2802, "event": "flush_started", "num_memtables": 1, "num_entries": 5000016, "num_deletes": 0, "memory_usage": 292966064} >2018-07-11 05:08:06.320662 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 2802] Level-0 flush table #35918: started >2018-07-11 05:08:08.986470 7f41649ec700 1 bluefs _allocate failed to allocate 0x8700000 on bdev 1, free 0x57fe000; fallback to bdev 2 >2018-07-11 05:08:10.785812 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285690785790, "cf_name": "default", "job": 2802, "event": "table_file_creation", "file_number": 35918, "file_size": 141288997, "table_properties": {"data_size": 127198867, "index_size": 1309411, "filter_size": 12779729, "raw_key_size": 144445142, "raw_average_key_size": 28, "raw_value_size": 59451578, "raw_average_value_size": 11, "num_data_blocks": 31252, "num_entries": 5000016, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:08:10.785836 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 2802] Level-0 flush table #35918: 141288997 bytes OK >2018-07-11 05:08:10.808283 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:242] adding log 35915 to recycle list > >2018-07-11 05:08:10.808297 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:08:10.785860) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #35918 started >2018-07-11 05:08:10.808300 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:08:10.808042) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #35918: memtable #1 done >2018-07-11 05:08:10.808302 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:08:10.808060) EVENT_LOG_v1 {"time_micros": 1531285690808053, "job": 2802, "event": "flush_finished", "lsm_state": [2, 7, 44, 445, 3642, 751, 0], "immutable_memtables": 0} >2018-07-11 05:08:10.808304 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:08:10.808196) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[2 7 44 445 3642 751 0] max score 1.05 > >2018-07-11 05:08:10.808430 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2803] Compacting 2@0 + 7@1 files to L1, score 1.05 >2018-07-11 05:08:10.808447 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2802 Base level 0, inputs: [35918(134MB) 35916(134MB)], [35891(71MB) 35896(65MB) 35864(6152B) 35814(5092B) 35790(5090B) 35761(5094B) 35729(54MB)] > >2018-07-11 05:08:10.808464 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285690808451, "job": 2803, "event": "compaction_started", "files_L0": [35918, 35916], "files_L1": [35891, 35896, 35864, 35814, 35790, 35761, 35729], "score": 1.05268, "input_data_size": 482116863} >2018-07-11 05:08:10.809177 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 2802] Try to delete WAL files size 178896775, prev total WAL file size 178896787, number of live WAL files 2. > >2018-07-11 05:08:13.938209 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2803] Generated table #35919: 2623423 keys, 74508005 bytes >2018-07-11 05:08:13.938243 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285693938228, "cf_name": "default", "job": 2803, "event": "table_file_creation", "file_number": 35919, "file_size": 74508005, "table_properties": {"data_size": 67109401, "index_size": 691466, "filter_size": 6706148, "raw_key_size": 75820882, "raw_average_key_size": 28, "raw_value_size": 31555696, "raw_average_value_size": 12, "num_data_blocks": 16475, "num_entries": 2623423, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:08:16.627444 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2803] Generated table #35920: 2638241 keys, 74546868 bytes >2018-07-11 05:08:16.627490 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285696627463, "cf_name": "default", "job": 2803, "event": "table_file_creation", "file_number": 35920, "file_size": 74546868, "table_properties": {"data_size": 67112119, "index_size": 690576, "filter_size": 6743183, "raw_key_size": 76215845, "raw_average_key_size": 28, "raw_value_size": 31365748, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:08:19.314973 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2803] Generated table #35921: 2638241 keys, 74546503 bytes >2018-07-11 05:08:19.315025 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285699315009, "cf_name": "default", "job": 2803, "event": "table_file_creation", "file_number": 35921, "file_size": 74546503, "table_properties": {"data_size": 67112153, "index_size": 690177, "filter_size": 6743183, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:08:22.885405 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2803] Generated table #35922: 2638241 keys, 74546405 bytes >2018-07-11 05:08:22.885440 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285702885426, "cf_name": "default", "job": 2803, "event": "table_file_creation", "file_number": 35922, "file_size": 74546405, "table_properties": {"data_size": 67112087, "index_size": 690145, "filter_size": 6743183, "raw_key_size": 76215853, "raw_average_key_size": 28, "raw_value_size": 31365756, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:08:25.297578 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2803] Generated table #35923: 2085397 keys, 58924958 bytes >2018-07-11 05:08:25.297625 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285705297597, "cf_name": "default", "job": 2803, "event": "table_file_creation", "file_number": 35923, "file_size": 58924958, "table_properties": {"data_size": 53048796, "index_size": 545034, "filter_size": 5330140, "raw_key_size": 60244815, "raw_average_key_size": 28, "raw_value_size": 24793066, "raw_average_value_size": 11, "num_data_blocks": 13034, "num_entries": 2085397, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:08:27.371910 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2803] Generated table #35924: 166188 keys, 68246516 bytes >2018-07-11 05:08:27.371963 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285707371946, "cf_name": "default", "job": 2803, "event": "table_file_creation", "file_number": 35924, "file_size": 68246516, "table_properties": {"data_size": 67111624, "index_size": 564852, "filter_size": 569052, "raw_key_size": 15376193, "raw_average_key_size": 92, "raw_value_size": 53112912, "raw_average_value_size": 319, "num_data_blocks": 15215, "num_entries": 166188, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "80898", "kMergeOperands": "0"}} >2018-07-11 05:08:28.434326 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2803] Generated table #35925: 138296 keys, 56785248 bytes >2018-07-11 05:08:28.434359 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285708434344, "cf_name": "default", "job": 2803, "event": "table_file_creation", "file_number": 35925, "file_size": 56785248, "table_properties": {"data_size": 55834474, "index_size": 475224, "filter_size": 474562, "raw_key_size": 6759149, "raw_average_key_size": 48, "raw_value_size": 49580572, "raw_average_value_size": 358, "num_data_blocks": 13434, "num_entries": 138296, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "58265"}} >2018-07-11 05:08:28.434427 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2803] Compacted 2@0 + 7@1 files to L1 => 482104503 bytes >2018-07-11 05:08:28.458061 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:08:28.457690) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 7 44 445 3642 751 0] max score 1.80, MB/sec: 27.4 rd, 27.4 wr, level 1, files in(2, 7) out(7) MB in(269.5, 190.3) out(459.8), read-write-amplify(3.4) write-amplify(1.7) OK, records in: 12928037, records dropped: 10 > >2018-07-11 05:08:28.458072 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:08:28.457716) EVENT_LOG_v1 {"time_micros": 1531285708457705, "job": 2803, "event": "compaction_finished", "compaction_time_micros": 17625944, "output_level": 1, "num_output_files": 7, "total_output_size": 482104503, "num_input_records": 12928037, "num_output_records": 12928027, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 7, 44, 445, 3642, 751, 0]} >2018-07-11 05:08:28.459018 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285708459002, "job": 2803, "event": "table_file_deletion", "file_number": 35918} >2018-07-11 05:08:28.459033 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285708459030, "job": 2803, "event": "table_file_deletion", "file_number": 35916} >2018-07-11 05:08:28.459045 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285708459043, "job": 2803, "event": "table_file_deletion", "file_number": 35896} >2018-07-11 05:08:28.459057 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285708459055, "job": 2803, "event": "table_file_deletion", "file_number": 35891} >2018-07-11 05:08:28.459068 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285708459066, "job": 2803, "event": "table_file_deletion", "file_number": 35864} >2018-07-11 05:08:28.459079 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285708459077, "job": 2803, "event": "table_file_deletion", "file_number": 35814} >2018-07-11 05:08:28.459093 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285708459090, "job": 2803, "event": "table_file_deletion", "file_number": 35790} >2018-07-11 05:08:28.459106 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285708459104, "job": 2803, "event": "table_file_deletion", "file_number": 35761} >2018-07-11 05:08:28.459120 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285708459117, "job": 2803, "event": "table_file_deletion", "file_number": 35729} >2018-07-11 05:08:28.459344 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_picker.cc:461] [default] Expanding@1 1+5(74546868+330065258 bytes) to 4+5 (282564734+330065258bytes) > >2018-07-11 05:08:28.459446 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2804] Compacting 4@1 + 5@2 files to L2, score 1.80 >2018-07-11 05:08:28.459459 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2803 Base level 1, inputs: [35920(71MB) 35921(71MB) 35922(71MB) 35923(56MB)], [35897(71MB) 35898(71MB) 35899(71MB) 35900(71MB) 35905(30MB)] > >2018-07-11 05:08:28.459474 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285708459463, "job": 2804, "event": "compaction_started", "files_L1": [35920, 35921, 35922, 35923], "files_L2": [35897, 35898, 35899, 35900, 35905], "score": 1.79598, "input_data_size": 612629992} >2018-07-11 05:08:31.665471 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2804] Generated table #35926: 2638239 keys, 74546239 bytes >2018-07-11 05:08:31.665517 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285711665498, "cf_name": "default", "job": 2804, "event": "table_file_creation", "file_number": 35926, "file_size": 74546239, "table_properties": {"data_size": 67112121, "index_size": 689949, "filter_size": 6743179, "raw_key_size": 76215786, "raw_average_key_size": 28, "raw_value_size": 31365723, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638239, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:08:35.201721 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2804] Generated table #35927: 2638239 keys, 74546279 bytes >2018-07-11 05:08:35.201751 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285715201738, "cf_name": "default", "job": 2804, "event": "table_file_creation", "file_number": 35927, "file_size": 74546279, "table_properties": {"data_size": 67112106, "index_size": 690004, "filter_size": 6743179, "raw_key_size": 76215790, "raw_average_key_size": 28, "raw_value_size": 31365727, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638239, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:08:38.470405 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2804] Generated table #35928: 2638241 keys, 74546346 bytes >2018-07-11 05:08:38.470458 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285718470442, "cf_name": "default", "job": 2804, "event": "table_file_creation", "file_number": 35928, "file_size": 74546346, "table_properties": {"data_size": 67112158, "index_size": 690014, "filter_size": 6743184, "raw_key_size": 76215853, "raw_average_key_size": 28, "raw_value_size": 31365756, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:08:41.713857 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2804] Generated table #35929: 2638240 keys, 74546323 bytes >2018-07-11 05:08:41.713903 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285721713877, "cf_name": "default", "job": 2804, "event": "table_file_creation", "file_number": 35929, "file_size": 74546323, "table_properties": {"data_size": 67112087, "index_size": 690065, "filter_size": 6743181, "raw_key_size": 76215828, "raw_average_key_size": 28, "raw_value_size": 31365748, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:08:44.873212 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2804] Generated table #35930: 2638240 keys, 74546909 bytes >2018-07-11 05:08:44.873264 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285724873232, "cf_name": "default", "job": 2804, "event": "table_file_creation", "file_number": 35930, "file_size": 74546909, "table_properties": {"data_size": 67112082, "index_size": 690656, "filter_size": 6743181, "raw_key_size": 76215819, "raw_average_key_size": 28, "raw_value_size": 31365739, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:08:47.918407 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2804] Generated table #35931: 2638241 keys, 74546388 bytes >2018-07-11 05:08:47.918455 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285727918428, "cf_name": "default", "job": 2804, "event": "table_file_creation", "file_number": 35931, "file_size": 74546388, "table_properties": {"data_size": 67112151, "index_size": 690064, "filter_size": 6743183, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:08:51.218254 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2804] Generated table #35932: 2638241 keys, 74546428 bytes >2018-07-11 05:08:51.218299 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285731218271, "cf_name": "default", "job": 2804, "event": "table_file_creation", "file_number": 35932, "file_size": 74546428, "table_properties": {"data_size": 67112090, "index_size": 690165, "filter_size": 6743183, "raw_key_size": 76215852, "raw_average_key_size": 28, "raw_value_size": 31365755, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:08:54.365428 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2804] Generated table #35933: 2638241 keys, 74546003 bytes >2018-07-11 05:08:54.365475 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285734365454, "cf_name": "default", "job": 2804, "event": "table_file_creation", "file_number": 35933, "file_size": 74546003, "table_properties": {"data_size": 67112181, "index_size": 689649, "filter_size": 6743183, "raw_key_size": 76215860, "raw_average_key_size": 28, "raw_value_size": 31365763, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:08:55.059437 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2804] Generated table #35934: 570960 keys, 16259527 bytes >2018-07-11 05:08:55.059483 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285735059455, "cf_name": "default", "job": 2804, "event": "table_file_creation", "file_number": 35934, "file_size": 16259527, "table_properties": {"data_size": 14648748, "index_size": 150163, "filter_size": 1459630, "raw_key_size": 16510152, "raw_average_key_size": 28, "raw_value_size": 6893454, "raw_average_value_size": 12, "num_data_blocks": 3601, "num_entries": 570960, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "11", "kMergeOperands": "0"}} >2018-07-11 05:08:55.059524 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2804] Compacted 4@1 + 5@2 files to L2 => 612630442 bytes >2018-07-11 05:08:55.082631 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:08:55.082307) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 3 48 445 3642 751 0] max score 1.08, MB/sec: 23.0 rd, 23.0 wr, level 2, files in(4, 5) out(9) MB in(269.5, 314.8) out(584.2), read-write-amplify(4.3) write-amplify(2.2) OK, records in: 21676882, records dropped: 0 > >2018-07-11 05:08:55.082643 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:08:55.082333) EVENT_LOG_v1 {"time_micros": 1531285735082321, "job": 2804, "event": "compaction_finished", "compaction_time_micros": 26600028, "output_level": 2, "num_output_files": 9, "total_output_size": 612630442, "num_input_records": 21676882, "num_output_records": 21676882, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 3, 48, 445, 3642, 751, 0]} >2018-07-11 05:08:55.083533 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285735083528, "job": 2804, "event": "table_file_deletion", "file_number": 35923} >2018-07-11 05:08:55.083557 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285735083554, "job": 2804, "event": "table_file_deletion", "file_number": 35922} >2018-07-11 05:08:55.083579 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285735083576, "job": 2804, "event": "table_file_deletion", "file_number": 35921} >2018-07-11 05:08:55.083600 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285735083596, "job": 2804, "event": "table_file_deletion", "file_number": 35920} >2018-07-11 05:08:55.083621 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285735083618, "job": 2804, "event": "table_file_deletion", "file_number": 35905} >2018-07-11 05:08:55.083644 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285735083640, "job": 2804, "event": "table_file_deletion", "file_number": 35900} >2018-07-11 05:08:55.083666 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285735083662, "job": 2804, "event": "table_file_deletion", "file_number": 35899} >2018-07-11 05:08:55.083692 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285735083688, "job": 2804, "event": "table_file_deletion", "file_number": 35898} >2018-07-11 05:08:55.083717 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285735083713, "job": 2804, "event": "table_file_deletion", "file_number": 35897} >2018-07-11 05:08:55.083930 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_picker.cc:461] [default] Expanding@2 1+2(74546909+121510903 bytes) to 4+2 (298185728+121510903bytes) > >2018-07-11 05:08:55.084030 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2805] Compacting 4@2 + 2@3 files to L3, score 1.08 >2018-07-11 05:08:55.084044 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2804 Base level 2, inputs: [35930(71MB) 35931(71MB) 35932(71MB) 35933(71MB)], [35909(71MB) 35911(44MB)] > >2018-07-11 05:08:55.084063 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285735084049, "job": 2805, "event": "compaction_started", "files_L2": [35930, 35931, 35932, 35933], "files_L3": [35909, 35911], "score": 1.07999, "input_data_size": 419696631} >2018-07-11 05:08:58.300339 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2805] Generated table #35935: 2638240 keys, 74545821 bytes >2018-07-11 05:08:58.300377 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285738300362, "cf_name": "default", "job": 2805, "event": "table_file_creation", "file_number": 35935, "file_size": 74545821, "table_properties": {"data_size": 67112122, "index_size": 689528, "filter_size": 6743181, "raw_key_size": 76215827, "raw_average_key_size": 28, "raw_value_size": 31365747, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:09:01.363297 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2805] Generated table #35936: 2638241 keys, 74546585 bytes >2018-07-11 05:09:01.363347 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285741363319, "cf_name": "default", "job": 2805, "event": "table_file_creation", "file_number": 35936, "file_size": 74546585, "table_properties": {"data_size": 67112155, "index_size": 690256, "filter_size": 6743184, "raw_key_size": 76215850, "raw_average_key_size": 28, "raw_value_size": 31365753, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:09:04.411811 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2805] Generated table #35937: 2638241 keys, 74546252 bytes >2018-07-11 05:09:04.411861 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285744411833, "cf_name": "default", "job": 2805, "event": "table_file_creation", "file_number": 35937, "file_size": 74546252, "table_properties": {"data_size": 67112163, "index_size": 689916, "filter_size": 6743183, "raw_key_size": 76215861, "raw_average_key_size": 28, "raw_value_size": 31365764, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:09:07.554583 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2805] Generated table #35938: 2638241 keys, 74545861 bytes >2018-07-11 05:09:07.554627 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285747554600, "cf_name": "default", "job": 2805, "event": "table_file_creation", "file_number": 35938, "file_size": 74545861, "table_properties": {"data_size": 67112100, "index_size": 689588, "filter_size": 6743183, "raw_key_size": 76215851, "raw_average_key_size": 28, "raw_value_size": 31365754, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:09:10.707105 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2805] Generated table #35939: 2638241 keys, 74546435 bytes >2018-07-11 05:09:10.707152 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285750707124, "cf_name": "default", "job": 2805, "event": "table_file_creation", "file_number": 35939, "file_size": 74546435, "table_properties": {"data_size": 67112167, "index_size": 690095, "filter_size": 6743183, "raw_key_size": 76215857, "raw_average_key_size": 28, "raw_value_size": 31365760, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:09:12.578994 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2805] Generated table #35940: 1519916 keys, 46965496 bytes >2018-07-11 05:09:12.579038 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285752579023, "cf_name": "default", "job": 2805, "event": "table_file_creation", "file_number": 35940, "file_size": 46965496, "table_properties": {"data_size": 42636631, "index_size": 433726, "filter_size": 3894151, "raw_key_size": 44348936, "raw_average_key_size": 29, "raw_value_size": 21522465, "raw_average_value_size": 14, "num_data_blocks": 10538, "num_entries": 1519916, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:09:12.579075 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2805] Compacted 4@2 + 2@3 files to L3 => 419696450 bytes >2018-07-11 05:09:12.598863 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:09:12.598637) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 3 44 449 3642 751 0] max score 1.01, MB/sec: 24.0 rd, 24.0 wr, level 3, files in(4, 2) out(6) MB in(284.4, 115.9) out(400.3), read-write-amplify(2.8) write-amplify(1.4) OK, records in: 14711120, records dropped: 0 > >2018-07-11 05:09:12.598874 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:09:12.598662) EVENT_LOG_v1 {"time_micros": 1531285752598651, "job": 2805, "event": "compaction_finished", "compaction_time_micros": 17494992, "output_level": 3, "num_output_files": 6, "total_output_size": 419696450, "num_input_records": 14711120, "num_output_records": 14711120, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 3, 44, 449, 3642, 751, 0]} >2018-07-11 05:09:12.599848 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285752599844, "job": 2805, "event": "table_file_deletion", "file_number": 35933} >2018-07-11 05:09:12.599874 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285752599872, "job": 2805, "event": "table_file_deletion", "file_number": 35932} >2018-07-11 05:09:12.599887 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285752599885, "job": 2805, "event": "table_file_deletion", "file_number": 35931} >2018-07-11 05:09:12.599898 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285752599896, "job": 2805, "event": "table_file_deletion", "file_number": 35930} >2018-07-11 05:09:12.599910 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285752599908, "job": 2805, "event": "table_file_deletion", "file_number": 35911} >2018-07-11 05:09:12.599922 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285752599919, "job": 2805, "event": "table_file_deletion", "file_number": 35909} >2018-07-11 05:09:12.623812 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:09:12.600243) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #35936 to level-4 74546585 bytes > >2018-07-11 05:09:12.623823 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:09:12.623605) EVENT_LOG_v1 {"time_micros": 1531285752623598, "job": 2806, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 74546585} >2018-07-11 05:09:12.623826 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:09:12.623613) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-4 74546585 bytes OK: base level 1 max bytes base 268435456 files[0 3 44 448 3643 751 0] max score 1.01 > >2018-07-11 05:09:12.640508 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:09:12.624709) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #35939 to level-4 74546435 bytes > >2018-07-11 05:09:12.640531 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:09:12.640309) EVENT_LOG_v1 {"time_micros": 1531285752640302, "job": 2807, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 74546435} >2018-07-11 05:09:12.640535 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:09:12.640317) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-4 74546435 bytes OK: base level 1 max bytes base 268435456 files[0 3 44 447 3644 751 0] max score 1.00 > >2018-07-11 05:09:12.657060 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:09:12.641403) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #35937 to level-4 74546252 bytes > >2018-07-11 05:09:12.657071 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:09:12.656965) EVENT_LOG_v1 {"time_micros": 1531285752656959, "job": 2808, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 74546252} >2018-07-11 05:09:12.657075 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:09:12.656973) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-4 74546252 bytes OK: base level 1 max bytes base 268435456 files[0 3 44 446 3645 751 0] max score 1.00 > >2018-07-11 05:09:12.690423 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:09:12.657923) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #16591 to level-5 74546847 bytes > >2018-07-11 05:09:12.690434 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:09:12.690330) EVENT_LOG_v1 {"time_micros": 1531285752690324, "job": 2809, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546847} >2018-07-11 05:09:12.690437 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:09:12.690338) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546847 bytes OK: base level 1 max bytes base 268435456 files[0 3 44 446 3644 752 0] max score 1.00 > >2018-07-11 05:09:12.707111 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:09:12.691281) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #17534 to level-5 74546847 bytes > >2018-07-11 05:09:12.707122 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:09:12.707018) EVENT_LOG_v1 {"time_micros": 1531285752707012, "job": 2810, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546847} >2018-07-11 05:09:12.707124 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:09:12.707026) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546847 bytes OK: base level 1 max bytes base 268435456 files[0 3 44 446 3643 753 0] max score 1.00 > >2018-07-11 05:09:12.832295 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:09:12.707967) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #32474 to level-5 74546847 bytes > >2018-07-11 05:09:12.832306 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:09:12.832068) EVENT_LOG_v1 {"time_micros": 1531285752832061, "job": 2811, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546847} >2018-07-11 05:09:12.832309 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:09:12.832079) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546847 bytes OK: base level 1 max bytes base 268435456 files[0 3 44 446 3642 754 0] max score 1.00 > >2018-07-11 05:09:12.857163 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:09:12.833544) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #3624 to level-4 74546044 bytes > >2018-07-11 05:09:12.857174 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:09:12.857071) EVENT_LOG_v1 {"time_micros": 1531285752857065, "job": 2812, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 74546044} >2018-07-11 05:09:12.857176 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:09:12.857079) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-4 74546044 bytes OK: base level 1 max bytes base 268435456 files[0 3 44 445 3643 754 0] max score 1.00 > >2018-07-11 05:09:12.874032 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:09:12.858028) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #17199 to level-5 74546847 bytes > >2018-07-11 05:09:12.874043 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:09:12.873823) EVENT_LOG_v1 {"time_micros": 1531285752873817, "job": 2813, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546847} >2018-07-11 05:09:12.874046 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:09:12.873832) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546847 bytes OK: base level 1 max bytes base 268435456 files[0 3 44 445 3642 755 0] max score 1.00 > >2018-07-11 05:10:45.207694 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:684] reusing log 35915 from recycle list > >2018-07-11 05:10:45.207814 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #35941. Immutable memtables: 0. > >2018-07-11 05:10:45.277633 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:10:45.207851) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1 >2018-07-11 05:10:45.277644 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 2814] Flushing memtable with next log file: 35941 > >2018-07-11 05:10:45.277661 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285845277652, "job": 2814, "event": "flush_started", "num_memtables": 1, "num_entries": 5000015, "num_deletes": 0, "memory_usage": 292938856} >2018-07-11 05:10:45.277664 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 2814] Level-0 flush table #35942: started >2018-07-11 05:10:49.209123 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285849209104, "cf_name": "default", "job": 2814, "event": "table_file_creation", "file_number": 35942, "file_size": 141286221, "table_properties": {"data_size": 127196936, "index_size": 1308576, "filter_size": 12779719, "raw_key_size": 144445077, "raw_average_key_size": 28, "raw_value_size": 59449772, "raw_average_value_size": 11, "num_data_blocks": 31252, "num_entries": 5000013, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:10:49.209144 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 2814] Level-0 flush table #35942: 141286221 bytes OK >2018-07-11 05:10:49.229848 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:242] adding log 35917 to recycle list > >2018-07-11 05:10:49.229859 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:10:49.209170) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #35942 started >2018-07-11 05:10:49.229863 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:10:49.229650) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #35942: memtable #1 done >2018-07-11 05:10:49.229865 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:10:49.229668) EVENT_LOG_v1 {"time_micros": 1531285849229661, "job": 2814, "event": "flush_finished", "lsm_state": [1, 3, 44, 445, 3642, 755, 0], "immutable_memtables": 0} >2018-07-11 05:10:49.229867 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:10:49.229796) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 3 44 445 3642 755 0] max score 1.00 > >2018-07-11 05:10:49.230759 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 2814] Try to delete WAL files size 178896755, prev total WAL file size 178896767, number of live WAL files 2. > >2018-07-11 05:12:07.501131 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:684] reusing log 35917 from recycle list > >2018-07-11 05:12:07.501244 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #35943. Immutable memtables: 0. > >2018-07-11 05:12:07.580637 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:12:07.501291) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1 >2018-07-11 05:12:07.580651 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 2815] Flushing memtable with next log file: 35943 > >2018-07-11 05:12:07.580668 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285927580659, "job": 2815, "event": "flush_started", "num_memtables": 1, "num_entries": 5000015, "num_deletes": 0, "memory_usage": 292951624} >2018-07-11 05:12:07.580672 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 2815] Level-0 flush table #35944: started >2018-07-11 05:12:11.577333 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285931577312, "cf_name": "default", "job": 2815, "event": "table_file_creation", "file_number": 35944, "file_size": 141286221, "table_properties": {"data_size": 127196936, "index_size": 1308576, "filter_size": 12779719, "raw_key_size": 144445077, "raw_average_key_size": 28, "raw_value_size": 59449772, "raw_average_value_size": 11, "num_data_blocks": 31252, "num_entries": 5000013, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:12:11.577356 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 2815] Level-0 flush table #35944: 141286221 bytes OK >2018-07-11 05:12:11.610826 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:242] adding log 35941 to recycle list > >2018-07-11 05:12:11.610840 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:12:11.577369) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #35944 started >2018-07-11 05:12:11.610844 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:12:11.610586) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #35944: memtable #1 done >2018-07-11 05:12:11.610846 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:12:11.610603) EVENT_LOG_v1 {"time_micros": 1531285931610596, "job": 2815, "event": "flush_finished", "lsm_state": [2, 3, 44, 445, 3642, 755, 0], "immutable_memtables": 0} >2018-07-11 05:12:11.610848 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:12:11.610772) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[2 3 44 445 3642 755 0] max score 1.05 > >2018-07-11 05:12:11.610961 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2816] Compacting 2@0 + 2@1 files to L1, score 1.05 >2018-07-11 05:12:11.610977 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2815 Base level 0, inputs: [35944(134MB) 35942(134MB)], [35919(71MB) 35924(65MB)] > >2018-07-11 05:12:11.610995 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285931610981, "job": 2816, "event": "compaction_started", "files_L0": [35944, 35942], "files_L1": [35919, 35924], "score": 1.05266, "input_data_size": 425326963} >2018-07-11 05:12:11.611730 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 2815] Try to delete WAL files size 178896755, prev total WAL file size 178896767, number of live WAL files 2. > >2018-07-11 05:12:14.288539 7f41641eb700 1 bluefs _allocate failed to allocate 0x4800000 on bdev 1, free 0x20fe000; fallback to bdev 2 >2018-07-11 05:12:15.831670 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2816] Generated table #35945: 2623393 keys, 74508695 bytes >2018-07-11 05:12:15.831720 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285935831692, "cf_name": "default", "job": 2816, "event": "table_file_creation", "file_number": 35945, "file_size": 74508695, "table_properties": {"data_size": 67110209, "index_size": 691425, "filter_size": 6706071, "raw_key_size": 75820226, "raw_average_key_size": 28, "raw_value_size": 31556840, "raw_average_value_size": 12, "num_data_blocks": 16475, "num_entries": 2623393, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:12:18.027331 7f41641eb700 1 bluefs _allocate failed to allocate 0x4800000 on bdev 1, free 0x20fe000; fallback to bdev 2 >2018-07-11 05:12:19.030230 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2816] Generated table #35946: 2638241 keys, 74546892 bytes >2018-07-11 05:12:19.030279 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285939030260, "cf_name": "default", "job": 2816, "event": "table_file_creation", "file_number": 35946, "file_size": 74546892, "table_properties": {"data_size": 67112121, "index_size": 690598, "filter_size": 6743183, "raw_key_size": 76215847, "raw_average_key_size": 28, "raw_value_size": 31365750, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:12:21.958154 7f41641eb700 1 bluefs _allocate failed to allocate 0x4800000 on bdev 1, free 0x20fe000; fallback to bdev 2 >2018-07-11 05:12:22.900321 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2816] Generated table #35947: 2638241 keys, 74546491 bytes >2018-07-11 05:12:22.900366 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285942900340, "cf_name": "default", "job": 2816, "event": "table_file_creation", "file_number": 35947, "file_size": 74546491, "table_properties": {"data_size": 67112154, "index_size": 690164, "filter_size": 6743183, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:12:24.972609 7f41641eb700 1 bluefs _allocate failed to allocate 0x4800000 on bdev 1, free 0x20fe000; fallback to bdev 2 >2018-07-11 05:12:25.994121 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2816] Generated table #35948: 2638241 keys, 74546381 bytes >2018-07-11 05:12:25.994166 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285945994149, "cf_name": "default", "job": 2816, "event": "table_file_creation", "file_number": 35948, "file_size": 74546381, "table_properties": {"data_size": 67112084, "index_size": 690124, "filter_size": 6743183, "raw_key_size": 76215850, "raw_average_key_size": 28, "raw_value_size": 31365753, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:12:28.339729 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2816] Generated table #35949: 2085317 keys, 58922727 bytes >2018-07-11 05:12:28.339775 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285948339747, "cf_name": "default", "job": 2816, "event": "table_file_creation", "file_number": 35949, "file_size": 58922727, "table_properties": {"data_size": 53046767, "index_size": 545032, "filter_size": 5329940, "raw_key_size": 60242505, "raw_average_key_size": 28, "raw_value_size": 24792116, "raw_average_value_size": 11, "num_data_blocks": 13034, "num_entries": 2085317, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:12:29.461014 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2816] Generated table #35950: 166193 keys, 68246771 bytes >2018-07-11 05:12:29.461058 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285949461043, "cf_name": "default", "job": 2816, "event": "table_file_creation", "file_number": 35950, "file_size": 68246771, "table_properties": {"data_size": 67111866, "index_size": 564853, "filter_size": 569064, "raw_key_size": 15376375, "raw_average_key_size": 92, "raw_value_size": 53112883, "raw_average_value_size": 319, "num_data_blocks": 15215, "num_entries": 166193, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "80898", "kMergeOperands": "0"}} >2018-07-11 05:12:29.494794 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2816] Generated table #35951: 5 keys, 5026 bytes >2018-07-11 05:12:29.494818 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285949494806, "cf_name": "default", "job": 2816, "event": "table_file_creation", "file_number": 35951, "file_size": 5026, "table_properties": {"data_size": 4008, "index_size": 28, "filter_size": 23, "raw_key_size": 428, "raw_average_key_size": 85, "raw_value_size": 3659, "raw_average_value_size": 731, "num_data_blocks": 1, "num_entries": 5, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:12:29.494884 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2816] Compacted 2@0 + 2@1 files to L1 => 425322983 bytes >2018-07-11 05:12:29.519046 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:12:29.518729) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 8 44 445 3642 755 0] max score 1.80, MB/sec: 23.8 rd, 23.8 wr, level 1, files in(2, 2) out(7) MB in(269.5, 136.1) out(405.6), read-write-amplify(3.0) write-amplify(1.5) OK, records in: 12789637, records dropped: 6 > >2018-07-11 05:12:29.519058 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:12:29.518755) EVENT_LOG_v1 {"time_micros": 1531285949518744, "job": 2816, "event": "compaction_finished", "compaction_time_micros": 17883871, "output_level": 1, "num_output_files": 7, "total_output_size": 425322983, "num_input_records": 12789637, "num_output_records": 12789631, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 8, 44, 445, 3642, 755, 0]} >2018-07-11 05:12:29.520010 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285949519993, "job": 2816, "event": "table_file_deletion", "file_number": 35944} >2018-07-11 05:12:29.520026 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285949520024, "job": 2816, "event": "table_file_deletion", "file_number": 35942} >2018-07-11 05:12:29.520037 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285949520035, "job": 2816, "event": "table_file_deletion", "file_number": 35924} >2018-07-11 05:12:29.520048 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285949520046, "job": 2816, "event": "table_file_deletion", "file_number": 35919} >2018-07-11 05:12:29.520275 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_picker.cc:461] [default] Expanding@1 1+5(74546892+314444714 bytes) to 4+5 (282562491+314444714bytes) > >2018-07-11 05:12:29.520376 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2817] Compacting 4@1 + 5@2 files to L2, score 1.80 >2018-07-11 05:12:29.520389 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2816 Base level 1, inputs: [35946(71MB) 35947(71MB) 35948(71MB) 35949(56MB)], [35926(71MB) 35927(71MB) 35928(71MB) 35929(71MB) 35934(15MB)] > >2018-07-11 05:12:29.520403 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285949520392, "job": 2817, "event": "compaction_started", "files_L1": [35946, 35947, 35948, 35949], "files_L2": [35926, 35927, 35928, 35929, 35934], "score": 1.79599, "input_data_size": 597007205} >2018-07-11 05:12:32.748030 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2817] Generated table #35952: 2638240 keys, 74545922 bytes >2018-07-11 05:12:32.748074 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285952748048, "cf_name": "default", "job": 2817, "event": "table_file_creation", "file_number": 35952, "file_size": 74545922, "table_properties": {"data_size": 67112143, "index_size": 689608, "filter_size": 6743181, "raw_key_size": 76215816, "raw_average_key_size": 28, "raw_value_size": 31365736, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:12:36.113307 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2817] Generated table #35953: 2638239 keys, 74546029 bytes >2018-07-11 05:12:36.113353 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285956113338, "cf_name": "default", "job": 2817, "event": "table_file_creation", "file_number": 35953, "file_size": 74546029, "table_properties": {"data_size": 67112084, "index_size": 689776, "filter_size": 6743179, "raw_key_size": 76215790, "raw_average_key_size": 28, "raw_value_size": 31365727, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638239, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:12:39.257736 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2817] Generated table #35954: 2638240 keys, 74546144 bytes >2018-07-11 05:12:39.257785 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285959257769, "cf_name": "default", "job": 2817, "event": "table_file_creation", "file_number": 35954, "file_size": 74546144, "table_properties": {"data_size": 67112137, "index_size": 689836, "filter_size": 6743181, "raw_key_size": 76215824, "raw_average_key_size": 28, "raw_value_size": 31365744, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:12:42.426948 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2817] Generated table #35955: 2638239 keys, 74546347 bytes >2018-07-11 05:12:42.426988 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285962426972, "cf_name": "default", "job": 2817, "event": "table_file_creation", "file_number": 35955, "file_size": 74546347, "table_properties": {"data_size": 67112081, "index_size": 690097, "filter_size": 6743179, "raw_key_size": 76215798, "raw_average_key_size": 28, "raw_value_size": 31365735, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638239, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:12:45.437406 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2817] Generated table #35956: 2638240 keys, 74546282 bytes >2018-07-11 05:12:45.437456 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285965437428, "cf_name": "default", "job": 2817, "event": "table_file_creation", "file_number": 35956, "file_size": 74546282, "table_properties": {"data_size": 67112146, "index_size": 689965, "filter_size": 6743181, "raw_key_size": 76215821, "raw_average_key_size": 28, "raw_value_size": 31365741, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:12:48.622757 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2817] Generated table #35957: 2638241 keys, 74545754 bytes >2018-07-11 05:12:48.622809 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285968622793, "cf_name": "default", "job": 2817, "event": "table_file_creation", "file_number": 35957, "file_size": 74545754, "table_properties": {"data_size": 67112106, "index_size": 689475, "filter_size": 6743183, "raw_key_size": 76215856, "raw_average_key_size": 28, "raw_value_size": 31365759, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:12:51.750950 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2817] Generated table #35958: 2638241 keys, 74546543 bytes >2018-07-11 05:12:51.750986 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285971750970, "cf_name": "default", "job": 2817, "event": "table_file_creation", "file_number": 35958, "file_size": 74546543, "table_properties": {"data_size": 67112173, "index_size": 690197, "filter_size": 6743183, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:12:54.998691 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2817] Generated table #35959: 2638241 keys, 74546775 bytes >2018-07-11 05:12:54.998729 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285974998713, "cf_name": "default", "job": 2817, "event": "table_file_creation", "file_number": 35959, "file_size": 74546775, "table_properties": {"data_size": 67112135, "index_size": 690467, "filter_size": 6743183, "raw_key_size": 76215858, "raw_average_key_size": 28, "raw_value_size": 31365761, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:12:55.036096 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2817] Generated table #35960: 18038 keys, 636525 bytes >2018-07-11 05:12:55.036121 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285975036109, "cf_name": "default", "job": 2817, "event": "table_file_creation", "file_number": 35960, "file_size": 636525, "table_properties": {"data_size": 583433, "index_size": 5703, "filter_size": 46409, "raw_key_size": 536848, "raw_average_key_size": 29, "raw_value_size": 319824, "raw_average_value_size": 17, "num_data_blocks": 146, "num_entries": 18038, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "11", "kMergeOperands": "0"}} >2018-07-11 05:12:55.036148 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2817] Compacted 4@1 + 5@2 files to L2 => 597006321 bytes >2018-07-11 05:12:55.051863 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:12:55.051597) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 4 48 445 3642 755 0] max score 1.07, MB/sec: 23.4 rd, 23.4 wr, level 2, files in(4, 5) out(9) MB in(269.5, 299.9) out(569.3), read-write-amplify(4.2) write-amplify(2.1) OK, records in: 21123959, records dropped: 0 > >2018-07-11 05:12:55.051874 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:12:55.051621) EVENT_LOG_v1 {"time_micros": 1531285975051610, "job": 2817, "event": "compaction_finished", "compaction_time_micros": 25515726, "output_level": 2, "num_output_files": 9, "total_output_size": 597006321, "num_input_records": 21123959, "num_output_records": 21123959, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 4, 48, 445, 3642, 755, 0]} >2018-07-11 05:12:55.052796 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285975052792, "job": 2817, "event": "table_file_deletion", "file_number": 35949} >2018-07-11 05:12:55.052810 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285975052808, "job": 2817, "event": "table_file_deletion", "file_number": 35948} >2018-07-11 05:12:55.052822 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285975052820, "job": 2817, "event": "table_file_deletion", "file_number": 35947} >2018-07-11 05:12:55.052832 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285975052830, "job": 2817, "event": "table_file_deletion", "file_number": 35946} >2018-07-11 05:12:55.052845 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285975052843, "job": 2817, "event": "table_file_deletion", "file_number": 35934} >2018-07-11 05:12:55.052858 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285975052856, "job": 2817, "event": "table_file_deletion", "file_number": 35929} >2018-07-11 05:12:55.052873 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285975052871, "job": 2817, "event": "table_file_deletion", "file_number": 35928} >2018-07-11 05:12:55.052887 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285975052885, "job": 2817, "event": "table_file_deletion", "file_number": 35927} >2018-07-11 05:12:55.052902 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285975052900, "job": 2817, "event": "table_file_deletion", "file_number": 35926} >2018-07-11 05:12:55.053117 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_picker.cc:461] [default] Expanding@2 1+1(74546775+46965496 bytes) to 3+1 (223639072+46965496bytes) > >2018-07-11 05:12:55.053217 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2818] Compacting 3@2 + 1@3 files to L3, score 1.07 >2018-07-11 05:12:55.053229 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2817 Base level 2, inputs: [35957(71MB) 35958(71MB) 35959(71MB)], [35940(44MB)] > >2018-07-11 05:12:55.053242 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285975053232, "job": 2818, "event": "compaction_started", "files_L2": [35957, 35958, 35959], "files_L3": [35940], "score": 1.07417, "input_data_size": 270604568} >2018-07-11 05:12:58.138461 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2818] Generated table #35961: 2638241 keys, 74546690 bytes >2018-07-11 05:12:58.138498 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285978138482, "cf_name": "default", "job": 2818, "event": "table_file_creation", "file_number": 35961, "file_size": 74546690, "table_properties": {"data_size": 67112129, "index_size": 690388, "filter_size": 6743183, "raw_key_size": 76215857, "raw_average_key_size": 28, "raw_value_size": 31365760, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:13:01.160547 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2818] Generated table #35962: 2638241 keys, 74546926 bytes >2018-07-11 05:13:01.160600 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285981160584, "cf_name": "default", "job": 2818, "event": "table_file_creation", "file_number": 35962, "file_size": 74546926, "table_properties": {"data_size": 67112163, "index_size": 690590, "filter_size": 6743183, "raw_key_size": 76215856, "raw_average_key_size": 28, "raw_value_size": 31365759, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:13:04.402788 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2818] Generated table #35963: 2638241 keys, 74546078 bytes >2018-07-11 05:13:04.402833 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285984402805, "cf_name": "default", "job": 2818, "event": "table_file_creation", "file_number": 35963, "file_size": 74546078, "table_properties": {"data_size": 67112128, "index_size": 689777, "filter_size": 6743183, "raw_key_size": 76215850, "raw_average_key_size": 28, "raw_value_size": 31365753, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:13:06.331540 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2818] Generated table #35964: 1519916 keys, 46965060 bytes >2018-07-11 05:13:06.331589 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285986331560, "cf_name": "default", "job": 2818, "event": "table_file_creation", "file_number": 35964, "file_size": 46965060, "table_properties": {"data_size": 42636630, "index_size": 433291, "filter_size": 3894151, "raw_key_size": 44348941, "raw_average_key_size": 29, "raw_value_size": 21522470, "raw_average_value_size": 14, "num_data_blocks": 10538, "num_entries": 1519916, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:13:06.331622 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2818] Compacted 3@2 + 1@3 files to L3 => 270604754 bytes >2018-07-11 05:13:06.343398 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:13:06.343092) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 4 45 448 3642 755 0] max score 1.01, MB/sec: 24.0 rd, 24.0 wr, level 3, files in(3, 1) out(4) MB in(213.3, 44.8) out(258.1), read-write-amplify(2.4) write-amplify(1.2) OK, records in: 9434639, records dropped: 0 > >2018-07-11 05:13:06.343409 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:13:06.343118) EVENT_LOG_v1 {"time_micros": 1531285986343107, "job": 2818, "event": "compaction_finished", "compaction_time_micros": 11278360, "output_level": 3, "num_output_files": 4, "total_output_size": 270604754, "num_input_records": 9434639, "num_output_records": 9434639, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 4, 45, 448, 3642, 755, 0]} >2018-07-11 05:13:06.344368 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285986344364, "job": 2818, "event": "table_file_deletion", "file_number": 35959} >2018-07-11 05:13:06.344381 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285986344379, "job": 2818, "event": "table_file_deletion", "file_number": 35958} >2018-07-11 05:13:06.344393 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285986344391, "job": 2818, "event": "table_file_deletion", "file_number": 35957} >2018-07-11 05:13:06.344411 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531285986344408, "job": 2818, "event": "table_file_deletion", "file_number": 35940} >2018-07-11 05:13:06.359870 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:13:06.344730) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #35962 to level-4 74546926 bytes > >2018-07-11 05:13:06.359880 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:13:06.359775) EVENT_LOG_v1 {"time_micros": 1531285986359769, "job": 2819, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 74546926} >2018-07-11 05:13:06.359883 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:13:06.359783) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-4 74546926 bytes OK: base level 1 max bytes base 268435456 files[0 4 45 447 3643 755 0] max score 1.00 > >2018-07-11 05:13:06.376674 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:13:06.360745) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #35961 to level-4 74546690 bytes > >2018-07-11 05:13:06.376685 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:13:06.376469) EVENT_LOG_v1 {"time_micros": 1531285986376463, "job": 2820, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 74546690} >2018-07-11 05:13:06.376688 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:13:06.376477) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-4 74546690 bytes OK: base level 1 max bytes base 268435456 files[0 4 45 446 3644 755 0] max score 1.00 > >2018-07-11 05:13:06.410166 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:13:06.377575) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #35962 to level-5 74546926 bytes > >2018-07-11 05:13:06.410177 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:13:06.409966) EVENT_LOG_v1 {"time_micros": 1531285986409961, "job": 2821, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546926} >2018-07-11 05:13:06.410179 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:13:06.409974) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546926 bytes OK: base level 1 max bytes base 268435456 files[0 4 45 446 3643 756 0] max score 1.00 > >2018-07-11 05:13:06.426727 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:13:06.411026) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #16418 to level-5 74546846 bytes > >2018-07-11 05:13:06.426737 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:13:06.426635) EVENT_LOG_v1 {"time_micros": 1531285986426630, "job": 2822, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546846} >2018-07-11 05:13:06.426740 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:13:06.426643) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546846 bytes OK: base level 1 max bytes base 268435456 files[0 4 45 446 3642 757 0] max score 1.00 > >2018-07-11 05:13:06.443294 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:13:06.427635) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #35963 to level-4 74546078 bytes > >2018-07-11 05:13:06.443304 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:13:06.443197) EVENT_LOG_v1 {"time_micros": 1531285986443191, "job": 2823, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 74546078} >2018-07-11 05:13:06.443307 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:13:06.443205) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-4 74546078 bytes OK: base level 1 max bytes base 268435456 files[0 4 45 445 3643 757 0] max score 1.00 > >2018-07-11 05:13:06.459981 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:13:06.444148) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #15545 to level-5 74546844 bytes > >2018-07-11 05:13:06.459991 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:13:06.459887) EVENT_LOG_v1 {"time_micros": 1531285986459881, "job": 2824, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546844} >2018-07-11 05:13:06.459994 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:13:06.459895) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546844 bytes OK: base level 1 max bytes base 268435456 files[0 4 45 445 3642 758 0] max score 1.00 > >2018-07-11 05:14:06.436233 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:684] reusing log 35941 from recycle list > >2018-07-11 05:14:06.436321 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #35965. Immutable memtables: 0. > >2018-07-11 05:14:06.513742 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:14:06.436357) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1 >2018-07-11 05:14:06.513755 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 2825] Flushing memtable with next log file: 35965 > >2018-07-11 05:14:06.513771 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286046513763, "job": 2825, "event": "flush_started", "num_memtables": 1, "num_entries": 5000015, "num_deletes": 0, "memory_usage": 292941928} >2018-07-11 05:14:06.513775 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 2825] Level-0 flush table #35966: started >2018-07-11 05:14:10.299085 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286050299064, "cf_name": "default", "job": 2825, "event": "table_file_creation", "file_number": 35966, "file_size": 141286219, "table_properties": {"data_size": 127196934, "index_size": 1308576, "filter_size": 12779719, "raw_key_size": 144445077, "raw_average_key_size": 28, "raw_value_size": 59449772, "raw_average_value_size": 11, "num_data_blocks": 31252, "num_entries": 5000013, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:14:10.299105 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 2825] Level-0 flush table #35966: 141286219 bytes OK >2018-07-11 05:14:10.308398 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:242] adding log 35943 to recycle list > >2018-07-11 05:14:10.308408 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:14:10.299130) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #35966 started >2018-07-11 05:14:10.308412 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:14:10.308272) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #35966: memtable #1 done >2018-07-11 05:14:10.308414 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:14:10.308287) EVENT_LOG_v1 {"time_micros": 1531286050308281, "job": 2825, "event": "flush_finished", "lsm_state": [1, 4, 45, 445, 3642, 758, 0], "immutable_memtables": 0} >2018-07-11 05:14:10.308416 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:14:10.308347) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 4 45 445 3642 758 0] max score 1.00 > >2018-07-11 05:14:10.309290 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 2825] Try to delete WAL files size 178896755, prev total WAL file size 178896767, number of live WAL files 2. > >2018-07-11 05:14:43.215454 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:684] reusing log 35943 from recycle list > >2018-07-11 05:14:43.215562 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #35967. Immutable memtables: 0. > >2018-07-11 05:14:43.268121 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:14:43.215597) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1 >2018-07-11 05:14:43.268133 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 2826] Flushing memtable with next log file: 35967 > >2018-07-11 05:14:43.268149 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286083268140, "job": 2826, "event": "flush_started", "num_memtables": 1, "num_entries": 5000015, "num_deletes": 0, "memory_usage": 292947744} >2018-07-11 05:14:43.268152 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 2826] Level-0 flush table #35968: started >2018-07-11 05:14:47.145153 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286087145132, "cf_name": "default", "job": 2826, "event": "table_file_creation", "file_number": 35968, "file_size": 141288967, "table_properties": {"data_size": 127198839, "index_size": 1309411, "filter_size": 12779727, "raw_key_size": 144445125, "raw_average_key_size": 28, "raw_value_size": 59451570, "raw_average_value_size": 11, "num_data_blocks": 31252, "num_entries": 5000015, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:14:47.145188 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 2826] Level-0 flush table #35968: 141288967 bytes OK >2018-07-11 05:14:47.157403 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:242] adding log 35965 to recycle list > >2018-07-11 05:14:47.157417 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:14:47.145201) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #35968 started >2018-07-11 05:14:47.157421 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:14:47.157252) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #35968: memtable #1 done >2018-07-11 05:14:47.157423 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:14:47.157267) EVENT_LOG_v1 {"time_micros": 1531286087157261, "job": 2826, "event": "flush_finished", "lsm_state": [2, 4, 45, 445, 3642, 758, 0], "immutable_memtables": 0} >2018-07-11 05:14:47.157425 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:14:47.157348) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[2 4 45 445 3642 758 0] max score 1.05 > >2018-07-11 05:14:47.157573 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2827] Compacting 2@0 + 2@1 files to L1, score 1.05 >2018-07-11 05:14:47.157591 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2826 Base level 0, inputs: [35968(134MB) 35966(134MB)], [35945(71MB) 35950(65MB)] > >2018-07-11 05:14:47.157608 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286087157595, "job": 2827, "event": "compaction_started", "files_L0": [35968, 35966], "files_L1": [35945, 35950], "score": 1.05267, "input_data_size": 425330652} >2018-07-11 05:14:47.158301 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 2826] Try to delete WAL files size 178896755, prev total WAL file size 178896767, number of live WAL files 2. > >2018-07-11 05:14:50.485121 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2827] Generated table #35969: 2623323 keys, 74508115 bytes >2018-07-11 05:14:50.485167 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286090485140, "cf_name": "default", "job": 2827, "event": "table_file_creation", "file_number": 35969, "file_size": 74508115, "table_properties": {"data_size": 67109909, "index_size": 691321, "filter_size": 6705895, "raw_key_size": 75818415, "raw_average_key_size": 28, "raw_value_size": 31557509, "raw_average_value_size": 12, "num_data_blocks": 16475, "num_entries": 2623323, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:14:52.519254 7f41641eb700 1 bluefs _allocate failed to allocate 0x4800000 on bdev 1, free 0x24fe000; fallback to bdev 2 >2018-07-11 05:14:53.612003 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2827] Generated table #35970: 2638242 keys, 74546424 bytes >2018-07-11 05:14:53.612048 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286093612022, "cf_name": "default", "job": 2827, "event": "table_file_creation", "file_number": 35970, "file_size": 74546424, "table_properties": {"data_size": 67112134, "index_size": 690114, "filter_size": 6743186, "raw_key_size": 76215874, "raw_average_key_size": 28, "raw_value_size": 31365760, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638242, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:14:55.621578 7f41641eb700 1 bluefs _allocate failed to allocate 0x4800000 on bdev 1, free 0x24fe000; fallback to bdev 2 >2018-07-11 05:14:56.586152 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2827] Generated table #35971: 2638241 keys, 74546257 bytes >2018-07-11 05:14:56.586198 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286096586170, "cf_name": "default", "job": 2827, "event": "table_file_creation", "file_number": 35971, "file_size": 74546257, "table_properties": {"data_size": 67112146, "index_size": 689938, "filter_size": 6743183, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:14:58.577725 7f41641eb700 1 bluefs _allocate failed to allocate 0x4800000 on bdev 1, free 0x24fe000; fallback to bdev 2 >2018-07-11 05:14:59.633490 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2827] Generated table #35972: 2638241 keys, 74545806 bytes >2018-07-11 05:14:59.633537 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286099633510, "cf_name": "default", "job": 2827, "event": "table_file_creation", "file_number": 35972, "file_size": 74545806, "table_properties": {"data_size": 67112087, "index_size": 689546, "filter_size": 6743183, "raw_key_size": 76215853, "raw_average_key_size": 28, "raw_value_size": 31365756, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:15:01.211705 7f41641eb700 1 bluefs _allocate failed to allocate 0x3900000 on bdev 1, free 0x24fe000; fallback to bdev 2 >2018-07-11 05:15:02.106180 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2827] Generated table #35973: 2085356 keys, 58924120 bytes >2018-07-11 05:15:02.106223 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286102106197, "cf_name": "default", "job": 2827, "event": "table_file_creation", "file_number": 35973, "file_size": 58924120, "table_properties": {"data_size": 53047745, "index_size": 545350, "filter_size": 5330037, "raw_key_size": 60243630, "raw_average_key_size": 28, "raw_value_size": 24792578, "raw_average_value_size": 11, "num_data_blocks": 13034, "num_entries": 2085356, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:15:02.619931 7f41641eb700 1 bluefs _allocate failed to allocate 0x4200000 on bdev 1, free 0x24fe000; fallback to bdev 2 >2018-07-11 05:15:03.601717 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2827] Generated table #35974: 166203 keys, 68251006 bytes >2018-07-11 05:15:03.601758 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286103601731, "cf_name": "default", "job": 2827, "event": "table_file_creation", "file_number": 35974, "file_size": 68251006, "table_properties": {"data_size": 67116040, "index_size": 564876, "filter_size": 569102, "raw_key_size": 15376985, "raw_average_key_size": 92, "raw_value_size": 53116513, "raw_average_value_size": 319, "num_data_blocks": 15216, "num_entries": 166203, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "80898", "kMergeOperands": "0"}} >2018-07-11 05:15:03.601822 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2827] Compacted 2@0 + 2@1 files to L1 => 425321728 bytes >2018-07-11 05:15:03.615535 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:15:03.615344) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 8 45 445 3642 758 0] max score 1.80, MB/sec: 25.9 rd, 25.9 wr, level 1, files in(2, 2) out(6) MB in(269.5, 136.1) out(405.6), read-write-amplify(3.0) write-amplify(1.5) OK, records in: 12789614, records dropped: 8 > >2018-07-11 05:15:03.615544 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:15:03.615367) EVENT_LOG_v1 {"time_micros": 1531286103615357, "job": 2827, "event": "compaction_finished", "compaction_time_micros": 16444197, "output_level": 1, "num_output_files": 6, "total_output_size": 425321728, "num_input_records": 12789614, "num_output_records": 12789606, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 8, 45, 445, 3642, 758, 0]} >2018-07-11 05:15:03.616544 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286103616540, "job": 2827, "event": "table_file_deletion", "file_number": 35968} >2018-07-11 05:15:03.616558 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286103616555, "job": 2827, "event": "table_file_deletion", "file_number": 35966} >2018-07-11 05:15:03.616568 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286103616566, "job": 2827, "event": "table_file_deletion", "file_number": 35950} >2018-07-11 05:15:03.616591 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286103616577, "job": 2827, "event": "table_file_deletion", "file_number": 35945} >2018-07-11 05:15:03.616820 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_picker.cc:461] [default] Expanding@1 1+6(74546424+373367249 bytes) to 4+6 (282562607+373367249bytes) > >2018-07-11 05:15:03.616925 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2828] Compacting 4@1 + 6@2 files to L2, score 1.80 >2018-07-11 05:15:03.616939 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2827 Base level 1, inputs: [35970(71MB) 35971(71MB) 35972(71MB) 35973(56MB)], [35952(71MB) 35953(71MB) 35954(71MB) 35955(71MB) 35956(71MB) 35960(621KB)] > >2018-07-11 05:15:03.616953 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286103616942, "job": 2828, "event": "compaction_started", "files_L1": [35970, 35971, 35972, 35973], "files_L2": [35952, 35953, 35954, 35955, 35956, 35960], "score": 1.79601, "input_data_size": 655929856} >2018-07-11 05:15:06.748722 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2828] Generated table #35975: 2638240 keys, 74545943 bytes >2018-07-11 05:15:06.748757 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286106748742, "cf_name": "default", "job": 2828, "event": "table_file_creation", "file_number": 35975, "file_size": 74545943, "table_properties": {"data_size": 67112144, "index_size": 689628, "filter_size": 6743181, "raw_key_size": 76215816, "raw_average_key_size": 28, "raw_value_size": 31365736, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:15:09.765691 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2828] Generated table #35976: 2638239 keys, 74545973 bytes >2018-07-11 05:15:09.765723 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286109765709, "cf_name": "default", "job": 2828, "event": "table_file_creation", "file_number": 35976, "file_size": 74545973, "table_properties": {"data_size": 67112074, "index_size": 689730, "filter_size": 6743179, "raw_key_size": 76215789, "raw_average_key_size": 28, "raw_value_size": 31365726, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638239, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:15:12.840674 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2828] Generated table #35977: 2638240 keys, 74546184 bytes >2018-07-11 05:15:12.840720 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286112840693, "cf_name": "default", "job": 2828, "event": "table_file_creation", "file_number": 35977, "file_size": 74546184, "table_properties": {"data_size": 67112140, "index_size": 689873, "filter_size": 6743181, "raw_key_size": 76215825, "raw_average_key_size": 28, "raw_value_size": 31365745, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:15:15.827699 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2828] Generated table #35978: 2638239 keys, 74546299 bytes >2018-07-11 05:15:15.827744 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286115827717, "cf_name": "default", "job": 2828, "event": "table_file_creation", "file_number": 35978, "file_size": 74546299, "table_properties": {"data_size": 67112080, "index_size": 690050, "filter_size": 6743179, "raw_key_size": 76215798, "raw_average_key_size": 28, "raw_value_size": 31365735, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638239, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:15:18.894200 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2828] Generated table #35979: 2638240 keys, 74546314 bytes >2018-07-11 05:15:18.894250 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286118894217, "cf_name": "default", "job": 2828, "event": "table_file_creation", "file_number": 35979, "file_size": 74546314, "table_properties": {"data_size": 67112147, "index_size": 689996, "filter_size": 6743181, "raw_key_size": 76215821, "raw_average_key_size": 28, "raw_value_size": 31365741, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:15:22.007890 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2828] Generated table #35980: 2638241 keys, 74546249 bytes >2018-07-11 05:15:22.007934 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286122007907, "cf_name": "default", "job": 2828, "event": "table_file_creation", "file_number": 35980, "file_size": 74546249, "table_properties": {"data_size": 67112146, "index_size": 689930, "filter_size": 6743183, "raw_key_size": 76215852, "raw_average_key_size": 28, "raw_value_size": 31365755, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:15:25.019508 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2828] Generated table #35981: 2638242 keys, 74546004 bytes >2018-07-11 05:15:25.019540 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286125019526, "cf_name": "default", "job": 2828, "event": "table_file_creation", "file_number": 35981, "file_size": 74546004, "table_properties": {"data_size": 67112159, "index_size": 689669, "filter_size": 6743186, "raw_key_size": 76215879, "raw_average_key_size": 28, "raw_value_size": 31365765, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638242, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:15:28.106655 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2828] Generated table #35982: 2638241 keys, 74547093 bytes >2018-07-11 05:15:28.106703 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286128106687, "cf_name": "default", "job": 2828, "event": "table_file_creation", "file_number": 35982, "file_size": 74547093, "table_properties": {"data_size": 67112167, "index_size": 690753, "filter_size": 6743183, "raw_key_size": 76215857, "raw_average_key_size": 28, "raw_value_size": 31365760, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:15:30.534340 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2828] Generated table #35983: 2103394 keys, 59559457 bytes >2018-07-11 05:15:30.534382 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286130534356, "cf_name": "default", "job": 2828, "event": "table_file_creation", "file_number": 35983, "file_size": 59559457, "table_properties": {"data_size": 53631088, "index_size": 550947, "filter_size": 5376433, "raw_key_size": 60780471, "raw_average_key_size": 28, "raw_value_size": 25112395, "raw_average_value_size": 11, "num_data_blocks": 13179, "num_entries": 2103394, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "11", "kMergeOperands": "0"}} >2018-07-11 05:15:30.534419 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2828] Compacted 4@1 + 6@2 files to L2 => 655929516 bytes >2018-07-11 05:15:30.548627 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:15:30.548310) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 4 48 445 3642 758 0] max score 1.10, MB/sec: 24.4 rd, 24.4 wr, level 2, files in(4, 6) out(9) MB in(269.5, 356.1) out(625.5), read-write-amplify(4.6) write-amplify(2.3) OK, records in: 23209316, records dropped: 0 > >2018-07-11 05:15:30.548638 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:15:30.548336) EVENT_LOG_v1 {"time_micros": 1531286130548325, "job": 2828, "event": "compaction_finished", "compaction_time_micros": 26917447, "output_level": 2, "num_output_files": 9, "total_output_size": 655929516, "num_input_records": 23209316, "num_output_records": 23209316, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 4, 48, 445, 3642, 758, 0]} >2018-07-11 05:15:30.549577 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286130549574, "job": 2828, "event": "table_file_deletion", "file_number": 35973} >2018-07-11 05:15:30.549604 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286130549601, "job": 2828, "event": "table_file_deletion", "file_number": 35972} >2018-07-11 05:15:30.549618 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286130549616, "job": 2828, "event": "table_file_deletion", "file_number": 35971} >2018-07-11 05:15:30.549628 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286130549626, "job": 2828, "event": "table_file_deletion", "file_number": 35970} >2018-07-11 05:15:30.549643 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286130549641, "job": 2828, "event": "table_file_deletion", "file_number": 35960} >2018-07-11 05:15:30.549655 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286130549653, "job": 2828, "event": "table_file_deletion", "file_number": 35956} >2018-07-11 05:15:30.549670 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286130549668, "job": 2828, "event": "table_file_deletion", "file_number": 35955} >2018-07-11 05:15:30.549683 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286130549681, "job": 2828, "event": "table_file_deletion", "file_number": 35954} >2018-07-11 05:15:30.549697 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286130549695, "job": 2828, "event": "table_file_deletion", "file_number": 35953} >2018-07-11 05:15:30.549708 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286130549705, "job": 2828, "event": "table_file_deletion", "file_number": 35952} >2018-07-11 05:15:30.550270 7f41641eb700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS ------- >2018-07-11 05:15:30.550276 7f41641eb700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] >** DB Stats ** >Uptime(secs): 50872.2 total, 612.8 interval >Cumulative writes: 196K writes, 1887M keys, 196K commit groups, 1.0 writes per commit group, ingest: 63.40 GB, 1.28 MB/s >Cumulative WAL: 196K writes, 56K syncs, 3.48 writes per sync, written: 63.40 GB, 1.28 MB/s >Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent >Interval writes: 54 writes, 27M keys, 54 commit groups, 1.0 writes per commit group, ingest: 921.29 MB, 1.50 MB/s >Interval WAL: 54 writes, 27 syncs, 1.93 writes per sync, written: 0.90 MB, 1.50 MB/s >Interval stall: 00:00:0.000 H:M:S, 0.0 percent > >** Compaction Stats [default] ** >Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop >---------------------------------------------------------------------------------------------------------------------------------------------------------- > L0 0/0 0.00 KB 0.0 0.0 0.0 0.0 49.7 49.7 0.0 1.0 0.0 32.4 1569 379 4.140 0 0 > L1 4/0 190.31 MB 0.7 82.0 49.8 32.2 81.9 49.7 0.0 1.6 25.7 25.7 3264 187 17.455 2593M 894K > L2 48/0 2.74 GB 1.1 97.0 49.8 47.2 96.9 49.7 0.0 1.9 21.6 21.5 4606 559 8.240 3620M 1358K > L3 445/0 24.93 GB 1.0 86.2 48.8 37.3 86.1 48.8 0.6 1.8 20.8 20.8 4245 294 14.438 3255M 4000K > L4 3642/0 250.00 GB 1.0 73.5 17.7 55.9 73.2 17.3 31.7 4.1 21.2 21.1 3557 237 15.009 2798M 15M > L5 758/0 52.60 GB 0.0 1.9 1.0 0.9 1.9 0.9 48.0 1.9 19.0 18.8 101 13 7.782 71M 1097K > Sum 4897/0 330.46 GB 0.0 340.5 167.0 173.5 389.6 216.1 80.4 7.8 20.1 23.0 17342 1669 10.391 12G 23M > Int 0/0 0.00 KB 0.0 4.2 2.4 1.7 4.9 3.2 1.5 6.3 20.5 24.4 208 16 12.986 148M 24 >Uptime(secs): 50872.2 total, 50872.2 interval >Flush(GB): cumulative 49.655, interval 0.790 >AddFile(GB): cumulative 0.000, interval 0.000 >AddFile(Total Files): cumulative 0, interval 0 >AddFile(L0 Files): cumulative 0, interval 0 >AddFile(Keys): cumulative 0, interval 0 >Cumulative compaction: 389.63 GB write, 7.84 MB/s write, 340.50 GB read, 6.85 MB/s read, 17342.3 seconds >Interval compaction: 4.94 GB write, 0.10 MB/s write, 4.15 GB read, 0.08 MB/s read, 207.8 seconds >Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count > >** File Read Latency Histogram By Level [default] ** > >** Compaction Stats [default] ** >Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop >---------------------------------------------------------------------------------------------------------------------------------------------------------- > L0 0/0 0.00 KB 0.0 0.0 0.0 0.0 49.7 49.7 0.0 1.0 0.0 32.4 1569 379 4.140 0 0 > L1 4/0 190.31 MB 0.7 82.0 49.8 32.2 81.9 49.7 0.0 1.6 25.7 25.7 3264 187 17.455 2593M 894K > L2 48/0 2.74 GB 1.1 97.0 49.8 47.2 96.9 49.7 0.0 1.9 21.6 21.5 4606 559 8.240 3620M 1358K > L3 445/0 24.93 GB 1.0 86.2 48.8 37.3 86.1 48.8 0.6 1.8 20.8 20.8 4245 294 14.438 3255M 4000K > L4 3642/0 250.00 GB 1.0 73.5 17.7 55.9 73.2 17.3 31.7 4.1 21.2 21.1 3557 237 15.009 2798M 15M > L5 758/0 52.60 GB 0.0 1.9 1.0 0.9 1.9 0.9 48.0 1.9 19.0 18.8 101 13 7.782 71M 1097K > Sum 4897/0 330.46 GB 0.0 340.5 167.0 173.5 389.6 216.1 80.4 7.8 20.1 23.0 17342 1669 10.391 12G 23M > Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 >Uptime(secs): 50872.2 total, 50872.2 interval >Flush(GB): cumulative 49.655, interval 0.000 >AddFile(GB): cumulative 0.000, interval 0.000 >AddFile(Total Files): cumulative 0, interval 0 >AddFile(L0 Files): cumulative 0, interval 0 >AddFile(Keys): cumulative 0, interval 0 >Cumulative compaction: 389.63 GB write, 7.84 MB/s write, 340.50 GB read, 6.85 MB/s read, 17342.3 seconds >Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds >Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count > >** File Read Latency Histogram By Level [default] ** > >2018-07-11 05:15:30.550301 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_picker.cc:461] [default] Expanding@2 1+1(74547093+46965060 bytes) to 2+1 (149093097+46965060bytes) > >2018-07-11 05:15:30.550392 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2829] Compacting 2@2 + 1@3 files to L3, score 1.10 >2018-07-11 05:15:30.550402 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2828 Base level 2, inputs: [35981(71MB) 35982(71MB)], [35964(44MB)] > >2018-07-11 05:15:30.550415 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286130550405, "job": 2829, "event": "compaction_started", "files_L2": [35981, 35982], "files_L3": [35964], "score": 1.09612, "input_data_size": 196058157} >2018-07-11 05:15:33.415525 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2829] Generated table #35984: 2638242 keys, 74546466 bytes >2018-07-11 05:15:33.415570 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286133415542, "cf_name": "default", "job": 2829, "event": "table_file_creation", "file_number": 35984, "file_size": 74546466, "table_properties": {"data_size": 67112183, "index_size": 690107, "filter_size": 6743186, "raw_key_size": 76215887, "raw_average_key_size": 28, "raw_value_size": 31365773, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638242, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:15:36.195403 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2829] Generated table #35985: 2638241 keys, 74546421 bytes >2018-07-11 05:15:36.195450 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286136195434, "cf_name": "default", "job": 2829, "event": "table_file_creation", "file_number": 35985, "file_size": 74546421, "table_properties": {"data_size": 67112145, "index_size": 690103, "filter_size": 6743183, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:15:38.054156 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2829] Generated table #35986: 1519916 keys, 46965204 bytes >2018-07-11 05:15:38.054198 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286138054183, "cf_name": "default", "job": 2829, "event": "table_file_creation", "file_number": 35986, "file_size": 46965204, "table_properties": {"data_size": 42636600, "index_size": 433465, "filter_size": 3894151, "raw_key_size": 44348936, "raw_average_key_size": 29, "raw_value_size": 21522465, "raw_average_value_size": 14, "num_data_blocks": 10538, "num_entries": 1519916, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:15:38.054231 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2829] Compacted 2@2 + 1@3 files to L3 => 196058091 bytes >2018-07-11 05:15:38.064898 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:15:38.064719) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 4 46 447 3642 758 0] max score 1.04, MB/sec: 26.1 rd, 26.1 wr, level 3, files in(2, 1) out(3) MB in(142.2, 44.8) out(187.0), read-write-amplify(2.6) write-amplify(1.3) OK, records in: 6796399, records dropped: 0 > >2018-07-11 05:15:38.064908 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:15:38.064742) EVENT_LOG_v1 {"time_micros": 1531286138064732, "job": 2829, "event": "compaction_finished", "compaction_time_micros": 7503802, "output_level": 3, "num_output_files": 3, "total_output_size": 196058091, "num_input_records": 6796399, "num_output_records": 6796399, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 4, 46, 447, 3642, 758, 0]} >2018-07-11 05:15:38.065874 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286138065858, "job": 2829, "event": "table_file_deletion", "file_number": 35982} >2018-07-11 05:15:38.065890 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286138065887, "job": 2829, "event": "table_file_deletion", "file_number": 35981} >2018-07-11 05:15:38.065905 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286138065902, "job": 2829, "event": "table_file_deletion", "file_number": 35964} >2018-07-11 05:15:38.066224 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2830] Compacting 1@2 + 2@3 files to L3, score 1.04 >2018-07-11 05:15:38.066236 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2829 Base level 2, inputs: [35979(71MB)], [35935(71MB) 35938(71MB)] > >2018-07-11 05:15:38.066268 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286138066239, "job": 2830, "event": "compaction_started", "files_L2": [35979], "files_L3": [35935, 35938], "score": 1.04058, "input_data_size": 223637996} >2018-07-11 05:15:41.258873 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2830] Generated table #35987: 2638241 keys, 74545730 bytes >2018-07-11 05:15:41.258918 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286141258903, "cf_name": "default", "job": 2830, "event": "table_file_creation", "file_number": 35987, "file_size": 74545730, "table_properties": {"data_size": 67112140, "index_size": 689417, "filter_size": 6743183, "raw_key_size": 76215855, "raw_average_key_size": 28, "raw_value_size": 31365758, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:15:44.194387 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2830] Generated table #35988: 2638240 keys, 74546349 bytes >2018-07-11 05:15:44.194432 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286144194405, "cf_name": "default", "job": 2830, "event": "table_file_creation", "file_number": 35988, "file_size": 74546349, "table_properties": {"data_size": 67112087, "index_size": 690091, "filter_size": 6743181, "raw_key_size": 76215823, "raw_average_key_size": 28, "raw_value_size": 31365743, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:15:47.129355 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2830] Generated table #35989: 2638240 keys, 74546313 bytes >2018-07-11 05:15:47.129388 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286147129373, "cf_name": "default", "job": 2830, "event": "table_file_creation", "file_number": 35989, "file_size": 74546313, "table_properties": {"data_size": 67112146, "index_size": 689996, "filter_size": 6743181, "raw_key_size": 76215821, "raw_average_key_size": 28, "raw_value_size": 31365741, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:15:47.129421 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2830] Compacted 1@2 + 2@3 files to L3 => 223638392 bytes >2018-07-11 05:15:47.139800 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:15:47.139578) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 4 45 448 3642 758 0] max score 1.01, MB/sec: 24.7 rd, 24.7 wr, level 3, files in(1, 2) out(3) MB in(71.1, 142.2) out(213.3), read-write-amplify(6.0) write-amplify(3.0) OK, records in: 7914721, records dropped: 0 > >2018-07-11 05:15:47.139811 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:15:47.139602) EVENT_LOG_v1 {"time_micros": 1531286147139591, "job": 2830, "event": "compaction_finished", "compaction_time_micros": 9063141, "output_level": 3, "num_output_files": 3, "total_output_size": 223638392, "num_input_records": 7914721, "num_output_records": 7914721, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 4, 45, 448, 3642, 758, 0]} >2018-07-11 05:15:47.140795 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286147140779, "job": 2830, "event": "table_file_deletion", "file_number": 35979} >2018-07-11 05:15:47.140811 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286147140809, "job": 2830, "event": "table_file_deletion", "file_number": 35938} >2018-07-11 05:15:47.140824 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286147140821, "job": 2830, "event": "table_file_deletion", "file_number": 35935} >2018-07-11 05:15:47.156483 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:15:47.141130) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #35978 to level-3 74546299 bytes > >2018-07-11 05:15:47.156510 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:15:47.156391) EVENT_LOG_v1 {"time_micros": 1531286147156385, "job": 2831, "event": "trivial_move", "destination_level": 3, "files": 1, "total_files_size": 74546299} >2018-07-11 05:15:47.156514 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:15:47.156399) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-3 74546299 bytes OK: base level 1 max bytes base 268435456 files[0 4 44 449 3642 758 0] max score 1.01 > >2018-07-11 05:15:47.173070 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:15:47.157359) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #35984 to level-4 74546466 bytes > >2018-07-11 05:15:47.173079 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:15:47.172984) EVENT_LOG_v1 {"time_micros": 1531286147172979, "job": 2832, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 74546466} >2018-07-11 05:15:47.173082 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:15:47.172992) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-4 74546466 bytes OK: base level 1 max bytes base 268435456 files[0 4 44 448 3643 758 0] max score 1.01 > >2018-07-11 05:15:47.189832 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:15:47.173910) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #35985 to level-4 74546421 bytes > >2018-07-11 05:15:47.189842 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:15:47.189638) EVENT_LOG_v1 {"time_micros": 1531286147189632, "job": 2833, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 74546421} >2018-07-11 05:15:47.189845 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:15:47.189646) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-4 74546421 bytes OK: base level 1 max bytes base 268435456 files[0 4 44 447 3644 758 0] max score 1.00 > >2018-07-11 05:15:47.190722 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2834] Compacting 1@3 + 2@4 files to L4, score 1.00 >2018-07-11 05:15:47.190733 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2833 Base level 3, inputs: [35988(71MB)], [35936(71MB) 35937(71MB)] > >2018-07-11 05:15:47.190760 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286147190736, "job": 2834, "event": "compaction_started", "files_L3": [35988], "files_L4": [35936, 35937], "score": 1.00291, "input_data_size": 223639186} >2018-07-11 05:15:50.299560 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2834] Generated table #35990: 2638241 keys, 74546442 bytes >2018-07-11 05:15:50.299605 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286150299578, "cf_name": "default", "job": 2834, "event": "table_file_creation", "file_number": 35990, "file_size": 74546442, "table_properties": {"data_size": 67112171, "index_size": 690098, "filter_size": 6743183, "raw_key_size": 76215851, "raw_average_key_size": 28, "raw_value_size": 31365754, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:15:53.303340 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2834] Generated table #35991: 2638241 keys, 74546355 bytes >2018-07-11 05:15:53.303386 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286153303358, "cf_name": "default", "job": 2834, "event": "table_file_creation", "file_number": 35991, "file_size": 74546355, "table_properties": {"data_size": 67112105, "index_size": 690077, "filter_size": 6743183, "raw_key_size": 76215864, "raw_average_key_size": 28, "raw_value_size": 31365767, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:15:56.269920 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2834] Generated table #35992: 2638240 keys, 74545927 bytes >2018-07-11 05:15:56.269968 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286156269940, "cf_name": "default", "job": 2834, "event": "table_file_creation", "file_number": 35992, "file_size": 74545927, "table_properties": {"data_size": 67112095, "index_size": 689672, "filter_size": 6743170, "raw_key_size": 76215819, "raw_average_key_size": 28, "raw_value_size": 31365739, "raw_average_value_size": 11, "num_data_blocks": 16489, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:15:56.270002 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2834] Compacted 1@3 + 2@4 files to L4 => 223638724 bytes >2018-07-11 05:15:56.281411 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:15:56.281219) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 4 44 446 3645 758 0] max score 1.00, MB/sec: 24.6 rd, 24.6 wr, level 4, files in(1, 2) out(3) MB in(71.1, 142.2) out(213.3), read-write-amplify(6.0) write-amplify(3.0) OK, records in: 7914722, records dropped: 0 > >2018-07-11 05:15:56.281421 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:15:56.281242) EVENT_LOG_v1 {"time_micros": 1531286156281232, "job": 2834, "event": "compaction_finished", "compaction_time_micros": 9079229, "output_level": 4, "num_output_files": 3, "total_output_size": 223638724, "num_input_records": 7914722, "num_output_records": 7914722, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 4, 44, 446, 3645, 758, 0]} >2018-07-11 05:15:56.282352 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286156282348, "job": 2834, "event": "table_file_deletion", "file_number": 35988} >2018-07-11 05:15:56.282370 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286156282368, "job": 2834, "event": "table_file_deletion", "file_number": 35937} >2018-07-11 05:15:56.282382 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286156282380, "job": 2834, "event": "table_file_deletion", "file_number": 35936} >2018-07-11 05:15:56.298105 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:15:56.282685) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #31197 to level-5 74546844 bytes > >2018-07-11 05:15:56.298115 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:15:56.297908) EVENT_LOG_v1 {"time_micros": 1531286156297902, "job": 2835, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546844} >2018-07-11 05:15:56.298118 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:15:56.297916) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546844 bytes OK: base level 1 max bytes base 268435456 files[0 4 44 446 3644 759 0] max score 1.00 > >2018-07-11 05:15:56.314682 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:15:56.298979) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #14375 to level-5 74546844 bytes > >2018-07-11 05:15:56.314693 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:15:56.314590) EVENT_LOG_v1 {"time_micros": 1531286156314584, "job": 2836, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546844} >2018-07-11 05:15:56.314696 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:15:56.314598) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546844 bytes OK: base level 1 max bytes base 268435456 files[0 4 44 446 3643 760 0] max score 1.00 > >2018-07-11 05:15:56.331383 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:15:56.315552) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #29525 to level-5 74546844 bytes > >2018-07-11 05:15:56.331392 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:15:56.331297) EVENT_LOG_v1 {"time_micros": 1531286156331291, "job": 2837, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546844} >2018-07-11 05:15:56.331394 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:15:56.331304) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546844 bytes OK: base level 1 max bytes base 268435456 files[0 4 44 446 3642 761 0] max score 1.00 > >2018-07-11 05:15:56.332212 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2838] Compacting 1@3 + 2@4 files to L4, score 1.00 >2018-07-11 05:15:56.332223 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2837 Base level 3, inputs: [35989(71MB)], [35939(71MB) 35961(71MB)] > >2018-07-11 05:15:56.332257 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286156332227, "job": 2838, "event": "compaction_started", "files_L3": [35989], "files_L4": [35939, 35961], "score": 1.00013, "input_data_size": 223639438} >2018-07-11 05:15:59.411696 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2838] Generated table #35993: 2638241 keys, 74546010 bytes >2018-07-11 05:15:59.411742 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286159411715, "cf_name": "default", "job": 2838, "event": "table_file_creation", "file_number": 35993, "file_size": 74546010, "table_properties": {"data_size": 67112168, "index_size": 689669, "filter_size": 6743183, "raw_key_size": 76215857, "raw_average_key_size": 28, "raw_value_size": 31365760, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:16:02.555375 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2838] Generated table #35994: 2638241 keys, 74546369 bytes >2018-07-11 05:16:02.555420 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286162555404, "cf_name": "default", "job": 2838, "event": "table_file_creation", "file_number": 35994, "file_size": 74546369, "table_properties": {"data_size": 67112118, "index_size": 690078, "filter_size": 6743183, "raw_key_size": 76215847, "raw_average_key_size": 28, "raw_value_size": 31365750, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:16:05.603646 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2838] Generated table #35995: 2638240 keys, 74546766 bytes >2018-07-11 05:16:05.603692 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286165603664, "cf_name": "default", "job": 2838, "event": "table_file_creation", "file_number": 35995, "file_size": 74546766, "table_properties": {"data_size": 67112115, "index_size": 690491, "filter_size": 6743170, "raw_key_size": 76215831, "raw_average_key_size": 28, "raw_value_size": 31365751, "raw_average_value_size": 11, "num_data_blocks": 16489, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:16:05.603724 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2838] Compacted 1@3 + 2@4 files to L4 => 223639145 bytes >2018-07-11 05:16:05.614764 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:16:05.614547) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 4 44 445 3643 761 0] max score 1.00, MB/sec: 24.1 rd, 24.1 wr, level 4, files in(1, 2) out(3) MB in(71.1, 142.2) out(213.3), read-write-amplify(6.0) write-amplify(3.0) OK, records in: 7914722, records dropped: 0 > >2018-07-11 05:16:05.614775 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:16:05.614572) EVENT_LOG_v1 {"time_micros": 1531286165614561, "job": 2838, "event": "compaction_finished", "compaction_time_micros": 9271454, "output_level": 4, "num_output_files": 3, "total_output_size": 223639145, "num_input_records": 7914722, "num_output_records": 7914722, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 4, 44, 445, 3643, 761, 0]} >2018-07-11 05:16:05.615733 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286165615729, "job": 2838, "event": "table_file_deletion", "file_number": 35989} >2018-07-11 05:16:05.615762 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286165615759, "job": 2838, "event": "table_file_deletion", "file_number": 35961} >2018-07-11 05:16:05.615773 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286165615771, "job": 2838, "event": "table_file_deletion", "file_number": 35939} >2018-07-11 05:16:05.631341 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:16:05.616065) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #6647 to level-5 74546844 bytes > >2018-07-11 05:16:05.631352 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:16:05.631234) EVENT_LOG_v1 {"time_micros": 1531286165631228, "job": 2839, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546844} >2018-07-11 05:16:05.631371 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:16:05.631242) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546844 bytes OK: base level 1 max bytes base 268435456 files[0 4 44 445 3642 762 0] max score 1.00 > >2018-07-11 05:16:16.415950 7f417420b700 1 bluestore(/var/lib/ceph/osd/ceph-10) _balance_bluefs_freespace gifting 0xd059e00000~80000000 to bluefs >2018-07-11 05:16:16.503415 7f417420b700 1 bluefs add_block_extent bdev 2 0xd059e00000~80000000 >2018-07-11 05:19:34.691977 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:684] reusing log 35965 from recycle list > >2018-07-11 05:19:34.692103 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #35996. Immutable memtables: 0. > >2018-07-11 05:19:34.754702 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:19:34.692163) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1 >2018-07-11 05:19:34.754714 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 2840] Flushing memtable with next log file: 35996 > >2018-07-11 05:19:34.754733 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286374754722, "job": 2840, "event": "flush_started", "num_memtables": 1, "num_entries": 5000016, "num_deletes": 0, "memory_usage": 292951008} >2018-07-11 05:19:34.754736 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 2840] Level-0 flush table #35997: started >2018-07-11 05:19:37.530905 7f41649ec700 1 bluefs _allocate failed to allocate 0x8700000 on bdev 1, free 0x24fe000; fallback to bdev 2 >2018-07-11 05:19:39.283686 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286379283664, "cf_name": "default", "job": 2840, "event": "table_file_creation", "file_number": 35997, "file_size": 141287595, "table_properties": {"data_size": 127198481, "index_size": 1308397, "filter_size": 12779727, "raw_key_size": 144445125, "raw_average_key_size": 28, "raw_value_size": 59451347, "raw_average_value_size": 11, "num_data_blocks": 31252, "num_entries": 5000015, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:19:39.283706 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 2840] Level-0 flush table #35997: 141287595 bytes OK >2018-07-11 05:19:39.300503 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:242] adding log 35967 to recycle list > >2018-07-11 05:19:39.300513 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:19:39.283732) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #35997 started >2018-07-11 05:19:39.300516 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:19:39.300376) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #35997: memtable #1 done >2018-07-11 05:19:39.300518 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:19:39.300391) EVENT_LOG_v1 {"time_micros": 1531286379300385, "job": 2840, "event": "flush_finished", "lsm_state": [1, 4, 44, 445, 3642, 762, 0], "immutable_memtables": 0} >2018-07-11 05:19:39.300520 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:19:39.300452) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 4 44 445 3642 762 0] max score 1.00 > >2018-07-11 05:19:39.301403 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 2840] Try to delete WAL files size 178897451, prev total WAL file size 178897463, number of live WAL files 2. > >2018-07-11 05:21:33.037216 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:684] reusing log 35967 from recycle list > >2018-07-11 05:21:33.037511 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #35998. Immutable memtables: 0. > >2018-07-11 05:21:33.106853 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:21:33.037589) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1 >2018-07-11 05:21:33.106865 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 2841] Flushing memtable with next log file: 35998 > >2018-07-11 05:21:33.106896 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286493106886, "job": 2841, "event": "flush_started", "num_memtables": 1, "num_entries": 5000018, "num_deletes": 0, "memory_usage": 292960808} >2018-07-11 05:21:33.106900 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 2841] Level-0 flush table #35999: started >2018-07-11 05:21:35.762136 7f41649ec700 1 bluefs _allocate failed to allocate 0x8700000 on bdev 1, free 0x24fe000; fallback to bdev 2 >2018-07-11 05:21:37.661028 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286497661006, "cf_name": "default", "job": 2841, "event": "table_file_creation", "file_number": 35999, "file_size": 141287881, "table_properties": {"data_size": 127198897, "index_size": 1308262, "filter_size": 12779732, "raw_key_size": 144445177, "raw_average_key_size": 28, "raw_value_size": 59451623, "raw_average_value_size": 11, "num_data_blocks": 31252, "num_entries": 5000017, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:21:37.661051 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 2841] Level-0 flush table #35999: 141287881 bytes OK >2018-07-11 05:21:37.672569 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:242] adding log 35996 to recycle list > >2018-07-11 05:21:37.672582 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:21:37.661076) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #35999 started >2018-07-11 05:21:37.672586 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:21:37.672308) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #35999: memtable #1 done >2018-07-11 05:21:37.672588 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:21:37.672326) EVENT_LOG_v1 {"time_micros": 1531286497672319, "job": 2841, "event": "flush_finished", "lsm_state": [2, 4, 44, 445, 3642, 762, 0], "immutable_memtables": 0} >2018-07-11 05:21:37.672590 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:21:37.672503) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[2 4 44 445 3642 762 0] max score 1.05 > >2018-07-11 05:21:37.672728 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2842] Compacting 2@0 + 4@1 files to L1, score 1.05 >2018-07-11 05:21:37.672746 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2841 Base level 0, inputs: [35999(134MB) 35997(134MB)], [35969(71MB) 35974(65MB) 35951(5026B) 35925(54MB)] > >2018-07-11 05:21:37.672764 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286497672750, "job": 2842, "event": "compaction_started", "files_L0": [35999, 35997], "files_L1": [35969, 35974, 35951, 35925], "score": 1.05268, "input_data_size": 482124871} >2018-07-11 05:21:37.673501 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 2841] Try to delete WAL files size 178897793, prev total WAL file size 178897805, number of live WAL files 2. > >2018-07-11 05:21:39.841551 7f41641eb700 1 bluefs _allocate failed to allocate 0x4800000 on bdev 1, free 0x24fe000; fallback to bdev 2 >2018-07-11 05:21:40.931775 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2842] Generated table #36000: 2623216 keys, 74507836 bytes >2018-07-11 05:21:40.931818 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286500931792, "cf_name": "default", "job": 2842, "event": "table_file_creation", "file_number": 36000, "file_size": 74507836, "table_properties": {"data_size": 67109707, "index_size": 691510, "filter_size": 6705629, "raw_key_size": 75815524, "raw_average_key_size": 28, "raw_value_size": 31558654, "raw_average_value_size": 12, "num_data_blocks": 16475, "num_entries": 2623216, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:21:42.955809 7f41641eb700 1 bluefs _allocate failed to allocate 0x4800000 on bdev 1, free 0x24fe000; fallback to bdev 2 >2018-07-11 05:21:43.998719 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2842] Generated table #36001: 2638241 keys, 74546869 bytes >2018-07-11 05:21:43.998764 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286503998737, "cf_name": "default", "job": 2842, "event": "table_file_creation", "file_number": 36001, "file_size": 74546869, "table_properties": {"data_size": 67112120, "index_size": 690576, "filter_size": 6743183, "raw_key_size": 76215845, "raw_average_key_size": 28, "raw_value_size": 31365748, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:21:46.741014 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2842] Generated table #36002: 2638241 keys, 74546503 bytes >2018-07-11 05:21:46.741058 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286506741032, "cf_name": "default", "job": 2842, "event": "table_file_creation", "file_number": 36002, "file_size": 74546503, "table_properties": {"data_size": 67112153, "index_size": 690177, "filter_size": 6743183, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:21:49.353684 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2842] Generated table #36003: 2638241 keys, 74546405 bytes >2018-07-11 05:21:49.353736 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286509353707, "cf_name": "default", "job": 2842, "event": "table_file_creation", "file_number": 36003, "file_size": 74546405, "table_properties": {"data_size": 67112087, "index_size": 690145, "filter_size": 6743183, "raw_key_size": 76215853, "raw_average_key_size": 28, "raw_value_size": 31365756, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:21:51.761434 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2842] Generated table #36004: 2085397 keys, 58924958 bytes >2018-07-11 05:21:51.761478 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286511761462, "cf_name": "default", "job": 2842, "event": "table_file_creation", "file_number": 36004, "file_size": 58924958, "table_properties": {"data_size": 53048796, "index_size": 545034, "filter_size": 5330140, "raw_key_size": 60244815, "raw_average_key_size": 28, "raw_value_size": 24793066, "raw_average_value_size": 11, "num_data_blocks": 13034, "num_entries": 2085397, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:21:53.160778 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2842] Generated table #36005: 166209 keys, 68248581 bytes >2018-07-11 05:21:53.160819 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286513160792, "cf_name": "default", "job": 2842, "event": "table_file_creation", "file_number": 36005, "file_size": 68248581, "table_properties": {"data_size": 67113670, "index_size": 564813, "filter_size": 569110, "raw_key_size": 15377257, "raw_average_key_size": 92, "raw_value_size": 53113921, "raw_average_value_size": 319, "num_data_blocks": 15216, "num_entries": 166209, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "80898", "kMergeOperands": "0"}} >2018-07-11 05:21:54.182043 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2842] Generated table #36006: 138305 keys, 56795920 bytes >2018-07-11 05:21:54.182085 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286514182071, "cf_name": "default", "job": 2842, "event": "table_file_creation", "file_number": 36006, "file_size": 56795920, "table_properties": {"data_size": 55844970, "index_size": 475355, "filter_size": 474607, "raw_key_size": 6759915, "raw_average_key_size": 48, "raw_value_size": 49590453, "raw_average_value_size": 358, "num_data_blocks": 13436, "num_entries": 138305, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "58265"}} >2018-07-11 05:21:54.182151 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2842] Compacted 2@0 + 4@1 files to L1 => 482117072 bytes >2018-07-11 05:21:54.197327 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:21:54.197068) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 7 44 445 3642 762 0] max score 1.80, MB/sec: 29.2 rd, 29.2 wr, level 1, files in(2, 4) out(7) MB in(269.5, 190.3) out(459.8), read-write-amplify(3.4) write-amplify(1.7) OK, records in: 12927859, records dropped: 9 > >2018-07-11 05:21:54.197338 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:21:54.197093) EVENT_LOG_v1 {"time_micros": 1531286514197082, "job": 2842, "event": "compaction_finished", "compaction_time_micros": 16509369, "output_level": 1, "num_output_files": 7, "total_output_size": 482117072, "num_input_records": 12927859, "num_output_records": 12927850, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 7, 44, 445, 3642, 762, 0]} >2018-07-11 05:21:54.198269 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286514198265, "job": 2842, "event": "table_file_deletion", "file_number": 35999} >2018-07-11 05:21:54.198285 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286514198283, "job": 2842, "event": "table_file_deletion", "file_number": 35997} >2018-07-11 05:21:54.198299 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286514198297, "job": 2842, "event": "table_file_deletion", "file_number": 35974} >2018-07-11 05:21:54.198312 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286514198310, "job": 2842, "event": "table_file_deletion", "file_number": 35969} >2018-07-11 05:21:54.198325 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286514198323, "job": 2842, "event": "table_file_deletion", "file_number": 35951} >2018-07-11 05:21:54.198337 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286514198335, "job": 2842, "event": "table_file_deletion", "file_number": 35925} >2018-07-11 05:21:54.198552 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_picker.cc:461] [default] Expanding@1 1+5(74546869+357743806 bytes) to 4+5 (282564735+357743806bytes) > >2018-07-11 05:21:54.198650 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2843] Compacting 4@1 + 5@2 files to L2, score 1.80 >2018-07-11 05:21:54.198663 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2842 Base level 1, inputs: [36001(71MB) 36002(71MB) 36003(71MB) 36004(56MB)], [35975(71MB) 35976(71MB) 35977(71MB) 35980(71MB) 35983(56MB)] > >2018-07-11 05:21:54.198678 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286514198666, "job": 2843, "event": "compaction_started", "files_L1": [36001, 36002, 36003, 36004], "files_L2": [35975, 35976, 35977, 35980, 35983], "score": 1.79603, "input_data_size": 640308541} >2018-07-11 05:21:57.411008 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2843] Generated table #36007: 2638240 keys, 74545798 bytes >2018-07-11 05:21:57.411054 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286517411038, "cf_name": "default", "job": 2843, "event": "table_file_creation", "file_number": 36007, "file_size": 74545798, "table_properties": {"data_size": 67112119, "index_size": 689508, "filter_size": 6743181, "raw_key_size": 76215816, "raw_average_key_size": 28, "raw_value_size": 31365736, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:22:00.618166 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2843] Generated table #36008: 2638239 keys, 74546197 bytes >2018-07-11 05:22:00.618211 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286520618183, "cf_name": "default", "job": 2843, "event": "table_file_creation", "file_number": 36008, "file_size": 74546197, "table_properties": {"data_size": 67112127, "index_size": 689901, "filter_size": 6743179, "raw_key_size": 76215789, "raw_average_key_size": 28, "raw_value_size": 31365726, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638239, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:22:03.782855 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2843] Generated table #36009: 2638240 keys, 74546302 bytes >2018-07-11 05:22:03.782896 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286523782879, "cf_name": "default", "job": 2843, "event": "table_file_creation", "file_number": 36009, "file_size": 74546302, "table_properties": {"data_size": 67112135, "index_size": 689995, "filter_size": 6743182, "raw_key_size": 76215825, "raw_average_key_size": 28, "raw_value_size": 31365745, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:22:06.950125 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2843] Generated table #36010: 2638240 keys, 74546824 bytes >2018-07-11 05:22:06.950169 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286526950151, "cf_name": "default", "job": 2843, "event": "table_file_creation", "file_number": 36010, "file_size": 74546824, "table_properties": {"data_size": 67112154, "index_size": 690499, "filter_size": 6743181, "raw_key_size": 76215823, "raw_average_key_size": 28, "raw_value_size": 31365743, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:22:10.104700 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2843] Generated table #36011: 2638241 keys, 74547004 bytes >2018-07-11 05:22:10.104744 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286530104728, "cf_name": "default", "job": 2843, "event": "table_file_creation", "file_number": 36011, "file_size": 74547004, "table_properties": {"data_size": 67112111, "index_size": 690720, "filter_size": 6743183, "raw_key_size": 76215855, "raw_average_key_size": 28, "raw_value_size": 31365758, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:22:13.493635 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2843] Generated table #36012: 2638241 keys, 74546332 bytes >2018-07-11 05:22:13.493687 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286533493670, "cf_name": "default", "job": 2843, "event": "table_file_creation", "file_number": 36012, "file_size": 74546332, "table_properties": {"data_size": 67112154, "index_size": 690005, "filter_size": 6743183, "raw_key_size": 76215849, "raw_average_key_size": 28, "raw_value_size": 31365752, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:22:17.514580 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2843] Generated table #36013: 2638241 keys, 74546468 bytes >2018-07-11 05:22:17.514623 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286537514606, "cf_name": "default", "job": 2843, "event": "table_file_creation", "file_number": 36013, "file_size": 74546468, "table_properties": {"data_size": 67112105, "index_size": 690190, "filter_size": 6743183, "raw_key_size": 76215856, "raw_average_key_size": 28, "raw_value_size": 31365759, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:22:21.716803 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2843] Generated table #36014: 2638241 keys, 74545949 bytes >2018-07-11 05:22:21.716859 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286541716839, "cf_name": "default", "job": 2843, "event": "table_file_creation", "file_number": 36014, "file_size": 74545949, "table_properties": {"data_size": 67112172, "index_size": 689604, "filter_size": 6743183, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:22:24.437016 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2843] Generated table #36015: 1550551 keys, 43938949 bytes >2018-07-11 05:22:24.437054 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286544437030, "cf_name": "default", "job": 2843, "event": "table_file_creation", "file_number": 36015, "file_size": 43938949, "table_properties": {"data_size": 39567815, "index_size": 406739, "filter_size": 3963407, "raw_key_size": 44809453, "raw_average_key_size": 28, "raw_value_size": 18539708, "raw_average_value_size": 11, "num_data_blocks": 9724, "num_entries": 1550551, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "11", "kMergeOperands": "0"}} >2018-07-11 05:22:24.437088 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2843] Compacted 4@1 + 5@2 files to L2 => 640309823 bytes >2018-07-11 05:22:24.463466 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:22:24.463091) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 3 48 445 3642 762 0] max score 1.09, MB/sec: 21.2 rd, 21.2 wr, level 2, files in(4, 5) out(9) MB in(269.5, 341.2) out(610.6), read-write-amplify(4.5) write-amplify(2.3) OK, records in: 22656474, records dropped: 0 > >2018-07-11 05:22:24.463478 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:22:24.463116) EVENT_LOG_v1 {"time_micros": 1531286544463104, "job": 2843, "event": "compaction_finished", "compaction_time_micros": 30238394, "output_level": 2, "num_output_files": 9, "total_output_size": 640309823, "num_input_records": 22656474, "num_output_records": 22656474, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 3, 48, 445, 3642, 762, 0]} >2018-07-11 05:22:24.464449 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286544464446, "job": 2843, "event": "table_file_deletion", "file_number": 36004} >2018-07-11 05:22:24.464464 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286544464461, "job": 2843, "event": "table_file_deletion", "file_number": 36003} >2018-07-11 05:22:24.464475 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286544464473, "job": 2843, "event": "table_file_deletion", "file_number": 36002} >2018-07-11 05:22:24.464487 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286544464485, "job": 2843, "event": "table_file_deletion", "file_number": 36001} >2018-07-11 05:22:24.464503 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286544464500, "job": 2843, "event": "table_file_deletion", "file_number": 35983} >2018-07-11 05:22:24.464521 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286544464518, "job": 2843, "event": "table_file_deletion", "file_number": 35980} >2018-07-11 05:22:24.464532 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286544464530, "job": 2843, "event": "table_file_deletion", "file_number": 35977} >2018-07-11 05:22:24.464545 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286544464543, "job": 2843, "event": "table_file_deletion", "file_number": 35976} >2018-07-11 05:22:24.464560 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286544464557, "job": 2843, "event": "table_file_deletion", "file_number": 35975} >2018-07-11 05:22:24.464796 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_picker.cc:461] [default] Expanding@2 1+1(74547004+46965204 bytes) to 4+1 (298185753+46965204bytes) > >2018-07-11 05:22:24.464894 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2844] Compacting 4@2 + 1@3 files to L3, score 1.09 >2018-07-11 05:22:24.464906 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2843 Base level 2, inputs: [36011(71MB) 36012(71MB) 36013(71MB) 36014(71MB)], [35986(44MB)] > >2018-07-11 05:22:24.464919 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286544464909, "job": 2844, "event": "compaction_started", "files_L2": [36011, 36012, 36013, 36014], "files_L3": [35986], "score": 1.0903, "input_data_size": 345150957} >2018-07-11 05:22:27.632235 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2844] Generated table #36016: 2638240 keys, 74547055 bytes >2018-07-11 05:22:27.632287 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286547632271, "cf_name": "default", "job": 2844, "event": "table_file_creation", "file_number": 36016, "file_size": 74547055, "table_properties": {"data_size": 67112147, "index_size": 690737, "filter_size": 6743181, "raw_key_size": 76215819, "raw_average_key_size": 28, "raw_value_size": 31365739, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:22:30.749743 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2844] Generated table #36017: 2638241 keys, 74546249 bytes >2018-07-11 05:22:30.749788 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286550749773, "cf_name": "default", "job": 2844, "event": "table_file_creation", "file_number": 36017, "file_size": 74546249, "table_properties": {"data_size": 67112108, "index_size": 689968, "filter_size": 6743183, "raw_key_size": 76215864, "raw_average_key_size": 28, "raw_value_size": 31365767, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:22:33.907373 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2844] Generated table #36018: 2638241 keys, 74546298 bytes >2018-07-11 05:22:33.907427 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286553907398, "cf_name": "default", "job": 2844, "event": "table_file_creation", "file_number": 36018, "file_size": 74546298, "table_properties": {"data_size": 67112155, "index_size": 689970, "filter_size": 6743183, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:22:37.023216 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2844] Generated table #36019: 2638241 keys, 74545744 bytes >2018-07-11 05:22:37.023267 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286557023251, "cf_name": "default", "job": 2844, "event": "table_file_creation", "file_number": 36019, "file_size": 74545744, "table_properties": {"data_size": 67112106, "index_size": 689465, "filter_size": 6743183, "raw_key_size": 76215852, "raw_average_key_size": 28, "raw_value_size": 31365755, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:22:38.902592 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2844] Generated table #36020: 1519917 keys, 46965251 bytes >2018-07-11 05:22:38.902638 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286558902611, "cf_name": "default", "job": 2844, "event": "table_file_creation", "file_number": 36020, "file_size": 46965251, "table_properties": {"data_size": 42636693, "index_size": 433417, "filter_size": 3894153, "raw_key_size": 44348961, "raw_average_key_size": 29, "raw_value_size": 21522473, "raw_average_value_size": 14, "num_data_blocks": 10538, "num_entries": 1519917, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:22:38.902699 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2844] Compacted 4@2 + 1@3 files to L3 => 345150597 bytes >2018-07-11 05:22:38.921461 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:22:38.921129) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 3 44 449 3642 762 0] max score 1.01, MB/sec: 23.9 rd, 23.9 wr, level 3, files in(4, 1) out(5) MB in(284.4, 44.8) out(329.2), read-write-amplify(2.3) write-amplify(1.2) OK, records in: 12072880, records dropped: 0 > >2018-07-11 05:22:38.921473 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:22:38.921155) EVENT_LOG_v1 {"time_micros": 1531286558921144, "job": 2844, "event": "compaction_finished", "compaction_time_micros": 14437750, "output_level": 3, "num_output_files": 5, "total_output_size": 345150597, "num_input_records": 12072880, "num_output_records": 12072880, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 3, 44, 449, 3642, 762, 0]} >2018-07-11 05:22:38.922406 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286558922402, "job": 2844, "event": "table_file_deletion", "file_number": 36014} >2018-07-11 05:22:38.922421 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286558922419, "job": 2844, "event": "table_file_deletion", "file_number": 36013} >2018-07-11 05:22:38.922433 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286558922430, "job": 2844, "event": "table_file_deletion", "file_number": 36012} >2018-07-11 05:22:38.922444 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286558922441, "job": 2844, "event": "table_file_deletion", "file_number": 36011} >2018-07-11 05:22:38.922457 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286558922455, "job": 2844, "event": "table_file_deletion", "file_number": 35986} >2018-07-11 05:22:38.922768 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2845] Compacting 1@3 + 2@4 files to L4, score 1.01 >2018-07-11 05:22:38.922791 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2844 Base level 3, inputs: [36016(71MB)], [35984(71MB) 35985(71MB)] > >2018-07-11 05:22:38.922805 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286558922795, "job": 2845, "event": "compaction_started", "files_L3": [36016], "files_L4": [35984, 35985], "score": 1.00846, "input_data_size": 223639942} >2018-07-11 05:22:42.192202 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2845] Generated table #36021: 2638241 keys, 74546279 bytes >2018-07-11 05:22:42.192262 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286562192239, "cf_name": "default", "job": 2845, "event": "table_file_creation", "file_number": 36021, "file_size": 74546279, "table_properties": {"data_size": 67112145, "index_size": 689961, "filter_size": 6743183, "raw_key_size": 76215860, "raw_average_key_size": 28, "raw_value_size": 31365763, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:22:45.252821 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2845] Generated table #36022: 2638241 keys, 74545935 bytes >2018-07-11 05:22:45.252856 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286565252840, "cf_name": "default", "job": 2845, "event": "table_file_creation", "file_number": 36022, "file_size": 74545935, "table_properties": {"data_size": 67112128, "index_size": 689634, "filter_size": 6743183, "raw_key_size": 76215852, "raw_average_key_size": 28, "raw_value_size": 31365755, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:22:48.253519 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2845] Generated table #36023: 2638241 keys, 74547071 bytes >2018-07-11 05:22:48.253578 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286568253560, "cf_name": "default", "job": 2845, "event": "table_file_creation", "file_number": 36023, "file_size": 74547071, "table_properties": {"data_size": 67112161, "index_size": 690737, "filter_size": 6743183, "raw_key_size": 76215848, "raw_average_key_size": 28, "raw_value_size": 31365751, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:22:48.253618 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2845] Compacted 1@3 + 2@4 files to L4 => 223639285 bytes >2018-07-11 05:22:48.271294 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:22:48.271032) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 3 44 448 3643 762 0] max score 1.01, MB/sec: 24.0 rd, 24.0 wr, level 4, files in(1, 2) out(3) MB in(71.1, 142.2) out(213.3), read-write-amplify(6.0) write-amplify(3.0) OK, records in: 7914723, records dropped: 0 > >2018-07-11 05:22:48.271306 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:22:48.271058) EVENT_LOG_v1 {"time_micros": 1531286568271047, "job": 2845, "event": "compaction_finished", "compaction_time_micros": 9330798, "output_level": 4, "num_output_files": 3, "total_output_size": 223639285, "num_input_records": 7914723, "num_output_records": 7914723, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 3, 44, 448, 3643, 762, 0]} >2018-07-11 05:22:48.272254 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286568272250, "job": 2845, "event": "table_file_deletion", "file_number": 36016} >2018-07-11 05:22:48.272271 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286568272269, "job": 2845, "event": "table_file_deletion", "file_number": 35985} >2018-07-11 05:22:48.272285 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286568272283, "job": 2845, "event": "table_file_deletion", "file_number": 35984} >2018-07-11 05:22:48.272601 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2846] Compacting 1@3 + 9@4 files to L4, score 1.01 >2018-07-11 05:22:48.272616 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2845 Base level 3, inputs: [35978(71MB)], [35882(71MB) 35884(1064B) 35880(71MB) 35881(71MB) 35847(71MB) 35848(71MB) 35825(71MB) 35826(71MB) 35912(71MB)] > >2018-07-11 05:22:48.272630 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286568272619, "job": 2846, "event": "compaction_started", "files_L3": [35978], "files_L4": [35882, 35884, 35880, 35881, 35847, 35848, 35825, 35826, 35912], "score": 1.00568, "input_data_size": 670918603} >2018-07-11 05:22:52.037542 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2846] Generated table #36024: 2638241 keys, 74546384 bytes >2018-07-11 05:22:52.037576 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286572037561, "cf_name": "default", "job": 2846, "event": "table_file_creation", "file_number": 36024, "file_size": 74546384, "table_properties": {"data_size": 67112107, "index_size": 690104, "filter_size": 6743183, "raw_key_size": 76215855, "raw_average_key_size": 28, "raw_value_size": 31365758, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:22:55.506975 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2846] Generated table #36025: 2638240 keys, 74547025 bytes >2018-07-11 05:22:55.507014 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286575506998, "cf_name": "default", "job": 2846, "event": "table_file_creation", "file_number": 36025, "file_size": 74547025, "table_properties": {"data_size": 67112156, "index_size": 690698, "filter_size": 6743181, "raw_key_size": 76215828, "raw_average_key_size": 28, "raw_value_size": 31365748, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:22:58.929274 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2846] Generated table #36026: 2638241 keys, 74546045 bytes >2018-07-11 05:22:58.929332 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286578929300, "cf_name": "default", "job": 2846, "event": "table_file_creation", "file_number": 36026, "file_size": 74546045, "table_properties": {"data_size": 67112117, "index_size": 689755, "filter_size": 6743183, "raw_key_size": 76215851, "raw_average_key_size": 28, "raw_value_size": 31365754, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:23:02.250162 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2846] Generated table #36027: 2638240 keys, 74546280 bytes >2018-07-11 05:23:02.250217 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286582250200, "cf_name": "default", "job": 2846, "event": "table_file_creation", "file_number": 36027, "file_size": 74546280, "table_properties": {"data_size": 67112155, "index_size": 689954, "filter_size": 6743181, "raw_key_size": 76215822, "raw_average_key_size": 28, "raw_value_size": 31365742, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:23:05.601276 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2846] Generated table #36028: 2638241 keys, 74545690 bytes >2018-07-11 05:23:05.601331 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286585601300, "cf_name": "default", "job": 2846, "event": "table_file_creation", "file_number": 36028, "file_size": 74545690, "table_properties": {"data_size": 67112137, "index_size": 689380, "filter_size": 6743183, "raw_key_size": 76215860, "raw_average_key_size": 28, "raw_value_size": 31365763, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:23:08.976837 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2846] Generated table #36029: 2638241 keys, 74546756 bytes >2018-07-11 05:23:08.976882 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286588976854, "cf_name": "default", "job": 2846, "event": "table_file_creation", "file_number": 36029, "file_size": 74546756, "table_properties": {"data_size": 67112144, "index_size": 690439, "filter_size": 6743183, "raw_key_size": 76215852, "raw_average_key_size": 28, "raw_value_size": 31365755, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:23:12.087238 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2846] Generated table #36030: 2638242 keys, 74546347 bytes >2018-07-11 05:23:12.087286 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286592087269, "cf_name": "default", "job": 2846, "event": "table_file_creation", "file_number": 36030, "file_size": 74546347, "table_properties": {"data_size": 67112157, "index_size": 690014, "filter_size": 6743186, "raw_key_size": 76215877, "raw_average_key_size": 28, "raw_value_size": 31365763, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638242, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:23:13.111078 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:684] reusing log 35996 from recycle list > >2018-07-11 05:23:13.111177 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #36032. Immutable memtables: 0. > >2018-07-11 05:23:13.182003 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:23:13.111211) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 0 >2018-07-11 05:23:13.182017 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 2847] Flushing memtable with next log file: 36032 > >2018-07-11 05:23:13.182040 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286593182028, "job": 2847, "event": "flush_started", "num_memtables": 1, "num_entries": 5000015, "num_deletes": 0, "memory_usage": 292949712} >2018-07-11 05:23:13.182045 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 2847] Level-0 flush table #36033: started >2018-07-11 05:23:15.264074 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2846] Generated table #36031: 2638241 keys, 74546117 bytes >2018-07-11 05:23:15.264118 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286595264101, "cf_name": "default", "job": 2846, "event": "table_file_creation", "file_number": 36031, "file_size": 74546117, "table_properties": {"data_size": 67112166, "index_size": 689778, "filter_size": 6743183, "raw_key_size": 76215856, "raw_average_key_size": 28, "raw_value_size": 31365759, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:23:17.148757 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286597148735, "cf_name": "default", "job": 2847, "event": "table_file_creation", "file_number": 36033, "file_size": 141286888, "table_properties": {"data_size": 127197776, "index_size": 1308397, "filter_size": 12779725, "raw_key_size": 144445101, "raw_average_key_size": 28, "raw_value_size": 59450671, "raw_average_value_size": 11, "num_data_blocks": 31252, "num_entries": 5000014, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:23:17.148782 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 2847] Level-0 flush table #36033: 141286888 bytes OK >2018-07-11 05:23:17.162218 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:242] adding log 35998 to recycle list > >2018-07-11 05:23:17.162230 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:23:17.148796) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #36033 started >2018-07-11 05:23:17.162236 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:23:17.162090) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #36033: memtable #1 done >2018-07-11 05:23:17.162239 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:23:17.162107) EVENT_LOG_v1 {"time_micros": 1531286597162100, "job": 2847, "event": "flush_finished", "lsm_state": [1, 3, 44, 448, 3643, 762, 0], "immutable_memtables": 0} >2018-07-11 05:23:17.162243 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:23:17.162125) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 3 44 448 3643 762 0] max score 1.00 > >2018-07-11 05:23:17.163369 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 2847] Try to delete WAL files size 178896755, prev total WAL file size 178896767, number of live WAL files 2. > >2018-07-11 05:23:18.660113 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2846] Generated table #36034: 2638240 keys, 74546082 bytes >2018-07-11 05:23:18.660156 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286598660129, "cf_name": "default", "job": 2846, "event": "table_file_creation", "file_number": 36034, "file_size": 74546082, "table_properties": {"data_size": 67112052, "index_size": 689870, "filter_size": 6743170, "raw_key_size": 76215827, "raw_average_key_size": 28, "raw_value_size": 31365747, "raw_average_value_size": 11, "num_data_blocks": 16489, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:23:18.660198 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2846] Compacted 1@3 + 9@4 files to L4 => 670916726 bytes >2018-07-11 05:23:18.679263 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:23:18.678856) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[1 3 44 447 3643 762 0] max score 1.00, MB/sec: 22.1 rd, 22.1 wr, level 4, files in(1, 9) out(9) MB in(71.1, 568.7) out(639.8), read-write-amplify(18.0) write-amplify(9.0) OK, records in: 23744167, records dropped: 0 > >2018-07-11 05:23:18.679274 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:23:18.678883) EVENT_LOG_v1 {"time_micros": 1531286598678871, "job": 2846, "event": "compaction_finished", "compaction_time_micros": 30387546, "output_level": 4, "num_output_files": 9, "total_output_size": 670916726, "num_input_records": 23744167, "num_output_records": 23744167, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 3, 44, 447, 3643, 762, 0]} >2018-07-11 05:23:18.680231 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286598680226, "job": 2846, "event": "table_file_deletion", "file_number": 35978} >2018-07-11 05:23:18.680263 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286598680259, "job": 2846, "event": "table_file_deletion", "file_number": 35912} >2018-07-11 05:23:18.680286 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286598680282, "job": 2846, "event": "table_file_deletion", "file_number": 35884} >2018-07-11 05:23:18.680314 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286598680310, "job": 2846, "event": "table_file_deletion", "file_number": 35882} >2018-07-11 05:23:18.680338 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286598680335, "job": 2846, "event": "table_file_deletion", "file_number": 35881} >2018-07-11 05:23:18.680362 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286598680358, "job": 2846, "event": "table_file_deletion", "file_number": 35880} >2018-07-11 05:23:18.680393 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286598680389, "job": 2846, "event": "table_file_deletion", "file_number": 35848} >2018-07-11 05:23:18.680422 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286598680417, "job": 2846, "event": "table_file_deletion", "file_number": 35847} >2018-07-11 05:23:18.680449 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286598680445, "job": 2846, "event": "table_file_deletion", "file_number": 35826} >2018-07-11 05:23:18.680474 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286598680470, "job": 2846, "event": "table_file_deletion", "file_number": 35825} >2018-07-11 05:23:18.695711 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:23:18.680794) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #36018 to level-4 74546298 bytes > >2018-07-11 05:23:18.695722 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:23:18.695508) EVENT_LOG_v1 {"time_micros": 1531286598695502, "job": 2848, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 74546298} >2018-07-11 05:23:18.695726 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:23:18.695516) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-4 74546298 bytes OK: base level 1 max bytes base 268435456 files[1 3 44 446 3644 762 0] max score 1.00 > >2018-07-11 05:23:18.712319 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:23:18.696618) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #36023 to level-5 74547071 bytes > >2018-07-11 05:23:18.712330 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:23:18.712220) EVENT_LOG_v1 {"time_micros": 1531286598712214, "job": 2849, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74547071} >2018-07-11 05:23:18.712332 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:23:18.712228) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74547071 bytes OK: base level 1 max bytes base 268435456 files[1 3 44 446 3643 763 0] max score 1.00 > >2018-07-11 05:23:18.713168 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2850] Compacting 1@4 + 2@5 files to L5, score 1.00 >2018-07-11 05:23:18.713179 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2849 Base level 4, inputs: [36025(71MB)], [35885(71MB) 35886(71MB)] > >2018-07-11 05:23:18.713207 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286598713196, "job": 2850, "event": "compaction_started", "files_L4": [36025], "files_L5": [35885, 35886], "score": 1.00055, "input_data_size": 223640136} >2018-07-11 05:23:22.067162 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2850] Generated table #36035: 2638241 keys, 74546998 bytes >2018-07-11 05:23:22.067202 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286602067182, "cf_name": "default", "job": 2850, "event": "table_file_creation", "file_number": 36035, "file_size": 74546998, "table_properties": {"data_size": 67112160, "index_size": 690665, "filter_size": 6743183, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:23:25.221914 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2850] Generated table #36036: 2638241 keys, 74546003 bytes >2018-07-11 05:23:25.221970 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286605221953, "cf_name": "default", "job": 2850, "event": "table_file_creation", "file_number": 36036, "file_size": 74546003, "table_properties": {"data_size": 67112101, "index_size": 689729, "filter_size": 6743183, "raw_key_size": 76215856, "raw_average_key_size": 28, "raw_value_size": 31365759, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:23:28.228577 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2850] Generated table #36037: 2638239 keys, 74546178 bytes >2018-07-11 05:23:28.228630 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286608228600, "cf_name": "default", "job": 2850, "event": "table_file_creation", "file_number": 36037, "file_size": 74546178, "table_properties": {"data_size": 67112095, "index_size": 689925, "filter_size": 6743168, "raw_key_size": 76215796, "raw_average_key_size": 28, "raw_value_size": 31365733, "raw_average_value_size": 11, "num_data_blocks": 16489, "num_entries": 2638239, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:23:28.228667 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2850] Compacted 1@4 + 2@5 files to L5 => 223639179 bytes >2018-07-11 05:23:28.245659 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:23:28.245364) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[1 3 44 446 3642 764 0] max score 1.00, MB/sec: 23.5 rd, 23.5 wr, level 5, files in(1, 2) out(3) MB in(71.1, 142.2) out(213.3), read-write-amplify(6.0) write-amplify(3.0) OK, records in: 7914721, records dropped: 0 > >2018-07-11 05:23:28.245670 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:23:28.245389) EVENT_LOG_v1 {"time_micros": 1531286608245378, "job": 2850, "event": "compaction_finished", "compaction_time_micros": 9515445, "output_level": 5, "num_output_files": 3, "total_output_size": 223639179, "num_input_records": 7914721, "num_output_records": 7914721, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 3, 44, 446, 3642, 764, 0]} >2018-07-11 05:23:28.246651 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286608246647, "job": 2850, "event": "table_file_deletion", "file_number": 36025} >2018-07-11 05:23:28.246681 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286608246679, "job": 2850, "event": "table_file_deletion", "file_number": 35886} >2018-07-11 05:23:28.246693 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286608246691, "job": 2850, "event": "table_file_deletion", "file_number": 35885} >2018-07-11 05:23:28.378837 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:23:28.247005) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #30684 to level-5 74546843 bytes > >2018-07-11 05:23:28.378849 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:23:28.378705) EVENT_LOG_v1 {"time_micros": 1531286608378697, "job": 2851, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546843} >2018-07-11 05:23:28.378852 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:23:28.378716) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546843 bytes OK: base level 1 max bytes base 268435456 files[1 3 44 446 3641 765 0] max score 1.00 > >2018-07-11 05:23:28.395484 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:23:28.380024) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #36017 to level-4 74546249 bytes > >2018-07-11 05:23:28.395495 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:23:28.395391) EVENT_LOG_v1 {"time_micros": 1531286608395385, "job": 2852, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 74546249} >2018-07-11 05:23:28.395498 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:23:28.395399) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-4 74546249 bytes OK: base level 1 max bytes base 268435456 files[1 3 44 445 3642 765 0] max score 1.00 > >2018-07-11 05:23:28.412171 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:23:28.396359) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #7688 to level-5 74546842 bytes > >2018-07-11 05:23:28.412181 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:23:28.412077) EVENT_LOG_v1 {"time_micros": 1531286608412071, "job": 2853, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546842} >2018-07-11 05:23:28.412184 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:23:28.412086) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546842 bytes OK: base level 1 max bytes base 268435456 files[1 3 44 445 3641 766 0] max score 1.00 > >2018-07-11 05:25:56.616128 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:684] reusing log 35998 from recycle list > >2018-07-11 05:25:56.616275 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #36038. Immutable memtables: 0. > >2018-07-11 05:25:56.666323 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:25:56.616338) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1 >2018-07-11 05:25:56.666338 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 2854] Flushing memtable with next log file: 36038 > >2018-07-11 05:25:56.666356 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286756666346, "job": 2854, "event": "flush_started", "num_memtables": 1, "num_entries": 5000015, "num_deletes": 0, "memory_usage": 292932944} >2018-07-11 05:25:56.666360 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 2854] Level-0 flush table #36039: started >2018-07-11 05:26:00.677144 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286760677123, "cf_name": "default", "job": 2854, "event": "table_file_creation", "file_number": 36039, "file_size": 141288967, "table_properties": {"data_size": 127198839, "index_size": 1309411, "filter_size": 12779727, "raw_key_size": 144445125, "raw_average_key_size": 28, "raw_value_size": 59451570, "raw_average_value_size": 11, "num_data_blocks": 31252, "num_entries": 5000015, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:26:00.677163 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 2854] Level-0 flush table #36039: 141288967 bytes OK >2018-07-11 05:26:00.691529 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:242] adding log 36032 to recycle list > >2018-07-11 05:26:00.691541 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:26:00.677187) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #36039 started >2018-07-11 05:26:00.691544 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:26:00.691288) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #36039: memtable #1 done >2018-07-11 05:26:00.691547 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:26:00.691306) EVENT_LOG_v1 {"time_micros": 1531286760691298, "job": 2854, "event": "flush_finished", "lsm_state": [2, 3, 44, 445, 3641, 766, 0], "immutable_memtables": 0} >2018-07-11 05:26:00.691549 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:26:00.691475) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[2 3 44 445 3641 766 0] max score 1.05 > >2018-07-11 05:26:00.691930 7f41641eb700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS ------- >2018-07-11 05:26:00.691939 7f41641eb700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] >** DB Stats ** >Uptime(secs): 51502.3 total, 630.1 interval >Cumulative writes: 196K writes, 1907M keys, 196K commit groups, 1.0 writes per commit group, ingest: 64.06 GB, 1.27 MB/s >Cumulative WAL: 196K writes, 56K syncs, 3.48 writes per sync, written: 64.06 GB, 1.27 MB/s >Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent >Interval writes: 42 writes, 20M keys, 42 commit groups, 1.0 writes per commit group, ingest: 682.44 MB, 1.08 MB/s >Interval WAL: 42 writes, 21 syncs, 1.91 writes per sync, written: 0.67 MB, 1.08 MB/s >Interval stall: 00:00:0.000 H:M:S, 0.0 percent > >** Compaction Stats [default] ** >Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop >---------------------------------------------------------------------------------------------------------------------------------------------------------- > L0 2/0 269.49 MB 1.1 0.0 0.0 0.0 50.2 50.2 0.0 1.0 0.0 32.4 1586 383 4.141 0 0 > L1 3/0 190.31 MB 0.7 82.4 50.0 32.4 82.4 50.0 0.0 1.6 25.7 25.7 3281 188 17.450 2606M 894K > L2 44/0 2.45 GB 1.0 97.6 50.0 47.5 97.5 50.0 0.0 1.9 21.5 21.5 4636 560 8.279 3643M 1358K > L3 445/0 24.93 GB 1.0 86.9 49.3 37.6 86.8 49.2 0.7 1.8 20.8 20.8 4276 297 14.397 3281M 4000K > L4 3641/0 250.00 GB 1.0 74.8 17.9 56.8 74.5 17.6 32.0 4.1 21.2 21.1 3615 241 15.000 2845M 15M > L5 766/0 53.16 GB 0.0 2.1 1.0 1.0 2.1 1.0 48.5 2.0 19.3 19.1 111 14 7.906 79M 1097K > Sum 4901/0 330.99 GB 0.0 343.7 168.4 175.4 393.4 218.0 81.2 7.8 20.1 23.0 17505 1683 10.401 12G 23M > Int 0/0 0.00 KB 0.0 3.2 1.4 1.9 3.7 1.9 0.8 7.1 20.3 23.6 162 14 11.600 117M 9 >Uptime(secs): 51502.3 total, 51502.3 interval >Flush(GB): cumulative 50.181, interval 0.526 >AddFile(GB): cumulative 0.000, interval 0.000 >AddFile(Total Files): cumulative 0, interval 0 >AddFile(L0 Files): cumulative 0, interval 0 >AddFile(Keys): cumulative 0, interval 0 >Cumulative compaction: 393.37 GB write, 7.82 MB/s write, 343.72 GB read, 6.83 MB/s read, 17504.7 seconds >Interval compaction: 3.74 GB write, 0.07 MB/s write, 3.22 GB read, 0.06 MB/s read, 162.4 seconds >Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count > >** File Read Latency Histogram By Level [default] ** > >** Compaction Stats [default] ** >Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop >---------------------------------------------------------------------------------------------------------------------------------------------------------- > L0 2/0 269.49 MB 1.1 0.0 0.0 0.0 50.2 50.2 0.0 1.0 0.0 32.4 1586 383 4.141 0 0 > L1 3/0 190.31 MB 0.7 82.4 50.0 32.4 82.4 50.0 0.0 1.6 25.7 25.7 3281 188 17.450 2606M 894K > L2 44/0 2.45 GB 1.0 97.6 50.0 47.5 97.5 50.0 0.0 1.9 21.5 21.5 4636 560 8.279 3643M 1358K > L3 445/0 24.93 GB 1.0 86.9 49.3 37.6 86.8 49.2 0.7 1.8 20.8 20.8 4276 297 14.397 3281M 4000K > L4 3641/0 250.00 GB 1.0 74.8 17.9 56.8 74.5 17.6 32.0 4.1 21.2 21.1 3615 241 15.000 2845M 15M > L5 766/0 53.16 GB 0.0 2.1 1.0 1.0 2.1 1.0 48.5 2.0 19.3 19.1 111 14 7.906 79M 1097K > Sum 4901/0 330.99 GB 0.0 343.7 168.4 175.4 393.4 218.0 81.2 7.8 20.1 23.0 17505 1683 10.401 12G 23M > Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 >Uptime(secs): 51502.3 total, 51502.3 interval >Flush(GB): cumulative 50.181, interval 0.000 >AddFile(GB): cumulative 0.000, interval 0.000 >AddFile(Total Files): cumulative 0, interval 0 >AddFile(L0 Files): cumulative 0, interval 0 >AddFile(Keys): cumulative 0, interval 0 >Cumulative compaction: 393.37 GB write, 7.82 MB/s write, 343.72 GB read, 6.83 MB/s read, 17504.7 seconds >Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds >Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count > >** File Read Latency Histogram By Level [default] ** > >2018-07-11 05:26:00.692059 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2855] Compacting 2@0 + 2@1 files to L1, score 1.05 >2018-07-11 05:26:00.692071 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2854 Base level 0, inputs: [36039(134MB) 36033(134MB)], [36000(71MB) 36005(65MB)] > >2018-07-11 05:26:00.692086 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286760692075, "job": 2855, "event": "compaction_started", "files_L0": [36039, 36033], "files_L1": [36000, 36005], "score": 1.05268, "input_data_size": 425332272} >2018-07-11 05:26:00.692430 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 2854] Try to delete WAL files size 178896755, prev total WAL file size 178896767, number of live WAL files 2. > >2018-07-11 05:26:03.742472 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2855] Generated table #36040: 2623146 keys, 74507366 bytes >2018-07-11 05:26:03.742524 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286763742505, "cf_name": "default", "job": 2855, "event": "table_file_creation", "file_number": 36040, "file_size": 74507366, "table_properties": {"data_size": 67109445, "index_size": 691477, "filter_size": 6705454, "raw_key_size": 75813716, "raw_average_key_size": 28, "raw_value_size": 31559326, "raw_average_value_size": 12, "num_data_blocks": 16475, "num_entries": 2623146, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:26:06.563824 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2855] Generated table #36041: 2638242 keys, 74546421 bytes >2018-07-11 05:26:06.563869 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286766563841, "cf_name": "default", "job": 2855, "event": "table_file_creation", "file_number": 36041, "file_size": 74546421, "table_properties": {"data_size": 67112131, "index_size": 690114, "filter_size": 6743186, "raw_key_size": 76215871, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638242, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:26:09.272843 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2855] Generated table #36042: 2638241 keys, 74546257 bytes >2018-07-11 05:26:09.272888 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286769272861, "cf_name": "default", "job": 2855, "event": "table_file_creation", "file_number": 36042, "file_size": 74546257, "table_properties": {"data_size": 67112146, "index_size": 689938, "filter_size": 6743183, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:26:11.238879 7f41641eb700 1 bluefs _allocate failed to allocate 0x4800000 on bdev 1, free 0x1dfe000; fallback to bdev 2 >2018-07-11 05:26:12.226668 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2855] Generated table #36043: 2638241 keys, 74545806 bytes >2018-07-11 05:26:12.226715 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286772226687, "cf_name": "default", "job": 2855, "event": "table_file_creation", "file_number": 36043, "file_size": 74545806, "table_properties": {"data_size": 67112087, "index_size": 689546, "filter_size": 6743183, "raw_key_size": 76215853, "raw_average_key_size": 28, "raw_value_size": 31365756, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:26:13.790037 7f41641eb700 1 bluefs _allocate failed to allocate 0x3900000 on bdev 1, free 0x1dfe000; fallback to bdev 2 >2018-07-11 05:26:14.655070 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2855] Generated table #36044: 2085356 keys, 58924120 bytes >2018-07-11 05:26:14.655120 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286774655104, "cf_name": "default", "job": 2855, "event": "table_file_creation", "file_number": 36044, "file_size": 58924120, "table_properties": {"data_size": 53047745, "index_size": 545350, "filter_size": 5330037, "raw_key_size": 60243630, "raw_average_key_size": 28, "raw_value_size": 24792578, "raw_average_value_size": 11, "num_data_blocks": 13034, "num_entries": 2085356, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:26:15.175162 7f41641eb700 1 bluefs _allocate failed to allocate 0x4200000 on bdev 1, free 0x1dfe000; fallback to bdev 2 >2018-07-11 05:26:15.962062 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2855] Generated table #36045: 166215 keys, 68248733 bytes >2018-07-11 05:26:15.962104 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286775962077, "cf_name": "default", "job": 2855, "event": "table_file_creation", "file_number": 36045, "file_size": 68248733, "table_properties": {"data_size": 67113784, "index_size": 564831, "filter_size": 569130, "raw_key_size": 15377532, "raw_average_key_size": 92, "raw_value_size": 53113800, "raw_average_value_size": 319, "num_data_blocks": 15217, "num_entries": 166215, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "80898", "kMergeOperands": "0"}} >2018-07-11 05:26:15.975437 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2855] Generated table #36046: 4 keys, 5093 bytes >2018-07-11 05:26:15.975474 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286775975450, "cf_name": "default", "job": 2855, "event": "table_file_creation", "file_number": 36046, "file_size": 5093, "table_properties": {"data_size": 4078, "index_size": 28, "filter_size": 20, "raw_key_size": 335, "raw_average_key_size": 83, "raw_value_size": 3751, "raw_average_value_size": 937, "num_data_blocks": 1, "num_entries": 4, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:26:15.975533 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2855] Compacted 2@0 + 2@1 files to L1 => 425323796 bytes >2018-07-11 05:26:15.991547 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:26:15.991230) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 8 44 445 3641 766 0] max score 1.80, MB/sec: 27.8 rd, 27.8 wr, level 1, files in(2, 2) out(7) MB in(269.5, 136.1) out(405.6), read-write-amplify(3.0) write-amplify(1.5) OK, records in: 12789454, records dropped: 9 > >2018-07-11 05:26:15.991557 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:26:15.991266) EVENT_LOG_v1 {"time_micros": 1531286775991255, "job": 2855, "event": "compaction_finished", "compaction_time_micros": 15283429, "output_level": 1, "num_output_files": 7, "total_output_size": 425323796, "num_input_records": 12789454, "num_output_records": 12789445, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 8, 44, 445, 3641, 766, 0]} >2018-07-11 05:26:15.992483 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286775992479, "job": 2855, "event": "table_file_deletion", "file_number": 36039} >2018-07-11 05:26:15.992496 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286775992493, "job": 2855, "event": "table_file_deletion", "file_number": 36033} >2018-07-11 05:26:15.992506 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286775992504, "job": 2855, "event": "table_file_deletion", "file_number": 36005} >2018-07-11 05:26:15.992519 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286775992516, "job": 2855, "event": "table_file_deletion", "file_number": 36000} >2018-07-11 05:26:15.992731 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_picker.cc:461] [default] Expanding@1 1+5(74546421+342124070 bytes) to 4+5 (282562604+342124070bytes) > >2018-07-11 05:26:15.992828 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2856] Compacting 4@1 + 5@2 files to L2, score 1.80 >2018-07-11 05:26:15.992841 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2855 Base level 1, inputs: [36041(71MB) 36042(71MB) 36043(71MB) 36044(56MB)], [36007(71MB) 36008(71MB) 36009(71MB) 36010(71MB) 36015(41MB)] > >2018-07-11 05:26:15.992855 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286775992844, "job": 2856, "event": "compaction_started", "files_L1": [36041, 36042, 36043, 36044], "files_L2": [36007, 36008, 36009, 36010, 36015], "score": 1.79604, "input_data_size": 624686674} >2018-07-11 05:26:19.132819 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2856] Generated table #36047: 2638239 keys, 74546239 bytes >2018-07-11 05:26:19.132852 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286779132837, "cf_name": "default", "job": 2856, "event": "table_file_creation", "file_number": 36047, "file_size": 74546239, "table_properties": {"data_size": 67112120, "index_size": 689950, "filter_size": 6743179, "raw_key_size": 76215782, "raw_average_key_size": 28, "raw_value_size": 31365719, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638239, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:26:22.189156 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2856] Generated table #36048: 2638239 keys, 74546280 bytes >2018-07-11 05:26:22.189202 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286782189175, "cf_name": "default", "job": 2856, "event": "table_file_creation", "file_number": 36048, "file_size": 74546280, "table_properties": {"data_size": 67112108, "index_size": 690003, "filter_size": 6743179, "raw_key_size": 76215791, "raw_average_key_size": 28, "raw_value_size": 31365728, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638239, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:26:25.209258 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2856] Generated table #36049: 2638241 keys, 74546347 bytes >2018-07-11 05:26:25.209302 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286785209275, "cf_name": "default", "job": 2856, "event": "table_file_creation", "file_number": 36049, "file_size": 74546347, "table_properties": {"data_size": 67112158, "index_size": 690015, "filter_size": 6743184, "raw_key_size": 76215852, "raw_average_key_size": 28, "raw_value_size": 31365755, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:26:28.302990 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2856] Generated table #36050: 2638240 keys, 74546849 bytes >2018-07-11 05:26:28.303038 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286788303008, "cf_name": "default", "job": 2856, "event": "table_file_creation", "file_number": 36050, "file_size": 74546849, "table_properties": {"data_size": 67112158, "index_size": 690520, "filter_size": 6743181, "raw_key_size": 76215825, "raw_average_key_size": 28, "raw_value_size": 31365745, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:26:31.373810 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2856] Generated table #36051: 2638241 keys, 74546791 bytes >2018-07-11 05:26:31.373860 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286791373843, "cf_name": "default", "job": 2856, "event": "table_file_creation", "file_number": 36051, "file_size": 74546791, "table_properties": {"data_size": 67112149, "index_size": 690469, "filter_size": 6743183, "raw_key_size": 76215856, "raw_average_key_size": 28, "raw_value_size": 31365759, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:26:34.480768 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2856] Generated table #36052: 2638241 keys, 74546641 bytes >2018-07-11 05:26:34.480819 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286794480804, "cf_name": "default", "job": 2856, "event": "table_file_creation", "file_number": 36052, "file_size": 74546641, "table_properties": {"data_size": 67112136, "index_size": 690332, "filter_size": 6743183, "raw_key_size": 76215856, "raw_average_key_size": 28, "raw_value_size": 31365759, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:26:37.825121 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2856] Generated table #36053: 2638242 keys, 74546152 bytes >2018-07-11 05:26:37.825176 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286797825159, "cf_name": "default", "job": 2856, "event": "table_file_creation", "file_number": 36053, "file_size": 74546152, "table_properties": {"data_size": 67112155, "index_size": 689821, "filter_size": 6743186, "raw_key_size": 76215878, "raw_average_key_size": 28, "raw_value_size": 31365764, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638242, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:26:40.936340 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2856] Generated table #36054: 2638241 keys, 74546094 bytes >2018-07-11 05:26:40.936374 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286800936358, "cf_name": "default", "job": 2856, "event": "table_file_creation", "file_number": 36054, "file_size": 74546094, "table_properties": {"data_size": 67112154, "index_size": 689767, "filter_size": 6743183, "raw_key_size": 76215850, "raw_average_key_size": 28, "raw_value_size": 31365753, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:26:42.105275 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2856] Generated table #36055: 997666 keys, 28316608 bytes >2018-07-11 05:26:42.105317 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286802105290, "cf_name": "default", "job": 2856, "event": "table_file_creation", "file_number": 36055, "file_size": 28316608, "table_properties": {"data_size": 25503372, "index_size": 261985, "filter_size": 2550263, "raw_key_size": 28837224, "raw_average_key_size": 28, "raw_value_size": 11966524, "raw_average_value_size": 11, "num_data_blocks": 6268, "num_entries": 997666, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "11", "kMergeOperands": "0"}} >2018-07-11 05:26:42.105351 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2856] Compacted 4@1 + 5@2 files to L2 => 624688001 bytes >2018-07-11 05:26:42.124349 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:26:42.124045) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 4 48 445 3641 766 0] max score 1.08, MB/sec: 23.9 rd, 23.9 wr, level 2, files in(4, 5) out(9) MB in(269.5, 326.3) out(595.7), read-write-amplify(4.4) write-amplify(2.2) OK, records in: 22103590, records dropped: 0 > >2018-07-11 05:26:42.124360 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:26:42.124070) EVENT_LOG_v1 {"time_micros": 1531286802124059, "job": 2856, "event": "compaction_finished", "compaction_time_micros": 26112479, "output_level": 2, "num_output_files": 9, "total_output_size": 624688001, "num_input_records": 22103590, "num_output_records": 22103590, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 4, 48, 445, 3641, 766, 0]} >2018-07-11 05:26:42.125295 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286802125291, "job": 2856, "event": "table_file_deletion", "file_number": 36044} >2018-07-11 05:26:42.125307 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286802125305, "job": 2856, "event": "table_file_deletion", "file_number": 36043} >2018-07-11 05:26:42.125320 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286802125318, "job": 2856, "event": "table_file_deletion", "file_number": 36042} >2018-07-11 05:26:42.125335 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286802125333, "job": 2856, "event": "table_file_deletion", "file_number": 36041} >2018-07-11 05:26:42.125351 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286802125349, "job": 2856, "event": "table_file_deletion", "file_number": 36015} >2018-07-11 05:26:42.125365 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286802125363, "job": 2856, "event": "table_file_deletion", "file_number": 36010} >2018-07-11 05:26:42.125378 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286802125375, "job": 2856, "event": "table_file_deletion", "file_number": 36009} >2018-07-11 05:26:42.125391 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286802125389, "job": 2856, "event": "table_file_deletion", "file_number": 36008} >2018-07-11 05:26:42.125404 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286802125402, "job": 2856, "event": "table_file_deletion", "file_number": 36007} >2018-07-11 05:26:42.125704 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2857] Compacting 1@2 + 1@3 files to L3, score 1.08 >2018-07-11 05:26:42.125715 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2856 Base level 2, inputs: [36050(71MB)], [35987(71MB)] > >2018-07-11 05:26:42.125727 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286802125718, "job": 2857, "event": "compaction_started", "files_L2": [36050], "files_L3": [35987], "score": 1.08448, "input_data_size": 149092579} >2018-07-11 05:26:45.354777 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2857] Generated table #36056: 2638239 keys, 74546224 bytes >2018-07-11 05:26:45.354820 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286805354794, "cf_name": "default", "job": 2857, "event": "table_file_creation", "file_number": 36056, "file_size": 74546224, "table_properties": {"data_size": 67112145, "index_size": 689910, "filter_size": 6743179, "raw_key_size": 76215800, "raw_average_key_size": 28, "raw_value_size": 31365737, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638239, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:26:48.309600 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2857] Generated table #36057: 2638240 keys, 74546007 bytes >2018-07-11 05:26:48.309643 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286808309617, "cf_name": "default", "job": 2857, "event": "table_file_creation", "file_number": 36057, "file_size": 74546007, "table_properties": {"data_size": 67112173, "index_size": 689663, "filter_size": 6743181, "raw_key_size": 76215822, "raw_average_key_size": 28, "raw_value_size": 31365742, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:26:48.316548 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2857] Generated table #36058: 2 keys, 1088 bytes >2018-07-11 05:26:48.316573 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286808316561, "cf_name": "default", "job": 2857, "event": "table_file_creation", "file_number": 36058, "file_size": 1088, "table_properties": {"data_size": 81, "index_size": 27, "filter_size": 18, "raw_key_size": 58, "raw_average_key_size": 29, "raw_value_size": 24, "raw_average_value_size": 12, "num_data_blocks": 1, "num_entries": 2, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:26:48.316608 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2857] Compacted 1@2 + 1@3 files to L3 => 149093319 bytes >2018-07-11 05:26:48.332580 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:26:48.332317) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 4 47 447 3641 766 0] max score 1.06, MB/sec: 24.1 rd, 24.1 wr, level 3, files in(1, 1) out(3) MB in(71.1, 71.1) out(142.2), read-write-amplify(4.0) write-amplify(2.0) OK, records in: 5276481, records dropped: 0 > >2018-07-11 05:26:48.332591 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:26:48.332341) EVENT_LOG_v1 {"time_micros": 1531286808332331, "job": 2857, "event": "compaction_finished", "compaction_time_micros": 6190868, "output_level": 3, "num_output_files": 3, "total_output_size": 149093319, "num_input_records": 5276481, "num_output_records": 5276481, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 4, 47, 447, 3641, 766, 0]} >2018-07-11 05:26:48.333540 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286808333523, "job": 2857, "event": "table_file_deletion", "file_number": 36050} >2018-07-11 05:26:48.333556 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286808333554, "job": 2857, "event": "table_file_deletion", "file_number": 35987} >2018-07-11 05:26:48.333781 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_picker.cc:461] [default] Expanding@2 1+2(74546791+121510995 bytes) to 4+2 (298185678+121510995bytes) > >2018-07-11 05:26:48.333879 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2858] Compacting 4@2 + 2@3 files to L3, score 1.06 >2018-07-11 05:26:48.333891 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2857 Base level 2, inputs: [36051(71MB) 36052(71MB) 36053(71MB) 36054(71MB)], [36019(71MB) 36020(44MB)] > >2018-07-11 05:26:48.333904 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286808333894, "job": 2858, "event": "compaction_started", "files_L2": [36051, 36052, 36053, 36054], "files_L3": [36019, 36020], "score": 1.05671, "input_data_size": 419696673} >2018-07-11 05:26:51.538726 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2858] Generated table #36059: 2638241 keys, 74545706 bytes >2018-07-11 05:26:51.538768 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286811538751, "cf_name": "default", "job": 2858, "event": "table_file_creation", "file_number": 36059, "file_size": 74545706, "table_properties": {"data_size": 67112105, "index_size": 689428, "filter_size": 6743183, "raw_key_size": 76215851, "raw_average_key_size": 28, "raw_value_size": 31365754, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:26:54.644833 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2858] Generated table #36060: 2638241 keys, 74546476 bytes >2018-07-11 05:26:54.644877 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286814644850, "cf_name": "default", "job": 2858, "event": "table_file_creation", "file_number": 36060, "file_size": 74546476, "table_properties": {"data_size": 67112172, "index_size": 690131, "filter_size": 6743183, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:26:57.687852 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2858] Generated table #36061: 2638241 keys, 74546802 bytes >2018-07-11 05:26:57.687897 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286817687870, "cf_name": "default", "job": 2858, "event": "table_file_creation", "file_number": 36061, "file_size": 74546802, "table_properties": {"data_size": 67112147, "index_size": 690482, "filter_size": 6743183, "raw_key_size": 76215852, "raw_average_key_size": 28, "raw_value_size": 31365755, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:27:00.756407 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2858] Generated table #36062: 2638241 keys, 74546271 bytes >2018-07-11 05:27:00.756451 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286820756424, "cf_name": "default", "job": 2858, "event": "table_file_creation", "file_number": 36062, "file_size": 74546271, "table_properties": {"data_size": 67112129, "index_size": 689969, "filter_size": 6743183, "raw_key_size": 76215859, "raw_average_key_size": 28, "raw_value_size": 31365762, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:27:03.817975 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2858] Generated table #36063: 2638242 keys, 74545965 bytes >2018-07-11 05:27:03.818019 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286823818000, "cf_name": "default", "job": 2858, "event": "table_file_creation", "file_number": 36063, "file_size": 74545965, "table_properties": {"data_size": 67112158, "index_size": 689631, "filter_size": 6743186, "raw_key_size": 76215880, "raw_average_key_size": 28, "raw_value_size": 31365766, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638242, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:27:05.748403 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2858] Generated table #36064: 1519917 keys, 46965591 bytes >2018-07-11 05:27:05.748447 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286825748432, "cf_name": "default", "job": 2858, "event": "table_file_creation", "file_number": 36064, "file_size": 46965591, "table_properties": {"data_size": 42636676, "index_size": 433774, "filter_size": 3894153, "raw_key_size": 44348957, "raw_average_key_size": 29, "raw_value_size": 21522469, "raw_average_value_size": 14, "num_data_blocks": 10538, "num_entries": 1519917, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:27:05.748483 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2858] Compacted 4@2 + 2@3 files to L3 => 419696811 bytes >2018-07-11 05:27:05.765613 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:27:05.765320) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 4 43 451 3641 766 0] max score 1.01, MB/sec: 24.1 rd, 24.1 wr, level 3, files in(4, 2) out(6) MB in(284.4, 115.9) out(400.3), read-write-amplify(2.8) write-amplify(1.4) OK, records in: 14711123, records dropped: 0 > >2018-07-11 05:27:05.765624 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:27:05.765345) EVENT_LOG_v1 {"time_micros": 1531286825765333, "job": 2858, "event": "compaction_finished", "compaction_time_micros": 17414561, "output_level": 3, "num_output_files": 6, "total_output_size": 419696811, "num_input_records": 14711123, "num_output_records": 14711123, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 4, 43, 451, 3641, 766, 0]} >2018-07-11 05:27:05.766588 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286825766570, "job": 2858, "event": "table_file_deletion", "file_number": 36054} >2018-07-11 05:27:05.766604 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286825766601, "job": 2858, "event": "table_file_deletion", "file_number": 36053} >2018-07-11 05:27:05.766615 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286825766613, "job": 2858, "event": "table_file_deletion", "file_number": 36052} >2018-07-11 05:27:05.766626 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286825766624, "job": 2858, "event": "table_file_deletion", "file_number": 36051} >2018-07-11 05:27:05.766639 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286825766637, "job": 2858, "event": "table_file_deletion", "file_number": 36020} >2018-07-11 05:27:05.766650 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286825766648, "job": 2858, "event": "table_file_deletion", "file_number": 36019} >2018-07-11 05:27:05.782072 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:27:05.766951) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #36061 to level-4 74546802 bytes > >2018-07-11 05:27:05.782083 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:27:05.781976) EVENT_LOG_v1 {"time_micros": 1531286825781970, "job": 2859, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 74546802} >2018-07-11 05:27:05.782086 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:27:05.781984) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-4 74546802 bytes OK: base level 1 max bytes base 268435456 files[0 4 43 450 3642 766 0] max score 1.01 > >2018-07-11 05:27:05.798765 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:27:05.782945) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #36060 to level-4 74546476 bytes > >2018-07-11 05:27:05.798776 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:27:05.798672) EVENT_LOG_v1 {"time_micros": 1531286825798667, "job": 2860, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 74546476} >2018-07-11 05:27:05.798779 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:27:05.798680) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-4 74546476 bytes OK: base level 1 max bytes base 268435456 files[0 4 43 449 3643 766 0] max score 1.01 > >2018-07-11 05:27:05.815460 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:27:05.799661) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #36062 to level-4 74546271 bytes > >2018-07-11 05:27:05.815471 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:27:05.815367) EVENT_LOG_v1 {"time_micros": 1531286825815361, "job": 2861, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 74546271} >2018-07-11 05:27:05.815474 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:27:05.815375) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-4 74546271 bytes OK: base level 1 max bytes base 268435456 files[0 4 43 448 3644 766 0] max score 1.00 > >2018-07-11 05:27:05.816347 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2862] Compacting 1@3 + 12@4 files to L4, score 1.00 >2018-07-11 05:27:05.816363 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2861 Base level 3, inputs: [36056(71MB)], [36024(71MB) 36026(71MB) 36027(71MB) 36028(71MB) 36029(71MB) 36030(71MB) 36031(71MB) 36034(71MB) 35913(71MB) 35914(1064B) 35907(71MB) 35908(71MB)] > >2018-07-11 05:27:05.816378 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286825816367, "job": 2862, "event": "compaction_started", "files_L3": [36056], "files_L4": [36024, 36026, 36027, 36028, 36029, 36030, 36031, 36034, 35913, 35914, 35907, 35908], "score": 1.00291, "input_data_size": 894555495} >2018-07-11 05:27:08.863800 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2862] Generated table #36065: 2638241 keys, 74546377 bytes >2018-07-11 05:27:08.863845 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286828863817, "cf_name": "default", "job": 2862, "event": "table_file_creation", "file_number": 36065, "file_size": 74546377, "table_properties": {"data_size": 67112141, "index_size": 690063, "filter_size": 6743183, "raw_key_size": 76215860, "raw_average_key_size": 28, "raw_value_size": 31365763, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:27:12.109149 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2862] Generated table #36066: 2638240 keys, 74546958 bytes >2018-07-11 05:27:12.109204 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286832109187, "cf_name": "default", "job": 2862, "event": "table_file_creation", "file_number": 36066, "file_size": 74546958, "table_properties": {"data_size": 67112084, "index_size": 690703, "filter_size": 6743181, "raw_key_size": 76215819, "raw_average_key_size": 28, "raw_value_size": 31365739, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:27:15.289571 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2862] Generated table #36067: 2638241 keys, 74546449 bytes >2018-07-11 05:27:15.289623 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286835289604, "cf_name": "default", "job": 2862, "event": "table_file_creation", "file_number": 36067, "file_size": 74546449, "table_properties": {"data_size": 67112143, "index_size": 690133, "filter_size": 6743183, "raw_key_size": 76215851, "raw_average_key_size": 28, "raw_value_size": 31365754, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:27:18.380353 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2862] Generated table #36068: 2638241 keys, 74545850 bytes >2018-07-11 05:27:18.380396 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286838380369, "cf_name": "default", "job": 2862, "event": "table_file_creation", "file_number": 36068, "file_size": 74545850, "table_properties": {"data_size": 67112144, "index_size": 689533, "filter_size": 6743183, "raw_key_size": 76215860, "raw_average_key_size": 28, "raw_value_size": 31365763, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:27:21.619479 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2862] Generated table #36069: 2638241 keys, 74546810 bytes >2018-07-11 05:27:21.619523 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286841619496, "cf_name": "default", "job": 2862, "event": "table_file_creation", "file_number": 36069, "file_size": 74546810, "table_properties": {"data_size": 67112138, "index_size": 690499, "filter_size": 6743183, "raw_key_size": 76215850, "raw_average_key_size": 28, "raw_value_size": 31365753, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:27:24.652397 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2862] Generated table #36070: 2638241 keys, 74546437 bytes >2018-07-11 05:27:24.652443 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286844652416, "cf_name": "default", "job": 2862, "event": "table_file_creation", "file_number": 36070, "file_size": 74546437, "table_properties": {"data_size": 67112123, "index_size": 690141, "filter_size": 6743183, "raw_key_size": 76215850, "raw_average_key_size": 28, "raw_value_size": 31365753, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:27:27.882991 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2862] Generated table #36071: 2638241 keys, 74546214 bytes >2018-07-11 05:27:27.883034 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286847883008, "cf_name": "default", "job": 2862, "event": "table_file_creation", "file_number": 36071, "file_size": 74546214, "table_properties": {"data_size": 67112169, "index_size": 689872, "filter_size": 6743183, "raw_key_size": 76215857, "raw_average_key_size": 28, "raw_value_size": 31365760, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:27:31.368348 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2862] Generated table #36072: 2638241 keys, 74545944 bytes >2018-07-11 05:27:31.368396 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286851368380, "cf_name": "default", "job": 2862, "event": "table_file_creation", "file_number": 36072, "file_size": 74545944, "table_properties": {"data_size": 67112101, "index_size": 689670, "filter_size": 6743183, "raw_key_size": 76215855, "raw_average_key_size": 28, "raw_value_size": 31365758, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:27:34.439172 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2862] Generated table #36073: 2638240 keys, 74546170 bytes >2018-07-11 05:27:34.439211 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286854439195, "cf_name": "default", "job": 2862, "event": "table_file_creation", "file_number": 36073, "file_size": 74546170, "table_properties": {"data_size": 67112157, "index_size": 689842, "filter_size": 6743181, "raw_key_size": 76215828, "raw_average_key_size": 28, "raw_value_size": 31365748, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:27:37.547965 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2862] Generated table #36074: 2638241 keys, 74546974 bytes >2018-07-11 05:27:37.548009 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286857547982, "cf_name": "default", "job": 2862, "event": "table_file_creation", "file_number": 36074, "file_size": 74546974, "table_properties": {"data_size": 67112124, "index_size": 690677, "filter_size": 6743183, "raw_key_size": 76215852, "raw_average_key_size": 28, "raw_value_size": 31365755, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:27:40.545231 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2862] Generated table #36075: 2638241 keys, 74546278 bytes >2018-07-11 05:27:40.545283 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286860545268, "cf_name": "default", "job": 2862, "event": "table_file_creation", "file_number": 36075, "file_size": 74546278, "table_properties": {"data_size": 67112161, "index_size": 689944, "filter_size": 6743183, "raw_key_size": 76215856, "raw_average_key_size": 28, "raw_value_size": 31365759, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:27:43.745766 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2862] Generated table #36076: 2638240 keys, 74546347 bytes >2018-07-11 05:27:43.745809 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286863745783, "cf_name": "default", "job": 2862, "event": "table_file_creation", "file_number": 36076, "file_size": 74546347, "table_properties": {"data_size": 67112069, "index_size": 690118, "filter_size": 6743170, "raw_key_size": 76215827, "raw_average_key_size": 28, "raw_value_size": 31365747, "raw_average_value_size": 11, "num_data_blocks": 16489, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:27:43.745855 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2862] Compacted 1@3 + 12@4 files to L4 => 894556808 bytes >2018-07-11 05:27:43.765077 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:27:43.764737) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 4 43 447 3644 766 0] max score 1.00, MB/sec: 23.6 rd, 23.6 wr, level 4, files in(1, 12) out(12) MB in(71.1, 782.0) out(853.1), read-write-amplify(24.0) write-amplify(12.0) OK, records in: 31658889, records dropped: 0 > >2018-07-11 05:27:43.765088 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:27:43.764764) EVENT_LOG_v1 {"time_micros": 1531286863764752, "job": 2862, "event": "compaction_finished", "compaction_time_micros": 37929451, "output_level": 4, "num_output_files": 12, "total_output_size": 894556808, "num_input_records": 31658889, "num_output_records": 31658889, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 4, 43, 447, 3644, 766, 0]} >2018-07-11 05:27:43.766012 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286863766009, "job": 2862, "event": "table_file_deletion", "file_number": 36056} >2018-07-11 05:27:43.766040 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286863766038, "job": 2862, "event": "table_file_deletion", "file_number": 36034} >2018-07-11 05:27:43.766051 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286863766049, "job": 2862, "event": "table_file_deletion", "file_number": 36031} >2018-07-11 05:27:43.766066 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286863766064, "job": 2862, "event": "table_file_deletion", "file_number": 36030} >2018-07-11 05:27:43.766078 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286863766076, "job": 2862, "event": "table_file_deletion", "file_number": 36029} >2018-07-11 05:27:43.766088 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286863766086, "job": 2862, "event": "table_file_deletion", "file_number": 36028} >2018-07-11 05:27:43.766102 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286863766100, "job": 2862, "event": "table_file_deletion", "file_number": 36027} >2018-07-11 05:27:43.766117 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286863766115, "job": 2862, "event": "table_file_deletion", "file_number": 36026} >2018-07-11 05:27:43.766127 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286863766125, "job": 2862, "event": "table_file_deletion", "file_number": 36024} >2018-07-11 05:27:43.766139 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286863766137, "job": 2862, "event": "table_file_deletion", "file_number": 35914} >2018-07-11 05:27:43.766153 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286863766151, "job": 2862, "event": "table_file_deletion", "file_number": 35913} >2018-07-11 05:27:43.766166 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286863766164, "job": 2862, "event": "table_file_deletion", "file_number": 35908} >2018-07-11 05:27:43.766179 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286863766176, "job": 2862, "event": "table_file_deletion", "file_number": 35907} >2018-07-11 05:27:43.781434 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:27:43.766475) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #36074 to level-5 74546974 bytes > >2018-07-11 05:27:43.781445 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:27:43.781339) EVENT_LOG_v1 {"time_micros": 1531286863781332, "job": 2863, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546974} >2018-07-11 05:27:43.781448 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:27:43.781347) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546974 bytes OK: base level 1 max bytes base 268435456 files[0 4 43 447 3643 767 0] max score 1.00 > >2018-07-11 05:27:43.782285 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2864] Compacting 1@4 + 3@5 files to L5, score 1.00 >2018-07-11 05:27:43.782297 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2863 Base level 4, inputs: [36066(71MB)], [36035(71MB) 36036(71MB) 36037(71MB)] > >2018-07-11 05:27:43.782311 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286863782300, "job": 2864, "event": "compaction_started", "files_L4": [36066], "files_L5": [36035, 36036, 36037], "score": 1.00082, "input_data_size": 298186137} >2018-07-11 05:27:46.939177 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2864] Generated table #36077: 2638241 keys, 74546964 bytes >2018-07-11 05:27:46.939228 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286866939199, "cf_name": "default", "job": 2864, "event": "table_file_creation", "file_number": 36077, "file_size": 74546964, "table_properties": {"data_size": 67112098, "index_size": 690693, "filter_size": 6743183, "raw_key_size": 76215851, "raw_average_key_size": 28, "raw_value_size": 31365754, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:27:50.081427 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2864] Generated table #36078: 2638241 keys, 74546256 bytes >2018-07-11 05:27:50.081472 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286870081445, "cf_name": "default", "job": 2864, "event": "table_file_creation", "file_number": 36078, "file_size": 74546256, "table_properties": {"data_size": 67112165, "index_size": 689918, "filter_size": 6743183, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:27:53.075601 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2864] Generated table #36079: 2638241 keys, 74546457 bytes >2018-07-11 05:27:53.075641 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286873075624, "cf_name": "default", "job": 2864, "event": "table_file_creation", "file_number": 36079, "file_size": 74546457, "table_properties": {"data_size": 67112138, "index_size": 690146, "filter_size": 6743183, "raw_key_size": 76215856, "raw_average_key_size": 28, "raw_value_size": 31365759, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:27:56.025719 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2864] Generated table #36080: 2638238 keys, 74545715 bytes >2018-07-11 05:27:56.025764 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286876025737, "cf_name": "default", "job": 2864, "event": "table_file_creation", "file_number": 36080, "file_size": 74545715, "table_properties": {"data_size": 67112035, "index_size": 689525, "filter_size": 6743165, "raw_key_size": 76215764, "raw_average_key_size": 28, "raw_value_size": 31365718, "raw_average_value_size": 11, "num_data_blocks": 16489, "num_entries": 2638238, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:27:56.025798 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2864] Compacted 1@4 + 3@5 files to L5 => 298185392 bytes >2018-07-11 05:27:56.039778 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:27:56.039495) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 4 43 447 3642 768 0] max score 1.00, MB/sec: 24.4 rd, 24.4 wr, level 5, files in(1, 3) out(4) MB in(71.1, 213.3) out(284.4), read-write-amplify(8.0) write-amplify(4.0) OK, records in: 10552961, records dropped: 0 > >2018-07-11 05:27:56.039789 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:27:56.039520) EVENT_LOG_v1 {"time_micros": 1531286876039510, "job": 2864, "event": "compaction_finished", "compaction_time_micros": 12243472, "output_level": 5, "num_output_files": 4, "total_output_size": 298185392, "num_input_records": 10552961, "num_output_records": 10552961, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 4, 43, 447, 3642, 768, 0]} >2018-07-11 05:27:56.040767 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286876040763, "job": 2864, "event": "table_file_deletion", "file_number": 36066} >2018-07-11 05:27:56.040796 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286876040793, "job": 2864, "event": "table_file_deletion", "file_number": 36037} >2018-07-11 05:27:56.040808 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286876040806, "job": 2864, "event": "table_file_deletion", "file_number": 36036} >2018-07-11 05:27:56.040819 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286876040817, "job": 2864, "event": "table_file_deletion", "file_number": 36035} >2018-07-11 05:27:56.056227 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:27:56.041126) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #13186 to level-5 74546842 bytes > >2018-07-11 05:27:56.056238 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:27:56.056131) EVENT_LOG_v1 {"time_micros": 1531286876056125, "job": 2865, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546842} >2018-07-11 05:27:56.056240 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:27:56.056139) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546842 bytes OK: base level 1 max bytes base 268435456 files[0 4 43 447 3641 769 0] max score 1.00 > >2018-07-11 05:27:56.073002 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:27:56.057076) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #30157 to level-5 74546841 bytes > >2018-07-11 05:27:56.073012 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:27:56.072913) EVENT_LOG_v1 {"time_micros": 1531286876072907, "job": 2866, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546841} >2018-07-11 05:27:56.073016 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:27:56.072920) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546841 bytes OK: base level 1 max bytes base 268435456 files[0 4 43 447 3640 770 0] max score 1.00 > >2018-07-11 05:27:56.073869 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2867] Compacting 1@3 + 2@4 files to L4, score 1.00 >2018-07-11 05:27:56.073881 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2866 Base level 3, inputs: [19101(71MB)], [14672(71MB) 14673(1137B)] > >2018-07-11 05:27:56.073907 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286876073884, "job": 2867, "event": "compaction_started", "files_L3": [19101], "files_L4": [14672, 14673], "score": 1.00013, "input_data_size": 149093479} >2018-07-11 05:27:59.215202 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2867] Generated table #36081: 2638241 keys, 74546302 bytes >2018-07-11 05:27:59.215238 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286879215222, "cf_name": "default", "job": 2867, "event": "table_file_creation", "file_number": 36081, "file_size": 74546302, "table_properties": {"data_size": 67112162, "index_size": 689967, "filter_size": 6743183, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:28:02.342015 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2867] Generated table #36082: 2638241 keys, 74545750 bytes >2018-07-11 05:28:02.342048 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286882342033, "cf_name": "default", "job": 2867, "event": "table_file_creation", "file_number": 36082, "file_size": 74545750, "table_properties": {"data_size": 67112117, "index_size": 689460, "filter_size": 6743183, "raw_key_size": 76215864, "raw_average_key_size": 28, "raw_value_size": 31365767, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:28:02.390365 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2867] Generated table #36083: 5 keys, 1170 bytes >2018-07-11 05:28:02.390391 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286882390378, "cf_name": "default", "job": 2867, "event": "table_file_creation", "file_number": 36083, "file_size": 1170, "table_properties": {"data_size": 153, "index_size": 28, "filter_size": 23, "raw_key_size": 145, "raw_average_key_size": 29, "raw_value_size": 60, "raw_average_value_size": 12, "num_data_blocks": 1, "num_entries": 5, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:28:02.390415 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2867] Compacted 1@3 + 2@4 files to L4 => 149093222 bytes >2018-07-11 05:28:02.406328 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:28:02.406056) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 4 43 446 3641 770 0] max score 1.00, MB/sec: 23.6 rd, 23.6 wr, level 4, files in(1, 2) out(3) MB in(71.1, 71.1) out(142.2), read-write-amplify(4.0) write-amplify(2.0) OK, records in: 5276487, records dropped: 0 > >2018-07-11 05:28:02.406339 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:28:02.406080) EVENT_LOG_v1 {"time_micros": 1531286882406070, "job": 2867, "event": "compaction_finished", "compaction_time_micros": 6316496, "output_level": 4, "num_output_files": 3, "total_output_size": 149093222, "num_input_records": 5276487, "num_output_records": 5276487, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 4, 43, 446, 3641, 770, 0]} >2018-07-11 05:28:02.407276 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286882407272, "job": 2867, "event": "table_file_deletion", "file_number": 19101} >2018-07-11 05:28:02.407291 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286882407289, "job": 2867, "event": "table_file_deletion", "file_number": 14673} >2018-07-11 05:28:02.407302 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286882407300, "job": 2867, "event": "table_file_deletion", "file_number": 14672} >2018-07-11 05:28:02.422881 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:28:02.407614) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #1289 to level-5 74546841 bytes > >2018-07-11 05:28:02.422891 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:28:02.422785) EVENT_LOG_v1 {"time_micros": 1531286882422779, "job": 2868, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546841} >2018-07-11 05:28:02.422894 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:28:02.422794) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546841 bytes OK: base level 1 max bytes base 268435456 files[0 4 43 446 3640 771 0] max score 1.00 > >2018-07-11 05:29:01.428837 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:684] reusing log 36032 from recycle list > >2018-07-11 05:29:01.429020 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #36084. Immutable memtables: 0. > >2018-07-11 05:29:01.514097 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:29:01.429083) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1 >2018-07-11 05:29:01.514109 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 2869] Flushing memtable with next log file: 36084 > >2018-07-11 05:29:01.514125 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286941514117, "job": 2869, "event": "flush_started", "num_memtables": 1, "num_entries": 5000015, "num_deletes": 0, "memory_usage": 292954680} >2018-07-11 05:29:01.514129 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 2869] Level-0 flush table #36085: started >2018-07-11 05:29:05.483381 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531286945483359, "cf_name": "default", "job": 2869, "event": "table_file_creation", "file_number": 36085, "file_size": 141286889, "table_properties": {"data_size": 127197777, "index_size": 1308397, "filter_size": 12779725, "raw_key_size": 144445101, "raw_average_key_size": 28, "raw_value_size": 59450671, "raw_average_value_size": 11, "num_data_blocks": 31252, "num_entries": 5000014, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:29:05.483404 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 2869] Level-0 flush table #36085: 141286889 bytes OK >2018-07-11 05:29:05.496438 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:242] adding log 36038 to recycle list > >2018-07-11 05:29:05.496449 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:29:05.483430) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #36085 started >2018-07-11 05:29:05.496466 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:29:05.496201) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #36085: memtable #1 done >2018-07-11 05:29:05.496468 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:29:05.496218) EVENT_LOG_v1 {"time_micros": 1531286945496212, "job": 2869, "event": "flush_finished", "lsm_state": [1, 4, 43, 446, 3640, 771, 0], "immutable_memtables": 0} >2018-07-11 05:29:05.496470 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:29:05.496386) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 4 43 446 3640 771 0] max score 1.00 > >2018-07-11 05:29:05.497352 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 2869] Try to delete WAL files size 178896755, prev total WAL file size 178896767, number of live WAL files 2. > >2018-07-11 05:30:28.217406 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:684] reusing log 36038 from recycle list > >2018-07-11 05:30:28.217563 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #36086. Immutable memtables: 0. > >2018-07-11 05:30:28.266980 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:30:28.217626) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1 >2018-07-11 05:30:28.266994 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 2870] Flushing memtable with next log file: 36086 > >2018-07-11 05:30:28.267012 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287028267002, "job": 2870, "event": "flush_started", "num_memtables": 1, "num_entries": 5000015, "num_deletes": 0, "memory_usage": 292946256} >2018-07-11 05:30:28.267016 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 2870] Level-0 flush table #36087: started >2018-07-11 05:30:32.184493 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287032184473, "cf_name": "default", "job": 2870, "event": "table_file_creation", "file_number": 36087, "file_size": 141286219, "table_properties": {"data_size": 127196934, "index_size": 1308576, "filter_size": 12779719, "raw_key_size": 144445077, "raw_average_key_size": 28, "raw_value_size": 59449772, "raw_average_value_size": 11, "num_data_blocks": 31252, "num_entries": 5000013, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:30:32.184527 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 2870] Level-0 flush table #36087: 141286219 bytes OK >2018-07-11 05:30:32.202367 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:242] adding log 36084 to recycle list > >2018-07-11 05:30:32.202379 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:30:32.184541) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #36087 started >2018-07-11 05:30:32.202383 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:30:32.202110) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #36087: memtable #1 done >2018-07-11 05:30:32.202385 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:30:32.202128) EVENT_LOG_v1 {"time_micros": 1531287032202121, "job": 2870, "event": "flush_finished", "lsm_state": [2, 4, 43, 446, 3640, 771, 0], "immutable_memtables": 0} >2018-07-11 05:30:32.202387 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:30:32.202312) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[2 4 43 446 3640 771 0] max score 1.05 > >2018-07-11 05:30:32.202521 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2871] Compacting 2@0 + 2@1 files to L1, score 1.05 >2018-07-11 05:30:32.202540 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2870 Base level 0, inputs: [36087(134MB) 36085(134MB)], [36040(71MB) 36045(65MB)] > >2018-07-11 05:30:32.202564 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287032202546, "job": 2871, "event": "compaction_started", "files_L0": [36087, 36085], "files_L1": [36040, 36045], "score": 1.05267, "input_data_size": 425329207} >2018-07-11 05:30:32.203267 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 2870] Try to delete WAL files size 178896755, prev total WAL file size 178896767, number of live WAL files 2. > >2018-07-11 05:30:35.145298 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2871] Generated table #36088: 2623076 keys, 74506984 bytes >2018-07-11 05:30:35.145344 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287035145329, "cf_name": "default", "job": 2871, "event": "table_file_creation", "file_number": 36088, "file_size": 74506984, "table_properties": {"data_size": 67109212, "index_size": 691503, "filter_size": 6705279, "raw_key_size": 75811903, "raw_average_key_size": 28, "raw_value_size": 31559993, "raw_average_value_size": 12, "num_data_blocks": 16475, "num_entries": 2623076, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:30:37.541556 7f41641eb700 1 bluefs _allocate failed to allocate 0x4800000 on bdev 1, free 0xffe000; fallback to bdev 2 >2018-07-11 05:30:38.588776 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2871] Generated table #36089: 2638242 keys, 74546426 bytes >2018-07-11 05:30:38.588820 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287038588803, "cf_name": "default", "job": 2871, "event": "table_file_creation", "file_number": 36089, "file_size": 74546426, "table_properties": {"data_size": 67112136, "index_size": 690114, "filter_size": 6743186, "raw_key_size": 76215876, "raw_average_key_size": 28, "raw_value_size": 31365762, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638242, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:30:41.166074 7f41641eb700 1 bluefs _allocate failed to allocate 0x4800000 on bdev 1, free 0xffe000; fallback to bdev 2 >2018-07-11 05:30:42.236555 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2871] Generated table #36090: 2638241 keys, 74546257 bytes >2018-07-11 05:30:42.236592 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287042236577, "cf_name": "default", "job": 2871, "event": "table_file_creation", "file_number": 36090, "file_size": 74546257, "table_properties": {"data_size": 67112146, "index_size": 689938, "filter_size": 6743183, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:30:44.261120 7f41641eb700 1 bluefs _allocate failed to allocate 0x4800000 on bdev 1, free 0xffe000; fallback to bdev 2 >2018-07-11 05:30:45.350547 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2871] Generated table #36091: 2638241 keys, 74545806 bytes >2018-07-11 05:30:45.350599 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287045350570, "cf_name": "default", "job": 2871, "event": "table_file_creation", "file_number": 36091, "file_size": 74545806, "table_properties": {"data_size": 67112087, "index_size": 689546, "filter_size": 6743183, "raw_key_size": 76215853, "raw_average_key_size": 28, "raw_value_size": 31365756, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:30:47.014195 7f41641eb700 1 bluefs _allocate failed to allocate 0x3b00000 on bdev 1, free 0xffe000; fallback to bdev 2 >2018-07-11 05:30:47.885027 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2871] Generated table #36092: 2089247 keys, 61206787 bytes >2018-07-11 05:30:47.885079 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287047885062, "cf_name": "default", "job": 2871, "event": "table_file_creation", "file_number": 36092, "file_size": 61206787, "table_properties": {"data_size": 55296128, "index_size": 564678, "filter_size": 5344993, "raw_key_size": 60632730, "raw_average_key_size": 29, "raw_value_size": 26671622, "raw_average_value_size": 12, "num_data_blocks": 13590, "num_entries": 2089247, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:30:48.580106 7f41641eb700 1 bluefs _allocate failed to allocate 0x3f00000 on bdev 1, free 0xffe000; fallback to bdev 2 >2018-07-11 05:30:49.426027 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2871] Generated table #36093: 162334 keys, 65970492 bytes >2018-07-11 05:30:49.426071 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287049426043, "cf_name": "default", "job": 2871, "event": "table_file_creation", "file_number": 36093, "file_size": 65970492, "table_properties": {"data_size": 64869606, "index_size": 545686, "filter_size": 554212, "raw_key_size": 14989042, "raw_average_key_size": 92, "raw_value_size": 51238386, "raw_average_value_size": 315, "num_data_blocks": 14662, "num_entries": 162334, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "80898", "kMergeOperands": "0"}} >2018-07-11 05:30:49.426135 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2871] Compacted 2@0 + 2@1 files to L1 => 425322752 bytes >2018-07-11 05:30:49.443633 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:30:49.443413) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 8 43 446 3640 771 0] max score 1.80, MB/sec: 24.7 rd, 24.7 wr, level 1, files in(2, 2) out(6) MB in(269.5, 136.1) out(405.6), read-write-amplify(3.0) write-amplify(1.5) OK, records in: 12789388, records dropped: 7 > >2018-07-11 05:30:49.443644 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:30:49.443438) EVENT_LOG_v1 {"time_micros": 1531287049443427, "job": 2871, "event": "compaction_finished", "compaction_time_micros": 17223551, "output_level": 1, "num_output_files": 6, "total_output_size": 425322752, "num_input_records": 12789388, "num_output_records": 12789381, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 8, 43, 446, 3640, 771, 0]} >2018-07-11 05:30:49.444632 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287049444615, "job": 2871, "event": "table_file_deletion", "file_number": 36087} >2018-07-11 05:30:49.444647 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287049444645, "job": 2871, "event": "table_file_deletion", "file_number": 36085} >2018-07-11 05:30:49.444660 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287049444657, "job": 2871, "event": "table_file_deletion", "file_number": 36045} >2018-07-11 05:30:49.444671 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287049444669, "job": 2871, "event": "table_file_deletion", "file_number": 36040} >2018-07-11 05:30:49.444893 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_picker.cc:461] [default] Expanding@1 1+4(74546426+251955474 bytes) to 3+4 (223638489+251955474bytes) > >2018-07-11 05:30:49.444995 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2872] Compacting 3@1 + 4@2 files to L2, score 1.80 >2018-07-11 05:30:49.445008 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2871 Base level 1, inputs: [36089(71MB) 36090(71MB) 36091(71MB)], [36047(71MB) 36048(71MB) 36049(71MB) 36055(27MB)] > >2018-07-11 05:30:49.445022 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287049445011, "job": 2872, "event": "compaction_started", "files_L1": [36089, 36090, 36091], "files_L2": [36047, 36048, 36049, 36055], "score": 1.79605, "input_data_size": 475593963} >2018-07-11 05:30:53.284997 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2872] Generated table #36094: 2638240 keys, 74545796 bytes >2018-07-11 05:30:53.285046 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287053285031, "cf_name": "default", "job": 2872, "event": "table_file_creation", "file_number": 36094, "file_size": 74545796, "table_properties": {"data_size": 67112117, "index_size": 689508, "filter_size": 6743181, "raw_key_size": 76215813, "raw_average_key_size": 28, "raw_value_size": 31365733, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:30:56.730383 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2872] Generated table #36095: 2638239 keys, 74546202 bytes >2018-07-11 05:30:56.730431 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287056730415, "cf_name": "default", "job": 2872, "event": "table_file_creation", "file_number": 36095, "file_size": 74546202, "table_properties": {"data_size": 67112131, "index_size": 689902, "filter_size": 6743179, "raw_key_size": 76215791, "raw_average_key_size": 28, "raw_value_size": 31365728, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638239, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:31:00.094848 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2872] Generated table #36096: 2638240 keys, 74546300 bytes >2018-07-11 05:31:00.094895 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287060094874, "cf_name": "default", "job": 2872, "event": "table_file_creation", "file_number": 36096, "file_size": 74546300, "table_properties": {"data_size": 67112135, "index_size": 689993, "filter_size": 6743182, "raw_key_size": 76215826, "raw_average_key_size": 28, "raw_value_size": 31365746, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:31:03.447658 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2872] Generated table #36097: 2638241 keys, 74547042 bytes >2018-07-11 05:31:03.447713 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287063447695, "cf_name": "default", "job": 2872, "event": "table_file_creation", "file_number": 36097, "file_size": 74547042, "table_properties": {"data_size": 67112109, "index_size": 690760, "filter_size": 6743183, "raw_key_size": 76215862, "raw_average_key_size": 28, "raw_value_size": 31365765, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:31:06.603227 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2872] Generated table #36098: 2638241 keys, 74546290 bytes >2018-07-11 05:31:06.603288 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287066603271, "cf_name": "default", "job": 2872, "event": "table_file_creation", "file_number": 36098, "file_size": 74546290, "table_properties": {"data_size": 67112154, "index_size": 689963, "filter_size": 6743183, "raw_key_size": 76215847, "raw_average_key_size": 28, "raw_value_size": 31365750, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:31:10.157356 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2872] Generated table #36099: 2638241 keys, 74546472 bytes >2018-07-11 05:31:10.157401 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287070157386, "cf_name": "default", "job": 2872, "event": "table_file_creation", "file_number": 36099, "file_size": 74546472, "table_properties": {"data_size": 67112124, "index_size": 690175, "filter_size": 6743183, "raw_key_size": 76215857, "raw_average_key_size": 28, "raw_value_size": 31365760, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:31:11.387970 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2872] Generated table #36100: 997667 keys, 28316192 bytes >2018-07-11 05:31:11.388022 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287071387994, "cf_name": "default", "job": 2872, "event": "table_file_creation", "file_number": 36100, "file_size": 28316192, "table_properties": {"data_size": 25503398, "index_size": 261541, "filter_size": 2550265, "raw_key_size": 28837236, "raw_average_key_size": 28, "raw_value_size": 11966519, "raw_average_value_size": 11, "num_data_blocks": 6268, "num_entries": 997667, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "11", "kMergeOperands": "0"}} >2018-07-11 05:31:11.388062 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2872] Compacted 3@1 + 4@2 files to L2 => 475594294 bytes >2018-07-11 05:31:11.401609 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:31:11.401376) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 5 46 446 3640 771 0] max score 1.03, MB/sec: 21.7 rd, 21.7 wr, level 2, files in(3, 4) out(7) MB in(213.3, 240.3) out(453.6), read-write-amplify(4.3) write-amplify(2.1) OK, records in: 16827109, records dropped: 0 > >2018-07-11 05:31:11.401620 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:31:11.401400) EVENT_LOG_v1 {"time_micros": 1531287071401389, "job": 2872, "event": "compaction_finished", "compaction_time_micros": 21943021, "output_level": 2, "num_output_files": 7, "total_output_size": 475594294, "num_input_records": 16827109, "num_output_records": 16827109, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 5, 46, 446, 3640, 771, 0]} >2018-07-11 05:31:11.402604 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287071402587, "job": 2872, "event": "table_file_deletion", "file_number": 36091} >2018-07-11 05:31:11.402619 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287071402616, "job": 2872, "event": "table_file_deletion", "file_number": 36090} >2018-07-11 05:31:11.402633 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287071402630, "job": 2872, "event": "table_file_deletion", "file_number": 36089} >2018-07-11 05:31:11.402647 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287071402645, "job": 2872, "event": "table_file_deletion", "file_number": 36055} >2018-07-11 05:31:11.402659 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287071402657, "job": 2872, "event": "table_file_deletion", "file_number": 36049} >2018-07-11 05:31:11.402670 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287071402668, "job": 2872, "event": "table_file_deletion", "file_number": 36048} >2018-07-11 05:31:11.402681 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287071402679, "job": 2872, "event": "table_file_deletion", "file_number": 36047} >2018-07-11 05:31:11.402901 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_picker.cc:461] [default] Expanding@2 1+5(74547042+270604357 bytes) to 3+5 (223639804+270604357bytes) > >2018-07-11 05:31:11.403004 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2873] Compacting 3@2 + 5@3 files to L3, score 1.03 >2018-07-11 05:31:11.403018 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2872 Base level 2, inputs: [36097(71MB) 36098(71MB) 36099(71MB)], [36057(71MB) 36058(1088B) 36059(71MB) 36063(71MB) 36064(44MB)] > >2018-07-11 05:31:11.403032 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287071403021, "job": 2873, "event": "compaction_started", "files_L2": [36097, 36098, 36099], "files_L3": [36057, 36058, 36059, 36063, 36064], "score": 1.02894, "input_data_size": 494244161} >2018-07-11 05:31:14.536097 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2873] Generated table #36101: 2638239 keys, 74546353 bytes >2018-07-11 05:31:14.536151 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287074536122, "cf_name": "default", "job": 2873, "event": "table_file_creation", "file_number": 36101, "file_size": 74546353, "table_properties": {"data_size": 67112070, "index_size": 690114, "filter_size": 6743179, "raw_key_size": 76215791, "raw_average_key_size": 28, "raw_value_size": 31365728, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638239, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:31:17.910065 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2873] Generated table #36102: 2638241 keys, 74547101 bytes >2018-07-11 05:31:17.910118 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287077910101, "cf_name": "default", "job": 2873, "event": "table_file_creation", "file_number": 36102, "file_size": 74547101, "table_properties": {"data_size": 67112171, "index_size": 690757, "filter_size": 6743183, "raw_key_size": 76215850, "raw_average_key_size": 28, "raw_value_size": 31365753, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:31:18.068916 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2873] Generated table #36103: 83 keys, 3351 bytes >2018-07-11 05:31:18.068964 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287078068949, "cf_name": "default", "job": 2873, "event": "table_file_creation", "file_number": 36103, "file_size": 3351, "table_properties": {"data_size": 2136, "index_size": 28, "filter_size": 218, "raw_key_size": 2398, "raw_average_key_size": 28, "raw_value_size": 987, "raw_average_value_size": 11, "num_data_blocks": 1, "num_entries": 83, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:31:21.074169 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2873] Generated table #36104: 2638242 keys, 74545965 bytes >2018-07-11 05:31:21.074217 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287081074197, "cf_name": "default", "job": 2873, "event": "table_file_creation", "file_number": 36104, "file_size": 74545965, "table_properties": {"data_size": 67112158, "index_size": 689631, "filter_size": 6743186, "raw_key_size": 76215880, "raw_average_key_size": 28, "raw_value_size": 31365766, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638242, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:31:24.069287 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2873] Generated table #36105: 2638241 keys, 74547041 bytes >2018-07-11 05:31:24.069332 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287084069304, "cf_name": "default", "job": 2873, "event": "table_file_creation", "file_number": 36105, "file_size": 74547041, "table_properties": {"data_size": 67112148, "index_size": 690720, "filter_size": 6743183, "raw_key_size": 76215851, "raw_average_key_size": 28, "raw_value_size": 31365754, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:31:27.087821 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2873] Generated table #36106: 2638241 keys, 74546122 bytes >2018-07-11 05:31:27.087867 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287087087839, "cf_name": "default", "job": 2873, "event": "table_file_creation", "file_number": 36106, "file_size": 74546122, "table_properties": {"data_size": 67112106, "index_size": 689843, "filter_size": 6743183, "raw_key_size": 76215858, "raw_average_key_size": 28, "raw_value_size": 31365761, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:31:30.180515 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2873] Generated table #36107: 2638241 keys, 74546327 bytes >2018-07-11 05:31:30.180571 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287090180554, "cf_name": "default", "job": 2873, "event": "table_file_creation", "file_number": 36107, "file_size": 74546327, "table_properties": {"data_size": 67112168, "index_size": 689986, "filter_size": 6743183, "raw_key_size": 76215852, "raw_average_key_size": 28, "raw_value_size": 31365755, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:31:32.062455 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2873] Generated table #36108: 1519837 keys, 46962495 bytes >2018-07-11 05:31:32.062496 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287092062480, "cf_name": "default", "job": 2873, "event": "table_file_creation", "file_number": 36108, "file_size": 46962495, "table_properties": {"data_size": 42634607, "index_size": 432961, "filter_size": 3893939, "raw_key_size": 44346654, "raw_average_key_size": 29, "raw_value_size": 21521526, "raw_average_value_size": 14, "num_data_blocks": 10537, "num_entries": 1519837, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:31:32.062538 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2873] Compacted 3@2 + 5@3 files to L3 => 494244755 bytes >2018-07-11 05:31:32.084678 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:31:32.084398) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 5 43 449 3640 771 0] max score 1.01, MB/sec: 23.9 rd, 23.9 wr, level 3, files in(3, 5) out(8) MB in(213.3, 258.1) out(471.3), read-write-amplify(4.4) write-amplify(2.2) OK, records in: 17349365, records dropped: 0 > >2018-07-11 05:31:32.084690 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:31:32.084425) EVENT_LOG_v1 {"time_micros": 1531287092084413, "job": 2873, "event": "compaction_finished", "compaction_time_micros": 20659484, "output_level": 3, "num_output_files": 8, "total_output_size": 494244755, "num_input_records": 17349365, "num_output_records": 17349365, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 5, 43, 449, 3640, 771, 0]} >2018-07-11 05:31:32.085632 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287092085627, "job": 2873, "event": "table_file_deletion", "file_number": 36099} >2018-07-11 05:31:32.085646 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287092085643, "job": 2873, "event": "table_file_deletion", "file_number": 36098} >2018-07-11 05:31:32.085660 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287092085658, "job": 2873, "event": "table_file_deletion", "file_number": 36097} >2018-07-11 05:31:32.085673 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287092085671, "job": 2873, "event": "table_file_deletion", "file_number": 36064} >2018-07-11 05:31:32.085685 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287092085683, "job": 2873, "event": "table_file_deletion", "file_number": 36063} >2018-07-11 05:31:32.085700 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287092085697, "job": 2873, "event": "table_file_deletion", "file_number": 36059} >2018-07-11 05:31:32.085715 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287092085712, "job": 2873, "event": "table_file_deletion", "file_number": 36058} >2018-07-11 05:31:32.085729 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287092085727, "job": 2873, "event": "table_file_deletion", "file_number": 36057} >2018-07-11 05:31:32.086044 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2874] Compacting 1@3 + 4@4 files to L4, score 1.01 >2018-07-11 05:31:32.086056 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2873 Base level 3, inputs: [36102(71MB)], [36021(71MB) 36022(71MB) 36017(71MB) 36018(71MB)] > >2018-07-11 05:31:32.086071 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287092086060, "job": 2874, "event": "compaction_started", "files_L3": [36102], "files_L4": [36021, 36022, 36017, 36018], "score": 1.00568, "input_data_size": 372731862} >2018-07-11 05:31:35.367193 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2874] Generated table #36109: 2638241 keys, 74546293 bytes >2018-07-11 05:31:35.367240 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287095367224, "cf_name": "default", "job": 2874, "event": "table_file_creation", "file_number": 36109, "file_size": 74546293, "table_properties": {"data_size": 67112151, "index_size": 689969, "filter_size": 6743183, "raw_key_size": 76215858, "raw_average_key_size": 28, "raw_value_size": 31365761, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:31:38.447716 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2874] Generated table #36110: 2638241 keys, 74546908 bytes >2018-07-11 05:31:38.447760 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287098447734, "cf_name": "default", "job": 2874, "event": "table_file_creation", "file_number": 36110, "file_size": 74546908, "table_properties": {"data_size": 67112133, "index_size": 690602, "filter_size": 6743183, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:31:41.532585 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2874] Generated table #36111: 2638240 keys, 74546229 bytes >2018-07-11 05:31:41.532620 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287101532604, "cf_name": "default", "job": 2874, "event": "table_file_creation", "file_number": 36111, "file_size": 74546229, "table_properties": {"data_size": 67112142, "index_size": 689916, "filter_size": 6743181, "raw_key_size": 76215833, "raw_average_key_size": 28, "raw_value_size": 31365753, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:31:44.662143 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2874] Generated table #36112: 2638241 keys, 74546386 bytes >2018-07-11 05:31:44.662187 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287104662160, "cf_name": "default", "job": 2874, "event": "table_file_creation", "file_number": 36112, "file_size": 74546386, "table_properties": {"data_size": 67112157, "index_size": 690056, "filter_size": 6743183, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:31:47.569106 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2874] Generated table #36113: 2638241 keys, 74545889 bytes >2018-07-11 05:31:47.569151 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287107569124, "cf_name": "default", "job": 2874, "event": "table_file_creation", "file_number": 36113, "file_size": 74545889, "table_properties": {"data_size": 67112111, "index_size": 689605, "filter_size": 6743183, "raw_key_size": 76215852, "raw_average_key_size": 28, "raw_value_size": 31365755, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:31:47.585334 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2874] Generated table #36114: 1 keys, 1064 bytes >2018-07-11 05:31:47.585360 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287107585347, "cf_name": "default", "job": 2874, "event": "table_file_creation", "file_number": 36114, "file_size": 1064, "table_properties": {"data_size": 57, "index_size": 27, "filter_size": 18, "raw_key_size": 29, "raw_average_key_size": 29, "raw_value_size": 12, "raw_average_value_size": 12, "num_data_blocks": 1, "num_entries": 1, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:31:47.585387 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2874] Compacted 1@3 + 4@4 files to L4 => 372732769 bytes >2018-07-11 05:31:47.601170 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:31:47.600922) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 5 43 448 3642 771 0] max score 1.00, MB/sec: 24.0 rd, 24.0 wr, level 4, files in(1, 4) out(6) MB in(71.1, 284.4) out(355.5), read-write-amplify(10.0) write-amplify(5.0) OK, records in: 13191205, records dropped: 0 > >2018-07-11 05:31:47.601180 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:31:47.600948) EVENT_LOG_v1 {"time_micros": 1531287107600937, "job": 2874, "event": "compaction_finished", "compaction_time_micros": 15499299, "output_level": 4, "num_output_files": 6, "total_output_size": 372732769, "num_input_records": 13191205, "num_output_records": 13191205, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 5, 43, 448, 3642, 771, 0]} >2018-07-11 05:31:47.602132 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287107602128, "job": 2874, "event": "table_file_deletion", "file_number": 36102} >2018-07-11 05:31:47.602148 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287107602146, "job": 2874, "event": "table_file_deletion", "file_number": 36022} >2018-07-11 05:31:47.602163 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287107602161, "job": 2874, "event": "table_file_deletion", "file_number": 36021} >2018-07-11 05:31:47.602177 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287107602175, "job": 2874, "event": "table_file_deletion", "file_number": 36018} >2018-07-11 05:31:47.602191 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287107602188, "job": 2874, "event": "table_file_deletion", "file_number": 36017} >2018-07-11 05:31:47.618862 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:31:47.602517) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #36105 to level-4 74547041 bytes > >2018-07-11 05:31:47.618872 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:31:47.618765) EVENT_LOG_v1 {"time_micros": 1531287107618759, "job": 2875, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 74547041} >2018-07-11 05:31:47.618876 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:31:47.618773) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-4 74547041 bytes OK: base level 1 max bytes base 268435456 files[0 5 43 447 3643 771 0] max score 1.00 > >2018-07-11 05:31:47.668921 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:31:47.619749) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #36105 to level-5 74547041 bytes > >2018-07-11 05:31:47.668932 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:31:47.668717) EVENT_LOG_v1 {"time_micros": 1531287107668711, "job": 2876, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74547041} >2018-07-11 05:31:47.668937 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:31:47.668724) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74547041 bytes OK: base level 1 max bytes base 268435456 files[0 5 43 447 3642 772 0] max score 1.00 > >2018-07-11 05:31:47.685582 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:31:47.669815) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #36110 to level-5 74546908 bytes > >2018-07-11 05:31:47.685593 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:31:47.685474) EVENT_LOG_v1 {"time_micros": 1531287107685468, "job": 2877, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546908} >2018-07-11 05:31:47.685595 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:31:47.685482) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546908 bytes OK: base level 1 max bytes base 268435456 files[0 5 43 447 3641 773 0] max score 1.00 > >2018-07-11 05:31:47.686513 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2878] Compacting 1@3 + 18@4 files to L4, score 1.00 >2018-07-11 05:31:47.686544 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2877 Base level 3, inputs: [36101(71MB)], [36065(71MB) 36067(71MB) 36068(71MB) 36069(71MB) 36070(71MB) 36071(71MB) 36072(71MB) 36073(71MB) 36075(71MB) 36076(71MB) 35990(71MB) 35991(71MB) 35992(71MB) 35993(71MB) 35994(71MB) 35995(71MB) 35963(71MB) 36109(71MB)] > >2018-07-11 05:31:47.686560 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287107686548, "job": 2878, "event": "compaction_started", "files_L3": [36101], "files_L4": [36065, 36067, 36068, 36069, 36070, 36071, 36072, 36073, 36075, 36076, 35990, 35991, 35992, 35993, 35994, 35995, 35963, 36109], "score": 1.00013, "input_data_size": 1416379469} >2018-07-11 05:31:51.111858 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2878] Generated table #36115: 2638241 keys, 74546360 bytes >2018-07-11 05:31:51.111904 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287111111877, "cf_name": "default", "job": 2878, "event": "table_file_creation", "file_number": 36115, "file_size": 74546360, "table_properties": {"data_size": 67112136, "index_size": 690051, "filter_size": 6743183, "raw_key_size": 76215855, "raw_average_key_size": 28, "raw_value_size": 31365758, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:31:54.389484 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2878] Generated table #36116: 2638240 keys, 74546297 bytes >2018-07-11 05:31:54.389532 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287114389513, "cf_name": "default", "job": 2878, "event": "table_file_creation", "file_number": 36116, "file_size": 74546297, "table_properties": {"data_size": 67112152, "index_size": 689974, "filter_size": 6743181, "raw_key_size": 76215825, "raw_average_key_size": 28, "raw_value_size": 31365745, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:31:57.554998 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2878] Generated table #36117: 2638241 keys, 74546461 bytes >2018-07-11 05:31:57.555041 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287117555014, "cf_name": "default", "job": 2878, "event": "table_file_creation", "file_number": 36117, "file_size": 74546461, "table_properties": {"data_size": 67112121, "index_size": 690167, "filter_size": 6743183, "raw_key_size": 76215858, "raw_average_key_size": 28, "raw_value_size": 31365761, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:32:00.834270 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2878] Generated table #36118: 2638241 keys, 74545893 bytes >2018-07-11 05:32:00.834312 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287120834295, "cf_name": "default", "job": 2878, "event": "table_file_creation", "file_number": 36118, "file_size": 74545893, "table_properties": {"data_size": 67112158, "index_size": 689562, "filter_size": 6743183, "raw_key_size": 76215851, "raw_average_key_size": 28, "raw_value_size": 31365754, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:32:01.248828 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:684] reusing log 36084 from recycle list > >2018-07-11 05:32:01.248989 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #36120. Immutable memtables: 0. > >2018-07-11 05:32:01.325774 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:32:01.249051) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 0 >2018-07-11 05:32:01.325788 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 2879] Flushing memtable with next log file: 36120 > >2018-07-11 05:32:01.325805 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287121325796, "job": 2879, "event": "flush_started", "num_memtables": 1, "num_entries": 5000015, "num_deletes": 0, "memory_usage": 292925440} >2018-07-11 05:32:01.325808 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 2879] Level-0 flush table #36121: started >2018-07-11 05:32:03.891113 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2878] Generated table #36119: 2638241 keys, 74546861 bytes >2018-07-11 05:32:03.891156 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287123891139, "cf_name": "default", "job": 2878, "event": "table_file_creation", "file_number": 36119, "file_size": 74546861, "table_properties": {"data_size": 67112152, "index_size": 690536, "filter_size": 6743183, "raw_key_size": 76215851, "raw_average_key_size": 28, "raw_value_size": 31365754, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:32:05.300746 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287125300722, "cf_name": "default", "job": 2879, "event": "table_file_creation", "file_number": 36121, "file_size": 141286222, "table_properties": {"data_size": 127196937, "index_size": 1308576, "filter_size": 12779719, "raw_key_size": 144445077, "raw_average_key_size": 28, "raw_value_size": 59449772, "raw_average_value_size": 11, "num_data_blocks": 31252, "num_entries": 5000013, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:32:05.300783 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 2879] Level-0 flush table #36121: 141286222 bytes OK >2018-07-11 05:32:05.318681 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:242] adding log 36086 to recycle list > >2018-07-11 05:32:05.318693 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:32:05.300796) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #36121 started >2018-07-11 05:32:05.318696 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:32:05.318537) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #36121: memtable #1 done >2018-07-11 05:32:05.318699 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:32:05.318555) EVENT_LOG_v1 {"time_micros": 1531287125318547, "job": 2879, "event": "flush_finished", "lsm_state": [1, 5, 43, 447, 3641, 773, 0], "immutable_memtables": 0} >2018-07-11 05:32:05.318700 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:32:05.318575) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 5 43 447 3641 773 0] max score 1.00 > >2018-07-11 05:32:05.319876 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 2879] Try to delete WAL files size 178896755, prev total WAL file size 178896767, number of live WAL files 2. > >2018-07-11 05:32:07.085432 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2878] Generated table #36122: 2638241 keys, 74546451 bytes >2018-07-11 05:32:07.085493 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287127085471, "cf_name": "default", "job": 2878, "event": "table_file_creation", "file_number": 36122, "file_size": 74546451, "table_properties": {"data_size": 67112125, "index_size": 690153, "filter_size": 6743183, "raw_key_size": 76215859, "raw_average_key_size": 28, "raw_value_size": 31365762, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:32:10.471665 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2878] Generated table #36123: 2638241 keys, 74546192 bytes >2018-07-11 05:32:10.471711 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287130471683, "cf_name": "default", "job": 2878, "event": "table_file_creation", "file_number": 36123, "file_size": 74546192, "table_properties": {"data_size": 67112148, "index_size": 689871, "filter_size": 6743183, "raw_key_size": 76215852, "raw_average_key_size": 28, "raw_value_size": 31365755, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:32:13.656926 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2878] Generated table #36124: 2638240 keys, 74545926 bytes >2018-07-11 05:32:13.656971 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287133656944, "cf_name": "default", "job": 2878, "event": "table_file_creation", "file_number": 36124, "file_size": 74545926, "table_properties": {"data_size": 67112080, "index_size": 689675, "filter_size": 6743181, "raw_key_size": 76215826, "raw_average_key_size": 28, "raw_value_size": 31365746, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:32:16.736617 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2878] Generated table #36125: 2638240 keys, 74546261 bytes >2018-07-11 05:32:16.736661 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287136736633, "cf_name": "default", "job": 2878, "event": "table_file_creation", "file_number": 36125, "file_size": 74546261, "table_properties": {"data_size": 67112154, "index_size": 689936, "filter_size": 6743181, "raw_key_size": 76215830, "raw_average_key_size": 28, "raw_value_size": 31365750, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:32:20.010821 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2878] Generated table #36126: 2638241 keys, 74547009 bytes >2018-07-11 05:32:20.010865 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287140010838, "cf_name": "default", "job": 2878, "event": "table_file_creation", "file_number": 36126, "file_size": 74547009, "table_properties": {"data_size": 67112109, "index_size": 690727, "filter_size": 6743183, "raw_key_size": 76215855, "raw_average_key_size": 28, "raw_value_size": 31365758, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:32:23.164871 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2878] Generated table #36127: 2638240 keys, 74546223 bytes >2018-07-11 05:32:23.164914 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287143164888, "cf_name": "default", "job": 2878, "event": "table_file_creation", "file_number": 36127, "file_size": 74546223, "table_properties": {"data_size": 67112142, "index_size": 689910, "filter_size": 6743181, "raw_key_size": 76215822, "raw_average_key_size": 28, "raw_value_size": 31365742, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:32:26.490075 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2878] Generated table #36128: 2638241 keys, 74546457 bytes >2018-07-11 05:32:26.490109 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287146490094, "cf_name": "default", "job": 2878, "event": "table_file_creation", "file_number": 36128, "file_size": 74546457, "table_properties": {"data_size": 67112129, "index_size": 690155, "filter_size": 6743183, "raw_key_size": 76215862, "raw_average_key_size": 28, "raw_value_size": 31365765, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:32:29.761183 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2878] Generated table #36129: 2638241 keys, 74545915 bytes >2018-07-11 05:32:29.761216 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287149761202, "cf_name": "default", "job": 2878, "event": "table_file_creation", "file_number": 36129, "file_size": 74545915, "table_properties": {"data_size": 67112139, "index_size": 689603, "filter_size": 6743183, "raw_key_size": 76215847, "raw_average_key_size": 28, "raw_value_size": 31365750, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:32:33.045240 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2878] Generated table #36130: 2638242 keys, 74546600 bytes >2018-07-11 05:32:33.045277 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287153045262, "cf_name": "default", "job": 2878, "event": "table_file_creation", "file_number": 36130, "file_size": 74546600, "table_properties": {"data_size": 67112173, "index_size": 690251, "filter_size": 6743186, "raw_key_size": 76215888, "raw_average_key_size": 28, "raw_value_size": 31365774, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638242, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:32:36.202799 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2878] Generated table #36131: 2638241 keys, 74546218 bytes >2018-07-11 05:32:36.202844 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287156202817, "cf_name": "default", "job": 2878, "event": "table_file_creation", "file_number": 36131, "file_size": 74546218, "table_properties": {"data_size": 67112160, "index_size": 689885, "filter_size": 6743183, "raw_key_size": 76215860, "raw_average_key_size": 28, "raw_value_size": 31365763, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:32:39.426294 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2878] Generated table #36132: 2638241 keys, 74545803 bytes >2018-07-11 05:32:39.426340 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287159426313, "cf_name": "default", "job": 2878, "event": "table_file_creation", "file_number": 36132, "file_size": 74545803, "table_properties": {"data_size": 67112090, "index_size": 689540, "filter_size": 6743183, "raw_key_size": 76215847, "raw_average_key_size": 28, "raw_value_size": 31365750, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:32:42.560559 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2878] Generated table #36133: 2638240 keys, 74546315 bytes >2018-07-11 05:32:42.560603 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287162560577, "cf_name": "default", "job": 2878, "event": "table_file_creation", "file_number": 36133, "file_size": 74546315, "table_properties": {"data_size": 67112154, "index_size": 689990, "filter_size": 6743181, "raw_key_size": 76215820, "raw_average_key_size": 28, "raw_value_size": 31365740, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:32:45.642257 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2878] Generated table #36134: 2638241 keys, 74546880 bytes >2018-07-11 05:32:45.642303 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287165642275, "cf_name": "default", "job": 2878, "event": "table_file_creation", "file_number": 36134, "file_size": 74546880, "table_properties": {"data_size": 67112136, "index_size": 690571, "filter_size": 6743183, "raw_key_size": 76215859, "raw_average_key_size": 28, "raw_value_size": 31365762, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:32:48.626028 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2878] Generated table #36135: 2638239 keys, 74546102 bytes >2018-07-11 05:32:48.626072 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287168626045, "cf_name": "default", "job": 2878, "event": "table_file_creation", "file_number": 36135, "file_size": 74546102, "table_properties": {"data_size": 67112065, "index_size": 689879, "filter_size": 6743168, "raw_key_size": 76215795, "raw_average_key_size": 28, "raw_value_size": 31365732, "raw_average_value_size": 11, "num_data_blocks": 16489, "num_entries": 2638239, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:32:48.626123 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2878] Compacted 1@3 + 18@4 files to L4 => 1416380224 bytes >2018-07-11 05:32:48.643316 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:32:48.642819) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[1 5 43 446 3642 773 0] max score 1.00, MB/sec: 23.2 rd, 23.2 wr, level 4, files in(1, 18) out(19) MB in(71.1, 1279.7) out(1350.8), read-write-amplify(38.0) write-amplify(19.0) OK, records in: 50126573, records dropped: 0 > >2018-07-11 05:32:48.643328 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:32:48.642847) EVENT_LOG_v1 {"time_micros": 1531287168642834, "job": 2878, "event": "compaction_finished", "compaction_time_micros": 60939532, "output_level": 4, "num_output_files": 19, "total_output_size": 1416380224, "num_input_records": 50126573, "num_output_records": 50126573, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 5, 43, 446, 3642, 773, 0]} >2018-07-11 05:32:48.644251 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287168644230, "job": 2878, "event": "table_file_deletion", "file_number": 36109} >2018-07-11 05:32:48.644268 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287168644266, "job": 2878, "event": "table_file_deletion", "file_number": 36101} >2018-07-11 05:32:48.644281 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287168644279, "job": 2878, "event": "table_file_deletion", "file_number": 36076} >2018-07-11 05:32:48.644293 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287168644291, "job": 2878, "event": "table_file_deletion", "file_number": 36075} >2018-07-11 05:32:48.644309 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287168644307, "job": 2878, "event": "table_file_deletion", "file_number": 36073} >2018-07-11 05:32:48.644322 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287168644320, "job": 2878, "event": "table_file_deletion", "file_number": 36072} >2018-07-11 05:32:48.644335 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287168644333, "job": 2878, "event": "table_file_deletion", "file_number": 36071} >2018-07-11 05:32:48.644348 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287168644346, "job": 2878, "event": "table_file_deletion", "file_number": 36070} >2018-07-11 05:32:48.644359 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287168644357, "job": 2878, "event": "table_file_deletion", "file_number": 36069} >2018-07-11 05:32:48.644370 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287168644368, "job": 2878, "event": "table_file_deletion", "file_number": 36068} >2018-07-11 05:32:48.644383 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287168644381, "job": 2878, "event": "table_file_deletion", "file_number": 36067} >2018-07-11 05:32:48.644395 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287168644393, "job": 2878, "event": "table_file_deletion", "file_number": 36065} >2018-07-11 05:32:48.644414 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287168644412, "job": 2878, "event": "table_file_deletion", "file_number": 35995} >2018-07-11 05:32:48.644427 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287168644425, "job": 2878, "event": "table_file_deletion", "file_number": 35994} >2018-07-11 05:32:48.644441 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287168644438, "job": 2878, "event": "table_file_deletion", "file_number": 35993} >2018-07-11 05:32:48.644454 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287168644452, "job": 2878, "event": "table_file_deletion", "file_number": 35992} >2018-07-11 05:32:48.644467 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287168644465, "job": 2878, "event": "table_file_deletion", "file_number": 35991} >2018-07-11 05:32:48.644479 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287168644477, "job": 2878, "event": "table_file_deletion", "file_number": 35990} >2018-07-11 05:32:48.644492 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287168644490, "job": 2878, "event": "table_file_deletion", "file_number": 35963} >2018-07-11 05:32:48.667889 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:32:48.644778) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #36126 to level-5 74547009 bytes > >2018-07-11 05:32:48.667899 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:32:48.667796) EVENT_LOG_v1 {"time_micros": 1531287168667790, "job": 2880, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74547009} >2018-07-11 05:32:48.667901 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:32:48.667804) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74547009 bytes OK: base level 1 max bytes base 268435456 files[1 5 43 446 3641 774 0] max score 1.00 > >2018-07-11 05:34:14.301296 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:684] reusing log 36086 from recycle list > >2018-07-11 05:34:14.301456 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #36136. Immutable memtables: 0. > >2018-07-11 05:34:14.360908 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:34:14.301523) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1 >2018-07-11 05:34:14.360924 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 2881] Flushing memtable with next log file: 36136 > >2018-07-11 05:34:14.360942 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287254360932, "job": 2881, "event": "flush_started", "num_memtables": 1, "num_entries": 5000015, "num_deletes": 0, "memory_usage": 292947088} >2018-07-11 05:34:14.360946 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 2881] Level-0 flush table #36137: started >2018-07-11 05:34:18.283101 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287258283080, "cf_name": "default", "job": 2881, "event": "table_file_creation", "file_number": 36137, "file_size": 141286221, "table_properties": {"data_size": 127196936, "index_size": 1308576, "filter_size": 12779719, "raw_key_size": 144445077, "raw_average_key_size": 28, "raw_value_size": 59449772, "raw_average_value_size": 11, "num_data_blocks": 31252, "num_entries": 5000013, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:34:18.283124 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 2881] Level-0 flush table #36137: 141286221 bytes OK >2018-07-11 05:34:18.298935 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:242] adding log 36120 to recycle list > >2018-07-11 05:34:18.298960 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:34:18.283148) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #36137 started >2018-07-11 05:34:18.298964 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:34:18.298717) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #36137: memtable #1 done >2018-07-11 05:34:18.298966 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:34:18.298734) EVENT_LOG_v1 {"time_micros": 1531287258298727, "job": 2881, "event": "flush_finished", "lsm_state": [2, 5, 43, 446, 3641, 774, 0], "immutable_memtables": 0} >2018-07-11 05:34:18.298968 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:34:18.298871) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[2 5 43 446 3641 774 0] max score 1.05 > >2018-07-11 05:34:18.299093 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2882] Compacting 2@0 + 3@1 files to L1, score 1.05 >2018-07-11 05:34:18.299110 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2881 Base level 0, inputs: [36137(134MB) 36121(134MB)], [36088(71MB) 36092(58MB) 36093(62MB)] > >2018-07-11 05:34:18.299128 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287258299114, "job": 2882, "event": "compaction_started", "files_L0": [36137, 36121], "files_L1": [36088, 36092, 36093], "score": 1.05266, "input_data_size": 484256706} >2018-07-11 05:34:18.299855 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 2881] Try to delete WAL files size 178896755, prev total WAL file size 178896767, number of live WAL files 2. > >2018-07-11 05:34:21.452617 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2882] Generated table #36138: 2623046 keys, 74507620 bytes >2018-07-11 05:34:21.452657 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287261452641, "cf_name": "default", "job": 2882, "event": "table_file_creation", "file_number": 36138, "file_size": 74507620, "table_properties": {"data_size": 67109977, "index_size": 691449, "filter_size": 6705204, "raw_key_size": 75811249, "raw_average_key_size": 28, "raw_value_size": 31561139, "raw_average_value_size": 12, "num_data_blocks": 16475, "num_entries": 2623046, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:34:24.291857 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2882] Generated table #36139: 2638240 keys, 74546073 bytes >2018-07-11 05:34:24.291904 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287264291888, "cf_name": "default", "job": 2882, "event": "table_file_creation", "file_number": 36139, "file_size": 74546073, "table_properties": {"data_size": 67112164, "index_size": 689738, "filter_size": 6743181, "raw_key_size": 76215828, "raw_average_key_size": 28, "raw_value_size": 31365748, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:34:27.101560 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2882] Generated table #36140: 2638241 keys, 74546415 bytes >2018-07-11 05:34:27.101597 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287267101582, "cf_name": "default", "job": 2882, "event": "table_file_creation", "file_number": 36140, "file_size": 74546415, "table_properties": {"data_size": 67112111, "index_size": 690131, "filter_size": 6743183, "raw_key_size": 76215850, "raw_average_key_size": 28, "raw_value_size": 31365753, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:34:29.917201 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2882] Generated table #36141: 2638241 keys, 74546928 bytes >2018-07-11 05:34:29.917253 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287269917220, "cf_name": "default", "job": 2882, "event": "table_file_creation", "file_number": 36141, "file_size": 74546928, "table_properties": {"data_size": 67112164, "index_size": 690591, "filter_size": 6743183, "raw_key_size": 76215856, "raw_average_key_size": 28, "raw_value_size": 31365759, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:34:32.641839 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2882] Generated table #36142: 2638241 keys, 74546079 bytes >2018-07-11 05:34:32.641891 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287272641874, "cf_name": "default", "job": 2882, "event": "table_file_creation", "file_number": 36142, "file_size": 74546079, "table_properties": {"data_size": 67112128, "index_size": 689778, "filter_size": 6743183, "raw_key_size": 76215849, "raw_average_key_size": 28, "raw_value_size": 31365752, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:34:34.368852 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2882] Generated table #36143: 1541028 keys, 49545928 bytes >2018-07-11 05:34:34.368904 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287274368888, "cf_name": "default", "job": 2882, "event": "table_file_creation", "file_number": 36143, "file_size": 49545928, "table_properties": {"data_size": 45147790, "index_size": 444792, "filter_size": 3952358, "raw_key_size": 45127665, "raw_average_key_size": 29, "raw_value_size": 23569511, "raw_average_value_size": 15, "num_data_blocks": 10852, "num_entries": 1541028, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:34:35.610972 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2882] Generated table #36144: 157640 keys, 62012156 bytes >2018-07-11 05:34:35.611016 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287275610990, "cf_name": "default", "job": 2882, "event": "table_file_creation", "file_number": 36144, "file_size": 62012156, "table_properties": {"data_size": 60956776, "index_size": 520745, "filter_size": 533647, "raw_key_size": 14521388, "raw_average_key_size": 92, "raw_value_size": 47770489, "raw_average_value_size": 303, "num_data_blocks": 13945, "num_entries": 157640, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "80898", "kMergeOperands": "0"}} >2018-07-11 05:34:35.611085 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2882] Compacted 2@0 + 3@1 files to L1 => 484251199 bytes >2018-07-11 05:34:35.631996 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:34:35.631773) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 9 43 446 3641 774 0] max score 2.02, MB/sec: 28.0 rd, 28.0 wr, level 1, files in(2, 3) out(7) MB in(269.5, 192.3) out(461.8), read-write-amplify(3.4) write-amplify(1.7) OK, records in: 14874683, records dropped: 6 > >2018-07-11 05:34:35.632007 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:34:35.631799) EVENT_LOG_v1 {"time_micros": 1531287275631788, "job": 2882, "event": "compaction_finished", "compaction_time_micros": 17311938, "output_level": 1, "num_output_files": 7, "total_output_size": 484251199, "num_input_records": 14874683, "num_output_records": 14874677, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 9, 43, 446, 3641, 774, 0]} >2018-07-11 05:34:35.632945 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287275632939, "job": 2882, "event": "table_file_deletion", "file_number": 36137} >2018-07-11 05:34:35.632969 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287275632965, "job": 2882, "event": "table_file_deletion", "file_number": 36121} >2018-07-11 05:34:35.632994 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287275632990, "job": 2882, "event": "table_file_deletion", "file_number": 36093} >2018-07-11 05:34:35.633017 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287275633013, "job": 2882, "event": "table_file_deletion", "file_number": 36092} >2018-07-11 05:34:35.633041 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287275633038, "job": 2882, "event": "table_file_deletion", "file_number": 36088} >2018-07-11 05:34:35.633268 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_picker.cc:461] [default] Expanding@1 1+1(74546928+28316192 bytes) to 3+1 (223639422+28316192bytes) > >2018-07-11 05:34:35.633368 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2883] Compacting 3@1 + 1@2 files to L2, score 2.02 >2018-07-11 05:34:35.633381 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2882 Base level 1, inputs: [36140(71MB) 36141(71MB) 36142(71MB)], [36100(27MB)] > >2018-07-11 05:34:35.633401 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287275633386, "job": 2883, "event": "compaction_started", "files_L1": [36140, 36141, 36142], "files_L2": [36100], "score": 2.01582, "input_data_size": 251955614} >2018-07-11 05:34:39.107496 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2883] Generated table #36145: 2638240 keys, 74546131 bytes >2018-07-11 05:34:39.107552 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287279107536, "cf_name": "default", "job": 2883, "event": "table_file_creation", "file_number": 36145, "file_size": 74546131, "table_properties": {"data_size": 67112139, "index_size": 689821, "filter_size": 6743181, "raw_key_size": 76215819, "raw_average_key_size": 28, "raw_value_size": 31365739, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:34:43.560792 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2883] Generated table #36146: 2638241 keys, 74545710 bytes >2018-07-11 05:34:43.560844 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287283560828, "cf_name": "default", "job": 2883, "event": "table_file_creation", "file_number": 36146, "file_size": 74545710, "table_properties": {"data_size": 67112135, "index_size": 689402, "filter_size": 6743183, "raw_key_size": 76215859, "raw_average_key_size": 28, "raw_value_size": 31365762, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:34:46.953116 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2883] Generated table #36147: 2638241 keys, 74546779 bytes >2018-07-11 05:34:46.953150 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287286953136, "cf_name": "default", "job": 2883, "event": "table_file_creation", "file_number": 36147, "file_size": 74546779, "table_properties": {"data_size": 67112133, "index_size": 690473, "filter_size": 6743183, "raw_key_size": 76215847, "raw_average_key_size": 28, "raw_value_size": 31365750, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:34:48.177800 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2883] Generated table #36148: 997668 keys, 28316373 bytes >2018-07-11 05:34:48.177831 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287288177817, "cf_name": "default", "job": 2883, "event": "table_file_creation", "file_number": 36148, "file_size": 28316373, "table_properties": {"data_size": 25503448, "index_size": 261669, "filter_size": 2550268, "raw_key_size": 28837266, "raw_average_key_size": 28, "raw_value_size": 11966532, "raw_average_value_size": 11, "num_data_blocks": 6268, "num_entries": 997668, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "11", "kMergeOperands": "0"}} >2018-07-11 05:34:48.177864 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2883] Compacted 3@1 + 1@2 files to L2 => 251954993 bytes >2018-07-11 05:34:48.198303 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:34:48.198096) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 6 46 446 3641 774 0] max score 1.18, MB/sec: 20.1 rd, 20.1 wr, level 2, files in(3, 1) out(4) MB in(213.3, 27.0) out(240.3), read-write-amplify(2.3) write-amplify(1.1) OK, records in: 8912390, records dropped: 0 > >2018-07-11 05:34:48.198314 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:34:48.198120) EVENT_LOG_v1 {"time_micros": 1531287288198109, "job": 2883, "event": "compaction_finished", "compaction_time_micros": 12544434, "output_level": 2, "num_output_files": 4, "total_output_size": 251954993, "num_input_records": 8912390, "num_output_records": 8912390, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 6, 46, 446, 3641, 774, 0]} >2018-07-11 05:34:48.199228 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287288199223, "job": 2883, "event": "table_file_deletion", "file_number": 36142} >2018-07-11 05:34:48.199243 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287288199241, "job": 2883, "event": "table_file_deletion", "file_number": 36141} >2018-07-11 05:34:48.199263 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287288199260, "job": 2883, "event": "table_file_deletion", "file_number": 36140} >2018-07-11 05:34:48.199279 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287288199277, "job": 2883, "event": "table_file_deletion", "file_number": 36100} >2018-07-11 05:34:48.199603 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2884] Compacting 1@1 + 4@2 files to L2, score 1.18 >2018-07-11 05:34:48.199615 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2883 Base level 1, inputs: [36139(71MB)], [36094(71MB) 36095(71MB) 36096(71MB) 36145(71MB)] > >2018-07-11 05:34:48.199641 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287288199618, "job": 2884, "event": "compaction_started", "files_L1": [36139], "files_L2": [36094, 36095, 36096, 36145], "score": 1.1827, "input_data_size": 372730502} >2018-07-11 05:34:51.463496 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2884] Generated table #36149: 2638240 keys, 74545936 bytes >2018-07-11 05:34:51.463546 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287291463529, "cf_name": "default", "job": 2884, "event": "table_file_creation", "file_number": 36149, "file_size": 74545936, "table_properties": {"data_size": 67112137, "index_size": 689628, "filter_size": 6743181, "raw_key_size": 76215811, "raw_average_key_size": 28, "raw_value_size": 31365731, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:34:54.835905 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2884] Generated table #36150: 2638239 keys, 74545975 bytes >2018-07-11 05:34:54.835960 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287294835931, "cf_name": "default", "job": 2884, "event": "table_file_creation", "file_number": 36150, "file_size": 74545975, "table_properties": {"data_size": 67112076, "index_size": 689730, "filter_size": 6743179, "raw_key_size": 76215791, "raw_average_key_size": 28, "raw_value_size": 31365728, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638239, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:34:58.259878 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2884] Generated table #36151: 2638240 keys, 74546182 bytes >2018-07-11 05:34:58.259912 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287298259897, "cf_name": "default", "job": 2884, "event": "table_file_creation", "file_number": 36151, "file_size": 74546182, "table_properties": {"data_size": 67112138, "index_size": 689873, "filter_size": 6743181, "raw_key_size": 76215826, "raw_average_key_size": 28, "raw_value_size": 31365746, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:35:01.532054 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2884] Generated table #36152: 2638240 keys, 74546346 bytes >2018-07-11 05:35:01.532105 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287301532089, "cf_name": "default", "job": 2884, "event": "table_file_creation", "file_number": 36152, "file_size": 74546346, "table_properties": {"data_size": 67112168, "index_size": 690007, "filter_size": 6743181, "raw_key_size": 76215832, "raw_average_key_size": 28, "raw_value_size": 31365752, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:35:04.677789 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2884] Generated table #36153: 2638240 keys, 74545658 bytes >2018-07-11 05:35:04.677829 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287304677813, "cf_name": "default", "job": 2884, "event": "table_file_creation", "file_number": 36153, "file_size": 74545658, "table_properties": {"data_size": 67112082, "index_size": 689416, "filter_size": 6743170, "raw_key_size": 76215817, "raw_average_key_size": 28, "raw_value_size": 31365737, "raw_average_value_size": 11, "num_data_blocks": 16489, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:35:04.677866 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2884] Compacted 1@1 + 4@2 files to L2 => 372730097 bytes >2018-07-11 05:35:04.698024 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:35:04.697807) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 5 47 446 3641 774 0] max score 1.06, MB/sec: 22.6 rd, 22.6 wr, level 2, files in(1, 4) out(5) MB in(71.1, 284.4) out(355.5), read-write-amplify(10.0) write-amplify(5.0) OK, records in: 13191199, records dropped: 0 > >2018-07-11 05:35:04.698036 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:35:04.697832) EVENT_LOG_v1 {"time_micros": 1531287304697821, "job": 2884, "event": "compaction_finished", "compaction_time_micros": 16478209, "output_level": 2, "num_output_files": 5, "total_output_size": 372730097, "num_input_records": 13191199, "num_output_records": 13191199, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 5, 47, 446, 3641, 774, 0]} >2018-07-11 05:35:04.698974 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287304698970, "job": 2884, "event": "table_file_deletion", "file_number": 36145} >2018-07-11 05:35:04.698990 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287304698988, "job": 2884, "event": "table_file_deletion", "file_number": 36139} >2018-07-11 05:35:04.699005 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287304699003, "job": 2884, "event": "table_file_deletion", "file_number": 36096} >2018-07-11 05:35:04.699019 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287304699017, "job": 2884, "event": "table_file_deletion", "file_number": 36095} >2018-07-11 05:35:04.699032 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287304699030, "job": 2884, "event": "table_file_deletion", "file_number": 36094} >2018-07-11 05:35:04.699263 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_picker.cc:461] [default] Expanding@2 1+1(74546779+46962495 bytes) to 3+1 (223638147+46962495bytes) > >2018-07-11 05:35:04.699363 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2885] Compacting 3@2 + 1@3 files to L3, score 1.06 >2018-07-11 05:35:04.699375 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2884 Base level 2, inputs: [36153(71MB) 36146(71MB) 36147(71MB)], [36108(44MB)] > >2018-07-11 05:35:04.699389 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287304699378, "job": 2885, "event": "compaction_started", "files_L2": [36153, 36146, 36147], "files_L3": [36108], "score": 1.05671, "input_data_size": 270600642} >2018-07-11 05:35:07.781618 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2885] Generated table #36154: 2638241 keys, 74545779 bytes >2018-07-11 05:35:07.781670 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287307781654, "cf_name": "default", "job": 2885, "event": "table_file_creation", "file_number": 36154, "file_size": 74545779, "table_properties": {"data_size": 67112139, "index_size": 689467, "filter_size": 6743183, "raw_key_size": 76215848, "raw_average_key_size": 28, "raw_value_size": 31365751, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:35:10.997238 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2885] Generated table #36155: 2638241 keys, 74546728 bytes >2018-07-11 05:35:10.997298 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287310997282, "cf_name": "default", "job": 2885, "event": "table_file_creation", "file_number": 36155, "file_size": 74546728, "table_properties": {"data_size": 67112148, "index_size": 690407, "filter_size": 6743183, "raw_key_size": 76215856, "raw_average_key_size": 28, "raw_value_size": 31365759, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:35:14.529704 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2885] Generated table #36156: 2638241 keys, 74546571 bytes >2018-07-11 05:35:14.529751 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287314529724, "cf_name": "default", "job": 2885, "event": "table_file_creation", "file_number": 36156, "file_size": 74546571, "table_properties": {"data_size": 67112125, "index_size": 690273, "filter_size": 6743183, "raw_key_size": 76215856, "raw_average_key_size": 28, "raw_value_size": 31365759, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:35:16.398624 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2885] Generated table #36157: 1519836 keys, 46962703 bytes >2018-07-11 05:35:16.398666 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287316398640, "cf_name": "default", "job": 2885, "event": "table_file_creation", "file_number": 36157, "file_size": 46962703, "table_properties": {"data_size": 42634558, "index_size": 433220, "filter_size": 3893937, "raw_key_size": 44346617, "raw_average_key_size": 29, "raw_value_size": 21521506, "raw_average_value_size": 14, "num_data_blocks": 10537, "num_entries": 1519836, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:35:16.398700 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2885] Compacted 3@2 + 1@3 files to L3 => 270601781 bytes >2018-07-11 05:35:16.422968 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:35:16.422728) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 5 44 449 3641 774 0] max score 1.01, MB/sec: 23.1 rd, 23.1 wr, level 3, files in(3, 1) out(4) MB in(213.3, 44.8) out(258.1), read-write-amplify(2.4) write-amplify(1.2) OK, records in: 9434559, records dropped: 0 > >2018-07-11 05:35:16.422979 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:35:16.422754) EVENT_LOG_v1 {"time_micros": 1531287316422743, "job": 2885, "event": "compaction_finished", "compaction_time_micros": 11699295, "output_level": 3, "num_output_files": 4, "total_output_size": 270601781, "num_input_records": 9434559, "num_output_records": 9434559, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 5, 44, 449, 3641, 774, 0]} >2018-07-11 05:35:16.423939 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287316423935, "job": 2885, "event": "table_file_deletion", "file_number": 36153} >2018-07-11 05:35:16.423966 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287316423964, "job": 2885, "event": "table_file_deletion", "file_number": 36147} >2018-07-11 05:35:16.423978 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287316423976, "job": 2885, "event": "table_file_deletion", "file_number": 36146} >2018-07-11 05:35:16.423990 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287316423988, "job": 2885, "event": "table_file_deletion", "file_number": 36108} >2018-07-11 05:35:16.447848 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:35:16.424288) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #36155 to level-4 74546728 bytes > >2018-07-11 05:35:16.447858 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:35:16.447757) EVENT_LOG_v1 {"time_micros": 1531287316447751, "job": 2886, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 74546728} >2018-07-11 05:35:16.447861 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:35:16.447764) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-4 74546728 bytes OK: base level 1 max bytes base 268435456 files[0 5 44 448 3642 774 0] max score 1.00 > >2018-07-11 05:35:16.472902 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:35:16.448773) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #36156 to level-4 74546571 bytes > >2018-07-11 05:35:16.472912 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:35:16.472813) EVENT_LOG_v1 {"time_micros": 1531287316472806, "job": 2887, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 74546571} >2018-07-11 05:35:16.472915 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:35:16.472820) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-4 74546571 bytes OK: base level 1 max bytes base 268435456 files[0 5 44 447 3643 774 0] max score 1.00 > >2018-07-11 05:35:16.489593 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:35:16.473763) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #36134 to level-5 74546880 bytes > >2018-07-11 05:35:16.489603 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:35:16.489502) EVENT_LOG_v1 {"time_micros": 1531287316489496, "job": 2888, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546880} >2018-07-11 05:35:16.489606 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:35:16.489509) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546880 bytes OK: base level 1 max bytes base 268435456 files[0 5 44 447 3642 775 0] max score 1.00 > >2018-07-11 05:35:16.506288 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:35:16.490456) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #36119 to level-5 74546861 bytes > >2018-07-11 05:35:16.506298 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:35:16.506191) EVENT_LOG_v1 {"time_micros": 1531287316506185, "job": 2889, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546861} >2018-07-11 05:35:16.506301 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:35:16.506198) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546861 bytes OK: base level 1 max bytes base 268435456 files[0 5 44 447 3641 776 0] max score 1.00 > >2018-07-11 05:35:16.522966 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:35:16.507140) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #36107 to level-4 74546327 bytes > >2018-07-11 05:35:16.522976 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:35:16.522877) EVENT_LOG_v1 {"time_micros": 1531287316522871, "job": 2890, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 74546327} >2018-07-11 05:35:16.522979 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:35:16.522884) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-4 74546327 bytes OK: base level 1 max bytes base 268435456 files[0 5 44 446 3642 776 0] max score 1.00 > >2018-07-11 05:35:16.548010 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:35:16.523830) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #11432 to level-5 74546840 bytes > >2018-07-11 05:35:16.548019 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:35:16.547920) EVENT_LOG_v1 {"time_micros": 1531287316547914, "job": 2891, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546840} >2018-07-11 05:35:16.548022 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:35:16.547928) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546840 bytes OK: base level 1 max bytes base 268435456 files[0 5 44 446 3641 777 0] max score 1.00 > >2018-07-11 05:35:52.502989 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:684] reusing log 36120 from recycle list > >2018-07-11 05:35:52.503111 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #36158. Immutable memtables: 0. > >2018-07-11 05:35:52.594361 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:35:52.503189) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1 >2018-07-11 05:35:52.594375 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 2892] Flushing memtable with next log file: 36158 > >2018-07-11 05:35:52.594396 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287352594388, "job": 2892, "event": "flush_started", "num_memtables": 1, "num_entries": 5000015, "num_deletes": 0, "memory_usage": 292945784} >2018-07-11 05:35:52.594400 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 2892] Level-0 flush table #36159: started >2018-07-11 05:35:56.480028 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287356480009, "cf_name": "default", "job": 2892, "event": "table_file_creation", "file_number": 36159, "file_size": 141286889, "table_properties": {"data_size": 127197777, "index_size": 1308397, "filter_size": 12779725, "raw_key_size": 144445101, "raw_average_key_size": 28, "raw_value_size": 59450671, "raw_average_value_size": 11, "num_data_blocks": 31252, "num_entries": 5000014, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:35:56.480046 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 2892] Level-0 flush table #36159: 141286889 bytes OK >2018-07-11 05:35:56.497060 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:242] adding log 36136 to recycle list > >2018-07-11 05:35:56.497071 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:35:56.480070) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #36159 started >2018-07-11 05:35:56.497074 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:35:56.496862) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #36159: memtable #1 done >2018-07-11 05:35:56.497076 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:35:56.496879) EVENT_LOG_v1 {"time_micros": 1531287356496872, "job": 2892, "event": "flush_finished", "lsm_state": [1, 5, 44, 446, 3641, 777, 0], "immutable_memtables": 0} >2018-07-11 05:35:56.497078 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:35:56.497009) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 5 44 446 3641 777 0] max score 1.00 > >2018-07-11 05:35:56.497975 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 2892] Try to delete WAL files size 178896755, prev total WAL file size 178896767, number of live WAL files 2. > >2018-07-11 05:37:45.891419 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:684] reusing log 36136 from recycle list > >2018-07-11 05:37:45.891538 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #36160. Immutable memtables: 0. > >2018-07-11 05:37:46.321671 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:37:45.891608) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1 >2018-07-11 05:37:46.321684 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 2893] Flushing memtable with next log file: 36160 > >2018-07-11 05:37:46.321701 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287466321692, "job": 2893, "event": "flush_started", "num_memtables": 1, "num_entries": 5000015, "num_deletes": 0, "memory_usage": 292929040} >2018-07-11 05:37:46.321704 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 2893] Level-0 flush table #36161: started >2018-07-11 05:37:50.485827 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287470485806, "cf_name": "default", "job": 2893, "event": "table_file_creation", "file_number": 36161, "file_size": 141286889, "table_properties": {"data_size": 127197777, "index_size": 1308397, "filter_size": 12779725, "raw_key_size": 144445101, "raw_average_key_size": 28, "raw_value_size": 59450671, "raw_average_value_size": 11, "num_data_blocks": 31252, "num_entries": 5000014, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:37:50.485850 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 2893] Level-0 flush table #36161: 141286889 bytes OK >2018-07-11 05:37:50.527214 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:242] adding log 36158 to recycle list > >2018-07-11 05:37:50.527226 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:37:50.485863) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #36161 started >2018-07-11 05:37:50.527229 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:37:50.526971) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #36161: memtable #1 done >2018-07-11 05:37:50.527231 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:37:50.526989) EVENT_LOG_v1 {"time_micros": 1531287470526982, "job": 2893, "event": "flush_finished", "lsm_state": [2, 5, 44, 446, 3641, 777, 0], "immutable_memtables": 0} >2018-07-11 05:37:50.527234 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:37:50.527160) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[2 5 44 446 3641 777 0] max score 1.05 > >2018-07-11 05:37:50.527632 7f41641eb700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS ------- >2018-07-11 05:37:50.527641 7f41641eb700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] >** DB Stats ** >Uptime(secs): 52212.2 total, 709.8 interval >Cumulative writes: 196K writes, 1937M keys, 196K commit groups, 1.0 writes per commit group, ingest: 65.06 GB, 1.28 MB/s >Cumulative WAL: 196K writes, 56K syncs, 3.48 writes per sync, written: 65.06 GB, 1.28 MB/s >Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent >Interval writes: 60 writes, 30M keys, 60 commit groups, 1.0 writes per commit group, ingest: 1023.66 MB, 1.44 MB/s >Interval WAL: 60 writes, 30 syncs, 1.94 writes per sync, written: 1.00 MB, 1.44 MB/s >Interval stall: 00:00:0.000 H:M:S, 0.0 percent > >** Compaction Stats [default] ** >Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop >---------------------------------------------------------------------------------------------------------------------------------------------------------- > L0 2/0 269.48 MB 1.1 0.0 0.0 0.0 51.0 51.0 0.0 1.0 0.0 32.4 1610 389 4.138 0 0 > L1 5/0 231.62 MB 0.9 83.6 50.8 32.8 83.6 50.8 0.0 1.6 25.7 25.7 3330 191 17.437 2647M 894K > L2 44/0 2.43 GB 1.0 99.2 50.8 48.4 99.1 50.7 0.0 2.0 21.5 21.5 4713 564 8.357 3704M 1358K > L3 446/0 24.93 GB 1.0 88.1 50.1 38.0 88.0 50.0 0.7 1.8 20.8 20.8 4332 301 14.392 3328M 4000K > L4 3641/0 250.00 GB 1.0 77.4 18.2 59.2 77.1 17.9 32.5 4.2 21.2 21.1 3736 245 15.248 2945M 15M > L5 777/0 53.92 GB 0.0 2.4 1.1 1.2 2.3 1.1 49.2 2.1 19.7 19.5 123 15 8.195 89M 1097K > Sum 4915/0 331.78 GB 0.0 350.7 171.0 179.7 401.2 221.5 82.4 7.9 20.1 23.0 17844 1705 10.466 12G 23M > Int 0/0 0.00 KB 0.0 7.0 2.6 4.4 7.8 3.4 1.2 9.9 21.1 23.5 340 22 15.437 259M 22 >Uptime(secs): 52212.2 total, 52212.2 interval >Flush(GB): cumulative 50.971, interval 0.790 >AddFile(GB): cumulative 0.000, interval 0.000 >AddFile(Total Files): cumulative 0, interval 0 >AddFile(L0 Files): cumulative 0, interval 0 >AddFile(Keys): cumulative 0, interval 0 >Cumulative compaction: 401.17 GB write, 7.87 MB/s write, 350.73 GB read, 6.88 MB/s read, 17844.3 seconds >Interval compaction: 7.80 GB write, 0.15 MB/s write, 7.01 GB read, 0.14 MB/s read, 339.6 seconds >Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count > >** File Read Latency Histogram By Level [default] ** > >** Compaction Stats [default] ** >Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop >---------------------------------------------------------------------------------------------------------------------------------------------------------- > L0 2/0 269.48 MB 1.1 0.0 0.0 0.0 51.0 51.0 0.0 1.0 0.0 32.4 1610 389 4.138 0 0 > L1 5/0 231.62 MB 0.9 83.6 50.8 32.8 83.6 50.8 0.0 1.6 25.7 25.7 3330 191 17.437 2647M 894K > L2 44/0 2.43 GB 1.0 99.2 50.8 48.4 99.1 50.7 0.0 2.0 21.5 21.5 4713 564 8.357 3704M 1358K > L3 446/0 24.93 GB 1.0 88.1 50.1 38.0 88.0 50.0 0.7 1.8 20.8 20.8 4332 301 14.392 3328M 4000K > L4 3641/0 250.00 GB 1.0 77.4 18.2 59.2 77.1 17.9 32.5 4.2 21.2 21.1 3736 245 15.248 2945M 15M > L5 777/0 53.92 GB 0.0 2.4 1.1 1.2 2.3 1.1 49.2 2.1 19.7 19.5 123 15 8.195 89M 1097K > Sum 4915/0 331.78 GB 0.0 350.7 171.0 179.7 401.2 221.5 82.4 7.9 20.1 23.0 17844 1705 10.466 12G 23M > Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 >Uptime(secs): 52212.2 total, 52212.2 interval >Flush(GB): cumulative 50.971, interval 0.000 >AddFile(GB): cumulative 0.000, interval 0.000 >AddFile(Total Files): cumulative 0, interval 0 >AddFile(L0 Files): cumulative 0, interval 0 >AddFile(Keys): cumulative 0, interval 0 >Cumulative compaction: 401.17 GB write, 7.87 MB/s write, 350.73 GB read, 6.88 MB/s read, 17844.3 seconds >Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds >Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count > >** File Read Latency Histogram By Level [default] ** > >2018-07-11 05:37:50.527764 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2894] Compacting 2@0 + 3@1 files to L1, score 1.05 >2018-07-11 05:37:50.527776 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2893 Base level 0, inputs: [36161(134MB) 36159(134MB)], [36138(71MB) 36143(47MB) 36144(59MB)] > >2018-07-11 05:37:50.527792 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287470527780, "job": 2894, "event": "compaction_started", "files_L0": [36161, 36159], "files_L1": [36138, 36143, 36144], "score": 1.05267, "input_data_size": 468639482} >2018-07-11 05:37:50.528114 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 2893] Try to delete WAL files size 178896755, prev total WAL file size 178896767, number of live WAL files 2. > >2018-07-11 05:37:54.098943 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2894] Generated table #36162: 2622976 keys, 74507320 bytes >2018-07-11 05:37:54.098992 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287474098964, "cf_name": "default", "job": 2894, "event": "table_file_creation", "file_number": 36162, "file_size": 74507320, "table_properties": {"data_size": 67109768, "index_size": 691533, "filter_size": 6705029, "raw_key_size": 75809436, "raw_average_key_size": 28, "raw_value_size": 31561806, "raw_average_value_size": 12, "num_data_blocks": 16475, "num_entries": 2622976, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:37:57.679706 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2894] Generated table #36163: 2638241 keys, 74546187 bytes >2018-07-11 05:37:57.679756 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287477679740, "cf_name": "default", "job": 2894, "event": "table_file_creation", "file_number": 36163, "file_size": 74546187, "table_properties": {"data_size": 67112147, "index_size": 689867, "filter_size": 6743183, "raw_key_size": 76215858, "raw_average_key_size": 28, "raw_value_size": 31365761, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:38:01.825675 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2894] Generated table #36164: 2638242 keys, 74546625 bytes >2018-07-11 05:38:01.825706 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287481825692, "cf_name": "default", "job": 2894, "event": "table_file_creation", "file_number": 36164, "file_size": 74546625, "table_properties": {"data_size": 67112168, "index_size": 690281, "filter_size": 6743186, "raw_key_size": 76215888, "raw_average_key_size": 28, "raw_value_size": 31365774, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638242, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:38:05.153922 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2894] Generated table #36165: 2638241 keys, 74546419 bytes >2018-07-11 05:38:05.153967 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287485153939, "cf_name": "default", "job": 2894, "event": "table_file_creation", "file_number": 36165, "file_size": 74546419, "table_properties": {"data_size": 67112142, "index_size": 690104, "filter_size": 6743183, "raw_key_size": 76215851, "raw_average_key_size": 28, "raw_value_size": 31365754, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:38:08.144474 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2894] Generated table #36166: 2638241 keys, 74546438 bytes >2018-07-11 05:38:08.144510 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287488144494, "cf_name": "default", "job": 2894, "event": "table_file_creation", "file_number": 36166, "file_size": 74546438, "table_properties": {"data_size": 67112091, "index_size": 690174, "filter_size": 6743183, "raw_key_size": 76215849, "raw_average_key_size": 28, "raw_value_size": 31365752, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:38:09.290851 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2894] Generated table #36167: 983439 keys, 29960976 bytes >2018-07-11 05:38:09.290881 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287489290867, "cf_name": "default", "job": 2894, "event": "table_file_creation", "file_number": 36167, "file_size": 29960976, "table_properties": {"data_size": 27166351, "index_size": 275023, "filter_size": 2518614, "raw_key_size": 28687168, "raw_average_key_size": 29, "raw_value_size": 13524796, "raw_average_value_size": 13, "num_data_blocks": 6678, "num_entries": 983439, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:38:10.814615 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2894] Generated table #36168: 162354 keys, 65979007 bytes >2018-07-11 05:38:10.814659 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287490814644, "cf_name": "default", "job": 2894, "event": "table_file_creation", "file_number": 36168, "file_size": 65979007, "table_properties": {"data_size": 64877987, "index_size": 545753, "filter_size": 554279, "raw_key_size": 14990262, "raw_average_key_size": 92, "raw_value_size": 51245646, "raw_average_value_size": 315, "num_data_blocks": 14664, "num_entries": 162354, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "80898", "kMergeOperands": "0"}} >2018-07-11 05:38:10.814726 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2894] Compacted 2@0 + 3@1 files to L1 => 468632972 bytes >2018-07-11 05:38:10.835216 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:38:10.834982) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 9 44 446 3641 777 0] max score 1.96, MB/sec: 23.1 rd, 23.1 wr, level 1, files in(2, 3) out(7) MB in(269.5, 177.4) out(446.9), read-write-amplify(3.3) write-amplify(1.7) OK, records in: 14321742, records dropped: 8 > >2018-07-11 05:38:10.835228 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:38:10.835007) EVENT_LOG_v1 {"time_micros": 1531287490834996, "job": 2894, "event": "compaction_finished", "compaction_time_micros": 20286916, "output_level": 1, "num_output_files": 7, "total_output_size": 468632972, "num_input_records": 14321742, "num_output_records": 14321734, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 9, 44, 446, 3641, 777, 0]} >2018-07-11 05:38:10.836204 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287490836199, "job": 2894, "event": "table_file_deletion", "file_number": 36161} >2018-07-11 05:38:10.836229 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287490836225, "job": 2894, "event": "table_file_deletion", "file_number": 36159} >2018-07-11 05:38:10.836262 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287490836258, "job": 2894, "event": "table_file_deletion", "file_number": 36144} >2018-07-11 05:38:10.836286 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287490836282, "job": 2894, "event": "table_file_deletion", "file_number": 36143} >2018-07-11 05:38:10.836309 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287490836305, "job": 2894, "event": "table_file_deletion", "file_number": 36138} >2018-07-11 05:38:10.836531 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_picker.cc:461] [default] Expanding@1 1+1(74546625+28316373 bytes) to 3+1 (223639482+28316373bytes) > >2018-07-11 05:38:10.836636 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2895] Compacting 3@1 + 1@2 files to L2, score 1.96 >2018-07-11 05:38:10.836649 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2894 Base level 1, inputs: [36164(71MB) 36165(71MB) 36166(71MB)], [36148(27MB)] > >2018-07-11 05:38:10.836669 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287490836654, "job": 2895, "event": "compaction_started", "files_L1": [36164, 36165, 36166], "files_L2": [36148], "score": 1.95739, "input_data_size": 251955855} >2018-07-11 05:38:14.342431 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2895] Generated table #36169: 2638241 keys, 74546323 bytes >2018-07-11 05:38:14.342475 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287494342450, "cf_name": "default", "job": 2895, "event": "table_file_creation", "file_number": 36169, "file_size": 74546323, "table_properties": {"data_size": 67112152, "index_size": 689997, "filter_size": 6743184, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:38:17.392152 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2895] Generated table #36170: 2638241 keys, 74546139 bytes >2018-07-11 05:38:17.392197 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287497392180, "cf_name": "default", "job": 2895, "event": "table_file_creation", "file_number": 36170, "file_size": 74546139, "table_properties": {"data_size": 67112146, "index_size": 689820, "filter_size": 6743183, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:38:20.683733 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2895] Generated table #36171: 2638241 keys, 74546293 bytes >2018-07-11 05:38:20.683786 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287500683756, "cf_name": "default", "job": 2895, "event": "table_file_creation", "file_number": 36171, "file_size": 74546293, "table_properties": {"data_size": 67112092, "index_size": 690028, "filter_size": 6743183, "raw_key_size": 76215851, "raw_average_key_size": 28, "raw_value_size": 31365754, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:38:21.899624 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2895] Generated table #36172: 997669 keys, 28316672 bytes >2018-07-11 05:38:21.899668 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287501899648, "cf_name": "default", "job": 2895, "event": "table_file_creation", "file_number": 36172, "file_size": 28316672, "table_properties": {"data_size": 25503458, "index_size": 261956, "filter_size": 2550270, "raw_key_size": 28837295, "raw_average_key_size": 28, "raw_value_size": 11966544, "raw_average_value_size": 11, "num_data_blocks": 6268, "num_entries": 997669, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "11", "kMergeOperands": "0"}} >2018-07-11 05:38:21.899710 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2895] Compacted 3@1 + 1@2 files to L2 => 251955427 bytes >2018-07-11 05:38:21.910017 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:38:21.909807) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 6 47 446 3641 777 0] max score 1.12, MB/sec: 22.8 rd, 22.8 wr, level 2, files in(3, 1) out(4) MB in(213.3, 27.0) out(240.3), read-write-amplify(2.3) write-amplify(1.1) OK, records in: 8912392, records dropped: 0 > >2018-07-11 05:38:21.910029 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:38:21.909831) EVENT_LOG_v1 {"time_micros": 1531287501909821, "job": 2895, "event": "compaction_finished", "compaction_time_micros": 11063019, "output_level": 2, "num_output_files": 4, "total_output_size": 251955427, "num_input_records": 8912392, "num_output_records": 8912392, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 6, 47, 446, 3641, 777, 0]} >2018-07-11 05:38:21.910993 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287501910987, "job": 2895, "event": "table_file_deletion", "file_number": 36166} >2018-07-11 05:38:21.911018 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287501911014, "job": 2895, "event": "table_file_deletion", "file_number": 36165} >2018-07-11 05:38:21.911041 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287501911037, "job": 2895, "event": "table_file_deletion", "file_number": 36164} >2018-07-11 05:38:21.911069 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287501911065, "job": 2895, "event": "table_file_deletion", "file_number": 36148} >2018-07-11 05:38:21.911390 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2896] Compacting 1@1 + 5@2 files to L2, score 1.12 >2018-07-11 05:38:21.911405 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2895 Base level 1, inputs: [36163(71MB)], [36149(71MB) 36150(71MB) 36151(71MB) 36152(71MB) 36169(71MB)] > >2018-07-11 05:38:21.911425 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287501911410, "job": 2896, "event": "compaction_started", "files_L1": [36163], "files_L2": [36149, 36150, 36151, 36152, 36169], "score": 1.12427, "input_data_size": 447276949} >2018-07-11 05:38:25.279752 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2896] Generated table #36173: 2638240 keys, 74545919 bytes >2018-07-11 05:38:25.279793 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287505279776, "cf_name": "default", "job": 2896, "event": "table_file_creation", "file_number": 36173, "file_size": 74545919, "table_properties": {"data_size": 67112140, "index_size": 689608, "filter_size": 6743181, "raw_key_size": 76215813, "raw_average_key_size": 28, "raw_value_size": 31365733, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:38:28.430728 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2896] Generated table #36174: 2638239 keys, 74546030 bytes >2018-07-11 05:38:28.430773 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287508430746, "cf_name": "default", "job": 2896, "event": "table_file_creation", "file_number": 36174, "file_size": 74546030, "table_properties": {"data_size": 67112086, "index_size": 689775, "filter_size": 6743179, "raw_key_size": 76215791, "raw_average_key_size": 28, "raw_value_size": 31365728, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638239, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:38:31.808590 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2896] Generated table #36175: 2638240 keys, 74546144 bytes >2018-07-11 05:38:31.808624 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287511808609, "cf_name": "default", "job": 2896, "event": "table_file_creation", "file_number": 36175, "file_size": 74546144, "table_properties": {"data_size": 67112138, "index_size": 689835, "filter_size": 6743181, "raw_key_size": 76215826, "raw_average_key_size": 28, "raw_value_size": 31365746, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:38:34.904832 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2896] Generated table #36176: 2638240 keys, 74546306 bytes >2018-07-11 05:38:34.904877 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287514904858, "cf_name": "default", "job": 2896, "event": "table_file_creation", "file_number": 36176, "file_size": 74546306, "table_properties": {"data_size": 67112167, "index_size": 689968, "filter_size": 6743181, "raw_key_size": 76215830, "raw_average_key_size": 28, "raw_value_size": 31365750, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:38:36.699304 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:684] reusing log 36158 from recycle list > >2018-07-11 05:38:36.699470 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #36178. Immutable memtables: 0. > >2018-07-11 05:38:36.856550 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:38:36.699507) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 0 >2018-07-11 05:38:36.856565 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 2897] Flushing memtable with next log file: 36178 > >2018-07-11 05:38:36.856593 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287516856581, "job": 2897, "event": "flush_started", "num_memtables": 1, "num_entries": 5000015, "num_deletes": 0, "memory_usage": 292946736} >2018-07-11 05:38:36.856598 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 2897] Level-0 flush table #36179: started >2018-07-11 05:38:38.535869 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2896] Generated table #36177: 2638242 keys, 74546389 bytes >2018-07-11 05:38:38.535912 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287518535895, "cf_name": "default", "job": 2896, "event": "table_file_creation", "file_number": 36177, "file_size": 74546389, "table_properties": {"data_size": 67112179, "index_size": 690034, "filter_size": 6743186, "raw_key_size": 76215891, "raw_average_key_size": 28, "raw_value_size": 31365777, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638242, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:38:39.645659 7f41649ec700 1 bluefs _allocate failed to allocate 0x8700000 on bdev 1, free 0x7fe000; fallback to bdev 2 >2018-07-11 05:38:41.795438 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287521795417, "cf_name": "default", "job": 2897, "event": "table_file_creation", "file_number": 36179, "file_size": 141286890, "table_properties": {"data_size": 127197778, "index_size": 1308397, "filter_size": 12779725, "raw_key_size": 144445101, "raw_average_key_size": 28, "raw_value_size": 59450671, "raw_average_value_size": 11, "num_data_blocks": 31252, "num_entries": 5000014, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:38:41.795473 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 2897] Level-0 flush table #36179: 141286890 bytes OK >2018-07-11 05:38:41.809585 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:242] adding log 36160 to recycle list > >2018-07-11 05:38:41.809596 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:38:41.795485) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #36179 started >2018-07-11 05:38:41.809600 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:38:41.809454) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #36179: memtable #1 done >2018-07-11 05:38:41.809602 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:38:41.809470) EVENT_LOG_v1 {"time_micros": 1531287521809463, "job": 2897, "event": "flush_finished", "lsm_state": [1, 6, 47, 446, 3641, 777, 0], "immutable_memtables": 0} >2018-07-11 05:38:41.809604 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:38:41.809490) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 6 47 446 3641 777 0] max score 1.00 > >2018-07-11 05:38:41.810738 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 2897] Try to delete WAL files size 178896755, prev total WAL file size 178896767, number of live WAL files 2. > >2018-07-11 05:38:43.463146 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2896] Generated table #36180: 2638240 keys, 74546044 bytes >2018-07-11 05:38:43.463179 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287523463164, "cf_name": "default", "job": 2896, "event": "table_file_creation", "file_number": 36180, "file_size": 74546044, "table_properties": {"data_size": 67112102, "index_size": 689782, "filter_size": 6743170, "raw_key_size": 76215821, "raw_average_key_size": 28, "raw_value_size": 31365741, "raw_average_value_size": 11, "num_data_blocks": 16489, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:38:43.463216 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2896] Compacted 1@1 + 5@2 files to L2 => 447276832 bytes >2018-07-11 05:38:43.476562 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:38:43.476161) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[1 5 48 446 3641 777 0] max score 1.08, MB/sec: 20.8 rd, 20.8 wr, level 2, files in(1, 5) out(6) MB in(71.1, 355.5) out(426.6), read-write-amplify(12.0) write-amplify(6.0) OK, records in: 15829441, records dropped: 0 > >2018-07-11 05:38:43.476572 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:38:43.476186) EVENT_LOG_v1 {"time_micros": 1531287523476175, "job": 2896, "event": "compaction_finished", "compaction_time_micros": 21551774, "output_level": 2, "num_output_files": 6, "total_output_size": 447276832, "num_input_records": 15829441, "num_output_records": 15829441, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 5, 48, 446, 3641, 777, 0]} >2018-07-11 05:38:43.477494 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287523477490, "job": 2896, "event": "table_file_deletion", "file_number": 36169} >2018-07-11 05:38:43.477513 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287523477511, "job": 2896, "event": "table_file_deletion", "file_number": 36163} >2018-07-11 05:38:43.477528 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287523477526, "job": 2896, "event": "table_file_deletion", "file_number": 36152} >2018-07-11 05:38:43.477540 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287523477537, "job": 2896, "event": "table_file_deletion", "file_number": 36151} >2018-07-11 05:38:43.477550 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287523477548, "job": 2896, "event": "table_file_deletion", "file_number": 36150} >2018-07-11 05:38:43.477563 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287523477561, "job": 2896, "event": "table_file_deletion", "file_number": 36149} >2018-07-11 05:38:43.477779 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_picker.cc:461] [default] Expanding@2 1+2(74546389+121508482 bytes) to 4+2 (298184865+121508482bytes) > >2018-07-11 05:38:43.477876 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2898] Compacting 4@2 + 2@3 files to L3, score 1.08 >2018-07-11 05:38:43.477889 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2897 Base level 2, inputs: [36177(71MB) 36180(71MB) 36170(71MB) 36171(71MB)], [36154(71MB) 36157(44MB)] > >2018-07-11 05:38:43.477902 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287523477892, "job": 2898, "event": "compaction_started", "files_L2": [36177, 36180, 36170, 36171], "files_L3": [36154, 36157], "score": 1.08448, "input_data_size": 419693347} >2018-07-11 05:38:46.715800 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2898] Generated table #36181: 2638240 keys, 74545987 bytes >2018-07-11 05:38:46.715846 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287526715820, "cf_name": "default", "job": 2898, "event": "table_file_creation", "file_number": 36181, "file_size": 74545987, "table_properties": {"data_size": 67112094, "index_size": 689722, "filter_size": 6743181, "raw_key_size": 76215821, "raw_average_key_size": 28, "raw_value_size": 31365741, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:38:50.316561 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2898] Generated table #36182: 2638241 keys, 74546682 bytes >2018-07-11 05:38:50.316614 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287530316598, "cf_name": "default", "job": 2898, "event": "table_file_creation", "file_number": 36182, "file_size": 74546682, "table_properties": {"data_size": 67112151, "index_size": 690358, "filter_size": 6743183, "raw_key_size": 76215849, "raw_average_key_size": 28, "raw_value_size": 31365752, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:38:53.469342 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2898] Generated table #36183: 2638242 keys, 74546144 bytes >2018-07-11 05:38:53.469381 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287533469364, "cf_name": "default", "job": 2898, "event": "table_file_creation", "file_number": 36183, "file_size": 74546144, "table_properties": {"data_size": 67112160, "index_size": 689808, "filter_size": 6743186, "raw_key_size": 76215880, "raw_average_key_size": 28, "raw_value_size": 31365766, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638242, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:38:56.742181 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2898] Generated table #36184: 2638241 keys, 74546056 bytes >2018-07-11 05:38:56.742229 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287536742208, "cf_name": "default", "job": 2898, "event": "table_file_creation", "file_number": 36184, "file_size": 74546056, "table_properties": {"data_size": 67112178, "index_size": 689705, "filter_size": 6743183, "raw_key_size": 76215864, "raw_average_key_size": 28, "raw_value_size": 31365767, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:38:59.771333 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2898] Generated table #36185: 2638241 keys, 74546992 bytes >2018-07-11 05:38:59.771384 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287539771367, "cf_name": "default", "job": 2898, "event": "table_file_creation", "file_number": 36185, "file_size": 74546992, "table_properties": {"data_size": 67112094, "index_size": 690725, "filter_size": 6743183, "raw_key_size": 76215847, "raw_average_key_size": 28, "raw_value_size": 31365750, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:39:01.729361 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2898] Generated table #36186: 1519836 keys, 46962779 bytes >2018-07-11 05:39:01.729392 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287541729379, "cf_name": "default", "job": 2898, "event": "table_file_creation", "file_number": 36186, "file_size": 46962779, "table_properties": {"data_size": 42634576, "index_size": 433278, "filter_size": 3893937, "raw_key_size": 44346621, "raw_average_key_size": 29, "raw_value_size": 21521510, "raw_average_value_size": 14, "num_data_blocks": 10537, "num_entries": 1519836, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:39:01.729428 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2898] Compacted 4@2 + 2@3 files to L3 => 419694640 bytes >2018-07-11 05:39:01.742706 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:39:01.742474) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[1 5 44 450 3641 777 0] max score 1.01, MB/sec: 23.0 rd, 23.0 wr, level 3, files in(4, 2) out(6) MB in(284.4, 115.9) out(400.3), read-write-amplify(2.8) write-amplify(1.4) OK, records in: 14711041, records dropped: 0 > >2018-07-11 05:39:01.742718 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:39:01.742499) EVENT_LOG_v1 {"time_micros": 1531287541742488, "job": 2898, "event": "compaction_finished", "compaction_time_micros": 18251508, "output_level": 3, "num_output_files": 6, "total_output_size": 419694640, "num_input_records": 14711041, "num_output_records": 14711041, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 5, 44, 450, 3641, 777, 0]} >2018-07-11 05:39:01.743681 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287541743675, "job": 2898, "event": "table_file_deletion", "file_number": 36180} >2018-07-11 05:39:01.743708 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287541743704, "job": 2898, "event": "table_file_deletion", "file_number": 36177} >2018-07-11 05:39:01.743735 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287541743731, "job": 2898, "event": "table_file_deletion", "file_number": 36171} >2018-07-11 05:39:01.743762 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287541743758, "job": 2898, "event": "table_file_deletion", "file_number": 36170} >2018-07-11 05:39:01.743789 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287541743785, "job": 2898, "event": "table_file_deletion", "file_number": 36157} >2018-07-11 05:39:01.743816 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287541743812, "job": 2898, "event": "table_file_deletion", "file_number": 36154} >2018-07-11 05:39:01.759226 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:39:01.744132) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #36185 to level-4 74546992 bytes > >2018-07-11 05:39:01.759236 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:39:01.759133) EVENT_LOG_v1 {"time_micros": 1531287541759127, "job": 2899, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 74546992} >2018-07-11 05:39:01.759243 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:39:01.759139) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-4 74546992 bytes OK: base level 1 max bytes base 268435456 files[1 5 44 449 3642 777 0] max score 1.01 > >2018-07-11 05:39:01.760136 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2900] Compacting 1@3 + 2@4 files to L4, score 1.01 >2018-07-11 05:39:01.760150 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2899 Base level 3, inputs: [36182(71MB)], [36155(71MB) 36156(71MB)] > >2018-07-11 05:39:01.760171 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287541760156, "job": 2900, "event": "compaction_started", "files_L3": [36182], "files_L4": [36155, 36156], "score": 1.00568, "input_data_size": 223639981} >2018-07-11 05:39:04.903995 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2900] Generated table #36187: 2638241 keys, 74545944 bytes >2018-07-11 05:39:04.904046 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287544904029, "cf_name": "default", "job": 2900, "event": "table_file_creation", "file_number": 36187, "file_size": 74545944, "table_properties": {"data_size": 67112175, "index_size": 689596, "filter_size": 6743183, "raw_key_size": 76215857, "raw_average_key_size": 28, "raw_value_size": 31365760, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:39:08.264341 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2900] Generated table #36188: 2638241 keys, 74546793 bytes >2018-07-11 05:39:08.264390 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287548264374, "cf_name": "default", "job": 2900, "event": "table_file_creation", "file_number": 36188, "file_size": 74546793, "table_properties": {"data_size": 67112132, "index_size": 690488, "filter_size": 6743183, "raw_key_size": 76215855, "raw_average_key_size": 28, "raw_value_size": 31365758, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:39:11.421584 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2900] Generated table #36189: 2638241 keys, 74546672 bytes >2018-07-11 05:39:11.421633 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287551421618, "cf_name": "default", "job": 2900, "event": "table_file_creation", "file_number": 36189, "file_size": 74546672, "table_properties": {"data_size": 67112141, "index_size": 690358, "filter_size": 6743183, "raw_key_size": 76215849, "raw_average_key_size": 28, "raw_value_size": 31365752, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:39:11.421669 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2900] Compacted 1@3 + 2@4 files to L4 => 223639409 bytes >2018-07-11 05:39:11.434210 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:39:11.434008) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[1 5 44 448 3643 777 0] max score 1.00, MB/sec: 23.1 rd, 23.1 wr, level 4, files in(1, 2) out(3) MB in(71.1, 142.2) out(213.3), read-write-amplify(6.0) write-amplify(3.0) OK, records in: 7914723, records dropped: 0 > >2018-07-11 05:39:11.434222 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:39:11.434033) EVENT_LOG_v1 {"time_micros": 1531287551434022, "job": 2900, "event": "compaction_finished", "compaction_time_micros": 9661478, "output_level": 4, "num_output_files": 3, "total_output_size": 223639409, "num_input_records": 7914723, "num_output_records": 7914723, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 5, 44, 448, 3643, 777, 0]} >2018-07-11 05:39:11.435186 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287551435169, "job": 2900, "event": "table_file_deletion", "file_number": 36182} >2018-07-11 05:39:11.435203 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287551435200, "job": 2900, "event": "table_file_deletion", "file_number": 36156} >2018-07-11 05:39:11.435214 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287551435212, "job": 2900, "event": "table_file_deletion", "file_number": 36155} >2018-07-11 05:39:11.450789 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:39:11.435537) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #36183 to level-4 74546144 bytes > >2018-07-11 05:39:11.450799 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:39:11.450699) EVENT_LOG_v1 {"time_micros": 1531287551450693, "job": 2901, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 74546144} >2018-07-11 05:39:11.450802 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:39:11.450707) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-4 74546144 bytes OK: base level 1 max bytes base 268435456 files[1 5 44 447 3644 777 0] max score 1.00 > >2018-07-11 05:39:11.484293 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:39:11.451698) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #36185 to level-5 74546992 bytes > >2018-07-11 05:39:11.484304 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:39:11.484076) EVENT_LOG_v1 {"time_micros": 1531287551484070, "job": 2902, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546992} >2018-07-11 05:39:11.484307 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:39:11.484084) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546992 bytes OK: base level 1 max bytes base 268435456 files[1 5 44 447 3643 778 0] max score 1.00 > >2018-07-11 05:39:11.500834 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:39:11.485146) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #9104 to level-5 74546840 bytes > >2018-07-11 05:39:11.500844 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:39:11.500742) EVENT_LOG_v1 {"time_micros": 1531287551500736, "job": 2903, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546840} >2018-07-11 05:39:11.500859 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:39:11.500750) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546840 bytes OK: base level 1 max bytes base 268435456 files[1 5 44 447 3642 779 0] max score 1.00 > >2018-07-11 05:39:11.517654 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:39:11.501751) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #28031 to level-5 74546839 bytes > >2018-07-11 05:39:11.517665 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:39:11.517449) EVENT_LOG_v1 {"time_micros": 1531287551517443, "job": 2904, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546839} >2018-07-11 05:39:11.517667 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:39:11.517457) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546839 bytes OK: base level 1 max bytes base 268435456 files[1 5 44 447 3641 780 0] max score 1.00 > >2018-07-11 05:39:11.534233 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:39:11.518560) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #36106 to level-4 74546122 bytes > >2018-07-11 05:39:11.534243 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:39:11.534142) EVENT_LOG_v1 {"time_micros": 1531287551534136, "job": 2905, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 74546122} >2018-07-11 05:39:11.534269 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:39:11.534149) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-4 74546122 bytes OK: base level 1 max bytes base 268435456 files[1 5 44 446 3642 780 0] max score 1.00 > >2018-07-11 05:39:11.550923 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:39:11.535096) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #20204 to level-5 74546839 bytes > >2018-07-11 05:39:11.550932 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:39:11.550832) EVENT_LOG_v1 {"time_micros": 1531287551550826, "job": 2906, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546839} >2018-07-11 05:39:11.550935 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:39:11.550840) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546839 bytes OK: base level 1 max bytes base 268435456 files[1 5 44 446 3641 781 0] max score 1.00 > >2018-07-11 05:40:53.574275 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:684] reusing log 36160 from recycle list > >2018-07-11 05:40:53.574512 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #36190. Immutable memtables: 0. > >2018-07-11 05:40:53.633194 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:40:53.574578) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1 >2018-07-11 05:40:53.633211 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 2907] Flushing memtable with next log file: 36190 > >2018-07-11 05:40:53.633227 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287653633219, "job": 2907, "event": "flush_started", "num_memtables": 1, "num_entries": 5000031, "num_deletes": 0, "memory_usage": 292999136} >2018-07-11 05:40:53.633231 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 2907] Level-0 flush table #36191: started >2018-07-11 05:40:56.304477 7f41649ec700 1 bluefs _allocate failed to allocate 0x8700000 on bdev 1, free 0x7fe000; fallback to bdev 2 >2018-07-11 05:40:58.224036 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287658224014, "cf_name": "default", "job": 2907, "event": "table_file_creation", "file_number": 36191, "file_size": 141350553, "table_properties": {"data_size": 127261035, "index_size": 1308619, "filter_size": 12779909, "raw_key_size": 144445833, "raw_average_key_size": 28, "raw_value_size": 59513272, "raw_average_value_size": 11, "num_data_blocks": 31257, "num_entries": 5000027, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:40:58.224058 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 2907] Level-0 flush table #36191: 141350553 bytes OK >2018-07-11 05:40:58.239759 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:242] adding log 36178 to recycle list > >2018-07-11 05:40:58.239771 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:40:58.224083) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #36191 started >2018-07-11 05:40:58.239775 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:40:58.239617) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #36191: memtable #1 done >2018-07-11 05:40:58.239777 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:40:58.239634) EVENT_LOG_v1 {"time_micros": 1531287658239627, "job": 2907, "event": "flush_finished", "lsm_state": [2, 5, 44, 446, 3641, 781, 0], "immutable_memtables": 0} >2018-07-11 05:40:58.239779 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:40:58.239704) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[2 5 44 446 3641 781 0] max score 1.05 > >2018-07-11 05:40:58.239911 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2908] Compacting 2@0 + 3@1 files to L1, score 1.05 >2018-07-11 05:40:58.239928 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2907 Base level 0, inputs: [36191(134MB) 36179(134MB)], [36162(71MB) 36167(28MB) 36168(62MB)] > >2018-07-11 05:40:58.239946 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287658239932, "job": 2908, "event": "compaction_started", "files_L0": [36191, 36179], "files_L1": [36162, 36167, 36168], "score": 1.05291, "input_data_size": 453084746} >2018-07-11 05:40:58.240672 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 2907] Try to delete WAL files size 178960785, prev total WAL file size 178960797, number of live WAL files 2. > >2018-07-11 05:41:00.555575 7f41641eb700 1 bluefs _allocate failed to allocate 0x4800000 on bdev 1, free 0x7fe000; fallback to bdev 2 >2018-07-11 05:41:01.497878 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2908] Generated table #36192: 2622954 keys, 74510245 bytes >2018-07-11 05:41:01.497923 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287661497896, "cf_name": "default", "job": 2908, "event": "table_file_creation", "file_number": 36192, "file_size": 74510245, "table_properties": {"data_size": 67112728, "index_size": 691548, "filter_size": 6704979, "raw_key_size": 75809109, "raw_average_key_size": 28, "raw_value_size": 31564951, "raw_average_value_size": 12, "num_data_blocks": 16476, "num_entries": 2622954, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:41:04.456200 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2908] Generated table #36193: 2638240 keys, 74546288 bytes >2018-07-11 05:41:04.456256 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287664456223, "cf_name": "default", "job": 2908, "event": "table_file_creation", "file_number": 36193, "file_size": 74546288, "table_properties": {"data_size": 67112171, "index_size": 689946, "filter_size": 6743181, "raw_key_size": 76215831, "raw_average_key_size": 28, "raw_value_size": 31365751, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:41:08.063935 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2908] Generated table #36194: 2638241 keys, 74545770 bytes >2018-07-11 05:41:08.063981 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287668063954, "cf_name": "default", "job": 2908, "event": "table_file_creation", "file_number": 36194, "file_size": 74545770, "table_properties": {"data_size": 67112109, "index_size": 689488, "filter_size": 6743183, "raw_key_size": 76215849, "raw_average_key_size": 28, "raw_value_size": 31365752, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:41:11.383814 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2908] Generated table #36195: 2638241 keys, 74546590 bytes >2018-07-11 05:41:11.383858 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287671383831, "cf_name": "default", "job": 2908, "event": "table_file_creation", "file_number": 36195, "file_size": 74546590, "table_properties": {"data_size": 67112162, "index_size": 690255, "filter_size": 6743183, "raw_key_size": 76215856, "raw_average_key_size": 28, "raw_value_size": 31365759, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:41:14.700996 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2908] Generated table #36196: 2638241 keys, 74546851 bytes >2018-07-11 05:41:14.701027 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287674701013, "cf_name": "default", "job": 2908, "event": "table_file_creation", "file_number": 36196, "file_size": 74546851, "table_properties": {"data_size": 67112128, "index_size": 690550, "filter_size": 6743183, "raw_key_size": 76215851, "raw_average_key_size": 28, "raw_value_size": 31365754, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:41:15.431798 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2908] Generated table #36197: 430516 keys, 14337866 bytes >2018-07-11 05:41:15.431840 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287675431813, "cf_name": "default", "job": 2908, "event": "table_file_creation", "file_number": 36197, "file_size": 14337866, "table_properties": {"data_size": 13100992, "index_size": 130501, "filter_size": 1105387, "raw_key_size": 12713835, "raw_average_key_size": 29, "raw_value_size": 6951154, "raw_average_value_size": 16, "num_data_blocks": 3223, "num_entries": 430516, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:41:16.541219 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2908] Generated table #36198: 162369 keys, 66044531 bytes >2018-07-11 05:41:16.541267 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287676541233, "cf_name": "default", "job": 2908, "event": "table_file_creation", "file_number": 36198, "file_size": 66044531, "table_properties": {"data_size": 64943200, "index_size": 545882, "filter_size": 554461, "raw_key_size": 14991277, "raw_average_key_size": 92, "raw_value_size": 51309878, "raw_average_value_size": 316, "num_data_blocks": 14668, "num_entries": 162369, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "80898", "kMergeOperands": "0"}} >2018-07-11 05:41:16.541330 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2908] Compacted 2@0 + 3@1 files to L1 => 453078141 bytes >2018-07-11 05:41:16.556316 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:41:16.556106) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 9 44 446 3641 781 0] max score 1.90, MB/sec: 24.8 rd, 24.8 wr, level 1, files in(2, 3) out(7) MB in(269.5, 162.6) out(432.1), read-write-amplify(3.2) write-amplify(1.6) OK, records in: 13768810, records dropped: 8 > >2018-07-11 05:41:16.556326 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:41:16.556128) EVENT_LOG_v1 {"time_micros": 1531287676556118, "job": 2908, "event": "compaction_finished", "compaction_time_micros": 18301367, "output_level": 1, "num_output_files": 7, "total_output_size": 453078141, "num_input_records": 13768810, "num_output_records": 13768802, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 9, 44, 446, 3641, 781, 0]} >2018-07-11 05:41:16.557217 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287676557213, "job": 2908, "event": "table_file_deletion", "file_number": 36191} >2018-07-11 05:41:16.557243 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287676557240, "job": 2908, "event": "table_file_deletion", "file_number": 36179} >2018-07-11 05:41:16.557261 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287676557259, "job": 2908, "event": "table_file_deletion", "file_number": 36168} >2018-07-11 05:41:16.557272 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287676557270, "job": 2908, "event": "table_file_deletion", "file_number": 36167} >2018-07-11 05:41:16.557283 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287676557281, "job": 2908, "event": "table_file_deletion", "file_number": 36162} >2018-07-11 05:41:16.557498 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_picker.cc:461] [default] Expanding@1 1+1(74546851+28316672 bytes) to 3+1 (223639211+28316672bytes) > >2018-07-11 05:41:16.557595 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2909] Compacting 3@1 + 1@2 files to L2, score 1.90 >2018-07-11 05:41:16.557606 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2908 Base level 1, inputs: [36194(71MB) 36195(71MB) 36196(71MB)], [36172(27MB)] > >2018-07-11 05:41:16.557619 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287676557609, "job": 2909, "event": "compaction_started", "files_L1": [36194, 36195, 36196], "files_L2": [36172], "score": 1.89945, "input_data_size": 251955883} >2018-07-11 05:41:19.974437 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2909] Generated table #36199: 2638241 keys, 74546959 bytes >2018-07-11 05:41:19.974484 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287679974469, "cf_name": "default", "job": 2909, "event": "table_file_creation", "file_number": 36199, "file_size": 74546959, "table_properties": {"data_size": 67112132, "index_size": 690654, "filter_size": 6743183, "raw_key_size": 76215853, "raw_average_key_size": 28, "raw_value_size": 31365756, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:41:23.139070 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2909] Generated table #36200: 2638241 keys, 74546244 bytes >2018-07-11 05:41:23.139104 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287683139089, "cf_name": "default", "job": 2909, "event": "table_file_creation", "file_number": 36200, "file_size": 74546244, "table_properties": {"data_size": 67112157, "index_size": 689914, "filter_size": 6743183, "raw_key_size": 76215852, "raw_average_key_size": 28, "raw_value_size": 31365755, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:41:26.271842 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2909] Generated table #36201: 2638241 keys, 74546438 bytes >2018-07-11 05:41:26.271888 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287686271861, "cf_name": "default", "job": 2909, "event": "table_file_creation", "file_number": 36201, "file_size": 74546438, "table_properties": {"data_size": 67112129, "index_size": 690136, "filter_size": 6743183, "raw_key_size": 76215855, "raw_average_key_size": 28, "raw_value_size": 31365758, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:41:27.555658 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2909] Generated table #36202: 997669 keys, 28316247 bytes >2018-07-11 05:41:27.555702 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287687555687, "cf_name": "default", "job": 2909, "event": "table_file_creation", "file_number": 36202, "file_size": 28316247, "table_properties": {"data_size": 25503445, "index_size": 261544, "filter_size": 2550270, "raw_key_size": 28837291, "raw_average_key_size": 28, "raw_value_size": 11966540, "raw_average_value_size": 11, "num_data_blocks": 6268, "num_entries": 997669, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "11", "kMergeOperands": "0"}} >2018-07-11 05:41:27.555732 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2909] Compacted 3@1 + 1@2 files to L2 => 251955888 bytes >2018-07-11 05:41:27.572818 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:41:27.572554) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 6 47 446 3641 781 0] max score 1.07, MB/sec: 22.9 rd, 22.9 wr, level 2, files in(3, 1) out(4) MB in(213.3, 27.0) out(240.3), read-write-amplify(2.3) write-amplify(1.1) OK, records in: 8912392, records dropped: 0 > >2018-07-11 05:41:27.572828 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:41:27.572578) EVENT_LOG_v1 {"time_micros": 1531287687572567, "job": 2909, "event": "compaction_finished", "compaction_time_micros": 10998098, "output_level": 2, "num_output_files": 4, "total_output_size": 251955888, "num_input_records": 8912392, "num_output_records": 8912392, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 6, 47, 446, 3641, 781, 0]} >2018-07-11 05:41:27.573804 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287687573800, "job": 2909, "event": "table_file_deletion", "file_number": 36196} >2018-07-11 05:41:27.573830 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287687573828, "job": 2909, "event": "table_file_deletion", "file_number": 36195} >2018-07-11 05:41:27.573841 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287687573839, "job": 2909, "event": "table_file_deletion", "file_number": 36194} >2018-07-11 05:41:27.573856 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287687573854, "job": 2909, "event": "table_file_deletion", "file_number": 36172} >2018-07-11 05:41:27.574170 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2910] Compacting 1@1 + 5@2 files to L2, score 1.07 >2018-07-11 05:41:27.574182 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2909 Base level 1, inputs: [36193(71MB)], [36173(71MB) 36174(71MB) 36175(71MB) 36176(71MB) 36199(71MB)] > >2018-07-11 05:41:27.574195 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287687574186, "job": 2910, "event": "compaction_started", "files_L1": [36193], "files_L2": [36173, 36174, 36175, 36176, 36199], "score": 1.06633, "input_data_size": 447277646} >2018-07-11 05:41:32.081384 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2910] Generated table #36203: 2638240 keys, 74545797 bytes >2018-07-11 05:41:32.081419 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287692081404, "cf_name": "default", "job": 2910, "event": "table_file_creation", "file_number": 36203, "file_size": 74545797, "table_properties": {"data_size": 67112118, "index_size": 689508, "filter_size": 6743181, "raw_key_size": 76215813, "raw_average_key_size": 28, "raw_value_size": 31365733, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:41:36.364042 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2910] Generated table #36204: 2638239 keys, 74546198 bytes >2018-07-11 05:41:36.364098 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287696364080, "cf_name": "default", "job": 2910, "event": "table_file_creation", "file_number": 36204, "file_size": 74546198, "table_properties": {"data_size": 67112128, "index_size": 689901, "filter_size": 6743179, "raw_key_size": 76215788, "raw_average_key_size": 28, "raw_value_size": 31365725, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638239, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:41:40.703516 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2910] Generated table #36205: 2638240 keys, 74546304 bytes >2018-07-11 05:41:40.703550 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287700703534, "cf_name": "default", "job": 2910, "event": "table_file_creation", "file_number": 36205, "file_size": 74546304, "table_properties": {"data_size": 67112139, "index_size": 689993, "filter_size": 6743182, "raw_key_size": 76215829, "raw_average_key_size": 28, "raw_value_size": 31365749, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:41:45.189666 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2910] Generated table #36206: 2638241 keys, 74545896 bytes >2018-07-11 05:41:45.189713 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287705189686, "cf_name": "default", "job": 2910, "event": "table_file_creation", "file_number": 36206, "file_size": 74545896, "table_properties": {"data_size": 67112162, "index_size": 689561, "filter_size": 6743183, "raw_key_size": 76215859, "raw_average_key_size": 28, "raw_value_size": 31365762, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:41:48.703466 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2910] Generated table #36207: 2638240 keys, 74547003 bytes >2018-07-11 05:41:48.703515 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287708703499, "cf_name": "default", "job": 2910, "event": "table_file_creation", "file_number": 36207, "file_size": 74547003, "table_properties": {"data_size": 67112102, "index_size": 690730, "filter_size": 6743181, "raw_key_size": 76215830, "raw_average_key_size": 28, "raw_value_size": 31365750, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:41:52.470385 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2910] Generated table #36208: 2638240 keys, 74546183 bytes >2018-07-11 05:41:52.470419 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287712470404, "cf_name": "default", "job": 2910, "event": "table_file_creation", "file_number": 36208, "file_size": 74546183, "table_properties": {"data_size": 67112110, "index_size": 689913, "filter_size": 6743170, "raw_key_size": 76215825, "raw_average_key_size": 28, "raw_value_size": 31365745, "raw_average_value_size": 11, "num_data_blocks": 16489, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:41:52.470454 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2910] Compacted 1@1 + 5@2 files to L2 => 447277381 bytes >2018-07-11 05:41:52.480747 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:41:52.480505) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 5 48 446 3641 781 0] max score 1.08, MB/sec: 18.0 rd, 18.0 wr, level 2, files in(1, 5) out(6) MB in(71.1, 355.5) out(426.6), read-write-amplify(12.0) write-amplify(6.0) OK, records in: 15829440, records dropped: 0 > >2018-07-11 05:41:52.480758 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:41:52.480530) EVENT_LOG_v1 {"time_micros": 1531287712480519, "job": 2910, "event": "compaction_finished", "compaction_time_micros": 24896244, "output_level": 2, "num_output_files": 6, "total_output_size": 447277381, "num_input_records": 15829440, "num_output_records": 15829440, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 5, 48, 446, 3641, 781, 0]} >2018-07-11 05:41:52.481747 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287712481742, "job": 2910, "event": "table_file_deletion", "file_number": 36199} >2018-07-11 05:41:52.481775 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287712481773, "job": 2910, "event": "table_file_deletion", "file_number": 36193} >2018-07-11 05:41:52.481788 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287712481786, "job": 2910, "event": "table_file_deletion", "file_number": 36176} >2018-07-11 05:41:52.481800 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287712481797, "job": 2910, "event": "table_file_deletion", "file_number": 36175} >2018-07-11 05:41:52.481810 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287712481808, "job": 2910, "event": "table_file_deletion", "file_number": 36174} >2018-07-11 05:41:52.481821 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287712481819, "job": 2910, "event": "table_file_deletion", "file_number": 36173} >2018-07-11 05:41:52.482043 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_picker.cc:461] [default] Expanding@2 1+3(74547003+196054822 bytes) to 4+3 (298185868+196054822bytes) > >2018-07-11 05:41:52.482141 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2911] Compacting 4@2 + 3@3 files to L3, score 1.08 >2018-07-11 05:41:52.482153 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2910 Base level 2, inputs: [36207(71MB) 36208(71MB) 36200(71MB) 36201(71MB)], [36181(71MB) 36184(71MB) 36186(44MB)] > >2018-07-11 05:41:52.482167 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287712482156, "job": 2911, "event": "compaction_started", "files_L2": [36207, 36208, 36200, 36201], "files_L3": [36181, 36184, 36186], "score": 1.08448, "input_data_size": 494240690} >2018-07-11 05:41:56.547465 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2911] Generated table #36209: 2638241 keys, 74546129 bytes >2018-07-11 05:41:56.547511 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287716547485, "cf_name": "default", "job": 2911, "event": "table_file_creation", "file_number": 36209, "file_size": 74546129, "table_properties": {"data_size": 67112135, "index_size": 689821, "filter_size": 6743183, "raw_key_size": 76215848, "raw_average_key_size": 28, "raw_value_size": 31365751, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:42:00.893007 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2911] Generated table #36210: 2638240 keys, 74547064 bytes >2018-07-11 05:42:00.893051 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287720893034, "cf_name": "default", "job": 2911, "event": "table_file_creation", "file_number": 36210, "file_size": 74547064, "table_properties": {"data_size": 67112165, "index_size": 690728, "filter_size": 6743181, "raw_key_size": 76215834, "raw_average_key_size": 28, "raw_value_size": 31365754, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:42:05.021265 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2911] Generated table #36211: 2638240 keys, 74546049 bytes >2018-07-11 05:42:05.021313 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287725021296, "cf_name": "default", "job": 2911, "event": "table_file_creation", "file_number": 36211, "file_size": 74546049, "table_properties": {"data_size": 67112109, "index_size": 689769, "filter_size": 6743181, "raw_key_size": 76215826, "raw_average_key_size": 28, "raw_value_size": 31365746, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:42:09.390225 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2911] Generated table #36212: 2638241 keys, 74546333 bytes >2018-07-11 05:42:09.390273 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287729390258, "cf_name": "default", "job": 2911, "event": "table_file_creation", "file_number": 36212, "file_size": 74546333, "table_properties": {"data_size": 67112161, "index_size": 689999, "filter_size": 6743183, "raw_key_size": 76215856, "raw_average_key_size": 28, "raw_value_size": 31365759, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:42:13.788447 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2911] Generated table #36213: 2638241 keys, 74545742 bytes >2018-07-11 05:42:13.788483 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287733788468, "cf_name": "default", "job": 2911, "event": "table_file_creation", "file_number": 36213, "file_size": 74545742, "table_properties": {"data_size": 67112143, "index_size": 689426, "filter_size": 6743183, "raw_key_size": 76215856, "raw_average_key_size": 28, "raw_value_size": 31365759, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:42:17.819205 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2911] Generated table #36214: 2638241 keys, 74546848 bytes >2018-07-11 05:42:17.819263 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287737819242, "cf_name": "default", "job": 2911, "event": "table_file_creation", "file_number": 36214, "file_size": 74546848, "table_properties": {"data_size": 67112135, "index_size": 690540, "filter_size": 6743183, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:42:20.456777 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2911] Generated table #36215: 1519835 keys, 46962830 bytes >2018-07-11 05:42:20.456819 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287740456793, "cf_name": "default", "job": 2911, "event": "table_file_creation", "file_number": 36215, "file_size": 46962830, "table_properties": {"data_size": 42634559, "index_size": 433349, "filter_size": 3893934, "raw_key_size": 44346594, "raw_average_key_size": 29, "raw_value_size": 21521500, "raw_average_value_size": 14, "num_data_blocks": 10537, "num_entries": 1519835, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:42:20.456853 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2911] Compacted 4@2 + 3@3 files to L3 => 494240995 bytes >2018-07-11 05:42:20.930200 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:42:20.929936) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 5 44 450 3641 781 0] max score 1.01, MB/sec: 17.7 rd, 17.7 wr, level 3, files in(4, 3) out(7) MB in(284.4, 187.0) out(471.3), read-write-amplify(3.3) write-amplify(1.7) OK, records in: 17349279, records dropped: 0 > >2018-07-11 05:42:20.930220 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:42:20.929959) EVENT_LOG_v1 {"time_micros": 1531287740929949, "job": 2911, "event": "compaction_finished", "compaction_time_micros": 27974669, "output_level": 3, "num_output_files": 7, "total_output_size": 494240995, "num_input_records": 17349279, "num_output_records": 17349279, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 5, 44, 450, 3641, 781, 0]} >2018-07-11 05:42:20.931190 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287740931173, "job": 2911, "event": "table_file_deletion", "file_number": 36208} >2018-07-11 05:42:20.931205 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287740931203, "job": 2911, "event": "table_file_deletion", "file_number": 36207} >2018-07-11 05:42:20.931217 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287740931215, "job": 2911, "event": "table_file_deletion", "file_number": 36201} >2018-07-11 05:42:20.931229 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287740931227, "job": 2911, "event": "table_file_deletion", "file_number": 36200} >2018-07-11 05:42:20.931244 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287740931242, "job": 2911, "event": "table_file_deletion", "file_number": 36186} >2018-07-11 05:42:20.931262 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287740931260, "job": 2911, "event": "table_file_deletion", "file_number": 36184} >2018-07-11 05:42:20.931274 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287740931272, "job": 2911, "event": "table_file_deletion", "file_number": 36181} >2018-07-11 05:42:20.931597 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2912] Compacting 1@3 + 4@4 files to L4, score 1.01 >2018-07-11 05:42:20.931610 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2911 Base level 3, inputs: [36210(71MB)], [36187(71MB) 36188(71MB) 36189(71MB) 36183(71MB)] > >2018-07-11 05:42:20.931623 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287740931613, "job": 2912, "event": "compaction_started", "files_L3": [36210], "files_L4": [36187, 36188, 36189, 36183], "score": 1.00846, "input_data_size": 372732617} >2018-07-11 05:42:25.248502 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2912] Generated table #36216: 2638241 keys, 74546723 bytes >2018-07-11 05:42:25.248536 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287745248521, "cf_name": "default", "job": 2912, "event": "table_file_creation", "file_number": 36216, "file_size": 74546723, "table_properties": {"data_size": 67112160, "index_size": 690390, "filter_size": 6743183, "raw_key_size": 76215856, "raw_average_key_size": 28, "raw_value_size": 31365759, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:42:29.809547 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2912] Generated table #36217: 2638241 keys, 74546309 bytes >2018-07-11 05:42:29.809606 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287749809574, "cf_name": "default", "job": 2912, "event": "table_file_creation", "file_number": 36217, "file_size": 74546309, "table_properties": {"data_size": 67112115, "index_size": 690021, "filter_size": 6743183, "raw_key_size": 76215855, "raw_average_key_size": 28, "raw_value_size": 31365758, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:42:34.282870 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2912] Generated table #36218: 2638241 keys, 74546266 bytes >2018-07-11 05:42:34.282913 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287754282887, "cf_name": "default", "job": 2912, "event": "table_file_creation", "file_number": 36218, "file_size": 74546266, "table_properties": {"data_size": 67112149, "index_size": 689944, "filter_size": 6743183, "raw_key_size": 76215852, "raw_average_key_size": 28, "raw_value_size": 31365755, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:42:38.845007 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2912] Generated table #36219: 2638241 keys, 74546125 bytes >2018-07-11 05:42:38.845062 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287758845044, "cf_name": "default", "job": 2912, "event": "table_file_creation", "file_number": 36219, "file_size": 74546125, "table_properties": {"data_size": 67112090, "index_size": 689862, "filter_size": 6743183, "raw_key_size": 76215850, "raw_average_key_size": 28, "raw_value_size": 31365753, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:42:42.850417 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2912] Generated table #36220: 2638241 keys, 74547076 bytes >2018-07-11 05:42:42.850451 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287762850435, "cf_name": "default", "job": 2912, "event": "table_file_creation", "file_number": 36220, "file_size": 74547076, "table_properties": {"data_size": 67112175, "index_size": 690728, "filter_size": 6743183, "raw_key_size": 76215862, "raw_average_key_size": 28, "raw_value_size": 31365765, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:42:42.850486 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2912] Compacted 1@3 + 4@4 files to L4 => 372732499 bytes >2018-07-11 05:42:42.863040 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:42:42.862791) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 5 44 449 3642 781 0] max score 1.01, MB/sec: 17.0 rd, 17.0 wr, level 4, files in(1, 4) out(5) MB in(71.1, 284.4) out(355.5), read-write-amplify(10.0) write-amplify(5.0) OK, records in: 13191205, records dropped: 0 > >2018-07-11 05:42:42.863052 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:42:42.862817) EVENT_LOG_v1 {"time_micros": 1531287762862806, "job": 2912, "event": "compaction_finished", "compaction_time_micros": 21918848, "output_level": 4, "num_output_files": 5, "total_output_size": 372732499, "num_input_records": 13191205, "num_output_records": 13191205, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 5, 44, 449, 3642, 781, 0]} >2018-07-11 05:42:42.864043 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287762864039, "job": 2912, "event": "table_file_deletion", "file_number": 36210} >2018-07-11 05:42:42.864063 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287762864061, "job": 2912, "event": "table_file_deletion", "file_number": 36189} >2018-07-11 05:42:42.864076 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287762864074, "job": 2912, "event": "table_file_deletion", "file_number": 36188} >2018-07-11 05:42:42.864089 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287762864087, "job": 2912, "event": "table_file_deletion", "file_number": 36187} >2018-07-11 05:42:42.864104 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287762864101, "job": 2912, "event": "table_file_deletion", "file_number": 36183} >2018-07-11 05:42:42.879556 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:42:42.864448) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #36214 to level-4 74546848 bytes > >2018-07-11 05:42:42.879567 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:42:42.879464) EVENT_LOG_v1 {"time_micros": 1531287762879458, "job": 2913, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 74546848} >2018-07-11 05:42:42.879569 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:42:42.879472) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-4 74546848 bytes OK: base level 1 max bytes base 268435456 files[0 5 44 448 3643 781 0] max score 1.00 > >2018-07-11 05:42:42.896299 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:42:42.880412) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #36212 to level-4 74546333 bytes > >2018-07-11 05:42:42.896310 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:42:42.896186) EVENT_LOG_v1 {"time_micros": 1531287762896180, "job": 2914, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 74546333} >2018-07-11 05:42:42.896313 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:42:42.896194) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-4 74546333 bytes OK: base level 1 max bytes base 268435456 files[0 5 44 447 3644 781 0] max score 1.00 > >2018-07-11 05:42:42.912911 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:42:42.897143) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #36220 to level-5 74547076 bytes > >2018-07-11 05:42:42.912921 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:42:42.912822) EVENT_LOG_v1 {"time_micros": 1531287762912816, "job": 2915, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74547076} >2018-07-11 05:42:42.912937 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:42:42.912829) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74547076 bytes OK: base level 1 max bytes base 268435456 files[0 5 44 447 3643 782 0] max score 1.00 > >2018-07-11 05:42:42.954737 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:42:42.913788) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #36214 to level-5 74546848 bytes > >2018-07-11 05:42:42.954748 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:42:42.954562) EVENT_LOG_v1 {"time_micros": 1531287762954556, "job": 2916, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546848} >2018-07-11 05:42:42.954750 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:42:42.954570) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546848 bytes OK: base level 1 max bytes base 268435456 files[0 5 44 447 3642 783 0] max score 1.00 > >2018-07-11 05:42:42.971222 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:42:42.955659) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #27973 to level-5 74546839 bytes > >2018-07-11 05:42:42.971232 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:42:42.971130) EVENT_LOG_v1 {"time_micros": 1531287762971124, "job": 2917, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546839} >2018-07-11 05:42:42.971235 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:42:42.971137) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546839 bytes OK: base level 1 max bytes base 268435456 files[0 5 44 447 3641 784 0] max score 1.00 > >2018-07-11 05:42:42.987931 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:42:42.972086) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #36209 to level-4 74546129 bytes > >2018-07-11 05:42:42.987943 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:42:42.987830) EVENT_LOG_v1 {"time_micros": 1531287762987823, "job": 2918, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 74546129} >2018-07-11 05:42:42.987946 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:42:42.987839) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-4 74546129 bytes OK: base level 1 max bytes base 268435456 files[0 5 44 446 3642 784 0] max score 1.00 > >2018-07-11 05:42:43.012932 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:42:42.988992) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #23523 to level-5 74546838 bytes > >2018-07-11 05:42:43.012943 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:42:43.012835) EVENT_LOG_v1 {"time_micros": 1531287763012829, "job": 2919, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546838} >2018-07-11 05:42:43.012946 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:42:43.012844) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546838 bytes OK: base level 1 max bytes base 268435456 files[0 5 44 446 3641 785 0] max score 1.00 > >2018-07-11 05:43:37.938962 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:684] reusing log 36178 from recycle list > >2018-07-11 05:43:37.939068 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #36221. Immutable memtables: 0. > >2018-07-11 05:43:38.031602 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:43:37.939129) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1 >2018-07-11 05:43:38.031616 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 2920] Flushing memtable with next log file: 36221 > >2018-07-11 05:43:38.031634 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287818031624, "job": 2920, "event": "flush_started", "num_memtables": 1, "num_entries": 5000042, "num_deletes": 0, "memory_usage": 293074664} >2018-07-11 05:43:38.031637 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 2920] Level-0 flush table #36222: started >2018-07-11 05:43:41.921741 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287821921720, "cf_name": "default", "job": 2920, "event": "table_file_creation", "file_number": 36222, "file_size": 141400749, "table_properties": {"data_size": 127310958, "index_size": 1308755, "filter_size": 12780046, "raw_key_size": 144446308, "raw_average_key_size": 28, "raw_value_size": 59562780, "raw_average_value_size": 11, "num_data_blocks": 31261, "num_entries": 5000033, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:43:41.921763 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 2920] Level-0 flush table #36222: 141400749 bytes OK >2018-07-11 05:43:41.936552 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:242] adding log 36190 to recycle list > >2018-07-11 05:43:41.936563 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:43:41.921776) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #36222 started >2018-07-11 05:43:41.936566 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:43:41.936317) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #36222: memtable #1 done >2018-07-11 05:43:41.936570 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:43:41.936334) EVENT_LOG_v1 {"time_micros": 1531287821936327, "job": 2920, "event": "flush_finished", "lsm_state": [1, 5, 44, 446, 3641, 785, 0], "immutable_memtables": 0} >2018-07-11 05:43:41.936573 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:43:41.936498) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 5 44 446 3641 785 0] max score 1.00 > >2018-07-11 05:43:41.937457 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 2920] Try to delete WAL files size 179011891, prev total WAL file size 179023805, number of live WAL files 2. > >2018-07-11 05:45:07.475735 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:684] reusing log 36190 from recycle list > >2018-07-11 05:45:07.475881 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #36223. Immutable memtables: 0. > >2018-07-11 05:45:07.534484 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:45:07.475916) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1 >2018-07-11 05:45:07.534499 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 2921] Flushing memtable with next log file: 36223 > >2018-07-11 05:45:07.534518 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287907534508, "job": 2921, "event": "flush_started", "num_memtables": 1, "num_entries": 5000039, "num_deletes": 0, "memory_usage": 293019520} >2018-07-11 05:45:07.534523 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 2921] Level-0 flush table #36224: started >2018-07-11 05:45:11.544169 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287911544147, "cf_name": "default", "job": 2921, "event": "table_file_creation", "file_number": 36224, "file_size": 141385320, "table_properties": {"data_size": 127295942, "index_size": 1308385, "filter_size": 12780003, "raw_key_size": 144446153, "raw_average_key_size": 28, "raw_value_size": 59547849, "raw_average_value_size": 11, "num_data_blocks": 31260, "num_entries": 5000030, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:45:11.544205 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 2921] Level-0 flush table #36224: 141385320 bytes OK >2018-07-11 05:45:11.559241 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:242] adding log 36221 to recycle list > >2018-07-11 05:45:11.559265 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:45:11.544218) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #36224 started >2018-07-11 05:45:11.559268 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:45:11.558983) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #36224: memtable #1 done >2018-07-11 05:45:11.559270 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:45:11.559000) EVENT_LOG_v1 {"time_micros": 1531287911558993, "job": 2921, "event": "flush_finished", "lsm_state": [2, 5, 44, 446, 3641, 785, 0], "immutable_memtables": 0} >2018-07-11 05:45:11.559272 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:45:11.559178) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[2 5 44 446 3641 785 0] max score 1.05 > >2018-07-11 05:45:11.559449 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2922] Compacting 2@0 + 3@1 files to L1, score 1.05 >2018-07-11 05:45:11.559467 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2921 Base level 0, inputs: [36224(134MB) 36222(134MB)], [36192(71MB) 36197(13MB) 36198(62MB)] > >2018-07-11 05:45:11.559484 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287911559471, "job": 2922, "event": "compaction_started", "files_L0": [36224, 36222], "files_L1": [36192, 36197, 36198], "score": 1.05346, "input_data_size": 437678711} >2018-07-11 05:45:11.560163 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 2921] Try to delete WAL files size 178997502, prev total WAL file size 179011410, number of live WAL files 2. > >2018-07-11 05:45:15.746757 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2922] Generated table #36225: 2622781 keys, 74510025 bytes >2018-07-11 05:45:15.746802 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287915746775, "cf_name": "default", "job": 2922, "event": "table_file_creation", "file_number": 36225, "file_size": 74510025, "table_properties": {"data_size": 67112885, "index_size": 691604, "filter_size": 6704546, "raw_key_size": 75804681, "raw_average_key_size": 28, "raw_value_size": 31567287, "raw_average_value_size": 12, "num_data_blocks": 16476, "num_entries": 2622781, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:45:18.515919 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2922] Generated table #36226: 2638240 keys, 74546720 bytes >2018-07-11 05:45:18.515966 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287918515939, "cf_name": "default", "job": 2922, "event": "table_file_creation", "file_number": 36226, "file_size": 74546720, "table_properties": {"data_size": 67112144, "index_size": 690405, "filter_size": 6743181, "raw_key_size": 76215826, "raw_average_key_size": 28, "raw_value_size": 31365746, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:45:21.475759 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2922] Generated table #36227: 2638241 keys, 74546062 bytes >2018-07-11 05:45:21.475804 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287921475777, "cf_name": "default", "job": 2922, "event": "table_file_creation", "file_number": 36227, "file_size": 74546062, "table_properties": {"data_size": 67112146, "index_size": 689743, "filter_size": 6743183, "raw_key_size": 76215860, "raw_average_key_size": 28, "raw_value_size": 31365763, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:45:24.188016 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2922] Generated table #36228: 2638241 keys, 74546139 bytes >2018-07-11 05:45:24.188075 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287924188035, "cf_name": "default", "job": 2922, "event": "table_file_creation", "file_number": 36228, "file_size": 74546139, "table_properties": {"data_size": 67112127, "index_size": 689839, "filter_size": 6743183, "raw_key_size": 76215849, "raw_average_key_size": 28, "raw_value_size": 31365752, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:45:27.066138 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2922] Generated table #36229: 2512103 keys, 70981813 bytes >2018-07-11 05:45:27.066182 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287927066155, "cf_name": "default", "job": 2922, "event": "table_file_creation", "file_number": 36229, "file_size": 70981813, "table_properties": {"data_size": 63903434, "index_size": 656618, "filter_size": 6420772, "raw_key_size": 72571878, "raw_average_key_size": 28, "raw_value_size": 29866127, "raw_average_value_size": 11, "num_data_blocks": 15701, "num_entries": 2512103, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:45:28.244733 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2922] Generated table #36230: 166011 keys, 68243131 bytes >2018-07-11 05:45:28.244774 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287928244759, "cf_name": "default", "job": 2922, "event": "table_file_creation", "file_number": 36230, "file_size": 68243131, "table_properties": {"data_size": 67110923, "index_size": 562617, "filter_size": 568603, "raw_key_size": 15359102, "raw_average_key_size": 92, "raw_value_size": 53127261, "raw_average_value_size": 320, "num_data_blocks": 15170, "num_entries": 166011, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "80898", "kMergeOperands": "0"}} >2018-07-11 05:45:28.284817 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2922] Generated table #36231: 276 keys, 298314 bytes >2018-07-11 05:45:28.284840 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287928284829, "cf_name": "default", "job": 2922, "event": "table_file_creation", "file_number": 36231, "file_size": 298314, "table_properties": {"data_size": 293109, "index_size": 2874, "filter_size": 1355, "raw_key_size": 23262, "raw_average_key_size": 84, "raw_value_size": 272011, "raw_average_value_size": 985, "num_data_blocks": 68, "num_entries": 276, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:45:28.284909 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2922] Compacted 2@0 + 3@1 files to L1 => 437672204 bytes >2018-07-11 05:45:28.300743 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:45:28.300523) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 9 44 446 3641 785 0] max score 1.84, MB/sec: 26.2 rd, 26.2 wr, level 1, files in(2, 3) out(7) MB in(269.7, 147.7) out(417.4), read-write-amplify(3.1) write-amplify(1.5) OK, records in: 13215902, records dropped: 9 > >2018-07-11 05:45:28.300753 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:45:28.300549) EVENT_LOG_v1 {"time_micros": 1531287928300537, "job": 2922, "event": "compaction_finished", "compaction_time_micros": 16725406, "output_level": 1, "num_output_files": 7, "total_output_size": 437672204, "num_input_records": 13215902, "num_output_records": 13215893, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 9, 44, 446, 3641, 785, 0]} >2018-07-11 05:45:28.301732 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287928301728, "job": 2922, "event": "table_file_deletion", "file_number": 36224} >2018-07-11 05:45:28.301757 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287928301754, "job": 2922, "event": "table_file_deletion", "file_number": 36222} >2018-07-11 05:45:28.301768 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287928301766, "job": 2922, "event": "table_file_deletion", "file_number": 36198} >2018-07-11 05:45:28.301778 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287928301776, "job": 2922, "event": "table_file_deletion", "file_number": 36197} >2018-07-11 05:45:28.301789 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287928301786, "job": 2922, "event": "table_file_deletion", "file_number": 36192} >2018-07-11 05:45:28.302013 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_picker.cc:461] [default] Expanding@1 1+5(74546720+326500442 bytes) to 4+5 (294620734+326500442bytes) > >2018-07-11 05:45:28.302110 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2923] Compacting 4@1 + 5@2 files to L2, score 1.84 >2018-07-11 05:45:28.302123 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2922 Base level 1, inputs: [36226(71MB) 36227(71MB) 36228(71MB) 36229(67MB)], [36203(71MB) 36204(71MB) 36205(71MB) 36206(71MB) 36202(27MB)] > >2018-07-11 05:45:28.302136 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287928302126, "job": 2923, "event": "compaction_started", "files_L1": [36226, 36227, 36228, 36229], "files_L2": [36203, 36204, 36205, 36206, 36202], "score": 1.84206, "input_data_size": 621121176} >2018-07-11 05:45:32.993385 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2923] Generated table #36232: 2638240 keys, 74545936 bytes >2018-07-11 05:45:32.993418 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287932993402, "cf_name": "default", "job": 2923, "event": "table_file_creation", "file_number": 36232, "file_size": 74545936, "table_properties": {"data_size": 67112137, "index_size": 689628, "filter_size": 6743181, "raw_key_size": 76215811, "raw_average_key_size": 28, "raw_value_size": 31365731, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:45:37.467513 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2923] Generated table #36233: 2638239 keys, 74545973 bytes >2018-07-11 05:45:37.467563 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287937467536, "cf_name": "default", "job": 2923, "event": "table_file_creation", "file_number": 36233, "file_size": 74545973, "table_properties": {"data_size": 67112075, "index_size": 689729, "filter_size": 6743179, "raw_key_size": 76215790, "raw_average_key_size": 28, "raw_value_size": 31365727, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638239, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:45:41.616939 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2923] Generated table #36234: 2638240 keys, 74546185 bytes >2018-07-11 05:45:41.616984 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287941616958, "cf_name": "default", "job": 2923, "event": "table_file_creation", "file_number": 36234, "file_size": 74546185, "table_properties": {"data_size": 67112141, "index_size": 689873, "filter_size": 6743181, "raw_key_size": 76215827, "raw_average_key_size": 28, "raw_value_size": 31365747, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:45:45.895598 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2923] Generated table #36235: 2638240 keys, 74546343 bytes >2018-07-11 05:45:45.895643 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287945895616, "cf_name": "default", "job": 2923, "event": "table_file_creation", "file_number": 36235, "file_size": 74546343, "table_properties": {"data_size": 67112166, "index_size": 690006, "filter_size": 6743181, "raw_key_size": 76215830, "raw_average_key_size": 28, "raw_value_size": 31365750, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:45:49.606153 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2923] Generated table #36236: 2638240 keys, 74546043 bytes >2018-07-11 05:45:49.606199 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287949606171, "cf_name": "default", "job": 2923, "event": "table_file_creation", "file_number": 36236, "file_size": 74546043, "table_properties": {"data_size": 67112124, "index_size": 689748, "filter_size": 6743181, "raw_key_size": 76215826, "raw_average_key_size": 28, "raw_value_size": 31365746, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:45:52.984139 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2923] Generated table #36237: 2638241 keys, 74546286 bytes >2018-07-11 05:45:52.984179 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287952984163, "cf_name": "default", "job": 2923, "event": "table_file_creation", "file_number": 36237, "file_size": 74546286, "table_properties": {"data_size": 67112146, "index_size": 689967, "filter_size": 6743183, "raw_key_size": 76215856, "raw_average_key_size": 28, "raw_value_size": 31365759, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:45:57.716762 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2923] Generated table #36238: 2638241 keys, 74545759 bytes >2018-07-11 05:45:57.716809 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287957716781, "cf_name": "default", "job": 2923, "event": "table_file_creation", "file_number": 36238, "file_size": 74545759, "table_properties": {"data_size": 67112142, "index_size": 689444, "filter_size": 6743183, "raw_key_size": 76215851, "raw_average_key_size": 28, "raw_value_size": 31365754, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:46:01.271722 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2923] Generated table #36239: 2638241 keys, 74546929 bytes >2018-07-11 05:46:01.271761 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287961271745, "cf_name": "default", "job": 2923, "event": "table_file_creation", "file_number": 36239, "file_size": 74546929, "table_properties": {"data_size": 67112128, "index_size": 690628, "filter_size": 6743183, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:46:02.543378 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2923] Generated table #36240: 871532 keys, 24752380 bytes >2018-07-11 05:46:02.543429 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287962543400, "cf_name": "default", "job": 2923, "event": "table_file_creation", "file_number": 36240, "file_size": 24752380, "table_properties": {"data_size": 22294791, "index_size": 228725, "filter_size": 2227876, "raw_key_size": 25193348, "raw_average_key_size": 28, "raw_value_size": 10466926, "raw_average_value_size": 12, "num_data_blocks": 5480, "num_entries": 871532, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "11", "kMergeOperands": "0"}} >2018-07-11 05:46:02.543469 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2923] Compacted 4@1 + 5@2 files to L2 => 621121834 bytes >2018-07-11 05:46:02.558441 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:46:02.558033) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 5 48 446 3641 785 0] max score 1.08, MB/sec: 18.1 rd, 18.1 wr, level 2, files in(4, 5) out(9) MB in(281.0, 311.4) out(592.3), read-write-amplify(4.2) write-amplify(2.1) OK, records in: 21977454, records dropped: 0 > >2018-07-11 05:46:02.558453 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:46:02.558060) EVENT_LOG_v1 {"time_micros": 1531287962558048, "job": 2923, "event": "compaction_finished", "compaction_time_micros": 34241312, "output_level": 2, "num_output_files": 9, "total_output_size": 621121834, "num_input_records": 21977454, "num_output_records": 21977454, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 5, 48, 446, 3641, 785, 0]} >2018-07-11 05:46:02.559386 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287962559382, "job": 2923, "event": "table_file_deletion", "file_number": 36229} >2018-07-11 05:46:02.559400 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287962559397, "job": 2923, "event": "table_file_deletion", "file_number": 36228} >2018-07-11 05:46:02.559411 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287962559409, "job": 2923, "event": "table_file_deletion", "file_number": 36227} >2018-07-11 05:46:02.559429 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287962559424, "job": 2923, "event": "table_file_deletion", "file_number": 36226} >2018-07-11 05:46:02.559447 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287962559444, "job": 2923, "event": "table_file_deletion", "file_number": 36206} >2018-07-11 05:46:02.559460 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287962559458, "job": 2923, "event": "table_file_deletion", "file_number": 36205} >2018-07-11 05:46:02.559475 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287962559473, "job": 2923, "event": "table_file_deletion", "file_number": 36204} >2018-07-11 05:46:02.559488 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287962559486, "job": 2923, "event": "table_file_deletion", "file_number": 36203} >2018-07-11 05:46:02.559502 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287962559500, "job": 2923, "event": "table_file_deletion", "file_number": 36202} >2018-07-11 05:46:02.559735 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_picker.cc:461] [default] Expanding@2 1+1(74546929+46962830 bytes) to 3+1 (223638974+46962830bytes) > >2018-07-11 05:46:02.559836 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2924] Compacting 3@2 + 1@3 files to L3, score 1.08 >2018-07-11 05:46:02.559848 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2923 Base level 2, inputs: [36237(71MB) 36238(71MB) 36239(71MB)], [36215(44MB)] > >2018-07-11 05:46:02.559862 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287962559851, "job": 2924, "event": "compaction_started", "files_L2": [36237, 36238, 36239], "files_L3": [36215], "score": 1.08316, "input_data_size": 270601804} >2018-07-11 05:46:06.786308 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2924] Generated table #36241: 2638241 keys, 74546630 bytes >2018-07-11 05:46:06.786354 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287966786340, "cf_name": "default", "job": 2924, "event": "table_file_creation", "file_number": 36241, "file_size": 74546630, "table_properties": {"data_size": 67112171, "index_size": 690285, "filter_size": 6743184, "raw_key_size": 76215852, "raw_average_key_size": 28, "raw_value_size": 31365755, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:46:11.339812 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2924] Generated table #36242: 2638241 keys, 74546389 bytes >2018-07-11 05:46:11.339862 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287971339845, "cf_name": "default", "job": 2924, "event": "table_file_creation", "file_number": 36242, "file_size": 74546389, "table_properties": {"data_size": 67112123, "index_size": 690093, "filter_size": 6743183, "raw_key_size": 76215857, "raw_average_key_size": 28, "raw_value_size": 31365760, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:46:15.564640 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2924] Generated table #36243: 2638241 keys, 74546133 bytes >2018-07-11 05:46:15.564696 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287975564679, "cf_name": "default", "job": 2924, "event": "table_file_creation", "file_number": 36243, "file_size": 74546133, "table_properties": {"data_size": 67112147, "index_size": 689813, "filter_size": 6743183, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:46:18.118926 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2924] Generated table #36244: 1519835 keys, 46962521 bytes >2018-07-11 05:46:18.118970 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287978118953, "cf_name": "default", "job": 2924, "event": "table_file_creation", "file_number": 36244, "file_size": 46962521, "table_properties": {"data_size": 42634534, "index_size": 433065, "filter_size": 3893934, "raw_key_size": 44346592, "raw_average_key_size": 29, "raw_value_size": 21521498, "raw_average_value_size": 14, "num_data_blocks": 10537, "num_entries": 1519835, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:46:18.119008 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2924] Compacted 3@2 + 1@3 files to L3 => 270601673 bytes >2018-07-11 05:46:18.283025 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:46:18.282793) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 5 45 449 3641 785 0] max score 1.01, MB/sec: 17.4 rd, 17.4 wr, level 3, files in(3, 1) out(4) MB in(213.3, 44.8) out(258.1), read-write-amplify(2.4) write-amplify(1.2) OK, records in: 9434558, records dropped: 0 > >2018-07-11 05:46:18.283036 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:46:18.282821) EVENT_LOG_v1 {"time_micros": 1531287978282809, "job": 2924, "event": "compaction_finished", "compaction_time_micros": 15559131, "output_level": 3, "num_output_files": 4, "total_output_size": 270601673, "num_input_records": 9434558, "num_output_records": 9434558, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 5, 45, 449, 3641, 785, 0]} >2018-07-11 05:46:18.284012 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287978283995, "job": 2924, "event": "table_file_deletion", "file_number": 36239} >2018-07-11 05:46:18.284027 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287978284025, "job": 2924, "event": "table_file_deletion", "file_number": 36238} >2018-07-11 05:46:18.284041 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287978284039, "job": 2924, "event": "table_file_deletion", "file_number": 36237} >2018-07-11 05:46:18.284054 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531287978284051, "job": 2924, "event": "table_file_deletion", "file_number": 36215} >2018-07-11 05:46:18.299503 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:46:18.284371) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #36241 to level-4 74546630 bytes > >2018-07-11 05:46:18.299513 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:46:18.299401) EVENT_LOG_v1 {"time_micros": 1531287978299395, "job": 2925, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 74546630} >2018-07-11 05:46:18.299516 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:46:18.299409) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-4 74546630 bytes OK: base level 1 max bytes base 268435456 files[0 5 45 448 3642 785 0] max score 1.00 > >2018-07-11 05:46:18.316186 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:46:18.300394) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #36242 to level-4 74546389 bytes > >2018-07-11 05:46:18.316197 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:46:18.316093) EVENT_LOG_v1 {"time_micros": 1531287978316087, "job": 2926, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 74546389} >2018-07-11 05:46:18.316200 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:46:18.316100) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-4 74546389 bytes OK: base level 1 max bytes base 268435456 files[0 5 45 447 3643 785 0] max score 1.00 > >2018-07-11 05:46:18.332880 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:46:18.317049) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #26359 to level-5 74546838 bytes > >2018-07-11 05:46:18.332891 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:46:18.332787) EVENT_LOG_v1 {"time_micros": 1531287978332781, "job": 2927, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546838} >2018-07-11 05:46:18.332894 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:46:18.332795) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546838 bytes OK: base level 1 max bytes base 268435456 files[0 5 45 447 3642 786 0] max score 1.00 > >2018-07-11 05:46:18.349556 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:46:18.333783) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #10760 to level-5 74546837 bytes > >2018-07-11 05:46:18.349566 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:46:18.349462) EVENT_LOG_v1 {"time_micros": 1531287978349456, "job": 2928, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546837} >2018-07-11 05:46:18.349569 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:46:18.349469) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546837 bytes OK: base level 1 max bytes base 268435456 files[0 5 45 447 3641 787 0] max score 1.00 > >2018-07-11 05:46:18.366275 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:46:18.350447) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #36243 to level-4 74546133 bytes > >2018-07-11 05:46:18.366285 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:46:18.366173) EVENT_LOG_v1 {"time_micros": 1531287978366167, "job": 2929, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 74546133} >2018-07-11 05:46:18.366288 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:46:18.366181) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-4 74546133 bytes OK: base level 1 max bytes base 268435456 files[0 5 45 446 3642 787 0] max score 1.00 > >2018-07-11 05:46:18.383095 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:46:18.367134) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #19677 to level-5 74546837 bytes > >2018-07-11 05:46:18.383105 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:46:18.382889) EVENT_LOG_v1 {"time_micros": 1531287978382883, "job": 2930, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546837} >2018-07-11 05:46:18.383108 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:46:18.382898) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546837 bytes OK: base level 1 max bytes base 268435456 files[0 5 45 446 3641 788 0] max score 1.00 > >2018-07-11 05:47:01.002076 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:684] reusing log 36221 from recycle list > >2018-07-11 05:47:01.002243 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #36245. Immutable memtables: 0. > >2018-07-11 05:47:01.092780 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:47:01.002291) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1 >2018-07-11 05:47:01.092794 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 2931] Flushing memtable with next log file: 36245 > >2018-07-11 05:47:01.092811 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288021092802, "job": 2931, "event": "flush_started", "num_memtables": 1, "num_entries": 5000042, "num_deletes": 0, "memory_usage": 293046616} >2018-07-11 05:47:01.092815 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 2931] Level-0 flush table #36246: started >2018-07-11 05:47:05.117862 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288025117842, "cf_name": "default", "job": 2931, "event": "table_file_creation", "file_number": 36246, "file_size": 141403635, "table_properties": {"data_size": 127314525, "index_size": 1308066, "filter_size": 12780054, "raw_key_size": 144446308, "raw_average_key_size": 28, "raw_value_size": 59566349, "raw_average_value_size": 11, "num_data_blocks": 31261, "num_entries": 5000033, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:47:05.117893 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 2931] Level-0 flush table #36246: 141403635 bytes OK >2018-07-11 05:47:05.131927 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:242] adding log 36223 to recycle list > >2018-07-11 05:47:05.131939 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:47:05.117906) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #36246 started >2018-07-11 05:47:05.131943 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:47:05.131796) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #36246: memtable #1 done >2018-07-11 05:47:05.131945 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:47:05.131814) EVENT_LOG_v1 {"time_micros": 1531288025131807, "job": 2931, "event": "flush_finished", "lsm_state": [1, 5, 45, 446, 3641, 788, 0], "immutable_memtables": 0} >2018-07-11 05:47:05.131947 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:47:05.131832) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 5 45 446 3641 788 0] max score 1.00 > >2018-07-11 05:47:05.133075 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 2931] Try to delete WAL files size 179015460, prev total WAL file size 179015472, number of live WAL files 2. > >2018-07-11 05:48:59.678272 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:684] reusing log 36223 from recycle list > >2018-07-11 05:48:59.678403 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #36247. Immutable memtables: 0. > >2018-07-11 05:48:59.728212 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:48:59.678480) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1 >2018-07-11 05:48:59.728225 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 2932] Flushing memtable with next log file: 36247 > >2018-07-11 05:48:59.728242 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288139728233, "job": 2932, "event": "flush_started", "num_memtables": 1, "num_entries": 5000039, "num_deletes": 0, "memory_usage": 293048352} >2018-07-11 05:48:59.728253 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 2932] Level-0 flush table #36248: started >2018-07-11 05:49:02.354722 7f41649ec700 1 bluefs _allocate failed to allocate 0x8700000 on bdev 1, free 0x4fe000; fallback to bdev 2 >2018-07-11 05:49:04.466610 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288144466589, "cf_name": "default", "job": 2932, "event": "table_file_creation", "file_number": 36248, "file_size": 141385815, "table_properties": {"data_size": 127295145, "index_size": 1309680, "filter_size": 12780000, "raw_key_size": 144446153, "raw_average_key_size": 28, "raw_value_size": 59547044, "raw_average_value_size": 11, "num_data_blocks": 31260, "num_entries": 5000030, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:49:04.466634 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 2932] Level-0 flush table #36248: 141385815 bytes OK >2018-07-11 05:49:04.478691 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:242] adding log 36245 to recycle list > >2018-07-11 05:49:04.478703 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:49:04.466658) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #36248 started >2018-07-11 05:49:04.478707 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:49:04.478442) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #36248: memtable #1 done >2018-07-11 05:49:04.478709 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:49:04.478460) EVENT_LOG_v1 {"time_micros": 1531288144478453, "job": 2932, "event": "flush_finished", "lsm_state": [2, 5, 45, 446, 3641, 788, 0], "immutable_memtables": 0} >2018-07-11 05:49:04.478711 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:49:04.478637) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[2 5 45 446 3641 788 0] max score 1.05 > >2018-07-11 05:49:04.479125 7f41641eb700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS ------- >2018-07-11 05:49:04.479135 7f41641eb700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] >** DB Stats ** >Uptime(secs): 52886.1 total, 674.0 interval >Cumulative writes: 196K writes, 1967M keys, 196K commit groups, 1.0 writes per commit group, ingest: 66.06 GB, 1.28 MB/s >Cumulative WAL: 196K writes, 56K syncs, 3.48 writes per sync, written: 66.06 GB, 1.28 MB/s >Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent >Interval writes: 138 writes, 30M keys, 138 commit groups, 1.0 writes per commit group, ingest: 1024.13 MB, 1.52 MB/s >Interval WAL: 138 writes, 69 syncs, 1.97 writes per sync, written: 1.00 MB, 1.52 MB/s >Interval stall: 00:00:0.000 H:M:S, 0.0 percent > >** Compaction Stats [default] ** >Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop >---------------------------------------------------------------------------------------------------------------------------------------------------------- > L0 2/0 269.69 MB 1.1 0.0 0.0 0.0 51.8 51.8 0.0 1.0 0.0 32.4 1636 395 4.142 0 0 > L1 5/0 190.59 MB 0.7 84.9 51.6 33.3 84.9 51.6 0.0 1.6 25.7 25.7 3386 194 17.452 2688M 894K > L2 45/0 2.50 GB 1.0 101.1 51.6 49.4 101.0 51.5 0.0 2.0 21.5 21.5 4816 569 8.464 3775M 1358K > L3 446/0 24.93 GB 1.0 89.2 50.8 38.4 89.2 50.8 0.7 1.8 20.8 20.8 4394 304 14.453 3370M 4000K > L4 3641/0 250.00 GB 1.0 78.0 18.4 59.6 77.7 18.0 33.1 4.2 21.2 21.1 3767 247 15.252 2966M 15M > L5 788/0 54.69 GB 0.0 2.4 1.1 1.2 2.3 1.1 50.0 2.1 19.7 19.5 123 15 8.195 89M 1097K > Sum 4927/0 332.57 GB 0.0 355.5 173.5 182.0 406.8 224.8 83.8 7.9 20.1 23.0 18122 1724 10.512 12G 23M > Int 0/0 0.00 KB 0.0 4.8 2.5 2.3 5.6 3.3 1.4 7.1 17.7 20.6 278 19 14.612 175M 25 >Uptime(secs): 52886.1 total, 52886.1 interval >Flush(GB): cumulative 51.761, interval 0.790 >AddFile(GB): cumulative 0.000, interval 0.000 >AddFile(Total Files): cumulative 0, interval 0 >AddFile(L0 Files): cumulative 0, interval 0 >AddFile(Keys): cumulative 0, interval 0 >Cumulative compaction: 406.76 GB write, 7.88 MB/s write, 355.53 GB read, 6.88 MB/s read, 18121.9 seconds >Interval compaction: 5.60 GB write, 0.11 MB/s write, 4.81 GB read, 0.09 MB/s read, 277.6 seconds >Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count > >** File Read Latency Histogram By Level [default] ** > >** Compaction Stats [default] ** >Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop >---------------------------------------------------------------------------------------------------------------------------------------------------------- > L0 2/0 269.69 MB 1.1 0.0 0.0 0.0 51.8 51.8 0.0 1.0 0.0 32.4 1636 395 4.142 0 0 > L1 5/0 190.59 MB 0.7 84.9 51.6 33.3 84.9 51.6 0.0 1.6 25.7 25.7 3386 194 17.452 2688M 894K > L2 45/0 2.50 GB 1.0 101.1 51.6 49.4 101.0 51.5 0.0 2.0 21.5 21.5 4816 569 8.464 3775M 1358K > L3 446/0 24.93 GB 1.0 89.2 50.8 38.4 89.2 50.8 0.7 1.8 20.8 20.8 4394 304 14.453 3370M 4000K > L4 3641/0 250.00 GB 1.0 78.0 18.4 59.6 77.7 18.0 33.1 4.2 21.2 21.1 3767 247 15.252 2966M 15M > L5 788/0 54.69 GB 0.0 2.4 1.1 1.2 2.3 1.1 50.0 2.1 19.7 19.5 123 15 8.195 89M 1097K > Sum 4927/0 332.57 GB 0.0 355.5 173.5 182.0 406.8 224.8 83.8 7.9 20.1 23.0 18122 1724 10.512 12G 23M > Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 >Uptime(secs): 52886.1 total, 52886.1 interval >Flush(GB): cumulative 51.761, interval 0.000 >AddFile(GB): cumulative 0.000, interval 0.000 >AddFile(Total Files): cumulative 0, interval 0 >AddFile(L0 Files): cumulative 0, interval 0 >AddFile(Keys): cumulative 0, interval 0 >Cumulative compaction: 406.76 GB write, 7.88 MB/s write, 355.53 GB read, 6.88 MB/s read, 18121.9 seconds >Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds >Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count > >** File Read Latency Histogram By Level [default] ** > >2018-07-11 05:49:04.479280 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2933] Compacting 2@0 + 2@1 files to L1, score 1.05 >2018-07-11 05:49:04.479295 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2932 Base level 0, inputs: [36248(134MB) 36246(134MB)], [36225(71MB) 36230(65MB)] > >2018-07-11 05:49:04.479317 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288144479300, "job": 2933, "event": "compaction_started", "files_L0": [36248, 36246], "files_L1": [36225, 36230], "score": 1.05347, "input_data_size": 425542606} >2018-07-11 05:49:04.479607 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 2932] Try to delete WAL files size 178996697, prev total WAL file size 178996709, number of live WAL files 2. > >2018-07-11 05:49:07.011089 7f41641eb700 1 bluefs _allocate failed to allocate 0x4800000 on bdev 1, free 0x4fe000; fallback to bdev 2 >2018-07-11 05:49:08.173734 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2933] Generated table #36249: 2622488 keys, 74506157 bytes >2018-07-11 05:49:08.173779 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288148173752, "cf_name": "default", "job": 2933, "event": "table_file_creation", "file_number": 36249, "file_size": 74506157, "table_properties": {"data_size": 67109869, "index_size": 691489, "filter_size": 6703809, "raw_key_size": 75796784, "raw_average_key_size": 28, "raw_value_size": 31568194, "raw_average_value_size": 12, "num_data_blocks": 16475, "num_entries": 2622488, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:49:10.223378 7f41641eb700 1 bluefs _allocate failed to allocate 0x4800000 on bdev 1, free 0x4fe000; fallback to bdev 2 >2018-07-11 05:49:11.214891 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2933] Generated table #36250: 2638242 keys, 74546376 bytes >2018-07-11 05:49:11.214935 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288151214908, "cf_name": "default", "job": 2933, "event": "table_file_creation", "file_number": 36250, "file_size": 74546376, "table_properties": {"data_size": 67112135, "index_size": 690065, "filter_size": 6743186, "raw_key_size": 76215876, "raw_average_key_size": 28, "raw_value_size": 31365762, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638242, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:49:13.580730 7f41641eb700 1 bluefs _allocate failed to allocate 0x4800000 on bdev 1, free 0x4fe000; fallback to bdev 2 >2018-07-11 05:49:14.680221 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2933] Generated table #36251: 2638241 keys, 74546214 bytes >2018-07-11 05:49:14.680272 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288154680240, "cf_name": "default", "job": 2933, "event": "table_file_creation", "file_number": 36251, "file_size": 74546214, "table_properties": {"data_size": 67112145, "index_size": 689896, "filter_size": 6743183, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:49:17.907861 7f41641eb700 1 bluefs _allocate failed to allocate 0x4800000 on bdev 1, free 0x4fe000; fallback to bdev 2 >2018-07-11 05:49:18.999152 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2933] Generated table #36252: 2638241 keys, 74545771 bytes >2018-07-11 05:49:18.999206 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288158999186, "cf_name": "default", "job": 2933, "event": "table_file_creation", "file_number": 36252, "file_size": 74545771, "table_properties": {"data_size": 67112090, "index_size": 689508, "filter_size": 6743183, "raw_key_size": 76215855, "raw_average_key_size": 28, "raw_value_size": 31365758, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:49:21.824638 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2933] Generated table #36253: 2085596 keys, 58930857 bytes >2018-07-11 05:49:21.824681 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288161824654, "cf_name": "default", "job": 2933, "event": "table_file_creation", "file_number": 36253, "file_size": 58930857, "table_properties": {"data_size": 53053846, "index_size": 545373, "filter_size": 5330650, "raw_key_size": 60250562, "raw_average_key_size": 28, "raw_value_size": 24795430, "raw_average_value_size": 11, "num_data_blocks": 13035, "num_entries": 2085596, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:49:22.905759 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2933] Generated table #36254: 165826 keys, 68240475 bytes >2018-07-11 05:49:22.905801 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288162905774, "cf_name": "default", "job": 2933, "event": "table_file_creation", "file_number": 36254, "file_size": 68240475, "table_properties": {"data_size": 67110506, "index_size": 560868, "filter_size": 568113, "raw_key_size": 15343194, "raw_average_key_size": 92, "raw_value_size": 53140464, "raw_average_value_size": 320, "num_data_blocks": 15131, "num_entries": 165826, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "80898", "kMergeOperands": "0"}} >2018-07-11 05:49:22.946003 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2933] Generated table #36255: 212 keys, 219644 bytes >2018-07-11 05:49:22.946027 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288162946015, "cf_name": "default", "job": 2933, "event": "table_file_creation", "file_number": 36255, "file_size": 219644, "table_properties": {"data_size": 215564, "index_size": 2090, "filter_size": 1014, "raw_key_size": 17895, "raw_average_key_size": 84, "raw_value_size": 199911, "raw_average_value_size": 942, "num_data_blocks": 50, "num_entries": 212, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:49:22.946098 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2933] Compacted 2@0 + 2@1 files to L1 => 425535494 bytes >2018-07-11 05:49:22.961828 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:49:22.961619) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 10 45 446 3641 788 0] max score 1.80, MB/sec: 23.0 rd, 23.0 wr, level 1, files in(2, 2) out(7) MB in(269.7, 136.1) out(405.8), read-write-amplify(3.0) write-amplify(1.5) OK, records in: 12788855, records dropped: 9 > >2018-07-11 05:49:22.961839 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:49:22.961644) EVENT_LOG_v1 {"time_micros": 1531288162961633, "job": 2933, "event": "compaction_finished", "compaction_time_micros": 18466759, "output_level": 1, "num_output_files": 7, "total_output_size": 425535494, "num_input_records": 12788855, "num_output_records": 12788846, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 10, 45, 446, 3641, 788, 0]} >2018-07-11 05:49:22.962836 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288162962820, "job": 2933, "event": "table_file_deletion", "file_number": 36248} >2018-07-11 05:49:22.962851 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288162962849, "job": 2933, "event": "table_file_deletion", "file_number": 36246} >2018-07-11 05:49:22.962863 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288162962861, "job": 2933, "event": "table_file_deletion", "file_number": 36230} >2018-07-11 05:49:22.962874 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288162962872, "job": 2933, "event": "table_file_deletion", "file_number": 36225} >2018-07-11 05:49:22.963115 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_picker.cc:461] [default] Expanding@1 1+6(74546376+397482860 bytes) to 4+6 (282569218+397482860bytes) > >2018-07-11 05:49:22.963213 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2934] Compacting 4@1 + 6@2 files to L2, score 1.80 >2018-07-11 05:49:22.963240 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2933 Base level 1, inputs: [36250(71MB) 36251(71MB) 36252(71MB) 36253(56MB)], [36232(71MB) 36233(71MB) 36234(71MB) 36235(71MB) 36236(71MB) 36240(23MB)] > >2018-07-11 05:49:22.963262 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288162963251, "job": 2934, "event": "compaction_started", "files_L1": [36250, 36251, 36252, 36253], "files_L2": [36232, 36233, 36234, 36235, 36236, 36240], "score": 1.79795, "input_data_size": 680052078} >2018-07-11 05:49:26.914696 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2934] Generated table #36256: 2638240 keys, 74545940 bytes >2018-07-11 05:49:26.914740 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288166914713, "cf_name": "default", "job": 2934, "event": "table_file_creation", "file_number": 36256, "file_size": 74545940, "table_properties": {"data_size": 67112141, "index_size": 689628, "filter_size": 6743181, "raw_key_size": 76215816, "raw_average_key_size": 28, "raw_value_size": 31365736, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:49:30.487966 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2934] Generated table #36257: 2638239 keys, 74545978 bytes >2018-07-11 05:49:30.488011 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288170487985, "cf_name": "default", "job": 2934, "event": "table_file_creation", "file_number": 36257, "file_size": 74545978, "table_properties": {"data_size": 67112079, "index_size": 689730, "filter_size": 6743179, "raw_key_size": 76215794, "raw_average_key_size": 28, "raw_value_size": 31365731, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638239, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:49:34.923990 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2934] Generated table #36258: 2638240 keys, 74546179 bytes >2018-07-11 05:49:34.924035 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288174924008, "cf_name": "default", "job": 2934, "event": "table_file_creation", "file_number": 36258, "file_size": 74546179, "table_properties": {"data_size": 67112135, "index_size": 689873, "filter_size": 6743181, "raw_key_size": 76215820, "raw_average_key_size": 28, "raw_value_size": 31365740, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:49:39.280902 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2934] Generated table #36259: 2638240 keys, 74546344 bytes >2018-07-11 05:49:39.280944 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288179280928, "cf_name": "default", "job": 2934, "event": "table_file_creation", "file_number": 36259, "file_size": 74546344, "table_properties": {"data_size": 67112166, "index_size": 690007, "filter_size": 6743181, "raw_key_size": 76215830, "raw_average_key_size": 28, "raw_value_size": 31365750, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:49:43.425706 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2934] Generated table #36260: 2638240 keys, 74546044 bytes >2018-07-11 05:49:43.425754 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288183425726, "cf_name": "default", "job": 2934, "event": "table_file_creation", "file_number": 36260, "file_size": 74546044, "table_properties": {"data_size": 67112125, "index_size": 689748, "filter_size": 6743181, "raw_key_size": 76215826, "raw_average_key_size": 28, "raw_value_size": 31365746, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:49:47.740108 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2934] Generated table #36261: 2638241 keys, 74546493 bytes >2018-07-11 05:49:47.740162 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288187740145, "cf_name": "default", "job": 2934, "event": "table_file_creation", "file_number": 36261, "file_size": 74546493, "table_properties": {"data_size": 67112163, "index_size": 690157, "filter_size": 6743183, "raw_key_size": 76215857, "raw_average_key_size": 28, "raw_value_size": 31365760, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:49:51.978072 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2934] Generated table #36262: 2638241 keys, 74546367 bytes >2018-07-11 05:49:51.978125 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288191978107, "cf_name": "default", "job": 2934, "event": "table_file_creation", "file_number": 36262, "file_size": 74546367, "table_properties": {"data_size": 67112094, "index_size": 690100, "filter_size": 6743183, "raw_key_size": 76215853, "raw_average_key_size": 28, "raw_value_size": 31365756, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:49:55.802396 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2934] Generated table #36263: 2638241 keys, 74546002 bytes >2018-07-11 05:49:55.802442 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288195802423, "cf_name": "default", "job": 2934, "event": "table_file_creation", "file_number": 36263, "file_size": 74546002, "table_properties": {"data_size": 67112163, "index_size": 689666, "filter_size": 6743183, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:50:00.205709 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2934] Generated table #36264: 2638241 keys, 74546323 bytes >2018-07-11 05:50:00.205762 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288200205733, "cf_name": "default", "job": 2934, "event": "table_file_creation", "file_number": 36264, "file_size": 74546323, "table_properties": {"data_size": 67112101, "index_size": 690049, "filter_size": 6743183, "raw_key_size": 76215852, "raw_average_key_size": 28, "raw_value_size": 31365755, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:50:00.676088 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2934] Generated table #36265: 318888 keys, 9136927 bytes >2018-07-11 05:50:00.676133 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288200676106, "cf_name": "default", "job": 2934, "event": "table_file_creation", "file_number": 36265, "file_size": 9136927, "table_properties": {"data_size": 8236515, "index_size": 84073, "filter_size": 815353, "raw_key_size": 9228077, "raw_average_key_size": 28, "raw_value_size": 3896603, "raw_average_value_size": 12, "num_data_blocks": 2026, "num_entries": 318888, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "11", "kMergeOperands": "0"}} >2018-07-11 05:50:00.676169 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2934] Compacted 4@1 + 6@2 files to L2 => 680052597 bytes >2018-07-11 05:50:00.694535 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:50:00.694110) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 6 49 446 3641 788 0] max score 1.11, MB/sec: 18.0 rd, 18.0 wr, level 2, files in(4, 6) out(10) MB in(269.5, 379.1) out(648.5), read-write-amplify(4.8) write-amplify(2.4) OK, records in: 24063051, records dropped: 0 > >2018-07-11 05:50:00.694547 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:50:00.694136) EVENT_LOG_v1 {"time_micros": 1531288200694124, "job": 2934, "event": "compaction_finished", "compaction_time_micros": 37712888, "output_level": 2, "num_output_files": 10, "total_output_size": 680052597, "num_input_records": 24063051, "num_output_records": 24063051, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 6, 49, 446, 3641, 788, 0]} >2018-07-11 05:50:00.695473 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288200695468, "job": 2934, "event": "table_file_deletion", "file_number": 36253} >2018-07-11 05:50:00.695498 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288200695494, "job": 2934, "event": "table_file_deletion", "file_number": 36252} >2018-07-11 05:50:00.695521 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288200695517, "job": 2934, "event": "table_file_deletion", "file_number": 36251} >2018-07-11 05:50:00.695539 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288200695536, "job": 2934, "event": "table_file_deletion", "file_number": 36250} >2018-07-11 05:50:00.695566 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288200695562, "job": 2934, "event": "table_file_deletion", "file_number": 36240} >2018-07-11 05:50:00.695589 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288200695585, "job": 2934, "event": "table_file_deletion", "file_number": 36236} >2018-07-11 05:50:00.695616 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288200695612, "job": 2934, "event": "table_file_deletion", "file_number": 36235} >2018-07-11 05:50:00.695639 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288200695635, "job": 2934, "event": "table_file_deletion", "file_number": 36234} >2018-07-11 05:50:00.695662 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288200695658, "job": 2934, "event": "table_file_deletion", "file_number": 36233} >2018-07-11 05:50:00.695685 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288200695680, "job": 2934, "event": "table_file_deletion", "file_number": 36232} >2018-07-11 05:50:00.695915 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_picker.cc:461] [default] Expanding@2 1+1(74546493+46962521 bytes) to 4+1 (298185185+46962521bytes) > >2018-07-11 05:50:00.696020 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2935] Compacting 4@2 + 1@3 files to L3, score 1.11 >2018-07-11 05:50:00.696034 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2934 Base level 2, inputs: [36261(71MB) 36262(71MB) 36263(71MB) 36264(71MB)], [36244(44MB)] > >2018-07-11 05:50:00.696054 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288200696039, "job": 2935, "event": "compaction_started", "files_L2": [36261, 36262, 36263, 36264], "files_L3": [36244], "score": 1.10511, "input_data_size": 345147706} >2018-07-11 05:50:05.115969 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2935] Generated table #36266: 2638241 keys, 74545879 bytes >2018-07-11 05:50:05.116009 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288205115993, "cf_name": "default", "job": 2935, "event": "table_file_creation", "file_number": 36266, "file_size": 74545879, "table_properties": {"data_size": 67112096, "index_size": 689610, "filter_size": 6743183, "raw_key_size": 76215852, "raw_average_key_size": 28, "raw_value_size": 31365755, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:50:09.093799 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2935] Generated table #36267: 2638241 keys, 74546219 bytes >2018-07-11 05:50:09.093848 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288209093820, "cf_name": "default", "job": 2935, "event": "table_file_creation", "file_number": 36267, "file_size": 74546219, "table_properties": {"data_size": 67112162, "index_size": 689884, "filter_size": 6743183, "raw_key_size": 76215853, "raw_average_key_size": 28, "raw_value_size": 31365756, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:50:13.113537 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2935] Generated table #36268: 2638241 keys, 74547004 bytes >2018-07-11 05:50:13.113585 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288213113564, "cf_name": "default", "job": 2935, "event": "table_file_creation", "file_number": 36268, "file_size": 74547004, "table_properties": {"data_size": 67112128, "index_size": 690703, "filter_size": 6743183, "raw_key_size": 76215859, "raw_average_key_size": 28, "raw_value_size": 31365762, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:50:17.181778 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2935] Generated table #36269: 2638241 keys, 74546284 bytes >2018-07-11 05:50:17.181823 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288217181807, "cf_name": "default", "job": 2935, "event": "table_file_creation", "file_number": 36269, "file_size": 74546284, "table_properties": {"data_size": 67112154, "index_size": 689957, "filter_size": 6743183, "raw_key_size": 76215850, "raw_average_key_size": 28, "raw_value_size": 31365753, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:50:19.939755 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2935] Generated table #36270: 1519835 keys, 46962876 bytes >2018-07-11 05:50:19.939807 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288219939791, "cf_name": "default", "job": 2935, "event": "table_file_creation", "file_number": 36270, "file_size": 46962876, "table_properties": {"data_size": 42634546, "index_size": 433408, "filter_size": 3893934, "raw_key_size": 44346594, "raw_average_key_size": 29, "raw_value_size": 21521500, "raw_average_value_size": 14, "num_data_blocks": 10537, "num_entries": 1519835, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:50:19.939843 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2935] Compacted 4@2 + 1@3 files to L3 => 345148262 bytes >2018-07-11 05:50:19.952301 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:50:19.952076) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 6 45 450 3641 788 0] max score 1.01, MB/sec: 17.9 rd, 17.9 wr, level 3, files in(4, 1) out(5) MB in(284.4, 44.8) out(329.2), read-write-amplify(2.3) write-amplify(1.2) OK, records in: 12072799, records dropped: 0 > >2018-07-11 05:50:19.952313 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:50:19.952100) EVENT_LOG_v1 {"time_micros": 1531288219952089, "job": 2935, "event": "compaction_finished", "compaction_time_micros": 19243770, "output_level": 3, "num_output_files": 5, "total_output_size": 345148262, "num_input_records": 12072799, "num_output_records": 12072799, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 6, 45, 450, 3641, 788, 0]} >2018-07-11 05:50:19.953273 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288219953267, "job": 2935, "event": "table_file_deletion", "file_number": 36264} >2018-07-11 05:50:19.953298 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288219953294, "job": 2935, "event": "table_file_deletion", "file_number": 36263} >2018-07-11 05:50:19.953324 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288219953320, "job": 2935, "event": "table_file_deletion", "file_number": 36262} >2018-07-11 05:50:19.953349 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288219953344, "job": 2935, "event": "table_file_deletion", "file_number": 36261} >2018-07-11 05:50:19.953374 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288219953371, "job": 2935, "event": "table_file_deletion", "file_number": 36244} >2018-07-11 05:50:19.968802 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:50:19.953692) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #36268 to level-4 74547004 bytes > >2018-07-11 05:50:19.968812 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:50:19.968707) EVENT_LOG_v1 {"time_micros": 1531288219968701, "job": 2936, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 74547004} >2018-07-11 05:50:19.968820 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:50:19.968714) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-4 74547004 bytes OK: base level 1 max bytes base 268435456 files[0 6 45 449 3642 788 0] max score 1.01 > >2018-07-11 05:50:19.985521 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:50:19.969723) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #36269 to level-4 74546284 bytes > >2018-07-11 05:50:19.985532 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:50:19.985428) EVENT_LOG_v1 {"time_micros": 1531288219985422, "job": 2937, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 74546284} >2018-07-11 05:50:19.985537 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:50:19.985435) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-4 74546284 bytes OK: base level 1 max bytes base 268435456 files[0 6 45 448 3643 788 0] max score 1.00 > >2018-07-11 05:50:20.002181 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:50:19.986419) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #36267 to level-4 74546219 bytes > >2018-07-11 05:50:20.002190 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:50:20.002084) EVENT_LOG_v1 {"time_micros": 1531288220002078, "job": 2938, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 74546219} >2018-07-11 05:50:20.002195 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:50:20.002092) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-4 74546219 bytes OK: base level 1 max bytes base 268435456 files[0 6 45 447 3644 788 0] max score 1.00 > >2018-07-11 05:50:20.060587 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:50:20.003068) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #36268 to level-5 74547004 bytes > >2018-07-11 05:50:20.060599 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:50:20.060477) EVENT_LOG_v1 {"time_micros": 1531288220060471, "job": 2939, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74547004} >2018-07-11 05:50:20.060603 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:50:20.060485) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74547004 bytes OK: base level 1 max bytes base 268435456 files[0 6 45 447 3643 789 0] max score 1.00 > >2018-07-11 05:50:20.077315 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:50:20.061539) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #1108 to level-5 74546837 bytes > >2018-07-11 05:50:20.077326 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:50:20.077216) EVENT_LOG_v1 {"time_micros": 1531288220077210, "job": 2940, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546837} >2018-07-11 05:50:20.077330 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:50:20.077223) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546837 bytes OK: base level 1 max bytes base 268435456 files[0 6 45 447 3642 790 0] max score 1.00 > >2018-07-11 05:50:20.093944 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:50:20.078151) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #32265 to level-5 74546836 bytes > >2018-07-11 05:50:20.093956 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:50:20.093851) EVENT_LOG_v1 {"time_micros": 1531288220093845, "job": 2941, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546836} >2018-07-11 05:50:20.093960 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:50:20.093859) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546836 bytes OK: base level 1 max bytes base 268435456 files[0 6 45 447 3641 791 0] max score 1.00 > >2018-07-11 05:50:20.110645 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:50:20.094851) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #36211 to level-4 74546049 bytes > >2018-07-11 05:50:20.110656 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:50:20.110552) EVENT_LOG_v1 {"time_micros": 1531288220110546, "job": 2942, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 74546049} >2018-07-11 05:50:20.110659 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:50:20.110560) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-4 74546049 bytes OK: base level 1 max bytes base 268435456 files[0 6 45 446 3642 791 0] max score 1.00 > >2018-07-11 05:50:20.127350 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:50:20.111575) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #31802 to level-5 74546836 bytes > >2018-07-11 05:50:20.127361 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:50:20.127256) EVENT_LOG_v1 {"time_micros": 1531288220127242, "job": 2943, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546836} >2018-07-11 05:50:20.127364 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:50:20.127265) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546836 bytes OK: base level 1 max bytes base 268435456 files[0 6 45 446 3641 792 0] max score 1.00 > >2018-07-11 05:50:37.406277 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:684] reusing log 36245 from recycle list > >2018-07-11 05:50:37.406500 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #36271. Immutable memtables: 0. > >2018-07-11 05:50:37.478393 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:50:37.406566) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1 >2018-07-11 05:50:37.478406 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 2944] Flushing memtable with next log file: 36271 > >2018-07-11 05:50:37.478423 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288237478415, "job": 2944, "event": "flush_started", "num_memtables": 1, "num_entries": 5000030, "num_deletes": 0, "memory_usage": 292998704} >2018-07-11 05:50:37.478426 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 2944] Level-0 flush table #36272: started >2018-07-11 05:50:41.383621 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288241383600, "cf_name": "default", "job": 2944, "event": "table_file_creation", "file_number": 36272, "file_size": 141349309, "table_properties": {"data_size": 127259798, "index_size": 1308619, "filter_size": 12779902, "raw_key_size": 144445808, "raw_average_key_size": 28, "raw_value_size": 59512017, "raw_average_value_size": 11, "num_data_blocks": 31257, "num_entries": 5000026, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:50:41.383643 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 2944] Level-0 flush table #36272: 141349309 bytes OK >2018-07-11 05:50:41.402135 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:242] adding log 36247 to recycle list > >2018-07-11 05:50:41.402147 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:50:41.383667) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #36272 started >2018-07-11 05:50:41.402150 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:50:41.401899) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #36272: memtable #1 done >2018-07-11 05:50:41.402152 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:50:41.401917) EVENT_LOG_v1 {"time_micros": 1531288241401910, "job": 2944, "event": "flush_finished", "lsm_state": [1, 6, 45, 446, 3641, 792, 0], "immutable_memtables": 0} >2018-07-11 05:50:41.402154 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:50:41.402080) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 6 45 446 3641 792 0] max score 1.00 > >2018-07-11 05:50:41.403033 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 2944] Try to delete WAL files size 178958801, prev total WAL file size 178958813, number of live WAL files 2. > >2018-07-11 05:51:55.474216 7f416d9fe700 1 osd.10 pg_epoch: 462 pg[10.1c( v 386'649 (0'0,386'649] local-lis/les=419/420 n=500 ec=193/193 lis/c 419/419 les/c/f 420/420/0 462/462/419) [11,4,8] r=-1 lpr=462 pi=[419,462)/1 luod=0'0 crt=386'649 lcod 0'0 active] start_peering_interval up [11,10,4] -> [11,4,8], acting [11,10,4] -> [11,4,8], acting_primary 11 -> 11, up_primary 11 -> 11, role 1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:51:55.474323 7f416d9fe700 1 osd.10 pg_epoch: 462 pg[10.1c( v 386'649 (0'0,386'649] local-lis/les=419/420 n=500 ec=193/193 lis/c 419/419 les/c/f 420/420/0 462/462/419) [11,4,8] r=-1 lpr=462 pi=[419,462)/1 crt=386'649 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:52:52.039145 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:684] reusing log 36247 from recycle list > >2018-07-11 05:52:52.039296 7f417420b700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #36273. Immutable memtables: 0. > >2018-07-11 05:52:52.096740 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:52:52.039359) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1 >2018-07-11 05:52:52.096754 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 2945] Flushing memtable with next log file: 36273 > >2018-07-11 05:52:52.096772 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288372096764, "job": 2945, "event": "flush_started", "num_memtables": 1, "num_entries": 5000065, "num_deletes": 0, "memory_usage": 293116624} >2018-07-11 05:52:52.096776 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 2945] Level-0 flush table #36274: started >2018-07-11 05:52:56.086728 7f41649ec700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288376086708, "cf_name": "default", "job": 2945, "event": "table_file_creation", "file_number": 36274, "file_size": 141465290, "table_properties": {"data_size": 127375227, "index_size": 1308835, "filter_size": 12780238, "raw_key_size": 144446996, "raw_average_key_size": 28, "raw_value_size": 59626368, "raw_average_value_size": 11, "num_data_blocks": 31265, "num_entries": 5000052, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "2"}} >2018-07-11 05:52:56.086767 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 2945] Level-0 flush table #36274: 141465290 bytes OK >2018-07-11 05:52:56.106864 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:242] adding log 36271 to recycle list > >2018-07-11 05:52:56.106876 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:52:56.086780) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #36274 started >2018-07-11 05:52:56.106893 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:52:56.106644) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #36274: memtable #1 done >2018-07-11 05:52:56.106895 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:52:56.106661) EVENT_LOG_v1 {"time_micros": 1531288376106654, "job": 2945, "event": "flush_finished", "lsm_state": [2, 6, 45, 446, 3641, 792, 0], "immutable_memtables": 0} >2018-07-11 05:52:56.106897 7f41649ec700 4 rocksdb: (Original Log Time 2018/07/11-05:52:56.106798) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[2 6 45 446 3641 792 0] max score 1.05 > >2018-07-11 05:52:56.107021 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2946] Compacting 2@0 + 6@1 files to L1, score 1.05 >2018-07-11 05:52:56.107040 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2945 Base level 0, inputs: [36274(134MB) 36272(134MB)], [36249(71MB) 36254(65MB) 36255(214KB) 36231(291KB) 36046(5093B) 36006(54MB)] > >2018-07-11 05:52:56.107058 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288376107044, "job": 2946, "event": "compaction_started", "files_L0": [36274, 36272], "files_L1": [36249, 36254, 36255, 36231, 36046, 36006], "score": 1.05357, "input_data_size": 482880202} >2018-07-11 05:52:56.107799 7f41649ec700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 2945] Try to delete WAL files size 179077160, prev total WAL file size 179088423, number of live WAL files 2. > >2018-07-11 05:53:00.071753 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2946] Generated table #36275: 2620556 keys, 74501299 bytes >2018-07-11 05:53:00.071787 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288380071772, "cf_name": "default", "job": 2946, "event": "table_file_creation", "file_number": 36275, "file_size": 74501299, "table_properties": {"data_size": 67110222, "index_size": 691104, "filter_size": 6698983, "raw_key_size": 75741479, "raw_average_key_size": 28, "raw_value_size": 31594581, "raw_average_value_size": 12, "num_data_blocks": 16466, "num_entries": 2620556, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:53:02.819456 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2946] Generated table #36276: 2638241 keys, 74546884 bytes >2018-07-11 05:53:02.819506 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288382819478, "cf_name": "default", "job": 2946, "event": "table_file_creation", "file_number": 36276, "file_size": 74546884, "table_properties": {"data_size": 67112113, "index_size": 690598, "filter_size": 6743183, "raw_key_size": 76215841, "raw_average_key_size": 28, "raw_value_size": 31365744, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:53:05.762514 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2946] Generated table #36277: 2638241 keys, 74546489 bytes >2018-07-11 05:53:05.762563 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288385762548, "cf_name": "default", "job": 2946, "event": "table_file_creation", "file_number": 36277, "file_size": 74546489, "table_properties": {"data_size": 67112152, "index_size": 690164, "filter_size": 6743183, "raw_key_size": 76215852, "raw_average_key_size": 28, "raw_value_size": 31365755, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:53:09.601814 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2946] Generated table #36278: 2638241 keys, 74546384 bytes >2018-07-11 05:53:09.601864 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288389601836, "cf_name": "default", "job": 2946, "event": "table_file_creation", "file_number": 36278, "file_size": 74546384, "table_properties": {"data_size": 67112088, "index_size": 690123, "filter_size": 6743183, "raw_key_size": 76215855, "raw_average_key_size": 28, "raw_value_size": 31365758, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:53:12.812365 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2946] Generated table #36279: 2087242 keys, 58977793 bytes >2018-07-11 05:53:12.812408 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288392812382, "cf_name": "default", "job": 2946, "event": "table_file_creation", "file_number": 36279, "file_size": 58977793, "table_properties": {"data_size": 53096410, "index_size": 545535, "filter_size": 5334860, "raw_key_size": 60298247, "raw_average_key_size": 28, "raw_value_size": 24815496, "raw_average_value_size": 11, "num_data_blocks": 13046, "num_entries": 2087242, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:53:14.240656 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2946] Generated table #36280: 165673 keys, 68241205 bytes >2018-07-11 05:53:14.240707 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288394240687, "cf_name": "default", "job": 2946, "event": "table_file_creation", "file_number": 36280, "file_size": 68241205, "table_properties": {"data_size": 67113315, "index_size": 559204, "filter_size": 567698, "raw_key_size": 15329964, "raw_average_key_size": 92, "raw_value_size": 53154183, "raw_average_value_size": 320, "num_data_blocks": 15099, "num_entries": 165673, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "80898", "kMergeOperands": "0"}} >2018-07-11 05:53:15.595060 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2946] Generated table #36281: 138976 keys, 57506872 bytes >2018-07-11 05:53:15.595097 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288395595079, "cf_name": "default", "job": 2946, "event": "table_file_creation", "file_number": 36281, "file_size": 57506872, "table_properties": {"data_size": 56545646, "index_size": 482395, "filter_size": 477843, "raw_key_size": 6816480, "raw_average_key_size": 49, "raw_value_size": 50241331, "raw_average_value_size": 361, "num_data_blocks": 13594, "num_entries": 138976, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "58266"}} >2018-07-11 05:53:15.595172 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2946] Compacted 2@0 + 6@1 files to L1 => 482866926 bytes >2018-07-11 05:53:15.614787 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:53:15.614508) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 7 45 446 3641 792 0] max score 1.80, MB/sec: 24.8 rd, 24.8 wr, level 1, files in(2, 6) out(7) MB in(269.7, 190.8) out(460.5), read-write-amplify(3.4) write-amplify(1.7) OK, records in: 12927189, records dropped: 18 > >2018-07-11 05:53:15.614798 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:53:15.614534) EVENT_LOG_v1 {"time_micros": 1531288395614522, "job": 2946, "event": "compaction_finished", "compaction_time_micros": 19488094, "output_level": 1, "num_output_files": 7, "total_output_size": 482866926, "num_input_records": 12927188, "num_output_records": 12927170, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 7, 45, 446, 3641, 792, 0]} >2018-07-11 05:53:15.615767 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288395615750, "job": 2946, "event": "table_file_deletion", "file_number": 36274} >2018-07-11 05:53:15.615782 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288395615779, "job": 2946, "event": "table_file_deletion", "file_number": 36272} >2018-07-11 05:53:15.615794 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288395615792, "job": 2946, "event": "table_file_deletion", "file_number": 36255} >2018-07-11 05:53:15.615806 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288395615803, "job": 2946, "event": "table_file_deletion", "file_number": 36254} >2018-07-11 05:53:15.615817 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288395615815, "job": 2946, "event": "table_file_deletion", "file_number": 36249} >2018-07-11 05:53:15.615832 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288395615829, "job": 2946, "event": "table_file_deletion", "file_number": 36231} >2018-07-11 05:53:15.615846 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288395615844, "job": 2946, "event": "table_file_deletion", "file_number": 36046} >2018-07-11 05:53:15.615859 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288395615857, "job": 2946, "event": "table_file_deletion", "file_number": 36006} >2018-07-11 05:53:15.616098 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_picker.cc:461] [default] Expanding@1 1+6(74546884+381867412 bytes) to 4+6 (282617550+381867412bytes) > >2018-07-11 05:53:15.616201 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2947] Compacting 4@1 + 6@2 files to L2, score 1.80 >2018-07-11 05:53:15.616214 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2946 Base level 1, inputs: [36276(71MB) 36277(71MB) 36278(71MB) 36279(56MB)], [36256(71MB) 36257(71MB) 36258(71MB) 36259(71MB) 36260(71MB) 36265(8922KB)] > >2018-07-11 05:53:15.616229 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288395616218, "job": 2947, "event": "compaction_started", "files_L1": [36276, 36277, 36278, 36279], "files_L2": [36256, 36257, 36258, 36259, 36260, 36265], "score": 1.79882, "input_data_size": 664484962} >2018-07-11 05:53:19.638476 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2947] Generated table #36282: 2638239 keys, 74546241 bytes >2018-07-11 05:53:19.638511 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288399638497, "cf_name": "default", "job": 2947, "event": "table_file_creation", "file_number": 36282, "file_size": 74546241, "table_properties": {"data_size": 67112120, "index_size": 689952, "filter_size": 6743179, "raw_key_size": 76215780, "raw_average_key_size": 28, "raw_value_size": 31365717, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638239, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:53:23.667439 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2947] Generated table #36283: 2638239 keys, 74546286 bytes >2018-07-11 05:53:23.667494 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288403667478, "cf_name": "default", "job": 2947, "event": "table_file_creation", "file_number": 36283, "file_size": 74546286, "table_properties": {"data_size": 67112111, "index_size": 690006, "filter_size": 6743179, "raw_key_size": 76215797, "raw_average_key_size": 28, "raw_value_size": 31365734, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638239, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:53:27.650263 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2947] Generated table #36284: 2638241 keys, 74546343 bytes >2018-07-11 05:53:27.650308 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288407650282, "cf_name": "default", "job": 2947, "event": "table_file_creation", "file_number": 36284, "file_size": 74546343, "table_properties": {"data_size": 67112155, "index_size": 690014, "filter_size": 6743184, "raw_key_size": 76215849, "raw_average_key_size": 28, "raw_value_size": 31365752, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:53:31.662173 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2947] Generated table #36285: 2638241 keys, 74545910 bytes >2018-07-11 05:53:31.662226 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288411662209, "cf_name": "default", "job": 2947, "event": "table_file_creation", "file_number": 36285, "file_size": 74545910, "table_properties": {"data_size": 67112167, "index_size": 689570, "filter_size": 6743183, "raw_key_size": 76215859, "raw_average_key_size": 28, "raw_value_size": 31365762, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:53:36.129578 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2947] Generated table #36286: 2638241 keys, 74545918 bytes >2018-07-11 05:53:36.129622 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288416129605, "cf_name": "default", "job": 2947, "event": "table_file_creation", "file_number": 36286, "file_size": 74545918, "table_properties": {"data_size": 67112150, "index_size": 689595, "filter_size": 6743183, "raw_key_size": 76215855, "raw_average_key_size": 28, "raw_value_size": 31365758, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:53:39.635212 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2947] Generated table #36287: 2638240 keys, 74546278 bytes >2018-07-11 05:53:39.635267 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288419635250, "cf_name": "default", "job": 2947, "event": "table_file_creation", "file_number": 36287, "file_size": 74546278, "table_properties": {"data_size": 67112152, "index_size": 689955, "filter_size": 6743181, "raw_key_size": 76215831, "raw_average_key_size": 28, "raw_value_size": 31365751, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:53:42.925642 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2947] Generated table #36288: 2638241 keys, 74546372 bytes >2018-07-11 05:53:42.925696 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288422925668, "cf_name": "default", "job": 2947, "event": "table_file_creation", "file_number": 36288, "file_size": 74546372, "table_properties": {"data_size": 67112142, "index_size": 690057, "filter_size": 6743183, "raw_key_size": 76215850, "raw_average_key_size": 28, "raw_value_size": 31365753, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:53:46.248358 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2947] Generated table #36289: 2638241 keys, 74545900 bytes >2018-07-11 05:53:46.248403 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288426248375, "cf_name": "default", "job": 2947, "event": "table_file_creation", "file_number": 36289, "file_size": 74545900, "table_properties": {"data_size": 67112134, "index_size": 689593, "filter_size": 6743183, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:53:49.285685 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2947] Generated table #36290: 2406128 keys, 68114896 bytes >2018-07-11 05:53:49.285729 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288429285702, "cf_name": "default", "job": 2947, "event": "table_file_creation", "file_number": 36290, "file_size": 68114896, "table_properties": {"data_size": 61332704, "index_size": 631002, "filter_size": 6150201, "raw_key_size": 69526226, "raw_average_key_size": 28, "raw_value_size": 28711967, "raw_average_value_size": 11, "num_data_blocks": 15071, "num_entries": 2406128, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "11", "kMergeOperands": "0"}} >2018-07-11 05:53:49.285767 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2947] Compacted 4@1 + 6@2 files to L2 => 664484144 bytes >2018-07-11 05:53:49.305919 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:53:49.305522) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 3 48 446 3641 792 0] max score 1.10, MB/sec: 19.7 rd, 19.7 wr, level 2, files in(4, 6) out(9) MB in(269.5, 364.2) out(633.7), read-write-amplify(4.7) write-amplify(2.4) OK, records in: 23512052, records dropped: 1 > >2018-07-11 05:53:49.305932 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:53:49.305549) EVENT_LOG_v1 {"time_micros": 1531288429305537, "job": 2947, "event": "compaction_finished", "compaction_time_micros": 33669517, "output_level": 2, "num_output_files": 9, "total_output_size": 664484144, "num_input_records": 23512052, "num_output_records": 23512051, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 3, 48, 446, 3641, 792, 0]} >2018-07-11 05:53:49.306898 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288429306892, "job": 2947, "event": "table_file_deletion", "file_number": 36279} >2018-07-11 05:53:49.306921 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288429306918, "job": 2947, "event": "table_file_deletion", "file_number": 36278} >2018-07-11 05:53:49.306947 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288429306943, "job": 2947, "event": "table_file_deletion", "file_number": 36277} >2018-07-11 05:53:49.306971 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288429306968, "job": 2947, "event": "table_file_deletion", "file_number": 36276} >2018-07-11 05:53:49.307000 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288429306996, "job": 2947, "event": "table_file_deletion", "file_number": 36265} >2018-07-11 05:53:49.307026 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288429307023, "job": 2947, "event": "table_file_deletion", "file_number": 36260} >2018-07-11 05:53:49.307050 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288429307046, "job": 2947, "event": "table_file_deletion", "file_number": 36259} >2018-07-11 05:53:49.307073 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288429307069, "job": 2947, "event": "table_file_deletion", "file_number": 36258} >2018-07-11 05:53:49.307091 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288429307088, "job": 2947, "event": "table_file_deletion", "file_number": 36257} >2018-07-11 05:53:49.307117 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288429307113, "job": 2947, "event": "table_file_deletion", "file_number": 36256} >2018-07-11 05:53:49.307350 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_picker.cc:461] [default] Expanding@2 1+1(74546372+46962876 bytes) to 2+1 (149092272+46962876bytes) > >2018-07-11 05:53:49.307453 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2948] Compacting 2@2 + 1@3 files to L3, score 1.10 >2018-07-11 05:53:49.307466 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2947 Base level 2, inputs: [36288(71MB) 36289(71MB)], [36270(44MB)] > >2018-07-11 05:53:49.307484 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288429307471, "job": 2948, "event": "compaction_started", "files_L2": [36288, 36289], "files_L3": [36270], "score": 1.09931, "input_data_size": 196055148} >2018-07-11 05:53:53.744367 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2948] Generated table #36291: 2638241 keys, 74546663 bytes >2018-07-11 05:53:53.744409 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288433744393, "cf_name": "default", "job": 2948, "event": "table_file_creation", "file_number": 36291, "file_size": 74546663, "table_properties": {"data_size": 67112125, "index_size": 690365, "filter_size": 6743183, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:53:57.250605 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2948] Generated table #36292: 2638241 keys, 74546475 bytes >2018-07-11 05:53:57.250646 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288437250627, "cf_name": "default", "job": 2948, "event": "table_file_creation", "file_number": 36292, "file_size": 74546475, "table_properties": {"data_size": 67112161, "index_size": 690141, "filter_size": 6743183, "raw_key_size": 76215848, "raw_average_key_size": 28, "raw_value_size": 31365751, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:53:59.540028 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2948] Generated table #36293: 1519835 keys, 46962818 bytes >2018-07-11 05:53:59.540068 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288439540049, "cf_name": "default", "job": 2948, "event": "table_file_creation", "file_number": 36293, "file_size": 46962818, "table_properties": {"data_size": 42634532, "index_size": 433364, "filter_size": 3893934, "raw_key_size": 44346596, "raw_average_key_size": 29, "raw_value_size": 21521502, "raw_average_value_size": 14, "num_data_blocks": 10537, "num_entries": 1519835, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:53:59.540109 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 2948] Compacted 2@2 + 1@3 files to L3 => 196055956 bytes >2018-07-11 05:53:59.555495 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:53:59.555303) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 3 46 448 3641 792 0] max score 1.04, MB/sec: 19.2 rd, 19.2 wr, level 3, files in(2, 1) out(3) MB in(142.2, 44.8) out(187.0), read-write-amplify(2.6) write-amplify(1.3) OK, records in: 6796317, records dropped: 0 > >2018-07-11 05:53:59.555505 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:53:59.555327) EVENT_LOG_v1 {"time_micros": 1531288439555316, "job": 2948, "event": "compaction_finished", "compaction_time_micros": 10232609, "output_level": 3, "num_output_files": 3, "total_output_size": 196055956, "num_input_records": 6796317, "num_output_records": 6796317, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 3, 46, 448, 3641, 792, 0]} >2018-07-11 05:53:59.556473 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288439556468, "job": 2948, "event": "table_file_deletion", "file_number": 36289} >2018-07-11 05:53:59.556500 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288439556496, "job": 2948, "event": "table_file_deletion", "file_number": 36288} >2018-07-11 05:53:59.556528 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288439556524, "job": 2948, "event": "table_file_deletion", "file_number": 36270} >2018-07-11 05:53:59.556863 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 2949] Compacting 1@2 + 1@3 files to L3, score 1.04 >2018-07-11 05:53:59.556876 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2948 Base level 2, inputs: [36284(71MB)], [35741(71MB)] > >2018-07-11 05:53:59.556895 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288439556881, "job": 2949, "event": "compaction_started", "files_L2": [36284], "files_L3": [35741], "score": 1.04377, "input_data_size": 149092290} >2018-07-11 05:53:59.943680 7f41651ed700 -1 received signal: Terminated from PID: 1 task name: /usr/lib/systemd/systemd --switched-root --system --deserialize 22 UID: 0 >2018-07-11 05:53:59.943703 7f41651ed700 -1 osd.10 463 *** Got signal Terminated *** >2018-07-11 05:53:59.943714 7f41651ed700 0 osd.10 463 prepare_to_stop telling mon we are shutting down >2018-07-11 05:54:00.095019 7f41782b3700 0 osd.10 463 got_stop_ack starting shutdown >2018-07-11 05:54:00.095036 7f41651ed700 0 osd.10 463 prepare_to_stop starting shutdown >2018-07-11 05:54:00.095042 7f41651ed700 -1 osd.10 463 shutdown >2018-07-11 05:54:02.804497 7f41651ed700 1 bluestore(/var/lib/ceph/osd/ceph-10) umount >2018-07-11 05:54:03.279780 7f41641eb700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 2949] Generated table #36294: 2638240 keys, 74546634 bytes >2018-07-11 05:54:03.279839 7f41641eb700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288443279821, "cf_name": "default", "job": 2949, "event": "table_file_creation", "file_number": 36294, "file_size": 74546634, "table_properties": {"data_size": 67112117, "index_size": 690346, "filter_size": 6743181, "raw_key_size": 76215817, "raw_average_key_size": 28, "raw_value_size": 31365737, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:54:04.045271 7f41651ed700 1 stupidalloc 0x0x562b86e6d0a0 shutdown >2018-07-11 05:54:04.045966 7f41651ed700 1 freelist shutdown >2018-07-11 05:54:04.046012 7f41651ed700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:217] Shutdown: canceling all background work >2018-07-11 05:54:04.355445 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:54:04.355238) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 3 46 448 3641 792 0] max score 1.02, MB/sec: 31.1 rd, 20.7 wr, level 3, files in(1, 1) out(2) MB in(71.1, 71.1) out(94.5), read-write-amplify(3.3) write-amplify(1.3) Shutdown in progress: Database shutdown or Column family drop during compaction, records in: 5276482, records dropped: 0 > >2018-07-11 05:54:04.355458 7f41641eb700 4 rocksdb: (Original Log Time 2018/07/11-05:54:04.355312) EVENT_LOG_v1 {"time_micros": 1531288444355270, "job": 2949, "event": "compaction_finished", "compaction_time_micros": 4798299, "output_level": 3, "num_output_files": 2, "total_output_size": 99109813, "num_input_records": 3603842, "num_output_records": 3603842, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 3, 46, 448, 3641, 792, 0]} >2018-07-11 05:54:04.359800 7f41651ed700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288444359793, "job": 2950, "event": "table_file_deletion", "file_number": 36295} >2018-07-11 05:54:04.359827 7f41651ed700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288444359822, "job": 2950, "event": "table_file_deletion", "file_number": 36294} >2018-07-11 05:54:04.394985 7f41651ed700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:343] Shutdown complete >2018-07-11 05:54:04.398437 7f41651ed700 1 bluefs umount >2018-07-11 05:54:04.405281 7f41651ed700 1 stupidalloc 0x0x562b86c95f10 shutdown >2018-07-11 05:54:04.405293 7f41651ed700 1 stupidalloc 0x0x562b86c95f80 shutdown >2018-07-11 05:54:04.405295 7f41651ed700 1 stupidalloc 0x0x562b86e6c000 shutdown >2018-07-11 05:54:04.407705 7f41651ed700 1 bdev(0x562b86d70200 /var/lib/ceph/osd/ceph-10/block.wal) close >2018-07-11 05:54:04.653674 7f41651ed700 1 bdev(0x562b86d71c00 /var/lib/ceph/osd/ceph-10/block.db) close >2018-07-11 05:54:04.799651 7f41651ed700 1 bdev(0x562b86d71800 /var/lib/ceph/osd/ceph-10/block) close >2018-07-11 05:54:04.993328 7f41651ed700 1 bdev(0x562b86d71a00 /var/lib/ceph/osd/ceph-10/block) close >2018-07-11 05:54:05.341447 7f3562aa9d80 0 set uid:gid to 167:167 (ceph:ceph) >2018-07-11 05:54:05.341468 7f3562aa9d80 0 ceph version 12.2.5-27.el7cp (6b7d1a8cf2e2376fd0a12424947c8c97aa1c0890) luminous (stable), process (unknown), pid 12665 >2018-07-11 05:54:05.347503 7f3562aa9d80 0 pidfile_write: ignore empty --pid-file >2018-07-11 05:54:05.384486 7f3562aa9d80 0 load: jerasure load: lrc load: isa >2018-07-11 05:54:05.384604 7f3562aa9d80 1 bdev create path /var/lib/ceph/osd/ceph-10/block type kernel >2018-07-11 05:54:05.384616 7f3562aa9d80 1 bdev(0x55a30f668e00 /var/lib/ceph/osd/ceph-10/block) open path /var/lib/ceph/osd/ceph-10/block >2018-07-11 05:54:05.384919 7f3562aa9d80 1 bdev(0x55a30f668e00 /var/lib/ceph/osd/ceph-10/block) open size 1000098959360 (0xe8da8b1000, 931 GB) block_size 4096 (4096 B) rotational >2018-07-11 05:54:05.385163 7f3562aa9d80 1 bluestore(/var/lib/ceph/osd/ceph-10) _set_cache_sizes max 0.5 < ratio 0.99 >2018-07-11 05:54:05.385199 7f3562aa9d80 1 bluestore(/var/lib/ceph/osd/ceph-10) _set_cache_sizes cache_size 1073741824 meta 0.5 kv 0.5 data 0 >2018-07-11 05:54:05.385208 7f3562aa9d80 1 bdev(0x55a30f668e00 /var/lib/ceph/osd/ceph-10/block) close >2018-07-11 05:54:05.639875 7f3562aa9d80 1 bluestore(/var/lib/ceph/osd/ceph-10) _mount path /var/lib/ceph/osd/ceph-10 >2018-07-11 05:54:05.640161 7f3562aa9d80 1 bdev create path /var/lib/ceph/osd/ceph-10/block type kernel >2018-07-11 05:54:05.640169 7f3562aa9d80 1 bdev(0x55a30f669a00 /var/lib/ceph/osd/ceph-10/block) open path /var/lib/ceph/osd/ceph-10/block >2018-07-11 05:54:05.640390 7f3562aa9d80 1 bdev(0x55a30f669a00 /var/lib/ceph/osd/ceph-10/block) open size 1000098959360 (0xe8da8b1000, 931 GB) block_size 4096 (4096 B) rotational >2018-07-11 05:54:05.640628 7f3562aa9d80 1 bluestore(/var/lib/ceph/osd/ceph-10) _set_cache_sizes max 0.5 < ratio 0.99 >2018-07-11 05:54:05.640642 7f3562aa9d80 1 bluestore(/var/lib/ceph/osd/ceph-10) _set_cache_sizes cache_size 1073741824 meta 0.5 kv 0.5 data 0 >2018-07-11 05:54:05.640730 7f3562aa9d80 1 bdev create path /var/lib/ceph/osd/ceph-10/block.db type kernel >2018-07-11 05:54:05.640738 7f3562aa9d80 1 bdev(0x55a30f669c00 /var/lib/ceph/osd/ceph-10/block.db) open path /var/lib/ceph/osd/ceph-10/block.db >2018-07-11 05:54:05.640940 7f3562aa9d80 1 bdev(0x55a30f669c00 /var/lib/ceph/osd/ceph-10/block.db) open size 1073741824 (0x40000000, 1024 MB) block_size 4096 (4096 B) rotational >2018-07-11 05:54:05.640952 7f3562aa9d80 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-10/block.db size 1024 MB >2018-07-11 05:54:05.648098 7f3562aa9d80 1 bdev create path /var/lib/ceph/osd/ceph-10/block type kernel >2018-07-11 05:54:05.648105 7f3562aa9d80 1 bdev(0x55a30f669800 /var/lib/ceph/osd/ceph-10/block) open path /var/lib/ceph/osd/ceph-10/block >2018-07-11 05:54:05.648308 7f3562aa9d80 1 bdev(0x55a30f669800 /var/lib/ceph/osd/ceph-10/block) open size 1000098959360 (0xe8da8b1000, 931 GB) block_size 4096 (4096 B) rotational >2018-07-11 05:54:05.648317 7f3562aa9d80 1 bluefs add_block_device bdev 2 path /var/lib/ceph/osd/ceph-10/block size 931 GB >2018-07-11 05:54:05.648358 7f3562aa9d80 1 bdev create path /var/lib/ceph/osd/ceph-10/block.wal type kernel >2018-07-11 05:54:05.648363 7f3562aa9d80 1 bdev(0x55a30f668200 /var/lib/ceph/osd/ceph-10/block.wal) open path /var/lib/ceph/osd/ceph-10/block.wal >2018-07-11 05:54:05.648538 7f3562aa9d80 1 bdev(0x55a30f668200 /var/lib/ceph/osd/ceph-10/block.wal) open size 603979776 (0x24000000, 576 MB) block_size 4096 (4096 B) rotational >2018-07-11 05:54:05.648546 7f3562aa9d80 1 bluefs add_block_device bdev 0 path /var/lib/ceph/osd/ceph-10/block.wal size 576 MB >2018-07-11 05:54:05.653287 7f3562aa9d80 1 bluefs mount >2018-07-11 05:54:05.750336 7f3562aa9d80 0 set rocksdb option compaction_readahead_size = 2097152 >2018-07-11 05:54:05.750352 7f3562aa9d80 0 set rocksdb option compression = kNoCompression >2018-07-11 05:54:05.750370 7f3562aa9d80 0 set rocksdb option max_write_buffer_number = 4 >2018-07-11 05:54:05.750374 7f3562aa9d80 0 set rocksdb option min_write_buffer_number_to_merge = 1 >2018-07-11 05:54:05.750377 7f3562aa9d80 0 set rocksdb option recycle_log_file_num = 4 >2018-07-11 05:54:05.750380 7f3562aa9d80 0 set rocksdb option writable_file_max_buffer_size = 0 >2018-07-11 05:54:05.750386 7f3562aa9d80 0 set rocksdb option write_buffer_size = 268435456 >2018-07-11 05:54:05.750411 7f3562aa9d80 0 set rocksdb option compaction_readahead_size = 2097152 >2018-07-11 05:54:05.750415 7f3562aa9d80 0 set rocksdb option compression = kNoCompression >2018-07-11 05:54:05.750419 7f3562aa9d80 0 set rocksdb option max_write_buffer_number = 4 >2018-07-11 05:54:05.750422 7f3562aa9d80 0 set rocksdb option min_write_buffer_number_to_merge = 1 >2018-07-11 05:54:05.750426 7f3562aa9d80 0 set rocksdb option recycle_log_file_num = 4 >2018-07-11 05:54:05.750429 7f3562aa9d80 0 set rocksdb option writable_file_max_buffer_size = 0 >2018-07-11 05:54:05.750432 7f3562aa9d80 0 set rocksdb option write_buffer_size = 268435456 >2018-07-11 05:54:05.750593 7f3562aa9d80 4 rocksdb: RocksDB version: 5.4.0 > >2018-07-11 05:54:05.750603 7f3562aa9d80 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ >2018-07-11 05:54:05.750604 7f3562aa9d80 4 rocksdb: Compile date Jun 28 2018 >2018-07-11 05:54:05.750608 7f3562aa9d80 4 rocksdb: DB SUMMARY > >2018-07-11 05:54:05.750623 7f3562aa9d80 4 rocksdb: CURRENT file: CURRENT > >2018-07-11 05:54:05.750629 7f3562aa9d80 4 rocksdb: IDENTITY file: IDENTITY > >2018-07-11 05:54:05.750639 7f3562aa9d80 4 rocksdb: MANIFEST file: MANIFEST-029828 size: 1022659 Bytes > >2018-07-11 05:54:05.750643 7f3562aa9d80 4 rocksdb: SST files in db dir, Total Num: 3, files: 036275.sst 036280.sst 036281.sst > >2018-07-11 05:54:05.753615 7f3562aa9d80 4 rocksdb: SST files in db.slow dir, Total Num: 4927, files: 000549.sst 000550.sst 000565.sst 000566.sst 000605.sst 000617.sst 000620.sst 000622.sst 000634.sst > >2018-07-11 05:54:05.753686 7f3562aa9d80 4 rocksdb: Write Ahead Log file in db.wal: 036271.log size: 179137682 ; 036273.log size: 107440091 ; > >2018-07-11 05:54:05.753692 7f3562aa9d80 4 rocksdb: Options.error_if_exists: 0 >2018-07-11 05:54:05.753693 7f3562aa9d80 4 rocksdb: Options.create_if_missing: 0 >2018-07-11 05:54:05.753694 7f3562aa9d80 4 rocksdb: Options.paranoid_checks: 1 >2018-07-11 05:54:05.753695 7f3562aa9d80 4 rocksdb: Options.env: 0x55a30f660f60 >2018-07-11 05:54:05.753696 7f3562aa9d80 4 rocksdb: Options.info_log: 0x55a30f9fa8c0 >2018-07-11 05:54:05.753697 7f3562aa9d80 4 rocksdb: Options.max_open_files: -1 >2018-07-11 05:54:05.753698 7f3562aa9d80 4 rocksdb: Options.max_file_opening_threads: 16 >2018-07-11 05:54:05.753699 7f3562aa9d80 4 rocksdb: Options.use_fsync: 0 >2018-07-11 05:54:05.753699 7f3562aa9d80 4 rocksdb: Options.max_log_file_size: 0 >2018-07-11 05:54:05.753700 7f3562aa9d80 4 rocksdb: Options.max_manifest_file_size: 18446744073709551615 >2018-07-11 05:54:05.753701 7f3562aa9d80 4 rocksdb: Options.log_file_time_to_roll: 0 >2018-07-11 05:54:05.753702 7f3562aa9d80 4 rocksdb: Options.keep_log_file_num: 1000 >2018-07-11 05:54:05.753703 7f3562aa9d80 4 rocksdb: Options.recycle_log_file_num: 4 >2018-07-11 05:54:05.753703 7f3562aa9d80 4 rocksdb: Options.allow_fallocate: 1 >2018-07-11 05:54:05.753704 7f3562aa9d80 4 rocksdb: Options.allow_mmap_reads: 0 >2018-07-11 05:54:05.753705 7f3562aa9d80 4 rocksdb: Options.allow_mmap_writes: 0 >2018-07-11 05:54:05.753706 7f3562aa9d80 4 rocksdb: Options.use_direct_reads: 0 >2018-07-11 05:54:05.753706 7f3562aa9d80 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 >2018-07-11 05:54:05.753707 7f3562aa9d80 4 rocksdb: Options.create_missing_column_families: 0 >2018-07-11 05:54:05.753708 7f3562aa9d80 4 rocksdb: Options.db_log_dir: >2018-07-11 05:54:05.753709 7f3562aa9d80 4 rocksdb: Options.wal_dir: db.wal >2018-07-11 05:54:05.753709 7f3562aa9d80 4 rocksdb: Options.table_cache_numshardbits: 6 >2018-07-11 05:54:05.753710 7f3562aa9d80 4 rocksdb: Options.max_subcompactions: 1 >2018-07-11 05:54:05.753711 7f3562aa9d80 4 rocksdb: Options.max_background_flushes: 1 >2018-07-11 05:54:05.753716 7f3562aa9d80 4 rocksdb: Options.WAL_ttl_seconds: 0 >2018-07-11 05:54:05.753717 7f3562aa9d80 4 rocksdb: Options.WAL_size_limit_MB: 0 >2018-07-11 05:54:05.753717 7f3562aa9d80 4 rocksdb: Options.manifest_preallocation_size: 4194304 >2018-07-11 05:54:05.753718 7f3562aa9d80 4 rocksdb: Options.is_fd_close_on_exec: 1 >2018-07-11 05:54:05.753719 7f3562aa9d80 4 rocksdb: Options.advise_random_on_open: 1 >2018-07-11 05:54:05.753720 7f3562aa9d80 4 rocksdb: Options.db_write_buffer_size: 0 >2018-07-11 05:54:05.753720 7f3562aa9d80 4 rocksdb: Options.access_hint_on_compaction_start: 1 >2018-07-11 05:54:05.753721 7f3562aa9d80 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 1 >2018-07-11 05:54:05.753722 7f3562aa9d80 4 rocksdb: Options.compaction_readahead_size: 2097152 >2018-07-11 05:54:05.753722 7f3562aa9d80 4 rocksdb: Options.random_access_max_buffer_size: 1048576 >2018-07-11 05:54:05.753723 7f3562aa9d80 4 rocksdb: Options.writable_file_max_buffer_size: 0 >2018-07-11 05:54:05.753724 7f3562aa9d80 4 rocksdb: Options.use_adaptive_mutex: 0 >2018-07-11 05:54:05.753725 7f3562aa9d80 4 rocksdb: Options.rate_limiter: (nil) >2018-07-11 05:54:05.753726 7f3562aa9d80 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 >2018-07-11 05:54:05.753726 7f3562aa9d80 4 rocksdb: Options.bytes_per_sync: 0 >2018-07-11 05:54:05.753727 7f3562aa9d80 4 rocksdb: Options.wal_bytes_per_sync: 0 >2018-07-11 05:54:05.753728 7f3562aa9d80 4 rocksdb: Options.wal_recovery_mode: 0 >2018-07-11 05:54:05.753728 7f3562aa9d80 4 rocksdb: Options.enable_thread_tracking: 0 >2018-07-11 05:54:05.753729 7f3562aa9d80 4 rocksdb: Options.allow_concurrent_memtable_write: 1 >2018-07-11 05:54:05.753730 7f3562aa9d80 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 >2018-07-11 05:54:05.753731 7f3562aa9d80 4 rocksdb: Options.write_thread_max_yield_usec: 100 >2018-07-11 05:54:05.753731 7f3562aa9d80 4 rocksdb: Options.write_thread_slow_yield_usec: 3 >2018-07-11 05:54:05.753732 7f3562aa9d80 4 rocksdb: Options.row_cache: None >2018-07-11 05:54:05.753733 7f3562aa9d80 4 rocksdb: Options.wal_filter: None >2018-07-11 05:54:05.753734 7f3562aa9d80 4 rocksdb: Options.avoid_flush_during_recovery: 0 >2018-07-11 05:54:05.753740 7f3562aa9d80 4 rocksdb: Options.base_background_compactions: 1 >2018-07-11 05:54:05.753741 7f3562aa9d80 4 rocksdb: Options.max_background_compactions: 1 >2018-07-11 05:54:05.753742 7f3562aa9d80 4 rocksdb: Options.avoid_flush_during_shutdown: 0 >2018-07-11 05:54:05.753742 7f3562aa9d80 4 rocksdb: Options.delayed_write_rate : 16777216 >2018-07-11 05:54:05.753743 7f3562aa9d80 4 rocksdb: Options.max_total_wal_size: 0 >2018-07-11 05:54:05.753744 7f3562aa9d80 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 >2018-07-11 05:54:05.753745 7f3562aa9d80 4 rocksdb: Options.stats_dump_period_sec: 600 >2018-07-11 05:54:05.753746 7f3562aa9d80 4 rocksdb: Compression algorithms supported: >2018-07-11 05:54:05.753749 7f3562aa9d80 4 rocksdb: Snappy supported: 0 >2018-07-11 05:54:05.753750 7f3562aa9d80 4 rocksdb: Zlib supported: 0 >2018-07-11 05:54:05.753750 7f3562aa9d80 4 rocksdb: Bzip supported: 0 >2018-07-11 05:54:05.753751 7f3562aa9d80 4 rocksdb: LZ4 supported: 0 >2018-07-11 05:54:05.753752 7f3562aa9d80 4 rocksdb: ZSTD supported: 0 >2018-07-11 05:54:05.753753 7f3562aa9d80 4 rocksdb: Fast CRC32 supported: 1 >2018-07-11 05:54:05.764444 7f3562aa9d80 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/version_set.cc:2609] Recovering from manifest file: MANIFEST-029828 > >2018-07-11 05:54:05.764506 7f3562aa9d80 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:407] --------------- Options for column family [default]: > >2018-07-11 05:54:05.764509 7f3562aa9d80 4 rocksdb: Options.comparator: leveldb.BytewiseComparator >2018-07-11 05:54:05.764537 7f3562aa9d80 4 rocksdb: Options.merge_operator: .T:int64_array.b:bitwise_xor >2018-07-11 05:54:05.764539 7f3562aa9d80 4 rocksdb: Options.compaction_filter: None >2018-07-11 05:54:05.764542 7f3562aa9d80 4 rocksdb: Options.compaction_filter_factory: None >2018-07-11 05:54:05.764543 7f3562aa9d80 4 rocksdb: Options.memtable_factory: SkipListFactory >2018-07-11 05:54:05.764544 7f3562aa9d80 4 rocksdb: Options.table_factory: BlockBasedTable >2018-07-11 05:54:05.764557 7f3562aa9d80 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55a30f300228) > cache_index_and_filter_blocks: 1 > cache_index_and_filter_blocks_with_high_priority: 1 > pin_l0_filter_and_index_blocks_in_cache: 1 > index_type: 0 > hash_index_allow_collision: 1 > checksum: 1 > no_block_cache: 0 > block_cache: 0x55a30f764328 > block_cache_name: LRUCache > block_cache_options: > capacity : 536870912 > num_shard_bits : 4 > strict_capacity_limit : 0 > high_pri_pool_ratio: 0.000 > block_cache_compressed: (nil) > persistent_cache: (nil) > block_size: 4096 > block_size_deviation: 10 > block_restart_interval: 16 > index_block_restart_interval: 1 > filter_policy: rocksdb.BuiltinBloomFilter > whole_key_filtering: 1 > format_version: 2 > >2018-07-11 05:54:05.764561 7f3562aa9d80 4 rocksdb: Options.write_buffer_size: 268435456 >2018-07-11 05:54:05.764562 7f3562aa9d80 4 rocksdb: Options.max_write_buffer_number: 4 >2018-07-11 05:54:05.764564 7f3562aa9d80 4 rocksdb: Options.compression: NoCompression >2018-07-11 05:54:05.764565 7f3562aa9d80 4 rocksdb: Options.bottommost_compression: Disabled >2018-07-11 05:54:05.764566 7f3562aa9d80 4 rocksdb: Options.prefix_extractor: nullptr >2018-07-11 05:54:05.764567 7f3562aa9d80 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr >2018-07-11 05:54:05.764567 7f3562aa9d80 4 rocksdb: Options.num_levels: 7 >2018-07-11 05:54:05.764568 7f3562aa9d80 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 >2018-07-11 05:54:05.764569 7f3562aa9d80 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 >2018-07-11 05:54:05.764570 7f3562aa9d80 4 rocksdb: Options.compression_opts.window_bits: -14 >2018-07-11 05:54:05.764571 7f3562aa9d80 4 rocksdb: Options.compression_opts.level: -1 >2018-07-11 05:54:05.764571 7f3562aa9d80 4 rocksdb: Options.compression_opts.strategy: 0 >2018-07-11 05:54:05.764572 7f3562aa9d80 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 >2018-07-11 05:54:05.764573 7f3562aa9d80 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 >2018-07-11 05:54:05.764573 7f3562aa9d80 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 >2018-07-11 05:54:05.764574 7f3562aa9d80 4 rocksdb: Options.level0_stop_writes_trigger: 36 >2018-07-11 05:54:05.764575 7f3562aa9d80 4 rocksdb: Options.target_file_size_base: 67108864 >2018-07-11 05:54:05.764576 7f3562aa9d80 4 rocksdb: Options.target_file_size_multiplier: 1 >2018-07-11 05:54:05.764579 7f3562aa9d80 4 rocksdb: Options.max_bytes_for_level_base: 268435456 >2018-07-11 05:54:05.764579 7f3562aa9d80 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0 >2018-07-11 05:54:05.764580 7f3562aa9d80 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 >2018-07-11 05:54:05.764583 7f3562aa9d80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 >2018-07-11 05:54:05.764584 7f3562aa9d80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 >2018-07-11 05:54:05.764585 7f3562aa9d80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 >2018-07-11 05:54:05.764585 7f3562aa9d80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 >2018-07-11 05:54:05.764586 7f3562aa9d80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 >2018-07-11 05:54:05.764587 7f3562aa9d80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 >2018-07-11 05:54:05.764588 7f3562aa9d80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 >2018-07-11 05:54:05.764589 7f3562aa9d80 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 >2018-07-11 05:54:05.764590 7f3562aa9d80 4 rocksdb: Options.max_compaction_bytes: 1677721600 >2018-07-11 05:54:05.764591 7f3562aa9d80 4 rocksdb: Options.arena_block_size: 33554432 >2018-07-11 05:54:05.764592 7f3562aa9d80 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 >2018-07-11 05:54:05.764593 7f3562aa9d80 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 >2018-07-11 05:54:05.764594 7f3562aa9d80 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 >2018-07-11 05:54:05.764594 7f3562aa9d80 4 rocksdb: Options.disable_auto_compactions: 0 >2018-07-11 05:54:05.764596 7f3562aa9d80 4 rocksdb: Options.compaction_style: kCompactionStyleLevel >2018-07-11 05:54:05.764597 7f3562aa9d80 4 rocksdb: Options.compaction_pri: kByCompensatedSize >2018-07-11 05:54:05.764598 7f3562aa9d80 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 >2018-07-11 05:54:05.764598 7f3562aa9d80 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 >2018-07-11 05:54:05.764599 7f3562aa9d80 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 >2018-07-11 05:54:05.764600 7f3562aa9d80 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 >2018-07-11 05:54:05.764601 7f3562aa9d80 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 >2018-07-11 05:54:05.764602 7f3562aa9d80 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 >2018-07-11 05:54:05.764603 7f3562aa9d80 4 rocksdb: Options.table_properties_collectors: >2018-07-11 05:54:05.764603 7f3562aa9d80 4 rocksdb: Options.inplace_update_support: 0 >2018-07-11 05:54:05.764604 7f3562aa9d80 4 rocksdb: Options.inplace_update_num_locks: 10000 >2018-07-11 05:54:05.764605 7f3562aa9d80 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 >2018-07-11 05:54:05.764606 7f3562aa9d80 4 rocksdb: Options.memtable_huge_page_size: 0 >2018-07-11 05:54:05.764607 7f3562aa9d80 4 rocksdb: Options.bloom_locality: 0 >2018-07-11 05:54:05.764608 7f3562aa9d80 4 rocksdb: Options.max_successive_merges: 0 >2018-07-11 05:54:05.764608 7f3562aa9d80 4 rocksdb: Options.optimize_filters_for_hits: 0 >2018-07-11 05:54:05.764609 7f3562aa9d80 4 rocksdb: Options.paranoid_file_checks: 0 >2018-07-11 05:54:05.764610 7f3562aa9d80 4 rocksdb: Options.force_consistency_checks: 0 >2018-07-11 05:54:05.764610 7f3562aa9d80 4 rocksdb: Options.report_bg_io_stats: 0 >2018-07-11 05:55:13.961259 7f3562aa9d80 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/version_set.cc:2859] Recovered from manifest file:db/MANIFEST-029828 succeeded,manifest_file_number is 29828, next_file_number is 36295, last_sequence is 12655313881, log_number is 0,prev_log_number is 0,max_column_family is 0 > >2018-07-11 05:55:13.961275 7f3562aa9d80 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/version_set.cc:2867] Column family [default] (ID 0), log number is 36273 > >2018-07-11 05:55:13.968450 7f3562aa9d80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288513968440, "job": 1, "event": "recovery_started", "log_files": [36271, 36273]} >2018-07-11 05:55:13.968458 7f3562aa9d80 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_open.cc:482] Recovering log #36271 mode 0 >2018-07-11 05:55:15.471163 7f3562aa9d80 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_open.cc:482] Recovering log #36273 mode 0 >2018-07-11 05:55:19.284411 7f3562aa9d80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288519284389, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 36295, "file_size": 84833893, "table_properties": {"data_size": 76379840, "index_size": 785023, "filter_size": 7668040, "raw_key_size": 86667998, "raw_average_key_size": 28, "raw_value_size": 35730710, "raw_average_value_size": 11, "num_data_blocks": 18757, "num_entries": 3000030, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "6", "kMergeOperands": "0"}} >2018-07-11 05:55:19.284725 7f3562aa9d80 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/version_set.cc:2395] Creating manifest 36296 > >2018-07-11 05:55:19.336877 7f3562aa9d80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288519336873, "job": 1, "event": "recovery_finished"} >2018-07-11 05:55:19.362319 7f3562aa9d80 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_open.cc:1063] DB pointer 0x55a30fa06000 >2018-07-11 05:55:19.362409 7f3562aa9d80 1 bluestore(/var/lib/ceph/osd/ceph-10) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152 >2018-07-11 05:55:19.362715 7f354e0ae700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS ------- >2018-07-11 05:55:19.362722 7f354e0ae700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] >** DB Stats ** >Uptime(secs): 73.6 total, 73.6 interval >Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s >Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s >Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent >Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s >Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s >Interval stall: 00:00:0.000 H:M:S, 0.0 percent > >** Compaction Stats [default] ** >Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop >---------------------------------------------------------------------------------------------------------------------------------------------------------- > L0 1/0 80.90 MB 0.3 0.0 0.0 0.0 0.1 0.1 0.0 1.0 0.0 34.6 2 1 2.341 0 0 > L1 3/0 190.97 MB 0.7 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 > L2 46/0 2.61 GB 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 > L3 448/0 25.07 GB 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 > L4 3641/0 250.00 GB 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 > L5 792/0 54.96 GB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 > Sum 4931/0 332.91 GB 0.0 0.0 0.0 0.0 0.1 0.1 0.0 1.0 0.0 34.6 2 1 2.341 0 0 > Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.1 0.1 0.0 1.0 0.0 34.6 2 1 2.341 0 0 >Uptime(secs): 73.6 total, 73.6 interval >Flush(GB): cumulative 0.079, interval 0.079 >AddFile(GB): cumulative 0.000, interval 0.000 >AddFile(Total Files): cumulative 0, interval 0 >AddFile(L0 Files): cumulative 0, interval 0 >AddFile(Keys): cumulative 0, interval 0 >Cumulative compaction: 0.08 GB write, 1.10 MB/s write, 0.00 GB read, 0.00 MB/s read, 2.3 seconds >Interval compaction: 0.08 GB write, 1.10 MB/s write, 0.00 GB read, 0.00 MB/s read, 2.3 seconds >Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count > >** File Read Latency Histogram By Level [default] ** > >** Compaction Stats [default] ** >Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop >---------------------------------------------------------------------------------------------------------------------------------------------------------- > L0 1/0 80.90 MB 0.3 0.0 0.0 0.0 0.1 0.1 0.0 1.0 0.0 34.6 2 1 2.341 0 0 > L1 3/0 190.97 MB 0.7 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 > L2 46/0 2.61 GB 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 > L3 448/0 25.07 GB 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 > L4 3641/0 250.00 GB 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 > L5 792/0 54.96 GB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 > Sum 4931/0 332.91 GB 0.0 0.0 0.0 0.0 0.1 0.1 0.0 1.0 0.0 34.6 2 1 2.341 0 0 > Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 >Uptime(secs): 73.6 total, 73.6 interval >Flush(GB): cumulative 0.079, interval 0.000 >AddFile(GB): cumulative 0.000, interval 0.000 >AddFile(Total Files): cumulative 0, interval 0 >AddFile(L0 Files): cumulative 0, interval 0 >AddFile(Keys): cumulative 0, interval 0 >Cumulative compaction: 0.08 GB write, 1.10 MB/s write, 0.00 GB read, 0.00 MB/s read, 2.3 seconds >Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds >Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count > >** File Read Latency Histogram By Level [default] ** > >2018-07-11 05:55:19.362879 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 3] Compacting 1@2 + 1@3 files to L3, score 1.04 >2018-07-11 05:55:19.362893 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2 Base level 2, inputs: [36284(71MB)], [35741(71MB)] > >2018-07-11 05:55:19.362908 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288519362897, "job": 3, "event": "compaction_started", "files_L2": [36284], "files_L3": [35741], "score": 1.04377, "input_data_size": 149092290} >2018-07-11 05:55:19.384765 7f3562aa9d80 1 freelist init >2018-07-11 05:55:19.767840 7f3562aa9d80 1 bluestore(/var/lib/ceph/osd/ceph-10) _open_alloc opening allocation metadata >2018-07-11 05:55:20.098509 7f3562aa9d80 1 bluestore(/var/lib/ceph/osd/ceph-10) _open_alloc loaded 495 G in 77663 extents >2018-07-11 05:55:20.314012 7f3562aa9d80 0 _get_class not permitted to load kvs >2018-07-11 05:55:20.314025 7f3562aa9d80 0 _get_class not permitted to load lua >2018-07-11 05:55:20.317066 7f3562aa9d80 0 <cls> /builddir/build/BUILD/ceph-12.2.5/src/cls/cephfs/cls_cephfs.cc:197: loading cephfs >2018-07-11 05:55:20.319372 7f3562aa9d80 0 <cls> /builddir/build/BUILD/ceph-12.2.5/src/cls/hello/cls_hello.cc:296: loading cls_hello >2018-07-11 05:55:20.320331 7f3562aa9d80 0 _get_class not permitted to load sdk >2018-07-11 05:55:20.331442 7f3562aa9d80 0 osd.10 463 crush map has features 288514119978713088, adjusting msgr requires for clients >2018-07-11 05:55:20.331455 7f3562aa9d80 0 osd.10 463 crush map has features 288514394856620032 was 8705, adjusting msgr requires for mons >2018-07-11 05:55:20.331474 7f3562aa9d80 0 osd.10 463 crush map has features 1009090335235915776, adjusting msgr requires for osds >2018-07-11 05:55:22.282726 7f3562aa9d80 0 osd.10 463 load_pgs >2018-07-11 05:55:23.591036 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 3] Generated table #36300: 2638240 keys, 74546634 bytes >2018-07-11 05:55:23.591078 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288523591062, "cf_name": "default", "job": 3, "event": "table_file_creation", "file_number": 36300, "file_size": 74546634, "table_properties": {"data_size": 67112117, "index_size": 690346, "filter_size": 6743181, "raw_key_size": 76215817, "raw_average_key_size": 28, "raw_value_size": 31365737, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:55:26.999164 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 3] Generated table #36301: 2638242 keys, 74546359 bytes >2018-07-11 05:55:26.999213 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288526999196, "cf_name": "default", "job": 3, "event": "table_file_creation", "file_number": 36301, "file_size": 74546359, "table_properties": {"data_size": 67112171, "index_size": 690012, "filter_size": 6743186, "raw_key_size": 76215879, "raw_average_key_size": 28, "raw_value_size": 31365765, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638242, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:55:26.999283 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 3] Compacted 1@2 + 1@3 files to L3 => 149092993 bytes >2018-07-11 05:55:27.012393 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-05:55:27.012266) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[1 3 45 449 3641 792 0] max score 1.02, MB/sec: 19.5 rd, 19.5 wr, level 3, files in(1, 1) out(2) MB in(71.1, 71.1) out(142.2), read-write-amplify(4.0) write-amplify(2.0) OK, records in: 5276482, records dropped: 0 > >2018-07-11 05:55:27.012403 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-05:55:27.012291) EVENT_LOG_v1 {"time_micros": 1531288527012280, "job": 3, "event": "compaction_finished", "compaction_time_micros": 7636353, "output_level": 3, "num_output_files": 2, "total_output_size": 149092993, "num_input_records": 5276482, "num_output_records": 5276482, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 3, 45, 449, 3641, 792, 0]} >2018-07-11 05:55:27.029127 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-05:55:27.013154) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #36283 to level-3 74546286 bytes > >2018-07-11 05:55:27.029139 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-05:55:27.028969) EVENT_LOG_v1 {"time_micros": 1531288527028957, "job": 4, "event": "trivial_move", "destination_level": 3, "files": 1, "total_files_size": 74546286} >2018-07-11 05:55:27.029142 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-05:55:27.028977) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-3 74546286 bytes OK: base level 1 max bytes base 268435456 files[1 3 44 450 3641 792 0] max score 1.01 > >2018-07-11 05:55:27.045841 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-05:55:27.029841) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #36291 to level-4 74546663 bytes > >2018-07-11 05:55:27.045851 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-05:55:27.045693) EVENT_LOG_v1 {"time_micros": 1531288527045686, "job": 5, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 74546663} >2018-07-11 05:55:27.045854 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-05:55:27.045700) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-4 74546663 bytes OK: base level 1 max bytes base 268435456 files[1 3 44 449 3642 792 0] max score 1.01 > >2018-07-11 05:55:27.046620 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 6] Compacting 1@3 + 7@4 files to L4, score 1.01 >2018-07-11 05:55:27.046634 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 5 Base level 3, inputs: [36300(71MB)], [35771(71MB) 35777(71MB) 35778(71MB) 35779(71MB) 35748(71MB) 35749(71MB) 35740(71MB)] > >2018-07-11 05:55:27.046662 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288527046650, "job": 6, "event": "compaction_started", "files_L3": [36300], "files_L4": [35771, 35777, 35778, 35779, 35748, 35749, 35740], "score": 1.00568, "input_data_size": 596370791} >2018-07-11 05:55:27.470204 7f3562aa9d80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288527470197, "job": 0, "event": "table_file_deletion", "file_number": 36284} >2018-07-11 05:55:27.470232 7f3562aa9d80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288527470230, "job": 0, "event": "table_file_deletion", "file_number": 35741} >2018-07-11 05:55:30.834091 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 6] Generated table #36302: 2638240 keys, 74545767 bytes >2018-07-11 05:55:30.834143 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288530834125, "cf_name": "default", "job": 6, "event": "table_file_creation", "file_number": 36302, "file_size": 74545767, "table_properties": {"data_size": 67112122, "index_size": 689474, "filter_size": 6743181, "raw_key_size": 76215825, "raw_average_key_size": 28, "raw_value_size": 31365745, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:55:34.543267 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 6] Generated table #36303: 2638241 keys, 74546982 bytes >2018-07-11 05:55:34.543309 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288534543294, "cf_name": "default", "job": 6, "event": "table_file_creation", "file_number": 36303, "file_size": 74546982, "table_properties": {"data_size": 67112090, "index_size": 690719, "filter_size": 6743183, "raw_key_size": 76215852, "raw_average_key_size": 28, "raw_value_size": 31365755, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:55:36.672897 7f3562aa9d80 0 osd.10 463 load_pgs opened 71 pgs >2018-07-11 05:55:36.672925 7f3562aa9d80 0 osd.10 463 using weightedpriority op queue with priority op cut off at 64. >2018-07-11 05:55:36.673738 7f3562aa9d80 -1 osd.10 463 log_to_monitors {default=true} >2018-07-11 05:55:36.713545 7f35478a1700 1 osd.10 pg_epoch: 463 pg[5.37( v 166'68122 (166'66622,166'68122] local-lis/les=414/415 n=68122 ec=71/71 lis/c 414/414 les/c/f 415/415/0 414/414/414) [4,10,8] r=1 lpr=456 crt=166'68122 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:36.713550 7f35480a2700 1 osd.10 pg_epoch: 463 pg[5.33( v 166'68409 (166'66894,166'68409] local-lis/les=403/404 n=68409 ec=71/71 lis/c 403/403 les/c/f 404/404/0 403/403/387) [2,10,8] r=1 lpr=461 crt=166'68409 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:36.713700 7f35480a2700 1 osd.10 pg_epoch: 463 pg[5.35( v 166'67898 (166'66348,166'67898] local-lis/les=419/420 n=67898 ec=71/71 lis/c 419/419 les/c/f 420/420/0 419/419/411) [0,11,10] r=2 lpr=461 crt=166'67898 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:36.713720 7f35478a1700 1 osd.10 pg_epoch: 463 pg[5.c( v 166'68190 (166'66593,166'68190] local-lis/les=414/415 n=68190 ec=71/71 lis/c 414/414 les/c/f 415/415/0 414/414/389) [1,10,4] r=1 lpr=457 crt=166'68190 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:36.713826 7f35480a2700 1 osd.10 pg_epoch: 463 pg[6.14( v 461'122884 (430'121291,461'122884] local-lis/les=414/415 n=25983 ec=76/76 lis/c 414/414 les/c/f 415/449/0 414/414/403) [10,4,3] r=0 lpr=461 crt=461'122884 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 05:55:36.745216 7f35480a2700 1 osd.10 pg_epoch: 463 pg[5.14( v 166'68424 (166'66924,166'68424] local-lis/les=422/423 n=68424 ec=71/71 lis/c 422/422 les/c/f 423/423/0 422/422/422) [7,10,2] r=1 lpr=461 crt=166'68424 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:36.755127 7f35478a1700 1 osd.10 pg_epoch: 463 pg[16.e( empty local-lis/les=442/443 n=0 ec=442/442 lis/c 442/442 les/c/f 443/443/0 442/442/442) [2,4,10] r=2 lpr=443 crt=0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:36.755196 7f35480a2700 1 osd.10 pg_epoch: 463 pg[6.17( v 430'108120 (430'106620,430'108120] local-lis/les=408/410 n=26407 ec=76/76 lis/c 408/408 les/c/f 410/449/0 408/408/403) [10,2,5] r=0 lpr=449 crt=430'108120 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 05:55:36.755240 7f35478a1700 1 osd.10 pg_epoch: 463 pg[5.24( v 166'68298 (166'66798,166'68298] local-lis/les=419/420 n=68298 ec=71/71 lis/c 419/419 les/c/f 420/420/0 419/419/403) [10,11,1] r=0 lpr=461 crt=166'68298 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 05:55:36.755344 7f35480a2700 1 osd.10 pg_epoch: 463 pg[6.21( v 461'128646 (430'127048,461'128646] local-lis/les=411/412 n=25982 ec=76/76 lis/c 411/411 les/c/f 412/431/0 411/411/411) [0,2,10] r=2 lpr=461 crt=461'128646 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:36.755404 7f35478a1700 1 osd.10 pg_epoch: 463 pg[13.ds1( v 377'10591 (355'1551,377'10591] local-lis/les=428/429 n=2744 ec=345/345 lis/c 428/428 les/c/f 429/429/0 428/428/411) [0,10,1,8,11,6]p0(0) r=1 lpr=461 crt=377'10591 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:36.755468 7f35480a2700 1 osd.10 pg_epoch: 463 pg[10.9( v 390'615 (0'0,390'615] local-lis/les=411/412 n=478 ec=193/193 lis/c 411/411 les/c/f 412/412/0 411/411/411) [0,2,10] r=2 lpr=461 crt=390'615 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:36.826501 7f35480a2700 1 osd.10 pg_epoch: 463 pg[9.a( v 461'571 (0'0,461'571] local-lis/les=419/420 n=231 ec=190/190 lis/c 419/419 les/c/f 420/431/0 419/419/411) [0,10,11] r=1 lpr=461 crt=461'571 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:36.826686 7f35478a1700 1 osd.10 pg_epoch: 463 pg[13.cs2( v 377'10679 (371'2117,377'10679] local-lis/les=428/430 n=2785 ec=345/345 lis/c 428/428 les/c/f 430/430/0 428/428/389) [1,11,10,4,8,6]p1(0) r=2 lpr=430 crt=377'10679 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:36.826725 7f35480a2700 1 osd.10 pg_epoch: 463 pg[5.1c( v 383'68804 (166'67302,383'68804] local-lis/les=403/404 n=68803 ec=71/71 lis/c 403/403 les/c/f 404/404/0 403/403/403) [10,8,2] r=0 lpr=445 crt=383'68804 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 05:55:36.826853 7f35480a2700 1 osd.10 pg_epoch: 463 pg[6.1f( v 430'114415 (430'112915,430'114415] local-lis/les=411/412 n=26413 ec=76/76 lis/c 411/411 les/c/f 412/440/0 411/411/391) [3,10,0] r=1 lpr=461 crt=430'114415 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:36.826929 7f35478a1700 1 osd.10 pg_epoch: 463 pg[16.10( empty local-lis/les=442/443 n=0 ec=442/442 lis/c 442/442 les/c/f 443/443/0 442/442/442) [2,10,7] r=1 lpr=443 crt=0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:36.826961 7f35480a2700 1 osd.10 pg_epoch: 463 pg[10.13( v 383'839 (0'0,383'839] local-lis/les=422/423 n=523 ec=193/193 lis/c 422/422 les/c/f 423/423/0 422/422/389) [1,10,7] r=1 lpr=461 crt=383'839 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:36.827036 7f35478a1700 1 osd.10 pg_epoch: 463 pg[10.a( v 386'2032 (203'500,386'2032] local-lis/les=408/410 n=516 ec=193/193 lis/c 408/408 les/c/f 410/449/0 408/408/403) [10,2,5] r=0 lpr=461 crt=386'2032 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 05:55:36.827066 7f35480a2700 1 osd.10 pg_epoch: 463 pg[5.9( v 166'68275 (166'66775,166'68275] local-lis/les=422/424 n=68275 ec=71/71 lis/c 422/422 les/c/f 424/424/0 422/422/403) [10,8,7] r=0 lpr=461 crt=166'68275 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 05:55:36.827171 7f35478a1700 1 osd.10 pg_epoch: 463 pg[5.23( v 166'68497 (166'66949,166'68497] local-lis/les=403/404 n=68497 ec=71/71 lis/c 403/403 les/c/f 404/404/0 403/403/401) [8,3,10] r=2 lpr=461 crt=166'68497 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:36.827258 7f35480a2700 1 osd.10 pg_epoch: 463 pg[6.a( v 430'109231 (430'107731,430'109231] local-lis/les=428/429 n=26267 ec=76/76 lis/c 428/428 les/c/f 429/453/0 428/428/408) [5,10,6] r=1 lpr=453 crt=430'109231 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:36.827283 7f35478a1700 1 osd.10 pg_epoch: 463 pg[10.f( v 383'568 (0'0,383'568] local-lis/les=419/420 n=494 ec=193/193 lis/c 419/419 les/c/f 420/420/0 419/419/419) [11,10,2] r=1 lpr=461 crt=383'568 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:36.827391 7f35478a1700 1 osd.10 pg_epoch: 463 pg[6.3( v 430'104329 (430'102829,430'104329] local-lis/les=419/420 n=25881 ec=76/76 lis/c 419/419 les/c/f 420/436/0 419/419/411) [0,10,11] r=1 lpr=461 crt=430'104329 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:36.827429 7f35480a2700 1 osd.10 pg_epoch: 463 pg[12.0( v 383'7619 (383'6093,383'7619] local-lis/les=414/415 n=7619 ec=331/331 lis/c 414/414 les/c/f 415/415/0 414/414/414) [4,10,3] r=1 lpr=456 crt=383'7619 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:36.909606 7f35478a1700 1 osd.10 pg_epoch: 463 pg[6.32( v 430'104474 (430'102974,430'104474] local-lis/les=414/415 n=25649 ec=76/76 lis/c 414/414 les/c/f 415/445/0 414/414/414) [4,2,10] r=2 lpr=445 crt=430'104474 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:36.909731 7f35480a2700 1 osd.10 pg_epoch: 463 pg[5.4( v 166'68216 (166'66716,166'68216] local-lis/les=422/423 n=68216 ec=71/71 lis/c 422/422 les/c/f 423/423/0 422/422/422) [7,2,10] r=2 lpr=423 crt=166'68216 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:36.909746 7f35478a1700 1 osd.10 pg_epoch: 463 pg[6.0( v 430'122181 (430'120681,430'122181] local-lis/les=422/424 n=25944 ec=76/76 lis/c 422/422 les/c/f 424/451/0 422/422/403) [10,7,2] r=0 lpr=461 crt=430'122181 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 05:55:36.909875 7f35480a2700 1 osd.10 pg_epoch: 463 pg[6.7( v 430'109641 (430'108141,430'109641] local-lis/les=428/429 n=25825 ec=76/76 lis/c 428/428 les/c/f 429/457/0 428/428/403) [10,6,3] r=0 lpr=457 crt=430'109641 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 05:55:36.909922 7f35478a1700 1 osd.10 pg_epoch: 463 pg[5.10( v 166'68154 (166'66598,166'68154] local-lis/les=414/415 n=68154 ec=71/71 lis/c 414/414 les/c/f 415/415/0 414/414/403) [10,8,4] r=0 lpr=456 crt=166'68154 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 05:55:36.910259 7f35480a2700 1 osd.10 pg_epoch: 463 pg[5.d( v 166'68209 (166'66709,166'68209] local-lis/les=422/424 n=68209 ec=71/71 lis/c 422/422 les/c/f 424/424/0 422/422/403) [10,7,0] r=0 lpr=424 crt=166'68209 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 05:55:36.910659 7f35480a2700 1 osd.10 pg_epoch: 463 pg[6.e( v 430'116090 (430'114517,430'116090] local-lis/les=414/415 n=25967 ec=76/76 lis/c 414/414 les/c/f 415/430/0 414/414/414) [4,1,10] r=2 lpr=430 crt=430'116090 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:36.910689 7f35478a1700 1 osd.10 pg_epoch: 463 pg[11.4s2( v 383'60130 (352'56449,383'60130] local-lis/les=428/429 n=60130 ec=267/267 lis/c 428/428 les/c/f 429/429/0 428/428/411) [0,6,10,8,3,11]p0(0) r=2 lpr=429 crt=383'60130 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:36.910827 7f35478a1700 1 osd.10 pg_epoch: 463 pg[13.1s1( v 377'9368 (355'1456,377'9368] local-lis/les=428/430 n=2661 ec=345/345 lis/c 428/428 les/c/f 430/430/0 428/428/401) [8,10,0,2,6,7]p8(0) r=1 lpr=461 crt=377'9368 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:36.911113 7f35480a2700 1 osd.10 pg_epoch: 463 pg[11.8s3( v 383'59629 (352'55946,383'59629] local-lis/les=428/430 n=59629 ec=267/267 lis/c 428/428 les/c/f 430/430/0 428/428/422) [7,1,6,10,5,8]p7(0) r=3 lpr=430 crt=383'59629 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:36.911419 7f35478a1700 1 osd.10 pg_epoch: 463 pg[11.3s5( v 383'60232 (352'56579,383'60232] local-lis/les=428/429 n=60232 ec=267/267 lis/c 428/428 les/c/f 429/429/0 428/428/428) [6,8,0,11,2,10]p6(0) r=5 lpr=429 crt=383'60232 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:36.911543 7f35478a1700 1 osd.10 pg_epoch: 463 pg[11.6s0( v 383'59732 (352'56233,383'59732] local-lis/les=428/429 n=59732 ec=267/267 lis/c 428/428 les/c/f 429/429/0 428/428/403) [10,1,11,6,5,8]p10(0) r=0 lpr=461 crt=383'59732 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 05:55:36.911588 7f35480a2700 1 osd.10 pg_epoch: 463 pg[13.8s5( v 377'9578 (367'1954,377'9578] local-lis/les=428/429 n=2631 ec=345/345 lis/c 428/428 les/c/f 429/429/0 428/428/387) [2,11,5,8,6,10]p2(0) r=5 lpr=429 crt=377'9578 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:36.911710 7f35478a1700 1 osd.10 pg_epoch: 463 pg[6.13( v 430'99276 (430'97776,430'99276] local-lis/les=422/424 n=26101 ec=76/76 lis/c 422/422 les/c/f 424/457/0 422/422/403) [10,7,0] r=0 lpr=457 crt=430'99276 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 05:55:36.911717 7f35480a2700 1 osd.10 pg_epoch: 463 pg[6.d( v 461'110631 (430'109050,461'110631] local-lis/les=422/424 n=25732 ec=76/76 lis/c 422/422 les/c/f 424/455/0 422/422/403) [10,7,2] r=0 lpr=461 crt=461'110631 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 05:55:36.911923 7f35478a1700 1 osd.10 pg_epoch: 463 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=414/415 n=68041 ec=71/71 lis/c 414/414 les/c/f 415/415/0 414/414/403) [10,2,4] r=0 lpr=456 crt=166'68041 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 05:55:36.911952 7f35480a2700 1 osd.10 pg_epoch: 463 pg[6.2( v 430'110237 (430'108737,430'110237] local-lis/les=422/423 n=25907 ec=76/76 lis/c 422/422 les/c/f 423/445/0 422/422/422) [7,10,2] r=1 lpr=445 crt=430'110237 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:36.912076 7f35480a2700 1 osd.10 pg_epoch: 463 pg[11.as3( v 383'60057 (367'56547,383'60057] local-lis/les=428/429 n=60057 ec=267/267 lis/c 428/428 les/c/f 429/429/0 428/428/391) [3,5,8,10,11,6]p3(0) r=3 lpr=461 crt=383'60057 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:36.912081 7f35478a1700 1 osd.10 pg_epoch: 463 pg[11.1s0( v 383'60378 (352'56674,383'60378] local-lis/les=428/429 n=60378 ec=267/267 lis/c 428/428 les/c/f 429/429/0 428/428/403) [10,5,2,8,6,11]p10(0) r=0 lpr=461 crt=383'60378 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 05:55:36.912190 7f35480a2700 1 osd.10 pg_epoch: 463 pg[13.bs4( v 377'9782 (355'1491,377'9782] local-lis/les=428/430 n=2694 ec=345/345 lis/c 428/428 les/c/f 430/430/0 428/428/419) [11,8,1,4,10,6]p11(0) r=4 lpr=461 crt=377'9782 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:36.912302 7f35480a2700 1 osd.10 pg_epoch: 463 pg[6.26( v 461'123787 (430'122194,461'123787] local-lis/les=422/423 n=26121 ec=76/76 lis/c 422/422 les/c/f 423/445/0 422/422/422) [7,10,8] r=1 lpr=461 crt=461'123787 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:36.912413 7f35480a2700 1 osd.10 pg_epoch: 463 pg[13.9s1( v 377'10260 (352'219,377'10260] local-lis/les=428/429 n=2727 ec=345/345 lis/c 428/428 les/c/f 429/429/0 428/428/408) [5,10,3,6,8,11]p5(0) r=1 lpr=461 crt=377'10254 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:36.912773 7f35478a1700 1 osd.10 pg_epoch: 463 pg[13.5s2( v 377'9560 (0'0,377'9560] local-lis/les=447/448 n=2691 ec=345/345 lis/c 447/447 les/c/f 448/455/0 446/447/414) [4,3,10,11,2147483647,8]/[4,3,10,11,8,8]p4(0) r=2 lpr=455 crt=377'9552 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:36.912897 7f35478a1700 1 osd.10 pg_epoch: 463 pg[13.4s3( v 377'9519 (355'1503,377'9519] local-lis/les=428/429 n=2672 ec=345/345 lis/c 428/428 les/c/f 429/429/0 428/428/408) [5,1,8,10,11,6]p5(0) r=3 lpr=461 crt=377'9519 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:36.912999 7f35478a1700 1 osd.10 pg_epoch: 463 pg[5.2f( v 166'67864 (166'66267,166'67864] local-lis/les=414/415 n=67864 ec=71/71 lis/c 414/414 les/c/f 415/415/0 414/414/401) [8,10,4] r=1 lpr=461 crt=166'67864 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:36.913105 7f35480a2700 1 osd.10 pg_epoch: 463 pg[11.ds3( v 383'59865 (367'56373,383'59865] local-lis/les=428/429 n=59865 ec=267/267 lis/c 428/428 les/c/f 429/429/0 428/428/387) [2,4,11,10,8,6]p2(0) r=3 lpr=429 crt=383'59865 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:36.913120 7f35478a1700 1 osd.10 pg_epoch: 463 pg[6.2c( v 457'120217 (430'118627,457'120217] local-lis/les=428/429 n=26249 ec=76/76 lis/c 428/428 les/c/f 429/449/0 428/428/414) [4,6,10] r=2 lpr=457 crt=457'120217 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:36.913214 7f35478a1700 1 osd.10 pg_epoch: 463 pg[10.1c( v 386'649 (0'0,386'649] local-lis/les=419/420 n=500 ec=193/193 lis/c 419/419 les/c/f 420/420/0 462/462/419) [11,4,8] r=-1 lpr=462 pi=[419,462)/1 crt=386'649 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:36.913324 7f35478a1700 1 osd.10 pg_epoch: 463 pg[10.7( v 386'575 (0'0,386'575] local-lis/les=428/429 n=507 ec=193/193 lis/c 428/428 les/c/f 429/457/0 428/428/403) [10,6,3] r=0 lpr=461 crt=386'575 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 05:55:36.913339 7f35480a2700 1 osd.10 pg_epoch: 463 pg[16.d( empty local-lis/les=442/444 n=0 ec=442/442 lis/c 442/442 les/c/f 444/444/0 442/442/442) [0,3,10] r=2 lpr=444 crt=0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:36.914060 7f35480a2700 1 osd.10 pg_epoch: 463 pg[13.fs4( v 377'8989 (0'0,377'8989] local-lis/les=428/429 n=2712 ec=345/345 lis/c 428/428 les/c/f 429/429/0 428/428/408) [5,7,3,6,10,8]p5(0) r=4 lpr=429 crt=377'8979 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:36.914269 7f35480a2700 1 osd.10 pg_epoch: 463 pg[16.1a( empty local-lis/les=442/443 n=0 ec=442/442 lis/c 442/442 les/c/f 443/443/0 442/442/442) [11,10,3] r=1 lpr=443 crt=0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:36.914364 7f35480a2700 1 osd.10 pg_epoch: 463 pg[9.1a( v 461'339 (0'0,461'339] local-lis/les=422/423 n=240 ec=190/190 lis/c 422/422 les/c/f 423/423/0 422/422/422) [7,2,10] r=2 lpr=461 crt=461'339 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:36.914595 7f35480a2700 1 osd.10 pg_epoch: 463 pg[16.3( empty local-lis/les=442/443 n=0 ec=442/442 lis/c 442/442 les/c/f 443/443/0 442/442/442) [5,2,10] r=2 lpr=443 crt=0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:36.914697 7f35480a2700 1 osd.10 pg_epoch: 463 pg[13.3s1( v 377'10405 (371'2081,377'10405] local-lis/les=428/430 n=2736 ec=345/345 lis/c 428/428 les/c/f 430/430/0 428/428/389) [1,10,8,6,7,5]p1(0) r=1 lpr=461 crt=377'10405 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:36.915216 7f35480a2700 1 osd.10 pg_epoch: 463 pg[11.fs3( v 383'63252 (352'59530,383'63252] local-lis/les=428/429 n=60320 ec=267/267 lis/c 428/428 les/c/f 429/429/0 428/428/411) [0,6,2,10,7,8]p0(0) r=3 lpr=429 crt=383'63252 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:36.915804 7f35480a2700 1 osd.10 pg_epoch: 463 pg[11.5s0( v 383'60461 (352'56750,383'60461] local-lis/les=428/429 n=60461 ec=267/267 lis/c 428/428 les/c/f 429/429/0 428/428/403) [10,5,8,6,1,7]p10(0) r=0 lpr=429 crt=383'60461 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 05:55:36.915956 7f35480a2700 1 osd.10 pg_epoch: 463 pg[11.2s2( v 383'59971 (377'58405,383'59971] local-lis/les=428/429 n=59971 ec=267/267 lis/c 428/428 les/c/f 429/429/0 428/428/387) [2,6,10,0,7,8]p2(0) r=2 lpr=461 crt=383'59970 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:36.916071 7f35480a2700 1 osd.10 pg_epoch: 463 pg[13.2s4( v 377'9209 (0'0,377'9209] local-lis/les=428/429 n=2712 ec=345/345 lis/c 428/428 les/c/f 429/429/0 428/428/408) [5,8,6,7,10,3]p5(0) r=4 lpr=461 crt=377'9201 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:36.916152 7f3562aa9d80 0 osd.10 463 done with init, starting boot process >2018-07-11 05:55:36.916197 7f3562aa9d80 1 osd.10 463 start_boot >2018-07-11 05:55:38.156658 7f354f0b0700 1 osd.10 492 state: booting -> active >2018-07-11 05:55:38.156894 7f35478a1700 1 osd.10 pg_epoch: 464 pg[5.37( v 166'68122 (166'66622,166'68122] local-lis/les=414/415 n=68122 ec=71/71 lis/c 414/414 les/c/f 415/415/0 464/464/464) [] r=-1 lpr=464 pi=[414,464)/1 crt=166'68122 lcod 0'0 unknown NOTIFY] start_peering_interval up [4,10,8] -> [], acting [4,10,8] -> [], acting_primary 4 -> -1, up_primary 4 -> -1, role 1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.156896 7f35480a2700 1 osd.10 pg_epoch: 464 pg[6.3( v 430'104329 (430'102829,430'104329] local-lis/les=419/420 n=25881 ec=76/76 lis/c 419/419 les/c/f 420/436/0 464/464/464) [11] r=-1 lpr=464 pi=[419,464)/1 crt=430'104329 lcod 0'0 unknown NOTIFY] start_peering_interval up [0,10,11] -> [11], acting [0,10,11] -> [11], acting_primary 0 -> 11, up_primary 0 -> 11, role 1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.157107 7f35478a1700 1 osd.10 pg_epoch: 472 pg[5.37( v 166'68122 (166'66622,166'68122] local-lis/les=414/415 n=68122 ec=71/71 lis/c 414/414 les/c/f 415/415/0 472/472/472) [8] r=-1 lpr=472 pi=[414,472)/1 crt=166'68122 lcod 0'0 unknown NOTIFY] start_peering_interval up [] -> [8], acting [] -> [8], acting_primary ? -> 8, up_primary ? -> 8, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.157153 7f35480a2700 1 osd.10 pg_epoch: 470 pg[6.3( v 430'104329 (430'102829,430'104329] local-lis/les=419/420 n=25881 ec=76/76 lis/c 419/419 les/c/f 420/436/0 470/470/470) [0,11] r=-1 lpr=470 pi=[419,470)/1 crt=430'104329 lcod 0'0 unknown NOTIFY] start_peering_interval up [11] -> [0,11], acting [11] -> [0,11], acting_primary 11 -> 0, up_primary 11 -> 0, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.157340 7f35478a1700 1 osd.10 pg_epoch: 487 pg[5.37( v 166'68122 (166'66622,166'68122] local-lis/les=414/415 n=68122 ec=71/71 lis/c 414/414 les/c/f 415/415/0 487/487/487) [4,8] r=-1 lpr=487 pi=[414,487)/1 crt=166'68122 lcod 0'0 unknown NOTIFY] start_peering_interval up [8] -> [4,8], acting [8] -> [4,8], acting_primary 8 -> 4, up_primary 8 -> 4, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.157360 7f35480a2700 1 osd.10 pg_epoch: 477 pg[6.3( v 430'104329 (430'102829,430'104329] local-lis/les=419/420 n=25881 ec=76/76 lis/c 419/419 les/c/f 420/436/0 477/477/470) [0] r=-1 lpr=477 pi=[419,477)/1 crt=430'104329 lcod 0'0 unknown NOTIFY] start_peering_interval up [0,11] -> [0], acting [0,11] -> [0], acting_primary 0 -> 0, up_primary 0 -> 0, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.157423 7f35478a1700 1 osd.10 pg_epoch: 492 pg[5.37( v 166'68122 (166'66622,166'68122] local-lis/les=414/415 n=68122 ec=71/71 lis/c 414/414 les/c/f 415/415/0 492/492/487) [4,10,8] r=1 lpr=492 pi=[414,492)/2 crt=166'68122 lcod 0'0 unknown NOTIFY] start_peering_interval up [4,8] -> [4,10,8], acting [4,8] -> [4,10,8], acting_primary 4 -> 4, up_primary 4 -> 4, role -1 -> 1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.157441 7f35480a2700 1 osd.10 pg_epoch: 478 pg[6.3( v 430'104329 (430'102829,430'104329] local-lis/les=419/420 n=25881 ec=76/76 lis/c 419/419 les/c/f 420/436/0 478/478/470) [0,11] r=-1 lpr=478 pi=[419,478)/1 crt=430'104329 lcod 0'0 unknown NOTIFY] start_peering_interval up [0] -> [0,11], acting [0] -> [0,11], acting_primary 0 -> 0, up_primary 0 -> 0, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.157605 7f35478a1700 1 osd.10 pg_epoch: 492 pg[5.37( v 166'68122 (166'66622,166'68122] local-lis/les=414/415 n=68122 ec=71/71 lis/c 414/414 les/c/f 415/415/0 492/492/487) [4,10,8] r=1 lpr=492 pi=[414,492)/2 crt=166'68122 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:38.157720 7f35480a2700 1 osd.10 pg_epoch: 492 pg[6.3( v 430'104329 (430'102829,430'104329] local-lis/les=419/420 n=25881 ec=76/76 lis/c 419/419 les/c/f 420/436/0 492/492/470) [0,10,11] r=1 lpr=492 pi=[419,492)/1 crt=430'104329 lcod 0'0 unknown NOTIFY] start_peering_interval up [0,11] -> [0,10,11], acting [0,11] -> [0,10,11], acting_primary 0 -> 0, up_primary 0 -> 0, role -1 -> 1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.157841 7f35480a2700 1 osd.10 pg_epoch: 492 pg[6.3( v 430'104329 (430'102829,430'104329] local-lis/les=419/420 n=25881 ec=76/76 lis/c 419/419 les/c/f 420/436/0 492/492/470) [0,10,11] r=1 lpr=492 pi=[419,492)/1 crt=430'104329 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:38.157863 7f35478a1700 1 osd.10 pg_epoch: 464 pg[5.33( v 166'68409 (166'66894,166'68409] local-lis/les=403/404 n=68409 ec=71/71 lis/c 403/403 les/c/f 404/404/0 464/464/464) [] r=-1 lpr=464 pi=[403,464)/1 crt=166'68409 lcod 0'0 unknown NOTIFY] start_peering_interval up [2,10,8] -> [], acting [2,10,8] -> [], acting_primary 2 -> -1, up_primary 2 -> -1, role 1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.157924 7f35478a1700 1 osd.10 pg_epoch: 465 pg[5.33( v 166'68409 (166'66894,166'68409] local-lis/les=403/404 n=68409 ec=71/71 lis/c 403/403 les/c/f 404/404/0 465/465/465) [2] r=-1 lpr=465 pi=[403,465)/1 crt=166'68409 lcod 0'0 unknown NOTIFY] start_peering_interval up [] -> [2], acting [] -> [2], acting_primary ? -> 2, up_primary ? -> 2, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.158057 7f35478a1700 1 osd.10 pg_epoch: 472 pg[5.33( v 166'68409 (166'66894,166'68409] local-lis/les=403/404 n=68409 ec=71/71 lis/c 403/403 les/c/f 404/404/0 472/472/465) [2,8] r=-1 lpr=472 pi=[403,472)/1 crt=166'68409 lcod 0'0 unknown NOTIFY] start_peering_interval up [2] -> [2,8], acting [2] -> [2,8], acting_primary 2 -> 2, up_primary 2 -> 2, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.158139 7f35480a2700 1 osd.10 pg_epoch: 464 pg[6.32( v 430'104474 (430'102974,430'104474] local-lis/les=414/415 n=25649 ec=76/76 lis/c 414/414 les/c/f 415/445/0 464/464/464) [] r=-1 lpr=464 pi=[414,464)/1 crt=430'104474 lcod 0'0 unknown NOTIFY] start_peering_interval up [4,2,10] -> [], acting [4,2,10] -> [], acting_primary 4 -> -1, up_primary 4 -> -1, role 2 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.158216 7f35480a2700 1 osd.10 pg_epoch: 465 pg[6.32( v 430'104474 (430'102974,430'104474] local-lis/les=414/415 n=25649 ec=76/76 lis/c 414/414 les/c/f 415/445/0 465/465/465) [2] r=-1 lpr=465 pi=[414,465)/1 crt=430'104474 lcod 0'0 unknown NOTIFY] start_peering_interval up [] -> [2], acting [] -> [2], acting_primary ? -> 2, up_primary ? -> 2, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.158264 7f35478a1700 1 osd.10 pg_epoch: 492 pg[5.33( v 166'68409 (166'66894,166'68409] local-lis/les=403/404 n=68409 ec=71/71 lis/c 403/403 les/c/f 404/404/0 492/492/465) [2,10,8] r=1 lpr=492 pi=[403,492)/2 crt=166'68409 lcod 0'0 unknown NOTIFY] start_peering_interval up [2,8] -> [2,10,8], acting [2,8] -> [2,10,8], acting_primary 2 -> 2, up_primary 2 -> 2, role -1 -> 1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.158344 7f35478a1700 1 osd.10 pg_epoch: 492 pg[5.33( v 166'68409 (166'66894,166'68409] local-lis/les=403/404 n=68409 ec=71/71 lis/c 403/403 les/c/f 404/404/0 492/492/465) [2,10,8] r=1 lpr=492 pi=[403,492)/2 crt=166'68409 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:38.158508 7f35478a1700 1 osd.10 pg_epoch: 464 pg[5.35( v 166'67898 (166'66348,166'67898] local-lis/les=419/420 n=67898 ec=71/71 lis/c 419/419 les/c/f 420/420/0 464/464/464) [11] r=-1 lpr=464 pi=[419,464)/1 crt=166'67898 lcod 0'0 unknown NOTIFY] start_peering_interval up [0,11,10] -> [11], acting [0,11,10] -> [11], acting_primary 0 -> 11, up_primary 0 -> 11, role 2 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.158537 7f35480a2700 1 osd.10 pg_epoch: 487 pg[6.32( v 430'104474 (430'102974,430'104474] local-lis/les=414/415 n=25649 ec=76/76 lis/c 414/414 les/c/f 415/445/0 487/487/487) [4,2] r=-1 lpr=487 pi=[414,487)/1 crt=430'104474 lcod 0'0 unknown NOTIFY] start_peering_interval up [2] -> [4,2], acting [2] -> [4,2], acting_primary 2 -> 4, up_primary 2 -> 4, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.158591 7f35478a1700 1 osd.10 pg_epoch: 470 pg[5.35( v 166'67898 (166'66348,166'67898] local-lis/les=419/420 n=67898 ec=71/71 lis/c 419/419 les/c/f 420/420/0 470/470/470) [0,11] r=-1 lpr=470 pi=[419,470)/1 crt=166'67898 lcod 0'0 unknown NOTIFY] start_peering_interval up [11] -> [0,11], acting [11] -> [0,11], acting_primary 11 -> 0, up_primary 11 -> 0, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.158621 7f35480a2700 1 osd.10 pg_epoch: 492 pg[6.32( v 430'104474 (430'102974,430'104474] local-lis/les=414/415 n=25649 ec=76/76 lis/c 414/414 les/c/f 415/445/0 492/492/487) [4,2,10] r=2 lpr=492 pi=[414,492)/2 crt=430'104474 lcod 0'0 unknown NOTIFY] start_peering_interval up [4,2] -> [4,2,10], acting [4,2] -> [4,2,10], acting_primary 4 -> 4, up_primary 4 -> 4, role -1 -> 2, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.158673 7f35478a1700 1 osd.10 pg_epoch: 477 pg[5.35( v 166'67898 (166'66348,166'67898] local-lis/les=419/420 n=67898 ec=71/71 lis/c 419/419 les/c/f 420/420/0 477/477/470) [0] r=-1 lpr=477 pi=[419,477)/1 crt=166'67898 lcod 0'0 unknown NOTIFY] start_peering_interval up [0,11] -> [0], acting [0,11] -> [0], acting_primary 0 -> 0, up_primary 0 -> 0, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.158691 7f35480a2700 1 osd.10 pg_epoch: 492 pg[6.32( v 430'104474 (430'102974,430'104474] local-lis/les=414/415 n=25649 ec=76/76 lis/c 414/414 les/c/f 415/445/0 492/492/487) [4,2,10] r=2 lpr=492 pi=[414,492)/2 crt=430'104474 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:38.158719 7f35478a1700 1 osd.10 pg_epoch: 478 pg[5.35( v 166'67898 (166'66348,166'67898] local-lis/les=419/420 n=67898 ec=71/71 lis/c 419/419 les/c/f 420/420/0 478/478/470) [0,11] r=-1 lpr=478 pi=[419,478)/1 crt=166'67898 lcod 0'0 unknown NOTIFY] start_peering_interval up [0] -> [0,11], acting [0] -> [0,11], acting_primary 0 -> 0, up_primary 0 -> 0, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.158784 7f35480a2700 1 osd.10 pg_epoch: 464 pg[6.0( v 430'122181 (430'120681,430'122181] local-lis/les=422/424 n=25944 ec=76/76 lis/c 422/422 les/c/f 424/451/0 422/422/403) [10,7,2] r=0 lpr=461 crt=430'122181 lcod 0'0 mlcod 0'0 peering] state<Started/Primary/Peering>: Peering, affected_by_map, going to Reset >2018-07-11 05:55:38.158838 7f35478a1700 1 osd.10 pg_epoch: 492 pg[5.35( v 166'67898 (166'66348,166'67898] local-lis/les=419/420 n=67898 ec=71/71 lis/c 419/419 les/c/f 420/420/0 492/492/470) [0,11,10] r=2 lpr=492 pi=[419,492)/1 crt=166'67898 lcod 0'0 unknown NOTIFY] start_peering_interval up [0,11] -> [0,11,10], acting [0,11] -> [0,11,10], acting_primary 0 -> 0, up_primary 0 -> 0, role -1 -> 2, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.158908 7f35478a1700 1 osd.10 pg_epoch: 492 pg[5.35( v 166'67898 (166'66348,166'67898] local-lis/les=419/420 n=67898 ec=71/71 lis/c 419/419 les/c/f 420/420/0 492/492/470) [0,11,10] r=2 lpr=492 pi=[419,492)/1 crt=166'67898 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:38.158904 7f35480a2700 1 osd.10 pg_epoch: 464 pg[6.0( v 430'122181 (430'120681,430'122181] local-lis/les=422/424 n=25944 ec=76/76 lis/c 422/422 les/c/f 424/451/0 464/464/464) [7] r=-1 lpr=464 pi=[422,464)/1 crt=430'122181 lcod 0'0 unknown] start_peering_interval up [10,7,2] -> [7], acting [10,7,2] -> [7], acting_primary 10 -> 7, up_primary 10 -> 7, role 0 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.158957 7f35480a2700 1 osd.10 pg_epoch: 465 pg[6.0( v 430'122181 (430'120681,430'122181] local-lis/les=422/424 n=25944 ec=76/76 lis/c 422/422 les/c/f 424/451/0 465/465/464) [7,2] r=-1 lpr=465 pi=[422,465)/1 crt=430'122181 lcod 0'0 unknown NOTIFY] start_peering_interval up [7] -> [7,2], acting [7] -> [7,2], acting_primary 7 -> 7, up_primary 7 -> 7, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.159006 7f35480a2700 1 osd.10 pg_epoch: 467 pg[6.0( v 430'122181 (430'120681,430'122181] local-lis/les=422/424 n=25944 ec=76/76 lis/c 422/422 les/c/f 424/451/0 467/467/467) [2] r=-1 lpr=467 pi=[422,467)/1 crt=430'122181 lcod 0'0 unknown NOTIFY] start_peering_interval up [7,2] -> [2], acting [7,2] -> [2], acting_primary 7 -> 2, up_primary 7 -> 2, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.159047 7f35478a1700 1 osd.10 pg_epoch: 464 pg[5.c( v 166'68190 (166'66593,166'68190] local-lis/les=414/415 n=68190 ec=71/71 lis/c 414/414 les/c/f 415/415/0 464/464/464) [] r=-1 lpr=464 pi=[414,464)/1 crt=166'68190 lcod 0'0 unknown NOTIFY] start_peering_interval up [1,10,4] -> [], acting [1,10,4] -> [], acting_primary 1 -> -1, up_primary 1 -> -1, role 1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.159048 7f35480a2700 1 osd.10 pg_epoch: 468 pg[6.0( v 430'122181 (430'120681,430'122181] local-lis/les=422/424 n=25944 ec=76/76 lis/c 422/422 les/c/f 424/451/0 468/468/468) [7,2] r=-1 lpr=468 pi=[422,468)/1 crt=430'122181 lcod 0'0 unknown NOTIFY] start_peering_interval up [2] -> [7,2], acting [2] -> [7,2], acting_primary 2 -> 7, up_primary 2 -> 7, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.159131 7f35478a1700 1 osd.10 pg_epoch: 470 pg[5.c( v 166'68190 (166'66593,166'68190] local-lis/les=414/415 n=68190 ec=71/71 lis/c 414/414 les/c/f 415/415/0 470/470/470) [1] r=-1 lpr=470 pi=[414,470)/1 crt=166'68190 lcod 0'0 unknown NOTIFY] start_peering_interval up [] -> [1], acting [] -> [1], acting_primary ? -> 1, up_primary ? -> 1, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.159267 7f35480a2700 1 osd.10 pg_epoch: 492 pg[6.0( v 430'122181 (430'120681,430'122181] local-lis/les=422/424 n=25944 ec=76/76 lis/c 422/422 les/c/f 424/451/0 492/492/492) [10,7,2] r=0 lpr=492 pi=[422,492)/1 crt=430'122181 lcod 0'0 mlcod 0'0 unknown NOTIFY] start_peering_interval up [7,2] -> [10,7,2], acting [7,2] -> [10,7,2], acting_primary 7 -> 10, up_primary 7 -> 10, role -1 -> 0, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.159305 7f35478a1700 1 osd.10 pg_epoch: 487 pg[5.c( v 166'68190 (166'66593,166'68190] local-lis/les=414/415 n=68190 ec=71/71 lis/c 414/414 les/c/f 415/415/0 487/487/470) [1,4] r=-1 lpr=487 pi=[414,487)/1 crt=166'68190 lcod 0'0 unknown NOTIFY] start_peering_interval up [1] -> [1,4], acting [1] -> [1,4], acting_primary 1 -> 1, up_primary 1 -> 1, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.159343 7f35480a2700 1 osd.10 pg_epoch: 492 pg[6.0( v 430'122181 (430'120681,430'122181] local-lis/les=422/424 n=25944 ec=76/76 lis/c 422/422 les/c/f 424/451/0 492/492/492) [10,7,2] r=0 lpr=492 pi=[422,492)/1 crt=430'122181 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 05:55:38.159387 7f35478a1700 1 osd.10 pg_epoch: 492 pg[5.c( v 166'68190 (166'66593,166'68190] local-lis/les=414/415 n=68190 ec=71/71 lis/c 414/414 les/c/f 415/415/0 492/492/470) [1,10,4] r=1 lpr=492 pi=[414,492)/2 crt=166'68190 lcod 0'0 unknown NOTIFY] start_peering_interval up [1,4] -> [1,10,4], acting [1,4] -> [1,10,4], acting_primary 1 -> 1, up_primary 1 -> 1, role -1 -> 1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.159452 7f35478a1700 1 osd.10 pg_epoch: 492 pg[5.c( v 166'68190 (166'66593,166'68190] local-lis/les=414/415 n=68190 ec=71/71 lis/c 414/414 les/c/f 415/415/0 492/492/470) [1,10,4] r=1 lpr=492 pi=[414,492)/2 crt=166'68190 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:38.159466 7f35480a2700 1 osd.10 pg_epoch: 464 pg[5.10( v 166'68154 (166'66598,166'68154] local-lis/les=414/415 n=68154 ec=71/71 lis/c 414/414 les/c/f 415/415/0 414/414/403) [10,8,4] r=0 lpr=456 crt=166'68154 lcod 0'0 mlcod 0'0 peering] state<Started/Primary/Peering>: Peering, affected_by_map, going to Reset >2018-07-11 05:55:38.159575 7f35480a2700 1 osd.10 pg_epoch: 464 pg[5.10( v 166'68154 (166'66598,166'68154] local-lis/les=414/415 n=68154 ec=71/71 lis/c 414/414 les/c/f 415/415/0 464/464/464) [] r=-1 lpr=464 pi=[414,464)/1 crt=166'68154 lcod 0'0 unknown] start_peering_interval up [10,8,4] -> [], acting [10,8,4] -> [], acting_primary 10 -> -1, up_primary 10 -> -1, role 0 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.159588 7f35478a1700 1 osd.10 pg_epoch: 464 pg[16.e( empty local-lis/les=442/443 n=0 ec=442/442 lis/c 442/442 les/c/f 443/443/0 464/464/464) [] r=-1 lpr=464 pi=[442,464)/1 crt=0'0 unknown NOTIFY] start_peering_interval up [2,4,10] -> [], acting [2,4,10] -> [], acting_primary 2 -> -1, up_primary 2 -> -1, role 2 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.159632 7f35478a1700 1 osd.10 pg_epoch: 465 pg[16.e( empty local-lis/les=442/443 n=0 ec=442/442 lis/c 442/442 les/c/f 443/443/0 465/465/465) [2] r=-1 lpr=465 pi=[442,465)/1 crt=0'0 unknown NOTIFY] start_peering_interval up [] -> [2], acting [] -> [2], acting_primary ? -> 2, up_primary ? -> 2, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.159666 7f35480a2700 1 osd.10 pg_epoch: 472 pg[5.10( v 166'68154 (166'66598,166'68154] local-lis/les=414/415 n=68154 ec=71/71 lis/c 414/414 les/c/f 415/415/0 472/472/472) [8] r=-1 lpr=472 pi=[414,472)/1 crt=166'68154 lcod 0'0 unknown NOTIFY] start_peering_interval up [] -> [8], acting [] -> [8], acting_primary ? -> 8, up_primary ? -> 8, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.159801 7f35480a2700 1 osd.10 pg_epoch: 487 pg[5.10( v 166'68154 (166'66598,166'68154] local-lis/les=414/415 n=68154 ec=71/71 lis/c 414/414 les/c/f 415/415/0 487/487/472) [8,4] r=-1 lpr=487 pi=[414,487)/1 crt=166'68154 lcod 0'0 unknown NOTIFY] start_peering_interval up [8] -> [8,4], acting [8] -> [8,4], acting_primary 8 -> 8, up_primary 8 -> 8, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.159817 7f35478a1700 1 osd.10 pg_epoch: 487 pg[16.e( empty local-lis/les=442/443 n=0 ec=442/442 lis/c 442/442 les/c/f 443/443/0 487/487/465) [2,4] r=-1 lpr=487 pi=[442,487)/1 crt=0'0 unknown NOTIFY] start_peering_interval up [2] -> [2,4], acting [2] -> [2,4], acting_primary 2 -> 2, up_primary 2 -> 2, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.159884 7f35480a2700 1 osd.10 pg_epoch: 492 pg[5.10( v 166'68154 (166'66598,166'68154] local-lis/les=414/415 n=68154 ec=71/71 lis/c 414/414 les/c/f 415/415/0 492/492/492) [10,8,4] r=0 lpr=492 pi=[414,492)/2 crt=166'68154 lcod 0'0 mlcod 0'0 unknown NOTIFY] start_peering_interval up [8,4] -> [10,8,4], acting [8,4] -> [10,8,4], acting_primary 8 -> 10, up_primary 8 -> 10, role -1 -> 0, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.159895 7f35478a1700 1 osd.10 pg_epoch: 492 pg[16.e( empty local-lis/les=442/443 n=0 ec=442/442 lis/c 442/442 les/c/f 443/443/0 492/492/465) [2,4,10] r=2 lpr=492 pi=[442,492)/1 crt=0'0 unknown NOTIFY] start_peering_interval up [2,4] -> [2,4,10], acting [2,4] -> [2,4,10], acting_primary 2 -> 2, up_primary 2 -> 2, role -1 -> 2, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.159956 7f35480a2700 1 osd.10 pg_epoch: 492 pg[5.10( v 166'68154 (166'66598,166'68154] local-lis/les=414/415 n=68154 ec=71/71 lis/c 414/414 les/c/f 415/415/0 492/492/492) [10,8,4] r=0 lpr=492 pi=[414,492)/2 crt=166'68154 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 05:55:38.159958 7f35478a1700 1 osd.10 pg_epoch: 492 pg[16.e( empty local-lis/les=442/443 n=0 ec=442/442 lis/c 442/442 les/c/f 443/443/0 492/492/465) [2,4,10] r=2 lpr=492 pi=[442,492)/1 crt=0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:38.160045 7f35478a1700 1 osd.10 pg_epoch: 464 pg[5.24( v 166'68298 (166'66798,166'68298] local-lis/les=419/420 n=68298 ec=71/71 lis/c 419/419 les/c/f 420/420/0 419/419/403) [10,11,1] r=0 lpr=461 crt=166'68298 lcod 0'0 mlcod 0'0 peering] state<Started/Primary/Peering>: Peering, affected_by_map, going to Reset >2018-07-11 05:55:38.160146 7f35478a1700 1 osd.10 pg_epoch: 464 pg[5.24( v 166'68298 (166'66798,166'68298] local-lis/les=419/420 n=68298 ec=71/71 lis/c 419/419 les/c/f 420/420/0 464/464/464) [11] r=-1 lpr=464 pi=[419,464)/1 crt=166'68298 lcod 0'0 unknown] start_peering_interval up [10,11,1] -> [11], acting [10,11,1] -> [11], acting_primary 10 -> 11, up_primary 10 -> 11, role 0 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.160157 7f35480a2700 1 osd.10 pg_epoch: 464 pg[11.4s2( v 383'60130 (352'56449,383'60130] local-lis/les=428/429 n=60130 ec=267/267 lis/c 428/428 les/c/f 429/429/0 464/464/464) [2147483647,6,2147483647,2147483647,2147483647,11]p6(1) r=-1 lpr=464 pi=[428,464)/1 crt=383'60130 lcod 0'0 unknown NOTIFY] start_peering_interval up [0,6,10,8,3,11] -> [2147483647,6,2147483647,2147483647,2147483647,11], acting [0,6,10,8,3,11] -> [2147483647,6,2147483647,2147483647,2147483647,11], acting_primary 0(0) -> 6, up_primary 0(0) -> 6, role 2 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.160232 7f35478a1700 1 osd.10 pg_epoch: 470 pg[5.24( v 166'68298 (166'66798,166'68298] local-lis/les=419/420 n=68298 ec=71/71 lis/c 419/419 les/c/f 420/420/0 470/470/464) [11,1] r=-1 lpr=470 pi=[419,470)/1 crt=166'68298 lcod 0'0 unknown NOTIFY] start_peering_interval up [11] -> [11,1], acting [11] -> [11,1], acting_primary 11 -> 11, up_primary 11 -> 11, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.160320 7f35478a1700 1 osd.10 pg_epoch: 477 pg[5.24( v 166'68298 (166'66798,166'68298] local-lis/les=419/420 n=68298 ec=71/71 lis/c 419/419 les/c/f 420/420/0 477/477/477) [1] r=-1 lpr=477 pi=[419,477)/1 crt=166'68298 lcod 0'0 unknown NOTIFY] start_peering_interval up [11,1] -> [1], acting [11,1] -> [1], acting_primary 11 -> 1, up_primary 11 -> 1, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.160334 7f35480a2700 1 osd.10 pg_epoch: 470 pg[11.4s2( v 383'60130 (352'56449,383'60130] local-lis/les=428/429 n=60130 ec=267/267 lis/c 428/428 les/c/f 429/429/0 470/470/470) [0,6,2147483647,2147483647,2147483647,11]p0(0) r=-1 lpr=470 pi=[428,470)/1 crt=383'60130 lcod 0'0 unknown NOTIFY] start_peering_interval up [2147483647,6,2147483647,2147483647,2147483647,11] -> [0,6,2147483647,2147483647,2147483647,11], acting [2147483647,6,2147483647,2147483647,2147483647,11] -> [0,6,2147483647,2147483647,2147483647,11], acting_primary 6(1) -> 0, up_primary 6(1) -> 0, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.160362 7f35478a1700 1 osd.10 pg_epoch: 478 pg[5.24( v 166'68298 (166'66798,166'68298] local-lis/les=419/420 n=68298 ec=71/71 lis/c 419/419 les/c/f 420/420/0 478/478/478) [11,1] r=-1 lpr=478 pi=[419,478)/1 crt=166'68298 lcod 0'0 unknown NOTIFY] start_peering_interval up [1] -> [11,1], acting [1] -> [11,1], acting_primary 1 -> 11, up_primary 1 -> 11, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.160416 7f35480a2700 1 osd.10 pg_epoch: 472 pg[11.4s2( v 383'60130 (352'56449,383'60130] local-lis/les=428/429 n=60130 ec=267/267 lis/c 428/428 les/c/f 429/429/0 472/472/470) [0,6,2147483647,8,2147483647,11]p0(0) r=-1 lpr=472 pi=[428,472)/1 crt=383'60130 lcod 0'0 unknown NOTIFY] start_peering_interval up [0,6,2147483647,2147483647,2147483647,11] -> [0,6,2147483647,8,2147483647,11], acting [0,6,2147483647,2147483647,2147483647,11] -> [0,6,2147483647,8,2147483647,11], acting_primary 0(0) -> 0, up_primary 0(0) -> 0, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.160486 7f35478a1700 1 osd.10 pg_epoch: 492 pg[5.24( v 166'68298 (166'66798,166'68298] local-lis/les=419/420 n=68298 ec=71/71 lis/c 419/419 les/c/f 420/420/0 492/492/492) [10,11,1] r=0 lpr=492 pi=[419,492)/1 crt=166'68298 lcod 0'0 mlcod 0'0 unknown NOTIFY] start_peering_interval up [11,1] -> [10,11,1], acting [11,1] -> [10,11,1], acting_primary 11 -> 10, up_primary 11 -> 10, role -1 -> 0, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.160545 7f35480a2700 1 osd.10 pg_epoch: 477 pg[11.4s2( v 383'60130 (352'56449,383'60130] local-lis/les=428/429 n=60130 ec=267/267 lis/c 428/428 les/c/f 429/429/0 477/477/470) [0,6,2147483647,8,2147483647,2147483647]p0(0) r=-1 lpr=477 pi=[428,477)/1 crt=383'60130 lcod 0'0 unknown NOTIFY] start_peering_interval up [0,6,2147483647,8,2147483647,11] -> [0,6,2147483647,8,2147483647,2147483647], acting [0,6,2147483647,8,2147483647,11] -> [0,6,2147483647,8,2147483647,2147483647], acting_primary 0(0) -> 0, up_primary 0(0) -> 0, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.160556 7f35478a1700 1 osd.10 pg_epoch: 492 pg[5.24( v 166'68298 (166'66798,166'68298] local-lis/les=419/420 n=68298 ec=71/71 lis/c 419/419 les/c/f 420/420/0 492/492/492) [10,11,1] r=0 lpr=492 pi=[419,492)/1 crt=166'68298 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 05:55:38.160603 7f35480a2700 1 osd.10 pg_epoch: 478 pg[11.4s2( v 383'60130 (352'56449,383'60130] local-lis/les=428/429 n=60130 ec=267/267 lis/c 428/428 les/c/f 429/429/0 478/478/470) [0,6,2147483647,8,2147483647,11]p0(0) r=-1 lpr=478 pi=[428,478)/1 crt=383'60130 lcod 0'0 unknown NOTIFY] start_peering_interval up [0,6,2147483647,8,2147483647,2147483647] -> [0,6,2147483647,8,2147483647,11], acting [0,6,2147483647,8,2147483647,2147483647] -> [0,6,2147483647,8,2147483647,11], acting_primary 0(0) -> 0, up_primary 0(0) -> 0, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.160678 7f35480a2700 1 osd.10 pg_epoch: 480 pg[11.4s2( v 383'60130 (352'56449,383'60130] local-lis/les=428/429 n=60130 ec=267/267 lis/c 428/428 les/c/f 429/429/0 480/480/470) [0,2147483647,2147483647,8,2147483647,11]p0(0) r=-1 lpr=480 pi=[428,480)/1 crt=383'60130 lcod 0'0 unknown NOTIFY] start_peering_interval up [0,6,2147483647,8,2147483647,11] -> [0,2147483647,2147483647,8,2147483647,11], acting [0,6,2147483647,8,2147483647,11] -> [0,2147483647,2147483647,8,2147483647,11], acting_primary 0(0) -> 0, up_primary 0(0) -> 0, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.160739 7f35478a1700 1 osd.10 pg_epoch: 464 pg[13.ds1( v 377'10591 (355'1551,377'10591] local-lis/les=428/429 n=2744 ec=345/345 lis/c 428/428 les/c/f 429/429/0 464/464/464) [2147483647,2147483647,2147483647,2147483647,11,6]p11(4) r=-1 lpr=464 pi=[428,464)/1 crt=377'10591 lcod 0'0 unknown NOTIFY] start_peering_interval up [0,10,1,8,11,6] -> [2147483647,2147483647,2147483647,2147483647,11,6], acting [0,10,1,8,11,6] -> [2147483647,2147483647,2147483647,2147483647,11,6], acting_primary 0(0) -> 11, up_primary 0(0) -> 11, role 1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.160743 7f35480a2700 1 osd.10 pg_epoch: 481 pg[11.4s2( v 383'60130 (352'56449,383'60130] local-lis/les=428/429 n=60130 ec=267/267 lis/c 428/428 les/c/f 429/429/0 481/481/470) [0,6,2147483647,8,2147483647,11]p0(0) r=-1 lpr=481 pi=[428,481)/1 crt=383'60130 lcod 0'0 unknown NOTIFY] start_peering_interval up [0,2147483647,2147483647,8,2147483647,11] -> [0,6,2147483647,8,2147483647,11], acting [0,2147483647,2147483647,8,2147483647,11] -> [0,6,2147483647,8,2147483647,11], acting_primary 0(0) -> 0, up_primary 0(0) -> 0, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.160835 7f35480a2700 1 osd.10 pg_epoch: 484 pg[11.4s2( v 383'60130 (352'56449,383'60130] local-lis/les=428/429 n=60130 ec=267/267 lis/c 428/428 les/c/f 429/429/0 484/484/470) [0,6,2147483647,8,3,11]p0(0) r=-1 lpr=484 pi=[428,484)/1 crt=383'60130 lcod 0'0 unknown NOTIFY] start_peering_interval up [0,6,2147483647,8,2147483647,11] -> [0,6,2147483647,8,3,11], acting [0,6,2147483647,8,2147483647,11] -> [0,6,2147483647,8,3,11], acting_primary 0(0) -> 0, up_primary 0(0) -> 0, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.160869 7f35478a1700 1 osd.10 pg_epoch: 470 pg[13.ds1( v 377'10591 (355'1551,377'10591] local-lis/les=428/429 n=2744 ec=345/345 lis/c 428/428 les/c/f 429/429/0 470/470/470) [0,2147483647,1,2147483647,11,6]p0(0) r=-1 lpr=470 pi=[428,470)/1 crt=377'10591 lcod 0'0 unknown NOTIFY] start_peering_interval up [2147483647,2147483647,2147483647,2147483647,11,6] -> [0,2147483647,1,2147483647,11,6], acting [2147483647,2147483647,2147483647,2147483647,11,6] -> [0,2147483647,1,2147483647,11,6], acting_primary 11(4) -> 0, up_primary 11(4) -> 0, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.160937 7f35478a1700 1 osd.10 pg_epoch: 472 pg[13.ds1( v 377'10591 (355'1551,377'10591] local-lis/les=428/429 n=2744 ec=345/345 lis/c 428/428 les/c/f 429/429/0 472/472/470) [0,2147483647,1,8,11,6]p0(0) r=-1 lpr=472 pi=[428,472)/1 crt=377'10591 lcod 0'0 unknown NOTIFY] start_peering_interval up [0,2147483647,1,2147483647,11,6] -> [0,2147483647,1,8,11,6], acting [0,2147483647,1,2147483647,11,6] -> [0,2147483647,1,8,11,6], acting_primary 0(0) -> 0, up_primary 0(0) -> 0, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.161017 7f35480a2700 1 osd.10 pg_epoch: 492 pg[11.4s2( v 383'60130 (352'56449,383'60130] local-lis/les=428/429 n=60130 ec=267/267 lis/c 428/428 les/c/f 429/429/0 492/492/470) [0,6,10,8,3,11]p0(0) r=2 lpr=492 pi=[428,492)/1 crt=383'60130 lcod 0'0 unknown NOTIFY] start_peering_interval up [0,6,2147483647,8,3,11] -> [0,6,10,8,3,11], acting [0,6,2147483647,8,3,11] -> [0,6,10,8,3,11], acting_primary 0(0) -> 0, up_primary 0(0) -> 0, role -1 -> 2, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.161049 7f35478a1700 1 osd.10 pg_epoch: 477 pg[13.ds1( v 377'10591 (355'1551,377'10591] local-lis/les=428/429 n=2744 ec=345/345 lis/c 428/428 les/c/f 429/429/0 477/477/470) [0,2147483647,1,8,2147483647,6]p0(0) r=-1 lpr=477 pi=[428,477)/1 crt=377'10591 lcod 0'0 unknown NOTIFY] start_peering_interval up [0,2147483647,1,8,11,6] -> [0,2147483647,1,8,2147483647,6], acting [0,2147483647,1,8,11,6] -> [0,2147483647,1,8,2147483647,6], acting_primary 0(0) -> 0, up_primary 0(0) -> 0, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.161091 7f35480a2700 1 osd.10 pg_epoch: 492 pg[11.4s2( v 383'60130 (352'56449,383'60130] local-lis/les=428/429 n=60130 ec=267/267 lis/c 428/428 les/c/f 429/429/0 492/492/470) [0,6,10,8,3,11]p0(0) r=2 lpr=492 pi=[428,492)/1 crt=383'60130 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:38.161111 7f35478a1700 1 osd.10 pg_epoch: 478 pg[13.ds1( v 377'10591 (355'1551,377'10591] local-lis/les=428/429 n=2744 ec=345/345 lis/c 428/428 les/c/f 429/429/0 478/478/470) [0,2147483647,1,8,11,6]p0(0) r=-1 lpr=478 pi=[428,478)/1 crt=377'10591 lcod 0'0 unknown NOTIFY] start_peering_interval up [0,2147483647,1,8,2147483647,6] -> [0,2147483647,1,8,11,6], acting [0,2147483647,1,8,2147483647,6] -> [0,2147483647,1,8,11,6], acting_primary 0(0) -> 0, up_primary 0(0) -> 0, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.161187 7f35478a1700 1 osd.10 pg_epoch: 480 pg[13.ds1( v 377'10591 (355'1551,377'10591] local-lis/les=428/429 n=2744 ec=345/345 lis/c 428/428 les/c/f 429/429/0 480/480/470) [0,2147483647,1,8,11,2147483647]p0(0) r=-1 lpr=480 pi=[428,480)/1 crt=377'10591 lcod 0'0 unknown NOTIFY] start_peering_interval up [0,2147483647,1,8,11,6] -> [0,2147483647,1,8,11,2147483647], acting [0,2147483647,1,8,11,6] -> [0,2147483647,1,8,11,2147483647], acting_primary 0(0) -> 0, up_primary 0(0) -> 0, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.161239 7f35478a1700 1 osd.10 pg_epoch: 481 pg[13.ds1( v 377'10591 (355'1551,377'10591] local-lis/les=428/429 n=2744 ec=345/345 lis/c 428/428 les/c/f 429/429/0 481/481/470) [0,2147483647,1,8,11,6]p0(0) r=-1 lpr=481 pi=[428,481)/1 crt=377'10591 lcod 0'0 unknown NOTIFY] start_peering_interval up [0,2147483647,1,8,11,2147483647] -> [0,2147483647,1,8,11,6], acting [0,2147483647,1,8,11,2147483647] -> [0,2147483647,1,8,11,6], acting_primary 0(0) -> 0, up_primary 0(0) -> 0, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.161265 7f35480a2700 1 osd.10 pg_epoch: 464 pg[13.1s1( v 377'9368 (355'1456,377'9368] local-lis/les=428/430 n=2661 ec=345/345 lis/c 428/428 les/c/f 430/430/0 464/464/464) [2147483647,2147483647,2147483647,2147483647,6,7]p6(4) r=-1 lpr=464 pi=[428,464)/1 crt=377'9368 lcod 0'0 unknown NOTIFY] start_peering_interval up [8,10,0,2,6,7] -> [2147483647,2147483647,2147483647,2147483647,6,7], acting [8,10,0,2,6,7] -> [2147483647,2147483647,2147483647,2147483647,6,7], acting_primary 8(0) -> 6, up_primary 8(0) -> 6, role 1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.161331 7f35480a2700 1 osd.10 pg_epoch: 465 pg[13.1s1( v 377'9368 (355'1456,377'9368] local-lis/les=428/430 n=2661 ec=345/345 lis/c 428/428 les/c/f 430/430/0 465/465/465) [2147483647,2147483647,2147483647,2,6,7]p2(3) r=-1 lpr=465 pi=[428,465)/1 crt=377'9368 lcod 0'0 unknown NOTIFY] start_peering_interval up [2147483647,2147483647,2147483647,2147483647,6,7] -> [2147483647,2147483647,2147483647,2,6,7], acting [2147483647,2147483647,2147483647,2147483647,6,7] -> [2147483647,2147483647,2147483647,2,6,7], acting_primary 6(4) -> 2, up_primary 6(4) -> 2, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.161410 7f35480a2700 1 osd.10 pg_epoch: 467 pg[13.1s1( v 377'9368 (355'1456,377'9368] local-lis/les=428/430 n=2661 ec=345/345 lis/c 428/428 les/c/f 430/430/0 467/467/465) [2147483647,2147483647,2147483647,2,6,2147483647]p2(3) r=-1 lpr=467 pi=[428,467)/1 crt=377'9368 lcod 0'0 unknown NOTIFY] start_peering_interval up [2147483647,2147483647,2147483647,2,6,7] -> [2147483647,2147483647,2147483647,2,6,2147483647], acting [2147483647,2147483647,2147483647,2,6,7] -> [2147483647,2147483647,2147483647,2,6,2147483647], acting_primary 2(3) -> 2, up_primary 2(3) -> 2, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.161436 7f35478a1700 1 osd.10 pg_epoch: 492 pg[13.ds1( v 377'10591 (355'1551,377'10591] local-lis/les=428/429 n=2744 ec=345/345 lis/c 428/428 les/c/f 429/429/0 492/492/470) [0,10,1,8,11,6]p0(0) r=1 lpr=492 pi=[428,492)/1 crt=377'10591 lcod 0'0 unknown NOTIFY] start_peering_interval up [0,2147483647,1,8,11,6] -> [0,10,1,8,11,6], acting [0,2147483647,1,8,11,6] -> [0,10,1,8,11,6], acting_primary 0(0) -> 0, up_primary 0(0) -> 0, role -1 -> 1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.161466 7f35480a2700 1 osd.10 pg_epoch: 468 pg[13.1s1( v 377'9368 (355'1456,377'9368] local-lis/les=428/430 n=2661 ec=345/345 lis/c 428/428 les/c/f 430/430/0 468/468/465) [2147483647,2147483647,2147483647,2,6,7]p2(3) r=-1 lpr=468 pi=[428,468)/1 crt=377'9368 lcod 0'0 unknown NOTIFY] start_peering_interval up [2147483647,2147483647,2147483647,2,6,2147483647] -> [2147483647,2147483647,2147483647,2,6,7], acting [2147483647,2147483647,2147483647,2,6,2147483647] -> [2147483647,2147483647,2147483647,2,6,7], acting_primary 2(3) -> 2, up_primary 2(3) -> 2, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.161508 7f35478a1700 1 osd.10 pg_epoch: 492 pg[13.ds1( v 377'10591 (355'1551,377'10591] local-lis/les=428/429 n=2744 ec=345/345 lis/c 428/428 les/c/f 429/429/0 492/492/470) [0,10,1,8,11,6]p0(0) r=1 lpr=492 pi=[428,492)/1 crt=377'10591 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:38.161539 7f35480a2700 1 osd.10 pg_epoch: 470 pg[13.1s1( v 377'9368 (355'1456,377'9368] local-lis/les=428/430 n=2661 ec=345/345 lis/c 428/428 les/c/f 430/430/0 470/470/470) [2147483647,2147483647,0,2,6,7]p0(2) r=-1 lpr=470 pi=[428,470)/1 crt=377'9368 lcod 0'0 unknown NOTIFY] start_peering_interval up [2147483647,2147483647,2147483647,2,6,7] -> [2147483647,2147483647,0,2,6,7], acting [2147483647,2147483647,2147483647,2,6,7] -> [2147483647,2147483647,0,2,6,7], acting_primary 2(3) -> 0, up_primary 2(3) -> 0, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.161611 7f35480a2700 1 osd.10 pg_epoch: 472 pg[13.1s1( v 377'9368 (355'1456,377'9368] local-lis/les=428/430 n=2661 ec=345/345 lis/c 428/428 les/c/f 430/430/0 472/472/472) [8,2147483647,0,2,6,7]p8(0) r=-1 lpr=472 pi=[428,472)/1 crt=377'9368 lcod 0'0 unknown NOTIFY] start_peering_interval up [2147483647,2147483647,0,2,6,7] -> [8,2147483647,0,2,6,7], acting [2147483647,2147483647,0,2,6,7] -> [8,2147483647,0,2,6,7], acting_primary 0(2) -> 8, up_primary 0(2) -> 8, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.161661 7f35478a1700 1 osd.10 pg_epoch: 464 pg[13.cs2( v 377'10679 (371'2117,377'10679] local-lis/les=428/430 n=2785 ec=345/345 lis/c 428/428 les/c/f 430/430/0 464/464/464) [2147483647,11,2147483647,2147483647,2147483647,6]p11(1) r=-1 lpr=464 pi=[428,464)/1 crt=377'10679 lcod 0'0 unknown NOTIFY] start_peering_interval up [1,11,10,4,8,6] -> [2147483647,11,2147483647,2147483647,2147483647,6], acting [1,11,10,4,8,6] -> [2147483647,11,2147483647,2147483647,2147483647,6], acting_primary 1(0) -> 11, up_primary 1(0) -> 11, role 2 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.161771 7f35478a1700 1 osd.10 pg_epoch: 470 pg[13.cs2( v 377'10679 (371'2117,377'10679] local-lis/les=428/430 n=2785 ec=345/345 lis/c 428/428 les/c/f 430/430/0 470/470/470) [1,11,2147483647,2147483647,2147483647,6]p1(0) r=-1 lpr=470 pi=[428,470)/1 crt=377'10679 lcod 0'0 unknown NOTIFY] start_peering_interval up [2147483647,11,2147483647,2147483647,2147483647,6] -> [1,11,2147483647,2147483647,2147483647,6], acting [2147483647,11,2147483647,2147483647,2147483647,6] -> [1,11,2147483647,2147483647,2147483647,6], acting_primary 11(1) -> 1, up_primary 11(1) -> 1, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.161801 7f35480a2700 1 osd.10 pg_epoch: 480 pg[13.1s1( v 377'9368 (355'1456,377'9368] local-lis/les=428/430 n=2661 ec=345/345 lis/c 428/428 les/c/f 430/430/0 480/480/472) [8,2147483647,0,2,2147483647,7]p8(0) r=-1 lpr=480 pi=[428,480)/1 crt=377'9368 lcod 0'0 unknown NOTIFY] start_peering_interval up [8,2147483647,0,2,6,7] -> [8,2147483647,0,2,2147483647,7], acting [8,2147483647,0,2,6,7] -> [8,2147483647,0,2,2147483647,7], acting_primary 8(0) -> 8, up_primary 8(0) -> 8, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.161833 7f35478a1700 1 osd.10 pg_epoch: 472 pg[13.cs2( v 377'10679 (371'2117,377'10679] local-lis/les=428/430 n=2785 ec=345/345 lis/c 428/428 les/c/f 430/430/0 472/472/470) [1,11,2147483647,2147483647,8,6]p1(0) r=-1 lpr=472 pi=[428,472)/1 crt=377'10679 lcod 0'0 unknown NOTIFY] start_peering_interval up [1,11,2147483647,2147483647,2147483647,6] -> [1,11,2147483647,2147483647,8,6], acting [1,11,2147483647,2147483647,2147483647,6] -> [1,11,2147483647,2147483647,8,6], acting_primary 1(0) -> 1, up_primary 1(0) -> 1, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.161863 7f35480a2700 1 osd.10 pg_epoch: 481 pg[13.1s1( v 377'9368 (355'1456,377'9368] local-lis/les=428/430 n=2661 ec=345/345 lis/c 428/428 les/c/f 430/430/0 481/481/472) [8,2147483647,0,2,6,7]p8(0) r=-1 lpr=481 pi=[428,481)/1 crt=377'9368 lcod 0'0 unknown NOTIFY] start_peering_interval up [8,2147483647,0,2,2147483647,7] -> [8,2147483647,0,2,6,7], acting [8,2147483647,0,2,2147483647,7] -> [8,2147483647,0,2,6,7], acting_primary 8(0) -> 8, up_primary 8(0) -> 8, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.161934 7f35478a1700 1 osd.10 pg_epoch: 477 pg[13.cs2( v 377'10679 (371'2117,377'10679] local-lis/les=428/430 n=2785 ec=345/345 lis/c 428/428 les/c/f 430/430/0 477/477/470) [1,2147483647,2147483647,2147483647,8,6]p1(0) r=-1 lpr=477 pi=[428,477)/1 crt=377'10679 lcod 0'0 unknown NOTIFY] start_peering_interval up [1,11,2147483647,2147483647,8,6] -> [1,2147483647,2147483647,2147483647,8,6], acting [1,11,2147483647,2147483647,8,6] -> [1,2147483647,2147483647,2147483647,8,6], acting_primary 1(0) -> 1, up_primary 1(0) -> 1, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.161984 7f35478a1700 1 osd.10 pg_epoch: 478 pg[13.cs2( v 377'10679 (371'2117,377'10679] local-lis/les=428/430 n=2785 ec=345/345 lis/c 428/428 les/c/f 430/430/0 478/478/470) [1,11,2147483647,2147483647,8,6]p1(0) r=-1 lpr=478 pi=[428,478)/1 crt=377'10679 lcod 0'0 unknown NOTIFY] start_peering_interval up [1,2147483647,2147483647,2147483647,8,6] -> [1,11,2147483647,2147483647,8,6], acting [1,2147483647,2147483647,2147483647,8,6] -> [1,11,2147483647,2147483647,8,6], acting_primary 1(0) -> 1, up_primary 1(0) -> 1, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.162045 7f35478a1700 1 osd.10 pg_epoch: 480 pg[13.cs2( v 377'10679 (371'2117,377'10679] local-lis/les=428/430 n=2785 ec=345/345 lis/c 428/428 les/c/f 430/430/0 480/480/470) [1,11,2147483647,2147483647,8,2147483647]p1(0) r=-1 lpr=480 pi=[428,480)/1 crt=377'10679 lcod 0'0 unknown NOTIFY] start_peering_interval up [1,11,2147483647,2147483647,8,6] -> [1,11,2147483647,2147483647,8,2147483647], acting [1,11,2147483647,2147483647,8,6] -> [1,11,2147483647,2147483647,8,2147483647], acting_primary 1(0) -> 1, up_primary 1(0) -> 1, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.162093 7f35478a1700 1 osd.10 pg_epoch: 481 pg[13.cs2( v 377'10679 (371'2117,377'10679] local-lis/les=428/430 n=2785 ec=345/345 lis/c 428/428 les/c/f 430/430/0 481/481/470) [1,11,2147483647,2147483647,8,6]p1(0) r=-1 lpr=481 pi=[428,481)/1 crt=377'10679 lcod 0'0 unknown NOTIFY] start_peering_interval up [1,11,2147483647,2147483647,8,2147483647] -> [1,11,2147483647,2147483647,8,6], acting [1,11,2147483647,2147483647,8,2147483647] -> [1,11,2147483647,2147483647,8,6], acting_primary 1(0) -> 1, up_primary 1(0) -> 1, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.162097 7f35480a2700 1 osd.10 pg_epoch: 492 pg[13.1s1( v 377'9368 (355'1456,377'9368] local-lis/les=428/430 n=2661 ec=345/345 lis/c 428/428 les/c/f 430/430/0 492/492/472) [8,10,0,2,6,7]p8(0) r=1 lpr=492 pi=[428,492)/1 crt=377'9368 lcod 0'0 unknown NOTIFY] start_peering_interval up [8,2147483647,0,2,6,7] -> [8,10,0,2,6,7], acting [8,2147483647,0,2,6,7] -> [8,10,0,2,6,7], acting_primary 8(0) -> 8, up_primary 8(0) -> 8, role -1 -> 1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.162168 7f35480a2700 1 osd.10 pg_epoch: 492 pg[13.1s1( v 377'9368 (355'1456,377'9368] local-lis/les=428/430 n=2661 ec=345/345 lis/c 428/428 les/c/f 430/430/0 492/492/472) [8,10,0,2,6,7]p8(0) r=1 lpr=492 pi=[428,492)/1 crt=377'9368 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:38.162193 7f35478a1700 1 osd.10 pg_epoch: 487 pg[13.cs2( v 377'10679 (371'2117,377'10679] local-lis/les=428/430 n=2785 ec=345/345 lis/c 428/428 les/c/f 430/430/0 487/487/470) [1,11,2147483647,4,8,6]p1(0) r=-1 lpr=487 pi=[428,487)/1 crt=377'10679 lcod 0'0 unknown NOTIFY] start_peering_interval up [1,11,2147483647,2147483647,8,6] -> [1,11,2147483647,4,8,6], acting [1,11,2147483647,2147483647,8,6] -> [1,11,2147483647,4,8,6], acting_primary 1(0) -> 1, up_primary 1(0) -> 1, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.162321 7f35478a1700 1 osd.10 pg_epoch: 492 pg[13.cs2( v 377'10679 (371'2117,377'10679] local-lis/les=428/430 n=2785 ec=345/345 lis/c 428/428 les/c/f 430/430/0 492/492/470) [1,11,10,4,8,6]p1(0) r=2 lpr=492 pi=[428,492)/1 crt=377'10679 lcod 0'0 unknown NOTIFY] start_peering_interval up [1,11,2147483647,4,8,6] -> [1,11,10,4,8,6], acting [1,11,2147483647,4,8,6] -> [1,11,10,4,8,6], acting_primary 1(0) -> 1, up_primary 1(0) -> 1, role -1 -> 2, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.162338 7f35480a2700 1 osd.10 pg_epoch: 464 pg[11.3s5( v 383'60232 (352'56579,383'60232] local-lis/les=428/429 n=60232 ec=267/267 lis/c 428/428 les/c/f 429/429/0 464/464/428) [6,2147483647,2147483647,11,2147483647,2147483647]p6(0) r=-1 lpr=464 pi=[428,464)/1 crt=383'60232 lcod 0'0 unknown NOTIFY] start_peering_interval up [6,8,0,11,2,10] -> [6,2147483647,2147483647,11,2147483647,2147483647], acting [6,8,0,11,2,10] -> [6,2147483647,2147483647,11,2147483647,2147483647], acting_primary 6(0) -> 6, up_primary 6(0) -> 6, role 5 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.162401 7f35478a1700 1 osd.10 pg_epoch: 492 pg[13.cs2( v 377'10679 (371'2117,377'10679] local-lis/les=428/430 n=2785 ec=345/345 lis/c 428/428 les/c/f 430/430/0 492/492/470) [1,11,10,4,8,6]p1(0) r=2 lpr=492 pi=[428,492)/1 crt=377'10679 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:38.162398 7f35480a2700 1 osd.10 pg_epoch: 465 pg[11.3s5( v 383'60232 (352'56579,383'60232] local-lis/les=428/429 n=60232 ec=267/267 lis/c 428/428 les/c/f 429/429/0 465/465/428) [6,2147483647,2147483647,11,2,2147483647]p6(0) r=-1 lpr=465 pi=[428,465)/1 crt=383'60232 lcod 0'0 unknown NOTIFY] start_peering_interval up [6,2147483647,2147483647,11,2147483647,2147483647] -> [6,2147483647,2147483647,11,2,2147483647], acting [6,2147483647,2147483647,11,2147483647,2147483647] -> [6,2147483647,2147483647,11,2,2147483647], acting_primary 6(0) -> 6, up_primary 6(0) -> 6, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.162505 7f35480a2700 1 osd.10 pg_epoch: 470 pg[11.3s5( v 383'60232 (352'56579,383'60232] local-lis/les=428/429 n=60232 ec=267/267 lis/c 428/428 les/c/f 429/429/0 470/470/428) [6,2147483647,0,11,2,2147483647]p6(0) r=-1 lpr=470 pi=[428,470)/1 crt=383'60232 lcod 0'0 unknown NOTIFY] start_peering_interval up [6,2147483647,2147483647,11,2,2147483647] -> [6,2147483647,0,11,2,2147483647], acting [6,2147483647,2147483647,11,2,2147483647] -> [6,2147483647,0,11,2,2147483647], acting_primary 6(0) -> 6, up_primary 6(0) -> 6, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.162545 7f35478a1700 1 osd.10 pg_epoch: 464 pg[16.10( empty local-lis/les=442/443 n=0 ec=442/442 lis/c 442/442 les/c/f 443/443/0 464/464/464) [7] r=-1 lpr=464 pi=[442,464)/1 crt=0'0 unknown NOTIFY] start_peering_interval up [2,10,7] -> [7], acting [2,10,7] -> [7], acting_primary 2 -> 7, up_primary 2 -> 7, role 1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.162570 7f35480a2700 1 osd.10 pg_epoch: 472 pg[11.3s5( v 383'60232 (352'56579,383'60232] local-lis/les=428/429 n=60232 ec=267/267 lis/c 428/428 les/c/f 429/429/0 472/472/428) [6,8,0,11,2,2147483647]p6(0) r=-1 lpr=472 pi=[428,472)/1 crt=383'60232 lcod 0'0 unknown NOTIFY] start_peering_interval up [6,2147483647,0,11,2,2147483647] -> [6,8,0,11,2,2147483647], acting [6,2147483647,0,11,2,2147483647] -> [6,8,0,11,2,2147483647], acting_primary 6(0) -> 6, up_primary 6(0) -> 6, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.162591 7f35478a1700 1 osd.10 pg_epoch: 465 pg[16.10( empty local-lis/les=442/443 n=0 ec=442/442 lis/c 442/442 les/c/f 443/443/0 465/465/465) [2,7] r=-1 lpr=465 pi=[442,465)/1 crt=0'0 unknown NOTIFY] start_peering_interval up [7] -> [2,7], acting [7] -> [2,7], acting_primary 7 -> 2, up_primary 7 -> 2, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.162643 7f35478a1700 1 osd.10 pg_epoch: 467 pg[16.10( empty local-lis/les=442/443 n=0 ec=442/442 lis/c 442/442 les/c/f 443/443/0 467/467/465) [2] r=-1 lpr=467 pi=[442,467)/1 crt=0'0 unknown NOTIFY] start_peering_interval up [2,7] -> [2], acting [2,7] -> [2], acting_primary 2 -> 2, up_primary 2 -> 2, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.162682 7f35478a1700 1 osd.10 pg_epoch: 468 pg[16.10( empty local-lis/les=442/443 n=0 ec=442/442 lis/c 442/442 les/c/f 443/443/0 468/468/465) [2,7] r=-1 lpr=468 pi=[442,468)/1 crt=0'0 unknown NOTIFY] start_peering_interval up [2] -> [2,7], acting [2] -> [2,7], acting_primary 2 -> 2, up_primary 2 -> 2, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.162688 7f35480a2700 1 osd.10 pg_epoch: 477 pg[11.3s5( v 383'60232 (352'56579,383'60232] local-lis/les=428/429 n=60232 ec=267/267 lis/c 428/428 les/c/f 429/429/0 477/477/428) [6,8,0,2147483647,2,2147483647]p6(0) r=-1 lpr=477 pi=[428,477)/1 crt=383'60232 lcod 0'0 unknown NOTIFY] start_peering_interval up [6,8,0,11,2,2147483647] -> [6,8,0,2147483647,2,2147483647], acting [6,8,0,11,2,2147483647] -> [6,8,0,2147483647,2,2147483647], acting_primary 6(0) -> 6, up_primary 6(0) -> 6, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.162740 7f35480a2700 1 osd.10 pg_epoch: 478 pg[11.3s5( v 383'60232 (352'56579,383'60232] local-lis/les=428/429 n=60232 ec=267/267 lis/c 428/428 les/c/f 429/429/0 478/478/428) [6,8,0,11,2,2147483647]p6(0) r=-1 lpr=478 pi=[428,478)/1 crt=383'60232 lcod 0'0 unknown NOTIFY] start_peering_interval up [6,8,0,2147483647,2,2147483647] -> [6,8,0,11,2,2147483647], acting [6,8,0,2147483647,2,2147483647] -> [6,8,0,11,2,2147483647], acting_primary 6(0) -> 6, up_primary 6(0) -> 6, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.162812 7f35480a2700 1 osd.10 pg_epoch: 480 pg[11.3s5( v 383'60232 (352'56579,383'60232] local-lis/les=428/429 n=60232 ec=267/267 lis/c 428/428 les/c/f 429/429/0 480/480/480) [2147483647,8,0,11,2,2147483647]p8(1) r=-1 lpr=480 pi=[428,480)/1 crt=383'60232 lcod 0'0 unknown NOTIFY] start_peering_interval up [6,8,0,11,2,2147483647] -> [2147483647,8,0,11,2,2147483647], acting [6,8,0,11,2,2147483647] -> [2147483647,8,0,11,2,2147483647], acting_primary 6(0) -> 8, up_primary 6(0) -> 8, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.162868 7f35480a2700 1 osd.10 pg_epoch: 481 pg[11.3s5( v 383'60232 (352'56579,383'60232] local-lis/les=428/429 n=60232 ec=267/267 lis/c 428/428 les/c/f 429/429/0 481/481/481) [6,8,0,11,2,2147483647]p6(0) r=-1 lpr=481 pi=[428,481)/1 crt=383'60232 lcod 0'0 unknown NOTIFY] start_peering_interval up [2147483647,8,0,11,2,2147483647] -> [6,8,0,11,2,2147483647], acting [2147483647,8,0,11,2,2147483647] -> [6,8,0,11,2,2147483647], acting_primary 8(1) -> 6, up_primary 8(1) -> 6, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.162887 7f35478a1700 1 osd.10 pg_epoch: 492 pg[16.10( empty local-lis/les=442/443 n=0 ec=442/442 lis/c 442/442 les/c/f 443/443/0 492/492/465) [2,10,7] r=1 lpr=492 pi=[442,492)/1 crt=0'0 unknown NOTIFY] start_peering_interval up [2,7] -> [2,10,7], acting [2,7] -> [2,10,7], acting_primary 2 -> 2, up_primary 2 -> 2, role -1 -> 1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.162952 7f35478a1700 1 osd.10 pg_epoch: 492 pg[16.10( empty local-lis/les=442/443 n=0 ec=442/442 lis/c 442/442 les/c/f 443/443/0 492/492/465) [2,10,7] r=1 lpr=492 pi=[442,492)/1 crt=0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:38.163033 7f35478a1700 1 osd.10 pg_epoch: 464 pg[10.a( v 386'2032 (203'500,386'2032] local-lis/les=408/410 n=516 ec=193/193 lis/c 408/408 les/c/f 410/449/0 408/408/403) [10,2,5] r=0 lpr=461 crt=386'2032 lcod 0'0 mlcod 0'0 peering] state<Started/Primary/Peering>: Peering, affected_by_map, going to Reset >2018-07-11 05:55:38.163065 7f35480a2700 1 osd.10 pg_epoch: 492 pg[11.3s5( v 383'60232 (352'56579,383'60232] local-lis/les=428/429 n=60232 ec=267/267 lis/c 428/428 les/c/f 429/429/0 492/492/481) [6,8,0,11,2,10]p6(0) r=5 lpr=492 pi=[428,492)/1 crt=383'60232 lcod 0'0 unknown NOTIFY] start_peering_interval up [6,8,0,11,2,2147483647] -> [6,8,0,11,2,10], acting [6,8,0,11,2,2147483647] -> [6,8,0,11,2,10], acting_primary 6(0) -> 6, up_primary 6(0) -> 6, role -1 -> 5, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.163136 7f35478a1700 1 osd.10 pg_epoch: 464 pg[10.a( v 386'2032 (203'500,386'2032] local-lis/les=408/410 n=516 ec=193/193 lis/c 408/408 les/c/f 410/449/0 464/464/464) [] r=-1 lpr=464 pi=[408,464)/1 crt=386'2032 lcod 0'0 unknown] start_peering_interval up [10,2,5] -> [], acting [10,2,5] -> [], acting_primary 10 -> -1, up_primary 10 -> -1, role 0 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.163144 7f35480a2700 1 osd.10 pg_epoch: 492 pg[11.3s5( v 383'60232 (352'56579,383'60232] local-lis/les=428/429 n=60232 ec=267/267 lis/c 428/428 les/c/f 429/429/0 492/492/481) [6,8,0,11,2,10]p6(0) r=5 lpr=492 pi=[428,492)/1 crt=383'60232 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:38.163180 7f35478a1700 1 osd.10 pg_epoch: 465 pg[10.a( v 386'2032 (203'500,386'2032] local-lis/les=408/410 n=516 ec=193/193 lis/c 408/408 les/c/f 410/449/0 465/465/465) [2] r=-1 lpr=465 pi=[408,465)/1 crt=386'2032 lcod 0'0 unknown NOTIFY] start_peering_interval up [] -> [2], acting [] -> [2], acting_primary ? -> 2, up_primary ? -> 2, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.163224 7f35478a1700 1 osd.10 pg_epoch: 467 pg[10.a( v 386'2032 (203'500,386'2032] local-lis/les=408/410 n=516 ec=193/193 lis/c 408/408 les/c/f 410/449/0 467/467/465) [2,5] r=-1 lpr=467 pi=[408,467)/1 crt=386'2032 lcod 0'0 unknown NOTIFY] start_peering_interval up [2] -> [2,5], acting [2] -> [2,5], acting_primary 2 -> 2, up_primary 2 -> 2, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.163235 7f35480a2700 1 osd.10 pg_epoch: 464 pg[11.6s0( v 383'59732 (352'56233,383'59732] local-lis/les=428/429 n=59732 ec=267/267 lis/c 428/428 les/c/f 429/429/0 428/428/403) [10,1,11,6,5,8]p10(0) r=0 lpr=461 crt=383'59732 lcod 0'0 mlcod 0'0 peering] state<Started/Primary/Peering>: Peering, affected_by_map, going to Reset >2018-07-11 05:55:38.163354 7f35480a2700 1 osd.10 pg_epoch: 464 pg[11.6s0( v 383'59732 (352'56233,383'59732] local-lis/les=428/429 n=59732 ec=267/267 lis/c 428/428 les/c/f 429/429/0 464/464/464) [2147483647,2147483647,11,6,2147483647,2147483647]p11(2) r=-1 lpr=464 pi=[428,464)/1 crt=383'59732 lcod 0'0 unknown] start_peering_interval up [10,1,11,6,5,8] -> [2147483647,2147483647,11,6,2147483647,2147483647], acting [10,1,11,6,5,8] -> [2147483647,2147483647,11,6,2147483647,2147483647], acting_primary 10(0) -> 11, up_primary 10(0) -> 11, role 0 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.163425 7f35478a1700 1 osd.10 pg_epoch: 492 pg[10.a( v 386'2032 (203'500,386'2032] local-lis/les=408/410 n=516 ec=193/193 lis/c 408/408 les/c/f 410/449/0 492/492/492) [10,2,5] r=0 lpr=492 pi=[408,492)/1 crt=386'2032 lcod 0'0 mlcod 0'0 unknown NOTIFY] start_peering_interval up [2,5] -> [10,2,5], acting [2,5] -> [10,2,5], acting_primary 2 -> 10, up_primary 2 -> 10, role -1 -> 0, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.163445 7f35480a2700 1 osd.10 pg_epoch: 467 pg[11.6s0( v 383'59732 (352'56233,383'59732] local-lis/les=428/429 n=59732 ec=267/267 lis/c 428/428 les/c/f 429/429/0 467/467/464) [2147483647,2147483647,11,6,5,2147483647]p11(2) r=-1 lpr=467 pi=[428,467)/1 crt=383'59732 lcod 0'0 unknown NOTIFY] start_peering_interval up [2147483647,2147483647,11,6,2147483647,2147483647] -> [2147483647,2147483647,11,6,5,2147483647], acting [2147483647,2147483647,11,6,2147483647,2147483647] -> [2147483647,2147483647,11,6,5,2147483647], acting_primary 11(2) -> 11, up_primary 11(2) -> 11, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.163495 7f35478a1700 1 osd.10 pg_epoch: 492 pg[10.a( v 386'2032 (203'500,386'2032] local-lis/les=408/410 n=516 ec=193/193 lis/c 408/408 les/c/f 410/449/0 492/492/492) [10,2,5] r=0 lpr=492 pi=[408,492)/1 crt=386'2032 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 05:55:38.163528 7f35480a2700 1 osd.10 pg_epoch: 470 pg[11.6s0( v 383'59732 (352'56233,383'59732] local-lis/les=428/429 n=59732 ec=267/267 lis/c 428/428 les/c/f 429/429/0 470/470/470) [2147483647,1,11,6,5,2147483647]p1(1) r=-1 lpr=470 pi=[428,470)/1 crt=383'59732 lcod 0'0 unknown NOTIFY] start_peering_interval up [2147483647,2147483647,11,6,5,2147483647] -> [2147483647,1,11,6,5,2147483647], acting [2147483647,2147483647,11,6,5,2147483647] -> [2147483647,1,11,6,5,2147483647], acting_primary 11(2) -> 1, up_primary 11(2) -> 1, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.163598 7f35480a2700 1 osd.10 pg_epoch: 472 pg[11.6s0( v 383'59732 (352'56233,383'59732] local-lis/les=428/429 n=59732 ec=267/267 lis/c 428/428 les/c/f 429/429/0 472/472/470) [2147483647,1,11,6,5,8]p1(1) r=-1 lpr=472 pi=[428,472)/1 crt=383'59732 lcod 0'0 unknown NOTIFY] start_peering_interval up [2147483647,1,11,6,5,2147483647] -> [2147483647,1,11,6,5,8], acting [2147483647,1,11,6,5,2147483647] -> [2147483647,1,11,6,5,8], acting_primary 1(1) -> 1, up_primary 1(1) -> 1, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.163663 7f35478a1700 1 osd.10 pg_epoch: 464 pg[5.23( v 166'68497 (166'66949,166'68497] local-lis/les=403/404 n=68497 ec=71/71 lis/c 403/403 les/c/f 404/404/0 464/464/464) [] r=-1 lpr=464 pi=[403,464)/1 crt=166'68497 lcod 0'0 unknown NOTIFY] start_peering_interval up [8,3,10] -> [], acting [8,3,10] -> [], acting_primary 8 -> -1, up_primary 8 -> -1, role 2 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.163712 7f35480a2700 1 osd.10 pg_epoch: 477 pg[11.6s0( v 383'59732 (352'56233,383'59732] local-lis/les=428/429 n=59732 ec=267/267 lis/c 428/428 les/c/f 429/429/0 477/477/470) [2147483647,1,2147483647,6,5,8]p1(1) r=-1 lpr=477 pi=[428,477)/1 crt=383'59732 lcod 0'0 unknown NOTIFY] start_peering_interval up [2147483647,1,11,6,5,8] -> [2147483647,1,2147483647,6,5,8], acting [2147483647,1,11,6,5,8] -> [2147483647,1,2147483647,6,5,8], acting_primary 1(1) -> 1, up_primary 1(1) -> 1, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.163754 7f35478a1700 1 osd.10 pg_epoch: 472 pg[5.23( v 166'68497 (166'66949,166'68497] local-lis/les=403/404 n=68497 ec=71/71 lis/c 403/403 les/c/f 404/404/0 472/472/472) [8] r=-1 lpr=472 pi=[403,472)/1 crt=166'68497 lcod 0'0 unknown NOTIFY] start_peering_interval up [] -> [8], acting [] -> [8], acting_primary ? -> 8, up_primary ? -> 8, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.163767 7f35480a2700 1 osd.10 pg_epoch: 478 pg[11.6s0( v 383'59732 (352'56233,383'59732] local-lis/les=428/429 n=59732 ec=267/267 lis/c 428/428 les/c/f 429/429/0 478/478/470) [2147483647,1,11,6,5,8]p1(1) r=-1 lpr=478 pi=[428,478)/1 crt=383'59732 lcod 0'0 unknown NOTIFY] start_peering_interval up [2147483647,1,2147483647,6,5,8] -> [2147483647,1,11,6,5,8], acting [2147483647,1,2147483647,6,5,8] -> [2147483647,1,11,6,5,8], acting_primary 1(1) -> 1, up_primary 1(1) -> 1, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.163844 7f35480a2700 1 osd.10 pg_epoch: 480 pg[11.6s0( v 383'59732 (352'56233,383'59732] local-lis/les=428/429 n=59732 ec=267/267 lis/c 428/428 les/c/f 429/429/0 480/480/470) [2147483647,1,11,2147483647,5,8]p1(1) r=-1 lpr=480 pi=[428,480)/1 crt=383'59732 lcod 0'0 unknown NOTIFY] start_peering_interval up [2147483647,1,11,6,5,8] -> [2147483647,1,11,2147483647,5,8], acting [2147483647,1,11,6,5,8] -> [2147483647,1,11,2147483647,5,8], acting_primary 1(1) -> 1, up_primary 1(1) -> 1, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.163877 7f35478a1700 1 osd.10 pg_epoch: 484 pg[5.23( v 166'68497 (166'66949,166'68497] local-lis/les=403/404 n=68497 ec=71/71 lis/c 403/403 les/c/f 404/404/0 484/484/472) [8,3] r=-1 lpr=484 pi=[403,484)/1 crt=166'68497 lcod 0'0 unknown NOTIFY] start_peering_interval up [8] -> [8,3], acting [8] -> [8,3], acting_primary 8 -> 8, up_primary 8 -> 8, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.163899 7f35480a2700 1 osd.10 pg_epoch: 481 pg[11.6s0( v 383'59732 (352'56233,383'59732] local-lis/les=428/429 n=59732 ec=267/267 lis/c 428/428 les/c/f 429/429/0 481/481/470) [2147483647,1,11,6,5,8]p1(1) r=-1 lpr=481 pi=[428,481)/1 crt=383'59732 lcod 0'0 unknown NOTIFY] start_peering_interval up [2147483647,1,11,2147483647,5,8] -> [2147483647,1,11,6,5,8], acting [2147483647,1,11,2147483647,5,8] -> [2147483647,1,11,6,5,8], acting_primary 1(1) -> 1, up_primary 1(1) -> 1, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.163969 7f35478a1700 1 osd.10 pg_epoch: 492 pg[5.23( v 166'68497 (166'66949,166'68497] local-lis/les=403/404 n=68497 ec=71/71 lis/c 403/403 les/c/f 404/404/0 492/492/472) [8,3,10] r=2 lpr=492 pi=[403,492)/2 crt=166'68497 lcod 0'0 unknown NOTIFY] start_peering_interval up [8,3] -> [8,3,10], acting [8,3] -> [8,3,10], acting_primary 8 -> 8, up_primary 8 -> 8, role -1 -> 2, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.164036 7f35478a1700 1 osd.10 pg_epoch: 492 pg[5.23( v 166'68497 (166'66949,166'68497] local-lis/les=403/404 n=68497 ec=71/71 lis/c 403/403 les/c/f 404/404/0 492/492/472) [8,3,10] r=2 lpr=492 pi=[403,492)/2 crt=166'68497 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:38.164095 7f35480a2700 1 osd.10 pg_epoch: 492 pg[11.6s0( v 383'59732 (352'56233,383'59732] local-lis/les=428/429 n=59732 ec=267/267 lis/c 428/428 les/c/f 429/429/0 492/492/492) [10,1,11,6,5,8]p10(0) r=0 lpr=492 pi=[428,492)/1 crt=383'59732 lcod 0'0 mlcod 0'0 unknown NOTIFY] start_peering_interval up [2147483647,1,11,6,5,8] -> [10,1,11,6,5,8], acting [2147483647,1,11,6,5,8] -> [10,1,11,6,5,8], acting_primary 1(1) -> 10, up_primary 1(1) -> 10, role -1 -> 0, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.164170 7f35480a2700 1 osd.10 pg_epoch: 492 pg[11.6s0( v 383'59732 (352'56233,383'59732] local-lis/les=428/429 n=59732 ec=267/267 lis/c 428/428 les/c/f 429/429/0 492/492/492) [10,1,11,6,5,8]p10(0) r=0 lpr=492 pi=[428,492)/1 crt=383'59732 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 05:55:38.164169 7f35478a1700 1 osd.10 pg_epoch: 464 pg[10.f( v 383'568 (0'0,383'568] local-lis/les=419/420 n=494 ec=193/193 lis/c 419/419 les/c/f 420/420/0 464/464/419) [11] r=-1 lpr=464 pi=[419,464)/1 crt=383'568 lcod 0'0 unknown NOTIFY] start_peering_interval up [11,10,2] -> [11], acting [11,10,2] -> [11], acting_primary 11 -> 11, up_primary 11 -> 11, role 1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.164211 7f35478a1700 1 osd.10 pg_epoch: 465 pg[10.f( v 383'568 (0'0,383'568] local-lis/les=419/420 n=494 ec=193/193 lis/c 419/419 les/c/f 420/420/0 465/465/419) [11,2] r=-1 lpr=465 pi=[419,465)/1 crt=383'568 lcod 0'0 unknown NOTIFY] start_peering_interval up [11] -> [11,2], acting [11] -> [11,2], acting_primary 11 -> 11, up_primary 11 -> 11, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.164306 7f35480a2700 1 osd.10 pg_epoch: 464 pg[6.13( v 430'99276 (430'97776,430'99276] local-lis/les=422/424 n=26101 ec=76/76 lis/c 422/422 les/c/f 424/457/0 422/422/403) [10,7,0] r=0 lpr=457 crt=430'99276 lcod 0'0 mlcod 0'0 peering] state<Started/Primary/Peering>: Peering, affected_by_map, going to Reset >2018-07-11 05:55:38.164324 7f35478a1700 1 osd.10 pg_epoch: 477 pg[10.f( v 383'568 (0'0,383'568] local-lis/les=419/420 n=494 ec=193/193 lis/c 419/419 les/c/f 420/420/0 477/477/477) [2] r=-1 lpr=477 pi=[419,477)/1 crt=383'568 lcod 0'0 unknown NOTIFY] start_peering_interval up [11,2] -> [2], acting [11,2] -> [2], acting_primary 11 -> 2, up_primary 11 -> 2, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.164366 7f35478a1700 1 osd.10 pg_epoch: 478 pg[10.f( v 383'568 (0'0,383'568] local-lis/les=419/420 n=494 ec=193/193 lis/c 419/419 les/c/f 420/420/0 478/478/478) [11,2] r=-1 lpr=478 pi=[419,478)/1 crt=383'568 lcod 0'0 unknown NOTIFY] start_peering_interval up [2] -> [11,2], acting [2] -> [11,2], acting_primary 2 -> 11, up_primary 2 -> 11, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.164401 7f35480a2700 1 osd.10 pg_epoch: 464 pg[6.13( v 430'99276 (430'97776,430'99276] local-lis/les=422/424 n=26101 ec=76/76 lis/c 422/422 les/c/f 424/457/0 464/464/464) [7] r=-1 lpr=464 pi=[422,464)/1 crt=430'99276 lcod 0'0 unknown] start_peering_interval up [10,7,0] -> [7], acting [10,7,0] -> [7], acting_primary 10 -> 7, up_primary 10 -> 7, role 0 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.164465 7f35480a2700 1 osd.10 pg_epoch: 467 pg[6.13( v 430'99276 (430'97776,430'99276] local-lis/les=422/424 n=26101 ec=76/76 lis/c 422/422 les/c/f 424/457/0 467/467/467) [] r=-1 lpr=467 pi=[422,467)/1 crt=430'99276 lcod 0'0 unknown NOTIFY] start_peering_interval up [7] -> [], acting [7] -> [], acting_primary 7 -> -1, up_primary 7 -> -1, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.164484 7f35478a1700 1 osd.10 pg_epoch: 492 pg[10.f( v 383'568 (0'0,383'568] local-lis/les=419/420 n=494 ec=193/193 lis/c 419/419 les/c/f 420/420/0 492/492/478) [11,10,2] r=1 lpr=492 pi=[419,492)/1 crt=383'568 lcod 0'0 unknown NOTIFY] start_peering_interval up [11,2] -> [11,10,2], acting [11,2] -> [11,10,2], acting_primary 11 -> 11, up_primary 11 -> 11, role -1 -> 1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.164505 7f35480a2700 1 osd.10 pg_epoch: 468 pg[6.13( v 430'99276 (430'97776,430'99276] local-lis/les=422/424 n=26101 ec=76/76 lis/c 422/422 les/c/f 424/457/0 468/468/468) [7] r=-1 lpr=468 pi=[422,468)/1 crt=430'99276 lcod 0'0 unknown NOTIFY] start_peering_interval up [] -> [7], acting [] -> [7], acting_primary ? -> 7, up_primary ? -> 7, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.164550 7f35478a1700 1 osd.10 pg_epoch: 492 pg[10.f( v 383'568 (0'0,383'568] local-lis/les=419/420 n=494 ec=193/193 lis/c 419/419 les/c/f 420/420/0 492/492/478) [11,10,2] r=1 lpr=492 pi=[419,492)/1 crt=383'568 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:38.164552 7f35480a2700 1 osd.10 pg_epoch: 470 pg[6.13( v 430'99276 (430'97776,430'99276] local-lis/les=422/424 n=26101 ec=76/76 lis/c 422/422 les/c/f 424/457/0 470/470/468) [7,0] r=-1 lpr=470 pi=[422,470)/1 crt=430'99276 lcod 0'0 unknown NOTIFY] start_peering_interval up [7] -> [7,0], acting [7] -> [7,0], acting_primary 7 -> 7, up_primary 7 -> 7, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.164721 7f35480a2700 1 osd.10 pg_epoch: 492 pg[6.13( v 430'99276 (430'97776,430'99276] local-lis/les=422/424 n=26101 ec=76/76 lis/c 422/422 les/c/f 424/457/0 492/492/492) [10,7,0] r=0 lpr=492 pi=[422,492)/2 crt=430'99276 lcod 0'0 mlcod 0'0 unknown NOTIFY] start_peering_interval up [7,0] -> [10,7,0], acting [7,0] -> [10,7,0], acting_primary 7 -> 10, up_primary 7 -> 10, role -1 -> 0, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.164791 7f35480a2700 1 osd.10 pg_epoch: 492 pg[6.13( v 430'99276 (430'97776,430'99276] local-lis/les=422/424 n=26101 ec=76/76 lis/c 422/422 les/c/f 424/457/0 492/492/492) [10,7,0] r=0 lpr=492 pi=[422,492)/2 crt=430'99276 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 05:55:38.164918 7f35478a1700 1 osd.10 pg_epoch: 464 pg[10.13( v 383'839 (0'0,383'839] local-lis/les=422/423 n=523 ec=193/193 lis/c 422/422 les/c/f 423/423/0 464/464/464) [7] r=-1 lpr=464 pi=[422,464)/1 crt=383'839 lcod 0'0 unknown NOTIFY] start_peering_interval up [1,10,7] -> [7], acting [1,10,7] -> [7], acting_primary 1 -> 7, up_primary 1 -> 7, role 1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.164948 7f35480a2700 1 osd.10 pg_epoch: 464 pg[9.1c( v 463'482 (0'0,463'482] local-lis/les=422/423 n=237 ec=190/190 lis/c 422/422 les/c/f 423/423/0 464/464/464) [7] r=-1 lpr=464 pi=[422,464)/1 crt=463'482 lcod 0'0 unknown NOTIFY] start_peering_interval up [1,10,7] -> [7], acting [1,10,7] -> [7], acting_primary 1 -> 7, up_primary 1 -> 7, role 1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.164980 7f35478a1700 1 osd.10 pg_epoch: 467 pg[10.13( v 383'839 (0'0,383'839] local-lis/les=422/423 n=523 ec=193/193 lis/c 422/422 les/c/f 423/423/0 467/467/467) [] r=-1 lpr=467 pi=[422,467)/1 crt=383'839 lcod 0'0 unknown NOTIFY] start_peering_interval up [7] -> [], acting [7] -> [], acting_primary 7 -> -1, up_primary 7 -> -1, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.165021 7f35478a1700 1 osd.10 pg_epoch: 468 pg[10.13( v 383'839 (0'0,383'839] local-lis/les=422/423 n=523 ec=193/193 lis/c 422/422 les/c/f 423/423/0 468/468/468) [7] r=-1 lpr=468 pi=[422,468)/1 crt=383'839 lcod 0'0 unknown NOTIFY] start_peering_interval up [] -> [7], acting [] -> [7], acting_primary ? -> 7, up_primary ? -> 7, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.165023 7f35480a2700 1 osd.10 pg_epoch: 467 pg[9.1c( v 463'482 (0'0,463'482] local-lis/les=422/423 n=237 ec=190/190 lis/c 422/422 les/c/f 423/423/0 467/467/467) [] r=-1 lpr=467 pi=[422,467)/1 crt=463'482 lcod 0'0 unknown NOTIFY] start_peering_interval up [7] -> [], acting [7] -> [], acting_primary 7 -> -1, up_primary 7 -> -1, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.165069 7f35478a1700 1 osd.10 pg_epoch: 470 pg[10.13( v 383'839 (0'0,383'839] local-lis/les=422/423 n=523 ec=193/193 lis/c 422/422 les/c/f 423/423/0 470/470/470) [1,7] r=-1 lpr=470 pi=[422,470)/1 crt=383'839 lcod 0'0 unknown NOTIFY] start_peering_interval up [7] -> [1,7], acting [7] -> [1,7], acting_primary 7 -> 1, up_primary 7 -> 1, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.165084 7f35480a2700 1 osd.10 pg_epoch: 468 pg[9.1c( v 463'482 (0'0,463'482] local-lis/les=422/423 n=237 ec=190/190 lis/c 422/422 les/c/f 423/423/0 468/468/468) [7] r=-1 lpr=468 pi=[422,468)/1 crt=463'482 lcod 0'0 unknown NOTIFY] start_peering_interval up [] -> [7], acting [] -> [7], acting_primary ? -> 7, up_primary ? -> 7, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.165167 7f35480a2700 1 osd.10 pg_epoch: 470 pg[9.1c( v 463'482 (0'0,463'482] local-lis/les=422/423 n=237 ec=190/190 lis/c 422/422 les/c/f 423/423/0 470/470/470) [1,7] r=-1 lpr=470 pi=[422,470)/1 crt=463'482 lcod 0'0 unknown NOTIFY] start_peering_interval up [7] -> [1,7], acting [7] -> [1,7], acting_primary 7 -> 1, up_primary 7 -> 1, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.165283 7f35478a1700 1 osd.10 pg_epoch: 492 pg[10.13( v 383'839 (0'0,383'839] local-lis/les=422/423 n=523 ec=193/193 lis/c 422/422 les/c/f 423/423/0 492/492/470) [1,10,7] r=1 lpr=492 pi=[422,492)/1 crt=383'839 lcod 0'0 unknown NOTIFY] start_peering_interval up [1,7] -> [1,10,7], acting [1,7] -> [1,10,7], acting_primary 1 -> 1, up_primary 1 -> 1, role -1 -> 1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.165351 7f35478a1700 1 osd.10 pg_epoch: 492 pg[10.13( v 383'839 (0'0,383'839] local-lis/les=422/423 n=523 ec=193/193 lis/c 422/422 les/c/f 423/423/0 492/492/470) [1,10,7] r=1 lpr=492 pi=[422,492)/1 crt=383'839 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:38.165429 7f35478a1700 1 osd.10 pg_epoch: 464 pg[5.9( v 166'68275 (166'66775,166'68275] local-lis/les=422/424 n=68275 ec=71/71 lis/c 422/422 les/c/f 424/424/0 422/422/403) [10,8,7] r=0 lpr=461 crt=166'68275 lcod 0'0 mlcod 0'0 peering] state<Started/Primary/Peering>: Peering, affected_by_map, going to Reset >2018-07-11 05:55:38.165466 7f35480a2700 1 osd.10 pg_epoch: 492 pg[9.1c( v 463'482 (0'0,463'482] local-lis/les=422/423 n=237 ec=190/190 lis/c 422/422 les/c/f 423/423/0 492/492/470) [1,10,7] r=1 lpr=492 pi=[422,492)/1 crt=463'482 lcod 0'0 unknown NOTIFY] start_peering_interval up [1,7] -> [1,10,7], acting [1,7] -> [1,10,7], acting_primary 1 -> 1, up_primary 1 -> 1, role -1 -> 1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.165520 7f35478a1700 1 osd.10 pg_epoch: 464 pg[5.9( v 166'68275 (166'66775,166'68275] local-lis/les=422/424 n=68275 ec=71/71 lis/c 422/422 les/c/f 424/424/0 464/464/464) [7] r=-1 lpr=464 pi=[422,464)/1 crt=166'68275 lcod 0'0 unknown] start_peering_interval up [10,8,7] -> [7], acting [10,8,7] -> [7], acting_primary 10 -> 7, up_primary 10 -> 7, role 0 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.165549 7f35480a2700 1 osd.10 pg_epoch: 492 pg[9.1c( v 463'482 (0'0,463'482] local-lis/les=422/423 n=237 ec=190/190 lis/c 422/422 les/c/f 423/423/0 492/492/470) [1,10,7] r=1 lpr=492 pi=[422,492)/1 crt=463'482 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:38.165583 7f35478a1700 1 osd.10 pg_epoch: 467 pg[5.9( v 166'68275 (166'66775,166'68275] local-lis/les=422/424 n=68275 ec=71/71 lis/c 422/422 les/c/f 424/424/0 467/467/467) [] r=-1 lpr=467 pi=[422,467)/1 crt=166'68275 lcod 0'0 unknown NOTIFY] start_peering_interval up [7] -> [], acting [7] -> [], acting_primary 7 -> -1, up_primary 7 -> -1, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.165624 7f35478a1700 1 osd.10 pg_epoch: 468 pg[5.9( v 166'68275 (166'66775,166'68275] local-lis/les=422/424 n=68275 ec=71/71 lis/c 422/422 les/c/f 424/424/0 468/468/468) [7] r=-1 lpr=468 pi=[422,468)/1 crt=166'68275 lcod 0'0 unknown NOTIFY] start_peering_interval up [] -> [7], acting [] -> [7], acting_primary ? -> 7, up_primary ? -> 7, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.165687 7f35478a1700 1 osd.10 pg_epoch: 472 pg[5.9( v 166'68275 (166'66775,166'68275] local-lis/les=422/424 n=68275 ec=71/71 lis/c 422/422 les/c/f 424/424/0 472/472/472) [8,7] r=-1 lpr=472 pi=[422,472)/1 crt=166'68275 lcod 0'0 unknown NOTIFY] start_peering_interval up [7] -> [8,7], acting [7] -> [8,7], acting_primary 7 -> 8, up_primary 7 -> 8, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.165698 7f35480a2700 1 osd.10 pg_epoch: 464 pg[9.1b( v 463'472 (0'0,463'472] local-lis/les=408/410 n=256 ec=190/190 lis/c 408/408 les/c/f 410/410/0 464/464/464) [] r=-1 lpr=464 pi=[408,464)/1 crt=463'472 lcod 0'0 unknown] start_peering_interval up [10,1,5] -> [], acting [10,1,5] -> [], acting_primary 10 -> -1, up_primary 10 -> -1, role 0 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.165797 7f35480a2700 1 osd.10 pg_epoch: 467 pg[9.1b( v 463'472 (0'0,463'472] local-lis/les=408/410 n=256 ec=190/190 lis/c 408/408 les/c/f 410/410/0 467/467/467) [5] r=-1 lpr=467 pi=[408,467)/1 crt=463'472 lcod 0'0 unknown NOTIFY] start_peering_interval up [] -> [5], acting [] -> [5], acting_primary ? -> 5, up_primary ? -> 5, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.165900 7f35478a1700 1 osd.10 pg_epoch: 492 pg[5.9( v 166'68275 (166'66775,166'68275] local-lis/les=422/424 n=68275 ec=71/71 lis/c 422/422 les/c/f 424/424/0 492/492/492) [10,8,7] r=0 lpr=492 pi=[422,492)/2 crt=166'68275 lcod 0'0 mlcod 0'0 unknown NOTIFY] start_peering_interval up [8,7] -> [10,8,7], acting [8,7] -> [10,8,7], acting_primary 8 -> 10, up_primary 8 -> 10, role -1 -> 0, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.165896 7f35480a2700 1 osd.10 pg_epoch: 470 pg[9.1b( v 463'472 (0'0,463'472] local-lis/les=408/410 n=256 ec=190/190 lis/c 408/408 les/c/f 410/410/0 470/470/470) [1,5] r=-1 lpr=470 pi=[408,470)/1 crt=463'472 lcod 0'0 unknown NOTIFY] start_peering_interval up [5] -> [1,5], acting [5] -> [1,5], acting_primary 5 -> 1, up_primary 5 -> 1, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.165970 7f35478a1700 1 osd.10 pg_epoch: 492 pg[5.9( v 166'68275 (166'66775,166'68275] local-lis/les=422/424 n=68275 ec=71/71 lis/c 422/422 les/c/f 424/424/0 492/492/492) [10,8,7] r=0 lpr=492 pi=[422,492)/2 crt=166'68275 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 05:55:38.166123 7f35480a2700 1 osd.10 pg_epoch: 492 pg[9.1b( v 463'472 (0'0,463'472] local-lis/les=408/410 n=256 ec=190/190 lis/c 408/408 les/c/f 410/410/0 492/492/492) [10,1,5] r=0 lpr=492 pi=[408,492)/1 crt=463'472 lcod 0'0 mlcod 0'0 unknown NOTIFY] start_peering_interval up [1,5] -> [10,1,5], acting [1,5] -> [10,1,5], acting_primary 1 -> 10, up_primary 1 -> 10, role -1 -> 0, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.166140 7f35478a1700 1 osd.10 pg_epoch: 464 pg[6.a( v 430'109231 (430'107731,430'109231] local-lis/les=428/429 n=26267 ec=76/76 lis/c 428/428 les/c/f 429/453/0 464/464/464) [6] r=-1 lpr=464 pi=[428,464)/1 crt=430'109231 lcod 0'0 unknown NOTIFY] start_peering_interval up [5,10,6] -> [6], acting [5,10,6] -> [6], acting_primary 5 -> 6, up_primary 5 -> 6, role 1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.166196 7f35480a2700 1 osd.10 pg_epoch: 492 pg[9.1b( v 463'472 (0'0,463'472] local-lis/les=408/410 n=256 ec=190/190 lis/c 408/408 les/c/f 410/410/0 492/492/492) [10,1,5] r=0 lpr=492 pi=[408,492)/1 crt=463'472 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 05:55:38.166200 7f35478a1700 1 osd.10 pg_epoch: 467 pg[6.a( v 430'109231 (430'107731,430'109231] local-lis/les=428/429 n=26267 ec=76/76 lis/c 428/428 les/c/f 429/453/0 467/467/467) [5,6] r=-1 lpr=467 pi=[428,467)/1 crt=430'109231 lcod 0'0 unknown NOTIFY] start_peering_interval up [6] -> [5,6], acting [6] -> [5,6], acting_primary 6 -> 5, up_primary 6 -> 5, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.166317 7f35480a2700 1 osd.10 pg_epoch: 464 pg[6.14( v 461'122884 (430'121291,461'122884] local-lis/les=414/415 n=25983 ec=76/76 lis/c 414/414 les/c/f 415/449/0 414/414/403) [10,4,3] r=0 lpr=461 crt=461'122884 lcod 0'0 mlcod 0'0 peering] state<Started/Primary/Peering>: Peering, affected_by_map, going to Reset >2018-07-11 05:55:38.166329 7f35478a1700 1 osd.10 pg_epoch: 480 pg[6.a( v 430'109231 (430'107731,430'109231] local-lis/les=428/429 n=26267 ec=76/76 lis/c 428/428 les/c/f 429/453/0 480/480/467) [5] r=-1 lpr=480 pi=[428,480)/1 crt=430'109231 lcod 0'0 unknown NOTIFY] start_peering_interval up [5,6] -> [5], acting [5,6] -> [5], acting_primary 5 -> 5, up_primary 5 -> 5, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.166370 7f35478a1700 1 osd.10 pg_epoch: 481 pg[6.a( v 430'109231 (430'107731,430'109231] local-lis/les=428/429 n=26267 ec=76/76 lis/c 428/428 les/c/f 429/453/0 481/481/467) [5,6] r=-1 lpr=481 pi=[428,481)/1 crt=430'109231 lcod 0'0 unknown NOTIFY] start_peering_interval up [5] -> [5,6], acting [5] -> [5,6], acting_primary 5 -> 5, up_primary 5 -> 5, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.166417 7f35480a2700 1 osd.10 pg_epoch: 464 pg[6.14( v 461'122884 (430'121291,461'122884] local-lis/les=414/415 n=25983 ec=76/76 lis/c 414/414 les/c/f 415/449/0 464/464/464) [] r=-1 lpr=464 pi=[414,464)/1 crt=461'122884 lcod 0'0 unknown] start_peering_interval up [10,4,3] -> [], acting [10,4,3] -> [], acting_primary 10 -> -1, up_primary 10 -> -1, role 0 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.166487 7f35478a1700 1 osd.10 pg_epoch: 492 pg[6.a( v 430'109231 (430'107731,430'109231] local-lis/les=428/429 n=26267 ec=76/76 lis/c 428/428 les/c/f 429/453/0 492/492/467) [5,10,6] r=1 lpr=492 pi=[428,492)/1 crt=430'109231 lcod 0'0 unknown NOTIFY] start_peering_interval up [5,6] -> [5,10,6], acting [5,6] -> [5,10,6], acting_primary 5 -> 5, up_primary 5 -> 5, role -1 -> 1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.166552 7f35478a1700 1 osd.10 pg_epoch: 492 pg[6.a( v 430'109231 (430'107731,430'109231] local-lis/les=428/429 n=26267 ec=76/76 lis/c 428/428 les/c/f 429/453/0 492/492/467) [5,10,6] r=1 lpr=492 pi=[428,492)/1 crt=430'109231 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:38.166592 7f35480a2700 1 osd.10 pg_epoch: 484 pg[6.14( v 461'122884 (430'121291,461'122884] local-lis/les=414/415 n=25983 ec=76/76 lis/c 414/414 les/c/f 415/449/0 484/484/484) [3] r=-1 lpr=484 pi=[414,484)/1 crt=461'122884 lcod 0'0 unknown NOTIFY] start_peering_interval up [] -> [3], acting [] -> [3], acting_primary ? -> 3, up_primary ? -> 3, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.166649 7f35480a2700 1 osd.10 pg_epoch: 487 pg[6.14( v 461'122884 (430'121291,461'122884] local-lis/les=414/415 n=25983 ec=76/76 lis/c 414/414 les/c/f 415/449/0 487/487/487) [4,3] r=-1 lpr=487 pi=[414,487)/1 crt=461'122884 lcod 0'0 unknown NOTIFY] start_peering_interval up [3] -> [4,3], acting [3] -> [4,3], acting_primary 3 -> 4, up_primary 3 -> 4, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.166681 7f35478a1700 1 osd.10 pg_epoch: 464 pg[12.0( v 383'7619 (383'6093,383'7619] local-lis/les=414/415 n=7619 ec=331/331 lis/c 414/414 les/c/f 415/415/0 464/464/464) [] r=-1 lpr=464 pi=[414,464)/1 crt=383'7619 lcod 0'0 unknown NOTIFY] start_peering_interval up [4,10,3] -> [], acting [4,10,3] -> [], acting_primary 4 -> -1, up_primary 4 -> -1, role 1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.166725 7f35480a2700 1 osd.10 pg_epoch: 492 pg[6.14( v 461'122884 (430'121291,461'122884] local-lis/les=414/415 n=25983 ec=76/76 lis/c 414/414 les/c/f 415/449/0 492/492/492) [10,4,3] r=0 lpr=492 pi=[414,492)/2 crt=461'122884 lcod 0'0 mlcod 0'0 unknown NOTIFY] start_peering_interval up [4,3] -> [10,4,3], acting [4,3] -> [10,4,3], acting_primary 4 -> 10, up_primary 4 -> 10, role -1 -> 0, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.166793 7f35480a2700 1 osd.10 pg_epoch: 492 pg[6.14( v 461'122884 (430'121291,461'122884] local-lis/les=414/415 n=25983 ec=76/76 lis/c 414/414 les/c/f 415/449/0 492/492/492) [10,4,3] r=0 lpr=492 pi=[414,492)/2 crt=461'122884 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 05:55:38.166884 7f35478a1700 1 osd.10 pg_epoch: 484 pg[12.0( v 383'7619 (383'6093,383'7619] local-lis/les=414/415 n=7619 ec=331/331 lis/c 414/414 les/c/f 415/415/0 484/484/484) [3] r=-1 lpr=484 pi=[414,484)/1 crt=383'7619 lcod 0'0 unknown NOTIFY] start_peering_interval up [] -> [3], acting [] -> [3], acting_primary ? -> 3, up_primary ? -> 3, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.166942 7f35478a1700 1 osd.10 pg_epoch: 487 pg[12.0( v 383'7619 (383'6093,383'7619] local-lis/les=414/415 n=7619 ec=331/331 lis/c 414/414 les/c/f 415/415/0 487/487/487) [4,3] r=-1 lpr=487 pi=[414,487)/1 crt=383'7619 lcod 0'0 unknown NOTIFY] start_peering_interval up [3] -> [4,3], acting [3] -> [4,3], acting_primary 3 -> 4, up_primary 3 -> 4, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.166957 7f35480a2700 1 osd.10 pg_epoch: 464 pg[5.14( v 166'68424 (166'66924,166'68424] local-lis/les=422/423 n=68424 ec=71/71 lis/c 422/422 les/c/f 423/423/0 464/464/422) [7] r=-1 lpr=464 pi=[422,464)/1 crt=166'68424 lcod 0'0 unknown NOTIFY] start_peering_interval up [7,10,2] -> [7], acting [7,10,2] -> [7], acting_primary 7 -> 7, up_primary 7 -> 7, role 1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.167008 7f35480a2700 1 osd.10 pg_epoch: 465 pg[5.14( v 166'68424 (166'66924,166'68424] local-lis/les=422/423 n=68424 ec=71/71 lis/c 422/422 les/c/f 423/423/0 465/465/422) [7,2] r=-1 lpr=465 pi=[422,465)/1 crt=166'68424 lcod 0'0 unknown NOTIFY] start_peering_interval up [7] -> [7,2], acting [7] -> [7,2], acting_primary 7 -> 7, up_primary 7 -> 7, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.167024 7f35478a1700 1 osd.10 pg_epoch: 492 pg[12.0( v 383'7619 (383'6093,383'7619] local-lis/les=414/415 n=7619 ec=331/331 lis/c 414/414 les/c/f 415/415/0 492/492/487) [4,10,3] r=1 lpr=492 pi=[414,492)/1 crt=383'7619 lcod 0'0 unknown NOTIFY] start_peering_interval up [4,3] -> [4,10,3], acting [4,3] -> [4,10,3], acting_primary 4 -> 4, up_primary 4 -> 4, role -1 -> 1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.167058 7f35480a2700 1 osd.10 pg_epoch: 467 pg[5.14( v 166'68424 (166'66924,166'68424] local-lis/les=422/423 n=68424 ec=71/71 lis/c 422/422 les/c/f 423/423/0 467/467/467) [2] r=-1 lpr=467 pi=[422,467)/1 crt=166'68424 lcod 0'0 unknown NOTIFY] start_peering_interval up [7,2] -> [2], acting [7,2] -> [2], acting_primary 7 -> 2, up_primary 7 -> 2, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.167087 7f35478a1700 1 osd.10 pg_epoch: 492 pg[12.0( v 383'7619 (383'6093,383'7619] local-lis/les=414/415 n=7619 ec=331/331 lis/c 414/414 les/c/f 415/415/0 492/492/487) [4,10,3] r=1 lpr=492 pi=[414,492)/1 crt=383'7619 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:38.167098 7f35480a2700 1 osd.10 pg_epoch: 468 pg[5.14( v 166'68424 (166'66924,166'68424] local-lis/les=422/423 n=68424 ec=71/71 lis/c 422/422 les/c/f 423/423/0 468/468/468) [7,2] r=-1 lpr=468 pi=[422,468)/1 crt=166'68424 lcod 0'0 unknown NOTIFY] start_peering_interval up [2] -> [7,2], acting [2] -> [7,2], acting_primary 2 -> 7, up_primary 2 -> 7, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.167216 7f35478a1700 1 osd.10 pg_epoch: 464 pg[5.4( v 166'68216 (166'66716,166'68216] local-lis/les=422/423 n=68216 ec=71/71 lis/c 422/422 les/c/f 423/423/0 464/464/422) [7] r=-1 lpr=464 pi=[422,464)/1 crt=166'68216 lcod 0'0 unknown NOTIFY] start_peering_interval up [7,2,10] -> [7], acting [7,2,10] -> [7], acting_primary 7 -> 7, up_primary 7 -> 7, role 2 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.167270 7f35478a1700 1 osd.10 pg_epoch: 465 pg[5.4( v 166'68216 (166'66716,166'68216] local-lis/les=422/423 n=68216 ec=71/71 lis/c 422/422 les/c/f 423/423/0 465/465/422) [7,2] r=-1 lpr=465 pi=[422,465)/1 crt=166'68216 lcod 0'0 unknown NOTIFY] start_peering_interval up [7] -> [7,2], acting [7] -> [7,2], acting_primary 7 -> 7, up_primary 7 -> 7, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.167291 7f35480a2700 1 osd.10 pg_epoch: 492 pg[5.14( v 166'68424 (166'66924,166'68424] local-lis/les=422/423 n=68424 ec=71/71 lis/c 422/422 les/c/f 423/423/0 492/492/468) [7,10,2] r=1 lpr=492 pi=[422,492)/1 crt=166'68424 lcod 0'0 unknown NOTIFY] start_peering_interval up [7,2] -> [7,10,2], acting [7,2] -> [7,10,2], acting_primary 7 -> 7, up_primary 7 -> 7, role -1 -> 1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.167320 7f35478a1700 1 osd.10 pg_epoch: 467 pg[5.4( v 166'68216 (166'66716,166'68216] local-lis/les=422/423 n=68216 ec=71/71 lis/c 422/422 les/c/f 423/423/0 467/467/467) [2] r=-1 lpr=467 pi=[422,467)/1 crt=166'68216 lcod 0'0 unknown NOTIFY] start_peering_interval up [7,2] -> [2], acting [7,2] -> [2], acting_primary 7 -> 2, up_primary 7 -> 2, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.167361 7f35480a2700 1 osd.10 pg_epoch: 492 pg[5.14( v 166'68424 (166'66924,166'68424] local-lis/les=422/423 n=68424 ec=71/71 lis/c 422/422 les/c/f 423/423/0 492/492/468) [7,10,2] r=1 lpr=492 pi=[422,492)/1 crt=166'68424 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:38.167361 7f35478a1700 1 osd.10 pg_epoch: 468 pg[5.4( v 166'68216 (166'66716,166'68216] local-lis/les=422/423 n=68216 ec=71/71 lis/c 422/422 les/c/f 423/423/0 468/468/468) [7,2] r=-1 lpr=468 pi=[422,468)/1 crt=166'68216 lcod 0'0 unknown NOTIFY] start_peering_interval up [2] -> [7,2], acting [2] -> [7,2], acting_primary 2 -> 7, up_primary 2 -> 7, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.167439 7f35480a2700 1 osd.10 pg_epoch: 464 pg[6.17( v 430'108120 (430'106620,430'108120] local-lis/les=408/410 n=26407 ec=76/76 lis/c 408/408 les/c/f 410/449/0 408/408/403) [10,2,5] r=0 lpr=449 crt=430'108120 lcod 0'0 mlcod 0'0 peering] state<Started/Primary/Peering>: Peering, affected_by_map, going to Reset >2018-07-11 05:55:38.167531 7f35478a1700 1 osd.10 pg_epoch: 492 pg[5.4( v 166'68216 (166'66716,166'68216] local-lis/les=422/423 n=68216 ec=71/71 lis/c 422/422 les/c/f 423/423/0 492/492/468) [7,2,10] r=2 lpr=492 pi=[422,492)/1 crt=166'68216 lcod 0'0 unknown NOTIFY] start_peering_interval up [7,2] -> [7,2,10], acting [7,2] -> [7,2,10], acting_primary 7 -> 7, up_primary 7 -> 7, role -1 -> 2, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.167538 7f35480a2700 1 osd.10 pg_epoch: 464 pg[6.17( v 430'108120 (430'106620,430'108120] local-lis/les=408/410 n=26407 ec=76/76 lis/c 408/408 les/c/f 410/449/0 464/464/464) [] r=-1 lpr=464 pi=[408,464)/1 crt=430'108120 lcod 0'0 unknown] start_peering_interval up [10,2,5] -> [], acting [10,2,5] -> [], acting_primary 10 -> -1, up_primary 10 -> -1, role 0 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.167581 7f35480a2700 1 osd.10 pg_epoch: 465 pg[6.17( v 430'108120 (430'106620,430'108120] local-lis/les=408/410 n=26407 ec=76/76 lis/c 408/408 les/c/f 410/449/0 465/465/465) [2] r=-1 lpr=465 pi=[408,465)/1 crt=430'108120 lcod 0'0 unknown NOTIFY] start_peering_interval up [] -> [2], acting [] -> [2], acting_primary ? -> 2, up_primary ? -> 2, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.167600 7f35478a1700 1 osd.10 pg_epoch: 492 pg[5.4( v 166'68216 (166'66716,166'68216] local-lis/les=422/423 n=68216 ec=71/71 lis/c 422/422 les/c/f 423/423/0 492/492/468) [7,2,10] r=2 lpr=492 pi=[422,492)/1 crt=166'68216 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:38.167629 7f35480a2700 1 osd.10 pg_epoch: 467 pg[6.17( v 430'108120 (430'106620,430'108120] local-lis/les=408/410 n=26407 ec=76/76 lis/c 408/408 les/c/f 410/449/0 467/467/465) [2,5] r=-1 lpr=467 pi=[408,467)/1 crt=430'108120 lcod 0'0 unknown NOTIFY] start_peering_interval up [2] -> [2,5], acting [2] -> [2,5], acting_primary 2 -> 2, up_primary 2 -> 2, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.167689 7f35478a1700 1 osd.10 pg_epoch: 464 pg[6.7( v 430'109641 (430'108141,430'109641] local-lis/les=428/429 n=25825 ec=76/76 lis/c 428/428 les/c/f 429/457/0 428/428/403) [10,6,3] r=0 lpr=457 crt=430'109641 lcod 0'0 mlcod 0'0 peering] state<Started/Primary/Peering>: Peering, affected_by_map, going to Reset >2018-07-11 05:55:38.167781 7f35478a1700 1 osd.10 pg_epoch: 464 pg[6.7( v 430'109641 (430'108141,430'109641] local-lis/les=428/429 n=25825 ec=76/76 lis/c 428/428 les/c/f 429/457/0 464/464/464) [6] r=-1 lpr=464 pi=[428,464)/1 crt=430'109641 lcod 0'0 unknown] start_peering_interval up [10,6,3] -> [6], acting [10,6,3] -> [6], acting_primary 10 -> 6, up_primary 10 -> 6, role 0 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.167822 7f35480a2700 1 osd.10 pg_epoch: 492 pg[6.17( v 430'108120 (430'106620,430'108120] local-lis/les=408/410 n=26407 ec=76/76 lis/c 408/408 les/c/f 410/449/0 492/492/492) [10,2,5] r=0 lpr=492 pi=[408,492)/2 crt=430'108120 lcod 0'0 mlcod 0'0 unknown NOTIFY] start_peering_interval up [2,5] -> [10,2,5], acting [2,5] -> [10,2,5], acting_primary 2 -> 10, up_primary 2 -> 10, role -1 -> 0, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.167891 7f35480a2700 1 osd.10 pg_epoch: 492 pg[6.17( v 430'108120 (430'106620,430'108120] local-lis/les=408/410 n=26407 ec=76/76 lis/c 408/408 les/c/f 410/449/0 492/492/492) [10,2,5] r=0 lpr=492 pi=[408,492)/2 crt=430'108120 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 05:55:38.167911 7f35478a1700 1 osd.10 pg_epoch: 480 pg[6.7( v 430'109641 (430'108141,430'109641] local-lis/les=428/429 n=25825 ec=76/76 lis/c 428/428 les/c/f 429/457/0 480/480/480) [] r=-1 lpr=480 pi=[428,480)/1 crt=430'109641 lcod 0'0 unknown NOTIFY] start_peering_interval up [6] -> [], acting [6] -> [], acting_primary 6 -> -1, up_primary 6 -> -1, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.167947 7f35478a1700 1 osd.10 pg_epoch: 481 pg[6.7( v 430'109641 (430'108141,430'109641] local-lis/les=428/429 n=25825 ec=76/76 lis/c 428/428 les/c/f 429/457/0 481/481/481) [6] r=-1 lpr=481 pi=[428,481)/1 crt=430'109641 lcod 0'0 unknown NOTIFY] start_peering_interval up [] -> [6], acting [] -> [6], acting_primary ? -> 6, up_primary ? -> 6, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.167998 7f35478a1700 1 osd.10 pg_epoch: 484 pg[6.7( v 430'109641 (430'108141,430'109641] local-lis/les=428/429 n=25825 ec=76/76 lis/c 428/428 les/c/f 429/457/0 484/484/481) [6,3] r=-1 lpr=484 pi=[428,484)/1 crt=430'109641 lcod 0'0 unknown NOTIFY] start_peering_interval up [6] -> [6,3], acting [6] -> [6,3], acting_primary 6 -> 6, up_primary 6 -> 6, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.168049 7f35480a2700 1 osd.10 pg_epoch: 464 pg[6.21( v 461'128646 (430'127048,461'128646] local-lis/les=411/412 n=25982 ec=76/76 lis/c 411/411 les/c/f 412/431/0 464/464/464) [] r=-1 lpr=464 pi=[411,464)/1 crt=461'128646 lcod 0'0 unknown NOTIFY] start_peering_interval up [0,2,10] -> [], acting [0,2,10] -> [], acting_primary 0 -> -1, up_primary 0 -> -1, role 2 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.168081 7f35478a1700 1 osd.10 pg_epoch: 492 pg[6.7( v 430'109641 (430'108141,430'109641] local-lis/les=428/429 n=25825 ec=76/76 lis/c 428/428 les/c/f 429/457/0 492/492/492) [10,6,3] r=0 lpr=492 pi=[428,492)/2 crt=430'109641 lcod 0'0 mlcod 0'0 unknown NOTIFY] start_peering_interval up [6,3] -> [10,6,3], acting [6,3] -> [10,6,3], acting_primary 6 -> 10, up_primary 6 -> 10, role -1 -> 0, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.168094 7f35480a2700 1 osd.10 pg_epoch: 465 pg[6.21( v 461'128646 (430'127048,461'128646] local-lis/les=411/412 n=25982 ec=76/76 lis/c 411/411 les/c/f 412/431/0 465/465/465) [2] r=-1 lpr=465 pi=[411,465)/1 crt=461'128646 lcod 0'0 unknown NOTIFY] start_peering_interval up [] -> [2], acting [] -> [2], acting_primary ? -> 2, up_primary ? -> 2, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.168151 7f35478a1700 1 osd.10 pg_epoch: 492 pg[6.7( v 430'109641 (430'108141,430'109641] local-lis/les=428/429 n=25825 ec=76/76 lis/c 428/428 les/c/f 429/457/0 492/492/492) [10,6,3] r=0 lpr=492 pi=[428,492)/2 crt=430'109641 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 05:55:38.168153 7f35480a2700 1 osd.10 pg_epoch: 470 pg[6.21( v 461'128646 (430'127048,461'128646] local-lis/les=411/412 n=25982 ec=76/76 lis/c 411/411 les/c/f 412/431/0 470/470/470) [0,2] r=-1 lpr=470 pi=[411,470)/1 crt=461'128646 lcod 0'0 unknown NOTIFY] start_peering_interval up [2] -> [0,2], acting [2] -> [0,2], acting_primary 2 -> 0, up_primary 2 -> 0, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.168264 7f35478a1700 1 osd.10 pg_epoch: 464 pg[5.d( v 166'68209 (166'66709,166'68209] local-lis/les=422/424 n=68209 ec=71/71 lis/c 422/422 les/c/f 424/424/0 422/422/403) [10,7,0] r=0 lpr=424 crt=166'68209 lcod 0'0 mlcod 0'0 peering] state<Started/Primary/Peering>: Peering, affected_by_map, going to Reset >2018-07-11 05:55:38.168324 7f35480a2700 1 osd.10 pg_epoch: 492 pg[6.21( v 461'128646 (430'127048,461'128646] local-lis/les=411/412 n=25982 ec=76/76 lis/c 411/411 les/c/f 412/431/0 492/492/470) [0,2,10] r=2 lpr=492 pi=[411,492)/2 crt=461'128646 lcod 0'0 unknown NOTIFY] start_peering_interval up [0,2] -> [0,2,10], acting [0,2] -> [0,2,10], acting_primary 0 -> 0, up_primary 0 -> 0, role -1 -> 2, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.168353 7f35478a1700 1 osd.10 pg_epoch: 464 pg[5.d( v 166'68209 (166'66709,166'68209] local-lis/les=422/424 n=68209 ec=71/71 lis/c 422/422 les/c/f 424/424/0 464/464/464) [7] r=-1 lpr=464 pi=[422,464)/1 crt=166'68209 lcod 0'0 unknown] start_peering_interval up [10,7,0] -> [7], acting [10,7,0] -> [7], acting_primary 10 -> 7, up_primary 10 -> 7, role 0 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.168396 7f35480a2700 1 osd.10 pg_epoch: 492 pg[6.21( v 461'128646 (430'127048,461'128646] local-lis/les=411/412 n=25982 ec=76/76 lis/c 411/411 les/c/f 412/431/0 492/492/470) [0,2,10] r=2 lpr=492 pi=[411,492)/2 crt=461'128646 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:38.168414 7f35478a1700 1 osd.10 pg_epoch: 467 pg[5.d( v 166'68209 (166'66709,166'68209] local-lis/les=422/424 n=68209 ec=71/71 lis/c 422/422 les/c/f 424/424/0 467/467/467) [] r=-1 lpr=467 pi=[422,467)/1 crt=166'68209 lcod 0'0 unknown NOTIFY] start_peering_interval up [7] -> [], acting [7] -> [], acting_primary 7 -> -1, up_primary 7 -> -1, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.168456 7f35478a1700 1 osd.10 pg_epoch: 468 pg[5.d( v 166'68209 (166'66709,166'68209] local-lis/les=422/424 n=68209 ec=71/71 lis/c 422/422 les/c/f 424/424/0 468/468/468) [7] r=-1 lpr=468 pi=[422,468)/1 crt=166'68209 lcod 0'0 unknown NOTIFY] start_peering_interval up [] -> [7], acting [] -> [7], acting_primary ? -> 7, up_primary ? -> 7, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.168501 7f35478a1700 1 osd.10 pg_epoch: 470 pg[5.d( v 166'68209 (166'66709,166'68209] local-lis/les=422/424 n=68209 ec=71/71 lis/c 422/422 les/c/f 424/424/0 470/470/468) [7,0] r=-1 lpr=470 pi=[422,470)/1 crt=166'68209 lcod 0'0 unknown NOTIFY] start_peering_interval up [7] -> [7,0], acting [7] -> [7,0], acting_primary 7 -> 7, up_primary 7 -> 7, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.168545 7f35480a2700 1 osd.10 pg_epoch: 464 pg[10.9( v 390'615 (0'0,390'615] local-lis/les=411/412 n=478 ec=193/193 lis/c 411/411 les/c/f 412/412/0 464/464/464) [] r=-1 lpr=464 pi=[411,464)/1 crt=390'615 lcod 0'0 unknown NOTIFY] start_peering_interval up [0,2,10] -> [], acting [0,2,10] -> [], acting_primary 0 -> -1, up_primary 0 -> -1, role 2 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.168591 7f35480a2700 1 osd.10 pg_epoch: 465 pg[10.9( v 390'615 (0'0,390'615] local-lis/les=411/412 n=478 ec=193/193 lis/c 411/411 les/c/f 412/412/0 465/465/465) [2] r=-1 lpr=465 pi=[411,465)/1 crt=390'615 lcod 0'0 unknown NOTIFY] start_peering_interval up [] -> [2], acting [] -> [2], acting_primary ? -> 2, up_primary ? -> 2, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.168665 7f35480a2700 1 osd.10 pg_epoch: 470 pg[10.9( v 390'615 (0'0,390'615] local-lis/les=411/412 n=478 ec=193/193 lis/c 411/411 les/c/f 412/412/0 470/470/470) [0,2] r=-1 lpr=470 pi=[411,470)/1 crt=390'615 lcod 0'0 unknown NOTIFY] start_peering_interval up [2] -> [0,2], acting [2] -> [0,2], acting_primary 2 -> 0, up_primary 2 -> 0, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.168671 7f35478a1700 1 osd.10 pg_epoch: 492 pg[5.d( v 166'68209 (166'66709,166'68209] local-lis/les=422/424 n=68209 ec=71/71 lis/c 422/422 les/c/f 424/424/0 492/492/492) [10,7,0] r=0 lpr=492 pi=[422,492)/2 crt=166'68209 lcod 0'0 mlcod 0'0 unknown NOTIFY] start_peering_interval up [7,0] -> [10,7,0], acting [7,0] -> [10,7,0], acting_primary 7 -> 10, up_primary 7 -> 10, role -1 -> 0, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.168739 7f35478a1700 1 osd.10 pg_epoch: 492 pg[5.d( v 166'68209 (166'66709,166'68209] local-lis/les=422/424 n=68209 ec=71/71 lis/c 422/422 les/c/f 424/424/0 492/492/492) [10,7,0] r=0 lpr=492 pi=[422,492)/2 crt=166'68209 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 05:55:38.168863 7f35480a2700 1 osd.10 pg_epoch: 492 pg[10.9( v 390'615 (0'0,390'615] local-lis/les=411/412 n=478 ec=193/193 lis/c 411/411 les/c/f 412/412/0 492/492/470) [0,2,10] r=2 lpr=492 pi=[411,492)/1 crt=390'615 lcod 0'0 unknown NOTIFY] start_peering_interval up [0,2] -> [0,2,10], acting [0,2] -> [0,2,10], acting_primary 0 -> 0, up_primary 0 -> 0, role -1 -> 2, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.168925 7f35478a1700 1 osd.10 pg_epoch: 464 pg[6.e( v 430'116090 (430'114517,430'116090] local-lis/les=414/415 n=25967 ec=76/76 lis/c 414/414 les/c/f 415/430/0 464/464/464) [] r=-1 lpr=464 pi=[414,464)/1 crt=430'116090 lcod 0'0 unknown NOTIFY] start_peering_interval up [4,1,10] -> [], acting [4,1,10] -> [], acting_primary 4 -> -1, up_primary 4 -> -1, role 2 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.168933 7f35480a2700 1 osd.10 pg_epoch: 492 pg[10.9( v 390'615 (0'0,390'615] local-lis/les=411/412 n=478 ec=193/193 lis/c 411/411 les/c/f 412/412/0 492/492/470) [0,2,10] r=2 lpr=492 pi=[411,492)/1 crt=390'615 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:38.169012 7f35478a1700 1 osd.10 pg_epoch: 470 pg[6.e( v 430'116090 (430'114517,430'116090] local-lis/les=414/415 n=25967 ec=76/76 lis/c 414/414 les/c/f 415/430/0 470/470/470) [1] r=-1 lpr=470 pi=[414,470)/1 crt=430'116090 lcod 0'0 unknown NOTIFY] start_peering_interval up [] -> [1], acting [] -> [1], acting_primary ? -> 1, up_primary ? -> 1, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.169063 7f35480a2700 1 osd.10 pg_epoch: 464 pg[9.a( v 461'571 (0'0,461'571] local-lis/les=419/420 n=231 ec=190/190 lis/c 419/419 les/c/f 420/431/0 464/464/464) [11] r=-1 lpr=464 pi=[419,464)/1 crt=461'571 lcod 0'0 unknown NOTIFY] start_peering_interval up [0,10,11] -> [11], acting [0,10,11] -> [11], acting_primary 0 -> 11, up_primary 0 -> 11, role 1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.169146 7f35480a2700 1 osd.10 pg_epoch: 470 pg[9.a( v 461'571 (0'0,461'571] local-lis/les=419/420 n=231 ec=190/190 lis/c 419/419 les/c/f 420/431/0 470/470/470) [0,11] r=-1 lpr=470 pi=[419,470)/1 crt=461'571 lcod 0'0 unknown NOTIFY] start_peering_interval up [11] -> [0,11], acting [11] -> [0,11], acting_primary 11 -> 0, up_primary 11 -> 0, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.169202 7f35478a1700 1 osd.10 pg_epoch: 487 pg[6.e( v 430'116090 (430'114517,430'116090] local-lis/les=414/415 n=25967 ec=76/76 lis/c 414/414 les/c/f 415/430/0 487/487/487) [4,1] r=-1 lpr=487 pi=[414,487)/1 crt=430'116090 lcod 0'0 unknown NOTIFY] start_peering_interval up [1] -> [4,1], acting [1] -> [4,1], acting_primary 1 -> 4, up_primary 1 -> 4, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.169229 7f35480a2700 1 osd.10 pg_epoch: 477 pg[9.a( v 461'571 (0'0,461'571] local-lis/les=419/420 n=231 ec=190/190 lis/c 419/419 les/c/f 420/431/0 477/477/470) [0] r=-1 lpr=477 pi=[419,477)/1 crt=461'571 lcod 0'0 unknown NOTIFY] start_peering_interval up [0,11] -> [0], acting [0,11] -> [0], acting_primary 0 -> 0, up_primary 0 -> 0, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.169280 7f35480a2700 1 osd.10 pg_epoch: 478 pg[9.a( v 461'571 (0'0,461'571] local-lis/les=419/420 n=231 ec=190/190 lis/c 419/419 les/c/f 420/431/0 478/478/470) [0,11] r=-1 lpr=478 pi=[419,478)/1 crt=461'571 lcod 0'0 unknown NOTIFY] start_peering_interval up [0] -> [0,11], acting [0] -> [0,11], acting_primary 0 -> 0, up_primary 0 -> 0, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.169291 7f35478a1700 1 osd.10 pg_epoch: 492 pg[6.e( v 430'116090 (430'114517,430'116090] local-lis/les=414/415 n=25967 ec=76/76 lis/c 414/414 les/c/f 415/430/0 492/492/487) [4,1,10] r=2 lpr=492 pi=[414,492)/2 crt=430'116090 lcod 0'0 unknown NOTIFY] start_peering_interval up [4,1] -> [4,1,10], acting [4,1] -> [4,1,10], acting_primary 4 -> 4, up_primary 4 -> 4, role -1 -> 2, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.169360 7f35478a1700 1 osd.10 pg_epoch: 492 pg[6.e( v 430'116090 (430'114517,430'116090] local-lis/les=414/415 n=25967 ec=76/76 lis/c 414/414 les/c/f 415/430/0 492/492/487) [4,1,10] r=2 lpr=492 pi=[414,492)/2 crt=430'116090 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:38.169399 7f35480a2700 1 osd.10 pg_epoch: 492 pg[9.a( v 461'571 (0'0,461'571] local-lis/les=419/420 n=231 ec=190/190 lis/c 419/419 les/c/f 420/431/0 492/492/470) [0,10,11] r=1 lpr=492 pi=[419,492)/1 crt=461'571 lcod 0'0 unknown NOTIFY] start_peering_interval up [0,11] -> [0,10,11], acting [0,11] -> [0,10,11], acting_primary 0 -> 0, up_primary 0 -> 0, role -1 -> 1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.169472 7f35480a2700 1 osd.10 pg_epoch: 492 pg[9.a( v 461'571 (0'0,461'571] local-lis/les=419/420 n=231 ec=190/190 lis/c 419/419 les/c/f 420/431/0 492/492/470) [0,10,11] r=1 lpr=492 pi=[419,492)/1 crt=461'571 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:38.169522 7f35478a1700 1 osd.10 pg_epoch: 464 pg[11.8s3( v 383'59629 (352'55946,383'59629] local-lis/les=428/430 n=59629 ec=267/267 lis/c 428/428 les/c/f 430/430/0 464/464/422) [7,2147483647,6,2147483647,2147483647,2147483647]p7(0) r=-1 lpr=464 pi=[428,464)/1 crt=383'59629 lcod 0'0 unknown NOTIFY] start_peering_interval up [7,1,6,10,5,8] -> [7,2147483647,6,2147483647,2147483647,2147483647], acting [7,1,6,10,5,8] -> [7,2147483647,6,2147483647,2147483647,2147483647], acting_primary 7(0) -> 7, up_primary 7(0) -> 7, role 3 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.169568 7f35480a2700 1 osd.10 pg_epoch: 464 pg[9.7( v 463'502 (0'0,463'502] local-lis/les=422/423 n=247 ec=190/190 lis/c 422/422 les/c/f 423/423/0 464/464/422) [7] r=-1 lpr=464 pi=[422,464)/1 crt=463'502 lcod 0'0 unknown NOTIFY] start_peering_interval up [7,10,2] -> [7], acting [7,10,2] -> [7], acting_primary 7 -> 7, up_primary 7 -> 7, role 1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.169602 7f35478a1700 1 osd.10 pg_epoch: 467 pg[11.8s3( v 383'59629 (352'55946,383'59629] local-lis/les=428/430 n=59629 ec=267/267 lis/c 428/428 les/c/f 430/430/0 467/467/467) [2147483647,2147483647,6,2147483647,5,2147483647]p6(2) r=-1 lpr=467 pi=[428,467)/1 crt=383'59629 lcod 0'0 unknown NOTIFY] start_peering_interval up [7,2147483647,6,2147483647,2147483647,2147483647] -> [2147483647,2147483647,6,2147483647,5,2147483647], acting [7,2147483647,6,2147483647,2147483647,2147483647] -> [2147483647,2147483647,6,2147483647,5,2147483647], acting_primary 7(0) -> 6, up_primary 7(0) -> 6, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.169615 7f35480a2700 1 osd.10 pg_epoch: 465 pg[9.7( v 463'502 (0'0,463'502] local-lis/les=422/423 n=247 ec=190/190 lis/c 422/422 les/c/f 423/423/0 465/465/422) [7,2] r=-1 lpr=465 pi=[422,465)/1 crt=463'502 lcod 0'0 unknown NOTIFY] start_peering_interval up [7] -> [7,2], acting [7] -> [7,2], acting_primary 7 -> 7, up_primary 7 -> 7, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.169655 7f35478a1700 1 osd.10 pg_epoch: 468 pg[11.8s3( v 383'59629 (352'55946,383'59629] local-lis/les=428/430 n=59629 ec=267/267 lis/c 428/428 les/c/f 430/430/0 468/468/468) [7,2147483647,6,2147483647,5,2147483647]p7(0) r=-1 lpr=468 pi=[428,468)/1 crt=383'59629 lcod 0'0 unknown NOTIFY] start_peering_interval up [2147483647,2147483647,6,2147483647,5,2147483647] -> [7,2147483647,6,2147483647,5,2147483647], acting [2147483647,2147483647,6,2147483647,5,2147483647] -> [7,2147483647,6,2147483647,5,2147483647], acting_primary 6(2) -> 7, up_primary 6(2) -> 7, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.169661 7f35480a2700 1 osd.10 pg_epoch: 467 pg[9.7( v 463'502 (0'0,463'502] local-lis/les=422/423 n=247 ec=190/190 lis/c 422/422 les/c/f 423/423/0 467/467/467) [2] r=-1 lpr=467 pi=[422,467)/1 crt=463'502 lcod 0'0 unknown NOTIFY] start_peering_interval up [7,2] -> [2], acting [7,2] -> [2], acting_primary 7 -> 2, up_primary 7 -> 2, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.169701 7f35480a2700 1 osd.10 pg_epoch: 468 pg[9.7( v 463'502 (0'0,463'502] local-lis/les=422/423 n=247 ec=190/190 lis/c 422/422 les/c/f 423/423/0 468/468/468) [7,2] r=-1 lpr=468 pi=[422,468)/1 crt=463'502 lcod 0'0 unknown NOTIFY] start_peering_interval up [2] -> [7,2], acting [2] -> [7,2], acting_primary 2 -> 7, up_primary 2 -> 7, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.169720 7f35478a1700 1 osd.10 pg_epoch: 470 pg[11.8s3( v 383'59629 (352'55946,383'59629] local-lis/les=428/430 n=59629 ec=267/267 lis/c 428/428 les/c/f 430/430/0 470/470/468) [7,1,6,2147483647,5,2147483647]p7(0) r=-1 lpr=470 pi=[428,470)/1 crt=383'59629 lcod 0'0 unknown NOTIFY] start_peering_interval up [7,2147483647,6,2147483647,5,2147483647] -> [7,1,6,2147483647,5,2147483647], acting [7,2147483647,6,2147483647,5,2147483647] -> [7,1,6,2147483647,5,2147483647], acting_primary 7(0) -> 7, up_primary 7(0) -> 7, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.169786 7f35478a1700 1 osd.10 pg_epoch: 472 pg[11.8s3( v 383'59629 (352'55946,383'59629] local-lis/les=428/430 n=59629 ec=267/267 lis/c 428/428 les/c/f 430/430/0 472/472/468) [7,1,6,2147483647,5,8]p7(0) r=-1 lpr=472 pi=[428,472)/1 crt=383'59629 lcod 0'0 unknown NOTIFY] start_peering_interval up [7,1,6,2147483647,5,2147483647] -> [7,1,6,2147483647,5,8], acting [7,1,6,2147483647,5,2147483647] -> [7,1,6,2147483647,5,8], acting_primary 7(0) -> 7, up_primary 7(0) -> 7, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.169903 7f35480a2700 1 osd.10 pg_epoch: 492 pg[9.7( v 463'502 (0'0,463'502] local-lis/les=422/423 n=247 ec=190/190 lis/c 422/422 les/c/f 423/423/0 492/492/468) [7,10,2] r=1 lpr=492 pi=[422,492)/1 crt=463'502 lcod 0'0 unknown NOTIFY] start_peering_interval up [7,2] -> [7,10,2], acting [7,2] -> [7,10,2], acting_primary 7 -> 7, up_primary 7 -> 7, role -1 -> 1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.169922 7f35478a1700 1 osd.10 pg_epoch: 480 pg[11.8s3( v 383'59629 (352'55946,383'59629] local-lis/les=428/430 n=59629 ec=267/267 lis/c 428/428 les/c/f 430/430/0 480/480/468) [7,1,2147483647,2147483647,5,8]p7(0) r=-1 lpr=480 pi=[428,480)/1 crt=383'59629 lcod 0'0 unknown NOTIFY] start_peering_interval up [7,1,6,2147483647,5,8] -> [7,1,2147483647,2147483647,5,8], acting [7,1,6,2147483647,5,8] -> [7,1,2147483647,2147483647,5,8], acting_primary 7(0) -> 7, up_primary 7(0) -> 7, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.169970 7f35480a2700 1 osd.10 pg_epoch: 492 pg[9.7( v 463'502 (0'0,463'502] local-lis/les=422/423 n=247 ec=190/190 lis/c 422/422 les/c/f 423/423/0 492/492/468) [7,10,2] r=1 lpr=492 pi=[422,492)/1 crt=463'502 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:38.169973 7f35478a1700 1 osd.10 pg_epoch: 481 pg[11.8s3( v 383'59629 (352'55946,383'59629] local-lis/les=428/430 n=59629 ec=267/267 lis/c 428/428 les/c/f 430/430/0 481/481/468) [7,1,6,2147483647,5,8]p7(0) r=-1 lpr=481 pi=[428,481)/1 crt=383'59629 lcod 0'0 unknown NOTIFY] start_peering_interval up [7,1,2147483647,2147483647,5,8] -> [7,1,6,2147483647,5,8], acting [7,1,2147483647,2147483647,5,8] -> [7,1,6,2147483647,5,8], acting_primary 7(0) -> 7, up_primary 7(0) -> 7, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.170045 7f35480a2700 1 osd.10 pg_epoch: 464 pg[5.1c( v 383'68804 (166'67302,383'68804] local-lis/les=403/404 n=68803 ec=71/71 lis/c 403/403 les/c/f 404/404/0 403/403/403) [10,8,2] r=0 lpr=445 crt=383'68804 lcod 0'0 mlcod 0'0 peering] state<Started/Primary/Peering>: Peering, affected_by_map, going to Reset >2018-07-11 05:55:38.170127 7f35478a1700 1 osd.10 pg_epoch: 492 pg[11.8s3( v 383'59629 (352'55946,383'59629] local-lis/les=428/430 n=59629 ec=267/267 lis/c 428/428 les/c/f 430/430/0 492/492/468) [7,1,6,10,5,8]p7(0) r=3 lpr=492 pi=[428,492)/1 crt=383'59629 lcod 0'0 unknown NOTIFY] start_peering_interval up [7,1,6,2147483647,5,8] -> [7,1,6,10,5,8], acting [7,1,6,2147483647,5,8] -> [7,1,6,10,5,8], acting_primary 7(0) -> 7, up_primary 7(0) -> 7, role -1 -> 3, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.170136 7f35480a2700 1 osd.10 pg_epoch: 464 pg[5.1c( v 383'68804 (166'67302,383'68804] local-lis/les=403/404 n=68803 ec=71/71 lis/c 403/403 les/c/f 404/404/0 464/464/464) [] r=-1 lpr=464 pi=[403,464)/1 crt=383'68804 lcod 0'0 unknown] start_peering_interval up [10,8,2] -> [], acting [10,8,2] -> [], acting_primary 10 -> -1, up_primary 10 -> -1, role 0 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.170177 7f35480a2700 1 osd.10 pg_epoch: 465 pg[5.1c( v 383'68804 (166'67302,383'68804] local-lis/les=403/404 n=68803 ec=71/71 lis/c 403/403 les/c/f 404/404/0 465/465/465) [2] r=-1 lpr=465 pi=[403,465)/1 crt=383'68804 lcod 0'0 unknown NOTIFY] start_peering_interval up [] -> [2], acting [] -> [2], acting_primary ? -> 2, up_primary ? -> 2, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.170206 7f35478a1700 1 osd.10 pg_epoch: 492 pg[11.8s3( v 383'59629 (352'55946,383'59629] local-lis/les=428/430 n=59629 ec=267/267 lis/c 428/428 les/c/f 430/430/0 492/492/468) [7,1,6,10,5,8]p7(0) r=3 lpr=492 pi=[428,492)/1 crt=383'59629 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:38.170262 7f35480a2700 1 osd.10 pg_epoch: 472 pg[5.1c( v 383'68804 (166'67302,383'68804] local-lis/les=403/404 n=68803 ec=71/71 lis/c 403/403 les/c/f 404/404/0 472/472/472) [8,2] r=-1 lpr=472 pi=[403,472)/1 crt=383'68804 lcod 0'0 unknown NOTIFY] start_peering_interval up [2] -> [8,2], acting [2] -> [8,2], acting_primary 2 -> 8, up_primary 2 -> 8, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.170385 7f35478a1700 1 osd.10 pg_epoch: 464 pg[13.8s5( v 377'9578 (367'1954,377'9578] local-lis/les=428/429 n=2631 ec=345/345 lis/c 428/428 les/c/f 429/429/0 464/464/464) [2147483647,11,2147483647,2147483647,6,2147483647]p11(1) r=-1 lpr=464 pi=[428,464)/1 crt=377'9578 lcod 0'0 unknown NOTIFY] start_peering_interval up [2,11,5,8,6,10] -> [2147483647,11,2147483647,2147483647,6,2147483647], acting [2,11,5,8,6,10] -> [2147483647,11,2147483647,2147483647,6,2147483647], acting_primary 2(0) -> 11, up_primary 2(0) -> 11, role 5 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.170425 7f35480a2700 1 osd.10 pg_epoch: 492 pg[5.1c( v 383'68804 (166'67302,383'68804] local-lis/les=403/404 n=68803 ec=71/71 lis/c 403/403 les/c/f 404/404/0 492/492/492) [10,8,2] r=0 lpr=492 pi=[403,492)/2 crt=383'68804 lcod 0'0 mlcod 0'0 unknown NOTIFY] start_peering_interval up [8,2] -> [10,8,2], acting [8,2] -> [10,8,2], acting_primary 8 -> 10, up_primary 8 -> 10, role -1 -> 0, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.170446 7f35478a1700 1 osd.10 pg_epoch: 465 pg[13.8s5( v 377'9578 (367'1954,377'9578] local-lis/les=428/429 n=2631 ec=345/345 lis/c 428/428 les/c/f 429/429/0 465/465/465) [2,11,2147483647,2147483647,6,2147483647]p2(0) r=-1 lpr=465 pi=[428,465)/1 crt=377'9578 lcod 0'0 unknown NOTIFY] start_peering_interval up [2147483647,11,2147483647,2147483647,6,2147483647] -> [2,11,2147483647,2147483647,6,2147483647], acting [2147483647,11,2147483647,2147483647,6,2147483647] -> [2,11,2147483647,2147483647,6,2147483647], acting_primary 11(1) -> 2, up_primary 11(1) -> 2, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.170492 7f35480a2700 1 osd.10 pg_epoch: 492 pg[5.1c( v 383'68804 (166'67302,383'68804] local-lis/les=403/404 n=68803 ec=71/71 lis/c 403/403 les/c/f 404/404/0 492/492/492) [10,8,2] r=0 lpr=492 pi=[403,492)/2 crt=383'68804 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 05:55:38.170509 7f35478a1700 1 osd.10 pg_epoch: 467 pg[13.8s5( v 377'9578 (367'1954,377'9578] local-lis/les=428/429 n=2631 ec=345/345 lis/c 428/428 les/c/f 429/429/0 467/467/465) [2,11,5,2147483647,6,2147483647]p2(0) r=-1 lpr=467 pi=[428,467)/1 crt=377'9578 lcod 0'0 unknown NOTIFY] start_peering_interval up [2,11,2147483647,2147483647,6,2147483647] -> [2,11,5,2147483647,6,2147483647], acting [2,11,2147483647,2147483647,6,2147483647] -> [2,11,5,2147483647,6,2147483647], acting_primary 2(0) -> 2, up_primary 2(0) -> 2, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.170599 7f35478a1700 1 osd.10 pg_epoch: 472 pg[13.8s5( v 377'9578 (367'1954,377'9578] local-lis/les=428/429 n=2631 ec=345/345 lis/c 428/428 les/c/f 429/429/0 472/472/465) [2,11,5,8,6,2147483647]p2(0) r=-1 lpr=472 pi=[428,472)/1 crt=377'9578 lcod 0'0 unknown NOTIFY] start_peering_interval up [2,11,5,2147483647,6,2147483647] -> [2,11,5,8,6,2147483647], acting [2,11,5,2147483647,6,2147483647] -> [2,11,5,8,6,2147483647], acting_primary 2(0) -> 2, up_primary 2(0) -> 2, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.170659 7f35480a2700 1 osd.10 pg_epoch: 464 pg[6.1f( v 430'114415 (430'112915,430'114415] local-lis/les=411/412 n=26413 ec=76/76 lis/c 411/411 les/c/f 412/440/0 464/464/464) [] r=-1 lpr=464 pi=[411,464)/1 crt=430'114415 lcod 0'0 unknown NOTIFY] start_peering_interval up [3,10,0] -> [], acting [3,10,0] -> [], acting_primary 3 -> -1, up_primary 3 -> -1, role 1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.170698 7f35478a1700 1 osd.10 pg_epoch: 477 pg[13.8s5( v 377'9578 (367'1954,377'9578] local-lis/les=428/429 n=2631 ec=345/345 lis/c 428/428 les/c/f 429/429/0 477/477/465) [2,2147483647,5,8,6,2147483647]p2(0) r=-1 lpr=477 pi=[428,477)/1 crt=377'9578 lcod 0'0 unknown NOTIFY] start_peering_interval up [2,11,5,8,6,2147483647] -> [2,2147483647,5,8,6,2147483647], acting [2,11,5,8,6,2147483647] -> [2,2147483647,5,8,6,2147483647], acting_primary 2(0) -> 2, up_primary 2(0) -> 2, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.170735 7f35480a2700 1 osd.10 pg_epoch: 470 pg[6.1f( v 430'114415 (430'112915,430'114415] local-lis/les=411/412 n=26413 ec=76/76 lis/c 411/411 les/c/f 412/440/0 470/470/470) [0] r=-1 lpr=470 pi=[411,470)/1 crt=430'114415 lcod 0'0 unknown NOTIFY] start_peering_interval up [] -> [0], acting [] -> [0], acting_primary ? -> 0, up_primary ? -> 0, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.170750 7f35478a1700 1 osd.10 pg_epoch: 478 pg[13.8s5( v 377'9578 (367'1954,377'9578] local-lis/les=428/429 n=2631 ec=345/345 lis/c 428/428 les/c/f 429/429/0 478/478/465) [2,11,5,8,6,2147483647]p2(0) r=-1 lpr=478 pi=[428,478)/1 crt=377'9578 lcod 0'0 unknown NOTIFY] start_peering_interval up [2,2147483647,5,8,6,2147483647] -> [2,11,5,8,6,2147483647], acting [2,2147483647,5,8,6,2147483647] -> [2,11,5,8,6,2147483647], acting_primary 2(0) -> 2, up_primary 2(0) -> 2, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.170817 7f35478a1700 1 osd.10 pg_epoch: 480 pg[13.8s5( v 377'9578 (367'1954,377'9578] local-lis/les=428/429 n=2631 ec=345/345 lis/c 428/428 les/c/f 429/429/0 480/480/465) [2,11,5,8,2147483647,2147483647]p2(0) r=-1 lpr=480 pi=[428,480)/1 crt=377'9578 lcod 0'0 unknown NOTIFY] start_peering_interval up [2,11,5,8,6,2147483647] -> [2,11,5,8,2147483647,2147483647], acting [2,11,5,8,6,2147483647] -> [2,11,5,8,2147483647,2147483647], acting_primary 2(0) -> 2, up_primary 2(0) -> 2, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.170851 7f35480a2700 1 osd.10 pg_epoch: 484 pg[6.1f( v 430'114415 (430'112915,430'114415] local-lis/les=411/412 n=26413 ec=76/76 lis/c 411/411 les/c/f 412/440/0 484/484/484) [3,0] r=-1 lpr=484 pi=[411,484)/1 crt=430'114415 lcod 0'0 unknown NOTIFY] start_peering_interval up [0] -> [3,0], acting [0] -> [3,0], acting_primary 0 -> 3, up_primary 0 -> 3, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.170868 7f35478a1700 1 osd.10 pg_epoch: 481 pg[13.8s5( v 377'9578 (367'1954,377'9578] local-lis/les=428/429 n=2631 ec=345/345 lis/c 428/428 les/c/f 429/429/0 481/481/465) [2,11,5,8,6,2147483647]p2(0) r=-1 lpr=481 pi=[428,481)/1 crt=377'9578 lcod 0'0 unknown NOTIFY] start_peering_interval up [2,11,5,8,2147483647,2147483647] -> [2,11,5,8,6,2147483647], acting [2,11,5,8,2147483647,2147483647] -> [2,11,5,8,6,2147483647], acting_primary 2(0) -> 2, up_primary 2(0) -> 2, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.170941 7f35480a2700 1 osd.10 pg_epoch: 492 pg[6.1f( v 430'114415 (430'112915,430'114415] local-lis/les=411/412 n=26413 ec=76/76 lis/c 411/411 les/c/f 412/440/0 492/492/484) [3,10,0] r=1 lpr=492 pi=[411,492)/2 crt=430'114415 lcod 0'0 unknown NOTIFY] start_peering_interval up [3,0] -> [3,10,0], acting [3,0] -> [3,10,0], acting_primary 3 -> 3, up_primary 3 -> 3, role -1 -> 1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.171009 7f35480a2700 1 osd.10 pg_epoch: 492 pg[6.1f( v 430'114415 (430'112915,430'114415] local-lis/les=411/412 n=26413 ec=76/76 lis/c 411/411 les/c/f 412/440/0 492/492/484) [3,10,0] r=1 lpr=492 pi=[411,492)/2 crt=430'114415 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:38.171031 7f35478a1700 1 osd.10 pg_epoch: 492 pg[13.8s5( v 377'9578 (367'1954,377'9578] local-lis/les=428/429 n=2631 ec=345/345 lis/c 428/428 les/c/f 429/429/0 492/492/465) [2,11,5,8,6,10]p2(0) r=5 lpr=492 pi=[428,492)/1 crt=377'9578 lcod 0'0 unknown NOTIFY] start_peering_interval up [2,11,5,8,6,2147483647] -> [2,11,5,8,6,10], acting [2,11,5,8,6,2147483647] -> [2,11,5,8,6,10], acting_primary 2(0) -> 2, up_primary 2(0) -> 2, role -1 -> 5, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.171104 7f35478a1700 1 osd.10 pg_epoch: 492 pg[13.8s5( v 377'9578 (367'1954,377'9578] local-lis/les=428/429 n=2631 ec=345/345 lis/c 428/428 les/c/f 429/429/0 492/492/465) [2,11,5,8,6,10]p2(0) r=5 lpr=492 pi=[428,492)/1 crt=377'9578 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:38.171104 7f35480a2700 1 osd.10 pg_epoch: 464 pg[9.10( v 463'393 (0'0,463'393] local-lis/les=419/421 n=250 ec=190/190 lis/c 419/419 les/c/f 421/449/0 464/464/464) [11] r=-1 lpr=464 pi=[419,464)/1 crt=463'393 lcod 0'0 unknown NOTIFY] start_peering_interval up [4,11,10] -> [11], acting [4,11,10] -> [11], acting_primary 4 -> 11, up_primary 4 -> 11, role 2 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.171187 7f35478a1700 1 osd.10 pg_epoch: 464 pg[6.d( v 461'110631 (430'109050,461'110631] local-lis/les=422/424 n=25732 ec=76/76 lis/c 422/422 les/c/f 424/455/0 422/422/403) [10,7,2] r=0 lpr=461 crt=461'110631 lcod 0'0 mlcod 0'0 peering] state<Started/Primary/Peering>: Peering, affected_by_map, going to Reset >2018-07-11 05:55:38.171220 7f35480a2700 1 osd.10 pg_epoch: 477 pg[9.10( v 463'393 (0'0,463'393] local-lis/les=419/421 n=250 ec=190/190 lis/c 419/419 les/c/f 421/449/0 477/477/477) [] r=-1 lpr=477 pi=[419,477)/1 crt=463'393 lcod 0'0 unknown NOTIFY] start_peering_interval up [11] -> [], acting [11] -> [], acting_primary 11 -> -1, up_primary 11 -> -1, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.171268 7f35480a2700 1 osd.10 pg_epoch: 478 pg[9.10( v 463'393 (0'0,463'393] local-lis/les=419/421 n=250 ec=190/190 lis/c 419/419 les/c/f 421/449/0 478/478/478) [11] r=-1 lpr=478 pi=[419,478)/1 crt=463'393 lcod 0'0 unknown NOTIFY] start_peering_interval up [] -> [11], acting [] -> [11], acting_primary ? -> 11, up_primary ? -> 11, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.171289 7f35478a1700 1 osd.10 pg_epoch: 464 pg[6.d( v 461'110631 (430'109050,461'110631] local-lis/les=422/424 n=25732 ec=76/76 lis/c 422/422 les/c/f 424/455/0 464/464/464) [7] r=-1 lpr=464 pi=[422,464)/1 crt=461'110631 lcod 0'0 unknown] start_peering_interval up [10,7,2] -> [7], acting [10,7,2] -> [7], acting_primary 10 -> 7, up_primary 10 -> 7, role 0 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.171337 7f35478a1700 1 osd.10 pg_epoch: 465 pg[6.d( v 461'110631 (430'109050,461'110631] local-lis/les=422/424 n=25732 ec=76/76 lis/c 422/422 les/c/f 424/455/0 465/465/464) [7,2] r=-1 lpr=465 pi=[422,465)/1 crt=461'110631 lcod 0'0 unknown NOTIFY] start_peering_interval up [7] -> [7,2], acting [7] -> [7,2], acting_primary 7 -> 7, up_primary 7 -> 7, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.171360 7f35480a2700 1 osd.10 pg_epoch: 487 pg[9.10( v 463'393 (0'0,463'393] local-lis/les=419/421 n=250 ec=190/190 lis/c 419/419 les/c/f 421/449/0 487/487/487) [4,11] r=-1 lpr=487 pi=[419,487)/1 crt=463'393 lcod 0'0 unknown NOTIFY] start_peering_interval up [11] -> [4,11], acting [11] -> [4,11], acting_primary 11 -> 4, up_primary 11 -> 4, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.171386 7f35478a1700 1 osd.10 pg_epoch: 467 pg[6.d( v 461'110631 (430'109050,461'110631] local-lis/les=422/424 n=25732 ec=76/76 lis/c 422/422 les/c/f 424/455/0 467/467/467) [2] r=-1 lpr=467 pi=[422,467)/1 crt=461'110631 lcod 0'0 unknown NOTIFY] start_peering_interval up [7,2] -> [2], acting [7,2] -> [2], acting_primary 7 -> 2, up_primary 7 -> 2, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.171427 7f35478a1700 1 osd.10 pg_epoch: 468 pg[6.d( v 461'110631 (430'109050,461'110631] local-lis/les=422/424 n=25732 ec=76/76 lis/c 422/422 les/c/f 424/455/0 468/468/468) [7,2] r=-1 lpr=468 pi=[422,468)/1 crt=461'110631 lcod 0'0 unknown NOTIFY] start_peering_interval up [2] -> [7,2], acting [2] -> [7,2], acting_primary 2 -> 7, up_primary 2 -> 7, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.171432 7f35480a2700 1 osd.10 pg_epoch: 492 pg[9.10( v 463'393 (0'0,463'393] local-lis/les=419/421 n=250 ec=190/190 lis/c 419/419 les/c/f 421/449/0 492/492/487) [4,11,10] r=2 lpr=492 pi=[419,492)/1 crt=463'393 lcod 0'0 unknown NOTIFY] start_peering_interval up [4,11] -> [4,11,10], acting [4,11] -> [4,11,10], acting_primary 4 -> 4, up_primary 4 -> 4, role -1 -> 2, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.171495 7f35480a2700 1 osd.10 pg_epoch: 492 pg[9.10( v 463'393 (0'0,463'393] local-lis/les=419/421 n=250 ec=190/190 lis/c 419/419 les/c/f 421/449/0 492/492/487) [4,11,10] r=2 lpr=492 pi=[419,492)/1 crt=463'393 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:38.171600 7f35478a1700 1 osd.10 pg_epoch: 492 pg[6.d( v 461'110631 (430'109050,461'110631] local-lis/les=422/424 n=25732 ec=76/76 lis/c 422/422 les/c/f 424/455/0 492/492/492) [10,7,2] r=0 lpr=492 pi=[422,492)/1 crt=461'110631 lcod 0'0 mlcod 0'0 unknown NOTIFY] start_peering_interval up [7,2] -> [10,7,2], acting [7,2] -> [10,7,2], acting_primary 7 -> 10, up_primary 7 -> 10, role -1 -> 0, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.171668 7f35478a1700 1 osd.10 pg_epoch: 492 pg[6.d( v 461'110631 (430'109050,461'110631] local-lis/les=422/424 n=25732 ec=76/76 lis/c 422/422 les/c/f 424/455/0 492/492/492) [10,7,2] r=0 lpr=492 pi=[422,492)/1 crt=461'110631 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 05:55:38.171807 7f35478a1700 1 osd.10 pg_epoch: 464 pg[9.d( v 463'449 (0'0,463'449] local-lis/les=428/429 n=227 ec=190/190 lis/c 428/428 les/c/f 429/455/0 464/464/464) [6] r=-1 lpr=464 pi=[428,464)/1 crt=463'449 lcod 0'0 unknown NOTIFY] start_peering_interval up [0,6,10] -> [6], acting [0,6,10] -> [6], acting_primary 0 -> 6, up_primary 0 -> 6, role 2 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.171875 7f35480a2700 1 osd.10 pg_epoch: 464 pg[16.1a( empty local-lis/les=442/443 n=0 ec=442/442 lis/c 442/442 les/c/f 443/443/0 464/464/442) [11] r=-1 lpr=464 pi=[442,464)/1 crt=0'0 unknown NOTIFY] start_peering_interval up [11,10,3] -> [11], acting [11,10,3] -> [11], acting_primary 11 -> 11, up_primary 11 -> 11, role 1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.171892 7f35478a1700 1 osd.10 pg_epoch: 470 pg[9.d( v 463'449 (0'0,463'449] local-lis/les=428/429 n=227 ec=190/190 lis/c 428/428 les/c/f 429/455/0 470/470/470) [0,6] r=-1 lpr=470 pi=[428,470)/1 crt=463'449 lcod 0'0 unknown NOTIFY] start_peering_interval up [6] -> [0,6], acting [6] -> [0,6], acting_primary 6 -> 0, up_primary 6 -> 0, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.172011 7f35480a2700 1 osd.10 pg_epoch: 477 pg[16.1a( empty local-lis/les=442/443 n=0 ec=442/442 lis/c 442/442 les/c/f 443/443/0 477/477/477) [] r=-1 lpr=477 pi=[442,477)/1 crt=0'0 unknown NOTIFY] start_peering_interval up [11] -> [], acting [11] -> [], acting_primary 11 -> -1, up_primary 11 -> -1, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.172039 7f35478a1700 1 osd.10 pg_epoch: 480 pg[9.d( v 463'449 (0'0,463'449] local-lis/les=428/429 n=227 ec=190/190 lis/c 428/428 les/c/f 429/455/0 480/480/470) [0] r=-1 lpr=480 pi=[428,480)/1 crt=463'449 lcod 0'0 unknown NOTIFY] start_peering_interval up [0,6] -> [0], acting [0,6] -> [0], acting_primary 0 -> 0, up_primary 0 -> 0, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.172050 7f35480a2700 1 osd.10 pg_epoch: 478 pg[16.1a( empty local-lis/les=442/443 n=0 ec=442/442 lis/c 442/442 les/c/f 443/443/0 478/478/478) [11] r=-1 lpr=478 pi=[442,478)/1 crt=0'0 unknown NOTIFY] start_peering_interval up [] -> [11], acting [] -> [11], acting_primary ? -> 11, up_primary ? -> 11, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.172089 7f35478a1700 1 osd.10 pg_epoch: 481 pg[9.d( v 463'449 (0'0,463'449] local-lis/les=428/429 n=227 ec=190/190 lis/c 428/428 les/c/f 429/455/0 481/481/470) [0,6] r=-1 lpr=481 pi=[428,481)/1 crt=463'449 lcod 0'0 unknown NOTIFY] start_peering_interval up [0] -> [0,6], acting [0] -> [0,6], acting_primary 0 -> 0, up_primary 0 -> 0, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.172141 7f35480a2700 1 osd.10 pg_epoch: 484 pg[16.1a( empty local-lis/les=442/443 n=0 ec=442/442 lis/c 442/442 les/c/f 443/443/0 484/484/478) [11,3] r=-1 lpr=484 pi=[442,484)/1 crt=0'0 unknown NOTIFY] start_peering_interval up [11] -> [11,3], acting [11] -> [11,3], acting_primary 11 -> 11, up_primary 11 -> 11, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.172228 7f35478a1700 1 osd.10 pg_epoch: 492 pg[9.d( v 463'449 (0'0,463'449] local-lis/les=428/429 n=227 ec=190/190 lis/c 428/428 les/c/f 429/455/0 492/492/470) [0,6,10] r=2 lpr=492 pi=[428,492)/1 crt=463'449 lcod 0'0 unknown NOTIFY] start_peering_interval up [0,6] -> [0,6,10], acting [0,6] -> [0,6,10], acting_primary 0 -> 0, up_primary 0 -> 0, role -1 -> 2, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.172286 7f35480a2700 1 osd.10 pg_epoch: 492 pg[16.1a( empty local-lis/les=442/443 n=0 ec=442/442 lis/c 442/442 les/c/f 443/443/0 492/492/478) [11,10,3] r=1 lpr=492 pi=[442,492)/1 crt=0'0 unknown NOTIFY] start_peering_interval up [11,3] -> [11,10,3], acting [11,3] -> [11,10,3], acting_primary 11 -> 11, up_primary 11 -> 11, role -1 -> 1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.172305 7f35478a1700 1 osd.10 pg_epoch: 492 pg[9.d( v 463'449 (0'0,463'449] local-lis/les=428/429 n=227 ec=190/190 lis/c 428/428 les/c/f 429/455/0 492/492/470) [0,6,10] r=2 lpr=492 pi=[428,492)/1 crt=463'449 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:38.172352 7f35480a2700 1 osd.10 pg_epoch: 492 pg[16.1a( empty local-lis/les=442/443 n=0 ec=442/442 lis/c 442/442 les/c/f 443/443/0 492/492/478) [11,10,3] r=1 lpr=492 pi=[442,492)/1 crt=0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:38.172453 7f35478a1700 1 osd.10 pg_epoch: 464 pg[6.2( v 430'110237 (430'108737,430'110237] local-lis/les=422/423 n=25907 ec=76/76 lis/c 422/422 les/c/f 423/445/0 464/464/422) [7] r=-1 lpr=464 pi=[422,464)/1 crt=430'110237 lcod 0'0 unknown NOTIFY] start_peering_interval up [7,10,2] -> [7], acting [7,10,2] -> [7], acting_primary 7 -> 7, up_primary 7 -> 7, role 1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.172485 7f35480a2700 1 osd.10 pg_epoch: 464 pg[9.1a( v 461'339 (0'0,461'339] local-lis/les=422/423 n=240 ec=190/190 lis/c 422/422 les/c/f 423/423/0 464/464/422) [7] r=-1 lpr=464 pi=[422,464)/1 crt=461'339 lcod 0'0 unknown NOTIFY] start_peering_interval up [7,2,10] -> [7], acting [7,2,10] -> [7], acting_primary 7 -> 7, up_primary 7 -> 7, role 2 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.172518 7f35478a1700 1 osd.10 pg_epoch: 465 pg[6.2( v 430'110237 (430'108737,430'110237] local-lis/les=422/423 n=25907 ec=76/76 lis/c 422/422 les/c/f 423/445/0 465/465/422) [7,2] r=-1 lpr=465 pi=[422,465)/1 crt=430'110237 lcod 0'0 unknown NOTIFY] start_peering_interval up [7] -> [7,2], acting [7] -> [7,2], acting_primary 7 -> 7, up_primary 7 -> 7, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.172531 7f35480a2700 1 osd.10 pg_epoch: 465 pg[9.1a( v 461'339 (0'0,461'339] local-lis/les=422/423 n=240 ec=190/190 lis/c 422/422 les/c/f 423/423/0 465/465/422) [7,2] r=-1 lpr=465 pi=[422,465)/1 crt=461'339 lcod 0'0 unknown NOTIFY] start_peering_interval up [7] -> [7,2], acting [7] -> [7,2], acting_primary 7 -> 7, up_primary 7 -> 7, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.172571 7f35478a1700 1 osd.10 pg_epoch: 467 pg[6.2( v 430'110237 (430'108737,430'110237] local-lis/les=422/423 n=25907 ec=76/76 lis/c 422/422 les/c/f 423/445/0 467/467/467) [2] r=-1 lpr=467 pi=[422,467)/1 crt=430'110237 lcod 0'0 unknown NOTIFY] start_peering_interval up [7,2] -> [2], acting [7,2] -> [2], acting_primary 7 -> 2, up_primary 7 -> 2, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.172578 7f35480a2700 1 osd.10 pg_epoch: 467 pg[9.1a( v 461'339 (0'0,461'339] local-lis/les=422/423 n=240 ec=190/190 lis/c 422/422 les/c/f 423/423/0 467/467/467) [2] r=-1 lpr=467 pi=[422,467)/1 crt=461'339 lcod 0'0 unknown NOTIFY] start_peering_interval up [7,2] -> [2], acting [7,2] -> [2], acting_primary 7 -> 2, up_primary 7 -> 2, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.172611 7f35478a1700 1 osd.10 pg_epoch: 468 pg[6.2( v 430'110237 (430'108737,430'110237] local-lis/les=422/423 n=25907 ec=76/76 lis/c 422/422 les/c/f 423/445/0 468/468/468) [7,2] r=-1 lpr=468 pi=[422,468)/1 crt=430'110237 lcod 0'0 unknown NOTIFY] start_peering_interval up [2] -> [7,2], acting [2] -> [7,2], acting_primary 2 -> 7, up_primary 2 -> 7, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.172615 7f35480a2700 1 osd.10 pg_epoch: 468 pg[9.1a( v 461'339 (0'0,461'339] local-lis/les=422/423 n=240 ec=190/190 lis/c 422/422 les/c/f 423/423/0 468/468/468) [7,2] r=-1 lpr=468 pi=[422,468)/1 crt=461'339 lcod 0'0 unknown NOTIFY] start_peering_interval up [2] -> [7,2], acting [2] -> [7,2], acting_primary 2 -> 7, up_primary 2 -> 7, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.172826 7f35478a1700 1 osd.10 pg_epoch: 492 pg[6.2( v 430'110237 (430'108737,430'110237] local-lis/les=422/423 n=25907 ec=76/76 lis/c 422/422 les/c/f 423/445/0 492/492/468) [7,10,2] r=1 lpr=492 pi=[422,492)/1 crt=430'110237 lcod 0'0 unknown NOTIFY] start_peering_interval up [7,2] -> [7,10,2], acting [7,2] -> [7,10,2], acting_primary 7 -> 7, up_primary 7 -> 7, role -1 -> 1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.172831 7f35480a2700 1 osd.10 pg_epoch: 492 pg[9.1a( v 461'339 (0'0,461'339] local-lis/les=422/423 n=240 ec=190/190 lis/c 422/422 les/c/f 423/423/0 492/492/468) [7,2,10] r=2 lpr=492 pi=[422,492)/1 crt=461'339 lcod 0'0 unknown NOTIFY] start_peering_interval up [7,2] -> [7,2,10], acting [7,2] -> [7,2,10], acting_primary 7 -> 7, up_primary 7 -> 7, role -1 -> 2, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.172900 7f35480a2700 1 osd.10 pg_epoch: 492 pg[9.1a( v 461'339 (0'0,461'339] local-lis/les=422/423 n=240 ec=190/190 lis/c 422/422 les/c/f 423/423/0 492/492/468) [7,2,10] r=2 lpr=492 pi=[422,492)/1 crt=461'339 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:38.172932 7f35478a1700 1 osd.10 pg_epoch: 492 pg[6.2( v 430'110237 (430'108737,430'110237] local-lis/les=422/423 n=25907 ec=76/76 lis/c 422/422 les/c/f 423/445/0 492/492/468) [7,10,2] r=1 lpr=492 pi=[422,492)/1 crt=430'110237 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:38.172999 7f35480a2700 1 osd.10 pg_epoch: 464 pg[6.15( v 463'106940 (430'105357,463'106940] local-lis/les=419/421 n=25757 ec=76/76 lis/c 419/419 les/c/f 421/430/0 464/464/464) [11] r=-1 lpr=464 pi=[419,464)/1 crt=463'106940 lcod 0'0 unknown NOTIFY] start_peering_interval up [5,10,11] -> [11], acting [5,10,11] -> [11], acting_primary 5 -> 11, up_primary 5 -> 11, role 1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.173066 7f35480a2700 1 osd.10 pg_epoch: 467 pg[6.15( v 463'106940 (430'105357,463'106940] local-lis/les=419/421 n=25757 ec=76/76 lis/c 419/419 les/c/f 421/430/0 467/467/467) [5,11] r=-1 lpr=467 pi=[419,467)/1 crt=463'106940 lcod 0'0 unknown NOTIFY] start_peering_interval up [11] -> [5,11], acting [11] -> [5,11], acting_primary 11 -> 5, up_primary 11 -> 5, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.173107 7f35478a1700 1 osd.10 pg_epoch: 464 pg[11.as3( v 383'60057 (367'56547,383'60057] local-lis/les=428/429 n=60057 ec=267/267 lis/c 428/428 les/c/f 429/429/0 464/464/464) [2147483647,2147483647,2147483647,2147483647,11,6]p11(4) r=-1 lpr=464 pi=[428,464)/1 crt=383'60057 lcod 0'0 unknown NOTIFY] start_peering_interval up [3,5,8,10,11,6] -> [2147483647,2147483647,2147483647,2147483647,11,6], acting [3,5,8,10,11,6] -> [2147483647,2147483647,2147483647,2147483647,11,6], acting_primary 3(0) -> 11, up_primary 3(0) -> 11, role 3 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.173187 7f35478a1700 1 osd.10 pg_epoch: 467 pg[11.as3( v 383'60057 (367'56547,383'60057] local-lis/les=428/429 n=60057 ec=267/267 lis/c 428/428 les/c/f 429/429/0 467/467/467) [2147483647,5,2147483647,2147483647,11,6]p5(1) r=-1 lpr=467 pi=[428,467)/1 crt=383'60057 lcod 0'0 unknown NOTIFY] start_peering_interval up [2147483647,2147483647,2147483647,2147483647,11,6] -> [2147483647,5,2147483647,2147483647,11,6], acting [2147483647,2147483647,2147483647,2147483647,11,6] -> [2147483647,5,2147483647,2147483647,11,6], acting_primary 11(4) -> 5, up_primary 11(4) -> 5, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.173191 7f35480a2700 1 osd.10 pg_epoch: 477 pg[6.15( v 463'106940 (430'105357,463'106940] local-lis/les=419/421 n=25757 ec=76/76 lis/c 419/419 les/c/f 421/430/0 477/477/467) [5] r=-1 lpr=477 pi=[419,477)/1 crt=463'106940 lcod 0'0 unknown NOTIFY] start_peering_interval up [5,11] -> [5], acting [5,11] -> [5], acting_primary 5 -> 5, up_primary 5 -> 5, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.173251 7f35480a2700 1 osd.10 pg_epoch: 478 pg[6.15( v 463'106940 (430'105357,463'106940] local-lis/les=419/421 n=25757 ec=76/76 lis/c 419/419 les/c/f 421/430/0 478/478/467) [5,11] r=-1 lpr=478 pi=[419,478)/1 crt=463'106940 lcod 0'0 unknown NOTIFY] start_peering_interval up [5] -> [5,11], acting [5] -> [5,11], acting_primary 5 -> 5, up_primary 5 -> 5, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.173297 7f35478a1700 1 osd.10 pg_epoch: 472 pg[11.as3( v 383'60057 (367'56547,383'60057] local-lis/les=428/429 n=60057 ec=267/267 lis/c 428/428 les/c/f 429/429/0 472/472/467) [2147483647,5,8,2147483647,11,6]p5(1) r=-1 lpr=472 pi=[428,472)/1 crt=383'60057 lcod 0'0 unknown NOTIFY] start_peering_interval up [2147483647,5,2147483647,2147483647,11,6] -> [2147483647,5,8,2147483647,11,6], acting [2147483647,5,2147483647,2147483647,11,6] -> [2147483647,5,8,2147483647,11,6], acting_primary 5(1) -> 5, up_primary 5(1) -> 5, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.173398 7f35478a1700 1 osd.10 pg_epoch: 477 pg[11.as3( v 383'60057 (367'56547,383'60057] local-lis/les=428/429 n=60057 ec=267/267 lis/c 428/428 les/c/f 429/429/0 477/477/467) [2147483647,5,8,2147483647,2147483647,6]p5(1) r=-1 lpr=477 pi=[428,477)/1 crt=383'60057 lcod 0'0 unknown NOTIFY] start_peering_interval up [2147483647,5,8,2147483647,11,6] -> [2147483647,5,8,2147483647,2147483647,6], acting [2147483647,5,8,2147483647,11,6] -> [2147483647,5,8,2147483647,2147483647,6], acting_primary 5(1) -> 5, up_primary 5(1) -> 5, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.173415 7f35480a2700 1 osd.10 pg_epoch: 492 pg[6.15( v 463'106940 (430'105357,463'106940] local-lis/les=419/421 n=25757 ec=76/76 lis/c 419/419 les/c/f 421/430/0 492/492/467) [5,10,11] r=1 lpr=492 pi=[419,492)/1 crt=463'106940 lcod 0'0 unknown NOTIFY] start_peering_interval up [5,11] -> [5,10,11], acting [5,11] -> [5,10,11], acting_primary 5 -> 5, up_primary 5 -> 5, role -1 -> 1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.173449 7f35478a1700 1 osd.10 pg_epoch: 478 pg[11.as3( v 383'60057 (367'56547,383'60057] local-lis/les=428/429 n=60057 ec=267/267 lis/c 428/428 les/c/f 429/429/0 478/478/467) [2147483647,5,8,2147483647,11,6]p5(1) r=-1 lpr=478 pi=[428,478)/1 crt=383'60057 lcod 0'0 unknown NOTIFY] start_peering_interval up [2147483647,5,8,2147483647,2147483647,6] -> [2147483647,5,8,2147483647,11,6], acting [2147483647,5,8,2147483647,2147483647,6] -> [2147483647,5,8,2147483647,11,6], acting_primary 5(1) -> 5, up_primary 5(1) -> 5, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.173481 7f35480a2700 1 osd.10 pg_epoch: 492 pg[6.15( v 463'106940 (430'105357,463'106940] local-lis/les=419/421 n=25757 ec=76/76 lis/c 419/419 les/c/f 421/430/0 492/492/467) [5,10,11] r=1 lpr=492 pi=[419,492)/1 crt=463'106940 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:38.173509 7f35478a1700 1 osd.10 pg_epoch: 480 pg[11.as3( v 383'60057 (367'56547,383'60057] local-lis/les=428/429 n=60057 ec=267/267 lis/c 428/428 les/c/f 429/429/0 480/480/467) [2147483647,5,8,2147483647,11,2147483647]p5(1) r=-1 lpr=480 pi=[428,480)/1 crt=383'60057 lcod 0'0 unknown NOTIFY] start_peering_interval up [2147483647,5,8,2147483647,11,6] -> [2147483647,5,8,2147483647,11,2147483647], acting [2147483647,5,8,2147483647,11,6] -> [2147483647,5,8,2147483647,11,2147483647], acting_primary 5(1) -> 5, up_primary 5(1) -> 5, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.173559 7f35478a1700 1 osd.10 pg_epoch: 481 pg[11.as3( v 383'60057 (367'56547,383'60057] local-lis/les=428/429 n=60057 ec=267/267 lis/c 428/428 les/c/f 429/429/0 481/481/467) [2147483647,5,8,2147483647,11,6]p5(1) r=-1 lpr=481 pi=[428,481)/1 crt=383'60057 lcod 0'0 unknown NOTIFY] start_peering_interval up [2147483647,5,8,2147483647,11,2147483647] -> [2147483647,5,8,2147483647,11,6], acting [2147483647,5,8,2147483647,11,2147483647] -> [2147483647,5,8,2147483647,11,6], acting_primary 5(1) -> 5, up_primary 5(1) -> 5, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.173604 7f35480a2700 1 osd.10 pg_epoch: 464 pg[16.3( empty local-lis/les=442/443 n=0 ec=442/442 lis/c 442/442 les/c/f 443/443/0 464/464/464) [] r=-1 lpr=464 pi=[442,464)/1 crt=0'0 unknown NOTIFY] start_peering_interval up [5,2,10] -> [], acting [5,2,10] -> [], acting_primary 5 -> -1, up_primary 5 -> -1, role 2 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.173626 7f35478a1700 1 osd.10 pg_epoch: 484 pg[11.as3( v 383'60057 (367'56547,383'60057] local-lis/les=428/429 n=60057 ec=267/267 lis/c 428/428 les/c/f 429/429/0 484/484/484) [3,5,8,2147483647,11,6]p3(0) r=-1 lpr=484 pi=[428,484)/1 crt=383'60057 lcod 0'0 unknown NOTIFY] start_peering_interval up [2147483647,5,8,2147483647,11,6] -> [3,5,8,2147483647,11,6], acting [2147483647,5,8,2147483647,11,6] -> [3,5,8,2147483647,11,6], acting_primary 5(1) -> 3, up_primary 5(1) -> 3, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.173651 7f35480a2700 1 osd.10 pg_epoch: 465 pg[16.3( empty local-lis/les=442/443 n=0 ec=442/442 lis/c 442/442 les/c/f 443/443/0 465/465/465) [2] r=-1 lpr=465 pi=[442,465)/1 crt=0'0 unknown NOTIFY] start_peering_interval up [] -> [2], acting [] -> [2], acting_primary ? -> 2, up_primary ? -> 2, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.173698 7f35480a2700 1 osd.10 pg_epoch: 467 pg[16.3( empty local-lis/les=442/443 n=0 ec=442/442 lis/c 442/442 les/c/f 443/443/0 467/467/467) [5,2] r=-1 lpr=467 pi=[442,467)/1 crt=0'0 unknown NOTIFY] start_peering_interval up [2] -> [5,2], acting [2] -> [5,2], acting_primary 2 -> 5, up_primary 2 -> 5, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.173762 7f35478a1700 1 osd.10 pg_epoch: 492 pg[11.as3( v 383'60057 (367'56547,383'60057] local-lis/les=428/429 n=60057 ec=267/267 lis/c 428/428 les/c/f 429/429/0 492/492/484) [3,5,8,10,11,6]p3(0) r=3 lpr=492 pi=[428,492)/1 crt=383'60057 lcod 0'0 unknown NOTIFY] start_peering_interval up [3,5,8,2147483647,11,6] -> [3,5,8,10,11,6], acting [3,5,8,2147483647,11,6] -> [3,5,8,10,11,6], acting_primary 3(0) -> 3, up_primary 3(0) -> 3, role -1 -> 3, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.173835 7f35478a1700 1 osd.10 pg_epoch: 492 pg[11.as3( v 383'60057 (367'56547,383'60057] local-lis/les=428/429 n=60057 ec=267/267 lis/c 428/428 les/c/f 429/429/0 492/492/484) [3,5,8,10,11,6]p3(0) r=3 lpr=492 pi=[428,492)/1 crt=383'60057 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:38.173921 7f35480a2700 1 osd.10 pg_epoch: 492 pg[16.3( empty local-lis/les=442/443 n=0 ec=442/442 lis/c 442/442 les/c/f 443/443/0 492/492/467) [5,2,10] r=2 lpr=492 pi=[442,492)/1 crt=0'0 unknown NOTIFY] start_peering_interval up [5,2] -> [5,2,10], acting [5,2] -> [5,2,10], acting_primary 5 -> 5, up_primary 5 -> 5, role -1 -> 2, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.173991 7f35480a2700 1 osd.10 pg_epoch: 492 pg[16.3( empty local-lis/les=442/443 n=0 ec=442/442 lis/c 442/442 les/c/f 443/443/0 492/492/467) [5,2,10] r=2 lpr=492 pi=[442,492)/1 crt=0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:38.173986 7f35478a1700 1 osd.10 pg_epoch: 464 pg[13.bs4( v 377'9782 (355'1491,377'9782] local-lis/les=428/430 n=2694 ec=345/345 lis/c 428/428 les/c/f 430/430/0 464/464/419) [11,2147483647,2147483647,2147483647,2147483647,6]p11(0) r=-1 lpr=464 pi=[428,464)/1 crt=377'9782 lcod 0'0 unknown NOTIFY] start_peering_interval up [11,8,1,4,10,6] -> [11,2147483647,2147483647,2147483647,2147483647,6], acting [11,8,1,4,10,6] -> [11,2147483647,2147483647,2147483647,2147483647,6], acting_primary 11(0) -> 11, up_primary 11(0) -> 11, role 4 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.174084 7f35478a1700 1 osd.10 pg_epoch: 470 pg[13.bs4( v 377'9782 (355'1491,377'9782] local-lis/les=428/430 n=2694 ec=345/345 lis/c 428/428 les/c/f 430/430/0 470/470/419) [11,2147483647,1,2147483647,2147483647,6]p11(0) r=-1 lpr=470 pi=[428,470)/1 crt=377'9782 lcod 0'0 unknown NOTIFY] start_peering_interval up [11,2147483647,2147483647,2147483647,2147483647,6] -> [11,2147483647,1,2147483647,2147483647,6], acting [11,2147483647,2147483647,2147483647,2147483647,6] -> [11,2147483647,1,2147483647,2147483647,6], acting_primary 11(0) -> 11, up_primary 11(0) -> 11, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.174130 7f35480a2700 1 osd.10 pg_epoch: 464 pg[13.3s1( v 377'10405 (371'2081,377'10405] local-lis/les=428/430 n=2736 ec=345/345 lis/c 428/428 les/c/f 430/430/0 464/464/464) [2147483647,2147483647,2147483647,6,7,2147483647]p6(3) r=-1 lpr=464 pi=[428,464)/1 crt=377'10405 lcod 0'0 unknown NOTIFY] start_peering_interval up [1,10,8,6,7,5] -> [2147483647,2147483647,2147483647,6,7,2147483647], acting [1,10,8,6,7,5] -> [2147483647,2147483647,2147483647,6,7,2147483647], acting_primary 1(0) -> 6, up_primary 1(0) -> 6, role 1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.174140 7f35478a1700 1 osd.10 pg_epoch: 472 pg[13.bs4( v 377'9782 (355'1491,377'9782] local-lis/les=428/430 n=2694 ec=345/345 lis/c 428/428 les/c/f 430/430/0 472/472/419) [11,8,1,2147483647,2147483647,6]p11(0) r=-1 lpr=472 pi=[428,472)/1 crt=377'9782 lcod 0'0 unknown NOTIFY] start_peering_interval up [11,2147483647,1,2147483647,2147483647,6] -> [11,8,1,2147483647,2147483647,6], acting [11,2147483647,1,2147483647,2147483647,6] -> [11,8,1,2147483647,2147483647,6], acting_primary 11(0) -> 11, up_primary 11(0) -> 11, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.174202 7f35480a2700 1 osd.10 pg_epoch: 467 pg[13.3s1( v 377'10405 (371'2081,377'10405] local-lis/les=428/430 n=2736 ec=345/345 lis/c 428/428 les/c/f 430/430/0 467/467/464) [2147483647,2147483647,2147483647,6,2147483647,5]p6(3) r=-1 lpr=467 pi=[428,467)/1 crt=377'10405 lcod 0'0 unknown NOTIFY] start_peering_interval up [2147483647,2147483647,2147483647,6,7,2147483647] -> [2147483647,2147483647,2147483647,6,2147483647,5], acting [2147483647,2147483647,2147483647,6,7,2147483647] -> [2147483647,2147483647,2147483647,6,2147483647,5], acting_primary 6(3) -> 6, up_primary 6(3) -> 6, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.174222 7f35478a1700 1 osd.10 pg_epoch: 477 pg[13.bs4( v 377'9782 (355'1491,377'9782] local-lis/les=428/430 n=2694 ec=345/345 lis/c 428/428 les/c/f 430/430/0 477/477/477) [2147483647,8,1,2147483647,2147483647,6]p8(1) r=-1 lpr=477 pi=[428,477)/1 crt=377'9782 lcod 0'0 unknown NOTIFY] start_peering_interval up [11,8,1,2147483647,2147483647,6] -> [2147483647,8,1,2147483647,2147483647,6], acting [11,8,1,2147483647,2147483647,6] -> [2147483647,8,1,2147483647,2147483647,6], acting_primary 11(0) -> 8, up_primary 11(0) -> 8, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.174261 7f35480a2700 1 osd.10 pg_epoch: 468 pg[13.3s1( v 377'10405 (371'2081,377'10405] local-lis/les=428/430 n=2736 ec=345/345 lis/c 428/428 les/c/f 430/430/0 468/468/464) [2147483647,2147483647,2147483647,6,7,5]p6(3) r=-1 lpr=468 pi=[428,468)/1 crt=377'10405 lcod 0'0 unknown NOTIFY] start_peering_interval up [2147483647,2147483647,2147483647,6,2147483647,5] -> [2147483647,2147483647,2147483647,6,7,5], acting [2147483647,2147483647,2147483647,6,2147483647,5] -> [2147483647,2147483647,2147483647,6,7,5], acting_primary 6(3) -> 6, up_primary 6(3) -> 6, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.174278 7f35478a1700 1 osd.10 pg_epoch: 478 pg[13.bs4( v 377'9782 (355'1491,377'9782] local-lis/les=428/430 n=2694 ec=345/345 lis/c 428/428 les/c/f 430/430/0 478/478/478) [11,8,1,2147483647,2147483647,6]p11(0) r=-1 lpr=478 pi=[428,478)/1 crt=377'9782 lcod 0'0 unknown NOTIFY] start_peering_interval up [2147483647,8,1,2147483647,2147483647,6] -> [11,8,1,2147483647,2147483647,6], acting [2147483647,8,1,2147483647,2147483647,6] -> [11,8,1,2147483647,2147483647,6], acting_primary 8(1) -> 11, up_primary 8(1) -> 11, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.174321 7f35480a2700 1 osd.10 pg_epoch: 470 pg[13.3s1( v 377'10405 (371'2081,377'10405] local-lis/les=428/430 n=2736 ec=345/345 lis/c 428/428 les/c/f 430/430/0 470/470/470) [1,2147483647,2147483647,6,7,5]p1(0) r=-1 lpr=470 pi=[428,470)/1 crt=377'10405 lcod 0'0 unknown NOTIFY] start_peering_interval up [2147483647,2147483647,2147483647,6,7,5] -> [1,2147483647,2147483647,6,7,5], acting [2147483647,2147483647,2147483647,6,7,5] -> [1,2147483647,2147483647,6,7,5], acting_primary 6(3) -> 1, up_primary 6(3) -> 1, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.174337 7f35478a1700 1 osd.10 pg_epoch: 480 pg[13.bs4( v 377'9782 (355'1491,377'9782] local-lis/les=428/430 n=2694 ec=345/345 lis/c 428/428 les/c/f 430/430/0 480/480/478) [11,8,1,2147483647,2147483647,2147483647]p11(0) r=-1 lpr=480 pi=[428,480)/1 crt=377'9782 lcod 0'0 unknown NOTIFY] start_peering_interval up [11,8,1,2147483647,2147483647,6] -> [11,8,1,2147483647,2147483647,2147483647], acting [11,8,1,2147483647,2147483647,6] -> [11,8,1,2147483647,2147483647,2147483647], acting_primary 11(0) -> 11, up_primary 11(0) -> 11, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.174378 7f35480a2700 1 osd.10 pg_epoch: 472 pg[13.3s1( v 377'10405 (371'2081,377'10405] local-lis/les=428/430 n=2736 ec=345/345 lis/c 428/428 les/c/f 430/430/0 472/472/470) [1,2147483647,8,6,7,5]p1(0) r=-1 lpr=472 pi=[428,472)/1 crt=377'10405 lcod 0'0 unknown NOTIFY] start_peering_interval up [1,2147483647,2147483647,6,7,5] -> [1,2147483647,8,6,7,5], acting [1,2147483647,2147483647,6,7,5] -> [1,2147483647,8,6,7,5], acting_primary 1(0) -> 1, up_primary 1(0) -> 1, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.174385 7f35478a1700 1 osd.10 pg_epoch: 481 pg[13.bs4( v 377'9782 (355'1491,377'9782] local-lis/les=428/430 n=2694 ec=345/345 lis/c 428/428 les/c/f 430/430/0 481/481/478) [11,8,1,2147483647,2147483647,6]p11(0) r=-1 lpr=481 pi=[428,481)/1 crt=377'9782 lcod 0'0 unknown NOTIFY] start_peering_interval up [11,8,1,2147483647,2147483647,2147483647] -> [11,8,1,2147483647,2147483647,6], acting [11,8,1,2147483647,2147483647,2147483647] -> [11,8,1,2147483647,2147483647,6], acting_primary 11(0) -> 11, up_primary 11(0) -> 11, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.174478 7f35478a1700 1 osd.10 pg_epoch: 487 pg[13.bs4( v 377'9782 (355'1491,377'9782] local-lis/les=428/430 n=2694 ec=345/345 lis/c 428/428 les/c/f 430/430/0 487/487/478) [11,8,1,4,2147483647,6]p11(0) r=-1 lpr=487 pi=[428,487)/1 crt=377'9782 lcod 0'0 unknown NOTIFY] start_peering_interval up [11,8,1,2147483647,2147483647,6] -> [11,8,1,4,2147483647,6], acting [11,8,1,2147483647,2147483647,6] -> [11,8,1,4,2147483647,6], acting_primary 11(0) -> 11, up_primary 11(0) -> 11, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.174493 7f35480a2700 1 osd.10 pg_epoch: 480 pg[13.3s1( v 377'10405 (371'2081,377'10405] local-lis/les=428/430 n=2736 ec=345/345 lis/c 428/428 les/c/f 430/430/0 480/480/470) [1,2147483647,8,2147483647,7,5]p1(0) r=-1 lpr=480 pi=[428,480)/1 crt=377'10405 lcod 0'0 unknown NOTIFY] start_peering_interval up [1,2147483647,8,6,7,5] -> [1,2147483647,8,2147483647,7,5], acting [1,2147483647,8,6,7,5] -> [1,2147483647,8,2147483647,7,5], acting_primary 1(0) -> 1, up_primary 1(0) -> 1, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.174542 7f35480a2700 1 osd.10 pg_epoch: 481 pg[13.3s1( v 377'10405 (371'2081,377'10405] local-lis/les=428/430 n=2736 ec=345/345 lis/c 428/428 les/c/f 430/430/0 481/481/470) [1,2147483647,8,6,7,5]p1(0) r=-1 lpr=481 pi=[428,481)/1 crt=377'10405 lcod 0'0 unknown NOTIFY] start_peering_interval up [1,2147483647,8,2147483647,7,5] -> [1,2147483647,8,6,7,5], acting [1,2147483647,8,2147483647,7,5] -> [1,2147483647,8,6,7,5], acting_primary 1(0) -> 1, up_primary 1(0) -> 1, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.174570 7f35478a1700 1 osd.10 pg_epoch: 492 pg[13.bs4( v 377'9782 (355'1491,377'9782] local-lis/les=428/430 n=2694 ec=345/345 lis/c 428/428 les/c/f 430/430/0 492/492/478) [11,8,1,4,10,6]p11(0) r=4 lpr=492 pi=[428,492)/1 crt=377'9782 lcod 0'0 unknown NOTIFY] start_peering_interval up [11,8,1,4,2147483647,6] -> [11,8,1,4,10,6], acting [11,8,1,4,2147483647,6] -> [11,8,1,4,10,6], acting_primary 11(0) -> 11, up_primary 11(0) -> 11, role -1 -> 4, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.174644 7f35478a1700 1 osd.10 pg_epoch: 492 pg[13.bs4( v 377'9782 (355'1491,377'9782] local-lis/les=428/430 n=2694 ec=345/345 lis/c 428/428 les/c/f 430/430/0 492/492/478) [11,8,1,4,10,6]p11(0) r=4 lpr=492 pi=[428,492)/1 crt=377'9782 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:38.174679 7f35480a2700 1 osd.10 pg_epoch: 492 pg[13.3s1( v 377'10405 (371'2081,377'10405] local-lis/les=428/430 n=2736 ec=345/345 lis/c 428/428 les/c/f 430/430/0 492/492/470) [1,10,8,6,7,5]p1(0) r=1 lpr=492 pi=[428,492)/1 crt=377'10405 lcod 0'0 unknown NOTIFY] start_peering_interval up [1,2147483647,8,6,7,5] -> [1,10,8,6,7,5], acting [1,2147483647,8,6,7,5] -> [1,10,8,6,7,5], acting_primary 1(0) -> 1, up_primary 1(0) -> 1, role -1 -> 1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.174757 7f35480a2700 1 osd.10 pg_epoch: 492 pg[13.3s1( v 377'10405 (371'2081,377'10405] local-lis/les=428/430 n=2736 ec=345/345 lis/c 428/428 les/c/f 430/430/0 492/492/470) [1,10,8,6,7,5]p1(0) r=1 lpr=492 pi=[428,492)/1 crt=377'10405 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:38.174786 7f35478a1700 1 osd.10 pg_epoch: 464 pg[6.26( v 461'123787 (430'122194,461'123787] local-lis/les=422/423 n=26121 ec=76/76 lis/c 422/422 les/c/f 423/445/0 464/464/422) [7] r=-1 lpr=464 pi=[422,464)/1 crt=461'123787 lcod 0'0 unknown NOTIFY] start_peering_interval up [7,10,8] -> [7], acting [7,10,8] -> [7], acting_primary 7 -> 7, up_primary 7 -> 7, role 1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.174846 7f35478a1700 1 osd.10 pg_epoch: 467 pg[6.26( v 461'123787 (430'122194,461'123787] local-lis/les=422/423 n=26121 ec=76/76 lis/c 422/422 les/c/f 423/445/0 467/467/467) [] r=-1 lpr=467 pi=[422,467)/1 crt=461'123787 lcod 0'0 unknown NOTIFY] start_peering_interval up [7] -> [], acting [7] -> [], acting_primary 7 -> -1, up_primary 7 -> -1, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.174884 7f35478a1700 1 osd.10 pg_epoch: 468 pg[6.26( v 461'123787 (430'122194,461'123787] local-lis/les=422/423 n=26121 ec=76/76 lis/c 422/422 les/c/f 423/445/0 468/468/468) [7] r=-1 lpr=468 pi=[422,468)/1 crt=461'123787 lcod 0'0 unknown NOTIFY] start_peering_interval up [] -> [7], acting [] -> [7], acting_primary ? -> 7, up_primary ? -> 7, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.174930 7f35480a2700 1 osd.10 pg_epoch: 464 pg[11.fs3( v 383'63252 (352'59530,383'63252] local-lis/les=428/429 n=60320 ec=267/267 lis/c 428/428 les/c/f 429/429/0 464/464/464) [2147483647,6,2147483647,2147483647,7,2147483647]p6(1) r=-1 lpr=464 pi=[428,464)/1 crt=383'63252 lcod 0'0 unknown NOTIFY] start_peering_interval up [0,6,2,10,7,8] -> [2147483647,6,2147483647,2147483647,7,2147483647], acting [0,6,2,10,7,8] -> [2147483647,6,2147483647,2147483647,7,2147483647], acting_primary 0(0) -> 6, up_primary 0(0) -> 6, role 3 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.174949 7f35478a1700 1 osd.10 pg_epoch: 472 pg[6.26( v 461'123787 (430'122194,461'123787] local-lis/les=422/423 n=26121 ec=76/76 lis/c 422/422 les/c/f 423/445/0 472/472/468) [7,8] r=-1 lpr=472 pi=[422,472)/1 crt=461'123787 lcod 0'0 unknown NOTIFY] start_peering_interval up [7] -> [7,8], acting [7] -> [7,8], acting_primary 7 -> 7, up_primary 7 -> 7, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.174991 7f35480a2700 1 osd.10 pg_epoch: 465 pg[11.fs3( v 383'63252 (352'59530,383'63252] local-lis/les=428/429 n=60320 ec=267/267 lis/c 428/428 les/c/f 429/429/0 465/465/464) [2147483647,6,2,2147483647,7,2147483647]p6(1) r=-1 lpr=465 pi=[428,465)/1 crt=383'63252 lcod 0'0 unknown NOTIFY] start_peering_interval up [2147483647,6,2147483647,2147483647,7,2147483647] -> [2147483647,6,2,2147483647,7,2147483647], acting [2147483647,6,2147483647,2147483647,7,2147483647] -> [2147483647,6,2,2147483647,7,2147483647], acting_primary 6(1) -> 6, up_primary 6(1) -> 6, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.175064 7f35480a2700 1 osd.10 pg_epoch: 467 pg[11.fs3( v 383'63252 (352'59530,383'63252] local-lis/les=428/429 n=60320 ec=267/267 lis/c 428/428 les/c/f 429/429/0 467/467/464) [2147483647,6,2,2147483647,2147483647,2147483647]p6(1) r=-1 lpr=467 pi=[428,467)/1 crt=383'63252 lcod 0'0 unknown NOTIFY] start_peering_interval up [2147483647,6,2,2147483647,7,2147483647] -> [2147483647,6,2,2147483647,2147483647,2147483647], acting [2147483647,6,2,2147483647,7,2147483647] -> [2147483647,6,2,2147483647,2147483647,2147483647], acting_primary 6(1) -> 6, up_primary 6(1) -> 6, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.175111 7f35478a1700 1 osd.10 pg_epoch: 492 pg[6.26( v 461'123787 (430'122194,461'123787] local-lis/les=422/423 n=26121 ec=76/76 lis/c 422/422 les/c/f 423/445/0 492/492/468) [7,10,8] r=1 lpr=492 pi=[422,492)/2 crt=461'123787 lcod 0'0 unknown NOTIFY] start_peering_interval up [7,8] -> [7,10,8], acting [7,8] -> [7,10,8], acting_primary 7 -> 7, up_primary 7 -> 7, role -1 -> 1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.175116 7f35480a2700 1 osd.10 pg_epoch: 468 pg[11.fs3( v 383'63252 (352'59530,383'63252] local-lis/les=428/429 n=60320 ec=267/267 lis/c 428/428 les/c/f 429/429/0 468/468/464) [2147483647,6,2,2147483647,7,2147483647]p6(1) r=-1 lpr=468 pi=[428,468)/1 crt=383'63252 lcod 0'0 unknown NOTIFY] start_peering_interval up [2147483647,6,2,2147483647,2147483647,2147483647] -> [2147483647,6,2,2147483647,7,2147483647], acting [2147483647,6,2,2147483647,2147483647,2147483647] -> [2147483647,6,2,2147483647,7,2147483647], acting_primary 6(1) -> 6, up_primary 6(1) -> 6, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.175180 7f35478a1700 1 osd.10 pg_epoch: 492 pg[6.26( v 461'123787 (430'122194,461'123787] local-lis/les=422/423 n=26121 ec=76/76 lis/c 422/422 les/c/f 423/445/0 492/492/468) [7,10,8] r=1 lpr=492 pi=[422,492)/2 crt=461'123787 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:38.175178 7f35480a2700 1 osd.10 pg_epoch: 470 pg[11.fs3( v 383'63252 (352'59530,383'63252] local-lis/les=428/429 n=60320 ec=267/267 lis/c 428/428 les/c/f 429/429/0 470/470/470) [0,6,2,2147483647,7,2147483647]p0(0) r=-1 lpr=470 pi=[428,470)/1 crt=383'63252 lcod 0'0 unknown NOTIFY] start_peering_interval up [2147483647,6,2,2147483647,7,2147483647] -> [0,6,2,2147483647,7,2147483647], acting [2147483647,6,2,2147483647,7,2147483647] -> [0,6,2,2147483647,7,2147483647], acting_primary 6(1) -> 0, up_primary 6(1) -> 0, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.175239 7f35480a2700 1 osd.10 pg_epoch: 472 pg[11.fs3( v 383'63252 (352'59530,383'63252] local-lis/les=428/429 n=60320 ec=267/267 lis/c 428/428 les/c/f 429/429/0 472/472/470) [0,6,2,2147483647,7,8]p0(0) r=-1 lpr=472 pi=[428,472)/1 crt=383'63252 lcod 0'0 unknown NOTIFY] start_peering_interval up [0,6,2,2147483647,7,2147483647] -> [0,6,2,2147483647,7,8], acting [0,6,2,2147483647,7,2147483647] -> [0,6,2,2147483647,7,8], acting_primary 0(0) -> 0, up_primary 0(0) -> 0, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.175334 7f35478a1700 1 osd.10 pg_epoch: 464 pg[13.9s1( v 377'10260 (352'219,377'10260] local-lis/les=428/429 n=2727 ec=345/345 lis/c 428/428 les/c/f 429/429/0 464/464/464) [2147483647,2147483647,2147483647,6,2147483647,11]p6(3) r=-1 lpr=464 pi=[428,464)/1 crt=377'10254 lcod 0'0 unknown NOTIFY] start_peering_interval up [5,10,3,6,8,11] -> [2147483647,2147483647,2147483647,6,2147483647,11], acting [5,10,3,6,8,11] -> [2147483647,2147483647,2147483647,6,2147483647,11], acting_primary 5(0) -> 6, up_primary 5(0) -> 6, role 1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.175402 7f35480a2700 1 osd.10 pg_epoch: 480 pg[11.fs3( v 383'63252 (352'59530,383'63252] local-lis/les=428/429 n=60320 ec=267/267 lis/c 428/428 les/c/f 429/429/0 480/480/470) [0,2147483647,2,2147483647,7,8]p0(0) r=-1 lpr=480 pi=[428,480)/1 crt=383'63252 lcod 0'0 unknown NOTIFY] start_peering_interval up [0,6,2,2147483647,7,8] -> [0,2147483647,2,2147483647,7,8], acting [0,6,2,2147483647,7,8] -> [0,2147483647,2,2147483647,7,8], acting_primary 0(0) -> 0, up_primary 0(0) -> 0, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.175410 7f35478a1700 1 osd.10 pg_epoch: 467 pg[13.9s1( v 377'10260 (352'219,377'10260] local-lis/les=428/429 n=2727 ec=345/345 lis/c 428/428 les/c/f 429/429/0 467/467/467) [5,2147483647,2147483647,6,2147483647,11]p5(0) r=-1 lpr=467 pi=[428,467)/1 crt=377'10254 lcod 0'0 unknown NOTIFY] start_peering_interval up [2147483647,2147483647,2147483647,6,2147483647,11] -> [5,2147483647,2147483647,6,2147483647,11], acting [2147483647,2147483647,2147483647,6,2147483647,11] -> [5,2147483647,2147483647,6,2147483647,11], acting_primary 6(3) -> 5, up_primary 6(3) -> 5, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.175458 7f35480a2700 1 osd.10 pg_epoch: 481 pg[11.fs3( v 383'63252 (352'59530,383'63252] local-lis/les=428/429 n=60320 ec=267/267 lis/c 428/428 les/c/f 429/429/0 481/481/470) [0,6,2,2147483647,7,8]p0(0) r=-1 lpr=481 pi=[428,481)/1 crt=383'63252 lcod 0'0 unknown NOTIFY] start_peering_interval up [0,2147483647,2,2147483647,7,8] -> [0,6,2,2147483647,7,8], acting [0,2147483647,2,2147483647,7,8] -> [0,6,2,2147483647,7,8], acting_primary 0(0) -> 0, up_primary 0(0) -> 0, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.175496 7f35478a1700 1 osd.10 pg_epoch: 472 pg[13.9s1( v 377'10260 (352'219,377'10260] local-lis/les=428/429 n=2727 ec=345/345 lis/c 428/428 les/c/f 429/429/0 472/472/467) [5,2147483647,2147483647,6,8,11]p5(0) r=-1 lpr=472 pi=[428,472)/1 crt=377'10254 lcod 0'0 unknown NOTIFY] start_peering_interval up [5,2147483647,2147483647,6,2147483647,11] -> [5,2147483647,2147483647,6,8,11], acting [5,2147483647,2147483647,6,2147483647,11] -> [5,2147483647,2147483647,6,8,11], acting_primary 5(0) -> 5, up_primary 5(0) -> 5, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.175632 7f35478a1700 1 osd.10 pg_epoch: 477 pg[13.9s1( v 377'10260 (352'219,377'10260] local-lis/les=428/429 n=2727 ec=345/345 lis/c 428/428 les/c/f 429/429/0 477/477/467) [5,2147483647,2147483647,6,8,2147483647]p5(0) r=-1 lpr=477 pi=[428,477)/1 crt=377'10254 lcod 0'0 unknown NOTIFY] start_peering_interval up [5,2147483647,2147483647,6,8,11] -> [5,2147483647,2147483647,6,8,2147483647], acting [5,2147483647,2147483647,6,8,11] -> [5,2147483647,2147483647,6,8,2147483647], acting_primary 5(0) -> 5, up_primary 5(0) -> 5, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.175647 7f35480a2700 1 osd.10 pg_epoch: 492 pg[11.fs3( v 383'63252 (352'59530,383'63252] local-lis/les=428/429 n=60320 ec=267/267 lis/c 428/428 les/c/f 429/429/0 492/492/470) [0,6,2,10,7,8]p0(0) r=3 lpr=492 pi=[428,492)/1 crt=383'63252 lcod 0'0 unknown NOTIFY] start_peering_interval up [0,6,2,2147483647,7,8] -> [0,6,2,10,7,8], acting [0,6,2,2147483647,7,8] -> [0,6,2,10,7,8], acting_primary 0(0) -> 0, up_primary 0(0) -> 0, role -1 -> 3, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.175720 7f35480a2700 1 osd.10 pg_epoch: 492 pg[11.fs3( v 383'63252 (352'59530,383'63252] local-lis/les=428/429 n=60320 ec=267/267 lis/c 428/428 les/c/f 429/429/0 492/492/470) [0,6,2,10,7,8]p0(0) r=3 lpr=492 pi=[428,492)/1 crt=383'63252 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:38.175729 7f35478a1700 1 osd.10 pg_epoch: 478 pg[13.9s1( v 377'10260 (352'219,377'10260] local-lis/les=428/429 n=2727 ec=345/345 lis/c 428/428 les/c/f 429/429/0 478/478/467) [5,2147483647,2147483647,6,8,11]p5(0) r=-1 lpr=478 pi=[428,478)/1 crt=377'10254 lcod 0'0 unknown NOTIFY] start_peering_interval up [5,2147483647,2147483647,6,8,2147483647] -> [5,2147483647,2147483647,6,8,11], acting [5,2147483647,2147483647,6,8,2147483647] -> [5,2147483647,2147483647,6,8,11], acting_primary 5(0) -> 5, up_primary 5(0) -> 5, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.175792 7f35478a1700 1 osd.10 pg_epoch: 480 pg[13.9s1( v 377'10260 (352'219,377'10260] local-lis/les=428/429 n=2727 ec=345/345 lis/c 428/428 les/c/f 429/429/0 480/480/467) [5,2147483647,2147483647,2147483647,8,11]p5(0) r=-1 lpr=480 pi=[428,480)/1 crt=377'10254 lcod 0'0 unknown NOTIFY] start_peering_interval up [5,2147483647,2147483647,6,8,11] -> [5,2147483647,2147483647,2147483647,8,11], acting [5,2147483647,2147483647,6,8,11] -> [5,2147483647,2147483647,2147483647,8,11], acting_primary 5(0) -> 5, up_primary 5(0) -> 5, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.175821 7f35480a2700 1 osd.10 pg_epoch: 464 pg[11.5s0( v 383'60461 (352'56750,383'60461] local-lis/les=428/429 n=60461 ec=267/267 lis/c 428/428 les/c/f 429/429/0 428/428/403) [10,5,8,6,1,7]p10(0) r=0 lpr=429 crt=383'60461 lcod 0'0 mlcod 0'0 peering] state<Started/Primary/Peering>: Peering, affected_by_map, going to Reset >2018-07-11 05:55:38.175842 7f35478a1700 1 osd.10 pg_epoch: 481 pg[13.9s1( v 377'10260 (352'219,377'10260] local-lis/les=428/429 n=2727 ec=345/345 lis/c 428/428 les/c/f 429/429/0 481/481/467) [5,2147483647,2147483647,6,8,11]p5(0) r=-1 lpr=481 pi=[428,481)/1 crt=377'10254 lcod 0'0 unknown NOTIFY] start_peering_interval up [5,2147483647,2147483647,2147483647,8,11] -> [5,2147483647,2147483647,6,8,11], acting [5,2147483647,2147483647,2147483647,8,11] -> [5,2147483647,2147483647,6,8,11], acting_primary 5(0) -> 5, up_primary 5(0) -> 5, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.175936 7f35480a2700 1 osd.10 pg_epoch: 464 pg[11.5s0( v 383'60461 (352'56750,383'60461] local-lis/les=428/429 n=60461 ec=267/267 lis/c 428/428 les/c/f 429/429/0 464/464/464) [2147483647,2147483647,2147483647,6,2147483647,7]p6(3) r=-1 lpr=464 pi=[428,464)/1 crt=383'60461 lcod 0'0 unknown] start_peering_interval up [10,5,8,6,1,7] -> [2147483647,2147483647,2147483647,6,2147483647,7], acting [10,5,8,6,1,7] -> [2147483647,2147483647,2147483647,6,2147483647,7], acting_primary 10(0) -> 6, up_primary 10(0) -> 6, role 0 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.175943 7f35478a1700 1 osd.10 pg_epoch: 484 pg[13.9s1( v 377'10260 (352'219,377'10260] local-lis/les=428/429 n=2727 ec=345/345 lis/c 428/428 les/c/f 429/429/0 484/484/467) [5,2147483647,3,6,8,11]p5(0) r=-1 lpr=484 pi=[428,484)/1 crt=377'10254 lcod 0'0 unknown NOTIFY] start_peering_interval up [5,2147483647,2147483647,6,8,11] -> [5,2147483647,3,6,8,11], acting [5,2147483647,2147483647,6,8,11] -> [5,2147483647,3,6,8,11], acting_primary 5(0) -> 5, up_primary 5(0) -> 5, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.176041 7f35480a2700 1 osd.10 pg_epoch: 467 pg[11.5s0( v 383'60461 (352'56750,383'60461] local-lis/les=428/429 n=60461 ec=267/267 lis/c 428/428 les/c/f 429/429/0 467/467/467) [2147483647,5,2147483647,6,2147483647,2147483647]p5(1) r=-1 lpr=467 pi=[428,467)/1 crt=383'60461 lcod 0'0 unknown NOTIFY] start_peering_interval up [2147483647,2147483647,2147483647,6,2147483647,7] -> [2147483647,5,2147483647,6,2147483647,2147483647], acting [2147483647,2147483647,2147483647,6,2147483647,7] -> [2147483647,5,2147483647,6,2147483647,2147483647], acting_primary 6(3) -> 5, up_primary 6(3) -> 5, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.176101 7f35478a1700 1 osd.10 pg_epoch: 492 pg[13.9s1( v 377'10260 (352'219,377'10260] local-lis/les=428/429 n=2727 ec=345/345 lis/c 428/428 les/c/f 429/429/0 492/492/467) [5,10,3,6,8,11]p5(0) r=1 lpr=492 pi=[428,492)/1 crt=377'10254 lcod 0'0 unknown NOTIFY] start_peering_interval up [5,2147483647,3,6,8,11] -> [5,10,3,6,8,11], acting [5,2147483647,3,6,8,11] -> [5,10,3,6,8,11], acting_primary 5(0) -> 5, up_primary 5(0) -> 5, role -1 -> 1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.176111 7f35480a2700 1 osd.10 pg_epoch: 468 pg[11.5s0( v 383'60461 (352'56750,383'60461] local-lis/les=428/429 n=60461 ec=267/267 lis/c 428/428 les/c/f 429/429/0 468/468/467) [2147483647,5,2147483647,6,2147483647,7]p5(1) r=-1 lpr=468 pi=[428,468)/1 crt=383'60461 lcod 0'0 unknown NOTIFY] start_peering_interval up [2147483647,5,2147483647,6,2147483647,2147483647] -> [2147483647,5,2147483647,6,2147483647,7], acting [2147483647,5,2147483647,6,2147483647,2147483647] -> [2147483647,5,2147483647,6,2147483647,7], acting_primary 5(1) -> 5, up_primary 5(1) -> 5, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.176179 7f35478a1700 1 osd.10 pg_epoch: 492 pg[13.9s1( v 377'10260 (352'219,377'10260] local-lis/les=428/429 n=2727 ec=345/345 lis/c 428/428 les/c/f 429/429/0 492/492/467) [5,10,3,6,8,11]p5(0) r=1 lpr=492 pi=[428,492)/1 crt=377'10254 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:38.176179 7f35480a2700 1 osd.10 pg_epoch: 470 pg[11.5s0( v 383'60461 (352'56750,383'60461] local-lis/les=428/429 n=60461 ec=267/267 lis/c 428/428 les/c/f 429/429/0 470/470/467) [2147483647,5,2147483647,6,1,7]p5(1) r=-1 lpr=470 pi=[428,470)/1 crt=383'60461 lcod 0'0 unknown NOTIFY] start_peering_interval up [2147483647,5,2147483647,6,2147483647,7] -> [2147483647,5,2147483647,6,1,7], acting [2147483647,5,2147483647,6,2147483647,7] -> [2147483647,5,2147483647,6,1,7], acting_primary 5(1) -> 5, up_primary 5(1) -> 5, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.176251 7f35480a2700 1 osd.10 pg_epoch: 472 pg[11.5s0( v 383'60461 (352'56750,383'60461] local-lis/les=428/429 n=60461 ec=267/267 lis/c 428/428 les/c/f 429/429/0 472/472/467) [2147483647,5,8,6,1,7]p5(1) r=-1 lpr=472 pi=[428,472)/1 crt=383'60461 lcod 0'0 unknown NOTIFY] start_peering_interval up [2147483647,5,2147483647,6,1,7] -> [2147483647,5,8,6,1,7], acting [2147483647,5,2147483647,6,1,7] -> [2147483647,5,8,6,1,7], acting_primary 5(1) -> 5, up_primary 5(1) -> 5, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.176345 7f35478a1700 1 osd.10 pg_epoch: 464 pg[11.ds3( v 383'59865 (367'56373,383'59865] local-lis/les=428/429 n=59865 ec=267/267 lis/c 428/428 les/c/f 429/429/0 464/464/464) [2147483647,2147483647,11,2147483647,2147483647,6]p11(2) r=-1 lpr=464 pi=[428,464)/1 crt=383'59865 lcod 0'0 unknown NOTIFY] start_peering_interval up [2,4,11,10,8,6] -> [2147483647,2147483647,11,2147483647,2147483647,6], acting [2,4,11,10,8,6] -> [2147483647,2147483647,11,2147483647,2147483647,6], acting_primary 2(0) -> 11, up_primary 2(0) -> 11, role 3 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.176411 7f35478a1700 1 osd.10 pg_epoch: 465 pg[11.ds3( v 383'59865 (367'56373,383'59865] local-lis/les=428/429 n=59865 ec=267/267 lis/c 428/428 les/c/f 429/429/0 465/465/465) [2,2147483647,11,2147483647,2147483647,6]p2(0) r=-1 lpr=465 pi=[428,465)/1 crt=383'59865 lcod 0'0 unknown NOTIFY] start_peering_interval up [2147483647,2147483647,11,2147483647,2147483647,6] -> [2,2147483647,11,2147483647,2147483647,6], acting [2147483647,2147483647,11,2147483647,2147483647,6] -> [2,2147483647,11,2147483647,2147483647,6], acting_primary 11(2) -> 2, up_primary 11(2) -> 2, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.176422 7f35480a2700 1 osd.10 pg_epoch: 480 pg[11.5s0( v 383'60461 (352'56750,383'60461] local-lis/les=428/429 n=60461 ec=267/267 lis/c 428/428 les/c/f 429/429/0 480/480/467) [2147483647,5,8,2147483647,1,7]p5(1) r=-1 lpr=480 pi=[428,480)/1 crt=383'60461 lcod 0'0 unknown NOTIFY] start_peering_interval up [2147483647,5,8,6,1,7] -> [2147483647,5,8,2147483647,1,7], acting [2147483647,5,8,6,1,7] -> [2147483647,5,8,2147483647,1,7], acting_primary 5(1) -> 5, up_primary 5(1) -> 5, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.176474 7f35480a2700 1 osd.10 pg_epoch: 481 pg[11.5s0( v 383'60461 (352'56750,383'60461] local-lis/les=428/429 n=60461 ec=267/267 lis/c 428/428 les/c/f 429/429/0 481/481/467) [2147483647,5,8,6,1,7]p5(1) r=-1 lpr=481 pi=[428,481)/1 crt=383'60461 lcod 0'0 unknown NOTIFY] start_peering_interval up [2147483647,5,8,2147483647,1,7] -> [2147483647,5,8,6,1,7], acting [2147483647,5,8,2147483647,1,7] -> [2147483647,5,8,6,1,7], acting_primary 5(1) -> 5, up_primary 5(1) -> 5, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.176579 7f35478a1700 1 osd.10 pg_epoch: 472 pg[11.ds3( v 383'59865 (367'56373,383'59865] local-lis/les=428/429 n=59865 ec=267/267 lis/c 428/428 les/c/f 429/429/0 472/472/465) [2,2147483647,11,2147483647,8,6]p2(0) r=-1 lpr=472 pi=[428,472)/1 crt=383'59865 lcod 0'0 unknown NOTIFY] start_peering_interval up [2,2147483647,11,2147483647,2147483647,6] -> [2,2147483647,11,2147483647,8,6], acting [2,2147483647,11,2147483647,2147483647,6] -> [2,2147483647,11,2147483647,8,6], acting_primary 2(0) -> 2, up_primary 2(0) -> 2, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.176686 7f35480a2700 1 osd.10 pg_epoch: 492 pg[11.5s0( v 383'60461 (352'56750,383'60461] local-lis/les=428/429 n=60461 ec=267/267 lis/c 428/428 les/c/f 429/429/0 492/492/492) [10,5,8,6,1,7]p10(0) r=0 lpr=492 pi=[428,492)/1 crt=383'60461 lcod 0'0 mlcod 0'0 unknown NOTIFY] start_peering_interval up [2147483647,5,8,6,1,7] -> [10,5,8,6,1,7], acting [2147483647,5,8,6,1,7] -> [10,5,8,6,1,7], acting_primary 5(1) -> 10, up_primary 5(1) -> 10, role -1 -> 0, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.176711 7f35478a1700 1 osd.10 pg_epoch: 477 pg[11.ds3( v 383'59865 (367'56373,383'59865] local-lis/les=428/429 n=59865 ec=267/267 lis/c 428/428 les/c/f 429/429/0 477/477/465) [2,2147483647,2147483647,2147483647,8,6]p2(0) r=-1 lpr=477 pi=[428,477)/1 crt=383'59865 lcod 0'0 unknown NOTIFY] start_peering_interval up [2,2147483647,11,2147483647,8,6] -> [2,2147483647,2147483647,2147483647,8,6], acting [2,2147483647,11,2147483647,8,6] -> [2,2147483647,2147483647,2147483647,8,6], acting_primary 2(0) -> 2, up_primary 2(0) -> 2, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.176759 7f35480a2700 1 osd.10 pg_epoch: 492 pg[11.5s0( v 383'60461 (352'56750,383'60461] local-lis/les=428/429 n=60461 ec=267/267 lis/c 428/428 les/c/f 429/429/0 492/492/492) [10,5,8,6,1,7]p10(0) r=0 lpr=492 pi=[428,492)/1 crt=383'60461 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 05:55:38.176769 7f35478a1700 1 osd.10 pg_epoch: 478 pg[11.ds3( v 383'59865 (367'56373,383'59865] local-lis/les=428/429 n=59865 ec=267/267 lis/c 428/428 les/c/f 429/429/0 478/478/465) [2,2147483647,11,2147483647,8,6]p2(0) r=-1 lpr=478 pi=[428,478)/1 crt=383'59865 lcod 0'0 unknown NOTIFY] start_peering_interval up [2,2147483647,2147483647,2147483647,8,6] -> [2,2147483647,11,2147483647,8,6], acting [2,2147483647,2147483647,2147483647,8,6] -> [2,2147483647,11,2147483647,8,6], acting_primary 2(0) -> 2, up_primary 2(0) -> 2, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.176843 7f35478a1700 1 osd.10 pg_epoch: 480 pg[11.ds3( v 383'59865 (367'56373,383'59865] local-lis/les=428/429 n=59865 ec=267/267 lis/c 428/428 les/c/f 429/429/0 480/480/465) [2,2147483647,11,2147483647,8,2147483647]p2(0) r=-1 lpr=480 pi=[428,480)/1 crt=383'59865 lcod 0'0 unknown NOTIFY] start_peering_interval up [2,2147483647,11,2147483647,8,6] -> [2,2147483647,11,2147483647,8,2147483647], acting [2,2147483647,11,2147483647,8,6] -> [2,2147483647,11,2147483647,8,2147483647], acting_primary 2(0) -> 2, up_primary 2(0) -> 2, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.176901 7f35478a1700 1 osd.10 pg_epoch: 481 pg[11.ds3( v 383'59865 (367'56373,383'59865] local-lis/les=428/429 n=59865 ec=267/267 lis/c 428/428 les/c/f 429/429/0 481/481/465) [2,2147483647,11,2147483647,8,6]p2(0) r=-1 lpr=481 pi=[428,481)/1 crt=383'59865 lcod 0'0 unknown NOTIFY] start_peering_interval up [2,2147483647,11,2147483647,8,2147483647] -> [2,2147483647,11,2147483647,8,6], acting [2,2147483647,11,2147483647,8,2147483647] -> [2,2147483647,11,2147483647,8,6], acting_primary 2(0) -> 2, up_primary 2(0) -> 2, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.176953 7f35480a2700 1 osd.10 pg_epoch: 464 pg[11.2s2( v 383'59971 (377'58405,383'59971] local-lis/les=428/429 n=59971 ec=267/267 lis/c 428/428 les/c/f 429/429/0 464/464/464) [2147483647,6,2147483647,2147483647,7,2147483647]p6(1) r=-1 lpr=464 pi=[428,464)/1 crt=383'59970 lcod 0'0 unknown NOTIFY] start_peering_interval up [2,6,10,0,7,8] -> [2147483647,6,2147483647,2147483647,7,2147483647], acting [2,6,10,0,7,8] -> [2147483647,6,2147483647,2147483647,7,2147483647], acting_primary 2(0) -> 6, up_primary 2(0) -> 6, role 2 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.177017 7f35480a2700 1 osd.10 pg_epoch: 465 pg[11.2s2( v 383'59971 (377'58405,383'59971] local-lis/les=428/429 n=59971 ec=267/267 lis/c 428/428 les/c/f 429/429/0 465/465/465) [2,6,2147483647,2147483647,7,2147483647]p2(0) r=-1 lpr=465 pi=[428,465)/1 crt=383'59970 lcod 0'0 unknown NOTIFY] start_peering_interval up [2147483647,6,2147483647,2147483647,7,2147483647] -> [2,6,2147483647,2147483647,7,2147483647], acting [2147483647,6,2147483647,2147483647,7,2147483647] -> [2,6,2147483647,2147483647,7,2147483647], acting_primary 6(1) -> 2, up_primary 6(1) -> 2, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.177049 7f35478a1700 1 osd.10 pg_epoch: 487 pg[11.ds3( v 383'59865 (367'56373,383'59865] local-lis/les=428/429 n=59865 ec=267/267 lis/c 428/428 les/c/f 429/429/0 487/487/465) [2,4,11,2147483647,8,6]p2(0) r=-1 lpr=487 pi=[428,487)/1 crt=383'59865 lcod 0'0 unknown NOTIFY] start_peering_interval up [2,2147483647,11,2147483647,8,6] -> [2,4,11,2147483647,8,6], acting [2,2147483647,11,2147483647,8,6] -> [2,4,11,2147483647,8,6], acting_primary 2(0) -> 2, up_primary 2(0) -> 2, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.177087 7f35480a2700 1 osd.10 pg_epoch: 467 pg[11.2s2( v 383'59971 (377'58405,383'59971] local-lis/les=428/429 n=59971 ec=267/267 lis/c 428/428 les/c/f 429/429/0 467/467/465) [2,6,2147483647,2147483647,2147483647,2147483647]p2(0) r=-1 lpr=467 pi=[428,467)/1 crt=383'59970 lcod 0'0 unknown NOTIFY] start_peering_interval up [2,6,2147483647,2147483647,7,2147483647] -> [2,6,2147483647,2147483647,2147483647,2147483647], acting [2,6,2147483647,2147483647,7,2147483647] -> [2,6,2147483647,2147483647,2147483647,2147483647], acting_primary 2(0) -> 2, up_primary 2(0) -> 2, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.177143 7f35480a2700 1 osd.10 pg_epoch: 468 pg[11.2s2( v 383'59971 (377'58405,383'59971] local-lis/les=428/429 n=59971 ec=267/267 lis/c 428/428 les/c/f 429/429/0 468/468/465) [2,6,2147483647,2147483647,7,2147483647]p2(0) r=-1 lpr=468 pi=[428,468)/1 crt=383'59970 lcod 0'0 unknown NOTIFY] start_peering_interval up [2,6,2147483647,2147483647,2147483647,2147483647] -> [2,6,2147483647,2147483647,7,2147483647], acting [2,6,2147483647,2147483647,2147483647,2147483647] -> [2,6,2147483647,2147483647,7,2147483647], acting_primary 2(0) -> 2, up_primary 2(0) -> 2, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.177186 7f35478a1700 1 osd.10 pg_epoch: 492 pg[11.ds3( v 383'59865 (367'56373,383'59865] local-lis/les=428/429 n=59865 ec=267/267 lis/c 428/428 les/c/f 429/429/0 492/492/465) [2,4,11,10,8,6]p2(0) r=3 lpr=492 pi=[428,492)/1 crt=383'59865 lcod 0'0 unknown NOTIFY] start_peering_interval up [2,4,11,2147483647,8,6] -> [2,4,11,10,8,6], acting [2,4,11,2147483647,8,6] -> [2,4,11,10,8,6], acting_primary 2(0) -> 2, up_primary 2(0) -> 2, role -1 -> 3, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.177211 7f35480a2700 1 osd.10 pg_epoch: 470 pg[11.2s2( v 383'59971 (377'58405,383'59971] local-lis/les=428/429 n=59971 ec=267/267 lis/c 428/428 les/c/f 429/429/0 470/470/465) [2,6,2147483647,0,7,2147483647]p2(0) r=-1 lpr=470 pi=[428,470)/1 crt=383'59970 lcod 0'0 unknown NOTIFY] start_peering_interval up [2,6,2147483647,2147483647,7,2147483647] -> [2,6,2147483647,0,7,2147483647], acting [2,6,2147483647,2147483647,7,2147483647] -> [2,6,2147483647,0,7,2147483647], acting_primary 2(0) -> 2, up_primary 2(0) -> 2, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.177264 7f35478a1700 1 osd.10 pg_epoch: 492 pg[11.ds3( v 383'59865 (367'56373,383'59865] local-lis/les=428/429 n=59865 ec=267/267 lis/c 428/428 les/c/f 429/429/0 492/492/465) [2,4,11,10,8,6]p2(0) r=3 lpr=492 pi=[428,492)/1 crt=383'59865 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:38.177290 7f35480a2700 1 osd.10 pg_epoch: 472 pg[11.2s2( v 383'59971 (377'58405,383'59971] local-lis/les=428/429 n=59971 ec=267/267 lis/c 428/428 les/c/f 429/429/0 472/472/465) [2,6,2147483647,0,7,8]p2(0) r=-1 lpr=472 pi=[428,472)/1 crt=383'59970 lcod 0'0 unknown NOTIFY] start_peering_interval up [2,6,2147483647,0,7,2147483647] -> [2,6,2147483647,0,7,8], acting [2,6,2147483647,0,7,2147483647] -> [2,6,2147483647,0,7,8], acting_primary 2(0) -> 2, up_primary 2(0) -> 2, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.177401 7f35478a1700 1 osd.10 pg_epoch: 464 pg[16.d( empty local-lis/les=442/444 n=0 ec=442/442 lis/c 442/442 les/c/f 444/444/0 464/464/464) [] r=-1 lpr=464 pi=[442,464)/1 crt=0'0 unknown NOTIFY] start_peering_interval up [0,3,10] -> [], acting [0,3,10] -> [], acting_primary 0 -> -1, up_primary 0 -> -1, role 2 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.177446 7f35480a2700 1 osd.10 pg_epoch: 480 pg[11.2s2( v 383'59971 (377'58405,383'59971] local-lis/les=428/429 n=59971 ec=267/267 lis/c 428/428 les/c/f 429/429/0 480/480/465) [2,2147483647,2147483647,0,7,8]p2(0) r=-1 lpr=480 pi=[428,480)/1 crt=383'59970 lcod 0'0 unknown NOTIFY] start_peering_interval up [2,6,2147483647,0,7,8] -> [2,2147483647,2147483647,0,7,8], acting [2,6,2147483647,0,7,8] -> [2,2147483647,2147483647,0,7,8], acting_primary 2(0) -> 2, up_primary 2(0) -> 2, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.177483 7f35478a1700 1 osd.10 pg_epoch: 470 pg[16.d( empty local-lis/les=442/444 n=0 ec=442/442 lis/c 442/442 les/c/f 444/444/0 470/470/470) [0] r=-1 lpr=470 pi=[442,470)/1 crt=0'0 unknown NOTIFY] start_peering_interval up [] -> [0], acting [] -> [0], acting_primary ? -> 0, up_primary ? -> 0, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.177503 7f35480a2700 1 osd.10 pg_epoch: 481 pg[11.2s2( v 383'59971 (377'58405,383'59971] local-lis/les=428/429 n=59971 ec=267/267 lis/c 428/428 les/c/f 429/429/0 481/481/465) [2,6,2147483647,0,7,8]p2(0) r=-1 lpr=481 pi=[428,481)/1 crt=383'59970 lcod 0'0 unknown NOTIFY] start_peering_interval up [2,2147483647,2147483647,0,7,8] -> [2,6,2147483647,0,7,8], acting [2,2147483647,2147483647,0,7,8] -> [2,6,2147483647,0,7,8], acting_primary 2(0) -> 2, up_primary 2(0) -> 2, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.177608 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 6] Generated table #36304: 2638241 keys, 74546334 bytes >2018-07-11 05:55:38.177620 7f35478a1700 1 osd.10 pg_epoch: 484 pg[16.d( empty local-lis/les=442/444 n=0 ec=442/442 lis/c 442/442 les/c/f 444/444/0 484/484/470) [0,3] r=-1 lpr=484 pi=[442,484)/1 crt=0'0 unknown NOTIFY] start_peering_interval up [0] -> [0,3], acting [0] -> [0,3], acting_primary 0 -> 0, up_primary 0 -> 0, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.177640 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288538177625, "cf_name": "default", "job": 6, "event": "table_file_creation", "file_number": 36304, "file_size": 74546334, "table_properties": {"data_size": 67112181, "index_size": 689980, "filter_size": 6743183, "raw_key_size": 76215860, "raw_average_key_size": 28, "raw_value_size": 31365763, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:55:38.177706 7f35480a2700 1 osd.10 pg_epoch: 492 pg[11.2s2( v 383'59971 (377'58405,383'59971] local-lis/les=428/429 n=59971 ec=267/267 lis/c 428/428 les/c/f 429/429/0 492/492/465) [2,6,10,0,7,8]p2(0) r=2 lpr=492 pi=[428,492)/1 crt=383'59970 lcod 0'0 unknown NOTIFY] start_peering_interval up [2,6,2147483647,0,7,8] -> [2,6,10,0,7,8], acting [2,6,2147483647,0,7,8] -> [2,6,10,0,7,8], acting_primary 2(0) -> 2, up_primary 2(0) -> 2, role -1 -> 2, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.177720 7f35478a1700 1 osd.10 pg_epoch: 492 pg[16.d( empty local-lis/les=442/444 n=0 ec=442/442 lis/c 442/442 les/c/f 444/444/0 492/492/470) [0,3,10] r=2 lpr=492 pi=[442,492)/1 crt=0'0 unknown NOTIFY] start_peering_interval up [0,3] -> [0,3,10], acting [0,3] -> [0,3,10], acting_primary 0 -> 0, up_primary 0 -> 0, role -1 -> 2, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.177782 7f35478a1700 1 osd.10 pg_epoch: 492 pg[16.d( empty local-lis/les=442/444 n=0 ec=442/442 lis/c 442/442 les/c/f 444/444/0 492/492/470) [0,3,10] r=2 lpr=492 pi=[442,492)/1 crt=0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:38.177781 7f35480a2700 1 osd.10 pg_epoch: 492 pg[11.2s2( v 383'59971 (377'58405,383'59971] local-lis/les=428/429 n=59971 ec=267/267 lis/c 428/428 les/c/f 429/429/0 492/492/465) [2,6,10,0,7,8]p2(0) r=2 lpr=492 pi=[428,492)/1 crt=383'59970 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:38.177880 7f35480a2700 1 osd.10 pg_epoch: 464 pg[9.1e( v 463'751 (0'0,463'751] local-lis/les=414/415 n=269 ec=190/190 lis/c 414/414 les/c/f 415/445/0 464/464/464) [] r=-1 lpr=464 pi=[414,464)/1 crt=463'751 lcod 0'0 unknown NOTIFY] start_peering_interval up [2,4,10] -> [], acting [2,4,10] -> [], acting_primary 2 -> -1, up_primary 2 -> -1, role 2 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.177928 7f35480a2700 1 osd.10 pg_epoch: 465 pg[9.1e( v 463'751 (0'0,463'751] local-lis/les=414/415 n=269 ec=190/190 lis/c 414/414 les/c/f 415/445/0 465/465/465) [2] r=-1 lpr=465 pi=[414,465)/1 crt=463'751 lcod 0'0 unknown NOTIFY] start_peering_interval up [] -> [2], acting [] -> [2], acting_primary ? -> 2, up_primary ? -> 2, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.177931 7f35478a1700 1 osd.10 pg_epoch: 464 pg[13.fs4( v 377'8989 (0'0,377'8989] local-lis/les=428/429 n=2712 ec=345/345 lis/c 428/428 les/c/f 429/429/0 464/464/464) [2147483647,7,2147483647,6,2147483647,2147483647]p7(1) r=-1 lpr=464 pi=[428,464)/1 crt=377'8979 lcod 0'0 unknown NOTIFY] start_peering_interval up [5,7,3,6,10,8] -> [2147483647,7,2147483647,6,2147483647,2147483647], acting [5,7,3,6,10,8] -> [2147483647,7,2147483647,6,2147483647,2147483647], acting_primary 5(0) -> 7, up_primary 5(0) -> 7, role 4 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.178037 7f35478a1700 1 osd.10 pg_epoch: 467 pg[13.fs4( v 377'8989 (0'0,377'8989] local-lis/les=428/429 n=2712 ec=345/345 lis/c 428/428 les/c/f 429/429/0 467/467/467) [5,2147483647,2147483647,6,2147483647,2147483647]p5(0) r=-1 lpr=467 pi=[428,467)/1 crt=377'8979 lcod 0'0 unknown NOTIFY] start_peering_interval up [2147483647,7,2147483647,6,2147483647,2147483647] -> [5,2147483647,2147483647,6,2147483647,2147483647], acting [2147483647,7,2147483647,6,2147483647,2147483647] -> [5,2147483647,2147483647,6,2147483647,2147483647], acting_primary 7(1) -> 5, up_primary 7(1) -> 5, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.178097 7f35478a1700 1 osd.10 pg_epoch: 468 pg[13.fs4( v 377'8989 (0'0,377'8989] local-lis/les=428/429 n=2712 ec=345/345 lis/c 428/428 les/c/f 429/429/0 468/468/467) [5,7,2147483647,6,2147483647,2147483647]p5(0) r=-1 lpr=468 pi=[428,468)/1 crt=377'8979 lcod 0'0 unknown NOTIFY] start_peering_interval up [5,2147483647,2147483647,6,2147483647,2147483647] -> [5,7,2147483647,6,2147483647,2147483647], acting [5,2147483647,2147483647,6,2147483647,2147483647] -> [5,7,2147483647,6,2147483647,2147483647], acting_primary 5(0) -> 5, up_primary 5(0) -> 5, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.178109 7f35480a2700 1 osd.10 pg_epoch: 487 pg[9.1e( v 463'751 (0'0,463'751] local-lis/les=414/415 n=269 ec=190/190 lis/c 414/414 les/c/f 415/445/0 487/487/465) [2,4] r=-1 lpr=487 pi=[414,487)/1 crt=463'751 lcod 0'0 unknown NOTIFY] start_peering_interval up [2] -> [2,4], acting [2] -> [2,4], acting_primary 2 -> 2, up_primary 2 -> 2, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.178178 7f35480a2700 1 osd.10 pg_epoch: 492 pg[9.1e( v 463'751 (0'0,463'751] local-lis/les=414/415 n=269 ec=190/190 lis/c 414/414 les/c/f 415/445/0 492/492/465) [2,4,10] r=2 lpr=492 pi=[414,492)/1 crt=463'751 lcod 0'0 unknown NOTIFY] start_peering_interval up [2,4] -> [2,4,10], acting [2,4] -> [2,4,10], acting_primary 2 -> 2, up_primary 2 -> 2, role -1 -> 2, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.178213 7f35478a1700 1 osd.10 pg_epoch: 472 pg[13.fs4( v 377'8989 (0'0,377'8989] local-lis/les=428/429 n=2712 ec=345/345 lis/c 428/428 les/c/f 429/429/0 472/472/467) [5,7,2147483647,6,2147483647,8]p5(0) r=-1 lpr=472 pi=[428,472)/1 crt=377'8979 lcod 0'0 unknown NOTIFY] start_peering_interval up [5,7,2147483647,6,2147483647,2147483647] -> [5,7,2147483647,6,2147483647,8], acting [5,7,2147483647,6,2147483647,2147483647] -> [5,7,2147483647,6,2147483647,8], acting_primary 5(0) -> 5, up_primary 5(0) -> 5, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.178242 7f35480a2700 1 osd.10 pg_epoch: 492 pg[9.1e( v 463'751 (0'0,463'751] local-lis/les=414/415 n=269 ec=190/190 lis/c 414/414 les/c/f 415/445/0 492/492/465) [2,4,10] r=2 lpr=492 pi=[414,492)/1 crt=463'751 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:38.178399 7f35480a2700 1 osd.10 pg_epoch: 464 pg[13.2s4( v 377'9209 (0'0,377'9209] local-lis/les=428/429 n=2712 ec=345/345 lis/c 428/428 les/c/f 429/429/0 464/464/464) [2147483647,2147483647,6,7,2147483647,2147483647]p6(2) r=-1 lpr=464 pi=[428,464)/1 crt=377'9201 lcod 0'0 unknown NOTIFY] start_peering_interval up [5,8,6,7,10,3] -> [2147483647,2147483647,6,7,2147483647,2147483647], acting [5,8,6,7,10,3] -> [2147483647,2147483647,6,7,2147483647,2147483647], acting_primary 5(0) -> 6, up_primary 5(0) -> 6, role 4 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.178403 7f35478a1700 1 osd.10 pg_epoch: 480 pg[13.fs4( v 377'8989 (0'0,377'8989] local-lis/les=428/429 n=2712 ec=345/345 lis/c 428/428 les/c/f 429/429/0 480/480/467) [5,7,2147483647,2147483647,2147483647,8]p5(0) r=-1 lpr=480 pi=[428,480)/1 crt=377'8979 lcod 0'0 unknown NOTIFY] start_peering_interval up [5,7,2147483647,6,2147483647,8] -> [5,7,2147483647,2147483647,2147483647,8], acting [5,7,2147483647,6,2147483647,8] -> [5,7,2147483647,2147483647,2147483647,8], acting_primary 5(0) -> 5, up_primary 5(0) -> 5, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.178458 7f35478a1700 1 osd.10 pg_epoch: 481 pg[13.fs4( v 377'8989 (0'0,377'8989] local-lis/les=428/429 n=2712 ec=345/345 lis/c 428/428 les/c/f 429/429/0 481/481/467) [5,7,2147483647,6,2147483647,8]p5(0) r=-1 lpr=481 pi=[428,481)/1 crt=377'8979 lcod 0'0 unknown NOTIFY] start_peering_interval up [5,7,2147483647,2147483647,2147483647,8] -> [5,7,2147483647,6,2147483647,8], acting [5,7,2147483647,2147483647,2147483647,8] -> [5,7,2147483647,6,2147483647,8], acting_primary 5(0) -> 5, up_primary 5(0) -> 5, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.178476 7f35480a2700 1 osd.10 pg_epoch: 467 pg[13.2s4( v 377'9209 (0'0,377'9209] local-lis/les=428/429 n=2712 ec=345/345 lis/c 428/428 les/c/f 429/429/0 467/467/467) [5,2147483647,6,2147483647,2147483647,2147483647]p5(0) r=-1 lpr=467 pi=[428,467)/1 crt=377'9201 lcod 0'0 unknown NOTIFY] start_peering_interval up [2147483647,2147483647,6,7,2147483647,2147483647] -> [5,2147483647,6,2147483647,2147483647,2147483647], acting [2147483647,2147483647,6,7,2147483647,2147483647] -> [5,2147483647,6,2147483647,2147483647,2147483647], acting_primary 6(2) -> 5, up_primary 6(2) -> 5, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.178528 7f35480a2700 1 osd.10 pg_epoch: 468 pg[13.2s4( v 377'9209 (0'0,377'9209] local-lis/les=428/429 n=2712 ec=345/345 lis/c 428/428 les/c/f 429/429/0 468/468/467) [5,2147483647,6,7,2147483647,2147483647]p5(0) r=-1 lpr=468 pi=[428,468)/1 crt=377'9201 lcod 0'0 unknown NOTIFY] start_peering_interval up [5,2147483647,6,2147483647,2147483647,2147483647] -> [5,2147483647,6,7,2147483647,2147483647], acting [5,2147483647,6,2147483647,2147483647,2147483647] -> [5,2147483647,6,7,2147483647,2147483647], acting_primary 5(0) -> 5, up_primary 5(0) -> 5, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.178556 7f35478a1700 1 osd.10 pg_epoch: 484 pg[13.fs4( v 377'8989 (0'0,377'8989] local-lis/les=428/429 n=2712 ec=345/345 lis/c 428/428 les/c/f 429/429/0 484/484/467) [5,7,3,6,2147483647,8]p5(0) r=-1 lpr=484 pi=[428,484)/1 crt=377'8979 lcod 0'0 unknown NOTIFY] start_peering_interval up [5,7,2147483647,6,2147483647,8] -> [5,7,3,6,2147483647,8], acting [5,7,2147483647,6,2147483647,8] -> [5,7,3,6,2147483647,8], acting_primary 5(0) -> 5, up_primary 5(0) -> 5, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.178610 7f35480a2700 1 osd.10 pg_epoch: 472 pg[13.2s4( v 377'9209 (0'0,377'9209] local-lis/les=428/429 n=2712 ec=345/345 lis/c 428/428 les/c/f 429/429/0 472/472/467) [5,8,6,7,2147483647,2147483647]p5(0) r=-1 lpr=472 pi=[428,472)/1 crt=377'9201 lcod 0'0 unknown NOTIFY] start_peering_interval up [5,2147483647,6,7,2147483647,2147483647] -> [5,8,6,7,2147483647,2147483647], acting [5,2147483647,6,7,2147483647,2147483647] -> [5,8,6,7,2147483647,2147483647], acting_primary 5(0) -> 5, up_primary 5(0) -> 5, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.178737 7f35480a2700 1 osd.10 pg_epoch: 480 pg[13.2s4( v 377'9209 (0'0,377'9209] local-lis/les=428/429 n=2712 ec=345/345 lis/c 428/428 les/c/f 429/429/0 480/480/467) [5,8,2147483647,7,2147483647,2147483647]p5(0) r=-1 lpr=480 pi=[428,480)/1 crt=377'9201 lcod 0'0 unknown NOTIFY] start_peering_interval up [5,8,6,7,2147483647,2147483647] -> [5,8,2147483647,7,2147483647,2147483647], acting [5,8,6,7,2147483647,2147483647] -> [5,8,2147483647,7,2147483647,2147483647], acting_primary 5(0) -> 5, up_primary 5(0) -> 5, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.178756 7f35478a1700 1 osd.10 pg_epoch: 492 pg[13.fs4( v 377'8989 (0'0,377'8989] local-lis/les=428/429 n=2712 ec=345/345 lis/c 428/428 les/c/f 429/429/0 492/492/467) [5,7,3,6,10,8]p5(0) r=4 lpr=492 pi=[428,492)/1 crt=377'8979 lcod 0'0 unknown NOTIFY] start_peering_interval up [5,7,3,6,2147483647,8] -> [5,7,3,6,10,8], acting [5,7,3,6,2147483647,8] -> [5,7,3,6,10,8], acting_primary 5(0) -> 5, up_primary 5(0) -> 5, role -1 -> 4, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.178787 7f35480a2700 1 osd.10 pg_epoch: 481 pg[13.2s4( v 377'9209 (0'0,377'9209] local-lis/les=428/429 n=2712 ec=345/345 lis/c 428/428 les/c/f 429/429/0 481/481/467) [5,8,6,7,2147483647,2147483647]p5(0) r=-1 lpr=481 pi=[428,481)/1 crt=377'9201 lcod 0'0 unknown NOTIFY] start_peering_interval up [5,8,2147483647,7,2147483647,2147483647] -> [5,8,6,7,2147483647,2147483647], acting [5,8,2147483647,7,2147483647,2147483647] -> [5,8,6,7,2147483647,2147483647], acting_primary 5(0) -> 5, up_primary 5(0) -> 5, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.178833 7f35478a1700 1 osd.10 pg_epoch: 492 pg[13.fs4( v 377'8989 (0'0,377'8989] local-lis/les=428/429 n=2712 ec=345/345 lis/c 428/428 les/c/f 429/429/0 492/492/467) [5,7,3,6,10,8]p5(0) r=4 lpr=492 pi=[428,492)/1 crt=377'8979 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:38.178861 7f35480a2700 1 osd.10 pg_epoch: 484 pg[13.2s4( v 377'9209 (0'0,377'9209] local-lis/les=428/429 n=2712 ec=345/345 lis/c 428/428 les/c/f 429/429/0 484/484/467) [5,8,6,7,2147483647,3]p5(0) r=-1 lpr=484 pi=[428,484)/1 crt=377'9201 lcod 0'0 unknown NOTIFY] start_peering_interval up [5,8,6,7,2147483647,2147483647] -> [5,8,6,7,2147483647,3], acting [5,8,6,7,2147483647,2147483647] -> [5,8,6,7,2147483647,3], acting_primary 5(0) -> 5, up_primary 5(0) -> 5, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.178989 7f35480a2700 1 osd.10 pg_epoch: 492 pg[13.2s4( v 377'9209 (0'0,377'9209] local-lis/les=428/429 n=2712 ec=345/345 lis/c 428/428 les/c/f 429/429/0 492/492/467) [5,8,6,7,10,3]p5(0) r=4 lpr=492 pi=[428,492)/1 crt=377'9201 lcod 0'0 unknown NOTIFY] start_peering_interval up [5,8,6,7,2147483647,3] -> [5,8,6,7,10,3], acting [5,8,6,7,2147483647,3] -> [5,8,6,7,10,3], acting_primary 5(0) -> 5, up_primary 5(0) -> 5, role -1 -> 4, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.179066 7f35480a2700 1 osd.10 pg_epoch: 492 pg[13.2s4( v 377'9209 (0'0,377'9209] local-lis/les=428/429 n=2712 ec=345/345 lis/c 428/428 les/c/f 429/429/0 492/492/467) [5,8,6,7,10,3]p5(0) r=4 lpr=492 pi=[428,492)/1 crt=377'9201 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:38.179161 7f35480a2700 1 osd.10 pg_epoch: 464 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=414/415 n=68041 ec=71/71 lis/c 414/414 les/c/f 415/415/0 414/414/403) [10,2,4] r=0 lpr=456 crt=166'68041 lcod 0'0 mlcod 0'0 peering] state<Started/Primary/Peering>: Peering, affected_by_map, going to Reset >2018-07-11 05:55:38.179279 7f35480a2700 1 osd.10 pg_epoch: 464 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=414/415 n=68041 ec=71/71 lis/c 414/414 les/c/f 415/415/0 464/464/464) [] r=-1 lpr=464 pi=[414,464)/1 crt=166'68041 lcod 0'0 unknown] start_peering_interval up [10,2,4] -> [], acting [10,2,4] -> [], acting_primary 10 -> -1, up_primary 10 -> -1, role 0 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.179328 7f35480a2700 1 osd.10 pg_epoch: 465 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=414/415 n=68041 ec=71/71 lis/c 414/414 les/c/f 415/415/0 465/465/465) [2] r=-1 lpr=465 pi=[414,465)/1 crt=166'68041 lcod 0'0 unknown NOTIFY] start_peering_interval up [] -> [2], acting [] -> [2], acting_primary ? -> 2, up_primary ? -> 2, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.179562 7f35480a2700 1 osd.10 pg_epoch: 487 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=414/415 n=68041 ec=71/71 lis/c 414/414 les/c/f 415/415/0 487/487/465) [2,4] r=-1 lpr=487 pi=[414,487)/1 crt=166'68041 lcod 0'0 unknown NOTIFY] start_peering_interval up [2] -> [2,4], acting [2] -> [2,4], acting_primary 2 -> 2, up_primary 2 -> 2, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.179645 7f35480a2700 1 osd.10 pg_epoch: 492 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=414/415 n=68041 ec=71/71 lis/c 414/414 les/c/f 415/415/0 492/492/492) [10,2,4] r=0 lpr=492 pi=[414,492)/2 crt=166'68041 lcod 0'0 mlcod 0'0 unknown NOTIFY] start_peering_interval up [2,4] -> [10,2,4], acting [2,4] -> [10,2,4], acting_primary 2 -> 10, up_primary 2 -> 10, role -1 -> 0, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.179712 7f35480a2700 1 osd.10 pg_epoch: 492 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=414/415 n=68041 ec=71/71 lis/c 414/414 les/c/f 415/415/0 492/492/492) [10,2,4] r=0 lpr=492 pi=[414,492)/2 crt=166'68041 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 05:55:38.179831 7f35480a2700 1 osd.10 pg_epoch: 464 pg[11.1s0( v 383'60378 (352'56674,383'60378] local-lis/les=428/429 n=60378 ec=267/267 lis/c 428/428 les/c/f 429/429/0 428/428/403) [10,5,2,8,6,11]p10(0) r=0 lpr=461 crt=383'60378 lcod 0'0 mlcod 0'0 peering] state<Started/Primary/Peering>: Peering, affected_by_map, going to Reset >2018-07-11 05:55:38.179938 7f35480a2700 1 osd.10 pg_epoch: 464 pg[11.1s0( v 383'60378 (352'56674,383'60378] local-lis/les=428/429 n=60378 ec=267/267 lis/c 428/428 les/c/f 429/429/0 464/464/464) [2147483647,2147483647,2147483647,2147483647,6,11]p6(4) r=-1 lpr=464 pi=[428,464)/1 crt=383'60378 lcod 0'0 unknown] start_peering_interval up [10,5,2,8,6,11] -> [2147483647,2147483647,2147483647,2147483647,6,11], acting [10,5,2,8,6,11] -> [2147483647,2147483647,2147483647,2147483647,6,11], acting_primary 10(0) -> 6, up_primary 10(0) -> 6, role 0 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.180002 7f35480a2700 1 osd.10 pg_epoch: 465 pg[11.1s0( v 383'60378 (352'56674,383'60378] local-lis/les=428/429 n=60378 ec=267/267 lis/c 428/428 les/c/f 429/429/0 465/465/465) [2147483647,2147483647,2,2147483647,6,11]p2(2) r=-1 lpr=465 pi=[428,465)/1 crt=383'60378 lcod 0'0 unknown NOTIFY] start_peering_interval up [2147483647,2147483647,2147483647,2147483647,6,11] -> [2147483647,2147483647,2,2147483647,6,11], acting [2147483647,2147483647,2147483647,2147483647,6,11] -> [2147483647,2147483647,2,2147483647,6,11], acting_primary 6(4) -> 2, up_primary 6(4) -> 2, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.180074 7f35480a2700 1 osd.10 pg_epoch: 467 pg[11.1s0( v 383'60378 (352'56674,383'60378] local-lis/les=428/429 n=60378 ec=267/267 lis/c 428/428 les/c/f 429/429/0 467/467/467) [2147483647,5,2,2147483647,6,11]p5(1) r=-1 lpr=467 pi=[428,467)/1 crt=383'60378 lcod 0'0 unknown NOTIFY] start_peering_interval up [2147483647,2147483647,2,2147483647,6,11] -> [2147483647,5,2,2147483647,6,11], acting [2147483647,2147483647,2,2147483647,6,11] -> [2147483647,5,2,2147483647,6,11], acting_primary 2(2) -> 5, up_primary 2(2) -> 5, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.180188 7f35480a2700 1 osd.10 pg_epoch: 472 pg[11.1s0( v 383'60378 (352'56674,383'60378] local-lis/les=428/429 n=60378 ec=267/267 lis/c 428/428 les/c/f 429/429/0 472/472/467) [2147483647,5,2,8,6,11]p5(1) r=-1 lpr=472 pi=[428,472)/1 crt=383'60378 lcod 0'0 unknown NOTIFY] start_peering_interval up [2147483647,5,2,2147483647,6,11] -> [2147483647,5,2,8,6,11], acting [2147483647,5,2,2147483647,6,11] -> [2147483647,5,2,8,6,11], acting_primary 5(1) -> 5, up_primary 5(1) -> 5, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.180318 7f35480a2700 1 osd.10 pg_epoch: 477 pg[11.1s0( v 383'60378 (352'56674,383'60378] local-lis/les=428/429 n=60378 ec=267/267 lis/c 428/428 les/c/f 429/429/0 477/477/467) [2147483647,5,2,8,6,2147483647]p5(1) r=-1 lpr=477 pi=[428,477)/1 crt=383'60378 lcod 0'0 unknown NOTIFY] start_peering_interval up [2147483647,5,2,8,6,11] -> [2147483647,5,2,8,6,2147483647], acting [2147483647,5,2,8,6,11] -> [2147483647,5,2,8,6,2147483647], acting_primary 5(1) -> 5, up_primary 5(1) -> 5, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.180374 7f35480a2700 1 osd.10 pg_epoch: 478 pg[11.1s0( v 383'60378 (352'56674,383'60378] local-lis/les=428/429 n=60378 ec=267/267 lis/c 428/428 les/c/f 429/429/0 478/478/467) [2147483647,5,2,8,6,11]p5(1) r=-1 lpr=478 pi=[428,478)/1 crt=383'60378 lcod 0'0 unknown NOTIFY] start_peering_interval up [2147483647,5,2,8,6,2147483647] -> [2147483647,5,2,8,6,11], acting [2147483647,5,2,8,6,2147483647] -> [2147483647,5,2,8,6,11], acting_primary 5(1) -> 5, up_primary 5(1) -> 5, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.180448 7f35480a2700 1 osd.10 pg_epoch: 480 pg[11.1s0( v 383'60378 (352'56674,383'60378] local-lis/les=428/429 n=60378 ec=267/267 lis/c 428/428 les/c/f 429/429/0 480/480/467) [2147483647,5,2,8,2147483647,11]p5(1) r=-1 lpr=480 pi=[428,480)/1 crt=383'60378 lcod 0'0 unknown NOTIFY] start_peering_interval up [2147483647,5,2,8,6,11] -> [2147483647,5,2,8,2147483647,11], acting [2147483647,5,2,8,6,11] -> [2147483647,5,2,8,2147483647,11], acting_primary 5(1) -> 5, up_primary 5(1) -> 5, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.180501 7f35480a2700 1 osd.10 pg_epoch: 481 pg[11.1s0( v 383'60378 (352'56674,383'60378] local-lis/les=428/429 n=60378 ec=267/267 lis/c 428/428 les/c/f 429/429/0 481/481/467) [2147483647,5,2,8,6,11]p5(1) r=-1 lpr=481 pi=[428,481)/1 crt=383'60378 lcod 0'0 unknown NOTIFY] start_peering_interval up [2147483647,5,2,8,2147483647,11] -> [2147483647,5,2,8,6,11], acting [2147483647,5,2,8,2147483647,11] -> [2147483647,5,2,8,6,11], acting_primary 5(1) -> 5, up_primary 5(1) -> 5, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.180714 7f35480a2700 1 osd.10 pg_epoch: 492 pg[11.1s0( v 383'60378 (352'56674,383'60378] local-lis/les=428/429 n=60378 ec=267/267 lis/c 428/428 les/c/f 429/429/0 492/492/492) [10,5,2,8,6,11]p10(0) r=0 lpr=492 pi=[428,492)/1 crt=383'60378 lcod 0'0 mlcod 0'0 unknown NOTIFY] start_peering_interval up [2147483647,5,2,8,6,11] -> [10,5,2,8,6,11], acting [2147483647,5,2,8,6,11] -> [10,5,2,8,6,11], acting_primary 5(1) -> 10, up_primary 5(1) -> 10, role -1 -> 0, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.180784 7f35480a2700 1 osd.10 pg_epoch: 492 pg[11.1s0( v 383'60378 (352'56674,383'60378] local-lis/les=428/429 n=60378 ec=267/267 lis/c 428/428 les/c/f 429/429/0 492/492/492) [10,5,2,8,6,11]p10(0) r=0 lpr=492 pi=[428,492)/1 crt=383'60378 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 05:55:38.181055 7f35480a2700 1 osd.10 pg_epoch: 464 pg[13.5s2( v 377'9560 (0'0,377'9560] local-lis/les=447/448 n=2691 ec=345/345 lis/c 447/447 les/c/f 448/455/0 464/464/464) [2147483647,2147483647,2147483647,11,2147483647,2147483647]p11(3) r=-1 lpr=464 pi=[447,464)/1 crt=377'9552 lcod 0'0 unknown NOTIFY] start_peering_interval up [4,3,10,11,2147483647,8] -> [2147483647,2147483647,2147483647,11,2147483647,2147483647], acting [4,3,10,11,8,8] -> [2147483647,2147483647,2147483647,11,2147483647,2147483647], acting_primary 4(0) -> 11, up_primary 4(0) -> 11, role 2 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.181636 7f35480a2700 1 osd.10 pg_epoch: 472 pg[13.5s2( v 377'9560 (0'0,377'9560] local-lis/les=447/448 n=2691 ec=345/345 lis/c 447/447 les/c/f 448/455/0 472/472/464) [2147483647,2147483647,2147483647,11,2147483647,8]p11(3) r=-1 lpr=472 pi=[447,472)/1 crt=377'9552 lcod 0'0 unknown NOTIFY] start_peering_interval up [2147483647,2147483647,2147483647,11,2147483647,2147483647] -> [2147483647,2147483647,2147483647,11,2147483647,8], acting [2147483647,2147483647,2147483647,11,2147483647,2147483647] -> [2147483647,2147483647,2147483647,11,2147483647,8], acting_primary 11(3) -> 11, up_primary 11(3) -> 11, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.182012 7f35480a2700 1 osd.10 pg_epoch: 477 pg[13.5s2( v 377'9560 (0'0,377'9560] local-lis/les=447/448 n=2691 ec=345/345 lis/c 447/447 les/c/f 448/455/0 477/477/477) [2147483647,2147483647,2147483647,2147483647,2147483647,8]p8(5) r=-1 lpr=477 pi=[447,477)/1 crt=377'9552 lcod 0'0 unknown NOTIFY] start_peering_interval up [2147483647,2147483647,2147483647,11,2147483647,8] -> [2147483647,2147483647,2147483647,2147483647,2147483647,8], acting [2147483647,2147483647,2147483647,11,2147483647,8] -> [2147483647,2147483647,2147483647,2147483647,2147483647,8], acting_primary 11(3) -> 8, up_primary 11(3) -> 8, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.182129 7f35480a2700 1 osd.10 pg_epoch: 478 pg[13.5s2( v 377'9560 (0'0,377'9560] local-lis/les=447/448 n=2691 ec=345/345 lis/c 447/447 les/c/f 448/455/0 478/478/478) [2147483647,2147483647,2147483647,11,2147483647,8]p11(3) r=-1 lpr=478 pi=[447,478)/1 crt=377'9552 lcod 0'0 unknown NOTIFY] start_peering_interval up [2147483647,2147483647,2147483647,2147483647,2147483647,8] -> [2147483647,2147483647,2147483647,11,2147483647,8], acting [2147483647,2147483647,2147483647,2147483647,2147483647,8] -> [2147483647,2147483647,2147483647,11,2147483647,8], acting_primary 8(5) -> 11, up_primary 8(5) -> 11, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.182563 7f35480a2700 1 osd.10 pg_epoch: 484 pg[13.5s2( v 377'9560 (0'0,377'9560] local-lis/les=447/448 n=2691 ec=345/345 lis/c 447/447 les/c/f 448/455/0 484/484/484) [2147483647,3,2147483647,11,2147483647,8]p3(1) r=-1 lpr=484 pi=[447,484)/1 crt=377'9552 lcod 0'0 unknown NOTIFY] start_peering_interval up [2147483647,2147483647,2147483647,11,2147483647,8] -> [2147483647,3,2147483647,11,2147483647,8], acting [2147483647,2147483647,2147483647,11,2147483647,8] -> [2147483647,3,2147483647,11,2147483647,8], acting_primary 11(3) -> 3, up_primary 11(3) -> 3, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.182797 7f35480a2700 1 osd.10 pg_epoch: 487 pg[13.5s2( v 377'9560 (0'0,377'9560] local-lis/les=447/448 n=2691 ec=345/345 lis/c 447/447 les/c/f 448/455/0 487/487/487) [4,3,2147483647,11,2147483647,8]p4(0) r=-1 lpr=487 pi=[447,487)/1 crt=377'9552 lcod 0'0 unknown NOTIFY] start_peering_interval up [2147483647,3,2147483647,11,2147483647,8] -> [4,3,2147483647,11,2147483647,8], acting [2147483647,3,2147483647,11,2147483647,8] -> [4,3,2147483647,11,2147483647,8], acting_primary 3(1) -> 4, up_primary 3(1) -> 4, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.183033 7f35480a2700 1 osd.10 pg_epoch: 490 pg[13.5s2( v 377'9560 (0'0,377'9560] local-lis/les=447/448 n=2691 ec=345/345 lis/c 447/447 les/c/f 448/455/0 487/490/487) [4,3,2147483647,11,2147483647,8]/[4,3,2147483647,11,8,8]p4(0) r=-1 lpr=490 pi=[447,490)/1 crt=377'9552 lcod 0'0 remapped NOTIFY] start_peering_interval up [4,3,2147483647,11,2147483647,8] -> [4,3,2147483647,11,2147483647,8], acting [4,3,2147483647,11,2147483647,8] -> [4,3,2147483647,11,8,8], acting_primary 4(0) -> 4, up_primary 4(0) -> 4, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.183210 7f35480a2700 1 osd.10 pg_epoch: 492 pg[13.5s2( v 377'9560 (0'0,377'9560] local-lis/les=447/448 n=2691 ec=345/345 lis/c 447/447 les/c/f 448/455/0 492/492/487) [4,3,10,11,2147483647,8]/[4,3,2147483647,11,8,8]p4(0) r=-1 lpr=492 pi=[447,492)/1 crt=377'9552 lcod 0'0 remapped NOTIFY] start_peering_interval up [4,3,2147483647,11,2147483647,8] -> [4,3,10,11,2147483647,8], acting [4,3,2147483647,11,8,8] -> [4,3,2147483647,11,8,8], acting_primary 4(0) -> 4, up_primary 4(0) -> 4, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.183292 7f35480a2700 1 osd.10 pg_epoch: 492 pg[13.5s2( v 377'9560 (0'0,377'9560] local-lis/les=447/448 n=2691 ec=345/345 lis/c 447/447 les/c/f 448/455/0 492/492/487) [4,3,10,11,2147483647,8]/[4,3,2147483647,11,8,8]p4(0) r=-1 lpr=492 pi=[447,492)/1 crt=377'9552 lcod 0'0 remapped NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:38.183449 7f35480a2700 1 osd.10 pg_epoch: 464 pg[13.4s3( v 377'9519 (355'1503,377'9519] local-lis/les=428/429 n=2672 ec=345/345 lis/c 428/428 les/c/f 429/429/0 464/464/464) [2147483647,2147483647,2147483647,2147483647,11,6]p11(4) r=-1 lpr=464 pi=[428,464)/1 crt=377'9519 lcod 0'0 unknown NOTIFY] start_peering_interval up [5,1,8,10,11,6] -> [2147483647,2147483647,2147483647,2147483647,11,6], acting [5,1,8,10,11,6] -> [2147483647,2147483647,2147483647,2147483647,11,6], acting_primary 5(0) -> 11, up_primary 5(0) -> 11, role 3 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.183543 7f35480a2700 1 osd.10 pg_epoch: 467 pg[13.4s3( v 377'9519 (355'1503,377'9519] local-lis/les=428/429 n=2672 ec=345/345 lis/c 428/428 les/c/f 429/429/0 467/467/467) [5,2147483647,2147483647,2147483647,11,6]p5(0) r=-1 lpr=467 pi=[428,467)/1 crt=377'9519 lcod 0'0 unknown NOTIFY] start_peering_interval up [2147483647,2147483647,2147483647,2147483647,11,6] -> [5,2147483647,2147483647,2147483647,11,6], acting [2147483647,2147483647,2147483647,2147483647,11,6] -> [5,2147483647,2147483647,2147483647,11,6], acting_primary 11(4) -> 5, up_primary 11(4) -> 5, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.183616 7f35480a2700 1 osd.10 pg_epoch: 470 pg[13.4s3( v 377'9519 (355'1503,377'9519] local-lis/les=428/429 n=2672 ec=345/345 lis/c 428/428 les/c/f 429/429/0 470/470/467) [5,1,2147483647,2147483647,11,6]p5(0) r=-1 lpr=470 pi=[428,470)/1 crt=377'9519 lcod 0'0 unknown NOTIFY] start_peering_interval up [5,2147483647,2147483647,2147483647,11,6] -> [5,1,2147483647,2147483647,11,6], acting [5,2147483647,2147483647,2147483647,11,6] -> [5,1,2147483647,2147483647,11,6], acting_primary 5(0) -> 5, up_primary 5(0) -> 5, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.183676 7f35480a2700 1 osd.10 pg_epoch: 472 pg[13.4s3( v 377'9519 (355'1503,377'9519] local-lis/les=428/429 n=2672 ec=345/345 lis/c 428/428 les/c/f 429/429/0 472/472/467) [5,1,8,2147483647,11,6]p5(0) r=-1 lpr=472 pi=[428,472)/1 crt=377'9519 lcod 0'0 unknown NOTIFY] start_peering_interval up [5,1,2147483647,2147483647,11,6] -> [5,1,8,2147483647,11,6], acting [5,1,2147483647,2147483647,11,6] -> [5,1,8,2147483647,11,6], acting_primary 5(0) -> 5, up_primary 5(0) -> 5, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.183774 7f35480a2700 1 osd.10 pg_epoch: 477 pg[13.4s3( v 377'9519 (355'1503,377'9519] local-lis/les=428/429 n=2672 ec=345/345 lis/c 428/428 les/c/f 429/429/0 477/477/467) [5,1,8,2147483647,2147483647,6]p5(0) r=-1 lpr=477 pi=[428,477)/1 crt=377'9519 lcod 0'0 unknown NOTIFY] start_peering_interval up [5,1,8,2147483647,11,6] -> [5,1,8,2147483647,2147483647,6], acting [5,1,8,2147483647,11,6] -> [5,1,8,2147483647,2147483647,6], acting_primary 5(0) -> 5, up_primary 5(0) -> 5, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.183823 7f35480a2700 1 osd.10 pg_epoch: 478 pg[13.4s3( v 377'9519 (355'1503,377'9519] local-lis/les=428/429 n=2672 ec=345/345 lis/c 428/428 les/c/f 429/429/0 478/478/467) [5,1,8,2147483647,11,6]p5(0) r=-1 lpr=478 pi=[428,478)/1 crt=377'9519 lcod 0'0 unknown NOTIFY] start_peering_interval up [5,1,8,2147483647,2147483647,6] -> [5,1,8,2147483647,11,6], acting [5,1,8,2147483647,2147483647,6] -> [5,1,8,2147483647,11,6], acting_primary 5(0) -> 5, up_primary 5(0) -> 5, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.183887 7f35480a2700 1 osd.10 pg_epoch: 480 pg[13.4s3( v 377'9519 (355'1503,377'9519] local-lis/les=428/429 n=2672 ec=345/345 lis/c 428/428 les/c/f 429/429/0 480/480/467) [5,1,8,2147483647,11,2147483647]p5(0) r=-1 lpr=480 pi=[428,480)/1 crt=377'9519 lcod 0'0 unknown NOTIFY] start_peering_interval up [5,1,8,2147483647,11,6] -> [5,1,8,2147483647,11,2147483647], acting [5,1,8,2147483647,11,6] -> [5,1,8,2147483647,11,2147483647], acting_primary 5(0) -> 5, up_primary 5(0) -> 5, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.183937 7f35480a2700 1 osd.10 pg_epoch: 481 pg[13.4s3( v 377'9519 (355'1503,377'9519] local-lis/les=428/429 n=2672 ec=345/345 lis/c 428/428 les/c/f 429/429/0 481/481/467) [5,1,8,2147483647,11,6]p5(0) r=-1 lpr=481 pi=[428,481)/1 crt=377'9519 lcod 0'0 unknown NOTIFY] start_peering_interval up [5,1,8,2147483647,11,2147483647] -> [5,1,8,2147483647,11,6], acting [5,1,8,2147483647,11,2147483647] -> [5,1,8,2147483647,11,6], acting_primary 5(0) -> 5, up_primary 5(0) -> 5, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.184103 7f35480a2700 1 osd.10 pg_epoch: 492 pg[13.4s3( v 377'9519 (355'1503,377'9519] local-lis/les=428/429 n=2672 ec=345/345 lis/c 428/428 les/c/f 429/429/0 492/492/467) [5,1,8,10,11,6]p5(0) r=3 lpr=492 pi=[428,492)/1 crt=377'9519 lcod 0'0 unknown NOTIFY] start_peering_interval up [5,1,8,2147483647,11,6] -> [5,1,8,10,11,6], acting [5,1,8,2147483647,11,6] -> [5,1,8,10,11,6], acting_primary 5(0) -> 5, up_primary 5(0) -> 5, role -1 -> 3, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.184189 7f35480a2700 1 osd.10 pg_epoch: 492 pg[13.4s3( v 377'9519 (355'1503,377'9519] local-lis/les=428/429 n=2672 ec=345/345 lis/c 428/428 les/c/f 429/429/0 492/492/467) [5,1,8,10,11,6]p5(0) r=3 lpr=492 pi=[428,492)/1 crt=377'9519 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:38.184330 7f35480a2700 1 osd.10 pg_epoch: 464 pg[5.2f( v 166'67864 (166'66267,166'67864] local-lis/les=414/415 n=67864 ec=71/71 lis/c 414/414 les/c/f 415/415/0 464/464/464) [] r=-1 lpr=464 pi=[414,464)/1 crt=166'67864 lcod 0'0 unknown NOTIFY] start_peering_interval up [8,10,4] -> [], acting [8,10,4] -> [], acting_primary 8 -> -1, up_primary 8 -> -1, role 1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.184415 7f35480a2700 1 osd.10 pg_epoch: 472 pg[5.2f( v 166'67864 (166'66267,166'67864] local-lis/les=414/415 n=67864 ec=71/71 lis/c 414/414 les/c/f 415/415/0 472/472/472) [8] r=-1 lpr=472 pi=[414,472)/1 crt=166'67864 lcod 0'0 unknown NOTIFY] start_peering_interval up [] -> [8], acting [] -> [8], acting_primary ? -> 8, up_primary ? -> 8, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.184537 7f35480a2700 1 osd.10 pg_epoch: 487 pg[5.2f( v 166'67864 (166'66267,166'67864] local-lis/les=414/415 n=67864 ec=71/71 lis/c 414/414 les/c/f 415/415/0 487/487/472) [8,4] r=-1 lpr=487 pi=[414,487)/1 crt=166'67864 lcod 0'0 unknown NOTIFY] start_peering_interval up [8] -> [8,4], acting [8] -> [8,4], acting_primary 8 -> 8, up_primary 8 -> 8, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.184605 7f35480a2700 1 osd.10 pg_epoch: 492 pg[5.2f( v 166'67864 (166'66267,166'67864] local-lis/les=414/415 n=67864 ec=71/71 lis/c 414/414 les/c/f 415/415/0 492/492/472) [8,10,4] r=1 lpr=492 pi=[414,492)/2 crt=166'67864 lcod 0'0 unknown NOTIFY] start_peering_interval up [8,4] -> [8,10,4], acting [8,4] -> [8,10,4], acting_primary 8 -> 8, up_primary 8 -> 8, role -1 -> 1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.184677 7f35480a2700 1 osd.10 pg_epoch: 492 pg[5.2f( v 166'67864 (166'66267,166'67864] local-lis/les=414/415 n=67864 ec=71/71 lis/c 414/414 les/c/f 415/415/0 492/492/472) [8,10,4] r=1 lpr=492 pi=[414,492)/2 crt=166'67864 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:38.184810 7f35480a2700 1 osd.10 pg_epoch: 464 pg[6.2c( v 457'120217 (430'118627,457'120217] local-lis/les=428/429 n=26249 ec=76/76 lis/c 428/428 les/c/f 429/449/0 464/464/464) [6] r=-1 lpr=464 pi=[428,464)/1 crt=457'120217 lcod 0'0 unknown NOTIFY] start_peering_interval up [4,6,10] -> [6], acting [4,6,10] -> [6], acting_primary 4 -> 6, up_primary 4 -> 6, role 2 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.184939 7f35480a2700 1 osd.10 pg_epoch: 480 pg[6.2c( v 457'120217 (430'118627,457'120217] local-lis/les=428/429 n=26249 ec=76/76 lis/c 428/428 les/c/f 429/449/0 480/480/480) [] r=-1 lpr=480 pi=[428,480)/1 crt=457'120217 lcod 0'0 unknown NOTIFY] start_peering_interval up [6] -> [], acting [6] -> [], acting_primary 6 -> -1, up_primary 6 -> -1, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.184978 7f35480a2700 1 osd.10 pg_epoch: 481 pg[6.2c( v 457'120217 (430'118627,457'120217] local-lis/les=428/429 n=26249 ec=76/76 lis/c 428/428 les/c/f 429/449/0 481/481/481) [6] r=-1 lpr=481 pi=[428,481)/1 crt=457'120217 lcod 0'0 unknown NOTIFY] start_peering_interval up [] -> [6], acting [] -> [6], acting_primary ? -> 6, up_primary ? -> 6, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.185045 7f35480a2700 1 osd.10 pg_epoch: 487 pg[6.2c( v 457'120217 (430'118627,457'120217] local-lis/les=428/429 n=26249 ec=76/76 lis/c 428/428 les/c/f 429/449/0 487/487/487) [4,6] r=-1 lpr=487 pi=[428,487)/1 crt=457'120217 lcod 0'0 unknown NOTIFY] start_peering_interval up [6] -> [4,6], acting [6] -> [4,6], acting_primary 6 -> 4, up_primary 6 -> 4, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.185112 7f35480a2700 1 osd.10 pg_epoch: 492 pg[6.2c( v 457'120217 (430'118627,457'120217] local-lis/les=428/429 n=26249 ec=76/76 lis/c 428/428 les/c/f 429/449/0 492/492/487) [4,6,10] r=2 lpr=492 pi=[428,492)/2 crt=457'120217 lcod 0'0 unknown NOTIFY] start_peering_interval up [4,6] -> [4,6,10], acting [4,6] -> [4,6,10], acting_primary 4 -> 4, up_primary 4 -> 4, role -1 -> 2, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.185182 7f35480a2700 1 osd.10 pg_epoch: 492 pg[6.2c( v 457'120217 (430'118627,457'120217] local-lis/les=428/429 n=26249 ec=76/76 lis/c 428/428 les/c/f 429/449/0 492/492/487) [4,6,10] r=2 lpr=492 pi=[428,492)/2 crt=457'120217 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:38.185321 7f35480a2700 1 osd.10 pg_epoch: 464 pg[10.1c( v 386'649 (0'0,386'649] local-lis/les=419/420 n=500 ec=193/193 lis/c 419/419 les/c/f 420/420/0 464/464/419) [11] r=-1 lpr=464 pi=[419,464)/2 crt=386'649 lcod 0'0 unknown NOTIFY] start_peering_interval up [11,4,8] -> [11], acting [11,4,8] -> [11], acting_primary 11 -> 11, up_primary 11 -> 11, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.185418 7f35480a2700 1 osd.10 pg_epoch: 472 pg[10.1c( v 386'649 (0'0,386'649] local-lis/les=419/420 n=500 ec=193/193 lis/c 419/419 les/c/f 420/420/0 472/472/419) [11,8] r=-1 lpr=472 pi=[419,472)/2 crt=386'649 lcod 0'0 unknown NOTIFY] start_peering_interval up [11] -> [11,8], acting [11] -> [11,8], acting_primary 11 -> 11, up_primary 11 -> 11, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.185492 7f35480a2700 1 osd.10 pg_epoch: 477 pg[10.1c( v 386'649 (0'0,386'649] local-lis/les=419/420 n=500 ec=193/193 lis/c 419/419 les/c/f 420/420/0 477/477/477) [8] r=-1 lpr=477 pi=[419,477)/2 crt=386'649 lcod 0'0 unknown NOTIFY] start_peering_interval up [11,8] -> [8], acting [11,8] -> [8], acting_primary 11 -> 8, up_primary 11 -> 8, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.185534 7f35480a2700 1 osd.10 pg_epoch: 478 pg[10.1c( v 386'649 (0'0,386'649] local-lis/les=419/420 n=500 ec=193/193 lis/c 419/419 les/c/f 420/420/0 478/478/478) [11,8] r=-1 lpr=478 pi=[419,478)/2 crt=386'649 lcod 0'0 unknown NOTIFY] start_peering_interval up [8] -> [11,8], acting [8] -> [11,8], acting_primary 8 -> 11, up_primary 8 -> 11, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.185637 7f35480a2700 1 osd.10 pg_epoch: 487 pg[10.1c( v 386'649 (0'0,386'649] local-lis/les=419/420 n=500 ec=193/193 lis/c 419/419 les/c/f 420/420/0 487/487/478) [11,4,8] r=-1 lpr=487 pi=[419,487)/2 crt=386'649 lcod 0'0 unknown NOTIFY] start_peering_interval up [11,8] -> [11,4,8], acting [11,8] -> [11,4,8], acting_primary 11 -> 11, up_primary 11 -> 11, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.185755 7f35480a2700 1 osd.10 pg_epoch: 492 pg[10.1c( v 386'649 (0'0,386'649] local-lis/les=419/420 n=500 ec=193/193 lis/c 419/419 les/c/f 420/420/0 487/487/478) [11,4,8] r=-1 lpr=487 pi=[419,487)/2 crt=386'649 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:38.185838 7f35480a2700 1 osd.10 pg_epoch: 464 pg[10.7( v 386'575 (0'0,386'575] local-lis/les=428/429 n=507 ec=193/193 lis/c 428/428 les/c/f 429/457/0 428/428/403) [10,6,3] r=0 lpr=461 crt=386'575 lcod 0'0 mlcod 0'0 peering] state<Started/Primary/Peering>: Peering, affected_by_map, going to Reset >2018-07-11 05:55:38.185928 7f35480a2700 1 osd.10 pg_epoch: 464 pg[10.7( v 386'575 (0'0,386'575] local-lis/les=428/429 n=507 ec=193/193 lis/c 428/428 les/c/f 429/457/0 464/464/464) [6] r=-1 lpr=464 pi=[428,464)/1 crt=386'575 lcod 0'0 unknown] start_peering_interval up [10,6,3] -> [6], acting [10,6,3] -> [6], acting_primary 10 -> 6, up_primary 10 -> 6, role 0 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.186055 7f35480a2700 1 osd.10 pg_epoch: 480 pg[10.7( v 386'575 (0'0,386'575] local-lis/les=428/429 n=507 ec=193/193 lis/c 428/428 les/c/f 429/457/0 480/480/480) [] r=-1 lpr=480 pi=[428,480)/1 crt=386'575 lcod 0'0 unknown NOTIFY] start_peering_interval up [6] -> [], acting [6] -> [], acting_primary 6 -> -1, up_primary 6 -> -1, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.186094 7f35480a2700 1 osd.10 pg_epoch: 481 pg[10.7( v 386'575 (0'0,386'575] local-lis/les=428/429 n=507 ec=193/193 lis/c 428/428 les/c/f 429/457/0 481/481/481) [6] r=-1 lpr=481 pi=[428,481)/1 crt=386'575 lcod 0'0 unknown NOTIFY] start_peering_interval up [] -> [6], acting [] -> [6], acting_primary ? -> 6, up_primary ? -> 6, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.186142 7f35480a2700 1 osd.10 pg_epoch: 484 pg[10.7( v 386'575 (0'0,386'575] local-lis/les=428/429 n=507 ec=193/193 lis/c 428/428 les/c/f 429/457/0 484/484/481) [6,3] r=-1 lpr=484 pi=[428,484)/1 crt=386'575 lcod 0'0 unknown NOTIFY] start_peering_interval up [6] -> [6,3], acting [6] -> [6,3], acting_primary 6 -> 6, up_primary 6 -> 6, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.186224 7f35480a2700 1 osd.10 pg_epoch: 492 pg[10.7( v 386'575 (0'0,386'575] local-lis/les=428/429 n=507 ec=193/193 lis/c 428/428 les/c/f 429/457/0 492/492/492) [10,6,3] r=0 lpr=492 pi=[428,492)/1 crt=386'575 lcod 0'0 mlcod 0'0 unknown NOTIFY] start_peering_interval up [6,3] -> [10,6,3], acting [6,3] -> [10,6,3], acting_primary 6 -> 10, up_primary 6 -> 10, role -1 -> 0, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.186298 7f35480a2700 1 osd.10 pg_epoch: 492 pg[10.7( v 386'575 (0'0,386'575] local-lis/les=428/429 n=507 ec=193/193 lis/c 428/428 les/c/f 429/457/0 492/492/492) [10,6,3] r=0 lpr=492 pi=[428,492)/1 crt=386'575 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 05:55:38.360947 7f35480a2700 1 osd.10 pg_epoch: 493 pg[13.5s2( v 377'9560 (0'0,377'9560] local-lis/les=447/448 n=2691 ec=345/345 lis/c 490/447 les/c/f 491/455/0 492/493/487) [4,3,10,11,2147483647,8]/[4,3,10,11,8,8]p4(0) r=2 lpr=493 pi=[447,493)/1 crt=377'9552 lcod 0'0 remapped NOTIFY] start_peering_interval up [4,3,10,11,2147483647,8] -> [4,3,10,11,2147483647,8], acting [4,3,2147483647,11,8,8] -> [4,3,10,11,8,8], acting_primary 4(0) -> 4, up_primary 4(0) -> 4, role -1 -> 2, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 05:55:38.361040 7f35480a2700 1 osd.10 pg_epoch: 493 pg[13.5s2( v 377'9560 (0'0,377'9560] local-lis/les=447/448 n=2691 ec=345/345 lis/c 490/447 les/c/f 491/455/0 492/493/487) [4,3,10,11,2147483647,8]/[4,3,10,11,8,8]p4(0) r=2 lpr=493 pi=[447,493)/1 crt=377'9552 lcod 0'0 remapped NOTIFY] state<Start>: transitioning to Stray >2018-07-11 05:55:38.975318 7f355d88e700 0 -- 10.8.128.107:6801/12665 >> 10.8.128.107:6805/12617 conn(0x55a324482800 :6801 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg: challenging authorizer >2018-07-11 05:55:38.975494 7f355d88e700 0 -- 10.8.128.107:6801/12665 >> 10.8.128.107:6805/12617 conn(0x55a324482800 :6801 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg accept connect_seq 0 vs existing csq=0 existing_state=STATE_CONNECTING_WAIT_CONNECT_REPLY >2018-07-11 05:55:38.975760 7f355d88e700 0 -- 10.8.128.107:6801/12665 >> 10.8.128.107:6805/12617 conn(0x55a324481000 :-1 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg: challenging authorizer >2018-07-11 05:55:41.315223 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 6] Generated table #36305: 2638241 keys, 74546440 bytes >2018-07-11 05:55:41.315260 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288541315241, "cf_name": "default", "job": 6, "event": "table_file_creation", "file_number": 36305, "file_size": 74546440, "table_properties": {"data_size": 67112121, "index_size": 690146, "filter_size": 6743183, "raw_key_size": 76215851, "raw_average_key_size": 28, "raw_value_size": 31365754, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:55:44.429621 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 6] Generated table #36306: 2638240 keys, 74545900 bytes >2018-07-11 05:55:44.429665 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288544429638, "cf_name": "default", "job": 6, "event": "table_file_creation", "file_number": 36306, "file_size": 74545900, "table_properties": {"data_size": 67112154, "index_size": 689575, "filter_size": 6743181, "raw_key_size": 76215829, "raw_average_key_size": 28, "raw_value_size": 31365749, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:55:47.525586 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 6] Generated table #36307: 2638241 keys, 74546728 bytes >2018-07-11 05:55:47.525622 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288547525606, "cf_name": "default", "job": 6, "event": "table_file_creation", "file_number": 36307, "file_size": 74546728, "table_properties": {"data_size": 67112148, "index_size": 690407, "filter_size": 6743183, "raw_key_size": 76215856, "raw_average_key_size": 28, "raw_value_size": 31365759, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:55:50.709257 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 6] Generated table #36308: 2638241 keys, 74546571 bytes >2018-07-11 05:55:50.709288 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288550709274, "cf_name": "default", "job": 6, "event": "table_file_creation", "file_number": 36308, "file_size": 74546571, "table_properties": {"data_size": 67112125, "index_size": 690273, "filter_size": 6743183, "raw_key_size": 76215856, "raw_average_key_size": 28, "raw_value_size": 31365759, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:55:53.663034 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 6] Generated table #36309: 2638239 keys, 74546000 bytes >2018-07-11 05:55:53.663078 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288553663051, "cf_name": "default", "job": 6, "event": "table_file_creation", "file_number": 36309, "file_size": 74546000, "table_properties": {"data_size": 67112045, "index_size": 689797, "filter_size": 6743168, "raw_key_size": 76215793, "raw_average_key_size": 28, "raw_value_size": 31365730, "raw_average_value_size": 11, "num_data_blocks": 16489, "num_entries": 2638239, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:55:53.663114 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 6] Compacted 1@3 + 7@4 files to L4 => 596370722 bytes >2018-07-11 05:55:53.676759 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-05:55:53.676480) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[1 3 44 448 3643 792 0] max score 1.00, MB/sec: 22.4 rd, 22.4 wr, level 4, files in(1, 7) out(8) MB in(71.1, 497.7) out(568.7), read-write-amplify(16.0) write-amplify(8.0) OK, records in: 21105924, records dropped: 0 > >2018-07-11 05:55:53.676771 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-05:55:53.676507) EVENT_LOG_v1 {"time_micros": 1531288553676495, "job": 6, "event": "compaction_finished", "compaction_time_micros": 26616437, "output_level": 4, "num_output_files": 8, "total_output_size": 596370722, "num_input_records": 21105924, "num_output_records": 21105924, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 3, 44, 448, 3643, 792, 0]} >2018-07-11 05:55:53.677359 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288553677354, "job": 6, "event": "table_file_deletion", "file_number": 36300} >2018-07-11 05:55:53.677376 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288553677374, "job": 6, "event": "table_file_deletion", "file_number": 35779} >2018-07-11 05:55:53.677391 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288553677389, "job": 6, "event": "table_file_deletion", "file_number": 35778} >2018-07-11 05:55:53.677405 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288553677403, "job": 6, "event": "table_file_deletion", "file_number": 35777} >2018-07-11 05:55:53.677416 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288553677414, "job": 6, "event": "table_file_deletion", "file_number": 35771} >2018-07-11 05:55:53.677428 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288553677425, "job": 6, "event": "table_file_deletion", "file_number": 35749} >2018-07-11 05:55:53.677441 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288553677439, "job": 6, "event": "table_file_deletion", "file_number": 35748} >2018-07-11 05:55:53.677455 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288553677453, "job": 6, "event": "table_file_deletion", "file_number": 35740} >2018-07-11 05:55:53.684870 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-05:55:53.677624) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #36292 to level-4 74546475 bytes > >2018-07-11 05:55:53.684881 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-05:55:53.684788) EVENT_LOG_v1 {"time_micros": 1531288553684782, "job": 7, "event": "trivial_move", "destination_level": 4, "files": 1, "total_files_size": 74546475} >2018-07-11 05:55:53.684884 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-05:55:53.684797) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-4 74546475 bytes OK: base level 1 max bytes base 268435456 files[1 3 44 447 3644 792 0] max score 1.00 > >2018-07-11 05:55:53.693315 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-05:55:53.685465) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #36303 to level-5 74546982 bytes > >2018-07-11 05:55:53.693325 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-05:55:53.693232) EVENT_LOG_v1 {"time_micros": 1531288553693226, "job": 8, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546982} >2018-07-11 05:55:53.693327 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-05:55:53.693239) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546982 bytes OK: base level 1 max bytes base 268435456 files[1 3 44 447 3643 793 0] max score 1.00 > >2018-07-11 05:55:53.701652 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-05:55:53.693892) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #605 to level-5 74546836 bytes > >2018-07-11 05:55:53.701663 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-05:55:53.701576) EVENT_LOG_v1 {"time_micros": 1531288553701570, "job": 9, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546836} >2018-07-11 05:55:53.701679 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-05:55:53.701583) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546836 bytes OK: base level 1 max bytes base 268435456 files[1 3 44 447 3642 794 0] max score 1.00 > >2018-07-11 05:55:53.710014 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-05:55:53.702228) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #26722 to level-5 74546836 bytes > >2018-07-11 05:55:53.710024 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-05:55:53.709939) EVENT_LOG_v1 {"time_micros": 1531288553709933, "job": 10, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546836} >2018-07-11 05:55:53.710027 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-05:55:53.709947) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546836 bytes OK: base level 1 max bytes base 268435456 files[1 3 44 447 3641 795 0] max score 1.00 > >2018-07-11 05:55:53.710637 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 11] Compacting 1@3 + 2@4 files to L4, score 1.00 >2018-07-11 05:55:53.710648 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 10 Base level 3, inputs: [36301(71MB)], [35742(71MB) 36115(71MB)] > >2018-07-11 05:55:53.710674 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288553710651, "job": 11, "event": "compaction_started", "files_L3": [36301], "files_L4": [35742, 36115], "score": 1.00013, "input_data_size": 223639469} >2018-07-11 05:55:56.874449 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 11] Generated table #36310: 2638241 keys, 74546607 bytes >2018-07-11 05:55:56.874492 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288556874465, "cf_name": "default", "job": 11, "event": "table_file_creation", "file_number": 36310, "file_size": 74546607, "table_properties": {"data_size": 67112139, "index_size": 690295, "filter_size": 6743183, "raw_key_size": 76215850, "raw_average_key_size": 28, "raw_value_size": 31365753, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:55:59.963458 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 11] Generated table #36311: 2638242 keys, 74546184 bytes >2018-07-11 05:55:59.963504 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288559963485, "cf_name": "default", "job": 11, "event": "table_file_creation", "file_number": 36311, "file_size": 74546184, "table_properties": {"data_size": 67112176, "index_size": 689832, "filter_size": 6743186, "raw_key_size": 76215889, "raw_average_key_size": 28, "raw_value_size": 31365775, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638242, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:56:02.982272 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 11] Generated table #36312: 2638241 keys, 74545987 bytes >2018-07-11 05:56:02.982316 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288562982288, "cf_name": "default", "job": 11, "event": "table_file_creation", "file_number": 36312, "file_size": 74545987, "table_properties": {"data_size": 67112139, "index_size": 689675, "filter_size": 6743183, "raw_key_size": 76215850, "raw_average_key_size": 28, "raw_value_size": 31365753, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 05:56:02.982377 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 11] Compacted 1@3 + 2@4 files to L4 => 223638778 bytes >2018-07-11 05:56:02.993194 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-05:56:02.993054) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[1 3 44 446 3642 795 0] max score 1.00, MB/sec: 24.1 rd, 24.1 wr, level 4, files in(1, 2) out(3) MB in(71.1, 142.2) out(213.3), read-write-amplify(6.0) write-amplify(3.0) OK, records in: 7914724, records dropped: 0 > >2018-07-11 05:56:02.993204 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-05:56:02.993079) EVENT_LOG_v1 {"time_micros": 1531288562993066, "job": 11, "event": "compaction_finished", "compaction_time_micros": 9271690, "output_level": 4, "num_output_files": 3, "total_output_size": 223638778, "num_input_records": 7914724, "num_output_records": 7914724, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 3, 44, 446, 3642, 795, 0]} >2018-07-11 05:56:02.993791 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288562993787, "job": 11, "event": "table_file_deletion", "file_number": 36301} >2018-07-11 05:56:02.993808 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288562993806, "job": 11, "event": "table_file_deletion", "file_number": 36115} >2018-07-11 05:56:02.993823 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288562993820, "job": 11, "event": "table_file_deletion", "file_number": 35742} >2018-07-11 05:56:03.001666 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-05:56:02.994035) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #23812 to level-5 74546835 bytes > >2018-07-11 05:56:03.001676 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-05:56:03.001505) EVENT_LOG_v1 {"time_micros": 1531288563001498, "job": 12, "event": "trivial_move", "destination_level": 5, "files": 1, "total_files_size": 74546835} >2018-07-11 05:56:03.001679 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-05:56:03.001513) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-5 74546835 bytes OK: base level 1 max bytes base 268435456 files[1 3 44 446 3641 796 0] max score 1.00 > >2018-07-11 05:56:30.388852 7f354f8b1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #36313. Immutable memtables: 0. > >2018-07-11 05:56:30.475693 7f353f090700 4 rocksdb: (Original Log Time 2018/07/11-05:56:30.389105) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1 >2018-07-11 05:56:30.475711 7f353f090700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 13] Flushing memtable with next log file: 36313 > >2018-07-11 05:56:30.475733 7f353f090700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288590475724, "job": 13, "event": "flush_started", "num_memtables": 1, "num_entries": 5001019, "num_deletes": 18, "memory_usage": 293916992} >2018-07-11 05:56:30.475736 7f353f090700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 13] Level-0 flush table #36314: started >2018-07-11 05:56:34.390551 7f353f090700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531288594390529, "cf_name": "default", "job": 13, "event": "table_file_creation", "file_number": 36314, "file_size": 141944003, "table_properties": {"data_size": 127851501, "index_size": 1309370, "filter_size": 12782142, "raw_key_size": 144460207, "raw_average_key_size": 28, "raw_value_size": 60091647, "raw_average_value_size": 12, "num_data_blocks": 31276, "num_entries": 5000453, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "17", "kMergeOperands": "31"}} >2018-07-11 05:56:34.390584 7f353f090700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 13] Level-0 flush table #36314: 141944003 bytes OK >2018-07-11 05:56:34.400950 7f353f090700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:242] adding log 36297 to recycle list > >2018-07-11 05:56:34.400962 7f353f090700 4 rocksdb: (Original Log Time 2018/07/11-05:56:34.390615) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #36314 started >2018-07-11 05:56:34.400965 7f353f090700 4 rocksdb: (Original Log Time 2018/07/11-05:56:34.400717) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #36314: memtable #1 done >2018-07-11 05:56:34.400967 7f353f090700 4 rocksdb: (Original Log Time 2018/07/11-05:56:34.400742) EVENT_LOG_v1 {"time_micros": 1531288594400733, "job": 13, "event": "flush_finished", "lsm_state": [2, 3, 44, 446, 3641, 796, 0], "immutable_memtables": 0} >2018-07-11 05:56:34.400969 7f353f090700 4 rocksdb: (Original Log Time 2018/07/11-05:56:34.400889) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[2 3 44 446 3641 796 0] max score 1.00 > >2018-07-11 05:56:34.401486 7f353f090700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 13] Try to delete WAL files size 179870188, prev total WAL file size 179870200, number of live WAL files 2. > >2018-07-11 05:57:14.685943 7f354589d700 0 log_channel(cluster) log [DBG] : 5.d deep-scrub starts >2018-07-11 06:09:15.588671 7f354589d700 0 log_channel(cluster) log [DBG] : 5.d deep-scrub ok >2018-07-11 07:12:45.296867 7f355c88c700 0 -- 10.8.128.107:6801/12665 >> 10.8.128.114:6801/15919 conn(0x55a3302ad000 :6801 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg: challenging authorizer >2018-07-11 07:12:45.297060 7f355c88c700 0 -- 10.8.128.107:6801/12665 >> 10.8.128.114:6801/15919 conn(0x55a3302ad000 :6801 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg accept connect_seq 1 vs existing csq=1 existing_state=STATE_STANDBY >2018-07-11 07:12:45.297276 7f355c88c700 0 -- 10.8.128.107:6801/12665 >> 10.8.128.114:6801/15919 conn(0x55a3302ad000 :6801 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg accept connect_seq 2 vs existing csq=1 existing_state=STATE_STANDBY >2018-07-11 07:12:45.297543 7f355c88c700 0 -- 10.8.128.107:6801/12665 >> 10.8.128.114:6801/15919 conn(0x55a317017800 :-1 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=15 cs=1 l=0).handle_connect_msg: challenging authorizer >2018-07-11 07:13:48.019953 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.020399 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.020792 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.021188 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.021609 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.021978 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.022359 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.022737 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.023129 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.023567 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.023946 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.024357 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.024775 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.025178 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.025607 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.026002 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.162164 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.162616 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.163012 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.163404 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.163790 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.164180 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.164588 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.164982 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.165368 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.165756 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.166152 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.166542 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.166970 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.167372 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.167773 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.168170 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.295688 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.296094 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.296491 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.296875 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.297298 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.297674 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.298074 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.298470 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.298848 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.299285 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.299685 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.300085 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.300534 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.300912 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.301308 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.301691 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.445878 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.446299 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.446696 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.447093 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.447495 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.447879 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.448296 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.448700 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.449097 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.449512 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.449928 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.450337 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.450735 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.451126 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.451553 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.451950 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.579417 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.579832 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.580273 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.580677 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.581077 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.581470 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.581858 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.582263 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.582670 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.583078 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.583488 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.583898 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.584296 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.584696 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.585096 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.585479 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.770194 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.770648 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.771043 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.771434 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.771816 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.772217 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.772658 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.773053 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.773446 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.773832 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.774277 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.774671 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.775065 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.775449 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.775847 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.776279 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.858123 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.858548 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.858953 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.859355 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.859744 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.860141 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.860555 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.860955 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.861351 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.861748 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.862146 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.862558 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.862965 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.863361 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.863755 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.864157 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.990400 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.990806 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.991193 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.991609 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.992021 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.992417 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.992817 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.993207 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.993607 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.994000 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.994388 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.994787 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.995181 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.995576 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.995972 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:48.996377 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.130917 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.131344 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.131737 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.132126 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.132529 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.132915 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.133307 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.133682 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.134085 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.134492 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.134883 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.135272 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.135655 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.136045 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.136438 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.136817 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.261867 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.262294 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.262695 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.263085 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.263484 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.263858 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.264271 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.264647 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.265038 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.265434 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.265809 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.266203 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.266630 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.267009 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.267418 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.267813 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.394088 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.394514 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.394917 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.395304 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.395687 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.396075 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.396465 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.396842 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.397232 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.397635 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.398023 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.398427 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.398839 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.399228 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.399653 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.400047 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.526325 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.526736 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.527124 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.527508 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.527921 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.528297 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.528678 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.529067 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.529468 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.529878 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.530274 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.530676 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.531084 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.531483 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.531881 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.532305 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.658539 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.658943 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.659329 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.659709 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.660105 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.660488 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.660864 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.661249 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.661633 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.662054 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.662459 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.662852 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.663265 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.663640 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.664039 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.664424 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.784405 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.784816 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.785206 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.785606 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.785991 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.786355 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.786733 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.787106 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.787502 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.787898 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.788282 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.788660 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.789057 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.789439 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.789831 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.790217 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.902385 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.902793 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.903184 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.903581 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.903977 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.904362 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.904746 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.905136 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.905534 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.905934 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.906318 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.906709 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.907115 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.907508 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.907907 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:49.908293 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.021628 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.022029 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.022412 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.022790 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.023185 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.023575 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.023977 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.024372 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.024770 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.025166 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.025582 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.025989 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.026380 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.026752 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.027150 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.027552 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.139602 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.140005 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.140391 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.140770 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.141165 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.141556 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.141950 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.142336 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.142750 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.143149 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.143560 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.143953 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.144345 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.144719 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.145115 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.145508 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.257618 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.258023 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.258402 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.258780 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.259174 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.259566 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.259966 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.260348 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.260742 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.261138 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.261532 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.261931 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.262342 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.262731 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.263128 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.263511 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.376880 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.377300 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.377698 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.378089 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.378487 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.378859 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.379272 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.379659 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.380073 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.380485 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.380870 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.381254 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.381649 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.382050 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.382455 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.382849 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.494867 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.495298 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.495676 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.496078 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.496491 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.496878 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.497271 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.497646 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.498033 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.498426 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.498800 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.499216 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.499626 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.499988 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.500367 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.500745 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.614165 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.614657 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.615059 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.615452 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.615834 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.616209 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.616600 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.616981 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.617358 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.617742 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.618114 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.618500 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.618879 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.619262 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.619646 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.620028 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.732186 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.732588 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.732963 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.733351 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:50.733751 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:13:57.355068 7f354f8b1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:684] reusing log 36297 from recycle list > >2018-07-11 07:13:57.355407 7f354f8b1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #36315. Immutable memtables: 0. > >2018-07-11 07:13:57.409571 7f353f090700 4 rocksdb: (Original Log Time 2018/07/11-07:13:57.355442) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 1 >2018-07-11 07:13:57.409585 7f353f090700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 14] Flushing memtable with next log file: 36315 > >2018-07-11 07:13:57.409603 7f353f090700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293237409593, "job": 14, "event": "flush_started", "num_memtables": 1, "num_entries": 32000046, "num_deletes": 29000030, "memory_usage": 1413090336} >2018-07-11 07:13:57.409606 7f353f090700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 14] Level-0 flush table #36316: started >2018-07-11 07:14:14.516428 7f353f090700 1 bluefs _allocate failed to allocate 0x21200000 on bdev 1, free 0x105fe000; fallback to bdev 2 >2018-07-11 07:14:22.357652 7f353f090700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293262357633, "cf_name": "default", "job": 14, "event": "table_file_creation", "file_number": 36316, "file_size": 554869144, "table_properties": {"data_size": 468920311, "index_size": 4868600, "filter_size": 81079235, "raw_key_size": 924446946, "raw_average_key_size": 28, "raw_value_size": 35672787, "raw_average_value_size": 1, "num_data_blocks": 115098, "num_entries": 32000046, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "29000030", "kMergeOperands": "0"}} >2018-07-11 07:14:22.357681 7f353f090700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 14] Level-0 flush table #36316: 554869144 bytes OK >2018-07-11 07:14:22.369255 7f353f090700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:242] adding log 36313 to recycle list > >2018-07-11 07:14:22.369267 7f353f090700 4 rocksdb: (Original Log Time 2018/07/11-07:14:22.357694) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #36316 started >2018-07-11 07:14:22.369271 7f353f090700 4 rocksdb: (Original Log Time 2018/07/11-07:14:22.369107) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #36316: memtable #1 done >2018-07-11 07:14:22.369273 7f353f090700 4 rocksdb: (Original Log Time 2018/07/11-07:14:22.369125) EVENT_LOG_v1 {"time_micros": 1531293262369117, "job": 14, "event": "flush_finished", "lsm_state": [3, 3, 44, 446, 3641, 796, 0], "immutable_memtables": 0} >2018-07-11 07:14:22.369275 7f353f090700 4 rocksdb: (Original Log Time 2018/07/11-07:14:22.369161) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[3 3 44 446 3641 796 0] max score 4.46 > >2018-07-11 07:14:22.369578 7f354e0ae700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS ------- >2018-07-11 07:14:22.369587 7f354e0ae700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] >** DB Stats ** >Uptime(secs): 4816.6 total, 4743.0 interval >Cumulative writes: 878 writes, 37M keys, 878 commit groups, 1.0 writes per commit group, ingest: 0.89 GB, 0.19 MB/s >Cumulative WAL: 878 writes, 243 syncs, 3.60 writes per sync, written: 0.89 GB, 0.19 MB/s >Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent >Interval writes: 878 writes, 37M keys, 878 commit groups, 1.0 writes per commit group, ingest: 906.93 MB, 0.19 MB/s >Interval WAL: 878 writes, 243 syncs, 3.60 writes per sync, written: 0.89 MB, 0.19 MB/s >Interval stall: 00:00:0.000 H:M:S, 0.0 percent > >** Compaction Stats [default] ** >Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop >---------------------------------------------------------------------------------------------------------------------------------------------------------- > L0 3/0 745.44 MB 4.5 0.0 0.0 0.0 0.7 0.7 0.0 1.0 0.0 23.9 31 3 10.401 0 0 > L1 3/0 190.97 MB 0.7 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 > L2 44/0 2.47 GB 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 > L3 446/0 24.93 GB 1.0 0.1 0.1 0.1 0.1 0.1 0.1 2.0 18.6 18.6 8 1 7.636 5276K 0 > L4 3641/0 250.00 GB 1.0 0.8 0.1 0.6 0.8 0.1 0.1 5.5 21.8 21.8 36 2 17.944 29M 0 > L5 796/0 55.24 GB 0.0 0.0 0.0 0.0 0.0 0.0 0.3 0.0 0.0 0.0 0 0 0.000 0 0 > Sum 4933/0 333.56 GB 0.0 0.9 0.2 0.7 1.6 0.9 0.5 2.2 12.4 22.3 75 6 12.455 34M 0 > Int 0/0 0.00 KB 0.0 0.9 0.2 0.7 1.6 0.9 0.5 2.4 12.8 21.9 72 5 14.478 34M 0 >Uptime(secs): 4816.6 total, 4816.6 interval >Flush(GB): cumulative 0.728, interval 0.649 >AddFile(GB): cumulative 0.000, interval 0.000 >AddFile(Total Files): cumulative 0, interval 0 >AddFile(L0 Files): cumulative 0, interval 0 >AddFile(Keys): cumulative 0, interval 0 >Cumulative compaction: 1.63 GB write, 0.35 MB/s write, 0.90 GB read, 0.19 MB/s read, 74.7 seconds >Interval compaction: 1.55 GB write, 0.33 MB/s write, 0.90 GB read, 0.19 MB/s read, 72.4 seconds >Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count > >** File Read Latency Histogram By Level [default] ** > >** Compaction Stats [default] ** >Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop >---------------------------------------------------------------------------------------------------------------------------------------------------------- > L0 3/0 745.44 MB 4.5 0.0 0.0 0.0 0.7 0.7 0.0 1.0 0.0 23.9 31 3 10.401 0 0 > L1 3/0 190.97 MB 0.7 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 > L2 44/0 2.47 GB 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 > L3 446/0 24.93 GB 1.0 0.1 0.1 0.1 0.1 0.1 0.1 2.0 18.6 18.6 8 1 7.636 5276K 0 > L4 3641/0 250.00 GB 1.0 0.8 0.1 0.6 0.8 0.1 0.1 5.5 21.8 21.8 36 2 17.944 29M 0 > L5 796/0 55.24 GB 0.0 0.0 0.0 0.0 0.0 0.0 0.3 0.0 0.0 0.0 0 0 0.000 0 0 > Sum 4933/0 333.56 GB 0.0 0.9 0.2 0.7 1.6 0.9 0.5 2.2 12.4 22.3 75 6 12.455 34M 0 > Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 >Uptime(secs): 4816.6 total, 4816.6 interval >Flush(GB): cumulative 0.728, interval 0.000 >AddFile(GB): cumulative 0.000, interval 0.000 >AddFile(Total Files): cumulative 0, interval 0 >AddFile(L0 Files): cumulative 0, interval 0 >AddFile(Keys): cumulative 0, interval 0 >Cumulative compaction: 1.63 GB write, 0.35 MB/s write, 0.90 GB read, 0.19 MB/s read, 74.7 seconds >Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds >Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count > >** File Read Latency Histogram By Level [default] ** > >2018-07-11 07:14:22.369711 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 15] Compacting 3@0 + 3@1 files to L1, score 4.46 >2018-07-11 07:14:22.369723 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 14 Base level 0, inputs: [36316(529MB) 36314(135MB) 36295(80MB)], [36275(71MB) 36280(65MB) 36281(54MB)] > >2018-07-11 07:14:22.369739 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293262369727, "job": 15, "event": "compaction_started", "files_L0": [36316, 36314, 36295], "files_L1": [36275, 36280, 36281], "score": 4.46158, "input_data_size": 981896416} >2018-07-11 07:14:22.369930 7f353f090700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 14] Try to delete WAL files size 771119652, prev total WAL file size 771119664, number of live WAL files 2. > >2018-07-11 07:14:27.689432 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 15] Generated table #36317: 4888760 keys, 80173883 bytes >2018-07-11 07:14:27.689470 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293267689454, "cf_name": "default", "job": 15, "event": "table_file_creation", "file_number": 36317, "file_size": 80173883, "table_properties": {"data_size": 67110185, "index_size": 685246, "filter_size": 12377461, "raw_key_size": 141267959, "raw_average_key_size": 28, "raw_value_size": 916542, "raw_average_value_size": 0, "num_data_blocks": 16336, "num_entries": 4888760, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4886796", "kMergeOperands": "0"}} >2018-07-11 07:14:32.979815 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 15] Generated table #36318: 4956341 keys, 80346473 bytes >2018-07-11 07:14:32.979856 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293272979840, "cf_name": "default", "job": 15, "event": "table_file_creation", "file_number": 36318, "file_size": 80346473, "table_properties": {"data_size": 67109658, "index_size": 689184, "filter_size": 12546640, "raw_key_size": 143183285, "raw_average_key_size": 28, "raw_value_size": 10824, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4956341, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4956286", "kMergeOperands": "0"}} >2018-07-11 07:14:38.364593 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 15] Generated table #36319: 4957165 keys, 80348269 bytes >2018-07-11 07:14:38.364628 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293278364613, "cf_name": "default", "job": 15, "event": "table_file_creation", "file_number": 36319, "file_size": 80348269, "table_properties": {"data_size": 67109369, "index_size": 689209, "filter_size": 12548701, "raw_key_size": 143206986, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957165, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957165", "kMergeOperands": "0"}} >2018-07-11 07:14:42.927001 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4d00000 on bdev 1, free 0x1efe000; fallback to bdev 2 >2018-07-11 07:14:44.080451 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 15] Generated table #36320: 4957165 keys, 80348260 bytes >2018-07-11 07:14:44.080504 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293284080487, "cf_name": "default", "job": 15, "event": "table_file_creation", "file_number": 36320, "file_size": 80348260, "table_properties": {"data_size": 67109357, "index_size": 689212, "filter_size": 12548701, "raw_key_size": 143207000, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957165, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957165", "kMergeOperands": "0"}} >2018-07-11 07:14:48.661199 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4d00000 on bdev 1, free 0x1efe000; fallback to bdev 2 >2018-07-11 07:14:49.907626 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 15] Generated table #36321: 4957165 keys, 80348267 bytes >2018-07-11 07:14:49.907682 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293289907663, "cf_name": "default", "job": 15, "event": "table_file_creation", "file_number": 36321, "file_size": 80348267, "table_properties": {"data_size": 67109361, "index_size": 689215, "filter_size": 12548701, "raw_key_size": 143206991, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957165, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957165", "kMergeOperands": "0"}} >2018-07-11 07:14:54.183654 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4c00000 on bdev 1, free 0x1efe000; fallback to bdev 2 >2018-07-11 07:14:55.407075 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 15] Generated table #36322: 4642807 keys, 79563212 bytes >2018-07-11 07:14:55.407124 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293295407108, "cf_name": "default", "job": 15, "event": "table_file_creation", "file_number": 36322, "file_size": 79563212, "table_properties": {"data_size": 67111181, "index_size": 689339, "filter_size": 11761699, "raw_key_size": 134125492, "raw_average_key_size": 28, "raw_value_size": 4253311, "raw_average_value_size": 0, "num_data_blocks": 16473, "num_entries": 4642807, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4285430", "kMergeOperands": "0"}} >2018-07-11 07:14:57.555664 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4800000 on bdev 1, free 0x1efe000; fallback to bdev 2 >2018-07-11 07:14:58.639236 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 15] Generated table #36323: 2638241 keys, 74546842 bytes >2018-07-11 07:14:58.639288 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293298639273, "cf_name": "default", "job": 15, "event": "table_file_creation", "file_number": 36323, "file_size": 74546842, "table_properties": {"data_size": 67112160, "index_size": 690509, "filter_size": 6743183, "raw_key_size": 76215849, "raw_average_key_size": 28, "raw_value_size": 31365752, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:15:00.727115 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4800000 on bdev 1, free 0x1efe000; fallback to bdev 2 >2018-07-11 07:15:01.772317 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 15] Generated table #36324: 2638241 keys, 74546976 bytes >2018-07-11 07:15:01.772363 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293301772348, "cf_name": "default", "job": 15, "event": "table_file_creation", "file_number": 36324, "file_size": 74546976, "table_properties": {"data_size": 67112123, "index_size": 690680, "filter_size": 6743183, "raw_key_size": 76215850, "raw_average_key_size": 28, "raw_value_size": 31365753, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:15:03.891913 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4800000 on bdev 1, free 0x1efe000; fallback to bdev 2 >2018-07-11 07:15:05.007059 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 15] Generated table #36325: 2638241 keys, 74546525 bytes >2018-07-11 07:15:05.007114 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293305007099, "cf_name": "default", "job": 15, "event": "table_file_creation", "file_number": 36325, "file_size": 74546525, "table_properties": {"data_size": 67112157, "index_size": 690195, "filter_size": 6743183, "raw_key_size": 76215862, "raw_average_key_size": 28, "raw_value_size": 31365765, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:15:07.134644 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4800000 on bdev 1, free 0x1efe000; fallback to bdev 2 >2018-07-11 07:15:08.098762 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 15] Generated table #36326: 2638241 keys, 74546353 bytes >2018-07-11 07:15:08.098810 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293308098781, "cf_name": "default", "job": 15, "event": "table_file_creation", "file_number": 36326, "file_size": 74546353, "table_properties": {"data_size": 67112093, "index_size": 690087, "filter_size": 6743183, "raw_key_size": 76215850, "raw_average_key_size": 28, "raw_value_size": 31365753, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:15:10.431008 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4800000 on bdev 1, free 0x1efe000; fallback to bdev 2 >2018-07-11 07:15:11.211960 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 15] Generated table #36327: 2638241 keys, 74546010 bytes >2018-07-11 07:15:11.211993 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293311211979, "cf_name": "default", "job": 15, "event": "table_file_creation", "file_number": 36327, "file_size": 74546010, "table_properties": {"data_size": 67112172, "index_size": 689665, "filter_size": 6743183, "raw_key_size": 76215859, "raw_average_key_size": 28, "raw_value_size": 31365762, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:15:11.328163 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 15] Generated table #36328: 70201 keys, 1993548 bytes >2018-07-11 07:15:11.328188 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293311328176, "cf_name": "default", "job": 15, "event": "table_file_creation", "file_number": 36328, "file_size": 1993548, "table_properties": {"data_size": 1795054, "index_size": 18054, "filter_size": 179459, "raw_key_size": 2030104, "raw_average_key_size": 28, "raw_value_size": 841046, "raw_average_value_size": 11, "num_data_blocks": 442, "num_entries": 70201, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "8", "kMergeOperands": "0"}} >2018-07-11 07:15:11.878412 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4200000 on bdev 1, free 0x1cfe000; fallback to bdev 2 >2018-07-11 07:15:12.582830 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 15] Generated table #36329: 165700 keys, 68239299 bytes >2018-07-11 07:15:12.582871 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293312582857, "cf_name": "default", "job": 15, "event": "table_file_creation", "file_number": 36329, "file_size": 68239299, "table_properties": {"data_size": 67111950, "index_size": 558626, "filter_size": 567735, "raw_key_size": 15329313, "raw_average_key_size": 92, "raw_value_size": 53152876, "raw_average_value_size": 320, "num_data_blocks": 15089, "num_entries": 165700, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "80930", "kMergeOperands": "0"}} >2018-07-11 07:15:13.129457 7f354e0ae700 1 bluefs _allocate failed to allocate 0x3700000 on bdev 1, free 0x1cfe000; fallback to bdev 2 >2018-07-11 07:15:13.659715 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 15] Generated table #36330: 139042 keys, 57577508 bytes >2018-07-11 07:15:13.659757 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293313659742, "cf_name": "default", "job": 15, "event": "table_file_creation", "file_number": 36330, "file_size": 57577508, "table_properties": {"data_size": 56615228, "index_size": 483124, "filter_size": 478168, "raw_key_size": 6822089, "raw_average_key_size": 49, "raw_value_size": 50306530, "raw_average_value_size": 361, "num_data_blocks": 13608, "num_entries": 139042, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "58266"}} >2018-07-11 07:15:13.659836 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 15] Compacted 3@0 + 3@1 files to L1 => 981671425 bytes >2018-07-11 07:15:13.668339 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:15:13.668089) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 14 44 446 3641 796 0] max score 5.36, MB/sec: 19.1 rd, 19.1 wr, level 1, files in(3, 3) out(14) MB in(745.4, 191.0) out(936.2), read-write-amplify(2.5) write-amplify(1.3) OK, records in: 42925734, records dropped: 146 > >2018-07-11 07:15:13.668350 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:15:13.668122) EVENT_LOG_v1 {"time_micros": 1531293313668102, "job": 15, "event": "compaction_finished", "compaction_time_micros": 51290076, "output_level": 1, "num_output_files": 14, "total_output_size": 981671425, "num_input_records": 42925697, "num_output_records": 42925551, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 14, 44, 446, 3641, 796, 0]} >2018-07-11 07:15:13.669319 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 16] Compacting 1@1 files to L2, score 5.36 >2018-07-11 07:15:13.669330 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 15 Base level 1, inputs: [36319(76MB)] > >2018-07-11 07:15:13.669343 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293313669333, "job": 16, "event": "compaction_started", "files_L1": [36319], "score": 5.36412, "input_data_size": 80348269} >2018-07-11 07:15:14.002938 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 16] Generated table #36331: 156925 keys, 2543995 bytes >2018-07-11 07:15:14.002983 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293314002967, "cf_name": "default", "job": 16, "event": "table_file_creation", "file_number": 36331, "file_size": 2543995, "table_properties": {"data_size": 2124426, "index_size": 21336, "filter_size": 397248, "raw_key_size": 4533394, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 522, "num_entries": 156925, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "156925", "kMergeOperands": "0"}} >2018-07-11 07:15:15.289922 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293315289904, "job": 0, "event": "table_file_deletion", "file_number": 36316} >2018-07-11 07:15:15.289938 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293315289935, "job": 0, "event": "table_file_deletion", "file_number": 36314} >2018-07-11 07:15:15.289949 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293315289947, "job": 0, "event": "table_file_deletion", "file_number": 36295} >2018-07-11 07:15:15.289960 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293315289958, "job": 0, "event": "table_file_deletion", "file_number": 36281} >2018-07-11 07:15:15.289972 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293315289970, "job": 0, "event": "table_file_deletion", "file_number": 36280} >2018-07-11 07:15:15.289985 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293315289983, "job": 0, "event": "table_file_deletion", "file_number": 36275} >2018-07-11 07:15:18.538818 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 16] Generated table #36332: 4800240 keys, 77804699 bytes >2018-07-11 07:15:18.538870 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293318538851, "cf_name": "default", "job": 16, "event": "table_file_creation", "file_number": 36332, "file_size": 77804699, "table_properties": {"data_size": 64984865, "index_size": 667399, "filter_size": 12151446, "raw_key_size": 138673592, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 15948, "num_entries": 4800240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4800240", "kMergeOperands": "0"}} >2018-07-11 07:15:18.538898 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 16] Compacted 1@1 files to L2 => 80348694 bytes >2018-07-11 07:15:18.551577 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:15:18.551438) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 13 46 446 3641 796 0] max score 4.77, MB/sec: 16.5 rd, 16.5 wr, level 2, files in(1, 0) out(2) MB in(76.6, 0.0) out(76.6), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 4957165, records dropped: 0 > >2018-07-11 07:15:18.551589 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:15:18.551463) EVENT_LOG_v1 {"time_micros": 1531293318551452, "job": 16, "event": "compaction_finished", "compaction_time_micros": 4869544, "output_level": 2, "num_output_files": 2, "total_output_size": 80348694, "num_input_records": 4957165, "num_output_records": 4957165, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 13, 46, 446, 3641, 796, 0]} >2018-07-11 07:15:18.552566 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 17] Compacting 1@1 files to L2, score 4.77 >2018-07-11 07:15:18.552577 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 16 Base level 1, inputs: [36321(76MB)] > >2018-07-11 07:15:18.552603 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293318552593, "job": 17, "event": "compaction_started", "files_L1": [36321], "score": 4.76933, "input_data_size": 80348267} >2018-07-11 07:15:20.474126 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293320474108, "job": 0, "event": "table_file_deletion", "file_number": 36319} >2018-07-11 07:15:23.604038 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.604484 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.604878 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.605408 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.606093 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.606758 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.607424 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.608084 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.608752 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.734356 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.735117 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.735888 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.736642 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.737402 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.738118 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.738857 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.739565 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.740295 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.740988 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.741697 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.742403 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.743145 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.743866 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.744539 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.875162 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.875607 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.875988 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.876378 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.876763 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.876771 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:23.877160 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.877168 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:23.877602 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.877621 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:23.878027 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.878035 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:23.878441 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.878449 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:23.878850 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.878857 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:23.879228 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.879236 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:23.879668 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.879676 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:23.880095 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.880102 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:23.880510 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.880517 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:23.880910 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.880917 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:23.881306 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.881316 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:23.881713 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.881721 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:23.882122 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.882129 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:23.882558 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.882566 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:23.882984 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.882992 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:23.883409 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.883417 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:23.883803 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.883810 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:23.884195 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.884202 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:23.884593 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.884600 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:23.884963 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.884971 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:23.885369 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.885376 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:23.885751 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.885758 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:23.886122 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.886129 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:23.886525 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.886533 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:23.886918 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.886925 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:23.887318 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.887326 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:23.887706 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.887729 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:23.888104 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.888111 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:23.888514 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.888522 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:23.888903 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.888910 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:23.889282 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.889289 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:23.889670 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.889678 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:23.890073 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.890081 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:23.890484 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.890492 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:23.890873 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.890880 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:23.891254 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.891261 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:23.891636 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.891643 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:23.892029 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.892036 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:23.892440 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.892448 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:23.892832 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.892840 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:23.893202 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.893210 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:23.893634 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.893642 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:23.894052 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.894072 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:23.894469 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.894476 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:23.894908 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.894916 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:23.895348 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.895355 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:23.895748 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.895755 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:23.896126 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.896133 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:23.896531 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.896539 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:23.897222 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.897242 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:23.897932 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.897939 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:23.898623 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.898630 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:23.899328 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.899335 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:23.900016 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.900023 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:23.900693 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.900700 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:23.901382 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.901389 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:23.901774 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.901782 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:23.902148 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.902155 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:23.902561 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.902569 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:23.902934 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.902941 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:23.903342 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.903350 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:23.903731 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.903738 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:23.904102 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.904109 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:23.904510 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.904517 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:23.904896 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.904903 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:23.905278 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.905285 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:23.905647 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.905654 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:23.906015 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.906023 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:23.906393 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.906400 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:23.906779 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.906787 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:23.907191 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.907198 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:23.907622 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.907630 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:23.908000 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.908007 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:23.908410 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.908418 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:23.908804 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.908811 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:23.909178 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.909185 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:23.909557 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.909564 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:23.909939 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.909946 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:23.910641 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:23.910648 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:24.100908 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:24.100918 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:24.101618 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:24.101624 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:24.102106 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:24.102113 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:24.102509 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:24.102528 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:24.102933 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:24.102940 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:24.103342 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:24.103349 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:24.103741 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:24.103748 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:24.104133 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:24.104140 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:24.104535 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:24.104541 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:24.104941 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:24.104948 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:24.105333 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:24.105340 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:24.105730 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:24.105737 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:24.106122 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:24.106129 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:24.106530 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:24.106537 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:24.106938 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:24.106945 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:24.277219 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:24.277226 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:24.277735 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:24.277744 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:24.278286 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:24.278295 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:24.278694 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:24.278702 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:24.279076 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:24.279083 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:24.279465 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:24.279472 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:24.279861 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:24.279868 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:24.280261 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:24.280268 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:24.280653 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:24.280660 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:24.281063 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:24.281070 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:24.281460 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:24.281468 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:24.281852 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:24.281859 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:24.282244 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:24.282256 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:24.282630 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:24.282636 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:24.283026 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:24.283033 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:24.283432 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:24.283439 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:24.466692 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:24.466703 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:24.467107 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:24.467114 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:24.467534 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:24.467541 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:24.467932 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:24.467939 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:24.468321 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:24.468328 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:24.468720 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:24.468727 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:24.469149 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:24.469156 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:24.469571 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:24.469594 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:24.469964 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:24.469971 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:24.470353 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:24.470360 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:24.470729 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:24.470736 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:24.471147 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:24.471154 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:24.471562 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:24.471570 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:24.471964 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:24.471971 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:24.472382 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:24.472390 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:24.697362 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:24.697372 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:24.697768 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:24.697775 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:24.698166 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:24.698186 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:24.698586 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:24.698593 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:24.698979 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:24.698986 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:24.699367 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:24.699374 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:24.699746 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:24.699752 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:24.700145 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:24.700153 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:24.700539 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:24.700546 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:24.700931 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:24.700938 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:24.701340 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:24.701347 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:24.701727 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:24.701734 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:24.702133 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:24.702140 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:24.702548 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:24.702555 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:24.702950 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:24.702959 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:24.854313 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:24.854324 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:24.854719 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:24.854726 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:24.855114 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:24.855121 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:24.855505 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:24.855512 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:24.855895 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:24.855915 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:24.856308 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:24.856316 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:24.856708 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:24.856715 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:24.857108 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:24.857116 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:24.857529 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:24.857536 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:24.857921 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:24.857928 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:24.858331 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:24.858338 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:24.858730 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:24.858737 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:24.859119 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:24.859126 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:24.859509 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:24.859528 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:24.859916 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:24.859923 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.058793 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.058805 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.059204 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.059210 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.059606 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.059613 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.060018 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.060025 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.060424 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.060431 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.060816 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.060823 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.061222 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.061229 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.061644 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.061651 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.062024 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.062030 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.062412 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.062418 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.062800 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.062807 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.063208 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.063215 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.063610 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.063617 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.063986 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.063992 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.064376 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.064383 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.064766 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.064772 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.138749 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.138762 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.139168 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.139175 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.139581 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.139589 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.139995 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.140002 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.140400 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.140407 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.140790 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.140797 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.141199 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.141206 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.141603 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.141611 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.141983 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.141990 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.142375 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.142381 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.142765 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.142773 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.143174 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.143181 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.143594 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.143601 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.143985 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.143992 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.144374 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.144380 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.144764 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.144771 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.313937 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.313950 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.314361 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.314369 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.314752 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.314759 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.315156 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.315163 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.315576 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.315584 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.315975 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.315982 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.316378 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.316385 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.316768 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.316775 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.317168 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.317175 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.317587 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.317594 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.317991 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.317998 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.318402 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.318410 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.318802 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.318809 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.319203 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.319211 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.319610 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.319618 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.532850 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.532861 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.533293 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.533302 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.533698 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.533705 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.534088 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.534094 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.534475 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.534482 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.534880 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.534887 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.535275 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.535282 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.535676 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.535683 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.536076 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.536082 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.536464 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.536471 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.536861 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.536868 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.537289 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.537296 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.537691 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.537698 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.538089 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.538096 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.692096 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.692107 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.692557 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.692564 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.692955 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.692962 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.693364 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.693374 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.693775 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.693781 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.694163 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.694170 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.694560 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.694567 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.694955 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.694962 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.695341 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.695348 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.695720 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.695727 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.696112 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.696119 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.696537 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.696543 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.696937 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.696944 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.697322 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.697329 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.697721 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.697728 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.698134 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.698141 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.859694 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.859704 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.860096 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.860103 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.860510 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.860530 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.860912 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.860921 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.861324 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.861331 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.861717 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.861724 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.862106 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.862112 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.862532 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.862539 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.862937 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.862944 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.863363 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.863371 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.863761 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.863768 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.864152 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.864158 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.864560 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.864567 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.864954 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.864961 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:25.865344 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:25.865351 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.078336 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.078347 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.078752 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.078759 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.079152 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.079159 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.079575 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.079582 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.079965 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.079974 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.080380 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.080387 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.080775 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.080781 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.081165 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.081171 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.081587 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.081594 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.081973 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.081980 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.082364 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.082371 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.082767 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.082774 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.083168 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.083175 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.083586 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.083592 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.083983 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.083990 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.084377 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.084384 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.276640 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.276650 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.277034 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.277041 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.277428 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.277435 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.277809 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.277816 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.278202 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.278210 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.278617 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.278625 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.279004 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.279011 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.279399 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.279406 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.279799 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.279805 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.280187 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.280193 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.280600 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.280607 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.281028 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.281034 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.281431 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.281438 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.281834 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.281841 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.282228 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.282235 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.349468 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.349480 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.349884 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.349891 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.350290 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.350297 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.350686 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.350693 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.351095 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.351102 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.351537 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.351548 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.351998 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.352006 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.352388 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.352395 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.352786 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.352793 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.353177 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.353184 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.353570 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.353577 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.353986 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.353993 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.354385 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.354392 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.354778 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.354785 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.355174 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.355181 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.355561 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.355568 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.597855 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.597867 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.598270 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.598278 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.598661 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.598668 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.599068 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.599075 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.599471 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.599478 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.599847 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.599855 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.600261 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.600268 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.600648 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.600655 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.601056 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.601063 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.601458 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.601465 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.601846 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.601853 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.602254 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.602261 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.602645 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.602652 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.603046 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.603053 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.603456 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.603463 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.729944 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.729955 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.730376 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.730383 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.730776 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.730784 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.731168 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.731175 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.731560 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.731567 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.731945 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.731952 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.732343 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.732352 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.732745 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.732753 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.733147 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.733154 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.733558 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.733565 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.733955 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.733962 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.734356 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.734364 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.734760 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.734766 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.735154 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.735161 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.735572 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.735591 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.942586 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.942597 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.943007 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.943014 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.943417 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.943425 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.943820 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.943827 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.944211 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.944218 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.944639 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.944647 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.945058 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.945067 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.945446 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.945453 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.945848 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.945855 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.946235 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.946241 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.946666 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.946673 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.947082 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.947089 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.947480 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.947488 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.947874 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.947881 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.948298 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.948306 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:26.948692 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:26.948698 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:27.277385 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:27.277396 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:27.277791 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:27.277798 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:27.278188 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:27.278194 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:27.278592 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:27.278599 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:27.278986 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:27.278993 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:27.279373 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:27.279380 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:27.279750 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:27.279757 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:27.280135 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:27.280142 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:27.280540 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:27.280547 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:27.280926 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:27.280933 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:27.281314 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:27.281321 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:27.281697 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:27.281704 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:27.282090 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:27.282096 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:27.282487 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:27.282494 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:27.282876 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:27.282883 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:27.503683 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:27.503693 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:27.504086 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:27.504093 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:27.504478 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:27.504485 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:27.504860 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:27.504867 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:27.505252 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:27.505259 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:27.505630 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:27.505637 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:27.506024 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:27.506030 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:27.506409 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:27.506416 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:27.506780 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:27.506787 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:27.507216 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:27.507223 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:27.507629 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:27.507636 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:27.508079 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:27.508086 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:27.508482 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:27.508489 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:27.508873 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:27.508880 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:27.509288 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:27.509295 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:27.619263 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 17] Generated table #36333: 4957165 keys, 80348267 bytes >2018-07-11 07:15:27.619316 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293327619287, "cf_name": "default", "job": 17, "event": "table_file_creation", "file_number": 36333, "file_size": 80348267, "table_properties": {"data_size": 67109361, "index_size": 689215, "filter_size": 12548701, "raw_key_size": 143206991, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957165, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957165", "kMergeOperands": "0"}} >2018-07-11 07:15:27.753417 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 17] Compacted 1@1 files to L2 => 80348267 bytes >2018-07-11 07:15:27.753578 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:27.753590 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:27.753974 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:27.753981 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:27.754372 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:27.754379 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:27.754749 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:27.754756 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:27.755155 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:27.755162 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:27.755593 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:27.755600 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:27.755980 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:27.755987 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:27.756385 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:27.756393 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:27.756784 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:27.756790 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:27.757168 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:27.757175 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:27.757575 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:27.757582 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:27.757980 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:27.757987 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:27.758381 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:27.758388 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:27.758781 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:27.758788 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:27.759167 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:27.759174 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:27.841184 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:27.841195 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:27.841615 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:27.841623 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:27.841982 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:27.841988 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:27.842361 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:27.842368 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:27.842738 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:27.842744 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:27.843136 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:27.843143 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:27.843545 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:27.843552 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:27.843940 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:27.843946 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:27.844332 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:27.844340 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:27.844715 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:27.844722 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:27.845108 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:27.845115 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:27.845539 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:27.845547 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:27.845947 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:27.845954 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:27.846352 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:27.846360 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:27.846749 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:27.846755 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:27.847136 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:27.847143 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:28.044443 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:28.044454 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:28.044858 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:28.044865 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:28.045239 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:28.045256 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:28.045634 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:28.045641 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:28.046026 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:28.046033 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:28.046435 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:28.046442 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:28.046832 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:28.046839 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:28.047220 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:28.047227 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:28.047645 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:28.047665 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:28.048058 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:28.048065 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:28.048469 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:28.048476 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:28.048876 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:28.048884 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:28.049279 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:28.049286 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:28.049679 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:28.049685 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:28.050078 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:28.050084 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:28.310222 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:28.310233 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:28.310648 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:28.310655 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:28.311040 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:28.311047 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:28.311447 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:28.311454 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:28.311836 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:28.311843 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:28.312201 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:28.312208 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:28.312589 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:28.312596 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:28.312985 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:28.312992 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:28.313387 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:28.313394 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:28.313784 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:28.313791 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:28.314174 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:28.314181 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:28.314571 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:28.314578 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:28.314980 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:28.314989 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:28.315388 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:28.315395 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:28.315784 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:28.315791 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:28.316196 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:28.316203 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:28.398493 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:28.398503 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:28.398971 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:28.398975 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:28.399372 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:28.399379 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:28.399751 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:28.399758 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:28.400138 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:28.400145 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:28.400524 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:28.400531 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:28.400906 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:28.400913 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:28.401299 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:28.401305 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:28.401674 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:28.401681 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:28.402062 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:28.402068 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:28.402457 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:28.402464 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:28.402832 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:28.402839 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:28.403219 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:28.403227 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:28.403605 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:28.403612 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:28.403985 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:28.403992 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:28.404378 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:28.404384 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:28.575678 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:28.575689 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:28.576082 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:28.576090 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:28.576502 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:28.576509 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:28.576894 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:28.576901 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:28.577288 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:28.577295 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:28.577662 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:28.577669 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:28.578056 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 0, free 0xff000; fallback to bdev 1 >2018-07-11 07:15:28.578063 7f354f8b1700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:28.840438 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:15:28.840264) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 12 47 446 3641 796 0] max score 4.17, MB/sec: 8.7 rd, 8.7 wr, level 2, files in(1, 0) out(1) MB in(76.6, 0.0) out(76.6), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 4957165, records dropped: 0 > >2018-07-11 07:15:28.840453 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:15:28.840296) EVENT_LOG_v1 {"time_micros": 1531293328840282, "job": 17, "event": "compaction_finished", "compaction_time_micros": 9200799, "output_level": 2, "num_output_files": 1, "total_output_size": 80348267, "num_input_records": 4957165, "num_output_records": 4957165, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 12, 47, 446, 3641, 796, 0]} >2018-07-11 07:15:28.841125 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293328841119, "job": 17, "event": "table_file_deletion", "file_number": 36321} >2018-07-11 07:15:28.841355 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 18] Compacting 1@1 + 3@2 files to L2, score 4.17 >2018-07-11 07:15:28.841369 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 17 Base level 1, inputs: [36320(76MB)], [32690(71MB) 32691(40MB) 32692(60MB)] > >2018-07-11 07:15:28.841390 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293328841374, "job": 18, "event": "compaction_started", "files_L1": [36320], "files_L2": [32690, 32691, 32692], "score": 4.17454, "input_data_size": 259952714} >2018-07-11 07:15:32.894833 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 18] Generated table #36334: 2731655 keys, 74779289 bytes >2018-07-11 07:15:32.894873 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293332894856, "cf_name": "default", "job": 18, "event": "table_file_creation", "file_number": 36334, "file_size": 74779289, "table_properties": {"data_size": 67111238, "index_size": 690010, "filter_size": 6977049, "raw_key_size": 78914497, "raw_average_key_size": 28, "raw_value_size": 30101424, "raw_average_value_size": 11, "num_data_blocks": 16489, "num_entries": 2731655, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "199760", "kMergeOperands": "0"}} >2018-07-11 07:15:35.064195 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 18] Generated table #36335: 1957237 keys, 43256439 bytes >2018-07-11 07:15:35.064250 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293335064228, "cf_name": "default", "job": 18, "event": "table_file_creation", "file_number": 36335, "file_size": 43256439, "table_properties": {"data_size": 37888267, "index_size": 389108, "filter_size": 4978074, "raw_key_size": 56542397, "raw_average_key_size": 28, "raw_value_size": 11380478, "raw_average_value_size": 5, "num_data_blocks": 9306, "num_entries": 1957237, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1000000", "kMergeOperands": "0"}} >2018-07-11 07:15:35.138242 7f354f8b1700 1 bluestore(/var/lib/ceph/osd/ceph-10) _balance_bluefs_freespace gifting 0xd0d9f00000~80000000 to bluefs >2018-07-11 07:15:35.138282 7f354f8b1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:684] reusing log 36313 from recycle list > >2018-07-11 07:15:35.138405 7f354f8b1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #36337. Immutable memtables: 0. > >2018-07-11 07:15:35.162537 7f354f8b1700 1 bluefs add_block_extent bdev 2 0xd0d9f00000~80000000 >2018-07-11 07:15:35.162569 7f353f090700 4 rocksdb: (Original Log Time 2018/07/11-07:15:35.138439) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 0 >2018-07-11 07:15:35.162578 7f353f090700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 19] Flushing memtable with next log file: 36337 > >2018-07-11 07:15:35.162594 7f353f090700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293335162585, "job": 19, "event": "flush_started", "num_memtables": 1, "num_entries": 30000030, "num_deletes": 30000030, "memory_usage": 1280019224} >2018-07-11 07:15:35.162597 7f353f090700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 19] Level-0 flush table #36338: started >2018-07-11 07:15:38.594740 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 18] Generated table #36336: 2638238 keys, 74546140 bytes >2018-07-11 07:15:38.594777 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293338594761, "cf_name": "default", "job": 18, "event": "table_file_creation", "file_number": 36336, "file_size": 74546140, "table_properties": {"data_size": 67112103, "index_size": 689870, "filter_size": 6743177, "raw_key_size": 76215766, "raw_average_key_size": 28, "raw_value_size": 31365720, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638238, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:15:42.355253 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 18] Generated table #36339: 3228950 keys, 55094888 bytes >2018-07-11 07:15:42.355302 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293342355286, "cf_name": "default", "job": 18, "event": "table_file_creation", "file_number": 36339, "file_size": 55094888, "table_properties": {"data_size": 46437642, "index_size": 476798, "filter_size": 8179456, "raw_key_size": 93280781, "raw_average_key_size": 28, "raw_value_size": 2721961, "raw_average_value_size": 0, "num_data_blocks": 11398, "num_entries": 3228950, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "3000000", "kMergeOperands": "0"}} >2018-07-11 07:15:43.187428 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 18] Generated table #36340: 757405 keys, 12276815 bytes >2018-07-11 07:15:43.187461 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293343187445, "cf_name": "default", "job": 18, "event": "table_file_creation", "file_number": 36340, "file_size": 12276815, "table_properties": {"data_size": 10253633, "index_size": 104876, "filter_size": 1917321, "raw_key_size": 21880582, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 2517, "num_entries": 757405, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "757405", "kMergeOperands": "0"}} >2018-07-11 07:15:43.187492 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 18] Compacted 1@1 + 3@2 files to L2 => 259953571 bytes >2018-07-11 07:15:43.201238 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:15:43.201106) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 11 49 446 3641 796 0] max score 3.58, MB/sec: 18.1 rd, 18.1 wr, level 2, files in(1, 3) out(5) MB in(76.6, 171.3) out(247.9), read-write-amplify(6.5) write-amplify(3.2) OK, records in: 11313485, records dropped: 0 > >2018-07-11 07:15:43.201260 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:15:43.201135) EVENT_LOG_v1 {"time_micros": 1531293343201121, "job": 18, "event": "compaction_finished", "compaction_time_micros": 14346087, "output_level": 2, "num_output_files": 5, "total_output_size": 259953571, "num_input_records": 11313485, "num_output_records": 11313485, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 11, 49, 446, 3641, 796, 0]} >2018-07-11 07:15:43.202269 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 20] Compacting 1@1 + 5@2 files to L2, score 3.58 >2018-07-11 07:15:43.202285 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 18 Base level 1, inputs: [36318(76MB)], [32685(1261B) 32686(1064B) 32687(66MB) 32688(18MB) 32689(8703KB)] > >2018-07-11 07:15:43.202308 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293343202291, "job": 20, "event": "compaction_started", "files_L1": [36318], "files_L2": [32685, 32686, 32687, 32688, 32689], "score": 3.57975, "input_data_size": 178949939} >2018-07-11 07:15:43.490042 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 20] Generated table #36341: 113219 keys, 1835889 bytes >2018-07-11 07:15:43.490072 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293343490058, "cf_name": "default", "job": 20, "event": "table_file_creation", "file_number": 36341, "file_size": 1835889, "table_properties": {"data_size": 1532913, "index_size": 15381, "filter_size": 286614, "raw_key_size": 3270777, "raw_average_key_size": 28, "raw_value_size": 95, "raw_average_value_size": 0, "num_data_blocks": 377, "num_entries": 113219, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "113211", "kMergeOperands": "0"}} >2018-07-11 07:15:43.577671 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 20] Generated table #36342: 1 keys, 1064 bytes >2018-07-11 07:15:43.577701 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293343577687, "cf_name": "default", "job": 20, "event": "table_file_creation", "file_number": 36342, "file_size": 1064, "table_properties": {"data_size": 57, "index_size": 27, "filter_size": 18, "raw_key_size": 29, "raw_average_key_size": 29, "raw_value_size": 12, "raw_average_value_size": 12, "num_data_blocks": 1, "num_entries": 1, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:15:47.555141 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 20] Generated table #36343: 2638717 keys, 74694279 bytes >2018-07-11 07:15:47.555177 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293347555161, "cf_name": "default", "job": 20, "event": "table_file_creation", "file_number": 36343, "file_size": 74694279, "table_properties": {"data_size": 67112305, "index_size": 832501, "filter_size": 6748481, "raw_key_size": 77161335, "raw_average_key_size": 29, "raw_value_size": 31055033, "raw_average_value_size": 11, "num_data_blocks": 16713, "num_entries": 2638717, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "2000000", "kMergeOperands": "0"}} >2018-07-11 07:15:49.316768 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 20] Generated table #36344: 702206 keys, 47412063 bytes >2018-07-11 07:15:49.316803 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293349316786, "cf_name": "default", "job": 20, "event": "table_file_creation", "file_number": 36344, "file_size": 47412063, "table_properties": {"data_size": 44931039, "index_size": 624802, "filter_size": 1855236, "raw_key_size": 21372051, "raw_average_key_size": 30, "raw_value_size": 35038423, "raw_average_value_size": 49, "num_data_blocks": 11303, "num_entries": 702206, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:15:51.219237 7f353f090700 1 bluefs _allocate failed to allocate 0x1d000000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:15:58.791854 7f353f090700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293358791832, "cf_name": "default", "job": 19, "event": "table_file_creation", "file_number": 36338, "file_size": 486288093, "table_properties": {"data_size": 406137025, "index_size": 4207268, "filter_size": 75942805, "raw_key_size": 866668569, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 99669, "num_entries": 30000030, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "30000030", "kMergeOperands": "0"}} >2018-07-11 07:15:58.791874 7f353f090700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 19] Level-0 flush table #36338: 486288093 bytes OK >2018-07-11 07:15:58.801107 7f353f090700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:242] adding log 36315 to recycle list > >2018-07-11 07:15:58.801120 7f353f090700 4 rocksdb: (Original Log Time 2018/07/11-07:15:58.792020) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #36338 started >2018-07-11 07:15:58.801123 7f353f090700 4 rocksdb: (Original Log Time 2018/07/11-07:15:58.800976) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #36338: memtable #1 done >2018-07-11 07:15:58.801126 7f353f090700 4 rocksdb: (Original Log Time 2018/07/11-07:15:58.800993) EVENT_LOG_v1 {"time_micros": 1531293358800986, "job": 19, "event": "flush_finished", "lsm_state": [1, 11, 49, 446, 3641, 796, 0], "immutable_memtables": 0} >2018-07-11 07:15:58.801128 7f353f090700 4 rocksdb: (Original Log Time 2018/07/11-07:15:58.801010) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 11 49 446 3641 796 0] max score 3.60 > >2018-07-11 07:15:58.801923 7f353f090700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 19] Try to delete WAL files size 686668413, prev total WAL file size 755336001, number of live WAL files 2. > >2018-07-11 07:15:58.801969 7f353f090700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293358801964, "job": 19, "event": "table_file_deletion", "file_number": 36320} >2018-07-11 07:15:58.801986 7f353f090700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293358801984, "job": 19, "event": "table_file_deletion", "file_number": 32692} >2018-07-11 07:15:58.802003 7f353f090700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293358802000, "job": 19, "event": "table_file_deletion", "file_number": 32691} >2018-07-11 07:15:58.802016 7f353f090700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293358802014, "job": 19, "event": "table_file_deletion", "file_number": 32690} >2018-07-11 07:15:58.868954 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 20] Generated table #36345: 2315389 keys, 41328906 bytes >2018-07-11 07:15:58.869000 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293358868985, "cf_name": "default", "job": 20, "event": "table_file_creation", "file_number": 36345, "file_size": 41328906, "table_properties": {"data_size": 35098665, "index_size": 360282, "filter_size": 5868968, "raw_key_size": 66889014, "raw_average_key_size": 28, "raw_value_size": 3749621, "raw_average_value_size": 1, "num_data_blocks": 8616, "num_entries": 2315389, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "2000000", "kMergeOperands": "0"}} >2018-07-11 07:15:59.755081 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 20] Generated table #36346: 843075 keys, 13665376 bytes >2018-07-11 07:15:59.755113 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293359755098, "cf_name": "default", "job": 20, "event": "table_file_creation", "file_number": 36346, "file_size": 13665376, "table_properties": {"data_size": 11413434, "index_size": 116773, "filter_size": 2134182, "raw_key_size": 24355496, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 2801, "num_entries": 843075, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "843075", "kMergeOperands": "0"}} >2018-07-11 07:15:59.755140 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 20] Compacted 1@1 + 5@2 files to L2 => 178937577 bytes >2018-07-11 07:15:59.767963 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:15:59.767534) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[1 10 50 446 3641 796 0] max score 3.60, MB/sec: 10.8 rd, 10.8 wr, level 2, files in(1, 5) out(6) MB in(76.6, 94.0) out(170.6), read-write-amplify(4.5) write-amplify(2.2) OK, records in: 6612662, records dropped: 55 > >2018-07-11 07:15:59.767974 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:15:59.767562) EVENT_LOG_v1 {"time_micros": 1531293359767548, "job": 20, "event": "compaction_finished", "compaction_time_micros": 16552815, "output_level": 2, "num_output_files": 6, "total_output_size": 178937577, "num_input_records": 6612662, "num_output_records": 6612607, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 10, 50, 446, 3641, 796, 0]} >2018-07-11 07:15:59.768884 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 21] Compacting 1@0 + 9@1 files to L1, score 3.60 >2018-07-11 07:15:59.768912 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 20 Base level 0, inputs: [36338(463MB)], [36317(76MB) 36322(75MB) 36323(71MB) 36324(71MB) 36325(71MB) 36326(71MB) 36327(71MB) 36328(1946KB) 36329(65MB)] > >2018-07-11 07:15:59.768927 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293359768916, "job": 21, "event": "compaction_started", "files_L0": [36338], "files_L1": [36317, 36322, 36323, 36324, 36325, 36326, 36327, 36328, 36329], "score": 3.59971, "input_data_size": 1088990741} >2018-07-11 07:16:00.397635 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293360397628, "job": 0, "event": "table_file_deletion", "file_number": 36318} >2018-07-11 07:16:00.397655 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293360397652, "job": 0, "event": "table_file_deletion", "file_number": 32689} >2018-07-11 07:16:00.397667 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293360397665, "job": 0, "event": "table_file_deletion", "file_number": 32688} >2018-07-11 07:16:00.397678 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293360397676, "job": 0, "event": "table_file_deletion", "file_number": 32687} >2018-07-11 07:16:00.397689 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293360397686, "job": 0, "event": "table_file_deletion", "file_number": 32686} >2018-07-11 07:16:00.397700 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293360397698, "job": 0, "event": "table_file_deletion", "file_number": 32685} >2018-07-11 07:16:04.072263 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4d00000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:16:05.216859 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 21] Generated table #36347: 4888760 keys, 80173883 bytes >2018-07-11 07:16:05.216912 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293365216884, "cf_name": "default", "job": 21, "event": "table_file_creation", "file_number": 36347, "file_size": 80173883, "table_properties": {"data_size": 67110185, "index_size": 685246, "filter_size": 12377461, "raw_key_size": 141267959, "raw_average_key_size": 28, "raw_value_size": 916542, "raw_average_value_size": 0, "num_data_blocks": 16336, "num_entries": 4888760, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4886796", "kMergeOperands": "0"}} >2018-07-11 07:16:09.693803 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4d00000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:16:10.932480 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 21] Generated table #36348: 4957166 keys, 80348286 bytes >2018-07-11 07:16:10.932529 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293370932512, "cf_name": "default", "job": 21, "event": "table_file_creation", "file_number": 36348, "file_size": 80348286, "table_properties": {"data_size": 67109371, "index_size": 689221, "filter_size": 12548704, "raw_key_size": 143207015, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957166, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957166", "kMergeOperands": "0"}} >2018-07-11 07:16:15.126925 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4d00000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:16:16.376240 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 21] Generated table #36349: 4957165 keys, 80348254 bytes >2018-07-11 07:16:16.376297 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293376376269, "cf_name": "default", "job": 21, "event": "table_file_creation", "file_number": 36349, "file_size": 80348254, "table_properties": {"data_size": 67109340, "index_size": 689223, "filter_size": 12548701, "raw_key_size": 143206995, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957165, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957165", "kMergeOperands": "0"}} >2018-07-11 07:16:20.447844 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4d00000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:16:21.778309 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 21] Generated table #36350: 4957165 keys, 80348251 bytes >2018-07-11 07:16:21.778361 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293381778344, "cf_name": "default", "job": 21, "event": "table_file_creation", "file_number": 36350, "file_size": 80348251, "table_properties": {"data_size": 67109335, "index_size": 689225, "filter_size": 12548701, "raw_key_size": 143206995, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957165, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957165", "kMergeOperands": "0"}} >2018-07-11 07:16:26.228555 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4d00000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:16:27.450184 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 21] Generated table #36351: 4957165 keys, 80348249 bytes >2018-07-11 07:16:27.450236 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293387450219, "cf_name": "default", "job": 21, "event": "table_file_creation", "file_number": 36351, "file_size": 80348249, "table_properties": {"data_size": 67109336, "index_size": 689222, "filter_size": 12548701, "raw_key_size": 143206995, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957165, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957165", "kMergeOperands": "0"}} >2018-07-11 07:16:31.532327 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4d00000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:16:32.683870 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 21] Generated table #36352: 4957165 keys, 80348244 bytes >2018-07-11 07:16:32.683925 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293392683896, "cf_name": "default", "job": 21, "event": "table_file_creation", "file_number": 36352, "file_size": 80348244, "table_properties": {"data_size": 67109326, "index_size": 689227, "filter_size": 12548701, "raw_key_size": 143206990, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957165, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957165", "kMergeOperands": "0"}} >2018-07-11 07:16:36.978782 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4d00000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:16:38.116605 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 21] Generated table #36353: 4957164 keys, 80348248 bytes >2018-07-11 07:16:38.116656 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293398116639, "cf_name": "default", "job": 21, "event": "table_file_creation", "file_number": 36353, "file_size": 80348248, "table_properties": {"data_size": 67109322, "index_size": 689238, "filter_size": 12548698, "raw_key_size": 143206974, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957164, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957164", "kMergeOperands": "0"}} >2018-07-11 07:16:42.608760 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4d00000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:16:43.860705 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 21] Generated table #36354: 4763020 keys, 79863518 bytes >2018-07-11 07:16:43.860742 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293403860726, "cf_name": "default", "job": 21, "event": "table_file_creation", "file_number": 36354, "file_size": 79863518, "table_properties": {"data_size": 67110594, "index_size": 689279, "filter_size": 12062652, "raw_key_size": 137598320, "raw_average_key_size": 28, "raw_value_size": 2626952, "raw_average_value_size": 0, "num_data_blocks": 16472, "num_entries": 4763020, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4542440", "kMergeOperands": "0"}} >2018-07-11 07:16:46.415824 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4800000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:16:47.398691 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 21] Generated table #36355: 2638241 keys, 74545830 bytes >2018-07-11 07:16:47.398735 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293407398720, "cf_name": "default", "job": 21, "event": "table_file_creation", "file_number": 36355, "file_size": 74545830, "table_properties": {"data_size": 67112138, "index_size": 689519, "filter_size": 6743183, "raw_key_size": 76215840, "raw_average_key_size": 28, "raw_value_size": 31365743, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:16:49.990404 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4800000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:16:51.050615 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 21] Generated table #36356: 2638241 keys, 74546028 bytes >2018-07-11 07:16:51.050670 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293411050655, "cf_name": "default", "job": 21, "event": "table_file_creation", "file_number": 36356, "file_size": 74546028, "table_properties": {"data_size": 67112175, "index_size": 689680, "filter_size": 6743183, "raw_key_size": 76215860, "raw_average_key_size": 28, "raw_value_size": 31365763, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:16:53.634578 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4800000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:16:54.639481 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 21] Generated table #36357: 2638241 keys, 74546365 bytes >2018-07-11 07:16:54.639525 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293414639510, "cf_name": "default", "job": 21, "event": "table_file_creation", "file_number": 36357, "file_size": 74546365, "table_properties": {"data_size": 67112124, "index_size": 690068, "filter_size": 6743183, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:16:57.211774 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4800000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:16:58.156339 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 21] Generated table #36358: 2638241 keys, 74546839 bytes >2018-07-11 07:16:58.156371 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293418156356, "cf_name": "default", "job": 21, "event": "table_file_creation", "file_number": 36358, "file_size": 74546839, "table_properties": {"data_size": 67112159, "index_size": 690507, "filter_size": 6743183, "raw_key_size": 76215852, "raw_average_key_size": 28, "raw_value_size": 31365755, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:17:00.622093 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4800000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:17:01.555293 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 21] Generated table #36359: 2638241 keys, 74546005 bytes >2018-07-11 07:17:01.555336 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293421555310, "cf_name": "default", "job": 21, "event": "table_file_creation", "file_number": 36359, "file_size": 74546005, "table_properties": {"data_size": 67112128, "index_size": 689704, "filter_size": 6743183, "raw_key_size": 76215855, "raw_average_key_size": 28, "raw_value_size": 31365758, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:17:01.771338 7f354e0ae700 1 bluefs _allocate failed to allocate 0x600000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:17:01.917419 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 21] Generated table #36360: 206991 keys, 5858204 bytes >2018-07-11 07:17:01.917461 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293421917447, "cf_name": "default", "job": 21, "event": "table_file_creation", "file_number": 36360, "file_size": 5858204, "table_properties": {"data_size": 5274455, "index_size": 53683, "filter_size": 529080, "raw_key_size": 5981720, "raw_average_key_size": 28, "raw_value_size": 2467211, "raw_average_value_size": 11, "num_data_blocks": 1296, "num_entries": 206991, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "8", "kMergeOperands": "0"}} >2018-07-11 07:17:02.762401 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4200000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:17:03.669698 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 21] Generated table #36361: 165733 keys, 68237400 bytes >2018-07-11 07:17:03.669740 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293423669725, "cf_name": "default", "job": 21, "event": "table_file_creation", "file_number": 36361, "file_size": 68237400, "table_properties": {"data_size": 67109966, "index_size": 558631, "filter_size": 567815, "raw_key_size": 15331149, "raw_average_key_size": 92, "raw_value_size": 53149329, "raw_average_value_size": 320, "num_data_blocks": 15089, "num_entries": 165733, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "80960", "kMergeOperands": "0"}} >2018-07-11 07:17:03.669838 7f354e0ae700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:17:03.709549 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 21] Generated table #36362: 4 keys, 5090 bytes >2018-07-11 07:17:03.709572 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293423709561, "cf_name": "default", "job": 21, "event": "table_file_creation", "file_number": 36362, "file_size": 5090, "table_properties": {"data_size": 4075, "index_size": 28, "filter_size": 20, "raw_key_size": 334, "raw_average_key_size": 83, "raw_value_size": 3750, "raw_average_value_size": 937, "num_data_blocks": 1, "num_entries": 4, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:17:03.709630 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 21] Compacted 1@0 + 9@1 files to L1 => 1088958694 bytes >2018-07-11 07:17:03.724571 7f354e0ae700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 25410370994 >2018-07-11 07:17:03.724707 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:17:03.724534) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 17 50 446 3641 796 0] max score 6.59, MB/sec: 17.0 rd, 17.0 wr, level 1, files in(1, 9) out(16) MB in(463.8, 574.8) out(1038.5), read-write-amplify(4.5) write-amplify(2.2) OK, records in: 52958703, records dropped: 0 > >2018-07-11 07:17:03.724716 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:17:03.724563) EVENT_LOG_v1 {"time_micros": 1531293423724548, "job": 21, "event": "compaction_finished", "compaction_time_micros": 63940683, "output_level": 1, "num_output_files": 16, "total_output_size": 1088958694, "num_input_records": 52958703, "num_output_records": 52958703, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 17, 50, 446, 3641, 796, 0]} >2018-07-11 07:17:03.725721 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 22] Compacting 1@1 + 7@2 files to L2, score 6.59 >2018-07-11 07:17:03.725746 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 21 Base level 1, inputs: [36348(76MB)], [36341(1792KB) 36342(1064B) 36343(71MB) 36344(45MB) 36345(39MB) 36346(13MB) 36331(2484KB)] > >2018-07-11 07:17:03.725761 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293423725750, "job": 22, "event": "compaction_started", "files_L1": [36348], "files_L2": [36341, 36342, 36343, 36344, 36345, 36346, 36331], "score": 6.59276, "input_data_size": 261829858} >2018-07-11 07:17:04.911513 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293424911505, "job": 0, "event": "table_file_deletion", "file_number": 36338} >2018-07-11 07:17:04.911539 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293424911535, "job": 0, "event": "table_file_deletion", "file_number": 36329} >2018-07-11 07:17:04.911557 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293424911554, "job": 0, "event": "table_file_deletion", "file_number": 36328} >2018-07-11 07:17:04.911580 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293424911576, "job": 0, "event": "table_file_deletion", "file_number": 36327} >2018-07-11 07:17:04.911602 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293424911599, "job": 0, "event": "table_file_deletion", "file_number": 36326} >2018-07-11 07:17:04.911625 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293424911621, "job": 0, "event": "table_file_deletion", "file_number": 36325} >2018-07-11 07:17:04.911648 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293424911644, "job": 0, "event": "table_file_deletion", "file_number": 36324} >2018-07-11 07:17:04.911671 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293424911666, "job": 0, "event": "table_file_deletion", "file_number": 36323} >2018-07-11 07:17:04.911693 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293424911690, "job": 0, "event": "table_file_deletion", "file_number": 36322} >2018-07-11 07:17:04.911715 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293424911711, "job": 0, "event": "table_file_deletion", "file_number": 36317} >2018-07-11 07:17:06.481601 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 22] Generated table #36363: 2113219 keys, 34252510 bytes >2018-07-11 07:17:06.481646 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293426481630, "cf_name": "default", "job": 22, "event": "table_file_creation", "file_number": 36363, "file_size": 34252510, "table_properties": {"data_size": 28608586, "index_size": 293476, "filter_size": 5349459, "raw_key_size": 61048557, "raw_average_key_size": 28, "raw_value_size": 95, "raw_average_value_size": 0, "num_data_blocks": 7021, "num_entries": 2113219, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "2113211", "kMergeOperands": "0"}} >2018-07-11 07:17:06.670776 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 22] Generated table #36364: 1 keys, 1064 bytes >2018-07-11 07:17:06.670818 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293426670792, "cf_name": "default", "job": 22, "event": "table_file_creation", "file_number": 36364, "file_size": 1064, "table_properties": {"data_size": 57, "index_size": 27, "filter_size": 18, "raw_key_size": 29, "raw_average_key_size": 29, "raw_value_size": 12, "raw_average_value_size": 12, "num_data_blocks": 1, "num_entries": 1, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:17:11.259564 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 22] Generated table #36365: 3614529 keys, 77055384 bytes >2018-07-11 07:17:11.259595 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293431259581, "cf_name": "default", "job": 22, "event": "table_file_creation", "file_number": 36365, "file_size": 77055384, "table_properties": {"data_size": 67110688, "index_size": 754451, "filter_size": 9189252, "raw_key_size": 104841254, "raw_average_key_size": 29, "raw_value_size": 18022859, "raw_average_value_size": 4, "num_data_blocks": 16577, "num_entries": 3614529, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "3000000", "kMergeOperands": "0"}} >2018-07-11 07:17:13.774858 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 22] Generated table #36366: 726394 keys, 61259067 bytes >2018-07-11 07:17:13.774903 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293433774887, "cf_name": "default", "job": 22, "event": "table_file_creation", "file_number": 36366, "file_size": 61259067, "table_properties": {"data_size": 58470303, "index_size": 841890, "filter_size": 1945888, "raw_key_size": 22581022, "raw_average_key_size": 31, "raw_value_size": 48070597, "raw_average_value_size": 66, "num_data_blocks": 14761, "num_entries": 726394, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:17:16.592495 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 22] Generated table #36367: 2315389 keys, 41328906 bytes >2018-07-11 07:17:16.592527 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293436592512, "cf_name": "default", "job": 22, "event": "table_file_creation", "file_number": 36367, "file_size": 41328906, "table_properties": {"data_size": 35098665, "index_size": 360282, "filter_size": 5868968, "raw_key_size": 66889014, "raw_average_key_size": 28, "raw_value_size": 3749621, "raw_average_value_size": 1, "num_data_blocks": 8616, "num_entries": 2315389, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "2000000", "kMergeOperands": "0"}} >2018-07-11 07:17:18.759730 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 22] Generated table #36368: 2000000 keys, 32417287 bytes >2018-07-11 07:17:18.759778 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293438759747, "cf_name": "default", "job": 22, "event": "table_file_creation", "file_number": 36368, "file_size": 32417287, "table_properties": {"data_size": 27075674, "index_size": 277774, "filter_size": 5062852, "raw_key_size": 57777780, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 6645, "num_entries": 2000000, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "2000000", "kMergeOperands": "0"}} >2018-07-11 07:17:19.809047 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 22] Generated table #36369: 957166 keys, 15514592 bytes >2018-07-11 07:17:19.809077 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293439809062, "cf_name": "default", "job": 22, "event": "table_file_creation", "file_number": 36369, "file_size": 15514592, "table_properties": {"data_size": 12957957, "index_size": 132650, "filter_size": 2422998, "raw_key_size": 27651455, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 3180, "num_entries": 957166, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "957166", "kMergeOperands": "0"}} >2018-07-11 07:17:19.809118 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 22] Compacted 1@1 + 7@2 files to L2 => 261828810 bytes >2018-07-11 07:17:19.824329 7f354e0ae700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 25337647338 >2018-07-11 07:17:19.824438 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:17:19.824295) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 16 50 446 3641 796 0] max score 6.00, MB/sec: 16.3 rd, 16.3 wr, level 2, files in(1, 7) out(7) MB in(76.6, 173.1) out(249.7), read-write-amplify(6.5) write-amplify(3.3) OK, records in: 11726698, records dropped: 0 > >2018-07-11 07:17:19.824447 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:17:19.824321) EVENT_LOG_v1 {"time_micros": 1531293439824309, "job": 22, "event": "compaction_finished", "compaction_time_micros": 16083342, "output_level": 2, "num_output_files": 7, "total_output_size": 261828810, "num_input_records": 11726698, "num_output_records": 11726698, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 16, 50, 446, 3641, 796, 0]} >2018-07-11 07:17:19.825440 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 23] Compacting 1@1 + 3@2 files to L2, score 6.00 >2018-07-11 07:17:19.825452 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 22 Base level 1, inputs: [36349(76MB)], [36332(74MB) 36334(71MB) 36335(41MB)] > >2018-07-11 07:17:19.825466 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293439825455, "job": 23, "event": "compaction_started", "files_L1": [36349], "files_L2": [36332, 36334, 36335], "score": 5.99797, "input_data_size": 276188681} >2018-07-11 07:17:20.506241 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293440506235, "job": 0, "event": "table_file_deletion", "file_number": 36348} >2018-07-11 07:17:20.506266 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293440506264, "job": 0, "event": "table_file_deletion", "file_number": 36346} >2018-07-11 07:17:20.506277 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293440506275, "job": 0, "event": "table_file_deletion", "file_number": 36345} >2018-07-11 07:17:20.506290 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293440506288, "job": 0, "event": "table_file_deletion", "file_number": 36344} >2018-07-11 07:17:20.506303 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293440506301, "job": 0, "event": "table_file_deletion", "file_number": 36343} >2018-07-11 07:17:20.506319 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293440506315, "job": 0, "event": "table_file_deletion", "file_number": 36342} >2018-07-11 07:17:20.506329 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293440506327, "job": 0, "event": "table_file_deletion", "file_number": 36341} >2018-07-11 07:17:20.506344 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293440506341, "job": 0, "event": "table_file_deletion", "file_number": 36331} >2018-07-11 07:17:23.639771 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 23] Generated table #36370: 3042834 keys, 49319996 bytes >2018-07-11 07:17:23.639819 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293443639802, "cf_name": "default", "job": 23, "event": "table_file_creation", "file_number": 36370, "file_size": 49319996, "table_properties": {"data_size": 41193432, "index_size": 422866, "filter_size": 7702709, "raw_key_size": 87904105, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 10110, "num_entries": 3042834, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "3042834", "kMergeOperands": "0"}} >2018-07-11 07:17:29.552093 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 23] Generated table #36371: 4957168 keys, 80348315 bytes >2018-07-11 07:17:29.552142 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293449552126, "cf_name": "default", "job": 23, "event": "table_file_creation", "file_number": 36371, "file_size": 80348315, "table_properties": {"data_size": 67109373, "index_size": 689242, "filter_size": 12548710, "raw_key_size": 143207073, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957168, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957168", "kMergeOperands": "0"}} >2018-07-11 07:17:33.886443 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 23] Generated table #36372: 2658303 keys, 74596780 bytes >2018-07-11 07:17:33.886480 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293453886464, "cf_name": "default", "job": 23, "event": "table_file_creation", "file_number": 36372, "file_size": 74596780, "table_properties": {"data_size": 67112829, "index_size": 689549, "filter_size": 6793410, "raw_key_size": 76795429, "raw_average_key_size": 28, "raw_value_size": 31095045, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2658303, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "42832", "kMergeOperands": "0"}} >2018-07-11 07:17:37.307307 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 23] Generated table #36373: 1937099 keys, 31672066 bytes >2018-07-11 07:17:37.307344 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293457307328, "cf_name": "default", "job": 23, "event": "table_file_creation", "file_number": 36373, "file_size": 31672066, "table_properties": {"data_size": 26495094, "index_size": 271800, "filter_size": 4904183, "raw_key_size": 55960627, "raw_average_key_size": 28, "raw_value_size": 270681, "raw_average_value_size": 0, "num_data_blocks": 6503, "num_entries": 1937099, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1914331", "kMergeOperands": "0"}} >2018-07-11 07:17:39.363148 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 23] Generated table #36374: 1850893 keys, 40251754 bytes >2018-07-11 07:17:39.363186 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293459363170, "cf_name": "default", "job": 23, "event": "table_file_creation", "file_number": 36374, "file_size": 40251754, "table_properties": {"data_size": 35183076, "index_size": 361431, "filter_size": 4706257, "raw_key_size": 53470247, "raw_average_key_size": 28, "raw_value_size": 10116176, "raw_average_value_size": 5, "num_data_blocks": 8641, "num_entries": 1850893, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1000000", "kMergeOperands": "0"}} >2018-07-11 07:17:39.363220 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 23] Compacted 1@1 + 3@2 files to L2 => 276188911 bytes >2018-07-11 07:17:39.373994 7f354e0ae700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 25253095568 >2018-07-11 07:17:39.374182 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:17:39.373958) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 15 52 446 3641 796 0] max score 5.40, MB/sec: 14.1 rd, 14.1 wr, level 2, files in(1, 3) out(5) MB in(76.6, 186.8) out(263.4), read-write-amplify(6.9) write-amplify(3.4) OK, records in: 14446297, records dropped: 0 > >2018-07-11 07:17:39.374192 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:17:39.373984) EVENT_LOG_v1 {"time_micros": 1531293459373973, "job": 23, "event": "compaction_finished", "compaction_time_micros": 19537740, "output_level": 2, "num_output_files": 5, "total_output_size": 276188911, "num_input_records": 14446297, "num_output_records": 14446297, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 15, 52, 446, 3641, 796, 0]} >2018-07-11 07:17:39.374926 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293459374922, "job": 23, "event": "table_file_deletion", "file_number": 36349} >2018-07-11 07:17:39.374944 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293459374941, "job": 23, "event": "table_file_deletion", "file_number": 36335} >2018-07-11 07:17:39.374960 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293459374957, "job": 23, "event": "table_file_deletion", "file_number": 36334} >2018-07-11 07:17:39.374975 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293459374973, "job": 23, "event": "table_file_deletion", "file_number": 36332} >2018-07-11 07:17:39.375159 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 24] Compacting 1@1 + 2@2 files to L2, score 5.40 >2018-07-11 07:17:39.375171 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 23 Base level 1, inputs: [36350(76MB)], [36374(38MB) 36336(71MB)] > >2018-07-11 07:17:39.375185 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293459375175, "job": 24, "event": "compaction_started", "files_L1": [36350], "files_L2": [36374, 36336], "score": 5.40318, "input_data_size": 195146145} >2018-07-11 07:17:43.515591 7f354f8b1700 1 bluestore(/var/lib/ceph/osd/ceph-10) _balance_bluefs_freespace gifting 0xd159f00000~80000000 to bluefs >2018-07-11 07:17:43.515628 7f354f8b1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:684] reusing log 36315 from recycle list > >2018-07-11 07:17:43.515721 7f354f8b1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #36376. Immutable memtables: 0. > >2018-07-11 07:17:43.515740 7f354f8b1700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 25253095568 >2018-07-11 07:17:44.063713 7f354f8b1700 1 bluefs add_block_extent bdev 2 0xd159f00000~80000000 >2018-07-11 07:17:44.063764 7f353f090700 4 rocksdb: (Original Log Time 2018/07/11-07:17:43.515802) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 0 >2018-07-11 07:17:44.063778 7f353f090700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 25] Flushing memtable with next log file: 36376 > >2018-07-11 07:17:44.063794 7f353f090700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293464063786, "job": 25, "event": "flush_started", "num_memtables": 1, "num_entries": 33000034, "num_deletes": 33000033, "memory_usage": 1407981400} >2018-07-11 07:17:44.063797 7f353f090700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 25] Level-0 flush table #36377: started >2018-07-11 07:17:44.652360 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 24] Generated table #36375: 4549560 keys, 79330414 bytes >2018-07-11 07:17:44.652415 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293464652398, "cf_name": "default", "job": 24, "event": "table_file_creation", "file_number": 36375, "file_size": 79330414, "table_properties": {"data_size": 67111826, "index_size": 689344, "filter_size": 11528251, "raw_key_size": 131431744, "raw_average_key_size": 28, "raw_value_size": 5515147, "raw_average_value_size": 1, "num_data_blocks": 16474, "num_entries": 4549560, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4085669", "kMergeOperands": "0"}} >2018-07-11 07:17:48.641420 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 24] Generated table #36378: 3106037 keys, 75717248 bytes >2018-07-11 07:17:48.641467 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293468641451, "cf_name": "default", "job": 24, "event": "table_file_creation", "file_number": 36378, "file_size": 75717248, "table_properties": {"data_size": 67111697, "index_size": 690228, "filter_size": 7914331, "raw_key_size": 89729971, "raw_average_key_size": 28, "raw_value_size": 25038452, "raw_average_value_size": 8, "num_data_blocks": 16486, "num_entries": 3106037, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1000000", "kMergeOperands": "0"}} >2018-07-11 07:17:49.683716 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 24] Generated table #36379: 776808 keys, 21950059 bytes >2018-07-11 07:17:49.683760 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293469683745, "cf_name": "default", "job": 24, "event": "table_file_creation", "file_number": 36379, "file_size": 21950059, "table_properties": {"data_size": 19760636, "index_size": 202959, "filter_size": 1985478, "raw_key_size": 22441121, "raw_average_key_size": 28, "raw_value_size": 9235385, "raw_average_value_size": 11, "num_data_blocks": 4856, "num_entries": 776808, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:17:50.886342 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 24] Generated table #36380: 1013891 keys, 18149566 bytes >2018-07-11 07:17:50.886377 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293470886360, "cf_name": "default", "job": 24, "event": "table_file_creation", "file_number": 36380, "file_size": 18149566, "table_properties": {"data_size": 15420468, "index_size": 158026, "filter_size": 2570083, "raw_key_size": 29290172, "raw_average_key_size": 28, "raw_value_size": 1692912, "raw_average_value_size": 1, "num_data_blocks": 3786, "num_entries": 1013891, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "871496", "kMergeOperands": "0"}} >2018-07-11 07:17:50.886408 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 24] Compacted 1@1 + 2@2 files to L2 => 195147287 bytes >2018-07-11 07:17:50.898775 7f354e0ae700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 25153554163 >2018-07-11 07:17:50.898885 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:17:50.898740) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 14 54 446 3641 796 0] max score 4.81, MB/sec: 17.0 rd, 17.0 wr, level 2, files in(1, 2) out(4) MB in(76.6, 109.5) out(186.1), read-write-amplify(4.9) write-amplify(2.4) OK, records in: 9446296, records dropped: 0 > >2018-07-11 07:17:50.898895 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:17:50.898766) EVENT_LOG_v1 {"time_micros": 1531293470898755, "job": 24, "event": "compaction_finished", "compaction_time_micros": 11511209, "output_level": 2, "num_output_files": 4, "total_output_size": 195147287, "num_input_records": 9446296, "num_output_records": 9446296, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 14, 54, 446, 3641, 796, 0]} >2018-07-11 07:17:50.899818 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_picker.cc:461] [default] Expanding@1 1+4(159662889+313305072 bytes) to 2+4 (319325773+313305072bytes) > >2018-07-11 07:17:50.899915 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 26] Compacting 2@1 + 4@2 files to L2, score 4.81 >2018-07-11 07:17:50.899931 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 24 Base level 1, inputs: [36351(76MB) 36352(76MB)], [36380(17MB) 36339(52MB) 36340(11MB) 36333(76MB)] > >2018-07-11 07:17:50.899952 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293470899936, "job": 26, "event": "compaction_started", "files_L1": [36351, 36352], "files_L2": [36380, 36339, 36340, 36333], "score": 4.80839, "input_data_size": 326566029} >2018-07-11 07:17:56.653483 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 26] Generated table #36381: 4630818 keys, 79532610 bytes >2018-07-11 07:17:56.653519 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293476653503, "cf_name": "default", "job": 26, "event": "table_file_creation", "file_number": 36381, "file_size": 79532610, "table_properties": {"data_size": 67110491, "index_size": 689447, "filter_size": 11731679, "raw_key_size": 133779172, "raw_average_key_size": 28, "raw_value_size": 4414873, "raw_average_value_size": 0, "num_data_blocks": 16473, "num_entries": 4630818, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4259473", "kMergeOperands": "0"}} >2018-07-11 07:18:00.965282 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 26] Generated table #36382: 3740527 keys, 60628482 bytes >2018-07-11 07:18:00.965335 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293480965318, "cf_name": "default", "job": 26, "event": "table_file_creation", "file_number": 36382, "file_size": 60628482, "table_properties": {"data_size": 50638699, "index_size": 519927, "filter_size": 9468867, "raw_key_size": 108059686, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 12428, "num_entries": 3740527, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "3740527", "kMergeOperands": "0"}} >2018-07-11 07:18:01.375033 7f353f090700 1 bluefs _allocate failed to allocate 0x1ff00000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:18:09.522489 7f353f090700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293489522468, "cf_name": "default", "job": 25, "event": "table_file_creation", "file_number": 36377, "file_size": 534924154, "table_properties": {"data_size": 446751439, "index_size": 4634631, "filter_size": 83537088, "raw_key_size": 953335452, "raw_average_key_size": 28, "raw_value_size": 692, "raw_average_value_size": 0, "num_data_blocks": 109636, "num_entries": 33000034, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "33000033", "kMergeOperands": "0"}} >2018-07-11 07:18:09.522509 7f353f090700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 25] Level-0 flush table #36377: 534924154 bytes OK >2018-07-11 07:18:09.540078 7f353f090700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 25153554163 >2018-07-11 07:18:09.540179 7f353f090700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:242] adding log 36337 to recycle list > >2018-07-11 07:18:09.540188 7f353f090700 4 rocksdb: (Original Log Time 2018/07/11-07:18:09.522654) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #36377 started >2018-07-11 07:18:09.540193 7f353f090700 4 rocksdb: (Original Log Time 2018/07/11-07:18:09.540052) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #36377: memtable #1 done >2018-07-11 07:18:09.540194 7f353f090700 4 rocksdb: (Original Log Time 2018/07/11-07:18:09.540070) EVENT_LOG_v1 {"time_micros": 1531293489540062, "job": 25, "event": "flush_finished", "lsm_state": [1, 14, 54, 446, 3641, 796, 0], "immutable_memtables": 0} >2018-07-11 07:18:09.540196 7f353f090700 4 rocksdb: (Original Log Time 2018/07/11-07:18:09.540095) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 14 54 446 3641 796 0] max score 3.96 > >2018-07-11 07:18:09.541018 7f353f090700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 25] Try to delete WAL files size 755335990, prev total WAL file size 755336714, number of live WAL files 2. > >2018-07-11 07:18:09.541055 7f353f090700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293489541051, "job": 25, "event": "table_file_deletion", "file_number": 36374} >2018-07-11 07:18:09.541068 7f353f090700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293489541066, "job": 25, "event": "table_file_deletion", "file_number": 36350} >2018-07-11 07:18:09.541081 7f353f090700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293489541079, "job": 25, "event": "table_file_deletion", "file_number": 36336} >2018-07-11 07:18:14.570125 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 26] Generated table #36383: 4957168 keys, 80348314 bytes >2018-07-11 07:18:14.570162 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293494570145, "cf_name": "default", "job": 26, "event": "table_file_creation", "file_number": 36383, "file_size": 80348314, "table_properties": {"data_size": 67109372, "index_size": 689242, "filter_size": 12548710, "raw_key_size": 143207073, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957168, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957168", "kMergeOperands": "0"}} >2018-07-11 07:18:20.419219 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 26] Generated table #36384: 4957164 keys, 80348246 bytes >2018-07-11 07:18:20.419266 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293500419249, "cf_name": "default", "job": 26, "event": "table_file_creation", "file_number": 36384, "file_size": 80348246, "table_properties": {"data_size": 67109344, "index_size": 689214, "filter_size": 12548698, "raw_key_size": 143206950, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957164, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957164", "kMergeOperands": "0"}} >2018-07-11 07:18:22.148404 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 26] Generated table #36385: 1586064 keys, 25708068 bytes >2018-07-11 07:18:22.148448 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293502148420, "cf_name": "default", "job": 26, "event": "table_file_creation", "file_number": 36385, "file_size": 25708068, "table_properties": {"data_size": 21471916, "index_size": 220160, "filter_size": 4015005, "raw_key_size": 45819630, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 5270, "num_entries": 1586064, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1586064", "kMergeOperands": "0"}} >2018-07-11 07:18:22.148477 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 26] Compacted 2@1 + 4@2 files to L2 => 326565720 bytes >2018-07-11 07:18:22.164858 7f354e0ae700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 24889980373 >2018-07-11 07:18:22.165371 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:18:22.164824) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[1 12 55 446 3641 796 0] max score 3.96, MB/sec: 10.5 rd, 10.5 wr, level 2, files in(2, 4) out(5) MB in(153.3, 158.2) out(311.4), read-write-amplify(4.1) write-amplify(2.0) OK, records in: 19871741, records dropped: 0 > >2018-07-11 07:18:22.165381 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:18:22.164849) EVENT_LOG_v1 {"time_micros": 1531293502164838, "job": 26, "event": "compaction_finished", "compaction_time_micros": 31248508, "output_level": 2, "num_output_files": 5, "total_output_size": 326565720, "num_input_records": 19871741, "num_output_records": 19871741, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 12, 55, 446, 3641, 796, 0]} >2018-07-11 07:18:22.166297 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 27] Compacting 1@0 + 12@1 files to L1, score 3.96 >2018-07-11 07:18:22.166313 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 26 Base level 0, inputs: [36377(510MB)], [36347(76MB) 36353(76MB) 36354(76MB) 36355(71MB) 36356(71MB) 36357(71MB) 36358(71MB) 36359(71MB) 36360(5720KB) 36361(65MB) 36362(5090B) 36330(54MB)] > >2018-07-11 07:18:22.166329 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293502166317, "job": 27, "event": "compaction_started", "files_L0": [36377], "files_L1": [36347, 36353, 36354, 36355, 36356, 36357, 36358, 36359, 36360, 36361, 36362, 36330], "score": 3.9597, "input_data_size": 1279719072} >2018-07-11 07:18:24.685914 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293504685893, "job": 0, "event": "table_file_deletion", "file_number": 36380} >2018-07-11 07:18:24.685931 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293504685929, "job": 0, "event": "table_file_deletion", "file_number": 36352} >2018-07-11 07:18:24.685942 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293504685940, "job": 0, "event": "table_file_deletion", "file_number": 36351} >2018-07-11 07:18:24.685954 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293504685952, "job": 0, "event": "table_file_deletion", "file_number": 36340} >2018-07-11 07:18:24.685966 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293504685963, "job": 0, "event": "table_file_deletion", "file_number": 36339} >2018-07-11 07:18:24.685979 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293504685977, "job": 0, "event": "table_file_deletion", "file_number": 36333} >2018-07-11 07:18:26.932176 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4d00000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:18:28.110856 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 27] Generated table #36386: 4888760 keys, 80173884 bytes >2018-07-11 07:18:28.110904 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293508110876, "cf_name": "default", "job": 27, "event": "table_file_creation", "file_number": 36386, "file_size": 80173884, "table_properties": {"data_size": 67110186, "index_size": 685246, "filter_size": 12377461, "raw_key_size": 141267959, "raw_average_key_size": 28, "raw_value_size": 916542, "raw_average_value_size": 0, "num_data_blocks": 16336, "num_entries": 4888760, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4886796", "kMergeOperands": "0"}} >2018-07-11 07:18:32.567474 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4d00000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:18:33.737653 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 27] Generated table #36387: 4957167 keys, 80348324 bytes >2018-07-11 07:18:33.737689 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293513737673, "cf_name": "default", "job": 27, "event": "table_file_creation", "file_number": 36387, "file_size": 80348324, "table_properties": {"data_size": 67109492, "index_size": 689135, "filter_size": 12548707, "raw_key_size": 143207044, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957167, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957167", "kMergeOperands": "0"}} >2018-07-11 07:18:37.841031 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4d00000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:18:39.020687 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 27] Generated table #36388: 4957165 keys, 80348257 bytes >2018-07-11 07:18:39.020732 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293519020716, "cf_name": "default", "job": 27, "event": "table_file_creation", "file_number": 36388, "file_size": 80348257, "table_properties": {"data_size": 67109373, "index_size": 689193, "filter_size": 12548701, "raw_key_size": 143206995, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957165, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957165", "kMergeOperands": "0"}} >2018-07-11 07:18:43.088599 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4d00000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:18:44.374342 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 27] Generated table #36389: 4957165 keys, 80348282 bytes >2018-07-11 07:18:44.374383 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293524374367, "cf_name": "default", "job": 27, "event": "table_file_creation", "file_number": 36389, "file_size": 80348282, "table_properties": {"data_size": 67109394, "index_size": 689197, "filter_size": 12548701, "raw_key_size": 143206995, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957165, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957165", "kMergeOperands": "0"}} >2018-07-11 07:18:48.838250 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4d00000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:18:50.155219 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 27] Generated table #36390: 4957165 keys, 80348263 bytes >2018-07-11 07:18:50.155276 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293530155259, "cf_name": "default", "job": 27, "event": "table_file_creation", "file_number": 36390, "file_size": 80348263, "table_properties": {"data_size": 67109366, "index_size": 689206, "filter_size": 12548701, "raw_key_size": 143206995, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957165, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957165", "kMergeOperands": "0"}} >2018-07-11 07:18:54.198310 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4d00000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:18:55.466288 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 27] Generated table #36391: 4957164 keys, 80348247 bytes >2018-07-11 07:18:55.466323 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293535466307, "cf_name": "default", "job": 27, "event": "table_file_creation", "file_number": 36391, "file_size": 80348247, "table_properties": {"data_size": 67109331, "index_size": 689228, "filter_size": 12548698, "raw_key_size": 143206972, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957164, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957164", "kMergeOperands": "0"}} >2018-07-11 07:19:00.011441 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4d00000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:19:01.206166 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 27] Generated table #36392: 4957164 keys, 80348268 bytes >2018-07-11 07:19:01.206216 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293541206199, "cf_name": "default", "job": 27, "event": "table_file_creation", "file_number": 36392, "file_size": 80348268, "table_properties": {"data_size": 67109356, "index_size": 689224, "filter_size": 12548698, "raw_key_size": 143206975, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957164, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957164", "kMergeOperands": "0"}} >2018-07-11 07:19:06.380466 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4d00000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:19:07.571304 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 27] Generated table #36393: 4957165 keys, 80348258 bytes >2018-07-11 07:19:07.571351 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293547571324, "cf_name": "default", "job": 27, "event": "table_file_creation", "file_number": 36393, "file_size": 80348258, "table_properties": {"data_size": 67109353, "index_size": 689214, "filter_size": 12548701, "raw_key_size": 143206996, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957165, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957165", "kMergeOperands": "0"}} >2018-07-11 07:19:12.057242 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4d00000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:19:13.235293 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 27] Generated table #36394: 4957165 keys, 80348275 bytes >2018-07-11 07:19:13.235335 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293553235317, "cf_name": "default", "job": 27, "event": "table_file_creation", "file_number": 36394, "file_size": 80348275, "table_properties": {"data_size": 67109381, "index_size": 689203, "filter_size": 12548701, "raw_key_size": 143206993, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957165, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957165", "kMergeOperands": "0"}} >2018-07-11 07:19:17.197677 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4b00000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:19:18.376426 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 27] Generated table #36395: 3967743 keys, 77874753 bytes >2018-07-11 07:19:18.376472 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293558376444, "cf_name": "default", "job": 27, "event": "table_file_creation", "file_number": 36395, "file_size": 77874753, "table_properties": {"data_size": 67112686, "index_size": 689426, "filter_size": 10071648, "raw_key_size": 114623647, "raw_average_key_size": 28, "raw_value_size": 13384957, "raw_average_value_size": 3, "num_data_blocks": 16479, "num_entries": 3967743, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "2842284", "kMergeOperands": "0"}} >2018-07-11 07:19:19.766944 7f354e0ae700 1 bluefs _allocate failed to allocate 0x2900000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:19:20.447456 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 27] Generated table #36396: 1493239 keys, 42193208 bytes >2018-07-11 07:19:20.447485 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293560447471, "cf_name": "default", "job": 27, "event": "table_file_creation", "file_number": 36396, "file_size": 42193208, "table_properties": {"data_size": 37985350, "index_size": 390246, "filter_size": 3816624, "raw_key_size": 43138019, "raw_average_key_size": 28, "raw_value_size": 17752956, "raw_average_value_size": 11, "num_data_blocks": 9333, "num_entries": 1493239, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:19:23.175824 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4800000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:19:24.237545 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 27] Generated table #36397: 2638242 keys, 74546384 bytes >2018-07-11 07:19:24.237591 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293564237563, "cf_name": "default", "job": 27, "event": "table_file_creation", "file_number": 36397, "file_size": 74546384, "table_properties": {"data_size": 67112132, "index_size": 690076, "filter_size": 6743186, "raw_key_size": 76215874, "raw_average_key_size": 28, "raw_value_size": 31365760, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638242, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:19:26.754772 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4800000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:19:31.458910 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 27] Generated table #36398: 2638241 keys, 74546216 bytes >2018-07-11 07:19:31.458955 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293571458928, "cf_name": "default", "job": 27, "event": "table_file_creation", "file_number": 36398, "file_size": 74546216, "table_properties": {"data_size": 67112147, "index_size": 689896, "filter_size": 6743183, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:19:35.371973 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4800000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:19:36.363833 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 27] Generated table #36399: 2638241 keys, 74545769 bytes >2018-07-11 07:19:36.363887 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293576363871, "cf_name": "default", "job": 27, "event": "table_file_creation", "file_number": 36399, "file_size": 74545769, "table_properties": {"data_size": 67112088, "index_size": 689508, "filter_size": 6743183, "raw_key_size": 76215853, "raw_average_key_size": 28, "raw_value_size": 31365756, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:19:38.615139 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4800000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:19:39.569923 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 27] Generated table #36400: 2638241 keys, 74546334 bytes >2018-07-11 07:19:39.569973 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293579569957, "cf_name": "default", "job": 27, "event": "table_file_creation", "file_number": 36400, "file_size": 74546334, "table_properties": {"data_size": 67112185, "index_size": 689976, "filter_size": 6743183, "raw_key_size": 76215859, "raw_average_key_size": 28, "raw_value_size": 31365762, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:19:39.595956 7f354f8b1700 1 bluestore(/var/lib/ceph/osd/ceph-10) _balance_bluefs_freespace gifting 0xd1d9f00000~80000000 to bluefs >2018-07-11 07:19:39.596000 7f354f8b1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:684] reusing log 36337 from recycle list > >2018-07-11 07:19:39.596105 7f354f8b1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #36402. Immutable memtables: 0. > >2018-07-11 07:19:39.596124 7f354f8b1700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 24889980373 >2018-07-11 07:19:39.622388 7f354f8b1700 1 bluefs add_block_extent bdev 2 0xd1d9f00000~80000000 >2018-07-11 07:19:39.622426 7f353f090700 4 rocksdb: (Original Log Time 2018/07/11-07:19:39.596171) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 0 >2018-07-11 07:19:39.622444 7f353f090700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 28] Flushing memtable with next log file: 36402 > >2018-07-11 07:19:39.622459 7f353f090700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293579622451, "job": 28, "event": "flush_started", "num_memtables": 1, "num_entries": 30000031, "num_deletes": 30000030, "memory_usage": 1279990904} >2018-07-11 07:19:39.622462 7f353f090700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 28] Level-0 flush table #36403: started >2018-07-11 07:19:40.895220 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4300000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:19:41.835354 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 27] Generated table #36401: 601599 keys, 69327400 bytes >2018-07-11 07:19:41.835387 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293581835371, "cf_name": "default", "job": 27, "event": "table_file_creation", "file_number": 36401, "file_size": 69327400, "table_properties": {"data_size": 67112480, "index_size": 557197, "filter_size": 1656735, "raw_key_size": 27298169, "raw_average_key_size": 45, "raw_value_size": 47901180, "raw_average_value_size": 79, "num_data_blocks": 15296, "num_entries": 601599, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "81001", "kMergeOperands": "0"}} >2018-07-11 07:19:42.661941 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4200000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:19:43.563531 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 27] Generated table #36404: 121828 keys, 68164636 bytes >2018-07-11 07:19:43.563565 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293583563550, "cf_name": "default", "job": 27, "event": "table_file_creation", "file_number": 36404, "file_size": 68164636, "table_properties": {"data_size": 67109214, "index_size": 595476, "filter_size": 458958, "raw_key_size": 7263071, "raw_average_key_size": 59, "raw_value_size": 60424468, "raw_average_value_size": 495, "num_data_blocks": 15979, "num_entries": 121828, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "29768"}} >2018-07-11 07:19:43.591286 7f354e0ae700 1 bluefs _allocate failed to allocate 0x100000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:19:43.623692 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 27] Generated table #36405: 28498 keys, 972093 bytes >2018-07-11 07:19:43.623730 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293583623704, "cf_name": "default", "job": 27, "event": "table_file_creation", "file_number": 36405, "file_size": 972093, "table_properties": {"data_size": 891862, "index_size": 6035, "filter_size": 73214, "raw_key_size": 512964, "raw_average_key_size": 18, "raw_value_size": 455968, "raw_average_value_size": 16, "num_data_blocks": 220, "num_entries": 28498, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "28498"}} >2018-07-11 07:19:43.623781 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 27] Compacted 1@0 + 12@1 files to L1 => 1279676851 bytes >2018-07-11 07:19:43.638205 7f354e0ae700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 34773957757 >2018-07-11 07:19:43.638338 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:19:43.638163) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 18 55 446 3641 796 0] max score 7.52, MB/sec: 15.7 rd, 15.7 wr, level 1, files in(1, 12) out(18) MB in(510.1, 710.3) out(1220.4), read-write-amplify(4.8) write-amplify(2.4) OK, records in: 61311953, records dropped: 1 > >2018-07-11 07:19:43.638347 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:19:43.638196) EVENT_LOG_v1 {"time_micros": 1531293583638177, "job": 27, "event": "compaction_finished", "compaction_time_micros": 81457429, "output_level": 1, "num_output_files": 18, "total_output_size": 1279676851, "num_input_records": 61311953, "num_output_records": 61311952, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 18, 55, 446, 3641, 796, 0]} >2018-07-11 07:19:43.639318 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 29] Compacting 1@1 + 5@2 files to L2, score 7.52 >2018-07-11 07:19:43.639330 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 27 Base level 1, inputs: [36387(76MB)], [36363(32MB) 36364(1064B) 36365(73MB) 36366(58MB) 36367(39MB)] > >2018-07-11 07:19:43.639345 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293583639334, "job": 29, "event": "compaction_started", "files_L1": [36387], "files_L2": [36363, 36364, 36365, 36366, 36367], "score": 7.52442, "input_data_size": 294245255} >2018-07-11 07:19:47.742955 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 29] Generated table #36406: 3113219 keys, 50460905 bytes >2018-07-11 07:19:47.742986 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293587742972, "cf_name": "default", "job": 29, "event": "table_file_creation", "file_number": 36406, "file_size": 50460905, "table_properties": {"data_size": 42146444, "index_size": 432590, "filter_size": 7880882, "raw_key_size": 89937447, "raw_average_key_size": 28, "raw_value_size": 95, "raw_average_value_size": 0, "num_data_blocks": 10343, "num_entries": 3113219, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "3113211", "kMergeOperands": "0"}} >2018-07-11 07:19:47.984201 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 29] Generated table #36407: 1 keys, 1064 bytes >2018-07-11 07:19:47.984234 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293587984219, "cf_name": "default", "job": 29, "event": "table_file_creation", "file_number": 36407, "file_size": 1064, "table_properties": {"data_size": 57, "index_size": 27, "filter_size": 18, "raw_key_size": 29, "raw_average_key_size": 29, "raw_value_size": 12, "raw_average_value_size": 12, "num_data_blocks": 1, "num_entries": 1, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:19:52.083919 7f354f8b1700 1 bluestore(/var/lib/ceph/osd/ceph-10) _balance_bluefs_freespace gifting 0xd259f00000~80000000 to bluefs >2018-07-11 07:19:52.096309 7f354f8b1700 1 bluefs add_block_extent bdev 2 0xd259f00000~80000000 >2018-07-11 07:19:53.257074 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 29] Generated table #36408: 4509432 keys, 79228658 bytes >2018-07-11 07:19:53.257126 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293593257110, "cf_name": "default", "job": 29, "event": "table_file_creation", "file_number": 36408, "file_size": 79228658, "table_properties": {"data_size": 67110504, "index_size": 689377, "filter_size": 11427784, "raw_key_size": 130272485, "raw_average_key_size": 28, "raw_value_size": 6056581, "raw_average_value_size": 1, "num_data_blocks": 16474, "num_entries": 4509432, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4000000", "kMergeOperands": "0"}} >2018-07-11 07:19:55.119612 7f353f090700 1 bluefs _allocate failed to allocate 0x1d000000 on bdev 1, free 0xfe000; fallback to bdev 2 >2018-07-11 07:20:01.794186 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 29] Generated table #36409: 639019 keys, 69855633 bytes >2018-07-11 07:20:01.794223 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293601794205, "cf_name": "default", "job": 29, "event": "table_file_creation", "file_number": 36409, "file_size": 69855633, "table_properties": {"data_size": 67112180, "index_size": 995619, "filter_size": 1746847, "raw_key_size": 20478377, "raw_average_key_size": 32, "raw_value_size": 57748595, "raw_average_value_size": 90, "num_data_blocks": 16984, "num_entries": 639019, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:20:02.938754 7f353f090700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293602938732, "cf_name": "default", "job": 28, "event": "table_file_creation", "file_number": 36403, "file_size": 486288803, "table_properties": {"data_size": 406137686, "index_size": 4207313, "filter_size": 75942808, "raw_key_size": 866668593, "raw_average_key_size": 28, "raw_value_size": 692, "raw_average_value_size": 0, "num_data_blocks": 99669, "num_entries": 30000031, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "30000030", "kMergeOperands": "0"}} >2018-07-11 07:20:02.938778 7f353f090700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 28] Level-0 flush table #36403: 486288803 bytes OK >2018-07-11 07:20:02.962870 7f353f090700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 34773957757 >2018-07-11 07:20:02.963084 7f353f090700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:242] adding log 36376 to recycle list > >2018-07-11 07:20:02.963094 7f353f090700 4 rocksdb: (Original Log Time 2018/07/11-07:20:02.946261) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #36403 started >2018-07-11 07:20:02.963098 7f353f090700 4 rocksdb: (Original Log Time 2018/07/11-07:20:02.962832) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #36403: memtable #1 done >2018-07-11 07:20:02.963100 7f353f090700 4 rocksdb: (Original Log Time 2018/07/11-07:20:02.962849) EVENT_LOG_v1 {"time_micros": 1531293602962842, "job": 28, "event": "flush_finished", "lsm_state": [1, 18, 55, 446, 3641, 796, 0], "immutable_memtables": 0} >2018-07-11 07:20:02.963102 7f353f090700 4 rocksdb: (Original Log Time 2018/07/11-07:20:02.962886) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[1 18 55 446 3641 796 0] max score 6.93 > >2018-07-11 07:20:02.963936 7f353f090700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 28] Try to delete WAL files size 686669125, prev total WAL file size 755337423, number of live WAL files 2. > >2018-07-11 07:20:02.963981 7f353f090700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293602963976, "job": 28, "event": "table_file_deletion", "file_number": 36377} >2018-07-11 07:20:02.963993 7f353f090700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293602963991, "job": 28, "event": "table_file_deletion", "file_number": 36362} >2018-07-11 07:20:02.964003 7f353f090700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293602964001, "job": 28, "event": "table_file_deletion", "file_number": 36361} >2018-07-11 07:20:02.964014 7f353f090700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293602964012, "job": 28, "event": "table_file_deletion", "file_number": 36360} >2018-07-11 07:20:02.964023 7f353f090700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293602964021, "job": 28, "event": "table_file_deletion", "file_number": 36359} >2018-07-11 07:20:02.964038 7f353f090700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293602964031, "job": 28, "event": "table_file_deletion", "file_number": 36358} >2018-07-11 07:20:02.964053 7f353f090700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293602964051, "job": 28, "event": "table_file_deletion", "file_number": 36357} >2018-07-11 07:20:02.964066 7f353f090700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293602964064, "job": 28, "event": "table_file_deletion", "file_number": 36356} >2018-07-11 07:20:02.964082 7f353f090700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293602964076, "job": 28, "event": "table_file_deletion", "file_number": 36355} >2018-07-11 07:20:02.964094 7f353f090700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293602964092, "job": 28, "event": "table_file_deletion", "file_number": 36354} >2018-07-11 07:20:02.964106 7f353f090700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293602964104, "job": 28, "event": "table_file_deletion", "file_number": 36353} >2018-07-11 07:20:02.964118 7f353f090700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293602964116, "job": 28, "event": "table_file_deletion", "file_number": 36347} >2018-07-11 07:20:02.964130 7f353f090700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293602964128, "job": 28, "event": "table_file_deletion", "file_number": 36330} >2018-07-11 07:20:04.298737 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 29] Generated table #36410: 1192472 keys, 21647380 bytes >2018-07-11 07:20:04.298776 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293604298751, "cf_name": "default", "job": 29, "event": "table_file_creation", "file_number": 36410, "file_size": 21647380, "table_properties": {"data_size": 18434032, "index_size": 188981, "filter_size": 3023377, "raw_key_size": 34449194, "raw_average_key_size": 28, "raw_value_size": 2288280, "raw_average_value_size": 1, "num_data_blocks": 4526, "num_entries": 1192472, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1000000", "kMergeOperands": "0"}} >2018-07-11 07:20:07.966855 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 29] Generated table #36411: 3315389 keys, 57537332 bytes >2018-07-11 07:20:07.966901 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293607966886, "cf_name": "default", "job": 29, "event": "table_file_creation", "file_number": 36411, "file_size": 57537332, "table_properties": {"data_size": 48636478, "index_size": 499471, "filter_size": 8400391, "raw_key_size": 95777904, "raw_average_key_size": 28, "raw_value_size": 3749621, "raw_average_value_size": 1, "num_data_blocks": 11938, "num_entries": 3315389, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "3000000", "kMergeOperands": "0"}} >2018-07-11 07:20:09.033297 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 29] Generated table #36412: 957167 keys, 15514606 bytes >2018-07-11 07:20:09.033325 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293609033311, "cf_name": "default", "job": 29, "event": "table_file_creation", "file_number": 36412, "file_size": 15514606, "table_properties": {"data_size": 12957969, "index_size": 132650, "filter_size": 2423000, "raw_key_size": 27651484, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 3180, "num_entries": 957167, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "957167", "kMergeOperands": "0"}} >2018-07-11 07:20:09.033354 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 29] Compacted 1@1 + 5@2 files to L2 => 294245578 bytes >2018-07-11 07:20:09.047286 7f354e0ae700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 34708537270 >2018-07-11 07:20:09.047782 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:20:09.047243) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[1 17 57 446 3641 796 0] max score 6.93, MB/sec: 11.6 rd, 11.6 wr, level 2, files in(1, 5) out(7) MB in(76.6, 204.0) out(280.6), read-write-amplify(7.3) write-amplify(3.7) OK, records in: 13726699, records dropped: 0 > >2018-07-11 07:20:09.047792 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:20:09.047277) EVENT_LOG_v1 {"time_micros": 1531293609047265, "job": 29, "event": "compaction_finished", "compaction_time_micros": 25393994, "output_level": 2, "num_output_files": 7, "total_output_size": 294245578, "num_input_records": 13726699, "num_output_records": 13726699, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 17, 57, 446, 3641, 796, 0]} >2018-07-11 07:20:09.048718 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 30] Compacting 1@1 + 5@2 files to L2, score 6.93 >2018-07-11 07:20:09.048743 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 29 Base level 1, inputs: [36389(76MB)], [36375(75MB) 36378(72MB) 36379(20MB) 36381(75MB) 36382(57MB)] > >2018-07-11 07:20:09.048758 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293609048747, "job": 30, "event": "compaction_started", "files_L1": [36389], "files_L2": [36375, 36378, 36379, 36381, 36382], "score": 6.92963, "input_data_size": 397507095} >2018-07-11 07:20:11.408923 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293611408915, "job": 0, "event": "table_file_deletion", "file_number": 36387} >2018-07-11 07:20:11.408942 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293611408939, "job": 0, "event": "table_file_deletion", "file_number": 36367} >2018-07-11 07:20:11.408953 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293611408951, "job": 0, "event": "table_file_deletion", "file_number": 36366} >2018-07-11 07:20:11.408964 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293611408961, "job": 0, "event": "table_file_deletion", "file_number": 36365} >2018-07-11 07:20:11.408977 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293611408975, "job": 0, "event": "table_file_deletion", "file_number": 36364} >2018-07-11 07:20:11.408988 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293611408986, "job": 0, "event": "table_file_deletion", "file_number": 36363} >2018-07-11 07:20:14.791692 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 30] Generated table #36413: 4698967 keys, 79705452 bytes >2018-07-11 07:20:14.791747 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293614791717, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 36413, "file_size": 79705452, "table_properties": {"data_size": 67112807, "index_size": 689356, "filter_size": 11902296, "raw_key_size": 135747941, "raw_average_key_size": 28, "raw_value_size": 3495477, "raw_average_value_size": 0, "num_data_blocks": 16473, "num_entries": 4698967, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4404955", "kMergeOperands": "0"}} >2018-07-11 07:20:18.957578 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 30] Generated table #36414: 3250585 keys, 76079150 bytes >2018-07-11 07:20:18.957628 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293618957611, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 36414, "file_size": 76079150, "table_properties": {"data_size": 67112499, "index_size": 689446, "filter_size": 8276213, "raw_key_size": 93905807, "raw_average_key_size": 28, "raw_value_size": 23084294, "raw_average_value_size": 7, "num_data_blocks": 16485, "num_entries": 3250585, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1308916", "kMergeOperands": "0"}} >2018-07-11 07:20:23.531326 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 30] Generated table #36415: 3568521 keys, 71225885 bytes >2018-07-11 07:20:23.531374 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293623531359, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 36415, "file_size": 71225885, "table_properties": {"data_size": 61532131, "index_size": 632083, "filter_size": 9060679, "raw_key_size": 103090619, "raw_average_key_size": 28, "raw_value_size": 13209213, "raw_average_value_size": 3, "num_data_blocks": 15109, "num_entries": 3568521, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "2457466", "kMergeOperands": "0"}} >2018-07-11 07:20:28.900581 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 30] Generated table #36416: 4630818 keys, 79532609 bytes >2018-07-11 07:20:28.900637 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293628900608, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 36416, "file_size": 79532609, "table_properties": {"data_size": 67110490, "index_size": 689447, "filter_size": 11731679, "raw_key_size": 133779172, "raw_average_key_size": 28, "raw_value_size": 4414873, "raw_average_value_size": 0, "num_data_blocks": 16473, "num_entries": 4630818, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4259473", "kMergeOperands": "0"}} >2018-07-11 07:20:34.619595 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 30] Generated table #36417: 4957165 keys, 80348250 bytes >2018-07-11 07:20:34.619650 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293634619622, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 36417, "file_size": 80348250, "table_properties": {"data_size": 67109330, "index_size": 689229, "filter_size": 12548701, "raw_key_size": 143206986, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957165, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957165", "kMergeOperands": "0"}} >2018-07-11 07:20:35.389242 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 30] Generated table #36418: 654859 keys, 10614703 bytes >2018-07-11 07:20:35.389286 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293635389272, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 36418, "file_size": 10614703, "table_properties": {"data_size": 8865389, "index_size": 90601, "filter_size": 1657728, "raw_key_size": 18918164, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 2176, "num_entries": 654859, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "654859", "kMergeOperands": "0"}} >2018-07-11 07:20:35.389314 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 30] Compacted 1@1 + 5@2 files to L2 => 397506049 bytes >2018-07-11 07:20:35.405087 7f354e0ae700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 34633698291 >2018-07-11 07:20:35.405191 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:20:35.405049) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[1 16 58 446 3641 796 0] max score 6.33, MB/sec: 15.1 rd, 15.1 wr, level 2, files in(1, 5) out(6) MB in(76.6, 302.5) out(379.1), read-write-amplify(9.9) write-amplify(4.9) OK, records in: 21760915, records dropped: 0 > >2018-07-11 07:20:35.405200 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:20:35.405078) EVENT_LOG_v1 {"time_micros": 1531293635405063, "job": 30, "event": "compaction_finished", "compaction_time_micros": 26340542, "output_level": 2, "num_output_files": 6, "total_output_size": 397506049, "num_input_records": 21760915, "num_output_records": 21760915, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 16, 58, 446, 3641, 796, 0]} >2018-07-11 07:20:35.406174 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 31] Compacting 1@1 files to L2, score 6.33 >2018-07-11 07:20:35.406184 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 30 Base level 1, inputs: [36394(76MB)] > >2018-07-11 07:20:35.406224 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293635406199, "job": 31, "event": "compaction_started", "files_L1": [36394], "score": 6.33484, "input_data_size": 80348275} >2018-07-11 07:20:37.333970 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293637333951, "job": 0, "event": "table_file_deletion", "file_number": 36389} >2018-07-11 07:20:37.333988 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293637333986, "job": 0, "event": "table_file_deletion", "file_number": 36382} >2018-07-11 07:20:37.334001 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293637333997, "job": 0, "event": "table_file_deletion", "file_number": 36381} >2018-07-11 07:20:37.334012 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293637334010, "job": 0, "event": "table_file_deletion", "file_number": 36379} >2018-07-11 07:20:37.334024 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293637334022, "job": 0, "event": "table_file_deletion", "file_number": 36378} >2018-07-11 07:20:37.334041 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293637334039, "job": 0, "event": "table_file_deletion", "file_number": 36375} >2018-07-11 07:20:40.805243 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 31] Generated table #36419: 4957165 keys, 80348275 bytes >2018-07-11 07:20:40.805297 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293640805269, "cf_name": "default", "job": 31, "event": "table_file_creation", "file_number": 36419, "file_size": 80348275, "table_properties": {"data_size": 67109381, "index_size": 689203, "filter_size": 12548701, "raw_key_size": 143206993, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957165, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957165", "kMergeOperands": "0"}} >2018-07-11 07:20:40.805328 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 31] Compacted 1@1 files to L2 => 80348275 bytes >2018-07-11 07:20:40.821605 7f354e0ae700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 34547300344 >2018-07-11 07:20:40.821739 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:20:40.821567) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[1 15 59 446 3641 796 0] max score 5.74, MB/sec: 14.9 rd, 14.9 wr, level 2, files in(1, 0) out(1) MB in(76.6, 0.0) out(76.6), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 4957165, records dropped: 0 > >2018-07-11 07:20:40.821752 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:20:40.821596) EVENT_LOG_v1 {"time_micros": 1531293640821582, "job": 31, "event": "compaction_finished", "compaction_time_micros": 5399092, "output_level": 2, "num_output_files": 1, "total_output_size": 80348275, "num_input_records": 4957165, "num_output_records": 4957165, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 15, 59, 446, 3641, 796, 0]} >2018-07-11 07:20:40.822970 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 32] Compacting 1@1 + 2@2 files to L2, score 5.74 >2018-07-11 07:20:40.822986 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 31 Base level 1, inputs: [36390(76MB)], [36417(76MB) 36418(10MB)] > >2018-07-11 07:20:40.823016 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293640822991, "job": 32, "event": "compaction_started", "files_L1": [36390], "files_L2": [36417, 36418], "score": 5.74005, "input_data_size": 171311216} >2018-07-11 07:20:41.023521 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293641023514, "job": 0, "event": "table_file_deletion", "file_number": 36394} >2018-07-11 07:20:46.604652 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 32] Generated table #36420: 4957165 keys, 80348257 bytes >2018-07-11 07:20:46.604704 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293646604676, "cf_name": "default", "job": 32, "event": "table_file_creation", "file_number": 36420, "file_size": 80348257, "table_properties": {"data_size": 67109342, "index_size": 689224, "filter_size": 12548701, "raw_key_size": 143207000, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957165, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957165", "kMergeOperands": "0"}} >2018-07-11 07:20:52.334607 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 32] Generated table #36421: 4957169 keys, 80348303 bytes >2018-07-11 07:20:52.334642 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293652334627, "cf_name": "default", "job": 32, "event": "table_file_creation", "file_number": 36421, "file_size": 80348303, "table_properties": {"data_size": 67109337, "index_size": 689263, "filter_size": 12548713, "raw_key_size": 143207106, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957169, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957169", "kMergeOperands": "0"}} >2018-07-11 07:20:53.166906 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 32] Generated table #36422: 654855 keys, 10614644 bytes >2018-07-11 07:20:53.166936 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293653166921, "cf_name": "default", "job": 32, "event": "table_file_creation", "file_number": 36422, "file_size": 10614644, "table_properties": {"data_size": 8865357, "index_size": 90584, "filter_size": 1657718, "raw_key_size": 18918039, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 2176, "num_entries": 654855, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "654855", "kMergeOperands": "0"}} >2018-07-11 07:20:53.166961 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 32] Compacted 1@1 + 2@2 files to L2 => 171311204 bytes >2018-07-11 07:20:53.179729 7f354e0ae700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 34446413965 >2018-07-11 07:20:53.179826 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:20:53.179695) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[1 14 60 446 3641 796 0] max score 5.15, MB/sec: 13.9 rd, 13.9 wr, level 2, files in(1, 2) out(3) MB in(76.6, 86.7) out(163.4), read-write-amplify(4.3) write-amplify(2.1) OK, records in: 10569189, records dropped: 0 > >2018-07-11 07:20:53.179834 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:20:53.179720) EVENT_LOG_v1 {"time_micros": 1531293653179708, "job": 32, "event": "compaction_finished", "compaction_time_micros": 12343932, "output_level": 2, "num_output_files": 3, "total_output_size": 171311204, "num_input_records": 10569189, "num_output_records": 10569189, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 14, 60, 446, 3641, 796, 0]} >2018-07-11 07:20:53.180815 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 33] Compacting 1@1 + 1@2 files to L2, score 5.15 >2018-07-11 07:20:53.180826 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 32 Base level 1, inputs: [36393(76MB)], [32694(39MB)] > >2018-07-11 07:20:53.180839 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293653180830, "job": 33, "event": "compaction_started", "files_L1": [36393], "files_L2": [32694], "score": 5.14525, "input_data_size": 122285051} >2018-07-11 07:20:53.765974 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293653765966, "job": 0, "event": "table_file_deletion", "file_number": 36418} >2018-07-11 07:20:53.765991 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293653765988, "job": 0, "event": "table_file_deletion", "file_number": 36417} >2018-07-11 07:20:53.766002 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293653766000, "job": 0, "event": "table_file_deletion", "file_number": 36390} >2018-07-11 07:20:57.884784 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 33] Generated table #36423: 3652560 keys, 77083378 bytes >2018-07-11 07:20:57.884842 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293657884826, "cf_name": "default", "job": 33, "event": "table_file_creation", "file_number": 36423, "file_size": 77083378, "table_properties": {"data_size": 67109927, "index_size": 689888, "filter_size": 9282570, "raw_key_size": 105518405, "raw_average_key_size": 28, "raw_value_size": 17645051, "raw_average_value_size": 4, "num_data_blocks": 16481, "num_entries": 3652560, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "2168397", "kMergeOperands": "0"}} >2018-07-11 07:21:01.123041 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 33] Generated table #36424: 2788768 keys, 45202046 bytes >2018-07-11 07:21:01.123086 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293661123058, "cf_name": "default", "job": 33, "event": "table_file_creation", "file_number": 36424, "file_size": 45202046, "table_properties": {"data_size": 37753939, "index_size": 387555, "filter_size": 7059563, "raw_key_size": 80564413, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 9266, "num_entries": 2788768, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "2788768", "kMergeOperands": "0"}} >2018-07-11 07:21:01.123118 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 33] Compacted 1@1 + 1@2 files to L2 => 122285424 bytes >2018-07-11 07:21:01.137894 7f354e0ae700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 34327067845 >2018-07-11 07:21:01.137988 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:21:01.137859) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[1 13 61 446 3641 796 0] max score 4.55, MB/sec: 15.4 rd, 15.4 wr, level 2, files in(1, 1) out(2) MB in(76.6, 40.0) out(116.6), read-write-amplify(3.0) write-amplify(1.5) OK, records in: 6441328, records dropped: 0 > >2018-07-11 07:21:01.137997 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:21:01.137884) EVENT_LOG_v1 {"time_micros": 1531293661137873, "job": 33, "event": "compaction_finished", "compaction_time_micros": 7942267, "output_level": 2, "num_output_files": 2, "total_output_size": 122285424, "num_input_records": 6441328, "num_output_records": 6441328, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 13, 61, 446, 3641, 796, 0]} >2018-07-11 07:21:01.138973 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 34] Compacting 1@1 + 7@2 files to L2, score 4.55 >2018-07-11 07:21:01.138986 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 33 Base level 1, inputs: [36388(76MB)], [36368(30MB) 36369(14MB) 36370(47MB) 36371(76MB) 36372(71MB) 36373(30MB) 36413(76MB)] > >2018-07-11 07:21:01.139014 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293661139003, "job": 34, "event": "compaction_started", "files_L1": [36388], "files_L2": [36368, 36369, 36370, 36371, 36372, 36373, 36413], "score": 4.55046, "input_data_size": 443922745} >2018-07-11 07:21:03.079483 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293663079476, "job": 0, "event": "table_file_deletion", "file_number": 36393} >2018-07-11 07:21:03.079514 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293663079510, "job": 0, "event": "table_file_deletion", "file_number": 32694} >2018-07-11 07:21:03.777533 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 34] Generated table #36425: 2042833 keys, 33111534 bytes >2018-07-11 07:21:03.777587 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293663777559, "cf_name": "default", "job": 34, "event": "table_file_creation", "file_number": 36425, "file_size": 33111534, "table_properties": {"data_size": 27655577, "index_size": 283691, "filter_size": 5171279, "raw_key_size": 59015186, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 6787, "num_entries": 2042833, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "2042833", "kMergeOperands": "0"}} >2018-07-11 07:21:07.382739 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 34] Generated table #36426: 3000000 keys, 48625679 bytes >2018-07-11 07:21:07.382771 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293667382757, "cf_name": "default", "job": 34, "event": "table_file_creation", "file_number": 36426, "file_size": 48625679, "table_properties": {"data_size": 40613507, "index_size": 416908, "filter_size": 7594275, "raw_key_size": 86666670, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 9967, "num_entries": 3000000, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "3000000", "kMergeOperands": "0"}} >2018-07-11 07:21:13.179739 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 34] Generated table #36427: 4957168 keys, 80348314 bytes >2018-07-11 07:21:13.179788 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293673179771, "cf_name": "default", "job": 34, "event": "table_file_creation", "file_number": 36427, "file_size": 80348314, "table_properties": {"data_size": 67109372, "index_size": 689242, "filter_size": 12548710, "raw_key_size": 143207073, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957168, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957168", "kMergeOperands": "0"}} >2018-07-11 07:21:18.476876 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 34] Generated table #36428: 4529496 keys, 79279545 bytes >2018-07-11 07:21:18.476928 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293678476900, "cf_name": "default", "job": 34, "event": "table_file_creation", "file_number": 36428, "file_size": 79279545, "table_properties": {"data_size": 67111129, "index_size": 689405, "filter_size": 11478018, "raw_key_size": 130852123, "raw_average_key_size": 28, "raw_value_size": 5785898, "raw_average_value_size": 1, "num_data_blocks": 16474, "num_entries": 4529496, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4042832", "kMergeOperands": "0"}} >2018-07-11 07:21:22.217386 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 34] Generated table #36429: 3094597 keys, 75688215 bytes >2018-07-11 07:21:22.217432 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293682217405, "cf_name": "default", "job": 34, "event": "table_file_creation", "file_number": 36429, "file_size": 75688215, "table_properties": {"data_size": 67111344, "index_size": 690189, "filter_size": 7885690, "raw_key_size": 89399461, "raw_average_key_size": 28, "raw_value_size": 25192839, "raw_average_value_size": 8, "num_data_blocks": 16486, "num_entries": 3094597, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "975573", "kMergeOperands": "0"}} >2018-07-11 07:21:22.272714 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 34] Generated table #36430: 26660 keys, 459979 bytes >2018-07-11 07:21:22.272751 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293682272727, "cf_name": "default", "job": 34, "event": "table_file_creation", "file_number": 36430, "file_size": 459979, "table_properties": {"data_size": 387549, "index_size": 3901, "filter_size": 67548, "raw_key_size": 770186, "raw_average_key_size": 28, "raw_value_size": 26549, "raw_average_value_size": 0, "num_data_blocks": 96, "num_entries": 26660, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "24427", "kMergeOperands": "0"}} >2018-07-11 07:21:28.128949 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 34] Generated table #36431: 4930748 keys, 80285331 bytes >2018-07-11 07:21:28.128998 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293688128970, "cf_name": "default", "job": 34, "event": "table_file_creation", "file_number": 36431, "file_size": 80285331, "table_properties": {"data_size": 67112510, "index_size": 689259, "filter_size": 12482570, "raw_key_size": 142443818, "raw_average_key_size": 28, "raw_value_size": 360440, "raw_average_value_size": 0, "num_data_blocks": 16471, "num_entries": 4930748, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4900430", "kMergeOperands": "0"}} >2018-07-11 07:21:31.283299 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 34] Generated table #36432: 2627200 keys, 46125427 bytes >2018-07-11 07:21:31.283343 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293691283327, "cf_name": "default", "job": 34, "event": "table_file_creation", "file_number": 36432, "file_size": 46125427, "table_properties": {"data_size": 39065494, "index_size": 401170, "filter_size": 6657771, "raw_key_size": 75896888, "raw_average_key_size": 28, "raw_value_size": 3495477, "raw_average_value_size": 1, "num_data_blocks": 9590, "num_entries": 2627200, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "2333188", "kMergeOperands": "0"}} >2018-07-11 07:21:31.283381 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 34] Compacted 1@1 + 7@2 files to L2 => 443924024 bytes >2018-07-11 07:21:31.303967 7f354e0ae700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 34183689757 >2018-07-11 07:21:31.304092 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:21:31.303932) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[1 12 62 446 3641 796 0] max score 3.96, MB/sec: 14.7 rd, 14.7 wr, level 2, files in(1, 7) out(8) MB in(76.6, 346.7) out(423.4), read-write-amplify(11.0) write-amplify(5.5) OK, records in: 25208702, records dropped: 0 > >2018-07-11 07:21:31.304101 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:21:31.303958) EVENT_LOG_v1 {"time_micros": 1531293691303946, "job": 34, "event": "compaction_finished", "compaction_time_micros": 30144351, "output_level": 2, "num_output_files": 8, "total_output_size": 443924024, "num_input_records": 25208702, "num_output_records": 25208702, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 12, 62, 446, 3641, 796, 0]} >2018-07-11 07:21:31.305118 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 35] Compacting 1@1 + 2@2 files to L2, score 3.96 >2018-07-11 07:21:31.305129 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 34 Base level 1, inputs: [36392(76MB)], [36384(76MB) 36385(24MB)] > >2018-07-11 07:21:31.305156 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293691305133, "job": 35, "event": "compaction_started", "files_L1": [36392], "files_L2": [36384, 36385], "score": 3.95567, "input_data_size": 186404582} >2018-07-11 07:21:32.289176 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293692289158, "job": 0, "event": "table_file_deletion", "file_number": 36413} >2018-07-11 07:21:32.289206 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293692289189, "job": 0, "event": "table_file_deletion", "file_number": 36388} >2018-07-11 07:21:32.289220 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293692289218, "job": 0, "event": "table_file_deletion", "file_number": 36373} >2018-07-11 07:21:32.289232 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293692289230, "job": 0, "event": "table_file_deletion", "file_number": 36372} >2018-07-11 07:21:32.289256 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293692289254, "job": 0, "event": "table_file_deletion", "file_number": 36371} >2018-07-11 07:21:32.289271 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293692289268, "job": 0, "event": "table_file_deletion", "file_number": 36370} >2018-07-11 07:21:32.289286 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293692289284, "job": 0, "event": "table_file_deletion", "file_number": 36369} >2018-07-11 07:21:32.289299 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293692289297, "job": 0, "event": "table_file_deletion", "file_number": 36368} >2018-07-11 07:21:37.573455 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 35] Generated table #36433: 4957164 keys, 80348255 bytes >2018-07-11 07:21:37.573491 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293697573475, "cf_name": "default", "job": 35, "event": "table_file_creation", "file_number": 36433, "file_size": 80348255, "table_properties": {"data_size": 67109336, "index_size": 689231, "filter_size": 12548698, "raw_key_size": 143206961, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957164, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957164", "kMergeOperands": "0"}} >2018-07-11 07:21:43.693369 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 35] Generated table #36434: 4957166 keys, 80348288 bytes >2018-07-11 07:21:43.693409 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293703693393, "cf_name": "default", "job": 35, "event": "table_file_creation", "file_number": 36434, "file_size": 80348288, "table_properties": {"data_size": 67109392, "index_size": 689202, "filter_size": 12548704, "raw_key_size": 143207029, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957166, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957166", "kMergeOperands": "0"}} >2018-07-11 07:21:45.470645 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 35] Generated table #36435: 1586062 keys, 25708014 bytes >2018-07-11 07:21:45.470690 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293705470675, "cf_name": "default", "job": 35, "event": "table_file_creation", "file_number": 36435, "file_size": 25708014, "table_properties": {"data_size": 21471852, "index_size": 220175, "filter_size": 4015000, "raw_key_size": 45819565, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 5270, "num_entries": 1586062, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1586062", "kMergeOperands": "0"}} >2018-07-11 07:21:45.470733 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 35] Compacted 1@1 + 2@2 files to L2 => 186404557 bytes >2018-07-11 07:21:45.487065 7f354e0ae700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 34008161659 >2018-07-11 07:21:45.487556 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:21:45.487019) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[1 11 63 446 3641 796 0] max score 3.60, MB/sec: 13.2 rd, 13.2 wr, level 2, files in(1, 2) out(3) MB in(76.6, 101.1) out(177.8), read-write-amplify(4.6) write-amplify(2.3) OK, records in: 11500392, records dropped: 0 > >2018-07-11 07:21:45.487565 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:21:45.487055) EVENT_LOG_v1 {"time_micros": 1531293705487033, "job": 35, "event": "compaction_finished", "compaction_time_micros": 14165564, "output_level": 2, "num_output_files": 3, "total_output_size": 186404557, "num_input_records": 11500392, "num_output_records": 11500392, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 11, 63, 446, 3641, 796, 0]} >2018-07-11 07:21:45.488191 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293705488174, "job": 35, "event": "table_file_deletion", "file_number": 36392} >2018-07-11 07:21:45.488210 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293705488208, "job": 35, "event": "table_file_deletion", "file_number": 36385} >2018-07-11 07:21:45.488222 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293705488220, "job": 35, "event": "table_file_deletion", "file_number": 36384} >2018-07-11 07:21:45.488427 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 36] Compacting 1@0 + 10@1 files to L1, score 3.60 >2018-07-11 07:21:45.488442 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 35 Base level 0, inputs: [36403(463MB)], [36386(76MB) 36391(76MB) 36395(74MB) 36396(40MB) 36397(71MB) 36398(71MB) 36399(71MB) 36400(71MB) 36401(66MB) 36404(65MB)] > >2018-07-11 07:21:45.488457 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293705488445, "job": 36, "event": "compaction_started", "files_L0": [36403], "files_L1": [36386, 36391, 36395, 36396, 36397, 36398, 36399, 36400, 36401, 36404], "score": 3.59971, "input_data_size": 1202555634} >2018-07-11 07:21:49.485811 7f354f8b1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:684] reusing log 36376 from recycle list > >2018-07-11 07:21:49.486053 7f354f8b1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #36437. Immutable memtables: 0. > >2018-07-11 07:21:49.486070 7f354f8b1700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 34008161659 >2018-07-11 07:21:49.538005 7f353f090700 4 rocksdb: (Original Log Time 2018/07/11-07:21:49.486117) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 0 >2018-07-11 07:21:49.538019 7f353f090700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 37] Flushing memtable with next log file: 36437 > >2018-07-11 07:21:49.538036 7f353f090700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293709538027, "job": 37, "event": "flush_started", "num_memtables": 1, "num_entries": 33000035, "num_deletes": 33000033, "memory_usage": 1408029896} >2018-07-11 07:21:49.538039 7f353f090700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 37] Level-0 flush table #36438: started >2018-07-11 07:21:50.739258 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 36] Generated table #36436: 4888760 keys, 80173882 bytes >2018-07-11 07:21:50.739299 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293710739282, "cf_name": "default", "job": 36, "event": "table_file_creation", "file_number": 36436, "file_size": 80173882, "table_properties": {"data_size": 67110184, "index_size": 685246, "filter_size": 12377461, "raw_key_size": 141267959, "raw_average_key_size": 28, "raw_value_size": 916542, "raw_average_value_size": 0, "num_data_blocks": 16336, "num_entries": 4888760, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4886796", "kMergeOperands": "0"}} >2018-07-11 07:21:56.065330 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 36] Generated table #36439: 4957166 keys, 80348285 bytes >2018-07-11 07:21:56.065365 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293716065350, "cf_name": "default", "job": 36, "event": "table_file_creation", "file_number": 36439, "file_size": 80348285, "table_properties": {"data_size": 67109353, "index_size": 689238, "filter_size": 12548704, "raw_key_size": 143207015, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957166, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957166", "kMergeOperands": "0"}} >2018-07-11 07:22:01.003991 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 36] Generated table #36440: 4957165 keys, 80348258 bytes >2018-07-11 07:22:01.004028 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293721004010, "cf_name": "default", "job": 36, "event": "table_file_creation", "file_number": 36440, "file_size": 80348258, "table_properties": {"data_size": 67109344, "index_size": 689223, "filter_size": 12548701, "raw_key_size": 143206995, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957165, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957165", "kMergeOperands": "0"}} >2018-07-11 07:22:06.057923 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 36] Generated table #36441: 4957165 keys, 80348252 bytes >2018-07-11 07:22:06.057961 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293726057946, "cf_name": "default", "job": 36, "event": "table_file_creation", "file_number": 36441, "file_size": 80348252, "table_properties": {"data_size": 67109336, "index_size": 689225, "filter_size": 12548701, "raw_key_size": 143206995, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957165, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957165", "kMergeOperands": "0"}} >2018-07-11 07:22:06.923066 7f353f090700 1 bluefs _allocate failed to allocate 0x1ff00000 on bdev 1, free 0x150fe000; fallback to bdev 2 >2018-07-11 07:22:15.164162 7f353f090700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293735164141, "cf_name": "default", "job": 37, "event": "table_file_creation", "file_number": 36438, "file_size": 534924140, "table_properties": {"data_size": 446751377, "index_size": 4634679, "filter_size": 83537088, "raw_key_size": 953335448, "raw_average_key_size": 28, "raw_value_size": 692, "raw_average_value_size": 0, "num_data_blocks": 109636, "num_entries": 33000034, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "33000033", "kMergeOperands": "0"}} >2018-07-11 07:22:15.164181 7f353f090700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 37] Level-0 flush table #36438: 534924140 bytes OK >2018-07-11 07:22:15.178104 7f353f090700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 34008161659 >2018-07-11 07:22:15.178218 7f353f090700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:242] adding log 36402 to recycle list > >2018-07-11 07:22:15.178228 7f353f090700 4 rocksdb: (Original Log Time 2018/07/11-07:22:15.164205) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #36438 started >2018-07-11 07:22:15.178231 7f353f090700 4 rocksdb: (Original Log Time 2018/07/11-07:22:15.178079) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #36438: memtable #1 done >2018-07-11 07:22:15.178233 7f353f090700 4 rocksdb: (Original Log Time 2018/07/11-07:22:15.178096) EVENT_LOG_v1 {"time_micros": 1531293735178088, "job": 37, "event": "flush_finished", "lsm_state": [2, 11, 63, 446, 3641, 796, 0], "immutable_memtables": 0} >2018-07-11 07:22:15.178235 7f353f090700 4 rocksdb: (Original Log Time 2018/07/11-07:22:15.178133) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[2 11 63 446 3641 796 0] max score 3.96 > >2018-07-11 07:22:15.179074 7f353f090700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 37] Try to delete WAL files size 755336698, prev total WAL file size 755336710, number of live WAL files 2. > >2018-07-11 07:22:17.376379 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 36] Generated table #36442: 4957166 keys, 80348286 bytes >2018-07-11 07:22:17.376430 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293737376413, "cf_name": "default", "job": 36, "event": "table_file_creation", "file_number": 36442, "file_size": 80348286, "table_properties": {"data_size": 67109401, "index_size": 689191, "filter_size": 12548704, "raw_key_size": 143207030, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957166, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957166", "kMergeOperands": "0"}} >2018-07-11 07:22:22.321970 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 36] Generated table #36443: 4957165 keys, 80348222 bytes >2018-07-11 07:22:22.322022 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293742322005, "cf_name": "default", "job": 36, "event": "table_file_creation", "file_number": 36443, "file_size": 80348222, "table_properties": {"data_size": 67109313, "index_size": 689218, "filter_size": 12548701, "raw_key_size": 143206990, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957165, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957165", "kMergeOperands": "0"}} >2018-07-11 07:22:27.441407 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 36] Generated table #36444: 4957165 keys, 80348304 bytes >2018-07-11 07:22:27.441457 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293747441440, "cf_name": "default", "job": 36, "event": "table_file_creation", "file_number": 36444, "file_size": 80348304, "table_properties": {"data_size": 67109378, "index_size": 689235, "filter_size": 12548701, "raw_key_size": 143206995, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957165, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957165", "kMergeOperands": "0"}} >2018-07-11 07:22:31.543163 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 36] Generated table #36445: 4214172 keys, 68305457 bytes >2018-07-11 07:22:31.543199 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293751543183, "cf_name": "default", "job": 36, "event": "table_file_creation", "file_number": 36445, "file_size": 68305457, "table_properties": {"data_size": 57050864, "index_size": 585745, "filter_size": 10667859, "raw_key_size": 121742762, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 14001, "num_entries": 4214172, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4214172", "kMergeOperands": "0"}} >2018-07-11 07:22:35.551380 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4c00000 on bdev 1, free 0x27fe000; fallback to bdev 2 >2018-07-11 07:22:36.658949 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 36] Generated table #36446: 4435540 keys, 79045341 bytes >2018-07-11 07:22:36.659002 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293756658985, "cf_name": "default", "job": 36, "event": "table_file_creation", "file_number": 36446, "file_size": 79045341, "table_properties": {"data_size": 67112229, "index_size": 689322, "filter_size": 11242797, "raw_key_size": 128137783, "raw_average_key_size": 28, "raw_value_size": 7057654, "raw_average_value_size": 1, "num_data_blocks": 16475, "num_entries": 4435540, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "3842284", "kMergeOperands": "0"}} >2018-07-11 07:22:39.389873 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4800000 on bdev 1, free 0x27fe000; fallback to bdev 2 >2018-07-11 07:22:40.446701 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 36] Generated table #36447: 2638241 keys, 74546278 bytes >2018-07-11 07:22:40.446746 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293760446718, "cf_name": "default", "job": 36, "event": "table_file_creation", "file_number": 36447, "file_size": 74546278, "table_properties": {"data_size": 67112144, "index_size": 689961, "filter_size": 6743183, "raw_key_size": 76215845, "raw_average_key_size": 28, "raw_value_size": 31365748, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:22:43.018228 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4800000 on bdev 1, free 0x27fe000; fallback to bdev 2 >2018-07-11 07:22:44.020913 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 36] Generated table #36448: 2638241 keys, 74545916 bytes >2018-07-11 07:22:44.020957 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293764020929, "cf_name": "default", "job": 36, "event": "table_file_creation", "file_number": 36448, "file_size": 74545916, "table_properties": {"data_size": 67112140, "index_size": 689603, "filter_size": 6743183, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:22:46.769938 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4800000 on bdev 1, free 0x27fe000; fallback to bdev 2 >2018-07-11 07:22:47.805856 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 36] Generated table #36449: 2638242 keys, 74546606 bytes >2018-07-11 07:22:47.805902 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293767805887, "cf_name": "default", "job": 36, "event": "table_file_creation", "file_number": 36449, "file_size": 74546606, "table_properties": {"data_size": 67112170, "index_size": 690260, "filter_size": 6743186, "raw_key_size": 76215881, "raw_average_key_size": 28, "raw_value_size": 31365767, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638242, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:22:50.313072 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4800000 on bdev 1, free 0x27fe000; fallback to bdev 2 >2018-07-11 07:22:51.308864 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 36] Generated table #36450: 2638241 keys, 74546211 bytes >2018-07-11 07:22:51.308908 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293771308894, "cf_name": "default", "job": 36, "event": "table_file_creation", "file_number": 36450, "file_size": 74546211, "table_properties": {"data_size": 67112157, "index_size": 689881, "filter_size": 6743183, "raw_key_size": 76215860, "raw_average_key_size": 28, "raw_value_size": 31365763, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:22:53.653157 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4700000 on bdev 1, free 0x27fe000; fallback to bdev 2 >2018-07-11 07:22:54.611077 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 36] Generated table #36451: 2478832 keys, 74136181 bytes >2018-07-11 07:22:54.611123 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293774611093, "cf_name": "default", "job": 36, "event": "table_file_creation", "file_number": 36451, "file_size": 74136181, "table_properties": {"data_size": 67112097, "index_size": 678060, "filter_size": 6345035, "raw_key_size": 72057444, "raw_average_key_size": 29, "raw_value_size": 33013382, "raw_average_value_size": 13, "num_data_blocks": 16372, "num_entries": 2478832, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "8", "kMergeOperands": "0"}} >2018-07-11 07:22:55.579485 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 36] Generated table #36452: 126193 keys, 38575339 bytes >2018-07-11 07:22:55.579515 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293775579501, "cf_name": "default", "job": 36, "event": "table_file_creation", "file_number": 36452, "file_size": 38575339, "table_properties": {"data_size": 37884380, "index_size": 288028, "filter_size": 401943, "raw_key_size": 11702392, "raw_average_key_size": 92, "raw_value_size": 27257238, "raw_average_value_size": 215, "num_data_blocks": 8245, "num_entries": 126193, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "80868", "kMergeOperands": "0"}} >2018-07-11 07:22:56.681501 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4100000 on bdev 1, free 0x2fe000; fallback to bdev 2 >2018-07-11 07:22:57.566066 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 36] Generated table #36453: 83932 keys, 68074201 bytes >2018-07-11 07:22:57.566107 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293777566093, "cf_name": "default", "job": 36, "event": "table_file_creation", "file_number": 36453, "file_size": 68074201, "table_properties": {"data_size": 67113104, "index_size": 595488, "filter_size": 364622, "raw_key_size": 7301929, "raw_average_key_size": 86, "raw_value_size": 60274154, "raw_average_value_size": 718, "num_data_blocks": 16079, "num_entries": 83932, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "155", "kMergeOperands": "0"}} >2018-07-11 07:22:57.746914 7f354e0ae700 1 bluefs _allocate failed to allocate 0xe00000 on bdev 1, free 0x2fe000; fallback to bdev 2 >2018-07-11 07:22:57.955919 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 36] Generated table #36454: 59942 keys, 13939175 bytes >2018-07-11 07:22:57.955947 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293777955933, "cf_name": "default", "job": 36, "event": "table_file_creation", "file_number": 36454, "file_size": 13939175, "table_properties": {"data_size": 13637326, "index_size": 119765, "filter_size": 181097, "raw_key_size": 2014117, "raw_average_key_size": 33, "raw_value_size": 11861133, "raw_average_value_size": 197, "num_data_blocks": 3241, "num_entries": 59942, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "29768"}} >2018-07-11 07:22:57.956042 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 36] Compacted 1@0 + 10@1 files to L1 => 1202524194 bytes >2018-07-11 07:22:57.977361 7f354e0ae700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 42537868626 >2018-07-11 07:22:57.977721 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:22:57.977325) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[1 18 63 446 3641 796 0] max score 6.99, MB/sec: 16.6 rd, 16.6 wr, level 1, files in(1, 10) out(17) MB in(463.8, 683.1) out(1146.8), read-write-amplify(4.9) write-amplify(2.5) OK, records in: 56583329, records dropped: 1 > >2018-07-11 07:22:57.977729 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:22:57.977352) EVENT_LOG_v1 {"time_micros": 1531293777977339, "job": 36, "event": "compaction_finished", "compaction_time_micros": 72467558, "output_level": 1, "num_output_files": 17, "total_output_size": 1202524194, "num_input_records": 56583329, "num_output_records": 56583328, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 18, 63, 446, 3641, 796, 0]} >2018-07-11 07:22:57.978763 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 38] Compacting 1@1 + 2@2 files to L2, score 6.99 >2018-07-11 07:22:57.978775 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 37 Base level 1, inputs: [36444(76MB)], [36433(76MB) 36434(76MB)] > >2018-07-11 07:22:57.978789 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293777978779, "job": 38, "event": "compaction_started", "files_L1": [36444], "files_L2": [36433, 36434], "score": 6.99431, "input_data_size": 241044847} >2018-07-11 07:23:00.037825 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293780037807, "job": 0, "event": "table_file_deletion", "file_number": 36404} >2018-07-11 07:23:00.037842 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293780037837, "job": 0, "event": "table_file_deletion", "file_number": 36403} >2018-07-11 07:23:00.037853 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293780037851, "job": 0, "event": "table_file_deletion", "file_number": 36401} >2018-07-11 07:23:00.037863 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293780037861, "job": 0, "event": "table_file_deletion", "file_number": 36400} >2018-07-11 07:23:00.037873 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293780037871, "job": 0, "event": "table_file_deletion", "file_number": 36399} >2018-07-11 07:23:00.037882 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293780037880, "job": 0, "event": "table_file_deletion", "file_number": 36398} >2018-07-11 07:23:00.037892 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293780037890, "job": 0, "event": "table_file_deletion", "file_number": 36397} >2018-07-11 07:23:00.037901 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293780037899, "job": 0, "event": "table_file_deletion", "file_number": 36396} >2018-07-11 07:23:00.037916 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293780037912, "job": 0, "event": "table_file_deletion", "file_number": 36395} >2018-07-11 07:23:00.037926 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293780037924, "job": 0, "event": "table_file_deletion", "file_number": 36391} >2018-07-11 07:23:00.037941 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293780037939, "job": 0, "event": "table_file_deletion", "file_number": 36386} >2018-07-11 07:23:03.839176 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 38] Generated table #36455: 4957164 keys, 80348291 bytes >2018-07-11 07:23:03.839215 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293783839199, "cf_name": "default", "job": 38, "event": "table_file_creation", "file_number": 36455, "file_size": 80348291, "table_properties": {"data_size": 67109374, "index_size": 689229, "filter_size": 12548698, "raw_key_size": 143206978, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957164, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957164", "kMergeOperands": "0"}} >2018-07-11 07:23:09.383097 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 38] Generated table #36456: 4957167 keys, 80348258 bytes >2018-07-11 07:23:09.383145 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293789383117, "cf_name": "default", "job": 38, "event": "table_file_creation", "file_number": 36456, "file_size": 80348258, "table_properties": {"data_size": 67109308, "index_size": 689253, "filter_size": 12548707, "raw_key_size": 143207050, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957167, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957167", "kMergeOperands": "0"}} >2018-07-11 07:23:14.901495 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 38] Generated table #36457: 4957164 keys, 80348178 bytes >2018-07-11 07:23:14.901557 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293794901529, "cf_name": "default", "job": 38, "event": "table_file_creation", "file_number": 36457, "file_size": 80348178, "table_properties": {"data_size": 67109343, "index_size": 689157, "filter_size": 12548688, "raw_key_size": 143206957, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16469, "num_entries": 4957164, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957164", "kMergeOperands": "0"}} >2018-07-11 07:23:14.901632 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 38] Compacted 1@1 + 2@2 files to L2 => 241044727 bytes >2018-07-11 07:23:14.918568 7f354e0ae700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 42455821987 >2018-07-11 07:23:14.918681 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:23:14.918529) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[1 17 64 446 3641 796 0] max score 6.40, MB/sec: 14.2 rd, 14.2 wr, level 2, files in(1, 2) out(3) MB in(76.6, 153.3) out(229.9), read-write-amplify(6.0) write-amplify(3.0) OK, records in: 14871495, records dropped: 0 > >2018-07-11 07:23:14.918691 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:23:14.918559) EVENT_LOG_v1 {"time_micros": 1531293794918543, "job": 38, "event": "compaction_finished", "compaction_time_micros": 16922829, "output_level": 2, "num_output_files": 3, "total_output_size": 241044727, "num_input_records": 14871495, "num_output_records": 14871495, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 17, 64, 446, 3641, 796, 0]} >2018-07-11 07:23:14.919710 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 39] Compacting 1@1 + 4@2 files to L2, score 6.40 >2018-07-11 07:23:14.919722 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 38 Base level 1, inputs: [36442(76MB)], [36420(76MB) 36421(76MB) 36422(10MB) 36383(76MB)] > >2018-07-11 07:23:14.919749 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293794919725, "job": 39, "event": "compaction_started", "files_L1": [36442], "files_L2": [36420, 36421, 36422, 36383], "score": 6.39952, "input_data_size": 332007804} >2018-07-11 07:23:14.980334 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293794980317, "job": 0, "event": "table_file_deletion", "file_number": 36444} >2018-07-11 07:23:14.980354 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293794980351, "job": 0, "event": "table_file_deletion", "file_number": 36434} >2018-07-11 07:23:14.980365 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293794980363, "job": 0, "event": "table_file_deletion", "file_number": 36433} >2018-07-11 07:23:20.812192 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 39] Generated table #36458: 4957164 keys, 80348267 bytes >2018-07-11 07:23:20.812250 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293800812228, "cf_name": "default", "job": 39, "event": "table_file_creation", "file_number": 36458, "file_size": 80348267, "table_properties": {"data_size": 67109347, "index_size": 689232, "filter_size": 12548698, "raw_key_size": 143206976, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957164, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957164", "kMergeOperands": "0"}} >2018-07-11 07:23:26.746108 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 39] Generated table #36459: 4957165 keys, 80348238 bytes >2018-07-11 07:23:26.746149 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293806746130, "cf_name": "default", "job": 39, "event": "table_file_creation", "file_number": 36459, "file_size": 80348238, "table_properties": {"data_size": 67109322, "index_size": 689225, "filter_size": 12548701, "raw_key_size": 143206991, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957165, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957165", "kMergeOperands": "0"}} >2018-07-11 07:23:37.057115 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 39] Generated table #36460: 4957165 keys, 80348253 bytes >2018-07-11 07:23:37.057175 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293817057157, "cf_name": "default", "job": 39, "event": "table_file_creation", "file_number": 36460, "file_size": 80348253, "table_properties": {"data_size": 67109339, "index_size": 689223, "filter_size": 12548701, "raw_key_size": 143206990, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957165, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957165", "kMergeOperands": "0"}} >2018-07-11 07:23:38.134828 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 39] Generated table #36461: 997537 keys, 16169000 bytes >2018-07-11 07:23:38.134878 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293818134862, "cf_name": "default", "job": 39, "event": "table_file_creation", "file_number": 36461, "file_size": 16169000, "table_properties": {"data_size": 13504516, "index_size": 138301, "filter_size": 2525196, "raw_key_size": 28817754, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 3315, "num_entries": 997537, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "997537", "kMergeOperands": "0"}} >2018-07-11 07:23:41.467063 7f354f8b1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:684] reusing log 36402 from recycle list > >2018-07-11 07:23:41.467178 7f354f8b1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #36463. Immutable memtables: 0. > >2018-07-11 07:23:41.467194 7f354f8b1700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 42455821987 >2018-07-11 07:23:41.492952 7f353f090700 4 rocksdb: (Original Log Time 2018/07/11-07:23:41.467243) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 0 >2018-07-11 07:23:41.492966 7f353f090700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 40] Flushing memtable with next log file: 36463 > >2018-07-11 07:23:41.492983 7f353f090700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293821492974, "job": 40, "event": "flush_started", "num_memtables": 1, "num_entries": 30000030, "num_deletes": 30000030, "memory_usage": 1279958200} >2018-07-11 07:23:41.492986 7f353f090700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 40] Level-0 flush table #36464: started >2018-07-11 07:23:43.131978 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 39] Generated table #36462: 4614492 keys, 74794054 bytes >2018-07-11 07:23:43.132015 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293823131999, "cf_name": "default", "job": 39, "event": "table_file_creation", "file_number": 36462, "file_size": 74794054, "table_properties": {"data_size": 62470292, "index_size": 641531, "filter_size": 11681242, "raw_key_size": 133307537, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 15331, "num_entries": 4614492, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4614492", "kMergeOperands": "0"}} >2018-07-11 07:23:43.132050 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 39] Compacted 1@1 + 4@2 files to L2 => 332007812 bytes >2018-07-11 07:23:43.151348 7f354e0ae700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 42361134179 >2018-07-11 07:23:43.151454 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:23:43.151313) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[1 16 65 446 3641 796 0] max score 5.80, MB/sec: 11.8 rd, 11.8 wr, level 2, files in(1, 4) out(5) MB in(76.6, 240.0) out(316.6), read-write-amplify(8.3) write-amplify(4.1) OK, records in: 20483523, records dropped: 0 > >2018-07-11 07:23:43.151462 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:23:43.151339) EVENT_LOG_v1 {"time_micros": 1531293823151327, "job": 39, "event": "compaction_finished", "compaction_time_micros": 28212287, "output_level": 2, "num_output_files": 5, "total_output_size": 332007812, "num_input_records": 20483523, "num_output_records": 20483523, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 16, 65, 446, 3641, 796, 0]} >2018-07-11 07:23:43.152472 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 41] Compacting 1@1 + 1@2 files to L2, score 5.80 >2018-07-11 07:23:43.152483 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 39 Base level 1, inputs: [36439(76MB)], [36406(48MB)] > >2018-07-11 07:23:43.152496 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293823152486, "job": 41, "event": "compaction_started", "files_L1": [36439], "files_L2": [36406], "score": 5.80473, "input_data_size": 130809190} >2018-07-11 07:23:49.063153 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 41] Generated table #36465: 4957159 keys, 80348277 bytes >2018-07-11 07:23:49.063212 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293829063195, "cf_name": "default", "job": 41, "event": "table_file_creation", "file_number": 36465, "file_size": 80348277, "table_properties": {"data_size": 67109420, "index_size": 689181, "filter_size": 12548686, "raw_key_size": 143206822, "raw_average_key_size": 28, "raw_value_size": 60, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957159, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957154", "kMergeOperands": "0"}} >2018-07-11 07:23:52.474057 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 41] Generated table #36466: 2785452 keys, 45148266 bytes >2018-07-11 07:23:52.474092 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293832474076, "cf_name": "default", "job": 41, "event": "table_file_creation", "file_number": 36466, "file_size": 45148266, "table_properties": {"data_size": 37709079, "index_size": 387025, "filter_size": 7051173, "raw_key_size": 80468600, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 9255, "num_entries": 2785452, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "2785452", "kMergeOperands": "0"}} >2018-07-11 07:23:52.946214 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 41] Generated table #36467: 327774 keys, 5313199 bytes >2018-07-11 07:23:52.946253 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293832946231, "cf_name": "default", "job": 41, "event": "table_file_creation", "file_number": 36467, "file_size": 5313199, "table_properties": {"data_size": 4437399, "index_size": 45078, "filter_size": 829737, "raw_key_size": 9469040, "raw_average_key_size": 28, "raw_value_size": 35, "raw_average_value_size": 0, "num_data_blocks": 1089, "num_entries": 327774, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "327771", "kMergeOperands": "0"}} >2018-07-11 07:23:52.946285 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 41] Compacted 1@1 + 1@2 files to L2 => 130809742 bytes >2018-07-11 07:23:52.967872 7f354e0ae700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 42250645176 >2018-07-11 07:23:52.968017 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:23:52.967837) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[1 15 67 446 3641 796 0] max score 5.21, MB/sec: 13.4 rd, 13.4 wr, level 2, files in(1, 1) out(3) MB in(76.6, 48.1) out(124.7), read-write-amplify(3.3) write-amplify(1.6) OK, records in: 8070385, records dropped: 0 > >2018-07-11 07:23:52.968027 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:23:52.967863) EVENT_LOG_v1 {"time_micros": 1531293832967852, "job": 41, "event": "compaction_finished", "compaction_time_micros": 9793775, "output_level": 2, "num_output_files": 3, "total_output_size": 130809742, "num_input_records": 8070385, "num_output_records": 8070385, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 15, 67, 446, 3641, 796, 0]} >2018-07-11 07:23:52.969182 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 42] Compacting 1@1 + 10@2 files to L2, score 5.21 >2018-07-11 07:23:52.969201 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 40 Base level 1, inputs: [36440(76MB)], [36466(43MB) 36467(5188KB) 36407(1064B) 36408(75MB) 36409(66MB) 36410(20MB) 36411(54MB) 36412(14MB) 36425(31MB) 36426(46MB)] > >2018-07-11 07:23:52.969224 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293832969206, "job": 42, "event": "compaction_started", "files_L1": [36440], "files_L2": [36466, 36467, 36407, 36408, 36409, 36410, 36411, 36412, 36425, 36426], "score": 5.20994, "input_data_size": 456331609} >2018-07-11 07:23:56.985137 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 42] Generated table #36468: 3156061 keys, 51155268 bytes >2018-07-11 07:23:56.985172 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293836985156, "cf_name": "default", "job": 42, "event": "table_file_creation", "file_number": 36468, "file_size": 51155268, "table_properties": {"data_size": 42726345, "index_size": 438601, "filter_size": 7989333, "raw_key_size": 91175104, "raw_average_key_size": 28, "raw_value_size": 47, "raw_average_value_size": 0, "num_data_blocks": 10486, "num_entries": 3156061, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "3156057", "kMergeOperands": "0"}} >2018-07-11 07:23:57.208083 7f353f090700 1 bluefs _allocate failed to allocate 0x1d000000 on bdev 1, free 0x4ffe000; fallback to bdev 2 >2018-07-11 07:24:05.158721 7f353f090700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293845158697, "cf_name": "default", "job": 40, "event": "table_file_creation", "file_number": 36464, "file_size": 486288088, "table_properties": {"data_size": 406136979, "index_size": 4207309, "filter_size": 75942805, "raw_key_size": 866668572, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 99669, "num_entries": 30000030, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "30000030", "kMergeOperands": "0"}} >2018-07-11 07:24:05.158741 7f353f090700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 40] Level-0 flush table #36464: 486288088 bytes OK >2018-07-11 07:24:05.176054 7f353f090700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 42250645176 >2018-07-11 07:24:05.176170 7f353f090700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:242] adding log 36437 to recycle list > >2018-07-11 07:24:05.176182 7f353f090700 4 rocksdb: (Original Log Time 2018/07/11-07:24:05.158929) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #36464 started >2018-07-11 07:24:05.176186 7f353f090700 4 rocksdb: (Original Log Time 2018/07/11-07:24:05.176028) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #36464: memtable #1 done >2018-07-11 07:24:05.176188 7f353f090700 4 rocksdb: (Original Log Time 2018/07/11-07:24:05.176046) EVENT_LOG_v1 {"time_micros": 1531293845176038, "job": 40, "event": "flush_finished", "lsm_state": [2, 15, 67, 446, 3641, 796, 0], "immutable_memtables": 0} >2018-07-11 07:24:05.176190 7f353f090700 4 rocksdb: (Original Log Time 2018/07/11-07:24:05.176083) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[2 15 67 446 3641 796 0] max score 7.56 > >2018-07-11 07:24:05.176994 7f353f090700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 40] Try to delete WAL files size 686668416, prev total WAL file size 755335288, number of live WAL files 2. > >2018-07-11 07:24:05.177040 7f353f090700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293845177034, "job": 40, "event": "table_file_deletion", "file_number": 36442} >2018-07-11 07:24:05.177053 7f353f090700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293845177051, "job": 40, "event": "table_file_deletion", "file_number": 36439} >2018-07-11 07:24:05.177066 7f353f090700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293845177064, "job": 40, "event": "table_file_deletion", "file_number": 36422} >2018-07-11 07:24:05.177077 7f353f090700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293845177075, "job": 40, "event": "table_file_deletion", "file_number": 36421} >2018-07-11 07:24:05.177088 7f353f090700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293845177087, "job": 40, "event": "table_file_deletion", "file_number": 36420} >2018-07-11 07:24:05.177103 7f353f090700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293845177101, "job": 40, "event": "table_file_deletion", "file_number": 36406} >2018-07-11 07:24:05.177116 7f353f090700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293845177114, "job": 40, "event": "table_file_deletion", "file_number": 36383} >2018-07-11 07:24:05.228827 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 42] Generated table #36469: 3000000 keys, 48625685 bytes >2018-07-11 07:24:05.228866 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293845228841, "cf_name": "default", "job": 42, "event": "table_file_creation", "file_number": 36469, "file_size": 48625685, "table_properties": {"data_size": 40613546, "index_size": 416876, "filter_size": 7594274, "raw_key_size": 86666670, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 9967, "num_entries": 3000000, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "3000000", "kMergeOperands": "0"}} >2018-07-11 07:24:10.104284 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 42] Generated table #36470: 3614529 keys, 77055379 bytes >2018-07-11 07:24:10.104332 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293850104305, "cf_name": "default", "job": 42, "event": "table_file_creation", "file_number": 36470, "file_size": 77055379, "table_properties": {"data_size": 67110650, "index_size": 754483, "filter_size": 9189253, "raw_key_size": 104841254, "raw_average_key_size": 29, "raw_value_size": 18022859, "raw_average_value_size": 4, "num_data_blocks": 16577, "num_entries": 3614529, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "3000000", "kMergeOperands": "0"}} >2018-07-11 07:24:13.235458 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 42] Generated table #36471: 1364524 keys, 71602235 bytes >2018-07-11 07:24:13.235491 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293853235476, "cf_name": "default", "job": 42, "event": "table_file_creation", "file_number": 36471, "file_size": 71602235, "table_properties": {"data_size": 67109249, "index_size": 930715, "filter_size": 3561280, "raw_key_size": 41015880, "raw_average_key_size": 30, "raw_value_size": 48070597, "raw_average_value_size": 35, "num_data_blocks": 16882, "num_entries": 1364524, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "638130", "kMergeOperands": "0"}} >2018-07-11 07:24:16.027413 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 42] Generated table #36472: 2361870 keys, 38282630 bytes >2018-07-11 07:24:16.027443 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293856027428, "cf_name": "default", "job": 42, "event": "table_file_creation", "file_number": 36472, "file_size": 38282630, "table_properties": {"data_size": 31974666, "index_size": 328078, "filter_size": 5978897, "raw_key_size": 68231812, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 7847, "num_entries": 2361870, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "2361870", "kMergeOperands": "0"}} >2018-07-11 07:24:19.928771 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 42] Generated table #36473: 3315389 keys, 57537347 bytes >2018-07-11 07:24:19.928824 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293859928804, "cf_name": "default", "job": 42, "event": "table_file_creation", "file_number": 36473, "file_size": 57537347, "table_properties": {"data_size": 48636551, "index_size": 499413, "filter_size": 8400391, "raw_key_size": 95777904, "raw_average_key_size": 28, "raw_value_size": 3749621, "raw_average_value_size": 1, "num_data_blocks": 11938, "num_entries": 3315389, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "3000000", "kMergeOperands": "0"}} >2018-07-11 07:24:24.562993 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 42] Generated table #36474: 4000000 keys, 64834129 bytes >2018-07-11 07:24:24.563028 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293864563012, "cf_name": "default", "job": 42, "event": "table_file_creation", "file_number": 36474, "file_size": 64834129, "table_properties": {"data_size": 54151363, "index_size": 556070, "filter_size": 10125707, "raw_key_size": 115555560, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 13290, "num_entries": 4000000, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4000000", "kMergeOperands": "0"}} >2018-07-11 07:24:27.593058 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 42] Generated table #36475: 2914331 keys, 47237177 bytes >2018-07-11 07:24:27.593093 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293867593078, "cf_name": "default", "job": 42, "event": "table_file_creation", "file_number": 36475, "file_size": 47237177, "table_properties": {"data_size": 39453768, "index_size": 405004, "filter_size": 7377416, "raw_key_size": 84191780, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 9683, "num_entries": 2914331, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "2914331", "kMergeOperands": "0"}} >2018-07-11 07:24:27.593132 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 42] Compacted 1@1 + 10@2 files to L2 => 456329850 bytes >2018-07-11 07:24:27.608830 7f354e0ae700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 42119990478 >2018-07-11 07:24:27.609087 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:24:27.608795) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[2 14 65 446 3641 796 0] max score 7.56, MB/sec: 13.2 rd, 13.2 wr, level 2, files in(1, 10) out(8) MB in(76.6, 358.6) out(435.2), read-write-amplify(11.4) write-amplify(5.7) OK, records in: 23726704, records dropped: 0 > >2018-07-11 07:24:27.609097 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:24:27.608822) EVENT_LOG_v1 {"time_micros": 1531293867608810, "job": 42, "event": "compaction_finished", "compaction_time_micros": 34623887, "output_level": 2, "num_output_files": 8, "total_output_size": 456329850, "num_input_records": 23726704, "num_output_records": 23726704, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [2, 14, 65, 446, 3641, 796, 0]} >2018-07-11 07:24:27.610350 7f354e0ae700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS ------- >2018-07-11 07:24:27.610357 7f354e0ae700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] >** DB Stats ** >Uptime(secs): 5421.8 total, 605.2 interval >Cumulative writes: 894 writes, 196M keys, 894 commit groups, 1.0 writes per commit group, ingest: 4.28 GB, 0.81 MB/s >Cumulative WAL: 894 writes, 251 syncs, 3.55 writes per sync, written: 4.28 GB, 0.81 MB/s >Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent >Interval writes: 16 writes, 159M keys, 16 commit groups, 0.9 writes per commit group, ingest: 3470.75 MB, 5.73 MB/s >Interval WAL: 16 writes, 8 syncs, 1.78 writes per sync, written: 3.39 MB, 5.73 MB/s >Interval stall: 00:00:0.000 H:M:S, 0.0 percent > >** Compaction Stats [default] ** >Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop >---------------------------------------------------------------------------------------------------------------------------------------------------------- > L0 2/0 973.90 MB 7.6 0.0 0.0 0.0 3.1 3.1 0.0 1.0 0.0 20.6 153 8 19.114 0 0 > L1 14/0 841.24 MB 4.6 4.2 2.1 2.1 4.2 2.1 0.0 2.0 16.1 16.1 269 4 67.289 213M 148 > L2 65/0 3.97 GB 2.1 4.2 1.5 2.7 4.2 1.5 0.0 2.8 12.9 12.9 335 19 17.612 244M 55 > L3 446/0 24.93 GB 1.0 0.1 0.1 0.1 0.1 0.1 0.1 2.0 18.6 18.6 8 1 7.636 5276K 0 > L4 3641/0 250.00 GB 1.0 0.8 0.1 0.6 0.8 0.1 0.1 5.5 21.8 21.8 36 2 17.944 29M 0 > L5 796/0 55.24 GB 0.0 0.0 0.0 0.0 0.0 0.0 0.3 0.0 0.0 0.0 0 0 0.000 0 0 > Sum 4964/0 335.91 GB 0.0 9.3 3.8 5.5 12.4 6.9 0.5 4.0 12.0 15.9 800 34 23.536 492M 203 > Int 0/0 0.00 KB 0.0 8.4 3.6 4.8 10.8 6.0 0.0 4.6 11.9 15.2 725 28 25.910 458M 203 >Uptime(secs): 5421.8 total, 5421.8 interval >Flush(GB): cumulative 3.083, interval 2.355 >AddFile(GB): cumulative 0.000, interval 0.000 >AddFile(Total Files): cumulative 0, interval 0 >AddFile(L0 Files): cumulative 0, interval 0 >AddFile(Keys): cumulative 0, interval 0 >Cumulative compaction: 12.43 GB write, 2.35 MB/s write, 9.35 GB read, 1.77 MB/s read, 800.2 seconds >Interval compaction: 10.80 GB write, 2.04 MB/s write, 8.45 GB read, 1.60 MB/s read, 725.5 seconds >Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count > >** File Read Latency Histogram By Level [default] ** > >** Compaction Stats [default] ** >Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop >---------------------------------------------------------------------------------------------------------------------------------------------------------- > L0 2/0 973.90 MB 7.6 0.0 0.0 0.0 3.1 3.1 0.0 1.0 0.0 20.6 153 8 19.114 0 0 > L1 14/0 841.24 MB 4.6 4.2 2.1 2.1 4.2 2.1 0.0 2.0 16.1 16.1 269 4 67.289 213M 148 > L2 65/0 3.97 GB 2.1 4.2 1.5 2.7 4.2 1.5 0.0 2.8 12.9 12.9 335 19 17.612 244M 55 > L3 446/0 24.93 GB 1.0 0.1 0.1 0.1 0.1 0.1 0.1 2.0 18.6 18.6 8 1 7.636 5276K 0 > L4 3641/0 250.00 GB 1.0 0.8 0.1 0.6 0.8 0.1 0.1 5.5 21.8 21.8 36 2 17.944 29M 0 > L5 796/0 55.24 GB 0.0 0.0 0.0 0.0 0.0 0.0 0.3 0.0 0.0 0.0 0 0 0.000 0 0 > Sum 4964/0 335.91 GB 0.0 9.3 3.8 5.5 12.4 6.9 0.5 4.0 12.0 15.9 800 34 23.536 492M 203 > Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 >Uptime(secs): 5421.8 total, 5421.8 interval >Flush(GB): cumulative 3.083, interval 0.000 >AddFile(GB): cumulative 0.000, interval 0.000 >AddFile(Total Files): cumulative 0, interval 0 >AddFile(L0 Files): cumulative 0, interval 0 >AddFile(Keys): cumulative 0, interval 0 >Cumulative compaction: 12.43 GB write, 2.35 MB/s write, 9.35 GB read, 1.77 MB/s read, 800.2 seconds >Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds >Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count > >** File Read Latency Histogram By Level [default] ** > >2018-07-11 07:24:27.610477 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 43] Compacting 2@0 + 13@1 files to L1, score 7.56 >2018-07-11 07:24:27.610499 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 42 Base level 0, inputs: [36464(463MB) 36438(510MB)], [36436(76MB) 36441(76MB) 36443(76MB) 36445(65MB) 36446(75MB) 36447(71MB) 36448(71MB) 36449(71MB) 36450(71MB) 36451(70MB) 36452(36MB) 36453(64MB) 36454(13MB)] > >2018-07-11 07:24:27.610535 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293867610505, "job": 43, "event": "compaction_started", "files_L0": [36464, 36438], "files_L1": [36436, 36441, 36443, 36445, 36446, 36447, 36448, 36449, 36450, 36451, 36452, 36453, 36454], "score": 7.55941, "input_data_size": 1902343289} >2018-07-11 07:24:29.511751 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293869511731, "job": 0, "event": "table_file_deletion", "file_number": 36467} >2018-07-11 07:24:29.511766 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293869511764, "job": 0, "event": "table_file_deletion", "file_number": 36466} >2018-07-11 07:24:29.511779 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293869511777, "job": 0, "event": "table_file_deletion", "file_number": 36440} >2018-07-11 07:24:29.511792 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293869511789, "job": 0, "event": "table_file_deletion", "file_number": 36426} >2018-07-11 07:24:29.511803 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293869511801, "job": 0, "event": "table_file_deletion", "file_number": 36425} >2018-07-11 07:24:29.511815 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293869511813, "job": 0, "event": "table_file_deletion", "file_number": 36412} >2018-07-11 07:24:29.511834 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293869511832, "job": 0, "event": "table_file_deletion", "file_number": 36411} >2018-07-11 07:24:29.511848 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293869511846, "job": 0, "event": "table_file_deletion", "file_number": 36410} >2018-07-11 07:24:29.511861 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293869511859, "job": 0, "event": "table_file_deletion", "file_number": 36409} >2018-07-11 07:24:29.511874 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293869511872, "job": 0, "event": "table_file_deletion", "file_number": 36408} >2018-07-11 07:24:29.511887 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293869511885, "job": 0, "event": "table_file_deletion", "file_number": 36407} >2018-07-11 07:24:30.076928 7f355d88e700 0 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6809/15287 conn(0x55a3380c2000 :6801 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg: challenging authorizer >2018-07-11 07:24:30.077164 7f355d88e700 0 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6809/15287 conn(0x55a3380c2000 :6801 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg accept connect_seq 1 vs existing csq=1 existing_state=STATE_STANDBY >2018-07-11 07:24:30.077348 7f355d88e700 0 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6809/15287 conn(0x55a3380c2000 :6801 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg accept connect_seq 2 vs existing csq=1 existing_state=STATE_STANDBY >2018-07-11 07:24:30.077612 7f355d88e700 0 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6809/15287 conn(0x55a314e44800 :-1 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=11 cs=1 l=0).handle_connect_msg: challenging authorizer >2018-07-11 07:24:33.095149 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 43] Generated table #36476: 4888760 keys, 80173882 bytes >2018-07-11 07:24:33.095202 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293873095185, "cf_name": "default", "job": 43, "event": "table_file_creation", "file_number": 36476, "file_size": 80173882, "table_properties": {"data_size": 67110184, "index_size": 685246, "filter_size": 12377461, "raw_key_size": 141267959, "raw_average_key_size": 28, "raw_value_size": 916542, "raw_average_value_size": 0, "num_data_blocks": 16336, "num_entries": 4888760, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4886796", "kMergeOperands": "0"}} >2018-07-11 07:24:37.432015 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4d00000 on bdev 1, free 0x2fe000; fallback to bdev 2 >2018-07-11 07:24:38.668356 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 43] Generated table #36477: 4957167 keys, 80348323 bytes >2018-07-11 07:24:38.668408 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293878668379, "cf_name": "default", "job": 43, "event": "table_file_creation", "file_number": 36477, "file_size": 80348323, "table_properties": {"data_size": 67109491, "index_size": 689135, "filter_size": 12548707, "raw_key_size": 143207044, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957167, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957167", "kMergeOperands": "0"}} >2018-07-11 07:24:42.710473 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4d00000 on bdev 1, free 0x2fe000; fallback to bdev 2 >2018-07-11 07:24:43.957200 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 43] Generated table #36478: 4957165 keys, 80348260 bytes >2018-07-11 07:24:43.957251 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293883957219, "cf_name": "default", "job": 43, "event": "table_file_creation", "file_number": 36478, "file_size": 80348260, "table_properties": {"data_size": 67109376, "index_size": 689193, "filter_size": 12548701, "raw_key_size": 143206995, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957165, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957165", "kMergeOperands": "0"}} >2018-07-11 07:24:48.241874 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4d00000 on bdev 1, free 0x2fe000; fallback to bdev 2 >2018-07-11 07:24:49.573091 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 43] Generated table #36479: 4957165 keys, 80348283 bytes >2018-07-11 07:24:49.573140 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293889573124, "cf_name": "default", "job": 43, "event": "table_file_creation", "file_number": 36479, "file_size": 80348283, "table_properties": {"data_size": 67109395, "index_size": 689197, "filter_size": 12548701, "raw_key_size": 143206995, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957165, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957165", "kMergeOperands": "0"}} >2018-07-11 07:24:54.073898 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4d00000 on bdev 1, free 0x2fe000; fallback to bdev 2 >2018-07-11 07:24:55.286179 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 43] Generated table #36480: 4957165 keys, 80348264 bytes >2018-07-11 07:24:55.286229 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293895286199, "cf_name": "default", "job": 43, "event": "table_file_creation", "file_number": 36480, "file_size": 80348264, "table_properties": {"data_size": 67109367, "index_size": 689206, "filter_size": 12548701, "raw_key_size": 143206995, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957165, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957165", "kMergeOperands": "0"}} >2018-07-11 07:24:59.492476 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4d00000 on bdev 1, free 0x2fe000; fallback to bdev 2 >2018-07-11 07:25:00.655798 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 43] Generated table #36481: 4957166 keys, 80348267 bytes >2018-07-11 07:25:00.655855 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293900655837, "cf_name": "default", "job": 43, "event": "table_file_creation", "file_number": 36481, "file_size": 80348267, "table_properties": {"data_size": 67109322, "index_size": 689251, "filter_size": 12548704, "raw_key_size": 143207025, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957166, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957166", "kMergeOperands": "0"}} >2018-07-11 07:25:04.862014 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4d00000 on bdev 1, free 0x2fe000; fallback to bdev 2 >2018-07-11 07:25:06.023486 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 43] Generated table #36482: 4957166 keys, 80348294 bytes >2018-07-11 07:25:06.023535 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293906023506, "cf_name": "default", "job": 43, "event": "table_file_creation", "file_number": 36482, "file_size": 80348294, "table_properties": {"data_size": 67109373, "index_size": 689227, "filter_size": 12548704, "raw_key_size": 143207028, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957166, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957166", "kMergeOperands": "0"}} >2018-07-11 07:25:10.022355 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4d00000 on bdev 1, free 0x2fe000; fallback to bdev 2 >2018-07-11 07:25:11.242211 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 43] Generated table #36483: 4957165 keys, 80348256 bytes >2018-07-11 07:25:11.242269 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293911242252, "cf_name": "default", "job": 43, "event": "table_file_creation", "file_number": 36483, "file_size": 80348256, "table_properties": {"data_size": 67109348, "index_size": 689217, "filter_size": 12548701, "raw_key_size": 143206991, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957165, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957165", "kMergeOperands": "0"}} >2018-07-11 07:25:15.486022 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4d00000 on bdev 1, free 0x2fe000; fallback to bdev 2 >2018-07-11 07:25:16.737292 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 43] Generated table #36484: 4957165 keys, 80348244 bytes >2018-07-11 07:25:16.737344 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293916737316, "cf_name": "default", "job": 43, "event": "table_file_creation", "file_number": 36484, "file_size": 80348244, "table_properties": {"data_size": 67109332, "index_size": 689221, "filter_size": 12548701, "raw_key_size": 143206986, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957165, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957165", "kMergeOperands": "0"}} >2018-07-11 07:25:21.223740 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4d00000 on bdev 1, free 0x2fe000; fallback to bdev 2 >2018-07-11 07:25:22.529646 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 43] Generated table #36485: 4957166 keys, 80348292 bytes >2018-07-11 07:25:22.529704 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293922529676, "cf_name": "default", "job": 43, "event": "table_file_creation", "file_number": 36485, "file_size": 80348292, "table_properties": {"data_size": 67109371, "index_size": 689227, "filter_size": 12548704, "raw_key_size": 143207028, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957166, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957166", "kMergeOperands": "0"}} >2018-07-11 07:25:27.002682 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4d00000 on bdev 1, free 0x2fe000; fallback to bdev 2 >2018-07-11 07:25:28.228335 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 43] Generated table #36486: 4957165 keys, 80348254 bytes >2018-07-11 07:25:28.228384 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293928228367, "cf_name": "default", "job": 43, "event": "table_file_creation", "file_number": 36486, "file_size": 80348254, "table_properties": {"data_size": 67109346, "index_size": 689217, "filter_size": 12548701, "raw_key_size": 143206991, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957165, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957165", "kMergeOperands": "0"}} >2018-07-11 07:25:32.263344 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4d00000 on bdev 1, free 0x2fe000; fallback to bdev 2 >2018-07-11 07:25:33.423695 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 43] Generated table #36487: 4957164 keys, 80348239 bytes >2018-07-11 07:25:33.423745 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293933423729, "cf_name": "default", "job": 43, "event": "table_file_creation", "file_number": 36487, "file_size": 80348239, "table_properties": {"data_size": 67109353, "index_size": 689198, "filter_size": 12548698, "raw_key_size": 143206970, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957164, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957164", "kMergeOperands": "0"}} >2018-07-11 07:25:37.460048 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4d00000 on bdev 1, free 0x2fe000; fallback to bdev 2 >2018-07-11 07:25:38.625090 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 43] Generated table #36488: 4957168 keys, 80348303 bytes >2018-07-11 07:25:38.625139 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293938625123, "cf_name": "default", "job": 43, "event": "table_file_creation", "file_number": 36488, "file_size": 80348303, "table_properties": {"data_size": 67109464, "index_size": 689139, "filter_size": 12548710, "raw_key_size": 143207085, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957168, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957168", "kMergeOperands": "0"}} >2018-07-11 07:25:45.893581 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4d00000 on bdev 1, free 0x2fe000; fallback to bdev 2 >2018-07-11 07:25:47.065322 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 43] Generated table #36489: 4957167 keys, 80348273 bytes >2018-07-11 07:25:47.065378 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293947065361, "cf_name": "default", "job": 43, "event": "table_file_creation", "file_number": 36489, "file_size": 80348273, "table_properties": {"data_size": 67109347, "index_size": 689229, "filter_size": 12548707, "raw_key_size": 143207044, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957167, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957167", "kMergeOperands": "0"}} >2018-07-11 07:25:48.934326 7f354e0ae700 1 bluefs _allocate failed to allocate 0x2700000 on bdev 1, free 0x2fe000; fallback to bdev 2 >2018-07-11 07:25:49.549914 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 43] Generated table #36490: 2471176 keys, 40054281 bytes >2018-07-11 07:25:49.549966 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293949549949, "cf_name": "default", "job": 43, "event": "table_file_creation", "file_number": 36490, "file_size": 40054281, "table_properties": {"data_size": 33454379, "index_size": 343314, "filter_size": 6255599, "raw_key_size": 71389538, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 8210, "num_entries": 2471176, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "2471176", "kMergeOperands": "0"}} >2018-07-11 07:25:50.244000 7f354f8b1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:684] reusing log 36437 from recycle list > >2018-07-11 07:25:50.244118 7f354f8b1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_write.cc:725] [default] New memtable created with log file: #36492. Immutable memtables: 0. > >2018-07-11 07:25:50.244135 7f354f8b1700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 42119990478 >2018-07-11 07:25:50.256230 7f353f090700 4 rocksdb: (Original Log Time 2018/07/11-07:25:50.244183) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1158] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots allowed 1, compaction slots scheduled 0 >2018-07-11 07:25:50.256243 7f353f090700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:264] [default] [JOB 44] Flushing memtable with next log file: 36492 > >2018-07-11 07:25:50.256268 7f353f090700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293950256258, "job": 44, "event": "flush_started", "num_memtables": 1, "num_entries": 33000033, "num_deletes": 33000033, "memory_usage": 1408022808} >2018-07-11 07:25:50.256271 7f353f090700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:293] [default] [JOB 44] Level-0 flush table #36493: started >2018-07-11 07:25:53.557286 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4d00000 on bdev 1, free 0x2fe000; fallback to bdev 2 >2018-07-11 07:25:54.884565 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 43] Generated table #36491: 4957168 keys, 80348300 bytes >2018-07-11 07:25:54.884619 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293954884602, "cf_name": "default", "job": 43, "event": "table_file_creation", "file_number": 36491, "file_size": 80348300, "table_properties": {"data_size": 67109454, "index_size": 689146, "filter_size": 12548710, "raw_key_size": 143207087, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957168, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957168", "kMergeOperands": "0"}} >2018-07-11 07:25:58.991862 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4d00000 on bdev 1, free 0x2fe000; fallback to bdev 2 >2018-07-11 07:26:00.278856 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 43] Generated table #36494: 4957166 keys, 80348274 bytes >2018-07-11 07:26:00.278891 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293960278875, "cf_name": "default", "job": 43, "event": "table_file_creation", "file_number": 36494, "file_size": 80348274, "table_properties": {"data_size": 67109370, "index_size": 689210, "filter_size": 12548704, "raw_key_size": 143207015, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957166, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957166", "kMergeOperands": "0"}} >2018-07-11 07:26:04.845317 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4c00000 on bdev 1, free 0x2fe000; fallback to bdev 2 >2018-07-11 07:26:06.015119 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 43] Generated table #36495: 4575822 keys, 79396920 bytes >2018-07-11 07:26:06.015155 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293966015139, "cf_name": "default", "job": 43, "event": "table_file_creation", "file_number": 36495, "file_size": 79396920, "table_properties": {"data_size": 67112573, "index_size": 689355, "filter_size": 11593999, "raw_key_size": 132190386, "raw_average_key_size": 28, "raw_value_size": 5160709, "raw_average_value_size": 1, "num_data_blocks": 16474, "num_entries": 4575822, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4142122", "kMergeOperands": "0"}} >2018-07-11 07:26:07.524685 7f353f090700 1 bluefs _allocate failed to allocate 0x1ff00000 on bdev 1, free 0x2fe000; fallback to bdev 2 >2018-07-11 07:26:14.836760 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4800000 on bdev 1, free 0x2fe000; fallback to bdev 2 >2018-07-11 07:26:15.826866 7f353f090700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293975826844, "cf_name": "default", "job": 44, "event": "table_file_creation", "file_number": 36493, "file_size": 534923425, "table_properties": {"data_size": 446750687, "index_size": 4634658, "filter_size": 83537085, "raw_key_size": 953335421, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 109636, "num_entries": 33000033, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "33000033", "kMergeOperands": "0"}} >2018-07-11 07:26:15.826885 7f353f090700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/flush_job.cc:319] [default] [JOB 44] Level-0 flush table #36493: 534923425 bytes OK >2018-07-11 07:26:15.847989 7f353f090700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 42119990478 >2018-07-11 07:26:15.848086 7f353f090700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:242] adding log 36463 to recycle list > >2018-07-11 07:26:15.848095 7f353f090700 4 rocksdb: (Original Log Time 2018/07/11-07:26:15.826898) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:360] [default] Level-0 commit table #36493 started >2018-07-11 07:26:15.848099 7f353f090700 4 rocksdb: (Original Log Time 2018/07/11-07:26:15.847964) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/memtable_list.cc:383] [default] Level-0 commit table #36493: memtable #1 done >2018-07-11 07:26:15.848101 7f353f090700 4 rocksdb: (Original Log Time 2018/07/11-07:26:15.847980) EVENT_LOG_v1 {"time_micros": 1531293975847973, "job": 44, "event": "flush_finished", "lsm_state": [3, 14, 65, 446, 3641, 796, 0], "immutable_memtables": 0} >2018-07-11 07:26:15.848103 7f353f090700 4 rocksdb: (Original Log Time 2018/07/11-07:26:15.848004) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] Level summary: base level 1 max bytes base 268435456 files[3 14 65 446 3641 796 0] max score 3.96 > >2018-07-11 07:26:15.848918 7f353f090700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:388] [JOB 44] Try to delete WAL files size 755335271, prev total WAL file size 755335283, number of live WAL files 2. > >2018-07-11 07:26:16.790439 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 43] Generated table #36496: 2638241 keys, 74545841 bytes >2018-07-11 07:26:16.790485 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293976790456, "cf_name": "default", "job": 43, "event": "table_file_creation", "file_number": 36496, "file_size": 74545841, "table_properties": {"data_size": 67112134, "index_size": 689534, "filter_size": 6743183, "raw_key_size": 76215847, "raw_average_key_size": 28, "raw_value_size": 31365750, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:26:19.321994 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4800000 on bdev 1, free 0x2fe000; fallback to bdev 2 >2018-07-11 07:26:20.407200 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 43] Generated table #36497: 2638241 keys, 74546007 bytes >2018-07-11 07:26:20.407250 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293980407231, "cf_name": "default", "job": 43, "event": "table_file_creation", "file_number": 36497, "file_size": 74546007, "table_properties": {"data_size": 67112158, "index_size": 689676, "filter_size": 6743183, "raw_key_size": 76215852, "raw_average_key_size": 28, "raw_value_size": 31365755, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:26:23.034491 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4800000 on bdev 1, free 0x2fe000; fallback to bdev 2 >2018-07-11 07:26:24.115027 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 43] Generated table #36498: 2638241 keys, 74546368 bytes >2018-07-11 07:26:24.115070 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293984115043, "cf_name": "default", "job": 43, "event": "table_file_creation", "file_number": 36498, "file_size": 74546368, "table_properties": {"data_size": 67112113, "index_size": 690082, "filter_size": 6743183, "raw_key_size": 76215859, "raw_average_key_size": 28, "raw_value_size": 31365762, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:26:26.636556 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4800000 on bdev 1, free 0x2fe000; fallback to bdev 2 >2018-07-11 07:26:27.630641 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 43] Generated table #36499: 2638241 keys, 74546878 bytes >2018-07-11 07:26:27.630675 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293987630659, "cf_name": "default", "job": 43, "event": "table_file_creation", "file_number": 36499, "file_size": 74546878, "table_properties": {"data_size": 67112164, "index_size": 690541, "filter_size": 6743183, "raw_key_size": 76215848, "raw_average_key_size": 28, "raw_value_size": 31365751, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:26:30.151964 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4800000 on bdev 1, free 0x2fe000; fallback to bdev 2 >2018-07-11 07:26:31.137448 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 43] Generated table #36500: 2632373 keys, 74530374 bytes >2018-07-11 07:26:31.137493 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293991137478, "cf_name": "default", "job": 43, "event": "table_file_creation", "file_number": 36500, "file_size": 74530374, "table_properties": {"data_size": 67111400, "index_size": 689456, "filter_size": 6728528, "raw_key_size": 76066365, "raw_average_key_size": 28, "raw_value_size": 31421172, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2632373, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "8", "kMergeOperands": "0"}} >2018-07-11 07:26:31.277728 7f354e0ae700 1 bluefs _allocate failed to allocate 0xb00000 on bdev 1, free 0x2fe000; fallback to bdev 2 >2018-07-11 07:26:31.487972 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 43] Generated table #36501: 15343 keys, 11131632 bytes >2018-07-11 07:26:31.488003 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293991487987, "cf_name": "default", "job": 43, "event": "table_file_creation", "file_number": 36501, "file_size": 11131632, "table_properties": {"data_size": 10988732, "index_size": 78485, "filter_size": 63431, "raw_key_size": 1530364, "raw_average_key_size": 99, "raw_value_size": 9534361, "raw_average_value_size": 621, "num_data_blocks": 2272, "num_entries": 15343, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:26:32.324464 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4200000 on bdev 1, free 0x2fe000; fallback to bdev 2 >2018-07-11 07:26:33.249679 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 43] Generated table #36502: 163471 keys, 68250761 bytes >2018-07-11 07:26:33.249708 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293993249694, "cf_name": "default", "job": 43, "event": "table_file_creation", "file_number": 36502, "file_size": 68250761, "table_properties": {"data_size": 67113009, "index_size": 574297, "filter_size": 562467, "raw_key_size": 14904496, "raw_average_key_size": 91, "raw_value_size": 53560126, "raw_average_value_size": 327, "num_data_blocks": 15455, "num_entries": 163471, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "81086", "kMergeOperands": "0"}} >2018-07-11 07:26:34.522701 7f354e0ae700 1 bluefs _allocate failed to allocate 0x2c00000 on bdev 1, free 0x2fe000; fallback to bdev 2 >2018-07-11 07:26:35.267801 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 43] Generated table #36503: 97332 keys, 45324813 bytes >2018-07-11 07:26:35.267843 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293995267816, "cf_name": "default", "job": 43, "event": "table_file_creation", "file_number": 36503, "file_size": 45324813, "table_properties": {"data_size": 44596353, "index_size": 381215, "filter_size": 346257, "raw_key_size": 5188014, "raw_average_key_size": 53, "raw_value_size": 39787210, "raw_average_value_size": 408, "num_data_blocks": 10719, "num_entries": 97332, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "29768"}} >2018-07-11 07:26:35.267909 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 43] Compacted 2@0 + 13@1 files to L1 => 1902271883 bytes >2018-07-11 07:26:35.289514 7f354e0ae700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 58149663922 >2018-07-11 07:26:35.289978 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:26:35.289479) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[1 27 65 446 3641 796 0] max score 12.18, MB/sec: 14.9 rd, 14.9 wr, level 1, files in(2, 13) out(26) MB in(973.9, 840.3) out(1814.1), read-write-amplify(3.7) write-amplify(1.9) OK, records in: 99754730, records dropped: 1 > >2018-07-11 07:26:35.289988 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:26:35.289505) EVENT_LOG_v1 {"time_micros": 1531293995289492, "job": 43, "event": "compaction_finished", "compaction_time_micros": 127657341, "output_level": 1, "num_output_files": 26, "total_output_size": 1902271883, "num_input_records": 99754730, "num_output_records": 99754729, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 27, 65, 446, 3641, 796, 0]} >2018-07-11 07:26:35.290885 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 45] Compacting 1@1 + 2@2 files to L2, score 12.18 >2018-07-11 07:26:35.290898 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 44 Base level 1, inputs: [36477(76MB)], [36465(76MB) 36468(48MB)] > >2018-07-11 07:26:35.290917 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531293995290903, "job": 45, "event": "compaction_started", "files_L1": [36477], "files_L2": [36465, 36468], "score": 12.1817, "input_data_size": 211851868} >2018-07-11 07:26:42.778659 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 45] Generated table #36504: 4957160 keys, 80348263 bytes >2018-07-11 07:26:42.778712 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294002778684, "cf_name": "default", "job": 45, "event": "table_file_creation", "file_number": 36504, "file_size": 80348263, "table_properties": {"data_size": 67109358, "index_size": 689226, "filter_size": 12548689, "raw_key_size": 143206851, "raw_average_key_size": 28, "raw_value_size": 60, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957160, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957155", "kMergeOperands": "0"}} >2018-07-11 07:26:49.984027 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 45] Generated table #36505: 4957165 keys, 80348239 bytes >2018-07-11 07:26:49.984083 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294009984054, "cf_name": "default", "job": 45, "event": "table_file_creation", "file_number": 36505, "file_size": 80348239, "table_properties": {"data_size": 67109319, "index_size": 689229, "filter_size": 12548701, "raw_key_size": 143206991, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957165, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957165", "kMergeOperands": "0"}} >2018-07-11 07:26:53.411704 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 45] Generated table #36506: 2198894 keys, 35641099 bytes >2018-07-11 07:26:53.411747 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294013411720, "cf_name": "default", "job": 45, "event": "table_file_creation", "file_number": 36506, "file_size": 35641099, "table_properties": {"data_size": 29768332, "index_size": 305440, "filter_size": 5566338, "raw_key_size": 63523615, "raw_average_key_size": 28, "raw_value_size": 35, "raw_average_value_size": 0, "num_data_blocks": 7306, "num_entries": 2198894, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "2198891", "kMergeOperands": "0"}} >2018-07-11 07:26:55.002278 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 45] Generated table #36507: 957168 keys, 15514662 bytes >2018-07-11 07:26:55.002309 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294015002294, "cf_name": "default", "job": 45, "event": "table_file_creation", "file_number": 36507, "file_size": 15514662, "table_properties": {"data_size": 12958048, "index_size": 132624, "filter_size": 2423003, "raw_key_size": 27651513, "raw_average_key_size": 28, "raw_value_size": 12, "raw_average_value_size": 0, "num_data_blocks": 3180, "num_entries": 957168, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "957167", "kMergeOperands": "0"}} >2018-07-11 07:26:55.002335 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 45] Compacted 1@1 + 2@2 files to L2 => 211852263 bytes >2018-07-11 07:26:55.014104 7f354e0ae700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 58111356258 >2018-07-11 07:26:55.014367 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:26:55.014072) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[1 26 67 446 3641 796 0] max score 11.59, MB/sec: 10.7 rd, 10.7 wr, level 2, files in(1, 2) out(4) MB in(76.6, 125.4) out(202.0), read-write-amplify(5.3) write-amplify(2.6) OK, records in: 13070387, records dropped: 0 > >2018-07-11 07:26:55.014377 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:26:55.014095) EVENT_LOG_v1 {"time_micros": 1531294015014084, "job": 45, "event": "compaction_finished", "compaction_time_micros": 19711403, "output_level": 2, "num_output_files": 4, "total_output_size": 211852263, "num_input_records": 13070387, "num_output_records": 13070387, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 26, 67, 446, 3641, 796, 0]} >2018-07-11 07:26:55.015187 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294015015183, "job": 45, "event": "table_file_deletion", "file_number": 36477} >2018-07-11 07:26:55.015400 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 46] Compacting 1@1 + 1@2 files to L2, score 11.59 >2018-07-11 07:26:55.015413 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 45 Base level 1, inputs: [36488(76MB)], [36455(76MB)] > >2018-07-11 07:26:55.015426 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294015015416, "job": 46, "event": "compaction_started", "files_L1": [36488], "files_L2": [36455], "score": 11.5869, "input_data_size": 160696594} >2018-07-11 07:27:02.673270 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 46] Generated table #36508: 4957165 keys, 80348315 bytes >2018-07-11 07:27:02.673326 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294022673298, "cf_name": "default", "job": 46, "event": "table_file_creation", "file_number": 36508, "file_size": 80348315, "table_properties": {"data_size": 67109414, "index_size": 689210, "filter_size": 12548701, "raw_key_size": 143207007, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957165, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957165", "kMergeOperands": "0"}} >2018-07-11 07:27:09.707391 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 46] Generated table #36509: 4957165 keys, 80348249 bytes >2018-07-11 07:27:09.707423 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294029707408, "cf_name": "default", "job": 46, "event": "table_file_creation", "file_number": 36509, "file_size": 80348249, "table_properties": {"data_size": 67109338, "index_size": 689220, "filter_size": 12548701, "raw_key_size": 143206998, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957165, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957165", "kMergeOperands": "0"}} >2018-07-11 07:27:09.787888 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 46] Generated table #36510: 2 keys, 1064 bytes >2018-07-11 07:27:09.787927 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294029787901, "cf_name": "default", "job": 46, "event": "table_file_creation", "file_number": 36510, "file_size": 1064, "table_properties": {"data_size": 57, "index_size": 27, "filter_size": 18, "raw_key_size": 58, "raw_average_key_size": 29, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 1, "num_entries": 2, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "2", "kMergeOperands": "0"}} >2018-07-11 07:27:09.787949 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 46] Compacted 1@1 + 1@2 files to L2 => 160697628 bytes >2018-07-11 07:27:09.797021 7f354e0ae700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 58069524415 >2018-07-11 07:27:09.797184 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:27:09.796991) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[1 25 69 446 3641 796 0] max score 10.99, MB/sec: 10.9 rd, 10.9 wr, level 2, files in(1, 1) out(3) MB in(76.6, 76.6) out(153.3), read-write-amplify(4.0) write-amplify(2.0) OK, records in: 9914332, records dropped: 0 > >2018-07-11 07:27:09.797192 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:27:09.797013) EVENT_LOG_v1 {"time_micros": 1531294029797003, "job": 46, "event": "compaction_finished", "compaction_time_micros": 14772510, "output_level": 2, "num_output_files": 3, "total_output_size": 160697628, "num_input_records": 9914332, "num_output_records": 9914332, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 25, 69, 446, 3641, 796, 0]} >2018-07-11 07:27:09.798026 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294029798022, "job": 46, "event": "table_file_deletion", "file_number": 36488} >2018-07-11 07:27:09.798144 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_picker.cc:461] [default] Expanding@1 1+5(159662988+548264179 bytes) to 2+5 (319325918+548264179bytes) > >2018-07-11 07:27:09.798233 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 47] Compacting 2@1 + 5@2 files to L2, score 10.99 >2018-07-11 07:27:09.798251 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 46 Base level 1, inputs: [36491(76MB) 36494(76MB)], [36457(76MB) 36435(24MB) 36423(73MB) 36424(43MB) 36419(76MB)] > >2018-07-11 07:27:09.798265 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294029798255, "job": 47, "event": "compaction_started", "files_L1": [36491, 36494], "files_L2": [36457, 36435, 36423, 36424, 36419], "score": 10.9921, "input_data_size": 469386465} >2018-07-11 07:27:11.624285 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294031624278, "job": 0, "event": "table_file_deletion", "file_number": 36468} >2018-07-11 07:27:11.624314 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294031624310, "job": 0, "event": "table_file_deletion", "file_number": 36465} >2018-07-11 07:27:11.624333 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294031624329, "job": 0, "event": "table_file_deletion", "file_number": 36464} >2018-07-11 07:27:11.624360 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294031624357, "job": 0, "event": "table_file_deletion", "file_number": 36455} >2018-07-11 07:27:11.624384 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294031624380, "job": 0, "event": "table_file_deletion", "file_number": 36454} >2018-07-11 07:27:11.624405 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294031624401, "job": 0, "event": "table_file_deletion", "file_number": 36453} >2018-07-11 07:27:11.624423 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294031624420, "job": 0, "event": "table_file_deletion", "file_number": 36452} >2018-07-11 07:27:11.624442 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294031624437, "job": 0, "event": "table_file_deletion", "file_number": 36451} >2018-07-11 07:27:11.624469 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294031624466, "job": 0, "event": "table_file_deletion", "file_number": 36450} >2018-07-11 07:27:11.624490 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294031624486, "job": 0, "event": "table_file_deletion", "file_number": 36449} >2018-07-11 07:27:11.624511 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294031624507, "job": 0, "event": "table_file_deletion", "file_number": 36448} >2018-07-11 07:27:11.624532 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294031624528, "job": 0, "event": "table_file_deletion", "file_number": 36447} >2018-07-11 07:27:11.624553 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294031624550, "job": 0, "event": "table_file_deletion", "file_number": 36446} >2018-07-11 07:27:11.624577 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294031624574, "job": 0, "event": "table_file_deletion", "file_number": 36445} >2018-07-11 07:27:11.624610 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294031624607, "job": 0, "event": "table_file_deletion", "file_number": 36443} >2018-07-11 07:27:11.624640 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294031624636, "job": 0, "event": "table_file_deletion", "file_number": 36441} >2018-07-11 07:27:11.624665 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294031624661, "job": 0, "event": "table_file_deletion", "file_number": 36438} >2018-07-11 07:27:11.624690 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294031624686, "job": 0, "event": "table_file_deletion", "file_number": 36436} >2018-07-11 07:27:17.013682 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 47] Generated table #36511: 4957167 keys, 80348317 bytes >2018-07-11 07:27:17.013739 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294037013709, "cf_name": "default", "job": 47, "event": "table_file_creation", "file_number": 36511, "file_size": 80348317, "table_properties": {"data_size": 67109464, "index_size": 689156, "filter_size": 12548707, "raw_key_size": 143207056, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957167, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957167", "kMergeOperands": "0"}} >2018-07-11 07:27:24.453329 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 47] Generated table #36512: 4769959 keys, 79881399 bytes >2018-07-11 07:27:24.453385 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294044453368, "cf_name": "default", "job": 47, "event": "table_file_creation", "file_number": 36512, "file_size": 79881399, "table_properties": {"data_size": 67111177, "index_size": 689201, "filter_size": 12080028, "raw_key_size": 137798827, "raw_average_key_size": 28, "raw_value_size": 2533690, "raw_average_value_size": 0, "num_data_blocks": 16472, "num_entries": 4769959, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4556845", "kMergeOperands": "0"}} >2018-07-11 07:27:30.790862 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 47] Generated table #36513: 3840065 keys, 77554767 bytes >2018-07-11 07:27:30.790915 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294050790899, "cf_name": "default", "job": 47, "event": "table_file_creation", "file_number": 36513, "file_size": 77554767, "table_properties": {"data_size": 67112185, "index_size": 689589, "filter_size": 9752000, "raw_key_size": 110935200, "raw_average_key_size": 28, "raw_value_size": 15111361, "raw_average_value_size": 3, "num_data_blocks": 16480, "num_entries": 3840065, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "2569016", "kMergeOperands": "0"}} >2018-07-11 07:27:38.023214 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 47] Generated table #36514: 4957167 keys, 80348346 bytes >2018-07-11 07:27:38.023276 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294058023242, "cf_name": "default", "job": 47, "event": "table_file_creation", "file_number": 36514, "file_size": 80348346, "table_properties": {"data_size": 67109429, "index_size": 689220, "filter_size": 12548707, "raw_key_size": 143207058, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957167, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957167", "kMergeOperands": "0"}} >2018-07-11 07:27:45.365047 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 47] Generated table #36515: 4957165 keys, 80348274 bytes >2018-07-11 07:27:45.365102 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294065365074, "cf_name": "default", "job": 47, "event": "table_file_creation", "file_number": 36515, "file_size": 80348274, "table_properties": {"data_size": 67109365, "index_size": 689218, "filter_size": 12548701, "raw_key_size": 143206991, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957165, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957165", "kMergeOperands": "0"}} >2018-07-11 07:27:51.535472 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 47] Generated table #36516: 4374530 keys, 70904655 bytes >2018-07-11 07:27:51.535524 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294071535496, "cf_name": "default", "job": 47, "event": "table_file_creation", "file_number": 36516, "file_size": 70904655, "table_properties": {"data_size": 59221737, "index_size": 608132, "filter_size": 11073797, "raw_key_size": 126375303, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 14534, "num_entries": 4374530, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4374530", "kMergeOperands": "0"}} >2018-07-11 07:27:51.535559 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 47] Compacted 2@1 + 5@2 files to L2 => 469385758 bytes >2018-07-11 07:27:51.546048 7f354e0ae700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 57973068228 >2018-07-11 07:27:51.546153 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:27:51.546013) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[1 23 70 446 3641 796 0] max score 9.80, MB/sec: 11.2 rd, 11.2 wr, level 2, files in(2, 5) out(6) MB in(153.3, 294.4) out(447.6), read-write-amplify(5.8) write-amplify(2.9) OK, records in: 27856053, records dropped: 0 > >2018-07-11 07:27:51.546162 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:27:51.546039) EVENT_LOG_v1 {"time_micros": 1531294071546027, "job": 47, "event": "compaction_finished", "compaction_time_micros": 41737277, "output_level": 2, "num_output_files": 6, "total_output_size": 469385758, "num_input_records": 27856053, "num_output_records": 27856053, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 23, 70, 446, 3641, 796, 0]} >2018-07-11 07:27:51.547149 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 48] Compacting 1@1 + 5@2 files to L2, score 9.80 >2018-07-11 07:27:51.547162 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 47 Base level 1, inputs: [36482(76MB)], [36430(449KB) 36431(76MB) 36432(43MB) 36414(72MB) 36415(67MB)] > >2018-07-11 07:27:51.547176 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294071547165, "job": 48, "event": "compaction_started", "files_L1": [36482], "files_L2": [36430, 36431, 36432, 36414, 36415], "score": 9.80252, "input_data_size": 354524066} >2018-07-11 07:27:58.965104 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294078965099, "job": 0, "event": "table_file_deletion", "file_number": 36494} >2018-07-11 07:27:58.965133 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294078965131, "job": 0, "event": "table_file_deletion", "file_number": 36491} >2018-07-11 07:27:58.965146 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294078965144, "job": 0, "event": "table_file_deletion", "file_number": 36457} >2018-07-11 07:27:58.965159 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294078965157, "job": 0, "event": "table_file_deletion", "file_number": 36435} >2018-07-11 07:27:58.965170 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294078965168, "job": 0, "event": "table_file_deletion", "file_number": 36424} >2018-07-11 07:27:58.965180 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294078965178, "job": 0, "event": "table_file_deletion", "file_number": 36423} >2018-07-11 07:27:58.965190 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294078965188, "job": 0, "event": "table_file_deletion", "file_number": 36419} >2018-07-11 07:27:59.370834 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 48] Generated table #36517: 4928777 keys, 80280326 bytes >2018-07-11 07:27:59.370893 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294079370863, "cf_name": "default", "job": 48, "event": "table_file_creation", "file_number": 36517, "file_size": 80280326, "table_properties": {"data_size": 67112483, "index_size": 689216, "filter_size": 12477635, "raw_key_size": 142386900, "raw_average_key_size": 28, "raw_value_size": 386989, "raw_average_value_size": 0, "num_data_blocks": 16471, "num_entries": 4928777, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4896226", "kMergeOperands": "0"}} >2018-07-11 07:28:06.076790 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 48] Generated table #36518: 4698968 keys, 79705429 bytes >2018-07-11 07:28:06.076839 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294086076810, "cf_name": "default", "job": 48, "event": "table_file_creation", "file_number": 36518, "file_size": 79705429, "table_properties": {"data_size": 67112866, "index_size": 689271, "filter_size": 11902299, "raw_key_size": 135747974, "raw_average_key_size": 28, "raw_value_size": 3495477, "raw_average_value_size": 0, "num_data_blocks": 16473, "num_entries": 4698968, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4404956", "kMergeOperands": "0"}} >2018-07-11 07:28:11.263797 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 48] Generated table #36519: 3218513 keys, 66569531 bytes >2018-07-11 07:28:11.263843 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294091263816, "cf_name": "default", "job": 48, "event": "table_file_creation", "file_number": 36519, "file_size": 66569531, "table_properties": {"data_size": 57797820, "index_size": 593996, "filter_size": 8176724, "raw_key_size": 92979274, "raw_average_key_size": 28, "raw_value_size": 14212300, "raw_average_value_size": 4, "num_data_blocks": 14193, "num_entries": 3218513, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "2023086", "kMergeOperands": "0"}} >2018-07-11 07:28:17.344233 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 48] Generated table #36520: 3691958 keys, 77182940 bytes >2018-07-11 07:28:17.344287 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294097344259, "cf_name": "default", "job": 48, "event": "table_file_creation", "file_number": 36520, "file_size": 77182940, "table_properties": {"data_size": 67111205, "index_size": 689534, "filter_size": 9381208, "raw_key_size": 106656558, "raw_average_key_size": 28, "raw_value_size": 17113476, "raw_average_value_size": 4, "num_data_blocks": 16481, "num_entries": 3691958, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "2252507", "kMergeOperands": "0"}} >2018-07-11 07:28:21.425962 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 48] Generated table #36521: 2822664 keys, 50785461 bytes >2018-07-11 07:28:21.426008 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294101425993, "cf_name": "default", "job": 48, "event": "table_file_creation", "file_number": 36521, "file_size": 50785461, "table_properties": {"data_size": 43185353, "index_size": 443512, "filter_size": 7155604, "raw_key_size": 81543640, "raw_average_key_size": 28, "raw_value_size": 4967731, "raw_average_value_size": 1, "num_data_blocks": 10601, "num_entries": 2822664, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "2404818", "kMergeOperands": "0"}} >2018-07-11 07:28:21.426041 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 48] Compacted 1@1 + 5@2 files to L2 => 354523687 bytes >2018-07-11 07:28:21.436937 7f354e0ae700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 57917098246 >2018-07-11 07:28:21.437034 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:28:21.436905) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[1 22 70 446 3641 796 0] max score 9.21, MB/sec: 11.9 rd, 11.9 wr, level 2, files in(1, 5) out(5) MB in(76.6, 261.5) out(338.1), read-write-amplify(8.8) write-amplify(4.4) OK, records in: 19360880, records dropped: 0 > >2018-07-11 07:28:21.437043 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:28:21.436929) EVENT_LOG_v1 {"time_micros": 1531294101436918, "job": 48, "event": "compaction_finished", "compaction_time_micros": 29878850, "output_level": 2, "num_output_files": 5, "total_output_size": 354523687, "num_input_records": 19360880, "num_output_records": 19360880, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 22, 70, 446, 3641, 796, 0]} >2018-07-11 07:28:21.438001 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 49] Compacting 1@1 + 2@2 files to L2, score 9.21 >2018-07-11 07:28:21.438013 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 48 Base level 1, inputs: [36485(76MB)], [36459(76MB) 36460(76MB)] > >2018-07-11 07:28:21.438026 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294101438016, "job": 49, "event": "compaction_started", "files_L1": [36485], "files_L2": [36459, 36460], "score": 9.20772, "input_data_size": 241044783} >2018-07-11 07:28:28.814899 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 49] Generated table #36522: 4957165 keys, 80348243 bytes >2018-07-11 07:28:28.814950 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294108814922, "cf_name": "default", "job": 49, "event": "table_file_creation", "file_number": 36522, "file_size": 80348243, "table_properties": {"data_size": 67109319, "index_size": 689233, "filter_size": 12548701, "raw_key_size": 143207008, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957165, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957165", "kMergeOperands": "0"}} >2018-07-11 07:28:36.241078 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 49] Generated table #36523: 4957165 keys, 80348251 bytes >2018-07-11 07:28:36.241120 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294116241103, "cf_name": "default", "job": 49, "event": "table_file_creation", "file_number": 36523, "file_size": 80348251, "table_properties": {"data_size": 67109331, "index_size": 689229, "filter_size": 12548701, "raw_key_size": 143206992, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957165, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957165", "kMergeOperands": "0"}} >2018-07-11 07:28:43.442686 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 49] Generated table #36524: 4957164 keys, 80348228 bytes >2018-07-11 07:28:43.442743 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294123442714, "cf_name": "default", "job": 49, "event": "table_file_creation", "file_number": 36524, "file_size": 80348228, "table_properties": {"data_size": 67109310, "index_size": 689230, "filter_size": 12548698, "raw_key_size": 143206951, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957164, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957164", "kMergeOperands": "0"}} >2018-07-11 07:28:43.478370 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 49] Generated table #36525: 2 keys, 1064 bytes >2018-07-11 07:28:43.478411 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294123478386, "cf_name": "default", "job": 49, "event": "table_file_creation", "file_number": 36525, "file_size": 1064, "table_properties": {"data_size": 57, "index_size": 27, "filter_size": 18, "raw_key_size": 58, "raw_average_key_size": 29, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 1, "num_entries": 2, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "2", "kMergeOperands": "0"}} >2018-07-11 07:28:43.478436 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 49] Compacted 1@1 + 2@2 files to L2 => 241045786 bytes >2018-07-11 07:28:43.486563 7f354e0ae700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 57854819590 >2018-07-11 07:28:43.486738 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:28:43.486532) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[1 21 72 446 3641 796 0] max score 8.61, MB/sec: 10.9 rd, 10.9 wr, level 2, files in(1, 2) out(4) MB in(76.6, 153.3) out(229.9), read-write-amplify(6.0) write-amplify(3.0) OK, records in: 14871496, records dropped: 0 > >2018-07-11 07:28:43.486748 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:28:43.486555) EVENT_LOG_v1 {"time_micros": 1531294123486545, "job": 49, "event": "compaction_finished", "compaction_time_micros": 22040395, "output_level": 2, "num_output_files": 4, "total_output_size": 241045786, "num_input_records": 14871496, "num_output_records": 14871496, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 21, 72, 446, 3641, 796, 0]} >2018-07-11 07:28:43.487689 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_picker.cc:461] [default] Expanding@1 1+3(159662945+319326915 bytes) to 2+3 (239256042+319326915bytes) > >2018-07-11 07:28:43.487787 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 50] Compacting 2@1 + 3@2 files to L2, score 8.61 >2018-07-11 07:28:43.487799 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 49 Base level 1, inputs: [36489(76MB) 36490(38MB)], [36509(76MB) 36510(1064B) 36456(76MB)] > >2018-07-11 07:28:43.487813 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294123487802, "job": 50, "event": "compaction_started", "files_L1": [36489, 36490], "files_L2": [36509, 36510, 36456], "score": 8.61293, "input_data_size": 281100125} >2018-07-11 07:28:50.595579 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 50] Generated table #36526: 4957165 keys, 80348252 bytes >2018-07-11 07:28:50.595634 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294130595605, "cf_name": "default", "job": 50, "event": "table_file_creation", "file_number": 36526, "file_size": 80348252, "table_properties": {"data_size": 67109357, "index_size": 689204, "filter_size": 12548701, "raw_key_size": 143206991, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957165, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957165", "kMergeOperands": "0"}} >2018-07-11 07:28:53.425877 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294133425871, "job": 0, "event": "table_file_deletion", "file_number": 36485} >2018-07-11 07:28:53.425893 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294133425891, "job": 0, "event": "table_file_deletion", "file_number": 36482} >2018-07-11 07:28:53.425908 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294133425906, "job": 0, "event": "table_file_deletion", "file_number": 36460} >2018-07-11 07:28:53.425919 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294133425917, "job": 0, "event": "table_file_deletion", "file_number": 36459} >2018-07-11 07:28:53.425931 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294133425928, "job": 0, "event": "table_file_deletion", "file_number": 36432} >2018-07-11 07:28:53.425941 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294133425939, "job": 0, "event": "table_file_deletion", "file_number": 36431} >2018-07-11 07:28:53.425955 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294133425952, "job": 0, "event": "table_file_deletion", "file_number": 36430} >2018-07-11 07:28:53.425971 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294133425968, "job": 0, "event": "table_file_deletion", "file_number": 36415} >2018-07-11 07:28:53.425984 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294133425982, "job": 0, "event": "table_file_deletion", "file_number": 36414} >2018-07-11 07:28:57.841566 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 50] Generated table #36527: 4957165 keys, 80348264 bytes >2018-07-11 07:28:57.841618 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294137841591, "cf_name": "default", "job": 50, "event": "table_file_creation", "file_number": 36527, "file_size": 80348264, "table_properties": {"data_size": 67109384, "index_size": 689189, "filter_size": 12548701, "raw_key_size": 143206995, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957165, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957165", "kMergeOperands": "0"}} >2018-07-11 07:29:05.244272 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 50] Generated table #36528: 4957165 keys, 80348247 bytes >2018-07-11 07:29:05.244325 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294145244297, "cf_name": "default", "job": 50, "event": "table_file_creation", "file_number": 36528, "file_size": 80348247, "table_properties": {"data_size": 67109367, "index_size": 689189, "filter_size": 12548701, "raw_key_size": 143206990, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957165, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957165", "kMergeOperands": "0"}} >2018-07-11 07:29:08.762254 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 50] Generated table #36529: 2471182 keys, 40054384 bytes >2018-07-11 07:29:08.762286 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294148762271, "cf_name": "default", "job": 50, "event": "table_file_creation", "file_number": 36529, "file_size": 40054384, "table_properties": {"data_size": 33454484, "index_size": 343297, "filter_size": 6255614, "raw_key_size": 71389712, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 8210, "num_entries": 2471182, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "2471182", "kMergeOperands": "0"}} >2018-07-11 07:29:08.762316 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 50] Compacted 2@1 + 3@2 files to L2 => 281099147 bytes >2018-07-11 07:29:08.777503 7f354e0ae700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 57747052972 >2018-07-11 07:29:08.777604 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:29:08.777469) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[1 19 73 446 3641 796 0] max score 7.72, MB/sec: 11.1 rd, 11.1 wr, level 2, files in(2, 3) out(4) MB in(114.8, 153.3) out(268.1), read-write-amplify(4.7) write-amplify(2.3) OK, records in: 17342677, records dropped: 0 > >2018-07-11 07:29:08.777612 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:29:08.777494) EVENT_LOG_v1 {"time_micros": 1531294148777483, "job": 50, "event": "compaction_finished", "compaction_time_micros": 25274491, "output_level": 2, "num_output_files": 4, "total_output_size": 281099147, "num_input_records": 17342677, "num_output_records": 17342677, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 19, 73, 446, 3641, 796, 0]} >2018-07-11 07:29:08.778620 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 51] Compacting 1@1 + 4@2 files to L2, score 7.72 >2018-07-11 07:29:08.778634 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 50 Base level 1, inputs: [36479(76MB)], [36471(68MB) 36472(36MB) 36473(54MB) 36474(61MB)] > >2018-07-11 07:29:08.778647 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294148778637, "job": 51, "event": "compaction_started", "files_L1": [36479], "files_L2": [36471, 36472, 36473, 36474], "score": 7.72164, "input_data_size": 312604624} >2018-07-11 07:29:13.317973 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 51] Generated table #36530: 1364524 keys, 71602263 bytes >2018-07-11 07:29:13.318003 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294153317989, "cf_name": "default", "job": 51, "event": "table_file_creation", "file_number": 36530, "file_size": 71602263, "table_properties": {"data_size": 67109270, "index_size": 930722, "filter_size": 3561280, "raw_key_size": 41015889, "raw_average_key_size": 30, "raw_value_size": 48070597, "raw_average_value_size": 35, "num_data_blocks": 16882, "num_entries": 1364524, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "638130", "kMergeOperands": "0"}} >2018-07-11 07:29:18.712258 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 51] Generated table #36531: 3447538 keys, 55879604 bytes >2018-07-11 07:29:18.712303 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294158712276, "cf_name": "default", "job": 51, "event": "table_file_creation", "file_number": 36531, "file_size": 55879604, "table_properties": {"data_size": 46672272, "index_size": 479157, "filter_size": 8727186, "raw_key_size": 99595554, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 11454, "num_entries": 3447538, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "3447538", "kMergeOperands": "0"}} >2018-07-11 07:29:24.416073 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 51] Generated table #36532: 4315389 keys, 73745783 bytes >2018-07-11 07:29:24.416126 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294164416098, "cf_name": "default", "job": 51, "event": "table_file_creation", "file_number": 36532, "file_size": 73745783, "table_properties": {"data_size": 62174388, "index_size": 638584, "filter_size": 10931819, "raw_key_size": 124666794, "raw_average_key_size": 28, "raw_value_size": 3749621, "raw_average_value_size": 0, "num_data_blocks": 15261, "num_entries": 4315389, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4000000", "kMergeOperands": "0"}} >2018-07-11 07:29:31.278225 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 51] Generated table #36533: 4957167 keys, 80348309 bytes >2018-07-11 07:29:31.278288 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294171278271, "cf_name": "default", "job": 51, "event": "table_file_creation", "file_number": 36533, "file_size": 80348309, "table_properties": {"data_size": 67109364, "index_size": 689248, "filter_size": 12548707, "raw_key_size": 143207034, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957167, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957167", "kMergeOperands": "0"}} >2018-07-11 07:29:33.830870 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 51] Generated table #36534: 1914330 keys, 31028683 bytes >2018-07-11 07:29:33.830914 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294173830886, "cf_name": "default", "job": 51, "event": "table_file_creation", "file_number": 36534, "file_size": 31028683, "table_properties": {"data_size": 25915866, "index_size": 265845, "filter_size": 4845985, "raw_key_size": 55302880, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 6360, "num_entries": 1914330, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1914330", "kMergeOperands": "0"}} >2018-07-11 07:29:33.830946 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 51] Compacted 1@1 + 4@2 files to L2 => 312604642 bytes >2018-07-11 07:29:33.843703 7f354e0ae700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 57663318163 >2018-07-11 07:29:33.843900 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:29:33.843665) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[1 18 74 446 3641 796 0] max score 7.13, MB/sec: 12.5 rd, 12.5 wr, level 2, files in(1, 4) out(5) MB in(76.6, 221.5) out(298.1), read-write-amplify(7.8) write-amplify(3.9) OK, records in: 15998948, records dropped: 0 > >2018-07-11 07:29:33.843909 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:29:33.843695) EVENT_LOG_v1 {"time_micros": 1531294173843679, "job": 51, "event": "compaction_finished", "compaction_time_micros": 25052283, "output_level": 2, "num_output_files": 5, "total_output_size": 312604642, "num_input_records": 15998948, "num_output_records": 15998948, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 18, 74, 446, 3641, 796, 0]} >2018-07-11 07:29:33.844930 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 52] Compacting 1@1 + 5@2 files to L2, score 7.13 >2018-07-11 07:29:33.844954 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 51 Base level 1, inputs: [36481(76MB)], [36475(45MB) 36427(76MB) 36428(75MB) 36429(72MB) 36517(76MB)] > >2018-07-11 07:29:33.844969 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294173844958, "job": 52, "event": "compaction_started", "files_L1": [36481], "files_L2": [36475, 36427, 36428, 36429, 36517], "score": 7.12685, "input_data_size": 443181844} >2018-07-11 07:29:39.953266 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294179953243, "job": 0, "event": "table_file_deletion", "file_number": 36510} >2018-07-11 07:29:39.953289 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294179953285, "job": 0, "event": "table_file_deletion", "file_number": 36509} >2018-07-11 07:29:39.953307 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294179953304, "job": 0, "event": "table_file_deletion", "file_number": 36490} >2018-07-11 07:29:39.953324 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294179953321, "job": 0, "event": "table_file_deletion", "file_number": 36489} >2018-07-11 07:29:39.953350 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294179953346, "job": 0, "event": "table_file_deletion", "file_number": 36479} >2018-07-11 07:29:39.953376 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294179953371, "job": 0, "event": "table_file_deletion", "file_number": 36474} >2018-07-11 07:29:39.953399 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294179953396, "job": 0, "event": "table_file_deletion", "file_number": 36473} >2018-07-11 07:29:39.953421 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294179953418, "job": 0, "event": "table_file_deletion", "file_number": 36472} >2018-07-11 07:29:39.953444 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294179953441, "job": 0, "event": "table_file_deletion", "file_number": 36471} >2018-07-11 07:29:39.953467 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294179953463, "job": 0, "event": "table_file_deletion", "file_number": 36456} >2018-07-11 07:29:41.816505 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 52] Generated table #36535: 4957165 keys, 80348263 bytes >2018-07-11 07:29:41.816561 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294181816531, "cf_name": "default", "job": 52, "event": "table_file_creation", "file_number": 36535, "file_size": 80348263, "table_properties": {"data_size": 67109361, "index_size": 689211, "filter_size": 12548701, "raw_key_size": 143206990, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957165, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957165", "kMergeOperands": "0"}} >2018-07-11 07:29:49.008204 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 52] Generated table #36536: 4957165 keys, 80348260 bytes >2018-07-11 07:29:49.008259 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294189008223, "cf_name": "default", "job": 52, "event": "table_file_creation", "file_number": 36536, "file_size": 80348260, "table_properties": {"data_size": 67109360, "index_size": 689209, "filter_size": 12548701, "raw_key_size": 143207000, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957165, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957165", "kMergeOperands": "0"}} >2018-07-11 07:29:53.302754 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 52] Generated table #36537: 2800204 keys, 45387362 bytes >2018-07-11 07:29:53.302799 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294193302772, "cf_name": "default", "job": 52, "event": "table_file_creation", "file_number": 36537, "file_size": 45387362, "table_properties": {"data_size": 37908785, "index_size": 389068, "filter_size": 7088520, "raw_key_size": 80894786, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 9304, "num_entries": 2800204, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "2800204", "kMergeOperands": "0"}} >2018-07-11 07:29:59.958423 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 52] Generated table #36538: 4255299 keys, 78592095 bytes >2018-07-11 07:29:59.958474 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294199958446, "cf_name": "default", "job": 52, "event": "table_file_creation", "file_number": 36538, "file_size": 78592095, "table_properties": {"data_size": 67110022, "index_size": 689529, "filter_size": 10791551, "raw_key_size": 122930866, "raw_average_key_size": 28, "raw_value_size": 9493216, "raw_average_value_size": 2, "num_data_blocks": 16476, "num_entries": 4255299, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "3456804", "kMergeOperands": "0"}} >2018-07-11 07:30:05.947726 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 52] Generated table #36539: 3368793 keys, 76375844 bytes >2018-07-11 07:30:05.947769 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294205947742, "cf_name": "default", "job": 52, "event": "table_file_creation", "file_number": 36539, "file_size": 76375844, "table_properties": {"data_size": 67112512, "index_size": 690189, "filter_size": 8572151, "raw_key_size": 97320681, "raw_average_key_size": 28, "raw_value_size": 21485521, "raw_average_value_size": 6, "num_data_blocks": 16484, "num_entries": 3368793, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1561600", "kMergeOperands": "0"}} >2018-07-11 07:30:13.044969 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 52] Generated table #36540: 4928777 keys, 80280309 bytes >2018-07-11 07:30:13.045016 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294213044989, "cf_name": "default", "job": 52, "event": "table_file_creation", "file_number": 36540, "file_size": 80280309, "table_properties": {"data_size": 67112438, "index_size": 689243, "filter_size": 12477636, "raw_key_size": 142386888, "raw_average_key_size": 28, "raw_value_size": 386989, "raw_average_value_size": 0, "num_data_blocks": 16471, "num_entries": 4928777, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4896226", "kMergeOperands": "0"}} >2018-07-11 07:30:13.371737 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 52] Generated table #36541: 114132 keys, 1850548 bytes >2018-07-11 07:30:13.371762 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294213371750, "cf_name": "default", "job": 52, "event": "table_file_creation", "file_number": 36541, "file_size": 1850548, "table_properties": {"data_size": 1545130, "index_size": 15512, "filter_size": 288925, "raw_key_size": 3297151, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 380, "num_entries": 114132, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "114132", "kMergeOperands": "0"}} >2018-07-11 07:30:13.371787 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 52] Compacted 1@1 + 5@2 files to L2 => 443182681 bytes >2018-07-11 07:30:13.384433 7f354e0ae700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 57567792009 >2018-07-11 07:30:13.384544 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:30:13.384403) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[1 17 76 446 3641 796 0] max score 6.53, MB/sec: 11.2 rd, 11.2 wr, level 2, files in(1, 5) out(7) MB in(76.6, 346.0) out(422.7), read-write-amplify(11.0) write-amplify(5.5) OK, records in: 25381535, records dropped: 0 > >2018-07-11 07:30:13.384553 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:30:13.384426) EVENT_LOG_v1 {"time_micros": 1531294213384415, "job": 52, "event": "compaction_finished", "compaction_time_micros": 39526804, "output_level": 2, "num_output_files": 7, "total_output_size": 443182681, "num_input_records": 25381535, "num_output_records": 25381535, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 17, 76, 446, 3641, 796, 0]} >2018-07-11 07:30:13.385549 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 53] Compacting 1@1 + 2@2 files to L2, score 6.53 >2018-07-11 07:30:13.385560 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 52 Base level 1, inputs: [36480(76MB)], [36533(76MB) 36534(29MB)] > >2018-07-11 07:30:13.385573 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294213385564, "job": 53, "event": "compaction_started", "files_L1": [36480], "files_L2": [36533, 36534], "score": 6.53205, "input_data_size": 191725256} >2018-07-11 07:30:20.854399 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 53] Generated table #36542: 4957168 keys, 80348314 bytes >2018-07-11 07:30:20.854454 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294220854437, "cf_name": "default", "job": 53, "event": "table_file_creation", "file_number": 36542, "file_size": 80348314, "table_properties": {"data_size": 67109372, "index_size": 689242, "filter_size": 12548710, "raw_key_size": 143207073, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957168, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957168", "kMergeOperands": "0"}} >2018-07-11 07:30:27.361479 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 53] Generated table #36543: 4957165 keys, 80348257 bytes >2018-07-11 07:30:27.361535 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294227361505, "cf_name": "default", "job": 53, "event": "table_file_creation", "file_number": 36543, "file_size": 80348257, "table_properties": {"data_size": 67109346, "index_size": 689220, "filter_size": 12548701, "raw_key_size": 143206995, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957165, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957165", "kMergeOperands": "0"}} >2018-07-11 07:30:27.570464 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 53] Generated table #36544: 85667 keys, 1389237 bytes >2018-07-11 07:30:27.570501 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294227570476, "cf_name": "default", "job": 53, "event": "table_file_creation", "file_number": 36544, "file_size": 1389237, "table_properties": {"data_size": 1159765, "index_size": 11627, "filter_size": 216864, "raw_key_size": 2474832, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 285, "num_entries": 85667, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "85667", "kMergeOperands": "0"}} >2018-07-11 07:30:28.925135 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294228925129, "job": 0, "event": "table_file_deletion", "file_number": 36517} >2018-07-11 07:30:28.925152 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294228925148, "job": 0, "event": "table_file_deletion", "file_number": 36481} >2018-07-11 07:30:28.925165 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294228925163, "job": 0, "event": "table_file_deletion", "file_number": 36475} >2018-07-11 07:30:28.925178 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294228925176, "job": 0, "event": "table_file_deletion", "file_number": 36429} >2018-07-11 07:30:28.925188 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294228925186, "job": 0, "event": "table_file_deletion", "file_number": 36428} >2018-07-11 07:30:28.925198 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294228925196, "job": 0, "event": "table_file_deletion", "file_number": 36427} >2018-07-11 07:30:29.907844 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 53] Generated table #36545: 1828662 keys, 29640198 bytes >2018-07-11 07:30:29.907887 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294229907860, "cf_name": "default", "job": 53, "event": "table_file_creation", "file_number": 36545, "file_size": 29640198, "table_properties": {"data_size": 24756162, "index_size": 253923, "filter_size": 4629126, "raw_key_size": 52828009, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 6076, "num_entries": 1828662, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1828662", "kMergeOperands": "0"}} >2018-07-11 07:30:29.907917 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 53] Compacted 1@1 + 2@2 files to L2 => 191726006 bytes >2018-07-11 07:30:29.917529 7f354e0ae700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 57457777005 >2018-07-11 07:30:29.917652 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:30:29.917496) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[1 16 78 446 3641 796 0] max score 5.94, MB/sec: 11.6 rd, 11.6 wr, level 2, files in(1, 2) out(4) MB in(76.6, 106.2) out(182.8), read-write-amplify(4.8) write-amplify(2.4) OK, records in: 11828662, records dropped: 0 > >2018-07-11 07:30:29.917661 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:30:29.917521) EVENT_LOG_v1 {"time_micros": 1531294229917510, "job": 53, "event": "compaction_finished", "compaction_time_micros": 16522331, "output_level": 2, "num_output_files": 4, "total_output_size": 191726006, "num_input_records": 11828662, "num_output_records": 11828662, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 16, 78, 446, 3641, 796, 0]} >2018-07-11 07:30:29.918665 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 54] Compacting 1@1 + 3@2 files to L2, score 5.94 >2018-07-11 07:30:29.918677 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 53 Base level 1, inputs: [36478(76MB)], [36469(46MB) 36470(73MB) 36530(68MB)] > >2018-07-11 07:30:29.918702 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294229918680, "job": 54, "event": "compaction_started", "files_L1": [36478], "files_L2": [36469, 36470, 36530], "score": 5.93726, "input_data_size": 277631587} >2018-07-11 07:30:36.497459 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 54] Generated table #36546: 4042833 keys, 65528355 bytes >2018-07-11 07:30:36.497510 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294236497482, "cf_name": "default", "job": 54, "event": "table_file_creation", "file_number": 36546, "file_size": 65528355, "table_properties": {"data_size": 54731248, "index_size": 561989, "filter_size": 10234129, "raw_key_size": 116792966, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 13432, "num_entries": 4042833, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4042833", "kMergeOperands": "0"}} >2018-07-11 07:30:43.796895 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 54] Generated table #36547: 4957168 keys, 80348313 bytes >2018-07-11 07:30:43.796951 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294243796920, "cf_name": "default", "job": 54, "event": "table_file_creation", "file_number": 36547, "file_size": 80348313, "table_properties": {"data_size": 67109357, "index_size": 689256, "filter_size": 12548710, "raw_key_size": 143207073, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957168, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957168", "kMergeOperands": "0"}} >2018-07-11 07:30:48.411303 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 54] Generated table #36548: 1705513 keys, 72433920 bytes >2018-07-11 07:30:48.411343 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294248411317, "cf_name": "default", "job": 54, "event": "table_file_creation", "file_number": 36548, "file_size": 72433920, "table_properties": {"data_size": 67111422, "index_size": 907181, "filter_size": 4414326, "raw_key_size": 50707736, "raw_average_key_size": 29, "raw_value_size": 43513496, "raw_average_value_size": 25, "num_data_blocks": 16836, "num_entries": 1705513, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1042832", "kMergeOperands": "0"}} >2018-07-11 07:30:52.585771 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 54] Generated table #36549: 2230704 keys, 59321027 bytes >2018-07-11 07:30:52.585813 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294252585787, "cf_name": "default", "job": 54, "event": "table_file_creation", "file_number": 36549, "file_size": 59321027, "table_properties": {"data_size": 52990730, "index_size": 632976, "filter_size": 5696330, "raw_key_size": 65023033, "raw_average_key_size": 29, "raw_value_size": 22579960, "raw_average_value_size": 10, "num_data_blocks": 13158, "num_entries": 2230704, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1552462", "kMergeOperands": "0"}} >2018-07-11 07:30:52.585845 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 54] Compacted 1@1 + 3@2 files to L2 => 277631615 bytes >2018-07-11 07:30:52.600345 7f354e0ae700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 57329698479 >2018-07-11 07:30:52.600599 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:30:52.600313) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[1 15 79 446 3641 796 0] max score 5.34, MB/sec: 12.2 rd, 12.2 wr, level 2, files in(1, 3) out(4) MB in(76.6, 188.1) out(264.8), read-write-amplify(6.9) write-amplify(3.5) OK, records in: 12936218, records dropped: 0 > >2018-07-11 07:30:52.600609 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:30:52.600338) EVENT_LOG_v1 {"time_micros": 1531294252600327, "job": 54, "event": "compaction_finished", "compaction_time_micros": 22667131, "output_level": 2, "num_output_files": 4, "total_output_size": 277631615, "num_input_records": 12936218, "num_output_records": 12936218, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 15, 79, 446, 3641, 796, 0]} >2018-07-11 07:30:52.601609 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 55] Compacting 1@1 + 2@2 files to L2, score 5.34 >2018-07-11 07:30:52.601620 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 54 Base level 1, inputs: [36483(76MB)], [36416(75MB) 36458(76MB)] > >2018-07-11 07:30:52.601645 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294252601623, "job": 55, "event": "compaction_started", "files_L1": [36483], "files_L2": [36416, 36458], "score": 5.34247, "input_data_size": 240229132} >2018-07-11 07:31:00.110901 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 55] Generated table #36550: 4630820 keys, 79532512 bytes >2018-07-11 07:31:00.110947 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294260110920, "cf_name": "default", "job": 55, "event": "table_file_creation", "file_number": 36550, "file_size": 79532512, "table_properties": {"data_size": 67110534, "index_size": 689301, "filter_size": 11731684, "raw_key_size": 133779240, "raw_average_key_size": 28, "raw_value_size": 4414873, "raw_average_value_size": 0, "num_data_blocks": 16473, "num_entries": 4630820, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4259475", "kMergeOperands": "0"}} >2018-07-11 07:31:07.462275 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 55] Generated table #36551: 4957165 keys, 80348271 bytes >2018-07-11 07:31:07.462333 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294267462304, "cf_name": "default", "job": 55, "event": "table_file_creation", "file_number": 36551, "file_size": 80348271, "table_properties": {"data_size": 67109407, "index_size": 689173, "filter_size": 12548701, "raw_key_size": 143207000, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957165, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957165", "kMergeOperands": "0"}} >2018-07-11 07:31:13.927028 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 55] Generated table #36552: 4957162 keys, 80348135 bytes >2018-07-11 07:31:13.927086 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294273927057, "cf_name": "default", "job": 55, "event": "table_file_creation", "file_number": 36552, "file_size": 80348135, "table_properties": {"data_size": 67109327, "index_size": 689136, "filter_size": 12548682, "raw_key_size": 143206899, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16469, "num_entries": 4957162, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957162", "kMergeOperands": "0"}} >2018-07-11 07:31:13.927119 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 55] Compacted 1@1 + 2@2 files to L2 => 240228918 bytes >2018-07-11 07:31:13.941518 7f354e0ae700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 57178722688 >2018-07-11 07:31:13.941747 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:31:13.941483) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[1 14 80 446 3641 796 0] max score 4.75, MB/sec: 11.3 rd, 11.3 wr, level 2, files in(1, 2) out(3) MB in(76.6, 152.5) out(229.1), read-write-amplify(6.0) write-amplify(3.0) OK, records in: 14545147, records dropped: 0 > >2018-07-11 07:31:13.941757 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:31:13.941509) EVENT_LOG_v1 {"time_micros": 1531294273941498, "job": 55, "event": "compaction_finished", "compaction_time_micros": 21325461, "output_level": 2, "num_output_files": 3, "total_output_size": 240228918, "num_input_records": 14545147, "num_output_records": 14545147, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 14, 80, 446, 3641, 796, 0]} >2018-07-11 07:31:13.942774 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 56] Compacting 1@1 + 4@2 files to L2, score 4.75 >2018-07-11 07:31:13.942785 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 55 Base level 1, inputs: [36486(76MB)], [36524(76MB) 36525(1064B) 36461(15MB) 36462(71MB)] > >2018-07-11 07:31:13.942812 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294273942788, "job": 56, "event": "compaction_started", "files_L1": [36486], "files_L2": [36524, 36525, 36461, 36462], "score": 4.74768, "input_data_size": 251660600} >2018-07-11 07:31:19.715680 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294279715674, "job": 0, "event": "table_file_deletion", "file_number": 36534} >2018-07-11 07:31:19.715705 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294279715701, "job": 0, "event": "table_file_deletion", "file_number": 36533} >2018-07-11 07:31:19.715717 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294279715715, "job": 0, "event": "table_file_deletion", "file_number": 36530} >2018-07-11 07:31:19.715731 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294279715729, "job": 0, "event": "table_file_deletion", "file_number": 36483} >2018-07-11 07:31:19.715745 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294279715743, "job": 0, "event": "table_file_deletion", "file_number": 36480} >2018-07-11 07:31:19.715758 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294279715755, "job": 0, "event": "table_file_deletion", "file_number": 36478} >2018-07-11 07:31:19.715770 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294279715768, "job": 0, "event": "table_file_deletion", "file_number": 36470} >2018-07-11 07:31:19.715781 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294279715779, "job": 0, "event": "table_file_deletion", "file_number": 36469} >2018-07-11 07:31:19.715793 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294279715790, "job": 0, "event": "table_file_deletion", "file_number": 36458} >2018-07-11 07:31:19.715804 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294279715802, "job": 0, "event": "table_file_deletion", "file_number": 36416} >2018-07-11 07:31:21.642423 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 56] Generated table #36553: 4957165 keys, 80348240 bytes >2018-07-11 07:31:21.642482 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294281642451, "cf_name": "default", "job": 56, "event": "table_file_creation", "file_number": 36553, "file_size": 80348240, "table_properties": {"data_size": 67109317, "index_size": 689232, "filter_size": 12548701, "raw_key_size": 143206998, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957165, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957165", "kMergeOperands": "0"}} >2018-07-11 07:31:29.231533 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 56] Generated table #36554: 4957165 keys, 80348234 bytes >2018-07-11 07:31:29.231580 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294289231552, "cf_name": "default", "job": 56, "event": "table_file_creation", "file_number": 36554, "file_size": 80348234, "table_properties": {"data_size": 67109327, "index_size": 689216, "filter_size": 12548701, "raw_key_size": 143206992, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957165, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957165", "kMergeOperands": "0"}} >2018-07-11 07:31:36.081443 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 56] Generated table #36555: 4957168 keys, 80348280 bytes >2018-07-11 07:31:36.081497 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294296081468, "cf_name": "default", "job": 56, "event": "table_file_creation", "file_number": 36555, "file_size": 80348280, "table_properties": {"data_size": 67109383, "index_size": 689197, "filter_size": 12548710, "raw_key_size": 143207061, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957168, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957168", "kMergeOperands": "0"}} >2018-07-11 07:31:37.000335 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 56] Generated table #36556: 654862 keys, 10614739 bytes >2018-07-11 07:31:37.000377 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294297000361, "cf_name": "default", "job": 56, "event": "table_file_creation", "file_number": 36556, "file_size": 10614739, "table_properties": {"data_size": 8865415, "index_size": 90603, "filter_size": 1657736, "raw_key_size": 18918240, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 2176, "num_entries": 654862, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "654862", "kMergeOperands": "0"}} >2018-07-11 07:31:37.000404 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 56] Compacted 1@1 + 4@2 files to L2 => 251659493 bytes >2018-07-11 07:31:37.015874 7f354e0ae700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 56998099742 >2018-07-11 07:31:37.015968 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:31:37.015843) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[1 13 80 446 3641 796 0] max score 4.15, MB/sec: 10.9 rd, 10.9 wr, level 2, files in(1, 4) out(4) MB in(76.6, 163.4) out(240.0), read-write-amplify(6.3) write-amplify(3.1) OK, records in: 15526360, records dropped: 0 > >2018-07-11 07:31:37.015976 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:31:37.015866) EVENT_LOG_v1 {"time_micros": 1531294297015855, "job": 56, "event": "compaction_finished", "compaction_time_micros": 23057580, "output_level": 2, "num_output_files": 4, "total_output_size": 251659493, "num_input_records": 15526360, "num_output_records": 15526360, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 13, 80, 446, 3641, 796, 0]} >2018-07-11 07:31:37.016963 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 57] Compacting 1@1 + 2@2 files to L2, score 4.15 >2018-07-11 07:31:37.016974 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 56 Base level 1, inputs: [36484(76MB)], [36552(76MB) 36522(76MB)] > >2018-07-11 07:31:37.017000 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294297016977, "job": 57, "event": "compaction_started", "files_L1": [36484], "files_L2": [36552, 36522], "score": 4.15289, "input_data_size": 241044622} >2018-07-11 07:31:44.127238 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 57] Generated table #36557: 4957165 keys, 80348241 bytes >2018-07-11 07:31:44.127290 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294304127262, "cf_name": "default", "job": 57, "event": "table_file_creation", "file_number": 36557, "file_size": 80348241, "table_properties": {"data_size": 67109368, "index_size": 689182, "filter_size": 12548701, "raw_key_size": 143206990, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957165, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957165", "kMergeOperands": "0"}} >2018-07-11 07:31:51.538988 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 57] Generated table #36558: 4957165 keys, 80348268 bytes >2018-07-11 07:31:51.539043 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294311539014, "cf_name": "default", "job": 57, "event": "table_file_creation", "file_number": 36558, "file_size": 80348268, "table_properties": {"data_size": 67109382, "index_size": 689195, "filter_size": 12548701, "raw_key_size": 143206995, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957165, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957165", "kMergeOperands": "0"}} >2018-07-11 07:31:58.353277 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 57] Generated table #36559: 4957162 keys, 80348127 bytes >2018-07-11 07:31:58.353331 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294318353303, "cf_name": "default", "job": 57, "event": "table_file_creation", "file_number": 36559, "file_size": 80348127, "table_properties": {"data_size": 67109286, "index_size": 689168, "filter_size": 12548683, "raw_key_size": 143206908, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16469, "num_entries": 4957162, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957162", "kMergeOperands": "0"}} >2018-07-11 07:31:58.353364 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 57] Compacted 1@1 + 2@2 files to L2 => 241044636 bytes >2018-07-11 07:31:58.365513 7f354e0ae700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 56778182695 >2018-07-11 07:31:58.365724 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:31:58.365479) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[1 12 81 446 3641 796 0] max score 3.96, MB/sec: 11.3 rd, 11.3 wr, level 2, files in(1, 2) out(3) MB in(76.6, 153.3) out(229.9), read-write-amplify(6.0) write-amplify(3.0) OK, records in: 14871492, records dropped: 0 > >2018-07-11 07:31:58.365733 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:31:58.365504) EVENT_LOG_v1 {"time_micros": 1531294318365493, "job": 57, "event": "compaction_finished", "compaction_time_micros": 21336352, "output_level": 2, "num_output_files": 3, "total_output_size": 241044636, "num_input_records": 14871492, "num_output_records": 14871492, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 12, 81, 446, 3641, 796, 0]} >2018-07-11 07:31:58.366758 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 58] Compacting 1@0 + 10@1 files to L1, score 3.96 >2018-07-11 07:31:58.366772 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 57 Base level 0, inputs: [36493(510MB)], [36476(76MB) 36487(76MB) 36495(75MB) 36496(71MB) 36497(71MB) 36498(71MB) 36499(71MB) 36500(71MB) 36501(10MB) 36502(65MB)] > >2018-07-11 07:31:58.366800 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294318366775, "job": 58, "event": "compaction_started", "files_L0": [36493], "files_L1": [36476, 36487, 36495, 36496, 36497, 36498, 36499, 36500, 36501, 36502], "score": 3.9597, "input_data_size": 1226940327} >2018-07-11 07:32:03.565646 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4d00000 on bdev 1, free 0x2fe000; fallback to bdev 2 >2018-07-11 07:32:04.886610 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 58] Generated table #36560: 4888760 keys, 80173878 bytes >2018-07-11 07:32:04.886668 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294324886638, "cf_name": "default", "job": 58, "event": "table_file_creation", "file_number": 36560, "file_size": 80173878, "table_properties": {"data_size": 67110180, "index_size": 685246, "filter_size": 12377461, "raw_key_size": 141267959, "raw_average_key_size": 28, "raw_value_size": 916542, "raw_average_value_size": 0, "num_data_blocks": 16336, "num_entries": 4888760, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4886796", "kMergeOperands": "0"}} >2018-07-11 07:32:05.964154 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294325964136, "job": 0, "event": "table_file_deletion", "file_number": 36552} >2018-07-11 07:32:05.964170 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294325964168, "job": 0, "event": "table_file_deletion", "file_number": 36525} >2018-07-11 07:32:05.964183 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294325964181, "job": 0, "event": "table_file_deletion", "file_number": 36524} >2018-07-11 07:32:05.964194 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294325964192, "job": 0, "event": "table_file_deletion", "file_number": 36522} >2018-07-11 07:32:05.964206 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294325964203, "job": 0, "event": "table_file_deletion", "file_number": 36486} >2018-07-11 07:32:05.964216 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294325964214, "job": 0, "event": "table_file_deletion", "file_number": 36484} >2018-07-11 07:32:05.964228 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294325964226, "job": 0, "event": "table_file_deletion", "file_number": 36462} >2018-07-11 07:32:05.964243 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294325964241, "job": 0, "event": "table_file_deletion", "file_number": 36461} >2018-07-11 07:32:09.836110 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4d00000 on bdev 1, free 0x2fe000; fallback to bdev 2 >2018-07-11 07:32:11.231230 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 58] Generated table #36561: 4957166 keys, 80348285 bytes >2018-07-11 07:32:11.231289 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294331231273, "cf_name": "default", "job": 58, "event": "table_file_creation", "file_number": 36561, "file_size": 80348285, "table_properties": {"data_size": 67109353, "index_size": 689238, "filter_size": 12548704, "raw_key_size": 143207015, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957166, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957166", "kMergeOperands": "0"}} >2018-07-11 07:32:16.284904 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4d00000 on bdev 1, free 0x2fe000; fallback to bdev 2 >2018-07-11 07:32:17.608904 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 58] Generated table #36562: 4957165 keys, 80348257 bytes >2018-07-11 07:32:17.608959 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294337608943, "cf_name": "default", "job": 58, "event": "table_file_creation", "file_number": 36562, "file_size": 80348257, "table_properties": {"data_size": 67109343, "index_size": 689223, "filter_size": 12548701, "raw_key_size": 143206995, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957165, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957165", "kMergeOperands": "0"}} >2018-07-11 07:32:23.125448 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4d00000 on bdev 1, free 0x2fe000; fallback to bdev 2 >2018-07-11 07:32:24.545840 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 58] Generated table #36563: 4957165 keys, 80348251 bytes >2018-07-11 07:32:24.545896 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294344545868, "cf_name": "default", "job": 58, "event": "table_file_creation", "file_number": 36563, "file_size": 80348251, "table_properties": {"data_size": 67109335, "index_size": 689225, "filter_size": 12548701, "raw_key_size": 143206995, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957165, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957165", "kMergeOperands": "0"}} >2018-07-11 07:32:26.856059 7f354e0ae700 1 bluefs _allocate failed to allocate 0x2100000 on bdev 1, free 0x2fe000; fallback to bdev 2 >2018-07-11 07:32:27.516804 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 58] Generated table #36564: 2128504 keys, 34500149 bytes >2018-07-11 07:32:27.516846 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294347516820, "cf_name": "default", "job": 58, "event": "table_file_creation", "file_number": 36564, "file_size": 34500149, "table_properties": {"data_size": 28815346, "index_size": 295662, "filter_size": 5388152, "raw_key_size": 61490125, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 7072, "num_entries": 2128504, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "2128504", "kMergeOperands": "0"}} >2018-07-11 07:32:32.720442 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4d00000 on bdev 1, free 0x2fe000; fallback to bdev 2 >2018-07-11 07:32:34.244108 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 58] Generated table #36565: 4957168 keys, 80348310 bytes >2018-07-11 07:32:34.244163 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294354244135, "cf_name": "default", "job": 58, "event": "table_file_creation", "file_number": 36565, "file_size": 80348310, "table_properties": {"data_size": 67109354, "index_size": 689256, "filter_size": 12548710, "raw_key_size": 143207073, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957168, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957168", "kMergeOperands": "0"}} >2018-07-11 07:32:39.638491 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4d00000 on bdev 1, free 0x2fe000; fallback to bdev 2 >2018-07-11 07:32:41.122007 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 58] Generated table #36566: 4957164 keys, 80348254 bytes >2018-07-11 07:32:41.122055 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294361122028, "cf_name": "default", "job": 58, "event": "table_file_creation", "file_number": 36566, "file_size": 80348254, "table_properties": {"data_size": 67109384, "index_size": 689182, "filter_size": 12548698, "raw_key_size": 143206978, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957164, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957164", "kMergeOperands": "0"}} >2018-07-11 07:32:46.654997 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4d00000 on bdev 1, free 0x2fe000; fallback to bdev 2 >2018-07-11 07:32:47.921505 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 58] Generated table #36567: 4957164 keys, 80348253 bytes >2018-07-11 07:32:47.921555 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294367921527, "cf_name": "default", "job": 58, "event": "table_file_creation", "file_number": 36567, "file_size": 80348253, "table_properties": {"data_size": 67109358, "index_size": 689207, "filter_size": 12548698, "raw_key_size": 143206958, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957164, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957164", "kMergeOperands": "0"}} >2018-07-11 07:32:53.439860 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4d00000 on bdev 1, free 0x2fe000; fallback to bdev 2 >2018-07-11 07:32:54.471827 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 58] Generated table #36568: 4957164 keys, 80348260 bytes >2018-07-11 07:32:54.471881 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294374471852, "cf_name": "default", "job": 58, "event": "table_file_creation", "file_number": 36568, "file_size": 80348260, "table_properties": {"data_size": 67109335, "index_size": 689237, "filter_size": 12548698, "raw_key_size": 143206976, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957164, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957164", "kMergeOperands": "0"}} >2018-07-11 07:33:00.151965 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4d00000 on bdev 1, free 0x2fe000; fallback to bdev 2 >2018-07-11 07:33:01.314218 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 58] Generated table #36569: 4957165 keys, 80348232 bytes >2018-07-11 07:33:01.314275 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294381314240, "cf_name": "default", "job": 58, "event": "table_file_creation", "file_number": 36569, "file_size": 80348232, "table_properties": {"data_size": 67109345, "index_size": 689196, "filter_size": 12548701, "raw_key_size": 143207009, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957165, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957165", "kMergeOperands": "0"}} >2018-07-11 07:33:01.639413 7f354e0ae700 1 bluefs _allocate failed to allocate 0x500000 on bdev 1, free 0x2fe000; fallback to bdev 2 >2018-07-11 07:33:01.784157 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 58] Generated table #36570: 313476 keys, 5086251 bytes >2018-07-11 07:33:01.784185 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294381784172, "cf_name": "default", "job": 58, "event": "table_file_creation", "file_number": 36570, "file_size": 5086251, "table_properties": {"data_size": 4248564, "index_size": 43146, "filter_size": 793555, "raw_key_size": 9055928, "raw_average_key_size": 28, "raw_value_size": 4680, "raw_average_value_size": 0, "num_data_blocks": 1043, "num_entries": 313476, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "313461", "kMergeOperands": "0"}} >2018-07-11 07:33:05.841951 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4800000 on bdev 1, free 0x2fe000; fallback to bdev 2 >2018-07-11 07:33:06.895899 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 58] Generated table #36571: 2638241 keys, 74546130 bytes >2018-07-11 07:33:06.895932 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294386895917, "cf_name": "default", "job": 58, "event": "table_file_creation", "file_number": 36571, "file_size": 74546130, "table_properties": {"data_size": 67112145, "index_size": 689812, "filter_size": 6743183, "raw_key_size": 76215842, "raw_average_key_size": 28, "raw_value_size": 31365745, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:33:10.925308 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4800000 on bdev 1, free 0x2fe000; fallback to bdev 2 >2018-07-11 07:33:12.073926 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 58] Generated table #36572: 2638241 keys, 74546299 bytes >2018-07-11 07:33:12.073975 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294392073944, "cf_name": "default", "job": 58, "event": "table_file_creation", "file_number": 36572, "file_size": 74546299, "table_properties": {"data_size": 67112148, "index_size": 689978, "filter_size": 6743183, "raw_key_size": 76215855, "raw_average_key_size": 28, "raw_value_size": 31365758, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:33:16.263421 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4800000 on bdev 1, free 0x2fe000; fallback to bdev 2 >2018-07-11 07:33:17.405380 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 58] Generated table #36573: 2638241 keys, 74545787 bytes >2018-07-11 07:33:17.405423 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294397405396, "cf_name": "default", "job": 58, "event": "table_file_creation", "file_number": 36573, "file_size": 74545787, "table_properties": {"data_size": 67112158, "index_size": 689456, "filter_size": 6743183, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:33:21.414677 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4800000 on bdev 1, free 0x2fe000; fallback to bdev 2 >2018-07-11 07:33:22.664272 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 58] Generated table #36574: 2638241 keys, 74546956 bytes >2018-07-11 07:33:22.664316 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294402664288, "cf_name": "default", "job": 58, "event": "table_file_creation", "file_number": 36574, "file_size": 74546956, "table_properties": {"data_size": 67112114, "index_size": 690669, "filter_size": 6743183, "raw_key_size": 76215855, "raw_average_key_size": 28, "raw_value_size": 31365758, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:33:26.701185 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4800000 on bdev 1, free 0x2fe000; fallback to bdev 2 >2018-07-11 07:33:27.735083 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 58] Generated table #36575: 2638241 keys, 74546539 bytes >2018-07-11 07:33:27.735114 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294407735100, "cf_name": "default", "job": 58, "event": "table_file_creation", "file_number": 36575, "file_size": 74546539, "table_properties": {"data_size": 67112151, "index_size": 690215, "filter_size": 6743183, "raw_key_size": 76215852, "raw_average_key_size": 28, "raw_value_size": 31365755, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:33:30.183700 7f354e0ae700 1 bluefs _allocate failed to allocate 0x3e00000 on bdev 1, free 0x2fe000; fallback to bdev 2 >2018-07-11 07:33:31.156743 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 58] Generated table #36576: 577817 keys, 64897190 bytes >2018-07-11 07:33:31.156784 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294411156758, "cf_name": "default", "job": 58, "event": "table_file_creation", "file_number": 36576, "file_size": 64897190, "table_properties": {"data_size": 62792336, "index_size": 516483, "filter_size": 1587383, "raw_key_size": 26371968, "raw_average_key_size": 45, "raw_value_size": 44176471, "raw_average_value_size": 76, "num_data_blocks": 14277, "num_entries": 577817, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "81127", "kMergeOperands": "0"}} >2018-07-11 07:33:31.990853 7f354e0ae700 1 bluefs _allocate failed to allocate 0x1a00000 on bdev 1, free 0x2fe000; fallback to bdev 2 >2018-07-11 07:33:32.482082 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 58] Generated table #36577: 28847 keys, 26725247 bytes >2018-07-11 07:33:32.482111 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294412482097, "cf_name": "default", "job": 58, "event": "table_file_creation", "file_number": 36577, "file_size": 26725247, "table_properties": {"data_size": 26342419, "index_size": 249430, "filter_size": 132412, "raw_key_size": 2444130, "raw_average_key_size": 84, "raw_value_size": 24129462, "raw_average_value_size": 836, "num_data_blocks": 6161, "num_entries": 28847, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:33:32.482180 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 58] Compacted 1@0 + 10@1 files to L1 => 1226900528 bytes >2018-07-11 07:33:32.496606 7f354e0ae700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 65088119716 >2018-07-11 07:33:32.496734 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:33:32.496572) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 20 81 446 3641 796 0] max score 7.54, MB/sec: 13.0 rd, 13.0 wr, level 1, files in(1, 10) out(18) MB in(510.1, 660.0) out(1170.1), read-write-amplify(4.6) write-amplify(2.3) OK, records in: 60785930, records dropped: 0 > >2018-07-11 07:33:32.496743 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:33:32.496598) EVENT_LOG_v1 {"time_micros": 1531294412496586, "job": 58, "event": "compaction_finished", "compaction_time_micros": 94115358, "output_level": 1, "num_output_files": 18, "total_output_size": 1226900528, "num_input_records": 60785930, "num_output_records": 60785930, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 20, 81, 446, 3641, 796, 0]} >2018-07-11 07:33:32.497750 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 59] Compacting 1@1 + 7@2 files to L2, score 7.54 >2018-07-11 07:33:32.497763 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 58 Base level 1, inputs: [36565(76MB)], [36551(76MB) 36557(76MB) 36558(76MB) 36559(76MB) 36523(76MB) 36553(76MB) 36554(76MB)] > >2018-07-11 07:33:32.497791 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294412497780, "job": 59, "event": "compaction_started", "files_L1": [36565], "files_L2": [36551, 36557, 36558, 36559, 36523, 36553, 36554], "score": 7.5435, "input_data_size": 642785942} >2018-07-11 07:33:39.674944 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 59] Generated table #36578: 4957165 keys, 80348270 bytes >2018-07-11 07:33:39.675001 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294419674984, "cf_name": "default", "job": 59, "event": "table_file_creation", "file_number": 36578, "file_size": 80348270, "table_properties": {"data_size": 67109406, "index_size": 689173, "filter_size": 12548701, "raw_key_size": 143207000, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957165, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957165", "kMergeOperands": "0"}} >2018-07-11 07:33:47.088212 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 59] Generated table #36579: 4957165 keys, 80348241 bytes >2018-07-11 07:33:47.088270 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294427088236, "cf_name": "default", "job": 59, "event": "table_file_creation", "file_number": 36579, "file_size": 80348241, "table_properties": {"data_size": 67109368, "index_size": 689182, "filter_size": 12548701, "raw_key_size": 143206990, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957165, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957165", "kMergeOperands": "0"}} >2018-07-11 07:33:47.171404 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294427171400, "job": 0, "event": "table_file_deletion", "file_number": 36502} >2018-07-11 07:33:47.171420 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294427171418, "job": 0, "event": "table_file_deletion", "file_number": 36501} >2018-07-11 07:33:47.171433 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294427171431, "job": 0, "event": "table_file_deletion", "file_number": 36500} >2018-07-11 07:33:47.171449 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294427171445, "job": 0, "event": "table_file_deletion", "file_number": 36499} >2018-07-11 07:33:47.171461 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294427171459, "job": 0, "event": "table_file_deletion", "file_number": 36498} >2018-07-11 07:33:47.171474 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294427171472, "job": 0, "event": "table_file_deletion", "file_number": 36497} >2018-07-11 07:33:47.171487 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294427171484, "job": 0, "event": "table_file_deletion", "file_number": 36496} >2018-07-11 07:33:47.171499 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294427171497, "job": 0, "event": "table_file_deletion", "file_number": 36495} >2018-07-11 07:33:47.171511 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294427171509, "job": 0, "event": "table_file_deletion", "file_number": 36493} >2018-07-11 07:33:47.171540 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294427171535, "job": 0, "event": "table_file_deletion", "file_number": 36487} >2018-07-11 07:33:47.171555 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294427171553, "job": 0, "event": "table_file_deletion", "file_number": 36476} >2018-07-11 07:33:54.556153 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 59] Generated table #36580: 4957165 keys, 80348269 bytes >2018-07-11 07:33:54.556200 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294434556174, "cf_name": "default", "job": 59, "event": "table_file_creation", "file_number": 36580, "file_size": 80348269, "table_properties": {"data_size": 67109383, "index_size": 689195, "filter_size": 12548701, "raw_key_size": 143206995, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957165, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957165", "kMergeOperands": "0"}} >2018-07-11 07:34:01.790529 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 59] Generated table #36581: 4957165 keys, 80348254 bytes >2018-07-11 07:34:01.790585 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294441790568, "cf_name": "default", "job": 59, "event": "table_file_creation", "file_number": 36581, "file_size": 80348254, "table_properties": {"data_size": 67109355, "index_size": 689208, "filter_size": 12548701, "raw_key_size": 143206994, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957165, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957165", "kMergeOperands": "0"}} >2018-07-11 07:34:08.500337 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 59] Generated table #36582: 4957165 keys, 80348256 bytes >2018-07-11 07:34:08.500374 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294448500358, "cf_name": "default", "job": 59, "event": "table_file_creation", "file_number": 36582, "file_size": 80348256, "table_properties": {"data_size": 67109362, "index_size": 689203, "filter_size": 12548701, "raw_key_size": 143206993, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957165, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957165", "kMergeOperands": "0"}} >2018-07-11 07:34:15.521885 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 59] Generated table #36583: 4957169 keys, 80348275 bytes >2018-07-11 07:34:15.521941 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294455521912, "cf_name": "default", "job": 59, "event": "table_file_creation", "file_number": 36583, "file_size": 80348275, "table_properties": {"data_size": 67109395, "index_size": 689177, "filter_size": 12548713, "raw_key_size": 143207113, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957169, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957169", "kMergeOperands": "0"}} >2018-07-11 07:34:16.424179 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 59] Generated table #36584: 717170 keys, 11624640 bytes >2018-07-11 07:34:16.424221 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294456424194, "cf_name": "default", "job": 59, "event": "table_file_creation", "file_number": 36584, "file_size": 11624640, "table_properties": {"data_size": 9708922, "index_size": 99269, "filter_size": 1815464, "raw_key_size": 20718243, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 2383, "num_entries": 717170, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "717170", "kMergeOperands": "0"}} >2018-07-11 07:34:23.718211 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 59] Generated table #36585: 4957167 keys, 80348326 bytes >2018-07-11 07:34:23.718273 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294463718238, "cf_name": "default", "job": 59, "event": "table_file_creation", "file_number": 36585, "file_size": 80348326, "table_properties": {"data_size": 67109413, "index_size": 689216, "filter_size": 12548707, "raw_key_size": 143207044, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957167, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957167", "kMergeOperands": "0"}} >2018-07-11 07:34:29.386818 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 59] Generated table #36586: 4239989 keys, 68723933 bytes >2018-07-11 07:34:29.386866 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294469386838, "cf_name": "default", "job": 59, "event": "table_file_creation", "file_number": 36586, "file_size": 68723933, "table_properties": {"data_size": 57400324, "index_size": 589406, "filter_size": 10733214, "raw_key_size": 122488576, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 14087, "num_entries": 4239989, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4239989", "kMergeOperands": "0"}} >2018-07-11 07:34:29.386902 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 59] Compacted 1@1 + 7@2 files to L2 => 642786464 bytes >2018-07-11 07:34:29.403628 7f354e0ae700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 64993007208 >2018-07-11 07:34:29.403754 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:34:29.403592) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 19 83 446 3641 796 0] max score 6.95, MB/sec: 11.3 rd, 11.3 wr, level 2, files in(1, 7) out(9) MB in(76.6, 536.4) out(613.0), read-write-amplify(16.0) write-amplify(8.0) OK, records in: 39657320, records dropped: 0 > >2018-07-11 07:34:29.403764 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:34:29.403619) EVENT_LOG_v1 {"time_micros": 1531294469403607, "job": 59, "event": "compaction_finished", "compaction_time_micros": 56889094, "output_level": 2, "num_output_files": 9, "total_output_size": 642786464, "num_input_records": 39657320, "num_output_records": 39657320, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 19, 83, 446, 3641, 796, 0]} >2018-07-11 07:34:29.404980 7f354e0ae700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS ------- >2018-07-11 07:34:29.404987 7f354e0ae700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] >** DB Stats ** >Uptime(secs): 6023.6 total, 601.8 interval >Cumulative writes: 896 writes, 226M keys, 896 commit groups, 1.0 writes per commit group, ingest: 4.91 GB, 0.84 MB/s >Cumulative WAL: 896 writes, 252 syncs, 3.54 writes per sync, written: 4.91 GB, 0.84 MB/s >Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent >Interval writes: 2 writes, 30M keys, 2 commit groups, 0.7 writes per commit group, ingest: 654.86 MB, 1.09 MB/s >Interval WAL: 2 writes, 1 syncs, 1.00 writes per sync, written: 0.64 MB, 1.09 MB/s >Interval stall: 00:00:0.000 H:M:S, 0.0 percent > >** Compaction Stats [default] ** >Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop >---------------------------------------------------------------------------------------------------------------------------------------------------------- > L0 0/0 0.00 KB 0.0 0.0 0.0 0.0 3.6 3.6 0.0 1.0 0.0 20.5 178 9 19.831 0 0 > L1 19/0 1.11 GB 6.9 7.2 3.6 3.6 7.2 3.6 0.0 2.0 14.9 14.9 491 6 81.821 374M 149 > L2 83/0 5.13 GB 3.0 8.2 2.7 5.6 8.2 2.7 0.0 3.1 11.8 11.8 714 33 21.649 497M 55 > L3 446/0 24.93 GB 1.0 0.1 0.1 0.1 0.1 0.1 0.1 2.0 18.6 18.6 8 1 7.636 5276K 0 > L4 3641/0 250.00 GB 1.0 0.8 0.1 0.6 0.8 0.1 0.1 5.5 21.8 21.8 36 2 17.944 29M 0 > L5 796/0 55.24 GB 0.0 0.0 0.0 0.0 0.0 0.0 0.3 0.0 0.0 0.0 0 0 0.000 0 0 > Sum 4985/0 336.41 GB 0.0 16.3 6.4 9.8 19.9 10.0 0.5 5.5 11.7 14.3 1427 51 27.987 906M 204 > Int 0/0 0.00 KB 0.0 6.9 2.6 4.3 7.4 3.1 0.0 14.9 11.3 12.1 627 17 36.890 413M 1 >Uptime(secs): 6023.6 total, 6023.6 interval >Flush(GB): cumulative 3.581, interval 0.498 >AddFile(GB): cumulative 0.000, interval 0.000 >AddFile(Total Files): cumulative 0, interval 0 >AddFile(L0 Files): cumulative 0, interval 0 >AddFile(Keys): cumulative 0, interval 0 >Cumulative compaction: 19.87 GB write, 3.38 MB/s write, 16.29 GB read, 2.77 MB/s read, 1427.4 seconds >Interval compaction: 7.44 GB write, 1.26 MB/s write, 6.94 GB read, 1.18 MB/s read, 627.1 seconds >Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count > >** File Read Latency Histogram By Level [default] ** > >** Compaction Stats [default] ** >Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop >---------------------------------------------------------------------------------------------------------------------------------------------------------- > L0 0/0 0.00 KB 0.0 0.0 0.0 0.0 3.6 3.6 0.0 1.0 0.0 20.5 178 9 19.831 0 0 > L1 19/0 1.11 GB 6.9 7.2 3.6 3.6 7.2 3.6 0.0 2.0 14.9 14.9 491 6 81.821 374M 149 > L2 83/0 5.13 GB 3.0 8.2 2.7 5.6 8.2 2.7 0.0 3.1 11.8 11.8 714 33 21.649 497M 55 > L3 446/0 24.93 GB 1.0 0.1 0.1 0.1 0.1 0.1 0.1 2.0 18.6 18.6 8 1 7.636 5276K 0 > L4 3641/0 250.00 GB 1.0 0.8 0.1 0.6 0.8 0.1 0.1 5.5 21.8 21.8 36 2 17.944 29M 0 > L5 796/0 55.24 GB 0.0 0.0 0.0 0.0 0.0 0.0 0.3 0.0 0.0 0.0 0 0 0.000 0 0 > Sum 4985/0 336.41 GB 0.0 16.3 6.4 9.8 19.9 10.0 0.5 5.5 11.7 14.3 1427 51 27.987 906M 204 > Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 >Uptime(secs): 6023.6 total, 6023.6 interval >Flush(GB): cumulative 3.581, interval 0.000 >AddFile(GB): cumulative 0.000, interval 0.000 >AddFile(Total Files): cumulative 0, interval 0 >AddFile(L0 Files): cumulative 0, interval 0 >AddFile(Keys): cumulative 0, interval 0 >Cumulative compaction: 19.87 GB write, 3.38 MB/s write, 16.29 GB read, 2.77 MB/s read, 1427.4 seconds >Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds >Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count > >** File Read Latency Histogram By Level [default] ** > >2018-07-11 07:34:29.405115 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 60] Compacting 1@1 + 5@2 files to L2, score 6.95 >2018-07-11 07:34:29.405128 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 59 Base level 1, inputs: [36561(76MB)], [36504(76MB) 36505(76MB) 36506(33MB) 36507(14MB) 36546(62MB)] > >2018-07-11 07:34:29.405143 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294469405131, "job": 60, "event": "compaction_started", "files_L1": [36561], "files_L2": [36504, 36505, 36506, 36507, 36546], "score": 6.94871, "input_data_size": 357728903} >2018-07-11 07:34:32.884718 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294472884700, "job": 0, "event": "table_file_deletion", "file_number": 36565} >2018-07-11 07:34:32.884738 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294472884736, "job": 0, "event": "table_file_deletion", "file_number": 36559} >2018-07-11 07:34:32.884751 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294472884749, "job": 0, "event": "table_file_deletion", "file_number": 36558} >2018-07-11 07:34:32.884767 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294472884765, "job": 0, "event": "table_file_deletion", "file_number": 36557} >2018-07-11 07:34:32.884786 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294472884784, "job": 0, "event": "table_file_deletion", "file_number": 36554} >2018-07-11 07:34:32.884802 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294472884800, "job": 0, "event": "table_file_deletion", "file_number": 36553} >2018-07-11 07:34:32.884815 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294472884813, "job": 0, "event": "table_file_deletion", "file_number": 36551} >2018-07-11 07:34:32.884830 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294472884828, "job": 0, "event": "table_file_deletion", "file_number": 36523} >2018-07-11 07:34:36.827062 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 60] Generated table #36587: 4957166 keys, 80348282 bytes >2018-07-11 07:34:36.827117 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294476827089, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 36587, "file_size": 80348282, "table_properties": {"data_size": 67109360, "index_size": 689228, "filter_size": 12548704, "raw_key_size": 143207024, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957166, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957166", "kMergeOperands": "0"}} >2018-07-11 07:34:43.781367 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 60] Generated table #36588: 4957160 keys, 80348286 bytes >2018-07-11 07:34:43.781405 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294483781389, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 36588, "file_size": 80348286, "table_properties": {"data_size": 67109393, "index_size": 689214, "filter_size": 12548689, "raw_key_size": 143206847, "raw_average_key_size": 28, "raw_value_size": 60, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957160, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957155", "kMergeOperands": "0"}} >2018-07-11 07:34:44.144995 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 60] Generated table #36589: 198890 keys, 3224173 bytes >2018-07-11 07:34:44.145034 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294484145009, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 36589, "file_size": 3224173, "table_properties": {"data_size": 2692554, "index_size": 27157, "filter_size": 503477, "raw_key_size": 5745720, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 661, "num_entries": 198890, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "198890", "kMergeOperands": "0"}} >2018-07-11 07:34:51.414610 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 60] Generated table #36590: 4957168 keys, 80348311 bytes >2018-07-11 07:34:51.414664 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294491414636, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 36590, "file_size": 80348311, "table_properties": {"data_size": 67109355, "index_size": 689256, "filter_size": 12548710, "raw_key_size": 143207073, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957168, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957168", "kMergeOperands": "0"}} >2018-07-11 07:34:53.333537 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 60] Generated table #36591: 1042836 keys, 16903262 bytes >2018-07-11 07:34:53.333568 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294493333554, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 36591, "file_size": 16903262, "table_properties": {"data_size": 14117846, "index_size": 144565, "filter_size": 2639864, "raw_key_size": 30126382, "raw_average_key_size": 28, "raw_value_size": 47, "raw_average_value_size": 0, "num_data_blocks": 3465, "num_entries": 1042836, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1042832", "kMergeOperands": "0"}} >2018-07-11 07:35:00.643184 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 60] Generated table #36592: 4957167 keys, 80348283 bytes >2018-07-11 07:35:00.643234 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294500643218, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 36592, "file_size": 80348283, "table_properties": {"data_size": 67109331, "index_size": 689255, "filter_size": 12548707, "raw_key_size": 143207039, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957167, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957167", "kMergeOperands": "0"}} >2018-07-11 07:35:02.180224 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 60] Generated table #36593: 999999 keys, 16208887 bytes >2018-07-11 07:35:02.180273 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294502180258, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 36593, "file_size": 16208887, "table_properties": {"data_size": 13537846, "index_size": 138627, "filter_size": 2531427, "raw_key_size": 28888866, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 3323, "num_entries": 999999, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "999999", "kMergeOperands": "0"}} >2018-07-11 07:35:02.180304 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 60] Compacted 1@1 + 5@2 files to L2 => 357729484 bytes >2018-07-11 07:35:02.194508 7f354e0ae700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 64884155848 >2018-07-11 07:35:02.194639 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:35:02.194473) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 18 85 446 3641 796 0] max score 6.35, MB/sec: 10.9 rd, 10.9 wr, level 2, files in(1, 5) out(7) MB in(76.6, 264.5) out(341.2), read-write-amplify(8.9) write-amplify(4.5) OK, records in: 22070386, records dropped: 0 > >2018-07-11 07:35:02.194651 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:35:02.194499) EVENT_LOG_v1 {"time_micros": 1531294502194488, "job": 60, "event": "compaction_finished", "compaction_time_micros": 32775144, "output_level": 2, "num_output_files": 7, "total_output_size": 357729484, "num_input_records": 22070386, "num_output_records": 22070386, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 18, 85, 446, 3641, 796, 0]} >2018-07-11 07:35:02.195681 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 61] Compacting 1@1 + 9@2 files to L2, score 6.35 >2018-07-11 07:35:02.195694 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 60 Base level 1, inputs: [36562(76MB)], [36592(76MB) 36593(15MB) 36547(76MB) 36548(69MB) 36549(56MB) 36531(53MB) 36532(70MB) 36542(76MB) 36543(76MB)] > >2018-07-11 07:35:02.195720 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294502195697, "job": 61, "event": "compaction_started", "files_L1": [36562], "files_L2": [36592, 36593, 36547, 36548, 36549, 36531, 36532, 36542, 36543], "score": 6.35392, "input_data_size": 679330645} >2018-07-11 07:35:09.845550 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 61] Generated table #36594: 4957167 keys, 80348309 bytes >2018-07-11 07:35:09.845602 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294509845573, "cf_name": "default", "job": 61, "event": "table_file_creation", "file_number": 36594, "file_size": 80348309, "table_properties": {"data_size": 67109393, "index_size": 689219, "filter_size": 12548707, "raw_key_size": 143207044, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957167, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957167", "kMergeOperands": "0"}} >2018-07-11 07:35:11.792322 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 61] Generated table #36595: 1042833 keys, 16903140 bytes >2018-07-11 07:35:11.792352 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294511792337, "cf_name": "default", "job": 61, "event": "table_file_creation", "file_number": 36595, "file_size": 16903140, "table_properties": {"data_size": 14117724, "index_size": 144573, "filter_size": 2639856, "raw_key_size": 30126296, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 3465, "num_entries": 1042833, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1042833", "kMergeOperands": "0"}} >2018-07-11 07:35:18.408386 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 61] Generated table #36596: 4957167 keys, 80348310 bytes >2018-07-11 07:35:18.408431 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294518408405, "cf_name": "default", "job": 61, "event": "table_file_creation", "file_number": 36596, "file_size": 80348310, "table_properties": {"data_size": 67109394, "index_size": 689219, "filter_size": 12548707, "raw_key_size": 143207044, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957167, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957167", "kMergeOperands": "0"}} >2018-07-11 07:35:19.280265 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294519280258, "job": 0, "event": "table_file_deletion", "file_number": 36561} >2018-07-11 07:35:19.280302 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294519280298, "job": 0, "event": "table_file_deletion", "file_number": 36546} >2018-07-11 07:35:19.280321 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294519280318, "job": 0, "event": "table_file_deletion", "file_number": 36507} >2018-07-11 07:35:19.280340 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294519280336, "job": 0, "event": "table_file_deletion", "file_number": 36506} >2018-07-11 07:35:19.280365 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294519280361, "job": 0, "event": "table_file_deletion", "file_number": 36505} >2018-07-11 07:35:19.280386 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294519280383, "job": 0, "event": "table_file_deletion", "file_number": 36504} >2018-07-11 07:35:23.353845 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 61] Generated table #36597: 2680208 keys, 74793582 bytes >2018-07-11 07:35:23.353889 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294523353863, "cf_name": "default", "job": 61, "event": "table_file_creation", "file_number": 36597, "file_size": 74793582, "table_properties": {"data_size": 67111025, "index_size": 829324, "filter_size": 6852241, "raw_key_size": 78331641, "raw_average_key_size": 29, "raw_value_size": 30499285, "raw_average_value_size": 11, "num_data_blocks": 16711, "num_entries": 2680208, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "2042833", "kMergeOperands": "0"}} >2018-07-11 07:35:28.449107 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 61] Generated table #36598: 2298857 keys, 73864103 bytes >2018-07-11 07:35:28.449150 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294528449124, "cf_name": "default", "job": 61, "event": "table_file_creation", "file_number": 36598, "file_size": 73864103, "table_properties": {"data_size": 67109070, "index_size": 855721, "filter_size": 5898320, "raw_key_size": 67525849, "raw_average_key_size": 29, "raw_value_size": 35594171, "raw_average_value_size": 15, "num_data_blocks": 16748, "num_entries": 2298857, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1595309", "kMergeOperands": "0"}} >2018-07-11 07:35:30.460732 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 61] Generated table #36599: 1404691 keys, 22768272 bytes >2018-07-11 07:35:30.460762 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294530460748, "cf_name": "default", "job": 61, "event": "table_file_creation", "file_number": 36599, "file_size": 22768272, "table_properties": {"data_size": 19016491, "index_size": 194922, "filter_size": 3555872, "raw_key_size": 40579972, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 4667, "num_entries": 1404691, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1404691", "kMergeOperands": "0"}} >2018-07-11 07:35:36.916162 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 61] Generated table #36600: 4680131 keys, 79657723 bytes >2018-07-11 07:35:36.916216 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294536916187, "cf_name": "default", "job": 61, "event": "table_file_creation", "file_number": 36600, "file_size": 79657723, "table_properties": {"data_size": 67112207, "index_size": 689382, "filter_size": 11855141, "raw_key_size": 135203775, "raw_average_key_size": 28, "raw_value_size": 3749621, "raw_average_value_size": 0, "num_data_blocks": 16473, "num_entries": 4680131, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4364742", "kMergeOperands": "0"}} >2018-07-11 07:35:37.775896 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 61] Generated table #36601: 635258 keys, 10297011 bytes >2018-07-11 07:35:37.775939 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294537775911, "cf_name": "default", "job": 61, "event": "table_file_creation", "file_number": 36601, "file_size": 10297011, "table_properties": {"data_size": 8600037, "index_size": 87877, "filter_size": 1608112, "raw_key_size": 18351909, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 2111, "num_entries": 635258, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "635258", "kMergeOperands": "0"}} >2018-07-11 07:35:44.787691 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 61] Generated table #36602: 4957168 keys, 80348314 bytes >2018-07-11 07:35:44.787726 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294544787711, "cf_name": "default", "job": 61, "event": "table_file_creation", "file_number": 36602, "file_size": 80348314, "table_properties": {"data_size": 67109372, "index_size": 689242, "filter_size": 12548710, "raw_key_size": 143207073, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957168, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957168", "kMergeOperands": "0"}} >2018-07-11 07:35:48.258555 7f354e0ae700 -1 bluefs _allocate failed to allocate 0x3000000 on bdev 2, free 0x1700000 >2018-07-11 07:35:48.258569 7f354e0ae700 0 stupidalloc 0x0x55a30f764070 dump free bin 0: 0 extents >2018-07-11 07:35:48.258574 7f354e0ae700 0 stupidalloc 0x0x55a30f764070 dump free bin 1: 0 extents >2018-07-11 07:35:48.258576 7f354e0ae700 0 stupidalloc 0x0x55a30f764070 dump free bin 2: 0 extents >2018-07-11 07:35:48.258577 7f354e0ae700 0 stupidalloc 0x0x55a30f764070 dump free bin 3: 0 extents >2018-07-11 07:35:48.258579 7f354e0ae700 0 stupidalloc 0x0x55a30f764070 dump free bin 4: 0 extents >2018-07-11 07:35:48.258580 7f354e0ae700 0 stupidalloc 0x0x55a30f764070 dump free bin 5: 0 extents >2018-07-11 07:35:48.258582 7f354e0ae700 0 stupidalloc 0x0x55a30f764070 dump free bin 6: 0 extents >2018-07-11 07:35:48.258584 7f354e0ae700 0 stupidalloc 0x0x55a30f764070 dump free bin 7: 0 extents >2018-07-11 07:35:48.258585 7f354e0ae700 0 stupidalloc 0x0x55a30f764070 dump free bin 8: 0 extents >2018-07-11 07:35:48.258587 7f354e0ae700 0 stupidalloc 0x0x55a30f764070 dump free bin 9: 14 extents >2018-07-11 07:35:48.258588 7f354e0ae700 0 stupidalloc 0x0x55a30f764070 dump 0x26fcc00000~100000 >2018-07-11 07:35:48.258590 7f354e0ae700 0 stupidalloc 0x0x55a30f764070 dump 0x26ff300000~100000 >2018-07-11 07:35:48.258591 7f354e0ae700 0 stupidalloc 0x0x55a30f764070 dump 0x33ed800000~100000 >2018-07-11 07:35:48.258592 7f354e0ae700 0 stupidalloc 0x0x55a30f764070 dump 0x7649900000~100000 >2018-07-11 07:35:48.258594 7f354e0ae700 0 stupidalloc 0x0x55a30f764070 dump 0x9c2d700000~100000 >2018-07-11 07:35:48.258595 7f354e0ae700 0 stupidalloc 0x0x55a30f764070 dump 0x9c4d200000~100000 >2018-07-11 07:35:48.258597 7f354e0ae700 0 stupidalloc 0x0x55a30f764070 dump 0x9d69d00000~200000 >2018-07-11 07:35:48.258598 7f354e0ae700 0 stupidalloc 0x0x55a30f764070 dump 0xa1f8e00000~100000 >2018-07-11 07:35:48.258599 7f354e0ae700 0 stupidalloc 0x0x55a30f764070 dump 0xa3b9d00000~100000 >2018-07-11 07:35:48.258601 7f354e0ae700 0 stupidalloc 0x0x55a30f764070 dump 0xc70a100000~500000 >2018-07-11 07:35:48.258602 7f354e0ae700 0 stupidalloc 0x0x55a30f764070 dump 0xc72bb00000~200000 >2018-07-11 07:35:48.258603 7f354e0ae700 0 stupidalloc 0x0x55a30f764070 dump 0xc76ae00000~100000 >2018-07-11 07:35:48.258604 7f354e0ae700 0 stupidalloc 0x0x55a30f764070 dump 0xc781400000~200000 >2018-07-11 07:35:48.258606 7f354e0ae700 0 stupidalloc 0x0x55a30f764070 dump 0xc98f300000~300000 >2018-07-11 07:35:48.258608 7f354e0ae700 -1 bluefs _flush_range allocated: 0x0 offset: 0x0 length: 0x2f0901c >2018-07-11 07:35:48.264086 7f354e0ae700 -1 /builddir/build/BUILD/ceph-12.2.5/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, uint64_t)' thread 7f354e0ae700 time 2018-07-11 07:35:48.258621 >/builddir/build/BUILD/ceph-12.2.5/src/os/bluestore/BlueFS.cc: 1525: FAILED assert(0 == "bluefs enospc") > > ceph version 12.2.5-27.el7cp (6b7d1a8cf2e2376fd0a12424947c8c97aa1c0890) luminous (stable) > 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x110) [0x55a304abd710] > 2: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1b1b) [0x55a304a3c13b] > 3: (BlueFS::_flush(BlueFS::FileWriter*, bool)+0x188) [0x55a304a3c6a8] > 4: (BlueRocksWritableFile::Flush()+0x3c) [0x55a304a5823c] > 5: (rocksdb::WritableFileWriter::Flush()+0xaf) [0x55a304e2755f] > 6: (rocksdb::WritableFileWriter::Sync(bool)+0x33) [0x55a304e27773] > 7: (rocksdb::CompactionJob::FinishCompactionOutputFile(rocksdb::Status const&, rocksdb::CompactionJob::SubcompactionState*, rocksdb::RangeDelAggregator*, CompactionIterationStats*, rocksdb::Slice const*)+0xa1b) [0x55a304eed7ab] > 8: (rocksdb::CompactionJob::ProcessKeyValueCompaction(rocksdb::CompactionJob::SubcompactionState*)+0xea2) [0x55a304ef0222] > 9: (rocksdb::CompactionJob::Run()+0x1de) [0x55a304ef0cde] > 10: (rocksdb::DBImpl::BackgroundCompaction(bool*, rocksdb::JobContext*, rocksdb::LogBuffer*, void*)+0x9c7) [0x55a304e7c687] > 11: (rocksdb::DBImpl::BackgroundCallCompaction(void*)+0x283) [0x55a304e7fc73] > 12: (rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long)+0x271) [0x55a304ee6e31] > 13: (rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*)+0x3d) [0x55a304ee701d] > 14: (()+0xb5070) [0x7f355fafc070] > 15: (()+0x7dd5) [0x7f356016fdd5] > 16: (clone()+0x6d) [0x7f355f260b3d] > NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. > >--- begin dump of recent events --- >-10000> 2018-07-11 07:32:19.451799 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:19.486263) v4 -- 0x55a314575000 con 0 > -9999> 2018-07-11 07:32:19.451798 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.2 10.8.128.106:0/12651 1816 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:19.486263) v4 ==== 2004+0+0 (3087042803 0 0) 0x55a324474c00 con 0x55a32169f000 > -9998> 2018-07-11 07:32:19.451821 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:19.486263) v4 -- 0x55a34e290000 con 0 > -9997> 2018-07-11 07:32:19.515025 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15255 conn(0x55a311b65800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1836 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:32:20.727133) v4 > -9996> 2018-07-11 07:32:19.515033 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15255 conn(0x55a311b64000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1836 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:32:20.727133) v4 > -9995> 2018-07-11 07:32:19.515049 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.0 10.8.128.113:0/15255 1836 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:20.727133) v4 ==== 2004+0+0 (3987030473 0 0) 0x55a34de6f000 con 0x55a311b64000 > -9994> 2018-07-11 07:32:19.515048 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.0 10.8.128.113:0/15255 1836 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:20.727133) v4 ==== 2004+0+0 (3987030473 0 0) 0x55a332e79a00 con 0x55a311b65800 > -9993> 2018-07-11 07:32:19.515058 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:20.727133) v4 -- 0x55a3131bca00 con 0 > -9992> 2018-07-11 07:32:19.515084 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:20.727133) v4 -- 0x55a33047ec00 con 0 > -9991> 2018-07-11 07:32:20.046169 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a3301eefc0 con 0 > -9990> 2018-07-11 07:32:20.059414 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5003 0x55a31e8de500 replica scrub(pg: 6.15,from:0'0,to:430'106147,epoch:494/492,start:6:a971c1f6:::benchmark_data_magna108_35617_object373220:0,end:6:a972bad3:::benchmark_data_magna108_41525_object53024:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -9989> 2018-07-11 07:32:20.059443 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5003 ==== replica scrub(pg: 6.15,from:0'0,to:430'106147,epoch:494/492,start:6:a971c1f6:::benchmark_data_magna108_35617_object373220:0,end:6:a972bad3:::benchmark_data_magna108_41525_object53024:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 217+0+0 (343175637 0 0) 0x55a31e8de500 con 0x55a317019000 > -9988> 2018-07-11 07:32:20.451854 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15885 conn(0x55a3170b5800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1828 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:32:21.075021) v4 > -9987> 2018-07-11 07:32:20.451877 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.7 10.8.128.114:0/15885 1828 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:21.075021) v4 ==== 2004+0+0 (123627377 0 0) 0x55a324474c00 con 0x55a3170b5800 > -9986> 2018-07-11 07:32:20.451887 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:21.075021) v4 -- 0x55a33047ec00 con 0 > -9985> 2018-07-11 07:32:20.451902 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15885 conn(0x55a311b76800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1828 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:32:21.075021) v4 > -9984> 2018-07-11 07:32:20.451924 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.7 10.8.128.114:0/15885 1828 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:21.075021) v4 ==== 2004+0+0 (123627377 0 0) 0x55a34de78600 con 0x55a311b76800 > -9983> 2018-07-11 07:32:20.451941 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:21.075021) v4 -- 0x55a32ccf6800 con 0 > -9982> 2018-07-11 07:32:20.645821 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15919 conn(0x55a3170b4000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1819 0x55a35772aa00 osd_ping(ping e494 stamp 2018-07-11 07:32:21.269016) v4 > -9981> 2018-07-11 07:32:20.645826 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15919 conn(0x55a317074800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1819 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:32:21.269016) v4 > -9980> 2018-07-11 07:32:20.645852 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.11 10.8.128.114:0/15919 1819 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:21.269016) v4 ==== 2004+0+0 (761791041 0 0) 0x55a332e79a00 con 0x55a317074800 > -9979> 2018-07-11 07:32:20.645850 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.11 10.8.128.114:0/15919 1819 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:21.269016) v4 ==== 2004+0+0 (761791041 0 0) 0x55a35772aa00 con 0x55a3170b4000 > -9978> 2018-07-11 07:32:20.645862 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:21.269016) v4 -- 0x55a324475000 con 0 > -9977> 2018-07-11 07:32:20.645886 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:21.269016) v4 -- 0x55a34e290e00 con 0 > -9976> 2018-07-11 07:32:20.790519 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a3314fcd80 con 0 > -9975> 2018-07-11 07:32:20.814822 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12677 conn(0x55a32169d800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1832 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:32:20.849404) v4 > -9974> 2018-07-11 07:32:20.814844 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.1 10.8.128.106:0/12677 1832 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:20.849404) v4 ==== 2004+0+0 (3879804534 0 0) 0x55a34de6f000 con 0x55a32169d800 > -9973> 2018-07-11 07:32:20.814861 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:20.849404) v4 -- 0x55a34de5bc00 con 0 > -9972> 2018-07-11 07:32:20.814859 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12677 conn(0x55a32169c000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1832 0x55a35772aa00 osd_ping(ping e494 stamp 2018-07-11 07:32:20.849404) v4 > -9971> 2018-07-11 07:32:20.814885 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.1 10.8.128.106:0/12677 1832 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:20.849404) v4 ==== 2004+0+0 (3879804534 0 0) 0x55a35772aa00 con 0x55a32169c000 > -9970> 2018-07-11 07:32:20.814896 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:20.849404) v4 -- 0x55a314574e00 con 0 > -9969> 2018-07-11 07:32:20.817924 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5004 0x55a31e8de780 replica scrub(pg: 6.15,from:0'0,to:430'106558,epoch:494/492,start:6:a972bad3:::benchmark_data_magna108_41525_object53024:0,end:6:a973dd6b:::benchmark_data_magna108_35617_object492734:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -9968> 2018-07-11 07:32:20.817953 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5004 ==== replica scrub(pg: 6.15,from:0'0,to:430'106558,epoch:494/492,start:6:a972bad3:::benchmark_data_magna108_41525_object53024:0,end:6:a973dd6b:::benchmark_data_magna108_35617_object492734:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 217+0+0 (3507728269 0 0) 0x55a31e8de780 con 0x55a317019000 > -9967> 2018-07-11 07:32:20.833871 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b8800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1760 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:32:20.584014) v4 > -9966> 2018-07-11 07:32:20.833892 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.8 10.8.128.108:0/11967 1760 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:20.584014) v4 ==== 2004+0+0 (1205199535 0 0) 0x55a324474c00 con 0x55a3170b8800 > -9965> 2018-07-11 07:32:20.833901 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:20.584014) v4 -- 0x55a324475000 con 0 > -9964> 2018-07-11 07:32:20.833902 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b7000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1760 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:32:20.584014) v4 > -9963> 2018-07-11 07:32:20.833924 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.8 10.8.128.108:0/11967 1760 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:20.584014) v4 ==== 2004+0+0 (1205199535 0 0) 0x55a34de78600 con 0x55a3170b7000 > -9962> 2018-07-11 07:32:20.833942 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:20.584014) v4 -- 0x55a34de5bc00 con 0 > -9961> 2018-07-11 07:32:20.947116 7f3543899700 5 osd.10 494 heartbeat: osd_stat(783 GB used, 148 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,1]) > -9960> 2018-07-11 07:32:20.947130 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6802/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:32:20.947129) v4 -- 0x55a33047f400 con 0 > -9959> 2018-07-11 07:32:20.947167 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6803/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:32:20.947129) v4 -- 0x55a343719800 con 0 > -9958> 2018-07-11 07:32:20.947184 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6806/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:32:20.947129) v4 -- 0x55a34e291e00 con 0 > -9957> 2018-07-11 07:32:20.947199 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6807/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:32:20.947129) v4 -- 0x55a33d673000 con 0 > -9956> 2018-07-11 07:32:20.947212 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6808/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:32:20.947129) v4 -- 0x55a327c0fe00 con 0 > -9955> 2018-07-11 07:32:20.947221 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6810/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:32:20.947129) v4 -- 0x55a33d673800 con 0 > -9954> 2018-07-11 07:32:20.947230 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6809/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:32:20.947129) v4 -- 0x55a33d673200 con 0 > -9953> 2018-07-11 07:32:20.947242 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6811/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:32:20.947129) v4 -- 0x55a33d673400 con 0 > -9952> 2018-07-11 07:32:20.947266 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6810/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:32:20.947129) v4 -- 0x55a33d673a00 con 0 > -9951> 2018-07-11 07:32:20.947277 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6811/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:32:20.947129) v4 -- 0x55a34e291c00 con 0 > -9950> 2018-07-11 07:32:20.947291 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6806/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:32:20.947129) v4 -- 0x55a3252c4000 con 0 > -9949> 2018-07-11 07:32:20.947309 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6807/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:32:20.947129) v4 -- 0x55a3252c4200 con 0 > -9948> 2018-07-11 07:32:20.947326 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6802/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:32:20.947129) v4 -- 0x55a33d672e00 con 0 > -9947> 2018-07-11 07:32:20.947345 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6803/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:32:20.947129) v4 -- 0x55a33d673600 con 0 > -9946> 2018-07-11 07:32:20.947358 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6806/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:32:20.947129) v4 -- 0x55a3252c4800 con 0 > -9945> 2018-07-11 07:32:20.947374 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6807/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:32:20.947129) v4 -- 0x55a33d673c00 con 0 > -9944> 2018-07-11 07:32:20.947388 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6802/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:32:20.947129) v4 -- 0x55a3252c4600 con 0 > -9943> 2018-07-11 07:32:20.947402 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6803/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:32:20.947129) v4 -- 0x55a3252c4c00 con 0 > -9942> 2018-07-11 07:32:20.947416 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6806/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:32:20.947129) v4 -- 0x55a3252c4e00 con 0 > -9941> 2018-07-11 07:32:20.947436 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6807/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:32:20.947129) v4 -- 0x55a3252c5400 con 0 > -9940> 2018-07-11 07:32:20.947456 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6802/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:32:20.947129) v4 -- 0x55a33d673e00 con 0 > -9939> 2018-07-11 07:32:20.947469 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6803/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:32:20.947129) v4 -- 0x55a3252c4a00 con 0 > -9938> 2018-07-11 07:32:20.947560 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6802/15255 conn(0x55a314e1c000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1811 0x55a314574e00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:20.947129) v4 > -9937> 2018-07-11 07:32:20.947577 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6802/15255 1811 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:20.947129) v4 ==== 2004+0+0 (3785654154 0 0) 0x55a314574e00 con 0x55a314e1c000 > -9936> 2018-07-11 07:32:20.947638 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6803/15255 conn(0x55a314e1d800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1811 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:20.947129) v4 > -9935> 2018-07-11 07:32:20.947638 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6808/12651 conn(0x55a314e96000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1811 0x55a314574e00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:20.947129) v4 > -9934> 2018-07-11 07:32:20.947661 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6803/15255 1811 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:20.947129) v4 ==== 2004+0+0 (3785654154 0 0) 0x55a324474c00 con 0x55a314e1d800 > -9933> 2018-07-11 07:32:20.947666 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6808/12651 1811 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:20.947129) v4 ==== 2004+0+0 (3785654154 0 0) 0x55a314574e00 con 0x55a314e96000 > -9932> 2018-07-11 07:32:20.947698 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6809/12650 conn(0x55a314e1f000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1811 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:20.947129) v4 > -9931> 2018-07-11 07:32:20.947712 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6807/12677 conn(0x55a314e85800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1811 0x55a314574e00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:20.947129) v4 > -9930> 2018-07-11 07:32:20.947733 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6809/12650 1811 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:20.947129) v4 ==== 2004+0+0 (3785654154 0 0) 0x55a34de6f000 con 0x55a314e1f000 > -9929> 2018-07-11 07:32:20.947739 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6807/12677 1811 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:20.947129) v4 ==== 2004+0+0 (3785654154 0 0) 0x55a314574e00 con 0x55a314e85800 > -9928> 2018-07-11 07:32:20.947737 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6806/12617 conn(0x55a314e26000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1811 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:20.947129) v4 > -9927> 2018-07-11 07:32:20.947758 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6806/12617 1811 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:20.947129) v4 ==== 2004+0+0 (3785654154 0 0) 0x55a324474c00 con 0x55a314e26000 > -9926> 2018-07-11 07:32:20.947787 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6811/12650 conn(0x55a314e20800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1811 0x55a314574e00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:20.947129) v4 > -9925> 2018-07-11 07:32:20.947773 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6807/12617 conn(0x55a314e27800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1811 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:20.947129) v4 > -9924> 2018-07-11 07:32:20.947803 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6811/12650 1811 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:20.947129) v4 ==== 2004+0+0 (3785654154 0 0) 0x55a314574e00 con 0x55a314e20800 > -9923> 2018-07-11 07:32:20.947810 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6807/12617 1811 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:20.947129) v4 ==== 2004+0+0 (3785654154 0 0) 0x55a34de6f000 con 0x55a314e27800 > -9922> 2018-07-11 07:32:20.947836 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6811/15287 conn(0x55a314e04800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1811 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:20.947129) v4 > -9921> 2018-07-11 07:32:20.947844 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6807/15263 conn(0x55a314e9a800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1811 0x55a314574e00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:20.947129) v4 > -9920> 2018-07-11 07:32:20.947857 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6807/15263 1811 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:20.947129) v4 ==== 2004+0+0 (3785654154 0 0) 0x55a314574e00 con 0x55a314e9a800 > -9919> 2018-07-11 07:32:20.947852 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6806/12677 conn(0x55a314e84000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1811 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:20.947129) v4 > -9918> 2018-07-11 07:32:20.947877 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6806/12677 1811 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:20.947129) v4 ==== 2004+0+0 (3785654154 0 0) 0x55a34de6f000 con 0x55a314e84000 > -9917> 2018-07-11 07:32:20.947857 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6811/15287 1811 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:20.947129) v4 ==== 2004+0+0 (3785654154 0 0) 0x55a324474c00 con 0x55a314e04800 > -9916> 2018-07-11 07:32:20.947907 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6810/15287 conn(0x55a314e03000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1811 0x55a314574e00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:20.947129) v4 > -9915> 2018-07-11 07:32:20.947922 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6806/15263 conn(0x55a314e99000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1811 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:20.947129) v4 > -9914> 2018-07-11 07:32:20.947926 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6810/15287 1811 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:20.947129) v4 ==== 2004+0+0 (3785654154 0 0) 0x55a314574e00 con 0x55a314e03000 > -9913> 2018-07-11 07:32:20.947935 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6806/15263 1811 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:20.947129) v4 ==== 2004+0+0 (3785654154 0 0) 0x55a34de6f000 con 0x55a314e99000 > -9912> 2018-07-11 07:32:20.947934 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6810/12651 conn(0x55a314e97800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1811 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:20.947129) v4 > -9911> 2018-07-11 07:32:20.947948 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6810/12651 1811 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:20.947129) v4 ==== 2004+0+0 (3785654154 0 0) 0x55a324474c00 con 0x55a314e97800 > -9910> 2018-07-11 07:32:20.947967 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6807/15885 conn(0x55a314e01800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1811 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:20.947129) v4 > -9909> 2018-07-11 07:32:20.947979 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6807/15885 1811 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:20.947129) v4 ==== 2004+0+0 (3785654154 0 0) 0x55a34de6f000 con 0x55a314e01800 > -9908> 2018-07-11 07:32:20.947975 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6803/11967 conn(0x55a314e13800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1811 0x55a314574e00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:20.947129) v4 > -9907> 2018-07-11 07:32:20.947994 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6803/11967 1811 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:20.947129) v4 ==== 2004+0+0 (3785654154 0 0) 0x55a314574e00 con 0x55a314e13800 > -9906> 2018-07-11 07:32:20.948000 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6806/15885 conn(0x55a314e00000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1811 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:20.947129) v4 > -9905> 2018-07-11 07:32:20.948014 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6802/11967 conn(0x55a314e12000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1811 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:20.947129) v4 > -9904> 2018-07-11 07:32:20.948015 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6806/15885 1811 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:20.947129) v4 ==== 2004+0+0 (3785654154 0 0) 0x55a324474c00 con 0x55a314e00000 > -9903> 2018-07-11 07:32:20.948031 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6802/11967 1811 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:20.947129) v4 ==== 2004+0+0 (3785654154 0 0) 0x55a34de6f000 con 0x55a314e12000 > -9902> 2018-07-11 07:32:20.948071 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6802/13323 conn(0x55a314e15000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1811 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:20.947129) v4 > -9901> 2018-07-11 07:32:20.948085 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6802/13323 1811 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:20.947129) v4 ==== 2004+0+0 (3785654154 0 0) 0x55a324474c00 con 0x55a314e15000 > -9900> 2018-07-11 07:32:20.948088 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6803/13323 conn(0x55a314e16800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1811 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:20.947129) v4 > -9899> 2018-07-11 07:32:20.948109 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6803/13323 1811 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:20.947129) v4 ==== 2004+0+0 (3785654154 0 0) 0x55a34de6f000 con 0x55a314e16800 > -9898> 2018-07-11 07:32:20.948117 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6802/15919 conn(0x55a314e87000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1811 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:20.947129) v4 > -9897> 2018-07-11 07:32:20.948130 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6802/15919 1811 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:20.947129) v4 ==== 2004+0+0 (3785654154 0 0) 0x55a324474c00 con 0x55a314e87000 > -9896> 2018-07-11 07:32:20.948143 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6803/15919 conn(0x55a314e88800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1811 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:20.947129) v4 > -9895> 2018-07-11 07:32:20.948155 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6803/15919 1811 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:20.947129) v4 ==== 2004+0+0 (3785654154 0 0) 0x55a34de6f000 con 0x55a314e88800 > -9894> 2018-07-11 07:32:21.234951 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.115:0/13323 conn(0x55a321665800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1824 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:32:21.571624) v4 > -9893> 2018-07-11 07:32:21.234970 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.6 10.8.128.115:0/13323 1824 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:21.571624) v4 ==== 2004+0+0 (3092423944 0 0) 0x55a324474c00 con 0x55a321665800 > -9892> 2018-07-11 07:32:21.234987 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:21.571624) v4 -- 0x55a324475000 con 0 > -9891> 2018-07-11 07:32:21.235115 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.115:0/13323 conn(0x55a321668800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1824 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:32:21.571624) v4 > -9890> 2018-07-11 07:32:21.235132 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.6 10.8.128.115:0/13323 1824 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:21.571624) v4 ==== 2004+0+0 (3092423944 0 0) 0x55a332e79a00 con 0x55a321668800 > -9889> 2018-07-11 07:32:21.235142 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:21.571624) v4 -- 0x55a33047ea00 con 0 > -9888> 2018-07-11 07:32:21.519046 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a331506d80 con 0 > -9887> 2018-07-11 07:32:21.534365 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5005 0x55a31e8dea00 replica scrub(pg: 6.15,from:0'0,to:430'106241,epoch:494/492,start:6:a973dd6b:::benchmark_data_magna108_35617_object492734:0,end:6:a974b81a:::benchmark_data_magna108.ceph.redhat.com_2478_object230510:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -9886> 2018-07-11 07:32:21.534396 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5005 ==== replica scrub(pg: 6.15,from:0'0,to:430'106241,epoch:494/492,start:6:a973dd6b:::benchmark_data_magna108_35617_object492734:0,end:6:a974b81a:::benchmark_data_magna108.ceph.redhat.com_2478_object230510:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (2233346048 0 0) 0x55a31e8dea00 con 0x55a317019000 > -9885> 2018-07-11 07:32:21.615227 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15287 conn(0x55a311b72000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1803 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:32:22.827331) v4 > -9884> 2018-07-11 07:32:21.615263 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.4 10.8.128.113:0/15287 1803 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:22.827331) v4 ==== 2004+0+0 (1964114195 0 0) 0x55a324474c00 con 0x55a311b72000 > -9883> 2018-07-11 07:32:21.615274 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:22.827331) v4 -- 0x55a33047ea00 con 0 > -9882> 2018-07-11 07:32:21.615273 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15287 conn(0x55a311b75000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1803 0x55a314574e00 osd_ping(ping e494 stamp 2018-07-11 07:32:22.827331) v4 > -9881> 2018-07-11 07:32:21.615295 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.4 10.8.128.113:0/15287 1803 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:22.827331) v4 ==== 2004+0+0 (1964114195 0 0) 0x55a314574e00 con 0x55a311b75000 > -9880> 2018-07-11 07:32:21.615306 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:22.827331) v4 -- 0x55a33047f400 con 0 > -9879> 2018-07-11 07:32:22.062198 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12650 conn(0x55a311b67000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1826 0x55a35772aa00 osd_ping(ping e494 stamp 2018-07-11 07:32:22.096756) v4 > -9878> 2018-07-11 07:32:22.062222 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.3 10.8.128.106:0/12650 1826 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:22.096756) v4 ==== 2004+0+0 (3840886750 0 0) 0x55a35772aa00 con 0x55a311b67000 > -9877> 2018-07-11 07:32:22.062236 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:22.096756) v4 -- 0x55a33047f400 con 0 > -9876> 2018-07-11 07:32:22.062291 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12650 conn(0x55a311b68800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1826 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:32:22.096756) v4 > -9875> 2018-07-11 07:32:22.062317 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.3 10.8.128.106:0/12650 1826 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:22.096756) v4 ==== 2004+0+0 (3840886750 0 0) 0x55a34de6f000 con 0x55a311b68800 > -9874> 2018-07-11 07:32:22.062331 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:22.096756) v4 -- 0x55a33d673200 con 0 > -9873> 2018-07-11 07:32:22.115792 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15287 conn(0x55a311b72000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1804 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:32:23.327893) v4 > -9872> 2018-07-11 07:32:22.115816 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.4 10.8.128.113:0/15287 1804 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:23.327893) v4 ==== 2004+0+0 (4011448953 0 0) 0x55a332e79a00 con 0x55a311b72000 > -9871> 2018-07-11 07:32:22.115827 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:23.327893) v4 -- 0x55a3131bd200 con 0 > -9870> 2018-07-11 07:32:22.115840 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15287 conn(0x55a311b75000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1804 0x55a314574e00 osd_ping(ping e494 stamp 2018-07-11 07:32:23.327893) v4 > -9869> 2018-07-11 07:32:22.115861 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.4 10.8.128.113:0/15287 1804 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:23.327893) v4 ==== 2004+0+0 (4011448953 0 0) 0x55a314574e00 con 0x55a311b75000 > -9868> 2018-07-11 07:32:22.115873 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:23.327893) v4 -- 0x55a34e290c00 con 0 > -9867> 2018-07-11 07:32:22.248281 7f35488a3700 1 -- 10.8.128.107:6800/12665 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55a324471b80 con 0 > -9866> 2018-07-11 07:32:22.248353 7f35488a3700 1 -- 10.8.128.107:6800/12665 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55a317010dc0 con 0 > -9865> 2018-07-11 07:32:22.255362 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a33151afc0 con 0 > -9864> 2018-07-11 07:32:22.268464 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5006 0x55a31e8dec80 replica scrub(pg: 6.15,from:0'0,to:430'105444,epoch:494/492,start:6:a974b81a:::benchmark_data_magna108.ceph.redhat.com_2478_object230510:0,end:6:a975fdc1:::benchmark_data_magna108_35617_object752821:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -9863> 2018-07-11 07:32:22.268487 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5006 ==== replica scrub(pg: 6.15,from:0'0,to:430'105444,epoch:494/492,start:6:a974b81a:::benchmark_data_magna108.ceph.redhat.com_2478_object230510:0,end:6:a975fdc1:::benchmark_data_magna108_35617_object752821:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (1591342797 0 0) 0x55a31e8dec80 con 0x55a317019000 > -9862> 2018-07-11 07:32:22.352144 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12651 conn(0x55a3216a0800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1817 0x55a35772aa00 osd_ping(ping e494 stamp 2018-07-11 07:32:22.386767) v4 > -9861> 2018-07-11 07:32:22.352165 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.2 10.8.128.106:0/12651 1817 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:22.386767) v4 ==== 2004+0+0 (846131475 0 0) 0x55a35772aa00 con 0x55a3216a0800 > -9860> 2018-07-11 07:32:22.352181 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:22.386767) v4 -- 0x55a33047fe00 con 0 > -9859> 2018-07-11 07:32:22.352266 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12651 conn(0x55a32169f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1817 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:32:22.386767) v4 > -9858> 2018-07-11 07:32:22.352291 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.2 10.8.128.106:0/12651 1817 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:22.386767) v4 ==== 2004+0+0 (846131475 0 0) 0x55a324474c00 con 0x55a32169f000 > -9857> 2018-07-11 07:32:22.352307 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:22.386767) v4 -- 0x55a3131bd200 con 0 > -9856> 2018-07-11 07:32:22.647632 7f3543899700 5 osd.10 494 heartbeat: osd_stat(783 GB used, 148 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,0,1]) > -9855> 2018-07-11 07:32:22.647646 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6802/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:32:22.647645) v4 -- 0x55a3252c4400 con 0 > -9854> 2018-07-11 07:32:22.647682 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6803/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:32:22.647645) v4 -- 0x55a3252c5200 con 0 > -9853> 2018-07-11 07:32:22.647696 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6806/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:32:22.647645) v4 -- 0x55a3252c5800 con 0 > -9852> 2018-07-11 07:32:22.647710 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6807/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:32:22.647645) v4 -- 0x55a3252c5600 con 0 > -9851> 2018-07-11 07:32:22.647733 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6808/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:32:22.647645) v4 -- 0x55a3252c5000 con 0 > -9850> 2018-07-11 07:32:22.647744 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6810/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:32:22.647645) v4 -- 0x55a34e290800 con 0 > -9849> 2018-07-11 07:32:22.647754 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6809/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:32:22.647645) v4 -- 0x55a34e291800 con 0 > -9848> 2018-07-11 07:32:22.647769 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6811/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:32:22.647645) v4 -- 0x55a32700b000 con 0 > -9847> 2018-07-11 07:32:22.647782 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6810/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:32:22.647645) v4 -- 0x55a34e291600 con 0 > -9846> 2018-07-11 07:32:22.647793 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6811/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:32:22.647645) v4 -- 0x55a34e290a00 con 0 > -9845> 2018-07-11 07:32:22.647805 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6806/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:32:22.647645) v4 -- 0x55a314574600 con 0 > -9844> 2018-07-11 07:32:22.647817 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6807/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:32:22.647645) v4 -- 0x55a314575600 con 0 > -9843> 2018-07-11 07:32:22.647829 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6802/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:32:22.647645) v4 -- 0x55a32fb19800 con 0 > -9842> 2018-07-11 07:32:22.647842 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6803/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:32:22.647645) v4 -- 0x55a32e457a00 con 0 > -9841> 2018-07-11 07:32:22.647861 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6806/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:32:22.647645) v4 -- 0x55a332febe00 con 0 > -9840> 2018-07-11 07:32:22.647875 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6807/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:32:22.647645) v4 -- 0x55a34da1ae00 con 0 > -9839> 2018-07-11 07:32:22.647892 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6802/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:32:22.647645) v4 -- 0x55a332fea400 con 0 > -9838> 2018-07-11 07:32:22.647905 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6803/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:32:22.647645) v4 -- 0x55a332348e00 con 0 > -9837> 2018-07-11 07:32:22.647918 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6806/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:32:22.647645) v4 -- 0x55a3314cea00 con 0 > -9836> 2018-07-11 07:32:22.647934 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6807/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:32:22.647645) v4 -- 0x55a332355400 con 0 > -9835> 2018-07-11 07:32:22.647974 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6802/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:32:22.647645) v4 -- 0x55a32eb15e00 con 0 > -9834> 2018-07-11 07:32:22.648000 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6803/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:32:22.647645) v4 -- 0x55a32ee77800 con 0 > -9833> 2018-07-11 07:32:22.648111 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6806/12617 conn(0x55a314e26000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1812 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:22.647645) v4 > -9832> 2018-07-11 07:32:22.648132 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6806/12617 1812 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:22.647645) v4 ==== 2004+0+0 (4046963008 0 0) 0x55a332e79a00 con 0x55a314e26000 > -9831> 2018-07-11 07:32:22.648125 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6802/15255 conn(0x55a314e1c000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1812 0x55a34e290c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:22.647645) v4 > -9830> 2018-07-11 07:32:22.648144 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6802/15255 1812 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:22.647645) v4 ==== 2004+0+0 (4046963008 0 0) 0x55a34e290c00 con 0x55a314e1c000 > -9829> 2018-07-11 07:32:22.648187 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6803/15255 conn(0x55a314e1d800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1812 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:22.647645) v4 > -9828> 2018-07-11 07:32:22.648201 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6803/15255 1812 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:22.647645) v4 ==== 2004+0+0 (4046963008 0 0) 0x55a332e79a00 con 0x55a314e1d800 > -9827> 2018-07-11 07:32:22.648194 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6807/12617 conn(0x55a314e27800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1812 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:22.647645) v4 > -9826> 2018-07-11 07:32:22.648219 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6807/12617 1812 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:22.647645) v4 ==== 2004+0+0 (4046963008 0 0) 0x55a34de78600 con 0x55a314e27800 > -9825> 2018-07-11 07:32:22.648235 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6810/12651 conn(0x55a314e97800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1812 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:22.647645) v4 > -9824> 2018-07-11 07:32:22.648272 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6810/12651 1812 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:22.647645) v4 ==== 2004+0+0 (4046963008 0 0) 0x55a332e79a00 con 0x55a314e97800 > -9823> 2018-07-11 07:32:22.648288 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6810/15287 conn(0x55a314e03000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1812 0x55a34e290c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:22.647645) v4 > -9822> 2018-07-11 07:32:22.648298 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6806/12677 conn(0x55a314e84000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1812 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:22.647645) v4 > -9821> 2018-07-11 07:32:22.648308 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6810/15287 1812 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:22.647645) v4 ==== 2004+0+0 (4046963008 0 0) 0x55a34e290c00 con 0x55a314e03000 > -9820> 2018-07-11 07:32:22.648309 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6811/15287 conn(0x55a314e04800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1812 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:22.647645) v4 > -9819> 2018-07-11 07:32:22.648313 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6806/12677 1812 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:22.647645) v4 ==== 2004+0+0 (4046963008 0 0) 0x55a34de78600 con 0x55a314e84000 > -9818> 2018-07-11 07:32:22.648320 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6811/15287 1812 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:22.647645) v4 ==== 2004+0+0 (4046963008 0 0) 0x55a332e79a00 con 0x55a314e04800 > -9817> 2018-07-11 07:32:22.648346 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6808/12651 conn(0x55a314e96000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1812 0x55a34e290c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:22.647645) v4 > -9816> 2018-07-11 07:32:22.648358 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6808/12651 1812 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:22.647645) v4 ==== 2004+0+0 (4046963008 0 0) 0x55a34e290c00 con 0x55a314e96000 > -9815> 2018-07-11 07:32:22.648389 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6802/13323 conn(0x55a314e15000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1812 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:22.647645) v4 > -9814> 2018-07-11 07:32:22.648387 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6809/12650 conn(0x55a314e1f000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1812 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:22.647645) v4 > -9813> 2018-07-11 07:32:22.648403 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6802/13323 1812 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:22.647645) v4 ==== 2004+0+0 (4046963008 0 0) 0x55a332e79a00 con 0x55a314e15000 > -9812> 2018-07-11 07:32:22.648405 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6809/12650 1812 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:22.647645) v4 ==== 2004+0+0 (4046963008 0 0) 0x55a34de78600 con 0x55a314e1f000 > -9811> 2018-07-11 07:32:22.648407 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6807/12677 conn(0x55a314e85800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1812 0x55a34e290c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:22.647645) v4 > -9810> 2018-07-11 07:32:22.648420 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6807/12677 1812 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:22.647645) v4 ==== 2004+0+0 (4046963008 0 0) 0x55a34e290c00 con 0x55a314e85800 > -9809> 2018-07-11 07:32:22.648433 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6806/15885 conn(0x55a314e00000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1812 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:22.647645) v4 > -9808> 2018-07-11 07:32:22.648449 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6806/15885 1812 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:22.647645) v4 ==== 2004+0+0 (4046963008 0 0) 0x55a332e79a00 con 0x55a314e00000 > -9807> 2018-07-11 07:32:22.648451 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6806/15263 conn(0x55a314e99000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1812 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:22.647645) v4 > -9806> 2018-07-11 07:32:22.648449 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6811/12650 conn(0x55a314e20800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1812 0x55a34e290c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:22.647645) v4 > -9805> 2018-07-11 07:32:22.648472 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6811/12650 1812 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:22.647645) v4 ==== 2004+0+0 (4046963008 0 0) 0x55a34e290c00 con 0x55a314e20800 > -9804> 2018-07-11 07:32:22.648469 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6806/15263 1812 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:22.647645) v4 ==== 2004+0+0 (4046963008 0 0) 0x55a34de78600 con 0x55a314e99000 > -9803> 2018-07-11 07:32:22.648481 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6802/15919 conn(0x55a314e87000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1812 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:22.647645) v4 > -9802> 2018-07-11 07:32:22.648493 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6802/15919 1812 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:22.647645) v4 ==== 2004+0+0 (4046963008 0 0) 0x55a332e79a00 con 0x55a314e87000 > -9801> 2018-07-11 07:32:22.648505 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6807/15263 conn(0x55a314e9a800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1812 0x55a34e290c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:22.647645) v4 > -9800> 2018-07-11 07:32:22.648518 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6807/15263 1812 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:22.647645) v4 ==== 2004+0+0 (4046963008 0 0) 0x55a34e290c00 con 0x55a314e9a800 > -9799> 2018-07-11 07:32:22.648524 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6803/13323 conn(0x55a314e16800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1812 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:22.647645) v4 > -9798> 2018-07-11 07:32:22.648541 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6803/13323 1812 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:22.647645) v4 ==== 2004+0+0 (4046963008 0 0) 0x55a34de78600 con 0x55a314e16800 > -9797> 2018-07-11 07:32:22.648547 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6803/11967 conn(0x55a314e13800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1812 0x55a34e290c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:22.647645) v4 > -9796> 2018-07-11 07:32:22.648557 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6803/11967 1812 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:22.647645) v4 ==== 2004+0+0 (4046963008 0 0) 0x55a34e290c00 con 0x55a314e13800 > -9795> 2018-07-11 07:32:22.648582 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6802/11967 conn(0x55a314e12000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1812 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:22.647645) v4 > -9794> 2018-07-11 07:32:22.648596 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6802/11967 1812 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:22.647645) v4 ==== 2004+0+0 (4046963008 0 0) 0x55a34de78600 con 0x55a314e12000 > -9793> 2018-07-11 07:32:22.648632 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6803/15919 conn(0x55a314e88800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1812 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:22.647645) v4 > -9792> 2018-07-11 07:32:22.648646 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6803/15919 1812 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:22.647645) v4 ==== 2004+0+0 (4046963008 0 0) 0x55a34de78600 con 0x55a314e88800 > -9791> 2018-07-11 07:32:22.648685 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6807/15885 conn(0x55a314e01800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1812 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:22.647645) v4 > -9790> 2018-07-11 07:32:22.648699 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6807/15885 1812 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:22.647645) v4 ==== 2004+0+0 (4046963008 0 0) 0x55a34de78600 con 0x55a314e01800 > -9789> 2018-07-11 07:32:22.862926 7f355d88e700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.107:0/12617 conn(0x55a32447f800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1785 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:32:22.862601) v4 > -9788> 2018-07-11 07:32:22.862929 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.107:0/12617 conn(0x55a32447e000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1785 0x55a34e290c00 osd_ping(ping e494 stamp 2018-07-11 07:32:22.862601) v4 > -9787> 2018-07-11 07:32:22.862948 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.9 10.8.128.107:0/12617 1785 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:22.862601) v4 ==== 2004+0+0 (4171757302 0 0) 0x55a34e290c00 con 0x55a32447e000 > -9786> 2018-07-11 07:32:22.862948 7f355d88e700 1 -- 10.8.128.107:6803/12665 <== osd.9 10.8.128.107:0/12617 1785 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:22.862601) v4 ==== 2004+0+0 (4171757302 0 0) 0x55a34de78600 con 0x55a32447f800 > -9785> 2018-07-11 07:32:22.862965 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:22.862601) v4 -- 0x55a3252c5600 con 0 > -9784> 2018-07-11 07:32:22.863022 7f355d88e700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:22.862601) v4 -- 0x55a3252c5800 con 0 > -9783> 2018-07-11 07:32:22.915644 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a34b4e0d80 con 0 > -9782> 2018-07-11 07:32:22.926976 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5007 0x55a31e8def00 replica scrub(pg: 6.15,from:0'0,to:430'106709,epoch:494/492,start:6:a975fdc1:::benchmark_data_magna108_35617_object752821:0,end:6:a976a51c:::benchmark_data_magna108_35617_object968258:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -9781> 2018-07-11 07:32:22.926999 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5007 ==== replica scrub(pg: 6.15,from:0'0,to:430'106709,epoch:494/492,start:6:a975fdc1:::benchmark_data_magna108_35617_object752821:0,end:6:a976a51c:::benchmark_data_magna108_35617_object968258:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (3001241902 0 0) 0x55a31e8def00 con 0x55a317019000 > -9780> 2018-07-11 07:32:23.115301 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12677 conn(0x55a32169d800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1833 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:32:23.149875) v4 > -9779> 2018-07-11 07:32:23.115309 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12677 conn(0x55a32169c000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1833 0x55a35772aa00 osd_ping(ping e494 stamp 2018-07-11 07:32:23.149875) v4 > -9778> 2018-07-11 07:32:23.115322 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.1 10.8.128.106:0/12677 1833 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:23.149875) v4 ==== 2004+0+0 (2768627348 0 0) 0x55a34de6f000 con 0x55a32169d800 > -9777> 2018-07-11 07:32:23.115332 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:23.149875) v4 -- 0x55a3252c5800 con 0 > -9776> 2018-07-11 07:32:23.115328 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.1 10.8.128.106:0/12677 1833 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:23.149875) v4 ==== 2004+0+0 (2768627348 0 0) 0x55a35772aa00 con 0x55a32169c000 > -9775> 2018-07-11 07:32:23.115358 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:23.149875) v4 -- 0x55a3252c4400 con 0 > -9774> 2018-07-11 07:32:23.125448 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4d00000 on bdev 1, free 0x2fe000; fallback to bdev 2 > -9773> 2018-07-11 07:32:23.352292 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15885 conn(0x55a311b76800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1829 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:32:23.975453) v4 > -9772> 2018-07-11 07:32:23.352300 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15885 conn(0x55a3170b5800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1829 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:32:23.975453) v4 > -9771> 2018-07-11 07:32:23.352315 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.7 10.8.128.114:0/15885 1829 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:23.975453) v4 ==== 2004+0+0 (4238891390 0 0) 0x55a34de78600 con 0x55a311b76800 > -9770> 2018-07-11 07:32:23.352318 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.7 10.8.128.114:0/15885 1829 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:23.975453) v4 ==== 2004+0+0 (4238891390 0 0) 0x55a332e79a00 con 0x55a3170b5800 > -9769> 2018-07-11 07:32:23.352326 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:23.975453) v4 -- 0x55a3252c5800 con 0 > -9768> 2018-07-11 07:32:23.352354 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:23.975453) v4 -- 0x55a32fb19800 con 0 > -9767> 2018-07-11 07:32:23.734340 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b7000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1761 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:32:23.484496) v4 > -9766> 2018-07-11 07:32:23.734361 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.8 10.8.128.108:0/11967 1761 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:23.484496) v4 ==== 2004+0+0 (3358189388 0 0) 0x55a34de6f000 con 0x55a3170b7000 > -9765> 2018-07-11 07:32:23.734370 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:23.484496) v4 -- 0x55a33d673200 con 0 > -9764> 2018-07-11 07:32:23.734389 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b8800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1761 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:32:23.484496) v4 > -9763> 2018-07-11 07:32:23.734418 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.8 10.8.128.108:0/11967 1761 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:23.484496) v4 ==== 2004+0+0 (3358189388 0 0) 0x55a324474c00 con 0x55a3170b8800 > -9762> 2018-07-11 07:32:23.734430 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:23.484496) v4 -- 0x55a34e290a00 con 0 > -9761> 2018-07-11 07:32:23.748187 7f3543899700 5 osd.10 494 heartbeat: osd_stat(783 GB used, 148 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,0,0,1]) > -9760> 2018-07-11 07:32:23.748202 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6802/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:32:23.748200) v4 -- 0x55a332fafe00 con 0 > -9759> 2018-07-11 07:32:23.748243 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6803/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:32:23.748200) v4 -- 0x55a32e457800 con 0 > -9758> 2018-07-11 07:32:23.748273 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6806/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:32:23.748200) v4 -- 0x55a332fe9200 con 0 > -9757> 2018-07-11 07:32:23.748287 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6807/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:32:23.748200) v4 -- 0x55a33234fc00 con 0 > -9756> 2018-07-11 07:32:23.748300 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6808/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:32:23.748200) v4 -- 0x55a332fe8800 con 0 > -9755> 2018-07-11 07:32:23.748311 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6810/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:32:23.748200) v4 -- 0x55a33234aa00 con 0 > -9754> 2018-07-11 07:32:23.748324 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6809/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:32:23.748200) v4 -- 0x55a34da1aa00 con 0 > -9753> 2018-07-11 07:32:23.748337 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6811/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:32:23.748200) v4 -- 0x55a34da1a800 con 0 > -9752> 2018-07-11 07:32:23.748350 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6810/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:32:23.748200) v4 -- 0x55a349f89200 con 0 > -9751> 2018-07-11 07:32:23.748366 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6811/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:32:23.748200) v4 -- 0x55a349f89a00 con 0 > -9750> 2018-07-11 07:32:23.748379 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6806/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:32:23.748200) v4 -- 0x55a349f89400 con 0 > -9749> 2018-07-11 07:32:23.748398 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6807/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:32:23.748200) v4 -- 0x55a349f89c00 con 0 > -9748> 2018-07-11 07:32:23.748416 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6802/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:32:23.748200) v4 -- 0x55a34da1a600 con 0 > -9747> 2018-07-11 07:32:23.748433 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6803/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:32:23.748200) v4 -- 0x55a349f89000 con 0 > -9746> 2018-07-11 07:32:23.748446 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6806/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:32:23.748200) v4 -- 0x55a349f89600 con 0 > -9745> 2018-07-11 07:32:23.748458 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6807/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:32:23.748200) v4 -- 0x55a349f89800 con 0 > -9744> 2018-07-11 07:32:23.748471 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6802/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:32:23.748200) v4 -- 0x55a331d5c200 con 0 > -9743> 2018-07-11 07:32:23.748482 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6803/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:32:23.748200) v4 -- 0x55a331d5c400 con 0 > -9742> 2018-07-11 07:32:23.748498 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6806/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:32:23.748200) v4 -- 0x55a331d5ca00 con 0 > -9741> 2018-07-11 07:32:23.748513 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6807/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:32:23.748200) v4 -- 0x55a331d5ce00 con 0 > -9740> 2018-07-11 07:32:23.748550 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6802/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:32:23.748200) v4 -- 0x55a331d5c800 con 0 > -9739> 2018-07-11 07:32:23.748573 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6803/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:32:23.748200) v4 -- 0x55a349f89e00 con 0 > -9738> 2018-07-11 07:32:23.748688 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6806/12617 conn(0x55a314e26000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1813 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:23.748200) v4 > -9737> 2018-07-11 07:32:23.748714 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6806/12617 1813 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:23.748200) v4 ==== 2004+0+0 (3126795820 0 0) 0x55a324474c00 con 0x55a314e26000 > -9736> 2018-07-11 07:32:23.748716 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6807/12617 conn(0x55a314e27800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1813 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:23.748200) v4 > -9735> 2018-07-11 07:32:23.748754 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6807/12617 1813 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:23.748200) v4 ==== 2004+0+0 (3126795820 0 0) 0x55a34de78600 con 0x55a314e27800 > -9734> 2018-07-11 07:32:23.748769 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6807/12677 conn(0x55a314e85800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1813 0x55a3252c5600 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:23.748200) v4 > -9733> 2018-07-11 07:32:23.748787 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6803/15255 conn(0x55a314e1d800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1813 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:23.748200) v4 > -9732> 2018-07-11 07:32:23.748805 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6803/15255 1813 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:23.748200) v4 ==== 2004+0+0 (3126795820 0 0) 0x55a324474c00 con 0x55a314e1d800 > -9731> 2018-07-11 07:32:23.748787 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6807/12677 1813 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:23.748200) v4 ==== 2004+0+0 (3126795820 0 0) 0x55a3252c5600 con 0x55a314e85800 > -9730> 2018-07-11 07:32:23.748839 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6810/12651 conn(0x55a314e97800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1813 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:23.748200) v4 > -9729> 2018-07-11 07:32:23.748834 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6809/12650 conn(0x55a314e1f000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1813 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:23.748200) v4 > -9728> 2018-07-11 07:32:23.748853 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6810/12651 1813 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:23.748200) v4 ==== 2004+0+0 (3126795820 0 0) 0x55a324474c00 con 0x55a314e97800 > -9727> 2018-07-11 07:32:23.748855 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6809/12650 1813 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:23.748200) v4 ==== 2004+0+0 (3126795820 0 0) 0x55a34de78600 con 0x55a314e1f000 > -9726> 2018-07-11 07:32:23.748868 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6810/15287 conn(0x55a314e03000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1813 0x55a3252c5600 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:23.748200) v4 > -9725> 2018-07-11 07:32:23.748886 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6810/15287 1813 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:23.748200) v4 ==== 2004+0+0 (3126795820 0 0) 0x55a3252c5600 con 0x55a314e03000 > -9724> 2018-07-11 07:32:23.748898 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6811/15287 conn(0x55a314e04800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1813 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:23.748200) v4 > -9723> 2018-07-11 07:32:23.748912 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6811/15287 1813 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:23.748200) v4 ==== 2004+0+0 (3126795820 0 0) 0x55a324474c00 con 0x55a314e04800 > -9722> 2018-07-11 07:32:23.748918 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6802/15255 conn(0x55a314e1c000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1813 0x55a3252c5600 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:23.748200) v4 > -9721> 2018-07-11 07:32:23.748937 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6802/15255 1813 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:23.748200) v4 ==== 2004+0+0 (3126795820 0 0) 0x55a3252c5600 con 0x55a314e1c000 > -9720> 2018-07-11 07:32:23.748961 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6806/12677 conn(0x55a314e84000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1813 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:23.748200) v4 > -9719> 2018-07-11 07:32:23.748981 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6802/15919 conn(0x55a314e87000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1813 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:23.748200) v4 > -9718> 2018-07-11 07:32:23.748986 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6806/12677 1813 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:23.748200) v4 ==== 2004+0+0 (3126795820 0 0) 0x55a34de78600 con 0x55a314e84000 > -9717> 2018-07-11 07:32:23.748982 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6811/12650 conn(0x55a314e20800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1813 0x55a3252c5600 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:23.748200) v4 > -9716> 2018-07-11 07:32:23.748997 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6802/15919 1813 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:23.748200) v4 ==== 2004+0+0 (3126795820 0 0) 0x55a324474c00 con 0x55a314e87000 > -9715> 2018-07-11 07:32:23.748998 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6811/12650 1813 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:23.748200) v4 ==== 2004+0+0 (3126795820 0 0) 0x55a3252c5600 con 0x55a314e20800 > -9714> 2018-07-11 07:32:23.749012 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6806/15263 conn(0x55a314e99000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1813 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:23.748200) v4 > -9713> 2018-07-11 07:32:23.749029 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6806/15263 1813 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:23.748200) v4 ==== 2004+0+0 (3126795820 0 0) 0x55a34de78600 con 0x55a314e99000 > -9712> 2018-07-11 07:32:23.749036 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6802/13323 conn(0x55a314e15000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1813 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:23.748200) v4 > -9711> 2018-07-11 07:32:23.749044 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6807/15263 conn(0x55a314e9a800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1813 0x55a3252c5600 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:23.748200) v4 > -9710> 2018-07-11 07:32:23.749054 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6802/13323 1813 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:23.748200) v4 ==== 2004+0+0 (3126795820 0 0) 0x55a324474c00 con 0x55a314e15000 > -9709> 2018-07-11 07:32:23.749065 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6807/15263 1813 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:23.748200) v4 ==== 2004+0+0 (3126795820 0 0) 0x55a3252c5600 con 0x55a314e9a800 > -9708> 2018-07-11 07:32:23.749078 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6803/13323 conn(0x55a314e16800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1813 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:23.748200) v4 > -9707> 2018-07-11 07:32:23.749082 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6806/15885 conn(0x55a314e00000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1813 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:23.748200) v4 > -9706> 2018-07-11 07:32:23.749095 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6806/15885 1813 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:23.748200) v4 ==== 2004+0+0 (3126795820 0 0) 0x55a324474c00 con 0x55a314e00000 > -9705> 2018-07-11 07:32:23.749094 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6803/13323 1813 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:23.748200) v4 ==== 2004+0+0 (3126795820 0 0) 0x55a34de78600 con 0x55a314e16800 > -9704> 2018-07-11 07:32:23.749112 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6808/12651 conn(0x55a314e96000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1813 0x55a3252c5600 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:23.748200) v4 > -9703> 2018-07-11 07:32:23.749130 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6808/12651 1813 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:23.748200) v4 ==== 2004+0+0 (3126795820 0 0) 0x55a3252c5600 con 0x55a314e96000 > -9702> 2018-07-11 07:32:23.749137 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6802/11967 conn(0x55a314e12000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1813 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:23.748200) v4 > -9701> 2018-07-11 07:32:23.749152 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6802/11967 1813 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:23.748200) v4 ==== 2004+0+0 (3126795820 0 0) 0x55a34de78600 con 0x55a314e12000 > -9700> 2018-07-11 07:32:23.749207 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6803/11967 conn(0x55a314e13800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1813 0x55a3252c5600 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:23.748200) v4 > -9699> 2018-07-11 07:32:23.749227 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6803/11967 1813 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:23.748200) v4 ==== 2004+0+0 (3126795820 0 0) 0x55a3252c5600 con 0x55a314e13800 > -9698> 2018-07-11 07:32:23.749239 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6803/15919 conn(0x55a314e88800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1813 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:23.748200) v4 > -9697> 2018-07-11 07:32:23.749265 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6803/15919 1813 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:23.748200) v4 ==== 2004+0+0 (3126795820 0 0) 0x55a34de78600 con 0x55a314e88800 > -9696> 2018-07-11 07:32:23.749302 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6807/15885 conn(0x55a314e01800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1813 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:23.748200) v4 > -9695> 2018-07-11 07:32:23.749315 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6807/15885 1813 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:23.748200) v4 ==== 2004+0+0 (3126795820 0 0) 0x55a34de78600 con 0x55a314e01800 > -9694> 2018-07-11 07:32:23.762796 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12650 conn(0x55a311b67000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1827 0x55a3252c5600 osd_ping(ping e494 stamp 2018-07-11 07:32:23.797333) v4 > -9693> 2018-07-11 07:32:23.762796 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12650 conn(0x55a311b68800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1827 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:32:23.797333) v4 > -9692> 2018-07-11 07:32:23.762826 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.3 10.8.128.106:0/12650 1827 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:23.797333) v4 ==== 2004+0+0 (1203593939 0 0) 0x55a3252c5600 con 0x55a311b67000 > -9691> 2018-07-11 07:32:23.762828 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.3 10.8.128.106:0/12650 1827 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:23.797333) v4 ==== 2004+0+0 (1203593939 0 0) 0x55a34de78600 con 0x55a311b68800 > -9690> 2018-07-11 07:32:23.762846 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:23.797333) v4 -- 0x55a33234fc00 con 0 > -9689> 2018-07-11 07:32:23.762874 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:23.797333) v4 -- 0x55a33d673200 con 0 > -9688> 2018-07-11 07:32:23.816226 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15287 conn(0x55a311b72000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1805 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:32:25.028378) v4 > -9687> 2018-07-11 07:32:23.816233 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15287 conn(0x55a311b75000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1805 0x55a35772aa00 osd_ping(ping e494 stamp 2018-07-11 07:32:25.028378) v4 > -9686> 2018-07-11 07:32:23.816258 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.4 10.8.128.113:0/15287 1805 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:25.028378) v4 ==== 2004+0+0 (405264478 0 0) 0x55a324474c00 con 0x55a311b72000 > -9685> 2018-07-11 07:32:23.816260 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.4 10.8.128.113:0/15287 1805 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:25.028378) v4 ==== 2004+0+0 (405264478 0 0) 0x55a35772aa00 con 0x55a311b75000 > -9684> 2018-07-11 07:32:23.816270 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:25.028378) v4 -- 0x55a32e457800 con 0 > -9683> 2018-07-11 07:32:23.816293 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:25.028378) v4 -- 0x55a33234fc00 con 0 > -9682> 2018-07-11 07:32:23.852760 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15885 conn(0x55a311b76800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1830 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:32:24.475906) v4 > -9681> 2018-07-11 07:32:23.852770 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15885 conn(0x55a3170b5800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1830 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:32:24.475906) v4 > -9680> 2018-07-11 07:32:23.852780 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.7 10.8.128.114:0/15885 1830 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:24.475906) v4 ==== 2004+0+0 (152049621 0 0) 0x55a34de6f000 con 0x55a311b76800 > -9679> 2018-07-11 07:32:23.852787 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.7 10.8.128.114:0/15885 1830 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:24.475906) v4 ==== 2004+0+0 (152049621 0 0) 0x55a332e79a00 con 0x55a3170b5800 > -9678> 2018-07-11 07:32:23.852789 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:24.475906) v4 -- 0x55a33d673200 con 0 > -9677> 2018-07-11 07:32:23.852811 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:24.475906) v4 -- 0x55a32e457800 con 0 > -9676> 2018-07-11 07:32:24.005102 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15263 conn(0x55a321664000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1808 0x55a3252c5600 osd_ping(ping e494 stamp 2018-07-11 07:32:25.217245) v4 > -9675> 2018-07-11 07:32:24.005133 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.5 10.8.128.113:0/15263 1808 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:25.217245) v4 ==== 2004+0+0 (4089340173 0 0) 0x55a3252c5600 con 0x55a321664000 > -9674> 2018-07-11 07:32:24.005147 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:25.217245) v4 -- 0x55a33234fc00 con 0 > -9673> 2018-07-11 07:32:24.005200 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15263 conn(0x55a321667000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1808 0x55a3252c5600 osd_ping(ping e494 stamp 2018-07-11 07:32:25.217245) v4 > -9672> 2018-07-11 07:32:24.005220 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.5 10.8.128.113:0/15263 1808 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:25.217245) v4 ==== 2004+0+0 (4089340173 0 0) 0x55a3252c5600 con 0x55a321667000 > -9671> 2018-07-11 07:32:24.005235 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:25.217245) v4 -- 0x55a332fafe00 con 0 > -9670> 2018-07-11 07:32:24.545840 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 58] Generated table #36563: 4957165 keys, 80348251 bytes > -9669> 2018-07-11 07:32:24.545896 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294344545868, "cf_name": "default", "job": 58, "event": "table_file_creation", "file_number": 36563, "file_size": 80348251, "table_properties": {"data_size": 67109335, "index_size": 689225, "filter_size": 12548701, "raw_key_size": 143206995, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957165, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957165", "kMergeOperands": "0"}} > -9668> 2018-07-11 07:32:24.563398 7f355d88e700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.107:0/12617 conn(0x55a32447f800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1786 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:32:24.563040) v4 > -9667> 2018-07-11 07:32:24.563395 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.107:0/12617 conn(0x55a32447e000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1786 0x55a35772aa00 osd_ping(ping e494 stamp 2018-07-11 07:32:24.563040) v4 > -9666> 2018-07-11 07:32:24.563419 7f355d88e700 1 -- 10.8.128.107:6803/12665 <== osd.9 10.8.128.107:0/12617 1786 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:24.563040) v4 ==== 2004+0+0 (4190350573 0 0) 0x55a34de78600 con 0x55a32447f800 > -9665> 2018-07-11 07:32:24.563434 7f355d88e700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:24.563040) v4 -- 0x55a33d673200 con 0 > -9664> 2018-07-11 07:32:24.563459 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.9 10.8.128.107:0/12617 1786 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:24.563040) v4 ==== 2004+0+0 (4190350573 0 0) 0x55a35772aa00 con 0x55a32447e000 > -9663> 2018-07-11 07:32:24.563476 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:24.563040) v4 -- 0x55a332fafe00 con 0 > -9662> 2018-07-11 07:32:24.755714 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a3166b6b40 con 0 > -9661> 2018-07-11 07:32:24.776335 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5008 0x55a31e8df180 replica scrub(pg: 6.15,from:0'0,to:430'106131,epoch:494/492,start:6:a976a51c:::benchmark_data_magna108_35617_object968258:0,end:6:a977d70a:::benchmark_data_magna108_35617_object702296:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -9660> 2018-07-11 07:32:24.776365 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5008 ==== replica scrub(pg: 6.15,from:0'0,to:430'106131,epoch:494/492,start:6:a976a51c:::benchmark_data_magna108_35617_object968258:0,end:6:a977d70a:::benchmark_data_magna108_35617_object702296:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (114632686 0 0) 0x55a31e8df180 con 0x55a317019000 > -9659> 2018-07-11 07:32:24.953289 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15885 conn(0x55a3170b5800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1831 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:32:25.576493) v4 > -9658> 2018-07-11 07:32:24.953296 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15885 conn(0x55a311b76800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1831 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:32:25.576493) v4 > -9657> 2018-07-11 07:32:24.953312 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.7 10.8.128.114:0/15885 1831 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:25.576493) v4 ==== 2004+0+0 (1734815595 0 0) 0x55a324474c00 con 0x55a3170b5800 > -9656> 2018-07-11 07:32:24.953313 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.7 10.8.128.114:0/15885 1831 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:25.576493) v4 ==== 2004+0+0 (1734815595 0 0) 0x55a34de6f000 con 0x55a311b76800 > -9655> 2018-07-11 07:32:24.953322 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:25.576493) v4 -- 0x55a32e457800 con 0 > -9654> 2018-07-11 07:32:24.953350 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:25.576493) v4 -- 0x55a33d673200 con 0 > -9653> 2018-07-11 07:32:25.105428 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15263 conn(0x55a321667000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1809 0x55a3252c5600 osd_ping(ping e494 stamp 2018-07-11 07:32:26.317636) v4 > -9652> 2018-07-11 07:32:25.105449 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.5 10.8.128.113:0/15263 1809 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:26.317636) v4 ==== 2004+0+0 (1852747439 0 0) 0x55a3252c5600 con 0x55a321667000 > -9651> 2018-07-11 07:32:25.105461 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:26.317636) v4 -- 0x55a332fafe00 con 0 > -9650> 2018-07-11 07:32:25.105500 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15263 conn(0x55a321664000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1809 0x55a3252c5600 osd_ping(ping e494 stamp 2018-07-11 07:32:26.317636) v4 > -9649> 2018-07-11 07:32:25.105513 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.5 10.8.128.113:0/15263 1809 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:26.317636) v4 ==== 2004+0+0 (1852747439 0 0) 0x55a3252c5600 con 0x55a321664000 > -9648> 2018-07-11 07:32:25.105523 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:26.317636) v4 -- 0x55a3252c4400 con 0 > -9647> 2018-07-11 07:32:25.335482 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.115:0/13323 conn(0x55a321668800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1825 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:32:25.672113) v4 > -9646> 2018-07-11 07:32:25.335503 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.6 10.8.128.115:0/13323 1825 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:25.672113) v4 ==== 2004+0+0 (363534810 0 0) 0x55a332e79a00 con 0x55a321668800 > -9645> 2018-07-11 07:32:25.335513 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:25.672113) v4 -- 0x55a34e290a00 con 0 > -9644> 2018-07-11 07:32:25.335560 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.115:0/13323 conn(0x55a321665800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1825 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:32:25.672113) v4 > -9643> 2018-07-11 07:32:25.335574 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.6 10.8.128.115:0/13323 1825 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:25.672113) v4 ==== 2004+0+0 (363534810 0 0) 0x55a324474c00 con 0x55a321665800 > -9642> 2018-07-11 07:32:25.335583 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:25.672113) v4 -- 0x55a34e290800 con 0 > -9641> 2018-07-11 07:32:25.346174 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15919 conn(0x55a317074800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1820 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:32:25.969400) v4 > -9640> 2018-07-11 07:32:25.346196 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.11 10.8.128.114:0/15919 1820 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:25.969400) v4 ==== 2004+0+0 (590820726 0 0) 0x55a332e79a00 con 0x55a317074800 > -9639> 2018-07-11 07:32:25.346206 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:25.969400) v4 -- 0x55a3252c5200 con 0 > -9638> 2018-07-11 07:32:25.346293 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15919 conn(0x55a3170b4000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1820 0x55a35772aa00 osd_ping(ping e494 stamp 2018-07-11 07:32:25.969400) v4 > -9637> 2018-07-11 07:32:25.346319 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.11 10.8.128.114:0/15919 1820 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:25.969400) v4 ==== 2004+0+0 (590820726 0 0) 0x55a35772aa00 con 0x55a3170b4000 > -9636> 2018-07-11 07:32:25.346335 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:25.969400) v4 -- 0x55a332fafe00 con 0 > -9635> 2018-07-11 07:32:25.367449 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a34537f200 con 0 > -9634> 2018-07-11 07:32:25.384662 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5009 0x55a31e8df400 replica scrub(pg: 6.15,from:0'0,to:461'106909,epoch:494/492,start:6:a977d70a:::benchmark_data_magna108_35617_object702296:0,end:6:a978a95a:::benchmark_data_magna108_35617_object745896:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -9633> 2018-07-11 07:32:25.384691 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5009 ==== replica scrub(pg: 6.15,from:0'0,to:461'106909,epoch:494/492,start:6:a977d70a:::benchmark_data_magna108_35617_object702296:0,end:6:a978a95a:::benchmark_data_magna108_35617_object745896:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (59071850 0 0) 0x55a31e8df400 con 0x55a317019000 > -9632> 2018-07-11 07:32:25.415369 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15255 conn(0x55a311b65800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1837 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:32:26.627580) v4 > -9631> 2018-07-11 07:32:25.415381 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15255 conn(0x55a311b64000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1837 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:32:26.627580) v4 > -9630> 2018-07-11 07:32:25.415396 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.0 10.8.128.113:0/15255 1837 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:26.627580) v4 ==== 2004+0+0 (1795379734 0 0) 0x55a324474c00 con 0x55a311b65800 > -9629> 2018-07-11 07:32:25.415404 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.0 10.8.128.113:0/15255 1837 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:26.627580) v4 ==== 2004+0+0 (1795379734 0 0) 0x55a34de78600 con 0x55a311b64000 > -9628> 2018-07-11 07:32:25.415414 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:26.627580) v4 -- 0x55a3252c5200 con 0 > -9627> 2018-07-11 07:32:25.415442 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:26.627580) v4 -- 0x55a33d673200 con 0 > -9626> 2018-07-11 07:32:25.415780 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12677 conn(0x55a32169d800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1834 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:32:25.450310) v4 > -9625> 2018-07-11 07:32:25.415797 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.1 10.8.128.106:0/12677 1834 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:25.450310) v4 ==== 2004+0+0 (4146724539 0 0) 0x55a34de6f000 con 0x55a32169d800 > -9624> 2018-07-11 07:32:25.415806 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:25.450310) v4 -- 0x55a34e291e00 con 0 > -9623> 2018-07-11 07:32:25.415944 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12677 conn(0x55a32169c000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1834 0x55a3252c5600 osd_ping(ping e494 stamp 2018-07-11 07:32:25.450310) v4 > -9622> 2018-07-11 07:32:25.415965 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.1 10.8.128.106:0/12677 1834 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:25.450310) v4 ==== 2004+0+0 (4146724539 0 0) 0x55a3252c5600 con 0x55a32169c000 > -9621> 2018-07-11 07:32:25.415981 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:25.450310) v4 -- 0x55a34e290400 con 0 > -9620> 2018-07-11 07:32:25.434739 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b7000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1762 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:32:25.184954) v4 > -9619> 2018-07-11 07:32:25.434759 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.8 10.8.128.108:0/11967 1762 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:25.184954) v4 ==== 2004+0+0 (1566137011 0 0) 0x55a34de78600 con 0x55a3170b7000 > -9618> 2018-07-11 07:32:25.434769 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:25.184954) v4 -- 0x55a34de5bc00 con 0 > -9617> 2018-07-11 07:32:25.434771 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b8800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1762 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:32:25.184954) v4 > -9616> 2018-07-11 07:32:25.434788 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.8 10.8.128.108:0/11967 1762 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:25.184954) v4 ==== 2004+0+0 (1566137011 0 0) 0x55a332e79a00 con 0x55a3170b8800 > -9615> 2018-07-11 07:32:25.434801 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:25.184954) v4 -- 0x55a3131bd200 con 0 > -9614> 2018-07-11 07:32:25.453708 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15885 conn(0x55a311b76800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1832 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:32:26.076931) v4 > -9613> 2018-07-11 07:32:25.453713 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15885 conn(0x55a3170b5800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1832 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:32:26.076931) v4 > -9612> 2018-07-11 07:32:25.453729 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.7 10.8.128.114:0/15885 1832 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:26.076931) v4 ==== 2004+0+0 (369984521 0 0) 0x55a34de6f000 con 0x55a311b76800 > -9611> 2018-07-11 07:32:25.453731 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.7 10.8.128.114:0/15885 1832 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:26.076931) v4 ==== 2004+0+0 (369984521 0 0) 0x55a324474c00 con 0x55a3170b5800 > -9610> 2018-07-11 07:32:25.453740 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:26.076931) v4 -- 0x55a3131bd400 con 0 > -9609> 2018-07-11 07:32:25.453766 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:26.076931) v4 -- 0x55a322455000 con 0 > -9608> 2018-07-11 07:32:25.835895 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.115:0/13323 conn(0x55a321665800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1826 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:32:26.172584) v4 > -9607> 2018-07-11 07:32:25.835922 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.6 10.8.128.115:0/13323 1826 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:26.172584) v4 ==== 2004+0+0 (141591337 0 0) 0x55a332e79a00 con 0x55a321665800 > -9606> 2018-07-11 07:32:25.835941 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:26.172584) v4 -- 0x55a3131bdc00 con 0 > -9605> 2018-07-11 07:32:25.836011 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.115:0/13323 conn(0x55a321668800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1826 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:32:26.172584) v4 > -9604> 2018-07-11 07:32:25.836026 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.6 10.8.128.115:0/13323 1826 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:26.172584) v4 ==== 2004+0+0 (141591337 0 0) 0x55a324474c00 con 0x55a321668800 > -9603> 2018-07-11 07:32:25.836036 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:26.172584) v4 -- 0x55a3131bdc00 con 0 > -9602> 2018-07-11 07:32:25.954279 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15885 conn(0x55a311b76800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1833 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:32:26.577464) v4 > -9601> 2018-07-11 07:32:25.954281 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15885 conn(0x55a3170b5800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1833 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:32:26.577464) v4 > -9600> 2018-07-11 07:32:25.954303 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.7 10.8.128.114:0/15885 1833 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:26.577464) v4 ==== 2004+0+0 (2338642783 0 0) 0x55a34de78600 con 0x55a311b76800 > -9599> 2018-07-11 07:32:25.954307 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.7 10.8.128.114:0/15885 1833 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:26.577464) v4 ==== 2004+0+0 (2338642783 0 0) 0x55a332e79a00 con 0x55a3170b5800 > -9598> 2018-07-11 07:32:25.954316 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:26.577464) v4 -- 0x55a322455000 con 0 > -9597> 2018-07-11 07:32:25.954339 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:26.577464) v4 -- 0x55a3131bdc00 con 0 > -9596> 2018-07-11 07:32:26.518854 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a311385680 con 0 > -9595> 2018-07-11 07:32:26.535975 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5010 0x55a31e8df680 replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:494/492,start:6:a978a95a:::benchmark_data_magna108_35617_object745896:0,end:6:a979e4ab:::benchmark_data_magna108_41525_object44172:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -9594> 2018-07-11 07:32:26.536005 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5010 ==== replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:494/492,start:6:a978a95a:::benchmark_data_magna108_35617_object745896:0,end:6:a979e4ab:::benchmark_data_magna108_41525_object44172:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 217+0+0 (3603803322 0 0) 0x55a31e8df680 con 0x55a317019000 > -9593> 2018-07-11 07:32:26.663114 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12650 conn(0x55a311b67000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1828 0x55a332fafe00 osd_ping(ping e494 stamp 2018-07-11 07:32:26.697722) v4 > -9592> 2018-07-11 07:32:26.663140 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.3 10.8.128.106:0/12650 1828 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:26.697722) v4 ==== 2004+0+0 (4099516364 0 0) 0x55a332fafe00 con 0x55a311b67000 > -9591> 2018-07-11 07:32:26.663155 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:26.697722) v4 -- 0x55a314575800 con 0 > -9590> 2018-07-11 07:32:26.663205 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12650 conn(0x55a311b68800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1828 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:32:26.697722) v4 > -9589> 2018-07-11 07:32:26.663228 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.3 10.8.128.106:0/12650 1828 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:26.697722) v4 ==== 2004+0+0 (4099516364 0 0) 0x55a34de6f000 con 0x55a311b68800 > -9588> 2018-07-11 07:32:26.663243 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:26.697722) v4 -- 0x55a3314cf400 con 0 > -9587> 2018-07-11 07:32:26.743271 7f35498a5700 10 monclient: tick > -9586> 2018-07-11 07:32:26.743285 7f35498a5700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2018-07-11 07:31:56.743283) > -9585> 2018-07-11 07:32:26.856059 7f354e0ae700 1 bluefs _allocate failed to allocate 0x2100000 on bdev 1, free 0x2fe000; fallback to bdev 2 > -9584> 2018-07-11 07:32:27.115843 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15255 conn(0x55a311b65800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1838 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:32:28.328044) v4 > -9583> 2018-07-11 07:32:27.115851 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15255 conn(0x55a311b64000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1838 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:32:28.328044) v4 > -9582> 2018-07-11 07:32:27.115873 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.0 10.8.128.113:0/15255 1838 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:28.328044) v4 ==== 2004+0+0 (2434022699 0 0) 0x55a324474c00 con 0x55a311b65800 > -9581> 2018-07-11 07:32:27.115878 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.0 10.8.128.113:0/15255 1838 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:28.328044) v4 ==== 2004+0+0 (2434022699 0 0) 0x55a34de78600 con 0x55a311b64000 > -9580> 2018-07-11 07:32:27.115887 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:28.328044) v4 -- 0x55a3131bd000 con 0 > -9579> 2018-07-11 07:32:27.115916 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:28.328044) v4 -- 0x55a331d5c600 con 0 > -9578> 2018-07-11 07:32:27.163755 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12650 conn(0x55a311b67000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1829 0x55a332fafe00 osd_ping(ping e494 stamp 2018-07-11 07:32:27.198279) v4 > -9577> 2018-07-11 07:32:27.163790 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.3 10.8.128.106:0/12650 1829 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:27.198279) v4 ==== 2004+0+0 (2983772816 0 0) 0x55a332fafe00 con 0x55a311b67000 > -9576> 2018-07-11 07:32:27.163800 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:27.198279) v4 -- 0x55a314574400 con 0 > -9575> 2018-07-11 07:32:27.163823 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12650 conn(0x55a311b68800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1829 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:32:27.198279) v4 > -9574> 2018-07-11 07:32:27.163844 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.3 10.8.128.106:0/12650 1829 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:27.198279) v4 ==== 2004+0+0 (2983772816 0 0) 0x55a34de6f000 con 0x55a311b68800 > -9573> 2018-07-11 07:32:27.163855 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:27.198279) v4 -- 0x55a331d5d000 con 0 > -9572> 2018-07-11 07:32:27.248583 7f35488a3700 1 -- 10.8.128.107:6800/12665 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55a3216de000 con 0 > -9571> 2018-07-11 07:32:27.248654 7f35488a3700 1 -- 10.8.128.107:6800/12665 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55a32e1bfb80 con 0 > -9570> 2018-07-11 07:32:27.516804 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 58] Generated table #36564: 2128504 keys, 34500149 bytes > -9569> 2018-07-11 07:32:27.516846 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294347516820, "cf_name": "default", "job": 58, "event": "table_file_creation", "file_number": 36564, "file_size": 34500149, "table_properties": {"data_size": 28815346, "index_size": 295662, "filter_size": 5388152, "raw_key_size": 61490125, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 7072, "num_entries": 2128504, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "2128504", "kMergeOperands": "0"}} > -9568> 2018-07-11 07:32:27.848722 7f3543899700 5 osd.10 494 heartbeat: osd_stat(783 GB used, 148 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,0,0,0,1]) > -9567> 2018-07-11 07:32:27.848739 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6802/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:32:27.848736) v4 -- 0x55a31dbc3000 con 0 > -9566> 2018-07-11 07:32:27.848779 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6803/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:32:27.848736) v4 -- 0x55a31dbc2200 con 0 > -9565> 2018-07-11 07:32:27.848795 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6806/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:32:27.848736) v4 -- 0x55a31dbc2800 con 0 > -9564> 2018-07-11 07:32:27.848818 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6807/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:32:27.848736) v4 -- 0x55a31dbc2a00 con 0 > -9563> 2018-07-11 07:32:27.848841 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6808/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:32:27.848736) v4 -- 0x55a31dbc2e00 con 0 > -9562> 2018-07-11 07:32:27.848860 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6810/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:32:27.848736) v4 -- 0x55a31dbc3600 con 0 > -9561> 2018-07-11 07:32:27.848878 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6809/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:32:27.848736) v4 -- 0x55a31dbc3400 con 0 > -9560> 2018-07-11 07:32:27.848897 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6811/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:32:27.848736) v4 -- 0x55a31dbc3200 con 0 > -9559> 2018-07-11 07:32:27.848922 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6810/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:32:27.848736) v4 -- 0x55a31dbc3a00 con 0 > -9558> 2018-07-11 07:32:27.848937 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6811/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:32:27.848736) v4 -- 0x55a31dbc3800 con 0 > -9557> 2018-07-11 07:32:27.848957 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6806/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:32:27.848736) v4 -- 0x55a31dbc3c00 con 0 > -9556> 2018-07-11 07:32:27.848979 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6807/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:32:27.848736) v4 -- 0x55a3484fe600 con 0 > -9555> 2018-07-11 07:32:27.849007 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6802/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:32:27.848736) v4 -- 0x55a327c0f400 con 0 > -9554> 2018-07-11 07:32:27.849024 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6803/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:32:27.848736) v4 -- 0x55a327982200 con 0 > -9553> 2018-07-11 07:32:27.849068 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6806/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:32:27.848736) v4 -- 0x55a327c0fa00 con 0 > -9552> 2018-07-11 07:32:27.849090 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6807/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:32:27.848736) v4 -- 0x55a327982000 con 0 > -9551> 2018-07-11 07:32:27.849111 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6802/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:32:27.848736) v4 -- 0x55a3252c5e00 con 0 > -9550> 2018-07-11 07:32:27.849133 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6803/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:32:27.848736) v4 -- 0x55a31d142400 con 0 > -9549> 2018-07-11 07:32:27.849172 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6806/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:32:27.848736) v4 -- 0x55a3484fea00 con 0 > -9548> 2018-07-11 07:32:27.849196 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6807/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:32:27.848736) v4 -- 0x55a3252c5c00 con 0 > -9547> 2018-07-11 07:32:27.849221 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6802/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:32:27.848736) v4 -- 0x55a3484ff200 con 0 > -9546> 2018-07-11 07:32:27.849267 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6802/15255 conn(0x55a314e1c000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1814 0x55a332fafe00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:27.848736) v4 > -9545> 2018-07-11 07:32:27.849286 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6802/15255 1814 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:27.848736) v4 ==== 2004+0+0 (3431612899 0 0) 0x55a332fafe00 con 0x55a314e1c000 > -9544> 2018-07-11 07:32:27.849285 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6803/15255 conn(0x55a314e1d800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1814 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:27.848736) v4 > -9543> 2018-07-11 07:32:27.849296 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6803/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:32:27.848736) v4 -- 0x55a3252c5a00 con 0 > -9542> 2018-07-11 07:32:27.849304 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6803/15255 1814 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:27.848736) v4 ==== 2004+0+0 (3431612899 0 0) 0x55a332e79a00 con 0x55a314e1d800 > -9541> 2018-07-11 07:32:27.849362 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6806/12677 conn(0x55a314e84000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1814 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:27.848736) v4 > -9540> 2018-07-11 07:32:27.849368 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6807/12677 conn(0x55a314e85800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1814 0x55a332fafe00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:27.848736) v4 > -9539> 2018-07-11 07:32:27.849389 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6806/12677 1814 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:27.848736) v4 ==== 2004+0+0 (3431612899 0 0) 0x55a34de78600 con 0x55a314e84000 > -9538> 2018-07-11 07:32:27.849396 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6807/12677 1814 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:27.848736) v4 ==== 2004+0+0 (3431612899 0 0) 0x55a332fafe00 con 0x55a314e85800 > -9537> 2018-07-11 07:32:27.849397 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6810/12651 conn(0x55a314e97800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1814 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:27.848736) v4 > -9536> 2018-07-11 07:32:27.849411 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6810/12651 1814 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:27.848736) v4 ==== 2004+0+0 (3431612899 0 0) 0x55a324474c00 con 0x55a314e97800 > -9535> 2018-07-11 07:32:27.849448 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6809/12650 conn(0x55a314e1f000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1814 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:27.848736) v4 > -9534> 2018-07-11 07:32:27.849460 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6811/15287 conn(0x55a314e04800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1814 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:27.848736) v4 > -9533> 2018-07-11 07:32:27.849473 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6809/12650 1814 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:27.848736) v4 ==== 2004+0+0 (3431612899 0 0) 0x55a34de78600 con 0x55a314e1f000 > -9532> 2018-07-11 07:32:27.849452 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6810/15287 conn(0x55a314e03000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1814 0x55a332fafe00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:27.848736) v4 > -9531> 2018-07-11 07:32:27.849478 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6811/15287 1814 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:27.848736) v4 ==== 2004+0+0 (3431612899 0 0) 0x55a324474c00 con 0x55a314e04800 > -9530> 2018-07-11 07:32:27.849486 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6810/15287 1814 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:27.848736) v4 ==== 2004+0+0 (3431612899 0 0) 0x55a332fafe00 con 0x55a314e03000 > -9529> 2018-07-11 07:32:27.849502 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6806/12617 conn(0x55a314e26000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1814 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:27.848736) v4 > -9528> 2018-07-11 07:32:27.849513 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6806/12617 1814 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:27.848736) v4 ==== 2004+0+0 (3431612899 0 0) 0x55a324474c00 con 0x55a314e26000 > -9527> 2018-07-11 07:32:27.849512 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6806/15263 conn(0x55a314e99000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1814 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:27.848736) v4 > -9526> 2018-07-11 07:32:27.849524 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6806/15263 1814 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:27.848736) v4 ==== 2004+0+0 (3431612899 0 0) 0x55a34de78600 con 0x55a314e99000 > -9525> 2018-07-11 07:32:27.849529 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6811/12650 conn(0x55a314e20800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1814 0x55a332fafe00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:27.848736) v4 > -9524> 2018-07-11 07:32:27.849543 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6811/12650 1814 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:27.848736) v4 ==== 2004+0+0 (3431612899 0 0) 0x55a332fafe00 con 0x55a314e20800 > -9523> 2018-07-11 07:32:27.849546 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6802/13323 conn(0x55a314e15000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1814 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:27.848736) v4 > -9522> 2018-07-11 07:32:27.849560 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6803/13323 conn(0x55a314e16800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1814 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:27.848736) v4 > -9521> 2018-07-11 07:32:27.849565 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6802/13323 1814 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:27.848736) v4 ==== 2004+0+0 (3431612899 0 0) 0x55a324474c00 con 0x55a314e15000 > -9520> 2018-07-11 07:32:27.849577 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6803/13323 1814 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:27.848736) v4 ==== 2004+0+0 (3431612899 0 0) 0x55a34de78600 con 0x55a314e16800 > -9519> 2018-07-11 07:32:27.849577 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6808/12651 conn(0x55a314e96000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1814 0x55a332fafe00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:27.848736) v4 > -9518> 2018-07-11 07:32:27.849587 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6808/12651 1814 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:27.848736) v4 ==== 2004+0+0 (3431612899 0 0) 0x55a332fafe00 con 0x55a314e96000 > -9517> 2018-07-11 07:32:27.849600 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6807/12617 conn(0x55a314e27800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1814 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:27.848736) v4 > -9516> 2018-07-11 07:32:27.849605 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6806/15885 conn(0x55a314e00000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1814 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:27.848736) v4 > -9515> 2018-07-11 07:32:27.849610 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6807/12617 1814 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:27.848736) v4 ==== 2004+0+0 (3431612899 0 0) 0x55a34de78600 con 0x55a314e27800 > -9514> 2018-07-11 07:32:27.849615 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6807/15263 conn(0x55a314e9a800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1814 0x55a332fafe00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:27.848736) v4 > -9513> 2018-07-11 07:32:27.849620 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6806/15885 1814 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:27.848736) v4 ==== 2004+0+0 (3431612899 0 0) 0x55a324474c00 con 0x55a314e00000 > -9512> 2018-07-11 07:32:27.849625 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6807/15263 1814 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:27.848736) v4 ==== 2004+0+0 (3431612899 0 0) 0x55a332fafe00 con 0x55a314e9a800 > -9511> 2018-07-11 07:32:27.849655 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6807/15885 conn(0x55a314e01800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1814 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:27.848736) v4 > -9510> 2018-07-11 07:32:27.849670 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6807/15885 1814 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:27.848736) v4 ==== 2004+0+0 (3431612899 0 0) 0x55a34de78600 con 0x55a314e01800 > -9509> 2018-07-11 07:32:27.849701 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6802/11967 conn(0x55a314e12000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1814 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:27.848736) v4 > -9508> 2018-07-11 07:32:27.849714 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6802/11967 1814 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:27.848736) v4 ==== 2004+0+0 (3431612899 0 0) 0x55a34de78600 con 0x55a314e12000 > -9507> 2018-07-11 07:32:27.849728 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6802/15919 conn(0x55a314e87000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1814 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:27.848736) v4 > -9506> 2018-07-11 07:32:27.849734 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6803/11967 conn(0x55a314e13800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1814 0x55a332fafe00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:27.848736) v4 > -9505> 2018-07-11 07:32:27.849742 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6802/15919 1814 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:27.848736) v4 ==== 2004+0+0 (3431612899 0 0) 0x55a324474c00 con 0x55a314e87000 > -9504> 2018-07-11 07:32:27.849752 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6803/11967 1814 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:27.848736) v4 ==== 2004+0+0 (3431612899 0 0) 0x55a332fafe00 con 0x55a314e13800 > -9503> 2018-07-11 07:32:27.849753 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6803/15919 conn(0x55a314e88800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1814 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:27.848736) v4 > -9502> 2018-07-11 07:32:27.849764 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6803/15919 1814 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:27.848736) v4 ==== 2004+0+0 (3431612899 0 0) 0x55a34de78600 con 0x55a314e88800 > -9501> 2018-07-11 07:32:27.900260 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a3315c5680 con 0 > -9500> 2018-07-11 07:32:27.916560 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15287 conn(0x55a311b72000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1806 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:32:29.128794) v4 > -9499> 2018-07-11 07:32:27.916585 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.4 10.8.128.113:0/15287 1806 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:29.128794) v4 ==== 2004+0+0 (1435731159 0 0) 0x55a324474c00 con 0x55a311b72000 > -9498> 2018-07-11 07:32:27.916578 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15287 conn(0x55a311b75000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1806 0x55a3252c5600 osd_ping(ping e494 stamp 2018-07-11 07:32:29.128794) v4 > -9497> 2018-07-11 07:32:27.916597 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:29.128794) v4 -- 0x55a31dbc3600 con 0 > -9496> 2018-07-11 07:32:27.916596 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.4 10.8.128.113:0/15287 1806 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:29.128794) v4 ==== 2004+0+0 (1435731159 0 0) 0x55a3252c5600 con 0x55a311b75000 > -9495> 2018-07-11 07:32:27.916623 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:29.128794) v4 -- 0x55a31dbc2a00 con 0 > -9494> 2018-07-11 07:32:27.919820 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5011 0x55a31e8df900 replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:494/492,start:6:a979e4ab:::benchmark_data_magna108_41525_object44172:0,end:6:a97ab548:::benchmark_data_magna108_35617_object1131253:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -9493> 2018-07-11 07:32:27.919850 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5011 ==== replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:494/492,start:6:a979e4ab:::benchmark_data_magna108_41525_object44172:0,end:6:a97ab548:::benchmark_data_magna108_35617_object1131253:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (4015749432 0 0) 0x55a31e8df900 con 0x55a317019000 > -9492> 2018-07-11 07:32:28.252692 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12651 conn(0x55a3216a0800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1818 0x55a3252c5600 osd_ping(ping e494 stamp 2018-07-11 07:32:28.287287) v4 > -9491> 2018-07-11 07:32:28.252702 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12651 conn(0x55a32169f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1818 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:32:28.287287) v4 > -9490> 2018-07-11 07:32:28.252717 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.2 10.8.128.106:0/12651 1818 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:28.287287) v4 ==== 2004+0+0 (3992450713 0 0) 0x55a3252c5600 con 0x55a3216a0800 > -9489> 2018-07-11 07:32:28.252719 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.2 10.8.128.106:0/12651 1818 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:28.287287) v4 ==== 2004+0+0 (3992450713 0 0) 0x55a332e79a00 con 0x55a32169f000 > -9488> 2018-07-11 07:32:28.252727 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:28.287287) v4 -- 0x55a31dbc2a00 con 0 > -9487> 2018-07-11 07:32:28.252752 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:28.287287) v4 -- 0x55a31dbc3600 con 0 > -9486> 2018-07-11 07:32:28.335216 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b7000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1763 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:32:28.085389) v4 > -9485> 2018-07-11 07:32:28.335237 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.8 10.8.128.108:0/11967 1763 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:28.085389) v4 ==== 2004+0+0 (2080677872 0 0) 0x55a34de78600 con 0x55a3170b7000 > -9484> 2018-07-11 07:32:28.335253 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:28.085389) v4 -- 0x55a31dbc3400 con 0 > -9483> 2018-07-11 07:32:28.335257 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b8800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1763 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:32:28.085389) v4 > -9482> 2018-07-11 07:32:28.335288 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.8 10.8.128.108:0/11967 1763 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:28.085389) v4 ==== 2004+0+0 (2080677872 0 0) 0x55a324474c00 con 0x55a3170b8800 > -9481> 2018-07-11 07:32:28.335305 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:28.085389) v4 -- 0x55a31dbc2200 con 0 > -9480> 2018-07-11 07:32:28.585294 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a34de24b40 con 0 > -9479> 2018-07-11 07:32:28.595374 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5012 0x55a31e8dfb80 replica scrub(pg: 6.15,from:0'0,to:430'105515,epoch:494/492,start:6:a97ab548:::benchmark_data_magna108_35617_object1131253:0,end:6:a97b9ab5:::benchmark_data_magna108_35617_object509090:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -9478> 2018-07-11 07:32:28.595403 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5012 ==== replica scrub(pg: 6.15,from:0'0,to:430'105515,epoch:494/492,start:6:a97ab548:::benchmark_data_magna108_35617_object1131253:0,end:6:a97b9ab5:::benchmark_data_magna108_35617_object509090:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 219+0+0 (3725515421 0 0) 0x55a31e8dfb80 con 0x55a317019000 > -9477> 2018-07-11 07:32:28.816318 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15255 conn(0x55a311b65800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1839 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:32:30.028448) v4 > -9476> 2018-07-11 07:32:28.816320 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15255 conn(0x55a311b64000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1839 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:32:30.028448) v4 > -9475> 2018-07-11 07:32:28.816346 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.0 10.8.128.113:0/15255 1839 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:30.028448) v4 ==== 2004+0+0 (2009438012 0 0) 0x55a332e79a00 con 0x55a311b65800 > -9474> 2018-07-11 07:32:28.816351 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.0 10.8.128.113:0/15255 1839 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:30.028448) v4 ==== 2004+0+0 (2009438012 0 0) 0x55a34de6f000 con 0x55a311b64000 > -9473> 2018-07-11 07:32:28.816359 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:30.028448) v4 -- 0x55a31dbc2200 con 0 > -9472> 2018-07-11 07:32:28.816382 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:30.028448) v4 -- 0x55a31dbc3400 con 0 > -9471> 2018-07-11 07:32:29.016969 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15287 conn(0x55a311b72000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1807 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:32:30.229236) v4 > -9470> 2018-07-11 07:32:29.016994 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.4 10.8.128.113:0/15287 1807 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:30.229236) v4 ==== 2004+0+0 (14147675 0 0) 0x55a324474c00 con 0x55a311b72000 > -9469> 2018-07-11 07:32:29.017003 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:30.229236) v4 -- 0x55a31dbc2200 con 0 > -9468> 2018-07-11 07:32:29.016997 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15287 conn(0x55a311b75000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1807 0x55a332fafe00 osd_ping(ping e494 stamp 2018-07-11 07:32:30.229236) v4 > -9467> 2018-07-11 07:32:29.017014 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.4 10.8.128.113:0/15287 1807 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:30.229236) v4 ==== 2004+0+0 (14147675 0 0) 0x55a332fafe00 con 0x55a311b75000 > -9466> 2018-07-11 07:32:29.017027 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:30.229236) v4 -- 0x55a314574400 con 0 > -9465> 2018-07-11 07:32:29.205830 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15263 conn(0x55a321667000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1810 0x55a31dbc2a00 osd_ping(ping e494 stamp 2018-07-11 07:32:30.418052) v4 > -9464> 2018-07-11 07:32:29.205855 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.5 10.8.128.113:0/15263 1810 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:30.418052) v4 ==== 2004+0+0 (1757758830 0 0) 0x55a31dbc2a00 con 0x55a321667000 > -9463> 2018-07-11 07:32:29.205865 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:30.418052) v4 -- 0x55a314575e00 con 0 > -9462> 2018-07-11 07:32:29.205901 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15263 conn(0x55a321664000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1810 0x55a31dbc2a00 osd_ping(ping e494 stamp 2018-07-11 07:32:30.418052) v4 > -9461> 2018-07-11 07:32:29.205914 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.5 10.8.128.113:0/15263 1810 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:30.418052) v4 ==== 2004+0+0 (1757758830 0 0) 0x55a31dbc2a00 con 0x55a321664000 > -9460> 2018-07-11 07:32:29.205923 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:30.418052) v4 -- 0x55a33d672000 con 0 > -9459> 2018-07-11 07:32:29.388135 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a331554900 con 0 > -9458> 2018-07-11 07:32:29.418444 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5013 0x55a33f8ce000 replica scrub(pg: 6.15,from:0'0,to:430'106686,epoch:494/492,start:6:a97b9ab5:::benchmark_data_magna108_35617_object509090:0,end:6:a97ca2e7:::benchmark_data_magna108_35617_object394519:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -9457> 2018-07-11 07:32:29.418468 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5013 ==== replica scrub(pg: 6.15,from:0'0,to:430'106686,epoch:494/492,start:6:a97b9ab5:::benchmark_data_magna108_35617_object509090:0,end:6:a97ca2e7:::benchmark_data_magna108_35617_object394519:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (198950273 0 0) 0x55a33f8ce000 con 0x55a317019000 > -9456> 2018-07-11 07:32:30.035758 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b8800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1764 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:32:29.785835) v4 > -9455> 2018-07-11 07:32:30.035756 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b7000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1764 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:32:29.785835) v4 > -9454> 2018-07-11 07:32:30.035786 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.8 10.8.128.108:0/11967 1764 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:29.785835) v4 ==== 2004+0+0 (2842680831 0 0) 0x55a332e79a00 con 0x55a3170b8800 > -9453> 2018-07-11 07:32:30.035790 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.8 10.8.128.108:0/11967 1764 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:29.785835) v4 ==== 2004+0+0 (2842680831 0 0) 0x55a34de78600 con 0x55a3170b7000 > -9452> 2018-07-11 07:32:30.035798 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:29.785835) v4 -- 0x55a31dbc2200 con 0 > -9451> 2018-07-11 07:32:30.035833 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:29.785835) v4 -- 0x55a31dbc3400 con 0 > -9450> 2018-07-11 07:32:30.151551 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a31556eb40 con 0 > -9449> 2018-07-11 07:32:30.170741 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5014 0x55a33f8ce280 replica scrub(pg: 6.15,from:0'0,to:430'106823,epoch:494/492,start:6:a97ca2e7:::benchmark_data_magna108_35617_object394519:0,end:6:a97d791e:::benchmark_data_magna108_35617_object96690:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -9448> 2018-07-11 07:32:30.170765 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5014 ==== replica scrub(pg: 6.15,from:0'0,to:430'106823,epoch:494/492,start:6:a97ca2e7:::benchmark_data_magna108_35617_object394519:0,end:6:a97d791e:::benchmark_data_magna108_35617_object96690:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 217+0+0 (4145591131 0 0) 0x55a33f8ce280 con 0x55a317019000 > -9447> 2018-07-11 07:32:30.463857 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.107:0/12617 conn(0x55a32447e000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1787 0x55a332fafe00 osd_ping(ping e494 stamp 2018-07-11 07:32:30.463540) v4 > -9446> 2018-07-11 07:32:30.463879 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.9 10.8.128.107:0/12617 1787 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:30.463540) v4 ==== 2004+0+0 (2908196814 0 0) 0x55a332fafe00 con 0x55a32447e000 > -9445> 2018-07-11 07:32:30.463889 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:30.463540) v4 -- 0x55a314575e00 con 0 > -9444> 2018-07-11 07:32:30.463897 7f355d88e700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.107:0/12617 conn(0x55a32447f800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1787 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:32:30.463540) v4 > -9443> 2018-07-11 07:32:30.463921 7f355d88e700 1 -- 10.8.128.107:6803/12665 <== osd.9 10.8.128.107:0/12617 1787 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:30.463540) v4 ==== 2004+0+0 (2908196814 0 0) 0x55a34de6f000 con 0x55a32447f800 > -9442> 2018-07-11 07:32:30.463959 7f355d88e700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:30.463540) v4 -- 0x55a31dbc2800 con 0 > -9441> 2018-07-11 07:32:30.536128 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b7000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1765 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:32:30.286296) v4 > -9440> 2018-07-11 07:32:30.536149 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.8 10.8.128.108:0/11967 1765 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:30.286296) v4 ==== 2004+0+0 (1403805556 0 0) 0x55a34de78600 con 0x55a3170b7000 > -9439> 2018-07-11 07:32:30.536158 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:30.286296) v4 -- 0x55a331d5d000 con 0 > -9438> 2018-07-11 07:32:30.536148 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b8800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1765 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:32:30.286296) v4 > -9437> 2018-07-11 07:32:30.536178 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.8 10.8.128.108:0/11967 1765 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:30.286296) v4 ==== 2004+0+0 (1403805556 0 0) 0x55a324474c00 con 0x55a3170b8800 > -9436> 2018-07-11 07:32:30.536189 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:30.286296) v4 -- 0x55a3131bd000 con 0 > -9435> 2018-07-11 07:32:30.646535 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15919 conn(0x55a3170b4000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1821 0x55a33d672000 osd_ping(ping e494 stamp 2018-07-11 07:32:31.269817) v4 > -9434> 2018-07-11 07:32:30.646556 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.11 10.8.128.114:0/15919 1821 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:31.269817) v4 ==== 2004+0+0 (2383261159 0 0) 0x55a33d672000 con 0x55a3170b4000 > -9433> 2018-07-11 07:32:30.646566 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:31.269817) v4 -- 0x55a314574c00 con 0 > -9432> 2018-07-11 07:32:30.646561 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15919 conn(0x55a317074800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1821 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:32:31.269817) v4 > -9431> 2018-07-11 07:32:30.646581 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.11 10.8.128.114:0/15919 1821 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:31.269817) v4 ==== 2004+0+0 (2383261159 0 0) 0x55a332e79a00 con 0x55a317074800 > -9430> 2018-07-11 07:32:30.646594 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:31.269817) v4 -- 0x55a3131bd600 con 0 > -9429> 2018-07-11 07:32:30.716094 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12677 conn(0x55a32169d800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1835 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:32:30.750728) v4 > -9428> 2018-07-11 07:32:30.716099 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12677 conn(0x55a32169c000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1835 0x55a332fafe00 osd_ping(ping e494 stamp 2018-07-11 07:32:30.750728) v4 > -9427> 2018-07-11 07:32:30.716115 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.1 10.8.128.106:0/12677 1835 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:30.750728) v4 ==== 2004+0+0 (3196470341 0 0) 0x55a34de6f000 con 0x55a32169d800 > -9426> 2018-07-11 07:32:30.716116 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.1 10.8.128.106:0/12677 1835 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:30.750728) v4 ==== 2004+0+0 (3196470341 0 0) 0x55a332fafe00 con 0x55a32169c000 > -9425> 2018-07-11 07:32:30.716124 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:30.750728) v4 -- 0x55a327786400 con 0 > -9424> 2018-07-11 07:32:30.716148 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:30.750728) v4 -- 0x55a33d672200 con 0 > -9423> 2018-07-11 07:32:30.900081 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a326e41d40 con 0 > -9422> 2018-07-11 07:32:30.925783 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5015 0x55a33f8ce500 replica scrub(pg: 6.15,from:0'0,to:430'106533,epoch:494/492,start:6:a97d791e:::benchmark_data_magna108_35617_object96690:0,end:6:a97e7e64:::benchmark_data_magna108.ceph.redhat.com_2478_object274115:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -9421> 2018-07-11 07:32:30.925814 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5015 ==== replica scrub(pg: 6.15,from:0'0,to:430'106533,epoch:494/492,start:6:a97d791e:::benchmark_data_magna108_35617_object96690:0,end:6:a97e7e64:::benchmark_data_magna108.ceph.redhat.com_2478_object274115:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 232+0+0 (2698217545 0 0) 0x55a33f8ce500 con 0x55a317019000 > -9420> 2018-07-11 07:32:31.216507 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12677 conn(0x55a32169d800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1836 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:32:31.251182) v4 > -9419> 2018-07-11 07:32:31.216510 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12677 conn(0x55a32169c000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1836 0x55a33d672000 osd_ping(ping e494 stamp 2018-07-11 07:32:31.251182) v4 > -9418> 2018-07-11 07:32:31.216533 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.1 10.8.128.106:0/12677 1836 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:31.251182) v4 ==== 2004+0+0 (3739675356 0 0) 0x55a34de78600 con 0x55a32169d800 > -9417> 2018-07-11 07:32:31.216536 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.1 10.8.128.106:0/12677 1836 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:31.251182) v4 ==== 2004+0+0 (3739675356 0 0) 0x55a33d672000 con 0x55a32169c000 > -9416> 2018-07-11 07:32:31.216545 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:31.251182) v4 -- 0x55a327786400 con 0 > -9415> 2018-07-11 07:32:31.216571 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:31.251182) v4 -- 0x55a33d672200 con 0 > -9414> 2018-07-11 07:32:31.660522 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a31eece900 con 0 > -9413> 2018-07-11 07:32:31.693533 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5016 0x55a33f8ce780 replica scrub(pg: 6.15,from:0'0,to:461'106910,epoch:494/492,start:6:a97e7e64:::benchmark_data_magna108.ceph.redhat.com_2478_object274115:0,end:6:a97fa027:::benchmark_data_magna108.ceph.redhat.com_2478_object305175:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -9412> 2018-07-11 07:32:31.693564 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5016 ==== replica scrub(pg: 6.15,from:0'0,to:461'106910,epoch:494/492,start:6:a97e7e64:::benchmark_data_magna108.ceph.redhat.com_2478_object274115:0,end:6:a97fa027:::benchmark_data_magna108.ceph.redhat.com_2478_object305175:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 248+0+0 (4101657766 0 0) 0x55a33f8ce780 con 0x55a317019000 > -9411> 2018-07-11 07:32:31.736389 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.115:0/13323 conn(0x55a321665800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1827 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:32:32.073056) v4 > -9410> 2018-07-11 07:32:31.736411 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.6 10.8.128.115:0/13323 1827 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:32.073056) v4 ==== 2004+0+0 (4158492472 0 0) 0x55a324474c00 con 0x55a321665800 > -9409> 2018-07-11 07:32:31.736421 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:32.073056) v4 -- 0x55a3131bd600 con 0 > -9408> 2018-07-11 07:32:31.736500 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.115:0/13323 conn(0x55a321668800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1827 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:32:32.073056) v4 > -9407> 2018-07-11 07:32:31.736520 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.6 10.8.128.115:0/13323 1827 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:32.073056) v4 ==== 2004+0+0 (4158492472 0 0) 0x55a332e79a00 con 0x55a321668800 > -9406> 2018-07-11 07:32:31.736536 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:32.073056) v4 -- 0x55a3131bd600 con 0 > -9405> 2018-07-11 07:32:31.854666 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15885 conn(0x55a311b76800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1834 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:32:32.477920) v4 > -9404> 2018-07-11 07:32:31.854688 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.7 10.8.128.114:0/15885 1834 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:32.477920) v4 ==== 2004+0+0 (1313593119 0 0) 0x55a34de6f000 con 0x55a311b76800 > -9403> 2018-07-11 07:32:31.854684 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15885 conn(0x55a3170b5800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1834 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:32:32.477920) v4 > -9402> 2018-07-11 07:32:31.854700 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:32.477920) v4 -- 0x55a327786400 con 0 > -9401> 2018-07-11 07:32:31.854715 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.7 10.8.128.114:0/15885 1834 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:32.477920) v4 ==== 2004+0+0 (1313593119 0 0) 0x55a324474c00 con 0x55a3170b5800 > -9400> 2018-07-11 07:32:31.854734 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:32.477920) v4 -- 0x55a3131bd600 con 0 > -9399> 2018-07-11 07:32:32.248866 7f35488a3700 1 -- 10.8.128.107:6800/12665 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55a346875600 con 0 > -9398> 2018-07-11 07:32:32.248939 7f35488a3700 1 -- 10.8.128.107:6800/12665 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55a32aa0d080 con 0 > -9397> 2018-07-11 07:32:32.600731 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a331556d80 con 0 > -9396> 2018-07-11 07:32:32.646119 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5017 0x55a33f8cea00 replica scrub(pg: 6.15,from:0'0,to:430'106725,epoch:494/492,start:6:a97fa027:::benchmark_data_magna108.ceph.redhat.com_2478_object305175:0,end:6:a980bf23:::benchmark_data_magna108.ceph.redhat.com_2478_object439649:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -9395> 2018-07-11 07:32:32.646152 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5017 ==== replica scrub(pg: 6.15,from:0'0,to:430'106725,epoch:494/492,start:6:a97fa027:::benchmark_data_magna108.ceph.redhat.com_2478_object305175:0,end:6:a980bf23:::benchmark_data_magna108.ceph.redhat.com_2478_object439649:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 248+0+0 (470621614 0 0) 0x55a33f8cea00 con 0x55a317019000 > -9394> 2018-07-11 07:32:32.720442 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4d00000 on bdev 1, free 0x2fe000; fallback to bdev 2 > -9393> 2018-07-11 07:32:33.064303 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12650 conn(0x55a311b68800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1830 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:32:33.098868) v4 > -9392> 2018-07-11 07:32:33.064303 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12650 conn(0x55a311b67000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1830 0x55a332fafe00 osd_ping(ping e494 stamp 2018-07-11 07:32:33.098868) v4 > -9391> 2018-07-11 07:32:33.064337 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.3 10.8.128.106:0/12650 1830 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:33.098868) v4 ==== 2004+0+0 (3503410428 0 0) 0x55a34de78600 con 0x55a311b68800 > -9390> 2018-07-11 07:32:33.064339 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.3 10.8.128.106:0/12650 1830 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:33.098868) v4 ==== 2004+0+0 (3503410428 0 0) 0x55a332fafe00 con 0x55a311b67000 > -9389> 2018-07-11 07:32:33.064360 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:33.098868) v4 -- 0x55a327786400 con 0 > -9388> 2018-07-11 07:32:33.064392 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:33.098868) v4 -- 0x55a33d672200 con 0 > -9387> 2018-07-11 07:32:33.117340 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15287 conn(0x55a311b75000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1808 0x55a33d672000 osd_ping(ping e494 stamp 2018-07-11 07:32:34.329627) v4 > -9386> 2018-07-11 07:32:33.117336 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15287 conn(0x55a311b72000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1808 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:32:34.329627) v4 > -9385> 2018-07-11 07:32:33.117363 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.4 10.8.128.113:0/15287 1808 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:34.329627) v4 ==== 2004+0+0 (2282447181 0 0) 0x55a33d672000 con 0x55a311b75000 > -9384> 2018-07-11 07:32:33.117368 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.4 10.8.128.113:0/15287 1808 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:34.329627) v4 ==== 2004+0+0 (2282447181 0 0) 0x55a332e79a00 con 0x55a311b72000 > -9383> 2018-07-11 07:32:33.117375 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:34.329627) v4 -- 0x55a33d672200 con 0 > -9382> 2018-07-11 07:32:33.117410 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:34.329627) v4 -- 0x55a3131bd600 con 0 > -9381> 2018-07-11 07:32:33.149481 7f3543899700 5 osd.10 494 heartbeat: osd_stat(783 GB used, 148 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,0,1]) > -9380> 2018-07-11 07:32:33.149494 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6802/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:32:33.149493) v4 -- 0x55a3484fe800 con 0 > -9379> 2018-07-11 07:32:33.149529 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6803/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:32:33.149493) v4 -- 0x55a3484fec00 con 0 > -9378> 2018-07-11 07:32:33.149545 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6806/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:32:33.149493) v4 -- 0x55a3484ffa00 con 0 > -9377> 2018-07-11 07:32:33.149566 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6807/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:32:33.149493) v4 -- 0x55a3484ff000 con 0 > -9376> 2018-07-11 07:32:33.149598 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6808/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:32:33.149493) v4 -- 0x55a34256c000 con 0 > -9375> 2018-07-11 07:32:33.149619 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6810/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:32:33.149493) v4 -- 0x55a3484fee00 con 0 > -9374> 2018-07-11 07:32:33.149638 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6809/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:32:33.149493) v4 -- 0x55a3484ff800 con 0 > -9373> 2018-07-11 07:32:33.149659 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6811/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:32:33.149493) v4 -- 0x55a3484ffe00 con 0 > -9372> 2018-07-11 07:32:33.149683 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6810/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:32:33.149493) v4 -- 0x55a34256c400 con 0 > -9371> 2018-07-11 07:32:33.149699 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6811/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:32:33.149493) v4 -- 0x55a3484ff400 con 0 > -9370> 2018-07-11 07:32:33.149720 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6806/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:32:33.149493) v4 -- 0x55a3272af000 con 0 > -9369> 2018-07-11 07:32:33.149742 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6807/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:32:33.149493) v4 -- 0x55a3272af200 con 0 > -9368> 2018-07-11 07:32:33.149769 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6802/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:32:33.149493) v4 -- 0x55a31d142a00 con 0 > -9367> 2018-07-11 07:32:33.149788 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6803/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:32:33.149493) v4 -- 0x55a3272af800 con 0 > -9366> 2018-07-11 07:32:33.149833 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6806/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:32:33.149493) v4 -- 0x55a3272aec00 con 0 > -9365> 2018-07-11 07:32:33.149856 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6807/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:32:33.149493) v4 -- 0x55a3272aee00 con 0 > -9364> 2018-07-11 07:32:33.149879 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6802/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:32:33.149493) v4 -- 0x55a3272af600 con 0 > -9363> 2018-07-11 07:32:33.149918 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6803/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:32:33.149493) v4 -- 0x55a3272afc00 con 0 > -9362> 2018-07-11 07:32:33.149942 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6806/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:32:33.149493) v4 -- 0x55a3272afe00 con 0 > -9361> 2018-07-11 07:32:33.149966 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6807/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:32:33.149493) v4 -- 0x55a3272afa00 con 0 > -9360> 2018-07-11 07:32:33.150018 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6802/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:32:33.149493) v4 -- 0x55a3272af400 con 0 > -9359> 2018-07-11 07:32:33.150005 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6807/12677 conn(0x55a314e85800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1815 0x55a332fafe00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:33.149493) v4 > -9358> 2018-07-11 07:32:33.150041 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6807/12677 1815 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:33.149493) v4 ==== 2004+0+0 (1947172494 0 0) 0x55a332fafe00 con 0x55a314e85800 > -9357> 2018-07-11 07:32:33.150046 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6803/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:32:33.149493) v4 -- 0x55a3456b0400 con 0 > -9356> 2018-07-11 07:32:33.150039 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6803/15255 conn(0x55a314e1d800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1815 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:33.149493) v4 > -9355> 2018-07-11 07:32:33.150061 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6806/12677 conn(0x55a314e84000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1815 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:33.149493) v4 > -9354> 2018-07-11 07:32:33.150072 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6803/15255 1815 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:33.149493) v4 ==== 2004+0+0 (1947172494 0 0) 0x55a324474c00 con 0x55a314e1d800 > -9353> 2018-07-11 07:32:33.150080 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6806/12677 1815 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:33.149493) v4 ==== 2004+0+0 (1947172494 0 0) 0x55a34de78600 con 0x55a314e84000 > -9352> 2018-07-11 07:32:33.150096 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6802/15255 conn(0x55a314e1c000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1815 0x55a332fafe00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:33.149493) v4 > -9351> 2018-07-11 07:32:33.150112 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6802/15255 1815 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:33.149493) v4 ==== 2004+0+0 (1947172494 0 0) 0x55a332fafe00 con 0x55a314e1c000 > -9350> 2018-07-11 07:32:33.150153 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6811/12650 conn(0x55a314e20800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1815 0x55a332fafe00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:33.149493) v4 > -9349> 2018-07-11 07:32:33.150157 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6810/12651 conn(0x55a314e97800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1815 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:33.149493) v4 > -9348> 2018-07-11 07:32:33.150171 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6811/12650 1815 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:33.149493) v4 ==== 2004+0+0 (1947172494 0 0) 0x55a332fafe00 con 0x55a314e20800 > -9347> 2018-07-11 07:32:33.150175 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6810/12651 1815 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:33.149493) v4 ==== 2004+0+0 (1947172494 0 0) 0x55a332e79a00 con 0x55a314e97800 > -9346> 2018-07-11 07:32:33.150177 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6809/12650 conn(0x55a314e1f000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1815 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:33.149493) v4 > -9345> 2018-07-11 07:32:33.150209 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6808/12651 conn(0x55a314e96000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1815 0x55a332fafe00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:33.149493) v4 > -9344> 2018-07-11 07:32:33.150217 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6809/12650 1815 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:33.149493) v4 ==== 2004+0+0 (1947172494 0 0) 0x55a34de6f000 con 0x55a314e1f000 > -9343> 2018-07-11 07:32:33.150226 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6808/12651 1815 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:33.149493) v4 ==== 2004+0+0 (1947172494 0 0) 0x55a332fafe00 con 0x55a314e96000 > -9342> 2018-07-11 07:32:33.150231 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6811/15287 conn(0x55a314e04800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1815 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:33.149493) v4 > -9341> 2018-07-11 07:32:33.150260 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6811/15287 1815 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:33.149493) v4 ==== 2004+0+0 (1947172494 0 0) 0x55a332e79a00 con 0x55a314e04800 > -9340> 2018-07-11 07:32:33.150286 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6810/15287 conn(0x55a314e03000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1815 0x55a332fafe00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:33.149493) v4 > -9339> 2018-07-11 07:32:33.150301 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6810/15287 1815 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:33.149493) v4 ==== 2004+0+0 (1947172494 0 0) 0x55a332fafe00 con 0x55a314e03000 > -9338> 2018-07-11 07:32:33.150299 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6806/12617 conn(0x55a314e26000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1815 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:33.149493) v4 > -9337> 2018-07-11 07:32:33.150295 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6806/15263 conn(0x55a314e99000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1815 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:33.149493) v4 > -9336> 2018-07-11 07:32:33.150313 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6806/15263 1815 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:33.149493) v4 ==== 2004+0+0 (1947172494 0 0) 0x55a34de6f000 con 0x55a314e99000 > -9335> 2018-07-11 07:32:33.150313 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6806/12617 1815 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:33.149493) v4 ==== 2004+0+0 (1947172494 0 0) 0x55a332e79a00 con 0x55a314e26000 > -9334> 2018-07-11 07:32:33.150331 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6807/15263 conn(0x55a314e9a800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1815 0x55a332fafe00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:33.149493) v4 > -9333> 2018-07-11 07:32:33.150343 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6807/15263 1815 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:33.149493) v4 ==== 2004+0+0 (1947172494 0 0) 0x55a332fafe00 con 0x55a314e9a800 > -9332> 2018-07-11 07:32:33.150354 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6807/12617 conn(0x55a314e27800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1815 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:33.149493) v4 > -9331> 2018-07-11 07:32:33.150365 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6802/13323 conn(0x55a314e15000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1815 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:33.149493) v4 > -9330> 2018-07-11 07:32:33.150381 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6807/12617 1815 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:33.149493) v4 ==== 2004+0+0 (1947172494 0 0) 0x55a34de6f000 con 0x55a314e27800 > -9329> 2018-07-11 07:32:33.150385 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6802/13323 1815 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:33.149493) v4 ==== 2004+0+0 (1947172494 0 0) 0x55a332e79a00 con 0x55a314e15000 > -9328> 2018-07-11 07:32:33.150434 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6806/15885 conn(0x55a314e00000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1815 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:33.149493) v4 > -9327> 2018-07-11 07:32:33.150431 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6803/13323 conn(0x55a314e16800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1815 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:33.149493) v4 > -9326> 2018-07-11 07:32:33.150453 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6806/15885 1815 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:33.149493) v4 ==== 2004+0+0 (1947172494 0 0) 0x55a332e79a00 con 0x55a314e00000 > -9325> 2018-07-11 07:32:33.150458 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6803/13323 1815 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:33.149493) v4 ==== 2004+0+0 (1947172494 0 0) 0x55a34de6f000 con 0x55a314e16800 > -9324> 2018-07-11 07:32:33.150464 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6803/11967 conn(0x55a314e13800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1815 0x55a332fafe00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:33.149493) v4 > -9323> 2018-07-11 07:32:33.150480 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6803/11967 1815 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:33.149493) v4 ==== 2004+0+0 (1947172494 0 0) 0x55a332fafe00 con 0x55a314e13800 > -9322> 2018-07-11 07:32:33.150505 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6807/15885 conn(0x55a314e01800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1815 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:33.149493) v4 > -9321> 2018-07-11 07:32:33.150505 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6802/15919 conn(0x55a314e87000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1815 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:33.149493) v4 > -9320> 2018-07-11 07:32:33.150526 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6807/15885 1815 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:33.149493) v4 ==== 2004+0+0 (1947172494 0 0) 0x55a34de6f000 con 0x55a314e01800 > -9319> 2018-07-11 07:32:33.150532 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6802/15919 1815 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:33.149493) v4 ==== 2004+0+0 (1947172494 0 0) 0x55a332e79a00 con 0x55a314e87000 > -9318> 2018-07-11 07:32:33.150574 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6802/11967 conn(0x55a314e12000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1815 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:33.149493) v4 > -9317> 2018-07-11 07:32:33.150588 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6802/11967 1815 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:33.149493) v4 ==== 2004+0+0 (1947172494 0 0) 0x55a34de6f000 con 0x55a314e12000 > -9316> 2018-07-11 07:32:33.150622 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6803/15919 conn(0x55a314e88800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1815 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:33.149493) v4 > -9315> 2018-07-11 07:32:33.150634 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6803/15919 1815 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:33.149493) v4 ==== 2004+0+0 (1947172494 0 0) 0x55a34de6f000 con 0x55a314e88800 > -9314> 2018-07-11 07:32:33.364332 7f355d88e700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.107:0/12617 conn(0x55a32447f800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1788 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:32:33.363987) v4 > -9313> 2018-07-11 07:32:33.364332 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.107:0/12617 conn(0x55a32447e000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1788 0x55a332fafe00 osd_ping(ping e494 stamp 2018-07-11 07:32:33.363987) v4 > -9312> 2018-07-11 07:32:33.364356 7f355d88e700 1 -- 10.8.128.107:6803/12665 <== osd.9 10.8.128.107:0/12617 1788 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:33.363987) v4 ==== 2004+0+0 (2830155659 0 0) 0x55a34de6f000 con 0x55a32447f800 > -9311> 2018-07-11 07:32:33.364362 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.9 10.8.128.107:0/12617 1788 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:33.363987) v4 ==== 2004+0+0 (2830155659 0 0) 0x55a332fafe00 con 0x55a32447e000 > -9310> 2018-07-11 07:32:33.364367 7f355d88e700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:33.363987) v4 -- 0x55a3484ffa00 con 0 > -9309> 2018-07-11 07:32:33.364399 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:33.363987) v4 -- 0x55a34256c000 con 0 > -9308> 2018-07-11 07:32:33.516996 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12677 conn(0x55a32169d800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1837 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:32:33.551694) v4 > -9307> 2018-07-11 07:32:33.517013 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.1 10.8.128.106:0/12677 1837 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:33.551694) v4 ==== 2004+0+0 (2971051037 0 0) 0x55a34de78600 con 0x55a32169d800 > -9306> 2018-07-11 07:32:33.517022 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:33.551694) v4 -- 0x55a327786400 con 0 > -9305> 2018-07-11 07:32:33.517061 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12677 conn(0x55a32169c000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1837 0x55a33d672000 osd_ping(ping e494 stamp 2018-07-11 07:32:33.551694) v4 > -9304> 2018-07-11 07:32:33.517082 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.1 10.8.128.106:0/12677 1837 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:33.551694) v4 ==== 2004+0+0 (2971051037 0 0) 0x55a33d672000 con 0x55a32169c000 > -9303> 2018-07-11 07:32:33.517097 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:33.551694) v4 -- 0x55a3484ff000 con 0 > -9302> 2018-07-11 07:32:33.555120 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15885 conn(0x55a311b76800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1835 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:32:34.178372) v4 > -9301> 2018-07-11 07:32:33.555136 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.7 10.8.128.114:0/15885 1835 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:34.178372) v4 ==== 2004+0+0 (802610057 0 0) 0x55a34de6f000 con 0x55a311b76800 > -9300> 2018-07-11 07:32:33.555145 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:34.178372) v4 -- 0x55a323f55800 con 0 > -9299> 2018-07-11 07:32:33.555144 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15885 conn(0x55a3170b5800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1835 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:32:34.178372) v4 > -9298> 2018-07-11 07:32:33.555161 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.7 10.8.128.114:0/15885 1835 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:34.178372) v4 ==== 2004+0+0 (802610057 0 0) 0x55a332e79a00 con 0x55a3170b5800 > -9297> 2018-07-11 07:32:33.555175 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:34.178372) v4 -- 0x55a3484fee00 con 0 > -9296> 2018-07-11 07:32:34.116629 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15255 conn(0x55a311b65800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1840 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:32:35.328986) v4 > -9295> 2018-07-11 07:32:34.116641 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15255 conn(0x55a311b64000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1840 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:32:35.328986) v4 > -9294> 2018-07-11 07:32:34.116650 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.0 10.8.128.113:0/15255 1840 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:35.328986) v4 ==== 2004+0+0 (77887316 0 0) 0x55a324474c00 con 0x55a311b65800 > -9293> 2018-07-11 07:32:34.116664 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.0 10.8.128.113:0/15255 1840 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:35.328986) v4 ==== 2004+0+0 (77887316 0 0) 0x55a34de78600 con 0x55a311b64000 > -9292> 2018-07-11 07:32:34.116670 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:35.328986) v4 -- 0x55a3484fec00 con 0 > -9291> 2018-07-11 07:32:34.116696 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:35.328986) v4 -- 0x55a349f88a00 con 0 > -9290> 2018-07-11 07:32:34.153197 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12651 conn(0x55a3216a0800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1819 0x55a332fafe00 osd_ping(ping e494 stamp 2018-07-11 07:32:34.187786) v4 > -9289> 2018-07-11 07:32:34.153222 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.2 10.8.128.106:0/12651 1819 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:34.187786) v4 ==== 2004+0+0 (3884324242 0 0) 0x55a332fafe00 con 0x55a3216a0800 > -9288> 2018-07-11 07:32:34.153216 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12651 conn(0x55a32169f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1819 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:32:34.187786) v4 > -9287> 2018-07-11 07:32:34.153231 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:34.187786) v4 -- 0x55a3484ff000 con 0 > -9286> 2018-07-11 07:32:34.153244 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.2 10.8.128.106:0/12651 1819 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:34.187786) v4 ==== 2004+0+0 (3884324242 0 0) 0x55a332e79a00 con 0x55a32169f000 > -9285> 2018-07-11 07:32:34.153274 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:34.187786) v4 -- 0x55a3484fec00 con 0 > -9284> 2018-07-11 07:32:34.164705 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12650 conn(0x55a311b67000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1831 0x55a33d672000 osd_ping(ping e494 stamp 2018-07-11 07:32:34.199386) v4 > -9283> 2018-07-11 07:32:34.164743 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.3 10.8.128.106:0/12650 1831 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:34.199386) v4 ==== 2004+0+0 (1794938735 0 0) 0x55a33d672000 con 0x55a311b67000 > -9282> 2018-07-11 07:32:34.164758 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:34.199386) v4 -- 0x55a3484ff000 con 0 > -9281> 2018-07-11 07:32:34.164835 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12650 conn(0x55a311b68800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1831 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:32:34.199386) v4 > -9280> 2018-07-11 07:32:34.164853 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.3 10.8.128.106:0/12650 1831 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:34.199386) v4 ==== 2004+0+0 (1794938735 0 0) 0x55a34de6f000 con 0x55a311b68800 > -9279> 2018-07-11 07:32:34.164863 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:34.199386) v4 -- 0x55a349f88a00 con 0 > -9278> 2018-07-11 07:32:34.244108 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 58] Generated table #36565: 4957168 keys, 80348310 bytes > -9277> 2018-07-11 07:32:34.244163 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294354244135, "cf_name": "default", "job": 58, "event": "table_file_creation", "file_number": 36565, "file_size": 80348310, "table_properties": {"data_size": 67109354, "index_size": 689256, "filter_size": 12548710, "raw_key_size": 143207073, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957168, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957168", "kMergeOperands": "0"}} > -9276> 2018-07-11 07:32:34.694141 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a34dc6db00 con 0 > -9275> 2018-07-11 07:32:34.713169 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5018 0x55a33f8cf180 replica scrub(pg: 6.15,from:0'0,to:430'106809,epoch:494/492,start:6:a980bf23:::benchmark_data_magna108.ceph.redhat.com_2478_object439649:0,end:6:a981d80e:::benchmark_data_magna108_35617_object933005:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -9274> 2018-07-11 07:32:34.713193 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5018 ==== replica scrub(pg: 6.15,from:0'0,to:430'106809,epoch:494/492,start:6:a980bf23:::benchmark_data_magna108.ceph.redhat.com_2478_object439649:0,end:6:a981d80e:::benchmark_data_magna108_35617_object933005:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (2809725502 0 0) 0x55a33f8cf180 con 0x55a317019000 > -9273> 2018-07-11 07:32:34.746986 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15919 conn(0x55a317074800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1822 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:32:35.370259) v4 > -9272> 2018-07-11 07:32:34.747008 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.11 10.8.128.114:0/15919 1822 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:35.370259) v4 ==== 2004+0+0 (4007963929 0 0) 0x55a324474c00 con 0x55a317074800 > -9271> 2018-07-11 07:32:34.747021 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:35.370259) v4 -- 0x55a3484fec00 con 0 > -9270> 2018-07-11 07:32:34.747076 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15919 conn(0x55a3170b4000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1822 0x55a332fafe00 osd_ping(ping e494 stamp 2018-07-11 07:32:35.370259) v4 > -9269> 2018-07-11 07:32:34.747103 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.11 10.8.128.114:0/15919 1822 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:35.370259) v4 ==== 2004+0+0 (4007963929 0 0) 0x55a332fafe00 con 0x55a3170b4000 > -9268> 2018-07-11 07:32:34.747119 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:35.370259) v4 -- 0x55a3484ff000 con 0 > -9267> 2018-07-11 07:32:35.064957 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.107:0/12617 conn(0x55a32447e000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1789 0x55a33d672000 osd_ping(ping e494 stamp 2018-07-11 07:32:35.064481) v4 > -9266> 2018-07-11 07:32:35.064959 7f355d88e700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.107:0/12617 conn(0x55a32447f800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1789 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:32:35.064481) v4 > -9265> 2018-07-11 07:32:35.064987 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.9 10.8.128.107:0/12617 1789 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:35.064481) v4 ==== 2004+0+0 (3798479123 0 0) 0x55a33d672000 con 0x55a32447e000 > -9264> 2018-07-11 07:32:35.064992 7f355d88e700 1 -- 10.8.128.107:6803/12665 <== osd.9 10.8.128.107:0/12617 1789 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:35.064481) v4 ==== 2004+0+0 (3798479123 0 0) 0x55a34de78600 con 0x55a32447f800 > -9263> 2018-07-11 07:32:35.065007 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:35.064481) v4 -- 0x55a3484ff000 con 0 > -9262> 2018-07-11 07:32:35.065032 7f355d88e700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:35.064481) v4 -- 0x55a349f88a00 con 0 > -9261> 2018-07-11 07:32:35.106131 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15263 conn(0x55a321664000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1811 0x55a332fafe00 osd_ping(ping e494 stamp 2018-07-11 07:32:36.318520) v4 > -9260> 2018-07-11 07:32:35.106159 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.5 10.8.128.113:0/15263 1811 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:36.318520) v4 ==== 2004+0+0 (3618683448 0 0) 0x55a332fafe00 con 0x55a321664000 > -9259> 2018-07-11 07:32:35.106172 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:36.318520) v4 -- 0x55a3484ff000 con 0 > -9258> 2018-07-11 07:32:35.106266 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15263 conn(0x55a321667000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1811 0x55a33d672000 osd_ping(ping e494 stamp 2018-07-11 07:32:36.318520) v4 > -9257> 2018-07-11 07:32:35.106283 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.5 10.8.128.113:0/15263 1811 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:36.318520) v4 ==== 2004+0+0 (3618683448 0 0) 0x55a33d672000 con 0x55a321667000 > -9256> 2018-07-11 07:32:35.106291 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:36.318520) v4 -- 0x55a3484fe800 con 0 > -9255> 2018-07-11 07:32:35.236943 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.115:0/13323 conn(0x55a321665800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1828 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:32:35.573723) v4 > -9254> 2018-07-11 07:32:35.236965 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.6 10.8.128.115:0/13323 1828 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:35.573723) v4 ==== 2004+0+0 (3177294384 0 0) 0x55a332e79a00 con 0x55a321665800 > -9253> 2018-07-11 07:32:35.236975 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:35.573723) v4 -- 0x55a3131bd600 con 0 > -9252> 2018-07-11 07:32:35.237036 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.115:0/13323 conn(0x55a321668800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1828 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:32:35.573723) v4 > -9251> 2018-07-11 07:32:35.237050 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.6 10.8.128.115:0/13323 1828 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:35.573723) v4 ==== 2004+0+0 (3177294384 0 0) 0x55a324474c00 con 0x55a321668800 > -9250> 2018-07-11 07:32:35.237060 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:35.573723) v4 -- 0x55a343718200 con 0 > -9249> 2018-07-11 07:32:35.361543 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a31f877680 con 0 > -9248> 2018-07-11 07:32:35.384995 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5019 0x55a33f8cf400 replica scrub(pg: 6.15,from:0'0,to:430'106824,epoch:494/492,start:6:a981d80e:::benchmark_data_magna108_35617_object933005:0,end:6:a983243f:::benchmark_data_magna108_35617_object694233:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -9247> 2018-07-11 07:32:35.385018 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5019 ==== replica scrub(pg: 6.15,from:0'0,to:430'106824,epoch:494/492,start:6:a981d80e:::benchmark_data_magna108_35617_object933005:0,end:6:a983243f:::benchmark_data_magna108_35617_object694233:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (1302805834 0 0) 0x55a33f8cf400 con 0x55a317019000 > -9246> 2018-07-11 07:32:35.417812 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15287 conn(0x55a311b75000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1809 0x55a332fafe00 osd_ping(ping e494 stamp 2018-07-11 07:32:36.630098) v4 > -9245> 2018-07-11 07:32:35.417818 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15287 conn(0x55a311b72000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1809 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:32:36.630098) v4 > -9244> 2018-07-11 07:32:35.417833 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.4 10.8.128.113:0/15287 1809 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:36.630098) v4 ==== 2004+0+0 (3841938501 0 0) 0x55a332fafe00 con 0x55a311b75000 > -9243> 2018-07-11 07:32:35.417835 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.4 10.8.128.113:0/15287 1809 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:36.630098) v4 ==== 2004+0+0 (3841938501 0 0) 0x55a332e79a00 con 0x55a311b72000 > -9242> 2018-07-11 07:32:35.417849 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:36.630098) v4 -- 0x55a33d672200 con 0 > -9241> 2018-07-11 07:32:35.417871 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:36.630098) v4 -- 0x55a343718200 con 0 > -9240> 2018-07-11 07:32:35.450172 7f3543899700 5 osd.10 494 heartbeat: osd_stat(783 GB used, 148 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,1]) > -9239> 2018-07-11 07:32:35.450186 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6802/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:32:35.450184) v4 -- 0x55a3456b0200 con 0 > -9238> 2018-07-11 07:32:35.450223 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6803/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:32:35.450184) v4 -- 0x55a3456b0000 con 0 > -9237> 2018-07-11 07:32:35.450260 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6806/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:32:35.450184) v4 -- 0x55a3456b0800 con 0 > -9236> 2018-07-11 07:32:35.450289 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6807/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:32:35.450184) v4 -- 0x55a3456b0600 con 0 > -9235> 2018-07-11 07:32:35.450310 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6808/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:32:35.450184) v4 -- 0x55a31d142600 con 0 > -9234> 2018-07-11 07:32:35.450330 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6810/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:32:35.450184) v4 -- 0x55a34256d200 con 0 > -9233> 2018-07-11 07:32:35.450355 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6809/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:32:35.450184) v4 -- 0x55a327982600 con 0 > -9232> 2018-07-11 07:32:35.450381 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6811/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:32:35.450184) v4 -- 0x55a327c0ec00 con 0 > -9231> 2018-07-11 07:32:35.450399 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6810/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:32:35.450184) v4 -- 0x55a31d142800 con 0 > -9230> 2018-07-11 07:32:35.450417 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6811/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:32:35.450184) v4 -- 0x55a327982400 con 0 > -9229> 2018-07-11 07:32:35.450463 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6806/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:32:35.450184) v4 -- 0x55a34256d600 con 0 > -9228> 2018-07-11 07:32:35.450486 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6807/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:32:35.450184) v4 -- 0x55a34256d400 con 0 > -9227> 2018-07-11 07:32:35.450502 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6802/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:32:35.450184) v4 -- 0x55a34256dc00 con 0 > -9226> 2018-07-11 07:32:35.450527 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6803/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:32:35.450184) v4 -- 0x55a34256d800 con 0 > -9225> 2018-07-11 07:32:35.450552 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6806/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:32:35.450184) v4 -- 0x55a34256de00 con 0 > -9224> 2018-07-11 07:32:35.450576 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6802/15255 conn(0x55a314e1c000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1816 0x55a332fafe00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:35.450184) v4 > -9223> 2018-07-11 07:32:35.450599 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6802/15255 1816 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:35.450184) v4 ==== 2004+0+0 (3160041786 0 0) 0x55a332fafe00 con 0x55a314e1c000 > -9222> 2018-07-11 07:32:35.450607 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6807/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:32:35.450184) v4 -- 0x55a34256da00 con 0 > -9221> 2018-07-11 07:32:35.450626 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6802/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:32:35.450184) v4 -- 0x55a31dbc2600 con 0 > -9220> 2018-07-11 07:32:35.450621 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6803/15255 conn(0x55a314e1d800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1816 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:35.450184) v4 > -9219> 2018-07-11 07:32:35.450639 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6803/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:32:35.450184) v4 -- 0x55a31dbc2000 con 0 > -9218> 2018-07-11 07:32:35.450645 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6803/15255 1816 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:35.450184) v4 ==== 2004+0+0 (3160041786 0 0) 0x55a324474c00 con 0x55a314e1d800 > -9217> 2018-07-11 07:32:35.450662 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6806/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:32:35.450184) v4 -- 0x55a31dbc2c00 con 0 > -9216> 2018-07-11 07:32:35.450675 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6807/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:32:35.450184) v4 -- 0x55a31dbc2400 con 0 > -9215> 2018-07-11 07:32:35.450687 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6802/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:32:35.450184) v4 -- 0x55a319404400 con 0 > -9214> 2018-07-11 07:32:35.450697 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6803/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:32:35.450184) v4 -- 0x55a31d143200 con 0 > -9213> 2018-07-11 07:32:35.450763 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6809/12650 conn(0x55a314e1f000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1816 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:35.450184) v4 > -9212> 2018-07-11 07:32:35.450776 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6808/12651 conn(0x55a314e96000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1816 0x55a3484fe800 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:35.450184) v4 > -9211> 2018-07-11 07:32:35.450791 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6808/12651 1816 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:35.450184) v4 ==== 2004+0+0 (3160041786 0 0) 0x55a3484fe800 con 0x55a314e96000 > -9210> 2018-07-11 07:32:35.450786 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6809/12650 1816 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:35.450184) v4 ==== 2004+0+0 (3160041786 0 0) 0x55a34de78600 con 0x55a314e1f000 > -9209> 2018-07-11 07:32:35.450831 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6807/12677 conn(0x55a314e85800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1816 0x55a3484fe800 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:35.450184) v4 > -9208> 2018-07-11 07:32:35.450844 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6807/12677 1816 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:35.450184) v4 ==== 2004+0+0 (3160041786 0 0) 0x55a3484fe800 con 0x55a314e85800 > -9207> 2018-07-11 07:32:35.450851 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6810/12651 conn(0x55a314e97800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1816 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:35.450184) v4 > -9206> 2018-07-11 07:32:35.450873 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6810/12651 1816 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:35.450184) v4 ==== 2004+0+0 (3160041786 0 0) 0x55a332e79a00 con 0x55a314e97800 > -9205> 2018-07-11 07:32:35.450877 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6811/12650 conn(0x55a314e20800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1816 0x55a3484fe800 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:35.450184) v4 > -9204> 2018-07-11 07:32:35.450889 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6811/12650 1816 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:35.450184) v4 ==== 2004+0+0 (3160041786 0 0) 0x55a3484fe800 con 0x55a314e20800 > -9203> 2018-07-11 07:32:35.450877 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6806/12677 conn(0x55a314e84000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1816 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:35.450184) v4 > -9202> 2018-07-11 07:32:35.450914 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6806/12677 1816 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:35.450184) v4 ==== 2004+0+0 (3160041786 0 0) 0x55a34de6f000 con 0x55a314e84000 > -9201> 2018-07-11 07:32:35.450920 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6807/15263 conn(0x55a314e9a800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1816 0x55a3484fe800 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:35.450184) v4 > -9200> 2018-07-11 07:32:35.450921 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6811/15287 conn(0x55a314e04800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1816 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:35.450184) v4 > -9199> 2018-07-11 07:32:35.450931 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6807/15263 1816 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:35.450184) v4 ==== 2004+0+0 (3160041786 0 0) 0x55a3484fe800 con 0x55a314e9a800 > -9198> 2018-07-11 07:32:35.450944 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6811/15287 1816 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:35.450184) v4 ==== 2004+0+0 (3160041786 0 0) 0x55a332e79a00 con 0x55a314e04800 > -9197> 2018-07-11 07:32:35.450974 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6810/15287 conn(0x55a314e03000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1816 0x55a3484fe800 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:35.450184) v4 > -9196> 2018-07-11 07:32:35.450952 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6807/12617 conn(0x55a314e27800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1816 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:35.450184) v4 > -9195> 2018-07-11 07:32:35.450991 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6810/15287 1816 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:35.450184) v4 ==== 2004+0+0 (3160041786 0 0) 0x55a3484fe800 con 0x55a314e03000 > -9194> 2018-07-11 07:32:35.450996 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6802/13323 conn(0x55a314e15000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1816 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:35.450184) v4 > -9193> 2018-07-11 07:32:35.450999 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6807/12617 1816 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:35.450184) v4 ==== 2004+0+0 (3160041786 0 0) 0x55a34de6f000 con 0x55a314e27800 > -9192> 2018-07-11 07:32:35.451016 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6802/13323 1816 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:35.450184) v4 ==== 2004+0+0 (3160041786 0 0) 0x55a332e79a00 con 0x55a314e15000 > -9191> 2018-07-11 07:32:35.451053 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6806/12617 conn(0x55a314e26000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1816 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:35.450184) v4 > -9190> 2018-07-11 07:32:35.451068 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6803/13323 conn(0x55a314e16800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1816 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:35.450184) v4 > -9189> 2018-07-11 07:32:35.451078 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6806/12617 1816 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:35.450184) v4 ==== 2004+0+0 (3160041786 0 0) 0x55a332e79a00 con 0x55a314e26000 > -9188> 2018-07-11 07:32:35.451084 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6803/13323 1816 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:35.450184) v4 ==== 2004+0+0 (3160041786 0 0) 0x55a34de6f000 con 0x55a314e16800 > -9187> 2018-07-11 07:32:35.451130 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6806/15263 conn(0x55a314e99000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1816 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:35.450184) v4 > -9186> 2018-07-11 07:32:35.451145 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6806/15263 1816 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:35.450184) v4 ==== 2004+0+0 (3160041786 0 0) 0x55a34de6f000 con 0x55a314e99000 > -9185> 2018-07-11 07:32:35.451174 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6803/11967 conn(0x55a314e13800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1816 0x55a3484fe800 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:35.450184) v4 > -9184> 2018-07-11 07:32:35.451190 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6803/11967 1816 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:35.450184) v4 ==== 2004+0+0 (3160041786 0 0) 0x55a3484fe800 con 0x55a314e13800 > -9183> 2018-07-11 07:32:35.451194 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6802/15919 conn(0x55a314e87000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1816 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:35.450184) v4 > -9182> 2018-07-11 07:32:35.451201 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6802/11967 conn(0x55a314e12000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1816 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:35.450184) v4 > -9181> 2018-07-11 07:32:35.451217 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6802/15919 1816 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:35.450184) v4 ==== 2004+0+0 (3160041786 0 0) 0x55a332e79a00 con 0x55a314e87000 > -9180> 2018-07-11 07:32:35.451220 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6802/11967 1816 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:35.450184) v4 ==== 2004+0+0 (3160041786 0 0) 0x55a34de6f000 con 0x55a314e12000 > -9179> 2018-07-11 07:32:35.451272 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6807/15885 conn(0x55a314e01800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1816 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:35.450184) v4 > -9178> 2018-07-11 07:32:35.451272 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6806/15885 conn(0x55a314e00000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1816 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:35.450184) v4 > -9177> 2018-07-11 07:32:35.451293 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6807/15885 1816 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:35.450184) v4 ==== 2004+0+0 (3160041786 0 0) 0x55a34de6f000 con 0x55a314e01800 > -9176> 2018-07-11 07:32:35.451299 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6806/15885 1816 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:35.450184) v4 ==== 2004+0+0 (3160041786 0 0) 0x55a332e79a00 con 0x55a314e00000 > -9175> 2018-07-11 07:32:35.451343 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6803/15919 conn(0x55a314e88800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1816 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:35.450184) v4 > -9174> 2018-07-11 07:32:35.451358 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6803/15919 1816 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:35.450184) v4 ==== 2004+0+0 (3160041786 0 0) 0x55a34de6f000 con 0x55a314e88800 > -9173> 2018-07-11 07:32:35.606488 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15263 conn(0x55a321667000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1812 0x55a3484fe800 osd_ping(ping e494 stamp 2018-07-11 07:32:36.818904) v4 > -9172> 2018-07-11 07:32:35.606512 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.5 10.8.128.113:0/15263 1812 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:36.818904) v4 ==== 2004+0+0 (946746876 0 0) 0x55a3484fe800 con 0x55a321667000 > -9171> 2018-07-11 07:32:35.606525 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:36.818904) v4 -- 0x55a3456b0200 con 0 > -9170> 2018-07-11 07:32:35.606575 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15263 conn(0x55a321664000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1812 0x55a332fafe00 osd_ping(ping e494 stamp 2018-07-11 07:32:36.818904) v4 > -9169> 2018-07-11 07:32:35.606588 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.5 10.8.128.113:0/15263 1812 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:36.818904) v4 ==== 2004+0+0 (946746876 0 0) 0x55a332fafe00 con 0x55a321664000 > -9168> 2018-07-11 07:32:35.606601 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:36.818904) v4 -- 0x55a3456b0200 con 0 > -9167> 2018-07-11 07:32:35.836745 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b7000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1766 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:32:35.586923) v4 > -9166> 2018-07-11 07:32:35.836767 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.8 10.8.128.108:0/11967 1766 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:35.586923) v4 ==== 2004+0+0 (4275396206 0 0) 0x55a34de6f000 con 0x55a3170b7000 > -9165> 2018-07-11 07:32:35.836778 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:35.586923) v4 -- 0x55a3456b0800 con 0 > -9164> 2018-07-11 07:32:35.836771 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b8800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1766 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:32:35.586923) v4 > -9163> 2018-07-11 07:32:35.836788 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.8 10.8.128.108:0/11967 1766 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:35.586923) v4 ==== 2004+0+0 (4275396206 0 0) 0x55a332e79a00 con 0x55a3170b8800 > -9162> 2018-07-11 07:32:35.836802 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:35.586923) v4 -- 0x55a3456b0000 con 0 > -9161> 2018-07-11 07:32:36.119140 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a33071e240 con 0 > -9160> 2018-07-11 07:32:36.138291 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5020 0x55a33f8cf680 replica scrub(pg: 6.15,from:0'0,to:430'106819,epoch:494/492,start:6:a983243f:::benchmark_data_magna108_35617_object694233:0,end:6:a984666f:::benchmark_data_magna108_35617_object704801:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -9159> 2018-07-11 07:32:36.138314 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5020 ==== replica scrub(pg: 6.15,from:0'0,to:430'106819,epoch:494/492,start:6:a983243f:::benchmark_data_magna108_35617_object694233:0,end:6:a984666f:::benchmark_data_magna108_35617_object704801:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (3471564931 0 0) 0x55a33f8cf680 con 0x55a317019000 > -9158> 2018-07-11 07:32:36.706946 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15263 conn(0x55a321664000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1813 0x55a3484fe800 osd_ping(ping e494 stamp 2018-07-11 07:32:37.919318) v4 > -9157> 2018-07-11 07:32:36.706975 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.5 10.8.128.113:0/15263 1813 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:37.919318) v4 ==== 2004+0+0 (3991842546 0 0) 0x55a3484fe800 con 0x55a321664000 > -9156> 2018-07-11 07:32:36.706993 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:37.919318) v4 -- 0x55a3456b0200 con 0 > -9155> 2018-07-11 07:32:36.707078 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15263 conn(0x55a321667000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1813 0x55a332fafe00 osd_ping(ping e494 stamp 2018-07-11 07:32:37.919318) v4 > -9154> 2018-07-11 07:32:36.707099 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.5 10.8.128.113:0/15263 1813 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:37.919318) v4 ==== 2004+0+0 (3991842546 0 0) 0x55a332fafe00 con 0x55a321667000 > -9153> 2018-07-11 07:32:36.707114 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:37.919318) v4 -- 0x55a3456b0200 con 0 > -9152> 2018-07-11 07:32:36.743428 7f35498a5700 10 monclient: tick > -9151> 2018-07-11 07:32:36.743441 7f35498a5700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2018-07-11 07:32:06.743439) > -9150> 2018-07-11 07:32:36.835353 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a334eaafc0 con 0 > -9149> 2018-07-11 07:32:36.859528 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5021 0x55a33f8cf900 replica scrub(pg: 6.15,from:0'0,to:430'106713,epoch:494/492,start:6:a984666f:::benchmark_data_magna108_35617_object704801:0,end:6:a9855fff:::benchmark_data_magna108_41525_object13430:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -9148> 2018-07-11 07:32:36.859558 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5021 ==== replica scrub(pg: 6.15,from:0'0,to:430'106713,epoch:494/492,start:6:a984666f:::benchmark_data_magna108_35617_object704801:0,end:6:a9855fff:::benchmark_data_magna108_41525_object13430:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 217+0+0 (1740920061 0 0) 0x55a33f8cf900 con 0x55a317019000 > -9147> 2018-07-11 07:32:37.047431 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15919 conn(0x55a317074800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1823 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:32:37.670697) v4 > -9146> 2018-07-11 07:32:37.047437 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15919 conn(0x55a3170b4000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1823 0x55a332fafe00 osd_ping(ping e494 stamp 2018-07-11 07:32:37.670697) v4 > -9145> 2018-07-11 07:32:37.047453 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.11 10.8.128.114:0/15919 1823 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:37.670697) v4 ==== 2004+0+0 (3285075827 0 0) 0x55a324474c00 con 0x55a317074800 > -9144> 2018-07-11 07:32:37.047463 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:37.670697) v4 -- 0x55a3456b0000 con 0 > -9143> 2018-07-11 07:32:37.047458 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.11 10.8.128.114:0/15919 1823 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:37.670697) v4 ==== 2004+0+0 (3285075827 0 0) 0x55a332fafe00 con 0x55a3170b4000 > -9142> 2018-07-11 07:32:37.047488 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:37.670697) v4 -- 0x55a33d672200 con 0 > -9141> 2018-07-11 07:32:37.249167 7f35488a3700 1 -- 10.8.128.107:6800/12665 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55a317050000 con 0 > -9140> 2018-07-11 07:32:37.249238 7f35488a3700 1 -- 10.8.128.107:6800/12665 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55a32e1bf8c0 con 0 > -9139> 2018-07-11 07:32:37.537472 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.115:0/13323 conn(0x55a321665800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1829 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:32:37.874179) v4 > -9138> 2018-07-11 07:32:37.537502 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.6 10.8.128.115:0/13323 1829 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:37.874179) v4 ==== 2004+0+0 (3714023564 0 0) 0x55a332e79a00 con 0x55a321665800 > -9137> 2018-07-11 07:32:37.537520 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:37.874179) v4 -- 0x55a3456b0000 con 0 > -9136> 2018-07-11 07:32:37.537608 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.115:0/13323 conn(0x55a321668800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1829 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:32:37.874179) v4 > -9135> 2018-07-11 07:32:37.537625 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.6 10.8.128.115:0/13323 1829 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:37.874179) v4 ==== 2004+0+0 (3714023564 0 0) 0x55a324474c00 con 0x55a321668800 > -9134> 2018-07-11 07:32:37.537634 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:37.874179) v4 -- 0x55a343718200 con 0 > -9133> 2018-07-11 07:32:37.617463 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12677 conn(0x55a32169d800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1838 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:32:37.652238) v4 > -9132> 2018-07-11 07:32:37.617485 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.1 10.8.128.106:0/12677 1838 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:37.652238) v4 ==== 2004+0+0 (56730048 0 0) 0x55a34de78600 con 0x55a32169d800 > -9131> 2018-07-11 07:32:37.617494 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:37.652238) v4 -- 0x55a349f88a00 con 0 > -9130> 2018-07-11 07:32:37.617526 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12677 conn(0x55a32169c000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1838 0x55a3456b0200 osd_ping(ping e494 stamp 2018-07-11 07:32:37.652238) v4 > -9129> 2018-07-11 07:32:37.617557 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.1 10.8.128.106:0/12677 1838 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:37.652238) v4 ==== 2004+0+0 (56730048 0 0) 0x55a3456b0200 con 0x55a32169c000 > -9128> 2018-07-11 07:32:37.617574 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:37.652238) v4 -- 0x55a314575c00 con 0 > -9127> 2018-07-11 07:32:37.653762 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12651 conn(0x55a32169f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1820 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:32:37.688331) v4 > -9126> 2018-07-11 07:32:37.653784 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.2 10.8.128.106:0/12651 1820 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:37.688331) v4 ==== 2004+0+0 (2134230272 0 0) 0x55a332e79a00 con 0x55a32169f000 > -9125> 2018-07-11 07:32:37.653798 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:37.688331) v4 -- 0x55a3131bd800 con 0 > -9124> 2018-07-11 07:32:37.653796 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12651 conn(0x55a3216a0800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1820 0x55a332fafe00 osd_ping(ping e494 stamp 2018-07-11 07:32:37.688331) v4 > -9123> 2018-07-11 07:32:37.653825 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.2 10.8.128.106:0/12651 1820 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:37.688331) v4 ==== 2004+0+0 (2134230272 0 0) 0x55a332fafe00 con 0x55a3216a0800 > -9122> 2018-07-11 07:32:37.653836 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:37.688331) v4 -- 0x55a314575200 con 0 > -9121> 2018-07-11 07:32:37.683618 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a319058900 con 0 > -9120> 2018-07-11 07:32:37.696800 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5022 0x55a33f8cfb80 replica scrub(pg: 6.15,from:0'0,to:430'106698,epoch:494/492,start:6:a9855fff:::benchmark_data_magna108_41525_object13430:0,end:6:a98667e9:::benchmark_data_magna108_35617_object1042730:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -9119> 2018-07-11 07:32:37.696821 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5022 ==== replica scrub(pg: 6.15,from:0'0,to:430'106698,epoch:494/492,start:6:a9855fff:::benchmark_data_magna108_41525_object13430:0,end:6:a98667e9:::benchmark_data_magna108_35617_object1042730:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (1470119448 0 0) 0x55a33f8cfb80 con 0x55a317019000 > -9118> 2018-07-11 07:32:38.154219 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12651 conn(0x55a3216a0800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1821 0x55a3456b0200 osd_ping(ping e494 stamp 2018-07-11 07:32:38.188892) v4 > -9117> 2018-07-11 07:32:38.154258 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.2 10.8.128.106:0/12651 1821 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:38.188892) v4 ==== 2004+0+0 (3004436405 0 0) 0x55a3456b0200 con 0x55a3216a0800 > -9116> 2018-07-11 07:32:38.154271 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:38.188892) v4 -- 0x55a314575200 con 0 > -9115> 2018-07-11 07:32:38.154268 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12651 conn(0x55a32169f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1821 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:32:38.188892) v4 > -9114> 2018-07-11 07:32:38.154291 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.2 10.8.128.106:0/12651 1821 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:38.188892) v4 ==== 2004+0+0 (3004436405 0 0) 0x55a324474c00 con 0x55a32169f000 > -9113> 2018-07-11 07:32:38.154310 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:38.188892) v4 -- 0x55a3131bd800 con 0 > -9112> 2018-07-11 07:32:38.399087 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a34adbb680 con 0 > -9111> 2018-07-11 07:32:38.425642 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5023 0x55a313a94000 replica scrub(pg: 6.15,from:0'0,to:430'106699,epoch:494/492,start:6:a98667e9:::benchmark_data_magna108_35617_object1042730:0,end:6:a9877b91:::benchmark_data_magna108_35617_object1064648:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -9110> 2018-07-11 07:32:38.425672 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5023 ==== replica scrub(pg: 6.15,from:0'0,to:430'106699,epoch:494/492,start:6:a98667e9:::benchmark_data_magna108_35617_object1042730:0,end:6:a9877b91:::benchmark_data_magna108_35617_object1064648:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 220+0+0 (1615084525 0 0) 0x55a313a94000 con 0x55a317019000 > -9109> 2018-07-11 07:32:38.855569 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15885 conn(0x55a311b76800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1836 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:32:39.478830) v4 > -9108> 2018-07-11 07:32:38.855567 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15885 conn(0x55a3170b5800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1836 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:32:39.478830) v4 > -9107> 2018-07-11 07:32:38.855598 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.7 10.8.128.114:0/15885 1836 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:39.478830) v4 ==== 2004+0+0 (3837457540 0 0) 0x55a34de6f000 con 0x55a311b76800 > -9106> 2018-07-11 07:32:38.855601 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.7 10.8.128.114:0/15885 1836 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:39.478830) v4 ==== 2004+0+0 (3837457540 0 0) 0x55a332e79a00 con 0x55a3170b5800 > -9105> 2018-07-11 07:32:38.855617 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:39.478830) v4 -- 0x55a349f88a00 con 0 > -9104> 2018-07-11 07:32:38.855653 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:39.478830) v4 -- 0x55a3131bd800 con 0 > -9103> 2018-07-11 07:32:38.918283 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15287 conn(0x55a311b75000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1810 0x55a332fafe00 osd_ping(ping e494 stamp 2018-07-11 07:32:40.130688) v4 > -9102> 2018-07-11 07:32:38.918294 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15287 conn(0x55a311b72000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1810 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:32:40.130688) v4 > -9101> 2018-07-11 07:32:38.918308 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.4 10.8.128.113:0/15287 1810 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:40.130688) v4 ==== 2004+0+0 (2470212759 0 0) 0x55a332fafe00 con 0x55a311b75000 > -9100> 2018-07-11 07:32:38.918311 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.4 10.8.128.113:0/15287 1810 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:40.130688) v4 ==== 2004+0+0 (2470212759 0 0) 0x55a324474c00 con 0x55a311b72000 > -9099> 2018-07-11 07:32:38.918319 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:40.130688) v4 -- 0x55a33d672800 con 0 > -9098> 2018-07-11 07:32:38.918343 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:40.130688) v4 -- 0x55a3131bd800 con 0 > -9097> 2018-07-11 07:32:39.165453 7f355d88e700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.107:0/12617 conn(0x55a32447f800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1790 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:32:39.165096) v4 > -9096> 2018-07-11 07:32:39.165468 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.107:0/12617 conn(0x55a32447e000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1790 0x55a314575200 osd_ping(ping e494 stamp 2018-07-11 07:32:39.165096) v4 > -9095> 2018-07-11 07:32:39.165474 7f355d88e700 1 -- 10.8.128.107:6803/12665 <== osd.9 10.8.128.107:0/12617 1790 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:39.165096) v4 ==== 2004+0+0 (1794908546 0 0) 0x55a34de78600 con 0x55a32447f800 > -9094> 2018-07-11 07:32:39.165494 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.9 10.8.128.107:0/12617 1790 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:39.165096) v4 ==== 2004+0+0 (1794908546 0 0) 0x55a314575200 con 0x55a32447e000 > -9093> 2018-07-11 07:32:39.165499 7f355d88e700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:39.165096) v4 -- 0x55a349f88800 con 0 > -9092> 2018-07-11 07:32:39.165549 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:39.165096) v4 -- 0x55a332fafa00 con 0 > -9091> 2018-07-11 07:32:39.212379 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a323e1bb00 con 0 > -9090> 2018-07-11 07:32:39.230449 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5024 0x55a313a94280 replica scrub(pg: 6.15,from:0'0,to:430'105941,epoch:494/492,start:6:a9877b91:::benchmark_data_magna108_35617_object1064648:0,end:6:a98852d5:::benchmark_data_magna108_35617_object798025:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -9089> 2018-07-11 07:32:39.230479 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5024 ==== replica scrub(pg: 6.15,from:0'0,to:430'105941,epoch:494/492,start:6:a9877b91:::benchmark_data_magna108_35617_object1064648:0,end:6:a98852d5:::benchmark_data_magna108_35617_object798025:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 219+0+0 (1557441531 0 0) 0x55a313a94280 con 0x55a317019000 > -9088> 2018-07-11 07:32:39.337305 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b8800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1767 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:32:39.087547) v4 > -9087> 2018-07-11 07:32:39.337327 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.8 10.8.128.108:0/11967 1767 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:39.087547) v4 ==== 2004+0+0 (3785990764 0 0) 0x55a332e79a00 con 0x55a3170b8800 > -9086> 2018-07-11 07:32:39.337321 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b7000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1767 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:32:39.087547) v4 > -9085> 2018-07-11 07:32:39.337336 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:39.087547) v4 -- 0x55a3131bd800 con 0 > -9084> 2018-07-11 07:32:39.337340 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.8 10.8.128.108:0/11967 1767 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:39.087547) v4 ==== 2004+0+0 (3785990764 0 0) 0x55a34de6f000 con 0x55a3170b7000 > -9083> 2018-07-11 07:32:39.337363 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:39.087547) v4 -- 0x55a349f88800 con 0 > -9082> 2018-07-11 07:32:39.638491 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4d00000 on bdev 1, free 0x2fe000; fallback to bdev 2 > -9081> 2018-07-11 07:32:40.016995 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15255 conn(0x55a311b64000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1841 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:32:41.229450) v4 > -9080> 2018-07-11 07:32:40.017000 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15255 conn(0x55a311b65800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1841 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:32:41.229450) v4 > -9079> 2018-07-11 07:32:40.017019 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.0 10.8.128.113:0/15255 1841 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:41.229450) v4 ==== 2004+0+0 (3422708193 0 0) 0x55a324474c00 con 0x55a311b65800 > -9078> 2018-07-11 07:32:40.017019 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.0 10.8.128.113:0/15255 1841 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:41.229450) v4 ==== 2004+0+0 (3422708193 0 0) 0x55a34de78600 con 0x55a311b64000 > -9077> 2018-07-11 07:32:40.017038 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:41.229450) v4 -- 0x55a343718600 con 0 > -9076> 2018-07-11 07:32:40.017064 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:41.229450) v4 -- 0x55a33ede3e00 con 0 > -9075> 2018-07-11 07:32:40.065219 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12650 conn(0x55a311b67000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1832 0x55a332fafe00 osd_ping(ping e494 stamp 2018-07-11 07:32:40.099945) v4 > -9074> 2018-07-11 07:32:40.065240 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.3 10.8.128.106:0/12650 1832 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:40.099945) v4 ==== 2004+0+0 (3598616446 0 0) 0x55a332fafe00 con 0x55a311b67000 > -9073> 2018-07-11 07:32:40.065234 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12650 conn(0x55a311b68800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1832 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:32:40.099945) v4 > -9072> 2018-07-11 07:32:40.065259 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.3 10.8.128.106:0/12650 1832 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:40.099945) v4 ==== 2004+0+0 (3598616446 0 0) 0x55a34de6f000 con 0x55a311b68800 > -9071> 2018-07-11 07:32:40.065260 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:40.099945) v4 -- 0x55a33d672c00 con 0 > -9070> 2018-07-11 07:32:40.065295 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:40.099945) v4 -- 0x55a33ede3800 con 0 > -9069> 2018-07-11 07:32:40.150855 7f3543899700 5 osd.10 494 heartbeat: osd_stat(783 GB used, 148 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,0,0,1]) > -9068> 2018-07-11 07:32:40.150869 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6802/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:32:40.150867) v4 -- 0x55a31d142e00 con 0 > -9067> 2018-07-11 07:32:40.150903 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6803/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:32:40.150867) v4 -- 0x55a3456b1c00 con 0 > -9066> 2018-07-11 07:32:40.150920 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6806/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:32:40.150867) v4 -- 0x55a3272ae600 con 0 > -9065> 2018-07-11 07:32:40.150934 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6807/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:32:40.150867) v4 -- 0x55a3456b1000 con 0 > -9064> 2018-07-11 07:32:40.150957 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6808/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:32:40.150867) v4 -- 0x55a319404200 con 0 > -9063> 2018-07-11 07:32:40.150971 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6810/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:32:40.150867) v4 -- 0x55a319404800 con 0 > -9062> 2018-07-11 07:32:40.150984 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6809/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:32:40.150867) v4 -- 0x55a319404600 con 0 > -9061> 2018-07-11 07:32:40.150996 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6811/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:32:40.150867) v4 -- 0x55a3456b1200 con 0 > -9060> 2018-07-11 07:32:40.151009 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6810/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:32:40.150867) v4 -- 0x55a319404a00 con 0 > -9059> 2018-07-11 07:32:40.151022 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6811/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:32:40.150867) v4 -- 0x55a319404e00 con 0 > -9058> 2018-07-11 07:32:40.151033 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6806/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:32:40.150867) v4 -- 0x55a3456b1800 con 0 > -9057> 2018-07-11 07:32:40.151046 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6807/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:32:40.150867) v4 -- 0x55a319405000 con 0 > -9056> 2018-07-11 07:32:40.151058 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6802/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:32:40.150867) v4 -- 0x55a3456b1a00 con 0 > -9055> 2018-07-11 07:32:40.151070 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6803/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:32:40.150867) v4 -- 0x55a319404000 con 0 > -9054> 2018-07-11 07:32:40.151083 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6806/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:32:40.150867) v4 -- 0x55a319405400 con 0 > -9053> 2018-07-11 07:32:40.151096 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6807/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:32:40.150867) v4 -- 0x55a319404c00 con 0 > -9052> 2018-07-11 07:32:40.151109 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6802/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:32:40.150867) v4 -- 0x55a319405200 con 0 > -9051> 2018-07-11 07:32:40.151124 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6803/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:32:40.150867) v4 -- 0x55a3272ae400 con 0 > -9050> 2018-07-11 07:32:40.151143 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6806/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:32:40.150867) v4 -- 0x55a327982e00 con 0 > -9049> 2018-07-11 07:32:40.151158 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6807/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:32:40.150867) v4 -- 0x55a3131ba000 con 0 > -9048> 2018-07-11 07:32:40.151175 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6802/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:32:40.150867) v4 -- 0x55a327982c00 con 0 > -9047> 2018-07-11 07:32:40.151187 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6803/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:32:40.150867) v4 -- 0x55a327982a00 con 0 > -9046> 2018-07-11 07:32:40.151358 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6803/15255 conn(0x55a314e1d800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1817 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:40.150867) v4 > -9045> 2018-07-11 07:32:40.151381 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6803/15255 1817 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:40.150867) v4 ==== 2004+0+0 (4064034874 0 0) 0x55a324474c00 con 0x55a314e1d800 > -9044> 2018-07-11 07:32:40.151406 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6806/12677 conn(0x55a314e84000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1817 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:40.150867) v4 > -9043> 2018-07-11 07:32:40.151432 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6806/12677 1817 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:40.150867) v4 ==== 2004+0+0 (4064034874 0 0) 0x55a34de78600 con 0x55a314e84000 > -9042> 2018-07-11 07:32:40.151473 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6807/12617 conn(0x55a314e27800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1817 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:40.150867) v4 > -9041> 2018-07-11 07:32:40.151471 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6802/15255 conn(0x55a314e1c000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1817 0x55a332fafa00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:40.150867) v4 > -9040> 2018-07-11 07:32:40.151471 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6806/12617 conn(0x55a314e26000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1817 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:40.150867) v4 > -9039> 2018-07-11 07:32:40.151486 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6807/12617 1817 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:40.150867) v4 ==== 2004+0+0 (4064034874 0 0) 0x55a34de78600 con 0x55a314e27800 > -9038> 2018-07-11 07:32:40.151491 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6806/12617 1817 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:40.150867) v4 ==== 2004+0+0 (4064034874 0 0) 0x55a324474c00 con 0x55a314e26000 > -9037> 2018-07-11 07:32:40.151491 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6802/15255 1817 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:40.150867) v4 ==== 2004+0+0 (4064034874 0 0) 0x55a332fafa00 con 0x55a314e1c000 > -9036> 2018-07-11 07:32:40.151529 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6808/12651 conn(0x55a314e96000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1817 0x55a332fafa00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:40.150867) v4 > -9035> 2018-07-11 07:32:40.151552 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6808/12651 1817 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:40.150867) v4 ==== 2004+0+0 (4064034874 0 0) 0x55a332fafa00 con 0x55a314e96000 > -9034> 2018-07-11 07:32:40.151533 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6809/12650 conn(0x55a314e1f000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1817 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:40.150867) v4 > -9033> 2018-07-11 07:32:40.151571 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6809/12650 1817 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:40.150867) v4 ==== 2004+0+0 (4064034874 0 0) 0x55a34de78600 con 0x55a314e1f000 > -9032> 2018-07-11 07:32:40.151584 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6810/12651 conn(0x55a314e97800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1817 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:40.150867) v4 > -9031> 2018-07-11 07:32:40.151603 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6807/15263 conn(0x55a314e9a800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1817 0x55a332fafa00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:40.150867) v4 > -9030> 2018-07-11 07:32:40.151605 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6810/12651 1817 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:40.150867) v4 ==== 2004+0+0 (4064034874 0 0) 0x55a324474c00 con 0x55a314e97800 > -9029> 2018-07-11 07:32:40.151621 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6807/15263 1817 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:40.150867) v4 ==== 2004+0+0 (4064034874 0 0) 0x55a332fafa00 con 0x55a314e9a800 > -9028> 2018-07-11 07:32:40.151620 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6806/15263 conn(0x55a314e99000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1817 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:40.150867) v4 > -9027> 2018-07-11 07:32:40.151645 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6806/15263 1817 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:40.150867) v4 ==== 2004+0+0 (4064034874 0 0) 0x55a34de78600 con 0x55a314e99000 > -9026> 2018-07-11 07:32:40.151661 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6811/15287 conn(0x55a314e04800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1817 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:40.150867) v4 > -9025> 2018-07-11 07:32:40.151669 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6810/15287 conn(0x55a314e03000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1817 0x55a332fafa00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:40.150867) v4 > -9024> 2018-07-11 07:32:40.151682 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6811/15287 1817 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:40.150867) v4 ==== 2004+0+0 (4064034874 0 0) 0x55a324474c00 con 0x55a314e04800 > -9023> 2018-07-11 07:32:40.151682 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6810/15287 1817 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:40.150867) v4 ==== 2004+0+0 (4064034874 0 0) 0x55a332fafa00 con 0x55a314e03000 > -9022> 2018-07-11 07:32:40.151716 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6803/13323 conn(0x55a314e16800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1817 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:40.150867) v4 > -9021> 2018-07-11 07:32:40.151722 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6807/12677 conn(0x55a314e85800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1817 0x55a332fafa00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:40.150867) v4 > -9020> 2018-07-11 07:32:40.151731 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6803/13323 1817 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:40.150867) v4 ==== 2004+0+0 (4064034874 0 0) 0x55a34de78600 con 0x55a314e16800 > -9019> 2018-07-11 07:32:40.151742 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6807/12677 1817 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:40.150867) v4 ==== 2004+0+0 (4064034874 0 0) 0x55a332fafa00 con 0x55a314e85800 > -9018> 2018-07-11 07:32:40.151766 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6802/13323 conn(0x55a314e15000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1817 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:40.150867) v4 > -9017> 2018-07-11 07:32:40.151770 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6802/11967 conn(0x55a314e12000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1817 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:40.150867) v4 > -9016> 2018-07-11 07:32:40.151791 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6802/13323 1817 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:40.150867) v4 ==== 2004+0+0 (4064034874 0 0) 0x55a324474c00 con 0x55a314e15000 > -9015> 2018-07-11 07:32:40.151788 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6811/12650 conn(0x55a314e20800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1817 0x55a332fafa00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:40.150867) v4 > -9014> 2018-07-11 07:32:40.151798 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6802/11967 1817 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:40.150867) v4 ==== 2004+0+0 (4064034874 0 0) 0x55a34de78600 con 0x55a314e12000 > -9013> 2018-07-11 07:32:40.151808 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6811/12650 1817 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:40.150867) v4 ==== 2004+0+0 (4064034874 0 0) 0x55a332fafa00 con 0x55a314e20800 > -9012> 2018-07-11 07:32:40.151851 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6803/15919 conn(0x55a314e88800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1817 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:40.150867) v4 > -9011> 2018-07-11 07:32:40.151853 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6803/11967 conn(0x55a314e13800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1817 0x55a332fafa00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:40.150867) v4 > -9010> 2018-07-11 07:32:40.151863 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6802/15919 conn(0x55a314e87000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1817 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:40.150867) v4 > -9009> 2018-07-11 07:32:40.151865 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6803/15919 1817 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:40.150867) v4 ==== 2004+0+0 (4064034874 0 0) 0x55a34de78600 con 0x55a314e88800 > -9008> 2018-07-11 07:32:40.151877 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6802/15919 1817 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:40.150867) v4 ==== 2004+0+0 (4064034874 0 0) 0x55a324474c00 con 0x55a314e87000 > -9007> 2018-07-11 07:32:40.151877 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6803/11967 1817 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:40.150867) v4 ==== 2004+0+0 (4064034874 0 0) 0x55a332fafa00 con 0x55a314e13800 > -9006> 2018-07-11 07:32:40.151904 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6807/15885 conn(0x55a314e01800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1817 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:40.150867) v4 > -9005> 2018-07-11 07:32:40.151915 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6806/15885 conn(0x55a314e00000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1817 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:40.150867) v4 > -9004> 2018-07-11 07:32:40.151918 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6807/15885 1817 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:40.150867) v4 ==== 2004+0+0 (4064034874 0 0) 0x55a34de78600 con 0x55a314e01800 > -9003> 2018-07-11 07:32:40.151933 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6806/15885 1817 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:40.150867) v4 ==== 2004+0+0 (4064034874 0 0) 0x55a324474c00 con 0x55a314e00000 > -9002> 2018-07-11 07:32:40.265839 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.107:0/12617 conn(0x55a32447e000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1791 0x55a332fafa00 osd_ping(ping e494 stamp 2018-07-11 07:32:40.265535) v4 > -9001> 2018-07-11 07:32:40.265859 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.9 10.8.128.107:0/12617 1791 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:40.265535) v4 ==== 2004+0+0 (631006719 0 0) 0x55a332fafa00 con 0x55a32447e000 > -9000> 2018-07-11 07:32:40.265852 7f355d88e700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.107:0/12617 conn(0x55a32447f800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1791 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:32:40.265535) v4 > -8999> 2018-07-11 07:32:40.265868 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:40.265535) v4 -- 0x55a31d142e00 con 0 > -8998> 2018-07-11 07:32:40.265884 7f355d88e700 1 -- 10.8.128.107:6803/12665 <== osd.9 10.8.128.107:0/12617 1791 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:40.265535) v4 ==== 2004+0+0 (631006719 0 0) 0x55a34de78600 con 0x55a32447f800 > -8997> 2018-07-11 07:32:40.265902 7f355d88e700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:40.265535) v4 -- 0x55a3272ae600 con 0 > -8996> 2018-07-11 07:32:40.517465 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15255 conn(0x55a311b65800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1842 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:32:41.729905) v4 > -8995> 2018-07-11 07:32:40.517490 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.0 10.8.128.113:0/15255 1842 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:41.729905) v4 ==== 2004+0+0 (388268968 0 0) 0x55a324474c00 con 0x55a311b65800 > -8994> 2018-07-11 07:32:40.517500 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:41.729905) v4 -- 0x55a3456b1c00 con 0 > -8993> 2018-07-11 07:32:40.517491 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15255 conn(0x55a311b64000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1842 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:32:41.729905) v4 > -8992> 2018-07-11 07:32:40.517520 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.0 10.8.128.113:0/15255 1842 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:41.729905) v4 ==== 2004+0+0 (388268968 0 0) 0x55a34de6f000 con 0x55a311b64000 > -8991> 2018-07-11 07:32:40.517532 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:41.729905) v4 -- 0x55a33ede3800 con 0 > -8990> 2018-07-11 07:32:40.555905 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15885 conn(0x55a311b76800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1837 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:32:41.179207) v4 > -8989> 2018-07-11 07:32:40.555915 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15885 conn(0x55a3170b5800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1837 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:32:41.179207) v4 > -8988> 2018-07-11 07:32:40.555926 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.7 10.8.128.114:0/15885 1837 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:41.179207) v4 ==== 2004+0+0 (2006625979 0 0) 0x55a34de78600 con 0x55a311b76800 > -8987> 2018-07-11 07:32:40.555937 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:41.179207) v4 -- 0x55a33ede3000 con 0 > -8986> 2018-07-11 07:32:40.555935 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.7 10.8.128.114:0/15885 1837 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:41.179207) v4 ==== 2004+0+0 (2006625979 0 0) 0x55a332e79a00 con 0x55a3170b5800 > -8985> 2018-07-11 07:32:40.555960 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:41.179207) v4 -- 0x55a343718600 con 0 > -8984> 2018-07-11 07:32:40.807410 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15263 conn(0x55a321664000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1814 0x55a332fafe00 osd_ping(ping e494 stamp 2018-07-11 07:32:42.019746) v4 > -8983> 2018-07-11 07:32:40.807436 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.5 10.8.128.113:0/15263 1814 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:42.019746) v4 ==== 2004+0+0 (2410230169 0 0) 0x55a332fafe00 con 0x55a321664000 > -8982> 2018-07-11 07:32:40.807459 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:42.019746) v4 -- 0x55a33d672c00 con 0 > -8981> 2018-07-11 07:32:40.807546 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15263 conn(0x55a321667000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1814 0x55a332fafa00 osd_ping(ping e494 stamp 2018-07-11 07:32:42.019746) v4 > -8980> 2018-07-11 07:32:40.807563 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.5 10.8.128.113:0/15263 1814 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:42.019746) v4 ==== 2004+0+0 (2410230169 0 0) 0x55a332fafa00 con 0x55a321667000 > -8979> 2018-07-11 07:32:40.807573 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:42.019746) v4 -- 0x55a33d672c00 con 0 > -8978> 2018-07-11 07:32:41.017936 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15255 conn(0x55a311b64000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1843 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:32:42.230331) v4 > -8977> 2018-07-11 07:32:41.017957 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.0 10.8.128.113:0/15255 1843 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:42.230331) v4 ==== 2004+0+0 (3658097296 0 0) 0x55a34de6f000 con 0x55a311b64000 > -8976> 2018-07-11 07:32:41.017957 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15255 conn(0x55a311b65800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1843 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:32:42.230331) v4 > -8975> 2018-07-11 07:32:41.017973 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:42.230331) v4 -- 0x55a33ede3000 con 0 > -8974> 2018-07-11 07:32:41.017974 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.0 10.8.128.113:0/15255 1843 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:42.230331) v4 ==== 2004+0+0 (3658097296 0 0) 0x55a324474c00 con 0x55a311b65800 > -8973> 2018-07-11 07:32:41.018001 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:42.230331) v4 -- 0x55a343718600 con 0 > -8972> 2018-07-11 07:32:41.037971 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.115:0/13323 conn(0x55a321665800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1830 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:32:41.374797) v4 > -8971> 2018-07-11 07:32:41.037998 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.6 10.8.128.115:0/13323 1830 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:41.374797) v4 ==== 2004+0+0 (1747533621 0 0) 0x55a332e79a00 con 0x55a321665800 > -8970> 2018-07-11 07:32:41.038015 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:41.374797) v4 -- 0x55a343718600 con 0 > -8969> 2018-07-11 07:32:41.038097 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.115:0/13323 conn(0x55a321668800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1830 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:32:41.374797) v4 > -8968> 2018-07-11 07:32:41.038114 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.6 10.8.128.115:0/13323 1830 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:41.374797) v4 ==== 2004+0+0 (1747533621 0 0) 0x55a324474c00 con 0x55a321668800 > -8967> 2018-07-11 07:32:41.038123 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:41.374797) v4 -- 0x55a343718600 con 0 > -8966> 2018-07-11 07:32:41.122007 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 58] Generated table #36566: 4957164 keys, 80348254 bytes > -8965> 2018-07-11 07:32:41.122055 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294361122028, "cf_name": "default", "job": 58, "event": "table_file_creation", "file_number": 36566, "file_size": 80348254, "table_properties": {"data_size": 67109384, "index_size": 689182, "filter_size": 12548698, "raw_key_size": 143206978, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957164, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957164", "kMergeOperands": "0"}} > -8964> 2018-07-11 07:32:41.188646 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a312369d40 con 0 > -8963> 2018-07-11 07:32:41.209842 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5025 0x55a313a94500 replica scrub(pg: 6.15,from:0'0,to:430'106092,epoch:494/492,start:6:a98852d5:::benchmark_data_magna108_35617_object798025:0,end:6:a989ac71:::benchmark_data_magna108_35617_object739998:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -8962> 2018-07-11 07:32:41.209869 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5025 ==== replica scrub(pg: 6.15,from:0'0,to:430'106092,epoch:494/492,start:6:a98852d5:::benchmark_data_magna108_35617_object798025:0,end:6:a989ac71:::benchmark_data_magna108_35617_object739998:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (3757596029 0 0) 0x55a313a94500 con 0x55a317019000 > -8961> 2018-07-11 07:32:41.818733 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15287 conn(0x55a311b75000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1811 0x55a332fafe00 osd_ping(ping e494 stamp 2018-07-11 07:32:43.031111) v4 > -8960> 2018-07-11 07:32:41.818738 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15287 conn(0x55a311b72000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1811 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:32:43.031111) v4 > -8959> 2018-07-11 07:32:41.818755 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.4 10.8.128.113:0/15287 1811 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:43.031111) v4 ==== 2004+0+0 (3626042732 0 0) 0x55a332fafe00 con 0x55a311b75000 > -8958> 2018-07-11 07:32:41.818762 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.4 10.8.128.113:0/15287 1811 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:43.031111) v4 ==== 2004+0+0 (3626042732 0 0) 0x55a332e79a00 con 0x55a311b72000 > -8957> 2018-07-11 07:32:41.818765 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:43.031111) v4 -- 0x55a33d672c00 con 0 > -8956> 2018-07-11 07:32:41.818797 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:43.031111) v4 -- 0x55a343718600 con 0 > -8955> 2018-07-11 07:32:41.829107 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a32c7466c0 con 0 > -8954> 2018-07-11 07:32:41.839185 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5026 0x55a313a94780 replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:494/492,start:6:a989ac71:::benchmark_data_magna108_35617_object739998:0,end:6:a98a82a4:::benchmark_data_magna108_35617_object486223:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -8953> 2018-07-11 07:32:41.839207 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5026 ==== replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:494/492,start:6:a989ac71:::benchmark_data_magna108_35617_object739998:0,end:6:a98a82a4:::benchmark_data_magna108_35617_object486223:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (1905273980 0 0) 0x55a313a94780 con 0x55a317019000 > -8952> 2018-07-11 07:32:42.249425 7f35488a3700 1 -- 10.8.128.107:6800/12665 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55a3216de840 con 0 > -8951> 2018-07-11 07:32:42.249499 7f35488a3700 1 -- 10.8.128.107:6800/12665 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55a32e1be2c0 con 0 > -8950> 2018-07-11 07:32:42.254721 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12651 conn(0x55a3216a0800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1822 0x55a332fafa00 osd_ping(ping e494 stamp 2018-07-11 07:32:42.289437) v4 > -8949> 2018-07-11 07:32:42.254722 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12651 conn(0x55a32169f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1822 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:32:42.289437) v4 > -8948> 2018-07-11 07:32:42.254746 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.2 10.8.128.106:0/12651 1822 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:42.289437) v4 ==== 2004+0+0 (3296231941 0 0) 0x55a332fafa00 con 0x55a3216a0800 > -8947> 2018-07-11 07:32:42.254747 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.2 10.8.128.106:0/12651 1822 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:42.289437) v4 ==== 2004+0+0 (3296231941 0 0) 0x55a324474c00 con 0x55a32169f000 > -8946> 2018-07-11 07:32:42.254760 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:42.289437) v4 -- 0x55a33d672c00 con 0 > -8945> 2018-07-11 07:32:42.254791 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:42.289437) v4 -- 0x55a343718600 con 0 > -8944> 2018-07-11 07:32:42.347753 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15919 conn(0x55a317074800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1824 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:32:42.971073) v4 > -8943> 2018-07-11 07:32:42.347775 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.11 10.8.128.114:0/15919 1824 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:42.971073) v4 ==== 2004+0+0 (2154072346 0 0) 0x55a332e79a00 con 0x55a317074800 > -8942> 2018-07-11 07:32:42.347784 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:42.971073) v4 -- 0x55a3131bda00 con 0 > -8941> 2018-07-11 07:32:42.347803 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15919 conn(0x55a3170b4000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1824 0x55a332fafe00 osd_ping(ping e494 stamp 2018-07-11 07:32:42.971073) v4 > -8940> 2018-07-11 07:32:42.347828 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.11 10.8.128.114:0/15919 1824 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:42.971073) v4 ==== 2004+0+0 (2154072346 0 0) 0x55a332fafe00 con 0x55a3170b4000 > -8939> 2018-07-11 07:32:42.347840 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:42.971073) v4 -- 0x55a33d672600 con 0 > -8938> 2018-07-11 07:32:42.618942 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a323e36b40 con 0 > -8937> 2018-07-11 07:32:42.630881 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5027 0x55a313a94a00 replica scrub(pg: 6.15,from:0'0,to:430'106682,epoch:494/492,start:6:a98a82a4:::benchmark_data_magna108_35617_object486223:0,end:6:a98b7f4f:::benchmark_data_magna108_35617_object1102061:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -8936> 2018-07-11 07:32:42.630913 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5027 ==== replica scrub(pg: 6.15,from:0'0,to:430'106682,epoch:494/492,start:6:a98a82a4:::benchmark_data_magna108_35617_object486223:0,end:6:a98b7f4f:::benchmark_data_magna108_35617_object1102061:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 219+0+0 (2255439997 0 0) 0x55a313a94a00 con 0x55a317019000 > -8935> 2018-07-11 07:32:42.755213 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12651 conn(0x55a32169f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1823 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:32:42.790009) v4 > -8934> 2018-07-11 07:32:42.755243 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.2 10.8.128.106:0/12651 1823 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:42.790009) v4 ==== 2004+0+0 (3862992576 0 0) 0x55a324474c00 con 0x55a32169f000 > -8933> 2018-07-11 07:32:42.755267 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:42.790009) v4 -- 0x55a3131bda00 con 0 > -8932> 2018-07-11 07:32:42.755260 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12651 conn(0x55a3216a0800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1823 0x55a332fafa00 osd_ping(ping e494 stamp 2018-07-11 07:32:42.790009) v4 > -8931> 2018-07-11 07:32:42.755282 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.2 10.8.128.106:0/12651 1823 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:42.790009) v4 ==== 2004+0+0 (3862992576 0 0) 0x55a332fafa00 con 0x55a3216a0800 > -8930> 2018-07-11 07:32:42.755302 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:42.790009) v4 -- 0x55a314575a00 con 0 > -8929> 2018-07-11 07:32:42.837828 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b8800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1768 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:32:42.588034) v4 > -8928> 2018-07-11 07:32:42.837848 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.8 10.8.128.108:0/11967 1768 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:42.588034) v4 ==== 2004+0+0 (747825056 0 0) 0x55a332e79a00 con 0x55a3170b8800 > -8927> 2018-07-11 07:32:42.837857 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:42.588034) v4 -- 0x55a343718000 con 0 > -8926> 2018-07-11 07:32:42.837902 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b7000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1768 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:32:42.588034) v4 > -8925> 2018-07-11 07:32:42.837925 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.8 10.8.128.108:0/11967 1768 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:42.588034) v4 ==== 2004+0+0 (747825056 0 0) 0x55a34de78600 con 0x55a3170b7000 > -8924> 2018-07-11 07:32:42.837937 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:42.588034) v4 -- 0x55a33ede2800 con 0 > -8923> 2018-07-11 07:32:42.919033 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15287 conn(0x55a311b72000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1812 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:32:44.131517) v4 > -8922> 2018-07-11 07:32:42.919054 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.4 10.8.128.113:0/15287 1812 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:44.131517) v4 ==== 2004+0+0 (3676024716 0 0) 0x55a324474c00 con 0x55a311b72000 > -8921> 2018-07-11 07:32:42.919049 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15287 conn(0x55a311b75000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1812 0x55a33d672600 osd_ping(ping e494 stamp 2018-07-11 07:32:44.131517) v4 > -8920> 2018-07-11 07:32:42.919063 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:44.131517) v4 -- 0x55a3131bde00 con 0 > -8919> 2018-07-11 07:32:42.919076 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.4 10.8.128.113:0/15287 1812 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:44.131517) v4 ==== 2004+0+0 (3676024716 0 0) 0x55a33d672600 con 0x55a311b75000 > -8918> 2018-07-11 07:32:42.919098 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:44.131517) v4 -- 0x55a33d672400 con 0 > -8917> 2018-07-11 07:32:43.107772 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15263 conn(0x55a321667000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1815 0x55a332fafa00 osd_ping(ping e494 stamp 2018-07-11 07:32:44.320283) v4 > -8916> 2018-07-11 07:32:43.107797 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.5 10.8.128.113:0/15263 1815 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:44.320283) v4 ==== 2004+0+0 (344755502 0 0) 0x55a332fafa00 con 0x55a321667000 > -8915> 2018-07-11 07:32:43.107806 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:44.320283) v4 -- 0x55a33d672a00 con 0 > -8914> 2018-07-11 07:32:43.107857 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15263 conn(0x55a321664000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1815 0x55a33d672600 osd_ping(ping e494 stamp 2018-07-11 07:32:44.320283) v4 > -8913> 2018-07-11 07:32:43.107871 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.5 10.8.128.113:0/15263 1815 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:44.320283) v4 ==== 2004+0+0 (344755502 0 0) 0x55a33d672600 con 0x55a321664000 > -8912> 2018-07-11 07:32:43.107883 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:44.320283) v4 -- 0x55a32700b800 con 0 > -8911> 2018-07-11 07:32:43.289900 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a357a4a480 con 0 > -8910> 2018-07-11 07:32:43.306178 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5028 0x55a313a94c80 replica scrub(pg: 6.15,from:0'0,to:430'105825,epoch:494/492,start:6:a98b7f4f:::benchmark_data_magna108_35617_object1102061:0,end:6:a98c9302:::benchmark_data_magna108_35617_object678645:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -8909> 2018-07-11 07:32:43.306209 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5028 ==== replica scrub(pg: 6.15,from:0'0,to:430'105825,epoch:494/492,start:6:a98b7f4f:::benchmark_data_magna108_35617_object1102061:0,end:6:a98c9302:::benchmark_data_magna108_35617_object678645:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 219+0+0 (3257102545 0 0) 0x55a313a94c80 con 0x55a317019000 > -8908> 2018-07-11 07:32:43.456343 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15885 conn(0x55a3170b5800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1838 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:32:44.079643) v4 > -8907> 2018-07-11 07:32:43.456367 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.7 10.8.128.114:0/15885 1838 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:44.079643) v4 ==== 2004+0+0 (2271024554 0 0) 0x55a332e79a00 con 0x55a3170b5800 > -8906> 2018-07-11 07:32:43.456378 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:44.079643) v4 -- 0x55a3131bde00 con 0 > -8905> 2018-07-11 07:32:43.456369 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15885 conn(0x55a311b76800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1838 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:32:44.079643) v4 > -8904> 2018-07-11 07:32:43.456389 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.7 10.8.128.114:0/15885 1838 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:44.079643) v4 ==== 2004+0+0 (2271024554 0 0) 0x55a34de6f000 con 0x55a311b76800 > -8903> 2018-07-11 07:32:43.456402 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:44.079643) v4 -- 0x55a33ede2800 con 0 > -8902> 2018-07-11 07:32:43.517839 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12677 conn(0x55a32169d800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1839 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:32:43.552651) v4 > -8901> 2018-07-11 07:32:43.517861 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.1 10.8.128.106:0/12677 1839 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:43.552651) v4 ==== 2004+0+0 (1088942522 0 0) 0x55a34de78600 con 0x55a32169d800 > -8900> 2018-07-11 07:32:43.517855 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12677 conn(0x55a32169c000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1839 0x55a332fafa00 osd_ping(ping e494 stamp 2018-07-11 07:32:43.552651) v4 > -8899> 2018-07-11 07:32:43.517870 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:43.552651) v4 -- 0x55a323f54600 con 0 > -8898> 2018-07-11 07:32:43.517872 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.1 10.8.128.106:0/12677 1839 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:43.552651) v4 ==== 2004+0+0 (1088942522 0 0) 0x55a332fafa00 con 0x55a32169c000 > -8897> 2018-07-11 07:32:43.517896 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:43.552651) v4 -- 0x55a34e290600 con 0 > -8896> 2018-07-11 07:32:43.855673 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12651 conn(0x55a3216a0800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1824 0x55a33d672600 osd_ping(ping e494 stamp 2018-07-11 07:32:43.890494) v4 > -8895> 2018-07-11 07:32:43.855699 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.2 10.8.128.106:0/12651 1824 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:43.890494) v4 ==== 2004+0+0 (1077196907 0 0) 0x55a33d672600 con 0x55a3216a0800 > -8894> 2018-07-11 07:32:43.855712 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:43.890494) v4 -- 0x55a327c0e200 con 0 > -8893> 2018-07-11 07:32:43.855763 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12651 conn(0x55a32169f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1824 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:32:43.890494) v4 > -8892> 2018-07-11 07:32:43.855785 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.2 10.8.128.106:0/12651 1824 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:43.890494) v4 ==== 2004+0+0 (1077196907 0 0) 0x55a324474c00 con 0x55a32169f000 > -8891> 2018-07-11 07:32:43.855795 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:43.890494) v4 -- 0x55a343718800 con 0 > -8890> 2018-07-11 07:32:43.938456 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.115:0/13323 conn(0x55a321665800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1831 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:32:44.275289) v4 > -8889> 2018-07-11 07:32:43.938488 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.6 10.8.128.115:0/13323 1831 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:44.275289) v4 ==== 2004+0+0 (3800534456 0 0) 0x55a332e79a00 con 0x55a321665800 > -8888> 2018-07-11 07:32:43.938503 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:44.275289) v4 -- 0x55a343718400 con 0 > -8887> 2018-07-11 07:32:43.938565 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.115:0/13323 conn(0x55a321668800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1831 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:32:44.275289) v4 > -8886> 2018-07-11 07:32:43.938580 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.6 10.8.128.115:0/13323 1831 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:44.275289) v4 ==== 2004+0+0 (3800534456 0 0) 0x55a324474c00 con 0x55a321668800 > -8885> 2018-07-11 07:32:43.938590 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:44.275289) v4 -- 0x55a343718a00 con 0 > -8884> 2018-07-11 07:32:44.046215 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a315f36fc0 con 0 > -8883> 2018-07-11 07:32:44.068611 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5029 0x55a313a94f00 replica scrub(pg: 6.15,from:0'0,to:430'106443,epoch:494/492,start:6:a98c9302:::benchmark_data_magna108_35617_object678645:0,end:6:a98da5bc:::benchmark_data_magna108_35617_object371974:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -8882> 2018-07-11 07:32:44.068635 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5029 ==== replica scrub(pg: 6.15,from:0'0,to:430'106443,epoch:494/492,start:6:a98c9302:::benchmark_data_magna108_35617_object678645:0,end:6:a98da5bc:::benchmark_data_magna108_35617_object371974:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (2702894958 0 0) 0x55a313a94f00 con 0x55a317019000 > -8881> 2018-07-11 07:32:44.744559 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a32c768240 con 0 > -8880> 2018-07-11 07:32:44.773262 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5030 0x55a313a95180 replica scrub(pg: 6.15,from:0'0,to:461'106911,epoch:494/492,start:6:a98da5bc:::benchmark_data_magna108_35617_object371974:0,end:6:a98e9530:::benchmark_data_magna108_35617_object67586:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -8879> 2018-07-11 07:32:44.773287 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5030 ==== replica scrub(pg: 6.15,from:0'0,to:461'106911,epoch:494/492,start:6:a98da5bc:::benchmark_data_magna108_35617_object371974:0,end:6:a98e9530:::benchmark_data_magna108_35617_object67586:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 217+0+0 (2666567149 0 0) 0x55a313a95180 con 0x55a317019000 > -8878> 2018-07-11 07:32:44.851320 7f3543899700 5 osd.10 494 heartbeat: osd_stat(783 GB used, 148 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,1]) > -8877> 2018-07-11 07:32:44.851340 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6802/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:32:44.851334) v4 -- 0x55a31d142c00 con 0 > -8876> 2018-07-11 07:32:44.851369 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6803/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:32:44.851334) v4 -- 0x55a327982800 con 0 > -8875> 2018-07-11 07:32:44.851385 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6806/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:32:44.851334) v4 -- 0x55a31d142200 con 0 > -8874> 2018-07-11 07:32:44.851404 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6807/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:32:44.851334) v4 -- 0x55a327c0e400 con 0 > -8873> 2018-07-11 07:32:44.851416 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6808/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:32:44.851334) v4 -- 0x55a3272ae000 con 0 > -8872> 2018-07-11 07:32:44.851429 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6810/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:32:44.851334) v4 -- 0x55a3272ae200 con 0 > -8871> 2018-07-11 07:32:44.851441 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6809/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:32:44.851334) v4 -- 0x55a3272ae800 con 0 > -8870> 2018-07-11 07:32:44.851455 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6811/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:32:44.851334) v4 -- 0x55a3272aea00 con 0 > -8869> 2018-07-11 07:32:44.851468 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6810/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:32:44.851334) v4 -- 0x55a3342d7e00 con 0 > -8868> 2018-07-11 07:32:44.851480 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6811/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:32:44.851334) v4 -- 0x55a333f6ea00 con 0 > -8867> 2018-07-11 07:32:44.851492 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6806/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:32:44.851334) v4 -- 0x55a3342d4000 con 0 > -8866> 2018-07-11 07:32:44.851512 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6807/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:32:44.851334) v4 -- 0x55a3342d4400 con 0 > -8865> 2018-07-11 07:32:44.851529 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6802/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:32:44.851334) v4 -- 0x55a3342d4200 con 0 > -8864> 2018-07-11 07:32:44.851543 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6803/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:32:44.851334) v4 -- 0x55a3342d4a00 con 0 > -8863> 2018-07-11 07:32:44.851556 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6806/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:32:44.851334) v4 -- 0x55a3342d4600 con 0 > -8862> 2018-07-11 07:32:44.851570 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6807/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:32:44.851334) v4 -- 0x55a3342d4800 con 0 > -8861> 2018-07-11 07:32:44.851583 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6802/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:32:44.851334) v4 -- 0x55a3342d5000 con 0 > -8860> 2018-07-11 07:32:44.851623 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6803/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:32:44.851334) v4 -- 0x55a3342d5400 con 0 > -8859> 2018-07-11 07:32:44.851645 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6806/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:32:44.851334) v4 -- 0x55a3342d4e00 con 0 > -8858> 2018-07-11 07:32:44.851673 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6807/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:32:44.851334) v4 -- 0x55a3342d5600 con 0 > -8857> 2018-07-11 07:32:44.851688 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6802/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:32:44.851334) v4 -- 0x55a3342d5200 con 0 > -8856> 2018-07-11 07:32:44.851700 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6803/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:32:44.851334) v4 -- 0x55a3342d5c00 con 0 > -8855> 2018-07-11 07:32:44.851793 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6803/15255 conn(0x55a314e1d800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1818 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:44.851334) v4 > -8854> 2018-07-11 07:32:44.851814 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6803/15255 1818 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:44.851334) v4 ==== 2004+0+0 (3816274086 0 0) 0x55a324474c00 con 0x55a314e1d800 > -8853> 2018-07-11 07:32:44.851832 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6802/15255 conn(0x55a314e1c000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1818 0x55a33d672600 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:44.851334) v4 > -8852> 2018-07-11 07:32:44.851851 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6802/15255 1818 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:44.851334) v4 ==== 2004+0+0 (3816274086 0 0) 0x55a33d672600 con 0x55a314e1c000 > -8851> 2018-07-11 07:32:44.851881 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6808/12651 conn(0x55a314e96000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1818 0x55a33d672600 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:44.851334) v4 > -8850> 2018-07-11 07:32:44.851892 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6808/12651 1818 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:44.851334) v4 ==== 2004+0+0 (3816274086 0 0) 0x55a33d672600 con 0x55a314e96000 > -8849> 2018-07-11 07:32:44.851916 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6810/12651 conn(0x55a314e97800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1818 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:44.851334) v4 > -8848> 2018-07-11 07:32:44.851920 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6806/12677 conn(0x55a314e84000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1818 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:44.851334) v4 > -8847> 2018-07-11 07:32:44.851931 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6810/15287 conn(0x55a314e03000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1818 0x55a33d672600 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:44.851334) v4 > -8846> 2018-07-11 07:32:44.851931 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6810/12651 1818 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:44.851334) v4 ==== 2004+0+0 (3816274086 0 0) 0x55a324474c00 con 0x55a314e97800 > -8845> 2018-07-11 07:32:44.851943 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6810/15287 1818 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:44.851334) v4 ==== 2004+0+0 (3816274086 0 0) 0x55a33d672600 con 0x55a314e03000 > -8844> 2018-07-11 07:32:44.851946 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6806/12677 1818 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:44.851334) v4 ==== 2004+0+0 (3816274086 0 0) 0x55a34de6f000 con 0x55a314e84000 > -8843> 2018-07-11 07:32:44.851976 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6806/12617 conn(0x55a314e26000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1818 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:44.851334) v4 > -8842> 2018-07-11 07:32:44.851995 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6806/12617 1818 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:44.851334) v4 ==== 2004+0+0 (3816274086 0 0) 0x55a324474c00 con 0x55a314e26000 > -8841> 2018-07-11 07:32:44.851995 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6811/12650 conn(0x55a314e20800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1818 0x55a33d672600 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:44.851334) v4 > -8840> 2018-07-11 07:32:44.852000 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6807/12617 conn(0x55a314e27800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1818 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:44.851334) v4 > -8839> 2018-07-11 07:32:44.852013 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6811/12650 1818 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:44.851334) v4 ==== 2004+0+0 (3816274086 0 0) 0x55a33d672600 con 0x55a314e20800 > -8838> 2018-07-11 07:32:44.852020 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6807/12617 1818 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:44.851334) v4 ==== 2004+0+0 (3816274086 0 0) 0x55a34de6f000 con 0x55a314e27800 > -8837> 2018-07-11 07:32:44.852032 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6811/15287 conn(0x55a314e04800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1818 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:44.851334) v4 > -8836> 2018-07-11 07:32:44.852045 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6811/15287 1818 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:44.851334) v4 ==== 2004+0+0 (3816274086 0 0) 0x55a324474c00 con 0x55a314e04800 > -8835> 2018-07-11 07:32:44.852055 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6807/12677 conn(0x55a314e85800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1818 0x55a33d672600 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:44.851334) v4 > -8834> 2018-07-11 07:32:44.852072 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6807/12677 1818 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:44.851334) v4 ==== 2004+0+0 (3816274086 0 0) 0x55a33d672600 con 0x55a314e85800 > -8833> 2018-07-11 07:32:44.852068 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6809/12650 conn(0x55a314e1f000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1818 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:44.851334) v4 > -8832> 2018-07-11 07:32:44.852081 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6806/15885 conn(0x55a314e00000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1818 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:44.851334) v4 > -8831> 2018-07-11 07:32:44.852084 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6809/12650 1818 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:44.851334) v4 ==== 2004+0+0 (3816274086 0 0) 0x55a34de6f000 con 0x55a314e1f000 > -8830> 2018-07-11 07:32:44.852097 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6806/15885 1818 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:44.851334) v4 ==== 2004+0+0 (3816274086 0 0) 0x55a324474c00 con 0x55a314e00000 > -8829> 2018-07-11 07:32:44.852106 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6807/15263 conn(0x55a314e9a800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1818 0x55a33d672600 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:44.851334) v4 > -8828> 2018-07-11 07:32:44.852123 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6807/15263 1818 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:44.851334) v4 ==== 2004+0+0 (3816274086 0 0) 0x55a33d672600 con 0x55a314e9a800 > -8827> 2018-07-11 07:32:44.852137 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6802/15919 conn(0x55a314e87000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1818 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:44.851334) v4 > -8826> 2018-07-11 07:32:44.852153 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6802/15919 1818 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:44.851334) v4 ==== 2004+0+0 (3816274086 0 0) 0x55a324474c00 con 0x55a314e87000 > -8825> 2018-07-11 07:32:44.852150 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6806/15263 conn(0x55a314e99000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1818 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:44.851334) v4 > -8824> 2018-07-11 07:32:44.852163 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6803/11967 conn(0x55a314e13800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1818 0x55a33d672600 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:44.851334) v4 > -8823> 2018-07-11 07:32:44.852171 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6806/15263 1818 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:44.851334) v4 ==== 2004+0+0 (3816274086 0 0) 0x55a34de6f000 con 0x55a314e99000 > -8822> 2018-07-11 07:32:44.852177 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6803/11967 1818 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:44.851334) v4 ==== 2004+0+0 (3816274086 0 0) 0x55a33d672600 con 0x55a314e13800 > -8821> 2018-07-11 07:32:44.852185 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6802/13323 conn(0x55a314e15000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1818 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:44.851334) v4 > -8820> 2018-07-11 07:32:44.852195 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6802/13323 1818 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:44.851334) v4 ==== 2004+0+0 (3816274086 0 0) 0x55a324474c00 con 0x55a314e15000 > -8819> 2018-07-11 07:32:44.852219 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6807/15885 conn(0x55a314e01800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1818 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:44.851334) v4 > -8818> 2018-07-11 07:32:44.852233 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6807/15885 1818 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:44.851334) v4 ==== 2004+0+0 (3816274086 0 0) 0x55a34de6f000 con 0x55a314e01800 > -8817> 2018-07-11 07:32:44.852280 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6803/13323 conn(0x55a314e16800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1818 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:44.851334) v4 > -8816> 2018-07-11 07:32:44.852294 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6803/13323 1818 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:44.851334) v4 ==== 2004+0+0 (3816274086 0 0) 0x55a34de6f000 con 0x55a314e16800 > -8815> 2018-07-11 07:32:44.852332 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6802/11967 conn(0x55a314e12000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1818 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:44.851334) v4 > -8814> 2018-07-11 07:32:44.852346 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6802/11967 1818 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:44.851334) v4 ==== 2004+0+0 (3816274086 0 0) 0x55a34de6f000 con 0x55a314e12000 > -8813> 2018-07-11 07:32:44.852386 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6803/15919 conn(0x55a314e88800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1818 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:44.851334) v4 > -8812> 2018-07-11 07:32:44.852399 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6803/15919 1818 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:44.851334) v4 ==== 2004+0+0 (3816274086 0 0) 0x55a34de6f000 con 0x55a314e88800 > -8811> 2018-07-11 07:32:45.219446 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15287 conn(0x55a311b75000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1813 0x55a33d672600 osd_ping(ping e494 stamp 2018-07-11 07:32:46.431920) v4 > -8810> 2018-07-11 07:32:45.219455 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15287 conn(0x55a311b72000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1813 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:32:46.431920) v4 > -8809> 2018-07-11 07:32:45.219478 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.4 10.8.128.113:0/15287 1813 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:46.431920) v4 ==== 2004+0+0 (2846915440 0 0) 0x55a33d672600 con 0x55a311b75000 > -8808> 2018-07-11 07:32:45.219483 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.4 10.8.128.113:0/15287 1813 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:46.431920) v4 ==== 2004+0+0 (2846915440 0 0) 0x55a324474c00 con 0x55a311b72000 > -8807> 2018-07-11 07:32:45.219491 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:46.431920) v4 -- 0x55a327c0e200 con 0 > -8806> 2018-07-11 07:32:45.219516 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:46.431920) v4 -- 0x55a327982800 con 0 > -8805> 2018-07-11 07:32:45.365666 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12650 conn(0x55a311b67000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1833 0x55a332fafa00 osd_ping(ping e494 stamp 2018-07-11 07:32:45.400364) v4 > -8804> 2018-07-11 07:32:45.365692 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.3 10.8.128.106:0/12650 1833 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:45.400364) v4 ==== 2004+0+0 (796877631 0 0) 0x55a332fafa00 con 0x55a311b67000 > -8803> 2018-07-11 07:32:45.365705 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:45.400364) v4 -- 0x55a327c0e200 con 0 > -8802> 2018-07-11 07:32:45.365765 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12650 conn(0x55a311b68800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1833 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:32:45.400364) v4 > -8801> 2018-07-11 07:32:45.365783 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.3 10.8.128.106:0/12650 1833 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:45.400364) v4 ==== 2004+0+0 (796877631 0 0) 0x55a34de6f000 con 0x55a311b68800 > -8800> 2018-07-11 07:32:45.365793 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:45.400364) v4 -- 0x55a31d142200 con 0 > -8799> 2018-07-11 07:32:45.408236 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15263 conn(0x55a321664000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1816 0x55a33d672600 osd_ping(ping e494 stamp 2018-07-11 07:32:46.620738) v4 > -8798> 2018-07-11 07:32:45.408265 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.5 10.8.128.113:0/15263 1816 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:46.620738) v4 ==== 2004+0+0 (3557095482 0 0) 0x55a33d672600 con 0x55a321664000 > -8797> 2018-07-11 07:32:45.408274 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:46.620738) v4 -- 0x55a327c0e200 con 0 > -8796> 2018-07-11 07:32:45.408314 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15263 conn(0x55a321667000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1816 0x55a33d672600 osd_ping(ping e494 stamp 2018-07-11 07:32:46.620738) v4 > -8795> 2018-07-11 07:32:45.408331 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.5 10.8.128.113:0/15263 1816 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:46.620738) v4 ==== 2004+0+0 (3557095482 0 0) 0x55a33d672600 con 0x55a321667000 > -8794> 2018-07-11 07:32:45.408340 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:46.620738) v4 -- 0x55a35772bc00 con 0 > -8793> 2018-07-11 07:32:45.583119 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a357566480 con 0 > -8792> 2018-07-11 07:32:45.714893 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5031 0x55a313a95400 replica scrub(pg: 6.15,from:0'0,to:430'105811,epoch:494/492,start:6:a98e9530:::benchmark_data_magna108_35617_object67586:0,end:6:a98fc1cb:::benchmark_data_magna108.ceph.redhat.com_2478_object200397:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -8791> 2018-07-11 07:32:45.714923 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5031 ==== replica scrub(pg: 6.15,from:0'0,to:430'105811,epoch:494/492,start:6:a98e9530:::benchmark_data_magna108_35617_object67586:0,end:6:a98fc1cb:::benchmark_data_magna108.ceph.redhat.com_2478_object200397:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 232+0+0 (2498579466 0 0) 0x55a313a95400 con 0x55a317019000 > -8790> 2018-07-11 07:32:45.718241 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15255 conn(0x55a311b64000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1844 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:32:46.930734) v4 > -8789> 2018-07-11 07:32:45.718243 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15255 conn(0x55a311b65800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1844 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:32:46.930734) v4 > -8788> 2018-07-11 07:32:45.718281 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.0 10.8.128.113:0/15255 1844 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:46.930734) v4 ==== 2004+0+0 (1249567029 0 0) 0x55a34de78600 con 0x55a311b64000 > -8787> 2018-07-11 07:32:45.718285 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.0 10.8.128.113:0/15255 1844 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:46.930734) v4 ==== 2004+0+0 (1249567029 0 0) 0x55a332e79a00 con 0x55a311b65800 > -8786> 2018-07-11 07:32:45.718299 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:46.930734) v4 -- 0x55a31d142200 con 0 > -8785> 2018-07-11 07:32:45.718328 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:46.930734) v4 -- 0x55a327982800 con 0 > -8784> 2018-07-11 07:32:46.166310 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.107:0/12617 conn(0x55a32447e000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1792 0x55a332fafa00 osd_ping(ping e494 stamp 2018-07-11 07:32:46.165967) v4 > -8783> 2018-07-11 07:32:46.166342 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.9 10.8.128.107:0/12617 1792 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:46.165967) v4 ==== 2004+0+0 (1595002221 0 0) 0x55a332fafa00 con 0x55a32447e000 > -8782> 2018-07-11 07:32:46.166341 7f355d88e700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.107:0/12617 conn(0x55a32447f800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1792 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:32:46.165967) v4 > -8781> 2018-07-11 07:32:46.166358 7f355d88e700 1 -- 10.8.128.107:6803/12665 <== osd.9 10.8.128.107:0/12617 1792 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:46.165967) v4 ==== 2004+0+0 (1595002221 0 0) 0x55a34de6f000 con 0x55a32447f800 > -8780> 2018-07-11 07:32:46.166360 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:46.165967) v4 -- 0x55a35772bc00 con 0 > -8779> 2018-07-11 07:32:46.166393 7f355d88e700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:46.165967) v4 -- 0x55a31d142200 con 0 > -8778> 2018-07-11 07:32:46.338242 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b7000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1769 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:32:46.088519) v4 > -8777> 2018-07-11 07:32:46.338267 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b8800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1769 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:32:46.088519) v4 > -8776> 2018-07-11 07:32:46.338273 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.8 10.8.128.108:0/11967 1769 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:46.088519) v4 ==== 2004+0+0 (3145592447 0 0) 0x55a34de78600 con 0x55a3170b7000 > -8775> 2018-07-11 07:32:46.338285 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.8 10.8.128.108:0/11967 1769 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:46.088519) v4 ==== 2004+0+0 (3145592447 0 0) 0x55a324474c00 con 0x55a3170b8800 > -8774> 2018-07-11 07:32:46.338287 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:46.088519) v4 -- 0x55a31d142200 con 0 > -8773> 2018-07-11 07:32:46.338312 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:46.088519) v4 -- 0x55a327982800 con 0 > -8772> 2018-07-11 07:32:46.356777 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15885 conn(0x55a311b76800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1839 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:32:46.980058) v4 > -8771> 2018-07-11 07:32:46.356802 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.7 10.8.128.114:0/15885 1839 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:46.980058) v4 ==== 2004+0+0 (1964733295 0 0) 0x55a34de6f000 con 0x55a311b76800 > -8770> 2018-07-11 07:32:46.356811 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:46.980058) v4 -- 0x55a31d142200 con 0 > -8769> 2018-07-11 07:32:46.356804 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15885 conn(0x55a3170b5800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1839 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:32:46.980058) v4 > -8768> 2018-07-11 07:32:46.356821 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.7 10.8.128.114:0/15885 1839 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:46.980058) v4 ==== 2004+0+0 (1964733295 0 0) 0x55a332e79a00 con 0x55a3170b5800 > -8767> 2018-07-11 07:32:46.356834 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:46.980058) v4 -- 0x55a343718a00 con 0 > -8766> 2018-07-11 07:32:46.440217 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a33bc75b00 con 0 > -8765> 2018-07-11 07:32:46.468448 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5032 0x55a313a95680 replica scrub(pg: 6.15,from:0'0,to:430'105575,epoch:494/492,start:6:a98fc1cb:::benchmark_data_magna108.ceph.redhat.com_2478_object200397:0,end:6:a990a49e:::benchmark_data_magna108_35617_object48441:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -8764> 2018-07-11 07:32:46.468478 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5032 ==== replica scrub(pg: 6.15,from:0'0,to:430'105575,epoch:494/492,start:6:a98fc1cb:::benchmark_data_magna108.ceph.redhat.com_2478_object200397:0,end:6:a990a49e:::benchmark_data_magna108_35617_object48441:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 232+0+0 (2633626026 0 0) 0x55a313a95680 con 0x55a317019000 > -8763> 2018-07-11 07:32:46.654997 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4d00000 on bdev 1, free 0x2fe000; fallback to bdev 2 > -8762> 2018-07-11 07:32:46.743560 7f35498a5700 10 monclient: tick > -8761> 2018-07-11 07:32:46.743573 7f35498a5700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2018-07-11 07:32:16.743571) > -8760> 2018-07-11 07:32:47.249679 7f35488a3700 1 -- 10.8.128.107:6800/12665 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55a317057340 con 0 > -8759> 2018-07-11 07:32:47.249752 7f35488a3700 1 -- 10.8.128.107:6800/12665 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55a32c6a2b00 con 0 > -8758> 2018-07-11 07:32:47.356257 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12651 conn(0x55a3216a0800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1825 0x55a35772bc00 osd_ping(ping e494 stamp 2018-07-11 07:32:47.391067) v4 > -8757> 2018-07-11 07:32:47.356281 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.2 10.8.128.106:0/12651 1825 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:47.391067) v4 ==== 2004+0+0 (4056746538 0 0) 0x55a35772bc00 con 0x55a3216a0800 > -8756> 2018-07-11 07:32:47.356299 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:47.391067) v4 -- 0x55a327c0e200 con 0 > -8755> 2018-07-11 07:32:47.356316 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12651 conn(0x55a32169f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1825 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:32:47.391067) v4 > -8754> 2018-07-11 07:32:47.356338 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.2 10.8.128.106:0/12651 1825 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:47.391067) v4 ==== 2004+0+0 (4056746538 0 0) 0x55a324474c00 con 0x55a32169f000 > -8753> 2018-07-11 07:32:47.356355 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:47.391067) v4 -- 0x55a343718a00 con 0 > -8752> 2018-07-11 07:32:47.438734 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b7000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1770 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:32:47.188974) v4 > -8751> 2018-07-11 07:32:47.438755 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.8 10.8.128.108:0/11967 1770 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:47.188974) v4 ==== 2004+0+0 (2598273705 0 0) 0x55a34de78600 con 0x55a3170b7000 > -8750> 2018-07-11 07:32:47.438765 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:47.188974) v4 -- 0x55a31d142200 con 0 > -8749> 2018-07-11 07:32:47.438754 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b8800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1770 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:32:47.188974) v4 > -8748> 2018-07-11 07:32:47.438776 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.8 10.8.128.108:0/11967 1770 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:47.188974) v4 ==== 2004+0+0 (2598273705 0 0) 0x55a332e79a00 con 0x55a3170b8800 > -8747> 2018-07-11 07:32:47.438795 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:47.188974) v4 -- 0x55a343718a00 con 0 > -8746> 2018-07-11 07:32:47.648101 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15919 conn(0x55a317074800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1825 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:32:48.271467) v4 > -8745> 2018-07-11 07:32:47.648123 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.11 10.8.128.114:0/15919 1825 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:48.271467) v4 ==== 2004+0+0 (1827463212 0 0) 0x55a324474c00 con 0x55a317074800 > -8744> 2018-07-11 07:32:47.648140 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:48.271467) v4 -- 0x55a343718a00 con 0 > -8743> 2018-07-11 07:32:47.648195 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15919 conn(0x55a3170b4000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1825 0x55a33d672600 osd_ping(ping e494 stamp 2018-07-11 07:32:48.271467) v4 > -8742> 2018-07-11 07:32:47.648218 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.11 10.8.128.114:0/15919 1825 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:48.271467) v4 ==== 2004+0+0 (1827463212 0 0) 0x55a33d672600 con 0x55a3170b4000 > -8741> 2018-07-11 07:32:47.648236 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:48.271467) v4 -- 0x55a35772b800 con 0 > -8740> 2018-07-11 07:32:47.921505 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 58] Generated table #36567: 4957164 keys, 80348253 bytes > -8739> 2018-07-11 07:32:47.921555 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294367921527, "cf_name": "default", "job": 58, "event": "table_file_creation", "file_number": 36567, "file_size": 80348253, "table_properties": {"data_size": 67109358, "index_size": 689207, "filter_size": 12548698, "raw_key_size": 143206958, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957164, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957164", "kMergeOperands": "0"}} > -8738> 2018-07-11 07:32:48.038885 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.115:0/13323 conn(0x55a321665800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1832 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:32:48.375761) v4 > -8737> 2018-07-11 07:32:48.038910 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.6 10.8.128.115:0/13323 1832 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:48.375761) v4 ==== 2004+0+0 (259550377 0 0) 0x55a332e79a00 con 0x55a321665800 > -8736> 2018-07-11 07:32:48.038925 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:48.375761) v4 -- 0x55a343718e00 con 0 > -8735> 2018-07-11 07:32:48.039000 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.115:0/13323 conn(0x55a321668800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1832 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:32:48.375761) v4 > -8734> 2018-07-11 07:32:48.039016 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.6 10.8.128.115:0/13323 1832 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:48.375761) v4 ==== 2004+0+0 (259550377 0 0) 0x55a324474c00 con 0x55a321668800 > -8733> 2018-07-11 07:32:48.039026 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:48.375761) v4 -- 0x55a343718c00 con 0 > -8732> 2018-07-11 07:32:48.339091 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a323e7e6c0 con 0 > -8731> 2018-07-11 07:32:48.348410 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5033 0x55a313a95900 replica scrub(pg: 6.15,from:0'0,to:430'106658,epoch:494/492,start:6:a990a49e:::benchmark_data_magna108_35617_object48441:0,end:6:a991e5c9:::benchmark_data_magna108_35617_object615241:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -8730> 2018-07-11 07:32:48.348439 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5033 ==== replica scrub(pg: 6.15,from:0'0,to:430'106658,epoch:494/492,start:6:a990a49e:::benchmark_data_magna108_35617_object48441:0,end:6:a991e5c9:::benchmark_data_magna108_35617_object615241:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 217+0+0 (1637752133 0 0) 0x55a313a95900 con 0x55a317019000 > -8729> 2018-07-11 07:32:48.748557 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15919 conn(0x55a317074800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1826 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:32:49.371897) v4 > -8728> 2018-07-11 07:32:48.748562 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15919 conn(0x55a3170b4000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1826 0x55a35772bc00 osd_ping(ping e494 stamp 2018-07-11 07:32:49.371897) v4 > -8727> 2018-07-11 07:32:48.748580 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.11 10.8.128.114:0/15919 1826 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:49.371897) v4 ==== 2004+0+0 (2825203715 0 0) 0x55a332e79a00 con 0x55a317074800 > -8726> 2018-07-11 07:32:48.748584 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.11 10.8.128.114:0/15919 1826 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:49.371897) v4 ==== 2004+0+0 (2825203715 0 0) 0x55a35772bc00 con 0x55a3170b4000 > -8725> 2018-07-11 07:32:48.748592 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:49.371897) v4 -- 0x55a343718c00 con 0 > -8724> 2018-07-11 07:32:48.748628 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:49.371897) v4 -- 0x55a31ea06400 con 0 > -8723> 2018-07-11 07:32:49.061198 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a323e84000 con 0 > -8722> 2018-07-11 07:32:49.084007 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5034 0x55a313a95b80 replica scrub(pg: 6.15,from:0'0,to:430'106412,epoch:494/492,start:6:a991e5c9:::benchmark_data_magna108_35617_object615241:0,end:6:a992d243:::benchmark_data_magna108.ceph.redhat.com_2478_object95796:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -8721> 2018-07-11 07:32:49.084037 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5034 ==== replica scrub(pg: 6.15,from:0'0,to:430'106412,epoch:494/492,start:6:a991e5c9:::benchmark_data_magna108_35617_object615241:0,end:6:a992d243:::benchmark_data_magna108.ceph.redhat.com_2478_object95796:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 232+0+0 (3221631040 0 0) 0x55a313a95b80 con 0x55a317019000 > -8720> 2018-07-11 07:32:49.418191 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12677 conn(0x55a32169d800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1840 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:32:49.453080) v4 > -8719> 2018-07-11 07:32:49.418218 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.1 10.8.128.106:0/12677 1840 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:49.453080) v4 ==== 2004+0+0 (101829414 0 0) 0x55a34de6f000 con 0x55a32169d800 > -8718> 2018-07-11 07:32:49.418229 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:49.453080) v4 -- 0x55a31d142200 con 0 > -8717> 2018-07-11 07:32:49.418286 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12677 conn(0x55a32169c000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1840 0x55a35772b800 osd_ping(ping e494 stamp 2018-07-11 07:32:49.453080) v4 > -8716> 2018-07-11 07:32:49.418309 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.1 10.8.128.106:0/12677 1840 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:49.453080) v4 ==== 2004+0+0 (101829414 0 0) 0x55a35772b800 con 0x55a32169c000 > -8715> 2018-07-11 07:32:49.418326 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:49.453080) v4 -- 0x55a31ea06400 con 0 > -8714> 2018-07-11 07:32:49.551851 7f3543899700 5 osd.10 494 heartbeat: osd_stat(783 GB used, 148 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,0,1]) > -8713> 2018-07-11 07:32:49.551865 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6802/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:32:49.551863) v4 -- 0x55a3342d4c00 con 0 > -8712> 2018-07-11 07:32:49.551902 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6803/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:32:49.551863) v4 -- 0x55a3342d5a00 con 0 > -8711> 2018-07-11 07:32:49.551919 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6806/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:32:49.551863) v4 -- 0x55a3342d5800 con 0 > -8710> 2018-07-11 07:32:49.551934 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6807/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:32:49.551863) v4 -- 0x55a3342d6000 con 0 > -8709> 2018-07-11 07:32:49.551946 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6808/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:32:49.551863) v4 -- 0x55a3342d5e00 con 0 > -8708> 2018-07-11 07:32:49.551960 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6810/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:32:49.551863) v4 -- 0x55a31d143e00 con 0 > -8707> 2018-07-11 07:32:49.551973 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6809/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:32:49.551863) v4 -- 0x55a3456b1600 con 0 > -8706> 2018-07-11 07:32:49.551984 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6811/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:32:49.551863) v4 -- 0x55a31d143600 con 0 > -8705> 2018-07-11 07:32:49.551996 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6810/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:32:49.551863) v4 -- 0x55a327983000 con 0 > -8704> 2018-07-11 07:32:49.552007 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6811/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:32:49.551863) v4 -- 0x55a31d143c00 con 0 > -8703> 2018-07-11 07:32:49.552017 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6806/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:32:49.551863) v4 -- 0x55a3456b1400 con 0 > -8702> 2018-07-11 07:32:49.552030 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6807/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:32:49.551863) v4 -- 0x55a31d143a00 con 0 > -8701> 2018-07-11 07:32:49.552048 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6802/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:32:49.551863) v4 -- 0x55a31d143800 con 0 > -8700> 2018-07-11 07:32:49.552061 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6803/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:32:49.551863) v4 -- 0x55a3456b0c00 con 0 > -8699> 2018-07-11 07:32:49.552077 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6806/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:32:49.551863) v4 -- 0x55a3456b0a00 con 0 > -8698> 2018-07-11 07:32:49.552094 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6807/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:32:49.551863) v4 -- 0x55a31d143000 con 0 > -8697> 2018-07-11 07:32:49.552107 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6802/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:32:49.551863) v4 -- 0x55a3456b1e00 con 0 > -8696> 2018-07-11 07:32:49.552119 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6803/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:32:49.551863) v4 -- 0x55a31d143400 con 0 > -8695> 2018-07-11 07:32:49.552158 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6806/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:32:49.551863) v4 -- 0x55a3342d7600 con 0 > -8694> 2018-07-11 07:32:49.552178 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6807/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:32:49.551863) v4 -- 0x55a31fa79a00 con 0 > -8693> 2018-07-11 07:32:49.552212 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6802/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:32:49.551863) v4 -- 0x55a31fa79c00 con 0 > -8692> 2018-07-11 07:32:49.552227 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6803/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:32:49.551863) v4 -- 0x55a333f6f200 con 0 > -8691> 2018-07-11 07:32:49.552316 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6802/15255 conn(0x55a314e1c000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1819 0x55a35772bc00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:49.551863) v4 > -8690> 2018-07-11 07:32:49.552336 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6802/15255 1819 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:49.551863) v4 ==== 2004+0+0 (2988831017 0 0) 0x55a35772bc00 con 0x55a314e1c000 > -8689> 2018-07-11 07:32:49.552394 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6807/12677 conn(0x55a314e85800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1819 0x55a35772bc00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:49.551863) v4 > -8688> 2018-07-11 07:32:49.552395 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6803/15255 conn(0x55a314e1d800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1819 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:49.551863) v4 > -8687> 2018-07-11 07:32:49.552417 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6807/12677 1819 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:49.551863) v4 ==== 2004+0+0 (2988831017 0 0) 0x55a35772bc00 con 0x55a314e85800 > -8686> 2018-07-11 07:32:49.552433 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6807/12617 conn(0x55a314e27800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1819 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:49.551863) v4 > -8685> 2018-07-11 07:32:49.552423 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6803/15255 1819 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:49.551863) v4 ==== 2004+0+0 (2988831017 0 0) 0x55a332e79a00 con 0x55a314e1d800 > -8684> 2018-07-11 07:32:49.552453 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6807/12617 1819 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:49.551863) v4 ==== 2004+0+0 (2988831017 0 0) 0x55a34de78600 con 0x55a314e27800 > -8683> 2018-07-11 07:32:49.552466 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6811/12650 conn(0x55a314e20800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1819 0x55a35772bc00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:49.551863) v4 > -8682> 2018-07-11 07:32:49.552475 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6806/12617 conn(0x55a314e26000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1819 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:49.551863) v4 > -8681> 2018-07-11 07:32:49.552486 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6811/12650 1819 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:49.551863) v4 ==== 2004+0+0 (2988831017 0 0) 0x55a35772bc00 con 0x55a314e20800 > -8680> 2018-07-11 07:32:49.552490 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6806/12617 1819 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:49.551863) v4 ==== 2004+0+0 (2988831017 0 0) 0x55a332e79a00 con 0x55a314e26000 > -8679> 2018-07-11 07:32:49.552504 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6809/12650 conn(0x55a314e1f000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1819 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:49.551863) v4 > -8678> 2018-07-11 07:32:49.552517 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6809/12650 1819 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:49.551863) v4 ==== 2004+0+0 (2988831017 0 0) 0x55a34de78600 con 0x55a314e1f000 > -8677> 2018-07-11 07:32:49.552544 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6806/15263 conn(0x55a314e99000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1819 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:49.551863) v4 > -8676> 2018-07-11 07:32:49.552538 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6808/12651 conn(0x55a314e96000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1819 0x55a35772bc00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:49.551863) v4 > -8675> 2018-07-11 07:32:49.552555 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6806/15263 1819 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:49.551863) v4 ==== 2004+0+0 (2988831017 0 0) 0x55a34de78600 con 0x55a314e99000 > -8674> 2018-07-11 07:32:49.552558 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6808/12651 1819 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:49.551863) v4 ==== 2004+0+0 (2988831017 0 0) 0x55a35772bc00 con 0x55a314e96000 > -8673> 2018-07-11 07:32:49.552585 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6803/13323 conn(0x55a314e16800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1819 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:49.551863) v4 > -8672> 2018-07-11 07:32:49.552597 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6803/13323 1819 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:49.551863) v4 ==== 2004+0+0 (2988831017 0 0) 0x55a34de78600 con 0x55a314e16800 > -8671> 2018-07-11 07:32:49.552611 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6810/12651 conn(0x55a314e97800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1819 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:49.551863) v4 > -8670> 2018-07-11 07:32:49.552614 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6810/15287 conn(0x55a314e03000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1819 0x55a35772bc00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:49.551863) v4 > -8669> 2018-07-11 07:32:49.552634 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6810/12651 1819 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:49.551863) v4 ==== 2004+0+0 (2988831017 0 0) 0x55a332e79a00 con 0x55a314e97800 > -8668> 2018-07-11 07:32:49.552636 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6810/15287 1819 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:49.551863) v4 ==== 2004+0+0 (2988831017 0 0) 0x55a35772bc00 con 0x55a314e03000 > -8667> 2018-07-11 07:32:49.552642 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6807/15885 conn(0x55a314e01800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1819 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:49.551863) v4 > -8666> 2018-07-11 07:32:49.552659 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6807/15885 1819 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:49.551863) v4 ==== 2004+0+0 (2988831017 0 0) 0x55a34de78600 con 0x55a314e01800 > -8665> 2018-07-11 07:32:49.552685 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6806/12677 conn(0x55a314e84000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1819 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:49.551863) v4 > -8664> 2018-07-11 07:32:49.552696 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6806/12677 1819 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:49.551863) v4 ==== 2004+0+0 (2988831017 0 0) 0x55a34de78600 con 0x55a314e84000 > -8663> 2018-07-11 07:32:49.552687 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6811/15287 conn(0x55a314e04800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1819 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:49.551863) v4 > -8662> 2018-07-11 07:32:49.552686 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6807/15263 conn(0x55a314e9a800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1819 0x55a35772bc00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:49.551863) v4 > -8661> 2018-07-11 07:32:49.552710 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6811/15287 1819 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:49.551863) v4 ==== 2004+0+0 (2988831017 0 0) 0x55a332e79a00 con 0x55a314e04800 > -8660> 2018-07-11 07:32:49.552717 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6807/15263 1819 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:49.551863) v4 ==== 2004+0+0 (2988831017 0 0) 0x55a35772bc00 con 0x55a314e9a800 > -8659> 2018-07-11 07:32:49.552727 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6803/15919 conn(0x55a314e88800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1819 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:49.551863) v4 > -8658> 2018-07-11 07:32:49.552739 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6803/15919 1819 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:49.551863) v4 ==== 2004+0+0 (2988831017 0 0) 0x55a34de78600 con 0x55a314e88800 > -8657> 2018-07-11 07:32:49.552758 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6802/13323 conn(0x55a314e15000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1819 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:49.551863) v4 > -8656> 2018-07-11 07:32:49.552770 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6802/11967 conn(0x55a314e12000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1819 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:49.551863) v4 > -8655> 2018-07-11 07:32:49.552760 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6803/11967 conn(0x55a314e13800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1819 0x55a35772bc00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:49.551863) v4 > -8654> 2018-07-11 07:32:49.552780 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6802/11967 1819 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:49.551863) v4 ==== 2004+0+0 (2988831017 0 0) 0x55a34de78600 con 0x55a314e12000 > -8653> 2018-07-11 07:32:49.552782 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6803/11967 1819 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:49.551863) v4 ==== 2004+0+0 (2988831017 0 0) 0x55a35772bc00 con 0x55a314e13800 > -8652> 2018-07-11 07:32:49.552779 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6802/13323 1819 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:49.551863) v4 ==== 2004+0+0 (2988831017 0 0) 0x55a332e79a00 con 0x55a314e15000 > -8651> 2018-07-11 07:32:49.552834 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6806/15885 conn(0x55a314e00000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1819 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:49.551863) v4 > -8650> 2018-07-11 07:32:49.552849 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6806/15885 1819 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:49.551863) v4 ==== 2004+0+0 (2988831017 0 0) 0x55a332e79a00 con 0x55a314e00000 > -8649> 2018-07-11 07:32:49.552878 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6802/15919 conn(0x55a314e87000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1819 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:49.551863) v4 > -8648> 2018-07-11 07:32:49.552894 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6802/15919 1819 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:49.551863) v4 ==== 2004+0+0 (2988831017 0 0) 0x55a332e79a00 con 0x55a314e87000 > -8647> 2018-07-11 07:32:49.666727 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.107:0/12617 conn(0x55a32447e000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1793 0x55a35772bc00 osd_ping(ping e494 stamp 2018-07-11 07:32:49.666412) v4 > -8646> 2018-07-11 07:32:49.666730 7f355d88e700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.107:0/12617 conn(0x55a32447f800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1793 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:32:49.666412) v4 > -8645> 2018-07-11 07:32:49.666757 7f355d88e700 1 -- 10.8.128.107:6803/12665 <== osd.9 10.8.128.107:0/12617 1793 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:49.666412) v4 ==== 2004+0+0 (1941292154 0 0) 0x55a34de78600 con 0x55a32447f800 > -8644> 2018-07-11 07:32:49.666772 7f355d88e700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:49.666412) v4 -- 0x55a3342d5800 con 0 > -8643> 2018-07-11 07:32:49.666754 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.9 10.8.128.107:0/12617 1793 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:49.666412) v4 ==== 2004+0+0 (1941292154 0 0) 0x55a35772bc00 con 0x55a32447e000 > -8642> 2018-07-11 07:32:49.666795 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:49.666412) v4 -- 0x55a3342d6000 con 0 > -8641> 2018-07-11 07:32:49.716033 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a33c0ca6c0 con 0 > -8640> 2018-07-11 07:32:49.723687 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5035 0x55a311168000 replica scrub(pg: 6.15,from:0'0,to:430'106206,epoch:494/492,start:6:a992d243:::benchmark_data_magna108.ceph.redhat.com_2478_object95796:0,end:6:a9935d86:::benchmark_data_magna108.ceph.redhat.com_2478_object363158:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -8639> 2018-07-11 07:32:49.723718 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5035 ==== replica scrub(pg: 6.15,from:0'0,to:430'106206,epoch:494/492,start:6:a992d243:::benchmark_data_magna108.ceph.redhat.com_2478_object95796:0,end:6:a9935d86:::benchmark_data_magna108.ceph.redhat.com_2478_object363158:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 247+0+0 (1033878668 0 0) 0x55a311168000 con 0x55a317019000 > -8638> 2018-07-11 07:32:49.818525 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15255 conn(0x55a311b64000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1845 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:32:51.031149) v4 > -8637> 2018-07-11 07:32:49.818552 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.0 10.8.128.113:0/15255 1845 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:51.031149) v4 ==== 2004+0+0 (1709821482 0 0) 0x55a34de6f000 con 0x55a311b64000 > -8636> 2018-07-11 07:32:49.818548 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15255 conn(0x55a311b65800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1845 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:32:51.031149) v4 > -8635> 2018-07-11 07:32:49.818567 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:51.031149) v4 -- 0x55a3342d5800 con 0 > -8634> 2018-07-11 07:32:49.818569 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.0 10.8.128.113:0/15255 1845 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:51.031149) v4 ==== 2004+0+0 (1709821482 0 0) 0x55a332e79a00 con 0x55a311b65800 > -8633> 2018-07-11 07:32:49.818599 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:51.031149) v4 -- 0x55a31d143e00 con 0 > -8632> 2018-07-11 07:32:49.857310 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15885 conn(0x55a311b76800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1840 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:32:50.480666) v4 > -8631> 2018-07-11 07:32:49.857315 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15885 conn(0x55a3170b5800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1840 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:32:50.480666) v4 > -8630> 2018-07-11 07:32:49.857335 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.7 10.8.128.114:0/15885 1840 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:50.480666) v4 ==== 2004+0+0 (1885235703 0 0) 0x55a324474c00 con 0x55a3170b5800 > -8629> 2018-07-11 07:32:49.857331 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.7 10.8.128.114:0/15885 1840 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:50.480666) v4 ==== 2004+0+0 (1885235703 0 0) 0x55a34de78600 con 0x55a311b76800 > -8628> 2018-07-11 07:32:49.857350 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:50.480666) v4 -- 0x55a31d143e00 con 0 > -8627> 2018-07-11 07:32:49.857372 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:50.480666) v4 -- 0x55a3342d5800 con 0 > -8626> 2018-07-11 07:32:50.066113 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12650 conn(0x55a311b68800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1834 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:32:50.100982) v4 > -8625> 2018-07-11 07:32:50.066113 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12650 conn(0x55a311b67000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1834 0x55a35772b800 osd_ping(ping e494 stamp 2018-07-11 07:32:50.100982) v4 > -8624> 2018-07-11 07:32:50.066141 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.3 10.8.128.106:0/12650 1834 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:50.100982) v4 ==== 2004+0+0 (307647840 0 0) 0x55a34de6f000 con 0x55a311b68800 > -8623> 2018-07-11 07:32:50.066144 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.3 10.8.128.106:0/12650 1834 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:50.100982) v4 ==== 2004+0+0 (307647840 0 0) 0x55a35772b800 con 0x55a311b67000 > -8622> 2018-07-11 07:32:50.066160 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:50.100982) v4 -- 0x55a3342d5800 con 0 > -8621> 2018-07-11 07:32:50.066187 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:50.100982) v4 -- 0x55a3342d4c00 con 0 > -8620> 2018-07-11 07:32:50.108527 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15263 conn(0x55a321667000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1817 0x55a3342d6000 osd_ping(ping e494 stamp 2018-07-11 07:32:51.321135) v4 > -8619> 2018-07-11 07:32:50.108548 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.5 10.8.128.113:0/15263 1817 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:51.321135) v4 ==== 2004+0+0 (1221559106 0 0) 0x55a3342d6000 con 0x55a321667000 > -8618> 2018-07-11 07:32:50.108557 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:51.321135) v4 -- 0x55a3342d4c00 con 0 > -8617> 2018-07-11 07:32:50.108611 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15263 conn(0x55a321664000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1817 0x55a35772b800 osd_ping(ping e494 stamp 2018-07-11 07:32:51.321135) v4 > -8616> 2018-07-11 07:32:50.108625 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.5 10.8.128.113:0/15263 1817 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:51.321135) v4 ==== 2004+0+0 (1221559106 0 0) 0x55a35772b800 con 0x55a321664000 > -8615> 2018-07-11 07:32:50.108634 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:51.321135) v4 -- 0x55a3342d4c00 con 0 > -8614> 2018-07-11 07:32:50.357730 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15885 conn(0x55a3170b5800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1841 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:32:50.981122) v4 > -8613> 2018-07-11 07:32:50.357735 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15885 conn(0x55a311b76800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1841 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:32:50.981122) v4 > -8612> 2018-07-11 07:32:50.357752 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.7 10.8.128.114:0/15885 1841 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:50.981122) v4 ==== 2004+0+0 (1588498460 0 0) 0x55a332e79a00 con 0x55a3170b5800 > -8611> 2018-07-11 07:32:50.357755 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.7 10.8.128.114:0/15885 1841 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:50.981122) v4 ==== 2004+0+0 (1588498460 0 0) 0x55a34de78600 con 0x55a311b76800 > -8610> 2018-07-11 07:32:50.357762 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:50.981122) v4 -- 0x55a31d143e00 con 0 > -8609> 2018-07-11 07:32:50.357797 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:50.981122) v4 -- 0x55a3342d5800 con 0 > -8608> 2018-07-11 07:32:50.367339 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a334b0ad80 con 0 > -8607> 2018-07-11 07:32:50.393222 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5036 0x55a311168280 replica scrub(pg: 6.15,from:0'0,to:430'105621,epoch:494/492,start:6:a9935d86:::benchmark_data_magna108.ceph.redhat.com_2478_object363158:0,end:6:a994f65b:::benchmark_data_magna108_35617_object848809:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -8606> 2018-07-11 07:32:50.393264 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5036 ==== replica scrub(pg: 6.15,from:0'0,to:430'105621,epoch:494/492,start:6:a9935d86:::benchmark_data_magna108.ceph.redhat.com_2478_object363158:0,end:6:a994f65b:::benchmark_data_magna108_35617_object848809:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (169788136 0 0) 0x55a311168280 con 0x55a317019000 > -8605> 2018-07-11 07:32:50.919070 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15255 conn(0x55a311b64000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1846 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:32:52.131578) v4 > -8604> 2018-07-11 07:32:50.919095 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.0 10.8.128.113:0/15255 1846 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:52.131578) v4 ==== 2004+0+0 (558848894 0 0) 0x55a34de6f000 con 0x55a311b64000 > -8603> 2018-07-11 07:32:50.919109 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:52.131578) v4 -- 0x55a3342d5800 con 0 > -8602> 2018-07-11 07:32:50.919097 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15255 conn(0x55a311b65800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1846 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:32:52.131578) v4 > -8601> 2018-07-11 07:32:50.919122 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.0 10.8.128.113:0/15255 1846 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:52.131578) v4 ==== 2004+0+0 (558848894 0 0) 0x55a324474c00 con 0x55a311b65800 > -8600> 2018-07-11 07:32:50.919141 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:52.131578) v4 -- 0x55a31d143e00 con 0 > -8599> 2018-07-11 07:32:51.048921 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15919 conn(0x55a3170b4000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1827 0x55a3342d6000 osd_ping(ping e494 stamp 2018-07-11 07:32:51.672308) v4 > -8598> 2018-07-11 07:32:51.048932 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15919 conn(0x55a317074800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1827 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:32:51.672308) v4 > -8597> 2018-07-11 07:32:51.048945 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.11 10.8.128.114:0/15919 1827 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:51.672308) v4 ==== 2004+0+0 (2314851074 0 0) 0x55a3342d6000 con 0x55a3170b4000 > -8596> 2018-07-11 07:32:51.048952 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.11 10.8.128.114:0/15919 1827 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:51.672308) v4 ==== 2004+0+0 (2314851074 0 0) 0x55a332e79a00 con 0x55a317074800 > -8595> 2018-07-11 07:32:51.048955 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:51.672308) v4 -- 0x55a31ea06400 con 0 > -8594> 2018-07-11 07:32:51.048978 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:51.672308) v4 -- 0x55a31d143e00 con 0 > -8593> 2018-07-11 07:32:51.119730 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15287 conn(0x55a311b72000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1814 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:32:52.332350) v4 > -8592> 2018-07-11 07:32:51.119738 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15287 conn(0x55a311b75000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1814 0x55a3342d4c00 osd_ping(ping e494 stamp 2018-07-11 07:32:52.332350) v4 > -8591> 2018-07-11 07:32:51.119750 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.4 10.8.128.113:0/15287 1814 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:52.332350) v4 ==== 2004+0+0 (1172668490 0 0) 0x55a324474c00 con 0x55a311b72000 > -8590> 2018-07-11 07:32:51.119755 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.4 10.8.128.113:0/15287 1814 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:52.332350) v4 ==== 2004+0+0 (1172668490 0 0) 0x55a3342d4c00 con 0x55a311b75000 > -8589> 2018-07-11 07:32:51.119760 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:52.332350) v4 -- 0x55a31d143e00 con 0 > -8588> 2018-07-11 07:32:51.119786 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:52.332350) v4 -- 0x55a31ea06400 con 0 > -8587> 2018-07-11 07:32:51.225558 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a323e95b00 con 0 > -8586> 2018-07-11 07:32:51.240595 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5037 0x55a311168500 replica scrub(pg: 6.15,from:0'0,to:430'106236,epoch:494/492,start:6:a994f65b:::benchmark_data_magna108_35617_object848809:0,end:6:a9959124:::benchmark_data_magna108_35617_object226281:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -8585> 2018-07-11 07:32:51.240622 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5037 ==== replica scrub(pg: 6.15,from:0'0,to:430'106236,epoch:494/492,start:6:a994f65b:::benchmark_data_magna108_35617_object848809:0,end:6:a9959124:::benchmark_data_magna108_35617_object226281:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (3418191168 0 0) 0x55a311168500 con 0x55a317019000 > -8584> 2018-07-11 07:32:51.620198 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15287 conn(0x55a311b72000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1815 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:32:52.832766) v4 > -8583> 2018-07-11 07:32:51.620212 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15287 conn(0x55a311b75000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1815 0x55a3342d6000 osd_ping(ping e494 stamp 2018-07-11 07:32:52.832766) v4 > -8582> 2018-07-11 07:32:51.620230 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.4 10.8.128.113:0/15287 1815 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:52.832766) v4 ==== 2004+0+0 (2509168260 0 0) 0x55a3342d6000 con 0x55a311b75000 > -8581> 2018-07-11 07:32:51.620227 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.4 10.8.128.113:0/15287 1815 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:52.832766) v4 ==== 2004+0+0 (2509168260 0 0) 0x55a332e79a00 con 0x55a311b72000 > -8580> 2018-07-11 07:32:51.620254 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:52.832766) v4 -- 0x55a31ea06400 con 0 > -8579> 2018-07-11 07:32:51.620276 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:52.832766) v4 -- 0x55a31d143e00 con 0 > -8578> 2018-07-11 07:32:51.718703 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12677 conn(0x55a32169d800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1841 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:32:51.753550) v4 > -8577> 2018-07-11 07:32:51.718706 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12677 conn(0x55a32169c000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1841 0x55a3342d4c00 osd_ping(ping e494 stamp 2018-07-11 07:32:51.753550) v4 > -8576> 2018-07-11 07:32:51.718731 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.1 10.8.128.106:0/12677 1841 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:51.753550) v4 ==== 2004+0+0 (2198819509 0 0) 0x55a34de78600 con 0x55a32169d800 > -8575> 2018-07-11 07:32:51.718741 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:51.753550) v4 -- 0x55a31d142200 con 0 > -8574> 2018-07-11 07:32:51.718734 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.1 10.8.128.106:0/12677 1841 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:51.753550) v4 ==== 2004+0+0 (2198819509 0 0) 0x55a3342d4c00 con 0x55a32169c000 > -8573> 2018-07-11 07:32:51.718769 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:51.753550) v4 -- 0x55a31fa78600 con 0 > -8572> 2018-07-11 07:32:51.852328 7f3543899700 5 osd.10 494 heartbeat: osd_stat(783 GB used, 148 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,0,0,1]) > -8571> 2018-07-11 07:32:51.852344 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6802/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:32:51.852342) v4 -- 0x55a333f6f800 con 0 > -8570> 2018-07-11 07:32:51.852370 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6803/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:32:51.852342) v4 -- 0x55a333f6fa00 con 0 > -8569> 2018-07-11 07:32:51.852386 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6806/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:32:51.852342) v4 -- 0x55a333f6ee00 con 0 > -8568> 2018-07-11 07:32:51.852410 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6807/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:32:51.852342) v4 -- 0x55a31fa79e00 con 0 > -8567> 2018-07-11 07:32:51.852435 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6808/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:32:51.852342) v4 -- 0x55a314658000 con 0 > -8566> 2018-07-11 07:32:51.852456 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6810/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:32:51.852342) v4 -- 0x55a314658600 con 0 > -8565> 2018-07-11 07:32:51.852471 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6809/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:32:51.852342) v4 -- 0x55a314658400 con 0 > -8564> 2018-07-11 07:32:51.852496 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6811/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:32:51.852342) v4 -- 0x55a314658e00 con 0 > -8563> 2018-07-11 07:32:51.852519 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6810/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:32:51.852342) v4 -- 0x55a314658c00 con 0 > -8562> 2018-07-11 07:32:51.852534 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6811/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:32:51.852342) v4 -- 0x55a314658200 con 0 > -8561> 2018-07-11 07:32:51.852552 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6806/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:32:51.852342) v4 -- 0x55a314659400 con 0 > -8560> 2018-07-11 07:32:51.852574 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6807/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:32:51.852342) v4 -- 0x55a314659200 con 0 > -8559> 2018-07-11 07:32:51.852597 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6802/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:32:51.852342) v4 -- 0x55a314658800 con 0 > -8558> 2018-07-11 07:32:51.852616 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6803/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:32:51.852342) v4 -- 0x55a314659800 con 0 > -8557> 2018-07-11 07:32:51.852641 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6806/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:32:51.852342) v4 -- 0x55a314658a00 con 0 > -8556> 2018-07-11 07:32:51.852664 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6807/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:32:51.852342) v4 -- 0x55a314659000 con 0 > -8555> 2018-07-11 07:32:51.852714 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6802/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:32:51.852342) v4 -- 0x55a314659a00 con 0 > -8554> 2018-07-11 07:32:51.852740 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6803/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:32:51.852342) v4 -- 0x55a314659600 con 0 > -8553> 2018-07-11 07:32:51.852762 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6806/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:32:51.852342) v4 -- 0x55a32d6e6000 con 0 > -8552> 2018-07-11 07:32:51.852813 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6807/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:32:51.852342) v4 -- 0x55a314659c00 con 0 > -8551> 2018-07-11 07:32:51.852839 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6802/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:32:51.852342) v4 -- 0x55a314659e00 con 0 > -8550> 2018-07-11 07:32:51.852855 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6808/12651 conn(0x55a314e96000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1820 0x55a3342d4c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:51.852342) v4 > -8549> 2018-07-11 07:32:51.852870 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6806/12677 conn(0x55a314e84000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1820 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:51.852342) v4 > -8548> 2018-07-11 07:32:51.852877 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6808/12651 1820 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:51.852342) v4 ==== 2004+0+0 (1698803067 0 0) 0x55a3342d4c00 con 0x55a314e96000 > -8547> 2018-07-11 07:32:51.852891 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6806/12677 1820 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:51.852342) v4 ==== 2004+0+0 (1698803067 0 0) 0x55a34de6f000 con 0x55a314e84000 > -8546> 2018-07-11 07:32:51.852884 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6810/12651 conn(0x55a314e97800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1820 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:51.852342) v4 > -8545> 2018-07-11 07:32:51.852862 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6803/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:32:51.852342) v4 -- 0x55a32d6e6400 con 0 > -8544> 2018-07-11 07:32:51.852903 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6810/12651 1820 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:51.852342) v4 ==== 2004+0+0 (1698803067 0 0) 0x55a324474c00 con 0x55a314e97800 > -8543> 2018-07-11 07:32:51.852950 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6802/15255 conn(0x55a314e1c000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1820 0x55a3342d4c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:51.852342) v4 > -8542> 2018-07-11 07:32:51.852965 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6802/15255 1820 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:51.852342) v4 ==== 2004+0+0 (1698803067 0 0) 0x55a3342d4c00 con 0x55a314e1c000 > -8541> 2018-07-11 07:32:51.853000 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6807/12677 conn(0x55a314e85800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1820 0x55a3342d4c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:51.852342) v4 > -8540> 2018-07-11 07:32:51.853020 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6807/12677 1820 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:51.852342) v4 ==== 2004+0+0 (1698803067 0 0) 0x55a3342d4c00 con 0x55a314e85800 > -8539> 2018-07-11 07:32:51.853019 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6809/12650 conn(0x55a314e1f000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1820 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:51.852342) v4 > -8538> 2018-07-11 07:32:51.853007 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6803/15255 conn(0x55a314e1d800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1820 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:51.852342) v4 > -8537> 2018-07-11 07:32:51.853032 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6809/12650 1820 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:51.852342) v4 ==== 2004+0+0 (1698803067 0 0) 0x55a34de78600 con 0x55a314e1f000 > -8536> 2018-07-11 07:32:51.853039 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6803/15255 1820 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:51.852342) v4 ==== 2004+0+0 (1698803067 0 0) 0x55a332e79a00 con 0x55a314e1d800 > -8535> 2018-07-11 07:32:51.853057 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6810/15287 conn(0x55a314e03000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1820 0x55a3342d4c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:51.852342) v4 > -8534> 2018-07-11 07:32:51.853061 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6806/15263 conn(0x55a314e99000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1820 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:51.852342) v4 > -8533> 2018-07-11 07:32:51.853070 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6810/15287 1820 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:51.852342) v4 ==== 2004+0+0 (1698803067 0 0) 0x55a3342d4c00 con 0x55a314e03000 > -8532> 2018-07-11 07:32:51.853071 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6806/15263 1820 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:51.852342) v4 ==== 2004+0+0 (1698803067 0 0) 0x55a34de78600 con 0x55a314e99000 > -8531> 2018-07-11 07:32:51.853083 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6811/15287 conn(0x55a314e04800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1820 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:51.852342) v4 > -8530> 2018-07-11 07:32:51.853097 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6811/15287 1820 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:51.852342) v4 ==== 2004+0+0 (1698803067 0 0) 0x55a332e79a00 con 0x55a314e04800 > -8529> 2018-07-11 07:32:51.853108 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6811/12650 conn(0x55a314e20800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1820 0x55a3342d4c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:51.852342) v4 > -8528> 2018-07-11 07:32:51.853116 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6807/15885 conn(0x55a314e01800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1820 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:51.852342) v4 > -8527> 2018-07-11 07:32:51.853122 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6811/12650 1820 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:51.852342) v4 ==== 2004+0+0 (1698803067 0 0) 0x55a3342d4c00 con 0x55a314e20800 > -8526> 2018-07-11 07:32:51.853122 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6806/12617 conn(0x55a314e26000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1820 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:51.852342) v4 > -8525> 2018-07-11 07:32:51.853131 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6806/12617 1820 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:51.852342) v4 ==== 2004+0+0 (1698803067 0 0) 0x55a332e79a00 con 0x55a314e26000 > -8524> 2018-07-11 07:32:51.853131 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6807/15885 1820 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:51.852342) v4 ==== 2004+0+0 (1698803067 0 0) 0x55a34de78600 con 0x55a314e01800 > -8523> 2018-07-11 07:32:51.853163 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6806/15885 conn(0x55a314e00000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1820 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:51.852342) v4 > -8522> 2018-07-11 07:32:51.853163 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6807/15263 conn(0x55a314e9a800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1820 0x55a3342d4c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:51.852342) v4 > -8521> 2018-07-11 07:32:51.853175 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6806/15885 1820 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:51.852342) v4 ==== 2004+0+0 (1698803067 0 0) 0x55a332e79a00 con 0x55a314e00000 > -8520> 2018-07-11 07:32:51.853177 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6807/15263 1820 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:51.852342) v4 ==== 2004+0+0 (1698803067 0 0) 0x55a3342d4c00 con 0x55a314e9a800 > -8519> 2018-07-11 07:32:51.853182 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6803/13323 conn(0x55a314e16800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1820 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:51.852342) v4 > -8518> 2018-07-11 07:32:51.853193 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6803/13323 1820 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:51.852342) v4 ==== 2004+0+0 (1698803067 0 0) 0x55a34de78600 con 0x55a314e16800 > -8517> 2018-07-11 07:32:51.853213 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6802/13323 conn(0x55a314e15000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1820 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:51.852342) v4 > -8516> 2018-07-11 07:32:51.853218 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6807/12617 conn(0x55a314e27800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1820 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:51.852342) v4 > -8515> 2018-07-11 07:32:51.853218 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6803/11967 conn(0x55a314e13800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1820 0x55a3342d4c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:51.852342) v4 > -8514> 2018-07-11 07:32:51.853227 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6807/12617 1820 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:51.852342) v4 ==== 2004+0+0 (1698803067 0 0) 0x55a34de78600 con 0x55a314e27800 > -8513> 2018-07-11 07:32:51.853226 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6802/13323 1820 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:51.852342) v4 ==== 2004+0+0 (1698803067 0 0) 0x55a332e79a00 con 0x55a314e15000 > -8512> 2018-07-11 07:32:51.853233 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6803/11967 1820 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:51.852342) v4 ==== 2004+0+0 (1698803067 0 0) 0x55a3342d4c00 con 0x55a314e13800 > -8511> 2018-07-11 07:32:51.853262 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6802/11967 conn(0x55a314e12000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1820 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:51.852342) v4 > -8510> 2018-07-11 07:32:51.853273 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6802/11967 1820 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:51.852342) v4 ==== 2004+0+0 (1698803067 0 0) 0x55a34de78600 con 0x55a314e12000 > -8509> 2018-07-11 07:32:51.853327 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6802/15919 conn(0x55a314e87000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1820 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:51.852342) v4 > -8508> 2018-07-11 07:32:51.853346 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6802/15919 1820 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:51.852342) v4 ==== 2004+0+0 (1698803067 0 0) 0x55a332e79a00 con 0x55a314e87000 > -8507> 2018-07-11 07:32:51.853352 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6803/15919 conn(0x55a314e88800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1820 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:51.852342) v4 > -8506> 2018-07-11 07:32:51.853367 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6803/15919 1820 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:51.852342) v4 ==== 2004+0+0 (1698803067 0 0) 0x55a34de78600 con 0x55a314e88800 > -8505> 2018-07-11 07:32:51.959480 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a323e9db00 con 0 > -8504> 2018-07-11 07:32:51.977238 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5038 0x55a311168780 replica scrub(pg: 6.15,from:0'0,to:430'105982,epoch:494/492,start:6:a9959124:::benchmark_data_magna108_35617_object226281:0,end:6:a99696ed:::benchmark_data_magna108_35617_object448889:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -8503> 2018-07-11 07:32:51.977277 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5038 ==== replica scrub(pg: 6.15,from:0'0,to:430'105982,epoch:494/492,start:6:a9959124:::benchmark_data_magna108_35617_object226281:0,end:6:a99696ed:::benchmark_data_magna108_35617_object448889:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (3719001537 0 0) 0x55a311168780 con 0x55a317019000 > -8502> 2018-07-11 07:32:52.056767 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12651 conn(0x55a3216a0800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1826 0x55a3342d4c00 osd_ping(ping e494 stamp 2018-07-11 07:32:52.091599) v4 > -8501> 2018-07-11 07:32:52.056789 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.2 10.8.128.106:0/12651 1826 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:52.091599) v4 ==== 2004+0+0 (2663635615 0 0) 0x55a3342d4c00 con 0x55a3216a0800 > -8500> 2018-07-11 07:32:52.056804 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:52.091599) v4 -- 0x55a333f6f800 con 0 > -8499> 2018-07-11 07:32:52.056833 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12651 conn(0x55a32169f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1826 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:32:52.091599) v4 > -8498> 2018-07-11 07:32:52.056854 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.2 10.8.128.106:0/12651 1826 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:52.091599) v4 ==== 2004+0+0 (2663635615 0 0) 0x55a332e79a00 con 0x55a32169f000 > -8497> 2018-07-11 07:32:52.056865 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:52.091599) v4 -- 0x55a314658600 con 0 > -8496> 2018-07-11 07:32:52.139271 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b7000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1771 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:32:51.889577) v4 > -8495> 2018-07-11 07:32:52.139268 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b8800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1771 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:32:51.889577) v4 > -8494> 2018-07-11 07:32:52.139297 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.8 10.8.128.108:0/11967 1771 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:51.889577) v4 ==== 2004+0+0 (1473081454 0 0) 0x55a34de78600 con 0x55a3170b7000 > -8493> 2018-07-11 07:32:52.139302 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.8 10.8.128.108:0/11967 1771 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:51.889577) v4 ==== 2004+0+0 (1473081454 0 0) 0x55a324474c00 con 0x55a3170b8800 > -8492> 2018-07-11 07:32:52.139309 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:51.889577) v4 -- 0x55a333f6ee00 con 0 > -8491> 2018-07-11 07:32:52.139345 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:51.889577) v4 -- 0x55a333f6fa00 con 0 > -8490> 2018-07-11 07:32:52.249987 7f35488a3700 1 -- 10.8.128.107:6800/12665 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55a317051b80 con 0 > -8489> 2018-07-11 07:32:52.250056 7f35488a3700 1 -- 10.8.128.107:6800/12665 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55a317056dc0 con 0 > -8488> 2018-07-11 07:32:52.366536 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12650 conn(0x55a311b68800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1835 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:32:52.401420) v4 > -8487> 2018-07-11 07:32:52.366536 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12650 conn(0x55a311b67000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1835 0x55a31fa79e00 osd_ping(ping e494 stamp 2018-07-11 07:32:52.401420) v4 > -8486> 2018-07-11 07:32:52.366566 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.3 10.8.128.106:0/12650 1835 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:52.401420) v4 ==== 2004+0+0 (165125370 0 0) 0x55a34de6f000 con 0x55a311b68800 > -8485> 2018-07-11 07:32:52.366568 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.3 10.8.128.106:0/12650 1835 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:52.401420) v4 ==== 2004+0+0 (165125370 0 0) 0x55a31fa79e00 con 0x55a311b67000 > -8484> 2018-07-11 07:32:52.366587 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:52.401420) v4 -- 0x55a31d142200 con 0 > -8483> 2018-07-11 07:32:52.366618 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:52.401420) v4 -- 0x55a31fa78600 con 0 > -8482> 2018-07-11 07:32:52.619566 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15255 conn(0x55a311b65800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1847 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:32:53.832164) v4 > -8481> 2018-07-11 07:32:52.619587 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.0 10.8.128.113:0/15255 1847 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:53.832164) v4 ==== 2004+0+0 (3012463787 0 0) 0x55a324474c00 con 0x55a311b65800 > -8480> 2018-07-11 07:32:52.619579 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15255 conn(0x55a311b64000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1847 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:32:53.832164) v4 > -8479> 2018-07-11 07:32:52.619596 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:53.832164) v4 -- 0x55a31d143e00 con 0 > -8478> 2018-07-11 07:32:52.619601 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.0 10.8.128.113:0/15255 1847 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:53.832164) v4 ==== 2004+0+0 (3012463787 0 0) 0x55a34de78600 con 0x55a311b64000 > -8477> 2018-07-11 07:32:52.619614 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:53.832164) v4 -- 0x55a323f54600 con 0 > -8476> 2018-07-11 07:32:52.639843 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b7000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1772 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:32:52.390143) v4 > -8475> 2018-07-11 07:32:52.639864 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.8 10.8.128.108:0/11967 1772 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:52.390143) v4 ==== 2004+0+0 (3650597489 0 0) 0x55a34de6f000 con 0x55a3170b7000 > -8474> 2018-07-11 07:32:52.639879 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:52.390143) v4 -- 0x55a3170dfe00 con 0 > -8473> 2018-07-11 07:32:52.639876 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b8800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1772 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:32:52.390143) v4 > -8472> 2018-07-11 07:32:52.639893 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.8 10.8.128.108:0/11967 1772 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:52.390143) v4 ==== 2004+0+0 (3650597489 0 0) 0x55a332e79a00 con 0x55a3170b8800 > -8471> 2018-07-11 07:32:52.639905 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:52.390143) v4 -- 0x55a3342d5a00 con 0 > -8470> 2018-07-11 07:32:52.732978 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a323ea7200 con 0 > -8469> 2018-07-11 07:32:52.748893 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5039 0x55a311168a00 replica scrub(pg: 6.15,from:0'0,to:430'106093,epoch:494/492,start:6:a99696ed:::benchmark_data_magna108_35617_object448889:0,end:6:a997832f:::benchmark_data_magna108_35617_object1058602:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -8468> 2018-07-11 07:32:52.748922 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5039 ==== replica scrub(pg: 6.15,from:0'0,to:430'106093,epoch:494/492,start:6:a99696ed:::benchmark_data_magna108_35617_object448889:0,end:6:a997832f:::benchmark_data_magna108_35617_object1058602:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 219+0+0 (1629546901 0 0) 0x55a311168a00 con 0x55a317019000 > -8467> 2018-07-11 07:32:53.317534 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a32d680240 con 0 > -8466> 2018-07-11 07:32:53.333068 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5040 0x55a311168c80 replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:494/492,start:6:a997832f:::benchmark_data_magna108_35617_object1058602:0,end:6:a998b2a6:::benchmark_data_magna108.ceph.redhat.com_2478_object343743:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -8465> 2018-07-11 07:32:53.333091 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5040 ==== replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:494/492,start:6:a997832f:::benchmark_data_magna108_35617_object1058602:0,end:6:a998b2a6:::benchmark_data_magna108.ceph.redhat.com_2478_object343743:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 234+0+0 (335481772 0 0) 0x55a311168c80 con 0x55a317019000 > -8464> 2018-07-11 07:32:53.339341 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.115:0/13323 conn(0x55a321665800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1833 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:32:53.676271) v4 > -8463> 2018-07-11 07:32:53.339365 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.6 10.8.128.115:0/13323 1833 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:53.676271) v4 ==== 2004+0+0 (636175707 0 0) 0x55a324474c00 con 0x55a321665800 > -8462> 2018-07-11 07:32:53.339374 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:53.676271) v4 -- 0x55a3342d5a00 con 0 > -8461> 2018-07-11 07:32:53.339415 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.115:0/13323 conn(0x55a321668800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1833 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:32:53.676271) v4 > -8460> 2018-07-11 07:32:53.339428 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.6 10.8.128.115:0/13323 1833 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:53.676271) v4 ==== 2004+0+0 (636175707 0 0) 0x55a324474c00 con 0x55a321668800 > -8459> 2018-07-11 07:32:53.339437 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:53.676271) v4 -- 0x55a343718c00 con 0 > -8458> 2018-07-11 07:32:53.439860 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4d00000 on bdev 1, free 0x2fe000; fallback to bdev 2 > -8457> 2018-07-11 07:32:53.466995 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12650 conn(0x55a311b68800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1836 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:32:53.501870) v4 > -8456> 2018-07-11 07:32:53.466995 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12650 conn(0x55a311b67000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1836 0x55a3342d4c00 osd_ping(ping e494 stamp 2018-07-11 07:32:53.501870) v4 > -8455> 2018-07-11 07:32:53.467025 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.3 10.8.128.106:0/12650 1836 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:53.501870) v4 ==== 2004+0+0 (492374483 0 0) 0x55a34de78600 con 0x55a311b68800 > -8454> 2018-07-11 07:32:53.467027 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.3 10.8.128.106:0/12650 1836 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:53.501870) v4 ==== 2004+0+0 (492374483 0 0) 0x55a3342d4c00 con 0x55a311b67000 > -8453> 2018-07-11 07:32:53.467046 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:53.501870) v4 -- 0x55a332355600 con 0 > -8452> 2018-07-11 07:32:53.467077 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:53.501870) v4 -- 0x55a31fa78c00 con 0 > -8451> 2018-07-11 07:32:53.967285 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12650 conn(0x55a311b67000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1837 0x55a31fa78600 osd_ping(ping e494 stamp 2018-07-11 07:32:54.002249) v4 > -8450> 2018-07-11 07:32:53.967288 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12650 conn(0x55a311b68800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1837 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:32:54.002249) v4 > -8449> 2018-07-11 07:32:53.967313 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.3 10.8.128.106:0/12650 1837 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:54.002249) v4 ==== 2004+0+0 (3042195405 0 0) 0x55a31fa78600 con 0x55a311b67000 > -8448> 2018-07-11 07:32:53.967315 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.3 10.8.128.106:0/12650 1837 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:54.002249) v4 ==== 2004+0+0 (3042195405 0 0) 0x55a34de6f000 con 0x55a311b68800 > -8447> 2018-07-11 07:32:53.967332 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:54.002249) v4 -- 0x55a333f6fc00 con 0 > -8446> 2018-07-11 07:32:53.967361 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:54.002249) v4 -- 0x55a33234a200 con 0 > -8445> 2018-07-11 07:32:54.208963 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15263 conn(0x55a321664000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1818 0x55a3342d4c00 osd_ping(ping e494 stamp 2018-07-11 07:32:55.421590) v4 > -8444> 2018-07-11 07:32:54.208984 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.5 10.8.128.113:0/15263 1818 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:55.421590) v4 ==== 2004+0+0 (2942838106 0 0) 0x55a3342d4c00 con 0x55a321664000 > -8443> 2018-07-11 07:32:54.208994 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:55.421590) v4 -- 0x55a31fa78a00 con 0 > -8442> 2018-07-11 07:32:54.209033 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15263 conn(0x55a321667000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1818 0x55a3342d4c00 osd_ping(ping e494 stamp 2018-07-11 07:32:55.421590) v4 > -8441> 2018-07-11 07:32:54.209045 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.5 10.8.128.113:0/15263 1818 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:55.421590) v4 ==== 2004+0+0 (2942838106 0 0) 0x55a3342d4c00 con 0x55a321667000 > -8440> 2018-07-11 07:32:54.209055 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:55.421590) v4 -- 0x55a31fa78e00 con 0 > -8439> 2018-07-11 07:32:54.471827 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 58] Generated table #36568: 4957164 keys, 80348260 bytes > -8438> 2018-07-11 07:32:54.471881 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294374471852, "cf_name": "default", "job": 58, "event": "table_file_creation", "file_number": 36568, "file_size": 80348260, "table_properties": {"data_size": 67109335, "index_size": 689237, "filter_size": 12548698, "raw_key_size": 143206976, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957164, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957164", "kMergeOperands": "0"}} > -8437> 2018-07-11 07:32:54.619189 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12677 conn(0x55a32169d800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1842 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:32:54.654071) v4 > -8436> 2018-07-11 07:32:54.619213 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.1 10.8.128.106:0/12677 1842 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:54.654071) v4 ==== 2004+0+0 (1284989889 0 0) 0x55a34de78600 con 0x55a32169d800 > -8435> 2018-07-11 07:32:54.619206 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12677 conn(0x55a32169c000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1842 0x55a31fa78600 osd_ping(ping e494 stamp 2018-07-11 07:32:54.654071) v4 > -8434> 2018-07-11 07:32:54.619227 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:54.654071) v4 -- 0x55a32eb21600 con 0 > -8433> 2018-07-11 07:32:54.619226 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.1 10.8.128.106:0/12677 1842 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:54.654071) v4 ==== 2004+0+0 (1284989889 0 0) 0x55a31fa78600 con 0x55a32169c000 > -8432> 2018-07-11 07:32:54.619262 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:54.654071) v4 -- 0x55a31fa78a00 con 0 > -8431> 2018-07-11 07:32:54.943244 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a34a2f5200 con 0 > -8430> 2018-07-11 07:32:54.967206 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.107:0/12617 conn(0x55a32447e000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1794 0x55a31fa78600 osd_ping(ping e494 stamp 2018-07-11 07:32:54.966890) v4 > -8429> 2018-07-11 07:32:54.967228 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.9 10.8.128.107:0/12617 1794 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:54.966890) v4 ==== 2004+0+0 (1166665017 0 0) 0x55a31fa78600 con 0x55a32447e000 > -8428> 2018-07-11 07:32:54.967221 7f355d88e700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.107:0/12617 conn(0x55a32447f800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1794 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:32:54.966890) v4 > -8427> 2018-07-11 07:32:54.967237 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:54.966890) v4 -- 0x55a31fa78a00 con 0 > -8426> 2018-07-11 07:32:54.967259 7f355d88e700 1 -- 10.8.128.107:6803/12665 <== osd.9 10.8.128.107:0/12617 1794 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:54.966890) v4 ==== 2004+0+0 (1166665017 0 0) 0x55a34de6f000 con 0x55a32447f800 > -8425> 2018-07-11 07:32:54.967316 7f355d88e700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:54.966890) v4 -- 0x55a32eb21600 con 0 > -8424> 2018-07-11 07:32:54.969335 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5041 0x55a311168f00 replica scrub(pg: 6.15,from:0'0,to:461'106912,epoch:494/492,start:6:a998b2a6:::benchmark_data_magna108.ceph.redhat.com_2478_object343743:0,end:6:a99a26fb:::benchmark_data_magna108.ceph.redhat.com_2478_object360620:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -8423> 2018-07-11 07:32:54.969359 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5041 ==== replica scrub(pg: 6.15,from:0'0,to:461'106912,epoch:494/492,start:6:a998b2a6:::benchmark_data_magna108.ceph.redhat.com_2478_object343743:0,end:6:a99a26fb:::benchmark_data_magna108.ceph.redhat.com_2478_object360620:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 248+0+0 (1579930659 0 0) 0x55a311168f00 con 0x55a317019000 > -8422> 2018-07-11 07:32:55.058129 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15885 conn(0x55a311b76800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1842 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:32:55.681535) v4 > -8421> 2018-07-11 07:32:55.058132 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15885 conn(0x55a3170b5800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1842 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:32:55.681535) v4 > -8420> 2018-07-11 07:32:55.058154 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.7 10.8.128.114:0/15885 1842 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:55.681535) v4 ==== 2004+0+0 (3215206792 0 0) 0x55a34de78600 con 0x55a311b76800 > -8419> 2018-07-11 07:32:55.058158 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.7 10.8.128.114:0/15885 1842 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:55.681535) v4 ==== 2004+0+0 (3215206792 0 0) 0x55a332e79a00 con 0x55a3170b5800 > -8418> 2018-07-11 07:32:55.058165 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:55.681535) v4 -- 0x55a32eb21600 con 0 > -8417> 2018-07-11 07:32:55.058186 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:55.681535) v4 -- 0x55a343718c00 con 0 > -8416> 2018-07-11 07:32:55.353034 7f3543899700 5 osd.10 494 heartbeat: osd_stat(783 GB used, 147 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,0,1]) > -8415> 2018-07-11 07:32:55.353049 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6802/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:32:55.353048) v4 -- 0x55a32d6e6200 con 0 > -8414> 2018-07-11 07:32:55.353088 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6803/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:32:55.353048) v4 -- 0x55a319405e00 con 0 > -8413> 2018-07-11 07:32:55.353103 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6806/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:32:55.353048) v4 -- 0x55a3342d6800 con 0 > -8412> 2018-07-11 07:32:55.353118 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6807/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:32:55.353048) v4 -- 0x55a31fa78000 con 0 > -8411> 2018-07-11 07:32:55.353138 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6808/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:32:55.353048) v4 -- 0x55a31fa78200 con 0 > -8410> 2018-07-11 07:32:55.353160 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6810/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:32:55.353048) v4 -- 0x55a333f6f600 con 0 > -8409> 2018-07-11 07:32:55.353184 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6809/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:32:55.353048) v4 -- 0x55a34256c200 con 0 > -8408> 2018-07-11 07:32:55.353200 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6811/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:32:55.353048) v4 -- 0x55a34256c600 con 0 > -8407> 2018-07-11 07:32:55.353220 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6810/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:32:55.353048) v4 -- 0x55a3484ff600 con 0 > -8406> 2018-07-11 07:32:55.353242 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6811/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:32:55.353048) v4 -- 0x55a34256cc00 con 0 > -8405> 2018-07-11 07:32:55.353283 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6806/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:32:55.353048) v4 -- 0x55a3484ffc00 con 0 > -8404> 2018-07-11 07:32:55.353304 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6807/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:32:55.353048) v4 -- 0x55a34256d000 con 0 > -8403> 2018-07-11 07:32:55.353326 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6802/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:32:55.353048) v4 -- 0x55a34256ca00 con 0 > -8402> 2018-07-11 07:32:55.353378 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6803/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:32:55.353048) v4 -- 0x55a34256c800 con 0 > -8401> 2018-07-11 07:32:55.353399 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6806/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:32:55.353048) v4 -- 0x55a34256ce00 con 0 > -8400> 2018-07-11 07:32:55.353420 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6807/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:32:55.353048) v4 -- 0x55a34e291a00 con 0 > -8399> 2018-07-11 07:32:55.353445 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6802/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:32:55.353048) v4 -- 0x55a3484fc600 con 0 > -8398> 2018-07-11 07:32:55.353468 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6803/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:32:55.353048) v4 -- 0x55a31d142000 con 0 > -8397> 2018-07-11 07:32:55.353491 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6806/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:32:55.353048) v4 -- 0x55a3484fc000 con 0 > -8396> 2018-07-11 07:32:55.353514 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6807/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:32:55.353048) v4 -- 0x55a3484fc400 con 0 > -8395> 2018-07-11 07:32:55.353558 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6802/15255 conn(0x55a314e1c000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1821 0x55a3342d4c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:55.353048) v4 > -8394> 2018-07-11 07:32:55.353579 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6809/12650 conn(0x55a314e1f000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1821 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:55.353048) v4 > -8393> 2018-07-11 07:32:55.353587 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6802/15255 1821 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:55.353048) v4 ==== 2004+0+0 (17442626 0 0) 0x55a3342d4c00 con 0x55a314e1c000 > -8392> 2018-07-11 07:32:55.353567 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6802/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:32:55.353048) v4 -- 0x55a3484fce00 con 0 > -8391> 2018-07-11 07:32:55.353599 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6809/12650 1821 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:55.353048) v4 ==== 2004+0+0 (17442626 0 0) 0x55a34de6f000 con 0x55a314e1f000 > -8390> 2018-07-11 07:32:55.353594 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6803/15255 conn(0x55a314e1d800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1821 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:55.353048) v4 > -8389> 2018-07-11 07:32:55.353617 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6803/15255 1821 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:55.353048) v4 ==== 2004+0+0 (17442626 0 0) 0x55a324474c00 con 0x55a314e1d800 > -8388> 2018-07-11 07:32:55.353619 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6803/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:32:55.353048) v4 -- 0x55a3484fc200 con 0 > -8387> 2018-07-11 07:32:55.353717 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6807/12677 conn(0x55a314e85800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1821 0x55a3342d4c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:55.353048) v4 > -8386> 2018-07-11 07:32:55.353724 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6810/12651 conn(0x55a314e97800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1821 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:55.353048) v4 > -8385> 2018-07-11 07:32:55.353765 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6807/12677 1821 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:55.353048) v4 ==== 2004+0+0 (17442626 0 0) 0x55a3342d4c00 con 0x55a314e85800 > -8384> 2018-07-11 07:32:55.353770 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6806/12677 conn(0x55a314e84000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1821 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:55.353048) v4 > -8383> 2018-07-11 07:32:55.353770 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6810/12651 1821 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:55.353048) v4 ==== 2004+0+0 (17442626 0 0) 0x55a324474c00 con 0x55a314e97800 > -8382> 2018-07-11 07:32:55.353783 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6806/12677 1821 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:55.353048) v4 ==== 2004+0+0 (17442626 0 0) 0x55a34de78600 con 0x55a314e84000 > -8381> 2018-07-11 07:32:55.353830 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6806/15263 conn(0x55a314e99000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1821 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:55.353048) v4 > -8380> 2018-07-11 07:32:55.353817 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6808/12651 conn(0x55a314e96000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1821 0x55a3342d4c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:55.353048) v4 > -8379> 2018-07-11 07:32:55.353844 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6811/15287 conn(0x55a314e04800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1821 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:55.353048) v4 > -8378> 2018-07-11 07:32:55.353849 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6806/15263 1821 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:55.353048) v4 ==== 2004+0+0 (17442626 0 0) 0x55a34de78600 con 0x55a314e99000 > -8377> 2018-07-11 07:32:55.353861 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6811/15287 1821 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:55.353048) v4 ==== 2004+0+0 (17442626 0 0) 0x55a332e79a00 con 0x55a314e04800 > -8376> 2018-07-11 07:32:55.353850 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6808/12651 1821 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:55.353048) v4 ==== 2004+0+0 (17442626 0 0) 0x55a3342d4c00 con 0x55a314e96000 > -8375> 2018-07-11 07:32:55.353888 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6806/12617 conn(0x55a314e26000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1821 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:55.353048) v4 > -8374> 2018-07-11 07:32:55.353895 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6803/13323 conn(0x55a314e16800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1821 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:55.353048) v4 > -8373> 2018-07-11 07:32:55.353908 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6803/13323 1821 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:55.353048) v4 ==== 2004+0+0 (17442626 0 0) 0x55a34de78600 con 0x55a314e16800 > -8372> 2018-07-11 07:32:55.353898 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6806/12617 1821 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:55.353048) v4 ==== 2004+0+0 (17442626 0 0) 0x55a332e79a00 con 0x55a314e26000 > -8371> 2018-07-11 07:32:55.353916 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6807/15263 conn(0x55a314e9a800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1821 0x55a3342d4c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:55.353048) v4 > -8370> 2018-07-11 07:32:55.353934 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6807/15263 1821 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:55.353048) v4 ==== 2004+0+0 (17442626 0 0) 0x55a3342d4c00 con 0x55a314e9a800 > -8369> 2018-07-11 07:32:55.353943 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6802/13323 conn(0x55a314e15000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1821 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:55.353048) v4 > -8368> 2018-07-11 07:32:55.353945 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6807/15885 conn(0x55a314e01800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1821 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:55.353048) v4 > -8367> 2018-07-11 07:32:55.353955 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6802/13323 1821 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:55.353048) v4 ==== 2004+0+0 (17442626 0 0) 0x55a332e79a00 con 0x55a314e15000 > -8366> 2018-07-11 07:32:55.353956 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6807/15885 1821 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:55.353048) v4 ==== 2004+0+0 (17442626 0 0) 0x55a34de78600 con 0x55a314e01800 > -8365> 2018-07-11 07:32:55.353975 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6810/15287 conn(0x55a314e03000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1821 0x55a3342d4c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:55.353048) v4 > -8364> 2018-07-11 07:32:55.353985 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6806/15885 conn(0x55a314e00000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1821 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:55.353048) v4 > -8363> 2018-07-11 07:32:55.353995 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6806/15885 1821 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:55.353048) v4 ==== 2004+0+0 (17442626 0 0) 0x55a332e79a00 con 0x55a314e00000 > -8362> 2018-07-11 07:32:55.353993 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6810/15287 1821 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:55.353048) v4 ==== 2004+0+0 (17442626 0 0) 0x55a3342d4c00 con 0x55a314e03000 > -8361> 2018-07-11 07:32:55.353995 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6807/12617 conn(0x55a314e27800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1821 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:55.353048) v4 > -8360> 2018-07-11 07:32:55.354006 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6807/12617 1821 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:55.353048) v4 ==== 2004+0+0 (17442626 0 0) 0x55a34de78600 con 0x55a314e27800 > -8359> 2018-07-11 07:32:55.354039 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6802/11967 conn(0x55a314e12000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1821 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:55.353048) v4 > -8358> 2018-07-11 07:32:55.354040 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6811/12650 conn(0x55a314e20800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1821 0x55a3342d4c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:55.353048) v4 > -8357> 2018-07-11 07:32:55.354052 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6802/11967 1821 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:55.353048) v4 ==== 2004+0+0 (17442626 0 0) 0x55a34de78600 con 0x55a314e12000 > -8356> 2018-07-11 07:32:55.354055 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6811/12650 1821 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:55.353048) v4 ==== 2004+0+0 (17442626 0 0) 0x55a3342d4c00 con 0x55a314e20800 > -8355> 2018-07-11 07:32:55.354103 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6802/15919 conn(0x55a314e87000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1821 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:55.353048) v4 > -8354> 2018-07-11 07:32:55.354105 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6803/11967 conn(0x55a314e13800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1821 0x55a3342d4c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:55.353048) v4 > -8353> 2018-07-11 07:32:55.354120 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6802/15919 1821 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:55.353048) v4 ==== 2004+0+0 (17442626 0 0) 0x55a332e79a00 con 0x55a314e87000 > -8352> 2018-07-11 07:32:55.354120 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6803/11967 1821 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:55.353048) v4 ==== 2004+0+0 (17442626 0 0) 0x55a3342d4c00 con 0x55a314e13800 > -8351> 2018-07-11 07:32:55.354139 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6803/15919 conn(0x55a314e88800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1821 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:55.353048) v4 > -8350> 2018-07-11 07:32:55.354153 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6803/15919 1821 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:55.353048) v4 ==== 2004+0+0 (17442626 0 0) 0x55a34de78600 con 0x55a314e88800 > -8349> 2018-07-11 07:32:55.520011 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15255 conn(0x55a311b65800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1848 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:32:56.732610) v4 > -8348> 2018-07-11 07:32:55.520032 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.0 10.8.128.113:0/15255 1848 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:56.732610) v4 ==== 2004+0+0 (1779868600 0 0) 0x55a332e79a00 con 0x55a311b65800 > -8347> 2018-07-11 07:32:55.520045 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:56.732610) v4 -- 0x55a333f6f600 con 0 > -8346> 2018-07-11 07:32:55.520138 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15255 conn(0x55a311b64000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1848 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:32:56.732610) v4 > -8345> 2018-07-11 07:32:55.520158 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.0 10.8.128.113:0/15255 1848 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:56.732610) v4 ==== 2004+0+0 (1779868600 0 0) 0x55a34de78600 con 0x55a311b64000 > -8344> 2018-07-11 07:32:55.520169 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:56.732610) v4 -- 0x55a3484ffc00 con 0 > -8343> 2018-07-11 07:32:55.720397 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15287 conn(0x55a311b75000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1816 0x55a3342d4c00 osd_ping(ping e494 stamp 2018-07-11 07:32:56.933137) v4 > -8342> 2018-07-11 07:32:55.720418 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.4 10.8.128.113:0/15287 1816 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:56.933137) v4 ==== 2004+0+0 (3825411364 0 0) 0x55a3342d4c00 con 0x55a311b75000 > -8341> 2018-07-11 07:32:55.720434 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:56.933137) v4 -- 0x55a31fa78000 con 0 > -8340> 2018-07-11 07:32:55.720425 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15287 conn(0x55a311b72000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1816 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:32:56.933137) v4 > -8339> 2018-07-11 07:32:55.720445 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.4 10.8.128.113:0/15287 1816 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:56.933137) v4 ==== 2004+0+0 (3825411364 0 0) 0x55a324474c00 con 0x55a311b72000 > -8338> 2018-07-11 07:32:55.720459 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:56.933137) v4 -- 0x55a319405e00 con 0 > -8337> 2018-07-11 07:32:55.872034 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a323ea8900 con 0 > -8336> 2018-07-11 07:32:55.899846 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5042 0x55a311169400 replica scrub(pg: 6.15,from:0'0,to:430'106235,epoch:494/492,start:6:a99a26fb:::benchmark_data_magna108.ceph.redhat.com_2478_object360620:0,end:6:a99b1e7c:::benchmark_data_magna108_35617_object527568:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -8335> 2018-07-11 07:32:55.899876 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5042 ==== replica scrub(pg: 6.15,from:0'0,to:430'106235,epoch:494/492,start:6:a99a26fb:::benchmark_data_magna108.ceph.redhat.com_2478_object360620:0,end:6:a99b1e7c:::benchmark_data_magna108_35617_object527568:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (3984061594 0 0) 0x55a311169400 con 0x55a317019000 > -8334> 2018-07-11 07:32:56.571918 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a333181200 con 0 > -8333> 2018-07-11 07:32:56.583295 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5043 0x55a311169680 replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:494/492,start:6:a99b1e7c:::benchmark_data_magna108_35617_object527568:0,end:6:a99c655e:::benchmark_data_magna108.ceph.redhat.com_2478_object181139:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -8332> 2018-07-11 07:32:56.583319 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5043 ==== replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:494/492,start:6:a99b1e7c:::benchmark_data_magna108_35617_object527568:0,end:6:a99c655e:::benchmark_data_magna108.ceph.redhat.com_2478_object181139:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (1942839839 0 0) 0x55a311169680 con 0x55a317019000 > -8331> 2018-07-11 07:32:56.667731 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.107:0/12617 conn(0x55a32447e000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1795 0x55a31fa78600 osd_ping(ping e494 stamp 2018-07-11 07:32:56.667406) v4 > -8330> 2018-07-11 07:32:56.667754 7f355d88e700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.107:0/12617 conn(0x55a32447f800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1795 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:32:56.667406) v4 > -8329> 2018-07-11 07:32:56.667758 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.9 10.8.128.107:0/12617 1795 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:56.667406) v4 ==== 2004+0+0 (2542160197 0 0) 0x55a31fa78600 con 0x55a32447e000 > -8328> 2018-07-11 07:32:56.667776 7f355d88e700 1 -- 10.8.128.107:6803/12665 <== osd.9 10.8.128.107:0/12617 1795 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:56.667406) v4 ==== 2004+0+0 (2542160197 0 0) 0x55a34de6f000 con 0x55a32447f800 > -8327> 2018-07-11 07:32:56.667780 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:56.667406) v4 -- 0x55a32d6e6200 con 0 > -8326> 2018-07-11 07:32:56.667820 7f355d88e700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:56.667406) v4 -- 0x55a3484ffc00 con 0 > -8325> 2018-07-11 07:32:56.743651 7f35498a5700 10 monclient: tick > -8324> 2018-07-11 07:32:56.743676 7f35498a5700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2018-07-11 07:32:26.743674) > -8323> 2018-07-11 07:32:56.949287 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15919 conn(0x55a3170b4000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1828 0x55a31fa78600 osd_ping(ping e494 stamp 2018-07-11 07:32:57.572709) v4 > -8322> 2018-07-11 07:32:56.949290 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15919 conn(0x55a317074800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1828 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:32:57.572709) v4 > -8321> 2018-07-11 07:32:56.949309 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.11 10.8.128.114:0/15919 1828 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:57.572709) v4 ==== 2004+0+0 (3506231878 0 0) 0x55a31fa78600 con 0x55a3170b4000 > -8320> 2018-07-11 07:32:56.949312 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.11 10.8.128.114:0/15919 1828 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:57.572709) v4 ==== 2004+0+0 (3506231878 0 0) 0x55a332e79a00 con 0x55a317074800 > -8319> 2018-07-11 07:32:56.949319 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:57.572709) v4 -- 0x55a32d6e6200 con 0 > -8318> 2018-07-11 07:32:56.949345 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:57.572709) v4 -- 0x55a319405e00 con 0 > -8317> 2018-07-11 07:32:57.053764 7f3543899700 5 osd.10 494 heartbeat: osd_stat(783 GB used, 147 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,0,1]) > -8316> 2018-07-11 07:32:57.053778 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6802/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:32:57.053776) v4 -- 0x55a3484fcc00 con 0 > -8315> 2018-07-11 07:32:57.053814 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6803/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:32:57.053776) v4 -- 0x55a3484fd400 con 0 > -8314> 2018-07-11 07:32:57.053829 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6806/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:32:57.053776) v4 -- 0x55a3484fc800 con 0 > -8313> 2018-07-11 07:32:57.053844 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6807/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:32:57.053776) v4 -- 0x55a3484fd200 con 0 > -8312> 2018-07-11 07:32:57.053858 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6808/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:32:57.053776) v4 -- 0x55a3484fd800 con 0 > -8311> 2018-07-11 07:32:57.053866 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6810/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:32:57.053776) v4 -- 0x55a3484fda00 con 0 > -8310> 2018-07-11 07:32:57.053882 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6809/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:32:57.053776) v4 -- 0x55a3484fca00 con 0 > -8309> 2018-07-11 07:32:57.053900 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6811/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:32:57.053776) v4 -- 0x55a3484fd600 con 0 > -8308> 2018-07-11 07:32:57.053914 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6810/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:32:57.053776) v4 -- 0x55a3484fe000 con 0 > -8307> 2018-07-11 07:32:57.053924 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6811/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:32:57.053776) v4 -- 0x55a3484fd000 con 0 > -8306> 2018-07-11 07:32:57.053936 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6806/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:32:57.053776) v4 -- 0x55a3484fde00 con 0 > -8305> 2018-07-11 07:32:57.053956 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6807/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:32:57.053776) v4 -- 0x55a3484fdc00 con 0 > -8304> 2018-07-11 07:32:57.053975 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6802/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:32:57.053776) v4 -- 0x55a3484fe400 con 0 > -8303> 2018-07-11 07:32:57.053990 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6803/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:32:57.053776) v4 -- 0x55a3484fe200 con 0 > -8302> 2018-07-11 07:32:57.054006 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6806/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:32:57.053776) v4 -- 0x55a34e291400 con 0 > -8301> 2018-07-11 07:32:57.054018 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6807/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:32:57.053776) v4 -- 0x55a3282bae00 con 0 > -8300> 2018-07-11 07:32:57.054035 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6802/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:32:57.053776) v4 -- 0x55a32316de00 con 0 > -8299> 2018-07-11 07:32:57.054048 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6803/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:32:57.053776) v4 -- 0x55a329322200 con 0 > -8298> 2018-07-11 07:32:57.054061 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6806/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:32:57.053776) v4 -- 0x55a329322400 con 0 > -8297> 2018-07-11 07:32:57.054102 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6807/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:32:57.053776) v4 -- 0x55a329322000 con 0 > -8296> 2018-07-11 07:32:57.054119 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6802/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:32:57.053776) v4 -- 0x55a329322600 con 0 > -8295> 2018-07-11 07:32:57.054131 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6803/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:32:57.053776) v4 -- 0x55a329322800 con 0 > -8294> 2018-07-11 07:32:57.054317 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6802/15255 conn(0x55a314e1c000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1822 0x55a31fa78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:57.053776) v4 > -8293> 2018-07-11 07:32:57.054335 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6802/15255 1822 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:57.053776) v4 ==== 2004+0+0 (1908227226 0 0) 0x55a31fa78600 con 0x55a314e1c000 > -8292> 2018-07-11 07:32:57.054330 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6803/15255 conn(0x55a314e1d800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1822 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:57.053776) v4 > -8291> 2018-07-11 07:32:57.054352 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6803/15255 1822 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:57.053776) v4 ==== 2004+0+0 (1908227226 0 0) 0x55a332e79a00 con 0x55a314e1d800 > -8290> 2018-07-11 07:32:57.054373 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6808/12651 conn(0x55a314e96000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1822 0x55a31fa78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:57.053776) v4 > -8289> 2018-07-11 07:32:57.054378 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6806/12617 conn(0x55a314e26000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1822 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:57.053776) v4 > -8288> 2018-07-11 07:32:57.054386 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6808/12651 1822 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:57.053776) v4 ==== 2004+0+0 (1908227226 0 0) 0x55a31fa78600 con 0x55a314e96000 > -8287> 2018-07-11 07:32:57.054360 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6809/12650 conn(0x55a314e1f000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1822 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:57.053776) v4 > -8286> 2018-07-11 07:32:57.054397 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6809/12650 1822 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:57.053776) v4 ==== 2004+0+0 (1908227226 0 0) 0x55a34de78600 con 0x55a314e1f000 > -8285> 2018-07-11 07:32:57.054387 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6806/12617 1822 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:57.053776) v4 ==== 2004+0+0 (1908227226 0 0) 0x55a332e79a00 con 0x55a314e26000 > -8284> 2018-07-11 07:32:57.054426 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6807/12617 conn(0x55a314e27800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1822 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:57.053776) v4 > -8283> 2018-07-11 07:32:57.054426 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6807/12677 conn(0x55a314e85800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1822 0x55a31fa78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:57.053776) v4 > -8282> 2018-07-11 07:32:57.054439 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6807/12617 1822 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:57.053776) v4 ==== 2004+0+0 (1908227226 0 0) 0x55a34de78600 con 0x55a314e27800 > -8281> 2018-07-11 07:32:57.054442 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6807/12677 1822 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:57.053776) v4 ==== 2004+0+0 (1908227226 0 0) 0x55a31fa78600 con 0x55a314e85800 > -8280> 2018-07-11 07:32:57.054444 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6811/15287 conn(0x55a314e04800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1822 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:57.053776) v4 > -8279> 2018-07-11 07:32:57.054457 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6811/15287 1822 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:57.053776) v4 ==== 2004+0+0 (1908227226 0 0) 0x55a332e79a00 con 0x55a314e04800 > -8278> 2018-07-11 07:32:57.054468 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6806/12677 conn(0x55a314e84000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1822 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:57.053776) v4 > -8277> 2018-07-11 07:32:57.054481 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6810/15287 conn(0x55a314e03000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1822 0x55a31fa78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:57.053776) v4 > -8276> 2018-07-11 07:32:57.054495 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6806/12677 1822 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:57.053776) v4 ==== 2004+0+0 (1908227226 0 0) 0x55a34de78600 con 0x55a314e84000 > -8275> 2018-07-11 07:32:57.054495 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6810/12651 conn(0x55a314e97800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1822 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:57.053776) v4 > -8274> 2018-07-11 07:32:57.054500 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6810/15287 1822 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:57.053776) v4 ==== 2004+0+0 (1908227226 0 0) 0x55a31fa78600 con 0x55a314e03000 > -8273> 2018-07-11 07:32:57.054505 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6810/12651 1822 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:57.053776) v4 ==== 2004+0+0 (1908227226 0 0) 0x55a332e79a00 con 0x55a314e97800 > -8272> 2018-07-11 07:32:57.054528 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6806/15263 conn(0x55a314e99000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1822 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:57.053776) v4 > -8271> 2018-07-11 07:32:57.054532 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6807/15263 conn(0x55a314e9a800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1822 0x55a31fa78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:57.053776) v4 > -8270> 2018-07-11 07:32:57.054536 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6806/15885 conn(0x55a314e00000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1822 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:57.053776) v4 > -8269> 2018-07-11 07:32:57.054544 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6807/15263 1822 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:57.053776) v4 ==== 2004+0+0 (1908227226 0 0) 0x55a31fa78600 con 0x55a314e9a800 > -8268> 2018-07-11 07:32:57.054540 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6806/15263 1822 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:57.053776) v4 ==== 2004+0+0 (1908227226 0 0) 0x55a34de78600 con 0x55a314e99000 > -8267> 2018-07-11 07:32:57.054550 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6806/15885 1822 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:57.053776) v4 ==== 2004+0+0 (1908227226 0 0) 0x55a332e79a00 con 0x55a314e00000 > -8266> 2018-07-11 07:32:57.054575 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6811/12650 conn(0x55a314e20800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1822 0x55a31fa78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:57.053776) v4 > -8265> 2018-07-11 07:32:57.054587 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6811/12650 1822 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:57.053776) v4 ==== 2004+0+0 (1908227226 0 0) 0x55a31fa78600 con 0x55a314e20800 > -8264> 2018-07-11 07:32:57.054592 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6802/11967 conn(0x55a314e12000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1822 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:57.053776) v4 > -8263> 2018-07-11 07:32:57.054604 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6802/11967 1822 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:57.053776) v4 ==== 2004+0+0 (1908227226 0 0) 0x55a34de78600 con 0x55a314e12000 > -8262> 2018-07-11 07:32:57.054611 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6802/13323 conn(0x55a314e15000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1822 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:57.053776) v4 > -8261> 2018-07-11 07:32:57.054637 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6802/13323 1822 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:57.053776) v4 ==== 2004+0+0 (1908227226 0 0) 0x55a332e79a00 con 0x55a314e15000 > -8260> 2018-07-11 07:32:57.054642 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6803/13323 conn(0x55a314e16800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1822 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:57.053776) v4 > -8259> 2018-07-11 07:32:57.054652 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6803/13323 1822 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:57.053776) v4 ==== 2004+0+0 (1908227226 0 0) 0x55a34de78600 con 0x55a314e16800 > -8258> 2018-07-11 07:32:57.054653 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6803/11967 conn(0x55a314e13800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1822 0x55a31fa78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:57.053776) v4 > -8257> 2018-07-11 07:32:57.054670 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6803/11967 1822 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:57.053776) v4 ==== 2004+0+0 (1908227226 0 0) 0x55a31fa78600 con 0x55a314e13800 > -8256> 2018-07-11 07:32:57.054680 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6807/15885 conn(0x55a314e01800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1822 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:57.053776) v4 > -8255> 2018-07-11 07:32:57.054691 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6807/15885 1822 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:57.053776) v4 ==== 2004+0+0 (1908227226 0 0) 0x55a34de78600 con 0x55a314e01800 > -8254> 2018-07-11 07:32:57.054695 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6802/15919 conn(0x55a314e87000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1822 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:57.053776) v4 > -8253> 2018-07-11 07:32:57.054711 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6803/15919 conn(0x55a314e88800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1822 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:32:57.053776) v4 > -8252> 2018-07-11 07:32:57.054714 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6802/15919 1822 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:57.053776) v4 ==== 2004+0+0 (1908227226 0 0) 0x55a332e79a00 con 0x55a314e87000 > -8251> 2018-07-11 07:32:57.054721 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6803/15919 1822 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:32:57.053776) v4 ==== 2004+0+0 (1908227226 0 0) 0x55a34de78600 con 0x55a314e88800 > -8250> 2018-07-11 07:32:57.109298 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15263 conn(0x55a321667000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1819 0x55a31fa78600 osd_ping(ping e494 stamp 2018-07-11 07:32:58.322034) v4 > -8249> 2018-07-11 07:32:57.109321 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.5 10.8.128.113:0/15263 1819 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:58.322034) v4 ==== 2004+0+0 (3810304140 0 0) 0x55a31fa78600 con 0x55a321667000 > -8248> 2018-07-11 07:32:57.109337 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:58.322034) v4 -- 0x55a3484fcc00 con 0 > -8247> 2018-07-11 07:32:57.109373 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15263 conn(0x55a321664000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1819 0x55a31fa78600 osd_ping(ping e494 stamp 2018-07-11 07:32:58.322034) v4 > -8246> 2018-07-11 07:32:57.109387 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.5 10.8.128.113:0/15263 1819 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:58.322034) v4 ==== 2004+0+0 (3810304140 0 0) 0x55a31fa78600 con 0x55a321664000 > -8245> 2018-07-11 07:32:57.109397 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:58.322034) v4 -- 0x55a31fa78a00 con 0 > -8244> 2018-07-11 07:32:57.214508 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a324673200 con 0 > -8243> 2018-07-11 07:32:57.243785 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5044 0x55a311169900 replica scrub(pg: 6.15,from:0'0,to:430'106869,epoch:494/492,start:6:a99c655e:::benchmark_data_magna108.ceph.redhat.com_2478_object181139:0,end:6:a99db26a:::benchmark_data_magna108.ceph.redhat.com_2478_object162247:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -8242> 2018-07-11 07:32:57.243815 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5044 ==== replica scrub(pg: 6.15,from:0'0,to:430'106869,epoch:494/492,start:6:a99c655e:::benchmark_data_magna108.ceph.redhat.com_2478_object181139:0,end:6:a99db26a:::benchmark_data_magna108.ceph.redhat.com_2478_object162247:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 248+0+0 (3792082694 0 0) 0x55a311169900 con 0x55a317019000 > -8241> 2018-07-11 07:32:57.250258 7f35488a3700 1 -- 10.8.128.107:6800/12665 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55a317050b00 con 0 > -8240> 2018-07-11 07:32:57.250317 7f35488a3700 1 -- 10.8.128.107:6800/12665 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55a332fec580 con 0 > -8239> 2018-07-11 07:32:57.439852 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.115:0/13323 conn(0x55a321665800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1834 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:32:57.776750) v4 > -8238> 2018-07-11 07:32:57.439875 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.6 10.8.128.115:0/13323 1834 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:57.776750) v4 ==== 2004+0+0 (722110938 0 0) 0x55a332e79a00 con 0x55a321665800 > -8237> 2018-07-11 07:32:57.439891 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:57.776750) v4 -- 0x55a3484fda00 con 0 > -8236> 2018-07-11 07:32:57.439962 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.115:0/13323 conn(0x55a321668800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1834 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:32:57.776750) v4 > -8235> 2018-07-11 07:32:57.439979 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.6 10.8.128.115:0/13323 1834 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:57.776750) v4 ==== 2004+0+0 (722110938 0 0) 0x55a324474c00 con 0x55a321668800 > -8234> 2018-07-11 07:32:57.439988 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:57.776750) v4 -- 0x55a3484fda00 con 0 > -8233> 2018-07-11 07:32:57.898788 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a323539b00 con 0 > -8232> 2018-07-11 07:32:57.918505 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5045 0x55a311169b80 replica scrub(pg: 6.15,from:0'0,to:430'106622,epoch:494/492,start:6:a99db26a:::benchmark_data_magna108.ceph.redhat.com_2478_object162247:0,end:6:a99e83d7:::benchmark_data_magna108_35617_object839233:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -8231> 2018-07-11 07:32:57.918536 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5045 ==== replica scrub(pg: 6.15,from:0'0,to:430'106622,epoch:494/492,start:6:a99db26a:::benchmark_data_magna108.ceph.redhat.com_2478_object162247:0,end:6:a99e83d7:::benchmark_data_magna108_35617_object839233:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (891363337 0 0) 0x55a311169b80 con 0x55a317019000 > -8230> 2018-07-11 07:32:57.957336 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12651 conn(0x55a32169f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1827 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:32:57.992178) v4 > -8229> 2018-07-11 07:32:57.957361 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.2 10.8.128.106:0/12651 1827 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:57.992178) v4 ==== 2004+0+0 (1852832655 0 0) 0x55a332e79a00 con 0x55a32169f000 > -8228> 2018-07-11 07:32:57.957377 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:57.992178) v4 -- 0x55a3484fda00 con 0 > -8227> 2018-07-11 07:32:57.957373 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12651 conn(0x55a3216a0800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1827 0x55a31fa78000 osd_ping(ping e494 stamp 2018-07-11 07:32:57.992178) v4 > -8226> 2018-07-11 07:32:57.957392 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.2 10.8.128.106:0/12651 1827 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:57.992178) v4 ==== 2004+0+0 (1852832655 0 0) 0x55a31fa78000 con 0x55a3216a0800 > -8225> 2018-07-11 07:32:57.957405 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:57.992178) v4 -- 0x55a3484fcc00 con 0 > -8224> 2018-07-11 07:32:58.540369 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b7000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1773 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:32:58.290664) v4 > -8223> 2018-07-11 07:32:58.540394 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.8 10.8.128.108:0/11967 1773 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:58.290664) v4 ==== 2004+0+0 (2127930154 0 0) 0x55a34de78600 con 0x55a3170b7000 > -8222> 2018-07-11 07:32:58.540382 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b8800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1773 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:32:58.290664) v4 > -8221> 2018-07-11 07:32:58.540405 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:58.290664) v4 -- 0x55a3484fca00 con 0 > -8220> 2018-07-11 07:32:58.540404 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.8 10.8.128.108:0/11967 1773 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:58.290664) v4 ==== 2004+0+0 (2127930154 0 0) 0x55a324474c00 con 0x55a3170b8800 > -8219> 2018-07-11 07:32:58.540434 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:58.290664) v4 -- 0x55a3484fda00 con 0 > -8218> 2018-07-11 07:32:58.571226 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a332f06fc0 con 0 > -8217> 2018-07-11 07:32:58.583530 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5046 0x55a317922000 replica scrub(pg: 6.15,from:0'0,to:430'106668,epoch:494/492,start:6:a99e83d7:::benchmark_data_magna108_35617_object839233:0,end:6:a99f7360:::benchmark_data_magna108_35617_object108013:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -8216> 2018-07-11 07:32:58.583558 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5046 ==== replica scrub(pg: 6.15,from:0'0,to:430'106668,epoch:494/492,start:6:a99e83d7:::benchmark_data_magna108_35617_object839233:0,end:6:a99f7360:::benchmark_data_magna108_35617_object108013:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (1622448567 0 0) 0x55a317922000 con 0x55a317019000 > -8215> 2018-07-11 07:32:58.620822 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15287 conn(0x55a311b72000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1817 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:32:59.833578) v4 > -8214> 2018-07-11 07:32:58.620841 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15287 conn(0x55a311b75000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1817 0x55a31fa78a00 osd_ping(ping e494 stamp 2018-07-11 07:32:59.833578) v4 > -8213> 2018-07-11 07:32:58.620849 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.4 10.8.128.113:0/15287 1817 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:59.833578) v4 ==== 2004+0+0 (1081486803 0 0) 0x55a332e79a00 con 0x55a311b72000 > -8212> 2018-07-11 07:32:58.620865 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.4 10.8.128.113:0/15287 1817 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:59.833578) v4 ==== 2004+0+0 (1081486803 0 0) 0x55a31fa78a00 con 0x55a311b75000 > -8211> 2018-07-11 07:32:58.620870 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:59.833578) v4 -- 0x55a3484fda00 con 0 > -8210> 2018-07-11 07:32:58.620901 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:59.833578) v4 -- 0x55a3484fcc00 con 0 > -8209> 2018-07-11 07:32:58.719640 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12677 conn(0x55a32169c000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1843 0x55a31fa78000 osd_ping(ping e494 stamp 2018-07-11 07:32:58.754591) v4 > -8208> 2018-07-11 07:32:58.719652 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12677 conn(0x55a32169d800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1843 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:32:58.754591) v4 > -8207> 2018-07-11 07:32:58.719669 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.1 10.8.128.106:0/12677 1843 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:58.754591) v4 ==== 2004+0+0 (3574636966 0 0) 0x55a34de6f000 con 0x55a32169d800 > -8206> 2018-07-11 07:32:58.719668 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.1 10.8.128.106:0/12677 1843 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:58.754591) v4 ==== 2004+0+0 (3574636966 0 0) 0x55a31fa78000 con 0x55a32169c000 > -8205> 2018-07-11 07:32:58.719678 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:58.754591) v4 -- 0x55a3484fc800 con 0 > -8204> 2018-07-11 07:32:58.719704 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:58.754591) v4 -- 0x55a31fa79200 con 0 > -8203> 2018-07-11 07:32:58.809719 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15263 conn(0x55a321667000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1820 0x55a31fa78a00 osd_ping(ping e494 stamp 2018-07-11 07:33:00.022478) v4 > -8202> 2018-07-11 07:32:58.809746 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.5 10.8.128.113:0/15263 1820 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:00.022478) v4 ==== 2004+0+0 (132512811 0 0) 0x55a31fa78a00 con 0x55a321667000 > -8201> 2018-07-11 07:32:58.809762 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:00.022478) v4 -- 0x55a31fa78400 con 0 > -8200> 2018-07-11 07:32:58.809808 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15263 conn(0x55a321664000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1820 0x55a31fa78a00 osd_ping(ping e494 stamp 2018-07-11 07:33:00.022478) v4 > -8199> 2018-07-11 07:32:58.809825 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.5 10.8.128.113:0/15263 1820 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:00.022478) v4 ==== 2004+0+0 (132512811 0 0) 0x55a31fa78a00 con 0x55a321664000 > -8198> 2018-07-11 07:32:58.809834 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:00.022478) v4 -- 0x55a31fa79000 con 0 > -8197> 2018-07-11 07:32:59.158566 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15885 conn(0x55a311b76800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1843 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:32:59.781974) v4 > -8196> 2018-07-11 07:32:59.158587 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.7 10.8.128.114:0/15885 1843 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:59.781974) v4 ==== 2004+0+0 (348204589 0 0) 0x55a34de78600 con 0x55a311b76800 > -8195> 2018-07-11 07:32:59.158583 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15885 conn(0x55a3170b5800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1843 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:32:59.781974) v4 > -8194> 2018-07-11 07:32:59.158598 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:59.781974) v4 -- 0x55a3484ffc00 con 0 > -8193> 2018-07-11 07:32:59.158610 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.7 10.8.128.114:0/15885 1843 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:59.781974) v4 ==== 2004+0+0 (348204589 0 0) 0x55a324474c00 con 0x55a3170b5800 > -8192> 2018-07-11 07:32:59.158630 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:59.781974) v4 -- 0x55a3484fd400 con 0 > -8191> 2018-07-11 07:32:59.213010 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a32883b8c0 con 0 > -8190> 2018-07-11 07:32:59.235781 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5047 0x55a317922280 replica scrub(pg: 6.15,from:0'0,to:430'105887,epoch:494/492,start:6:a99f7360:::benchmark_data_magna108_35617_object108013:0,end:6:a9a087c8:::benchmark_data_magna108_41525_object15296:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -8189> 2018-07-11 07:32:59.235804 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5047 ==== replica scrub(pg: 6.15,from:0'0,to:430'105887,epoch:494/492,start:6:a99f7360:::benchmark_data_magna108_35617_object108013:0,end:6:a9a087c8:::benchmark_data_magna108_41525_object15296:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 217+0+0 (2867009946 0 0) 0x55a317922280 con 0x55a317019000 > -8188> 2018-07-11 07:32:59.659039 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15885 conn(0x55a311b76800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1844 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:33:00.282431) v4 > -8187> 2018-07-11 07:32:59.659039 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15885 conn(0x55a3170b5800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1844 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:33:00.282431) v4 > -8186> 2018-07-11 07:32:59.659070 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.7 10.8.128.114:0/15885 1844 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:00.282431) v4 ==== 2004+0+0 (2603293487 0 0) 0x55a34de6f000 con 0x55a311b76800 > -8185> 2018-07-11 07:32:59.659071 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.7 10.8.128.114:0/15885 1844 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:00.282431) v4 ==== 2004+0+0 (2603293487 0 0) 0x55a332e79a00 con 0x55a3170b5800 > -8184> 2018-07-11 07:32:59.659090 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:00.282431) v4 -- 0x55a3484ffc00 con 0 > -8183> 2018-07-11 07:32:59.659121 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:00.282431) v4 -- 0x55a3484fd400 con 0 > -8182> 2018-07-11 07:32:59.867793 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12650 conn(0x55a311b68800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1838 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:32:59.902680) v4 > -8181> 2018-07-11 07:32:59.867793 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12650 conn(0x55a311b67000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1838 0x55a31fa78000 osd_ping(ping e494 stamp 2018-07-11 07:32:59.902680) v4 > -8180> 2018-07-11 07:32:59.867821 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.3 10.8.128.106:0/12650 1838 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:59.902680) v4 ==== 2004+0+0 (275098347 0 0) 0x55a34de78600 con 0x55a311b68800 > -8179> 2018-07-11 07:32:59.867824 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.3 10.8.128.106:0/12650 1838 ==== osd_ping(ping e494 stamp 2018-07-11 07:32:59.902680) v4 ==== 2004+0+0 (275098347 0 0) 0x55a31fa78000 con 0x55a311b67000 > -8178> 2018-07-11 07:32:59.867841 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:59.902680) v4 -- 0x55a3484ffc00 con 0 > -8177> 2018-07-11 07:32:59.867872 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:32:59.902680) v4 -- 0x55a31fa78400 con 0 > -8176> 2018-07-11 07:32:59.963306 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a31f81d200 con 0 > -8175> 2018-07-11 07:32:59.975599 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5048 0x55a317922500 replica scrub(pg: 6.15,from:0'0,to:430'106167,epoch:494/492,start:6:a9a087c8:::benchmark_data_magna108_41525_object15296:0,end:6:a9a18981:::benchmark_data_magna108_35617_object1106095:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -8174> 2018-07-11 07:32:59.975629 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5048 ==== replica scrub(pg: 6.15,from:0'0,to:430'106167,epoch:494/492,start:6:a9a087c8:::benchmark_data_magna108_41525_object15296:0,end:6:a9a18981:::benchmark_data_magna108_35617_object1106095:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (1575894315 0 0) 0x55a317922500 con 0x55a317019000 > -8173> 2018-07-11 07:33:00.151965 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4d00000 on bdev 1, free 0x2fe000; fallback to bdev 2 > -8172> 2018-07-11 07:33:00.220405 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15255 conn(0x55a311b64000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1849 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:33:01.433240) v4 > -8171> 2018-07-11 07:33:00.220430 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.0 10.8.128.113:0/15255 1849 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:01.433240) v4 ==== 2004+0+0 (1715177126 0 0) 0x55a34de6f000 con 0x55a311b64000 > -8170> 2018-07-11 07:33:00.220435 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15255 conn(0x55a311b65800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1849 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:33:01.433240) v4 > -8169> 2018-07-11 07:33:00.220447 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:01.433240) v4 -- 0x55a3484ffc00 con 0 > -8168> 2018-07-11 07:33:00.220462 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.0 10.8.128.113:0/15255 1849 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:01.433240) v4 ==== 2004+0+0 (1715177126 0 0) 0x55a324474c00 con 0x55a311b65800 > -8167> 2018-07-11 07:33:00.220484 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:01.433240) v4 -- 0x55a3484fd400 con 0 > -8166> 2018-07-11 07:33:00.768192 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.107:0/12617 conn(0x55a32447e000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1796 0x55a31fa79000 osd_ping(ping e494 stamp 2018-07-11 07:33:00.767881) v4 > -8165> 2018-07-11 07:33:00.768212 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.9 10.8.128.107:0/12617 1796 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:00.767881) v4 ==== 2004+0+0 (3387534495 0 0) 0x55a31fa79000 con 0x55a32447e000 > -8164> 2018-07-11 07:33:00.768224 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:00.767881) v4 -- 0x55a31fa79600 con 0 > -8163> 2018-07-11 07:33:00.768217 7f355d88e700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.107:0/12617 conn(0x55a32447f800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1796 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:33:00.767881) v4 > -8162> 2018-07-11 07:33:00.768234 7f355d88e700 1 -- 10.8.128.107:6803/12665 <== osd.9 10.8.128.107:0/12617 1796 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:00.767881) v4 ==== 2004+0+0 (3387534495 0 0) 0x55a34de78600 con 0x55a32447f800 > -8161> 2018-07-11 07:33:00.768260 7f355d88e700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:00.767881) v4 -- 0x55a34256c200 con 0 > -8160> 2018-07-11 07:33:00.841021 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b7000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1774 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:33:00.591316) v4 > -8159> 2018-07-11 07:33:00.841042 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.8 10.8.128.108:0/11967 1774 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:00.591316) v4 ==== 2004+0+0 (1071339784 0 0) 0x55a34de6f000 con 0x55a3170b7000 > -8158> 2018-07-11 07:33:00.841052 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:00.591316) v4 -- 0x55a3342d6800 con 0 > -8157> 2018-07-11 07:33:00.841133 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b8800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1774 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:33:00.591316) v4 > -8156> 2018-07-11 07:33:00.841162 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.8 10.8.128.108:0/11967 1774 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:00.591316) v4 ==== 2004+0+0 (1071339784 0 0) 0x55a332e79a00 con 0x55a3170b8800 > -8155> 2018-07-11 07:33:00.841179 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:00.591316) v4 -- 0x55a319405e00 con 0 > -8154> 2018-07-11 07:33:00.940449 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.115:0/13323 conn(0x55a321665800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1835 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:33:01.277416) v4 > -8153> 2018-07-11 07:33:00.940471 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.6 10.8.128.115:0/13323 1835 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:01.277416) v4 ==== 2004+0+0 (3687415588 0 0) 0x55a324474c00 con 0x55a321665800 > -8152> 2018-07-11 07:33:00.940486 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:01.277416) v4 -- 0x55a343718c00 con 0 > -8151> 2018-07-11 07:33:00.940565 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.115:0/13323 conn(0x55a321668800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1835 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:33:01.277416) v4 > -8150> 2018-07-11 07:33:00.940580 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.6 10.8.128.115:0/13323 1835 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:01.277416) v4 ==== 2004+0+0 (3687415588 0 0) 0x55a332e79a00 con 0x55a321668800 > -8149> 2018-07-11 07:33:00.940590 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:01.277416) v4 -- 0x55a3342d5a00 con 0 > -8148> 2018-07-11 07:33:01.314218 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 58] Generated table #36569: 4957165 keys, 80348232 bytes > -8147> 2018-07-11 07:33:01.314275 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294381314240, "cf_name": "default", "job": 58, "event": "table_file_creation", "file_number": 36569, "file_size": 80348232, "table_properties": {"data_size": 67109345, "index_size": 689196, "filter_size": 12548701, "raw_key_size": 143207009, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957165, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957165", "kMergeOperands": "0"}} > -8146> 2018-07-11 07:33:01.639413 7f354e0ae700 1 bluefs _allocate failed to allocate 0x500000 on bdev 1, free 0x2fe000; fallback to bdev 2 > -8145> 2018-07-11 07:33:01.641279 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a31f82e240 con 0 > -8144> 2018-07-11 07:33:01.659192 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5049 0x55a317922780 replica scrub(pg: 6.15,from:0'0,to:430'106200,epoch:494/492,start:6:a9a18981:::benchmark_data_magna108_35617_object1106095:0,end:6:a9a286d5:::benchmark_data_magna108_35617_object199995:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -8143> 2018-07-11 07:33:01.659216 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5049 ==== replica scrub(pg: 6.15,from:0'0,to:430'106200,epoch:494/492,start:6:a9a18981:::benchmark_data_magna108_35617_object1106095:0,end:6:a9a286d5:::benchmark_data_magna108_35617_object199995:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 219+0+0 (3195322393 0 0) 0x55a317922780 con 0x55a317019000 > -8142> 2018-07-11 07:33:01.784157 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 58] Generated table #36570: 313476 keys, 5086251 bytes > -8141> 2018-07-11 07:33:01.784185 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294381784172, "cf_name": "default", "job": 58, "event": "table_file_creation", "file_number": 36570, "file_size": 5086251, "table_properties": {"data_size": 4248564, "index_size": 43146, "filter_size": 793555, "raw_key_size": 9055928, "raw_average_key_size": 28, "raw_value_size": 4680, "raw_average_value_size": 0, "num_data_blocks": 1043, "num_entries": 313476, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "313461", "kMergeOperands": "0"}} > -8140> 2018-07-11 07:33:01.920905 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15255 conn(0x55a311b65800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1850 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:33:03.133677) v4 > -8139> 2018-07-11 07:33:01.920910 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15255 conn(0x55a311b64000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1850 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:33:03.133677) v4 > -8138> 2018-07-11 07:33:01.920929 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.0 10.8.128.113:0/15255 1850 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:03.133677) v4 ==== 2004+0+0 (3730179019 0 0) 0x55a324474c00 con 0x55a311b65800 > -8137> 2018-07-11 07:33:01.920930 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.0 10.8.128.113:0/15255 1850 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:03.133677) v4 ==== 2004+0+0 (3730179019 0 0) 0x55a34de78600 con 0x55a311b64000 > -8136> 2018-07-11 07:33:01.920945 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:03.133677) v4 -- 0x55a3342d5a00 con 0 > -8135> 2018-07-11 07:33:01.920970 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:03.133677) v4 -- 0x55a3342d6800 con 0 > -8134> 2018-07-11 07:33:02.057828 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12651 conn(0x55a3216a0800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1828 0x55a31fa78400 osd_ping(ping e494 stamp 2018-07-11 07:33:02.092793) v4 > -8133> 2018-07-11 07:33:02.057850 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.2 10.8.128.106:0/12651 1828 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:02.092793) v4 ==== 2004+0+0 (3882495964 0 0) 0x55a31fa78400 con 0x55a3216a0800 > -8132> 2018-07-11 07:33:02.057862 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:02.092793) v4 -- 0x55a31fa79600 con 0 > -8131> 2018-07-11 07:33:02.057850 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12651 conn(0x55a32169f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1828 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:33:02.092793) v4 > -8130> 2018-07-11 07:33:02.057881 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.2 10.8.128.106:0/12651 1828 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:02.092793) v4 ==== 2004+0+0 (3882495964 0 0) 0x55a332e79a00 con 0x55a32169f000 > -8129> 2018-07-11 07:33:02.057895 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:02.092793) v4 -- 0x55a343719000 con 0 > -8128> 2018-07-11 07:33:02.250520 7f35488a3700 1 -- 10.8.128.107:6800/12665 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55a32161e840 con 0 > -8127> 2018-07-11 07:33:02.250609 7f35488a3700 1 -- 10.8.128.107:6800/12665 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55a32a5c1b80 con 0 > -8126> 2018-07-11 07:33:02.354318 7f3543899700 5 osd.10 494 heartbeat: osd_stat(783 GB used, 147 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,0,0,1]) > -8125> 2018-07-11 07:33:02.354332 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6802/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:33:02.354330) v4 -- 0x55a325c0f600 con 0 > -8124> 2018-07-11 07:33:02.354368 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6803/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:33:02.354330) v4 -- 0x55a325c0f200 con 0 > -8123> 2018-07-11 07:33:02.354386 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6806/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:33:02.354330) v4 -- 0x55a32d6e6a00 con 0 > -8122> 2018-07-11 07:33:02.354400 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6807/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:33:02.354330) v4 -- 0x55a342825600 con 0 > -8121> 2018-07-11 07:33:02.354424 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6808/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:33:02.354330) v4 -- 0x55a325c0f400 con 0 > -8120> 2018-07-11 07:33:02.354436 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6810/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:33:02.354330) v4 -- 0x55a342825e00 con 0 > -8119> 2018-07-11 07:33:02.354448 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6809/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:33:02.354330) v4 -- 0x55a342825c00 con 0 > -8118> 2018-07-11 07:33:02.354461 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6811/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:33:02.354330) v4 -- 0x55a344ac8400 con 0 > -8117> 2018-07-11 07:33:02.354473 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6810/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:33:02.354330) v4 -- 0x55a325c0e400 con 0 > -8116> 2018-07-11 07:33:02.354483 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6811/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:33:02.354330) v4 -- 0x55a342825200 con 0 > -8115> 2018-07-11 07:33:02.354495 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6806/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:33:02.354330) v4 -- 0x55a342825800 con 0 > -8114> 2018-07-11 07:33:02.354511 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6807/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:33:02.354330) v4 -- 0x55a344ac8600 con 0 > -8113> 2018-07-11 07:33:02.354524 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6802/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:33:02.354330) v4 -- 0x55a344ac8c00 con 0 > -8112> 2018-07-11 07:33:02.354538 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6803/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:33:02.354330) v4 -- 0x55a344ac9000 con 0 > -8111> 2018-07-11 07:33:02.354551 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6806/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:33:02.354330) v4 -- 0x55a344ac8a00 con 0 > -8110> 2018-07-11 07:33:02.354570 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6807/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:33:02.354330) v4 -- 0x55a342825a00 con 0 > -8109> 2018-07-11 07:33:02.354583 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6802/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:33:02.354330) v4 -- 0x55a344ac9200 con 0 > -8108> 2018-07-11 07:33:02.354595 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6803/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:33:02.354330) v4 -- 0x55a344ac8e00 con 0 > -8107> 2018-07-11 07:33:02.354608 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6806/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:33:02.354330) v4 -- 0x55a344ac8000 con 0 > -8106> 2018-07-11 07:33:02.354621 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6807/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:33:02.354330) v4 -- 0x55a344ac9800 con 0 > -8105> 2018-07-11 07:33:02.354640 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6802/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:33:02.354330) v4 -- 0x55a344ac8200 con 0 > -8104> 2018-07-11 07:33:02.354657 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6803/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:33:02.354330) v4 -- 0x55a344ac9600 con 0 > -8103> 2018-07-11 07:33:02.354794 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6806/12617 conn(0x55a314e26000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1823 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:02.354330) v4 > -8102> 2018-07-11 07:33:02.354816 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6806/12617 1823 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:02.354330) v4 ==== 2004+0+0 (3721285938 0 0) 0x55a332e79a00 con 0x55a314e26000 > -8101> 2018-07-11 07:33:02.354834 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6807/12617 conn(0x55a314e27800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1823 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:02.354330) v4 > -8100> 2018-07-11 07:33:02.354855 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6807/12617 1823 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:02.354330) v4 ==== 2004+0+0 (3721285938 0 0) 0x55a34de6f000 con 0x55a314e27800 > -8099> 2018-07-11 07:33:02.354917 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6807/12677 conn(0x55a314e85800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1823 0x55a31fa79000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:02.354330) v4 > -8098> 2018-07-11 07:33:02.354925 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6809/12650 conn(0x55a314e1f000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1823 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:02.354330) v4 > -8097> 2018-07-11 07:33:02.354938 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6809/12650 1823 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:02.354330) v4 ==== 2004+0+0 (3721285938 0 0) 0x55a34de6f000 con 0x55a314e1f000 > -8096> 2018-07-11 07:33:02.354937 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6807/12677 1823 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:02.354330) v4 ==== 2004+0+0 (3721285938 0 0) 0x55a31fa79000 con 0x55a314e85800 > -8095> 2018-07-11 07:33:02.354947 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6810/12651 conn(0x55a314e97800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1823 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:02.354330) v4 > -8094> 2018-07-11 07:33:02.354967 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6810/12651 1823 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:02.354330) v4 ==== 2004+0+0 (3721285938 0 0) 0x55a332e79a00 con 0x55a314e97800 > -8093> 2018-07-11 07:33:02.354992 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6802/15255 conn(0x55a314e1c000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1823 0x55a31fa79000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:02.354330) v4 > -8092> 2018-07-11 07:33:02.354975 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6806/12677 conn(0x55a314e84000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1823 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:02.354330) v4 > -8091> 2018-07-11 07:33:02.355007 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6802/15255 1823 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:02.354330) v4 ==== 2004+0+0 (3721285938 0 0) 0x55a31fa79000 con 0x55a314e1c000 > -8090> 2018-07-11 07:33:02.355011 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6806/12677 1823 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:02.354330) v4 ==== 2004+0+0 (3721285938 0 0) 0x55a34de6f000 con 0x55a314e84000 > -8089> 2018-07-11 07:33:02.355003 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6803/15255 conn(0x55a314e1d800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1823 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:02.354330) v4 > -8088> 2018-07-11 07:33:02.355023 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6803/15255 1823 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:02.354330) v4 ==== 2004+0+0 (3721285938 0 0) 0x55a332e79a00 con 0x55a314e1d800 > -8087> 2018-07-11 07:33:02.355045 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6811/12650 conn(0x55a314e20800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1823 0x55a31fa79000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:02.354330) v4 > -8086> 2018-07-11 07:33:02.355057 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6811/12650 1823 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:02.354330) v4 ==== 2004+0+0 (3721285938 0 0) 0x55a31fa79000 con 0x55a314e20800 > -8085> 2018-07-11 07:33:02.355060 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6806/15263 conn(0x55a314e99000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1823 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:02.354330) v4 > -8084> 2018-07-11 07:33:02.355074 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6806/15263 1823 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:02.354330) v4 ==== 2004+0+0 (3721285938 0 0) 0x55a34de6f000 con 0x55a314e99000 > -8083> 2018-07-11 07:33:02.355075 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6811/15287 conn(0x55a314e04800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1823 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:02.354330) v4 > -8082> 2018-07-11 07:33:02.355086 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6811/15287 1823 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:02.354330) v4 ==== 2004+0+0 (3721285938 0 0) 0x55a332e79a00 con 0x55a314e04800 > -8081> 2018-07-11 07:33:02.355094 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6810/15287 conn(0x55a314e03000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1823 0x55a31fa79000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:02.354330) v4 > -8080> 2018-07-11 07:33:02.355107 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6810/15287 1823 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:02.354330) v4 ==== 2004+0+0 (3721285938 0 0) 0x55a31fa79000 con 0x55a314e03000 > -8079> 2018-07-11 07:33:02.355104 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6803/13323 conn(0x55a314e16800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1823 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:02.354330) v4 > -8078> 2018-07-11 07:33:02.355114 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6802/13323 conn(0x55a314e15000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1823 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:02.354330) v4 > -8077> 2018-07-11 07:33:02.355124 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6802/13323 1823 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:02.354330) v4 ==== 2004+0+0 (3721285938 0 0) 0x55a332e79a00 con 0x55a314e15000 > -8076> 2018-07-11 07:33:02.355136 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6803/13323 1823 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:02.354330) v4 ==== 2004+0+0 (3721285938 0 0) 0x55a34de6f000 con 0x55a314e16800 > -8075> 2018-07-11 07:33:02.355138 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6808/12651 conn(0x55a314e96000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1823 0x55a31fa79000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:02.354330) v4 > -8074> 2018-07-11 07:33:02.355154 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6808/12651 1823 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:02.354330) v4 ==== 2004+0+0 (3721285938 0 0) 0x55a31fa79000 con 0x55a314e96000 > -8073> 2018-07-11 07:33:02.355176 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6802/11967 conn(0x55a314e12000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1823 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:02.354330) v4 > -8072> 2018-07-11 07:33:02.355188 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6802/11967 1823 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:02.354330) v4 ==== 2004+0+0 (3721285938 0 0) 0x55a34de6f000 con 0x55a314e12000 > -8071> 2018-07-11 07:33:02.355187 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6807/15263 conn(0x55a314e9a800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1823 0x55a31fa79000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:02.354330) v4 > -8070> 2018-07-11 07:33:02.355197 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6807/15263 1823 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:02.354330) v4 ==== 2004+0+0 (3721285938 0 0) 0x55a31fa79000 con 0x55a314e9a800 > -8069> 2018-07-11 07:33:02.355195 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6806/15885 conn(0x55a314e00000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1823 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:02.354330) v4 > -8068> 2018-07-11 07:33:02.355212 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6806/15885 1823 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:02.354330) v4 ==== 2004+0+0 (3721285938 0 0) 0x55a332e79a00 con 0x55a314e00000 > -8067> 2018-07-11 07:33:02.355217 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6807/15885 conn(0x55a314e01800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1823 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:02.354330) v4 > -8066> 2018-07-11 07:33:02.355226 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6807/15885 1823 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:02.354330) v4 ==== 2004+0+0 (3721285938 0 0) 0x55a34de6f000 con 0x55a314e01800 > -8065> 2018-07-11 07:33:02.355229 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6803/11967 conn(0x55a314e13800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1823 0x55a31fa79000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:02.354330) v4 > -8064> 2018-07-11 07:33:02.355242 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6803/11967 1823 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:02.354330) v4 ==== 2004+0+0 (3721285938 0 0) 0x55a31fa79000 con 0x55a314e13800 > -8063> 2018-07-11 07:33:02.355266 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6803/15919 conn(0x55a314e88800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1823 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:02.354330) v4 > -8062> 2018-07-11 07:33:02.355269 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6802/15919 conn(0x55a314e87000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1823 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:02.354330) v4 > -8061> 2018-07-11 07:33:02.355278 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6803/15919 1823 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:02.354330) v4 ==== 2004+0+0 (3721285938 0 0) 0x55a34de6f000 con 0x55a314e88800 > -8060> 2018-07-11 07:33:02.355281 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6802/15919 1823 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:02.354330) v4 ==== 2004+0+0 (3721285938 0 0) 0x55a332e79a00 con 0x55a314e87000 > -8059> 2018-07-11 07:33:02.473539 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a31f832240 con 0 > -8058> 2018-07-11 07:33:02.484018 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5050 0x55a317922a00 replica scrub(pg: 6.15,from:0'0,to:430'106493,epoch:494/492,start:6:a9a286d5:::benchmark_data_magna108_35617_object199995:0,end:6:a9a337ab:::benchmark_data_magna108_35617_object424624:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -8057> 2018-07-11 07:33:02.484042 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5050 ==== replica scrub(pg: 6.15,from:0'0,to:430'106493,epoch:494/492,start:6:a9a286d5:::benchmark_data_magna108_35617_object199995:0,end:6:a9a337ab:::benchmark_data_magna108_35617_object424624:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (3225204759 0 0) 0x55a317922a00 con 0x55a317019000 > -8056> 2018-07-11 07:33:02.640794 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.115:0/13323 conn(0x55a321665800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1836 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:33:02.977840) v4 > -8055> 2018-07-11 07:33:02.640811 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.6 10.8.128.115:0/13323 1836 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:02.977840) v4 ==== 2004+0+0 (325953640 0 0) 0x55a332e79a00 con 0x55a321665800 > -8054> 2018-07-11 07:33:02.640820 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:02.977840) v4 -- 0x55a342825200 con 0 > -8053> 2018-07-11 07:33:02.640923 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.115:0/13323 conn(0x55a321668800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1836 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:33:02.977840) v4 > -8052> 2018-07-11 07:33:02.640940 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.6 10.8.128.115:0/13323 1836 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:02.977840) v4 ==== 2004+0+0 (325953640 0 0) 0x55a324474c00 con 0x55a321668800 > -8051> 2018-07-11 07:33:02.640954 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:02.977840) v4 -- 0x55a342825200 con 0 > -8050> 2018-07-11 07:33:02.849694 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15919 conn(0x55a317074800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1829 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:33:03.473140) v4 > -8049> 2018-07-11 07:33:02.849718 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.11 10.8.128.114:0/15919 1829 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:03.473140) v4 ==== 2004+0+0 (4253725858 0 0) 0x55a332e79a00 con 0x55a317074800 > -8048> 2018-07-11 07:33:02.849728 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:03.473140) v4 -- 0x55a342825e00 con 0 > -8047> 2018-07-11 07:33:02.849792 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15919 conn(0x55a3170b4000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1829 0x55a31fa79000 osd_ping(ping e494 stamp 2018-07-11 07:33:03.473140) v4 > -8046> 2018-07-11 07:33:02.849818 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.11 10.8.128.114:0/15919 1829 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:03.473140) v4 ==== 2004+0+0 (4253725858 0 0) 0x55a31fa79000 con 0x55a3170b4000 > -8045> 2018-07-11 07:33:02.849835 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:03.473140) v4 -- 0x55a31fa79600 con 0 > -8044> 2018-07-11 07:33:03.367083 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a31f8338c0 con 0 > -8043> 2018-07-11 07:33:03.394458 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5051 0x55a317922c80 replica scrub(pg: 6.15,from:0'0,to:430'106565,epoch:494/492,start:6:a9a337ab:::benchmark_data_magna108_35617_object424624:0,end:6:a9a45548:::benchmark_data_magna108_35617_object603996:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -8042> 2018-07-11 07:33:03.394482 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5051 ==== replica scrub(pg: 6.15,from:0'0,to:430'106565,epoch:494/492,start:6:a9a337ab:::benchmark_data_magna108_35617_object424624:0,end:6:a9a45548:::benchmark_data_magna108_35617_object603996:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (419994368 0 0) 0x55a317922c80 con 0x55a317019000 > -8041> 2018-07-11 07:33:03.420133 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12677 conn(0x55a32169d800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1844 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:33:03.455122) v4 > -8040> 2018-07-11 07:33:03.420156 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.1 10.8.128.106:0/12677 1844 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:03.455122) v4 ==== 2004+0+0 (2239885389 0 0) 0x55a34de6f000 con 0x55a32169d800 > -8039> 2018-07-11 07:33:03.420167 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:03.455122) v4 -- 0x55a342825c00 con 0 > -8038> 2018-07-11 07:33:03.420224 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12677 conn(0x55a32169c000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1844 0x55a342825600 osd_ping(ping e494 stamp 2018-07-11 07:33:03.455122) v4 > -8037> 2018-07-11 07:33:03.420257 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.1 10.8.128.106:0/12677 1844 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:03.455122) v4 ==== 2004+0+0 (2239885389 0 0) 0x55a342825600 con 0x55a32169c000 > -8036> 2018-07-11 07:33:03.420275 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:03.455122) v4 -- 0x55a325c0e200 con 0 > -8035> 2018-07-11 07:33:03.968281 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12650 conn(0x55a311b67000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1839 0x55a31fa79600 osd_ping(ping e494 stamp 2018-07-11 07:33:04.003262) v4 > -8034> 2018-07-11 07:33:03.968281 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12650 conn(0x55a311b68800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1839 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:33:04.003262) v4 > -8033> 2018-07-11 07:33:03.968316 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.3 10.8.128.106:0/12650 1839 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:04.003262) v4 ==== 2004+0+0 (267692581 0 0) 0x55a31fa79600 con 0x55a311b67000 > -8032> 2018-07-11 07:33:03.968319 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.3 10.8.128.106:0/12650 1839 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:04.003262) v4 ==== 2004+0+0 (267692581 0 0) 0x55a34de78600 con 0x55a311b68800 > -8031> 2018-07-11 07:33:03.968336 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:04.003262) v4 -- 0x55a319405c00 con 0 > -8030> 2018-07-11 07:33:03.968366 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:04.003262) v4 -- 0x55a32d6e6a00 con 0 > -8029> 2018-07-11 07:33:04.221517 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a32eb4ab40 con 0 > -8028> 2018-07-11 07:33:04.244019 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5052 0x55a317922f00 replica scrub(pg: 6.15,from:0'0,to:430'106782,epoch:494/492,start:6:a9a45548:::benchmark_data_magna108_35617_object603996:0,end:6:a9a52e28:::benchmark_data_magna108_35617_object722530:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -8027> 2018-07-11 07:33:04.244043 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5052 ==== replica scrub(pg: 6.15,from:0'0,to:430'106782,epoch:494/492,start:6:a9a45548:::benchmark_data_magna108_35617_object603996:0,end:6:a9a52e28:::benchmark_data_magna108_35617_object722530:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (717165984 0 0) 0x55a317922f00 con 0x55a317019000 > -8026> 2018-07-11 07:33:04.268785 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.107:0/12617 conn(0x55a32447e000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1797 0x55a342825600 osd_ping(ping e494 stamp 2018-07-11 07:33:04.268332) v4 > -8025> 2018-07-11 07:33:04.268810 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.9 10.8.128.107:0/12617 1797 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:04.268332) v4 ==== 2004+0+0 (1094590439 0 0) 0x55a342825600 con 0x55a32447e000 > -8024> 2018-07-11 07:33:04.268803 7f355d88e700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.107:0/12617 conn(0x55a32447f800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1797 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:33:04.268332) v4 > -8023> 2018-07-11 07:33:04.268825 7f355d88e700 1 -- 10.8.128.107:6803/12665 <== osd.9 10.8.128.107:0/12617 1797 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:04.268332) v4 ==== 2004+0+0 (1094590439 0 0) 0x55a34de6f000 con 0x55a32447f800 > -8022> 2018-07-11 07:33:04.268826 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:04.268332) v4 -- 0x55a3342d7000 con 0 > -8021> 2018-07-11 07:33:04.268880 7f355d88e700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:04.268332) v4 -- 0x55a32d6e6a00 con 0 > -8020> 2018-07-11 07:33:04.521273 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15287 conn(0x55a311b75000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1818 0x55a319405c00 osd_ping(ping e494 stamp 2018-07-11 07:33:05.734051) v4 > -8019> 2018-07-11 07:33:04.521270 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15287 conn(0x55a311b72000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1818 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:33:05.734051) v4 > -8018> 2018-07-11 07:33:04.521299 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.4 10.8.128.113:0/15287 1818 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:05.734051) v4 ==== 2004+0+0 (231136941 0 0) 0x55a319405c00 con 0x55a311b75000 > -8017> 2018-07-11 07:33:04.521311 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:05.734051) v4 -- 0x55a3342d7000 con 0 > -8016> 2018-07-11 07:33:04.521305 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.4 10.8.128.113:0/15287 1818 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:05.734051) v4 ==== 2004+0+0 (231136941 0 0) 0x55a324474c00 con 0x55a311b72000 > -8015> 2018-07-11 07:33:04.521335 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:05.734051) v4 -- 0x55a342825e00 con 0 > -8014> 2018-07-11 07:33:04.709997 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15263 conn(0x55a321664000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1821 0x55a342825600 osd_ping(ping e494 stamp 2018-07-11 07:33:05.922892) v4 > -8013> 2018-07-11 07:33:04.710023 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.5 10.8.128.113:0/15263 1821 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:05.922892) v4 ==== 2004+0+0 (1503585855 0 0) 0x55a342825600 con 0x55a321664000 > -8012> 2018-07-11 07:33:04.710032 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:05.922892) v4 -- 0x55a3342d7c00 con 0 > -8011> 2018-07-11 07:33:04.710092 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15263 conn(0x55a321667000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1821 0x55a319405c00 osd_ping(ping e494 stamp 2018-07-11 07:33:05.922892) v4 > -8010> 2018-07-11 07:33:04.710107 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.5 10.8.128.113:0/15263 1821 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:05.922892) v4 ==== 2004+0+0 (1503585855 0 0) 0x55a319405c00 con 0x55a321667000 > -8009> 2018-07-11 07:33:04.710120 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:05.922892) v4 -- 0x55a3342d6200 con 0 > -8008> 2018-07-11 07:33:04.821290 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15255 conn(0x55a311b65800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1851 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:33:06.034090) v4 > -8007> 2018-07-11 07:33:04.821315 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.0 10.8.128.113:0/15255 1851 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:06.034090) v4 ==== 2004+0+0 (2814579685 0 0) 0x55a332e79a00 con 0x55a311b65800 > -8006> 2018-07-11 07:33:04.821325 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:06.034090) v4 -- 0x55a325c0f200 con 0 > -8005> 2018-07-11 07:33:04.821321 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15255 conn(0x55a311b64000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1851 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:33:06.034090) v4 > -8004> 2018-07-11 07:33:04.821339 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.0 10.8.128.113:0/15255 1851 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:06.034090) v4 ==== 2004+0+0 (2814579685 0 0) 0x55a34de78600 con 0x55a311b64000 > -8003> 2018-07-11 07:33:04.821348 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:06.034090) v4 -- 0x55a3342d6800 con 0 > -8002> 2018-07-11 07:33:04.883371 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a3221b6240 con 0 > -8001> 2018-07-11 07:33:04.901282 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5053 0x55a317923180 replica scrub(pg: 6.15,from:0'0,to:430'106016,epoch:494/492,start:6:a9a52e28:::benchmark_data_magna108_35617_object722530:0,end:6:a9a68553:::benchmark_data_magna108.ceph.redhat.com_2478_object76467:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -8000> 2018-07-11 07:33:04.901312 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5053 ==== replica scrub(pg: 6.15,from:0'0,to:430'106016,epoch:494/492,start:6:a9a52e28:::benchmark_data_magna108_35617_object722530:0,end:6:a9a68553:::benchmark_data_magna108.ceph.redhat.com_2478_object76467:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 232+0+0 (2959325140 0 0) 0x55a317923180 con 0x55a317019000 > -7999> 2018-07-11 07:33:04.959452 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15885 conn(0x55a311b76800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1845 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:33:05.582923) v4 > -7998> 2018-07-11 07:33:04.959457 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15885 conn(0x55a3170b5800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1845 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:33:05.582923) v4 > -7997> 2018-07-11 07:33:04.959474 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.7 10.8.128.114:0/15885 1845 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:05.582923) v4 ==== 2004+0+0 (3993682868 0 0) 0x55a34de6f000 con 0x55a311b76800 > -7996> 2018-07-11 07:33:04.959476 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.7 10.8.128.114:0/15885 1845 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:05.582923) v4 ==== 2004+0+0 (3993682868 0 0) 0x55a324474c00 con 0x55a3170b5800 > -7995> 2018-07-11 07:33:04.959483 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:05.582923) v4 -- 0x55a3342d6800 con 0 > -7994> 2018-07-11 07:33:04.959511 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:05.582923) v4 -- 0x55a325c0f200 con 0 > -7993> 2018-07-11 07:33:05.210553 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15263 conn(0x55a321664000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1822 0x55a342825600 osd_ping(ping e494 stamp 2018-07-11 07:33:06.423302) v4 > -7992> 2018-07-11 07:33:05.210580 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.5 10.8.128.113:0/15263 1822 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:06.423302) v4 ==== 2004+0+0 (2981060585 0 0) 0x55a342825600 con 0x55a321664000 > -7991> 2018-07-11 07:33:05.210600 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:06.423302) v4 -- 0x55a3342d6c00 con 0 > -7990> 2018-07-11 07:33:05.210678 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15263 conn(0x55a321667000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1822 0x55a3342d6200 osd_ping(ping e494 stamp 2018-07-11 07:33:06.423302) v4 > -7989> 2018-07-11 07:33:05.210698 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.5 10.8.128.113:0/15263 1822 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:06.423302) v4 ==== 2004+0+0 (2981060585 0 0) 0x55a3342d6200 con 0x55a321667000 > -7988> 2018-07-11 07:33:05.210709 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:06.423302) v4 -- 0x55a3342d6c00 con 0 > -7987> 2018-07-11 07:33:05.541313 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.115:0/13323 conn(0x55a321665800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1837 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:33:05.878323) v4 > -7986> 2018-07-11 07:33:05.541335 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.6 10.8.128.115:0/13323 1837 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:05.878323) v4 ==== 2004+0+0 (2922126917 0 0) 0x55a332e79a00 con 0x55a321665800 > -7985> 2018-07-11 07:33:05.541354 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:05.878323) v4 -- 0x55a325c0f200 con 0 > -7984> 2018-07-11 07:33:05.541422 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.115:0/13323 conn(0x55a321668800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1837 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:33:05.878323) v4 > -7983> 2018-07-11 07:33:05.541436 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.6 10.8.128.115:0/13323 1837 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:05.878323) v4 ==== 2004+0+0 (2922126917 0 0) 0x55a324474c00 con 0x55a321668800 > -7982> 2018-07-11 07:33:05.541444 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:05.878323) v4 -- 0x55a343719000 con 0 > -7981> 2018-07-11 07:33:05.630620 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a32eb558c0 con 0 > -7980> 2018-07-11 07:33:05.652768 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5054 0x55a317923400 replica scrub(pg: 6.15,from:0'0,to:430'105367,epoch:494/492,start:6:a9a68553:::benchmark_data_magna108.ceph.redhat.com_2478_object76467:0,end:6:a9a7bbaf:::benchmark_data_magna108_35617_object952394:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -7979> 2018-07-11 07:33:05.652791 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5054 ==== replica scrub(pg: 6.15,from:0'0,to:430'105367,epoch:494/492,start:6:a9a68553:::benchmark_data_magna108.ceph.redhat.com_2478_object76467:0,end:6:a9a7bbaf:::benchmark_data_magna108_35617_object952394:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 232+0+0 (1402745117 0 0) 0x55a317923400 con 0x55a317019000 > -7978> 2018-07-11 07:33:05.668679 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12650 conn(0x55a311b67000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1840 0x55a342825600 osd_ping(ping e494 stamp 2018-07-11 07:33:05.703731) v4 > -7977> 2018-07-11 07:33:05.668673 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12650 conn(0x55a311b68800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1840 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:33:05.703731) v4 > -7976> 2018-07-11 07:33:05.668702 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.3 10.8.128.106:0/12650 1840 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:05.703731) v4 ==== 2004+0+0 (1199673318 0 0) 0x55a342825600 con 0x55a311b67000 > -7975> 2018-07-11 07:33:05.668705 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.3 10.8.128.106:0/12650 1840 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:05.703731) v4 ==== 2004+0+0 (1199673318 0 0) 0x55a34de78600 con 0x55a311b68800 > -7974> 2018-07-11 07:33:05.668716 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:05.703731) v4 -- 0x55a333f6e000 con 0 > -7973> 2018-07-11 07:33:05.668745 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:05.703731) v4 -- 0x55a3342d6800 con 0 > -7972> 2018-07-11 07:33:05.720642 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12677 conn(0x55a32169c000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1845 0x55a3342d6c00 osd_ping(ping e494 stamp 2018-07-11 07:33:05.755643) v4 > -7971> 2018-07-11 07:33:05.720634 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12677 conn(0x55a32169d800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1845 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:33:05.755643) v4 > -7970> 2018-07-11 07:33:05.720671 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.1 10.8.128.106:0/12677 1845 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:05.755643) v4 ==== 2004+0+0 (923735805 0 0) 0x55a3342d6c00 con 0x55a32169c000 > -7969> 2018-07-11 07:33:05.720673 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.1 10.8.128.106:0/12677 1845 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:05.755643) v4 ==== 2004+0+0 (923735805 0 0) 0x55a34de6f000 con 0x55a32169d800 > -7968> 2018-07-11 07:33:05.720692 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:05.755643) v4 -- 0x55a333f6e000 con 0 > -7967> 2018-07-11 07:33:05.720722 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:05.755643) v4 -- 0x55a3342d6800 con 0 > -7966> 2018-07-11 07:33:05.841951 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4800000 on bdev 1, free 0x2fe000; fallback to bdev 2 > -7965> 2018-07-11 07:33:05.854831 7f3543899700 5 osd.10 494 heartbeat: osd_stat(783 GB used, 147 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,1]) > -7964> 2018-07-11 07:33:05.854845 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6802/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:33:05.854843) v4 -- 0x55a344ac9c00 con 0 > -7963> 2018-07-11 07:33:05.854884 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6803/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:33:05.854843) v4 -- 0x55a344ac9a00 con 0 > -7962> 2018-07-11 07:33:05.854902 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6806/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:33:05.854843) v4 -- 0x55a344ac8800 con 0 > -7961> 2018-07-11 07:33:05.854916 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6807/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:33:05.854843) v4 -- 0x55a344ac9400 con 0 > -7960> 2018-07-11 07:33:05.854941 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6808/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:33:05.854843) v4 -- 0x55a343481200 con 0 > -7959> 2018-07-11 07:33:05.854952 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6810/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:33:05.854843) v4 -- 0x55a343481600 con 0 > -7958> 2018-07-11 07:33:05.854963 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6809/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:33:05.854843) v4 -- 0x55a343480600 con 0 > -7957> 2018-07-11 07:33:05.854973 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6811/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:33:05.854843) v4 -- 0x55a343480c00 con 0 > -7956> 2018-07-11 07:33:05.854982 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6810/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:33:05.854843) v4 -- 0x55a343481c00 con 0 > -7955> 2018-07-11 07:33:05.854992 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6811/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:33:05.854843) v4 -- 0x55a343482000 con 0 > -7954> 2018-07-11 07:33:05.855004 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6806/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:33:05.854843) v4 -- 0x55a343481a00 con 0 > -7953> 2018-07-11 07:33:05.855016 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6807/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:33:05.854843) v4 -- 0x55a343481e00 con 0 > -7952> 2018-07-11 07:33:05.855029 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6802/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:33:05.854843) v4 -- 0x55a343481800 con 0 > -7951> 2018-07-11 07:33:05.855042 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6803/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:33:05.854843) v4 -- 0x55a324966a00 con 0 > -7950> 2018-07-11 07:33:05.855054 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6806/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:33:05.854843) v4 -- 0x55a329322e00 con 0 > -7949> 2018-07-11 07:33:05.855071 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6807/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:33:05.854843) v4 -- 0x55a31e8f4200 con 0 > -7948> 2018-07-11 07:33:05.855083 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6802/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:33:05.854843) v4 -- 0x55a324966800 con 0 > -7947> 2018-07-11 07:33:05.855093 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6803/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:33:05.854843) v4 -- 0x55a329323200 con 0 > -7946> 2018-07-11 07:33:05.855105 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6806/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:33:05.854843) v4 -- 0x55a329323600 con 0 > -7945> 2018-07-11 07:33:05.855118 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6807/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:33:05.854843) v4 -- 0x55a329323e00 con 0 > -7944> 2018-07-11 07:33:05.855129 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6802/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:33:05.854843) v4 -- 0x55a329322a00 con 0 > -7943> 2018-07-11 07:33:05.855141 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6803/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:33:05.854843) v4 -- 0x55a329323000 con 0 > -7942> 2018-07-11 07:33:05.855306 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6806/12617 conn(0x55a314e26000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1824 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:05.854843) v4 > -7941> 2018-07-11 07:33:05.855328 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6806/12617 1824 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:05.854843) v4 ==== 2004+0+0 (2653454182 0 0) 0x55a332e79a00 con 0x55a314e26000 > -7940> 2018-07-11 07:33:05.855381 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6806/12677 conn(0x55a314e84000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1824 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:05.854843) v4 > -7939> 2018-07-11 07:33:05.855401 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6806/12677 1824 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:05.854843) v4 ==== 2004+0+0 (2653454182 0 0) 0x55a34de6f000 con 0x55a314e84000 > -7938> 2018-07-11 07:33:05.855426 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6807/12617 conn(0x55a314e27800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1824 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:05.854843) v4 > -7937> 2018-07-11 07:33:05.855420 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6807/12677 conn(0x55a314e85800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1824 0x55a3342d6c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:05.854843) v4 > -7936> 2018-07-11 07:33:05.855453 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6807/12617 1824 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:05.854843) v4 ==== 2004+0+0 (2653454182 0 0) 0x55a34de6f000 con 0x55a314e27800 > -7935> 2018-07-11 07:33:05.855449 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6803/15255 conn(0x55a314e1d800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1824 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:05.854843) v4 > -7934> 2018-07-11 07:33:05.855464 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6803/15255 1824 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:05.854843) v4 ==== 2004+0+0 (2653454182 0 0) 0x55a332e79a00 con 0x55a314e1d800 > -7933> 2018-07-11 07:33:05.855459 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6807/12677 1824 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:05.854843) v4 ==== 2004+0+0 (2653454182 0 0) 0x55a3342d6c00 con 0x55a314e85800 > -7932> 2018-07-11 07:33:05.855488 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6809/12650 conn(0x55a314e1f000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1824 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:05.854843) v4 > -7931> 2018-07-11 07:33:05.855501 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6809/12650 1824 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:05.854843) v4 ==== 2004+0+0 (2653454182 0 0) 0x55a34de6f000 con 0x55a314e1f000 > -7930> 2018-07-11 07:33:05.855527 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6808/12651 conn(0x55a314e96000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1824 0x55a3342d6c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:05.854843) v4 > -7929> 2018-07-11 07:33:05.855533 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6811/15287 conn(0x55a314e04800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1824 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:05.854843) v4 > -7928> 2018-07-11 07:33:05.855542 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6808/12651 1824 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:05.854843) v4 ==== 2004+0+0 (2653454182 0 0) 0x55a3342d6c00 con 0x55a314e96000 > -7927> 2018-07-11 07:33:05.855549 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6811/15287 1824 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:05.854843) v4 ==== 2004+0+0 (2653454182 0 0) 0x55a332e79a00 con 0x55a314e04800 > -7926> 2018-07-11 07:33:05.855552 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6806/15263 conn(0x55a314e99000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1824 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:05.854843) v4 > -7925> 2018-07-11 07:33:05.855562 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6802/15255 conn(0x55a314e1c000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1824 0x55a3342d6c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:05.854843) v4 > -7924> 2018-07-11 07:33:05.855565 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6806/15263 1824 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:05.854843) v4 ==== 2004+0+0 (2653454182 0 0) 0x55a34de6f000 con 0x55a314e99000 > -7923> 2018-07-11 07:33:05.855572 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6810/12651 conn(0x55a314e97800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1824 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:05.854843) v4 > -7922> 2018-07-11 07:33:05.855572 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6802/15255 1824 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:05.854843) v4 ==== 2004+0+0 (2653454182 0 0) 0x55a3342d6c00 con 0x55a314e1c000 > -7921> 2018-07-11 07:33:05.855605 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6810/15287 conn(0x55a314e03000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1824 0x55a3342d6c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:05.854843) v4 > -7920> 2018-07-11 07:33:05.855583 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6810/12651 1824 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:05.854843) v4 ==== 2004+0+0 (2653454182 0 0) 0x55a332e79a00 con 0x55a314e97800 > -7919> 2018-07-11 07:33:05.855611 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6803/13323 conn(0x55a314e16800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1824 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:05.854843) v4 > -7918> 2018-07-11 07:33:05.855617 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6810/15287 1824 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:05.854843) v4 ==== 2004+0+0 (2653454182 0 0) 0x55a3342d6c00 con 0x55a314e03000 > -7917> 2018-07-11 07:33:05.855622 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6803/13323 1824 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:05.854843) v4 ==== 2004+0+0 (2653454182 0 0) 0x55a34de6f000 con 0x55a314e16800 > -7916> 2018-07-11 07:33:05.855642 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6802/13323 conn(0x55a314e15000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1824 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:05.854843) v4 > -7915> 2018-07-11 07:33:05.855649 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6811/12650 conn(0x55a314e20800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1824 0x55a3342d6c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:05.854843) v4 > -7914> 2018-07-11 07:33:05.855655 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6802/13323 1824 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:05.854843) v4 ==== 2004+0+0 (2653454182 0 0) 0x55a332e79a00 con 0x55a314e15000 > -7913> 2018-07-11 07:33:05.855660 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6811/12650 1824 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:05.854843) v4 ==== 2004+0+0 (2653454182 0 0) 0x55a3342d6c00 con 0x55a314e20800 > -7912> 2018-07-11 07:33:05.855652 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6803/15919 conn(0x55a314e88800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1824 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:05.854843) v4 > -7911> 2018-07-11 07:33:05.855690 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6803/15919 1824 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:05.854843) v4 ==== 2004+0+0 (2653454182 0 0) 0x55a34de6f000 con 0x55a314e88800 > -7910> 2018-07-11 07:33:05.855708 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6806/15885 conn(0x55a314e00000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1824 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:05.854843) v4 > -7909> 2018-07-11 07:33:05.855710 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6803/11967 conn(0x55a314e13800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1824 0x55a3342d6c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:05.854843) v4 > -7908> 2018-07-11 07:33:05.855722 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6806/15885 1824 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:05.854843) v4 ==== 2004+0+0 (2653454182 0 0) 0x55a332e79a00 con 0x55a314e00000 > -7907> 2018-07-11 07:33:05.855722 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6803/11967 1824 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:05.854843) v4 ==== 2004+0+0 (2653454182 0 0) 0x55a3342d6c00 con 0x55a314e13800 > -7906> 2018-07-11 07:33:05.855727 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6802/11967 conn(0x55a314e12000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1824 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:05.854843) v4 > -7905> 2018-07-11 07:33:05.855740 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6802/11967 1824 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:05.854843) v4 ==== 2004+0+0 (2653454182 0 0) 0x55a34de6f000 con 0x55a314e12000 > -7904> 2018-07-11 07:33:05.855760 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6802/15919 conn(0x55a314e87000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1824 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:05.854843) v4 > -7903> 2018-07-11 07:33:05.855768 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6807/15885 conn(0x55a314e01800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1824 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:05.854843) v4 > -7902> 2018-07-11 07:33:05.855775 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6802/15919 1824 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:05.854843) v4 ==== 2004+0+0 (2653454182 0 0) 0x55a332e79a00 con 0x55a314e87000 > -7901> 2018-07-11 07:33:05.855761 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6807/15263 conn(0x55a314e9a800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1824 0x55a3342d6c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:05.854843) v4 > -7900> 2018-07-11 07:33:05.855781 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6807/15885 1824 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:05.854843) v4 ==== 2004+0+0 (2653454182 0 0) 0x55a34de6f000 con 0x55a314e01800 > -7899> 2018-07-11 07:33:05.855786 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6807/15263 1824 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:05.854843) v4 ==== 2004+0+0 (2653454182 0 0) 0x55a3342d6c00 con 0x55a314e9a800 > -7898> 2018-07-11 07:33:06.141637 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b8800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1775 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:33:05.891968) v4 > -7897> 2018-07-11 07:33:06.141646 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b7000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1775 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:33:05.891968) v4 > -7896> 2018-07-11 07:33:06.141665 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.8 10.8.128.108:0/11967 1775 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:05.891968) v4 ==== 2004+0+0 (2636861812 0 0) 0x55a332e79a00 con 0x55a3170b8800 > -7895> 2018-07-11 07:33:06.141669 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.8 10.8.128.108:0/11967 1775 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:05.891968) v4 ==== 2004+0+0 (2636861812 0 0) 0x55a34de6f000 con 0x55a3170b7000 > -7894> 2018-07-11 07:33:06.141683 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:05.891968) v4 -- 0x55a343481600 con 0 > -7893> 2018-07-11 07:33:06.141712 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:05.891968) v4 -- 0x55a344ac8800 con 0 > -7892> 2018-07-11 07:33:06.158256 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12651 conn(0x55a3216a0800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1829 0x55a3342d6c00 osd_ping(ping e494 stamp 2018-07-11 07:33:06.193296) v4 > -7891> 2018-07-11 07:33:06.158285 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.2 10.8.128.106:0/12651 1829 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:06.193296) v4 ==== 2004+0+0 (2096507438 0 0) 0x55a3342d6c00 con 0x55a3216a0800 > -7890> 2018-07-11 07:33:06.158277 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12651 conn(0x55a32169f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1829 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:33:06.193296) v4 > -7889> 2018-07-11 07:33:06.158303 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:06.193296) v4 -- 0x55a344ac9c00 con 0 > -7888> 2018-07-11 07:33:06.158308 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.2 10.8.128.106:0/12651 1829 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:06.193296) v4 ==== 2004+0+0 (2096507438 0 0) 0x55a324474c00 con 0x55a32169f000 > -7887> 2018-07-11 07:33:06.158325 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:06.193296) v4 -- 0x55a344ac9a00 con 0 > -7886> 2018-07-11 07:33:06.743793 7f35498a5700 10 monclient: tick > -7885> 2018-07-11 07:33:06.743805 7f35498a5700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2018-07-11 07:32:36.743804) > -7884> 2018-07-11 07:33:06.895899 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 58] Generated table #36571: 2638241 keys, 74546130 bytes > -7883> 2018-07-11 07:33:06.895932 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294386895917, "cf_name": "default", "job": 58, "event": "table_file_creation", "file_number": 36571, "file_size": 74546130, "table_properties": {"data_size": 67112145, "index_size": 689812, "filter_size": 6743183, "raw_key_size": 76215842, "raw_average_key_size": 28, "raw_value_size": 31365745, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} > -7882> 2018-07-11 07:33:07.242338 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b7000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1776 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:33:06.992657) v4 > -7881> 2018-07-11 07:33:07.242364 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.8 10.8.128.108:0/11967 1776 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:06.992657) v4 ==== 2004+0+0 (3149264286 0 0) 0x55a34de78600 con 0x55a3170b7000 > -7880> 2018-07-11 07:33:07.242364 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b8800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1776 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:33:06.992657) v4 > -7879> 2018-07-11 07:33:07.242382 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.8 10.8.128.108:0/11967 1776 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:06.992657) v4 ==== 2004+0+0 (3149264286 0 0) 0x55a332e79a00 con 0x55a3170b8800 > -7878> 2018-07-11 07:33:07.242382 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:06.992657) v4 -- 0x55a3342d6800 con 0 > -7877> 2018-07-11 07:33:07.242421 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:06.992657) v4 -- 0x55a343719000 con 0 > -7876> 2018-07-11 07:33:07.250803 7f35488a3700 1 -- 10.8.128.107:6800/12665 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55a32a5c5b80 con 0 > -7875> 2018-07-11 07:33:07.250875 7f35488a3700 1 -- 10.8.128.107:6800/12665 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55a32a5c58c0 con 0 > -7874> 2018-07-11 07:33:07.350355 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a3221b7440 con 0 > -7873> 2018-07-11 07:33:07.367890 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5055 0x55a317923680 replica scrub(pg: 6.15,from:0'0,to:430'106728,epoch:494/492,start:6:a9a7bbaf:::benchmark_data_magna108_35617_object952394:0,end:6:a9a8f245:::benchmark_data_magna108.ceph.redhat.com_2478_object208123:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -7872> 2018-07-11 07:33:07.367914 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5055 ==== replica scrub(pg: 6.15,from:0'0,to:430'106728,epoch:494/492,start:6:a9a7bbaf:::benchmark_data_magna108_35617_object952394:0,end:6:a9a8f245:::benchmark_data_magna108.ceph.redhat.com_2478_object208123:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (3690859319 0 0) 0x55a317923680 con 0x55a317019000 > -7871> 2018-07-11 07:33:07.421112 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12677 conn(0x55a32169d800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1846 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:33:07.456153) v4 > -7870> 2018-07-11 07:33:07.421123 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12677 conn(0x55a32169c000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1846 0x55a3342d6c00 osd_ping(ping e494 stamp 2018-07-11 07:33:07.456153) v4 > -7869> 2018-07-11 07:33:07.421135 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.1 10.8.128.106:0/12677 1846 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:07.456153) v4 ==== 2004+0+0 (4010889638 0 0) 0x55a34de6f000 con 0x55a32169d800 > -7868> 2018-07-11 07:33:07.421143 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.1 10.8.128.106:0/12677 1846 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:07.456153) v4 ==== 2004+0+0 (4010889638 0 0) 0x55a3342d6c00 con 0x55a32169c000 > -7867> 2018-07-11 07:33:07.421150 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:07.456153) v4 -- 0x55a3342d6800 con 0 > -7866> 2018-07-11 07:33:07.421175 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:07.456153) v4 -- 0x55a333f6e000 con 0 > -7865> 2018-07-11 07:33:07.510979 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15263 conn(0x55a321664000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1823 0x55a344ac9c00 osd_ping(ping e494 stamp 2018-07-11 07:33:08.723867) v4 > -7864> 2018-07-11 07:33:07.511006 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.5 10.8.128.113:0/15263 1823 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:08.723867) v4 ==== 2004+0+0 (2071416569 0 0) 0x55a344ac9c00 con 0x55a321664000 > -7863> 2018-07-11 07:33:07.511023 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:08.723867) v4 -- 0x55a333f6e000 con 0 > -7862> 2018-07-11 07:33:07.511076 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15263 conn(0x55a321667000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1823 0x55a344ac9c00 osd_ping(ping e494 stamp 2018-07-11 07:33:08.723867) v4 > -7861> 2018-07-11 07:33:07.511098 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.5 10.8.128.113:0/15263 1823 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:08.723867) v4 ==== 2004+0+0 (2071416569 0 0) 0x55a344ac9c00 con 0x55a321667000 > -7860> 2018-07-11 07:33:07.511120 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:08.723867) v4 -- 0x55a3342d7400 con 0 > -7859> 2018-07-11 07:33:07.550128 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15919 conn(0x55a3170b4000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1830 0x55a3342d6c00 osd_ping(ping e494 stamp 2018-07-11 07:33:08.173576) v4 > -7858> 2018-07-11 07:33:07.550149 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.11 10.8.128.114:0/15919 1830 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:08.173576) v4 ==== 2004+0+0 (3695118516 0 0) 0x55a3342d6c00 con 0x55a3170b4000 > -7857> 2018-07-11 07:33:07.550159 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:08.173576) v4 -- 0x55a3342d7400 con 0 > -7856> 2018-07-11 07:33:07.550154 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15919 conn(0x55a317074800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1830 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:33:08.173576) v4 > -7855> 2018-07-11 07:33:07.550178 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.11 10.8.128.114:0/15919 1830 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:08.173576) v4 ==== 2004+0+0 (3695118516 0 0) 0x55a324474c00 con 0x55a317074800 > -7854> 2018-07-11 07:33:07.550191 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:08.173576) v4 -- 0x55a343719200 con 0 > -7853> 2018-07-11 07:33:07.555332 7f3543899700 5 osd.10 494 heartbeat: osd_stat(783 GB used, 147 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,1]) > -7852> 2018-07-11 07:33:07.555346 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6802/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:33:07.555345) v4 -- 0x55a329323800 con 0 > -7851> 2018-07-11 07:33:07.555382 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6803/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:33:07.555345) v4 -- 0x55a31e8f4400 con 0 > -7850> 2018-07-11 07:33:07.555396 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6806/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:33:07.555345) v4 -- 0x55a31e8f4600 con 0 > -7849> 2018-07-11 07:33:07.555409 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6807/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:33:07.555345) v4 -- 0x55a31e8f4800 con 0 > -7848> 2018-07-11 07:33:07.555432 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6808/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:33:07.555345) v4 -- 0x55a329323a00 con 0 > -7847> 2018-07-11 07:33:07.555443 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6810/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:33:07.555345) v4 -- 0x55a31e8f4a00 con 0 > -7846> 2018-07-11 07:33:07.555453 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6809/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:33:07.555345) v4 -- 0x55a31e8f4e00 con 0 > -7845> 2018-07-11 07:33:07.555465 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6811/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:33:07.555345) v4 -- 0x55a31e8f4000 con 0 > -7844> 2018-07-11 07:33:07.555477 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6810/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:33:07.555345) v4 -- 0x55a31e8f5000 con 0 > -7843> 2018-07-11 07:33:07.555488 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6811/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:33:07.555345) v4 -- 0x55a31e8f4c00 con 0 > -7842> 2018-07-11 07:33:07.555499 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6806/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:33:07.555345) v4 -- 0x55a31e8f5400 con 0 > -7841> 2018-07-11 07:33:07.555511 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6807/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:33:07.555345) v4 -- 0x55a31e8f5200 con 0 > -7840> 2018-07-11 07:33:07.555522 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6802/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:33:07.555345) v4 -- 0x55a324966000 con 0 > -7839> 2018-07-11 07:33:07.555534 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6803/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:33:07.555345) v4 -- 0x55a327983e00 con 0 > -7838> 2018-07-11 07:33:07.555548 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6806/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:33:07.555345) v4 -- 0x55a324966400 con 0 > -7837> 2018-07-11 07:33:07.555559 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6807/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:33:07.555345) v4 -- 0x55a327983800 con 0 > -7836> 2018-07-11 07:33:07.555572 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6802/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:33:07.555345) v4 -- 0x55a324966200 con 0 > -7835> 2018-07-11 07:33:07.555584 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6803/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:33:07.555345) v4 -- 0x55a327983c00 con 0 > -7834> 2018-07-11 07:33:07.555597 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6806/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:33:07.555345) v4 -- 0x55a325c0f800 con 0 > -7833> 2018-07-11 07:33:07.555610 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6807/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:33:07.555345) v4 -- 0x55a325c0fe00 con 0 > -7832> 2018-07-11 07:33:07.555635 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6802/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:33:07.555345) v4 -- 0x55a3456b0e00 con 0 > -7831> 2018-07-11 07:33:07.555657 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6803/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:33:07.555345) v4 -- 0x55a327983a00 con 0 > -7830> 2018-07-11 07:33:07.555773 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6806/12617 conn(0x55a314e26000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1825 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:07.555345) v4 > -7829> 2018-07-11 07:33:07.555804 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6802/15255 conn(0x55a314e1c000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1825 0x55a3342d6c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:07.555345) v4 > -7828> 2018-07-11 07:33:07.555818 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6806/12617 1825 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:07.555345) v4 ==== 2004+0+0 (3092140071 0 0) 0x55a332e79a00 con 0x55a314e26000 > -7827> 2018-07-11 07:33:07.555821 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6802/15255 1825 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:07.555345) v4 ==== 2004+0+0 (3092140071 0 0) 0x55a3342d6c00 con 0x55a314e1c000 > -7826> 2018-07-11 07:33:07.555819 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6807/12617 conn(0x55a314e27800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1825 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:07.555345) v4 > -7825> 2018-07-11 07:33:07.555842 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6807/12617 1825 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:07.555345) v4 ==== 2004+0+0 (3092140071 0 0) 0x55a34de78600 con 0x55a314e27800 > -7824> 2018-07-11 07:33:07.555862 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6803/15255 conn(0x55a314e1d800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1825 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:07.555345) v4 > -7823> 2018-07-11 07:33:07.555875 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6803/15255 1825 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:07.555345) v4 ==== 2004+0+0 (3092140071 0 0) 0x55a332e79a00 con 0x55a314e1d800 > -7822> 2018-07-11 07:33:07.555915 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6807/12677 conn(0x55a314e85800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1825 0x55a3342d6c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:07.555345) v4 > -7821> 2018-07-11 07:33:07.555936 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6807/12677 1825 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:07.555345) v4 ==== 2004+0+0 (3092140071 0 0) 0x55a3342d6c00 con 0x55a314e85800 > -7820> 2018-07-11 07:33:07.555955 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6806/12677 conn(0x55a314e84000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1825 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:07.555345) v4 > -7819> 2018-07-11 07:33:07.555970 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6806/12677 1825 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:07.555345) v4 ==== 2004+0+0 (3092140071 0 0) 0x55a34de78600 con 0x55a314e84000 > -7818> 2018-07-11 07:33:07.555971 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6810/12651 conn(0x55a314e97800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1825 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:07.555345) v4 > -7817> 2018-07-11 07:33:07.555980 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6808/12651 conn(0x55a314e96000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1825 0x55a3342d6c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:07.555345) v4 > -7816> 2018-07-11 07:33:07.556004 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6808/12651 1825 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:07.555345) v4 ==== 2004+0+0 (3092140071 0 0) 0x55a3342d6c00 con 0x55a314e96000 > -7815> 2018-07-11 07:33:07.556006 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6806/15263 conn(0x55a314e99000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1825 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:07.555345) v4 > -7814> 2018-07-11 07:33:07.555999 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6810/12651 1825 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:07.555345) v4 ==== 2004+0+0 (3092140071 0 0) 0x55a332e79a00 con 0x55a314e97800 > -7813> 2018-07-11 07:33:07.556016 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6806/15263 1825 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:07.555345) v4 ==== 2004+0+0 (3092140071 0 0) 0x55a34de78600 con 0x55a314e99000 > -7812> 2018-07-11 07:33:07.556054 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6810/15287 conn(0x55a314e03000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1825 0x55a3342d6c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:07.555345) v4 > -7811> 2018-07-11 07:33:07.556057 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6809/12650 conn(0x55a314e1f000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1825 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:07.555345) v4 > -7810> 2018-07-11 07:33:07.556077 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6809/12650 1825 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:07.555345) v4 ==== 2004+0+0 (3092140071 0 0) 0x55a34de78600 con 0x55a314e1f000 > -7809> 2018-07-11 07:33:07.556074 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6810/15287 1825 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:07.555345) v4 ==== 2004+0+0 (3092140071 0 0) 0x55a3342d6c00 con 0x55a314e03000 > -7808> 2018-07-11 07:33:07.556076 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6811/15287 conn(0x55a314e04800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1825 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:07.555345) v4 > -7807> 2018-07-11 07:33:07.556096 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6811/15287 1825 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:07.555345) v4 ==== 2004+0+0 (3092140071 0 0) 0x55a332e79a00 con 0x55a314e04800 > -7806> 2018-07-11 07:33:07.556115 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6803/13323 conn(0x55a314e16800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1825 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:07.555345) v4 > -7805> 2018-07-11 07:33:07.556121 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6807/15263 conn(0x55a314e9a800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1825 0x55a3342d6c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:07.555345) v4 > -7804> 2018-07-11 07:33:07.556136 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6803/13323 1825 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:07.555345) v4 ==== 2004+0+0 (3092140071 0 0) 0x55a34de78600 con 0x55a314e16800 > -7803> 2018-07-11 07:33:07.556138 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6807/15263 1825 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:07.555345) v4 ==== 2004+0+0 (3092140071 0 0) 0x55a3342d6c00 con 0x55a314e9a800 > -7802> 2018-07-11 07:33:07.556143 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6806/15885 conn(0x55a314e00000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1825 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:07.555345) v4 > -7801> 2018-07-11 07:33:07.556160 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6806/15885 1825 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:07.555345) v4 ==== 2004+0+0 (3092140071 0 0) 0x55a332e79a00 con 0x55a314e00000 > -7800> 2018-07-11 07:33:07.556175 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6811/12650 conn(0x55a314e20800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1825 0x55a3342d6c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:07.555345) v4 > -7799> 2018-07-11 07:33:07.556188 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6811/12650 1825 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:07.555345) v4 ==== 2004+0+0 (3092140071 0 0) 0x55a3342d6c00 con 0x55a314e20800 > -7798> 2018-07-11 07:33:07.556219 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6803/11967 conn(0x55a314e13800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1825 0x55a3342d6c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:07.555345) v4 > -7797> 2018-07-11 07:33:07.556205 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6802/13323 conn(0x55a314e15000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1825 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:07.555345) v4 > -7796> 2018-07-11 07:33:07.556233 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6803/11967 1825 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:07.555345) v4 ==== 2004+0+0 (3092140071 0 0) 0x55a3342d6c00 con 0x55a314e13800 > -7795> 2018-07-11 07:33:07.556234 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6802/13323 1825 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:07.555345) v4 ==== 2004+0+0 (3092140071 0 0) 0x55a332e79a00 con 0x55a314e15000 > -7794> 2018-07-11 07:33:07.556236 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6802/11967 conn(0x55a314e12000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1825 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:07.555345) v4 > -7793> 2018-07-11 07:33:07.556265 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6802/11967 1825 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:07.555345) v4 ==== 2004+0+0 (3092140071 0 0) 0x55a34de78600 con 0x55a314e12000 > -7792> 2018-07-11 07:33:07.556285 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6802/15919 conn(0x55a314e87000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1825 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:07.555345) v4 > -7791> 2018-07-11 07:33:07.556300 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6802/15919 1825 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:07.555345) v4 ==== 2004+0+0 (3092140071 0 0) 0x55a332e79a00 con 0x55a314e87000 > -7790> 2018-07-11 07:33:07.556305 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6807/15885 conn(0x55a314e01800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1825 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:07.555345) v4 > -7789> 2018-07-11 07:33:07.556315 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6807/15885 1825 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:07.555345) v4 ==== 2004+0+0 (3092140071 0 0) 0x55a34de78600 con 0x55a314e01800 > -7788> 2018-07-11 07:33:07.556349 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6803/15919 conn(0x55a314e88800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1825 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:07.555345) v4 > -7787> 2018-07-11 07:33:07.556361 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6803/15919 1825 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:07.555345) v4 ==== 2004+0+0 (3092140071 0 0) 0x55a34de78600 con 0x55a314e88800 > -7786> 2018-07-11 07:33:08.259179 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a31c079200 con 0 > -7785> 2018-07-11 07:33:08.278063 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5056 0x55a317923900 replica scrub(pg: 6.15,from:0'0,to:430'105662,epoch:494/492,start:6:a9a8f245:::benchmark_data_magna108.ceph.redhat.com_2478_object208123:0,end:6:a9a9e545:::benchmark_data_magna108_35617_object120072:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -7784> 2018-07-11 07:33:08.278096 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5056 ==== replica scrub(pg: 6.15,from:0'0,to:430'105662,epoch:494/492,start:6:a9a8f245:::benchmark_data_magna108.ceph.redhat.com_2478_object208123:0,end:6:a9a9e545:::benchmark_data_magna108_35617_object120072:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (3758117604 0 0) 0x55a317923900 con 0x55a317019000 > -7783> 2018-07-11 07:33:08.969441 7f355d88e700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.107:0/12617 conn(0x55a32447f800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1798 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:33:08.968960) v4 > -7782> 2018-07-11 07:33:08.969465 7f355d88e700 1 -- 10.8.128.107:6803/12665 <== osd.9 10.8.128.107:0/12617 1798 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:08.968960) v4 ==== 2004+0+0 (1707024906 0 0) 0x55a34de78600 con 0x55a32447f800 > -7781> 2018-07-11 07:33:08.969478 7f355d88e700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:08.968960) v4 -- 0x55a31e8f5400 con 0 > -7780> 2018-07-11 07:33:08.969523 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.107:0/12617 conn(0x55a32447e000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1798 0x55a3342d6c00 osd_ping(ping e494 stamp 2018-07-11 07:33:08.968960) v4 > -7779> 2018-07-11 07:33:08.969545 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.9 10.8.128.107:0/12617 1798 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:08.968960) v4 ==== 2004+0+0 (1707024906 0 0) 0x55a3342d6c00 con 0x55a32447e000 > -7778> 2018-07-11 07:33:08.969556 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:08.968960) v4 -- 0x55a31e8f4800 con 0 > -7777> 2018-07-11 07:33:08.993575 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a324102fc0 con 0 > -7776> 2018-07-11 07:33:09.010111 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5057 0x55a317923b80 replica scrub(pg: 6.15,from:0'0,to:430'105763,epoch:494/492,start:6:a9a9e545:::benchmark_data_magna108_35617_object120072:0,end:6:a9aa8c04:::benchmark_data_magna108_35617_object919518:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -7775> 2018-07-11 07:33:09.010136 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5057 ==== replica scrub(pg: 6.15,from:0'0,to:430'105763,epoch:494/492,start:6:a9a9e545:::benchmark_data_magna108_35617_object120072:0,end:6:a9aa8c04:::benchmark_data_magna108_35617_object919518:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (3482477285 0 0) 0x55a317923b80 con 0x55a317019000 > -7774> 2018-07-11 07:33:09.641854 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.115:0/13323 conn(0x55a321665800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1838 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:33:09.978815) v4 > -7773> 2018-07-11 07:33:09.641886 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.6 10.8.128.115:0/13323 1838 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:09.978815) v4 ==== 2004+0+0 (658940446 0 0) 0x55a332e79a00 con 0x55a321665800 > -7772> 2018-07-11 07:33:09.641902 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:09.978815) v4 -- 0x55a31e8f4a00 con 0 > -7771> 2018-07-11 07:33:09.641986 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.115:0/13323 conn(0x55a321668800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1838 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:33:09.978815) v4 > -7770> 2018-07-11 07:33:09.642003 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.6 10.8.128.115:0/13323 1838 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:09.978815) v4 ==== 2004+0+0 (658940446 0 0) 0x55a324474c00 con 0x55a321668800 > -7769> 2018-07-11 07:33:09.642013 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:09.978815) v4 -- 0x55a31e8f4a00 con 0 > -7768> 2018-07-11 07:33:09.658780 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12651 conn(0x55a3216a0800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1830 0x55a329323a00 osd_ping(ping e494 stamp 2018-07-11 07:33:09.693859) v4 > -7767> 2018-07-11 07:33:09.658784 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12651 conn(0x55a32169f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1830 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:33:09.693859) v4 > -7766> 2018-07-11 07:33:09.658808 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.2 10.8.128.106:0/12651 1830 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:09.693859) v4 ==== 2004+0+0 (202578331 0 0) 0x55a329323a00 con 0x55a3216a0800 > -7765> 2018-07-11 07:33:09.658811 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.2 10.8.128.106:0/12651 1830 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:09.693859) v4 ==== 2004+0+0 (202578331 0 0) 0x55a332e79a00 con 0x55a32169f000 > -7764> 2018-07-11 07:33:09.658827 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:09.693859) v4 -- 0x55a329323800 con 0 > -7763> 2018-07-11 07:33:09.658862 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:09.693859) v4 -- 0x55a31e8f4400 con 0 > -7762> 2018-07-11 07:33:09.821552 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15287 conn(0x55a311b75000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1819 0x55a3342d6c00 osd_ping(ping e494 stamp 2018-07-11 07:33:11.034527) v4 > -7761> 2018-07-11 07:33:09.821554 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15287 conn(0x55a311b72000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1819 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:33:11.034527) v4 > -7760> 2018-07-11 07:33:09.821579 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.4 10.8.128.113:0/15287 1819 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:11.034527) v4 ==== 2004+0+0 (3515075957 0 0) 0x55a3342d6c00 con 0x55a311b75000 > -7759> 2018-07-11 07:33:09.821581 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.4 10.8.128.113:0/15287 1819 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:11.034527) v4 ==== 2004+0+0 (3515075957 0 0) 0x55a324474c00 con 0x55a311b72000 > -7758> 2018-07-11 07:33:09.821597 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:11.034527) v4 -- 0x55a3342d7400 con 0 > -7757> 2018-07-11 07:33:09.821621 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:11.034527) v4 -- 0x55a343719200 con 0 > -7756> 2018-07-11 07:33:09.825116 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a3123eb440 con 0 > -7755> 2018-07-11 07:33:09.835580 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5058 0x55a327d44000 replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:494/492,start:6:a9aa8c04:::benchmark_data_magna108_35617_object919518:0,end:6:a9ab63d9:::benchmark_data_magna108_35617_object861132:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -7754> 2018-07-11 07:33:09.835609 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5058 ==== replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:494/492,start:6:a9aa8c04:::benchmark_data_magna108_35617_object919518:0,end:6:a9ab63d9:::benchmark_data_magna108_35617_object861132:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (3043321695 0 0) 0x55a327d44000 con 0x55a317019000 > -7753> 2018-07-11 07:33:09.855859 7f3543899700 5 osd.10 494 heartbeat: osd_stat(783 GB used, 147 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,1]) > -7752> 2018-07-11 07:33:09.855872 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6802/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:33:09.855871) v4 -- 0x55a327983600 con 0 > -7751> 2018-07-11 07:33:09.855906 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6803/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:33:09.855871) v4 -- 0x55a325c0fc00 con 0 > -7750> 2018-07-11 07:33:09.855921 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6806/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:33:09.855871) v4 -- 0x55a319405a00 con 0 > -7749> 2018-07-11 07:33:09.855942 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6807/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:33:09.855871) v4 -- 0x55a327983400 con 0 > -7748> 2018-07-11 07:33:09.855975 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6808/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:33:09.855871) v4 -- 0x55a3342d6600 con 0 > -7747> 2018-07-11 07:33:09.855993 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6810/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:33:09.855871) v4 -- 0x55a3342d6e00 con 0 > -7746> 2018-07-11 07:33:09.856011 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6809/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:33:09.855871) v4 -- 0x55a325c0fa00 con 0 > -7745> 2018-07-11 07:33:09.856033 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6811/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:33:09.855871) v4 -- 0x55a342825400 con 0 > -7744> 2018-07-11 07:33:09.856057 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6810/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:33:09.855871) v4 -- 0x55a344ac9e00 con 0 > -7743> 2018-07-11 07:33:09.856072 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6811/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:33:09.855871) v4 -- 0x55a32316c400 con 0 > -7742> 2018-07-11 07:33:09.856093 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6806/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:33:09.855871) v4 -- 0x55a32316c200 con 0 > -7741> 2018-07-11 07:33:09.856115 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6807/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:33:09.855871) v4 -- 0x55a32316c600 con 0 > -7740> 2018-07-11 07:33:09.856160 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6802/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:33:09.855871) v4 -- 0x55a32316c000 con 0 > -7739> 2018-07-11 07:33:09.856197 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6803/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:33:09.855871) v4 -- 0x55a32316ca00 con 0 > -7738> 2018-07-11 07:33:09.856223 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6806/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:33:09.855871) v4 -- 0x55a32316d200 con 0 > -7737> 2018-07-11 07:33:09.856261 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6807/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:33:09.855871) v4 -- 0x55a32316c800 con 0 > -7736> 2018-07-11 07:33:09.856309 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6802/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:33:09.855871) v4 -- 0x55a32316cc00 con 0 > -7735> 2018-07-11 07:33:09.856330 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6802/15255 conn(0x55a314e1c000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1826 0x55a3342d7400 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:09.855871) v4 > -7734> 2018-07-11 07:33:09.856340 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6803/15255 conn(0x55a314e1d800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1826 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:09.855871) v4 > -7733> 2018-07-11 07:33:09.856364 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6802/15255 1826 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:09.855871) v4 ==== 2004+0+0 (808418689 0 0) 0x55a3342d7400 con 0x55a314e1c000 > -7732> 2018-07-11 07:33:09.856364 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6803/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:33:09.855871) v4 -- 0x55a32316d800 con 0 > -7731> 2018-07-11 07:33:09.856368 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6803/15255 1826 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:09.855871) v4 ==== 2004+0+0 (808418689 0 0) 0x55a324474c00 con 0x55a314e1d800 > -7730> 2018-07-11 07:33:09.856366 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6806/12677 conn(0x55a314e84000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1826 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:09.855871) v4 > -7729> 2018-07-11 07:33:09.856388 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6806/12677 1826 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:09.855871) v4 ==== 2004+0+0 (808418689 0 0) 0x55a34de78600 con 0x55a314e84000 > -7728> 2018-07-11 07:33:09.856403 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6806/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:33:09.855871) v4 -- 0x55a32316d000 con 0 > -7727> 2018-07-11 07:33:09.856421 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6807/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:33:09.855871) v4 -- 0x55a32316d600 con 0 > -7726> 2018-07-11 07:33:09.856433 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6802/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:33:09.855871) v4 -- 0x55a32316ce00 con 0 > -7725> 2018-07-11 07:33:09.856444 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6803/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:33:09.855871) v4 -- 0x55a32316dc00 con 0 > -7724> 2018-07-11 07:33:09.856509 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6809/12650 conn(0x55a314e1f000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1826 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:09.855871) v4 > -7723> 2018-07-11 07:33:09.856524 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6809/12650 1826 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:09.855871) v4 ==== 2004+0+0 (808418689 0 0) 0x55a34de78600 con 0x55a314e1f000 > -7722> 2018-07-11 07:33:09.856529 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6808/12651 conn(0x55a314e96000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1826 0x55a329323a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:09.855871) v4 > -7721> 2018-07-11 07:33:09.856545 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6808/12651 1826 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:09.855871) v4 ==== 2004+0+0 (808418689 0 0) 0x55a329323a00 con 0x55a314e96000 > -7720> 2018-07-11 07:33:09.856555 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6810/12651 conn(0x55a314e97800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1826 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:09.855871) v4 > -7719> 2018-07-11 07:33:09.856572 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6810/12651 1826 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:09.855871) v4 ==== 2004+0+0 (808418689 0 0) 0x55a332e79a00 con 0x55a314e97800 > -7718> 2018-07-11 07:33:09.856596 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6807/12677 conn(0x55a314e85800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1826 0x55a329323a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:09.855871) v4 > -7717> 2018-07-11 07:33:09.856609 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6807/12677 1826 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:09.855871) v4 ==== 2004+0+0 (808418689 0 0) 0x55a329323a00 con 0x55a314e85800 > -7716> 2018-07-11 07:33:09.856608 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6811/15287 conn(0x55a314e04800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1826 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:09.855871) v4 > -7715> 2018-07-11 07:33:09.856618 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6811/15287 1826 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:09.855871) v4 ==== 2004+0+0 (808418689 0 0) 0x55a332e79a00 con 0x55a314e04800 > -7714> 2018-07-11 07:33:09.856614 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6806/15263 conn(0x55a314e99000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1826 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:09.855871) v4 > -7713> 2018-07-11 07:33:09.856635 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6806/15263 1826 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:09.855871) v4 ==== 2004+0+0 (808418689 0 0) 0x55a34de6f000 con 0x55a314e99000 > -7712> 2018-07-11 07:33:09.856639 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6811/12650 conn(0x55a314e20800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1826 0x55a329323a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:09.855871) v4 > -7711> 2018-07-11 07:33:09.856651 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6811/12650 1826 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:09.855871) v4 ==== 2004+0+0 (808418689 0 0) 0x55a329323a00 con 0x55a314e20800 > -7710> 2018-07-11 07:33:09.856654 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6806/15885 conn(0x55a314e00000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1826 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:09.855871) v4 > -7709> 2018-07-11 07:33:09.856666 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6806/15885 1826 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:09.855871) v4 ==== 2004+0+0 (808418689 0 0) 0x55a332e79a00 con 0x55a314e00000 > -7708> 2018-07-11 07:33:09.856691 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6803/13323 conn(0x55a314e16800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1826 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:09.855871) v4 > -7707> 2018-07-11 07:33:09.856696 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6802/13323 conn(0x55a314e15000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1826 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:09.855871) v4 > -7706> 2018-07-11 07:33:09.856701 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6810/15287 conn(0x55a314e03000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1826 0x55a329323a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:09.855871) v4 > -7705> 2018-07-11 07:33:09.856714 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6802/13323 1826 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:09.855871) v4 ==== 2004+0+0 (808418689 0 0) 0x55a332e79a00 con 0x55a314e15000 > -7704> 2018-07-11 07:33:09.856704 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6803/13323 1826 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:09.855871) v4 ==== 2004+0+0 (808418689 0 0) 0x55a34de6f000 con 0x55a314e16800 > -7703> 2018-07-11 07:33:09.856722 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6810/15287 1826 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:09.855871) v4 ==== 2004+0+0 (808418689 0 0) 0x55a329323a00 con 0x55a314e03000 > -7702> 2018-07-11 07:33:09.856738 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6806/12617 conn(0x55a314e26000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1826 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:09.855871) v4 > -7701> 2018-07-11 07:33:09.856752 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6807/12617 conn(0x55a314e27800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1826 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:09.855871) v4 > -7700> 2018-07-11 07:33:09.856755 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6806/12617 1826 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:09.855871) v4 ==== 2004+0+0 (808418689 0 0) 0x55a332e79a00 con 0x55a314e26000 > -7699> 2018-07-11 07:33:09.856766 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6807/12617 1826 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:09.855871) v4 ==== 2004+0+0 (808418689 0 0) 0x55a34de6f000 con 0x55a314e27800 > -7698> 2018-07-11 07:33:09.856766 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6807/15263 conn(0x55a314e9a800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1826 0x55a329323a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:09.855871) v4 > -7697> 2018-07-11 07:33:09.856777 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6807/15263 1826 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:09.855871) v4 ==== 2004+0+0 (808418689 0 0) 0x55a329323a00 con 0x55a314e9a800 > -7696> 2018-07-11 07:33:09.856797 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6807/15885 conn(0x55a314e01800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1826 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:09.855871) v4 > -7695> 2018-07-11 07:33:09.856799 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6802/15919 conn(0x55a314e87000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1826 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:09.855871) v4 > -7694> 2018-07-11 07:33:09.856809 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6807/15885 1826 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:09.855871) v4 ==== 2004+0+0 (808418689 0 0) 0x55a34de6f000 con 0x55a314e01800 > -7693> 2018-07-11 07:33:09.856812 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6802/15919 1826 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:09.855871) v4 ==== 2004+0+0 (808418689 0 0) 0x55a332e79a00 con 0x55a314e87000 > -7692> 2018-07-11 07:33:09.856915 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6803/11967 conn(0x55a314e13800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1826 0x55a329323a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:09.855871) v4 > -7691> 2018-07-11 07:33:09.856931 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6803/11967 1826 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:09.855871) v4 ==== 2004+0+0 (808418689 0 0) 0x55a329323a00 con 0x55a314e13800 > -7690> 2018-07-11 07:33:09.856929 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6802/11967 conn(0x55a314e12000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1826 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:09.855871) v4 > -7689> 2018-07-11 07:33:09.856941 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6802/11967 1826 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:09.855871) v4 ==== 2004+0+0 (808418689 0 0) 0x55a34de6f000 con 0x55a314e12000 > -7688> 2018-07-11 07:33:09.856991 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6803/15919 conn(0x55a314e88800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1826 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:09.855871) v4 > -7687> 2018-07-11 07:33:09.857016 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6803/15919 1826 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:09.855871) v4 ==== 2004+0+0 (808418689 0 0) 0x55a34de6f000 con 0x55a314e88800 > -7686> 2018-07-11 07:33:10.121694 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15255 conn(0x55a311b64000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1852 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:33:11.334511) v4 > -7685> 2018-07-11 07:33:10.121716 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.0 10.8.128.113:0/15255 1852 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:11.334511) v4 ==== 2004+0+0 (1664864940 0 0) 0x55a34de6f000 con 0x55a311b64000 > -7684> 2018-07-11 07:33:10.121725 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:11.334511) v4 -- 0x55a31e8f4e00 con 0 > -7683> 2018-07-11 07:33:10.121735 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15255 conn(0x55a311b65800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1852 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:33:11.334511) v4 > -7682> 2018-07-11 07:33:10.121757 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.0 10.8.128.113:0/15255 1852 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:11.334511) v4 ==== 2004+0+0 (1664864940 0 0) 0x55a332e79a00 con 0x55a311b65800 > -7681> 2018-07-11 07:33:10.121769 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:11.334511) v4 -- 0x55a343719200 con 0 > -7680> 2018-07-11 07:33:10.259839 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15885 conn(0x55a3170b5800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1846 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:33:10.883320) v4 > -7679> 2018-07-11 07:33:10.259842 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15885 conn(0x55a311b76800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1846 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:33:10.883320) v4 > -7678> 2018-07-11 07:33:10.259864 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.7 10.8.128.114:0/15885 1846 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:10.883320) v4 ==== 2004+0+0 (1601666723 0 0) 0x55a324474c00 con 0x55a3170b5800 > -7677> 2018-07-11 07:33:10.259869 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.7 10.8.128.114:0/15885 1846 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:10.883320) v4 ==== 2004+0+0 (1601666723 0 0) 0x55a34de78600 con 0x55a311b76800 > -7676> 2018-07-11 07:33:10.259876 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:10.883320) v4 -- 0x55a343719200 con 0 > -7675> 2018-07-11 07:33:10.259899 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:10.883320) v4 -- 0x55a31e8f4e00 con 0 > -7674> 2018-07-11 07:33:10.659920 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a3123ea000 con 0 > -7673> 2018-07-11 07:33:10.686096 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5059 0x55a327d44280 replica scrub(pg: 6.15,from:0'0,to:430'106777,epoch:494/492,start:6:a9ab63d9:::benchmark_data_magna108_35617_object861132:0,end:6:a9acc76d:::benchmark_data_magna108.ceph.redhat.com_2478_object352093:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -7672> 2018-07-11 07:33:10.686121 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5059 ==== replica scrub(pg: 6.15,from:0'0,to:430'106777,epoch:494/492,start:6:a9ab63d9:::benchmark_data_magna108_35617_object861132:0,end:6:a9acc76d:::benchmark_data_magna108.ceph.redhat.com_2478_object352093:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (1498639232 0 0) 0x55a327d44280 con 0x55a317019000 > -7671> 2018-07-11 07:33:10.925308 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4800000 on bdev 1, free 0x2fe000; fallback to bdev 2 > -7670> 2018-07-11 07:33:11.011309 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15263 conn(0x55a321664000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1824 0x55a329323a00 osd_ping(ping e494 stamp 2018-07-11 07:33:12.224274) v4 > -7669> 2018-07-11 07:33:11.011337 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.5 10.8.128.113:0/15263 1824 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:12.224274) v4 ==== 2004+0+0 (2561808589 0 0) 0x55a329323a00 con 0x55a321664000 > -7668> 2018-07-11 07:33:11.011356 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:12.224274) v4 -- 0x55a327983400 con 0 > -7667> 2018-07-11 07:33:11.011451 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15263 conn(0x55a321667000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1824 0x55a3342d7400 osd_ping(ping e494 stamp 2018-07-11 07:33:12.224274) v4 > -7666> 2018-07-11 07:33:11.011472 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.5 10.8.128.113:0/15263 1824 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:12.224274) v4 ==== 2004+0+0 (2561808589 0 0) 0x55a3342d7400 con 0x55a321667000 > -7665> 2018-07-11 07:33:11.011482 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:12.224274) v4 -- 0x55a327983400 con 0 > -7664> 2018-07-11 07:33:11.342938 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b8800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1777 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:33:11.093309) v4 > -7663> 2018-07-11 07:33:11.342945 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b7000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1777 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:33:11.093309) v4 > -7662> 2018-07-11 07:33:11.342962 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.8 10.8.128.108:0/11967 1777 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:11.093309) v4 ==== 2004+0+0 (260795581 0 0) 0x55a332e79a00 con 0x55a3170b8800 > -7661> 2018-07-11 07:33:11.342963 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.8 10.8.128.108:0/11967 1777 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:11.093309) v4 ==== 2004+0+0 (260795581 0 0) 0x55a34de6f000 con 0x55a3170b7000 > -7660> 2018-07-11 07:33:11.342972 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:11.093309) v4 -- 0x55a343719200 con 0 > -7659> 2018-07-11 07:33:11.342996 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:11.093309) v4 -- 0x55a31e8f4e00 con 0 > -7658> 2018-07-11 07:33:11.522030 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15287 conn(0x55a311b75000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1820 0x55a329323a00 osd_ping(ping e494 stamp 2018-07-11 07:33:12.734965) v4 > -7657> 2018-07-11 07:33:11.522030 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15287 conn(0x55a311b72000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1820 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:33:12.734965) v4 > -7656> 2018-07-11 07:33:11.522060 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.4 10.8.128.113:0/15287 1820 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:12.734965) v4 ==== 2004+0+0 (507830026 0 0) 0x55a324474c00 con 0x55a311b72000 > -7655> 2018-07-11 07:33:11.522059 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.4 10.8.128.113:0/15287 1820 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:12.734965) v4 ==== 2004+0+0 (507830026 0 0) 0x55a329323a00 con 0x55a311b75000 > -7654> 2018-07-11 07:33:11.522079 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:12.734965) v4 -- 0x55a343719200 con 0 > -7653> 2018-07-11 07:33:11.522108 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:12.734965) v4 -- 0x55a327983400 con 0 > -7652> 2018-07-11 07:33:11.568896 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12650 conn(0x55a311b68800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1841 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:33:11.604127) v4 > -7651> 2018-07-11 07:33:11.568919 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.3 10.8.128.106:0/12650 1841 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:11.604127) v4 ==== 2004+0+0 (578486384 0 0) 0x55a34de78600 con 0x55a311b68800 > -7650> 2018-07-11 07:33:11.568935 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:11.604127) v4 -- 0x55a31e8f4600 con 0 > -7649> 2018-07-11 07:33:11.568986 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12650 conn(0x55a311b67000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1841 0x55a3342d7400 osd_ping(ping e494 stamp 2018-07-11 07:33:11.604127) v4 > -7648> 2018-07-11 07:33:11.569011 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.3 10.8.128.106:0/12650 1841 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:11.604127) v4 ==== 2004+0+0 (578486384 0 0) 0x55a3342d7400 con 0x55a311b67000 > -7647> 2018-07-11 07:33:11.569026 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:11.604127) v4 -- 0x55a327983600 con 0 > -7646> 2018-07-11 07:33:12.069449 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12650 conn(0x55a311b67000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1842 0x55a329323a00 osd_ping(ping e494 stamp 2018-07-11 07:33:12.104574) v4 > -7645> 2018-07-11 07:33:12.069471 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.3 10.8.128.106:0/12650 1842 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:12.104574) v4 ==== 2004+0+0 (3711870144 0 0) 0x55a329323a00 con 0x55a311b67000 > -7644> 2018-07-11 07:33:12.069486 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:12.104574) v4 -- 0x55a333f6e000 con 0 > -7643> 2018-07-11 07:33:12.069588 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12650 conn(0x55a311b68800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1842 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:33:12.104574) v4 > -7642> 2018-07-11 07:33:12.069609 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.3 10.8.128.106:0/12650 1842 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:12.104574) v4 ==== 2004+0+0 (3711870144 0 0) 0x55a34de6f000 con 0x55a311b68800 > -7641> 2018-07-11 07:33:12.069625 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:12.104574) v4 -- 0x55a3342d6800 con 0 > -7640> 2018-07-11 07:33:12.073926 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 58] Generated table #36572: 2638241 keys, 74546299 bytes > -7639> 2018-07-11 07:33:12.073975 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294392073944, "cf_name": "default", "job": 58, "event": "table_file_creation", "file_number": 36572, "file_size": 74546299, "table_properties": {"data_size": 67112148, "index_size": 689978, "filter_size": 6743183, "raw_key_size": 76215855, "raw_average_key_size": 28, "raw_value_size": 31365758, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} > -7638> 2018-07-11 07:33:12.121573 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12677 conn(0x55a32169d800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1847 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:33:12.156696) v4 > -7637> 2018-07-11 07:33:12.121574 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12677 conn(0x55a32169c000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1847 0x55a3342d7400 osd_ping(ping e494 stamp 2018-07-11 07:33:12.156696) v4 > -7636> 2018-07-11 07:33:12.121599 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.1 10.8.128.106:0/12677 1847 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:12.156696) v4 ==== 2004+0+0 (3848536218 0 0) 0x55a34de78600 con 0x55a32169d800 > -7635> 2018-07-11 07:33:12.121603 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.1 10.8.128.106:0/12677 1847 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:12.156696) v4 ==== 2004+0+0 (3848536218 0 0) 0x55a3342d7400 con 0x55a32169c000 > -7634> 2018-07-11 07:33:12.121611 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:12.156696) v4 -- 0x55a32eb21600 con 0 > -7633> 2018-07-11 07:33:12.121644 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:12.156696) v4 -- 0x55a3342d6a00 con 0 > -7632> 2018-07-11 07:33:12.251063 7f35488a3700 1 -- 10.8.128.107:6800/12665 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55a32a57a580 con 0 > -7631> 2018-07-11 07:33:12.251124 7f35488a3700 1 -- 10.8.128.107:6800/12665 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55a32a57b340 con 0 > -7630> 2018-07-11 07:33:12.462350 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a3123ebb00 con 0 > -7629> 2018-07-11 07:33:12.477575 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5060 0x55a327d44500 replica scrub(pg: 6.15,from:0'0,to:430'106468,epoch:494/492,start:6:a9acc76d:::benchmark_data_magna108.ceph.redhat.com_2478_object352093:0,end:6:a9add609:::benchmark_data_magna108.ceph.redhat.com_2478_object192833:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -7628> 2018-07-11 07:33:12.477599 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5060 ==== replica scrub(pg: 6.15,from:0'0,to:430'106468,epoch:494/492,start:6:a9acc76d:::benchmark_data_magna108.ceph.redhat.com_2478_object352093:0,end:6:a9add609:::benchmark_data_magna108.ceph.redhat.com_2478_object192833:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 248+0+0 (3352457348 0 0) 0x55a327d44500 con 0x55a317019000 > -7627> 2018-07-11 07:33:12.622001 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12677 conn(0x55a32169d800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1848 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:33:12.657119) v4 > -7626> 2018-07-11 07:33:12.622024 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.1 10.8.128.106:0/12677 1848 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:12.657119) v4 ==== 2004+0+0 (3062426940 0 0) 0x55a34de6f000 con 0x55a32169d800 > -7625> 2018-07-11 07:33:12.622015 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12677 conn(0x55a32169c000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1848 0x55a329323a00 osd_ping(ping e494 stamp 2018-07-11 07:33:12.657119) v4 > -7624> 2018-07-11 07:33:12.622035 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:12.657119) v4 -- 0x55a32eb21600 con 0 > -7623> 2018-07-11 07:33:12.622040 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.1 10.8.128.106:0/12677 1848 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:12.657119) v4 ==== 2004+0+0 (3062426940 0 0) 0x55a329323a00 con 0x55a32169c000 > -7622> 2018-07-11 07:33:12.622056 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:12.657119) v4 -- 0x55a325c0e000 con 0 > -7621> 2018-07-11 07:33:13.022150 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15255 conn(0x55a311b65800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1853 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:33:14.235090) v4 > -7620> 2018-07-11 07:33:13.022149 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15255 conn(0x55a311b64000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1853 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:33:14.235090) v4 > -7619> 2018-07-11 07:33:13.022182 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.0 10.8.128.113:0/15255 1853 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:14.235090) v4 ==== 2004+0+0 (986349223 0 0) 0x55a332e79a00 con 0x55a311b65800 > -7618> 2018-07-11 07:33:13.022185 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.0 10.8.128.113:0/15255 1853 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:14.235090) v4 ==== 2004+0+0 (986349223 0 0) 0x55a34de78600 con 0x55a311b64000 > -7617> 2018-07-11 07:33:13.022200 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:14.235090) v4 -- 0x55a343719200 con 0 > -7616> 2018-07-11 07:33:13.022230 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:14.235090) v4 -- 0x55a32eb21600 con 0 > -7615> 2018-07-11 07:33:13.069947 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.107:0/12617 conn(0x55a32447e000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1799 0x55a3342d6a00 osd_ping(ping e494 stamp 2018-07-11 07:33:13.069611) v4 > -7614> 2018-07-11 07:33:13.069956 7f355d88e700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.107:0/12617 conn(0x55a32447f800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1799 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:33:13.069611) v4 > -7613> 2018-07-11 07:33:13.069969 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.9 10.8.128.107:0/12617 1799 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:13.069611) v4 ==== 2004+0+0 (2819163522 0 0) 0x55a3342d6a00 con 0x55a32447e000 > -7612> 2018-07-11 07:33:13.069977 7f355d88e700 1 -- 10.8.128.107:6803/12665 <== osd.9 10.8.128.107:0/12617 1799 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:13.069611) v4 ==== 2004+0+0 (2819163522 0 0) 0x55a34de6f000 con 0x55a32447f800 > -7611> 2018-07-11 07:33:13.069987 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:13.069611) v4 -- 0x55a325c0e000 con 0 > -7610> 2018-07-11 07:33:13.070017 7f355d88e700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:13.069611) v4 -- 0x55a32eb21600 con 0 > -7609> 2018-07-11 07:33:13.169945 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12650 conn(0x55a311b68800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1843 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:33:13.205113) v4 > -7608> 2018-07-11 07:33:13.169965 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.3 10.8.128.106:0/12650 1843 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:13.205113) v4 ==== 2004+0+0 (450787442 0 0) 0x55a34de78600 con 0x55a311b68800 > -7607> 2018-07-11 07:33:13.169981 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:13.205113) v4 -- 0x55a32eb21600 con 0 > -7606> 2018-07-11 07:33:13.170093 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12650 conn(0x55a311b67000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1843 0x55a329323a00 osd_ping(ping e494 stamp 2018-07-11 07:33:13.205113) v4 > -7605> 2018-07-11 07:33:13.170111 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.3 10.8.128.106:0/12650 1843 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:13.205113) v4 ==== 2004+0+0 (450787442 0 0) 0x55a329323a00 con 0x55a311b67000 > -7604> 2018-07-11 07:33:13.170121 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:13.205113) v4 -- 0x55a325c0e000 con 0 > -7603> 2018-07-11 07:33:13.356306 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a33b29a900 con 0 > -7602> 2018-07-11 07:33:13.377706 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5061 0x55a327d44a00 replica scrub(pg: 6.15,from:0'0,to:430'106653,epoch:494/492,start:6:a9add609:::benchmark_data_magna108.ceph.redhat.com_2478_object192833:0,end:6:a9af1fe6:::benchmark_data_magna108_35617_object972804:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -7601> 2018-07-11 07:33:13.377729 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5061 ==== replica scrub(pg: 6.15,from:0'0,to:430'106653,epoch:494/492,start:6:a9add609:::benchmark_data_magna108.ceph.redhat.com_2478_object192833:0,end:6:a9af1fe6:::benchmark_data_magna108_35617_object972804:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (2602169262 0 0) 0x55a327d44a00 con 0x55a317019000 > -7600> 2018-07-11 07:33:13.450470 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15919 conn(0x55a317074800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1831 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:33:14.074000) v4 > -7599> 2018-07-11 07:33:13.450488 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15919 conn(0x55a3170b4000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1831 0x55a3342d6a00 osd_ping(ping e494 stamp 2018-07-11 07:33:14.074000) v4 > -7598> 2018-07-11 07:33:13.450497 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.11 10.8.128.114:0/15919 1831 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:14.074000) v4 ==== 2004+0+0 (771270282 0 0) 0x55a324474c00 con 0x55a317074800 > -7597> 2018-07-11 07:33:13.450509 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.11 10.8.128.114:0/15919 1831 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:14.074000) v4 ==== 2004+0+0 (771270282 0 0) 0x55a3342d6a00 con 0x55a3170b4000 > -7596> 2018-07-11 07:33:13.450512 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:14.074000) v4 -- 0x55a343719200 con 0 > -7595> 2018-07-11 07:33:13.450546 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:14.074000) v4 -- 0x55a3342d7800 con 0 > -7594> 2018-07-11 07:33:14.077381 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a322700b40 con 0 > -7593> 2018-07-11 07:33:14.086279 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5062 0x55a327d44c80 replica scrub(pg: 6.15,from:0'0,to:430'106841,epoch:494/492,start:6:a9af1fe6:::benchmark_data_magna108_35617_object972804:0,end:6:a9b02c14:::benchmark_data_magna108_35617_object1149094:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -7592> 2018-07-11 07:33:14.086304 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5062 ==== replica scrub(pg: 6.15,from:0'0,to:430'106841,epoch:494/492,start:6:a9af1fe6:::benchmark_data_magna108_35617_object972804:0,end:6:a9b02c14:::benchmark_data_magna108_35617_object1149094:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 219+0+0 (1976849324 0 0) 0x55a327d44c80 con 0x55a317019000 > -7591> 2018-07-11 07:33:14.170593 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.107:0/12617 conn(0x55a32447e000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1800 0x55a325c0e000 osd_ping(ping e494 stamp 2018-07-11 07:33:14.170105) v4 > -7590> 2018-07-11 07:33:14.170599 7f355d88e700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.107:0/12617 conn(0x55a32447f800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1800 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:33:14.170105) v4 > -7589> 2018-07-11 07:33:14.170617 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.9 10.8.128.107:0/12617 1800 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:14.170105) v4 ==== 2004+0+0 (2409772978 0 0) 0x55a325c0e000 con 0x55a32447e000 > -7588> 2018-07-11 07:33:14.170622 7f355d88e700 1 -- 10.8.128.107:6803/12665 <== osd.9 10.8.128.107:0/12617 1800 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:14.170105) v4 ==== 2004+0+0 (2409772978 0 0) 0x55a34de6f000 con 0x55a32447f800 > -7587> 2018-07-11 07:33:14.170628 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:14.170105) v4 -- 0x55a3342d7800 con 0 > -7586> 2018-07-11 07:33:14.170652 7f355d88e700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:14.170105) v4 -- 0x55a32eb21600 con 0 > -7585> 2018-07-11 07:33:14.243376 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b7000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1778 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:33:13.993772) v4 > -7584> 2018-07-11 07:33:14.243398 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.8 10.8.128.108:0/11967 1778 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:13.993772) v4 ==== 2004+0+0 (1408928953 0 0) 0x55a34de78600 con 0x55a3170b7000 > -7583> 2018-07-11 07:33:14.243407 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:13.993772) v4 -- 0x55a32eb21600 con 0 > -7582> 2018-07-11 07:33:14.243415 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b8800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1778 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:33:13.993772) v4 > -7581> 2018-07-11 07:33:14.243435 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.8 10.8.128.108:0/11967 1778 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:13.993772) v4 ==== 2004+0+0 (1408928953 0 0) 0x55a332e79a00 con 0x55a3170b8800 > -7580> 2018-07-11 07:33:14.243446 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:13.993772) v4 -- 0x55a343719200 con 0 > -7579> 2018-07-11 07:33:14.422366 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15287 conn(0x55a311b75000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1821 0x55a3342d6a00 osd_ping(ping e494 stamp 2018-07-11 07:33:15.635377) v4 > -7578> 2018-07-11 07:33:14.422392 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.4 10.8.128.113:0/15287 1821 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:15.635377) v4 ==== 2004+0+0 (3254185814 0 0) 0x55a3342d6a00 con 0x55a311b75000 > -7577> 2018-07-11 07:33:14.422391 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15287 conn(0x55a311b72000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1821 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:33:15.635377) v4 > -7576> 2018-07-11 07:33:14.422408 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.4 10.8.128.113:0/15287 1821 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:15.635377) v4 ==== 2004+0+0 (3254185814 0 0) 0x55a324474c00 con 0x55a311b72000 > -7575> 2018-07-11 07:33:14.422410 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:15.635377) v4 -- 0x55a333f6e600 con 0 > -7574> 2018-07-11 07:33:14.422449 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:15.635377) v4 -- 0x55a33047e400 con 0 > -7573> 2018-07-11 07:33:14.928442 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a324016240 con 0 > -7572> 2018-07-11 07:33:14.942315 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.115:0/13323 conn(0x55a321665800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1839 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:33:15.279438) v4 > -7571> 2018-07-11 07:33:14.942343 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.6 10.8.128.115:0/13323 1839 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:15.279438) v4 ==== 2004+0+0 (334301715 0 0) 0x55a332e79a00 con 0x55a321665800 > -7570> 2018-07-11 07:33:14.942359 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:15.279438) v4 -- 0x55a3131ba800 con 0 > -7569> 2018-07-11 07:33:14.942446 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.115:0/13323 conn(0x55a321668800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1839 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:33:15.279438) v4 > -7568> 2018-07-11 07:33:14.942463 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.6 10.8.128.115:0/13323 1839 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:15.279438) v4 ==== 2004+0+0 (334301715 0 0) 0x55a324474c00 con 0x55a321668800 > -7567> 2018-07-11 07:33:14.942473 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:15.279438) v4 -- 0x55a34866a600 con 0 > -7566> 2018-07-11 07:33:14.953189 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5063 0x55a327d44f00 replica scrub(pg: 6.15,from:0'0,to:430'105715,epoch:494/492,start:6:a9b02c14:::benchmark_data_magna108_35617_object1149094:0,end:6:a9b136fe:::benchmark_data_magna108_35617_object841600:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -7565> 2018-07-11 07:33:14.953220 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5063 ==== replica scrub(pg: 6.15,from:0'0,to:430'105715,epoch:494/492,start:6:a9b02c14:::benchmark_data_magna108_35617_object1149094:0,end:6:a9b136fe:::benchmark_data_magna108_35617_object841600:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 219+0+0 (2149516289 0 0) 0x55a327d44f00 con 0x55a317019000 > -7564> 2018-07-11 07:33:14.959106 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12651 conn(0x55a3216a0800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1831 0x55a325c0e000 osd_ping(ping e494 stamp 2018-07-11 07:33:14.994335) v4 > -7563> 2018-07-11 07:33:14.959144 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.2 10.8.128.106:0/12651 1831 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:14.994335) v4 ==== 2004+0+0 (3549374190 0 0) 0x55a325c0e000 con 0x55a3216a0800 > -7562> 2018-07-11 07:33:14.959154 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:14.994335) v4 -- 0x55a3342d7200 con 0 > -7561> 2018-07-11 07:33:14.959202 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12651 conn(0x55a32169f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1831 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:33:14.994335) v4 > -7560> 2018-07-11 07:33:14.959223 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.2 10.8.128.106:0/12651 1831 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:14.994335) v4 ==== 2004+0+0 (3549374190 0 0) 0x55a332e79a00 con 0x55a32169f000 > -7559> 2018-07-11 07:33:14.959234 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:14.994335) v4 -- 0x55a34866a600 con 0 > -7558> 2018-07-11 07:33:15.156602 7f3543899700 5 osd.10 494 heartbeat: osd_stat(783 GB used, 147 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,1]) > -7557> 2018-07-11 07:33:15.156616 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6802/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:33:15.156615) v4 -- 0x55a32316da00 con 0 > -7556> 2018-07-11 07:33:15.156650 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6803/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:33:15.156615) v4 -- 0x55a32316d400 con 0 > -7555> 2018-07-11 07:33:15.156665 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6806/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:33:15.156615) v4 -- 0x55a343482800 con 0 > -7554> 2018-07-11 07:33:15.156678 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6807/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:33:15.156615) v4 -- 0x55a329ac3a00 con 0 > -7553> 2018-07-11 07:33:15.156686 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6808/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:33:15.156615) v4 -- 0x55a33610c600 con 0 > -7552> 2018-07-11 07:33:15.156694 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6810/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:33:15.156615) v4 -- 0x55a33610c000 con 0 > -7551> 2018-07-11 07:33:15.156714 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6809/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:33:15.156615) v4 -- 0x55a33610c200 con 0 > -7550> 2018-07-11 07:33:15.156723 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6811/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:33:15.156615) v4 -- 0x55a33610cc00 con 0 > -7549> 2018-07-11 07:33:15.156730 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6810/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:33:15.156615) v4 -- 0x55a33610ce00 con 0 > -7548> 2018-07-11 07:33:15.156740 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6811/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:33:15.156615) v4 -- 0x55a33610d200 con 0 > -7547> 2018-07-11 07:33:15.156752 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6806/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:33:15.156615) v4 -- 0x55a33610d400 con 0 > -7546> 2018-07-11 07:33:15.156764 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6807/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:33:15.156615) v4 -- 0x55a33610d800 con 0 > -7545> 2018-07-11 07:33:15.156775 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6802/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:33:15.156615) v4 -- 0x55a33610da00 con 0 > -7544> 2018-07-11 07:33:15.156783 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6803/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:33:15.156615) v4 -- 0x55a33610c800 con 0 > -7543> 2018-07-11 07:33:15.156793 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6806/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:33:15.156615) v4 -- 0x55a33610d600 con 0 > -7542> 2018-07-11 07:33:15.156804 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6807/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:33:15.156615) v4 -- 0x55a33610e000 con 0 > -7541> 2018-07-11 07:33:15.156816 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6802/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:33:15.156615) v4 -- 0x55a33610ca00 con 0 > -7540> 2018-07-11 07:33:15.156827 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6803/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:33:15.156615) v4 -- 0x55a33610d000 con 0 > -7539> 2018-07-11 07:33:15.156840 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6806/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:33:15.156615) v4 -- 0x55a33610de00 con 0 > -7538> 2018-07-11 07:33:15.156853 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6807/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:33:15.156615) v4 -- 0x55a33610e400 con 0 > -7537> 2018-07-11 07:33:15.156866 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6802/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:33:15.156615) v4 -- 0x55a33610dc00 con 0 > -7536> 2018-07-11 07:33:15.156902 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6803/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:33:15.156615) v4 -- 0x55a33610e200 con 0 > -7535> 2018-07-11 07:33:15.157031 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6806/12617 conn(0x55a314e26000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1827 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:15.156615) v4 > -7534> 2018-07-11 07:33:15.157053 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6806/12617 1827 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:15.156615) v4 ==== 2004+0+0 (557056153 0 0) 0x55a332e79a00 con 0x55a314e26000 > -7533> 2018-07-11 07:33:15.157059 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6807/12617 conn(0x55a314e27800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1827 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:15.156615) v4 > -7532> 2018-07-11 07:33:15.157080 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6807/12617 1827 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:15.156615) v4 ==== 2004+0+0 (557056153 0 0) 0x55a34de6f000 con 0x55a314e27800 > -7531> 2018-07-11 07:33:15.157083 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6802/15255 conn(0x55a314e1c000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1827 0x55a325c0e000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:15.156615) v4 > -7530> 2018-07-11 07:33:15.157101 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6802/15255 1827 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:15.156615) v4 ==== 2004+0+0 (557056153 0 0) 0x55a325c0e000 con 0x55a314e1c000 > -7529> 2018-07-11 07:33:15.157108 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6803/15255 conn(0x55a314e1d800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1827 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:15.156615) v4 > -7528> 2018-07-11 07:33:15.157147 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6809/12650 conn(0x55a314e1f000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1827 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:15.156615) v4 > -7527> 2018-07-11 07:33:15.157149 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6803/15255 1827 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:15.156615) v4 ==== 2004+0+0 (557056153 0 0) 0x55a332e79a00 con 0x55a314e1d800 > -7526> 2018-07-11 07:33:15.157162 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6809/12650 1827 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:15.156615) v4 ==== 2004+0+0 (557056153 0 0) 0x55a34de6f000 con 0x55a314e1f000 > -7525> 2018-07-11 07:33:15.157182 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6807/12677 conn(0x55a314e85800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1827 0x55a325c0e000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:15.156615) v4 > -7524> 2018-07-11 07:33:15.157203 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6807/12677 1827 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:15.156615) v4 ==== 2004+0+0 (557056153 0 0) 0x55a325c0e000 con 0x55a314e85800 > -7523> 2018-07-11 07:33:15.157197 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6810/12651 conn(0x55a314e97800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1827 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:15.156615) v4 > -7522> 2018-07-11 07:33:15.157221 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6810/12651 1827 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:15.156615) v4 ==== 2004+0+0 (557056153 0 0) 0x55a332e79a00 con 0x55a314e97800 > -7521> 2018-07-11 07:33:15.157229 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6806/12677 conn(0x55a314e84000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1827 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:15.156615) v4 > -7520> 2018-07-11 07:33:15.157266 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6806/12677 1827 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:15.156615) v4 ==== 2004+0+0 (557056153 0 0) 0x55a34de6f000 con 0x55a314e84000 > -7519> 2018-07-11 07:33:15.157269 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6808/12651 conn(0x55a314e96000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1827 0x55a325c0e000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:15.156615) v4 > -7518> 2018-07-11 07:33:15.157282 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6808/12651 1827 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:15.156615) v4 ==== 2004+0+0 (557056153 0 0) 0x55a325c0e000 con 0x55a314e96000 > -7517> 2018-07-11 07:33:15.157315 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6810/15287 conn(0x55a314e03000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1827 0x55a325c0e000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:15.156615) v4 > -7516> 2018-07-11 07:33:15.157330 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6810/15287 1827 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:15.156615) v4 ==== 2004+0+0 (557056153 0 0) 0x55a325c0e000 con 0x55a314e03000 > -7515> 2018-07-11 07:33:15.157342 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6811/15287 conn(0x55a314e04800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1827 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:15.156615) v4 > -7514> 2018-07-11 07:33:15.157358 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6811/15287 1827 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:15.156615) v4 ==== 2004+0+0 (557056153 0 0) 0x55a332e79a00 con 0x55a314e04800 > -7513> 2018-07-11 07:33:15.157362 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6811/12650 conn(0x55a314e20800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1827 0x55a325c0e000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:15.156615) v4 > -7512> 2018-07-11 07:33:15.157374 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6811/12650 1827 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:15.156615) v4 ==== 2004+0+0 (557056153 0 0) 0x55a325c0e000 con 0x55a314e20800 > -7511> 2018-07-11 07:33:15.157368 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6803/13323 conn(0x55a314e16800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1827 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:15.156615) v4 > -7510> 2018-07-11 07:33:15.157389 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6803/13323 1827 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:15.156615) v4 ==== 2004+0+0 (557056153 0 0) 0x55a34de6f000 con 0x55a314e16800 > -7509> 2018-07-11 07:33:15.157401 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6806/15885 conn(0x55a314e00000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1827 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:15.156615) v4 > -7508> 2018-07-11 07:33:15.157404 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6807/15263 conn(0x55a314e9a800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1827 0x55a325c0e000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:15.156615) v4 > -7507> 2018-07-11 07:33:15.157414 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6806/15885 1827 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:15.156615) v4 ==== 2004+0+0 (557056153 0 0) 0x55a332e79a00 con 0x55a314e00000 > -7506> 2018-07-11 07:33:15.157415 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6807/15263 1827 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:15.156615) v4 ==== 2004+0+0 (557056153 0 0) 0x55a325c0e000 con 0x55a314e9a800 > -7505> 2018-07-11 07:33:15.157443 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6806/15263 conn(0x55a314e99000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1827 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:15.156615) v4 > -7504> 2018-07-11 07:33:15.157452 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6802/13323 conn(0x55a314e15000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1827 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:15.156615) v4 > -7503> 2018-07-11 07:33:15.157455 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6803/11967 conn(0x55a314e13800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1827 0x55a325c0e000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:15.156615) v4 > -7502> 2018-07-11 07:33:15.157457 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6806/15263 1827 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:15.156615) v4 ==== 2004+0+0 (557056153 0 0) 0x55a34de6f000 con 0x55a314e99000 > -7501> 2018-07-11 07:33:15.157462 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6802/13323 1827 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:15.156615) v4 ==== 2004+0+0 (557056153 0 0) 0x55a332e79a00 con 0x55a314e15000 > -7500> 2018-07-11 07:33:15.157465 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6803/11967 1827 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:15.156615) v4 ==== 2004+0+0 (557056153 0 0) 0x55a325c0e000 con 0x55a314e13800 > -7499> 2018-07-11 07:33:15.157492 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6802/15919 conn(0x55a314e87000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1827 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:15.156615) v4 > -7498> 2018-07-11 07:33:15.157504 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6802/15919 1827 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:15.156615) v4 ==== 2004+0+0 (557056153 0 0) 0x55a332e79a00 con 0x55a314e87000 > -7497> 2018-07-11 07:33:15.157503 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6807/15885 conn(0x55a314e01800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1827 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:15.156615) v4 > -7496> 2018-07-11 07:33:15.157522 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6807/15885 1827 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:15.156615) v4 ==== 2004+0+0 (557056153 0 0) 0x55a34de6f000 con 0x55a314e01800 > -7495> 2018-07-11 07:33:15.157559 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6802/11967 conn(0x55a314e12000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1827 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:15.156615) v4 > -7494> 2018-07-11 07:33:15.157573 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6802/11967 1827 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:15.156615) v4 ==== 2004+0+0 (557056153 0 0) 0x55a34de6f000 con 0x55a314e12000 > -7493> 2018-07-11 07:33:15.157613 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6803/15919 conn(0x55a314e88800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1827 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:15.156615) v4 > -7492> 2018-07-11 07:33:15.157627 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6803/15919 1827 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:15.156615) v4 ==== 2004+0+0 (557056153 0 0) 0x55a34de6f000 con 0x55a314e88800 > -7491> 2018-07-11 07:33:15.821745 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a332dd4b40 con 0 > -7490> 2018-07-11 07:33:15.836855 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5064 0x55a327d45180 replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:494/492,start:6:a9b136fe:::benchmark_data_magna108_35617_object841600:0,end:6:a9b25cda:::benchmark_data_magna108.ceph.redhat.com_2478_object288024:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -7489> 2018-07-11 07:33:15.836880 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5064 ==== replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:494/492,start:6:a9b136fe:::benchmark_data_magna108_35617_object841600:0,end:6:a9b25cda:::benchmark_data_magna108.ceph.redhat.com_2478_object288024:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (1554847791 0 0) 0x55a327d45180 con 0x55a317019000 > -7488> 2018-07-11 07:33:16.122752 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15287 conn(0x55a311b75000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1822 0x55a325c0e000 osd_ping(ping e494 stamp 2018-07-11 07:33:17.335760) v4 > -7487> 2018-07-11 07:33:16.122767 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15287 conn(0x55a311b72000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1822 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:33:17.335760) v4 > -7486> 2018-07-11 07:33:16.122781 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.4 10.8.128.113:0/15287 1822 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:17.335760) v4 ==== 2004+0+0 (3594040659 0 0) 0x55a325c0e000 con 0x55a311b75000 > -7485> 2018-07-11 07:33:16.122785 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.4 10.8.128.113:0/15287 1822 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:17.335760) v4 ==== 2004+0+0 (3594040659 0 0) 0x55a332e79a00 con 0x55a311b72000 > -7484> 2018-07-11 07:33:16.122796 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:17.335760) v4 -- 0x55a32316da00 con 0 > -7483> 2018-07-11 07:33:16.122821 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:17.335760) v4 -- 0x55a34866a600 con 0 > -7482> 2018-07-11 07:33:16.160196 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15885 conn(0x55a311b76800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1847 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:33:16.783748) v4 > -7481> 2018-07-11 07:33:16.160200 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15885 conn(0x55a3170b5800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1847 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:33:16.783748) v4 > -7480> 2018-07-11 07:33:16.160217 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.7 10.8.128.114:0/15885 1847 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:16.783748) v4 ==== 2004+0+0 (3811946604 0 0) 0x55a34de6f000 con 0x55a311b76800 > -7479> 2018-07-11 07:33:16.160219 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.7 10.8.128.114:0/15885 1847 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:16.783748) v4 ==== 2004+0+0 (3811946604 0 0) 0x55a324474c00 con 0x55a3170b5800 > -7478> 2018-07-11 07:33:16.160237 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:16.783748) v4 -- 0x55a343482800 con 0 > -7477> 2018-07-11 07:33:16.160290 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:16.783748) v4 -- 0x55a34866a600 con 0 > -7476> 2018-07-11 07:33:16.263421 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4800000 on bdev 1, free 0x2fe000; fallback to bdev 2 > -7475> 2018-07-11 07:33:16.311759 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15263 conn(0x55a321664000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1825 0x55a329ac3a00 osd_ping(ping e494 stamp 2018-07-11 07:33:17.524713) v4 > -7474> 2018-07-11 07:33:16.311783 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.5 10.8.128.113:0/15263 1825 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:17.524713) v4 ==== 2004+0+0 (3882870818 0 0) 0x55a329ac3a00 con 0x55a321664000 > -7473> 2018-07-11 07:33:16.311796 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:17.524713) v4 -- 0x55a32316da00 con 0 > -7472> 2018-07-11 07:33:16.311875 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15263 conn(0x55a321667000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1825 0x55a325c0e000 osd_ping(ping e494 stamp 2018-07-11 07:33:17.524713) v4 > -7471> 2018-07-11 07:33:16.311890 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.5 10.8.128.113:0/15263 1825 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:17.524713) v4 ==== 2004+0+0 (3882870818 0 0) 0x55a325c0e000 con 0x55a321667000 > -7470> 2018-07-11 07:33:16.311904 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:17.524713) v4 -- 0x55a3342d7200 con 0 > -7469> 2018-07-11 07:33:16.471086 7f355d88e700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.107:0/12617 conn(0x55a32447f800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1801 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:33:16.470748) v4 > -7468> 2018-07-11 07:33:16.471106 7f355d88e700 1 -- 10.8.128.107:6803/12665 <== osd.9 10.8.128.107:0/12617 1801 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:16.470748) v4 ==== 2004+0+0 (1828267233 0 0) 0x55a34de78600 con 0x55a32447f800 > -7467> 2018-07-11 07:33:16.471115 7f355d88e700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:16.470748) v4 -- 0x55a32eb21600 con 0 > -7466> 2018-07-11 07:33:16.471108 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.107:0/12617 conn(0x55a32447e000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1801 0x55a329ac3a00 osd_ping(ping e494 stamp 2018-07-11 07:33:16.470748) v4 > -7465> 2018-07-11 07:33:16.471129 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.9 10.8.128.107:0/12617 1801 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:16.470748) v4 ==== 2004+0+0 (1828267233 0 0) 0x55a329ac3a00 con 0x55a32447e000 > -7464> 2018-07-11 07:33:16.471142 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:16.470748) v4 -- 0x55a31dbc3e00 con 0 > -7463> 2018-07-11 07:33:16.743941 7f35498a5700 10 monclient: tick > -7462> 2018-07-11 07:33:16.743955 7f35498a5700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2018-07-11 07:32:46.743953) > -7461> 2018-07-11 07:33:17.223258 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15287 conn(0x55a311b72000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1823 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:33:18.436269) v4 > -7460> 2018-07-11 07:33:17.223262 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15287 conn(0x55a311b75000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1823 0x55a329ac3a00 osd_ping(ping e494 stamp 2018-07-11 07:33:18.436269) v4 > -7459> 2018-07-11 07:33:17.223288 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.4 10.8.128.113:0/15287 1823 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:18.436269) v4 ==== 2004+0+0 (2865797304 0 0) 0x55a329ac3a00 con 0x55a311b75000 > -7458> 2018-07-11 07:33:17.223284 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.4 10.8.128.113:0/15287 1823 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:18.436269) v4 ==== 2004+0+0 (2865797304 0 0) 0x55a332e79a00 con 0x55a311b72000 > -7457> 2018-07-11 07:33:17.223302 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:18.436269) v4 -- 0x55a333f6e200 con 0 > -7456> 2018-07-11 07:33:17.223327 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:18.436269) v4 -- 0x55a3131bb000 con 0 > -7455> 2018-07-11 07:33:17.251379 7f35488a3700 1 -- 10.8.128.107:6800/12665 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55a32a5c4000 con 0 > -7454> 2018-07-11 07:33:17.251446 7f35488a3700 1 -- 10.8.128.107:6800/12665 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55a32aa0d600 con 0 > -7453> 2018-07-11 07:33:17.405380 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 58] Generated table #36573: 2638241 keys, 74545787 bytes > -7452> 2018-07-11 07:33:17.405423 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294397405396, "cf_name": "default", "job": 58, "event": "table_file_creation", "file_number": 36573, "file_size": 74545787, "table_properties": {"data_size": 67112158, "index_size": 689456, "filter_size": 6743183, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} > -7451> 2018-07-11 07:33:17.457100 7f3543899700 5 osd.10 494 heartbeat: osd_stat(783 GB used, 147 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,0,0,0,1]) > -7450> 2018-07-11 07:33:17.457115 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6802/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:33:17.457113) v4 -- 0x55a343483400 con 0 > -7449> 2018-07-11 07:33:17.457154 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6803/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:33:17.457113) v4 -- 0x55a3331d8e00 con 0 > -7448> 2018-07-11 07:33:17.457170 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6806/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:33:17.457113) v4 -- 0x55a3331d9000 con 0 > -7447> 2018-07-11 07:33:17.457184 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6807/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:33:17.457113) v4 -- 0x55a3331d9400 con 0 > -7446> 2018-07-11 07:33:17.457209 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6808/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:33:17.457113) v4 -- 0x55a329ac2200 con 0 > -7445> 2018-07-11 07:33:17.457218 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6810/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:33:17.457113) v4 -- 0x55a3331d9800 con 0 > -7444> 2018-07-11 07:33:17.457228 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6809/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:33:17.457113) v4 -- 0x55a329ac2800 con 0 > -7443> 2018-07-11 07:33:17.457242 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6811/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:33:17.457113) v4 -- 0x55a329ac2000 con 0 > -7442> 2018-07-11 07:33:17.457275 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6810/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:33:17.457113) v4 -- 0x55a3331d9600 con 0 > -7441> 2018-07-11 07:33:17.457287 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6811/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:33:17.457113) v4 -- 0x55a329ac2c00 con 0 > -7440> 2018-07-11 07:33:17.457299 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6806/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:33:17.457113) v4 -- 0x55a329ac2600 con 0 > -7439> 2018-07-11 07:33:17.457314 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6807/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:33:17.457113) v4 -- 0x55a3331d9c00 con 0 > -7438> 2018-07-11 07:33:17.457337 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6802/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:33:17.457113) v4 -- 0x55a324967000 con 0 > -7437> 2018-07-11 07:33:17.457351 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6803/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:33:17.457113) v4 -- 0x55a31e8f5600 con 0 > -7436> 2018-07-11 07:33:17.457374 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6806/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:33:17.457113) v4 -- 0x55a33610f200 con 0 > -7435> 2018-07-11 07:33:17.457388 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6807/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:33:17.457113) v4 -- 0x55a324966c00 con 0 > -7434> 2018-07-11 07:33:17.457402 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6802/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:33:17.457113) v4 -- 0x55a33dff0000 con 0 > -7433> 2018-07-11 07:33:17.457417 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6803/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:33:17.457113) v4 -- 0x55a32447ac00 con 0 > -7432> 2018-07-11 07:33:17.457430 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6806/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:33:17.457113) v4 -- 0x55a329ac3c00 con 0 > -7431> 2018-07-11 07:33:17.457443 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6807/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:33:17.457113) v4 -- 0x55a33610fa00 con 0 > -7430> 2018-07-11 07:33:17.457485 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6802/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:33:17.457113) v4 -- 0x55a33610e600 con 0 > -7429> 2018-07-11 07:33:17.457505 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6803/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:33:17.457113) v4 -- 0x55a33610e800 con 0 > -7428> 2018-07-11 07:33:17.457629 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6806/12617 conn(0x55a314e26000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1828 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:17.457113) v4 > -7427> 2018-07-11 07:33:17.457651 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6806/12617 1828 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:17.457113) v4 ==== 2004+0+0 (3922291873 0 0) 0x55a324474c00 con 0x55a314e26000 > -7426> 2018-07-11 07:33:17.457666 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6807/12617 conn(0x55a314e27800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1828 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:17.457113) v4 > -7425> 2018-07-11 07:33:17.457685 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6807/12617 1828 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:17.457113) v4 ==== 2004+0+0 (3922291873 0 0) 0x55a34de6f000 con 0x55a314e27800 > -7424> 2018-07-11 07:33:17.457714 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6810/12651 conn(0x55a314e97800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1828 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:17.457113) v4 > -7423> 2018-07-11 07:33:17.457712 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6802/15255 conn(0x55a314e1c000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1828 0x55a325c0e000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:17.457113) v4 > -7422> 2018-07-11 07:33:17.457729 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6810/12651 1828 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:17.457113) v4 ==== 2004+0+0 (3922291873 0 0) 0x55a324474c00 con 0x55a314e97800 > -7421> 2018-07-11 07:33:17.457734 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6802/15255 1828 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:17.457113) v4 ==== 2004+0+0 (3922291873 0 0) 0x55a325c0e000 con 0x55a314e1c000 > -7420> 2018-07-11 07:33:17.457731 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6806/12677 conn(0x55a314e84000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1828 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:17.457113) v4 > -7419> 2018-07-11 07:33:17.457752 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6806/12677 1828 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:17.457113) v4 ==== 2004+0+0 (3922291873 0 0) 0x55a34de6f000 con 0x55a314e84000 > -7418> 2018-07-11 07:33:17.457750 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6803/15255 conn(0x55a314e1d800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1828 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:17.457113) v4 > -7417> 2018-07-11 07:33:17.457776 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6807/12677 conn(0x55a314e85800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1828 0x55a325c0e000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:17.457113) v4 > -7416> 2018-07-11 07:33:17.457768 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6803/15255 1828 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:17.457113) v4 ==== 2004+0+0 (3922291873 0 0) 0x55a324474c00 con 0x55a314e1d800 > -7415> 2018-07-11 07:33:17.457790 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6807/12677 1828 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:17.457113) v4 ==== 2004+0+0 (3922291873 0 0) 0x55a325c0e000 con 0x55a314e85800 > -7414> 2018-07-11 07:33:17.457791 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6809/12650 conn(0x55a314e1f000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1828 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:17.457113) v4 > -7413> 2018-07-11 07:33:17.457809 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6809/12650 1828 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:17.457113) v4 ==== 2004+0+0 (3922291873 0 0) 0x55a34de6f000 con 0x55a314e1f000 > -7412> 2018-07-11 07:33:17.457815 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6811/15287 conn(0x55a314e04800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1828 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:17.457113) v4 > -7411> 2018-07-11 07:33:17.457832 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6811/15287 1828 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:17.457113) v4 ==== 2004+0+0 (3922291873 0 0) 0x55a324474c00 con 0x55a314e04800 > -7410> 2018-07-11 07:33:17.457840 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6808/12651 conn(0x55a314e96000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1828 0x55a325c0e000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:17.457113) v4 > -7409> 2018-07-11 07:33:17.457847 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6806/15263 conn(0x55a314e99000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1828 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:17.457113) v4 > -7408> 2018-07-11 07:33:17.457859 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6806/15263 1828 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:17.457113) v4 ==== 2004+0+0 (3922291873 0 0) 0x55a34de6f000 con 0x55a314e99000 > -7407> 2018-07-11 07:33:17.457864 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6806/15885 conn(0x55a314e00000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1828 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:17.457113) v4 > -7406> 2018-07-11 07:33:17.457855 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6808/12651 1828 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:17.457113) v4 ==== 2004+0+0 (3922291873 0 0) 0x55a325c0e000 con 0x55a314e96000 > -7405> 2018-07-11 07:33:17.457878 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6806/15885 1828 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:17.457113) v4 ==== 2004+0+0 (3922291873 0 0) 0x55a324474c00 con 0x55a314e00000 > -7404> 2018-07-11 07:33:17.457924 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6810/15287 conn(0x55a314e03000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1828 0x55a325c0e000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:17.457113) v4 > -7403> 2018-07-11 07:33:17.457931 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6802/13323 conn(0x55a314e15000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1828 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:17.457113) v4 > -7402> 2018-07-11 07:33:17.457934 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6803/13323 conn(0x55a314e16800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1828 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:17.457113) v4 > -7401> 2018-07-11 07:33:17.457940 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6810/15287 1828 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:17.457113) v4 ==== 2004+0+0 (3922291873 0 0) 0x55a325c0e000 con 0x55a314e03000 > -7400> 2018-07-11 07:33:17.457948 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6803/13323 1828 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:17.457113) v4 ==== 2004+0+0 (3922291873 0 0) 0x55a34de6f000 con 0x55a314e16800 > -7399> 2018-07-11 07:33:17.457947 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6802/13323 1828 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:17.457113) v4 ==== 2004+0+0 (3922291873 0 0) 0x55a324474c00 con 0x55a314e15000 > -7398> 2018-07-11 07:33:17.457985 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6802/15919 conn(0x55a314e87000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1828 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:17.457113) v4 > -7397> 2018-07-11 07:33:17.457986 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6802/11967 conn(0x55a314e12000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1828 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:17.457113) v4 > -7396> 2018-07-11 07:33:17.457993 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6811/12650 conn(0x55a314e20800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1828 0x55a325c0e000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:17.457113) v4 > -7395> 2018-07-11 07:33:17.458000 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6802/15919 1828 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:17.457113) v4 ==== 2004+0+0 (3922291873 0 0) 0x55a324474c00 con 0x55a314e87000 > -7394> 2018-07-11 07:33:17.458005 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6811/12650 1828 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:17.457113) v4 ==== 2004+0+0 (3922291873 0 0) 0x55a325c0e000 con 0x55a314e20800 > -7393> 2018-07-11 07:33:17.458005 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6802/11967 1828 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:17.457113) v4 ==== 2004+0+0 (3922291873 0 0) 0x55a34de6f000 con 0x55a314e12000 > -7392> 2018-07-11 07:33:17.458038 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6807/15263 conn(0x55a314e9a800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1828 0x55a325c0e000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:17.457113) v4 > -7391> 2018-07-11 07:33:17.458050 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6807/15263 1828 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:17.457113) v4 ==== 2004+0+0 (3922291873 0 0) 0x55a325c0e000 con 0x55a314e9a800 > -7390> 2018-07-11 07:33:17.458048 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6803/15919 conn(0x55a314e88800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1828 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:17.457113) v4 > -7389> 2018-07-11 07:33:17.458060 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6803/15919 1828 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:17.457113) v4 ==== 2004+0+0 (3922291873 0 0) 0x55a34de6f000 con 0x55a314e88800 > -7388> 2018-07-11 07:33:17.458084 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6803/11967 conn(0x55a314e13800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1828 0x55a325c0e000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:17.457113) v4 > -7387> 2018-07-11 07:33:17.458092 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6807/15885 conn(0x55a314e01800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1828 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:17.457113) v4 > -7386> 2018-07-11 07:33:17.458098 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6803/11967 1828 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:17.457113) v4 ==== 2004+0+0 (3922291873 0 0) 0x55a325c0e000 con 0x55a314e13800 > -7385> 2018-07-11 07:33:17.458105 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6807/15885 1828 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:17.457113) v4 ==== 2004+0+0 (3922291873 0 0) 0x55a34de6f000 con 0x55a314e01800 > -7384> 2018-07-11 07:33:17.550848 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15919 conn(0x55a317074800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1832 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:33:18.174392) v4 > -7383> 2018-07-11 07:33:17.550874 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.11 10.8.128.114:0/15919 1832 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:18.174392) v4 ==== 2004+0+0 (4265476453 0 0) 0x55a324474c00 con 0x55a317074800 > -7382> 2018-07-11 07:33:17.550890 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:18.174392) v4 -- 0x55a3331d8e00 con 0 > -7381> 2018-07-11 07:33:17.550923 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15919 conn(0x55a3170b4000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1832 0x55a325c0e000 osd_ping(ping e494 stamp 2018-07-11 07:33:18.174392) v4 > -7380> 2018-07-11 07:33:17.550948 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.11 10.8.128.114:0/15919 1832 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:18.174392) v4 ==== 2004+0+0 (4265476453 0 0) 0x55a325c0e000 con 0x55a3170b4000 > -7379> 2018-07-11 07:33:17.550964 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:18.174392) v4 -- 0x55a3331d9400 con 0 > -7378> 2018-07-11 07:33:17.662959 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a33b621680 con 0 > -7377> 2018-07-11 07:33:17.679486 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5065 0x55a327d45400 replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:494/492,start:6:a9b25cda:::benchmark_data_magna108.ceph.redhat.com_2478_object288024:0,end:6:a9b31f48:::benchmark_data_magna108_35617_object193126:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -7376> 2018-07-11 07:33:17.679510 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5065 ==== replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:494/492,start:6:a9b25cda:::benchmark_data_magna108.ceph.redhat.com_2478_object288024:0,end:6:a9b31f48:::benchmark_data_magna108_35617_object193126:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (802143956 0 0) 0x55a327d45400 con 0x55a317019000 > -7375> 2018-07-11 07:33:17.870352 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12650 conn(0x55a311b67000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1844 0x55a329ac3a00 osd_ping(ping e494 stamp 2018-07-11 07:33:17.905523) v4 > -7374> 2018-07-11 07:33:17.870380 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.3 10.8.128.106:0/12650 1844 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:17.905523) v4 ==== 2004+0+0 (325281160 0 0) 0x55a329ac3a00 con 0x55a311b67000 > -7373> 2018-07-11 07:33:17.870384 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12650 conn(0x55a311b68800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1844 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:33:17.905523) v4 > -7372> 2018-07-11 07:33:17.870398 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:17.905523) v4 -- 0x55a3331d9400 con 0 > -7371> 2018-07-11 07:33:17.870400 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.3 10.8.128.106:0/12650 1844 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:17.905523) v4 ==== 2004+0+0 (325281160 0 0) 0x55a34de6f000 con 0x55a311b68800 > -7370> 2018-07-11 07:33:17.870431 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:17.905523) v4 -- 0x55a329ac2800 con 0 > -7369> 2018-07-11 07:33:17.922319 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12677 conn(0x55a32169d800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1849 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:33:17.957506) v4 > -7368> 2018-07-11 07:33:17.922341 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.1 10.8.128.106:0/12677 1849 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:17.957506) v4 ==== 2004+0+0 (2139090296 0 0) 0x55a34de78600 con 0x55a32169d800 > -7367> 2018-07-11 07:33:17.922336 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12677 conn(0x55a32169c000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1849 0x55a325c0e000 osd_ping(ping e494 stamp 2018-07-11 07:33:17.957506) v4 > -7366> 2018-07-11 07:33:17.922350 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:17.957506) v4 -- 0x55a329ac2800 con 0 > -7365> 2018-07-11 07:33:17.922353 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.1 10.8.128.106:0/12677 1849 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:17.957506) v4 ==== 2004+0+0 (2139090296 0 0) 0x55a325c0e000 con 0x55a32169c000 > -7364> 2018-07-11 07:33:17.922372 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:17.957506) v4 -- 0x55a3331d9400 con 0 > -7363> 2018-07-11 07:33:18.537192 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a317b326c0 con 0 > -7362> 2018-07-11 07:33:18.552807 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5066 0x55a327d45680 replica scrub(pg: 6.15,from:0'0,to:430'106261,epoch:494/492,start:6:a9b31f48:::benchmark_data_magna108_35617_object193126:0,end:6:a9b3de91:::benchmark_data_magna108_35617_object52910:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -7361> 2018-07-11 07:33:18.552831 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5066 ==== replica scrub(pg: 6.15,from:0'0,to:430'106261,epoch:494/492,start:6:a9b31f48:::benchmark_data_magna108_35617_object193126:0,end:6:a9b3de91:::benchmark_data_magna108_35617_object52910:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 217+0+0 (647463512 0 0) 0x55a327d45680 con 0x55a317019000 > -7360> 2018-07-11 07:33:18.922423 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15255 conn(0x55a311b64000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1854 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:33:20.135483) v4 > -7359> 2018-07-11 07:33:18.922426 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15255 conn(0x55a311b65800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1854 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:33:20.135483) v4 > -7358> 2018-07-11 07:33:18.922458 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.0 10.8.128.113:0/15255 1854 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:20.135483) v4 ==== 2004+0+0 (1706581476 0 0) 0x55a34de6f000 con 0x55a311b64000 > -7357> 2018-07-11 07:33:18.922460 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.0 10.8.128.113:0/15255 1854 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:20.135483) v4 ==== 2004+0+0 (1706581476 0 0) 0x55a332e79a00 con 0x55a311b65800 > -7356> 2018-07-11 07:33:18.922480 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:20.135483) v4 -- 0x55a3331d9000 con 0 > -7355> 2018-07-11 07:33:18.922518 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:20.135483) v4 -- 0x55a3131bb000 con 0 > -7354> 2018-07-11 07:33:18.943865 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b7000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1779 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:33:18.694219) v4 > -7353> 2018-07-11 07:33:18.943891 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.8 10.8.128.108:0/11967 1779 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:18.694219) v4 ==== 2004+0+0 (3956645962 0 0) 0x55a34de78600 con 0x55a3170b7000 > -7352> 2018-07-11 07:33:18.943887 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b8800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1779 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:33:18.694219) v4 > -7351> 2018-07-11 07:33:18.943911 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:18.694219) v4 -- 0x55a32eb21600 con 0 > -7350> 2018-07-11 07:33:18.943916 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.8 10.8.128.108:0/11967 1779 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:18.694219) v4 ==== 2004+0+0 (3956645962 0 0) 0x55a324474c00 con 0x55a3170b8800 > -7349> 2018-07-11 07:33:18.943940 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:18.694219) v4 -- 0x55a33047e800 con 0 > -7348> 2018-07-11 07:33:19.022614 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12677 conn(0x55a32169d800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1850 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:33:19.057855) v4 > -7347> 2018-07-11 07:33:19.022641 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.1 10.8.128.106:0/12677 1850 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:19.057855) v4 ==== 2004+0+0 (3408130050 0 0) 0x55a34de6f000 con 0x55a32169d800 > -7346> 2018-07-11 07:33:19.022657 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:19.057855) v4 -- 0x55a34de79c00 con 0 > -7345> 2018-07-11 07:33:19.022678 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12677 conn(0x55a32169c000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1850 0x55a329ac3a00 osd_ping(ping e494 stamp 2018-07-11 07:33:19.057855) v4 > -7344> 2018-07-11 07:33:19.022699 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.1 10.8.128.106:0/12677 1850 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:19.057855) v4 ==== 2004+0+0 (3408130050 0 0) 0x55a329ac3a00 con 0x55a32169c000 > -7343> 2018-07-11 07:33:19.022714 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:19.057855) v4 -- 0x55a333f6e200 con 0 > -7342> 2018-07-11 07:33:19.042834 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.115:0/13323 conn(0x55a321668800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1840 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:33:19.379906) v4 > -7341> 2018-07-11 07:33:19.042855 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.6 10.8.128.115:0/13323 1840 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:19.379906) v4 ==== 2004+0+0 (3776748014 0 0) 0x55a332e79a00 con 0x55a321668800 > -7340> 2018-07-11 07:33:19.042867 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:19.379906) v4 -- 0x55a34866b600 con 0 > -7339> 2018-07-11 07:33:19.042898 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.115:0/13323 conn(0x55a321665800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1840 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:33:19.379906) v4 > -7338> 2018-07-11 07:33:19.042917 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.6 10.8.128.115:0/13323 1840 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:19.379906) v4 ==== 2004+0+0 (3776748014 0 0) 0x55a332e79a00 con 0x55a321665800 > -7337> 2018-07-11 07:33:19.042931 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:19.379906) v4 -- 0x55a33047e600 con 0 > -7336> 2018-07-11 07:33:19.157654 7f3543899700 5 osd.10 494 heartbeat: osd_stat(783 GB used, 147 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,0,0,1]) > -7335> 2018-07-11 07:33:19.157670 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6802/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:33:19.157668) v4 -- 0x55a33610f000 con 0 > -7334> 2018-07-11 07:33:19.157708 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6803/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:33:19.157668) v4 -- 0x55a33610ec00 con 0 > -7333> 2018-07-11 07:33:19.157723 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6806/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:33:19.157668) v4 -- 0x55a33610f800 con 0 > -7332> 2018-07-11 07:33:19.157739 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6807/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:33:19.157668) v4 -- 0x55a33610fe00 con 0 > -7331> 2018-07-11 07:33:19.157752 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6808/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:33:19.157668) v4 -- 0x55a312362000 con 0 > -7330> 2018-07-11 07:33:19.157762 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6810/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:33:19.157668) v4 -- 0x55a33610ee00 con 0 > -7329> 2018-07-11 07:33:19.157773 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6809/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:33:19.157668) v4 -- 0x55a312362400 con 0 > -7328> 2018-07-11 07:33:19.157787 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6811/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:33:19.157668) v4 -- 0x55a33610f400 con 0 > -7327> 2018-07-11 07:33:19.157800 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6810/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:33:19.157668) v4 -- 0x55a33610f600 con 0 > -7326> 2018-07-11 07:33:19.157810 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6811/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:33:19.157668) v4 -- 0x55a312362200 con 0 > -7325> 2018-07-11 07:33:19.157823 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6806/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:33:19.157668) v4 -- 0x55a312362600 con 0 > -7324> 2018-07-11 07:33:19.157836 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6807/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:33:19.157668) v4 -- 0x55a312362a00 con 0 > -7323> 2018-07-11 07:33:19.157847 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6802/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:33:19.157668) v4 -- 0x55a33610fc00 con 0 > -7322> 2018-07-11 07:33:19.157859 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6803/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:33:19.157668) v4 -- 0x55a312362c00 con 0 > -7321> 2018-07-11 07:33:19.157872 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6806/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:33:19.157668) v4 -- 0x55a312362800 con 0 > -7320> 2018-07-11 07:33:19.157890 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6807/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:33:19.157668) v4 -- 0x55a312363000 con 0 > -7319> 2018-07-11 07:33:19.157903 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6802/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:33:19.157668) v4 -- 0x55a312362e00 con 0 > -7318> 2018-07-11 07:33:19.157915 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6803/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:33:19.157668) v4 -- 0x55a30f329e00 con 0 > -7317> 2018-07-11 07:33:19.157936 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6806/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:33:19.157668) v4 -- 0x55a33b42c000 con 0 > -7316> 2018-07-11 07:33:19.157950 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6807/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:33:19.157668) v4 -- 0x55a343482a00 con 0 > -7315> 2018-07-11 07:33:19.157964 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6802/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:33:19.157668) v4 -- 0x55a343482200 con 0 > -7314> 2018-07-11 07:33:19.157976 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6803/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:33:19.157668) v4 -- 0x55a33610c400 con 0 > -7313> 2018-07-11 07:33:19.158134 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6806/12617 conn(0x55a314e26000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1829 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:19.157668) v4 > -7312> 2018-07-11 07:33:19.158140 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6807/12617 conn(0x55a314e27800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1829 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:19.157668) v4 > -7311> 2018-07-11 07:33:19.158160 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6807/12617 1829 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:19.157668) v4 ==== 2004+0+0 (2890697417 0 0) 0x55a34de78600 con 0x55a314e27800 > -7310> 2018-07-11 07:33:19.158160 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6806/12617 1829 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:19.157668) v4 ==== 2004+0+0 (2890697417 0 0) 0x55a332e79a00 con 0x55a314e26000 > -7309> 2018-07-11 07:33:19.158201 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6802/15255 conn(0x55a314e1c000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1829 0x55a3331d9400 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:19.157668) v4 > -7308> 2018-07-11 07:33:19.158224 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6802/15255 1829 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:19.157668) v4 ==== 2004+0+0 (2890697417 0 0) 0x55a3331d9400 con 0x55a314e1c000 > -7307> 2018-07-11 07:33:19.158231 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6806/15263 conn(0x55a314e99000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1829 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:19.157668) v4 > -7306> 2018-07-11 07:33:19.158269 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6810/12651 conn(0x55a314e97800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1829 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:19.157668) v4 > -7305> 2018-07-11 07:33:19.158287 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6810/12651 1829 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:19.157668) v4 ==== 2004+0+0 (2890697417 0 0) 0x55a332e79a00 con 0x55a314e97800 > -7304> 2018-07-11 07:33:19.158265 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6806/15263 1829 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:19.157668) v4 ==== 2004+0+0 (2890697417 0 0) 0x55a34de78600 con 0x55a314e99000 > -7303> 2018-07-11 07:33:19.158322 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6811/15287 conn(0x55a314e04800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1829 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:19.157668) v4 > -7302> 2018-07-11 07:33:19.158334 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6811/15287 1829 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:19.157668) v4 ==== 2004+0+0 (2890697417 0 0) 0x55a332e79a00 con 0x55a314e04800 > -7301> 2018-07-11 07:33:19.158330 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6810/15287 conn(0x55a314e03000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1829 0x55a3331d9400 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:19.157668) v4 > -7300> 2018-07-11 07:33:19.158343 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6806/12677 conn(0x55a314e84000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1829 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:19.157668) v4 > -7299> 2018-07-11 07:33:19.158352 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6810/15287 1829 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:19.157668) v4 ==== 2004+0+0 (2890697417 0 0) 0x55a3331d9400 con 0x55a314e03000 > -7298> 2018-07-11 07:33:19.158359 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6806/12677 1829 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:19.157668) v4 ==== 2004+0+0 (2890697417 0 0) 0x55a34de78600 con 0x55a314e84000 > -7297> 2018-07-11 07:33:19.158376 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6803/15255 conn(0x55a314e1d800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1829 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:19.157668) v4 > -7296> 2018-07-11 07:33:19.158389 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6807/12677 conn(0x55a314e85800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1829 0x55a3331d9400 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:19.157668) v4 > -7295> 2018-07-11 07:33:19.158420 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6803/15255 1829 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:19.157668) v4 ==== 2004+0+0 (2890697417 0 0) 0x55a332e79a00 con 0x55a314e1d800 > -7294> 2018-07-11 07:33:19.158425 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6807/12677 1829 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:19.157668) v4 ==== 2004+0+0 (2890697417 0 0) 0x55a3331d9400 con 0x55a314e85800 > -7293> 2018-07-11 07:33:19.158448 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6809/12650 conn(0x55a314e1f000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1829 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:19.157668) v4 > -7292> 2018-07-11 07:33:19.158460 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6802/13323 conn(0x55a314e15000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1829 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:19.157668) v4 > -7291> 2018-07-11 07:33:19.158475 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6808/12651 conn(0x55a314e96000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1829 0x55a3331d9400 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:19.157668) v4 > -7290> 2018-07-11 07:33:19.158498 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6802/13323 1829 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:19.157668) v4 ==== 2004+0+0 (2890697417 0 0) 0x55a332e79a00 con 0x55a314e15000 > -7289> 2018-07-11 07:33:19.158498 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6808/12651 1829 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:19.157668) v4 ==== 2004+0+0 (2890697417 0 0) 0x55a3331d9400 con 0x55a314e96000 > -7288> 2018-07-11 07:33:19.158495 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6809/12650 1829 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:19.157668) v4 ==== 2004+0+0 (2890697417 0 0) 0x55a34de78600 con 0x55a314e1f000 > -7287> 2018-07-11 07:33:19.158538 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6806/15885 conn(0x55a314e00000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1829 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:19.157668) v4 > -7286> 2018-07-11 07:33:19.158550 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6807/15263 conn(0x55a314e9a800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1829 0x55a3331d9400 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:19.157668) v4 > -7285> 2018-07-11 07:33:19.158550 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6802/11967 conn(0x55a314e12000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1829 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:19.157668) v4 > -7284> 2018-07-11 07:33:19.158565 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6806/15885 1829 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:19.157668) v4 ==== 2004+0+0 (2890697417 0 0) 0x55a332e79a00 con 0x55a314e00000 > -7283> 2018-07-11 07:33:19.158567 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6807/15263 1829 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:19.157668) v4 ==== 2004+0+0 (2890697417 0 0) 0x55a3331d9400 con 0x55a314e9a800 > -7282> 2018-07-11 07:33:19.158568 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6802/11967 1829 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:19.157668) v4 ==== 2004+0+0 (2890697417 0 0) 0x55a34de78600 con 0x55a314e12000 > -7281> 2018-07-11 07:33:19.158595 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6802/15919 conn(0x55a314e87000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1829 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:19.157668) v4 > -7280> 2018-07-11 07:33:19.158607 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6802/15919 1829 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:19.157668) v4 ==== 2004+0+0 (2890697417 0 0) 0x55a332e79a00 con 0x55a314e87000 > -7279> 2018-07-11 07:33:19.158621 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6811/12650 conn(0x55a314e20800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1829 0x55a3331d9400 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:19.157668) v4 > -7278> 2018-07-11 07:33:19.158621 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6803/13323 conn(0x55a314e16800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1829 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:19.157668) v4 > -7277> 2018-07-11 07:33:19.158642 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6811/12650 1829 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:19.157668) v4 ==== 2004+0+0 (2890697417 0 0) 0x55a3331d9400 con 0x55a314e20800 > -7276> 2018-07-11 07:33:19.158648 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6803/13323 1829 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:19.157668) v4 ==== 2004+0+0 (2890697417 0 0) 0x55a34de78600 con 0x55a314e16800 > -7275> 2018-07-11 07:33:19.158689 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6803/11967 conn(0x55a314e13800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1829 0x55a3331d9400 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:19.157668) v4 > -7274> 2018-07-11 07:33:19.158689 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6807/15885 conn(0x55a314e01800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1829 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:19.157668) v4 > -7273> 2018-07-11 07:33:19.158710 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6803/11967 1829 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:19.157668) v4 ==== 2004+0+0 (2890697417 0 0) 0x55a3331d9400 con 0x55a314e13800 > -7272> 2018-07-11 07:33:19.158714 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6807/15885 1829 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:19.157668) v4 ==== 2004+0+0 (2890697417 0 0) 0x55a34de78600 con 0x55a314e01800 > -7271> 2018-07-11 07:33:19.158766 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6803/15919 conn(0x55a314e88800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1829 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:19.157668) v4 > -7270> 2018-07-11 07:33:19.158780 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6803/15919 1829 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:19.157668) v4 ==== 2004+0+0 (2890697417 0 0) 0x55a34de78600 con 0x55a314e88800 > -7269> 2018-07-11 07:33:19.350821 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a313f6c480 con 0 > -7268> 2018-07-11 07:33:19.362845 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5067 0x55a327d45900 replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:494/492,start:6:a9b3de91:::benchmark_data_magna108_35617_object52910:0,end:6:a9b48c67:::benchmark_data_magna108_35617_object598775:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -7267> 2018-07-11 07:33:19.362868 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5067 ==== replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:494/492,start:6:a9b3de91:::benchmark_data_magna108_35617_object52910:0,end:6:a9b48c67:::benchmark_data_magna108_35617_object598775:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 217+0+0 (3521307066 0 0) 0x55a327d45900 con 0x55a317019000 > -7266> 2018-07-11 07:33:19.422986 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15255 conn(0x55a311b64000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1855 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:33:20.635935) v4 > -7265> 2018-07-11 07:33:19.423002 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15255 conn(0x55a311b65800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1855 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:33:20.635935) v4 > -7264> 2018-07-11 07:33:19.423014 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.0 10.8.128.113:0/15255 1855 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:20.635935) v4 ==== 2004+0+0 (2642640514 0 0) 0x55a34de78600 con 0x55a311b64000 > -7263> 2018-07-11 07:33:19.423021 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.0 10.8.128.113:0/15255 1855 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:20.635935) v4 ==== 2004+0+0 (2642640514 0 0) 0x55a332e79a00 con 0x55a311b65800 > -7262> 2018-07-11 07:33:19.423033 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:20.635935) v4 -- 0x55a312362400 con 0 > -7261> 2018-07-11 07:33:19.423062 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:20.635935) v4 -- 0x55a33610ec00 con 0 > -7260> 2018-07-11 07:33:19.570865 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12650 conn(0x55a311b68800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1845 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:33:19.606075) v4 > -7259> 2018-07-11 07:33:19.570888 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.3 10.8.128.106:0/12650 1845 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:19.606075) v4 ==== 2004+0+0 (2070452680 0 0) 0x55a34de6f000 con 0x55a311b68800 > -7258> 2018-07-11 07:33:19.570904 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:19.606075) v4 -- 0x55a33610f800 con 0 > -7257> 2018-07-11 07:33:19.571003 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12650 conn(0x55a311b67000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1845 0x55a3331d9400 osd_ping(ping e494 stamp 2018-07-11 07:33:19.606075) v4 > -7256> 2018-07-11 07:33:19.571020 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.3 10.8.128.106:0/12650 1845 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:19.606075) v4 ==== 2004+0+0 (2070452680 0 0) 0x55a3331d9400 con 0x55a311b67000 > -7255> 2018-07-11 07:33:19.571029 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:19.606075) v4 -- 0x55a333f6e200 con 0 > -7254> 2018-07-11 07:33:20.066194 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a32403bb00 con 0 > -7253> 2018-07-11 07:33:20.079752 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5068 0x55a327d45b80 replica scrub(pg: 6.15,from:0'0,to:430'105697,epoch:494/492,start:6:a9b48c67:::benchmark_data_magna108_35617_object598775:0,end:6:a9b562ae:::benchmark_data_magna108_35617_object540539:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -7252> 2018-07-11 07:33:20.079777 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5068 ==== replica scrub(pg: 6.15,from:0'0,to:430'105697,epoch:494/492,start:6:a9b48c67:::benchmark_data_magna108_35617_object598775:0,end:6:a9b562ae:::benchmark_data_magna108_35617_object540539:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (1398099817 0 0) 0x55a327d45b80 con 0x55a317019000 > -7251> 2018-07-11 07:33:20.412158 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15263 conn(0x55a321664000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1826 0x55a329ac3a00 osd_ping(ping e494 stamp 2018-07-11 07:33:21.625313) v4 > -7250> 2018-07-11 07:33:20.412191 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.5 10.8.128.113:0/15263 1826 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:21.625313) v4 ==== 2004+0+0 (2411577257 0 0) 0x55a329ac3a00 con 0x55a321664000 > -7249> 2018-07-11 07:33:20.412201 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:21.625313) v4 -- 0x55a333f6e400 con 0 > -7248> 2018-07-11 07:33:20.412286 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15263 conn(0x55a321667000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1826 0x55a333f6e200 osd_ping(ping e494 stamp 2018-07-11 07:33:21.625313) v4 > -7247> 2018-07-11 07:33:20.412302 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.5 10.8.128.113:0/15263 1826 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:21.625313) v4 ==== 2004+0+0 (2411577257 0 0) 0x55a333f6e200 con 0x55a321667000 > -7246> 2018-07-11 07:33:20.412312 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:21.625313) v4 -- 0x55a333f6e400 con 0 > -7245> 2018-07-11 07:33:20.859645 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12651 conn(0x55a32169f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1832 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:33:20.894874) v4 > -7244> 2018-07-11 07:33:20.859673 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.2 10.8.128.106:0/12651 1832 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:20.894874) v4 ==== 2004+0+0 (676817224 0 0) 0x55a324474c00 con 0x55a32169f000 > -7243> 2018-07-11 07:33:20.859671 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12651 conn(0x55a3216a0800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1832 0x55a329ac3a00 osd_ping(ping e494 stamp 2018-07-11 07:33:20.894874) v4 > -7242> 2018-07-11 07:33:20.859687 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:20.894874) v4 -- 0x55a33047e600 con 0 > -7241> 2018-07-11 07:33:20.859689 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.2 10.8.128.106:0/12651 1832 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:20.894874) v4 ==== 2004+0+0 (676817224 0 0) 0x55a329ac3a00 con 0x55a3216a0800 > -7240> 2018-07-11 07:33:20.859714 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:20.894874) v4 -- 0x55a333f6e800 con 0 > -7239> 2018-07-11 07:33:20.945759 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a3128ce480 con 0 > -7238> 2018-07-11 07:33:20.977113 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5069 0x55a32fdd4000 replica scrub(pg: 6.15,from:0'0,to:430'105801,epoch:494/492,start:6:a9b562ae:::benchmark_data_magna108_35617_object540539:0,end:6:a9b688e3:::benchmark_data_magna108_35617_object224735:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -7237> 2018-07-11 07:33:20.977141 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5069 ==== replica scrub(pg: 6.15,from:0'0,to:430'105801,epoch:494/492,start:6:a9b562ae:::benchmark_data_magna108_35617_object540539:0,end:6:a9b688e3:::benchmark_data_magna108_35617_object224735:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (2489514028 0 0) 0x55a32fdd4000 con 0x55a317019000 > -7236> 2018-07-11 07:33:21.244515 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b7000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1780 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:33:20.994908) v4 > -7235> 2018-07-11 07:33:21.244540 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.8 10.8.128.108:0/11967 1780 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:20.994908) v4 ==== 2004+0+0 (1853055286 0 0) 0x55a34de78600 con 0x55a3170b7000 > -7234> 2018-07-11 07:33:21.244544 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b8800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1780 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:33:20.994908) v4 > -7233> 2018-07-11 07:33:21.244554 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:20.994908) v4 -- 0x55a33610f800 con 0 > -7232> 2018-07-11 07:33:21.244561 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.8 10.8.128.108:0/11967 1780 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:20.994908) v4 ==== 2004+0+0 (1853055286 0 0) 0x55a332e79a00 con 0x55a3170b8800 > -7231> 2018-07-11 07:33:21.244581 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:20.994908) v4 -- 0x55a33047e600 con 0 > -7230> 2018-07-11 07:33:21.343302 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.115:0/13323 conn(0x55a321665800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1841 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:33:21.680408) v4 > -7229> 2018-07-11 07:33:21.343328 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.6 10.8.128.115:0/13323 1841 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:21.680408) v4 ==== 2004+0+0 (2470796819 0 0) 0x55a324474c00 con 0x55a321665800 > -7228> 2018-07-11 07:33:21.343344 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:21.680408) v4 -- 0x55a33047e600 con 0 > -7227> 2018-07-11 07:33:21.343401 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.115:0/13323 conn(0x55a321668800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1841 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:33:21.680408) v4 > -7226> 2018-07-11 07:33:21.343419 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.6 10.8.128.115:0/13323 1841 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:21.680408) v4 ==== 2004+0+0 (2470796819 0 0) 0x55a324474c00 con 0x55a321668800 > -7225> 2018-07-11 07:33:21.343430 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:21.680408) v4 -- 0x55a3435eb600 con 0 > -7224> 2018-07-11 07:33:21.360122 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12651 conn(0x55a3216a0800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1833 0x55a333f6e200 osd_ping(ping e494 stamp 2018-07-11 07:33:21.395335) v4 > -7223> 2018-07-11 07:33:21.360145 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.2 10.8.128.106:0/12651 1833 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:21.395335) v4 ==== 2004+0+0 (3807279878 0 0) 0x55a333f6e200 con 0x55a3216a0800 > -7222> 2018-07-11 07:33:21.360160 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:21.395335) v4 -- 0x55a3342d7a00 con 0 > -7221> 2018-07-11 07:33:21.360193 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12651 conn(0x55a32169f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1833 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:33:21.395335) v4 > -7220> 2018-07-11 07:33:21.360213 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.2 10.8.128.106:0/12651 1833 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:21.395335) v4 ==== 2004+0+0 (3807279878 0 0) 0x55a332e79a00 con 0x55a32169f000 > -7219> 2018-07-11 07:33:21.360224 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:21.395335) v4 -- 0x55a33047e600 con 0 > -7218> 2018-07-11 07:33:21.414677 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4800000 on bdev 1, free 0x2fe000; fallback to bdev 2 > -7217> 2018-07-11 07:33:21.771591 7f355d88e700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.107:0/12617 conn(0x55a32447f800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1802 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:33:21.771230) v4 > -7216> 2018-07-11 07:33:21.771605 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.107:0/12617 conn(0x55a32447e000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1802 0x55a333f6e800 osd_ping(ping e494 stamp 2018-07-11 07:33:21.771230) v4 > -7215> 2018-07-11 07:33:21.771614 7f355d88e700 1 -- 10.8.128.107:6803/12665 <== osd.9 10.8.128.107:0/12617 1802 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:21.771230) v4 ==== 2004+0+0 (3264067488 0 0) 0x55a34de6f000 con 0x55a32447f800 > -7214> 2018-07-11 07:33:21.771622 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.9 10.8.128.107:0/12617 1802 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:21.771230) v4 ==== 2004+0+0 (3264067488 0 0) 0x55a333f6e800 con 0x55a32447e000 > -7213> 2018-07-11 07:33:21.771627 7f355d88e700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:21.771230) v4 -- 0x55a34de79c00 con 0 > -7212> 2018-07-11 07:33:21.771660 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:21.771230) v4 -- 0x55a31fca4e00 con 0 > -7211> 2018-07-11 07:33:21.923503 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15287 conn(0x55a311b75000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1824 0x55a333f6e200 osd_ping(ping e494 stamp 2018-07-11 07:33:23.136667) v4 > -7210> 2018-07-11 07:33:21.923525 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.4 10.8.128.113:0/15287 1824 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:23.136667) v4 ==== 2004+0+0 (207211245 0 0) 0x55a333f6e200 con 0x55a311b75000 > -7209> 2018-07-11 07:33:21.923545 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:23.136667) v4 -- 0x55a348fbc400 con 0 > -7208> 2018-07-11 07:33:21.923536 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15287 conn(0x55a311b72000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1824 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:33:23.136667) v4 > -7207> 2018-07-11 07:33:21.923555 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.4 10.8.128.113:0/15287 1824 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:23.136667) v4 ==== 2004+0+0 (207211245 0 0) 0x55a324474c00 con 0x55a311b72000 > -7206> 2018-07-11 07:33:21.923569 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:23.136667) v4 -- 0x55a33047e200 con 0 > -7205> 2018-07-11 07:33:22.058134 7f3543899700 5 osd.10 494 heartbeat: osd_stat(783 GB used, 147 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,0,0,0,1]) > -7204> 2018-07-11 07:33:22.058147 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6802/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:33:22.058145) v4 -- 0x55a329ac2a00 con 0 > -7203> 2018-07-11 07:33:22.058182 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6803/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:33:22.058145) v4 -- 0x55a3331d9e00 con 0 > -7202> 2018-07-11 07:33:22.058205 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6806/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:33:22.058145) v4 -- 0x55a329ac2e00 con 0 > -7201> 2018-07-11 07:33:22.058241 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6807/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:33:22.058145) v4 -- 0x55a329ac3200 con 0 > -7200> 2018-07-11 07:33:22.058276 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6808/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:33:22.058145) v4 -- 0x55a329ac2400 con 0 > -7199> 2018-07-11 07:33:22.058296 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6810/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:33:22.058145) v4 -- 0x55a329ac3400 con 0 > -7198> 2018-07-11 07:33:22.058321 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6809/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:33:22.058145) v4 -- 0x55a329ac3600 con 0 > -7197> 2018-07-11 07:33:22.058344 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6811/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:33:22.058145) v4 -- 0x55a329ac3000 con 0 > -7196> 2018-07-11 07:33:22.058366 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6810/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:33:22.058145) v4 -- 0x55a329ac3800 con 0 > -7195> 2018-07-11 07:33:22.058387 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6811/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:33:22.058145) v4 -- 0x55a31e8f5c00 con 0 > -7194> 2018-07-11 07:33:22.058410 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6806/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:33:22.058145) v4 -- 0x55a343483c00 con 0 > -7193> 2018-07-11 07:33:22.058440 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6807/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:33:22.058145) v4 -- 0x55a343483a00 con 0 > -7192> 2018-07-11 07:33:22.058465 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6802/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:33:22.058145) v4 -- 0x55a343483000 con 0 > -7191> 2018-07-11 07:33:22.058513 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6803/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:33:22.058145) v4 -- 0x55a3331d8000 con 0 > -7190> 2018-07-11 07:33:22.058556 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6806/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:33:22.058145) v4 -- 0x55a343483e00 con 0 > -7189> 2018-07-11 07:33:22.058561 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6802/15255 conn(0x55a314e1c000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1830 0x55a333f6e800 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:22.058145) v4 > -7188> 2018-07-11 07:33:22.058579 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6802/15255 1830 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:22.058145) v4 ==== 2004+0+0 (3545423048 0 0) 0x55a333f6e800 con 0x55a314e1c000 > -7187> 2018-07-11 07:33:22.058583 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6807/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:33:22.058145) v4 -- 0x55a3331d8200 con 0 > -7186> 2018-07-11 07:33:22.058610 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6802/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:33:22.058145) v4 -- 0x55a3331d8600 con 0 > -7185> 2018-07-11 07:33:22.058625 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6803/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:33:22.058145) v4 -- 0x55a343483200 con 0 > -7184> 2018-07-11 07:33:22.058663 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6806/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:33:22.058145) v4 -- 0x55a3331d8800 con 0 > -7183> 2018-07-11 07:33:22.058676 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6807/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:33:22.058145) v4 -- 0x55a3331d8a00 con 0 > -7182> 2018-07-11 07:33:22.058670 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6803/15255 conn(0x55a314e1d800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1830 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:22.058145) v4 > -7181> 2018-07-11 07:33:22.058689 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6802/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:33:22.058145) v4 -- 0x55a343483800 con 0 > -7180> 2018-07-11 07:33:22.058702 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6803/15255 1830 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:22.058145) v4 ==== 2004+0+0 (3545423048 0 0) 0x55a324474c00 con 0x55a314e1d800 > -7179> 2018-07-11 07:33:22.058709 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6803/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:33:22.058145) v4 -- 0x55a3331d8c00 con 0 > -7178> 2018-07-11 07:33:22.058721 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6806/12677 conn(0x55a314e84000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1830 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:22.058145) v4 > -7177> 2018-07-11 07:33:22.058741 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6806/12677 1830 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:22.058145) v4 ==== 2004+0+0 (3545423048 0 0) 0x55a34de78600 con 0x55a314e84000 > -7176> 2018-07-11 07:33:22.058814 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6807/12677 conn(0x55a314e85800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1830 0x55a333f6e200 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:22.058145) v4 > -7175> 2018-07-11 07:33:22.058831 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6807/12677 1830 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:22.058145) v4 ==== 2004+0+0 (3545423048 0 0) 0x55a333f6e200 con 0x55a314e85800 > -7174> 2018-07-11 07:33:22.058866 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6811/12650 conn(0x55a314e20800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1830 0x55a333f6e200 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:22.058145) v4 > -7173> 2018-07-11 07:33:22.058860 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6810/12651 conn(0x55a314e97800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1830 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:22.058145) v4 > -7172> 2018-07-11 07:33:22.058860 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6809/12650 conn(0x55a314e1f000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1830 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:22.058145) v4 > -7171> 2018-07-11 07:33:22.058883 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6811/12650 1830 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:22.058145) v4 ==== 2004+0+0 (3545423048 0 0) 0x55a333f6e200 con 0x55a314e20800 > -7170> 2018-07-11 07:33:22.058888 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6810/12651 1830 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:22.058145) v4 ==== 2004+0+0 (3545423048 0 0) 0x55a332e79a00 con 0x55a314e97800 > -7169> 2018-07-11 07:33:22.058894 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6809/12650 1830 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:22.058145) v4 ==== 2004+0+0 (3545423048 0 0) 0x55a34de6f000 con 0x55a314e1f000 > -7168> 2018-07-11 07:33:22.058922 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6807/15263 conn(0x55a314e9a800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1830 0x55a333f6e200 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:22.058145) v4 > -7167> 2018-07-11 07:33:22.058934 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6807/15263 1830 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:22.058145) v4 ==== 2004+0+0 (3545423048 0 0) 0x55a333f6e200 con 0x55a314e9a800 > -7166> 2018-07-11 07:33:22.058938 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6811/15287 conn(0x55a314e04800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1830 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:22.058145) v4 > -7165> 2018-07-11 07:33:22.058938 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6806/15263 conn(0x55a314e99000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1830 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:22.058145) v4 > -7164> 2018-07-11 07:33:22.058960 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6811/15287 1830 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:22.058145) v4 ==== 2004+0+0 (3545423048 0 0) 0x55a332e79a00 con 0x55a314e04800 > -7163> 2018-07-11 07:33:22.058965 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6806/15263 1830 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:22.058145) v4 ==== 2004+0+0 (3545423048 0 0) 0x55a34de6f000 con 0x55a314e99000 > -7162> 2018-07-11 07:33:22.058963 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6808/12651 conn(0x55a314e96000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1830 0x55a333f6e200 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:22.058145) v4 > -7161> 2018-07-11 07:33:22.058983 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6808/12651 1830 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:22.058145) v4 ==== 2004+0+0 (3545423048 0 0) 0x55a333f6e200 con 0x55a314e96000 > -7160> 2018-07-11 07:33:22.059003 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6807/12617 conn(0x55a314e27800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1830 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:22.058145) v4 > -7159> 2018-07-11 07:33:22.059017 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6810/15287 conn(0x55a314e03000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1830 0x55a333f6e200 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:22.058145) v4 > -7158> 2018-07-11 07:33:22.059017 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6802/13323 conn(0x55a314e15000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1830 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:22.058145) v4 > -7157> 2018-07-11 07:33:22.059029 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6810/15287 1830 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:22.058145) v4 ==== 2004+0+0 (3545423048 0 0) 0x55a333f6e200 con 0x55a314e03000 > -7156> 2018-07-11 07:33:22.059033 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6802/13323 1830 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:22.058145) v4 ==== 2004+0+0 (3545423048 0 0) 0x55a332e79a00 con 0x55a314e15000 > -7155> 2018-07-11 07:33:22.059040 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6807/12617 1830 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:22.058145) v4 ==== 2004+0+0 (3545423048 0 0) 0x55a34de6f000 con 0x55a314e27800 > -7154> 2018-07-11 07:33:22.059067 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6806/12617 conn(0x55a314e26000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1830 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:22.058145) v4 > -7153> 2018-07-11 07:33:22.059087 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6806/12617 1830 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:22.058145) v4 ==== 2004+0+0 (3545423048 0 0) 0x55a332e79a00 con 0x55a314e26000 > -7152> 2018-07-11 07:33:22.059088 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6803/13323 conn(0x55a314e16800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1830 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:22.058145) v4 > -7151> 2018-07-11 07:33:22.059097 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6803/11967 conn(0x55a314e13800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1830 0x55a333f6e200 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:22.058145) v4 > -7150> 2018-07-11 07:33:22.059126 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6803/11967 1830 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:22.058145) v4 ==== 2004+0+0 (3545423048 0 0) 0x55a333f6e200 con 0x55a314e13800 > -7149> 2018-07-11 07:33:22.059106 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6803/13323 1830 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:22.058145) v4 ==== 2004+0+0 (3545423048 0 0) 0x55a34de6f000 con 0x55a314e16800 > -7148> 2018-07-11 07:33:22.059141 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6806/15885 conn(0x55a314e00000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1830 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:22.058145) v4 > -7147> 2018-07-11 07:33:22.059161 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6806/15885 1830 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:22.058145) v4 ==== 2004+0+0 (3545423048 0 0) 0x55a332e79a00 con 0x55a314e00000 > -7146> 2018-07-11 07:33:22.059176 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6802/11967 conn(0x55a314e12000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1830 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:22.058145) v4 > -7145> 2018-07-11 07:33:22.059194 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6802/11967 1830 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:22.058145) v4 ==== 2004+0+0 (3545423048 0 0) 0x55a34de6f000 con 0x55a314e12000 > -7144> 2018-07-11 07:33:22.059211 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6802/15919 conn(0x55a314e87000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1830 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:22.058145) v4 > -7143> 2018-07-11 07:33:22.059230 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6802/15919 1830 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:22.058145) v4 ==== 2004+0+0 (3545423048 0 0) 0x55a332e79a00 con 0x55a314e87000 > -7142> 2018-07-11 07:33:22.059239 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6807/15885 conn(0x55a314e01800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1830 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:22.058145) v4 > -7141> 2018-07-11 07:33:22.059263 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6807/15885 1830 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:22.058145) v4 ==== 2004+0+0 (3545423048 0 0) 0x55a34de6f000 con 0x55a314e01800 > -7140> 2018-07-11 07:33:22.059307 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6803/15919 conn(0x55a314e88800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1830 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:22.058145) v4 > -7139> 2018-07-11 07:33:22.059320 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6803/15919 1830 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:22.058145) v4 ==== 2004+0+0 (3545423048 0 0) 0x55a34de6f000 con 0x55a314e88800 > -7138> 2018-07-11 07:33:22.060625 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15885 conn(0x55a3170b5800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1848 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:33:22.684176) v4 > -7137> 2018-07-11 07:33:22.060648 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.7 10.8.128.114:0/15885 1848 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:22.684176) v4 ==== 2004+0+0 (966124566 0 0) 0x55a332e79a00 con 0x55a3170b5800 > -7136> 2018-07-11 07:33:22.060659 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:22.684176) v4 -- 0x55a31e8f5c00 con 0 > -7135> 2018-07-11 07:33:22.060682 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15885 conn(0x55a311b76800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1848 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:33:22.684176) v4 > -7134> 2018-07-11 07:33:22.060698 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.7 10.8.128.114:0/15885 1848 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:22.684176) v4 ==== 2004+0+0 (966124566 0 0) 0x55a34de6f000 con 0x55a311b76800 > -7133> 2018-07-11 07:33:22.060708 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:22.684176) v4 -- 0x55a3331d8000 con 0 > -7132> 2018-07-11 07:33:22.251670 7f35488a3700 1 -- 10.8.128.107:6800/12665 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55a32aa0d340 con 0 > -7131> 2018-07-11 07:33:22.251740 7f35488a3700 1 -- 10.8.128.107:6800/12665 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55a30fa24b00 con 0 > -7130> 2018-07-11 07:33:22.664272 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 58] Generated table #36574: 2638241 keys, 74546956 bytes > -7129> 2018-07-11 07:33:22.664316 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294402664288, "cf_name": "default", "job": 58, "event": "table_file_creation", "file_number": 36574, "file_size": 74546956, "table_properties": {"data_size": 67112114, "index_size": 690669, "filter_size": 6743183, "raw_key_size": 76215855, "raw_average_key_size": 28, "raw_value_size": 31365758, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} > -7128> 2018-07-11 07:33:22.705333 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a3177d9d40 con 0 > -7127> 2018-07-11 07:33:22.721423 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5070 0x55a32fdd4280 replica scrub(pg: 6.15,from:0'0,to:430'106721,epoch:494/492,start:6:a9b688e3:::benchmark_data_magna108_35617_object224735:0,end:6:a9b7a2c5:::benchmark_data_magna108_35617_object1088160:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -7126> 2018-07-11 07:33:22.721447 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5070 ==== replica scrub(pg: 6.15,from:0'0,to:430'106721,epoch:494/492,start:6:a9b688e3:::benchmark_data_magna108_35617_object224735:0,end:6:a9b7a2c5:::benchmark_data_magna108_35617_object1088160:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 219+0+0 (2654903795 0 0) 0x55a32fdd4280 con 0x55a317019000 > -7125> 2018-07-11 07:33:22.851280 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15919 conn(0x55a3170b4000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1833 0x55a333f6e200 osd_ping(ping e494 stamp 2018-07-11 07:33:23.474819) v4 > -7124> 2018-07-11 07:33:22.851301 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.11 10.8.128.114:0/15919 1833 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:23.474819) v4 ==== 2004+0+0 (3932188024 0 0) 0x55a333f6e200 con 0x55a3170b4000 > -7123> 2018-07-11 07:33:22.851289 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15919 conn(0x55a317074800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1833 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:33:23.474819) v4 > -7122> 2018-07-11 07:33:22.851314 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:23.474819) v4 -- 0x55a329ac3200 con 0 > -7121> 2018-07-11 07:33:22.851313 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.11 10.8.128.114:0/15919 1833 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:23.474819) v4 ==== 2004+0+0 (3932188024 0 0) 0x55a324474c00 con 0x55a317074800 > -7120> 2018-07-11 07:33:22.851343 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:23.474819) v4 -- 0x55a31e8f5c00 con 0 > -7119> 2018-07-11 07:33:23.123051 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12677 conn(0x55a32169d800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1851 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:33:23.158273) v4 > -7118> 2018-07-11 07:33:23.123073 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.1 10.8.128.106:0/12677 1851 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:23.158273) v4 ==== 2004+0+0 (3400564740 0 0) 0x55a34de78600 con 0x55a32169d800 > -7117> 2018-07-11 07:33:23.123083 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:23.158273) v4 -- 0x55a3331d8000 con 0 > -7116> 2018-07-11 07:33:23.123115 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12677 conn(0x55a32169c000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1851 0x55a333f6e800 osd_ping(ping e494 stamp 2018-07-11 07:33:23.158273) v4 > -7115> 2018-07-11 07:33:23.123141 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.1 10.8.128.106:0/12677 1851 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:23.158273) v4 ==== 2004+0+0 (3400564740 0 0) 0x55a333f6e800 con 0x55a32169c000 > -7114> 2018-07-11 07:33:23.123160 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:23.158273) v4 -- 0x55a329ac2a00 con 0 > -7113> 2018-07-11 07:33:23.158900 7f3543899700 5 osd.10 494 heartbeat: osd_stat(783 GB used, 147 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,0,1]) > -7112> 2018-07-11 07:33:23.158914 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6802/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:33:23.158912) v4 -- 0x55a3331d8400 con 0 > -7111> 2018-07-11 07:33:23.158949 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6803/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:33:23.158912) v4 -- 0x55a324966600 con 0 > -7110> 2018-07-11 07:33:23.158964 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6806/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:33:23.158912) v4 -- 0x55a31e8f5800 con 0 > -7109> 2018-07-11 07:33:23.158978 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6807/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:33:23.158912) v4 -- 0x55a343480000 con 0 > -7108> 2018-07-11 07:33:23.158989 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6808/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:33:23.158912) v4 -- 0x55a343480800 con 0 > -7107> 2018-07-11 07:33:23.159015 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6810/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:33:23.158912) v4 -- 0x55a343481000 con 0 > -7106> 2018-07-11 07:33:23.159026 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6809/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:33:23.158912) v4 -- 0x55a343480e00 con 0 > -7105> 2018-07-11 07:33:23.159039 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6811/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:33:23.158912) v4 -- 0x55a31e8f5e00 con 0 > -7104> 2018-07-11 07:33:23.159051 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6810/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:33:23.158912) v4 -- 0x55a343481400 con 0 > -7103> 2018-07-11 07:33:23.159061 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6811/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:33:23.158912) v4 -- 0x55a343480400 con 0 > -7102> 2018-07-11 07:33:23.159075 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6806/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:33:23.158912) v4 -- 0x55a30fa76000 con 0 > -7101> 2018-07-11 07:33:23.159087 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6807/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:33:23.158912) v4 -- 0x55a30fa77200 con 0 > -7100> 2018-07-11 07:33:23.159099 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6802/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:33:23.158912) v4 -- 0x55a30fa77400 con 0 > -7099> 2018-07-11 07:33:23.159112 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6803/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:33:23.158912) v4 -- 0x55a30fa76c00 con 0 > -7098> 2018-07-11 07:33:23.159125 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6806/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:33:23.158912) v4 -- 0x55a33be6e200 con 0 > -7097> 2018-07-11 07:33:23.159137 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6807/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:33:23.158912) v4 -- 0x55a3331d9200 con 0 > -7096> 2018-07-11 07:33:23.159149 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6802/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:33:23.158912) v4 -- 0x55a324967a00 con 0 > -7095> 2018-07-11 07:33:23.159161 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6803/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:33:23.158912) v4 -- 0x55a33be6e000 con 0 > -7094> 2018-07-11 07:33:23.159174 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6806/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:33:23.158912) v4 -- 0x55a324967800 con 0 > -7093> 2018-07-11 07:33:23.159193 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6807/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:33:23.158912) v4 -- 0x55a329323c00 con 0 > -7092> 2018-07-11 07:33:23.159207 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6802/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:33:23.158912) v4 -- 0x55a324967600 con 0 > -7091> 2018-07-11 07:33:23.159229 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6803/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:33:23.158912) v4 -- 0x55a3331d9a00 con 0 > -7090> 2018-07-11 07:33:23.159370 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6807/12617 conn(0x55a314e27800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1831 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:23.158912) v4 > -7089> 2018-07-11 07:33:23.159392 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6807/12617 1831 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:23.158912) v4 ==== 2004+0+0 (1210952611 0 0) 0x55a34de6f000 con 0x55a314e27800 > -7088> 2018-07-11 07:33:23.159412 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6806/12617 conn(0x55a314e26000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1831 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:23.158912) v4 > -7087> 2018-07-11 07:33:23.159432 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6806/12617 1831 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:23.158912) v4 ==== 2004+0+0 (1210952611 0 0) 0x55a332e79a00 con 0x55a314e26000 > -7086> 2018-07-11 07:33:23.159443 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6806/12677 conn(0x55a314e84000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1831 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:23.158912) v4 > -7085> 2018-07-11 07:33:23.159458 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6806/12677 1831 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:23.158912) v4 ==== 2004+0+0 (1210952611 0 0) 0x55a34de6f000 con 0x55a314e84000 > -7084> 2018-07-11 07:33:23.159484 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6803/15255 conn(0x55a314e1d800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1831 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:23.158912) v4 > -7083> 2018-07-11 07:33:23.159481 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6808/12651 conn(0x55a314e96000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1831 0x55a333f6e200 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:23.158912) v4 > -7082> 2018-07-11 07:33:23.159492 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6809/12650 conn(0x55a314e1f000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1831 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:23.158912) v4 > -7081> 2018-07-11 07:33:23.159505 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6809/12650 1831 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:23.158912) v4 ==== 2004+0+0 (1210952611 0 0) 0x55a34de6f000 con 0x55a314e1f000 > -7080> 2018-07-11 07:33:23.159499 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6803/15255 1831 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:23.158912) v4 ==== 2004+0+0 (1210952611 0 0) 0x55a332e79a00 con 0x55a314e1d800 > -7079> 2018-07-11 07:33:23.159500 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6808/12651 1831 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:23.158912) v4 ==== 2004+0+0 (1210952611 0 0) 0x55a333f6e200 con 0x55a314e96000 > -7078> 2018-07-11 07:33:23.159544 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6810/12651 conn(0x55a314e97800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1831 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:23.158912) v4 > -7077> 2018-07-11 07:33:23.159557 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6810/12651 1831 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:23.158912) v4 ==== 2004+0+0 (1210952611 0 0) 0x55a332e79a00 con 0x55a314e97800 > -7076> 2018-07-11 07:33:23.159560 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6806/15263 conn(0x55a314e99000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1831 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:23.158912) v4 > -7075> 2018-07-11 07:33:23.159576 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6806/15263 1831 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:23.158912) v4 ==== 2004+0+0 (1210952611 0 0) 0x55a34de6f000 con 0x55a314e99000 > -7074> 2018-07-11 07:33:23.159575 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6807/12677 conn(0x55a314e85800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1831 0x55a333f6e200 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:23.158912) v4 > -7073> 2018-07-11 07:33:23.159588 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6807/12677 1831 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:23.158912) v4 ==== 2004+0+0 (1210952611 0 0) 0x55a333f6e200 con 0x55a314e85800 > -7072> 2018-07-11 07:33:23.159588 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6811/15287 conn(0x55a314e04800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1831 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:23.158912) v4 > -7071> 2018-07-11 07:33:23.159614 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6811/15287 1831 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:23.158912) v4 ==== 2004+0+0 (1210952611 0 0) 0x55a332e79a00 con 0x55a314e04800 > -7070> 2018-07-11 07:33:23.159641 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6811/12650 conn(0x55a314e20800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1831 0x55a333f6e200 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:23.158912) v4 > -7069> 2018-07-11 07:33:23.159654 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6811/12650 1831 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:23.158912) v4 ==== 2004+0+0 (1210952611 0 0) 0x55a333f6e200 con 0x55a314e20800 > -7068> 2018-07-11 07:33:23.159653 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6803/13323 conn(0x55a314e16800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1831 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:23.158912) v4 > -7067> 2018-07-11 07:33:23.159674 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6802/15255 conn(0x55a314e1c000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1831 0x55a333f6e200 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:23.158912) v4 > -7066> 2018-07-11 07:33:23.159684 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6802/15255 1831 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:23.158912) v4 ==== 2004+0+0 (1210952611 0 0) 0x55a333f6e200 con 0x55a314e1c000 > -7065> 2018-07-11 07:33:23.159678 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6803/13323 1831 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:23.158912) v4 ==== 2004+0+0 (1210952611 0 0) 0x55a34de6f000 con 0x55a314e16800 > -7064> 2018-07-11 07:33:23.159717 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6810/15287 conn(0x55a314e03000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1831 0x55a333f6e200 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:23.158912) v4 > -7063> 2018-07-11 07:33:23.159726 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6806/15885 conn(0x55a314e00000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1831 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:23.158912) v4 > -7062> 2018-07-11 07:33:23.159733 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6810/15287 1831 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:23.158912) v4 ==== 2004+0+0 (1210952611 0 0) 0x55a333f6e200 con 0x55a314e03000 > -7061> 2018-07-11 07:33:23.159746 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6806/15885 1831 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:23.158912) v4 ==== 2004+0+0 (1210952611 0 0) 0x55a332e79a00 con 0x55a314e00000 > -7060> 2018-07-11 07:33:23.159792 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6807/15885 conn(0x55a314e01800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1831 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:23.158912) v4 > -7059> 2018-07-11 07:33:23.159798 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6802/13323 conn(0x55a314e15000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1831 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:23.158912) v4 > -7058> 2018-07-11 07:33:23.159799 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6807/15263 conn(0x55a314e9a800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1831 0x55a333f6e200 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:23.158912) v4 > -7057> 2018-07-11 07:33:23.159809 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6802/13323 1831 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:23.158912) v4 ==== 2004+0+0 (1210952611 0 0) 0x55a332e79a00 con 0x55a314e15000 > -7056> 2018-07-11 07:33:23.159810 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6807/15263 1831 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:23.158912) v4 ==== 2004+0+0 (1210952611 0 0) 0x55a333f6e200 con 0x55a314e9a800 > -7055> 2018-07-11 07:33:23.159808 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6807/15885 1831 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:23.158912) v4 ==== 2004+0+0 (1210952611 0 0) 0x55a34de6f000 con 0x55a314e01800 > -7054> 2018-07-11 07:33:23.159843 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6802/15919 conn(0x55a314e87000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1831 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:23.158912) v4 > -7053> 2018-07-11 07:33:23.159855 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6802/15919 1831 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:23.158912) v4 ==== 2004+0+0 (1210952611 0 0) 0x55a332e79a00 con 0x55a314e87000 > -7052> 2018-07-11 07:33:23.159853 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6803/11967 conn(0x55a314e13800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1831 0x55a333f6e200 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:23.158912) v4 > -7051> 2018-07-11 07:33:23.159850 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6802/11967 conn(0x55a314e12000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1831 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:23.158912) v4 > -7050> 2018-07-11 07:33:23.159866 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6803/11967 1831 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:23.158912) v4 ==== 2004+0+0 (1210952611 0 0) 0x55a333f6e200 con 0x55a314e13800 > -7049> 2018-07-11 07:33:23.159869 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6802/11967 1831 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:23.158912) v4 ==== 2004+0+0 (1210952611 0 0) 0x55a34de6f000 con 0x55a314e12000 > -7048> 2018-07-11 07:33:23.159909 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6803/15919 conn(0x55a314e88800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1831 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:23.158912) v4 > -7047> 2018-07-11 07:33:23.159923 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6803/15919 1831 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:23.158912) v4 ==== 2004+0+0 (1210952611 0 0) 0x55a34de6f000 con 0x55a314e88800 > -7046> 2018-07-11 07:33:23.523538 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15255 conn(0x55a311b65800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1856 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:33:24.736550) v4 > -7045> 2018-07-11 07:33:23.523564 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.0 10.8.128.113:0/15255 1856 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:24.736550) v4 ==== 2004+0+0 (781483484 0 0) 0x55a332e79a00 con 0x55a311b65800 > -7044> 2018-07-11 07:33:23.523557 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15255 conn(0x55a311b64000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1856 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:33:24.736550) v4 > -7043> 2018-07-11 07:33:23.523586 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:24.736550) v4 -- 0x55a31e8f5c00 con 0 > -7042> 2018-07-11 07:33:23.523590 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.0 10.8.128.113:0/15255 1856 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:24.736550) v4 ==== 2004+0+0 (781483484 0 0) 0x55a34de6f000 con 0x55a311b64000 > -7041> 2018-07-11 07:33:23.523610 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:24.736550) v4 -- 0x55a3331d8000 con 0 > -7040> 2018-07-11 07:33:23.545074 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b7000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1781 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:33:23.295495) v4 > -7039> 2018-07-11 07:33:23.545097 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.8 10.8.128.108:0/11967 1781 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:23.295495) v4 ==== 2004+0+0 (447585 0 0) 0x55a34de78600 con 0x55a3170b7000 > -7038> 2018-07-11 07:33:23.545096 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b8800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1781 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:33:23.295495) v4 > -7037> 2018-07-11 07:33:23.545114 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.8 10.8.128.108:0/11967 1781 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:23.295495) v4 ==== 2004+0+0 (447585 0 0) 0x55a324474c00 con 0x55a3170b8800 > -7036> 2018-07-11 07:33:23.545115 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:23.295495) v4 -- 0x55a343483c00 con 0 > -7035> 2018-07-11 07:33:23.545148 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:23.295495) v4 -- 0x55a329ac3400 con 0 > -7034> 2018-07-11 07:33:23.630896 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a313f75440 con 0 > -7033> 2018-07-11 07:33:23.638435 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5071 0x55a32fdd4500 replica scrub(pg: 6.15,from:0'0,to:430'106348,epoch:494/492,start:6:a9b7a2c5:::benchmark_data_magna108_35617_object1088160:0,end:6:a9b8af02:::benchmark_data_magna108_35617_object838705:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -7032> 2018-07-11 07:33:23.638458 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5071 ==== replica scrub(pg: 6.15,from:0'0,to:430'106348,epoch:494/492,start:6:a9b7a2c5:::benchmark_data_magna108_35617_object1088160:0,end:6:a9b8af02:::benchmark_data_magna108_35617_object838705:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 219+0+0 (1661026659 0 0) 0x55a32fdd4500 con 0x55a317019000 > -7031> 2018-07-11 07:33:23.951629 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15919 conn(0x55a317074800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1834 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:33:24.575211) v4 > -7030> 2018-07-11 07:33:23.951657 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.11 10.8.128.114:0/15919 1834 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:24.575211) v4 ==== 2004+0+0 (3720129934 0 0) 0x55a332e79a00 con 0x55a317074800 > -7029> 2018-07-11 07:33:23.951671 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:24.575211) v4 -- 0x55a329ac3400 con 0 > -7028> 2018-07-11 07:33:23.951716 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15919 conn(0x55a3170b4000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1834 0x55a333f6e200 osd_ping(ping e494 stamp 2018-07-11 07:33:24.575211) v4 > -7027> 2018-07-11 07:33:23.951742 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.11 10.8.128.114:0/15919 1834 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:24.575211) v4 ==== 2004+0+0 (3720129934 0 0) 0x55a333f6e200 con 0x55a3170b4000 > -7026> 2018-07-11 07:33:23.951759 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:24.575211) v4 -- 0x55a3331d8400 con 0 > -7025> 2018-07-11 07:33:24.072170 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.107:0/12617 conn(0x55a32447e000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1803 0x55a333f6e800 osd_ping(ping e494 stamp 2018-07-11 07:33:24.071701) v4 > -7024> 2018-07-11 07:33:24.072196 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.9 10.8.128.107:0/12617 1803 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:24.071701) v4 ==== 2004+0+0 (313632518 0 0) 0x55a333f6e800 con 0x55a32447e000 > -7023> 2018-07-11 07:33:24.072195 7f355d88e700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.107:0/12617 conn(0x55a32447f800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1803 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:33:24.071701) v4 > -7022> 2018-07-11 07:33:24.072212 7f355d88e700 1 -- 10.8.128.107:6803/12665 <== osd.9 10.8.128.107:0/12617 1803 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:24.071701) v4 ==== 2004+0+0 (313632518 0 0) 0x55a34de6f000 con 0x55a32447f800 > -7021> 2018-07-11 07:33:24.072213 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:24.071701) v4 -- 0x55a329ac2a00 con 0 > -7020> 2018-07-11 07:33:24.072287 7f355d88e700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:24.071701) v4 -- 0x55a329ac3600 con 0 > -7019> 2018-07-11 07:33:24.271209 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12650 conn(0x55a311b67000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1846 0x55a333f6e200 osd_ping(ping e494 stamp 2018-07-11 07:33:24.306509) v4 > -7018> 2018-07-11 07:33:24.271209 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12650 conn(0x55a311b68800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1846 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:33:24.306509) v4 > -7017> 2018-07-11 07:33:24.271237 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.3 10.8.128.106:0/12650 1846 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:24.306509) v4 ==== 2004+0+0 (2861430030 0 0) 0x55a333f6e200 con 0x55a311b67000 > -7016> 2018-07-11 07:33:24.271240 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.3 10.8.128.106:0/12650 1846 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:24.306509) v4 ==== 2004+0+0 (2861430030 0 0) 0x55a34de78600 con 0x55a311b68800 > -7015> 2018-07-11 07:33:24.271268 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:24.306509) v4 -- 0x55a348fbc400 con 0 > -7014> 2018-07-11 07:33:24.271301 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:24.306509) v4 -- 0x55a329ac2e00 con 0 > -7013> 2018-07-11 07:33:24.483491 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a324076d80 con 0 > -7012> 2018-07-11 07:33:24.502742 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5072 0x55a32fdd4780 replica scrub(pg: 6.15,from:0'0,to:430'105819,epoch:494/492,start:6:a9b8af02:::benchmark_data_magna108_35617_object838705:0,end:6:a9b970ef:::benchmark_data_magna108_35617_object294973:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -7011> 2018-07-11 07:33:24.502767 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5072 ==== replica scrub(pg: 6.15,from:0'0,to:430'105819,epoch:494/492,start:6:a9b8af02:::benchmark_data_magna108_35617_object838705:0,end:6:a9b970ef:::benchmark_data_magna108_35617_object294973:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (1076697832 0 0) 0x55a32fdd4780 con 0x55a317019000 > -7010> 2018-07-11 07:33:24.512609 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15263 conn(0x55a321664000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1827 0x55a333f6e800 osd_ping(ping e494 stamp 2018-07-11 07:33:25.725761) v4 > -7009> 2018-07-11 07:33:24.512636 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.5 10.8.128.113:0/15263 1827 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:25.725761) v4 ==== 2004+0+0 (4078088406 0 0) 0x55a333f6e800 con 0x55a321664000 > -7008> 2018-07-11 07:33:24.512649 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:25.725761) v4 -- 0x55a31fca5400 con 0 > -7007> 2018-07-11 07:33:24.512702 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15263 conn(0x55a321667000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1827 0x55a333f6e800 osd_ping(ping e494 stamp 2018-07-11 07:33:25.725761) v4 > -7006> 2018-07-11 07:33:24.512723 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.5 10.8.128.113:0/15263 1827 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:25.725761) v4 ==== 2004+0+0 (4078088406 0 0) 0x55a333f6e800 con 0x55a321667000 > -7005> 2018-07-11 07:33:24.512739 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:25.725761) v4 -- 0x55a348fbc200 con 0 > -7004> 2018-07-11 07:33:24.645523 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b7000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1782 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:33:24.395979) v4 > -7003> 2018-07-11 07:33:24.645545 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.8 10.8.128.108:0/11967 1782 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:24.395979) v4 ==== 2004+0+0 (1869372186 0 0) 0x55a34de6f000 con 0x55a3170b7000 > -7002> 2018-07-11 07:33:24.645554 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:24.395979) v4 -- 0x55a34de79c00 con 0 > -7001> 2018-07-11 07:33:24.645560 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b8800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1782 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:33:24.395979) v4 > -7000> 2018-07-11 07:33:24.645586 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.8 10.8.128.108:0/11967 1782 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:24.395979) v4 ==== 2004+0+0 (1869372186 0 0) 0x55a324474c00 con 0x55a3170b8800 > -6999> 2018-07-11 07:33:24.645597 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:24.395979) v4 -- 0x55a3331d9e00 con 0 > -6998> 2018-07-11 07:33:24.823957 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15287 conn(0x55a311b75000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1825 0x55a348fbc400 osd_ping(ping e494 stamp 2018-07-11 07:33:26.037077) v4 > -6997> 2018-07-11 07:33:24.823979 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.4 10.8.128.113:0/15287 1825 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:26.037077) v4 ==== 2004+0+0 (4271394078 0 0) 0x55a348fbc400 con 0x55a311b75000 > -6996> 2018-07-11 07:33:24.823991 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:26.037077) v4 -- 0x55a31fca5400 con 0 > -6995> 2018-07-11 07:33:24.824001 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15287 conn(0x55a311b72000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1825 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:33:26.037077) v4 > -6994> 2018-07-11 07:33:24.824022 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.4 10.8.128.113:0/15287 1825 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:26.037077) v4 ==== 2004+0+0 (4271394078 0 0) 0x55a332e79a00 con 0x55a311b72000 > -6993> 2018-07-11 07:33:24.824038 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:26.037077) v4 -- 0x55a33047e200 con 0 > -6992> 2018-07-11 07:33:25.013083 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15263 conn(0x55a321664000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1828 0x55a333f6e800 osd_ping(ping e494 stamp 2018-07-11 07:33:26.226206) v4 > -6991> 2018-07-11 07:33:25.013104 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.5 10.8.128.113:0/15263 1828 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:26.226206) v4 ==== 2004+0+0 (2152460321 0 0) 0x55a333f6e800 con 0x55a321664000 > -6990> 2018-07-11 07:33:25.013115 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:26.226206) v4 -- 0x55a348fbc000 con 0 > -6989> 2018-07-11 07:33:25.013179 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15263 conn(0x55a321667000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1828 0x55a348fbc400 osd_ping(ping e494 stamp 2018-07-11 07:33:26.226206) v4 > -6988> 2018-07-11 07:33:25.013193 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.5 10.8.128.113:0/15263 1828 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:26.226206) v4 ==== 2004+0+0 (2152460321 0 0) 0x55a348fbc400 con 0x55a321667000 > -6987> 2018-07-11 07:33:25.013203 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:26.226206) v4 -- 0x55a348fbc800 con 0 > -6986> 2018-07-11 07:33:25.239687 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a3126898c0 con 0 > -6985> 2018-07-11 07:33:25.255344 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5073 0x55a32fdd4a00 replica scrub(pg: 6.15,from:0'0,to:430'106457,epoch:494/492,start:6:a9b970ef:::benchmark_data_magna108_35617_object294973:0,end:6:a9ba90f3:::benchmark_data_magna108_35617_object2504:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -6984> 2018-07-11 07:33:25.255374 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5073 ==== replica scrub(pg: 6.15,from:0'0,to:430'106457,epoch:494/492,start:6:a9b970ef:::benchmark_data_magna108_35617_object294973:0,end:6:a9ba90f3:::benchmark_data_magna108_35617_object2504:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 216+0+0 (2662484525 0 0) 0x55a32fdd4a00 con 0x55a317019000 > -6983> 2018-07-11 07:33:25.460641 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12651 conn(0x55a3216a0800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1834 0x55a333f6e800 osd_ping(ping e494 stamp 2018-07-11 07:33:25.495868) v4 > -6982> 2018-07-11 07:33:25.460650 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12651 conn(0x55a32169f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1834 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:33:25.495868) v4 > -6981> 2018-07-11 07:33:25.460679 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.2 10.8.128.106:0/12651 1834 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:25.495868) v4 ==== 2004+0+0 (3582509063 0 0) 0x55a333f6e800 con 0x55a3216a0800 > -6980> 2018-07-11 07:33:25.460683 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.2 10.8.128.106:0/12651 1834 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:25.495868) v4 ==== 2004+0+0 (3582509063 0 0) 0x55a324474c00 con 0x55a32169f000 > -6979> 2018-07-11 07:33:25.460692 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:25.495868) v4 -- 0x55a348fbc800 con 0 > -6978> 2018-07-11 07:33:25.460731 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:25.495868) v4 -- 0x55a33047e200 con 0 > -6977> 2018-07-11 07:33:25.772629 7f355d88e700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.107:0/12617 conn(0x55a32447f800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1804 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:33:25.772301) v4 > -6976> 2018-07-11 07:33:25.772636 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.107:0/12617 conn(0x55a32447e000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1804 0x55a348fbc400 osd_ping(ping e494 stamp 2018-07-11 07:33:25.772301) v4 > -6975> 2018-07-11 07:33:25.772651 7f355d88e700 1 -- 10.8.128.107:6803/12665 <== osd.9 10.8.128.107:0/12617 1804 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:25.772301) v4 ==== 2004+0+0 (1939172292 0 0) 0x55a34de78600 con 0x55a32447f800 > -6974> 2018-07-11 07:33:25.772653 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.9 10.8.128.107:0/12617 1804 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:25.772301) v4 ==== 2004+0+0 (1939172292 0 0) 0x55a348fbc400 con 0x55a32447e000 > -6973> 2018-07-11 07:33:25.772662 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:25.772301) v4 -- 0x55a348fbc600 con 0 > -6972> 2018-07-11 07:33:25.772712 7f355d88e700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:25.772301) v4 -- 0x55a314e7ba00 con 0 > -6971> 2018-07-11 07:33:25.971663 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12650 conn(0x55a311b68800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1847 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:33:26.006926) v4 > -6970> 2018-07-11 07:33:25.971663 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12650 conn(0x55a311b67000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1847 0x55a333f6e800 osd_ping(ping e494 stamp 2018-07-11 07:33:26.006926) v4 > -6969> 2018-07-11 07:33:25.971696 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.3 10.8.128.106:0/12650 1847 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:26.006926) v4 ==== 2004+0+0 (3567058163 0 0) 0x55a34de6f000 con 0x55a311b68800 > -6968> 2018-07-11 07:33:25.971699 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.3 10.8.128.106:0/12650 1847 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:26.006926) v4 ==== 2004+0+0 (3567058163 0 0) 0x55a333f6e800 con 0x55a311b67000 > -6967> 2018-07-11 07:33:25.971711 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:26.006926) v4 -- 0x55a32e456a00 con 0 > -6966> 2018-07-11 07:33:25.971738 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:26.006926) v4 -- 0x55a312363200 con 0 > -6965> 2018-07-11 07:33:26.061924 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a315000d80 con 0 > -6964> 2018-07-11 07:33:26.086011 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5074 0x55a32fdd4c80 replica scrub(pg: 6.15,from:0'0,to:430'106214,epoch:494/492,start:6:a9ba90f3:::benchmark_data_magna108_35617_object2504:0,end:6:a9bb7406:::benchmark_data_magna108_35617_object873246:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -6963> 2018-07-11 07:33:26.086034 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5074 ==== replica scrub(pg: 6.15,from:0'0,to:430'106214,epoch:494/492,start:6:a9ba90f3:::benchmark_data_magna108_35617_object2504:0,end:6:a9bb7406:::benchmark_data_magna108_35617_object873246:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 216+0+0 (3051741244 0 0) 0x55a32fdd4c80 con 0x55a317019000 > -6962> 2018-07-11 07:33:26.346071 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b7000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1783 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:33:26.096495) v4 > -6961> 2018-07-11 07:33:26.346080 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b8800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1783 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:33:26.096495) v4 > -6960> 2018-07-11 07:33:26.346096 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.8 10.8.128.108:0/11967 1783 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:26.096495) v4 ==== 2004+0+0 (3694031188 0 0) 0x55a34de78600 con 0x55a3170b7000 > -6959> 2018-07-11 07:33:26.346097 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.8 10.8.128.108:0/11967 1783 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:26.096495) v4 ==== 2004+0+0 (3694031188 0 0) 0x55a332e79a00 con 0x55a3170b8800 > -6958> 2018-07-11 07:33:26.346107 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:26.096495) v4 -- 0x55a32e456a00 con 0 > -6957> 2018-07-11 07:33:26.346133 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:26.096495) v4 -- 0x55a33047e200 con 0 > -6956> 2018-07-11 07:33:26.623561 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12677 conn(0x55a32169c000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1852 0x55a348fbc400 osd_ping(ping e494 stamp 2018-07-11 07:33:26.658827) v4 > -6955> 2018-07-11 07:33:26.623586 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.1 10.8.128.106:0/12677 1852 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:26.658827) v4 ==== 2004+0+0 (992171326 0 0) 0x55a348fbc400 con 0x55a32169c000 > -6954> 2018-07-11 07:33:26.623586 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12677 conn(0x55a32169d800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1852 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:33:26.658827) v4 > -6953> 2018-07-11 07:33:26.623604 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.1 10.8.128.106:0/12677 1852 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:26.658827) v4 ==== 2004+0+0 (992171326 0 0) 0x55a34de6f000 con 0x55a32169d800 > -6952> 2018-07-11 07:33:26.623605 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:26.658827) v4 -- 0x55a325c0ee00 con 0 > -6951> 2018-07-11 07:33:26.623638 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:26.658827) v4 -- 0x55a332e85600 con 0 > -6950> 2018-07-11 07:33:26.643771 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.115:0/13323 conn(0x55a321665800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1842 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:33:26.980897) v4 > -6949> 2018-07-11 07:33:26.643796 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.6 10.8.128.115:0/13323 1842 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:26.980897) v4 ==== 2004+0+0 (2926835997 0 0) 0x55a324474c00 con 0x55a321665800 > -6948> 2018-07-11 07:33:26.643811 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:26.980897) v4 -- 0x55a34866ae00 con 0 > -6947> 2018-07-11 07:33:26.643867 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.115:0/13323 conn(0x55a321668800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1842 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:33:26.980897) v4 > -6946> 2018-07-11 07:33:26.643887 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.6 10.8.128.115:0/13323 1842 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:26.980897) v4 ==== 2004+0+0 (2926835997 0 0) 0x55a324474c00 con 0x55a321668800 > -6945> 2018-07-11 07:33:26.643898 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:26.980897) v4 -- 0x55a33047e000 con 0 > -6944> 2018-07-11 07:33:26.701185 7f354e0ae700 1 bluefs _allocate failed to allocate 0x4800000 on bdev 1, free 0x2fe000; fallback to bdev 2 > -6943> 2018-07-11 07:33:26.713752 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15263 conn(0x55a321667000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1829 0x55a312363200 osd_ping(ping e494 stamp 2018-07-11 07:33:27.926775) v4 > -6942> 2018-07-11 07:33:26.713775 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.5 10.8.128.113:0/15263 1829 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:27.926775) v4 ==== 2004+0+0 (2564675538 0 0) 0x55a312363200 con 0x55a321667000 > -6941> 2018-07-11 07:33:26.713787 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:27.926775) v4 -- 0x55a312363400 con 0 > -6940> 2018-07-11 07:33:26.713833 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15263 conn(0x55a321664000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1829 0x55a312363200 osd_ping(ping e494 stamp 2018-07-11 07:33:27.926775) v4 > -6939> 2018-07-11 07:33:26.713847 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.5 10.8.128.113:0/15263 1829 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:27.926775) v4 ==== 2004+0+0 (2564675538 0 0) 0x55a312363200 con 0x55a321664000 > -6938> 2018-07-11 07:33:26.713855 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:27.926775) v4 -- 0x55a3265d6600 con 0 > -6937> 2018-07-11 07:33:26.744107 7f35498a5700 10 monclient: tick > -6936> 2018-07-11 07:33:26.744121 7f35498a5700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2018-07-11 07:32:56.744119) > -6935> 2018-07-11 07:33:27.251975 7f35488a3700 1 -- 10.8.128.107:6800/12665 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55a32ebb98c0 con 0 > -6934> 2018-07-11 07:33:27.252046 7f35488a3700 1 -- 10.8.128.107:6800/12665 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55a3170ea000 con 0 > -6933> 2018-07-11 07:33:27.723939 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12677 conn(0x55a32169d800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1853 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:33:27.759231) v4 > -6932> 2018-07-11 07:33:27.723962 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.1 10.8.128.106:0/12677 1853 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:27.759231) v4 ==== 2004+0+0 (940508747 0 0) 0x55a34de78600 con 0x55a32169d800 > -6931> 2018-07-11 07:33:27.723972 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:27.759231) v4 -- 0x55a34dec7200 con 0 > -6930> 2018-07-11 07:33:27.724003 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12677 conn(0x55a32169c000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1853 0x55a312363200 osd_ping(ping e494 stamp 2018-07-11 07:33:27.759231) v4 > -6929> 2018-07-11 07:33:27.724024 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.1 10.8.128.106:0/12677 1853 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:27.759231) v4 ==== 2004+0+0 (940508747 0 0) 0x55a312363200 con 0x55a32169c000 > -6928> 2018-07-11 07:33:27.724035 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:27.759231) v4 -- 0x55a3265d6600 con 0 > -6927> 2018-07-11 07:33:27.735083 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 58] Generated table #36575: 2638241 keys, 74546539 bytes > -6926> 2018-07-11 07:33:27.735114 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294407735100, "cf_name": "default", "job": 58, "event": "table_file_creation", "file_number": 36575, "file_size": 74546539, "table_properties": {"data_size": 67112151, "index_size": 690215, "filter_size": 6743183, "raw_key_size": 76215852, "raw_average_key_size": 28, "raw_value_size": 31365755, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} > -6925> 2018-07-11 07:33:27.747921 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a3128f7440 con 0 > -6924> 2018-07-11 07:33:27.763864 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5075 0x55a32fdd4f00 replica scrub(pg: 6.15,from:0'0,to:430'106477,epoch:494/492,start:6:a9bb7406:::benchmark_data_magna108_35617_object873246:0,end:6:a9bc5a1c:::benchmark_data_magna108_35617_object638504:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -6923> 2018-07-11 07:33:27.763899 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5075 ==== replica scrub(pg: 6.15,from:0'0,to:430'106477,epoch:494/492,start:6:a9bb7406:::benchmark_data_magna108_35617_object873246:0,end:6:a9bc5a1c:::benchmark_data_magna108_35617_object638504:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (2657429482 0 0) 0x55a32fdd4f00 con 0x55a317019000 > -6922> 2018-07-11 07:33:27.960983 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15885 conn(0x55a3170b5800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1849 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:33:28.584608) v4 > -6921> 2018-07-11 07:33:27.960996 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15885 conn(0x55a311b76800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1849 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:33:28.584608) v4 > -6920> 2018-07-11 07:33:27.961006 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.7 10.8.128.114:0/15885 1849 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:28.584608) v4 ==== 2004+0+0 (2479921144 0 0) 0x55a324474c00 con 0x55a3170b5800 > -6919> 2018-07-11 07:33:27.961013 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.7 10.8.128.114:0/15885 1849 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:28.584608) v4 ==== 2004+0+0 (2479921144 0 0) 0x55a34de6f000 con 0x55a311b76800 > -6918> 2018-07-11 07:33:27.961023 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:28.584608) v4 -- 0x55a33047e000 con 0 > -6917> 2018-07-11 07:33:27.961056 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:28.584608) v4 -- 0x55a34dec7200 con 0 > -6916> 2018-07-11 07:33:28.414085 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15263 conn(0x55a321664000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1830 0x55a348fbc400 osd_ping(ping e494 stamp 2018-07-11 07:33:29.627342) v4 > -6915> 2018-07-11 07:33:28.414116 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.5 10.8.128.113:0/15263 1830 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:29.627342) v4 ==== 2004+0+0 (2818218589 0 0) 0x55a348fbc400 con 0x55a321664000 > -6914> 2018-07-11 07:33:28.414132 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:29.627342) v4 -- 0x55a3265d6600 con 0 > -6913> 2018-07-11 07:33:28.414192 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15263 conn(0x55a321667000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1830 0x55a348fbc400 osd_ping(ping e494 stamp 2018-07-11 07:33:29.627342) v4 > -6912> 2018-07-11 07:33:28.414215 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.5 10.8.128.113:0/15263 1830 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:29.627342) v4 ==== 2004+0+0 (2818218589 0 0) 0x55a348fbc400 con 0x55a321667000 > -6911> 2018-07-11 07:33:28.414225 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:29.627342) v4 -- 0x55a312363c00 con 0 > -6910> 2018-07-11 07:33:28.459377 7f3543899700 5 osd.10 494 heartbeat: osd_stat(783 GB used, 147 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,0,0,1]) > -6909> 2018-07-11 07:33:28.459392 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6802/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:33:28.459390) v4 -- 0x55a324967200 con 0 > -6908> 2018-07-11 07:33:28.459419 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6803/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:33:28.459390) v4 -- 0x55a329ac3e00 con 0 > -6907> 2018-07-11 07:33:28.459436 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6806/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:33:28.459390) v4 -- 0x55a3265d7e00 con 0 > -6906> 2018-07-11 07:33:28.459451 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6807/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:33:28.459390) v4 -- 0x55a31fca4000 con 0 > -6905> 2018-07-11 07:33:28.459471 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6808/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:33:28.459390) v4 -- 0x55a33610ea00 con 0 > -6904> 2018-07-11 07:33:28.459492 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6810/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:33:28.459390) v4 -- 0x55a329322c00 con 0 > -6903> 2018-07-11 07:33:28.459515 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6809/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:33:28.459390) v4 -- 0x55a312363600 con 0 > -6902> 2018-07-11 07:33:28.459533 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6811/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:33:28.459390) v4 -- 0x55a31fca4400 con 0 > -6901> 2018-07-11 07:33:28.459553 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6810/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:33:28.459390) v4 -- 0x55a31fca4200 con 0 > -6900> 2018-07-11 07:33:28.459575 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6811/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:33:28.459390) v4 -- 0x55a31fca4a00 con 0 > -6899> 2018-07-11 07:33:28.459599 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6806/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:33:28.459390) v4 -- 0x55a31fca4800 con 0 > -6898> 2018-07-11 07:33:28.459622 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6807/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:33:28.459390) v4 -- 0x55a31fca5200 con 0 > -6897> 2018-07-11 07:33:28.459646 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6802/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:33:28.459390) v4 -- 0x55a31fca4600 con 0 > -6896> 2018-07-11 07:33:28.459669 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6803/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:33:28.459390) v4 -- 0x55a31fca5800 con 0 > -6895> 2018-07-11 07:33:28.459710 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6806/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:33:28.459390) v4 -- 0x55a31fca5000 con 0 > -6894> 2018-07-11 07:33:28.459761 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6807/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:33:28.459390) v4 -- 0x55a31fca4c00 con 0 > -6893> 2018-07-11 07:33:28.459789 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6802/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:33:28.459390) v4 -- 0x55a31fca5c00 con 0 > -6892> 2018-07-11 07:33:28.459782 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6803/15255 conn(0x55a314e1d800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1832 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:28.459390) v4 > -6891> 2018-07-11 07:33:28.459812 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6803/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:33:28.459390) v4 -- 0x55a31fca5600 con 0 > -6890> 2018-07-11 07:33:28.459813 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6803/15255 1832 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:28.459390) v4 ==== 2004+0+0 (3084288556 0 0) 0x55a332e79a00 con 0x55a314e1d800 > -6889> 2018-07-11 07:33:28.459830 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6806/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:33:28.459390) v4 -- 0x55a31fca5a00 con 0 > -6888> 2018-07-11 07:33:28.459848 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6807/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:33:28.459390) v4 -- 0x55a31fca5e00 con 0 > -6887> 2018-07-11 07:33:28.459902 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6802/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:33:28.459390) v4 -- 0x55a3203f6a00 con 0 > -6886> 2018-07-11 07:33:28.459896 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6802/15255 conn(0x55a314e1c000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1832 0x55a348fbc400 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:28.459390) v4 > -6885> 2018-07-11 07:33:28.459920 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6802/15255 1832 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:28.459390) v4 ==== 2004+0+0 (3084288556 0 0) 0x55a348fbc400 con 0x55a314e1c000 > -6884> 2018-07-11 07:33:28.459924 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6803/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:33:28.459390) v4 -- 0x55a31bbec200 con 0 > -6883> 2018-07-11 07:33:28.459964 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6806/12677 conn(0x55a314e84000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1832 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:28.459390) v4 > -6882> 2018-07-11 07:33:28.459985 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6806/12677 1832 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:28.459390) v4 ==== 2004+0+0 (3084288556 0 0) 0x55a34de6f000 con 0x55a314e84000 > -6881> 2018-07-11 07:33:28.460003 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6808/12651 conn(0x55a314e96000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1832 0x55a348fbc400 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:28.459390) v4 > -6880> 2018-07-11 07:33:28.460024 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6808/12651 1832 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:28.459390) v4 ==== 2004+0+0 (3084288556 0 0) 0x55a348fbc400 con 0x55a314e96000 > -6879> 2018-07-11 07:33:28.460048 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6809/12650 conn(0x55a314e1f000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1832 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:28.459390) v4 > -6878> 2018-07-11 07:33:28.460067 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6809/12650 1832 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:28.459390) v4 ==== 2004+0+0 (3084288556 0 0) 0x55a34de78600 con 0x55a314e1f000 > -6877> 2018-07-11 07:33:28.460064 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6807/15263 conn(0x55a314e9a800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1832 0x55a348fbc400 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:28.459390) v4 > -6876> 2018-07-11 07:33:28.460066 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6810/12651 conn(0x55a314e97800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1832 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:28.459390) v4 > -6875> 2018-07-11 07:33:28.460080 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6807/15263 1832 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:28.459390) v4 ==== 2004+0+0 (3084288556 0 0) 0x55a348fbc400 con 0x55a314e9a800 > -6874> 2018-07-11 07:33:28.460088 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6810/12651 1832 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:28.459390) v4 ==== 2004+0+0 (3084288556 0 0) 0x55a324474c00 con 0x55a314e97800 > -6873> 2018-07-11 07:33:28.460102 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6806/15263 conn(0x55a314e99000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1832 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:28.459390) v4 > -6872> 2018-07-11 07:33:28.460115 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6806/15263 1832 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:28.459390) v4 ==== 2004+0+0 (3084288556 0 0) 0x55a34de78600 con 0x55a314e99000 > -6871> 2018-07-11 07:33:28.460118 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6810/15287 conn(0x55a314e03000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1832 0x55a348fbc400 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:28.459390) v4 > -6870> 2018-07-11 07:33:28.460131 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6810/15287 1832 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:28.459390) v4 ==== 2004+0+0 (3084288556 0 0) 0x55a348fbc400 con 0x55a314e03000 > -6869> 2018-07-11 07:33:28.460138 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6811/15287 conn(0x55a314e04800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1832 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:28.459390) v4 > -6868> 2018-07-11 07:33:28.460151 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6803/13323 conn(0x55a314e16800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1832 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:28.459390) v4 > -6867> 2018-07-11 07:33:28.460155 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6811/15287 1832 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:28.459390) v4 ==== 2004+0+0 (3084288556 0 0) 0x55a324474c00 con 0x55a314e04800 > -6866> 2018-07-11 07:33:28.460164 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6803/13323 1832 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:28.459390) v4 ==== 2004+0+0 (3084288556 0 0) 0x55a34de78600 con 0x55a314e16800 > -6865> 2018-07-11 07:33:28.460163 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6807/12677 conn(0x55a314e85800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1832 0x55a348fbc400 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:28.459390) v4 > -6864> 2018-07-11 07:33:28.460178 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6807/12677 1832 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:28.459390) v4 ==== 2004+0+0 (3084288556 0 0) 0x55a348fbc400 con 0x55a314e85800 > -6863> 2018-07-11 07:33:28.460191 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6807/12617 conn(0x55a314e27800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1832 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:28.459390) v4 > -6862> 2018-07-11 07:33:28.460204 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6807/12617 1832 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:28.459390) v4 ==== 2004+0+0 (3084288556 0 0) 0x55a34de78600 con 0x55a314e27800 > -6861> 2018-07-11 07:33:28.460198 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6802/13323 conn(0x55a314e15000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1832 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:28.459390) v4 > -6860> 2018-07-11 07:33:28.460212 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6811/12650 conn(0x55a314e20800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1832 0x55a348fbc400 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:28.459390) v4 > -6859> 2018-07-11 07:33:28.460224 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6811/12650 1832 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:28.459390) v4 ==== 2004+0+0 (3084288556 0 0) 0x55a348fbc400 con 0x55a314e20800 > -6858> 2018-07-11 07:33:28.460231 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6802/13323 1832 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:28.459390) v4 ==== 2004+0+0 (3084288556 0 0) 0x55a324474c00 con 0x55a314e15000 > -6857> 2018-07-11 07:33:28.460243 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6802/11967 conn(0x55a314e12000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1832 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:28.459390) v4 > -6856> 2018-07-11 07:33:28.460266 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6802/11967 1832 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:28.459390) v4 ==== 2004+0+0 (3084288556 0 0) 0x55a34de78600 con 0x55a314e12000 > -6855> 2018-07-11 07:33:28.460282 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6806/12617 conn(0x55a314e26000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1832 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:28.459390) v4 > -6854> 2018-07-11 07:33:28.460288 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6807/15885 conn(0x55a314e01800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1832 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:28.459390) v4 > -6853> 2018-07-11 07:33:28.460304 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6806/12617 1832 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:28.459390) v4 ==== 2004+0+0 (3084288556 0 0) 0x55a324474c00 con 0x55a314e26000 > -6852> 2018-07-11 07:33:28.460308 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6807/15885 1832 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:28.459390) v4 ==== 2004+0+0 (3084288556 0 0) 0x55a34de78600 con 0x55a314e01800 > -6851> 2018-07-11 07:33:28.460306 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6803/11967 conn(0x55a314e13800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1832 0x55a348fbc400 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:28.459390) v4 > -6850> 2018-07-11 07:33:28.460342 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6803/11967 1832 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:28.459390) v4 ==== 2004+0+0 (3084288556 0 0) 0x55a348fbc400 con 0x55a314e13800 > -6849> 2018-07-11 07:33:28.460396 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6803/15919 conn(0x55a314e88800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1832 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:28.459390) v4 > -6848> 2018-07-11 07:33:28.460410 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6806/15885 conn(0x55a314e00000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1832 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:28.459390) v4 > -6847> 2018-07-11 07:33:28.460412 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6803/15919 1832 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:28.459390) v4 ==== 2004+0+0 (3084288556 0 0) 0x55a34de78600 con 0x55a314e88800 > -6846> 2018-07-11 07:33:28.460428 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6806/15885 1832 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:28.459390) v4 ==== 2004+0+0 (3084288556 0 0) 0x55a324474c00 con 0x55a314e00000 > -6845> 2018-07-11 07:33:28.460550 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6802/15919 conn(0x55a314e87000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1832 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:28.459390) v4 > -6844> 2018-07-11 07:33:28.460565 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6802/15919 1832 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:28.459390) v4 ==== 2004+0+0 (3084288556 0 0) 0x55a324474c00 con 0x55a314e87000 > -6843> 2018-07-11 07:33:28.670814 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a3112ccfc0 con 0 > -6842> 2018-07-11 07:33:28.673085 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.107:0/12617 conn(0x55a32447e000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1805 0x55a312363200 osd_ping(ping e494 stamp 2018-07-11 07:33:28.672758) v4 > -6841> 2018-07-11 07:33:28.673089 7f355d88e700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.107:0/12617 conn(0x55a32447f800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1805 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:33:28.672758) v4 > -6840> 2018-07-11 07:33:28.673117 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.9 10.8.128.107:0/12617 1805 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:28.672758) v4 ==== 2004+0+0 (2366214090 0 0) 0x55a312363200 con 0x55a32447e000 > -6839> 2018-07-11 07:33:28.673124 7f355d88e700 1 -- 10.8.128.107:6803/12665 <== osd.9 10.8.128.107:0/12617 1805 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:28.672758) v4 ==== 2004+0+0 (2366214090 0 0) 0x55a34de78600 con 0x55a32447f800 > -6838> 2018-07-11 07:33:28.673139 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:28.672758) v4 -- 0x55a312363c00 con 0 > -6837> 2018-07-11 07:33:28.673164 7f355d88e700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:28.672758) v4 -- 0x55a312363600 con 0 > -6836> 2018-07-11 07:33:28.702500 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5076 0x55a32fdd5180 replica scrub(pg: 6.15,from:0'0,to:430'106630,epoch:494/492,start:6:a9bc5a1c:::benchmark_data_magna108_35617_object638504:0,end:6:a9bd66e5:::benchmark_data_magna108_35617_object492704:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -6835> 2018-07-11 07:33:28.702522 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5076 ==== replica scrub(pg: 6.15,from:0'0,to:430'106630,epoch:494/492,start:6:a9bc5a1c:::benchmark_data_magna108_35617_object638504:0,end:6:a9bd66e5:::benchmark_data_magna108_35617_object492704:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (4263450137 0 0) 0x55a32fdd5180 con 0x55a317019000 > -6834> 2018-07-11 07:33:29.252059 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15919 conn(0x55a3170b4000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1835 0x55a312363200 osd_ping(ping e494 stamp 2018-07-11 07:33:29.875657) v4 > -6833> 2018-07-11 07:33:29.252090 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.11 10.8.128.114:0/15919 1835 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:29.875657) v4 ==== 2004+0+0 (3066970230 0 0) 0x55a312363200 con 0x55a3170b4000 > -6832> 2018-07-11 07:33:29.252090 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15919 conn(0x55a317074800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1835 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:33:29.875657) v4 > -6831> 2018-07-11 07:33:29.252106 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:29.875657) v4 -- 0x55a312363c00 con 0 > -6830> 2018-07-11 07:33:29.252110 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.11 10.8.128.114:0/15919 1835 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:29.875657) v4 ==== 2004+0+0 (3066970230 0 0) 0x55a324474c00 con 0x55a317074800 > -6829> 2018-07-11 07:33:29.252141 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:29.875657) v4 -- 0x55a31fca4a00 con 0 > -6828> 2018-07-11 07:33:29.423913 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15255 conn(0x55a311b65800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1857 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:33:30.637149) v4 > -6827> 2018-07-11 07:33:29.423936 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.0 10.8.128.113:0/15255 1857 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:30.637149) v4 ==== 2004+0+0 (3047995155 0 0) 0x55a332e79a00 con 0x55a311b65800 > -6826> 2018-07-11 07:33:29.423945 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:30.637149) v4 -- 0x55a31fca4a00 con 0 > -6825> 2018-07-11 07:33:29.423954 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15255 conn(0x55a311b64000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1857 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:33:30.637149) v4 > -6824> 2018-07-11 07:33:29.423977 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.0 10.8.128.113:0/15255 1857 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:30.637149) v4 ==== 2004+0+0 (3047995155 0 0) 0x55a34de6f000 con 0x55a311b64000 > -6823> 2018-07-11 07:33:29.423989 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:30.637149) v4 -- 0x55a3265d7e00 con 0 > -6822> 2018-07-11 07:33:29.472123 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12650 conn(0x55a311b68800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1848 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:33:29.507463) v4 > -6821> 2018-07-11 07:33:29.472123 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12650 conn(0x55a311b67000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1848 0x55a348fbc400 osd_ping(ping e494 stamp 2018-07-11 07:33:29.507463) v4 > -6820> 2018-07-11 07:33:29.472152 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.3 10.8.128.106:0/12650 1848 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:29.507463) v4 ==== 2004+0+0 (1538809711 0 0) 0x55a34de78600 con 0x55a311b68800 > -6819> 2018-07-11 07:33:29.472161 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.3 10.8.128.106:0/12650 1848 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:29.507463) v4 ==== 2004+0+0 (1538809711 0 0) 0x55a348fbc400 con 0x55a311b67000 > -6818> 2018-07-11 07:33:29.472171 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:29.507463) v4 -- 0x55a34dec7200 con 0 > -6817> 2018-07-11 07:33:29.472199 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:29.507463) v4 -- 0x55a3265d6600 con 0 > -6816> 2018-07-11 07:33:29.524277 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15287 conn(0x55a311b72000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1826 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:33:30.737509) v4 > -6815> 2018-07-11 07:33:29.524297 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.4 10.8.128.113:0/15287 1826 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:30.737509) v4 ==== 2004+0+0 (1075763555 0 0) 0x55a324474c00 con 0x55a311b72000 > -6814> 2018-07-11 07:33:29.524306 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:30.737509) v4 -- 0x55a329322c00 con 0 > -6813> 2018-07-11 07:33:29.524317 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15287 conn(0x55a311b75000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1826 0x55a312363200 osd_ping(ping e494 stamp 2018-07-11 07:33:30.737509) v4 > -6812> 2018-07-11 07:33:29.524338 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.4 10.8.128.113:0/15287 1826 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:30.737509) v4 ==== 2004+0+0 (1075763555 0 0) 0x55a312363200 con 0x55a311b75000 > -6811> 2018-07-11 07:33:29.524349 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:30.737509) v4 -- 0x55a3265d6c00 con 0 > -6810> 2018-07-11 07:33:29.661478 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15885 conn(0x55a311b76800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1850 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:33:30.285060) v4 > -6809> 2018-07-11 07:33:29.661501 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.7 10.8.128.114:0/15885 1850 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:30.285060) v4 ==== 2004+0+0 (306299892 0 0) 0x55a34de6f000 con 0x55a311b76800 > -6808> 2018-07-11 07:33:29.661510 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:30.285060) v4 -- 0x55a344e84000 con 0 > -6807> 2018-07-11 07:33:29.661515 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15885 conn(0x55a3170b5800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1850 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:33:30.285060) v4 > -6806> 2018-07-11 07:33:29.661541 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.7 10.8.128.114:0/15885 1850 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:30.285060) v4 ==== 2004+0+0 (306299892 0 0) 0x55a332e79a00 con 0x55a3170b5800 > -6805> 2018-07-11 07:33:29.661554 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:30.285060) v4 -- 0x55a329ac3e00 con 0 > -6804> 2018-07-11 07:33:29.724535 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a313fb3b00 con 0 > -6803> 2018-07-11 07:33:29.739496 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5077 0x55a32fdd5400 replica scrub(pg: 6.15,from:0'0,to:430'106239,epoch:494/492,start:6:a9bd66e5:::benchmark_data_magna108_35617_object492704:0,end:6:a9be345c:::benchmark_data_magna108.ceph.redhat.com_2478_object244045:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -6802> 2018-07-11 07:33:29.739519 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5077 ==== replica scrub(pg: 6.15,from:0'0,to:430'106239,epoch:494/492,start:6:a9bd66e5:::benchmark_data_magna108_35617_object492704:0,end:6:a9be345c:::benchmark_data_magna108.ceph.redhat.com_2478_object244045:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (3444462655 0 0) 0x55a32fdd5400 con 0x55a317019000 > -6801> 2018-07-11 07:33:29.752479 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15919 conn(0x55a3170b4000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1836 0x55a348fbc400 osd_ping(ping e494 stamp 2018-07-11 07:33:30.376100) v4 > -6800> 2018-07-11 07:33:29.752503 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.11 10.8.128.114:0/15919 1836 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:30.376100) v4 ==== 2004+0+0 (2265723490 0 0) 0x55a348fbc400 con 0x55a3170b4000 > -6799> 2018-07-11 07:33:29.752503 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15919 conn(0x55a317074800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1836 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:33:30.376100) v4 > -6798> 2018-07-11 07:33:29.752515 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:30.376100) v4 -- 0x55a3265d6c00 con 0 > -6797> 2018-07-11 07:33:29.752526 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.11 10.8.128.114:0/15919 1836 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:30.376100) v4 ==== 2004+0+0 (2265723490 0 0) 0x55a324474c00 con 0x55a317074800 > -6796> 2018-07-11 07:33:29.752545 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:30.376100) v4 -- 0x55a329ac3e00 con 0 > -6795> 2018-07-11 07:33:30.183700 7f354e0ae700 1 bluefs _allocate failed to allocate 0x3e00000 on bdev 1, free 0x2fe000; fallback to bdev 2 > -6794> 2018-07-11 07:33:30.760113 7f3543899700 5 osd.10 494 heartbeat: osd_stat(783 GB used, 147 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,0,0,1]) > -6793> 2018-07-11 07:33:30.760127 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6802/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:33:30.760125) v4 -- 0x55a3203f7000 con 0 > -6792> 2018-07-11 07:33:30.760165 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6803/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:33:30.760125) v4 -- 0x55a3203f7e00 con 0 > -6791> 2018-07-11 07:33:30.760183 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6806/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:33:30.760125) v4 -- 0x55a3203f7200 con 0 > -6790> 2018-07-11 07:33:30.760197 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6807/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:33:30.760125) v4 -- 0x55a31bbec800 con 0 > -6789> 2018-07-11 07:33:30.760221 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6808/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:33:30.760125) v4 -- 0x55a31bbec600 con 0 > -6788> 2018-07-11 07:33:30.760232 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6810/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:33:30.760125) v4 -- 0x55a3203f7800 con 0 > -6787> 2018-07-11 07:33:30.760262 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6809/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:33:30.760125) v4 -- 0x55a31bbecc00 con 0 > -6786> 2018-07-11 07:33:30.760277 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6811/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:33:30.760125) v4 -- 0x55a31bbeca00 con 0 > -6785> 2018-07-11 07:33:30.760290 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6810/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:33:30.760125) v4 -- 0x55a31bbec400 con 0 > -6784> 2018-07-11 07:33:30.760299 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6811/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:33:30.760125) v4 -- 0x55a324967c00 con 0 > -6783> 2018-07-11 07:33:30.760313 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6806/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:33:30.760125) v4 -- 0x55a32f9b8a00 con 0 > -6782> 2018-07-11 07:33:30.760326 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6807/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:33:30.760125) v4 -- 0x55a30fa77600 con 0 > -6781> 2018-07-11 07:33:30.760338 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6802/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:33:30.760125) v4 -- 0x55a33be6e600 con 0 > -6780> 2018-07-11 07:33:30.760359 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6803/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:33:30.760125) v4 -- 0x55a343482e00 con 0 > -6779> 2018-07-11 07:33:30.760373 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6806/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:33:30.760125) v4 -- 0x55a32f9b8600 con 0 > -6778> 2018-07-11 07:33:30.760386 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6807/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:33:30.760125) v4 -- 0x55a32f9b8400 con 0 > -6777> 2018-07-11 07:33:30.760403 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6802/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:33:30.760125) v4 -- 0x55a33be6e400 con 0 > -6776> 2018-07-11 07:33:30.760416 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6803/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:33:30.760125) v4 -- 0x55a348fbcc00 con 0 > -6775> 2018-07-11 07:33:30.760437 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6806/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:33:30.760125) v4 -- 0x55a32f9b8200 con 0 > -6774> 2018-07-11 07:33:30.760480 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6807/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:33:30.760125) v4 -- 0x55a343483600 con 0 > -6773> 2018-07-11 07:33:30.760495 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6802/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:33:30.760125) v4 -- 0x55a348fbd400 con 0 > -6772> 2018-07-11 07:33:30.760507 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6803/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:33:30.760125) v4 -- 0x55a348fbce00 con 0 > -6771> 2018-07-11 07:33:30.760606 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6803/15255 conn(0x55a314e1d800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1833 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:30.760125) v4 > -6770> 2018-07-11 07:33:30.760629 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6803/15255 1833 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:30.760125) v4 ==== 2004+0+0 (2037971130 0 0) 0x55a324474c00 con 0x55a314e1d800 > -6769> 2018-07-11 07:33:30.760674 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6802/15255 conn(0x55a314e1c000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1833 0x55a312363200 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:30.760125) v4 > -6768> 2018-07-11 07:33:30.760672 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6806/12617 conn(0x55a314e26000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1833 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:30.760125) v4 > -6767> 2018-07-11 07:33:30.760698 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6802/15255 1833 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:30.760125) v4 ==== 2004+0+0 (2037971130 0 0) 0x55a312363200 con 0x55a314e1c000 > -6766> 2018-07-11 07:33:30.760701 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6806/12617 1833 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:30.760125) v4 ==== 2004+0+0 (2037971130 0 0) 0x55a324474c00 con 0x55a314e26000 > -6765> 2018-07-11 07:33:30.760708 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6807/12617 conn(0x55a314e27800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1833 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:30.760125) v4 > -6764> 2018-07-11 07:33:30.760742 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6808/12651 conn(0x55a314e96000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1833 0x55a312363200 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:30.760125) v4 > -6763> 2018-07-11 07:33:30.760746 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6810/12651 conn(0x55a314e97800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1833 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:30.760125) v4 > -6762> 2018-07-11 07:33:30.760743 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6807/12617 1833 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:30.760125) v4 ==== 2004+0+0 (2037971130 0 0) 0x55a34de78600 con 0x55a314e27800 > -6761> 2018-07-11 07:33:30.760756 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6810/12651 1833 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:30.760125) v4 ==== 2004+0+0 (2037971130 0 0) 0x55a324474c00 con 0x55a314e97800 > -6760> 2018-07-11 07:33:30.760756 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6808/12651 1833 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:30.760125) v4 ==== 2004+0+0 (2037971130 0 0) 0x55a312363200 con 0x55a314e96000 > -6759> 2018-07-11 07:33:30.760789 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6806/15263 conn(0x55a314e99000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1833 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:30.760125) v4 > -6758> 2018-07-11 07:33:30.760795 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6807/12677 conn(0x55a314e85800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1833 0x55a312363200 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:30.760125) v4 > -6757> 2018-07-11 07:33:30.760804 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6806/15263 1833 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:30.760125) v4 ==== 2004+0+0 (2037971130 0 0) 0x55a34de78600 con 0x55a314e99000 > -6756> 2018-07-11 07:33:30.760805 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6807/12677 1833 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:30.760125) v4 ==== 2004+0+0 (2037971130 0 0) 0x55a312363200 con 0x55a314e85800 > -6755> 2018-07-11 07:33:30.760861 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6806/12677 conn(0x55a314e84000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1833 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:30.760125) v4 > -6754> 2018-07-11 07:33:30.760868 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6807/15263 conn(0x55a314e9a800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1833 0x55a312363200 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:30.760125) v4 > -6753> 2018-07-11 07:33:30.760878 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6806/12677 1833 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:30.760125) v4 ==== 2004+0+0 (2037971130 0 0) 0x55a34de78600 con 0x55a314e84000 > -6752> 2018-07-11 07:33:30.760880 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6807/15263 1833 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:30.760125) v4 ==== 2004+0+0 (2037971130 0 0) 0x55a312363200 con 0x55a314e9a800 > -6751> 2018-07-11 07:33:30.760898 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6811/15287 conn(0x55a314e04800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1833 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:30.760125) v4 > -6750> 2018-07-11 07:33:30.760912 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6809/12650 conn(0x55a314e1f000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1833 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:30.760125) v4 > -6749> 2018-07-11 07:33:30.760918 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6811/15287 1833 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:30.760125) v4 ==== 2004+0+0 (2037971130 0 0) 0x55a324474c00 con 0x55a314e04800 > -6748> 2018-07-11 07:33:30.760924 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6809/12650 1833 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:30.760125) v4 ==== 2004+0+0 (2037971130 0 0) 0x55a34de78600 con 0x55a314e1f000 > -6747> 2018-07-11 07:33:30.760941 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6811/12650 conn(0x55a314e20800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1833 0x55a312363200 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:30.760125) v4 > -6746> 2018-07-11 07:33:30.760954 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6811/12650 1833 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:30.760125) v4 ==== 2004+0+0 (2037971130 0 0) 0x55a312363200 con 0x55a314e20800 > -6745> 2018-07-11 07:33:30.760956 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6802/13323 conn(0x55a314e15000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1833 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:30.760125) v4 > -6744> 2018-07-11 07:33:30.760960 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6803/13323 conn(0x55a314e16800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1833 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:30.760125) v4 > -6743> 2018-07-11 07:33:30.760970 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6802/13323 1833 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:30.760125) v4 ==== 2004+0+0 (2037971130 0 0) 0x55a324474c00 con 0x55a314e15000 > -6742> 2018-07-11 07:33:30.760971 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6803/13323 1833 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:30.760125) v4 ==== 2004+0+0 (2037971130 0 0) 0x55a34de78600 con 0x55a314e16800 > -6741> 2018-07-11 07:33:30.760986 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6810/15287 conn(0x55a314e03000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1833 0x55a312363200 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:30.760125) v4 > -6740> 2018-07-11 07:33:30.760996 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6810/15287 1833 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:30.760125) v4 ==== 2004+0+0 (2037971130 0 0) 0x55a312363200 con 0x55a314e03000 > -6739> 2018-07-11 07:33:30.761009 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6802/11967 conn(0x55a314e12000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1833 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:30.760125) v4 > -6738> 2018-07-11 07:33:30.761018 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6802/15919 conn(0x55a314e87000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1833 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:30.760125) v4 > -6737> 2018-07-11 07:33:30.761024 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6803/11967 conn(0x55a314e13800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1833 0x55a312363200 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:30.760125) v4 > -6736> 2018-07-11 07:33:30.761023 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6802/11967 1833 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:30.760125) v4 ==== 2004+0+0 (2037971130 0 0) 0x55a34de78600 con 0x55a314e12000 > -6735> 2018-07-11 07:33:30.761033 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6802/15919 1833 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:30.760125) v4 ==== 2004+0+0 (2037971130 0 0) 0x55a324474c00 con 0x55a314e87000 > -6734> 2018-07-11 07:33:30.761033 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6803/11967 1833 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:30.760125) v4 ==== 2004+0+0 (2037971130 0 0) 0x55a312363200 con 0x55a314e13800 > -6733> 2018-07-11 07:33:30.761079 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6803/15919 conn(0x55a314e88800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1833 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:30.760125) v4 > -6732> 2018-07-11 07:33:30.761092 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6803/15919 1833 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:30.760125) v4 ==== 2004+0+0 (2037971130 0 0) 0x55a34de78600 con 0x55a314e88800 > -6731> 2018-07-11 07:33:30.761089 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6806/15885 conn(0x55a314e00000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1833 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:30.760125) v4 > -6730> 2018-07-11 07:33:30.761102 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6806/15885 1833 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:30.760125) v4 ==== 2004+0+0 (2037971130 0 0) 0x55a324474c00 con 0x55a314e00000 > -6729> 2018-07-11 07:33:30.761123 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6807/15885 conn(0x55a314e01800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1833 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:30.760125) v4 > -6728> 2018-07-11 07:33:30.761135 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6807/15885 1833 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:30.760125) v4 ==== 2004+0+0 (2037971130 0 0) 0x55a34de78600 con 0x55a314e01800 > -6727> 2018-07-11 07:33:31.046636 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b8800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1784 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:33:30.797065) v4 > -6726> 2018-07-11 07:33:31.046657 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.8 10.8.128.108:0/11967 1784 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:30.797065) v4 ==== 2004+0+0 (2675625966 0 0) 0x55a324474c00 con 0x55a3170b8800 > -6725> 2018-07-11 07:33:31.046651 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b7000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1784 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:33:30.797065) v4 > -6724> 2018-07-11 07:33:31.046668 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:30.797065) v4 -- 0x55a324967c00 con 0 > -6723> 2018-07-11 07:33:31.046668 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.8 10.8.128.108:0/11967 1784 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:30.797065) v4 ==== 2004+0+0 (2675625966 0 0) 0x55a34de78600 con 0x55a3170b7000 > -6722> 2018-07-11 07:33:31.046692 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:30.797065) v4 -- 0x55a31bbecc00 con 0 > -6721> 2018-07-11 07:33:31.156743 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 58] Generated table #36576: 577817 keys, 64897190 bytes > -6720> 2018-07-11 07:33:31.156784 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294411156758, "cf_name": "default", "job": 58, "event": "table_file_creation", "file_number": 36576, "file_size": 64897190, "table_properties": {"data_size": 62792336, "index_size": 516483, "filter_size": 1587383, "raw_key_size": 26371968, "raw_average_key_size": 45, "raw_value_size": 44176471, "raw_average_value_size": 76, "num_data_blocks": 14277, "num_entries": 577817, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "81127", "kMergeOperands": "0"}} > -6719> 2018-07-11 07:33:31.224361 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12677 conn(0x55a32169d800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1854 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:33:31.259654) v4 > -6718> 2018-07-11 07:33:31.224370 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12677 conn(0x55a32169c000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1854 0x55a312363200 osd_ping(ping e494 stamp 2018-07-11 07:33:31.259654) v4 > -6717> 2018-07-11 07:33:31.224384 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.1 10.8.128.106:0/12677 1854 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:31.259654) v4 ==== 2004+0+0 (676471639 0 0) 0x55a34de6f000 con 0x55a32169d800 > -6716> 2018-07-11 07:33:31.224387 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.1 10.8.128.106:0/12677 1854 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:31.259654) v4 ==== 2004+0+0 (676471639 0 0) 0x55a312363200 con 0x55a32169c000 > -6715> 2018-07-11 07:33:31.224398 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:31.259654) v4 -- 0x55a3203f7200 con 0 > -6714> 2018-07-11 07:33:31.224427 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:31.259654) v4 -- 0x55a3203f7000 con 0 > -6713> 2018-07-11 07:33:31.361106 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12651 conn(0x55a3216a0800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1835 0x55a348fbc400 osd_ping(ping e494 stamp 2018-07-11 07:33:31.396435) v4 > -6712> 2018-07-11 07:33:31.361117 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12651 conn(0x55a32169f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1835 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:33:31.396435) v4 > -6711> 2018-07-11 07:33:31.361131 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.2 10.8.128.106:0/12651 1835 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:31.396435) v4 ==== 2004+0+0 (2067421818 0 0) 0x55a348fbc400 con 0x55a3216a0800 > -6710> 2018-07-11 07:33:31.361135 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.2 10.8.128.106:0/12651 1835 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:31.396435) v4 ==== 2004+0+0 (2067421818 0 0) 0x55a332e79a00 con 0x55a32169f000 > -6709> 2018-07-11 07:33:31.361141 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:31.396435) v4 -- 0x55a3265d6c00 con 0 > -6708> 2018-07-11 07:33:31.361174 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:31.396435) v4 -- 0x55a3203f7800 con 0 > -6707> 2018-07-11 07:33:31.719609 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a3240aa240 con 0 > -6706> 2018-07-11 07:33:31.731066 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5078 0x55a32fdd5680 replica scrub(pg: 6.15,from:0'0,to:430'105435,epoch:494/492,start:6:a9be345c:::benchmark_data_magna108.ceph.redhat.com_2478_object244045:0,end:6:a9bf0682:::benchmark_data_magna108.ceph.redhat.com_2478_object150497:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -6705> 2018-07-11 07:33:31.731090 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5078 ==== replica scrub(pg: 6.15,from:0'0,to:430'105435,epoch:494/492,start:6:a9be345c:::benchmark_data_magna108.ceph.redhat.com_2478_object244045:0,end:6:a9bf0682:::benchmark_data_magna108.ceph.redhat.com_2478_object150497:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 248+0+0 (3041559460 0 0) 0x55a32fdd5680 con 0x55a317019000 > -6704> 2018-07-11 07:33:31.772540 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12650 conn(0x55a311b67000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1849 0x55a312363200 osd_ping(ping e494 stamp 2018-07-11 07:33:31.807919) v4 > -6703> 2018-07-11 07:33:31.772563 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.3 10.8.128.106:0/12650 1849 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:31.807919) v4 ==== 2004+0+0 (3389514313 0 0) 0x55a312363200 con 0x55a311b67000 > -6702> 2018-07-11 07:33:31.772580 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:31.807919) v4 -- 0x55a3265d6400 con 0 > -6701> 2018-07-11 07:33:31.772611 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12650 conn(0x55a311b68800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1849 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:33:31.807919) v4 > -6700> 2018-07-11 07:33:31.772633 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.3 10.8.128.106:0/12650 1849 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:31.807919) v4 ==== 2004+0+0 (3389514313 0 0) 0x55a34de78600 con 0x55a311b68800 > -6699> 2018-07-11 07:33:31.772648 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:31.807919) v4 -- 0x55a3203f7200 con 0 > -6698> 2018-07-11 07:33:31.944193 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.115:0/13323 conn(0x55a321665800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1843 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:33:32.281368) v4 > -6697> 2018-07-11 07:33:31.944222 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.6 10.8.128.115:0/13323 1843 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:32.281368) v4 ==== 2004+0+0 (1100936580 0 0) 0x55a324474c00 con 0x55a321665800 > -6696> 2018-07-11 07:33:31.944240 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:32.281368) v4 -- 0x55a3203f7800 con 0 > -6695> 2018-07-11 07:33:31.944335 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.115:0/13323 conn(0x55a321668800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1843 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:33:32.281368) v4 > -6694> 2018-07-11 07:33:31.944351 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.6 10.8.128.115:0/13323 1843 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:32.281368) v4 ==== 2004+0+0 (1100936580 0 0) 0x55a332e79a00 con 0x55a321668800 > -6693> 2018-07-11 07:33:31.944361 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:32.281368) v4 -- 0x55a3203f7800 con 0 > -6692> 2018-07-11 07:33:31.990853 7f354e0ae700 1 bluefs _allocate failed to allocate 0x1a00000 on bdev 1, free 0x2fe000; fallback to bdev 2 > -6691> 2018-07-11 07:33:32.252265 7f35488a3700 1 -- 10.8.128.107:6800/12665 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55a3216de000 con 0 > -6690> 2018-07-11 07:33:32.252339 7f35488a3700 1 -- 10.8.128.107:6800/12665 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55a346874840 con 0 > -6689> 2018-07-11 07:33:32.460647 7f3543899700 5 osd.10 494 heartbeat: osd_stat(783 GB used, 147 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,0,0,1]) > -6688> 2018-07-11 07:33:32.460664 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6802/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:33:32.460662) v4 -- 0x55a32f9b8000 con 0 > -6687> 2018-07-11 07:33:32.460689 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6803/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:33:32.460662) v4 -- 0x55a348fbdc00 con 0 > -6686> 2018-07-11 07:33:32.460704 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6806/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:33:32.460662) v4 -- 0x55a324967e00 con 0 > -6685> 2018-07-11 07:33:32.460723 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6807/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:33:32.460662) v4 -- 0x55a348fbd600 con 0 > -6684> 2018-07-11 07:33:32.460735 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6808/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:33:32.460662) v4 -- 0x55a348fbde00 con 0 > -6683> 2018-07-11 07:33:32.460748 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6810/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:33:32.460662) v4 -- 0x55a324967400 con 0 > -6682> 2018-07-11 07:33:32.460760 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6809/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:33:32.460662) v4 -- 0x55a30fa77800 con 0 > -6681> 2018-07-11 07:33:32.460772 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6811/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:33:32.460662) v4 -- 0x55a30fa77a00 con 0 > -6680> 2018-07-11 07:33:32.460786 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6810/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:33:32.460662) v4 -- 0x55a31334ca00 con 0 > -6679> 2018-07-11 07:33:32.460798 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6811/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:33:32.460662) v4 -- 0x55a31334d400 con 0 > -6678> 2018-07-11 07:33:32.460810 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6806/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:33:32.460662) v4 -- 0x55a31bbede00 con 0 > -6677> 2018-07-11 07:33:32.460834 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6807/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:33:32.460662) v4 -- 0x55a31334d200 con 0 > -6676> 2018-07-11 07:33:32.460852 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6802/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:33:32.460662) v4 -- 0x55a31334d800 con 0 > -6675> 2018-07-11 07:33:32.460867 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6803/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:33:32.460662) v4 -- 0x55a31334c400 con 0 > -6674> 2018-07-11 07:33:32.460882 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6806/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:33:32.460662) v4 -- 0x55a31334d600 con 0 > -6673> 2018-07-11 07:33:32.460897 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6807/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:33:32.460662) v4 -- 0x55a31334d000 con 0 > -6672> 2018-07-11 07:33:32.460909 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6802/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:33:32.460662) v4 -- 0x55a32f9b9800 con 0 > -6671> 2018-07-11 07:33:32.460956 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6803/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:33:32.460662) v4 -- 0x55a343480200 con 0 > -6670> 2018-07-11 07:33:32.460974 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6806/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:33:32.460662) v4 -- 0x55a33be6ee00 con 0 > -6669> 2018-07-11 07:33:32.460985 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6807/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:33:32.460662) v4 -- 0x55a32f9b9400 con 0 > -6668> 2018-07-11 07:33:32.460999 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6802/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:33:32.460662) v4 -- 0x55a343480a00 con 0 > -6667> 2018-07-11 07:33:32.461015 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6803/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:33:32.460662) v4 -- 0x55a33be6ec00 con 0 > -6666> 2018-07-11 07:33:32.461095 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6807/12677 conn(0x55a314e85800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1834 0x55a312363200 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:32.460662) v4 > -6665> 2018-07-11 07:33:32.461085 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6806/12677 conn(0x55a314e84000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1834 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:32.460662) v4 > -6664> 2018-07-11 07:33:32.461114 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6807/12677 1834 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:32.460662) v4 ==== 2004+0+0 (4155348757 0 0) 0x55a312363200 con 0x55a314e85800 > -6663> 2018-07-11 07:33:32.461122 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6806/12677 1834 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:32.460662) v4 ==== 2004+0+0 (4155348757 0 0) 0x55a34de6f000 con 0x55a314e84000 > -6662> 2018-07-11 07:33:32.461177 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6803/15255 conn(0x55a314e1d800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1834 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:32.460662) v4 > -6661> 2018-07-11 07:33:32.461185 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6808/12651 conn(0x55a314e96000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1834 0x55a312363200 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:32.460662) v4 > -6660> 2018-07-11 07:33:32.461219 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6803/15255 1834 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:32.460662) v4 ==== 2004+0+0 (4155348757 0 0) 0x55a332e79a00 con 0x55a314e1d800 > -6659> 2018-07-11 07:33:32.461202 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6809/12650 conn(0x55a314e1f000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1834 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:32.460662) v4 > -6658> 2018-07-11 07:33:32.461224 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6808/12651 1834 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:32.460662) v4 ==== 2004+0+0 (4155348757 0 0) 0x55a312363200 con 0x55a314e96000 > -6657> 2018-07-11 07:33:32.461242 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6809/12650 1834 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:32.460662) v4 ==== 2004+0+0 (4155348757 0 0) 0x55a34de6f000 con 0x55a314e1f000 > -6656> 2018-07-11 07:33:32.461268 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6802/15255 conn(0x55a314e1c000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1834 0x55a312363200 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:32.460662) v4 > -6655> 2018-07-11 07:33:32.461288 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6802/15255 1834 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:32.460662) v4 ==== 2004+0+0 (4155348757 0 0) 0x55a312363200 con 0x55a314e1c000 > -6654> 2018-07-11 07:33:32.461300 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6807/12617 conn(0x55a314e27800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1834 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:32.460662) v4 > -6653> 2018-07-11 07:33:32.461313 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6807/12617 1834 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:32.460662) v4 ==== 2004+0+0 (4155348757 0 0) 0x55a34de6f000 con 0x55a314e27800 > -6652> 2018-07-11 07:33:32.461281 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6806/12617 conn(0x55a314e26000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1834 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:32.460662) v4 > -6651> 2018-07-11 07:33:32.461334 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6811/12650 conn(0x55a314e20800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1834 0x55a312363200 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:32.460662) v4 > -6650> 2018-07-11 07:33:32.461342 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6806/12617 1834 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:32.460662) v4 ==== 2004+0+0 (4155348757 0 0) 0x55a332e79a00 con 0x55a314e26000 > -6649> 2018-07-11 07:33:32.461346 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6806/15263 conn(0x55a314e99000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1834 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:32.460662) v4 > -6648> 2018-07-11 07:33:32.461351 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6811/12650 1834 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:32.460662) v4 ==== 2004+0+0 (4155348757 0 0) 0x55a312363200 con 0x55a314e20800 > -6647> 2018-07-11 07:33:32.461362 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6806/15263 1834 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:32.460662) v4 ==== 2004+0+0 (4155348757 0 0) 0x55a34de6f000 con 0x55a314e99000 > -6646> 2018-07-11 07:33:32.461395 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6807/15885 conn(0x55a314e01800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1834 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:32.460662) v4 > -6645> 2018-07-11 07:33:32.461395 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6807/15263 conn(0x55a314e9a800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1834 0x55a312363200 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:32.460662) v4 > -6644> 2018-07-11 07:33:32.461404 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6810/12651 conn(0x55a314e97800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1834 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:32.460662) v4 > -6643> 2018-07-11 07:33:32.461407 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6807/15885 1834 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:32.460662) v4 ==== 2004+0+0 (4155348757 0 0) 0x55a34de6f000 con 0x55a314e01800 > -6642> 2018-07-11 07:33:32.461419 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6807/15263 1834 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:32.460662) v4 ==== 2004+0+0 (4155348757 0 0) 0x55a312363200 con 0x55a314e9a800 > -6641> 2018-07-11 07:33:32.461421 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6810/12651 1834 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:32.460662) v4 ==== 2004+0+0 (4155348757 0 0) 0x55a332e79a00 con 0x55a314e97800 > -6640> 2018-07-11 07:33:32.461445 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6803/13323 conn(0x55a314e16800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1834 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:32.460662) v4 > -6639> 2018-07-11 07:33:32.461458 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6803/13323 1834 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:32.460662) v4 ==== 2004+0+0 (4155348757 0 0) 0x55a34de6f000 con 0x55a314e16800 > -6638> 2018-07-11 07:33:32.461472 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6810/15287 conn(0x55a314e03000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1834 0x55a312363200 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:32.460662) v4 > -6637> 2018-07-11 07:33:32.461489 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6810/15287 1834 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:32.460662) v4 ==== 2004+0+0 (4155348757 0 0) 0x55a312363200 con 0x55a314e03000 > -6636> 2018-07-11 07:33:32.461468 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6811/15287 conn(0x55a314e04800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1834 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:32.460662) v4 > -6635> 2018-07-11 07:33:32.461509 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6811/15287 1834 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:32.460662) v4 ==== 2004+0+0 (4155348757 0 0) 0x55a332e79a00 con 0x55a314e04800 > -6634> 2018-07-11 07:33:32.461519 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6803/15919 conn(0x55a314e88800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1834 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:32.460662) v4 > -6633> 2018-07-11 07:33:32.461542 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6803/15919 1834 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:32.460662) v4 ==== 2004+0+0 (4155348757 0 0) 0x55a34de6f000 con 0x55a314e88800 > -6632> 2018-07-11 07:33:32.461542 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6803/11967 conn(0x55a314e13800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1834 0x55a312363200 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:32.460662) v4 > -6631> 2018-07-11 07:33:32.461562 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6803/11967 1834 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:32.460662) v4 ==== 2004+0+0 (4155348757 0 0) 0x55a312363200 con 0x55a314e13800 > -6630> 2018-07-11 07:33:32.461562 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6802/13323 conn(0x55a314e15000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1834 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:32.460662) v4 > -6629> 2018-07-11 07:33:32.461583 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6802/11967 conn(0x55a314e12000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1834 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:32.460662) v4 > -6628> 2018-07-11 07:33:32.461582 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6802/13323 1834 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:32.460662) v4 ==== 2004+0+0 (4155348757 0 0) 0x55a332e79a00 con 0x55a314e15000 > -6627> 2018-07-11 07:33:32.461598 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6802/11967 1834 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:32.460662) v4 ==== 2004+0+0 (4155348757 0 0) 0x55a34de6f000 con 0x55a314e12000 > -6626> 2018-07-11 07:33:32.461628 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6806/15885 conn(0x55a314e00000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1834 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:32.460662) v4 > -6625> 2018-07-11 07:33:32.461641 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6806/15885 1834 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:32.460662) v4 ==== 2004+0+0 (4155348757 0 0) 0x55a332e79a00 con 0x55a314e00000 > -6624> 2018-07-11 07:33:32.461674 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6802/15919 conn(0x55a314e87000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1834 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:32.460662) v4 > -6623> 2018-07-11 07:33:32.461686 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6802/15919 1834 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:32.460662) v4 ==== 2004+0+0 (4155348757 0 0) 0x55a332e79a00 con 0x55a314e87000 > -6622> 2018-07-11 07:33:32.482082 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 58] Generated table #36577: 28847 keys, 26725247 bytes > -6621> 2018-07-11 07:33:32.482111 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294412482097, "cf_name": "default", "job": 58, "event": "table_file_creation", "file_number": 36577, "file_size": 26725247, "table_properties": {"data_size": 26342419, "index_size": 249430, "filter_size": 132412, "raw_key_size": 2444130, "raw_average_key_size": 84, "raw_value_size": 24129462, "raw_average_value_size": 836, "num_data_blocks": 6161, "num_entries": 28847, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} > -6620> 2018-07-11 07:33:32.482180 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 58] Compacted 1@0 + 10@1 files to L1 => 1226900528 bytes > -6619> 2018-07-11 07:33:32.496606 7f354e0ae700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 65088119716 > -6618> 2018-07-11 07:33:32.496734 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:33:32.496572) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 20 81 446 3641 796 0] max score 7.54, MB/sec: 13.0 rd, 13.0 wr, level 1, files in(1, 10) out(18) MB in(510.1, 660.0) out(1170.1), read-write-amplify(4.6) write-amplify(2.3) OK, records in: 60785930, records dropped: 0 > > -6617> 2018-07-11 07:33:32.496743 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:33:32.496598) EVENT_LOG_v1 {"time_micros": 1531294412496586, "job": 58, "event": "compaction_finished", "compaction_time_micros": 94115358, "output_level": 1, "num_output_files": 18, "total_output_size": 1226900528, "num_input_records": 60785930, "num_output_records": 60785930, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 20, 81, 446, 3641, 796, 0]} > -6616> 2018-07-11 07:33:32.497750 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 59] Compacting 1@1 + 7@2 files to L2, score 7.54 > -6615> 2018-07-11 07:33:32.497763 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 58 Base level 1, inputs: [36565(76MB)], [36551(76MB) 36557(76MB) 36558(76MB) 36559(76MB) 36523(76MB) 36553(76MB) 36554(76MB)] > > -6614> 2018-07-11 07:33:32.497791 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294412497780, "job": 59, "event": "compaction_started", "files_L1": [36565], "files_L2": [36551, 36557, 36558, 36559, 36523, 36553, 36554], "score": 7.5435, "input_data_size": 642785942} > -6613> 2018-07-11 07:33:32.514449 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15263 conn(0x55a321667000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1831 0x55a312363200 osd_ping(ping e494 stamp 2018-07-11 07:33:33.727759) v4 > -6612> 2018-07-11 07:33:32.514473 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.5 10.8.128.113:0/15263 1831 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:33.727759) v4 ==== 2004+0+0 (1203988560 0 0) 0x55a312363200 con 0x55a321667000 > -6611> 2018-07-11 07:33:32.514483 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:33.727759) v4 -- 0x55a348fbde00 con 0 > -6610> 2018-07-11 07:33:32.514519 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15263 conn(0x55a321664000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1831 0x55a312363200 osd_ping(ping e494 stamp 2018-07-11 07:33:33.727759) v4 > -6609> 2018-07-11 07:33:32.514533 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.5 10.8.128.113:0/15263 1831 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:33.727759) v4 ==== 2004+0+0 (1203988560 0 0) 0x55a312363200 con 0x55a321664000 > -6608> 2018-07-11 07:33:32.514542 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:33.727759) v4 -- 0x55a32f9b8000 con 0 > -6607> 2018-07-11 07:33:32.938838 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a31597afc0 con 0 > -6606> 2018-07-11 07:33:32.969097 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5079 0x55a32fdd5b80 replica scrub(pg: 6.15,from:0'0,to:430'106645,epoch:494/492,start:6:a9bf0682:::benchmark_data_magna108.ceph.redhat.com_2478_object150497:0,end:6:a9c00c7b:::benchmark_data_magna108.ceph.redhat.com_2478_object11029:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -6605> 2018-07-11 07:33:32.969129 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5079 ==== replica scrub(pg: 6.15,from:0'0,to:430'106645,epoch:494/492,start:6:a9bf0682:::benchmark_data_magna108.ceph.redhat.com_2478_object150497:0,end:6:a9c00c7b:::benchmark_data_magna108.ceph.redhat.com_2478_object11029:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 247+0+0 (3772377454 0 0) 0x55a32fdd5b80 con 0x55a317019000 > -6604> 2018-07-11 07:33:33.014859 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15263 conn(0x55a321667000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1832 0x55a3265d6c00 osd_ping(ping e494 stamp 2018-07-11 07:33:34.228205) v4 > -6603> 2018-07-11 07:33:33.014885 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.5 10.8.128.113:0/15263 1832 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:34.228205) v4 ==== 2004+0+0 (2178076509 0 0) 0x55a3265d6c00 con 0x55a321667000 > -6602> 2018-07-11 07:33:33.014895 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:34.228205) v4 -- 0x55a348fbde00 con 0 > -6601> 2018-07-11 07:33:33.014933 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15263 conn(0x55a321664000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1832 0x55a3265d6c00 osd_ping(ping e494 stamp 2018-07-11 07:33:34.228205) v4 > -6600> 2018-07-11 07:33:33.014946 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.5 10.8.128.113:0/15263 1832 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:34.228205) v4 ==== 2004+0+0 (2178076509 0 0) 0x55a3265d6c00 con 0x55a321664000 > -6599> 2018-07-11 07:33:33.014956 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:34.228205) v4 -- 0x55a3265d6400 con 0 > -6598> 2018-07-11 07:33:33.044829 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.115:0/13323 conn(0x55a321668800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1844 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:33:33.382002) v4 > -6597> 2018-07-11 07:33:33.044851 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.6 10.8.128.115:0/13323 1844 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:33.382002) v4 ==== 2004+0+0 (1036022837 0 0) 0x55a332e79a00 con 0x55a321668800 > -6596> 2018-07-11 07:33:33.044861 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:33.382002) v4 -- 0x55a31334d400 con 0 > -6595> 2018-07-11 07:33:33.044910 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.115:0/13323 conn(0x55a321665800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1844 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:33:33.382002) v4 > -6594> 2018-07-11 07:33:33.044923 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.6 10.8.128.115:0/13323 1844 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:33.382002) v4 ==== 2004+0+0 (1036022837 0 0) 0x55a324474c00 con 0x55a321665800 > -6593> 2018-07-11 07:33:33.044933 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:33.382002) v4 -- 0x55a31334d400 con 0 > -6592> 2018-07-11 07:33:33.561924 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a32403fb00 con 0 > -6591> 2018-07-11 07:33:33.594332 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5080 0x55a3486b0500 replica scrub(pg: 6.15,from:0'0,to:430'106748,epoch:494/492,start:6:a9c00c7b:::benchmark_data_magna108.ceph.redhat.com_2478_object11029:0,end:6:a9c110e4:::benchmark_data_magna108.ceph.redhat.com_2478_object49073:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -6590> 2018-07-11 07:33:33.594355 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5080 ==== replica scrub(pg: 6.15,from:0'0,to:430'106748,epoch:494/492,start:6:a9c00c7b:::benchmark_data_magna108.ceph.redhat.com_2478_object11029:0,end:6:a9c110e4:::benchmark_data_magna108.ceph.redhat.com_2478_object49073:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 246+0+0 (2129926234 0 0) 0x55a3486b0500 con 0x55a317019000 > -6589> 2018-07-11 07:33:33.761898 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15885 conn(0x55a3170b5800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1851 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:33:34.385516) v4 > -6588> 2018-07-11 07:33:33.761905 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15885 conn(0x55a311b76800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1851 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:33:34.385516) v4 > -6587> 2018-07-11 07:33:33.761920 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.7 10.8.128.114:0/15885 1851 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:34.385516) v4 ==== 2004+0+0 (3546651049 0 0) 0x55a332e79a00 con 0x55a3170b5800 > -6586> 2018-07-11 07:33:33.761923 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.7 10.8.128.114:0/15885 1851 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:34.385516) v4 ==== 2004+0+0 (3546651049 0 0) 0x55a34de6f000 con 0x55a311b76800 > -6585> 2018-07-11 07:33:33.761930 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:34.385516) v4 -- 0x55a31334d400 con 0 > -6584> 2018-07-11 07:33:33.761955 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:34.385516) v4 -- 0x55a31334c400 con 0 > -6583> 2018-07-11 07:33:33.973564 7f355d88e700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.107:0/12617 conn(0x55a32447f800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1806 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:33:33.973205) v4 > -6582> 2018-07-11 07:33:33.973551 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.107:0/12617 conn(0x55a32447e000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1806 0x55a32f9b8000 osd_ping(ping e494 stamp 2018-07-11 07:33:33.973205) v4 > -6581> 2018-07-11 07:33:33.973610 7f355d88e700 1 -- 10.8.128.107:6803/12665 <== osd.9 10.8.128.107:0/12617 1806 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:33.973205) v4 ==== 2004+0+0 (2307489013 0 0) 0x55a34de78600 con 0x55a32447f800 > -6580> 2018-07-11 07:33:33.973624 7f355d88e700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:33.973205) v4 -- 0x55a31334c400 con 0 > -6579> 2018-07-11 07:33:33.973623 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.9 10.8.128.107:0/12617 1806 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:33.973205) v4 ==== 2004+0+0 (2307489013 0 0) 0x55a32f9b8000 con 0x55a32447e000 > -6578> 2018-07-11 07:33:33.973654 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:33.973205) v4 -- 0x55a3265d6400 con 0 > -6577> 2018-07-11 07:33:34.115304 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15263 conn(0x55a321664000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1833 0x55a3265d6c00 osd_ping(ping e494 stamp 2018-07-11 07:33:35.328618) v4 > -6576> 2018-07-11 07:33:34.115333 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.5 10.8.128.113:0/15263 1833 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:35.328618) v4 ==== 2004+0+0 (1739265054 0 0) 0x55a3265d6c00 con 0x55a321664000 > -6575> 2018-07-11 07:33:34.115350 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:35.328618) v4 -- 0x55a3265d6400 con 0 > -6574> 2018-07-11 07:33:34.115407 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15263 conn(0x55a321667000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1833 0x55a3265d6c00 osd_ping(ping e494 stamp 2018-07-11 07:33:35.328618) v4 > -6573> 2018-07-11 07:33:34.115432 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.5 10.8.128.113:0/15263 1833 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:35.328618) v4 ==== 2004+0+0 (1739265054 0 0) 0x55a3265d6c00 con 0x55a321667000 > -6572> 2018-07-11 07:33:34.115443 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:35.328618) v4 -- 0x55a348fbde00 con 0 > -6571> 2018-07-11 07:33:34.145322 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.115:0/13323 conn(0x55a321668800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1845 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:33:34.482482) v4 > -6570> 2018-07-11 07:33:34.145344 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.6 10.8.128.115:0/13323 1845 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:34.482482) v4 ==== 2004+0+0 (1237882144 0 0) 0x55a324474c00 con 0x55a321668800 > -6569> 2018-07-11 07:33:34.145360 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:34.482482) v4 -- 0x55a31334d400 con 0 > -6568> 2018-07-11 07:33:34.145388 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.115:0/13323 conn(0x55a321665800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1845 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:33:34.482482) v4 > -6567> 2018-07-11 07:33:34.145399 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.6 10.8.128.115:0/13323 1845 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:34.482482) v4 ==== 2004+0+0 (1237882144 0 0) 0x55a324474c00 con 0x55a321665800 > -6566> 2018-07-11 07:33:34.145409 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:34.482482) v4 -- 0x55a324967400 con 0 > -6565> 2018-07-11 07:33:34.241089 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a31bbefb00 con 0 > -6564> 2018-07-11 07:33:34.265161 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5081 0x55a3486b0a00 replica scrub(pg: 6.15,from:0'0,to:461'106913,epoch:494/492,start:6:a9c110e4:::benchmark_data_magna108.ceph.redhat.com_2478_object49073:0,end:6:a9c2281b:::benchmark_data_magna108_35617_object1062727:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -6563> 2018-07-11 07:33:34.265187 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5081 ==== replica scrub(pg: 6.15,from:0'0,to:461'106913,epoch:494/492,start:6:a9c110e4:::benchmark_data_magna108.ceph.redhat.com_2478_object49073:0,end:6:a9c2281b:::benchmark_data_magna108_35617_object1062727:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (690772670 0 0) 0x55a3486b0a00 con 0x55a317019000 > -6562> 2018-07-11 07:33:34.452812 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15919 conn(0x55a3170b4000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1837 0x55a32f9b8000 osd_ping(ping e494 stamp 2018-07-11 07:33:35.076495) v4 > -6561> 2018-07-11 07:33:34.452838 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.11 10.8.128.114:0/15919 1837 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:35.076495) v4 ==== 2004+0+0 (3789271652 0 0) 0x55a32f9b8000 con 0x55a3170b4000 > -6560> 2018-07-11 07:33:34.452852 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:35.076495) v4 -- 0x55a3265d6400 con 0 > -6559> 2018-07-11 07:33:34.452854 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15919 conn(0x55a317074800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1837 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:33:35.076495) v4 > -6558> 2018-07-11 07:33:34.452879 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.11 10.8.128.114:0/15919 1837 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:35.076495) v4 ==== 2004+0+0 (3789271652 0 0) 0x55a332e79a00 con 0x55a317074800 > -6557> 2018-07-11 07:33:34.452890 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:35.076495) v4 -- 0x55a324967400 con 0 > -6556> 2018-07-11 07:33:34.724285 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15255 conn(0x55a311b64000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1858 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:33:35.937598) v4 > -6555> 2018-07-11 07:33:34.724292 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15255 conn(0x55a311b65800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1858 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:33:35.937598) v4 > -6554> 2018-07-11 07:33:34.724316 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.0 10.8.128.113:0/15255 1858 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:35.937598) v4 ==== 2004+0+0 (3964658228 0 0) 0x55a34de6f000 con 0x55a311b64000 > -6553> 2018-07-11 07:33:34.724319 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.0 10.8.128.113:0/15255 1858 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:35.937598) v4 ==== 2004+0+0 (3964658228 0 0) 0x55a324474c00 con 0x55a311b65800 > -6552> 2018-07-11 07:33:34.724336 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:35.937598) v4 -- 0x55a31334c400 con 0 > -6551> 2018-07-11 07:33:34.724373 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:35.937598) v4 -- 0x55a324967400 con 0 > -6550> 2018-07-11 07:33:34.724771 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12677 conn(0x55a32169d800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1855 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:33:34.760034) v4 > -6549> 2018-07-11 07:33:34.724788 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.1 10.8.128.106:0/12677 1855 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:34.760034) v4 ==== 2004+0+0 (3049677661 0 0) 0x55a34de78600 con 0x55a32169d800 > -6548> 2018-07-11 07:33:34.724798 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:34.760034) v4 -- 0x55a31334c400 con 0 > -6547> 2018-07-11 07:33:34.724852 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12677 conn(0x55a32169c000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1855 0x55a348fbde00 osd_ping(ping e494 stamp 2018-07-11 07:33:34.760034) v4 > -6546> 2018-07-11 07:33:34.724873 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.1 10.8.128.106:0/12677 1855 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:34.760034) v4 ==== 2004+0+0 (3049677661 0 0) 0x55a348fbde00 con 0x55a32169c000 > -6545> 2018-07-11 07:33:34.724885 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:34.760034) v4 -- 0x55a3265d7000 con 0 > -6544> 2018-07-11 07:33:34.824692 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15287 conn(0x55a311b72000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1827 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:33:36.038020) v4 > -6543> 2018-07-11 07:33:34.824716 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.4 10.8.128.113:0/15287 1827 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:36.038020) v4 ==== 2004+0+0 (3939443122 0 0) 0x55a332e79a00 con 0x55a311b72000 > -6542> 2018-07-11 07:33:34.824727 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:36.038020) v4 -- 0x55a31334d400 con 0 > -6541> 2018-07-11 07:33:34.824718 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15287 conn(0x55a311b75000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1827 0x55a32f9b8000 osd_ping(ping e494 stamp 2018-07-11 07:33:36.038020) v4 > -6540> 2018-07-11 07:33:34.824742 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.4 10.8.128.113:0/15287 1827 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:36.038020) v4 ==== 2004+0+0 (3939443122 0 0) 0x55a32f9b8000 con 0x55a311b75000 > -6539> 2018-07-11 07:33:34.824757 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:36.038020) v4 -- 0x55a312363a00 con 0 > -6538> 2018-07-11 07:33:35.216155 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a3240ccfc0 con 0 > -6537> 2018-07-11 07:33:35.245710 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.115:0/13323 conn(0x55a321665800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1846 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:33:35.582951) v4 > -6536> 2018-07-11 07:33:35.245732 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.6 10.8.128.115:0/13323 1846 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:35.582951) v4 ==== 2004+0+0 (2846373701 0 0) 0x55a324474c00 con 0x55a321665800 > -6535> 2018-07-11 07:33:35.245748 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:35.582951) v4 -- 0x55a348fbdc00 con 0 > -6534> 2018-07-11 07:33:35.245818 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.115:0/13323 conn(0x55a321668800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1846 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:33:35.582951) v4 > -6533> 2018-07-11 07:33:35.245833 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.6 10.8.128.115:0/13323 1846 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:35.582951) v4 ==== 2004+0+0 (2846373701 0 0) 0x55a332e79a00 con 0x55a321668800 > -6532> 2018-07-11 07:33:35.245841 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:35.582951) v4 -- 0x55a3203f7800 con 0 > -6531> 2018-07-11 07:33:35.248280 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5082 0x55a3486b0c80 replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:494/492,start:6:a9c2281b:::benchmark_data_magna108_35617_object1062727:0,end:6:a9c30fb6:::benchmark_data_magna108.ceph.redhat.com_2478_object370901:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -6530> 2018-07-11 07:33:35.248310 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5082 ==== replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:494/492,start:6:a9c2281b:::benchmark_data_magna108_35617_object1062727:0,end:6:a9c30fb6:::benchmark_data_magna108.ceph.redhat.com_2478_object370901:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 234+0+0 (3216092310 0 0) 0x55a3486b0c80 con 0x55a317019000 > -6529> 2018-07-11 07:33:35.747208 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b7000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1785 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:33:35.497707) v4 > -6528> 2018-07-11 07:33:35.747232 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.8 10.8.128.108:0/11967 1785 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:35.497707) v4 ==== 2004+0+0 (4224803755 0 0) 0x55a34de6f000 con 0x55a3170b7000 > -6527> 2018-07-11 07:33:35.747241 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:35.497707) v4 -- 0x55a31334c400 con 0 > -6526> 2018-07-11 07:33:35.747267 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b8800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1785 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:33:35.497707) v4 > -6525> 2018-07-11 07:33:35.747288 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.8 10.8.128.108:0/11967 1785 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:35.497707) v4 ==== 2004+0+0 (4224803755 0 0) 0x55a324474c00 con 0x55a3170b8800 > -6524> 2018-07-11 07:33:35.747300 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:35.497707) v4 -- 0x55a3203f7800 con 0 > -6523> 2018-07-11 07:33:36.002182 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a31fceed80 con 0 > -6522> 2018-07-11 07:33:36.026931 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5083 0x55a3486b0f00 replica scrub(pg: 6.15,from:0'0,to:430'106822,epoch:494/492,start:6:a9c30fb6:::benchmark_data_magna108.ceph.redhat.com_2478_object370901:0,end:6:a9c417ce:::benchmark_data_magna108.ceph.redhat.com_2478_object133568:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -6521> 2018-07-11 07:33:36.026961 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5083 ==== replica scrub(pg: 6.15,from:0'0,to:430'106822,epoch:494/492,start:6:a9c30fb6:::benchmark_data_magna108.ceph.redhat.com_2478_object370901:0,end:6:a9c417ce:::benchmark_data_magna108.ceph.redhat.com_2478_object133568:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 248+0+0 (2304386306 0 0) 0x55a3486b0f00 con 0x55a317019000 > -6520> 2018-07-11 07:33:36.274034 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.107:0/12617 conn(0x55a32447e000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1807 0x55a348fbde00 osd_ping(ping e494 stamp 2018-07-11 07:33:36.273708) v4 > -6519> 2018-07-11 07:33:36.274042 7f355d88e700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.107:0/12617 conn(0x55a32447f800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1807 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:33:36.273708) v4 > -6518> 2018-07-11 07:33:36.274056 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.9 10.8.128.107:0/12617 1807 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:36.273708) v4 ==== 2004+0+0 (1236487790 0 0) 0x55a348fbde00 con 0x55a32447e000 > -6517> 2018-07-11 07:33:36.274059 7f355d88e700 1 -- 10.8.128.107:6803/12665 <== osd.9 10.8.128.107:0/12617 1807 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:36.273708) v4 ==== 2004+0+0 (1236487790 0 0) 0x55a34de78600 con 0x55a32447f800 > -6516> 2018-07-11 07:33:36.274067 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:36.273708) v4 -- 0x55a3265d7200 con 0 > -6515> 2018-07-11 07:33:36.274109 7f355d88e700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:36.273708) v4 -- 0x55a31334c400 con 0 > -6514> 2018-07-11 07:33:36.473054 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12650 conn(0x55a311b68800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1850 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:33:36.508503) v4 > -6513> 2018-07-11 07:33:36.473063 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12650 conn(0x55a311b67000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1850 0x55a312363a00 osd_ping(ping e494 stamp 2018-07-11 07:33:36.508503) v4 > -6512> 2018-07-11 07:33:36.473073 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.3 10.8.128.106:0/12650 1850 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:36.508503) v4 ==== 2004+0+0 (371185743 0 0) 0x55a34de6f000 con 0x55a311b68800 > -6511> 2018-07-11 07:33:36.473087 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.3 10.8.128.106:0/12650 1850 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:36.508503) v4 ==== 2004+0+0 (371185743 0 0) 0x55a312363a00 con 0x55a311b67000 > -6510> 2018-07-11 07:33:36.473093 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:36.508503) v4 -- 0x55a31334c400 con 0 > -6509> 2018-07-11 07:33:36.473118 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:36.508503) v4 -- 0x55a3265d7200 con 0 > -6508> 2018-07-11 07:33:36.603988 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a3240d38c0 con 0 > -6507> 2018-07-11 07:33:36.623969 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5084 0x55a3486b1680 replica scrub(pg: 6.15,from:0'0,to:430'106215,epoch:494/492,start:6:a9c417ce:::benchmark_data_magna108.ceph.redhat.com_2478_object133568:0,end:6:a9c52078:::benchmark_data_magna108_35617_object716799:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -6506> 2018-07-11 07:33:36.623992 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5084 ==== replica scrub(pg: 6.15,from:0'0,to:430'106215,epoch:494/492,start:6:a9c417ce:::benchmark_data_magna108.ceph.redhat.com_2478_object133568:0,end:6:a9c52078:::benchmark_data_magna108_35617_object716799:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (1936707410 0 0) 0x55a3486b1680 con 0x55a317019000 > -6505> 2018-07-11 07:33:36.661486 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12651 conn(0x55a3216a0800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1836 0x55a348fbde00 osd_ping(ping e494 stamp 2018-07-11 07:33:36.696887) v4 > -6504> 2018-07-11 07:33:36.661507 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.2 10.8.128.106:0/12651 1836 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:36.696887) v4 ==== 2004+0+0 (2115870519 0 0) 0x55a348fbde00 con 0x55a3216a0800 > -6503> 2018-07-11 07:33:36.661516 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:36.696887) v4 -- 0x55a3265d7200 con 0 > -6502> 2018-07-11 07:33:36.661508 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12651 conn(0x55a32169f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1836 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:33:36.696887) v4 > -6501> 2018-07-11 07:33:36.661534 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.2 10.8.128.106:0/12651 1836 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:36.696887) v4 ==== 2004+0+0 (2115870519 0 0) 0x55a332e79a00 con 0x55a32169f000 > -6500> 2018-07-11 07:33:36.661551 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:36.696887) v4 -- 0x55a3203f7800 con 0 > -6499> 2018-07-11 07:33:36.744212 7f35498a5700 10 monclient: tick > -6498> 2018-07-11 07:33:36.744227 7f35498a5700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2018-07-11 07:33:06.744224) > -6497> 2018-07-11 07:33:36.753278 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15919 conn(0x55a317074800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1838 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:33:37.376903) v4 > -6496> 2018-07-11 07:33:36.753299 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.11 10.8.128.114:0/15919 1838 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:37.376903) v4 ==== 2004+0+0 (2544088900 0 0) 0x55a324474c00 con 0x55a317074800 > -6495> 2018-07-11 07:33:36.753310 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:37.376903) v4 -- 0x55a3203f7800 con 0 > -6494> 2018-07-11 07:33:36.753358 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15919 conn(0x55a3170b4000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1838 0x55a348fbde00 osd_ping(ping e494 stamp 2018-07-11 07:33:37.376903) v4 > -6493> 2018-07-11 07:33:36.753379 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.11 10.8.128.114:0/15919 1838 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:37.376903) v4 ==== 2004+0+0 (2544088900 0 0) 0x55a348fbde00 con 0x55a3170b4000 > -6492> 2018-07-11 07:33:36.753400 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:37.376903) v4 -- 0x55a3265d7200 con 0 > -6491> 2018-07-11 07:33:37.161183 7f3543899700 5 osd.10 494 heartbeat: osd_stat(783 GB used, 147 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,0,0,1]) > -6490> 2018-07-11 07:33:37.161198 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6802/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:33:37.161196) v4 -- 0x55a32f9b9600 con 0 > -6489> 2018-07-11 07:33:37.161235 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6803/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:33:37.161196) v4 -- 0x55a343482600 con 0 > -6488> 2018-07-11 07:33:37.161267 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6806/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:33:37.161196) v4 -- 0x55a32f9b8800 con 0 > -6487> 2018-07-11 07:33:37.161284 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6807/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:33:37.161196) v4 -- 0x55a33be6ea00 con 0 > -6486> 2018-07-11 07:33:37.161296 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6808/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:33:37.161196) v4 -- 0x55a3203f6400 con 0 > -6485> 2018-07-11 07:33:37.161314 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6810/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:33:37.161196) v4 -- 0x55a32f9b9200 con 0 > -6484> 2018-07-11 07:33:37.161326 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6809/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:33:37.161196) v4 -- 0x55a32f9b8c00 con 0 > -6483> 2018-07-11 07:33:37.161339 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6811/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:33:37.161196) v4 -- 0x55a3203f6c00 con 0 > -6482> 2018-07-11 07:33:37.161351 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6810/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:33:37.161196) v4 -- 0x55a32f9b8e00 con 0 > -6481> 2018-07-11 07:33:37.161361 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6811/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:33:37.161196) v4 -- 0x55a3203f6600 con 0 > -6480> 2018-07-11 07:33:37.161374 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6806/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:33:37.161196) v4 -- 0x55a3203f6200 con 0 > -6479> 2018-07-11 07:33:37.161392 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6807/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:33:37.161196) v4 -- 0x55a3203f6800 con 0 > -6478> 2018-07-11 07:33:37.161405 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6802/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:33:37.161196) v4 -- 0x55a3203f6e00 con 0 > -6477> 2018-07-11 07:33:37.161423 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6803/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:33:37.161196) v4 -- 0x55a3203f7600 con 0 > -6476> 2018-07-11 07:33:37.161440 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6806/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:33:37.161196) v4 -- 0x55a3203f7c00 con 0 > -6475> 2018-07-11 07:33:37.161454 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6807/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:33:37.161196) v4 -- 0x55a3203f7400 con 0 > -6474> 2018-07-11 07:33:37.161474 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6802/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:33:37.161196) v4 -- 0x55a31bbec000 con 0 > -6473> 2018-07-11 07:33:37.161509 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6803/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:33:37.161196) v4 -- 0x55a3203f7a00 con 0 > -6472> 2018-07-11 07:33:37.161525 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6806/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:33:37.161196) v4 -- 0x55a331c65400 con 0 > -6471> 2018-07-11 07:33:37.161540 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6807/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:33:37.161196) v4 -- 0x55a331c65800 con 0 > -6470> 2018-07-11 07:33:37.161561 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6802/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:33:37.161196) v4 -- 0x55a331c64c00 con 0 > -6469> 2018-07-11 07:33:37.161601 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6803/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:33:37.161196) v4 -- 0x55a324af4000 con 0 > -6468> 2018-07-11 07:33:37.161669 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6802/15255 conn(0x55a314e1c000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1835 0x55a312363a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:37.161196) v4 > -6467> 2018-07-11 07:33:37.161691 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6802/15255 1835 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:37.161196) v4 ==== 2004+0+0 (2831566491 0 0) 0x55a312363a00 con 0x55a314e1c000 > -6466> 2018-07-11 07:33:37.161735 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6803/15255 conn(0x55a314e1d800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1835 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:37.161196) v4 > -6465> 2018-07-11 07:33:37.161751 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6803/15255 1835 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:37.161196) v4 ==== 2004+0+0 (2831566491 0 0) 0x55a324474c00 con 0x55a314e1d800 > -6464> 2018-07-11 07:33:37.161758 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6807/12617 conn(0x55a314e27800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1835 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:37.161196) v4 > -6463> 2018-07-11 07:33:37.161761 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6807/12677 conn(0x55a314e85800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1835 0x55a312363a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:37.161196) v4 > -6462> 2018-07-11 07:33:37.161786 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6807/12617 1835 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:37.161196) v4 ==== 2004+0+0 (2831566491 0 0) 0x55a34de78600 con 0x55a314e27800 > -6461> 2018-07-11 07:33:37.161774 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6806/12617 conn(0x55a314e26000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1835 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:37.161196) v4 > -6460> 2018-07-11 07:33:37.161791 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6807/12677 1835 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:37.161196) v4 ==== 2004+0+0 (2831566491 0 0) 0x55a312363a00 con 0x55a314e85800 > -6459> 2018-07-11 07:33:37.161798 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6806/12617 1835 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:37.161196) v4 ==== 2004+0+0 (2831566491 0 0) 0x55a324474c00 con 0x55a314e26000 > -6458> 2018-07-11 07:33:37.161832 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6809/12650 conn(0x55a314e1f000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1835 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:37.161196) v4 > -6457> 2018-07-11 07:33:37.161849 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6809/12650 1835 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:37.161196) v4 ==== 2004+0+0 (2831566491 0 0) 0x55a34de78600 con 0x55a314e1f000 > -6456> 2018-07-11 07:33:37.161846 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6810/12651 conn(0x55a314e97800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1835 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:37.161196) v4 > -6455> 2018-07-11 07:33:37.161838 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6810/15287 conn(0x55a314e03000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1835 0x55a312363a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:37.161196) v4 > -6454> 2018-07-11 07:33:37.161858 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6810/12651 1835 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:37.161196) v4 ==== 2004+0+0 (2831566491 0 0) 0x55a324474c00 con 0x55a314e97800 > -6453> 2018-07-11 07:33:37.161866 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6810/15287 1835 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:37.161196) v4 ==== 2004+0+0 (2831566491 0 0) 0x55a312363a00 con 0x55a314e03000 > -6452> 2018-07-11 07:33:37.161884 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6806/12677 conn(0x55a314e84000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1835 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:37.161196) v4 > -6451> 2018-07-11 07:33:37.161896 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6806/12677 1835 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:37.161196) v4 ==== 2004+0+0 (2831566491 0 0) 0x55a34de78600 con 0x55a314e84000 > -6450> 2018-07-11 07:33:37.161913 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6807/15263 conn(0x55a314e9a800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1835 0x55a312363a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:37.161196) v4 > -6449> 2018-07-11 07:33:37.161919 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6811/15287 conn(0x55a314e04800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1835 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:37.161196) v4 > -6448> 2018-07-11 07:33:37.161927 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6807/15263 1835 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:37.161196) v4 ==== 2004+0+0 (2831566491 0 0) 0x55a312363a00 con 0x55a314e9a800 > -6447> 2018-07-11 07:33:37.161932 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6811/15287 1835 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:37.161196) v4 ==== 2004+0+0 (2831566491 0 0) 0x55a324474c00 con 0x55a314e04800 > -6446> 2018-07-11 07:33:37.161928 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6806/15263 conn(0x55a314e99000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1835 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:37.161196) v4 > -6445> 2018-07-11 07:33:37.161946 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6806/15263 1835 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:37.161196) v4 ==== 2004+0+0 (2831566491 0 0) 0x55a34de78600 con 0x55a314e99000 > -6444> 2018-07-11 07:33:37.161957 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6811/12650 conn(0x55a314e20800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1835 0x55a312363a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:37.161196) v4 > -6443> 2018-07-11 07:33:37.161969 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6811/12650 1835 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:37.161196) v4 ==== 2004+0+0 (2831566491 0 0) 0x55a312363a00 con 0x55a314e20800 > -6442> 2018-07-11 07:33:37.161964 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6806/15885 conn(0x55a314e00000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1835 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:37.161196) v4 > -6441> 2018-07-11 07:33:37.161991 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6803/13323 conn(0x55a314e16800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1835 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:37.161196) v4 > -6440> 2018-07-11 07:33:37.161999 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6806/15885 1835 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:37.161196) v4 ==== 2004+0+0 (2831566491 0 0) 0x55a324474c00 con 0x55a314e00000 > -6439> 2018-07-11 07:33:37.162002 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6808/12651 conn(0x55a314e96000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1835 0x55a312363a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:37.161196) v4 > -6438> 2018-07-11 07:33:37.162006 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6803/13323 1835 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:37.161196) v4 ==== 2004+0+0 (2831566491 0 0) 0x55a34de78600 con 0x55a314e16800 > -6437> 2018-07-11 07:33:37.162016 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6808/12651 1835 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:37.161196) v4 ==== 2004+0+0 (2831566491 0 0) 0x55a312363a00 con 0x55a314e96000 > -6436> 2018-07-11 07:33:37.162033 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6802/13323 conn(0x55a314e15000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1835 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:37.161196) v4 > -6435> 2018-07-11 07:33:37.162045 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6802/13323 1835 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:37.161196) v4 ==== 2004+0+0 (2831566491 0 0) 0x55a324474c00 con 0x55a314e15000 > -6434> 2018-07-11 07:33:37.162046 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6803/15919 conn(0x55a314e88800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1835 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:37.161196) v4 > -6433> 2018-07-11 07:33:37.162057 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6803/15919 1835 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:37.161196) v4 ==== 2004+0+0 (2831566491 0 0) 0x55a34de78600 con 0x55a314e88800 > -6432> 2018-07-11 07:33:37.162054 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6803/11967 conn(0x55a314e13800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1835 0x55a312363a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:37.161196) v4 > -6431> 2018-07-11 07:33:37.162072 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6803/11967 1835 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:37.161196) v4 ==== 2004+0+0 (2831566491 0 0) 0x55a312363a00 con 0x55a314e13800 > -6430> 2018-07-11 07:33:37.162076 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6802/15919 conn(0x55a314e87000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1835 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:37.161196) v4 > -6429> 2018-07-11 07:33:37.162092 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6802/15919 1835 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:37.161196) v4 ==== 2004+0+0 (2831566491 0 0) 0x55a324474c00 con 0x55a314e87000 > -6428> 2018-07-11 07:33:37.162090 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6807/15885 conn(0x55a314e01800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1835 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:37.161196) v4 > -6427> 2018-07-11 07:33:37.162104 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6807/15885 1835 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:37.161196) v4 ==== 2004+0+0 (2831566491 0 0) 0x55a34de78600 con 0x55a314e01800 > -6426> 2018-07-11 07:33:37.162137 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6802/11967 conn(0x55a314e12000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1835 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:37.161196) v4 > -6425> 2018-07-11 07:33:37.162149 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6802/11967 1835 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:37.161196) v4 ==== 2004+0+0 (2831566491 0 0) 0x55a34de78600 con 0x55a314e12000 > -6424> 2018-07-11 07:33:37.252556 7f35488a3700 1 -- 10.8.128.107:6800/12665 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55a346875600 con 0 > -6423> 2018-07-11 07:33:37.252630 7f35488a3700 1 -- 10.8.128.107:6800/12665 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55a30f75f600 con 0 > -6422> 2018-07-11 07:33:37.262326 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15885 conn(0x55a311b76800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1852 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:33:37.885973) v4 > -6421> 2018-07-11 07:33:37.262326 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15885 conn(0x55a3170b5800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1852 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:33:37.885973) v4 > -6420> 2018-07-11 07:33:37.262355 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.7 10.8.128.114:0/15885 1852 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:37.885973) v4 ==== 2004+0+0 (3593531747 0 0) 0x55a34de78600 con 0x55a311b76800 > -6419> 2018-07-11 07:33:37.262357 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.7 10.8.128.114:0/15885 1852 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:37.885973) v4 ==== 2004+0+0 (3593531747 0 0) 0x55a324474c00 con 0x55a3170b5800 > -6418> 2018-07-11 07:33:37.262376 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:37.885973) v4 -- 0x55a32f9b8c00 con 0 > -6417> 2018-07-11 07:33:37.262404 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:37.885973) v4 -- 0x55a32f9b9200 con 0 > -6416> 2018-07-11 07:33:37.272773 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a330c6cb40 con 0 > -6415> 2018-07-11 07:33:37.290200 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5085 0x55a3486b1900 replica scrub(pg: 6.15,from:0'0,to:430'105660,epoch:494/492,start:6:a9c52078:::benchmark_data_magna108_35617_object716799:0,end:6:a9c66b76:::benchmark_data_magna108_35617_object412205:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -6414> 2018-07-11 07:33:37.290230 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5085 ==== replica scrub(pg: 6.15,from:0'0,to:430'105660,epoch:494/492,start:6:a9c52078:::benchmark_data_magna108_35617_object716799:0,end:6:a9c66b76:::benchmark_data_magna108_35617_object412205:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (440813929 0 0) 0x55a3486b1900 con 0x55a317019000 > -6413> 2018-07-11 07:33:37.615651 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15263 conn(0x55a321667000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1834 0x55a312363a00 osd_ping(ping e494 stamp 2018-07-11 07:33:38.829034) v4 > -6412> 2018-07-11 07:33:37.615685 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.5 10.8.128.113:0/15263 1834 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:38.829034) v4 ==== 2004+0+0 (2849446971 0 0) 0x55a312363a00 con 0x55a321667000 > -6411> 2018-07-11 07:33:37.615701 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:38.829034) v4 -- 0x55a33be6ea00 con 0 > -6410> 2018-07-11 07:33:37.615728 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15263 conn(0x55a321664000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1834 0x55a312363a00 osd_ping(ping e494 stamp 2018-07-11 07:33:38.829034) v4 > -6409> 2018-07-11 07:33:37.615740 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.5 10.8.128.113:0/15263 1834 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:38.829034) v4 ==== 2004+0+0 (2849446971 0 0) 0x55a312363a00 con 0x55a321664000 > -6408> 2018-07-11 07:33:37.615750 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:38.829034) v4 -- 0x55a32f9b9600 con 0 > -6407> 2018-07-11 07:33:37.624968 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12677 conn(0x55a32169d800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1856 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:33:37.660432) v4 > -6406> 2018-07-11 07:33:37.624998 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.1 10.8.128.106:0/12677 1856 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:37.660432) v4 ==== 2004+0+0 (3230226186 0 0) 0x55a34de6f000 con 0x55a32169d800 > -6405> 2018-07-11 07:33:37.625003 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12677 conn(0x55a32169c000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1856 0x55a3203f6400 osd_ping(ping e494 stamp 2018-07-11 07:33:37.660432) v4 > -6404> 2018-07-11 07:33:37.625016 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:37.660432) v4 -- 0x55a32f9b8800 con 0 > -6403> 2018-07-11 07:33:37.625021 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.1 10.8.128.106:0/12677 1856 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:37.660432) v4 ==== 2004+0+0 (3230226186 0 0) 0x55a3203f6400 con 0x55a32169c000 > -6402> 2018-07-11 07:33:37.625047 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:37.660432) v4 -- 0x55a33be6ea00 con 0 > -6401> 2018-07-11 07:33:37.762731 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15885 conn(0x55a3170b5800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1853 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:33:38.386369) v4 > -6400> 2018-07-11 07:33:37.762732 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15885 conn(0x55a311b76800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1853 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:33:38.386369) v4 > -6399> 2018-07-11 07:33:37.762755 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.7 10.8.128.114:0/15885 1853 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:38.386369) v4 ==== 2004+0+0 (2857580810 0 0) 0x55a332e79a00 con 0x55a3170b5800 > -6398> 2018-07-11 07:33:37.762759 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.7 10.8.128.114:0/15885 1853 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:38.386369) v4 ==== 2004+0+0 (2857580810 0 0) 0x55a34de78600 con 0x55a311b76800 > -6397> 2018-07-11 07:33:37.762765 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:38.386369) v4 -- 0x55a343482600 con 0 > -6396> 2018-07-11 07:33:37.762787 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:38.386369) v4 -- 0x55a31334c400 con 0 > -6395> 2018-07-11 07:33:37.853706 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15919 conn(0x55a317074800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1839 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:33:38.477321) v4 > -6394> 2018-07-11 07:33:37.853726 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.11 10.8.128.114:0/15919 1839 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:38.477321) v4 ==== 2004+0+0 (1366642921 0 0) 0x55a324474c00 con 0x55a317074800 > -6393> 2018-07-11 07:33:37.853738 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:38.477321) v4 -- 0x55a3203f7800 con 0 > -6392> 2018-07-11 07:33:37.853834 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15919 conn(0x55a3170b4000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1839 0x55a312363a00 osd_ping(ping e494 stamp 2018-07-11 07:33:38.477321) v4 > -6391> 2018-07-11 07:33:37.853854 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.11 10.8.128.114:0/15919 1839 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:38.477321) v4 ==== 2004+0+0 (1366642921 0 0) 0x55a312363a00 con 0x55a3170b4000 > -6390> 2018-07-11 07:33:37.853865 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:38.477321) v4 -- 0x55a3265d7200 con 0 > -6389> 2018-07-11 07:33:37.992919 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a332beb200 con 0 > -6388> 2018-07-11 07:33:38.019357 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5086 0x55a3486b1b80 replica scrub(pg: 6.15,from:0'0,to:430'106076,epoch:494/492,start:6:a9c66b76:::benchmark_data_magna108_35617_object412205:0,end:6:a9c79b5a:::benchmark_data_magna108_35617_object1149216:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -6387> 2018-07-11 07:33:38.019380 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5086 ==== replica scrub(pg: 6.15,from:0'0,to:430'106076,epoch:494/492,start:6:a9c66b76:::benchmark_data_magna108_35617_object412205:0,end:6:a9c79b5a:::benchmark_data_magna108_35617_object1149216:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 219+0+0 (2482169869 0 0) 0x55a3486b1b80 con 0x55a317019000 > -6386> 2018-07-11 07:33:38.224581 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15255 conn(0x55a311b65800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1859 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:33:39.437987) v4 > -6385> 2018-07-11 07:33:38.224609 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.0 10.8.128.113:0/15255 1859 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:39.437987) v4 ==== 2004+0+0 (3149502837 0 0) 0x55a332e79a00 con 0x55a311b65800 > -6384> 2018-07-11 07:33:38.224622 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:39.437987) v4 -- 0x55a3203f7800 con 0 > -6383> 2018-07-11 07:33:38.224618 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15255 conn(0x55a311b64000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1859 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:33:39.437987) v4 > -6382> 2018-07-11 07:33:38.224635 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.0 10.8.128.113:0/15255 1859 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:39.437987) v4 ==== 2004+0+0 (3149502837 0 0) 0x55a34de6f000 con 0x55a311b64000 > -6381> 2018-07-11 07:33:38.224648 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:39.437987) v4 -- 0x55a31334c400 con 0 > -6380> 2018-07-11 07:33:38.574661 7f355d88e700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.107:0/12617 conn(0x55a32447f800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1808 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:33:38.574188) v4 > -6379> 2018-07-11 07:33:38.574658 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.107:0/12617 conn(0x55a32447e000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1808 0x55a3203f6400 osd_ping(ping e494 stamp 2018-07-11 07:33:38.574188) v4 > -6378> 2018-07-11 07:33:38.574705 7f355d88e700 1 -- 10.8.128.107:6803/12665 <== osd.9 10.8.128.107:0/12617 1808 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:38.574188) v4 ==== 2004+0+0 (2112044402 0 0) 0x55a34de78600 con 0x55a32447f800 > -6377> 2018-07-11 07:33:38.574728 7f355d88e700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:38.574188) v4 -- 0x55a31bbede00 con 0 > -6376> 2018-07-11 07:33:38.574722 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.9 10.8.128.107:0/12617 1808 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:38.574188) v4 ==== 2004+0+0 (2112044402 0 0) 0x55a3203f6400 con 0x55a32447e000 > -6375> 2018-07-11 07:33:38.574756 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:38.574188) v4 -- 0x55a3265d6000 con 0 > -6374> 2018-07-11 07:33:38.647646 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b7000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1786 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:33:38.398151) v4 > -6373> 2018-07-11 07:33:38.647641 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b8800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1786 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:33:38.398151) v4 > -6372> 2018-07-11 07:33:38.647672 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.8 10.8.128.108:0/11967 1786 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:38.398151) v4 ==== 2004+0+0 (4028325465 0 0) 0x55a34de6f000 con 0x55a3170b7000 > -6371> 2018-07-11 07:33:38.647676 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.8 10.8.128.108:0/11967 1786 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:38.398151) v4 ==== 2004+0+0 (4028325465 0 0) 0x55a324474c00 con 0x55a3170b8800 > -6370> 2018-07-11 07:33:38.647682 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:38.398151) v4 -- 0x55a30fa77800 con 0 > -6369> 2018-07-11 07:33:38.647714 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:38.398151) v4 -- 0x55a3203f7e00 con 0 > -6368> 2018-07-11 07:33:38.863117 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15885 conn(0x55a311b76800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1854 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:33:39.486755) v4 > -6367> 2018-07-11 07:33:38.863121 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15885 conn(0x55a3170b5800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1854 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:33:39.486755) v4 > -6366> 2018-07-11 07:33:38.863139 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.7 10.8.128.114:0/15885 1854 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:39.486755) v4 ==== 2004+0+0 (4024357504 0 0) 0x55a34de78600 con 0x55a311b76800 > -6365> 2018-07-11 07:33:38.863140 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.7 10.8.128.114:0/15885 1854 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:39.486755) v4 ==== 2004+0+0 (4024357504 0 0) 0x55a332e79a00 con 0x55a3170b5800 > -6364> 2018-07-11 07:33:38.863148 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:39.486755) v4 -- 0x55a324967e00 con 0 > -6363> 2018-07-11 07:33:38.863173 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:39.486755) v4 -- 0x55a329ac3e00 con 0 > -6362> 2018-07-11 07:33:39.525080 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15287 conn(0x55a311b75000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1828 0x55a3265d7200 osd_ping(ping e494 stamp 2018-07-11 07:33:40.738472) v4 > -6361> 2018-07-11 07:33:39.525102 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.4 10.8.128.113:0/15287 1828 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:40.738472) v4 ==== 2004+0+0 (21630679 0 0) 0x55a3265d7200 con 0x55a311b75000 > -6360> 2018-07-11 07:33:39.525117 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:40.738472) v4 -- 0x55a3265d7800 con 0 > -6359> 2018-07-11 07:33:39.525112 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15287 conn(0x55a311b72000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1828 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:33:40.738472) v4 > -6358> 2018-07-11 07:33:39.525134 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.4 10.8.128.113:0/15287 1828 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:40.738472) v4 ==== 2004+0+0 (21630679 0 0) 0x55a324474c00 con 0x55a311b72000 > -6357> 2018-07-11 07:33:39.525145 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:40.738472) v4 -- 0x55a33047e000 con 0 > -6356> 2018-07-11 07:33:39.554127 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15919 conn(0x55a317074800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1840 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:33:40.177754) v4 > -6355> 2018-07-11 07:33:39.554130 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15919 conn(0x55a3170b4000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1840 0x55a3203f6400 osd_ping(ping e494 stamp 2018-07-11 07:33:40.177754) v4 > -6354> 2018-07-11 07:33:39.554148 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.11 10.8.128.114:0/15919 1840 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:40.177754) v4 ==== 2004+0+0 (2639754284 0 0) 0x55a332e79a00 con 0x55a317074800 > -6353> 2018-07-11 07:33:39.554149 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.11 10.8.128.114:0/15919 1840 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:40.177754) v4 ==== 2004+0+0 (2639754284 0 0) 0x55a3203f6400 con 0x55a3170b4000 > -6352> 2018-07-11 07:33:39.554157 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:40.177754) v4 -- 0x55a34866ae00 con 0 > -6351> 2018-07-11 07:33:39.554178 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:40.177754) v4 -- 0x55a3265d6200 con 0 > -6350> 2018-07-11 07:33:39.674944 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 59] Generated table #36578: 4957165 keys, 80348270 bytes > -6349> 2018-07-11 07:33:39.675001 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294419674984, "cf_name": "default", "job": 59, "event": "table_file_creation", "file_number": 36578, "file_size": 80348270, "table_properties": {"data_size": 67109406, "index_size": 689173, "filter_size": 12548701, "raw_key_size": 143207000, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957165, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957165", "kMergeOperands": "0"}} > -6348> 2018-07-11 07:33:39.933235 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a33c2466c0 con 0 > -6347> 2018-07-11 07:33:39.946171 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.115:0/13323 conn(0x55a321665800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1847 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:33:40.283434) v4 > -6346> 2018-07-11 07:33:39.946194 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.6 10.8.128.115:0/13323 1847 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:40.283434) v4 ==== 2004+0+0 (2843964507 0 0) 0x55a324474c00 con 0x55a321665800 > -6345> 2018-07-11 07:33:39.946203 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:40.283434) v4 -- 0x55a34866ae00 con 0 > -6344> 2018-07-11 07:33:39.946253 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.115:0/13323 conn(0x55a321668800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1847 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:33:40.283434) v4 > -6343> 2018-07-11 07:33:39.946269 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.6 10.8.128.115:0/13323 1847 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:40.283434) v4 ==== 2004+0+0 (2843964507 0 0) 0x55a324474c00 con 0x55a321668800 > -6342> 2018-07-11 07:33:39.946278 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:40.283434) v4 -- 0x55a3131ba600 con 0 > -6341> 2018-07-11 07:33:39.949139 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5087 0x55a321714000 replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:494/492,start:6:a9c79b5a:::benchmark_data_magna108_35617_object1149216:0,end:6:a9c88ce2:::benchmark_data_magna108_35617_object3280:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -6340> 2018-07-11 07:33:39.949163 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5087 ==== replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:494/492,start:6:a9c79b5a:::benchmark_data_magna108_35617_object1149216:0,end:6:a9c88ce2:::benchmark_data_magna108_35617_object3280:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 217+0+0 (4006827280 0 0) 0x55a321714000 con 0x55a317019000 > -6339> 2018-07-11 07:33:40.061732 7f3543899700 5 osd.10 494 heartbeat: osd_stat(784 GB used, 147 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,1]) > -6338> 2018-07-11 07:33:40.061748 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6802/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:33:40.061746) v4 -- 0x55a331c64a00 con 0 > -6337> 2018-07-11 07:33:40.061785 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6803/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:33:40.061746) v4 -- 0x55a31334da00 con 0 > -6336> 2018-07-11 07:33:40.061802 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6806/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:33:40.061746) v4 -- 0x55a331c64000 con 0 > -6335> 2018-07-11 07:33:40.061817 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6807/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:33:40.061746) v4 -- 0x55a331c65200 con 0 > -6334> 2018-07-11 07:33:40.061836 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6808/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:33:40.061746) v4 -- 0x55a331c64200 con 0 > -6333> 2018-07-11 07:33:40.061858 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6810/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:33:40.061746) v4 -- 0x55a331c65600 con 0 > -6332> 2018-07-11 07:33:40.061883 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6809/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:33:40.061746) v4 -- 0x55a331c65e00 con 0 > -6331> 2018-07-11 07:33:40.061900 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6811/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:33:40.061746) v4 -- 0x55a331c64800 con 0 > -6330> 2018-07-11 07:33:40.061921 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6810/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:33:40.061746) v4 -- 0x55a331c65a00 con 0 > -6329> 2018-07-11 07:33:40.061942 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6811/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:33:40.061746) v4 -- 0x55a32f9b9000 con 0 > -6328> 2018-07-11 07:33:40.061967 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6806/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:33:40.061746) v4 -- 0x55a331c65000 con 0 > -6327> 2018-07-11 07:33:40.061984 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6807/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:33:40.061746) v4 -- 0x55a331c66000 con 0 > -6326> 2018-07-11 07:33:40.062010 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6802/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:33:40.061746) v4 -- 0x55a32f9b9a00 con 0 > -6325> 2018-07-11 07:33:40.062035 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6803/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:33:40.061746) v4 -- 0x55a331c66200 con 0 > -6324> 2018-07-11 07:33:40.062076 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6806/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:33:40.061746) v4 -- 0x55a331c66400 con 0 > -6323> 2018-07-11 07:33:40.062105 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6807/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:33:40.061746) v4 -- 0x55a31bbed400 con 0 > -6322> 2018-07-11 07:33:40.062123 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6802/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:33:40.061746) v4 -- 0x55a31bbed800 con 0 > -6321> 2018-07-11 07:33:40.062143 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6803/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:33:40.061746) v4 -- 0x55a32f9b9c00 con 0 > -6320> 2018-07-11 07:33:40.062199 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6806/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:33:40.061746) v4 -- 0x55a31bbeda00 con 0 > -6319> 2018-07-11 07:33:40.062199 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6803/15255 conn(0x55a314e1d800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1836 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:40.061746) v4 > -6318> 2018-07-11 07:33:40.062216 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6803/15255 1836 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:40.061746) v4 ==== 2004+0+0 (3793346237 0 0) 0x55a324474c00 con 0x55a314e1d800 > -6317> 2018-07-11 07:33:40.062218 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6807/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:33:40.061746) v4 -- 0x55a31bbece00 con 0 > -6316> 2018-07-11 07:33:40.062209 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6802/15255 conn(0x55a314e1c000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1836 0x55a3265d7200 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:40.061746) v4 > -6315> 2018-07-11 07:33:40.062260 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6802/15255 1836 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:40.061746) v4 ==== 2004+0+0 (3793346237 0 0) 0x55a3265d7200 con 0x55a314e1c000 > -6314> 2018-07-11 07:33:40.062269 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6802/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:33:40.061746) v4 -- 0x55a31bbed600 con 0 > -6313> 2018-07-11 07:33:40.062291 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6803/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:33:40.061746) v4 -- 0x55a31334c200 con 0 > -6312> 2018-07-11 07:33:40.062376 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6809/12650 conn(0x55a314e1f000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1836 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:40.061746) v4 > -6311> 2018-07-11 07:33:40.062400 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6809/12650 1836 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:40.061746) v4 ==== 2004+0+0 (3793346237 0 0) 0x55a34de6f000 con 0x55a314e1f000 > -6310> 2018-07-11 07:33:40.062437 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6810/12651 conn(0x55a314e97800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1836 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:40.061746) v4 > -6309> 2018-07-11 07:33:40.062414 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6807/12677 conn(0x55a314e85800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1836 0x55a3265d6200 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:40.061746) v4 > -6308> 2018-07-11 07:33:40.062448 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6806/12677 conn(0x55a314e84000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1836 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:40.061746) v4 > -6307> 2018-07-11 07:33:40.062460 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6806/12677 1836 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:40.061746) v4 ==== 2004+0+0 (3793346237 0 0) 0x55a34de6f000 con 0x55a314e84000 > -6306> 2018-07-11 07:33:40.062461 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6807/12677 1836 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:40.061746) v4 ==== 2004+0+0 (3793346237 0 0) 0x55a3265d6200 con 0x55a314e85800 > -6305> 2018-07-11 07:33:40.062459 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6810/12651 1836 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:40.061746) v4 ==== 2004+0+0 (3793346237 0 0) 0x55a332e79a00 con 0x55a314e97800 > -6304> 2018-07-11 07:33:40.062500 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6806/15263 conn(0x55a314e99000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1836 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:40.061746) v4 > -6303> 2018-07-11 07:33:40.062503 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6811/15287 conn(0x55a314e04800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1836 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:40.061746) v4 > -6302> 2018-07-11 07:33:40.062519 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6806/15263 1836 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:40.061746) v4 ==== 2004+0+0 (3793346237 0 0) 0x55a34de6f000 con 0x55a314e99000 > -6301> 2018-07-11 07:33:40.062518 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6810/15287 conn(0x55a314e03000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1836 0x55a3265d6200 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:40.061746) v4 > -6300> 2018-07-11 07:33:40.062523 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6811/15287 1836 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:40.061746) v4 ==== 2004+0+0 (3793346237 0 0) 0x55a332e79a00 con 0x55a314e04800 > -6299> 2018-07-11 07:33:40.062534 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6810/15287 1836 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:40.061746) v4 ==== 2004+0+0 (3793346237 0 0) 0x55a3265d6200 con 0x55a314e03000 > -6298> 2018-07-11 07:33:40.062549 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6807/12617 conn(0x55a314e27800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1836 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:40.061746) v4 > -6297> 2018-07-11 07:33:40.062561 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6807/12617 1836 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:40.061746) v4 ==== 2004+0+0 (3793346237 0 0) 0x55a34de6f000 con 0x55a314e27800 > -6296> 2018-07-11 07:33:40.062573 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6802/13323 conn(0x55a314e15000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1836 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:40.061746) v4 > -6295> 2018-07-11 07:33:40.062590 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6802/13323 1836 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:40.061746) v4 ==== 2004+0+0 (3793346237 0 0) 0x55a332e79a00 con 0x55a314e15000 > -6294> 2018-07-11 07:33:40.062590 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6807/15885 conn(0x55a314e01800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1836 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:40.061746) v4 > -6293> 2018-07-11 07:33:40.062584 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6807/15263 conn(0x55a314e9a800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1836 0x55a3265d6200 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:40.061746) v4 > -6292> 2018-07-11 07:33:40.062600 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6807/15885 1836 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:40.061746) v4 ==== 2004+0+0 (3793346237 0 0) 0x55a34de6f000 con 0x55a314e01800 > -6291> 2018-07-11 07:33:40.062602 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6807/15263 1836 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:40.061746) v4 ==== 2004+0+0 (3793346237 0 0) 0x55a3265d6200 con 0x55a314e9a800 > -6290> 2018-07-11 07:33:40.062623 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6806/15885 conn(0x55a314e00000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1836 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:40.061746) v4 > -6289> 2018-07-11 07:33:40.062639 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6806/15885 1836 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:40.061746) v4 ==== 2004+0+0 (3793346237 0 0) 0x55a332e79a00 con 0x55a314e00000 > -6288> 2018-07-11 07:33:40.062635 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6803/13323 conn(0x55a314e16800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1836 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:40.061746) v4 > -6287> 2018-07-11 07:33:40.062645 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6811/12650 conn(0x55a314e20800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1836 0x55a3265d6200 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:40.061746) v4 > -6286> 2018-07-11 07:33:40.062657 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6803/13323 1836 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:40.061746) v4 ==== 2004+0+0 (3793346237 0 0) 0x55a34de6f000 con 0x55a314e16800 > -6285> 2018-07-11 07:33:40.062663 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6806/12617 conn(0x55a314e26000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1836 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:40.061746) v4 > -6284> 2018-07-11 07:33:40.062662 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6811/12650 1836 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:40.061746) v4 ==== 2004+0+0 (3793346237 0 0) 0x55a3265d6200 con 0x55a314e20800 > -6283> 2018-07-11 07:33:40.062673 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6806/12617 1836 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:40.061746) v4 ==== 2004+0+0 (3793346237 0 0) 0x55a332e79a00 con 0x55a314e26000 > -6282> 2018-07-11 07:33:40.062691 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6802/11967 conn(0x55a314e12000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1836 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:40.061746) v4 > -6281> 2018-07-11 07:33:40.062703 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6802/11967 1836 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:40.061746) v4 ==== 2004+0+0 (3793346237 0 0) 0x55a34de6f000 con 0x55a314e12000 > -6280> 2018-07-11 07:33:40.062717 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6808/12651 conn(0x55a314e96000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1836 0x55a3265d6200 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:40.061746) v4 > -6279> 2018-07-11 07:33:40.062736 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6808/12651 1836 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:40.061746) v4 ==== 2004+0+0 (3793346237 0 0) 0x55a3265d6200 con 0x55a314e96000 > -6278> 2018-07-11 07:33:40.062741 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6803/15919 conn(0x55a314e88800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1836 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:40.061746) v4 > -6277> 2018-07-11 07:33:40.062760 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6803/15919 1836 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:40.061746) v4 ==== 2004+0+0 (3793346237 0 0) 0x55a34de6f000 con 0x55a314e88800 > -6276> 2018-07-11 07:33:40.062776 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6802/15919 conn(0x55a314e87000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1836 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:40.061746) v4 > -6275> 2018-07-11 07:33:40.062779 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6803/11967 conn(0x55a314e13800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1836 0x55a3265d6200 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:40.061746) v4 > -6274> 2018-07-11 07:33:40.062798 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6802/15919 1836 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:40.061746) v4 ==== 2004+0+0 (3793346237 0 0) 0x55a332e79a00 con 0x55a314e87000 > -6273> 2018-07-11 07:33:40.062804 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6803/11967 1836 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:40.061746) v4 ==== 2004+0+0 (3793346237 0 0) 0x55a3265d6200 con 0x55a314e13800 > -6272> 2018-07-11 07:33:40.573449 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12650 conn(0x55a311b67000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1851 0x55a3265d6200 osd_ping(ping e494 stamp 2018-07-11 07:33:40.608881) v4 > -6271> 2018-07-11 07:33:40.573478 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.3 10.8.128.106:0/12650 1851 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:40.608881) v4 ==== 2004+0+0 (2774571577 0 0) 0x55a3265d6200 con 0x55a311b67000 > -6270> 2018-07-11 07:33:40.573498 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:40.608881) v4 -- 0x55a331c64a00 con 0 > -6269> 2018-07-11 07:33:40.573505 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12650 conn(0x55a311b68800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1851 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:33:40.608881) v4 > -6268> 2018-07-11 07:33:40.573523 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.3 10.8.128.106:0/12650 1851 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:40.608881) v4 ==== 2004+0+0 (2774571577 0 0) 0x55a34de6f000 con 0x55a311b68800 > -6267> 2018-07-11 07:33:40.573539 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:40.608881) v4 -- 0x55a331c64000 con 0 > -6266> 2018-07-11 07:33:40.600082 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a324ad6240 con 0 > -6265> 2018-07-11 07:33:40.615680 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5088 0x55a321714280 replica scrub(pg: 6.15,from:0'0,to:430'106802,epoch:494/492,start:6:a9c88ce2:::benchmark_data_magna108_35617_object3280:0,end:6:a9c91a90:::benchmark_data_magna108_35617_object354083:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -6264> 2018-07-11 07:33:40.615703 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5088 ==== replica scrub(pg: 6.15,from:0'0,to:430'106802,epoch:494/492,start:6:a9c88ce2:::benchmark_data_magna108_35617_object3280:0,end:6:a9c91a90:::benchmark_data_magna108_35617_object354083:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 216+0+0 (2533116117 0 0) 0x55a321714280 con 0x55a317019000 > -6263> 2018-07-11 07:33:40.761957 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12651 conn(0x55a3216a0800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1837 0x55a3265d7200 osd_ping(ping e494 stamp 2018-07-11 07:33:40.797377) v4 > -6262> 2018-07-11 07:33:40.761961 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12651 conn(0x55a32169f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1837 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:33:40.797377) v4 > -6261> 2018-07-11 07:33:40.761983 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.2 10.8.128.106:0/12651 1837 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:40.797377) v4 ==== 2004+0+0 (1585379553 0 0) 0x55a3265d7200 con 0x55a3216a0800 > -6260> 2018-07-11 07:33:40.761987 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.2 10.8.128.106:0/12651 1837 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:40.797377) v4 ==== 2004+0+0 (1585379553 0 0) 0x55a332e79a00 con 0x55a32169f000 > -6259> 2018-07-11 07:33:40.761995 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:40.797377) v4 -- 0x55a3265d6e00 con 0 > -6258> 2018-07-11 07:33:40.762024 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:40.797377) v4 -- 0x55a32f9b9000 con 0 > -6257> 2018-07-11 07:33:41.124932 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15255 conn(0x55a311b65800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1860 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:33:42.338389) v4 > -6256> 2018-07-11 07:33:41.124956 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.0 10.8.128.113:0/15255 1860 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:42.338389) v4 ==== 2004+0+0 (2975233138 0 0) 0x55a324474c00 con 0x55a311b65800 > -6255> 2018-07-11 07:33:41.124958 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15255 conn(0x55a311b64000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1860 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:33:42.338389) v4 > -6254> 2018-07-11 07:33:41.124973 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.338389) v4 -- 0x55a331c65600 con 0 > -6253> 2018-07-11 07:33:41.124981 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.0 10.8.128.113:0/15255 1860 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:42.338389) v4 ==== 2004+0+0 (2975233138 0 0) 0x55a34de78600 con 0x55a311b64000 > -6252> 2018-07-11 07:33:41.125005 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.338389) v4 -- 0x55a324967e00 con 0 > -6251> 2018-07-11 07:33:41.475101 7f355d88e700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.107:0/12617 conn(0x55a32447f800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1809 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:33:41.474770) v4 > -6250> 2018-07-11 07:33:41.475108 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.107:0/12617 conn(0x55a32447e000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1809 0x55a331c64a00 osd_ping(ping e494 stamp 2018-07-11 07:33:41.474770) v4 > -6249> 2018-07-11 07:33:41.475126 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.9 10.8.128.107:0/12617 1809 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:41.474770) v4 ==== 2004+0+0 (1195800378 0 0) 0x55a331c64a00 con 0x55a32447e000 > -6248> 2018-07-11 07:33:41.475126 7f355d88e700 1 -- 10.8.128.107:6803/12665 <== osd.9 10.8.128.107:0/12617 1809 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:41.474770) v4 ==== 2004+0+0 (1195800378 0 0) 0x55a34de6f000 con 0x55a32447f800 > -6247> 2018-07-11 07:33:41.475137 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:41.474770) v4 -- 0x55a3265d7c00 con 0 > -6246> 2018-07-11 07:33:41.475168 7f355d88e700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:41.474770) v4 -- 0x55a3203f7200 con 0 > -6245> 2018-07-11 07:33:41.506647 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a31efc4240 con 0 > -6244> 2018-07-11 07:33:41.526883 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5089 0x55a321714500 replica scrub(pg: 6.15,from:0'0,to:430'106172,epoch:494/492,start:6:a9c91a90:::benchmark_data_magna108_35617_object354083:0,end:6:a9caac0f:::benchmark_data_magna108_35617_object381435:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -6243> 2018-07-11 07:33:41.526908 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5089 ==== replica scrub(pg: 6.15,from:0'0,to:430'106172,epoch:494/492,start:6:a9c91a90:::benchmark_data_magna108_35617_object354083:0,end:6:a9caac0f:::benchmark_data_magna108_35617_object381435:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (1158354968 0 0) 0x55a321714500 con 0x55a317019000 > -6242> 2018-07-11 07:33:41.646630 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.115:0/13323 conn(0x55a321668800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1848 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:33:41.983911) v4 > -6241> 2018-07-11 07:33:41.646651 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.6 10.8.128.115:0/13323 1848 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:41.983911) v4 ==== 2004+0+0 (3426809984 0 0) 0x55a332e79a00 con 0x55a321668800 > -6240> 2018-07-11 07:33:41.646663 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:41.983911) v4 -- 0x55a331c65600 con 0 > -6239> 2018-07-11 07:33:41.646715 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.115:0/13323 conn(0x55a321665800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1848 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:33:41.983911) v4 > -6238> 2018-07-11 07:33:41.646729 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.6 10.8.128.115:0/13323 1848 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:41.983911) v4 ==== 2004+0+0 (3426809984 0 0) 0x55a324474c00 con 0x55a321665800 > -6237> 2018-07-11 07:33:41.646738 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:41.983911) v4 -- 0x55a331c65600 con 0 > -6236> 2018-07-11 07:33:41.716176 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15263 conn(0x55a321664000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1835 0x55a3265d7200 osd_ping(ping e494 stamp 2018-07-11 07:33:42.929494) v4 > -6235> 2018-07-11 07:33:41.716199 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.5 10.8.128.113:0/15263 1835 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:42.929494) v4 ==== 2004+0+0 (1113817426 0 0) 0x55a3265d7200 con 0x55a321664000 > -6234> 2018-07-11 07:33:41.716213 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.929494) v4 -- 0x55a3265d7600 con 0 > -6233> 2018-07-11 07:33:41.716303 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15263 conn(0x55a321667000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1835 0x55a3265d7c00 osd_ping(ping e494 stamp 2018-07-11 07:33:42.929494) v4 > -6232> 2018-07-11 07:33:41.716319 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.5 10.8.128.113:0/15263 1835 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:42.929494) v4 ==== 2004+0+0 (1113817426 0 0) 0x55a3265d7c00 con 0x55a321667000 > -6231> 2018-07-11 07:33:41.716329 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.929494) v4 -- 0x55a3265d7400 con 0 > -6230> 2018-07-11 07:33:41.825422 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15287 conn(0x55a311b75000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1829 0x55a3265d7200 osd_ping(ping e494 stamp 2018-07-11 07:33:43.038845) v4 > -6229> 2018-07-11 07:33:41.825421 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15287 conn(0x55a311b72000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1829 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:33:43.038845) v4 > -6228> 2018-07-11 07:33:41.825449 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.4 10.8.128.113:0/15287 1829 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:43.038845) v4 ==== 2004+0+0 (189507026 0 0) 0x55a3265d7200 con 0x55a311b75000 > -6227> 2018-07-11 07:33:41.825452 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.4 10.8.128.113:0/15287 1829 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:43.038845) v4 ==== 2004+0+0 (189507026 0 0) 0x55a332e79a00 con 0x55a311b72000 > -6226> 2018-07-11 07:33:41.825468 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:43.038845) v4 -- 0x55a3265d7a00 con 0 > -6225> 2018-07-11 07:33:41.825497 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:43.038845) v4 -- 0x55a31334da00 con 0 > -6224> 2018-07-11 07:33:42.154836 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a321cf9d40 con 0 > -6223> 2018-07-11 07:33:42.166159 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5090 0x55a321714780 replica scrub(pg: 6.15,from:0'0,to:430'105823,epoch:494/492,start:6:a9caac0f:::benchmark_data_magna108_35617_object381435:0,end:6:a9cba892:::benchmark_data_magna108_35617_object1126049:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -6222> 2018-07-11 07:33:42.166187 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5090 ==== replica scrub(pg: 6.15,from:0'0,to:430'105823,epoch:494/492,start:6:a9caac0f:::benchmark_data_magna108_35617_object381435:0,end:6:a9cba892:::benchmark_data_magna108_35617_object1126049:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 219+0+0 (2901495631 0 0) 0x55a321714780 con 0x55a317019000 > -6221> 2018-07-11 07:33:42.252876 7f35488a3700 1 -- 10.8.128.107:6800/12665 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55a3170ea580 con 0 > -6220> 2018-07-11 07:33:42.252952 7f35488a3700 1 -- 10.8.128.107:6800/12665 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55a30f75eb00 con 0 > -6219> 2018-07-11 07:33:42.325930 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15287 conn(0x55a311b72000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1830 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:33:43.539324) v4 > -6218> 2018-07-11 07:33:42.325957 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.4 10.8.128.113:0/15287 1830 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:43.539324) v4 ==== 2004+0+0 (2203617761 0 0) 0x55a324474c00 con 0x55a311b72000 > -6217> 2018-07-11 07:33:42.325950 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15287 conn(0x55a311b75000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1830 0x55a3265d7c00 osd_ping(ping e494 stamp 2018-07-11 07:33:43.539324) v4 > -6216> 2018-07-11 07:33:42.325966 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:43.539324) v4 -- 0x55a31334da00 con 0 > -6215> 2018-07-11 07:33:42.325967 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.4 10.8.128.113:0/15287 1830 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:43.539324) v4 ==== 2004+0+0 (2203617761 0 0) 0x55a3265d7c00 con 0x55a311b75000 > -6214> 2018-07-11 07:33:42.325992 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:43.539324) v4 -- 0x55a329323400 con 0 > -6213> 2018-07-11 07:33:42.362454 7f3543899700 5 osd.10 494 heartbeat: osd_stat(784 GB used, 147 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,1]) > -6212> 2018-07-11 07:33:42.362468 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6802/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:33:42.362467) v4 -- 0x55a31334c000 con 0 > -6211> 2018-07-11 07:33:42.362492 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6803/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:33:42.362467) v4 -- 0x55a31334c600 con 0 > -6210> 2018-07-11 07:33:42.362509 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6806/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:33:42.362467) v4 -- 0x55a31334c800 con 0 > -6209> 2018-07-11 07:33:42.362532 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6807/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:33:42.362467) v4 -- 0x55a31334cc00 con 0 > -6208> 2018-07-11 07:33:42.362556 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6808/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:33:42.362467) v4 -- 0x55a31334ce00 con 0 > -6207> 2018-07-11 07:33:42.362577 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6810/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:33:42.362467) v4 -- 0x55a31bbedc00 con 0 > -6206> 2018-07-11 07:33:42.362592 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6809/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:33:42.362467) v4 -- 0x55a331c66a00 con 0 > -6205> 2018-07-11 07:33:42.362612 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6811/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:33:42.362467) v4 -- 0x55a324af4400 con 0 > -6204> 2018-07-11 07:33:42.362636 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6810/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:33:42.362467) v4 -- 0x55a324af4800 con 0 > -6203> 2018-07-11 07:33:42.362659 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6811/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:33:42.362467) v4 -- 0x55a324af4600 con 0 > -6202> 2018-07-11 07:33:42.362686 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6806/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:33:42.362467) v4 -- 0x55a331c66600 con 0 > -6201> 2018-07-11 07:33:42.362707 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6807/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:33:42.362467) v4 -- 0x55a31bbed000 con 0 > -6200> 2018-07-11 07:33:42.362731 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6802/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:33:42.362467) v4 -- 0x55a331c66c00 con 0 > -6199> 2018-07-11 07:33:42.362752 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6803/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:33:42.362467) v4 -- 0x55a31334dc00 con 0 > -6198> 2018-07-11 07:33:42.362821 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6806/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:33:42.362467) v4 -- 0x55a331c67200 con 0 > -6197> 2018-07-11 07:33:42.362847 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6807/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:33:42.362467) v4 -- 0x55a331c66800 con 0 > -6196> 2018-07-11 07:33:42.362821 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6802/15255 conn(0x55a314e1c000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1837 0x55a3265d7a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.362467) v4 > -6195> 2018-07-11 07:33:42.362864 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6802/15255 1837 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.362467) v4 ==== 2004+0+0 (936433745 0 0) 0x55a3265d7a00 con 0x55a314e1c000 > -6194> 2018-07-11 07:33:42.362872 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6802/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:33:42.362467) v4 -- 0x55a331c67a00 con 0 > -6193> 2018-07-11 07:33:42.362893 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6803/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:33:42.362467) v4 -- 0x55a331c67000 con 0 > -6192> 2018-07-11 07:33:42.362935 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6803/15255 conn(0x55a314e1d800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1837 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.362467) v4 > -6191> 2018-07-11 07:33:42.362955 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6803/15255 1837 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.362467) v4 ==== 2004+0+0 (936433745 0 0) 0x55a324474c00 con 0x55a314e1d800 > -6190> 2018-07-11 07:33:42.362972 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6806/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:33:42.362467) v4 -- 0x55a32f9b9e00 con 0 > -6189> 2018-07-11 07:33:42.362991 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6807/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:33:42.362467) v4 -- 0x55a32e330000 con 0 > -6188> 2018-07-11 07:33:42.363005 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6802/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:33:42.362467) v4 -- 0x55a331c67800 con 0 > -6187> 2018-07-11 07:33:42.363016 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6803/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:33:42.362467) v4 -- 0x55a32e330400 con 0 > -6186> 2018-07-11 07:33:42.363065 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6806/12677 conn(0x55a314e84000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1837 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.362467) v4 > -6185> 2018-07-11 07:33:42.363083 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6807/12677 conn(0x55a314e85800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1837 0x55a3265d7c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.362467) v4 > -6184> 2018-07-11 07:33:42.363089 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6806/12677 1837 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.362467) v4 ==== 2004+0+0 (936433745 0 0) 0x55a34de6f000 con 0x55a314e84000 > -6183> 2018-07-11 07:33:42.363099 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6807/12677 1837 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.362467) v4 ==== 2004+0+0 (936433745 0 0) 0x55a3265d7c00 con 0x55a314e85800 > -6182> 2018-07-11 07:33:42.363117 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6810/12651 conn(0x55a314e97800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1837 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.362467) v4 > -6181> 2018-07-11 07:33:42.363133 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6810/12651 1837 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.362467) v4 ==== 2004+0+0 (936433745 0 0) 0x55a332e79a00 con 0x55a314e97800 > -6180> 2018-07-11 07:33:42.363134 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6808/12651 conn(0x55a314e96000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1837 0x55a3265d7c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.362467) v4 > -6179> 2018-07-11 07:33:42.363144 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6808/12651 1837 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.362467) v4 ==== 2004+0+0 (936433745 0 0) 0x55a3265d7c00 con 0x55a314e96000 > -6178> 2018-07-11 07:33:42.363162 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6809/12650 conn(0x55a314e1f000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1837 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.362467) v4 > -6177> 2018-07-11 07:33:42.363174 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6810/15287 conn(0x55a314e03000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1837 0x55a3265d7c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.362467) v4 > -6176> 2018-07-11 07:33:42.363182 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6809/12650 1837 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.362467) v4 ==== 2004+0+0 (936433745 0 0) 0x55a34de6f000 con 0x55a314e1f000 > -6175> 2018-07-11 07:33:42.363184 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6810/15287 1837 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.362467) v4 ==== 2004+0+0 (936433745 0 0) 0x55a3265d7c00 con 0x55a314e03000 > -6174> 2018-07-11 07:33:42.363173 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6811/15287 conn(0x55a314e04800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1837 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.362467) v4 > -6173> 2018-07-11 07:33:42.363209 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6811/15287 1837 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.362467) v4 ==== 2004+0+0 (936433745 0 0) 0x55a332e79a00 con 0x55a314e04800 > -6172> 2018-07-11 07:33:42.363230 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6811/12650 conn(0x55a314e20800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1837 0x55a3265d7c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.362467) v4 > -6171> 2018-07-11 07:33:42.363233 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6806/15263 conn(0x55a314e99000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1837 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.362467) v4 > -6170> 2018-07-11 07:33:42.363242 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6811/12650 1837 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.362467) v4 ==== 2004+0+0 (936433745 0 0) 0x55a3265d7c00 con 0x55a314e20800 > -6169> 2018-07-11 07:33:42.363261 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6806/15263 1837 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.362467) v4 ==== 2004+0+0 (936433745 0 0) 0x55a34de78600 con 0x55a314e99000 > -6168> 2018-07-11 07:33:42.363261 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6802/13323 conn(0x55a314e15000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1837 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.362467) v4 > -6167> 2018-07-11 07:33:42.363277 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6802/13323 1837 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.362467) v4 ==== 2004+0+0 (936433745 0 0) 0x55a332e79a00 con 0x55a314e15000 > -6166> 2018-07-11 07:33:42.363279 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6807/15263 conn(0x55a314e9a800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1837 0x55a3265d7c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.362467) v4 > -6165> 2018-07-11 07:33:42.363288 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6807/15263 1837 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.362467) v4 ==== 2004+0+0 (936433745 0 0) 0x55a3265d7c00 con 0x55a314e9a800 > -6164> 2018-07-11 07:33:42.363298 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6803/13323 conn(0x55a314e16800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1837 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.362467) v4 > -6163> 2018-07-11 07:33:42.363306 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6806/12617 conn(0x55a314e26000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1837 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.362467) v4 > -6162> 2018-07-11 07:33:42.363314 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6803/13323 1837 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.362467) v4 ==== 2004+0+0 (936433745 0 0) 0x55a34de78600 con 0x55a314e16800 > -6161> 2018-07-11 07:33:42.363315 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6806/12617 1837 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.362467) v4 ==== 2004+0+0 (936433745 0 0) 0x55a332e79a00 con 0x55a314e26000 > -6160> 2018-07-11 07:33:42.363341 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6807/12617 conn(0x55a314e27800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1837 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.362467) v4 > -6159> 2018-07-11 07:33:42.363351 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6807/12617 1837 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.362467) v4 ==== 2004+0+0 (936433745 0 0) 0x55a34de78600 con 0x55a314e27800 > -6158> 2018-07-11 07:33:42.363383 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6802/11967 conn(0x55a314e12000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1837 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.362467) v4 > -6157> 2018-07-11 07:33:42.363395 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6802/11967 1837 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.362467) v4 ==== 2004+0+0 (936433745 0 0) 0x55a34de78600 con 0x55a314e12000 > -6156> 2018-07-11 07:33:42.363414 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15885 conn(0x55a3170b5800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1855 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:33:42.987223) v4 > -6155> 2018-07-11 07:33:42.363427 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15885 conn(0x55a311b76800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1855 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:33:42.987223) v4 > -6154> 2018-07-11 07:33:42.363434 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.7 10.8.128.114:0/15885 1855 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:42.987223) v4 ==== 2004+0+0 (3511656371 0 0) 0x55a332e79a00 con 0x55a3170b5800 > -6153> 2018-07-11 07:33:42.363444 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.7 10.8.128.114:0/15885 1855 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:42.987223) v4 ==== 2004+0+0 (3511656371 0 0) 0x55a34de78600 con 0x55a311b76800 > -6152> 2018-07-11 07:33:42.363466 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6803/11967 conn(0x55a314e13800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1837 0x55a3265d7c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.362467) v4 > -6151> 2018-07-11 07:33:42.363458 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.987223) v4 -- 0x55a324af4600 con 0 > -6150> 2018-07-11 07:33:42.363486 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6803/11967 1837 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.362467) v4 ==== 2004+0+0 (936433745 0 0) 0x55a3265d7c00 con 0x55a314e13800 > -6149> 2018-07-11 07:33:42.363505 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.987223) v4 -- 0x55a31334dc00 con 0 > -6148> 2018-07-11 07:33:42.363548 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6802/15919 conn(0x55a314e87000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1837 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.362467) v4 > -6147> 2018-07-11 07:33:42.363563 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6802/15919 1837 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.362467) v4 ==== 2004+0+0 (936433745 0 0) 0x55a324474c00 con 0x55a314e87000 > -6146> 2018-07-11 07:33:42.363665 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6806/15885 conn(0x55a314e00000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1837 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.362467) v4 > -6145> 2018-07-11 07:33:42.363684 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6806/15885 1837 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.362467) v4 ==== 2004+0+0 (936433745 0 0) 0x55a324474c00 con 0x55a314e00000 > -6144> 2018-07-11 07:33:42.363739 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6803/15919 conn(0x55a314e88800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1837 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.362467) v4 > -6143> 2018-07-11 07:33:42.363754 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6803/15919 1837 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.362467) v4 ==== 2004+0+0 (936433745 0 0) 0x55a34de6f000 con 0x55a314e88800 > -6142> 2018-07-11 07:33:42.363779 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6807/15885 conn(0x55a314e01800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1837 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.362467) v4 > -6141> 2018-07-11 07:33:42.363789 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6807/15885 1837 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.362467) v4 ==== 2004+0+0 (936433745 0 0) 0x55a34de6f000 con 0x55a314e01800 > -6140> 2018-07-11 07:33:42.795371 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a324aef680 con 0 > -6139> 2018-07-11 07:33:42.808196 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5091 0x55a321714a00 replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:494/492,start:6:a9cba892:::benchmark_data_magna108_35617_object1126049:0,end:6:a9cc802f:::benchmark_data_magna108_35617_object1061012:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -6138> 2018-07-11 07:33:42.808219 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5091 ==== replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:494/492,start:6:a9cba892:::benchmark_data_magna108_35617_object1126049:0,end:6:a9cc802f:::benchmark_data_magna108_35617_object1061012:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 220+0+0 (3422956987 0 0) 0x55a321714a00 con 0x55a317019000 > -6137> 2018-07-11 07:33:42.863150 7f3543899700 5 osd.10 494 heartbeat: osd_stat(784 GB used, 147 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,1]) > -6136> 2018-07-11 07:33:42.863164 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6802/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:33:42.863163) v4 -- 0x55a331c66e00 con 0 > -6135> 2018-07-11 07:33:42.863204 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6803/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:33:42.863163) v4 -- 0x55a331c67400 con 0 > -6134> 2018-07-11 07:33:42.863220 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6806/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:33:42.863163) v4 -- 0x55a331c67e00 con 0 > -6133> 2018-07-11 07:33:42.863236 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6807/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:33:42.863163) v4 -- 0x55a32e330600 con 0 > -6132> 2018-07-11 07:33:42.863264 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6808/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:33:42.863163) v4 -- 0x55a331c67600 con 0 > -6131> 2018-07-11 07:33:42.863277 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6810/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:33:42.863163) v4 -- 0x55a32e330200 con 0 > -6130> 2018-07-11 07:33:42.863290 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6809/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:33:42.863163) v4 -- 0x55a32e330c00 con 0 > -6129> 2018-07-11 07:33:42.863301 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6811/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:33:42.863163) v4 -- 0x55a32e330a00 con 0 > -6128> 2018-07-11 07:33:42.863313 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6810/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:33:42.863163) v4 -- 0x55a331c67c00 con 0 > -6127> 2018-07-11 07:33:42.863324 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6811/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:33:42.863163) v4 -- 0x55a32e331000 con 0 > -6126> 2018-07-11 07:33:42.863336 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6806/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:33:42.863163) v4 -- 0x55a32e330800 con 0 > -6125> 2018-07-11 07:33:42.863355 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6807/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:33:42.863163) v4 -- 0x55a32e330e00 con 0 > -6124> 2018-07-11 07:33:42.863370 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6802/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:33:42.863163) v4 -- 0x55a331c64600 con 0 > -6123> 2018-07-11 07:33:42.863387 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6803/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:33:42.863163) v4 -- 0x55a331c64e00 con 0 > -6122> 2018-07-11 07:33:42.863403 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6806/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:33:42.863163) v4 -- 0x55a324af4200 con 0 > -6121> 2018-07-11 07:33:42.863416 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6807/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:33:42.863163) v4 -- 0x55a331c64400 con 0 > -6120> 2018-07-11 07:33:42.863430 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6802/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:33:42.863163) v4 -- 0x55a33be6f600 con 0 > -6119> 2018-07-11 07:33:42.863441 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6803/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:33:42.863163) v4 -- 0x55a33be6f200 con 0 > -6118> 2018-07-11 07:33:42.863454 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6806/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:33:42.863163) v4 -- 0x55a31e8f5a00 con 0 > -6117> 2018-07-11 07:33:42.863466 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6807/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:33:42.863163) v4 -- 0x55a324af5000 con 0 > -6116> 2018-07-11 07:33:42.863500 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6802/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:33:42.863163) v4 -- 0x55a33be6f000 con 0 > -6115> 2018-07-11 07:33:42.863516 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6803/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:33:42.863163) v4 -- 0x55a31bbed200 con 0 > -6114> 2018-07-11 07:33:42.863608 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6802/15255 conn(0x55a314e1c000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1838 0x55a31334cc00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.863163) v4 > -6113> 2018-07-11 07:33:42.863625 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6802/15255 1838 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.863163) v4 ==== 2004+0+0 (4250454317 0 0) 0x55a31334cc00 con 0x55a314e1c000 > -6112> 2018-07-11 07:33:42.863638 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6806/12617 conn(0x55a314e26000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1838 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.863163) v4 > -6111> 2018-07-11 07:33:42.863681 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6806/12617 1838 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.863163) v4 ==== 2004+0+0 (4250454317 0 0) 0x55a332e79a00 con 0x55a314e26000 > -6110> 2018-07-11 07:33:42.863681 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6807/12617 conn(0x55a314e27800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1838 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.863163) v4 > -6109> 2018-07-11 07:33:42.863717 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6807/12617 1838 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.863163) v4 ==== 2004+0+0 (4250454317 0 0) 0x55a34de78600 con 0x55a314e27800 > -6108> 2018-07-11 07:33:42.863717 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6808/12651 conn(0x55a314e96000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1838 0x55a31334cc00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.863163) v4 > -6107> 2018-07-11 07:33:42.863730 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6808/12651 1838 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.863163) v4 ==== 2004+0+0 (4250454317 0 0) 0x55a31334cc00 con 0x55a314e96000 > -6106> 2018-07-11 07:33:42.863730 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6803/15255 conn(0x55a314e1d800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1838 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.863163) v4 > -6105> 2018-07-11 07:33:42.863741 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6803/15255 1838 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.863163) v4 ==== 2004+0+0 (4250454317 0 0) 0x55a332e79a00 con 0x55a314e1d800 > -6104> 2018-07-11 07:33:42.863757 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6809/12650 conn(0x55a314e1f000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1838 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.863163) v4 > -6103> 2018-07-11 07:33:42.863766 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6807/12677 conn(0x55a314e85800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1838 0x55a31334cc00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.863163) v4 > -6102> 2018-07-11 07:33:42.863770 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6809/12650 1838 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.863163) v4 ==== 2004+0+0 (4250454317 0 0) 0x55a34de78600 con 0x55a314e1f000 > -6101> 2018-07-11 07:33:42.863775 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6810/12651 conn(0x55a314e97800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1838 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.863163) v4 > -6100> 2018-07-11 07:33:42.863790 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6810/12651 1838 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.863163) v4 ==== 2004+0+0 (4250454317 0 0) 0x55a332e79a00 con 0x55a314e97800 > -6099> 2018-07-11 07:33:42.863792 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6807/12677 1838 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.863163) v4 ==== 2004+0+0 (4250454317 0 0) 0x55a31334cc00 con 0x55a314e85800 > -6098> 2018-07-11 07:33:42.863821 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6811/15287 conn(0x55a314e04800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1838 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.863163) v4 > -6097> 2018-07-11 07:33:42.863825 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6806/12677 conn(0x55a314e84000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1838 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.863163) v4 > -6096> 2018-07-11 07:33:42.863833 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6811/15287 1838 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.863163) v4 ==== 2004+0+0 (4250454317 0 0) 0x55a332e79a00 con 0x55a314e04800 > -6095> 2018-07-11 07:33:42.863836 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6806/12677 1838 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.863163) v4 ==== 2004+0+0 (4250454317 0 0) 0x55a34de78600 con 0x55a314e84000 > -6094> 2018-07-11 07:33:42.863827 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6810/15287 conn(0x55a314e03000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1838 0x55a31334cc00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.863163) v4 > -6093> 2018-07-11 07:33:42.863854 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6810/15287 1838 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.863163) v4 ==== 2004+0+0 (4250454317 0 0) 0x55a31334cc00 con 0x55a314e03000 > -6092> 2018-07-11 07:33:42.863901 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6806/15263 conn(0x55a314e99000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1838 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.863163) v4 > -6091> 2018-07-11 07:33:42.863914 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6806/15263 1838 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.863163) v4 ==== 2004+0+0 (4250454317 0 0) 0x55a34de78600 con 0x55a314e99000 > -6090> 2018-07-11 07:33:42.863916 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6811/12650 conn(0x55a314e20800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1838 0x55a31334cc00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.863163) v4 > -6089> 2018-07-11 07:33:42.863933 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6803/13323 conn(0x55a314e16800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1838 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.863163) v4 > -6088> 2018-07-11 07:33:42.863934 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6811/12650 1838 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.863163) v4 ==== 2004+0+0 (4250454317 0 0) 0x55a31334cc00 con 0x55a314e20800 > -6087> 2018-07-11 07:33:42.863943 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6803/13323 1838 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.863163) v4 ==== 2004+0+0 (4250454317 0 0) 0x55a34de78600 con 0x55a314e16800 > -6086> 2018-07-11 07:33:42.863949 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6802/13323 conn(0x55a314e15000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1838 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.863163) v4 > -6085> 2018-07-11 07:33:42.863974 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6802/13323 1838 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.863163) v4 ==== 2004+0+0 (4250454317 0 0) 0x55a332e79a00 con 0x55a314e15000 > -6084> 2018-07-11 07:33:42.863997 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6807/15263 conn(0x55a314e9a800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1838 0x55a31334cc00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.863163) v4 > -6083> 2018-07-11 07:33:42.864011 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6807/15263 1838 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.863163) v4 ==== 2004+0+0 (4250454317 0 0) 0x55a31334cc00 con 0x55a314e9a800 > -6082> 2018-07-11 07:33:42.864008 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15885 conn(0x55a311b76800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1856 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:33:43.487616) v4 > -6081> 2018-07-11 07:33:42.864021 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.7 10.8.128.114:0/15885 1856 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:43.487616) v4 ==== 2004+0+0 (566379797 0 0) 0x55a34de78600 con 0x55a311b76800 > -6080> 2018-07-11 07:33:42.864032 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:43.487616) v4 -- 0x55a331c64400 con 0 > -6079> 2018-07-11 07:33:42.864023 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6806/15885 conn(0x55a314e00000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1838 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.863163) v4 > -6078> 2018-07-11 07:33:42.864052 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6806/15885 1838 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.863163) v4 ==== 2004+0+0 (4250454317 0 0) 0x55a332e79a00 con 0x55a314e00000 > -6077> 2018-07-11 07:33:42.864070 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6807/15885 conn(0x55a314e01800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1838 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.863163) v4 > -6076> 2018-07-11 07:33:42.864082 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6807/15885 1838 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.863163) v4 ==== 2004+0+0 (4250454317 0 0) 0x55a34de78600 con 0x55a314e01800 > -6075> 2018-07-11 07:33:42.864103 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15885 conn(0x55a3170b5800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1856 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:33:43.487616) v4 > -6074> 2018-07-11 07:33:42.864122 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.7 10.8.128.114:0/15885 1856 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:43.487616) v4 ==== 2004+0+0 (566379797 0 0) 0x55a332e79a00 con 0x55a3170b5800 > -6073> 2018-07-11 07:33:42.864133 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6802/11967 conn(0x55a314e12000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1838 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.863163) v4 > -6072> 2018-07-11 07:33:42.864144 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:43.487616) v4 -- 0x55a331c67400 con 0 > -6071> 2018-07-11 07:33:42.864145 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6802/11967 1838 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.863163) v4 ==== 2004+0+0 (4250454317 0 0) 0x55a34de6f000 con 0x55a314e12000 > -6070> 2018-07-11 07:33:42.864181 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6803/11967 conn(0x55a314e13800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1838 0x55a31334cc00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.863163) v4 > -6069> 2018-07-11 07:33:42.864197 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6803/15919 conn(0x55a314e88800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1838 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.863163) v4 > -6068> 2018-07-11 07:33:42.864204 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6803/11967 1838 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.863163) v4 ==== 2004+0+0 (4250454317 0 0) 0x55a31334cc00 con 0x55a314e13800 > -6067> 2018-07-11 07:33:42.864212 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6803/15919 1838 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.863163) v4 ==== 2004+0+0 (4250454317 0 0) 0x55a34de6f000 con 0x55a314e88800 > -6066> 2018-07-11 07:33:42.864221 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6802/15919 conn(0x55a314e87000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1838 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.863163) v4 > -6065> 2018-07-11 07:33:42.864241 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6802/15919 1838 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.863163) v4 ==== 2004+0+0 (4250454317 0 0) 0x55a324474c00 con 0x55a314e87000 > -6064> 2018-07-11 07:33:42.925369 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12677 conn(0x55a32169d800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1857 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:33:42.960897) v4 > -6063> 2018-07-11 07:33:42.925390 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.1 10.8.128.106:0/12677 1857 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:42.960897) v4 ==== 2004+0+0 (3897523719 0 0) 0x55a34de6f000 con 0x55a32169d800 > -6062> 2018-07-11 07:33:42.925402 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.960897) v4 -- 0x55a331c67e00 con 0 > -6061> 2018-07-11 07:33:42.925522 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12677 conn(0x55a32169c000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1857 0x55a31334cc00 osd_ping(ping e494 stamp 2018-07-11 07:33:42.960897) v4 > -6060> 2018-07-11 07:33:42.925541 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.1 10.8.128.106:0/12677 1857 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:42.960897) v4 ==== 2004+0+0 (3897523719 0 0) 0x55a31334cc00 con 0x55a32169c000 > -6059> 2018-07-11 07:33:42.925552 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:42.960897) v4 -- 0x55a32e330a00 con 0 > -6058> 2018-07-11 07:33:43.175538 7f355d88e700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.107:0/12617 conn(0x55a32447f800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1810 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:33:43.175184) v4 > -6057> 2018-07-11 07:33:43.175538 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.107:0/12617 conn(0x55a32447e000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1810 0x55a3265d7c00 osd_ping(ping e494 stamp 2018-07-11 07:33:43.175184) v4 > -6056> 2018-07-11 07:33:43.175565 7f355d88e700 1 -- 10.8.128.107:6803/12665 <== osd.9 10.8.128.107:0/12617 1810 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:43.175184) v4 ==== 2004+0+0 (915390258 0 0) 0x55a34de78600 con 0x55a32447f800 > -6055> 2018-07-11 07:33:43.175567 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.9 10.8.128.107:0/12617 1810 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:43.175184) v4 ==== 2004+0+0 (915390258 0 0) 0x55a3265d7c00 con 0x55a32447e000 > -6054> 2018-07-11 07:33:43.175607 7f355d88e700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:43.175184) v4 -- 0x55a331c67e00 con 0 > -6053> 2018-07-11 07:33:43.175638 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:43.175184) v4 -- 0x55a32e330a00 con 0 > -6052> 2018-07-11 07:33:43.348084 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b7000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1787 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:33:43.098616) v4 > -6051> 2018-07-11 07:33:43.348105 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.8 10.8.128.108:0/11967 1787 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:43.098616) v4 ==== 2004+0+0 (3086168632 0 0) 0x55a34de6f000 con 0x55a3170b7000 > -6050> 2018-07-11 07:33:43.348114 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:43.098616) v4 -- 0x55a331c67e00 con 0 > -6049> 2018-07-11 07:33:43.348117 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b8800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1787 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:33:43.098616) v4 > -6048> 2018-07-11 07:33:43.348150 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.8 10.8.128.108:0/11967 1787 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:43.098616) v4 ==== 2004+0+0 (3086168632 0 0) 0x55a324474c00 con 0x55a3170b8800 > -6047> 2018-07-11 07:33:43.348167 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:43.098616) v4 -- 0x55a31334c600 con 0 > -6046> 2018-07-11 07:33:43.363622 7f3543899700 5 osd.10 494 heartbeat: osd_stat(784 GB used, 147 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,0,0,1]) > -6045> 2018-07-11 07:33:43.363635 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6802/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:33:43.363634) v4 -- 0x55a31334de00 con 0 > -6044> 2018-07-11 07:33:43.363671 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6803/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:33:43.363634) v4 -- 0x55a33be6e800 con 0 > -6043> 2018-07-11 07:33:43.363687 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6806/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:33:43.363634) v4 -- 0x55a331c65c00 con 0 > -6042> 2018-07-11 07:33:43.363701 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6807/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:33:43.363634) v4 -- 0x55a32e331400 con 0 > -6041> 2018-07-11 07:33:43.363713 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6808/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:33:43.363634) v4 -- 0x55a324af4c00 con 0 > -6040> 2018-07-11 07:33:43.363723 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6810/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:33:43.363634) v4 -- 0x55a32e331200 con 0 > -6039> 2018-07-11 07:33:43.363736 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6809/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:33:43.363634) v4 -- 0x55a32e331c00 con 0 > -6038> 2018-07-11 07:33:43.363748 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6811/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:33:43.363634) v4 -- 0x55a32e331600 con 0 > -6037> 2018-07-11 07:33:43.363759 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6810/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:33:43.363634) v4 -- 0x55a34de62400 con 0 > -6036> 2018-07-11 07:33:43.363769 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6811/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:33:43.363634) v4 -- 0x55a32e331800 con 0 > -6035> 2018-07-11 07:33:43.363782 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6806/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:33:43.363634) v4 -- 0x55a32e331e00 con 0 > -6034> 2018-07-11 07:33:43.363794 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6807/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:33:43.363634) v4 -- 0x55a332e79200 con 0 > -6033> 2018-07-11 07:33:43.363817 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6802/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:33:43.363634) v4 -- 0x55a32e331a00 con 0 > -6032> 2018-07-11 07:33:43.363836 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6803/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:33:43.363634) v4 -- 0x55a332e85000 con 0 > -6031> 2018-07-11 07:33:43.363849 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6806/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:33:43.363634) v4 -- 0x55a32e456200 con 0 > -6030> 2018-07-11 07:33:43.363861 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6807/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:33:43.363634) v4 -- 0x55a330935200 con 0 > -6029> 2018-07-11 07:33:43.363874 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6802/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:33:43.363634) v4 -- 0x55a32eb15400 con 0 > -6028> 2018-07-11 07:33:43.363886 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6803/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:33:43.363634) v4 -- 0x55a32eb29a00 con 0 > -6027> 2018-07-11 07:33:43.363922 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6806/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:33:43.363634) v4 -- 0x55a34dec7a00 con 0 > -6026> 2018-07-11 07:33:43.363938 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6807/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:33:43.363634) v4 -- 0x55a332e84600 con 0 > -6025> 2018-07-11 07:33:43.363951 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6802/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:33:43.363634) v4 -- 0x55a313704800 con 0 > -6024> 2018-07-11 07:33:43.363969 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6803/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:33:43.363634) v4 -- 0x55a33b42d200 con 0 > -6023> 2018-07-11 07:33:43.364098 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6802/15255 conn(0x55a314e1c000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1839 0x55a3265d7c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:43.363634) v4 > -6022> 2018-07-11 07:33:43.364105 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6807/12617 conn(0x55a314e27800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1839 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:43.363634) v4 > -6021> 2018-07-11 07:33:43.364123 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6807/12617 1839 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:43.363634) v4 ==== 2004+0+0 (2891473127 0 0) 0x55a34de78600 con 0x55a314e27800 > -6020> 2018-07-11 07:33:43.364120 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6802/15255 1839 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:43.363634) v4 ==== 2004+0+0 (2891473127 0 0) 0x55a3265d7c00 con 0x55a314e1c000 > -6019> 2018-07-11 07:33:43.364138 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6803/15255 conn(0x55a314e1d800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1839 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:43.363634) v4 > -6018> 2018-07-11 07:33:43.364176 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6803/15255 1839 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:43.363634) v4 ==== 2004+0+0 (2891473127 0 0) 0x55a324474c00 con 0x55a314e1d800 > -6017> 2018-07-11 07:33:43.364213 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6806/12617 conn(0x55a314e26000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1839 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:43.363634) v4 > -6016> 2018-07-11 07:33:43.364214 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6806/12677 conn(0x55a314e84000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1839 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:43.363634) v4 > -6015> 2018-07-11 07:33:43.364225 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6806/12617 1839 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:43.363634) v4 ==== 2004+0+0 (2891473127 0 0) 0x55a324474c00 con 0x55a314e26000 > -6014> 2018-07-11 07:33:43.364237 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6806/12677 1839 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:43.363634) v4 ==== 2004+0+0 (2891473127 0 0) 0x55a34de78600 con 0x55a314e84000 > -6013> 2018-07-11 07:33:43.364289 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6809/12650 conn(0x55a314e1f000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1839 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:43.363634) v4 > -6012> 2018-07-11 07:33:43.364287 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6808/12651 conn(0x55a314e96000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1839 0x55a3265d7c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:43.363634) v4 > -6011> 2018-07-11 07:33:43.364304 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6809/12650 1839 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:43.363634) v4 ==== 2004+0+0 (2891473127 0 0) 0x55a34de78600 con 0x55a314e1f000 > -6010> 2018-07-11 07:33:43.364312 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6810/12651 conn(0x55a314e97800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1839 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:43.363634) v4 > -6009> 2018-07-11 07:33:43.364327 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6810/12651 1839 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:43.363634) v4 ==== 2004+0+0 (2891473127 0 0) 0x55a324474c00 con 0x55a314e97800 > -6008> 2018-07-11 07:33:43.364304 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6808/12651 1839 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:43.363634) v4 ==== 2004+0+0 (2891473127 0 0) 0x55a3265d7c00 con 0x55a314e96000 > -6007> 2018-07-11 07:33:43.364345 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6806/15263 conn(0x55a314e99000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1839 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:43.363634) v4 > -6006> 2018-07-11 07:33:43.364357 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6806/15263 1839 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:43.363634) v4 ==== 2004+0+0 (2891473127 0 0) 0x55a34de78600 con 0x55a314e99000 > -6005> 2018-07-11 07:33:43.364371 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6810/15287 conn(0x55a314e03000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1839 0x55a3265d7c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:43.363634) v4 > -6004> 2018-07-11 07:33:43.364385 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6810/15287 1839 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:43.363634) v4 ==== 2004+0+0 (2891473127 0 0) 0x55a3265d7c00 con 0x55a314e03000 > -6003> 2018-07-11 07:33:43.364385 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6811/15287 conn(0x55a314e04800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1839 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:43.363634) v4 > -6002> 2018-07-11 07:33:43.364398 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6811/15287 1839 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:43.363634) v4 ==== 2004+0+0 (2891473127 0 0) 0x55a324474c00 con 0x55a314e04800 > -6001> 2018-07-11 07:33:43.364399 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6803/13323 conn(0x55a314e16800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1839 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:43.363634) v4 > -6000> 2018-07-11 07:33:43.364410 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6803/13323 1839 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:43.363634) v4 ==== 2004+0+0 (2891473127 0 0) 0x55a34de78600 con 0x55a314e16800 > -5999> 2018-07-11 07:33:43.364423 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6811/12650 conn(0x55a314e20800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1839 0x55a3265d7c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:43.363634) v4 > -5998> 2018-07-11 07:33:43.364432 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6802/13323 conn(0x55a314e15000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1839 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:43.363634) v4 > -5997> 2018-07-11 07:33:43.364437 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6811/12650 1839 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:43.363634) v4 ==== 2004+0+0 (2891473127 0 0) 0x55a3265d7c00 con 0x55a314e20800 > -5996> 2018-07-11 07:33:43.364441 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6802/11967 conn(0x55a314e12000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1839 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:43.363634) v4 > -5995> 2018-07-11 07:33:43.364445 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6802/13323 1839 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:43.363634) v4 ==== 2004+0+0 (2891473127 0 0) 0x55a324474c00 con 0x55a314e15000 > -5994> 2018-07-11 07:33:43.364455 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6802/11967 1839 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:43.363634) v4 ==== 2004+0+0 (2891473127 0 0) 0x55a34de78600 con 0x55a314e12000 > -5993> 2018-07-11 07:33:43.364468 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6807/12677 conn(0x55a314e85800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1839 0x55a3265d7c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:43.363634) v4 > -5992> 2018-07-11 07:33:43.364482 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6807/12677 1839 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:43.363634) v4 ==== 2004+0+0 (2891473127 0 0) 0x55a3265d7c00 con 0x55a314e85800 > -5991> 2018-07-11 07:33:43.364479 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6802/15919 conn(0x55a314e87000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1839 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:43.363634) v4 > -5990> 2018-07-11 07:33:43.364486 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6807/15885 conn(0x55a314e01800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1839 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:43.363634) v4 > -5989> 2018-07-11 07:33:43.364502 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6802/15919 1839 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:43.363634) v4 ==== 2004+0+0 (2891473127 0 0) 0x55a324474c00 con 0x55a314e87000 > -5988> 2018-07-11 07:33:43.364503 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6807/15263 conn(0x55a314e9a800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1839 0x55a3265d7c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:43.363634) v4 > -5987> 2018-07-11 07:33:43.364506 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6807/15885 1839 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:43.363634) v4 ==== 2004+0+0 (2891473127 0 0) 0x55a34de78600 con 0x55a314e01800 > -5986> 2018-07-11 07:33:43.364512 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6807/15263 1839 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:43.363634) v4 ==== 2004+0+0 (2891473127 0 0) 0x55a3265d7c00 con 0x55a314e9a800 > -5985> 2018-07-11 07:33:43.364542 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6803/15919 conn(0x55a314e88800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1839 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:43.363634) v4 > -5984> 2018-07-11 07:33:43.364557 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6803/15919 1839 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:43.363634) v4 ==== 2004+0+0 (2891473127 0 0) 0x55a34de78600 con 0x55a314e88800 > -5983> 2018-07-11 07:33:43.364570 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6803/11967 conn(0x55a314e13800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1839 0x55a3265d7c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:43.363634) v4 > -5982> 2018-07-11 07:33:43.364585 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6803/11967 1839 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:43.363634) v4 ==== 2004+0+0 (2891473127 0 0) 0x55a3265d7c00 con 0x55a314e13800 > -5981> 2018-07-11 07:33:43.364624 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6806/15885 conn(0x55a314e00000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1839 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:43.363634) v4 > -5980> 2018-07-11 07:33:43.364638 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6806/15885 1839 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:43.363634) v4 ==== 2004+0+0 (2891473127 0 0) 0x55a324474c00 con 0x55a314e00000 > -5979> 2018-07-11 07:33:43.426209 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15287 conn(0x55a311b75000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1831 0x55a3265d7c00 osd_ping(ping e494 stamp 2018-07-11 07:33:44.639702) v4 > -5978> 2018-07-11 07:33:43.426209 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15287 conn(0x55a311b72000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1831 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:33:44.639702) v4 > -5977> 2018-07-11 07:33:43.426237 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.4 10.8.128.113:0/15287 1831 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:44.639702) v4 ==== 2004+0+0 (134553644 0 0) 0x55a3265d7c00 con 0x55a311b75000 > -5976> 2018-07-11 07:33:43.426241 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.4 10.8.128.113:0/15287 1831 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:44.639702) v4 ==== 2004+0+0 (134553644 0 0) 0x55a324474c00 con 0x55a311b72000 > -5975> 2018-07-11 07:33:43.426262 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:44.639702) v4 -- 0x55a32eb29a00 con 0 > -5974> 2018-07-11 07:33:43.426290 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:44.639702) v4 -- 0x55a313704800 con 0 > -5973> 2018-07-11 07:33:43.455430 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a33248dd40 con 0 > -5972> 2018-07-11 07:33:43.477030 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5092 0x55a321714c80 replica scrub(pg: 6.15,from:0'0,to:430'105518,epoch:494/492,start:6:a9cc802f:::benchmark_data_magna108_35617_object1061012:0,end:6:a9cd4b01:::benchmark_data_magna108_35617_object667926:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -5971> 2018-07-11 07:33:43.477053 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5092 ==== replica scrub(pg: 6.15,from:0'0,to:430'105518,epoch:494/492,start:6:a9cc802f:::benchmark_data_magna108_35617_object1061012:0,end:6:a9cd4b01:::benchmark_data_magna108_35617_object667926:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 219+0+0 (2178613428 0 0) 0x55a321714c80 con 0x55a317019000 > -5970> 2018-07-11 07:33:44.145213 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a3141bad80 con 0 > -5969> 2018-07-11 07:33:44.175288 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5093 0x55a321714f00 replica scrub(pg: 6.15,from:0'0,to:461'106914,epoch:494/492,start:6:a9cd4b01:::benchmark_data_magna108_35617_object667926:0,end:6:a9ce1f36:::benchmark_data_magna108.ceph.redhat.com_2478_object144149:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -5968> 2018-07-11 07:33:44.175318 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5093 ==== replica scrub(pg: 6.15,from:0'0,to:461'106914,epoch:494/492,start:6:a9cd4b01:::benchmark_data_magna108_35617_object667926:0,end:6:a9ce1f36:::benchmark_data_magna108.ceph.redhat.com_2478_object144149:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (2331149310 0 0) 0x55a321714f00 con 0x55a317019000 > -5967> 2018-07-11 07:33:44.673982 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12650 conn(0x55a311b67000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1852 0x55a31334cc00 osd_ping(ping e494 stamp 2018-07-11 07:33:44.709404) v4 > -5966> 2018-07-11 07:33:44.674011 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.3 10.8.128.106:0/12650 1852 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:44.709404) v4 ==== 2004+0+0 (3737476124 0 0) 0x55a31334cc00 con 0x55a311b67000 > -5965> 2018-07-11 07:33:44.674023 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:44.709404) v4 -- 0x55a32eb29a00 con 0 > -5964> 2018-07-11 07:33:44.674045 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12650 conn(0x55a311b68800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1852 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:33:44.709404) v4 > -5963> 2018-07-11 07:33:44.674068 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.3 10.8.128.106:0/12650 1852 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:44.709404) v4 ==== 2004+0+0 (3737476124 0 0) 0x55a34de78600 con 0x55a311b68800 > -5962> 2018-07-11 07:33:44.674081 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:44.709404) v4 -- 0x55a33b42d200 con 0 > -5961> 2018-07-11 07:33:45.064164 7f3543899700 5 osd.10 494 heartbeat: osd_stat(784 GB used, 147 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,0,0,1]) > -5960> 2018-07-11 07:33:45.064179 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6802/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:33:45.064177) v4 -- 0x55a33b42c600 con 0 > -5959> 2018-07-11 07:33:45.064217 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6803/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:33:45.064177) v4 -- 0x55a32adf0000 con 0 > -5958> 2018-07-11 07:33:45.064234 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6806/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:33:45.064177) v4 -- 0x55a33b42dc00 con 0 > -5957> 2018-07-11 07:33:45.064280 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6807/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:33:45.064177) v4 -- 0x55a33b42ce00 con 0 > -5956> 2018-07-11 07:33:45.064305 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6808/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:33:45.064177) v4 -- 0x55a33be6fc00 con 0 > -5955> 2018-07-11 07:33:45.064324 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6810/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:33:45.064177) v4 -- 0x55a324af5800 con 0 > -5954> 2018-07-11 07:33:45.064339 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6809/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:33:45.064177) v4 -- 0x55a33be6fa00 con 0 > -5953> 2018-07-11 07:33:45.064362 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6811/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:33:45.064177) v4 -- 0x55a33be6f400 con 0 > -5952> 2018-07-11 07:33:45.064385 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6810/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:33:45.064177) v4 -- 0x55a32adf0400 con 0 > -5951> 2018-07-11 07:33:45.064400 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6811/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:33:45.064177) v4 -- 0x55a34866a800 con 0 > -5950> 2018-07-11 07:33:45.064439 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6806/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:33:45.064177) v4 -- 0x55a32adf0c00 con 0 > -5949> 2018-07-11 07:33:45.064470 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6807/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:33:45.064177) v4 -- 0x55a324af4a00 con 0 > -5948> 2018-07-11 07:33:45.064490 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6802/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:33:45.064177) v4 -- 0x55a32adf1400 con 0 > -5947> 2018-07-11 07:33:45.064507 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6803/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:33:45.064177) v4 -- 0x55a324af4e00 con 0 > -5946> 2018-07-11 07:33:45.064528 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6806/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:33:45.064177) v4 -- 0x55a32adf0600 con 0 > -5945> 2018-07-11 07:33:45.064566 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6807/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:33:45.064177) v4 -- 0x55a324af5200 con 0 > -5944> 2018-07-11 07:33:45.064592 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6802/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:33:45.064177) v4 -- 0x55a32adf0800 con 0 > -5943> 2018-07-11 07:33:45.064614 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6803/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:33:45.064177) v4 -- 0x55a32adf1a00 con 0 > -5942> 2018-07-11 07:33:45.064655 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6806/12677 conn(0x55a314e84000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1840 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:45.064177) v4 > -5941> 2018-07-11 07:33:45.064638 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6806/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:33:45.064177) v4 -- 0x55a32adf0e00 con 0 > -5940> 2018-07-11 07:33:45.064678 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6806/12677 1840 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:45.064177) v4 ==== 2004+0+0 (993229865 0 0) 0x55a34de78600 con 0x55a314e84000 > -5939> 2018-07-11 07:33:45.064686 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6807/12677 conn(0x55a314e85800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1840 0x55a3265d7c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:45.064177) v4 > -5938> 2018-07-11 07:33:45.064696 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6803/15255 conn(0x55a314e1d800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1840 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:45.064177) v4 > -5937> 2018-07-11 07:33:45.064694 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6807/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:33:45.064177) v4 -- 0x55a32adf0200 con 0 > -5936> 2018-07-11 07:33:45.064712 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6807/12677 1840 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:45.064177) v4 ==== 2004+0+0 (993229865 0 0) 0x55a3265d7c00 con 0x55a314e85800 > -5935> 2018-07-11 07:33:45.064715 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6803/15255 1840 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:45.064177) v4 ==== 2004+0+0 (993229865 0 0) 0x55a324474c00 con 0x55a314e1d800 > -5934> 2018-07-11 07:33:45.064720 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6802/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:33:45.064177) v4 -- 0x55a32adf1600 con 0 > -5933> 2018-07-11 07:33:45.064761 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6803/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:33:45.064177) v4 -- 0x55a32adf1e00 con 0 > -5932> 2018-07-11 07:33:45.064817 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6802/15255 conn(0x55a314e1c000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1840 0x55a3265d7c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:45.064177) v4 > -5931> 2018-07-11 07:33:45.064834 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6802/15255 1840 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:45.064177) v4 ==== 2004+0+0 (993229865 0 0) 0x55a3265d7c00 con 0x55a314e1c000 > -5930> 2018-07-11 07:33:45.064874 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6809/12650 conn(0x55a314e1f000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1840 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:45.064177) v4 > -5929> 2018-07-11 07:33:45.064876 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6811/12650 conn(0x55a314e20800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1840 0x55a3265d7c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:45.064177) v4 > -5928> 2018-07-11 07:33:45.064890 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6809/12650 1840 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:45.064177) v4 ==== 2004+0+0 (993229865 0 0) 0x55a34de6f000 con 0x55a314e1f000 > -5927> 2018-07-11 07:33:45.064884 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6810/12651 conn(0x55a314e97800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1840 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:45.064177) v4 > -5926> 2018-07-11 07:33:45.064895 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6811/12650 1840 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:45.064177) v4 ==== 2004+0+0 (993229865 0 0) 0x55a3265d7c00 con 0x55a314e20800 > -5925> 2018-07-11 07:33:45.064907 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6810/12651 1840 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:45.064177) v4 ==== 2004+0+0 (993229865 0 0) 0x55a332e79a00 con 0x55a314e97800 > -5924> 2018-07-11 07:33:45.064934 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6808/12651 conn(0x55a314e96000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1840 0x55a3265d7c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:45.064177) v4 > -5923> 2018-07-11 07:33:45.064951 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6808/12651 1840 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:45.064177) v4 ==== 2004+0+0 (993229865 0 0) 0x55a3265d7c00 con 0x55a314e96000 > -5922> 2018-07-11 07:33:45.064945 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6811/15287 conn(0x55a314e04800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1840 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:45.064177) v4 > -5921> 2018-07-11 07:33:45.064966 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6803/13323 conn(0x55a314e16800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1840 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:45.064177) v4 > -5920> 2018-07-11 07:33:45.064976 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6811/15287 1840 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:45.064177) v4 ==== 2004+0+0 (993229865 0 0) 0x55a332e79a00 con 0x55a314e04800 > -5919> 2018-07-11 07:33:45.064995 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6803/13323 1840 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:45.064177) v4 ==== 2004+0+0 (993229865 0 0) 0x55a34de6f000 con 0x55a314e16800 > -5918> 2018-07-11 07:33:45.064995 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6810/15287 conn(0x55a314e03000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1840 0x55a3265d7c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:45.064177) v4 > -5917> 2018-07-11 07:33:45.065006 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6810/15287 1840 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:45.064177) v4 ==== 2004+0+0 (993229865 0 0) 0x55a3265d7c00 con 0x55a314e03000 > -5916> 2018-07-11 07:33:45.065040 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6806/15263 conn(0x55a314e99000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1840 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:45.064177) v4 > -5915> 2018-07-11 07:33:45.065046 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6806/15885 conn(0x55a314e00000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1840 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:45.064177) v4 > -5914> 2018-07-11 07:33:45.065049 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6807/15263 conn(0x55a314e9a800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1840 0x55a3265d7c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:45.064177) v4 > -5913> 2018-07-11 07:33:45.065058 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6806/15885 1840 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:45.064177) v4 ==== 2004+0+0 (993229865 0 0) 0x55a332e79a00 con 0x55a314e00000 > -5912> 2018-07-11 07:33:45.065058 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6806/15263 1840 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:45.064177) v4 ==== 2004+0+0 (993229865 0 0) 0x55a34de6f000 con 0x55a314e99000 > -5911> 2018-07-11 07:33:45.065060 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6807/15263 1840 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:45.064177) v4 ==== 2004+0+0 (993229865 0 0) 0x55a3265d7c00 con 0x55a314e9a800 > -5910> 2018-07-11 07:33:45.065089 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6807/12617 conn(0x55a314e27800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1840 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:45.064177) v4 > -5909> 2018-07-11 07:33:45.065096 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6802/13323 conn(0x55a314e15000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1840 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:45.064177) v4 > -5908> 2018-07-11 07:33:45.065104 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6807/12617 1840 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:45.064177) v4 ==== 2004+0+0 (993229865 0 0) 0x55a34de6f000 con 0x55a314e27800 > -5907> 2018-07-11 07:33:45.065108 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6802/13323 1840 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:45.064177) v4 ==== 2004+0+0 (993229865 0 0) 0x55a332e79a00 con 0x55a314e15000 > -5906> 2018-07-11 07:33:45.065103 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6803/11967 conn(0x55a314e13800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1840 0x55a3265d7c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:45.064177) v4 > -5905> 2018-07-11 07:33:45.065117 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6803/11967 1840 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:45.064177) v4 ==== 2004+0+0 (993229865 0 0) 0x55a3265d7c00 con 0x55a314e13800 > -5904> 2018-07-11 07:33:45.065129 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6806/12617 conn(0x55a314e26000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1840 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:45.064177) v4 > -5903> 2018-07-11 07:33:45.065135 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6807/15885 conn(0x55a314e01800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1840 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:45.064177) v4 > -5902> 2018-07-11 07:33:45.065139 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6806/12617 1840 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:45.064177) v4 ==== 2004+0+0 (993229865 0 0) 0x55a332e79a00 con 0x55a314e26000 > -5901> 2018-07-11 07:33:45.065149 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6807/15885 1840 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:45.064177) v4 ==== 2004+0+0 (993229865 0 0) 0x55a34de6f000 con 0x55a314e01800 > -5900> 2018-07-11 07:33:45.065187 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6802/15919 conn(0x55a314e87000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1840 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:45.064177) v4 > -5899> 2018-07-11 07:33:45.065194 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6803/15919 conn(0x55a314e88800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1840 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:45.064177) v4 > -5898> 2018-07-11 07:33:45.065202 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6802/15919 1840 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:45.064177) v4 ==== 2004+0+0 (993229865 0 0) 0x55a332e79a00 con 0x55a314e87000 > -5897> 2018-07-11 07:33:45.065206 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6803/15919 1840 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:45.064177) v4 ==== 2004+0+0 (993229865 0 0) 0x55a34de6f000 con 0x55a314e88800 > -5896> 2018-07-11 07:33:45.065243 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6802/11967 conn(0x55a314e12000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1840 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:45.064177) v4 > -5895> 2018-07-11 07:33:45.065271 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6802/11967 1840 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:45.064177) v4 ==== 2004+0+0 (993229865 0 0) 0x55a34de6f000 con 0x55a314e12000 > -5894> 2018-07-11 07:33:45.177419 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a324b20000 con 0 > -5893> 2018-07-11 07:33:45.191775 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5094 0x55a321715180 replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:494/492,start:6:a9ce1f36:::benchmark_data_magna108.ceph.redhat.com_2478_object144149:0,end:6:a9cef4cc:::benchmark_data_magna108_35617_object1050312:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -5892> 2018-07-11 07:33:45.191807 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5094 ==== replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:494/492,start:6:a9ce1f36:::benchmark_data_magna108.ceph.redhat.com_2478_object144149:0,end:6:a9cef4cc:::benchmark_data_magna108_35617_object1050312:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 234+0+0 (2059796478 0 0) 0x55a321715180 con 0x55a317019000 > -5891> 2018-07-11 07:33:45.225424 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15255 conn(0x55a311b65800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1861 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:33:46.438807) v4 > -5890> 2018-07-11 07:33:45.225442 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.0 10.8.128.113:0/15255 1861 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:46.438807) v4 ==== 2004+0+0 (2716344945 0 0) 0x55a332e79a00 con 0x55a311b65800 > -5889> 2018-07-11 07:33:45.225453 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:46.438807) v4 -- 0x55a34866a800 con 0 > -5888> 2018-07-11 07:33:45.225526 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15255 conn(0x55a311b64000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1861 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:33:46.438807) v4 > -5887> 2018-07-11 07:33:45.225546 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.0 10.8.128.113:0/15255 1861 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:46.438807) v4 ==== 2004+0+0 (2716344945 0 0) 0x55a34de6f000 con 0x55a311b64000 > -5886> 2018-07-11 07:33:45.225561 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:46.438807) v4 -- 0x55a32adf0c00 con 0 > -5885> 2018-07-11 07:33:45.454396 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15919 conn(0x55a3170b4000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1841 0x55a3265d7c00 osd_ping(ping e494 stamp 2018-07-11 07:33:46.078131) v4 > -5884> 2018-07-11 07:33:45.454422 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.11 10.8.128.114:0/15919 1841 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:46.078131) v4 ==== 2004+0+0 (2549132477 0 0) 0x55a3265d7c00 con 0x55a3170b4000 > -5883> 2018-07-11 07:33:45.454436 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:46.078131) v4 -- 0x55a33b42c600 con 0 > -5882> 2018-07-11 07:33:45.454431 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15919 conn(0x55a317074800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1841 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:33:46.078131) v4 > -5881> 2018-07-11 07:33:45.454448 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.11 10.8.128.114:0/15919 1841 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:46.078131) v4 ==== 2004+0+0 (2549132477 0 0) 0x55a324474c00 con 0x55a317074800 > -5880> 2018-07-11 07:33:45.454465 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:46.078131) v4 -- 0x55a324af5800 con 0 > -5879> 2018-07-11 07:33:45.462341 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12651 conn(0x55a32169f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1838 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:33:45.497884) v4 > -5878> 2018-07-11 07:33:45.462363 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.2 10.8.128.106:0/12651 1838 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:45.497884) v4 ==== 2004+0+0 (1341117427 0 0) 0x55a332e79a00 con 0x55a32169f000 > -5877> 2018-07-11 07:33:45.462380 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:45.497884) v4 -- 0x55a32adf0000 con 0 > -5876> 2018-07-11 07:33:45.462394 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12651 conn(0x55a3216a0800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1838 0x55a33b42ce00 osd_ping(ping e494 stamp 2018-07-11 07:33:45.497884) v4 > -5875> 2018-07-11 07:33:45.462419 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.2 10.8.128.106:0/12651 1838 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:45.497884) v4 ==== 2004+0+0 (1341117427 0 0) 0x55a33b42ce00 con 0x55a3216a0800 > -5874> 2018-07-11 07:33:45.462430 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:45.497884) v4 -- 0x55a32eb29a00 con 0 > -5873> 2018-07-11 07:33:45.476054 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.107:0/12617 conn(0x55a32447e000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1811 0x55a3265d7c00 osd_ping(ping e494 stamp 2018-07-11 07:33:45.475628) v4 > -5872> 2018-07-11 07:33:45.476076 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.9 10.8.128.107:0/12617 1811 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:45.475628) v4 ==== 2004+0+0 (6146580 0 0) 0x55a3265d7c00 con 0x55a32447e000 > -5871> 2018-07-11 07:33:45.476080 7f355d88e700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.107:0/12617 conn(0x55a32447f800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1811 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:33:45.475628) v4 > -5870> 2018-07-11 07:33:45.476104 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:45.475628) v4 -- 0x55a332e79200 con 0 > -5869> 2018-07-11 07:33:45.476108 7f355d88e700 1 -- 10.8.128.107:6803/12665 <== osd.9 10.8.128.107:0/12617 1811 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:45.475628) v4 ==== 2004+0+0 (6146580 0 0) 0x55a34de78600 con 0x55a32447f800 > -5868> 2018-07-11 07:33:45.476127 7f355d88e700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:45.475628) v4 -- 0x55a33be6fa00 con 0 > -5867> 2018-07-11 07:33:45.648490 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b8800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1788 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:33:45.399062) v4 > -5866> 2018-07-11 07:33:45.648492 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b7000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1788 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:33:45.399062) v4 > -5865> 2018-07-11 07:33:45.648511 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.8 10.8.128.108:0/11967 1788 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:45.399062) v4 ==== 2004+0+0 (756942497 0 0) 0x55a324474c00 con 0x55a3170b8800 > -5864> 2018-07-11 07:33:45.648514 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.8 10.8.128.108:0/11967 1788 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:45.399062) v4 ==== 2004+0+0 (756942497 0 0) 0x55a34de6f000 con 0x55a3170b7000 > -5863> 2018-07-11 07:33:45.648520 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:45.399062) v4 -- 0x55a313704800 con 0 > -5862> 2018-07-11 07:33:45.648544 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:45.399062) v4 -- 0x55a33b42dc00 con 0 > -5861> 2018-07-11 07:33:45.774481 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12650 conn(0x55a311b68800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1853 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:33:45.809988) v4 > -5860> 2018-07-11 07:33:45.774503 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.3 10.8.128.106:0/12650 1853 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:45.809988) v4 ==== 2004+0+0 (3221287056 0 0) 0x55a34de78600 con 0x55a311b68800 > -5859> 2018-07-11 07:33:45.774512 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:45.809988) v4 -- 0x55a33b42dc00 con 0 > -5858> 2018-07-11 07:33:45.774510 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12650 conn(0x55a311b67000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1853 0x55a33b42ce00 osd_ping(ping e494 stamp 2018-07-11 07:33:45.809988) v4 > -5857> 2018-07-11 07:33:45.774527 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.3 10.8.128.106:0/12650 1853 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:45.809988) v4 ==== 2004+0+0 (3221287056 0 0) 0x55a33b42ce00 con 0x55a311b67000 > -5856> 2018-07-11 07:33:45.774538 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:45.809988) v4 -- 0x55a332e79200 con 0 > -5855> 2018-07-11 07:33:45.823478 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a324b29200 con 0 > -5854> 2018-07-11 07:33:45.833808 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5095 0x55a321715400 replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:494/492,start:6:a9cef4cc:::benchmark_data_magna108_35617_object1050312:0,end:6:a9cfe566:::benchmark_data_magna108_35617_object1041629:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -5853> 2018-07-11 07:33:45.833834 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5095 ==== replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:494/492,start:6:a9cef4cc:::benchmark_data_magna108_35617_object1050312:0,end:6:a9cfe566:::benchmark_data_magna108_35617_object1041629:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 220+0+0 (1075909183 0 0) 0x55a321715400 con 0x55a317019000 > -5852> 2018-07-11 07:33:46.416562 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15263 conn(0x55a321664000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1836 0x55a3265d7c00 osd_ping(ping e494 stamp 2018-07-11 07:33:47.630102) v4 > -5851> 2018-07-11 07:33:46.416588 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.5 10.8.128.113:0/15263 1836 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:47.630102) v4 ==== 2004+0+0 (2359079466 0 0) 0x55a3265d7c00 con 0x55a321664000 > -5850> 2018-07-11 07:33:46.416599 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:47.630102) v4 -- 0x55a332e79200 con 0 > -5849> 2018-07-11 07:33:46.416663 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15263 conn(0x55a321667000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1836 0x55a33b42ce00 osd_ping(ping e494 stamp 2018-07-11 07:33:47.630102) v4 > -5848> 2018-07-11 07:33:46.416677 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.5 10.8.128.113:0/15263 1836 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:47.630102) v4 ==== 2004+0+0 (2359079466 0 0) 0x55a33b42ce00 con 0x55a321667000 > -5847> 2018-07-11 07:33:46.416687 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:47.630102) v4 -- 0x55a34de62400 con 0 > -5846> 2018-07-11 07:33:46.576624 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.107:0/12617 conn(0x55a32447e000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1812 0x55a3265d7c00 osd_ping(ping e494 stamp 2018-07-11 07:33:46.576287) v4 > -5845> 2018-07-11 07:33:46.576630 7f355d88e700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.107:0/12617 conn(0x55a32447f800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1812 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:33:46.576287) v4 > -5844> 2018-07-11 07:33:46.576645 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.9 10.8.128.107:0/12617 1812 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:46.576287) v4 ==== 2004+0+0 (2638058280 0 0) 0x55a3265d7c00 con 0x55a32447e000 > -5843> 2018-07-11 07:33:46.576661 7f355d88e700 1 -- 10.8.128.107:6803/12665 <== osd.9 10.8.128.107:0/12617 1812 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:46.576287) v4 ==== 2004+0+0 (2638058280 0 0) 0x55a34de6f000 con 0x55a32447f800 > -5842> 2018-07-11 07:33:46.576668 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:46.576287) v4 -- 0x55a32e331600 con 0 > -5841> 2018-07-11 07:33:46.576699 7f355d88e700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:46.576287) v4 -- 0x55a33b42d200 con 0 > -5840> 2018-07-11 07:33:46.744333 7f35498a5700 10 monclient: tick > -5839> 2018-07-11 07:33:46.744348 7f35498a5700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2018-07-11 07:33:16.744346) > -5838> 2018-07-11 07:33:46.764865 7f3543899700 5 osd.10 494 heartbeat: osd_stat(784 GB used, 147 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,0,0,1]) > -5837> 2018-07-11 07:33:46.764879 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6802/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:33:46.764878) v4 -- 0x55a32adf0a00 con 0 > -5836> 2018-07-11 07:33:46.764908 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6803/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:33:46.764878) v4 -- 0x55a32adf1c00 con 0 > -5835> 2018-07-11 07:33:46.764924 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6806/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:33:46.764878) v4 -- 0x55a32adf1000 con 0 > -5834> 2018-07-11 07:33:46.764940 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6807/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:33:46.764878) v4 -- 0x55a3170de600 con 0 > -5833> 2018-07-11 07:33:46.764952 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6808/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:33:46.764878) v4 -- 0x55a332355200 con 0 > -5832> 2018-07-11 07:33:46.764964 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6810/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:33:46.764878) v4 -- 0x55a32adf1200 con 0 > -5831> 2018-07-11 07:33:46.764976 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6809/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:33:46.764878) v4 -- 0x55a330934200 con 0 > -5830> 2018-07-11 07:33:46.764988 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6811/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:33:46.764878) v4 -- 0x55a32adf1800 con 0 > -5829> 2018-07-11 07:33:46.765001 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6810/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:33:46.764878) v4 -- 0x55a34de5a400 con 0 > -5828> 2018-07-11 07:33:46.765015 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6811/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:33:46.764878) v4 -- 0x55a32eb21000 con 0 > -5827> 2018-07-11 07:33:46.765033 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6806/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:33:46.764878) v4 -- 0x55a32eb20000 con 0 > -5826> 2018-07-11 07:33:46.765048 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6807/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:33:46.764878) v4 -- 0x55a332354200 con 0 > -5825> 2018-07-11 07:33:46.765070 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6802/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:33:46.764878) v4 -- 0x55a34866b000 con 0 > -5824> 2018-07-11 07:33:46.765088 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6803/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:33:46.764878) v4 -- 0x55a33be6f800 con 0 > -5823> 2018-07-11 07:33:46.765101 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6806/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:33:46.764878) v4 -- 0x55a3203f4200 con 0 > -5822> 2018-07-11 07:33:46.765114 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6807/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:33:46.764878) v4 -- 0x55a3203f4a00 con 0 > -5821> 2018-07-11 07:33:46.765126 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6802/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:33:46.764878) v4 -- 0x55a3203f5000 con 0 > -5820> 2018-07-11 07:33:46.765136 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6803/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:33:46.764878) v4 -- 0x55a30fa77c00 con 0 > -5819> 2018-07-11 07:33:46.765156 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6806/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:33:46.764878) v4 -- 0x55a3203f4000 con 0 > -5818> 2018-07-11 07:33:46.765194 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6807/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:33:46.764878) v4 -- 0x55a3203f5400 con 0 > -5817> 2018-07-11 07:33:46.765210 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6802/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:33:46.764878) v4 -- 0x55a30fa77e00 con 0 > -5816> 2018-07-11 07:33:46.765222 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6803/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:33:46.764878) v4 -- 0x55a3203f4800 con 0 > -5815> 2018-07-11 07:33:46.765330 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6802/15255 conn(0x55a314e1c000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1841 0x55a33b42ce00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:46.764878) v4 > -5814> 2018-07-11 07:33:46.765352 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6802/15255 1841 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:46.764878) v4 ==== 2004+0+0 (549657174 0 0) 0x55a33b42ce00 con 0x55a314e1c000 > -5813> 2018-07-11 07:33:46.765376 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6806/12617 conn(0x55a314e26000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1841 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:46.764878) v4 > -5812> 2018-07-11 07:33:46.765397 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6806/12617 1841 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:46.764878) v4 ==== 2004+0+0 (549657174 0 0) 0x55a324474c00 con 0x55a314e26000 > -5811> 2018-07-11 07:33:46.765408 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6807/12617 conn(0x55a314e27800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1841 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:46.764878) v4 > -5810> 2018-07-11 07:33:46.765417 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6807/12677 conn(0x55a314e85800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1841 0x55a33b42ce00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:46.764878) v4 > -5809> 2018-07-11 07:33:46.765431 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6807/12677 1841 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:46.764878) v4 ==== 2004+0+0 (549657174 0 0) 0x55a33b42ce00 con 0x55a314e85800 > -5808> 2018-07-11 07:33:46.765432 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6807/12617 1841 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:46.764878) v4 ==== 2004+0+0 (549657174 0 0) 0x55a34de6f000 con 0x55a314e27800 > -5807> 2018-07-11 07:33:46.765437 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6803/15255 conn(0x55a314e1d800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1841 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:46.764878) v4 > -5806> 2018-07-11 07:33:46.765448 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6803/15255 1841 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:46.764878) v4 ==== 2004+0+0 (549657174 0 0) 0x55a324474c00 con 0x55a314e1d800 > -5805> 2018-07-11 07:33:46.765461 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6808/12651 conn(0x55a314e96000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1841 0x55a33b42ce00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:46.764878) v4 > -5804> 2018-07-11 07:33:46.765470 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6808/12651 1841 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:46.764878) v4 ==== 2004+0+0 (549657174 0 0) 0x55a33b42ce00 con 0x55a314e96000 > -5803> 2018-07-11 07:33:46.765487 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6810/12651 conn(0x55a314e97800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1841 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:46.764878) v4 > -5802> 2018-07-11 07:33:46.765488 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6806/12677 conn(0x55a314e84000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1841 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:46.764878) v4 > -5801> 2018-07-11 07:33:46.765500 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6810/12651 1841 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:46.764878) v4 ==== 2004+0+0 (549657174 0 0) 0x55a324474c00 con 0x55a314e97800 > -5800> 2018-07-11 07:33:46.765503 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6810/15287 conn(0x55a314e03000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1841 0x55a33b42ce00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:46.764878) v4 > -5799> 2018-07-11 07:33:46.765502 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6806/12677 1841 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:46.764878) v4 ==== 2004+0+0 (549657174 0 0) 0x55a34de6f000 con 0x55a314e84000 > -5798> 2018-07-11 07:33:46.765529 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6810/15287 1841 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:46.764878) v4 ==== 2004+0+0 (549657174 0 0) 0x55a33b42ce00 con 0x55a314e03000 > -5797> 2018-07-11 07:33:46.765544 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6811/15287 conn(0x55a314e04800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1841 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:46.764878) v4 > -5796> 2018-07-11 07:33:46.765558 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6811/15287 1841 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:46.764878) v4 ==== 2004+0+0 (549657174 0 0) 0x55a324474c00 con 0x55a314e04800 > -5795> 2018-07-11 07:33:46.765572 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6809/12650 conn(0x55a314e1f000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1841 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:46.764878) v4 > -5794> 2018-07-11 07:33:46.765576 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6811/12650 conn(0x55a314e20800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1841 0x55a33b42ce00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:46.764878) v4 > -5793> 2018-07-11 07:33:46.765589 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6811/12650 1841 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:46.764878) v4 ==== 2004+0+0 (549657174 0 0) 0x55a33b42ce00 con 0x55a314e20800 > -5792> 2018-07-11 07:33:46.765588 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6809/12650 1841 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:46.764878) v4 ==== 2004+0+0 (549657174 0 0) 0x55a34de6f000 con 0x55a314e1f000 > -5791> 2018-07-11 07:33:46.765586 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6802/13323 conn(0x55a314e15000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1841 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:46.764878) v4 > -5790> 2018-07-11 07:33:46.765606 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6802/13323 1841 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:46.764878) v4 ==== 2004+0+0 (549657174 0 0) 0x55a324474c00 con 0x55a314e15000 > -5789> 2018-07-11 07:33:46.765627 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6807/15263 conn(0x55a314e9a800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1841 0x55a33b42ce00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:46.764878) v4 > -5788> 2018-07-11 07:33:46.765639 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6807/15263 1841 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:46.764878) v4 ==== 2004+0+0 (549657174 0 0) 0x55a33b42ce00 con 0x55a314e9a800 > -5787> 2018-07-11 07:33:46.765640 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6806/15885 conn(0x55a314e00000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1841 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:46.764878) v4 > -5786> 2018-07-11 07:33:46.765642 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6806/15263 conn(0x55a314e99000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1841 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:46.764878) v4 > -5785> 2018-07-11 07:33:46.765650 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6806/15885 1841 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:46.764878) v4 ==== 2004+0+0 (549657174 0 0) 0x55a324474c00 con 0x55a314e00000 > -5784> 2018-07-11 07:33:46.765663 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6806/15263 1841 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:46.764878) v4 ==== 2004+0+0 (549657174 0 0) 0x55a34de6f000 con 0x55a314e99000 > -5783> 2018-07-11 07:33:46.765685 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6803/11967 conn(0x55a314e13800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1841 0x55a33b42ce00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:46.764878) v4 > -5782> 2018-07-11 07:33:46.765688 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6802/15919 conn(0x55a314e87000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1841 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:46.764878) v4 > -5781> 2018-07-11 07:33:46.765700 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6803/11967 1841 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:46.764878) v4 ==== 2004+0+0 (549657174 0 0) 0x55a33b42ce00 con 0x55a314e13800 > -5780> 2018-07-11 07:33:46.765700 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6802/15919 1841 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:46.764878) v4 ==== 2004+0+0 (549657174 0 0) 0x55a324474c00 con 0x55a314e87000 > -5779> 2018-07-11 07:33:46.765708 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6803/13323 conn(0x55a314e16800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1841 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:46.764878) v4 > -5778> 2018-07-11 07:33:46.765726 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6803/13323 1841 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:46.764878) v4 ==== 2004+0+0 (549657174 0 0) 0x55a34de6f000 con 0x55a314e16800 > -5777> 2018-07-11 07:33:46.765764 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6807/15885 conn(0x55a314e01800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1841 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:46.764878) v4 > -5776> 2018-07-11 07:33:46.765778 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6807/15885 1841 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:46.764878) v4 ==== 2004+0+0 (549657174 0 0) 0x55a34de6f000 con 0x55a314e01800 > -5775> 2018-07-11 07:33:46.765819 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6803/15919 conn(0x55a314e88800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1841 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:46.764878) v4 > -5774> 2018-07-11 07:33:46.765833 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6803/15919 1841 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:46.764878) v4 ==== 2004+0+0 (549657174 0 0) 0x55a34de6f000 con 0x55a314e88800 > -5773> 2018-07-11 07:33:46.765870 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6802/11967 conn(0x55a314e12000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1841 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:46.764878) v4 > -5772> 2018-07-11 07:33:46.765884 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6802/11967 1841 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:46.764878) v4 ==== 2004+0+0 (549657174 0 0) 0x55a34de6f000 con 0x55a314e12000 > -5771> 2018-07-11 07:33:46.947302 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.115:0/13323 conn(0x55a321665800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1849 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:33:47.284554) v4 > -5770> 2018-07-11 07:33:46.947328 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.6 10.8.128.115:0/13323 1849 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:47.284554) v4 ==== 2004+0+0 (2499239397 0 0) 0x55a324474c00 con 0x55a321665800 > -5769> 2018-07-11 07:33:46.947342 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:47.284554) v4 -- 0x55a313704800 con 0 > -5768> 2018-07-11 07:33:46.947429 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.115:0/13323 conn(0x55a321668800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1849 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:33:47.284554) v4 > -5767> 2018-07-11 07:33:46.947448 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.6 10.8.128.115:0/13323 1849 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:47.284554) v4 ==== 2004+0+0 (2499239397 0 0) 0x55a332e79a00 con 0x55a321668800 > -5766> 2018-07-11 07:33:46.947459 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:47.284554) v4 -- 0x55a34dec7a00 con 0 > -5765> 2018-07-11 07:33:46.964326 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15885 conn(0x55a311b76800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1857 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:33:47.588049) v4 > -5764> 2018-07-11 07:33:46.964334 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15885 conn(0x55a3170b5800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1857 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:33:47.588049) v4 > -5763> 2018-07-11 07:33:46.964348 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.7 10.8.128.114:0/15885 1857 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:47.588049) v4 ==== 2004+0+0 (1697193332 0 0) 0x55a34de6f000 con 0x55a311b76800 > -5762> 2018-07-11 07:33:46.964358 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:47.588049) v4 -- 0x55a33b42d200 con 0 > -5761> 2018-07-11 07:33:46.964356 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.7 10.8.128.114:0/15885 1857 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:47.588049) v4 ==== 2004+0+0 (1697193332 0 0) 0x55a324474c00 con 0x55a3170b5800 > -5760> 2018-07-11 07:33:46.964380 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:47.588049) v4 -- 0x55a34dec7a00 con 0 > -5759> 2018-07-11 07:33:47.025899 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12677 conn(0x55a32169d800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1858 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:33:47.061422) v4 > -5758> 2018-07-11 07:33:47.025919 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.1 10.8.128.106:0/12677 1858 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:47.061422) v4 ==== 2004+0+0 (124976646 0 0) 0x55a34de78600 con 0x55a32169d800 > -5757> 2018-07-11 07:33:47.025929 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:47.061422) v4 -- 0x55a33b42d200 con 0 > -5756> 2018-07-11 07:33:47.026054 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12677 conn(0x55a32169c000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1858 0x55a33b42ce00 osd_ping(ping e494 stamp 2018-07-11 07:33:47.061422) v4 > -5755> 2018-07-11 07:33:47.026075 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.1 10.8.128.106:0/12677 1858 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:47.061422) v4 ==== 2004+0+0 (124976646 0 0) 0x55a33b42ce00 con 0x55a32169c000 > -5754> 2018-07-11 07:33:47.026086 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:47.061422) v4 -- 0x55a32adf0a00 con 0 > -5753> 2018-07-11 07:33:47.088212 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 59] Generated table #36579: 4957165 keys, 80348241 bytes > -5752> 2018-07-11 07:33:47.088270 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294427088236, "cf_name": "default", "job": 59, "event": "table_file_creation", "file_number": 36579, "file_size": 80348241, "table_properties": {"data_size": 67109368, "index_size": 689182, "filter_size": 12548701, "raw_key_size": 143206990, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957165, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957165", "kMergeOperands": "0"}} > -5751> 2018-07-11 07:33:47.171388 7f354489b700 5 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:307] [JOB 0] Delete db/036502.sst type=2 #36502 -- OK > > -5750> 2018-07-11 07:33:47.171404 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294427171400, "job": 0, "event": "table_file_deletion", "file_number": 36502} > -5749> 2018-07-11 07:33:47.171415 7f354489b700 5 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:307] [JOB 0] Delete db/036501.sst type=2 #36501 -- OK > > -5748> 2018-07-11 07:33:47.171420 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294427171418, "job": 0, "event": "table_file_deletion", "file_number": 36501} > -5747> 2018-07-11 07:33:47.171428 7f354489b700 5 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:307] [JOB 0] Delete db/036500.sst type=2 #36500 -- OK > > -5746> 2018-07-11 07:33:47.171433 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294427171431, "job": 0, "event": "table_file_deletion", "file_number": 36500} > -5745> 2018-07-11 07:33:47.171442 7f354489b700 5 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:307] [JOB 0] Delete db/036499.sst type=2 #36499 -- OK > > -5744> 2018-07-11 07:33:47.171449 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294427171445, "job": 0, "event": "table_file_deletion", "file_number": 36499} > -5743> 2018-07-11 07:33:47.171456 7f354489b700 5 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:307] [JOB 0] Delete db/036498.sst type=2 #36498 -- OK > > -5742> 2018-07-11 07:33:47.171461 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294427171459, "job": 0, "event": "table_file_deletion", "file_number": 36498} > -5741> 2018-07-11 07:33:47.171469 7f354489b700 5 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:307] [JOB 0] Delete db/036497.sst type=2 #36497 -- OK > > -5740> 2018-07-11 07:33:47.171474 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294427171472, "job": 0, "event": "table_file_deletion", "file_number": 36497} > -5739> 2018-07-11 07:33:47.171481 7f354489b700 5 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:307] [JOB 0] Delete db/036496.sst type=2 #36496 -- OK > > -5738> 2018-07-11 07:33:47.171487 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294427171484, "job": 0, "event": "table_file_deletion", "file_number": 36496} > -5737> 2018-07-11 07:33:47.171494 7f354489b700 5 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:307] [JOB 0] Delete db/036495.sst type=2 #36495 -- OK > > -5736> 2018-07-11 07:33:47.171499 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294427171497, "job": 0, "event": "table_file_deletion", "file_number": 36495} > -5735> 2018-07-11 07:33:47.171506 7f354489b700 5 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:307] [JOB 0] Delete db/036493.sst type=2 #36493 -- OK > > -5734> 2018-07-11 07:33:47.171511 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294427171509, "job": 0, "event": "table_file_deletion", "file_number": 36493} > -5733> 2018-07-11 07:33:47.171532 7f354489b700 5 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:307] [JOB 0] Delete db/036487.sst type=2 #36487 -- OK > > -5732> 2018-07-11 07:33:47.171540 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294427171535, "job": 0, "event": "table_file_deletion", "file_number": 36487} > -5731> 2018-07-11 07:33:47.171550 7f354489b700 5 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:307] [JOB 0] Delete db/036476.sst type=2 #36476 -- OK > > -5730> 2018-07-11 07:33:47.171555 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294427171553, "job": 0, "event": "table_file_deletion", "file_number": 36476} > -5729> 2018-07-11 07:33:47.253178 7f35488a3700 1 -- 10.8.128.107:6800/12665 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55a32aa0d340 con 0 > -5728> 2018-07-11 07:33:47.253257 7f35488a3700 1 -- 10.8.128.107:6800/12665 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55a346875600 con 0 > -5727> 2018-07-11 07:33:47.265364 7f3543899700 5 osd.10 494 heartbeat: osd_stat(784 GB used, 147 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,0,0,0,1]) > -5726> 2018-07-11 07:33:47.265378 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6802/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:33:47.265377) v4 -- 0x55a3203f5600 con 0 > -5725> 2018-07-11 07:33:47.265418 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6803/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:33:47.265377) v4 -- 0x55a3203f4400 con 0 > -5724> 2018-07-11 07:33:47.265434 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6806/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:33:47.265377) v4 -- 0x55a3203f5c00 con 0 > -5723> 2018-07-11 07:33:47.265450 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6807/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:33:47.265377) v4 -- 0x55a3203f4e00 con 0 > -5722> 2018-07-11 07:33:47.265463 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6808/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:33:47.265377) v4 -- 0x55a3203f4600 con 0 > -5721> 2018-07-11 07:33:47.265473 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6810/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:33:47.265377) v4 -- 0x55a3203f5200 con 0 > -5720> 2018-07-11 07:33:47.265485 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6809/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:33:47.265377) v4 -- 0x55a3203f6000 con 0 > -5719> 2018-07-11 07:33:47.265497 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6811/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:33:47.265377) v4 -- 0x55a3203f4c00 con 0 > -5718> 2018-07-11 07:33:47.265508 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6810/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:33:47.265377) v4 -- 0x55a3203f5a00 con 0 > -5717> 2018-07-11 07:33:47.265518 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6811/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:33:47.265377) v4 -- 0x55a3203f5800 con 0 > -5716> 2018-07-11 07:33:47.265531 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6806/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:33:47.265377) v4 -- 0x55a3203f5e00 con 0 > -5715> 2018-07-11 07:33:47.265543 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6807/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:33:47.265377) v4 -- 0x55a324966e00 con 0 > -5714> 2018-07-11 07:33:47.265563 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6802/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:33:47.265377) v4 -- 0x55a30fa76200 con 0 > -5713> 2018-07-11 07:33:47.265584 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6803/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:33:47.265377) v4 -- 0x55a348fbca00 con 0 > -5712> 2018-07-11 07:33:47.265598 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6806/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:33:47.265377) v4 -- 0x55a348fbd000 con 0 > -5711> 2018-07-11 07:33:47.265610 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6807/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:33:47.265377) v4 -- 0x55a30fa76400 con 0 > -5710> 2018-07-11 07:33:47.265628 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6802/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:33:47.265377) v4 -- 0x55a30fa76600 con 0 > -5709> 2018-07-11 07:33:47.265640 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6803/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:33:47.265377) v4 -- 0x55a348fbd200 con 0 > -5708> 2018-07-11 07:33:47.265683 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6806/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:33:47.265377) v4 -- 0x55a30fa76800 con 0 > -5707> 2018-07-11 07:33:47.265701 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6807/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:33:47.265377) v4 -- 0x55a348fbd800 con 0 > -5706> 2018-07-11 07:33:47.265727 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6802/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:33:47.265377) v4 -- 0x55a30fa76a00 con 0 > -5705> 2018-07-11 07:33:47.265743 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6803/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:33:47.265377) v4 -- 0x55a30fa76e00 con 0 > -5704> 2018-07-11 07:33:47.265843 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6803/15255 conn(0x55a314e1d800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1842 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:47.265377) v4 > -5703> 2018-07-11 07:33:47.265880 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6808/12651 conn(0x55a314e96000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1842 0x55a33b42ce00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:47.265377) v4 > -5702> 2018-07-11 07:33:47.265921 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6808/12651 1842 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:47.265377) v4 ==== 2004+0+0 (1218348138 0 0) 0x55a33b42ce00 con 0x55a314e96000 > -5701> 2018-07-11 07:33:47.265887 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6803/15255 1842 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:47.265377) v4 ==== 2004+0+0 (1218348138 0 0) 0x55a324474c00 con 0x55a314e1d800 > -5700> 2018-07-11 07:33:47.265928 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6806/12677 conn(0x55a314e84000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1842 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:47.265377) v4 > -5699> 2018-07-11 07:33:47.265949 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6806/12677 1842 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:47.265377) v4 ==== 2004+0+0 (1218348138 0 0) 0x55a34de6f000 con 0x55a314e84000 > -5698> 2018-07-11 07:33:47.265955 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6802/15255 conn(0x55a314e1c000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1842 0x55a33b42ce00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:47.265377) v4 > -5697> 2018-07-11 07:33:47.265966 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6802/15255 1842 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:47.265377) v4 ==== 2004+0+0 (1218348138 0 0) 0x55a33b42ce00 con 0x55a314e1c000 > -5696> 2018-07-11 07:33:47.265979 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6807/12617 conn(0x55a314e27800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1842 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:47.265377) v4 > -5695> 2018-07-11 07:33:47.265991 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6807/12617 1842 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:47.265377) v4 ==== 2004+0+0 (1218348138 0 0) 0x55a34de6f000 con 0x55a314e27800 > -5694> 2018-07-11 07:33:47.265989 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6806/12617 conn(0x55a314e26000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1842 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:47.265377) v4 > -5693> 2018-07-11 07:33:47.265997 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6807/12677 conn(0x55a314e85800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1842 0x55a33b42ce00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:47.265377) v4 > -5692> 2018-07-11 07:33:47.266013 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6806/12617 1842 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:47.265377) v4 ==== 2004+0+0 (1218348138 0 0) 0x55a324474c00 con 0x55a314e26000 > -5691> 2018-07-11 07:33:47.266018 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6807/12677 1842 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:47.265377) v4 ==== 2004+0+0 (1218348138 0 0) 0x55a33b42ce00 con 0x55a314e85800 > -5690> 2018-07-11 07:33:47.266036 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6809/12650 conn(0x55a314e1f000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1842 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:47.265377) v4 > -5689> 2018-07-11 07:33:47.266049 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6809/12650 1842 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:47.265377) v4 ==== 2004+0+0 (1218348138 0 0) 0x55a34de6f000 con 0x55a314e1f000 > -5688> 2018-07-11 07:33:47.266067 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6810/12651 conn(0x55a314e97800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1842 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:47.265377) v4 > -5687> 2018-07-11 07:33:47.266067 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6811/12650 conn(0x55a314e20800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1842 0x55a33b42ce00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:47.265377) v4 > -5686> 2018-07-11 07:33:47.266082 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6810/12651 1842 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:47.265377) v4 ==== 2004+0+0 (1218348138 0 0) 0x55a324474c00 con 0x55a314e97800 > -5685> 2018-07-11 07:33:47.266086 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6811/12650 1842 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:47.265377) v4 ==== 2004+0+0 (1218348138 0 0) 0x55a33b42ce00 con 0x55a314e20800 > -5684> 2018-07-11 07:33:47.266122 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6811/15287 conn(0x55a314e04800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1842 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:47.265377) v4 > -5683> 2018-07-11 07:33:47.266136 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6811/15287 1842 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:47.265377) v4 ==== 2004+0+0 (1218348138 0 0) 0x55a324474c00 con 0x55a314e04800 > -5682> 2018-07-11 07:33:47.266147 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6810/15287 conn(0x55a314e03000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1842 0x55a33b42ce00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:47.265377) v4 > -5681> 2018-07-11 07:33:47.266157 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6806/15263 conn(0x55a314e99000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1842 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:47.265377) v4 > -5680> 2018-07-11 07:33:47.266167 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6810/15287 1842 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:47.265377) v4 ==== 2004+0+0 (1218348138 0 0) 0x55a33b42ce00 con 0x55a314e03000 > -5679> 2018-07-11 07:33:47.266171 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6806/15263 1842 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:47.265377) v4 ==== 2004+0+0 (1218348138 0 0) 0x55a34de6f000 con 0x55a314e99000 > -5678> 2018-07-11 07:33:47.266197 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6802/13323 conn(0x55a314e15000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1842 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:47.265377) v4 > -5677> 2018-07-11 07:33:47.266212 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6802/13323 1842 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:47.265377) v4 ==== 2004+0+0 (1218348138 0 0) 0x55a324474c00 con 0x55a314e15000 > -5676> 2018-07-11 07:33:47.266218 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6807/15263 conn(0x55a314e9a800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1842 0x55a33b42ce00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:47.265377) v4 > -5675> 2018-07-11 07:33:47.266218 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6807/15885 conn(0x55a314e01800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1842 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:47.265377) v4 > -5674> 2018-07-11 07:33:47.266240 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6807/15263 1842 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:47.265377) v4 ==== 2004+0+0 (1218348138 0 0) 0x55a33b42ce00 con 0x55a314e9a800 > -5673> 2018-07-11 07:33:47.266241 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6807/15885 1842 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:47.265377) v4 ==== 2004+0+0 (1218348138 0 0) 0x55a34de6f000 con 0x55a314e01800 > -5672> 2018-07-11 07:33:47.266262 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6806/15885 conn(0x55a314e00000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1842 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:47.265377) v4 > -5671> 2018-07-11 07:33:47.266279 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6806/15885 1842 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:47.265377) v4 ==== 2004+0+0 (1218348138 0 0) 0x55a324474c00 con 0x55a314e00000 > -5670> 2018-07-11 07:33:47.266302 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6803/13323 conn(0x55a314e16800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1842 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:47.265377) v4 > -5669> 2018-07-11 07:33:47.266299 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6803/11967 conn(0x55a314e13800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1842 0x55a33b42ce00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:47.265377) v4 > -5668> 2018-07-11 07:33:47.266316 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6802/15919 conn(0x55a314e87000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1842 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:47.265377) v4 > -5667> 2018-07-11 07:33:47.266321 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6803/13323 1842 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:47.265377) v4 ==== 2004+0+0 (1218348138 0 0) 0x55a34de6f000 con 0x55a314e16800 > -5666> 2018-07-11 07:33:47.266324 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6803/11967 1842 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:47.265377) v4 ==== 2004+0+0 (1218348138 0 0) 0x55a33b42ce00 con 0x55a314e13800 > -5665> 2018-07-11 07:33:47.266333 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6802/15919 1842 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:47.265377) v4 ==== 2004+0+0 (1218348138 0 0) 0x55a324474c00 con 0x55a314e87000 > -5664> 2018-07-11 07:33:47.266371 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6803/15919 conn(0x55a314e88800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1842 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:47.265377) v4 > -5663> 2018-07-11 07:33:47.266385 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6803/15919 1842 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:47.265377) v4 ==== 2004+0+0 (1218348138 0 0) 0x55a34de6f000 con 0x55a314e88800 > -5662> 2018-07-11 07:33:47.266416 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6802/11967 conn(0x55a314e12000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1842 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:47.265377) v4 > -5661> 2018-07-11 07:33:47.266428 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6802/11967 1842 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:47.265377) v4 ==== 2004+0+0 (1218348138 0 0) 0x55a34de6f000 con 0x55a314e12000 > -5660> 2018-07-11 07:33:47.525821 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15255 conn(0x55a311b65800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1862 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:33:48.739412) v4 > -5659> 2018-07-11 07:33:47.525852 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.0 10.8.128.113:0/15255 1862 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:48.739412) v4 ==== 2004+0+0 (461890395 0 0) 0x55a324474c00 con 0x55a311b65800 > -5658> 2018-07-11 07:33:47.525867 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:48.739412) v4 -- 0x55a30fa76a00 con 0 > -5657> 2018-07-11 07:33:47.525897 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15255 conn(0x55a311b64000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1862 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:33:48.739412) v4 > -5656> 2018-07-11 07:33:47.525925 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.0 10.8.128.113:0/15255 1862 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:48.739412) v4 ==== 2004+0+0 (461890395 0 0) 0x55a34de6f000 con 0x55a311b64000 > -5655> 2018-07-11 07:33:47.525942 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:48.739412) v4 -- 0x55a30fa76e00 con 0 > -5654> 2018-07-11 07:33:47.753365 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a31369a900 con 0 > -5653> 2018-07-11 07:33:47.777718 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5096 0x55a321715680 replica scrub(pg: 6.15,from:0'0,to:430'105377,epoch:494/492,start:6:a9cfe566:::benchmark_data_magna108_35617_object1041629:0,end:6:a9d0c72c:::benchmark_data_magna108_35617_object1023320:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -5652> 2018-07-11 07:33:47.777749 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5096 ==== replica scrub(pg: 6.15,from:0'0,to:430'105377,epoch:494/492,start:6:a9cfe566:::benchmark_data_magna108_35617_object1041629:0,end:6:a9d0c72c:::benchmark_data_magna108_35617_object1023320:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 220+0+0 (1623555168 0 0) 0x55a321715680 con 0x55a317019000 > -5651> 2018-07-11 07:33:48.365888 7f3543899700 5 osd.10 494 heartbeat: osd_stat(784 GB used, 147 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,0,0,1]) > -5650> 2018-07-11 07:33:48.365906 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6802/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:33:48.365903) v4 -- 0x55a348fbda00 con 0 > -5649> 2018-07-11 07:33:48.365936 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6803/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:33:48.365903) v4 -- 0x55a30fa77000 con 0 > -5648> 2018-07-11 07:33:48.365953 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6806/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:33:48.365903) v4 -- 0x55a34dea8e00 con 0 > -5647> 2018-07-11 07:33:48.365976 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6807/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:33:48.365903) v4 -- 0x55a34de84800 con 0 > -5646> 2018-07-11 07:33:48.365999 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6808/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:33:48.365903) v4 -- 0x55a332e98c00 con 0 > -5645> 2018-07-11 07:33:48.366018 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6810/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:33:48.365903) v4 -- 0x55a32eb28600 con 0 > -5644> 2018-07-11 07:33:48.366036 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6809/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:33:48.365903) v4 -- 0x55a3170d1800 con 0 > -5643> 2018-07-11 07:33:48.366058 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6811/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:33:48.365903) v4 -- 0x55a33235ba00 con 0 > -5642> 2018-07-11 07:33:48.366082 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6810/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:33:48.365903) v4 -- 0x55a332e85800 con 0 > -5641> 2018-07-11 07:33:48.366097 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6811/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:33:48.365903) v4 -- 0x55a32eb21200 con 0 > -5640> 2018-07-11 07:33:48.366117 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6806/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:33:48.365903) v4 -- 0x55a332364600 con 0 > -5639> 2018-07-11 07:33:48.366139 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6807/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:33:48.365903) v4 -- 0x55a332eadc00 con 0 > -5638> 2018-07-11 07:33:48.366164 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6802/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:33:48.365903) v4 -- 0x55a332e84c00 con 0 > -5637> 2018-07-11 07:33:48.366184 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6803/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:33:48.365903) v4 -- 0x55a32eb43e00 con 0 > -5636> 2018-07-11 07:33:48.366231 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6806/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:33:48.365903) v4 -- 0x55a32eb43c00 con 0 > -5635> 2018-07-11 07:33:48.366268 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6807/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:33:48.365903) v4 -- 0x55a34dec7800 con 0 > -5634> 2018-07-11 07:33:48.366291 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6802/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:33:48.365903) v4 -- 0x55a33234ec00 con 0 > -5633> 2018-07-11 07:33:48.366338 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6803/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:33:48.365903) v4 -- 0x55a32eb3a600 con 0 > -5632> 2018-07-11 07:33:48.366355 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6806/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:33:48.365903) v4 -- 0x55a34dec6200 con 0 > -5631> 2018-07-11 07:33:48.366368 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6807/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:33:48.365903) v4 -- 0x55a332ead800 con 0 > -5630> 2018-07-11 07:33:48.366389 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6802/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:33:48.365903) v4 -- 0x55a32eb3b000 con 0 > -5629> 2018-07-11 07:33:48.366445 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6806/12677 conn(0x55a314e84000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1843 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:48.365903) v4 > -5628> 2018-07-11 07:33:48.366456 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6810/12651 conn(0x55a314e97800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1843 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:48.365903) v4 > -5627> 2018-07-11 07:33:48.366454 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6808/12651 conn(0x55a314e96000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1843 0x55a3265d7c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:48.365903) v4 > -5626> 2018-07-11 07:33:48.366471 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6810/12651 1843 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:48.365903) v4 ==== 2004+0+0 (631015558 0 0) 0x55a332e79a00 con 0x55a314e97800 > -5625> 2018-07-11 07:33:48.366473 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6806/12677 1843 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:48.365903) v4 ==== 2004+0+0 (631015558 0 0) 0x55a34de78600 con 0x55a314e84000 > -5624> 2018-07-11 07:33:48.366453 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6803/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:33:48.365903) v4 -- 0x55a32eb43200 con 0 > -5623> 2018-07-11 07:33:48.366483 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6808/12651 1843 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:48.365903) v4 ==== 2004+0+0 (631015558 0 0) 0x55a3265d7c00 con 0x55a314e96000 > -5622> 2018-07-11 07:33:48.366555 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6803/15255 conn(0x55a314e1d800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1843 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:48.365903) v4 > -5621> 2018-07-11 07:33:48.366571 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6803/15255 1843 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:48.365903) v4 ==== 2004+0+0 (631015558 0 0) 0x55a332e79a00 con 0x55a314e1d800 > -5620> 2018-07-11 07:33:48.366573 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6802/15255 conn(0x55a314e1c000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1843 0x55a3265d7c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:48.365903) v4 > -5619> 2018-07-11 07:33:48.366614 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6809/12650 conn(0x55a314e1f000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1843 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:48.365903) v4 > -5618> 2018-07-11 07:33:48.366628 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6809/12650 1843 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:48.365903) v4 ==== 2004+0+0 (631015558 0 0) 0x55a34de6f000 con 0x55a314e1f000 > -5617> 2018-07-11 07:33:48.366628 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6802/15255 1843 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:48.365903) v4 ==== 2004+0+0 (631015558 0 0) 0x55a3265d7c00 con 0x55a314e1c000 > -5616> 2018-07-11 07:33:48.366634 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6811/15287 conn(0x55a314e04800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1843 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:48.365903) v4 > -5615> 2018-07-11 07:33:48.366655 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6811/15287 1843 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:48.365903) v4 ==== 2004+0+0 (631015558 0 0) 0x55a324474c00 con 0x55a314e04800 > -5614> 2018-07-11 07:33:48.366678 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6806/15263 conn(0x55a314e99000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1843 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:48.365903) v4 > -5613> 2018-07-11 07:33:48.366682 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6806/12617 conn(0x55a314e26000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1843 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:48.365903) v4 > -5612> 2018-07-11 07:33:48.366695 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6806/12617 1843 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:48.365903) v4 ==== 2004+0+0 (631015558 0 0) 0x55a324474c00 con 0x55a314e26000 > -5611> 2018-07-11 07:33:48.366688 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6810/15287 conn(0x55a314e03000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1843 0x55a3265d7c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:48.365903) v4 > -5610> 2018-07-11 07:33:48.366692 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6806/15263 1843 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:48.365903) v4 ==== 2004+0+0 (631015558 0 0) 0x55a34de6f000 con 0x55a314e99000 > -5609> 2018-07-11 07:33:48.366706 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6810/15287 1843 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:48.365903) v4 ==== 2004+0+0 (631015558 0 0) 0x55a3265d7c00 con 0x55a314e03000 > -5608> 2018-07-11 07:33:48.366739 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6802/13323 conn(0x55a314e15000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1843 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:48.365903) v4 > -5607> 2018-07-11 07:33:48.366753 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6802/13323 1843 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:48.365903) v4 ==== 2004+0+0 (631015558 0 0) 0x55a324474c00 con 0x55a314e15000 > -5606> 2018-07-11 07:33:48.366750 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6803/13323 conn(0x55a314e16800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1843 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:48.365903) v4 > -5605> 2018-07-11 07:33:48.366749 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6807/12677 conn(0x55a314e85800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1843 0x55a3265d7c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:48.365903) v4 > -5604> 2018-07-11 07:33:48.366765 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6803/13323 1843 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:48.365903) v4 ==== 2004+0+0 (631015558 0 0) 0x55a34de6f000 con 0x55a314e16800 > -5603> 2018-07-11 07:33:48.366766 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6807/12677 1843 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:48.365903) v4 ==== 2004+0+0 (631015558 0 0) 0x55a3265d7c00 con 0x55a314e85800 > -5602> 2018-07-11 07:33:48.366783 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6806/15885 conn(0x55a314e00000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1843 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:48.365903) v4 > -5601> 2018-07-11 07:33:48.366792 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6806/15885 1843 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:48.365903) v4 ==== 2004+0+0 (631015558 0 0) 0x55a324474c00 con 0x55a314e00000 > -5600> 2018-07-11 07:33:48.366824 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6802/11967 conn(0x55a314e12000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1843 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:48.365903) v4 > -5599> 2018-07-11 07:33:48.366822 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6811/12650 conn(0x55a314e20800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1843 0x55a3265d7c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:48.365903) v4 > -5598> 2018-07-11 07:33:48.366840 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6811/12650 1843 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:48.365903) v4 ==== 2004+0+0 (631015558 0 0) 0x55a3265d7c00 con 0x55a314e20800 > -5597> 2018-07-11 07:33:48.366840 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6802/11967 1843 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:48.365903) v4 ==== 2004+0+0 (631015558 0 0) 0x55a34de6f000 con 0x55a314e12000 > -5596> 2018-07-11 07:33:48.366888 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6807/15263 conn(0x55a314e9a800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1843 0x55a3265d7c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:48.365903) v4 > -5595> 2018-07-11 07:33:48.366896 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6807/15885 conn(0x55a314e01800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1843 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:48.365903) v4 > -5594> 2018-07-11 07:33:48.366907 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6807/15263 1843 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:48.365903) v4 ==== 2004+0+0 (631015558 0 0) 0x55a3265d7c00 con 0x55a314e9a800 > -5593> 2018-07-11 07:33:48.366909 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6807/15885 1843 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:48.365903) v4 ==== 2004+0+0 (631015558 0 0) 0x55a34de6f000 con 0x55a314e01800 > -5592> 2018-07-11 07:33:48.366940 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6802/15919 conn(0x55a314e87000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1843 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:48.365903) v4 > -5591> 2018-07-11 07:33:48.366956 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6802/15919 1843 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:48.365903) v4 ==== 2004+0+0 (631015558 0 0) 0x55a324474c00 con 0x55a314e87000 > -5590> 2018-07-11 07:33:48.366962 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6807/12617 conn(0x55a314e27800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1843 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:48.365903) v4 > -5589> 2018-07-11 07:33:48.366961 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6803/11967 conn(0x55a314e13800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1843 0x55a3265d7c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:48.365903) v4 > -5588> 2018-07-11 07:33:48.366982 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6807/12617 1843 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:48.365903) v4 ==== 2004+0+0 (631015558 0 0) 0x55a34de6f000 con 0x55a314e27800 > -5587> 2018-07-11 07:33:48.366988 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6803/11967 1843 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:48.365903) v4 ==== 2004+0+0 (631015558 0 0) 0x55a3265d7c00 con 0x55a314e13800 > -5586> 2018-07-11 07:33:48.367029 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6803/15919 conn(0x55a314e88800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1843 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:48.365903) v4 > -5585> 2018-07-11 07:33:48.367043 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6803/15919 1843 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:48.365903) v4 ==== 2004+0+0 (631015558 0 0) 0x55a34de6f000 con 0x55a314e88800 > -5584> 2018-07-11 07:33:48.626286 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15255 conn(0x55a311b65800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1863 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:33:49.839835) v4 > -5583> 2018-07-11 07:33:48.626315 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.0 10.8.128.113:0/15255 1863 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:49.839835) v4 ==== 2004+0+0 (1689016441 0 0) 0x55a324474c00 con 0x55a311b65800 > -5582> 2018-07-11 07:33:48.626310 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15255 conn(0x55a311b64000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1863 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:33:49.839835) v4 > -5581> 2018-07-11 07:33:48.626330 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:49.839835) v4 -- 0x55a32eb28600 con 0 > -5580> 2018-07-11 07:33:48.626330 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.0 10.8.128.113:0/15255 1863 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:49.839835) v4 ==== 2004+0+0 (1689016441 0 0) 0x55a34de6f000 con 0x55a311b64000 > -5579> 2018-07-11 07:33:48.626356 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:49.839835) v4 -- 0x55a3170d1800 con 0 > -5578> 2018-07-11 07:33:48.726432 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15287 conn(0x55a311b72000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1832 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:33:49.940054) v4 > -5577> 2018-07-11 07:33:48.726443 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15287 conn(0x55a311b75000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1832 0x55a3265d7c00 osd_ping(ping e494 stamp 2018-07-11 07:33:49.940054) v4 > -5576> 2018-07-11 07:33:48.726469 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.4 10.8.128.113:0/15287 1832 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:49.940054) v4 ==== 2004+0+0 (3482701391 0 0) 0x55a332e79a00 con 0x55a311b72000 > -5575> 2018-07-11 07:33:48.726472 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.4 10.8.128.113:0/15287 1832 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:49.940054) v4 ==== 2004+0+0 (3482701391 0 0) 0x55a3265d7c00 con 0x55a311b75000 > -5574> 2018-07-11 07:33:48.726481 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:49.940054) v4 -- 0x55a30fa77000 con 0 > -5573> 2018-07-11 07:33:48.726507 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:49.940054) v4 -- 0x55a34de84800 con 0 > -5572> 2018-07-11 07:33:48.815935 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a324b2c6c0 con 0 > -5571> 2018-07-11 07:33:48.833805 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5097 0x55a321715900 replica scrub(pg: 6.15,from:0'0,to:430'106881,epoch:494/492,start:6:a9d0c72c:::benchmark_data_magna108_35617_object1023320:0,end:6:a9d20071:::benchmark_data_magna108_35617_object841343:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -5570> 2018-07-11 07:33:48.833836 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5097 ==== replica scrub(pg: 6.15,from:0'0,to:430'106881,epoch:494/492,start:6:a9d0c72c:::benchmark_data_magna108_35617_object1023320:0,end:6:a9d20071:::benchmark_data_magna108_35617_object841343:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 219+0+0 (3978669609 0 0) 0x55a321715900 con 0x55a317019000 > -5569> 2018-07-11 07:33:48.954769 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15919 conn(0x55a3170b4000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1842 0x55a33b42ce00 osd_ping(ping e494 stamp 2018-07-11 07:33:49.578518) v4 > -5568> 2018-07-11 07:33:48.954796 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.11 10.8.128.114:0/15919 1842 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:49.578518) v4 ==== 2004+0+0 (2690195573 0 0) 0x55a33b42ce00 con 0x55a3170b4000 > -5567> 2018-07-11 07:33:48.954797 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15919 conn(0x55a317074800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1842 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:33:49.578518) v4 > -5566> 2018-07-11 07:33:48.954810 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:49.578518) v4 -- 0x55a348fbda00 con 0 > -5565> 2018-07-11 07:33:48.954815 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.11 10.8.128.114:0/15919 1842 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:49.578518) v4 ==== 2004+0+0 (2690195573 0 0) 0x55a324474c00 con 0x55a317074800 > -5564> 2018-07-11 07:33:48.954838 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:49.578518) v4 -- 0x55a30fa77000 con 0 > -5563> 2018-07-11 07:33:49.317073 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15263 conn(0x55a321664000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1837 0x55a34de84800 osd_ping(ping e494 stamp 2018-07-11 07:33:50.530648) v4 > -5562> 2018-07-11 07:33:49.317102 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.5 10.8.128.113:0/15263 1837 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:50.530648) v4 ==== 2004+0+0 (1403725274 0 0) 0x55a34de84800 con 0x55a321664000 > -5561> 2018-07-11 07:33:49.317116 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:50.530648) v4 -- 0x55a348fbda00 con 0 > -5560> 2018-07-11 07:33:49.317167 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15263 conn(0x55a321667000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1837 0x55a34de84800 osd_ping(ping e494 stamp 2018-07-11 07:33:50.530648) v4 > -5559> 2018-07-11 07:33:49.317187 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.5 10.8.128.113:0/15263 1837 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:50.530648) v4 ==== 2004+0+0 (1403725274 0 0) 0x55a34de84800 con 0x55a321667000 > -5558> 2018-07-11 07:33:49.317209 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:50.530648) v4 -- 0x55a324966e00 con 0 > -5557> 2018-07-11 07:33:49.477044 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.107:0/12617 conn(0x55a32447e000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1813 0x55a33b42ce00 osd_ping(ping e494 stamp 2018-07-11 07:33:49.476747) v4 > -5556> 2018-07-11 07:33:49.477068 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.9 10.8.128.107:0/12617 1813 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:49.476747) v4 ==== 2004+0+0 (3340663146 0 0) 0x55a33b42ce00 con 0x55a32447e000 > -5555> 2018-07-11 07:33:49.477065 7f355d88e700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.107:0/12617 conn(0x55a32447f800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1813 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:33:49.476747) v4 > -5554> 2018-07-11 07:33:49.477085 7f355d88e700 1 -- 10.8.128.107:6803/12665 <== osd.9 10.8.128.107:0/12617 1813 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:49.476747) v4 ==== 2004+0+0 (3340663146 0 0) 0x55a34de78600 con 0x55a32447f800 > -5553> 2018-07-11 07:33:49.477086 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:49.476747) v4 -- 0x55a348fbda00 con 0 > -5552> 2018-07-11 07:33:49.477114 7f355d88e700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:49.476747) v4 -- 0x55a34dea8e00 con 0 > -5551> 2018-07-11 07:33:49.524833 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a324b43440 con 0 > -5550> 2018-07-11 07:33:49.543653 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5098 0x55a321715b80 replica scrub(pg: 6.15,from:0'0,to:430'106509,epoch:494/492,start:6:a9d20071:::benchmark_data_magna108_35617_object841343:0,end:6:a9d2e416:::benchmark_data_magna108.ceph.redhat.com_2478_object96831:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -5549> 2018-07-11 07:33:49.543677 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5098 ==== replica scrub(pg: 6.15,from:0'0,to:430'106509,epoch:494/492,start:6:a9d20071:::benchmark_data_magna108_35617_object841343:0,end:6:a9d2e416:::benchmark_data_magna108.ceph.redhat.com_2478_object96831:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 232+0+0 (2679795010 0 0) 0x55a321715b80 con 0x55a317019000 > -5548> 2018-07-11 07:33:49.748963 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b7000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1789 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:33:49.499537) v4 > -5547> 2018-07-11 07:33:49.748985 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.8 10.8.128.108:0/11967 1789 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:49.499537) v4 ==== 2004+0+0 (2092528628 0 0) 0x55a34de6f000 con 0x55a3170b7000 > -5546> 2018-07-11 07:33:49.748995 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:49.499537) v4 -- 0x55a34dea8e00 con 0 > -5545> 2018-07-11 07:33:49.749006 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b8800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1789 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:33:49.499537) v4 > -5544> 2018-07-11 07:33:49.749032 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.8 10.8.128.108:0/11967 1789 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:49.499537) v4 ==== 2004+0+0 (2092528628 0 0) 0x55a332e79a00 con 0x55a3170b8800 > -5543> 2018-07-11 07:33:49.749043 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:49.499537) v4 -- 0x55a30fa77000 con 0 > -5542> 2018-07-11 07:33:49.826890 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15287 conn(0x55a311b75000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1833 0x55a34de84800 osd_ping(ping e494 stamp 2018-07-11 07:33:51.040451) v4 > -5541> 2018-07-11 07:33:49.826911 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.4 10.8.128.113:0/15287 1833 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:51.040451) v4 ==== 2004+0+0 (2759935110 0 0) 0x55a34de84800 con 0x55a311b75000 > -5540> 2018-07-11 07:33:49.826927 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:51.040451) v4 -- 0x55a3203f5a00 con 0 > -5539> 2018-07-11 07:33:49.826925 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15287 conn(0x55a311b72000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1833 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:33:51.040451) v4 > -5538> 2018-07-11 07:33:49.826943 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.4 10.8.128.113:0/15287 1833 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:51.040451) v4 ==== 2004+0+0 (2759935110 0 0) 0x55a324474c00 con 0x55a311b72000 > -5537> 2018-07-11 07:33:49.826956 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:51.040451) v4 -- 0x55a30fa77000 con 0 > -5536> 2018-07-11 07:33:49.926459 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12677 conn(0x55a32169d800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1859 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:33:49.961984) v4 > -5535> 2018-07-11 07:33:49.926465 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12677 conn(0x55a32169c000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1859 0x55a348fbda00 osd_ping(ping e494 stamp 2018-07-11 07:33:49.961984) v4 > -5534> 2018-07-11 07:33:49.926481 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.1 10.8.128.106:0/12677 1859 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:49.961984) v4 ==== 2004+0+0 (1007089217 0 0) 0x55a34de78600 con 0x55a32169d800 > -5533> 2018-07-11 07:33:49.926487 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.1 10.8.128.106:0/12677 1859 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:49.961984) v4 ==== 2004+0+0 (1007089217 0 0) 0x55a348fbda00 con 0x55a32169c000 > -5532> 2018-07-11 07:33:49.926490 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:49.961984) v4 -- 0x55a30fa76e00 con 0 > -5531> 2018-07-11 07:33:49.926523 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:49.961984) v4 -- 0x55a3203f4c00 con 0 > -5530> 2018-07-11 07:33:50.162750 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12651 conn(0x55a3216a0800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1839 0x55a34de84800 osd_ping(ping e494 stamp 2018-07-11 07:33:50.198333) v4 > -5529> 2018-07-11 07:33:50.162784 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.2 10.8.128.106:0/12651 1839 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:50.198333) v4 ==== 2004+0+0 (1468739046 0 0) 0x55a34de84800 con 0x55a3216a0800 > -5528> 2018-07-11 07:33:50.162795 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:50.198333) v4 -- 0x55a3203f4600 con 0 > -5527> 2018-07-11 07:33:50.162786 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12651 conn(0x55a32169f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1839 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:33:50.198333) v4 > -5526> 2018-07-11 07:33:50.162806 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.2 10.8.128.106:0/12651 1839 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:50.198333) v4 ==== 2004+0+0 (1468739046 0 0) 0x55a332e79a00 con 0x55a32169f000 > -5525> 2018-07-11 07:33:50.162819 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:50.198333) v4 -- 0x55a30fa76a00 con 0 > -5524> 2018-07-11 07:33:50.246760 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a324b4eb40 con 0 > -5523> 2018-07-11 07:33:50.259296 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5099 0x55a333658000 replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:494/492,start:6:a9d2e416:::benchmark_data_magna108.ceph.redhat.com_2478_object96831:0,end:6:a9d3c488:::benchmark_data_magna108_35617_object711005:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -5522> 2018-07-11 07:33:50.259327 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5099 ==== replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:494/492,start:6:a9d2e416:::benchmark_data_magna108.ceph.redhat.com_2478_object96831:0,end:6:a9d3c488:::benchmark_data_magna108_35617_object711005:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 232+0+0 (1882385941 0 0) 0x55a333658000 con 0x55a317019000 > -5521> 2018-07-11 07:33:50.464779 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15885 conn(0x55a311b76800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1858 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:33:51.088497) v4 > -5520> 2018-07-11 07:33:50.464792 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15885 conn(0x55a3170b5800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1858 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:33:51.088497) v4 > -5519> 2018-07-11 07:33:50.464803 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.7 10.8.128.114:0/15885 1858 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:51.088497) v4 ==== 2004+0+0 (3942829916 0 0) 0x55a34de6f000 con 0x55a311b76800 > -5518> 2018-07-11 07:33:50.464809 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.7 10.8.128.114:0/15885 1858 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:51.088497) v4 ==== 2004+0+0 (3942829916 0 0) 0x55a324474c00 con 0x55a3170b5800 > -5517> 2018-07-11 07:33:50.464813 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:51.088497) v4 -- 0x55a30fa76e00 con 0 > -5516> 2018-07-11 07:33:50.464848 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:51.088497) v4 -- 0x55a30fa76a00 con 0 > -5515> 2018-07-11 07:33:50.875543 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a324b52000 con 0 > -5514> 2018-07-11 07:33:50.884311 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5100 0x55a333658280 replica scrub(pg: 6.15,from:0'0,to:430'106743,epoch:494/492,start:6:a9d3c488:::benchmark_data_magna108_35617_object711005:0,end:6:a9d4ee10:::benchmark_data_magna108.ceph.redhat.com_2478_object100036:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -5513> 2018-07-11 07:33:50.884334 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5100 ==== replica scrub(pg: 6.15,from:0'0,to:430'106743,epoch:494/492,start:6:a9d3c488:::benchmark_data_magna108_35617_object711005:0,end:6:a9d4ee10:::benchmark_data_magna108.ceph.redhat.com_2478_object100036:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (1383756598 0 0) 0x55a333658280 con 0x55a317019000 > -5512> 2018-07-11 07:33:51.266635 7f3543899700 5 osd.10 494 heartbeat: osd_stat(784 GB used, 147 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,0,1]) > -5511> 2018-07-11 07:33:51.266650 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6802/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:33:51.266648) v4 -- 0x55a32eb3b200 con 0 > -5510> 2018-07-11 07:33:51.266690 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6803/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:33:51.266648) v4 -- 0x55a32eb3b800 con 0 > -5509> 2018-07-11 07:33:51.266706 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6806/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:33:51.266648) v4 -- 0x55a33234fa00 con 0 > -5508> 2018-07-11 07:33:51.266721 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6807/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:33:51.266648) v4 -- 0x55a32eb3a000 con 0 > -5507> 2018-07-11 07:33:51.266740 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6808/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:33:51.266648) v4 -- 0x55a3170dec00 con 0 > -5506> 2018-07-11 07:33:51.266762 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6810/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:33:51.266648) v4 -- 0x55a32eb42a00 con 0 > -5505> 2018-07-11 07:33:51.266786 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6809/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:33:51.266648) v4 -- 0x55a32162ae00 con 0 > -5504> 2018-07-11 07:33:51.266804 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6811/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:33:51.266648) v4 -- 0x55a32eb3a400 con 0 > -5503> 2018-07-11 07:33:51.266824 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6810/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:33:51.266648) v4 -- 0x55a324af5c00 con 0 > -5502> 2018-07-11 07:33:51.266847 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6811/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:33:51.266648) v4 -- 0x55a33be6fe00 con 0 > -5501> 2018-07-11 07:33:51.266873 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6806/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:33:51.266648) v4 -- 0x55a344e85c00 con 0 > -5500> 2018-07-11 07:33:51.266895 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6807/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:33:51.266648) v4 -- 0x55a324af5a00 con 0 > -5499> 2018-07-11 07:33:51.266923 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6802/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:33:51.266648) v4 -- 0x55a31ce7da00 con 0 > -5498> 2018-07-11 07:33:51.266946 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6803/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:33:51.266648) v4 -- 0x55a31ce7d600 con 0 > -5497> 2018-07-11 07:33:51.266988 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6806/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:33:51.266648) v4 -- 0x55a332e78200 con 0 > -5496> 2018-07-11 07:33:51.267013 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6807/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:33:51.266648) v4 -- 0x55a332e78600 con 0 > -5495> 2018-07-11 07:33:51.267035 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6802/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:33:51.266648) v4 -- 0x55a34de63c00 con 0 > -5494> 2018-07-11 07:33:51.267060 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6802/15255 conn(0x55a314e1c000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1844 0x55a348fbda00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:51.266648) v4 > -5493> 2018-07-11 07:33:51.267084 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6802/15255 1844 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:51.266648) v4 ==== 2004+0+0 (1660980428 0 0) 0x55a348fbda00 con 0x55a314e1c000 > -5492> 2018-07-11 07:33:51.267089 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6803/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:33:51.266648) v4 -- 0x55a31ce7d800 con 0 > -5491> 2018-07-11 07:33:51.267097 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6803/15255 conn(0x55a314e1d800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1844 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:51.266648) v4 > -5490> 2018-07-11 07:33:51.267112 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6806/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:33:51.266648) v4 -- 0x55a34de62200 con 0 > -5489> 2018-07-11 07:33:51.267119 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6803/15255 1844 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:51.266648) v4 ==== 2004+0+0 (1660980428 0 0) 0x55a324474c00 con 0x55a314e1d800 > -5488> 2018-07-11 07:33:51.267132 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6807/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:33:51.266648) v4 -- 0x55a34dec6400 con 0 > -5487> 2018-07-11 07:33:51.267164 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6802/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:33:51.266648) v4 -- 0x55a34dec6a00 con 0 > -5486> 2018-07-11 07:33:51.267183 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6803/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:33:51.266648) v4 -- 0x55a31ce7de00 con 0 > -5485> 2018-07-11 07:33:51.267267 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6809/12650 conn(0x55a314e1f000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1844 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:51.266648) v4 > -5484> 2018-07-11 07:33:51.267290 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6809/12650 1844 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:51.266648) v4 ==== 2004+0+0 (1660980428 0 0) 0x55a34de78600 con 0x55a314e1f000 > -5483> 2018-07-11 07:33:51.267324 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6810/12651 conn(0x55a314e97800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1844 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:51.266648) v4 > -5482> 2018-07-11 07:33:51.267340 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6810/12651 1844 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:51.266648) v4 ==== 2004+0+0 (1660980428 0 0) 0x55a332e79a00 con 0x55a314e97800 > -5481> 2018-07-11 07:33:51.267343 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6806/12677 conn(0x55a314e84000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1844 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:51.266648) v4 > -5480> 2018-07-11 07:33:51.267355 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6806/12677 1844 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:51.266648) v4 ==== 2004+0+0 (1660980428 0 0) 0x55a34de78600 con 0x55a314e84000 > -5479> 2018-07-11 07:33:51.267318 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6808/12651 conn(0x55a314e96000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1844 0x55a3203f4600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:51.266648) v4 > -5478> 2018-07-11 07:33:51.267371 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6811/15287 conn(0x55a314e04800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1844 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:51.266648) v4 > -5477> 2018-07-11 07:33:51.267374 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6808/12651 1844 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:51.266648) v4 ==== 2004+0+0 (1660980428 0 0) 0x55a3203f4600 con 0x55a314e96000 > -5476> 2018-07-11 07:33:51.267383 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6811/15287 1844 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:51.266648) v4 ==== 2004+0+0 (1660980428 0 0) 0x55a332e79a00 con 0x55a314e04800 > -5475> 2018-07-11 07:33:51.267389 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6806/15263 conn(0x55a314e99000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1844 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:51.266648) v4 > -5474> 2018-07-11 07:33:51.267422 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6806/15263 1844 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:51.266648) v4 ==== 2004+0+0 (1660980428 0 0) 0x55a34de78600 con 0x55a314e99000 > -5473> 2018-07-11 07:33:51.267432 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6807/12677 conn(0x55a314e85800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1844 0x55a3203f4600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:51.266648) v4 > -5472> 2018-07-11 07:33:51.267439 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6802/13323 conn(0x55a314e15000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1844 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:51.266648) v4 > -5471> 2018-07-11 07:33:51.267452 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6802/13323 1844 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:51.266648) v4 ==== 2004+0+0 (1660980428 0 0) 0x55a332e79a00 con 0x55a314e15000 > -5470> 2018-07-11 07:33:51.267452 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6807/12677 1844 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:51.266648) v4 ==== 2004+0+0 (1660980428 0 0) 0x55a3203f4600 con 0x55a314e85800 > -5469> 2018-07-11 07:33:51.267452 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6807/12617 conn(0x55a314e27800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1844 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:51.266648) v4 > -5468> 2018-07-11 07:33:51.267465 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6807/12617 1844 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:51.266648) v4 ==== 2004+0+0 (1660980428 0 0) 0x55a34de78600 con 0x55a314e27800 > -5467> 2018-07-11 07:33:51.267483 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6806/12617 conn(0x55a314e26000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1844 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:51.266648) v4 > -5466> 2018-07-11 07:33:51.267498 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6807/15263 conn(0x55a314e9a800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1844 0x55a3203f4600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:51.266648) v4 > -5465> 2018-07-11 07:33:51.267509 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6806/12617 1844 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:51.266648) v4 ==== 2004+0+0 (1660980428 0 0) 0x55a332e79a00 con 0x55a314e26000 > -5464> 2018-07-11 07:33:51.267509 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6803/13323 conn(0x55a314e16800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1844 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:51.266648) v4 > -5463> 2018-07-11 07:33:51.267516 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6807/15263 1844 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:51.266648) v4 ==== 2004+0+0 (1660980428 0 0) 0x55a3203f4600 con 0x55a314e9a800 > -5462> 2018-07-11 07:33:51.267523 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6803/13323 1844 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:51.266648) v4 ==== 2004+0+0 (1660980428 0 0) 0x55a34de78600 con 0x55a314e16800 > -5461> 2018-07-11 07:33:51.267564 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6810/15287 conn(0x55a314e03000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1844 0x55a3203f4600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:51.266648) v4 > -5460> 2018-07-11 07:33:51.267573 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6807/15885 conn(0x55a314e01800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1844 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:51.266648) v4 > -5459> 2018-07-11 07:33:51.267590 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6807/15885 1844 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:51.266648) v4 ==== 2004+0+0 (1660980428 0 0) 0x55a34de78600 con 0x55a314e01800 > -5458> 2018-07-11 07:33:51.267582 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6810/15287 1844 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:51.266648) v4 ==== 2004+0+0 (1660980428 0 0) 0x55a3203f4600 con 0x55a314e03000 > -5457> 2018-07-11 07:33:51.267590 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6806/15885 conn(0x55a314e00000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1844 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:51.266648) v4 > -5456> 2018-07-11 07:33:51.267614 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6806/15885 1844 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:51.266648) v4 ==== 2004+0+0 (1660980428 0 0) 0x55a332e79a00 con 0x55a314e00000 > -5455> 2018-07-11 07:33:51.267626 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6802/11967 conn(0x55a314e12000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1844 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:51.266648) v4 > -5454> 2018-07-11 07:33:51.267638 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6802/11967 1844 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:51.266648) v4 ==== 2004+0+0 (1660980428 0 0) 0x55a34de78600 con 0x55a314e12000 > -5453> 2018-07-11 07:33:51.267637 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6811/12650 conn(0x55a314e20800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1844 0x55a3203f4600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:51.266648) v4 > -5452> 2018-07-11 07:33:51.267654 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6802/15919 conn(0x55a314e87000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1844 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:51.266648) v4 > -5451> 2018-07-11 07:33:51.267656 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6811/12650 1844 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:51.266648) v4 ==== 2004+0+0 (1660980428 0 0) 0x55a3203f4600 con 0x55a314e20800 > -5450> 2018-07-11 07:33:51.267668 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6802/15919 1844 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:51.266648) v4 ==== 2004+0+0 (1660980428 0 0) 0x55a332e79a00 con 0x55a314e87000 > -5449> 2018-07-11 07:33:51.267675 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6803/15919 conn(0x55a314e88800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1844 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:51.266648) v4 > -5448> 2018-07-11 07:33:51.267688 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6803/15919 1844 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:51.266648) v4 ==== 2004+0+0 (1660980428 0 0) 0x55a34de78600 con 0x55a314e88800 > -5447> 2018-07-11 07:33:51.267703 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6803/11967 conn(0x55a314e13800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1844 0x55a3203f4600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:51.266648) v4 > -5446> 2018-07-11 07:33:51.267720 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6803/11967 1844 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:51.266648) v4 ==== 2004+0+0 (1660980428 0 0) 0x55a3203f4600 con 0x55a314e13800 > -5445> 2018-07-11 07:33:51.570561 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a324b53d40 con 0 > -5444> 2018-07-11 07:33:51.594098 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5101 0x55a333658500 replica scrub(pg: 6.15,from:0'0,to:430'106015,epoch:494/492,start:6:a9d4ee10:::benchmark_data_magna108.ceph.redhat.com_2478_object100036:0,end:6:a9d5ed18:::benchmark_data_magna108.ceph.redhat.com_2478_object117144:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -5443> 2018-07-11 07:33:51.594129 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5101 ==== replica scrub(pg: 6.15,from:0'0,to:430'106015,epoch:494/492,start:6:a9d4ee10:::benchmark_data_magna108.ceph.redhat.com_2478_object100036:0,end:6:a9d5ed18:::benchmark_data_magna108.ceph.redhat.com_2478_object117144:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 248+0+0 (2554543722 0 0) 0x55a333658500 con 0x55a317019000 > -5442> 2018-07-11 07:33:51.647899 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.115:0/13323 conn(0x55a321665800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1850 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:33:51.985202) v4 > -5441> 2018-07-11 07:33:51.647926 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.6 10.8.128.115:0/13323 1850 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:51.985202) v4 ==== 2004+0+0 (3905946298 0 0) 0x55a332e79a00 con 0x55a321665800 > -5440> 2018-07-11 07:33:51.647939 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:51.985202) v4 -- 0x55a31ce7da00 con 0 > -5439> 2018-07-11 07:33:51.647995 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.115:0/13323 conn(0x55a321668800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1850 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:33:51.985202) v4 > -5438> 2018-07-11 07:33:51.648016 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.6 10.8.128.115:0/13323 1850 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:51.985202) v4 ==== 2004+0+0 (3905946298 0 0) 0x55a332e79a00 con 0x55a321668800 > -5437> 2018-07-11 07:33:51.648031 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:51.985202) v4 -- 0x55a33be6fe00 con 0 > -5436> 2018-07-11 07:33:51.674857 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12650 conn(0x55a311b67000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1854 0x55a3203f4600 osd_ping(ping e494 stamp 2018-07-11 07:33:51.710446) v4 > -5435> 2018-07-11 07:33:51.674881 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.3 10.8.128.106:0/12650 1854 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:51.710446) v4 ==== 2004+0+0 (146628034 0 0) 0x55a3203f4600 con 0x55a311b67000 > -5434> 2018-07-11 07:33:51.674895 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:51.710446) v4 -- 0x55a32eb3a000 con 0 > -5433> 2018-07-11 07:33:51.674946 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12650 conn(0x55a311b68800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1854 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:33:51.710446) v4 > -5432> 2018-07-11 07:33:51.674968 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.3 10.8.128.106:0/12650 1854 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:51.710446) v4 ==== 2004+0+0 (146628034 0 0) 0x55a34de78600 con 0x55a311b68800 > -5431> 2018-07-11 07:33:51.674980 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:51.710446) v4 -- 0x55a31ce7d600 con 0 > -5430> 2018-07-11 07:33:52.233914 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a322082480 con 0 > -5429> 2018-07-11 07:33:52.242683 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5102 0x55a333658780 replica scrub(pg: 6.15,from:0'0,to:430'106361,epoch:494/492,start:6:a9d5ed18:::benchmark_data_magna108.ceph.redhat.com_2478_object117144:0,end:6:a9d6c03d:::benchmark_data_magna108.ceph.redhat.com_2478_object398604:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -5428> 2018-07-11 07:33:52.242713 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5102 ==== replica scrub(pg: 6.15,from:0'0,to:430'106361,epoch:494/492,start:6:a9d5ed18:::benchmark_data_magna108.ceph.redhat.com_2478_object117144:0,end:6:a9d6c03d:::benchmark_data_magna108.ceph.redhat.com_2478_object398604:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 248+0+0 (167864425 0 0) 0x55a333658780 con 0x55a317019000 > -5427> 2018-07-11 07:33:52.253452 7f35488a3700 1 -- 10.8.128.107:6800/12665 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55a32161fb80 con 0 > -5426> 2018-07-11 07:33:52.253530 7f35488a3700 1 -- 10.8.128.107:6800/12665 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55a3216de000 con 0 > -5425> 2018-07-11 07:33:52.889782 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a34a81afc0 con 0 > -5424> 2018-07-11 07:33:52.910756 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5103 0x55a333658a00 replica scrub(pg: 6.15,from:0'0,to:430'106411,epoch:494/492,start:6:a9d6c03d:::benchmark_data_magna108.ceph.redhat.com_2478_object398604:0,end:6:a9d80670:::benchmark_data_magna108.ceph.redhat.com_2478_object50414:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -5423> 2018-07-11 07:33:52.910788 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5103 ==== replica scrub(pg: 6.15,from:0'0,to:430'106411,epoch:494/492,start:6:a9d6c03d:::benchmark_data_magna108.ceph.redhat.com_2478_object398604:0,end:6:a9d80670:::benchmark_data_magna108.ceph.redhat.com_2478_object50414:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 247+0+0 (3648101615 0 0) 0x55a333658a00 con 0x55a317019000 > -5422> 2018-07-11 07:33:53.249495 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b7000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1790 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:33:52.999999) v4 > -5421> 2018-07-11 07:33:53.249515 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b8800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1790 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:33:52.999999) v4 > -5420> 2018-07-11 07:33:53.249530 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.8 10.8.128.108:0/11967 1790 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:52.999999) v4 ==== 2004+0+0 (3299296364 0 0) 0x55a34de6f000 con 0x55a3170b7000 > -5419> 2018-07-11 07:33:53.249534 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.8 10.8.128.108:0/11967 1790 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:52.999999) v4 ==== 2004+0+0 (3299296364 0 0) 0x55a324474c00 con 0x55a3170b8800 > -5418> 2018-07-11 07:33:53.249551 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:52.999999) v4 -- 0x55a31ce7d600 con 0 > -5417> 2018-07-11 07:33:53.249587 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:52.999999) v4 -- 0x55a33be6fe00 con 0 > -5416> 2018-07-11 07:33:53.327299 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15287 conn(0x55a311b72000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1834 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:33:54.540879) v4 > -5415> 2018-07-11 07:33:53.327320 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.4 10.8.128.113:0/15287 1834 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:54.540879) v4 ==== 2004+0+0 (1193362309 0 0) 0x55a332e79a00 con 0x55a311b72000 > -5414> 2018-07-11 07:33:53.327329 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:54.540879) v4 -- 0x55a33be6fe00 con 0 > -5413> 2018-07-11 07:33:53.327334 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15287 conn(0x55a311b75000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1834 0x55a3170dec00 osd_ping(ping e494 stamp 2018-07-11 07:33:54.540879) v4 > -5412> 2018-07-11 07:33:53.327361 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.4 10.8.128.113:0/15287 1834 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:54.540879) v4 ==== 2004+0+0 (1193362309 0 0) 0x55a3170dec00 con 0x55a311b75000 > -5411> 2018-07-11 07:33:53.327377 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:54.540879) v4 -- 0x55a32eb3a000 con 0 > -5410> 2018-07-11 07:33:53.926758 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15255 conn(0x55a311b64000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1864 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:33:55.140273) v4 > -5409> 2018-07-11 07:33:53.926760 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15255 conn(0x55a311b65800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1864 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:33:55.140273) v4 > -5408> 2018-07-11 07:33:53.926783 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.0 10.8.128.113:0/15255 1864 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:55.140273) v4 ==== 2004+0+0 (1798452219 0 0) 0x55a34de78600 con 0x55a311b64000 > -5407> 2018-07-11 07:33:53.926787 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.0 10.8.128.113:0/15255 1864 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:55.140273) v4 ==== 2004+0+0 (1798452219 0 0) 0x55a324474c00 con 0x55a311b65800 > -5406> 2018-07-11 07:33:53.926795 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:55.140273) v4 -- 0x55a31ce7d600 con 0 > -5405> 2018-07-11 07:33:53.926822 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:55.140273) v4 -- 0x55a33be6fe00 con 0 > -5404> 2018-07-11 07:33:53.965147 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15885 conn(0x55a311b76800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1859 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:33:54.588914) v4 > -5403> 2018-07-11 07:33:53.965150 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15885 conn(0x55a3170b5800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1859 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:33:54.588914) v4 > -5402> 2018-07-11 07:33:53.965168 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.7 10.8.128.114:0/15885 1859 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:54.588914) v4 ==== 2004+0+0 (1641161866 0 0) 0x55a34de6f000 con 0x55a311b76800 > -5401> 2018-07-11 07:33:53.965169 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.7 10.8.128.114:0/15885 1859 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:54.588914) v4 ==== 2004+0+0 (1641161866 0 0) 0x55a332e79a00 con 0x55a3170b5800 > -5400> 2018-07-11 07:33:53.965177 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:54.588914) v4 -- 0x55a31ce7d600 con 0 > -5399> 2018-07-11 07:33:53.965202 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:54.588914) v4 -- 0x55a33be6fe00 con 0 > -5398> 2018-07-11 07:33:54.263149 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12651 conn(0x55a3216a0800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1840 0x55a3203f4600 osd_ping(ping e494 stamp 2018-07-11 07:33:54.298774) v4 > -5397> 2018-07-11 07:33:54.263164 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12651 conn(0x55a32169f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1840 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:33:54.298774) v4 > -5396> 2018-07-11 07:33:54.263172 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.2 10.8.128.106:0/12651 1840 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:54.298774) v4 ==== 2004+0+0 (887959054 0 0) 0x55a3203f4600 con 0x55a3216a0800 > -5395> 2018-07-11 07:33:54.263189 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.2 10.8.128.106:0/12651 1840 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:54.298774) v4 ==== 2004+0+0 (887959054 0 0) 0x55a324474c00 con 0x55a32169f000 > -5394> 2018-07-11 07:33:54.263195 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:54.298774) v4 -- 0x55a32eb3a000 con 0 > -5393> 2018-07-11 07:33:54.263237 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:54.298774) v4 -- 0x55a33be6fe00 con 0 > -5392> 2018-07-11 07:33:54.465531 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15885 conn(0x55a311b76800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1860 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:33:55.089330) v4 > -5391> 2018-07-11 07:33:54.465544 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15885 conn(0x55a3170b5800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1860 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:33:55.089330) v4 > -5390> 2018-07-11 07:33:54.465555 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.7 10.8.128.114:0/15885 1860 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:55.089330) v4 ==== 2004+0+0 (4239624661 0 0) 0x55a34de78600 con 0x55a311b76800 > -5389> 2018-07-11 07:33:54.465561 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.7 10.8.128.114:0/15885 1860 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:55.089330) v4 ==== 2004+0+0 (4239624661 0 0) 0x55a332e79a00 con 0x55a3170b5800 > -5388> 2018-07-11 07:33:54.465564 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:55.089330) v4 -- 0x55a344e85c00 con 0 > -5387> 2018-07-11 07:33:54.465586 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:55.089330) v4 -- 0x55a31ce7da00 con 0 > -5386> 2018-07-11 07:33:54.556153 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 59] Generated table #36580: 4957165 keys, 80348269 bytes > -5385> 2018-07-11 07:33:54.556200 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294434556174, "cf_name": "default", "job": 59, "event": "table_file_creation", "file_number": 36580, "file_size": 80348269, "table_properties": {"data_size": 67109383, "index_size": 689195, "filter_size": 12548701, "raw_key_size": 143206995, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957165, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957165", "kMergeOperands": "0"}} > -5384> 2018-07-11 07:33:54.575347 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12650 conn(0x55a311b67000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1855 0x55a3170dec00 osd_ping(ping e494 stamp 2018-07-11 07:33:54.610962) v4 > -5383> 2018-07-11 07:33:54.575375 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.3 10.8.128.106:0/12650 1855 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:54.610962) v4 ==== 2004+0+0 (1293003323 0 0) 0x55a3170dec00 con 0x55a311b67000 > -5382> 2018-07-11 07:33:54.575396 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:54.610962) v4 -- 0x55a32eb3b200 con 0 > -5381> 2018-07-11 07:33:54.575423 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12650 conn(0x55a311b68800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1855 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:33:54.610962) v4 > -5380> 2018-07-11 07:33:54.575444 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.3 10.8.128.106:0/12650 1855 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:54.610962) v4 ==== 2004+0+0 (1293003323 0 0) 0x55a34de6f000 con 0x55a311b68800 > -5379> 2018-07-11 07:33:54.575462 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:54.610962) v4 -- 0x55a32162ae00 con 0 > -5378> 2018-07-11 07:33:54.745131 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a31ece3680 con 0 > -5377> 2018-07-11 07:33:54.763617 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12651 conn(0x55a32169f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1841 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:33:54.799223) v4 > -5376> 2018-07-11 07:33:54.763626 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12651 conn(0x55a3216a0800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1841 0x55a3170dec00 osd_ping(ping e494 stamp 2018-07-11 07:33:54.799223) v4 > -5375> 2018-07-11 07:33:54.763643 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.2 10.8.128.106:0/12651 1841 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:54.799223) v4 ==== 2004+0+0 (3108033092 0 0) 0x55a3170dec00 con 0x55a3216a0800 > -5374> 2018-07-11 07:33:54.763642 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.2 10.8.128.106:0/12651 1841 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:54.799223) v4 ==== 2004+0+0 (3108033092 0 0) 0x55a324474c00 con 0x55a32169f000 > -5373> 2018-07-11 07:33:54.763658 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:54.799223) v4 -- 0x55a3203f4e00 con 0 > -5372> 2018-07-11 07:33:54.763684 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:54.799223) v4 -- 0x55a32eb42a00 con 0 > -5371> 2018-07-11 07:33:54.767334 7f3543899700 5 osd.10 494 heartbeat: osd_stat(784 GB used, 147 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist []) > -5370> 2018-07-11 07:33:54.767348 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6802/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:33:54.767347) v4 -- 0x55a332fae400 con 0 > -5369> 2018-07-11 07:33:54.767372 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6803/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:33:54.767347) v4 -- 0x55a34de84a00 con 0 > -5368> 2018-07-11 07:33:54.767389 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6806/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:33:54.767347) v4 -- 0x55a34dec7000 con 0 > -5367> 2018-07-11 07:33:54.767403 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6807/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:33:54.767347) v4 -- 0x55a34de5b400 con 0 > -5366> 2018-07-11 07:33:54.767417 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6808/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:33:54.767347) v4 -- 0x55a324af5600 con 0 > -5365> 2018-07-11 07:33:54.767428 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6810/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:33:54.767347) v4 -- 0x55a31ce7ac00 con 0 > -5364> 2018-07-11 07:33:54.767441 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6809/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:33:54.767347) v4 -- 0x55a31ce7b800 con 0 > -5363> 2018-07-11 07:33:54.767454 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6811/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:33:54.767347) v4 -- 0x55a31ce7ae00 con 0 > -5362> 2018-07-11 07:33:54.767466 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6810/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:33:54.767347) v4 -- 0x55a324af5400 con 0 > -5361> 2018-07-11 07:33:54.767478 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6811/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:33:54.767347) v4 -- 0x55a31ce7b400 con 0 > -5360> 2018-07-11 07:33:54.767496 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6806/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:33:54.767347) v4 -- 0x55a31ce7c000 con 0 > -5359> 2018-07-11 07:33:54.767512 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6807/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:33:54.767347) v4 -- 0x55a31ce7c600 con 0 > -5358> 2018-07-11 07:33:54.767529 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6802/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:33:54.767347) v4 -- 0x55a31ce7c400 con 0 > -5357> 2018-07-11 07:33:54.767542 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6803/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:33:54.767347) v4 -- 0x55a31ce7be00 con 0 > -5356> 2018-07-11 07:33:54.767556 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6806/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:33:54.767347) v4 -- 0x55a332e84a00 con 0 > -5355> 2018-07-11 07:33:54.767568 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6807/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:33:54.767347) v4 -- 0x55a332e84200 con 0 > -5354> 2018-07-11 07:33:54.767581 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6802/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:33:54.767347) v4 -- 0x55a32eb3ae00 con 0 > -5353> 2018-07-11 07:33:54.767625 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6803/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:33:54.767347) v4 -- 0x55a332e99400 con 0 > -5352> 2018-07-11 07:33:54.767656 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6806/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:33:54.767347) v4 -- 0x55a34dec6800 con 0 > -5351> 2018-07-11 07:33:54.767671 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6807/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:33:54.767347) v4 -- 0x55a32eb28400 con 0 > -5350> 2018-07-11 07:33:54.767684 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6802/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:33:54.767347) v4 -- 0x55a332365800 con 0 > -5349> 2018-07-11 07:33:54.767697 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6803/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:33:54.767347) v4 -- 0x55a32eb21a00 con 0 > -5348> 2018-07-11 07:33:54.767753 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6802/15255 conn(0x55a314e1c000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1845 0x55a3203f4600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:54.767347) v4 > -5347> 2018-07-11 07:33:54.767771 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6802/15255 1845 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:54.767347) v4 ==== 2004+0+0 (3536978768 0 0) 0x55a3203f4600 con 0x55a314e1c000 > -5346> 2018-07-11 07:33:54.767839 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6803/15255 conn(0x55a314e1d800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1845 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:54.767347) v4 > -5345> 2018-07-11 07:33:54.767856 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6803/15255 1845 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:54.767347) v4 ==== 2004+0+0 (3536978768 0 0) 0x55a324474c00 con 0x55a314e1d800 > -5344> 2018-07-11 07:33:54.767844 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6807/12677 conn(0x55a314e85800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1845 0x55a3203f4600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:54.767347) v4 > -5343> 2018-07-11 07:33:54.767851 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6806/12677 conn(0x55a314e84000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1845 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:54.767347) v4 > -5342> 2018-07-11 07:33:54.767877 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6807/12677 1845 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:54.767347) v4 ==== 2004+0+0 (3536978768 0 0) 0x55a3203f4600 con 0x55a314e85800 > -5341> 2018-07-11 07:33:54.767883 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6806/12677 1845 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:54.767347) v4 ==== 2004+0+0 (3536978768 0 0) 0x55a34de78600 con 0x55a314e84000 > -5340> 2018-07-11 07:33:54.767896 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6810/12651 conn(0x55a314e97800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1845 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:54.767347) v4 > -5339> 2018-07-11 07:33:54.767908 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6810/12651 1845 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:54.767347) v4 ==== 2004+0+0 (3536978768 0 0) 0x55a324474c00 con 0x55a314e97800 > -5338> 2018-07-11 07:33:54.767917 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6809/12650 conn(0x55a314e1f000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1845 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:54.767347) v4 > -5337> 2018-07-11 07:33:54.767929 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6809/12650 1845 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:54.767347) v4 ==== 2004+0+0 (3536978768 0 0) 0x55a34de78600 con 0x55a314e1f000 > -5336> 2018-07-11 07:33:54.767935 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6808/12651 conn(0x55a314e96000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1845 0x55a3203f4600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:54.767347) v4 > -5335> 2018-07-11 07:33:54.767950 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6808/12651 1845 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:54.767347) v4 ==== 2004+0+0 (3536978768 0 0) 0x55a3203f4600 con 0x55a314e96000 > -5334> 2018-07-11 07:33:54.767965 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6811/15287 conn(0x55a314e04800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1845 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:54.767347) v4 > -5333> 2018-07-11 07:33:54.767967 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6807/12617 conn(0x55a314e27800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1845 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:54.767347) v4 > -5332> 2018-07-11 07:33:54.767980 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6811/15287 1845 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:54.767347) v4 ==== 2004+0+0 (3536978768 0 0) 0x55a324474c00 con 0x55a314e04800 > -5331> 2018-07-11 07:33:54.767980 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6810/15287 conn(0x55a314e03000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1845 0x55a3203f4600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:54.767347) v4 > -5330> 2018-07-11 07:33:54.767996 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6810/15287 1845 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:54.767347) v4 ==== 2004+0+0 (3536978768 0 0) 0x55a3203f4600 con 0x55a314e03000 > -5329> 2018-07-11 07:33:54.767985 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6807/12617 1845 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:54.767347) v4 ==== 2004+0+0 (3536978768 0 0) 0x55a34de78600 con 0x55a314e27800 > -5328> 2018-07-11 07:33:54.768027 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6807/15263 conn(0x55a314e9a800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1845 0x55a3203f4600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:54.767347) v4 > -5327> 2018-07-11 07:33:54.768039 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6807/15263 1845 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:54.767347) v4 ==== 2004+0+0 (3536978768 0 0) 0x55a3203f4600 con 0x55a314e9a800 > -5326> 2018-07-11 07:33:54.768012 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6806/12617 conn(0x55a314e26000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1845 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:54.767347) v4 > -5325> 2018-07-11 07:33:54.768036 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6803/13323 conn(0x55a314e16800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1845 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:54.767347) v4 > -5324> 2018-07-11 07:33:54.768055 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6806/12617 1845 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:54.767347) v4 ==== 2004+0+0 (3536978768 0 0) 0x55a324474c00 con 0x55a314e26000 > -5323> 2018-07-11 07:33:54.768063 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6803/13323 1845 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:54.767347) v4 ==== 2004+0+0 (3536978768 0 0) 0x55a34de78600 con 0x55a314e16800 > -5322> 2018-07-11 07:33:54.768070 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6811/12650 conn(0x55a314e20800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1845 0x55a3203f4600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:54.767347) v4 > -5321> 2018-07-11 07:33:54.768082 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6811/12650 1845 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:54.767347) v4 ==== 2004+0+0 (3536978768 0 0) 0x55a3203f4600 con 0x55a314e20800 > -5320> 2018-07-11 07:33:54.768086 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6806/15885 conn(0x55a314e00000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1845 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:54.767347) v4 > -5319> 2018-07-11 07:33:54.768094 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6806/15263 conn(0x55a314e99000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1845 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:54.767347) v4 > -5318> 2018-07-11 07:33:54.768108 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6806/15263 1845 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:54.767347) v4 ==== 2004+0+0 (3536978768 0 0) 0x55a34de78600 con 0x55a314e99000 > -5317> 2018-07-11 07:33:54.768098 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6806/15885 1845 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:54.767347) v4 ==== 2004+0+0 (3536978768 0 0) 0x55a324474c00 con 0x55a314e00000 > -5316> 2018-07-11 07:33:54.768140 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6802/11967 conn(0x55a314e12000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1845 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:54.767347) v4 > -5315> 2018-07-11 07:33:54.768152 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6802/11967 1845 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:54.767347) v4 ==== 2004+0+0 (3536978768 0 0) 0x55a34de78600 con 0x55a314e12000 > -5314> 2018-07-11 07:33:54.768149 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6803/11967 conn(0x55a314e13800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1845 0x55a3203f4600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:54.767347) v4 > -5313> 2018-07-11 07:33:54.768160 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6803/11967 1845 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:54.767347) v4 ==== 2004+0+0 (3536978768 0 0) 0x55a3203f4600 con 0x55a314e13800 > -5312> 2018-07-11 07:33:54.768157 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6802/13323 conn(0x55a314e15000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1845 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:54.767347) v4 > -5311> 2018-07-11 07:33:54.768176 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6802/13323 1845 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:54.767347) v4 ==== 2004+0+0 (3536978768 0 0) 0x55a324474c00 con 0x55a314e15000 > -5310> 2018-07-11 07:33:54.768180 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6807/15885 conn(0x55a314e01800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1845 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:54.767347) v4 > -5309> 2018-07-11 07:33:54.768191 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6807/15885 1845 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:54.767347) v4 ==== 2004+0+0 (3536978768 0 0) 0x55a34de78600 con 0x55a314e01800 > -5308> 2018-07-11 07:33:54.768206 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6802/15919 conn(0x55a314e87000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1845 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:54.767347) v4 > -5307> 2018-07-11 07:33:54.768221 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6802/15919 1845 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:54.767347) v4 ==== 2004+0+0 (3536978768 0 0) 0x55a324474c00 con 0x55a314e87000 > -5306> 2018-07-11 07:33:54.768222 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6803/15919 conn(0x55a314e88800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1845 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:54.767347) v4 > -5305> 2018-07-11 07:33:54.768232 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6803/15919 1845 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:54.767347) v4 ==== 2004+0+0 (3536978768 0 0) 0x55a34de78600 con 0x55a314e88800 > -5304> 2018-07-11 07:33:54.768921 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5104 0x55a333659180 replica scrub(pg: 6.15,from:0'0,to:430'105516,epoch:494/492,start:6:a9d80670:::benchmark_data_magna108.ceph.redhat.com_2478_object50414:0,end:6:a9d91678:::benchmark_data_magna108_35617_object1136981:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -5303> 2018-07-11 07:33:54.768950 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5104 ==== replica scrub(pg: 6.15,from:0'0,to:430'105516,epoch:494/492,start:6:a9d80670:::benchmark_data_magna108.ceph.redhat.com_2478_object50414:0,end:6:a9d91678:::benchmark_data_magna108_35617_object1136981:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (3322546115 0 0) 0x55a333659180 con 0x55a317019000 > -5302> 2018-07-11 07:33:54.777482 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.107:0/12617 conn(0x55a32447e000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1814 0x55a3170dec00 osd_ping(ping e494 stamp 2018-07-11 07:33:54.777139) v4 > -5301> 2018-07-11 07:33:54.777487 7f355d88e700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.107:0/12617 conn(0x55a32447f800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1814 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:33:54.777139) v4 > -5300> 2018-07-11 07:33:54.777517 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.9 10.8.128.107:0/12617 1814 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:54.777139) v4 ==== 2004+0+0 (1650808366 0 0) 0x55a3170dec00 con 0x55a32447e000 > -5299> 2018-07-11 07:33:54.777521 7f355d88e700 1 -- 10.8.128.107:6803/12665 <== osd.9 10.8.128.107:0/12617 1814 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:54.777139) v4 ==== 2004+0+0 (1650808366 0 0) 0x55a34de78600 con 0x55a32447f800 > -5298> 2018-07-11 07:33:54.777535 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:54.777139) v4 -- 0x55a34de5b400 con 0 > -5297> 2018-07-11 07:33:54.777566 7f355d88e700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:54.777139) v4 -- 0x55a31ce7b800 con 0 > -5296> 2018-07-11 07:33:54.855102 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15919 conn(0x55a317074800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1843 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:33:55.478891) v4 > -5295> 2018-07-11 07:33:54.855125 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.11 10.8.128.114:0/15919 1843 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:55.478891) v4 ==== 2004+0+0 (3239377314 0 0) 0x55a324474c00 con 0x55a317074800 > -5294> 2018-07-11 07:33:54.855139 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:55.478891) v4 -- 0x55a34de84a00 con 0 > -5293> 2018-07-11 07:33:54.855134 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15919 conn(0x55a3170b4000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1843 0x55a3203f4600 osd_ping(ping e494 stamp 2018-07-11 07:33:55.478891) v4 > -5292> 2018-07-11 07:33:54.855155 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.11 10.8.128.114:0/15919 1843 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:55.478891) v4 ==== 2004+0+0 (3239377314 0 0) 0x55a3203f4600 con 0x55a3170b4000 > -5291> 2018-07-11 07:33:54.855172 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:55.478891) v4 -- 0x55a34de5b400 con 0 > -5290> 2018-07-11 07:33:55.148333 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.115:0/13323 conn(0x55a321665800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1851 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:33:55.485718) v4 > -5289> 2018-07-11 07:33:55.148356 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.6 10.8.128.115:0/13323 1851 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:55.485718) v4 ==== 2004+0+0 (3324057296 0 0) 0x55a332e79a00 con 0x55a321665800 > -5288> 2018-07-11 07:33:55.148365 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:55.485718) v4 -- 0x55a34de84a00 con 0 > -5287> 2018-07-11 07:33:55.148402 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.115:0/13323 conn(0x55a321668800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1851 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:33:55.485718) v4 > -5286> 2018-07-11 07:33:55.148415 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.6 10.8.128.115:0/13323 1851 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:55.485718) v4 ==== 2004+0+0 (3324057296 0 0) 0x55a332e79a00 con 0x55a321668800 > -5285> 2018-07-11 07:33:55.148424 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:55.485718) v4 -- 0x55a32eb3b800 con 0 > -5284> 2018-07-11 07:33:55.217455 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15263 conn(0x55a321664000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1838 0x55a3170dec00 osd_ping(ping e494 stamp 2018-07-11 07:33:56.431056) v4 > -5283> 2018-07-11 07:33:55.217480 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.5 10.8.128.113:0/15263 1838 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:56.431056) v4 ==== 2004+0+0 (3565142469 0 0) 0x55a3170dec00 con 0x55a321664000 > -5282> 2018-07-11 07:33:55.217493 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:56.431056) v4 -- 0x55a332fae400 con 0 > -5281> 2018-07-11 07:33:55.217570 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15263 conn(0x55a321667000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1838 0x55a3203f4600 osd_ping(ping e494 stamp 2018-07-11 07:33:56.431056) v4 > -5280> 2018-07-11 07:33:55.217590 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.5 10.8.128.113:0/15263 1838 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:56.431056) v4 ==== 2004+0+0 (3565142469 0 0) 0x55a3203f4600 con 0x55a321667000 > -5279> 2018-07-11 07:33:55.217605 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:56.431056) v4 -- 0x55a3203f4e00 con 0 > -5278> 2018-07-11 07:33:55.226756 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12677 conn(0x55a32169d800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1860 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:33:55.262467) v4 > -5277> 2018-07-11 07:33:55.226780 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.1 10.8.128.106:0/12677 1860 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:55.262467) v4 ==== 2004+0+0 (3940940349 0 0) 0x55a34de6f000 con 0x55a32169d800 > -5276> 2018-07-11 07:33:55.226790 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:55.262467) v4 -- 0x55a34dec7000 con 0 > -5275> 2018-07-11 07:33:55.226799 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12677 conn(0x55a32169c000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1860 0x55a3170dec00 osd_ping(ping e494 stamp 2018-07-11 07:33:55.262467) v4 > -5274> 2018-07-11 07:33:55.226820 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.1 10.8.128.106:0/12677 1860 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:55.262467) v4 ==== 2004+0+0 (3940940349 0 0) 0x55a3170dec00 con 0x55a32169c000 > -5273> 2018-07-11 07:33:55.226831 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:55.262467) v4 -- 0x55a3203f5600 con 0 > -5272> 2018-07-11 07:33:55.389383 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a32c8e66c0 con 0 > -5271> 2018-07-11 07:33:55.401528 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5105 0x55a333659400 replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:494/492,start:6:a9d91678:::benchmark_data_magna108_35617_object1136981:0,end:6:a9da5496:::benchmark_data_magna108.ceph.redhat.com_2478_object73106:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -5270> 2018-07-11 07:33:55.401555 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5105 ==== replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:494/492,start:6:a9d91678:::benchmark_data_magna108_35617_object1136981:0,end:6:a9da5496:::benchmark_data_magna108.ceph.redhat.com_2478_object73106:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (2196604400 0 0) 0x55a333659400 con 0x55a317019000 > -5269> 2018-07-11 07:33:55.627697 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15287 conn(0x55a311b72000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1835 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:33:56.841406) v4 > -5268> 2018-07-11 07:33:55.627723 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.4 10.8.128.113:0/15287 1835 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:56.841406) v4 ==== 2004+0+0 (1226290206 0 0) 0x55a324474c00 con 0x55a311b72000 > -5267> 2018-07-11 07:33:55.627715 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15287 conn(0x55a311b75000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1835 0x55a3203f4600 osd_ping(ping e494 stamp 2018-07-11 07:33:56.841406) v4 > -5266> 2018-07-11 07:33:55.627732 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:56.841406) v4 -- 0x55a32eb3b800 con 0 > -5265> 2018-07-11 07:33:55.627737 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.4 10.8.128.113:0/15287 1835 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:56.841406) v4 ==== 2004+0+0 (1226290206 0 0) 0x55a3203f4600 con 0x55a311b75000 > -5264> 2018-07-11 07:33:55.627750 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:56.841406) v4 -- 0x55a32adf0a00 con 0 > -5263> 2018-07-11 07:33:56.081738 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a324400000 con 0 > -5262> 2018-07-11 07:33:56.101779 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5106 0x55a333659680 replica scrub(pg: 6.15,from:0'0,to:430'106837,epoch:494/492,start:6:a9da5496:::benchmark_data_magna108.ceph.redhat.com_2478_object73106:0,end:6:a9db0d2f:::benchmark_data_magna108.ceph.redhat.com_2478_object404160:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -5261> 2018-07-11 07:33:56.101809 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5106 ==== replica scrub(pg: 6.15,from:0'0,to:430'106837,epoch:494/492,start:6:a9da5496:::benchmark_data_magna108.ceph.redhat.com_2478_object73106:0,end:6:a9db0d2f:::benchmark_data_magna108.ceph.redhat.com_2478_object404160:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 247+0+0 (4202848785 0 0) 0x55a333659680 con 0x55a317019000 > -5260> 2018-07-11 07:33:56.128127 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15287 conn(0x55a311b75000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1836 0x55a3203f5600 osd_ping(ping e494 stamp 2018-07-11 07:33:57.341818) v4 > -5259> 2018-07-11 07:33:56.128149 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.4 10.8.128.113:0/15287 1836 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:57.341818) v4 ==== 2004+0+0 (2571061858 0 0) 0x55a3203f5600 con 0x55a311b75000 > -5258> 2018-07-11 07:33:56.128143 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15287 conn(0x55a311b72000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1836 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:33:57.341818) v4 > -5257> 2018-07-11 07:33:56.128176 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.4 10.8.128.113:0/15287 1836 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:57.341818) v4 ==== 2004+0+0 (2571061858 0 0) 0x55a332e79a00 con 0x55a311b72000 > -5256> 2018-07-11 07:33:56.128169 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:57.341818) v4 -- 0x55a32adf0a00 con 0 > -5255> 2018-07-11 07:33:56.128209 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:57.341818) v4 -- 0x55a32eb3b800 con 0 > -5254> 2018-07-11 07:33:56.709000 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a324b78d80 con 0 > -5253> 2018-07-11 07:33:56.735128 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5107 0x55a333659b80 replica scrub(pg: 6.15,from:0'0,to:430'106538,epoch:494/492,start:6:a9db0d2f:::benchmark_data_magna108.ceph.redhat.com_2478_object404160:0,end:6:a9dbffb9:::benchmark_data_magna108_35617_object748958:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -5252> 2018-07-11 07:33:56.735157 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5107 ==== replica scrub(pg: 6.15,from:0'0,to:430'106538,epoch:494/492,start:6:a9db0d2f:::benchmark_data_magna108.ceph.redhat.com_2478_object404160:0,end:6:a9dbffb9:::benchmark_data_magna108_35617_object748958:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (2458890785 0 0) 0x55a333659b80 con 0x55a317019000 > -5251> 2018-07-11 07:33:56.744441 7f35498a5700 10 monclient: tick > -5250> 2018-07-11 07:33:56.744455 7f35498a5700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2018-07-11 07:33:26.744453) > -5249> 2018-07-11 07:33:56.927276 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12677 conn(0x55a32169d800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1861 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:33:56.962911) v4 > -5248> 2018-07-11 07:33:56.927298 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.1 10.8.128.106:0/12677 1861 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:56.962911) v4 ==== 2004+0+0 (1486197329 0 0) 0x55a34de78600 con 0x55a32169d800 > -5247> 2018-07-11 07:33:56.927294 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12677 conn(0x55a32169c000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1861 0x55a3203f5600 osd_ping(ping e494 stamp 2018-07-11 07:33:56.962911) v4 > -5246> 2018-07-11 07:33:56.927309 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:56.962911) v4 -- 0x55a34dec7000 con 0 > -5245> 2018-07-11 07:33:56.927316 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.1 10.8.128.106:0/12677 1861 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:56.962911) v4 ==== 2004+0+0 (1486197329 0 0) 0x55a3203f5600 con 0x55a32169c000 > -5244> 2018-07-11 07:33:56.927334 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:56.962911) v4 -- 0x55a32adf0a00 con 0 > -5243> 2018-07-11 07:33:57.077913 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.107:0/12617 conn(0x55a32447e000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1815 0x55a3203f4600 osd_ping(ping e494 stamp 2018-07-11 07:33:57.077600) v4 > -5242> 2018-07-11 07:33:57.077934 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.9 10.8.128.107:0/12617 1815 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:57.077600) v4 ==== 2004+0+0 (587155123 0 0) 0x55a3203f4600 con 0x55a32447e000 > -5241> 2018-07-11 07:33:57.077922 7f355d88e700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.107:0/12617 conn(0x55a32447f800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1815 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:33:57.077600) v4 > -5240> 2018-07-11 07:33:57.077943 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:57.077600) v4 -- 0x55a32adf0a00 con 0 > -5239> 2018-07-11 07:33:57.077944 7f355d88e700 1 -- 10.8.128.107:6803/12665 <== osd.9 10.8.128.107:0/12617 1815 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:57.077600) v4 ==== 2004+0+0 (587155123 0 0) 0x55a34de6f000 con 0x55a32447f800 > -5238> 2018-07-11 07:33:57.077964 7f355d88e700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:57.077600) v4 -- 0x55a34dec7000 con 0 > -5237> 2018-07-11 07:33:57.253724 7f35488a3700 1 -- 10.8.128.107:6800/12665 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55a32c6a3600 con 0 > -5236> 2018-07-11 07:33:57.253801 7f35488a3700 1 -- 10.8.128.107:6800/12665 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55a330933b80 con 0 > -5235> 2018-07-11 07:33:57.366018 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15885 conn(0x55a311b76800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1861 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:33:57.989788) v4 > -5234> 2018-07-11 07:33:57.366025 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15885 conn(0x55a3170b5800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1861 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:33:57.989788) v4 > -5233> 2018-07-11 07:33:57.366039 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.7 10.8.128.114:0/15885 1861 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:57.989788) v4 ==== 2004+0+0 (928059986 0 0) 0x55a34de78600 con 0x55a311b76800 > -5232> 2018-07-11 07:33:57.366042 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.7 10.8.128.114:0/15885 1861 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:57.989788) v4 ==== 2004+0+0 (928059986 0 0) 0x55a324474c00 con 0x55a3170b5800 > -5231> 2018-07-11 07:33:57.366048 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:57.989788) v4 -- 0x55a32162ae00 con 0 > -5230> 2018-07-11 07:33:57.366072 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:57.989788) v4 -- 0x55a34de84a00 con 0 > -5229> 2018-07-11 07:33:57.388361 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a3433eafc0 con 0 > -5228> 2018-07-11 07:33:57.401910 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5108 0x55a35634c500 replica scrub(pg: 6.15,from:0'0,to:430'106563,epoch:494/492,start:6:a9dbffb9:::benchmark_data_magna108_35617_object748958:0,end:6:a9dcd77e:::benchmark_data_magna108_35617_object591333:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -5227> 2018-07-11 07:33:57.401938 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5108 ==== replica scrub(pg: 6.15,from:0'0,to:430'106563,epoch:494/492,start:6:a9dbffb9:::benchmark_data_magna108_35617_object748958:0,end:6:a9dcd77e:::benchmark_data_magna108_35617_object591333:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (758309034 0 0) 0x55a35634c500 con 0x55a317019000 > -5226> 2018-07-11 07:33:57.667821 7f3543899700 5 osd.10 494 heartbeat: osd_stat(784 GB used, 147 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,0,1]) > -5225> 2018-07-11 07:33:57.667837 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6802/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:33:57.667834) v4 -- 0x55a34de62000 con 0 > -5224> 2018-07-11 07:33:57.667873 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6803/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:33:57.667834) v4 -- 0x55a33234a800 con 0 > -5223> 2018-07-11 07:33:57.667889 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6806/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:33:57.667834) v4 -- 0x55a34de6f800 con 0 > -5222> 2018-07-11 07:33:57.667903 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6807/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:33:57.667834) v4 -- 0x55a3170dea00 con 0 > -5221> 2018-07-11 07:33:57.667916 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6808/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:33:57.667834) v4 -- 0x55a32eb14a00 con 0 > -5220> 2018-07-11 07:33:57.667927 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6810/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:33:57.667834) v4 -- 0x55a330934400 con 0 > -5219> 2018-07-11 07:33:57.667939 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6809/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:33:57.667834) v4 -- 0x55a332e98200 con 0 > -5218> 2018-07-11 07:33:57.667958 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6811/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:33:57.667834) v4 -- 0x55a31ce7a400 con 0 > -5217> 2018-07-11 07:33:57.667978 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6810/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:33:57.667834) v4 -- 0x55a32eb29400 con 0 > -5216> 2018-07-11 07:33:57.667991 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6811/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:33:57.667834) v4 -- 0x55a3170d0000 con 0 > -5215> 2018-07-11 07:33:57.668004 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6806/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:33:57.667834) v4 -- 0x55a32aa0f800 con 0 > -5214> 2018-07-11 07:33:57.668020 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6807/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:33:57.667834) v4 -- 0x55a32eb20c00 con 0 > -5213> 2018-07-11 07:33:57.668037 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6802/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:33:57.667834) v4 -- 0x55a332e79600 con 0 > -5212> 2018-07-11 07:33:57.668054 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6803/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:33:57.667834) v4 -- 0x55a332e78c00 con 0 > -5211> 2018-07-11 07:33:57.668068 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6806/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:33:57.667834) v4 -- 0x55a332e84400 con 0 > -5210> 2018-07-11 07:33:57.668081 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6807/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:33:57.667834) v4 -- 0x55a3170df200 con 0 > -5209> 2018-07-11 07:33:57.668094 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6802/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:33:57.667834) v4 -- 0x55a30f329800 con 0 > -5208> 2018-07-11 07:33:57.668110 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6803/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:33:57.667834) v4 -- 0x55a332e85c00 con 0 > -5207> 2018-07-11 07:33:57.668128 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6806/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:33:57.667834) v4 -- 0x55a3170de800 con 0 > -5206> 2018-07-11 07:33:57.668145 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6807/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:33:57.667834) v4 -- 0x55a3170d1400 con 0 > -5205> 2018-07-11 07:33:57.668159 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6802/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:33:57.667834) v4 -- 0x55a332e76000 con 0 > -5204> 2018-07-11 07:33:57.668171 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6803/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:33:57.667834) v4 -- 0x55a332e77c00 con 0 > -5203> 2018-07-11 07:33:57.668261 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6802/15255 conn(0x55a314e1c000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1846 0x55a3203f5600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:57.667834) v4 > -5202> 2018-07-11 07:33:57.668283 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6802/15255 1846 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:57.667834) v4 ==== 2004+0+0 (619694405 0 0) 0x55a3203f5600 con 0x55a314e1c000 > -5201> 2018-07-11 07:33:57.668376 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6803/15255 conn(0x55a314e1d800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1846 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:57.667834) v4 > -5200> 2018-07-11 07:33:57.668407 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6803/15255 1846 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:57.667834) v4 ==== 2004+0+0 (619694405 0 0) 0x55a324474c00 con 0x55a314e1d800 > -5199> 2018-07-11 07:33:57.668400 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6807/12677 conn(0x55a314e85800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1846 0x55a3203f5600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:57.667834) v4 > -5198> 2018-07-11 07:33:57.668405 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6806/12677 conn(0x55a314e84000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1846 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:57.667834) v4 > -5197> 2018-07-11 07:33:57.668419 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6807/12677 1846 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:57.667834) v4 ==== 2004+0+0 (619694405 0 0) 0x55a3203f5600 con 0x55a314e85800 > -5196> 2018-07-11 07:33:57.668426 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6806/12677 1846 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:57.667834) v4 ==== 2004+0+0 (619694405 0 0) 0x55a34de6f000 con 0x55a314e84000 > -5195> 2018-07-11 07:33:57.668433 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6806/12617 conn(0x55a314e26000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1846 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:57.667834) v4 > -5194> 2018-07-11 07:33:57.668444 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6806/12617 1846 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:57.667834) v4 ==== 2004+0+0 (619694405 0 0) 0x55a324474c00 con 0x55a314e26000 > -5193> 2018-07-11 07:33:57.668454 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6807/12617 conn(0x55a314e27800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1846 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:57.667834) v4 > -5192> 2018-07-11 07:33:57.668465 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6807/12617 1846 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:57.667834) v4 ==== 2004+0+0 (619694405 0 0) 0x55a34de6f000 con 0x55a314e27800 > -5191> 2018-07-11 07:33:57.668469 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6810/15287 conn(0x55a314e03000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1846 0x55a3203f5600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:57.667834) v4 > -5190> 2018-07-11 07:33:57.668484 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6810/15287 1846 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:57.667834) v4 ==== 2004+0+0 (619694405 0 0) 0x55a3203f5600 con 0x55a314e03000 > -5189> 2018-07-11 07:33:57.668499 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6810/12651 conn(0x55a314e97800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1846 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:57.667834) v4 > -5188> 2018-07-11 07:33:57.668508 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6809/12650 conn(0x55a314e1f000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1846 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:57.667834) v4 > -5187> 2018-07-11 07:33:57.668516 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6807/15263 conn(0x55a314e9a800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1846 0x55a3203f5600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:57.667834) v4 > -5186> 2018-07-11 07:33:57.668524 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6809/12650 1846 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:57.667834) v4 ==== 2004+0+0 (619694405 0 0) 0x55a34de6f000 con 0x55a314e1f000 > -5185> 2018-07-11 07:33:57.668513 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6810/12651 1846 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:57.667834) v4 ==== 2004+0+0 (619694405 0 0) 0x55a324474c00 con 0x55a314e97800 > -5184> 2018-07-11 07:33:57.668530 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6807/15263 1846 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:57.667834) v4 ==== 2004+0+0 (619694405 0 0) 0x55a3203f5600 con 0x55a314e9a800 > -5183> 2018-07-11 07:33:57.668562 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6806/15263 conn(0x55a314e99000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1846 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:57.667834) v4 > -5182> 2018-07-11 07:33:57.668575 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6806/15263 1846 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:57.667834) v4 ==== 2004+0+0 (619694405 0 0) 0x55a34de6f000 con 0x55a314e99000 > -5181> 2018-07-11 07:33:57.668575 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6811/12650 conn(0x55a314e20800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1846 0x55a3203f5600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:57.667834) v4 > -5180> 2018-07-11 07:33:57.668585 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6811/12650 1846 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:57.667834) v4 ==== 2004+0+0 (619694405 0 0) 0x55a3203f5600 con 0x55a314e20800 > -5179> 2018-07-11 07:33:57.668603 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6803/13323 conn(0x55a314e16800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1846 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:57.667834) v4 > -5178> 2018-07-11 07:33:57.668603 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6811/15287 conn(0x55a314e04800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1846 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:57.667834) v4 > -5177> 2018-07-11 07:33:57.668615 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6803/13323 1846 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:57.667834) v4 ==== 2004+0+0 (619694405 0 0) 0x55a34de6f000 con 0x55a314e16800 > -5176> 2018-07-11 07:33:57.668615 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6808/12651 conn(0x55a314e96000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1846 0x55a3203f5600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:57.667834) v4 > -5175> 2018-07-11 07:33:57.668616 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6811/15287 1846 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:57.667834) v4 ==== 2004+0+0 (619694405 0 0) 0x55a324474c00 con 0x55a314e04800 > -5174> 2018-07-11 07:33:57.668624 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6808/12651 1846 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:57.667834) v4 ==== 2004+0+0 (619694405 0 0) 0x55a3203f5600 con 0x55a314e96000 > -5173> 2018-07-11 07:33:57.668656 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6806/15885 conn(0x55a314e00000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1846 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:57.667834) v4 > -5172> 2018-07-11 07:33:57.668661 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6803/11967 conn(0x55a314e13800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1846 0x55a3203f5600 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:57.667834) v4 > -5171> 2018-07-11 07:33:57.668662 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6807/15885 conn(0x55a314e01800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1846 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:57.667834) v4 > -5170> 2018-07-11 07:33:57.668670 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6806/15885 1846 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:57.667834) v4 ==== 2004+0+0 (619694405 0 0) 0x55a324474c00 con 0x55a314e00000 > -5169> 2018-07-11 07:33:57.668673 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6807/15885 1846 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:57.667834) v4 ==== 2004+0+0 (619694405 0 0) 0x55a34de6f000 con 0x55a314e01800 > -5168> 2018-07-11 07:33:57.668671 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6803/11967 1846 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:57.667834) v4 ==== 2004+0+0 (619694405 0 0) 0x55a3203f5600 con 0x55a314e13800 > -5167> 2018-07-11 07:33:57.668704 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6802/11967 conn(0x55a314e12000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1846 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:57.667834) v4 > -5166> 2018-07-11 07:33:57.668716 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6802/11967 1846 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:57.667834) v4 ==== 2004+0+0 (619694405 0 0) 0x55a34de6f000 con 0x55a314e12000 > -5165> 2018-07-11 07:33:57.668719 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6802/13323 conn(0x55a314e15000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1846 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:57.667834) v4 > -5164> 2018-07-11 07:33:57.668732 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6802/13323 1846 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:57.667834) v4 ==== 2004+0+0 (619694405 0 0) 0x55a324474c00 con 0x55a314e15000 > -5163> 2018-07-11 07:33:57.668747 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6803/15919 conn(0x55a314e88800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1846 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:57.667834) v4 > -5162> 2018-07-11 07:33:57.668759 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6803/15919 1846 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:57.667834) v4 ==== 2004+0+0 (619694405 0 0) 0x55a34de6f000 con 0x55a314e88800 > -5161> 2018-07-11 07:33:57.668766 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6802/15919 conn(0x55a314e87000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1846 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:33:57.667834) v4 > -5160> 2018-07-11 07:33:57.668779 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6802/15919 1846 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:33:57.667834) v4 ==== 2004+0+0 (619694405 0 0) 0x55a324474c00 con 0x55a314e87000 > -5159> 2018-07-11 07:33:57.755494 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15919 conn(0x55a3170b4000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1844 0x55a3203f5600 osd_ping(ping e494 stamp 2018-07-11 07:33:58.379307) v4 > -5158> 2018-07-11 07:33:57.755508 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15919 conn(0x55a317074800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1844 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:33:58.379307) v4 > -5157> 2018-07-11 07:33:57.755517 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.11 10.8.128.114:0/15919 1844 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:58.379307) v4 ==== 2004+0+0 (3776761260 0 0) 0x55a3203f5600 con 0x55a3170b4000 > -5156> 2018-07-11 07:33:57.755532 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.11 10.8.128.114:0/15919 1844 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:58.379307) v4 ==== 2004+0+0 (3776761260 0 0) 0x55a324474c00 con 0x55a317074800 > -5155> 2018-07-11 07:33:57.755538 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:58.379307) v4 -- 0x55a32eb14a00 con 0 > -5154> 2018-07-11 07:33:57.755569 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:58.379307) v4 -- 0x55a330934400 con 0 > -5153> 2018-07-11 07:33:58.027174 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15255 conn(0x55a311b65800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1865 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:33:59.240886) v4 > -5152> 2018-07-11 07:33:58.027195 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.0 10.8.128.113:0/15255 1865 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:59.240886) v4 ==== 2004+0+0 (2021688059 0 0) 0x55a332e79a00 con 0x55a311b65800 > -5151> 2018-07-11 07:33:58.027205 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:59.240886) v4 -- 0x55a33234a800 con 0 > -5150> 2018-07-11 07:33:58.027236 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15255 conn(0x55a311b64000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1865 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:33:59.240886) v4 > -5149> 2018-07-11 07:33:58.027264 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.0 10.8.128.113:0/15255 1865 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:59.240886) v4 ==== 2004+0+0 (2021688059 0 0) 0x55a34de6f000 con 0x55a311b64000 > -5148> 2018-07-11 07:33:58.027277 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:59.240886) v4 -- 0x55a332e98200 con 0 > -5147> 2018-07-11 07:33:58.062495 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a323b31d40 con 0 > -5146> 2018-07-11 07:33:58.085944 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5109 0x55a35634c780 replica scrub(pg: 6.15,from:0'0,to:430'106094,epoch:494/492,start:6:a9dcd77e:::benchmark_data_magna108_35617_object591333:0,end:6:a9ddc5c7:::benchmark_data_magna108_35617_object1012056:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -5145> 2018-07-11 07:33:58.085967 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5109 ==== replica scrub(pg: 6.15,from:0'0,to:430'106094,epoch:494/492,start:6:a9dcd77e:::benchmark_data_magna108_35617_object591333:0,end:6:a9ddc5c7:::benchmark_data_magna108_35617_object1012056:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 219+0+0 (2694011216 0 0) 0x55a35634c780 con 0x55a317019000 > -5144> 2018-07-11 07:33:58.931685 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a324b89b00 con 0 > -5143> 2018-07-11 07:33:58.943916 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5110 0x55a35634ca00 replica scrub(pg: 6.15,from:0'0,to:430'106425,epoch:494/492,start:6:a9ddc5c7:::benchmark_data_magna108_35617_object1012056:0,end:6:a9de86e9:::benchmark_data_magna108_35617_object171547:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -5142> 2018-07-11 07:33:58.943949 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5110 ==== replica scrub(pg: 6.15,from:0'0,to:430'106425,epoch:494/492,start:6:a9ddc5c7:::benchmark_data_magna108_35617_object1012056:0,end:6:a9de86e9:::benchmark_data_magna108_35617_object171547:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 219+0+0 (3306319568 0 0) 0x55a35634ca00 con 0x55a317019000 > -5141> 2018-07-11 07:33:59.028504 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15287 conn(0x55a311b72000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1837 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:34:00.242191) v4 > -5140> 2018-07-11 07:33:59.028527 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.4 10.8.128.113:0/15287 1837 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:00.242191) v4 ==== 2004+0+0 (1355348587 0 0) 0x55a324474c00 con 0x55a311b72000 > -5139> 2018-07-11 07:33:59.028521 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15287 conn(0x55a311b75000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1837 0x55a32adf0a00 osd_ping(ping e494 stamp 2018-07-11 07:34:00.242191) v4 > -5138> 2018-07-11 07:33:59.028536 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:00.242191) v4 -- 0x55a33234a800 con 0 > -5137> 2018-07-11 07:33:59.028538 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.4 10.8.128.113:0/15287 1837 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:00.242191) v4 ==== 2004+0+0 (1355348587 0 0) 0x55a32adf0a00 con 0x55a311b75000 > -5136> 2018-07-11 07:33:59.028561 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:00.242191) v4 -- 0x55a3170dea00 con 0 > -5135> 2018-07-11 07:33:59.066435 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15885 conn(0x55a311b76800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1862 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:33:59.690217) v4 > -5134> 2018-07-11 07:33:59.066439 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15885 conn(0x55a3170b5800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1862 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:33:59.690217) v4 > -5133> 2018-07-11 07:33:59.066456 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.7 10.8.128.114:0/15885 1862 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:59.690217) v4 ==== 2004+0+0 (3540968706 0 0) 0x55a34de78600 con 0x55a311b76800 > -5132> 2018-07-11 07:33:59.066458 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.7 10.8.128.114:0/15885 1862 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:59.690217) v4 ==== 2004+0+0 (3540968706 0 0) 0x55a332e79a00 con 0x55a3170b5800 > -5131> 2018-07-11 07:33:59.066466 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:59.690217) v4 -- 0x55a332e98200 con 0 > -5130> 2018-07-11 07:33:59.066492 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:59.690217) v4 -- 0x55a33234a800 con 0 > -5129> 2018-07-11 07:33:59.149818 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b8800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1791 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:33:58.900427) v4 > -5128> 2018-07-11 07:33:59.149838 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.8 10.8.128.108:0/11967 1791 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:58.900427) v4 ==== 2004+0+0 (2704380807 0 0) 0x55a324474c00 con 0x55a3170b8800 > -5127> 2018-07-11 07:33:59.149849 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:58.900427) v4 -- 0x55a34de84a00 con 0 > -5126> 2018-07-11 07:33:59.149845 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b7000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1791 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:33:58.900427) v4 > -5125> 2018-07-11 07:33:59.149867 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.8 10.8.128.108:0/11967 1791 ==== osd_ping(ping e494 stamp 2018-07-11 07:33:58.900427) v4 ==== 2004+0+0 (2704380807 0 0) 0x55a34de6f000 con 0x55a3170b7000 > -5124> 2018-07-11 07:33:59.149880 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:33:58.900427) v4 -- 0x55a34de6f800 con 0 > -5123> 2018-07-11 07:33:59.638998 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a31f9de900 con 0 > -5122> 2018-07-11 07:33:59.652872 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5111 0x55a35634cc80 replica scrub(pg: 6.15,from:0'0,to:430'106182,epoch:494/492,start:6:a9de86e9:::benchmark_data_magna108_35617_object171547:0,end:6:a9df346a:::benchmark_data_magna108_35617_object689783:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -5121> 2018-07-11 07:33:59.652896 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5111 ==== replica scrub(pg: 6.15,from:0'0,to:430'106182,epoch:494/492,start:6:a9de86e9:::benchmark_data_magna108_35617_object171547:0,end:6:a9df346a:::benchmark_data_magna108_35617_object689783:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (1821874848 0 0) 0x55a35634cc80 con 0x55a317019000 > -5120> 2018-07-11 07:33:59.848764 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.115:0/13323 conn(0x55a321665800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1852 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:34:00.186182) v4 > -5119> 2018-07-11 07:33:59.848790 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.6 10.8.128.115:0/13323 1852 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:00.186182) v4 ==== 2004+0+0 (2558070496 0 0) 0x55a332e79a00 con 0x55a321665800 > -5118> 2018-07-11 07:33:59.848805 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:00.186182) v4 -- 0x55a34de84a00 con 0 > -5117> 2018-07-11 07:33:59.848860 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.115:0/13323 conn(0x55a321668800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1852 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:34:00.186182) v4 > -5116> 2018-07-11 07:33:59.848879 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.6 10.8.128.115:0/13323 1852 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:00.186182) v4 ==== 2004+0+0 (2558070496 0 0) 0x55a332e79a00 con 0x55a321668800 > -5115> 2018-07-11 07:33:59.848896 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:00.186182) v4 -- 0x55a30fa76a00 con 0 > -5114> 2018-07-11 07:34:00.270797 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a3150858c0 con 0 > -5113> 2018-07-11 07:34:00.286354 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5112 0x55a35634cf00 replica scrub(pg: 6.15,from:0'0,to:430'105892,epoch:494/492,start:6:a9df346a:::benchmark_data_magna108_35617_object689783:0,end:6:a9e059aa:::benchmark_data_magna108_35617_object854010:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -5112> 2018-07-11 07:34:00.286377 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5112 ==== replica scrub(pg: 6.15,from:0'0,to:430'105892,epoch:494/492,start:6:a9df346a:::benchmark_data_magna108_35617_object689783:0,end:6:a9e059aa:::benchmark_data_magna108_35617_object854010:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (3505402881 0 0) 0x55a35634cf00 con 0x55a317019000 > -5111> 2018-07-11 07:34:00.327486 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15255 conn(0x55a311b64000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1866 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:34:01.541286) v4 > -5110> 2018-07-11 07:34:00.327510 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.0 10.8.128.113:0/15255 1866 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:01.541286) v4 ==== 2004+0+0 (3065341442 0 0) 0x55a34de78600 con 0x55a311b64000 > -5109> 2018-07-11 07:34:00.327505 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15255 conn(0x55a311b65800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1866 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:34:01.541286) v4 > -5108> 2018-07-11 07:34:00.327521 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:01.541286) v4 -- 0x55a34de6f800 con 0 > -5107> 2018-07-11 07:34:00.327522 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.0 10.8.128.113:0/15255 1866 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:01.541286) v4 ==== 2004+0+0 (3065341442 0 0) 0x55a324474c00 con 0x55a311b65800 > -5106> 2018-07-11 07:34:00.327552 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:01.541286) v4 -- 0x55a30fa76a00 con 0 > -5105> 2018-07-11 07:34:00.475815 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12650 conn(0x55a311b67000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1856 0x55a32eb14a00 osd_ping(ping e494 stamp 2018-07-11 07:34:00.511539) v4 > -5104> 2018-07-11 07:34:00.475836 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.3 10.8.128.106:0/12650 1856 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:00.511539) v4 ==== 2004+0+0 (935884817 0 0) 0x55a32eb14a00 con 0x55a311b67000 > -5103> 2018-07-11 07:34:00.475845 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:00.511539) v4 -- 0x55a324af4c00 con 0 > -5102> 2018-07-11 07:34:00.475836 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12650 conn(0x55a311b68800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1856 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:34:00.511539) v4 > -5101> 2018-07-11 07:34:00.475865 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.3 10.8.128.106:0/12650 1856 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:00.511539) v4 ==== 2004+0+0 (935884817 0 0) 0x55a34de6f000 con 0x55a311b68800 > -5100> 2018-07-11 07:34:00.475877 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:00.511539) v4 -- 0x55a32162ae00 con 0 > -5099> 2018-07-11 07:34:00.664028 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12651 conn(0x55a32169f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1842 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:34:00.699677) v4 > -5098> 2018-07-11 07:34:00.664032 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12651 conn(0x55a3216a0800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1842 0x55a34de62000 osd_ping(ping e494 stamp 2018-07-11 07:34:00.699677) v4 > -5097> 2018-07-11 07:34:00.664057 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.2 10.8.128.106:0/12651 1842 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:00.699677) v4 ==== 2004+0+0 (1972913838 0 0) 0x55a332e79a00 con 0x55a32169f000 > -5096> 2018-07-11 07:34:00.664060 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.2 10.8.128.106:0/12651 1842 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:00.699677) v4 ==== 2004+0+0 (1972913838 0 0) 0x55a34de62000 con 0x55a3216a0800 > -5095> 2018-07-11 07:34:00.664068 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:00.699677) v4 -- 0x55a34de84a00 con 0 > -5094> 2018-07-11 07:34:00.664092 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:00.699677) v4 -- 0x55a32e331400 con 0 > -5093> 2018-07-11 07:34:01.117984 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15263 conn(0x55a321664000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1839 0x55a32eb14a00 osd_ping(ping e494 stamp 2018-07-11 07:34:02.331643) v4 > -5092> 2018-07-11 07:34:01.118010 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.5 10.8.128.113:0/15263 1839 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:02.331643) v4 ==== 2004+0+0 (2998044582 0 0) 0x55a32eb14a00 con 0x55a321664000 > -5091> 2018-07-11 07:34:01.118026 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:02.331643) v4 -- 0x55a31334de00 con 0 > -5090> 2018-07-11 07:34:01.118097 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15263 conn(0x55a321667000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1839 0x55a34de62000 osd_ping(ping e494 stamp 2018-07-11 07:34:02.331643) v4 > -5089> 2018-07-11 07:34:01.118112 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.5 10.8.128.113:0/15263 1839 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:02.331643) v4 ==== 2004+0+0 (2998044582 0 0) 0x55a34de62000 con 0x55a321667000 > -5088> 2018-07-11 07:34:01.118122 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:02.331643) v4 -- 0x55a32e330a00 con 0 > -5087> 2018-07-11 07:34:01.427842 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15255 conn(0x55a311b64000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1867 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:34:02.641686) v4 > -5086> 2018-07-11 07:34:01.427848 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15255 conn(0x55a311b65800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1867 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:34:02.641686) v4 > -5085> 2018-07-11 07:34:01.427867 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.0 10.8.128.113:0/15255 1867 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:02.641686) v4 ==== 2004+0+0 (1670411786 0 0) 0x55a324474c00 con 0x55a311b65800 > -5084> 2018-07-11 07:34:01.427867 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.0 10.8.128.113:0/15255 1867 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:02.641686) v4 ==== 2004+0+0 (1670411786 0 0) 0x55a34de78600 con 0x55a311b64000 > -5083> 2018-07-11 07:34:01.427877 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:02.641686) v4 -- 0x55a30fa76800 con 0 > -5082> 2018-07-11 07:34:01.427900 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:02.641686) v4 -- 0x55a33234fa00 con 0 > -5081> 2018-07-11 07:34:01.778421 7f355d88e700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.107:0/12617 conn(0x55a32447f800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1816 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:34:01.778056) v4 > -5080> 2018-07-11 07:34:01.778443 7f355d88e700 1 -- 10.8.128.107:6803/12665 <== osd.9 10.8.128.107:0/12617 1816 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:01.778056) v4 ==== 2004+0+0 (132140885 0 0) 0x55a34de6f000 con 0x55a32447f800 > -5079> 2018-07-11 07:34:01.778453 7f355d88e700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:01.778056) v4 -- 0x55a33234fa00 con 0 > -5078> 2018-07-11 07:34:01.778443 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.107:0/12617 conn(0x55a32447e000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1816 0x55a32eb14a00 osd_ping(ping e494 stamp 2018-07-11 07:34:01.778056) v4 > -5077> 2018-07-11 07:34:01.778464 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.9 10.8.128.107:0/12617 1816 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:01.778056) v4 ==== 2004+0+0 (132140885 0 0) 0x55a32eb14a00 con 0x55a32447e000 > -5076> 2018-07-11 07:34:01.778479 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:01.778056) v4 -- 0x55a32e330a00 con 0 > -5075> 2018-07-11 07:34:01.790529 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 59] Generated table #36581: 4957165 keys, 80348254 bytes > -5074> 2018-07-11 07:34:01.790585 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294441790568, "cf_name": "default", "job": 59, "event": "table_file_creation", "file_number": 36581, "file_size": 80348254, "table_properties": {"data_size": 67109355, "index_size": 689208, "filter_size": 12548701, "raw_key_size": 143206994, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957165, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957165", "kMergeOperands": "0"}} > -5073> 2018-07-11 07:34:01.855925 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15919 conn(0x55a317074800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1845 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:34:02.479714) v4 > -5072> 2018-07-11 07:34:01.855952 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.11 10.8.128.114:0/15919 1845 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:02.479714) v4 ==== 2004+0+0 (214593870 0 0) 0x55a332e79a00 con 0x55a317074800 > -5071> 2018-07-11 07:34:01.855950 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15919 conn(0x55a3170b4000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1845 0x55a34de62000 osd_ping(ping e494 stamp 2018-07-11 07:34:02.479714) v4 > -5070> 2018-07-11 07:34:01.855968 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.11 10.8.128.114:0/15919 1845 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:02.479714) v4 ==== 2004+0+0 (214593870 0 0) 0x55a34de62000 con 0x55a3170b4000 > -5069> 2018-07-11 07:34:01.855968 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:02.479714) v4 -- 0x55a30fa76800 con 0 > -5068> 2018-07-11 07:34:01.856002 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:02.479714) v4 -- 0x55a32e330a00 con 0 > -5067> 2018-07-11 07:34:01.966798 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15885 conn(0x55a311b76800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1863 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:34:02.590632) v4 > -5066> 2018-07-11 07:34:01.966795 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15885 conn(0x55a3170b5800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1863 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:34:02.590632) v4 > -5065> 2018-07-11 07:34:01.966823 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.7 10.8.128.114:0/15885 1863 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:02.590632) v4 ==== 2004+0+0 (2794965598 0 0) 0x55a34de78600 con 0x55a311b76800 > -5064> 2018-07-11 07:34:01.966826 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.7 10.8.128.114:0/15885 1863 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:02.590632) v4 ==== 2004+0+0 (2794965598 0 0) 0x55a324474c00 con 0x55a3170b5800 > -5063> 2018-07-11 07:34:01.966843 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:02.590632) v4 -- 0x55a33234fa00 con 0 > -5062> 2018-07-11 07:34:01.966871 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:02.590632) v4 -- 0x55a30fa76800 con 0 > -5061> 2018-07-11 07:34:02.007665 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a3122078c0 con 0 > -5060> 2018-07-11 07:34:02.019838 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5113 0x55a35634d180 replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:494/492,start:6:a9e059aa:::benchmark_data_magna108_35617_object854010:0,end:6:a9e14e95:::benchmark_data_magna108_35617_object99724:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -5059> 2018-07-11 07:34:02.019868 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5113 ==== replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:494/492,start:6:a9e059aa:::benchmark_data_magna108_35617_object854010:0,end:6:a9e14e95:::benchmark_data_magna108_35617_object99724:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 217+0+0 (3166421078 0 0) 0x55a35634d180 con 0x55a317019000 > -5058> 2018-07-11 07:34:02.176259 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12650 conn(0x55a311b67000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1857 0x55a32eb14a00 osd_ping(ping e494 stamp 2018-07-11 07:34:02.211965) v4 > -5057> 2018-07-11 07:34:02.176286 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.3 10.8.128.106:0/12650 1857 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:02.211965) v4 ==== 2004+0+0 (3956099244 0 0) 0x55a32eb14a00 con 0x55a311b67000 > -5056> 2018-07-11 07:34:02.176304 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:02.211965) v4 -- 0x55a32e330600 con 0 > -5055> 2018-07-11 07:34:02.176322 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12650 conn(0x55a311b68800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1857 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:34:02.211965) v4 > -5054> 2018-07-11 07:34:02.176343 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.3 10.8.128.106:0/12650 1857 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:02.211965) v4 ==== 2004+0+0 (3956099244 0 0) 0x55a34de6f000 con 0x55a311b68800 > -5053> 2018-07-11 07:34:02.176360 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:02.211965) v4 -- 0x55a30fa76e00 con 0 > -5052> 2018-07-11 07:34:02.227584 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12677 conn(0x55a32169d800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1862 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:34:02.263312) v4 > -5051> 2018-07-11 07:34:02.227605 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.1 10.8.128.106:0/12677 1862 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:02.263312) v4 ==== 2004+0+0 (2999621091 0 0) 0x55a34de78600 con 0x55a32169d800 > -5050> 2018-07-11 07:34:02.227617 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:02.263312) v4 -- 0x55a348fbd800 con 0 > -5049> 2018-07-11 07:34:02.227659 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12677 conn(0x55a32169c000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1862 0x55a32e330a00 osd_ping(ping e494 stamp 2018-07-11 07:34:02.263312) v4 > -5048> 2018-07-11 07:34:02.227685 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.1 10.8.128.106:0/12677 1862 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:02.263312) v4 ==== 2004+0+0 (2999621091 0 0) 0x55a32e330a00 con 0x55a32169c000 > -5047> 2018-07-11 07:34:02.227698 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:02.263312) v4 -- 0x55a331c66e00 con 0 > -5046> 2018-07-11 07:34:02.253996 7f35488a3700 1 -- 10.8.128.107:6800/12665 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55a3170118c0 con 0 > -5045> 2018-07-11 07:34:02.254076 7f35488a3700 1 -- 10.8.128.107:6800/12665 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55a32a5c0000 con 0 > -5044> 2018-07-11 07:34:02.528920 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15287 conn(0x55a311b72000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1838 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:34:03.742680) v4 > -5043> 2018-07-11 07:34:02.528941 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.4 10.8.128.113:0/15287 1838 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:03.742680) v4 ==== 2004+0+0 (4177831820 0 0) 0x55a332e79a00 con 0x55a311b72000 > -5042> 2018-07-11 07:34:02.528951 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:03.742680) v4 -- 0x55a348fbd000 con 0 > -5041> 2018-07-11 07:34:02.528941 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15287 conn(0x55a311b75000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1838 0x55a32eb14a00 osd_ping(ping e494 stamp 2018-07-11 07:34:03.742680) v4 > -5040> 2018-07-11 07:34:02.528967 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.4 10.8.128.113:0/15287 1838 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:03.742680) v4 ==== 2004+0+0 (4177831820 0 0) 0x55a32eb14a00 con 0x55a311b75000 > -5039> 2018-07-11 07:34:02.528979 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:03.742680) v4 -- 0x55a31334c000 con 0 > -5038> 2018-07-11 07:34:02.676802 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12650 conn(0x55a311b68800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1858 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:34:02.712532) v4 > -5037> 2018-07-11 07:34:02.676823 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.3 10.8.128.106:0/12650 1858 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:02.712532) v4 ==== 2004+0+0 (1233402502 0 0) 0x55a34de6f000 con 0x55a311b68800 > -5036> 2018-07-11 07:34:02.676834 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:02.712532) v4 -- 0x55a348fbd800 con 0 > -5035> 2018-07-11 07:34:02.676940 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12650 conn(0x55a311b67000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1858 0x55a32e330a00 osd_ping(ping e494 stamp 2018-07-11 07:34:02.712532) v4 > -5034> 2018-07-11 07:34:02.676956 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.3 10.8.128.106:0/12650 1858 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:02.712532) v4 ==== 2004+0+0 (1233402502 0 0) 0x55a32e330a00 con 0x55a311b67000 > -5033> 2018-07-11 07:34:02.676966 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:02.712532) v4 -- 0x55a31334c000 con 0 > -5032> 2018-07-11 07:34:02.778100 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a33bf5f680 con 0 > -5031> 2018-07-11 07:34:02.794829 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5114 0x55a35634d400 replica scrub(pg: 6.15,from:0'0,to:430'105919,epoch:494/492,start:6:a9e14e95:::benchmark_data_magna108_35617_object99724:0,end:6:a9e21c0f:::benchmark_data_magna108_35617_object534353:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -5030> 2018-07-11 07:34:02.794858 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5114 ==== replica scrub(pg: 6.15,from:0'0,to:430'105919,epoch:494/492,start:6:a9e14e95:::benchmark_data_magna108_35617_object99724:0,end:6:a9e21c0f:::benchmark_data_magna108_35617_object534353:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 217+0+0 (3344132489 0 0) 0x55a35634d400 con 0x55a317019000 > -5029> 2018-07-11 07:34:02.818332 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15263 conn(0x55a321664000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1840 0x55a32eb14a00 osd_ping(ping e494 stamp 2018-07-11 07:34:04.032179) v4 > -5028> 2018-07-11 07:34:02.818359 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.5 10.8.128.113:0/15263 1840 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:04.032179) v4 ==== 2004+0+0 (2703081416 0 0) 0x55a32eb14a00 con 0x55a321664000 > -5027> 2018-07-11 07:34:02.818373 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:04.032179) v4 -- 0x55a31334c000 con 0 > -5026> 2018-07-11 07:34:02.818456 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15263 conn(0x55a321667000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1840 0x55a32e330a00 osd_ping(ping e494 stamp 2018-07-11 07:34:04.032179) v4 > -5025> 2018-07-11 07:34:02.818473 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.5 10.8.128.113:0/15263 1840 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:04.032179) v4 ==== 2004+0+0 (2703081416 0 0) 0x55a32e330a00 con 0x55a321667000 > -5024> 2018-07-11 07:34:02.818483 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:04.032179) v4 -- 0x55a31334c000 con 0 > -5023> 2018-07-11 07:34:02.956316 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15919 conn(0x55a317074800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1846 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:34:03.580131) v4 > -5022> 2018-07-11 07:34:02.956342 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.11 10.8.128.114:0/15919 1846 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:03.580131) v4 ==== 2004+0+0 (2070484466 0 0) 0x55a324474c00 con 0x55a317074800 > -5021> 2018-07-11 07:34:02.956353 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:03.580131) v4 -- 0x55a30fa76200 con 0 > -5020> 2018-07-11 07:34:02.956414 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15919 conn(0x55a3170b4000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1846 0x55a32eb14a00 osd_ping(ping e494 stamp 2018-07-11 07:34:03.580131) v4 > -5019> 2018-07-11 07:34:02.956435 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.11 10.8.128.114:0/15919 1846 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:03.580131) v4 ==== 2004+0+0 (2070484466 0 0) 0x55a32eb14a00 con 0x55a3170b4000 > -5018> 2018-07-11 07:34:02.956446 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:03.580131) v4 -- 0x55a329323400 con 0 > -5017> 2018-07-11 07:34:02.968319 7f3543899700 5 osd.10 494 heartbeat: osd_stat(784 GB used, 147 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,1]) > -5016> 2018-07-11 07:34:02.968334 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6802/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:34:02.968332) v4 -- 0x55a332385600 con 0 > -5015> 2018-07-11 07:34:02.968359 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6803/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:34:02.968332) v4 -- 0x55a31ce7b200 con 0 > -5014> 2018-07-11 07:34:02.968375 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6806/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:34:02.968332) v4 -- 0x55a31ce7c200 con 0 > -5013> 2018-07-11 07:34:02.968389 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6807/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:34:02.968332) v4 -- 0x55a31ce7c800 con 0 > -5012> 2018-07-11 07:34:02.968408 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6808/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:34:02.968332) v4 -- 0x55a31ce7d200 con 0 > -5011> 2018-07-11 07:34:02.968431 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6810/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:34:02.968332) v4 -- 0x55a31ce7ce00 con 0 > -5010> 2018-07-11 07:34:02.968464 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6809/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:34:02.968332) v4 -- 0x55a31ce7cc00 con 0 > -5009> 2018-07-11 07:34:02.968480 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6811/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:34:02.968332) v4 -- 0x55a31ce7d400 con 0 > -5008> 2018-07-11 07:34:02.968505 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6810/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:34:02.968332) v4 -- 0x55a31ce7d000 con 0 > -5007> 2018-07-11 07:34:02.968528 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6811/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:34:02.968332) v4 -- 0x55a31ce7dc00 con 0 > -5006> 2018-07-11 07:34:02.968551 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6806/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:34:02.968332) v4 -- 0x55a332385e00 con 0 > -5005> 2018-07-11 07:34:02.968566 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6807/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:34:02.968332) v4 -- 0x55a332364c00 con 0 > -5004> 2018-07-11 07:34:02.968586 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6802/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:34:02.968332) v4 -- 0x55a32eb15600 con 0 > -5003> 2018-07-11 07:34:02.968610 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6803/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:34:02.968332) v4 -- 0x55a34de63600 con 0 > -5002> 2018-07-11 07:34:02.968626 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6806/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:34:02.968332) v4 -- 0x55a330935c00 con 0 > -5001> 2018-07-11 07:34:02.968647 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6807/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:34:02.968332) v4 -- 0x55a32eb15800 con 0 > -5000> 2018-07-11 07:34:02.968675 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6802/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:34:02.968332) v4 -- 0x55a332384c00 con 0 > -4999> 2018-07-11 07:34:02.968698 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6803/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:34:02.968332) v4 -- 0x55a332384000 con 0 > -4998> 2018-07-11 07:34:02.968720 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6806/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:34:02.968332) v4 -- 0x55a31ce7aa00 con 0 > -4997> 2018-07-11 07:34:02.968744 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6807/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:34:02.968332) v4 -- 0x55a32eb28e00 con 0 > -4996> 2018-07-11 07:34:02.968763 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6806/12677 conn(0x55a314e84000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1847 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:02.968332) v4 > -4995> 2018-07-11 07:34:02.968772 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6802/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:34:02.968332) v4 -- 0x55a332e98a00 con 0 > -4994> 2018-07-11 07:34:02.968786 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6806/12677 1847 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:02.968332) v4 ==== 2004+0+0 (3964419285 0 0) 0x55a34de78600 con 0x55a314e84000 > -4993> 2018-07-11 07:34:02.968784 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6802/15255 conn(0x55a314e1c000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1847 0x55a32eb14a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:02.968332) v4 > -4992> 2018-07-11 07:34:02.968801 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6802/15255 1847 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:02.968332) v4 ==== 2004+0+0 (3964419285 0 0) 0x55a32eb14a00 con 0x55a314e1c000 > -4991> 2018-07-11 07:34:02.968800 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6803/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:34:02.968332) v4 -- 0x55a332eac000 con 0 > -4990> 2018-07-11 07:34:02.968901 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6803/15255 conn(0x55a314e1d800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1847 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:02.968332) v4 > -4989> 2018-07-11 07:34:02.968923 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6803/15255 1847 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:02.968332) v4 ==== 2004+0+0 (3964419285 0 0) 0x55a324474c00 con 0x55a314e1d800 > -4988> 2018-07-11 07:34:02.968900 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6807/12677 conn(0x55a314e85800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1847 0x55a32eb14a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:02.968332) v4 > -4987> 2018-07-11 07:34:02.968935 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6807/12677 1847 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:02.968332) v4 ==== 2004+0+0 (3964419285 0 0) 0x55a32eb14a00 con 0x55a314e85800 > -4986> 2018-07-11 07:34:02.968939 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6809/12650 conn(0x55a314e1f000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1847 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:02.968332) v4 > -4985> 2018-07-11 07:34:02.968956 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6809/12650 1847 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:02.968332) v4 ==== 2004+0+0 (3964419285 0 0) 0x55a34de6f000 con 0x55a314e1f000 > -4984> 2018-07-11 07:34:02.968960 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6806/12617 conn(0x55a314e26000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1847 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:02.968332) v4 > -4983> 2018-07-11 07:34:02.968977 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6806/12617 1847 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:02.968332) v4 ==== 2004+0+0 (3964419285 0 0) 0x55a324474c00 con 0x55a314e26000 > -4982> 2018-07-11 07:34:02.969017 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6811/15287 conn(0x55a314e04800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1847 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:02.968332) v4 > -4981> 2018-07-11 07:34:02.969029 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6811/15287 1847 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:02.968332) v4 ==== 2004+0+0 (3964419285 0 0) 0x55a324474c00 con 0x55a314e04800 > -4980> 2018-07-11 07:34:02.968983 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6808/12651 conn(0x55a314e96000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1847 0x55a32eb14a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:02.968332) v4 > -4979> 2018-07-11 07:34:02.969035 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6807/12617 conn(0x55a314e27800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1847 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:02.968332) v4 > -4978> 2018-07-11 07:34:02.969054 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6808/12651 1847 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:02.968332) v4 ==== 2004+0+0 (3964419285 0 0) 0x55a32eb14a00 con 0x55a314e96000 > -4977> 2018-07-11 07:34:02.969058 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6807/12617 1847 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:02.968332) v4 ==== 2004+0+0 (3964419285 0 0) 0x55a34de6f000 con 0x55a314e27800 > -4976> 2018-07-11 07:34:02.969058 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6810/12651 conn(0x55a314e97800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1847 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:02.968332) v4 > -4975> 2018-07-11 07:34:02.969069 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6810/12651 1847 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:02.968332) v4 ==== 2004+0+0 (3964419285 0 0) 0x55a324474c00 con 0x55a314e97800 > -4974> 2018-07-11 07:34:02.969087 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6806/15263 conn(0x55a314e99000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1847 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:02.968332) v4 > -4973> 2018-07-11 07:34:02.969101 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6806/15263 1847 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:02.968332) v4 ==== 2004+0+0 (3964419285 0 0) 0x55a34de6f000 con 0x55a314e99000 > -4972> 2018-07-11 07:34:02.969105 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6806/15885 conn(0x55a314e00000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1847 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:02.968332) v4 > -4971> 2018-07-11 07:34:02.969117 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6806/15885 1847 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:02.968332) v4 ==== 2004+0+0 (3964419285 0 0) 0x55a324474c00 con 0x55a314e00000 > -4970> 2018-07-11 07:34:02.969119 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6811/12650 conn(0x55a314e20800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1847 0x55a32eb14a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:02.968332) v4 > -4969> 2018-07-11 07:34:02.969123 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6803/13323 conn(0x55a314e16800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1847 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:02.968332) v4 > -4968> 2018-07-11 07:34:02.969132 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6811/12650 1847 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:02.968332) v4 ==== 2004+0+0 (3964419285 0 0) 0x55a32eb14a00 con 0x55a314e20800 > -4967> 2018-07-11 07:34:02.969132 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6803/13323 1847 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:02.968332) v4 ==== 2004+0+0 (3964419285 0 0) 0x55a34de6f000 con 0x55a314e16800 > -4966> 2018-07-11 07:34:02.969146 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6802/13323 conn(0x55a314e15000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1847 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:02.968332) v4 > -4965> 2018-07-11 07:34:02.969156 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6802/13323 1847 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:02.968332) v4 ==== 2004+0+0 (3964419285 0 0) 0x55a324474c00 con 0x55a314e15000 > -4964> 2018-07-11 07:34:02.969166 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6810/15287 conn(0x55a314e03000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1847 0x55a32eb14a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:02.968332) v4 > -4963> 2018-07-11 07:34:02.969176 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6807/15885 conn(0x55a314e01800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1847 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:02.968332) v4 > -4962> 2018-07-11 07:34:02.969182 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6810/15287 1847 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:02.968332) v4 ==== 2004+0+0 (3964419285 0 0) 0x55a32eb14a00 con 0x55a314e03000 > -4961> 2018-07-11 07:34:02.969189 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6807/15885 1847 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:02.968332) v4 ==== 2004+0+0 (3964419285 0 0) 0x55a34de6f000 con 0x55a314e01800 > -4960> 2018-07-11 07:34:02.969186 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6802/15919 conn(0x55a314e87000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1847 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:02.968332) v4 > -4959> 2018-07-11 07:34:02.969205 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6802/15919 1847 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:02.968332) v4 ==== 2004+0+0 (3964419285 0 0) 0x55a324474c00 con 0x55a314e87000 > -4958> 2018-07-11 07:34:02.969211 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6807/15263 conn(0x55a314e9a800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1847 0x55a32eb14a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:02.968332) v4 > -4957> 2018-07-11 07:34:02.969222 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6802/11967 conn(0x55a314e12000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1847 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:02.968332) v4 > -4956> 2018-07-11 07:34:02.969227 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6807/15263 1847 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:02.968332) v4 ==== 2004+0+0 (3964419285 0 0) 0x55a32eb14a00 con 0x55a314e9a800 > -4955> 2018-07-11 07:34:02.969234 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6802/11967 1847 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:02.968332) v4 ==== 2004+0+0 (3964419285 0 0) 0x55a34de6f000 con 0x55a314e12000 > -4954> 2018-07-11 07:34:02.969269 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6803/11967 conn(0x55a314e13800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1847 0x55a32eb14a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:02.968332) v4 > -4953> 2018-07-11 07:34:02.969288 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6803/11967 1847 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:02.968332) v4 ==== 2004+0+0 (3964419285 0 0) 0x55a32eb14a00 con 0x55a314e13800 > -4952> 2018-07-11 07:34:02.969365 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6803/15919 conn(0x55a314e88800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1847 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:02.968332) v4 > -4951> 2018-07-11 07:34:02.969390 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6803/15919 1847 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:02.968332) v4 ==== 2004+0+0 (3964419285 0 0) 0x55a34de6f000 con 0x55a314e88800 > -4950> 2018-07-11 07:34:03.250225 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b7000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1792 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:34:03.000899) v4 > -4949> 2018-07-11 07:34:03.250253 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.8 10.8.128.108:0/11967 1792 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:03.000899) v4 ==== 2004+0+0 (2586556927 0 0) 0x55a34de6f000 con 0x55a3170b7000 > -4948> 2018-07-11 07:34:03.250265 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:03.000899) v4 -- 0x55a31ce7c200 con 0 > -4947> 2018-07-11 07:34:03.250285 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b8800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1792 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:34:03.000899) v4 > -4946> 2018-07-11 07:34:03.250306 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.8 10.8.128.108:0/11967 1792 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:03.000899) v4 ==== 2004+0+0 (2586556927 0 0) 0x55a324474c00 con 0x55a3170b8800 > -4945> 2018-07-11 07:34:03.250321 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:03.000899) v4 -- 0x55a31ce7ce00 con 0 > -4944> 2018-07-11 07:34:03.326670 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a31f9d2000 con 0 > -4943> 2018-07-11 07:34:03.352835 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5115 0x55a35634d680 replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:494/492,start:6:a9e21c0f:::benchmark_data_magna108_35617_object534353:0,end:6:a9e29a8f:::benchmark_data_magna108.ceph.redhat.com_2478_object282721:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -4942> 2018-07-11 07:34:03.352866 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5115 ==== replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:494/492,start:6:a9e21c0f:::benchmark_data_magna108_35617_object534353:0,end:6:a9e29a8f:::benchmark_data_magna108.ceph.redhat.com_2478_object282721:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (1276989721 0 0) 0x55a35634d680 con 0x55a317019000 > -4941> 2018-07-11 07:34:03.888942 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a31133d440 con 0 > -4940> 2018-07-11 07:34:03.903457 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5116 0x55a35634d900 replica scrub(pg: 6.15,from:0'0,to:430'106614,epoch:494/492,start:6:a9e29a8f:::benchmark_data_magna108.ceph.redhat.com_2478_object282721:0,end:6:a9e3691f:::benchmark_data_magna108_35617_object639112:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -4939> 2018-07-11 07:34:03.903487 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5116 ==== replica scrub(pg: 6.15,from:0'0,to:430'106614,epoch:494/492,start:6:a9e29a8f:::benchmark_data_magna108.ceph.redhat.com_2478_object282721:0,end:6:a9e3691f:::benchmark_data_magna108_35617_object639112:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (3290942675 0 0) 0x55a35634d900 con 0x55a317019000 > -4938> 2018-07-11 07:34:03.928068 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12677 conn(0x55a32169d800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1863 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:34:03.963814) v4 > -4937> 2018-07-11 07:34:03.928093 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.1 10.8.128.106:0/12677 1863 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:03.963814) v4 ==== 2004+0+0 (1109666895 0 0) 0x55a34de78600 con 0x55a32169d800 > -4936> 2018-07-11 07:34:03.928104 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:03.963814) v4 -- 0x55a31ce7c200 con 0 > -4935> 2018-07-11 07:34:03.928109 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12677 conn(0x55a32169c000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1863 0x55a32eb14a00 osd_ping(ping e494 stamp 2018-07-11 07:34:03.963814) v4 > -4934> 2018-07-11 07:34:03.928126 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.1 10.8.128.106:0/12677 1863 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:03.963814) v4 ==== 2004+0+0 (1109666895 0 0) 0x55a32eb14a00 con 0x55a32169c000 > -4933> 2018-07-11 07:34:03.928153 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:03.963814) v4 -- 0x55a31ce7c800 con 0 > -4932> 2018-07-11 07:34:04.164391 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12651 conn(0x55a3216a0800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1843 0x55a32e330a00 osd_ping(ping e494 stamp 2018-07-11 07:34:04.200154) v4 > -4931> 2018-07-11 07:34:04.164391 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12651 conn(0x55a32169f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1843 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:34:04.200154) v4 > -4930> 2018-07-11 07:34:04.164414 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.2 10.8.128.106:0/12651 1843 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:04.200154) v4 ==== 2004+0+0 (2022197751 0 0) 0x55a32e330a00 con 0x55a3216a0800 > -4929> 2018-07-11 07:34:04.164418 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.2 10.8.128.106:0/12651 1843 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:04.200154) v4 ==== 2004+0+0 (2022197751 0 0) 0x55a332e79a00 con 0x55a32169f000 > -4928> 2018-07-11 07:34:04.164429 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:04.200154) v4 -- 0x55a31ce7c800 con 0 > -4927> 2018-07-11 07:34:04.164452 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:04.200154) v4 -- 0x55a31ce7ce00 con 0 > -4926> 2018-07-11 07:34:04.532112 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a31f9d3440 con 0 > -4925> 2018-07-11 07:34:04.549226 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.115:0/13323 conn(0x55a321665800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1853 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:34:04.886687) v4 > -4924> 2018-07-11 07:34:04.549267 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.6 10.8.128.115:0/13323 1853 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:04.886687) v4 ==== 2004+0+0 (206102936 0 0) 0x55a324474c00 con 0x55a321665800 > -4923> 2018-07-11 07:34:04.549284 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:04.886687) v4 -- 0x55a31ce7b200 con 0 > -4922> 2018-07-11 07:34:04.549347 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.115:0/13323 conn(0x55a321668800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1853 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:34:04.886687) v4 > -4921> 2018-07-11 07:34:04.549362 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.6 10.8.128.115:0/13323 1853 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:04.886687) v4 ==== 2004+0+0 (206102936 0 0) 0x55a332e79a00 con 0x55a321668800 > -4920> 2018-07-11 07:34:04.549371 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:04.886687) v4 -- 0x55a30fa76200 con 0 > -4919> 2018-07-11 07:34:04.562055 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5117 0x55a35634db80 replica scrub(pg: 6.15,from:0'0,to:430'106670,epoch:494/492,start:6:a9e3691f:::benchmark_data_magna108_35617_object639112:0,end:6:a9e4775c:::benchmark_data_magna108_35617_object54243:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -4918> 2018-07-11 07:34:04.562085 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5117 ==== replica scrub(pg: 6.15,from:0'0,to:430'106670,epoch:494/492,start:6:a9e3691f:::benchmark_data_magna108_35617_object639112:0,end:6:a9e4775c:::benchmark_data_magna108_35617_object54243:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 217+0+0 (1181817397 0 0) 0x55a35634db80 con 0x55a317019000 > -4917> 2018-07-11 07:34:04.867219 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15885 conn(0x55a3170b5800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1864 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:34:05.491039) v4 > -4916> 2018-07-11 07:34:04.867241 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.7 10.8.128.114:0/15885 1864 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:05.491039) v4 ==== 2004+0+0 (3715710316 0 0) 0x55a324474c00 con 0x55a3170b5800 > -4915> 2018-07-11 07:34:04.867231 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15885 conn(0x55a311b76800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1864 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:34:05.491039) v4 > -4914> 2018-07-11 07:34:04.867261 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:05.491039) v4 -- 0x55a30fa76200 con 0 > -4913> 2018-07-11 07:34:04.867263 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.7 10.8.128.114:0/15885 1864 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:05.491039) v4 ==== 2004+0+0 (3715710316 0 0) 0x55a34de6f000 con 0x55a311b76800 > -4912> 2018-07-11 07:34:04.867287 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:05.491039) v4 -- 0x55a31ce7c200 con 0 > -4911> 2018-07-11 07:34:05.049686 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.115:0/13323 conn(0x55a321665800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1854 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:34:05.387163) v4 > -4910> 2018-07-11 07:34:05.049709 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.6 10.8.128.115:0/13323 1854 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:05.387163) v4 ==== 2004+0+0 (3305154222 0 0) 0x55a332e79a00 con 0x55a321665800 > -4909> 2018-07-11 07:34:05.049719 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:05.387163) v4 -- 0x55a30fa76200 con 0 > -4908> 2018-07-11 07:34:05.049759 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.115:0/13323 conn(0x55a321668800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1854 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:34:05.387163) v4 > -4907> 2018-07-11 07:34:05.049774 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.6 10.8.128.115:0/13323 1854 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:05.387163) v4 ==== 2004+0+0 (3305154222 0 0) 0x55a332e79a00 con 0x55a321668800 > -4906> 2018-07-11 07:34:05.049784 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:05.387163) v4 -- 0x55a3203f5200 con 0 > -4905> 2018-07-11 07:34:05.118990 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15263 conn(0x55a321664000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1841 0x55a32eb14a00 osd_ping(ping e494 stamp 2018-07-11 07:34:06.332642) v4 > -4904> 2018-07-11 07:34:05.119011 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.5 10.8.128.113:0/15263 1841 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:06.332642) v4 ==== 2004+0+0 (1766935132 0 0) 0x55a32eb14a00 con 0x55a321664000 > -4903> 2018-07-11 07:34:05.119021 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:06.332642) v4 -- 0x55a329323400 con 0 > -4902> 2018-07-11 07:34:05.119082 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15263 conn(0x55a321667000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1841 0x55a31ce7c800 osd_ping(ping e494 stamp 2018-07-11 07:34:06.332642) v4 > -4901> 2018-07-11 07:34:05.119097 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.5 10.8.128.113:0/15263 1841 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:06.332642) v4 ==== 2004+0+0 (1766935132 0 0) 0x55a31ce7c800 con 0x55a321667000 > -4900> 2018-07-11 07:34:05.119107 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:06.332642) v4 -- 0x55a343482400 con 0 > -4899> 2018-07-11 07:34:05.268960 7f3543899700 5 osd.10 494 heartbeat: osd_stat(784 GB used, 147 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,0,0,1]) > -4898> 2018-07-11 07:34:05.268975 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6802/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:34:05.268973) v4 -- 0x55a31707c400 con 0 > -4897> 2018-07-11 07:34:05.269012 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6803/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:34:05.268973) v4 -- 0x55a332faf000 con 0 > -4896> 2018-07-11 07:34:05.269029 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6806/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:34:05.268973) v4 -- 0x55a33235a800 con 0 > -4895> 2018-07-11 07:34:05.269043 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6807/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:34:05.268973) v4 -- 0x55a34de6e600 con 0 > -4894> 2018-07-11 07:34:05.269056 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6808/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:34:05.268973) v4 -- 0x55a31ce7a600 con 0 > -4893> 2018-07-11 07:34:05.269066 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6810/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:34:05.268973) v4 -- 0x55a32eb43a00 con 0 > -4892> 2018-07-11 07:34:05.269079 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6809/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:34:05.268973) v4 -- 0x55a34dea8200 con 0 > -4891> 2018-07-11 07:34:05.269095 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6811/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:34:05.268973) v4 -- 0x55a32447b200 con 0 > -4890> 2018-07-11 07:34:05.269117 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6810/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:34:05.268973) v4 -- 0x55a34dea9200 con 0 > -4889> 2018-07-11 07:34:05.269131 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6811/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:34:05.268973) v4 -- 0x55a34de5ae00 con 0 > -4888> 2018-07-11 07:34:05.269144 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6806/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:34:05.268973) v4 -- 0x55a34de5b800 con 0 > -4887> 2018-07-11 07:34:05.269162 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6807/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:34:05.268973) v4 -- 0x55a332e84000 con 0 > -4886> 2018-07-11 07:34:05.269176 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6802/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:34:05.268973) v4 -- 0x55a332e79e00 con 0 > -4885> 2018-07-11 07:34:05.269189 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6803/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:34:05.268973) v4 -- 0x55a34dea8600 con 0 > -4884> 2018-07-11 07:34:05.269206 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6806/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:34:05.268973) v4 -- 0x55a32eb14200 con 0 > -4883> 2018-07-11 07:34:05.269219 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6807/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:34:05.268973) v4 -- 0x55a34de5be00 con 0 > -4882> 2018-07-11 07:34:05.269236 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6802/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:34:05.268973) v4 -- 0x55a313704a00 con 0 > -4881> 2018-07-11 07:34:05.269263 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6803/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:34:05.268973) v4 -- 0x55a332365e00 con 0 > -4880> 2018-07-11 07:34:05.269277 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6806/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:34:05.268973) v4 -- 0x55a332355800 con 0 > -4879> 2018-07-11 07:34:05.269290 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6807/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:34:05.268973) v4 -- 0x55a32eb15a00 con 0 > -4878> 2018-07-11 07:34:05.269306 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6802/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:34:05.268973) v4 -- 0x55a32eb28800 con 0 > -4877> 2018-07-11 07:34:05.269321 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6803/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:34:05.268973) v4 -- 0x55a332365c00 con 0 > -4876> 2018-07-11 07:34:05.269469 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6808/12651 conn(0x55a314e96000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1848 0x55a32eb14a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:05.268973) v4 > -4875> 2018-07-11 07:34:05.269472 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6807/12617 conn(0x55a314e27800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1848 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:05.268973) v4 > -4874> 2018-07-11 07:34:05.269492 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6803/15255 conn(0x55a314e1d800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1848 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:05.268973) v4 > -4873> 2018-07-11 07:34:05.269491 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6808/12651 1848 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:05.268973) v4 ==== 2004+0+0 (96412886 0 0) 0x55a32eb14a00 con 0x55a314e96000 > -4872> 2018-07-11 07:34:05.269508 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6807/12617 1848 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:05.268973) v4 ==== 2004+0+0 (96412886 0 0) 0x55a34de78600 con 0x55a314e27800 > -4871> 2018-07-11 07:34:05.269512 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6803/15255 1848 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:05.268973) v4 ==== 2004+0+0 (96412886 0 0) 0x55a332e79a00 con 0x55a314e1d800 > -4870> 2018-07-11 07:34:05.269537 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6802/15255 conn(0x55a314e1c000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1848 0x55a32eb14a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:05.268973) v4 > -4869> 2018-07-11 07:34:05.269543 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6806/12617 conn(0x55a314e26000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1848 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:05.268973) v4 > -4868> 2018-07-11 07:34:05.269550 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6802/15255 1848 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:05.268973) v4 ==== 2004+0+0 (96412886 0 0) 0x55a32eb14a00 con 0x55a314e1c000 > -4867> 2018-07-11 07:34:05.269552 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6806/12617 1848 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:05.268973) v4 ==== 2004+0+0 (96412886 0 0) 0x55a332e79a00 con 0x55a314e26000 > -4866> 2018-07-11 07:34:05.269559 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6806/12677 conn(0x55a314e84000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1848 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:05.268973) v4 > -4865> 2018-07-11 07:34:05.269580 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6807/12677 conn(0x55a314e85800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1848 0x55a32eb14a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:05.268973) v4 > -4864> 2018-07-11 07:34:05.269609 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6807/12677 1848 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:05.268973) v4 ==== 2004+0+0 (96412886 0 0) 0x55a32eb14a00 con 0x55a314e85800 > -4863> 2018-07-11 07:34:05.269585 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6811/15287 conn(0x55a314e04800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1848 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:05.268973) v4 > -4862> 2018-07-11 07:34:05.269573 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6806/12677 1848 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:05.268973) v4 ==== 2004+0+0 (96412886 0 0) 0x55a34de78600 con 0x55a314e84000 > -4861> 2018-07-11 07:34:05.269624 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6811/15287 1848 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:05.268973) v4 ==== 2004+0+0 (96412886 0 0) 0x55a332e79a00 con 0x55a314e04800 > -4860> 2018-07-11 07:34:05.269655 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6810/15287 conn(0x55a314e03000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1848 0x55a32eb14a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:05.268973) v4 > -4859> 2018-07-11 07:34:05.269663 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6809/12650 conn(0x55a314e1f000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1848 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:05.268973) v4 > -4858> 2018-07-11 07:34:05.269669 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6810/15287 1848 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:05.268973) v4 ==== 2004+0+0 (96412886 0 0) 0x55a32eb14a00 con 0x55a314e03000 > -4857> 2018-07-11 07:34:05.269671 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6810/12651 conn(0x55a314e97800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1848 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:05.268973) v4 > -4856> 2018-07-11 07:34:05.269679 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6809/12650 1848 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:05.268973) v4 ==== 2004+0+0 (96412886 0 0) 0x55a34de78600 con 0x55a314e1f000 > -4855> 2018-07-11 07:34:05.269709 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6811/12650 conn(0x55a314e20800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1848 0x55a32eb14a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:05.268973) v4 > -4854> 2018-07-11 07:34:05.269716 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6806/15263 conn(0x55a314e99000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1848 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:05.268973) v4 > -4853> 2018-07-11 07:34:05.269722 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6811/12650 1848 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:05.268973) v4 ==== 2004+0+0 (96412886 0 0) 0x55a32eb14a00 con 0x55a314e20800 > -4852> 2018-07-11 07:34:05.269685 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6810/12651 1848 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:05.268973) v4 ==== 2004+0+0 (96412886 0 0) 0x55a332e79a00 con 0x55a314e97800 > -4851> 2018-07-11 07:34:05.269729 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6806/15263 1848 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:05.268973) v4 ==== 2004+0+0 (96412886 0 0) 0x55a34de78600 con 0x55a314e99000 > -4850> 2018-07-11 07:34:05.269755 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6807/15263 conn(0x55a314e9a800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1848 0x55a32eb14a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:05.268973) v4 > -4849> 2018-07-11 07:34:05.269768 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6807/15263 1848 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:05.268973) v4 ==== 2004+0+0 (96412886 0 0) 0x55a32eb14a00 con 0x55a314e9a800 > -4848> 2018-07-11 07:34:05.269763 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6802/13323 conn(0x55a314e15000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1848 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:05.268973) v4 > -4847> 2018-07-11 07:34:05.269772 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6803/13323 conn(0x55a314e16800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1848 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:05.268973) v4 > -4846> 2018-07-11 07:34:05.269783 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6803/13323 1848 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:05.268973) v4 ==== 2004+0+0 (96412886 0 0) 0x55a34de78600 con 0x55a314e16800 > -4845> 2018-07-11 07:34:05.269783 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6802/13323 1848 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:05.268973) v4 ==== 2004+0+0 (96412886 0 0) 0x55a332e79a00 con 0x55a314e15000 > -4844> 2018-07-11 07:34:05.269810 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6803/11967 conn(0x55a314e13800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1848 0x55a32eb14a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:05.268973) v4 > -4843> 2018-07-11 07:34:05.269813 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6803/15919 conn(0x55a314e88800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1848 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:05.268973) v4 > -4842> 2018-07-11 07:34:05.269820 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6803/11967 1848 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:05.268973) v4 ==== 2004+0+0 (96412886 0 0) 0x55a32eb14a00 con 0x55a314e13800 > -4841> 2018-07-11 07:34:05.269823 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6803/15919 1848 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:05.268973) v4 ==== 2004+0+0 (96412886 0 0) 0x55a34de78600 con 0x55a314e88800 > -4840> 2018-07-11 07:34:05.269846 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6806/15885 conn(0x55a314e00000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1848 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:05.268973) v4 > -4839> 2018-07-11 07:34:05.269859 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6807/15885 conn(0x55a314e01800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1848 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:05.268973) v4 > -4838> 2018-07-11 07:34:05.269864 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6806/15885 1848 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:05.268973) v4 ==== 2004+0+0 (96412886 0 0) 0x55a332e79a00 con 0x55a314e00000 > -4837> 2018-07-11 07:34:05.269874 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6807/15885 1848 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:05.268973) v4 ==== 2004+0+0 (96412886 0 0) 0x55a34de78600 con 0x55a314e01800 > -4836> 2018-07-11 07:34:05.269900 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6802/15919 conn(0x55a314e87000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1848 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:05.268973) v4 > -4835> 2018-07-11 07:34:05.269905 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6802/11967 conn(0x55a314e12000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1848 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:05.268973) v4 > -4834> 2018-07-11 07:34:05.269913 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6802/15919 1848 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:05.268973) v4 ==== 2004+0+0 (96412886 0 0) 0x55a332e79a00 con 0x55a314e87000 > -4833> 2018-07-11 07:34:05.269915 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6802/11967 1848 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:05.268973) v4 ==== 2004+0+0 (96412886 0 0) 0x55a34de78600 con 0x55a314e12000 > -4832> 2018-07-11 07:34:05.278817 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.107:0/12617 conn(0x55a32447e000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1817 0x55a32eb14a00 osd_ping(ping e494 stamp 2018-07-11 07:34:05.278518) v4 > -4831> 2018-07-11 07:34:05.278832 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.9 10.8.128.107:0/12617 1817 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:05.278518) v4 ==== 2004+0+0 (2521366258 0 0) 0x55a32eb14a00 con 0x55a32447e000 > -4830> 2018-07-11 07:34:05.278833 7f355d88e700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.107:0/12617 conn(0x55a32447f800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1817 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:34:05.278518) v4 > -4829> 2018-07-11 07:34:05.278842 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:05.278518) v4 -- 0x55a343482400 con 0 > -4828> 2018-07-11 07:34:05.278849 7f355d88e700 1 -- 10.8.128.107:6803/12665 <== osd.9 10.8.128.107:0/12617 1817 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:05.278518) v4 ==== 2004+0+0 (2521366258 0 0) 0x55a34de78600 con 0x55a32447f800 > -4827> 2018-07-11 07:34:05.278878 7f355d88e700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:05.278518) v4 -- 0x55a31ce7c200 con 0 > -4826> 2018-07-11 07:34:05.469204 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a3138b6900 con 0 > -4825> 2018-07-11 07:34:05.487958 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5118 0x55a31367e000 replica scrub(pg: 6.15,from:0'0,to:430'105397,epoch:494/492,start:6:a9e4775c:::benchmark_data_magna108_35617_object54243:0,end:6:a9e56f64:::benchmark_data_magna108_35617_object274367:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -4824> 2018-07-11 07:34:05.487990 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5118 ==== replica scrub(pg: 6.15,from:0'0,to:430'105397,epoch:494/492,start:6:a9e4775c:::benchmark_data_magna108_35617_object54243:0,end:6:a9e56f64:::benchmark_data_magna108_35617_object274367:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 217+0+0 (2051072247 0 0) 0x55a31367e000 con 0x55a317019000 > -4823> 2018-07-11 07:34:05.550750 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b7000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1793 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:34:05.301386) v4 > -4822> 2018-07-11 07:34:05.550773 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.8 10.8.128.108:0/11967 1793 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:05.301386) v4 ==== 2004+0+0 (694940957 0 0) 0x55a34de6f000 con 0x55a3170b7000 > -4821> 2018-07-11 07:34:05.550785 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:05.301386) v4 -- 0x55a31ce7c200 con 0 > -4820> 2018-07-11 07:34:05.550806 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b8800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1793 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:34:05.301386) v4 > -4819> 2018-07-11 07:34:05.550840 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.8 10.8.128.108:0/11967 1793 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:05.301386) v4 ==== 2004+0+0 (694940957 0 0) 0x55a332e79a00 con 0x55a3170b8800 > -4818> 2018-07-11 07:34:05.550858 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:05.301386) v4 -- 0x55a332faf000 con 0 > -4817> 2018-07-11 07:34:05.967529 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15885 conn(0x55a311b76800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1865 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:34:06.591417) v4 > -4816> 2018-07-11 07:34:05.967553 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.7 10.8.128.114:0/15885 1865 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:06.591417) v4 ==== 2004+0+0 (1543643045 0 0) 0x55a34de78600 con 0x55a311b76800 > -4815> 2018-07-11 07:34:05.967563 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:06.591417) v4 -- 0x55a348fbd800 con 0 > -4814> 2018-07-11 07:34:05.967555 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15885 conn(0x55a3170b5800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1865 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:34:06.591417) v4 > -4813> 2018-07-11 07:34:05.967573 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.7 10.8.128.114:0/15885 1865 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:06.591417) v4 ==== 2004+0+0 (1543643045 0 0) 0x55a324474c00 con 0x55a3170b5800 > -4812> 2018-07-11 07:34:05.967586 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:06.591417) v4 -- 0x55a3203f5200 con 0 > -4811> 2018-07-11 07:34:06.165588 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a312c31440 con 0 > -4810> 2018-07-11 07:34:06.179793 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5119 0x55a31367e280 replica scrub(pg: 6.15,from:0'0,to:430'106271,epoch:494/492,start:6:a9e56f64:::benchmark_data_magna108_35617_object274367:0,end:6:a9e64915:::benchmark_data_magna108_35617_object683572:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -4809> 2018-07-11 07:34:06.179817 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5119 ==== replica scrub(pg: 6.15,from:0'0,to:430'106271,epoch:494/492,start:6:a9e56f64:::benchmark_data_magna108_35617_object274367:0,end:6:a9e64915:::benchmark_data_magna108_35617_object683572:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (747290537 0 0) 0x55a31367e280 con 0x55a317019000 > -4808> 2018-07-11 07:34:06.728284 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15255 conn(0x55a311b65800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1868 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:34:07.942095) v4 > -4807> 2018-07-11 07:34:06.728291 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15255 conn(0x55a311b64000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1868 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:34:07.942095) v4 > -4806> 2018-07-11 07:34:06.728309 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.0 10.8.128.113:0/15255 1868 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:07.942095) v4 ==== 2004+0+0 (4099430112 0 0) 0x55a332e79a00 con 0x55a311b65800 > -4805> 2018-07-11 07:34:06.728311 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.0 10.8.128.113:0/15255 1868 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:07.942095) v4 ==== 2004+0+0 (4099430112 0 0) 0x55a34de6f000 con 0x55a311b64000 > -4804> 2018-07-11 07:34:06.728319 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:07.942095) v4 -- 0x55a3203f5200 con 0 > -4803> 2018-07-11 07:34:06.728348 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:07.942095) v4 -- 0x55a348fbd800 con 0 > -4802> 2018-07-11 07:34:06.744574 7f35498a5700 10 monclient: tick > -4801> 2018-07-11 07:34:06.744601 7f35498a5700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2018-07-11 07:33:36.744599) > -4800> 2018-07-11 07:34:06.750216 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.115:0/13323 conn(0x55a321665800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1855 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:34:07.087674) v4 > -4799> 2018-07-11 07:34:06.750240 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.6 10.8.128.115:0/13323 1855 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:07.087674) v4 ==== 2004+0+0 (211873858 0 0) 0x55a324474c00 con 0x55a321665800 > -4798> 2018-07-11 07:34:06.750259 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:07.087674) v4 -- 0x55a3203f5200 con 0 > -4797> 2018-07-11 07:34:06.750299 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.115:0/13323 conn(0x55a321668800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1855 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:34:07.087674) v4 > -4796> 2018-07-11 07:34:06.750312 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.6 10.8.128.115:0/13323 1855 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:07.087674) v4 ==== 2004+0+0 (211873858 0 0) 0x55a324474c00 con 0x55a321668800 > -4795> 2018-07-11 07:34:06.750322 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:07.087674) v4 -- 0x55a3203f4400 con 0 > -4794> 2018-07-11 07:34:06.777280 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12650 conn(0x55a311b67000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1859 0x55a343482400 osd_ping(ping e494 stamp 2018-07-11 07:34:06.812980) v4 > -4793> 2018-07-11 07:34:06.777306 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.3 10.8.128.106:0/12650 1859 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:06.812980) v4 ==== 2004+0+0 (953029202 0 0) 0x55a343482400 con 0x55a311b67000 > -4792> 2018-07-11 07:34:06.777325 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:06.812980) v4 -- 0x55a332e77a00 con 0 > -4791> 2018-07-11 07:34:06.777345 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12650 conn(0x55a311b68800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1859 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:34:06.812980) v4 > -4790> 2018-07-11 07:34:06.777365 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.3 10.8.128.106:0/12650 1859 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:06.812980) v4 ==== 2004+0+0 (953029202 0 0) 0x55a34de78600 con 0x55a311b68800 > -4789> 2018-07-11 07:34:06.777382 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:06.812980) v4 -- 0x55a30fa76600 con 0 > -4788> 2018-07-11 07:34:06.828506 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12677 conn(0x55a32169d800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1864 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:34:06.864239) v4 > -4787> 2018-07-11 07:34:06.828506 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12677 conn(0x55a32169c000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1864 0x55a31ce7c800 osd_ping(ping e494 stamp 2018-07-11 07:34:06.864239) v4 > -4786> 2018-07-11 07:34:06.828530 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.1 10.8.128.106:0/12677 1864 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:06.864239) v4 ==== 2004+0+0 (1259603937 0 0) 0x55a34de6f000 con 0x55a32169d800 > -4785> 2018-07-11 07:34:06.828534 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.1 10.8.128.106:0/12677 1864 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:06.864239) v4 ==== 2004+0+0 (1259603937 0 0) 0x55a31ce7c800 con 0x55a32169c000 > -4784> 2018-07-11 07:34:06.828540 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:06.864239) v4 -- 0x55a30fa76400 con 0 > -4783> 2018-07-11 07:34:06.828563 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:06.864239) v4 -- 0x55a332365200 con 0 > -4782> 2018-07-11 07:34:06.872941 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a324bc3200 con 0 > -4781> 2018-07-11 07:34:06.894413 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5120 0x55a31367e500 replica scrub(pg: 6.15,from:0'0,to:430'106056,epoch:494/492,start:6:a9e64915:::benchmark_data_magna108_35617_object683572:0,end:6:a9e767f7:::benchmark_data_magna108_35617_object62636:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -4780> 2018-07-11 07:34:06.894436 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5120 ==== replica scrub(pg: 6.15,from:0'0,to:430'106056,epoch:494/492,start:6:a9e64915:::benchmark_data_magna108_35617_object683572:0,end:6:a9e767f7:::benchmark_data_magna108_35617_object62636:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 217+0+0 (2467477312 0 0) 0x55a31367e500 con 0x55a317019000 > -4779> 2018-07-11 07:34:07.229267 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15287 conn(0x55a311b75000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1839 0x55a343482400 osd_ping(ping e494 stamp 2018-07-11 07:34:08.443101) v4 > -4778> 2018-07-11 07:34:07.229289 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.4 10.8.128.113:0/15287 1839 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:08.443101) v4 ==== 2004+0+0 (2432614313 0 0) 0x55a343482400 con 0x55a311b75000 > -4777> 2018-07-11 07:34:07.229306 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:08.443101) v4 -- 0x55a332355e00 con 0 > -4776> 2018-07-11 07:34:07.229302 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15287 conn(0x55a311b72000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1839 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:34:08.443101) v4 > -4775> 2018-07-11 07:34:07.229323 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.4 10.8.128.113:0/15287 1839 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:08.443101) v4 ==== 2004+0+0 (2432614313 0 0) 0x55a332e79a00 con 0x55a311b72000 > -4774> 2018-07-11 07:34:07.229334 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:08.443101) v4 -- 0x55a3203f4400 con 0 > -4773> 2018-07-11 07:34:07.254292 7f35488a3700 1 -- 10.8.128.107:6800/12665 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55a317056840 con 0 > -4772> 2018-07-11 07:34:07.254365 7f35488a3700 1 -- 10.8.128.107:6800/12665 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55a32e1bedc0 con 0 > -4771> 2018-07-11 07:34:08.249818 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a328bb6000 con 0 > -4770> 2018-07-11 07:34:08.256671 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15919 conn(0x55a317074800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1847 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:34:08.880531) v4 > -4769> 2018-07-11 07:34:08.256690 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15919 conn(0x55a3170b4000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1847 0x55a343482400 osd_ping(ping e494 stamp 2018-07-11 07:34:08.880531) v4 > -4768> 2018-07-11 07:34:08.256696 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.11 10.8.128.114:0/15919 1847 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:08.880531) v4 ==== 2004+0+0 (1705433681 0 0) 0x55a324474c00 con 0x55a317074800 > -4767> 2018-07-11 07:34:08.256707 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.11 10.8.128.114:0/15919 1847 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:08.880531) v4 ==== 2004+0+0 (1705433681 0 0) 0x55a343482400 con 0x55a3170b4000 > -4766> 2018-07-11 07:34:08.256710 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:08.880531) v4 -- 0x55a3203f5200 con 0 > -4765> 2018-07-11 07:34:08.256744 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:08.880531) v4 -- 0x55a330935e00 con 0 > -4764> 2018-07-11 07:34:08.263521 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5121 0x55a31367e780 replica scrub(pg: 6.15,from:0'0,to:430'106847,epoch:494/492,start:6:a9e767f7:::benchmark_data_magna108_35617_object62636:0,end:6:a9e88809:::benchmark_data_magna108.ceph.redhat.com_2478_object70493:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -4763> 2018-07-11 07:34:08.263545 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5121 ==== replica scrub(pg: 6.15,from:0'0,to:430'106847,epoch:494/492,start:6:a9e767f7:::benchmark_data_magna108_35617_object62636:0,end:6:a9e88809:::benchmark_data_magna108.ceph.redhat.com_2478_object70493:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 231+0+0 (197214579 0 0) 0x55a31367e780 con 0x55a317019000 > -4762> 2018-07-11 07:34:08.267999 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15885 conn(0x55a311b76800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1866 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:34:08.891843) v4 > -4761> 2018-07-11 07:34:08.268005 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15885 conn(0x55a3170b5800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1866 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:34:08.891843) v4 > -4760> 2018-07-11 07:34:08.268021 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.7 10.8.128.114:0/15885 1866 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:08.891843) v4 ==== 2004+0+0 (3361787086 0 0) 0x55a34de78600 con 0x55a311b76800 > -4759> 2018-07-11 07:34:08.268024 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.7 10.8.128.114:0/15885 1866 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:08.891843) v4 ==== 2004+0+0 (3361787086 0 0) 0x55a332e79a00 con 0x55a3170b5800 > -4758> 2018-07-11 07:34:08.268030 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:08.891843) v4 -- 0x55a30fa76400 con 0 > -4757> 2018-07-11 07:34:08.268054 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:08.891843) v4 -- 0x55a3203f5200 con 0 > -4756> 2018-07-11 07:34:08.500337 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 59] Generated table #36582: 4957165 keys, 80348256 bytes > -4755> 2018-07-11 07:34:08.500374 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294448500358, "cf_name": "default", "job": 59, "event": "table_file_creation", "file_number": 36582, "file_size": 80348256, "table_properties": {"data_size": 67109362, "index_size": 689203, "filter_size": 12548701, "raw_key_size": 143206993, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957165, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957165", "kMergeOperands": "0"}} > -4754> 2018-07-11 07:34:08.864891 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12651 conn(0x55a3216a0800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1844 0x55a343482400 osd_ping(ping e494 stamp 2018-07-11 07:34:08.900665) v4 > -4753> 2018-07-11 07:34:08.864893 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12651 conn(0x55a32169f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1844 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:34:08.900665) v4 > -4752> 2018-07-11 07:34:08.864914 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.2 10.8.128.106:0/12651 1844 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:08.900665) v4 ==== 2004+0+0 (28327726 0 0) 0x55a343482400 con 0x55a3216a0800 > -4751> 2018-07-11 07:34:08.864915 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.2 10.8.128.106:0/12651 1844 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:08.900665) v4 ==== 2004+0+0 (28327726 0 0) 0x55a324474c00 con 0x55a32169f000 > -4750> 2018-07-11 07:34:08.864932 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:08.900665) v4 -- 0x55a332349200 con 0 > -4749> 2018-07-11 07:34:08.864968 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:08.900665) v4 -- 0x55a3203f5200 con 0 > -4748> 2018-07-11 07:34:09.151105 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a31fb04000 con 0 > -4747> 2018-07-11 07:34:09.163565 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5122 0x55a31367ea00 replica scrub(pg: 6.15,from:0'0,to:430'106154,epoch:494/492,start:6:a9e88809:::benchmark_data_magna108.ceph.redhat.com_2478_object70493:0,end:6:a9e96f62:::benchmark_data_magna108.ceph.redhat.com_2478_object167119:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -4746> 2018-07-11 07:34:09.163588 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5122 ==== replica scrub(pg: 6.15,from:0'0,to:430'106154,epoch:494/492,start:6:a9e88809:::benchmark_data_magna108.ceph.redhat.com_2478_object70493:0,end:6:a9e96f62:::benchmark_data_magna108.ceph.redhat.com_2478_object167119:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 247+0+0 (4224948197 0 0) 0x55a31367ea00 con 0x55a317019000 > -4745> 2018-07-11 07:34:09.219384 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15263 conn(0x55a321667000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1842 0x55a330935e00 osd_ping(ping e494 stamp 2018-07-11 07:34:10.433282) v4 > -4744> 2018-07-11 07:34:09.219412 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.5 10.8.128.113:0/15263 1842 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:10.433282) v4 ==== 2004+0+0 (4000422136 0 0) 0x55a330935e00 con 0x55a321667000 > -4743> 2018-07-11 07:34:09.219427 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:10.433282) v4 -- 0x55a332349200 con 0 > -4742> 2018-07-11 07:34:09.219476 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15263 conn(0x55a321664000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1842 0x55a330935e00 osd_ping(ping e494 stamp 2018-07-11 07:34:10.433282) v4 > -4741> 2018-07-11 07:34:09.219490 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.5 10.8.128.113:0/15263 1842 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:10.433282) v4 ==== 2004+0+0 (4000422136 0 0) 0x55a330935e00 con 0x55a321664000 > -4740> 2018-07-11 07:34:09.219498 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:10.433282) v4 -- 0x55a332ff2e00 con 0 > -4739> 2018-07-11 07:34:09.365351 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12651 conn(0x55a3216a0800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1845 0x55a343482400 osd_ping(ping e494 stamp 2018-07-11 07:34:09.401147) v4 > -4738> 2018-07-11 07:34:09.365377 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.2 10.8.128.106:0/12651 1845 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:09.401147) v4 ==== 2004+0+0 (1823060571 0 0) 0x55a343482400 con 0x55a3216a0800 > -4737> 2018-07-11 07:34:09.365376 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12651 conn(0x55a32169f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1845 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:34:09.401147) v4 > -4736> 2018-07-11 07:34:09.365395 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.2 10.8.128.106:0/12651 1845 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:09.401147) v4 ==== 2004+0+0 (1823060571 0 0) 0x55a332e79a00 con 0x55a32169f000 > -4735> 2018-07-11 07:34:09.365396 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:09.401147) v4 -- 0x55a332ff2e00 con 0 > -4734> 2018-07-11 07:34:09.365432 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:09.401147) v4 -- 0x55a3203f5200 con 0 > -4733> 2018-07-11 07:34:09.369473 7f3543899700 5 osd.10 494 heartbeat: osd_stat(784 GB used, 147 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,1]) > -4732> 2018-07-11 07:34:09.369487 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6802/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:34:09.369486) v4 -- 0x55a33234e600 con 0 > -4731> 2018-07-11 07:34:09.369523 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6803/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:34:09.369486) v4 -- 0x55a31707da00 con 0 > -4730> 2018-07-11 07:34:09.369539 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6806/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:34:09.369486) v4 -- 0x55a31ce7b600 con 0 > -4729> 2018-07-11 07:34:09.369553 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6807/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:34:09.369486) v4 -- 0x55a31ce7ba00 con 0 > -4728> 2018-07-11 07:34:09.369565 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6808/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:34:09.369486) v4 -- 0x55a33234ba00 con 0 > -4727> 2018-07-11 07:34:09.369576 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6810/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:34:09.369486) v4 -- 0x55a34de63a00 con 0 > -4726> 2018-07-11 07:34:09.369589 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6809/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:34:09.369486) v4 -- 0x55a31ce7bc00 con 0 > -4725> 2018-07-11 07:34:09.369601 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6811/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:34:09.369486) v4 -- 0x55a34dea8400 con 0 > -4724> 2018-07-11 07:34:09.369614 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6810/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:34:09.369486) v4 -- 0x55a33532ce00 con 0 > -4723> 2018-07-11 07:34:09.369642 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6811/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:34:09.369486) v4 -- 0x55a34de62c00 con 0 > -4722> 2018-07-11 07:34:09.369658 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6806/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:34:09.369486) v4 -- 0x55a33235be00 con 0 > -4721> 2018-07-11 07:34:09.369675 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6807/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:34:09.369486) v4 -- 0x55a32162a000 con 0 > -4720> 2018-07-11 07:34:09.369689 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6802/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:34:09.369486) v4 -- 0x55a32162a800 con 0 > -4719> 2018-07-11 07:34:09.369702 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6803/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:34:09.369486) v4 -- 0x55a34de63e00 con 0 > -4718> 2018-07-11 07:34:09.369718 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6806/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:34:09.369486) v4 -- 0x55a32162aa00 con 0 > -4717> 2018-07-11 07:34:09.369731 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6807/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:34:09.369486) v4 -- 0x55a3170d1a00 con 0 > -4716> 2018-07-11 07:34:09.369746 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6802/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:34:09.369486) v4 -- 0x55a34de78a00 con 0 > -4715> 2018-07-11 07:34:09.369763 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6803/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:34:09.369486) v4 -- 0x55a32442b600 con 0 > -4714> 2018-07-11 07:34:09.369777 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6806/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:34:09.369486) v4 -- 0x55a3170d1e00 con 0 > -4713> 2018-07-11 07:34:09.369790 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6807/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:34:09.369486) v4 -- 0x55a33234b800 con 0 > -4712> 2018-07-11 07:34:09.369802 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6802/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:34:09.369486) v4 -- 0x55a34dea9000 con 0 > -4711> 2018-07-11 07:34:09.369817 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6803/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:34:09.369486) v4 -- 0x55a33234a400 con 0 > -4710> 2018-07-11 07:34:09.369904 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6802/15255 conn(0x55a314e1c000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1849 0x55a330935e00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:09.369486) v4 > -4709> 2018-07-11 07:34:09.369928 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6802/15255 1849 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:09.369486) v4 ==== 2004+0+0 (3119464535 0 0) 0x55a330935e00 con 0x55a314e1c000 > -4708> 2018-07-11 07:34:09.369937 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6803/15255 conn(0x55a314e1d800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1849 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:09.369486) v4 > -4707> 2018-07-11 07:34:09.369970 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6803/15255 1849 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:09.369486) v4 ==== 2004+0+0 (3119464535 0 0) 0x55a332e79a00 con 0x55a314e1d800 > -4706> 2018-07-11 07:34:09.369988 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6807/12677 conn(0x55a314e85800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1849 0x55a330935e00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:09.369486) v4 > -4705> 2018-07-11 07:34:09.370003 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6809/12650 conn(0x55a314e1f000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1849 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:09.369486) v4 > -4704> 2018-07-11 07:34:09.370003 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6807/12677 1849 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:09.369486) v4 ==== 2004+0+0 (3119464535 0 0) 0x55a330935e00 con 0x55a314e85800 > -4703> 2018-07-11 07:34:09.370027 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6809/12650 1849 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:09.369486) v4 ==== 2004+0+0 (3119464535 0 0) 0x55a34de6f000 con 0x55a314e1f000 > -4702> 2018-07-11 07:34:09.370029 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6806/12617 conn(0x55a314e26000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1849 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:09.369486) v4 > -4701> 2018-07-11 07:34:09.370048 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6806/12617 1849 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:09.369486) v4 ==== 2004+0+0 (3119464535 0 0) 0x55a332e79a00 con 0x55a314e26000 > -4700> 2018-07-11 07:34:09.370059 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6808/12651 conn(0x55a314e96000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1849 0x55a330935e00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:09.369486) v4 > -4699> 2018-07-11 07:34:09.370072 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6808/12651 1849 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:09.369486) v4 ==== 2004+0+0 (3119464535 0 0) 0x55a330935e00 con 0x55a314e96000 > -4698> 2018-07-11 07:34:09.370085 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6806/12677 conn(0x55a314e84000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1849 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:09.369486) v4 > -4697> 2018-07-11 07:34:09.370108 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6810/15287 conn(0x55a314e03000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1849 0x55a330935e00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:09.369486) v4 > -4696> 2018-07-11 07:34:09.370107 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6806/12677 1849 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:09.369486) v4 ==== 2004+0+0 (3119464535 0 0) 0x55a34de6f000 con 0x55a314e84000 > -4695> 2018-07-11 07:34:09.370120 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6810/15287 1849 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:09.369486) v4 ==== 2004+0+0 (3119464535 0 0) 0x55a330935e00 con 0x55a314e03000 > -4694> 2018-07-11 07:34:09.370130 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6810/12651 conn(0x55a314e97800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1849 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:09.369486) v4 > -4693> 2018-07-11 07:34:09.370151 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6807/12617 conn(0x55a314e27800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1849 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:09.369486) v4 > -4692> 2018-07-11 07:34:09.370159 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6807/15263 conn(0x55a314e9a800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1849 0x55a330935e00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:09.369486) v4 > -4691> 2018-07-11 07:34:09.370148 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6810/12651 1849 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:09.369486) v4 ==== 2004+0+0 (3119464535 0 0) 0x55a332e79a00 con 0x55a314e97800 > -4690> 2018-07-11 07:34:09.370167 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6807/12617 1849 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:09.369486) v4 ==== 2004+0+0 (3119464535 0 0) 0x55a34de6f000 con 0x55a314e27800 > -4689> 2018-07-11 07:34:09.370171 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6807/15263 1849 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:09.369486) v4 ==== 2004+0+0 (3119464535 0 0) 0x55a330935e00 con 0x55a314e9a800 > -4688> 2018-07-11 07:34:09.370217 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6811/15287 conn(0x55a314e04800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1849 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:09.369486) v4 > -4687> 2018-07-11 07:34:09.370227 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6811/12650 conn(0x55a314e20800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1849 0x55a330935e00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:09.369486) v4 > -4686> 2018-07-11 07:34:09.370217 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6806/15263 conn(0x55a314e99000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1849 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:09.369486) v4 > -4685> 2018-07-11 07:34:09.370239 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6811/12650 1849 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:09.369486) v4 ==== 2004+0+0 (3119464535 0 0) 0x55a330935e00 con 0x55a314e20800 > -4684> 2018-07-11 07:34:09.370252 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6806/15263 1849 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:09.369486) v4 ==== 2004+0+0 (3119464535 0 0) 0x55a34de6f000 con 0x55a314e99000 > -4683> 2018-07-11 07:34:09.370238 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6811/15287 1849 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:09.369486) v4 ==== 2004+0+0 (3119464535 0 0) 0x55a332e79a00 con 0x55a314e04800 > -4682> 2018-07-11 07:34:09.370302 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6807/15885 conn(0x55a314e01800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1849 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:09.369486) v4 > -4681> 2018-07-11 07:34:09.370309 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6803/11967 conn(0x55a314e13800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1849 0x55a330935e00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:09.369486) v4 > -4680> 2018-07-11 07:34:09.370323 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6803/11967 1849 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:09.369486) v4 ==== 2004+0+0 (3119464535 0 0) 0x55a330935e00 con 0x55a314e13800 > -4679> 2018-07-11 07:34:09.370322 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6807/15885 1849 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:09.369486) v4 ==== 2004+0+0 (3119464535 0 0) 0x55a34de6f000 con 0x55a314e01800 > -4678> 2018-07-11 07:34:09.370337 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6806/15885 conn(0x55a314e00000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1849 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:09.369486) v4 > -4677> 2018-07-11 07:34:09.370355 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6806/15885 1849 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:09.369486) v4 ==== 2004+0+0 (3119464535 0 0) 0x55a332e79a00 con 0x55a314e00000 > -4676> 2018-07-11 07:34:09.370380 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6803/15919 conn(0x55a314e88800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1849 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:09.369486) v4 > -4675> 2018-07-11 07:34:09.370399 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6803/15919 1849 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:09.369486) v4 ==== 2004+0+0 (3119464535 0 0) 0x55a34de6f000 con 0x55a314e88800 > -4674> 2018-07-11 07:34:09.370403 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6802/13323 conn(0x55a314e15000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1849 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:09.369486) v4 > -4673> 2018-07-11 07:34:09.370415 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6802/13323 1849 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:09.369486) v4 ==== 2004+0+0 (3119464535 0 0) 0x55a332e79a00 con 0x55a314e15000 > -4672> 2018-07-11 07:34:09.370446 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6803/13323 conn(0x55a314e16800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1849 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:09.369486) v4 > -4671> 2018-07-11 07:34:09.370454 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6802/15919 conn(0x55a314e87000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1849 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:09.369486) v4 > -4670> 2018-07-11 07:34:09.370466 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6802/15919 1849 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:09.369486) v4 ==== 2004+0+0 (3119464535 0 0) 0x55a332e79a00 con 0x55a314e87000 > -4669> 2018-07-11 07:34:09.370466 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6803/13323 1849 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:09.369486) v4 ==== 2004+0+0 (3119464535 0 0) 0x55a34de6f000 con 0x55a314e16800 > -4668> 2018-07-11 07:34:09.370502 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6802/11967 conn(0x55a314e12000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1849 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:09.369486) v4 > -4667> 2018-07-11 07:34:09.370518 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6802/11967 1849 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:09.369486) v4 ==== 2004+0+0 (3119464535 0 0) 0x55a34de6f000 con 0x55a314e12000 > -4666> 2018-07-11 07:34:09.805402 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a3526ed440 con 0 > -4665> 2018-07-11 07:34:09.822076 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5123 0x55a31367f180 replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:494/492,start:6:a9e96f62:::benchmark_data_magna108.ceph.redhat.com_2478_object167119:0,end:6:a9ea6c00:::benchmark_data_magna108_41525_object43844:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -4664> 2018-07-11 07:34:09.822107 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5123 ==== replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:494/492,start:6:a9e96f62:::benchmark_data_magna108.ceph.redhat.com_2478_object167119:0,end:6:a9ea6c00:::benchmark_data_magna108_41525_object43844:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 232+0+0 (2550717395 0 0) 0x55a31367f180 con 0x55a317019000 > -4663> 2018-07-11 07:34:09.979293 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.107:0/12617 conn(0x55a32447e000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1818 0x55a330935e00 osd_ping(ping e494 stamp 2018-07-11 07:34:09.978952) v4 > -4662> 2018-07-11 07:34:09.979319 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.9 10.8.128.107:0/12617 1818 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:09.978952) v4 ==== 2004+0+0 (205079849 0 0) 0x55a330935e00 con 0x55a32447e000 > -4661> 2018-07-11 07:34:09.979314 7f355d88e700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.107:0/12617 conn(0x55a32447f800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1818 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:34:09.978952) v4 > -4660> 2018-07-11 07:34:09.979341 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:09.978952) v4 -- 0x55a33234e600 con 0 > -4659> 2018-07-11 07:34:09.979343 7f355d88e700 1 -- 10.8.128.107:6803/12665 <== osd.9 10.8.128.107:0/12617 1818 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:09.978952) v4 ==== 2004+0+0 (205079849 0 0) 0x55a34de6f000 con 0x55a32447f800 > -4658> 2018-07-11 07:34:09.979374 7f355d88e700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:09.978952) v4 -- 0x55a31ce7b600 con 0 > -4657> 2018-07-11 07:34:10.129581 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15287 conn(0x55a311b75000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1840 0x55a31ce7ba00 osd_ping(ping e494 stamp 2018-07-11 07:34:11.343524) v4 > -4656> 2018-07-11 07:34:10.129581 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15287 conn(0x55a311b72000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1840 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:34:11.343524) v4 > -4655> 2018-07-11 07:34:10.129609 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.4 10.8.128.113:0/15287 1840 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:11.343524) v4 ==== 2004+0+0 (470844380 0 0) 0x55a31ce7ba00 con 0x55a311b75000 > -4654> 2018-07-11 07:34:10.129612 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.4 10.8.128.113:0/15287 1840 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:11.343524) v4 ==== 2004+0+0 (470844380 0 0) 0x55a332e79a00 con 0x55a311b72000 > -4653> 2018-07-11 07:34:10.129628 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:11.343524) v4 -- 0x55a33234e600 con 0 > -4652> 2018-07-11 07:34:10.129657 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:11.343524) v4 -- 0x55a34de63a00 con 0 > -4651> 2018-07-11 07:34:10.251152 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b7000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1794 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:34:10.001831) v4 > -4650> 2018-07-11 07:34:10.251153 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b8800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1794 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:34:10.001831) v4 > -4649> 2018-07-11 07:34:10.251173 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.8 10.8.128.108:0/11967 1794 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:10.001831) v4 ==== 2004+0+0 (3768179658 0 0) 0x55a34de78600 con 0x55a3170b7000 > -4648> 2018-07-11 07:34:10.251182 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:10.001831) v4 -- 0x55a31ce7b600 con 0 > -4647> 2018-07-11 07:34:10.251177 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.8 10.8.128.108:0/11967 1794 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:10.001831) v4 ==== 2004+0+0 (3768179658 0 0) 0x55a324474c00 con 0x55a3170b8800 > -4646> 2018-07-11 07:34:10.251212 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:10.001831) v4 -- 0x55a34de63a00 con 0 > -4645> 2018-07-11 07:34:10.328916 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12677 conn(0x55a32169d800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1865 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:34:10.364777) v4 > -4644> 2018-07-11 07:34:10.328944 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.1 10.8.128.106:0/12677 1865 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:10.364777) v4 ==== 2004+0+0 (3585471036 0 0) 0x55a34de6f000 con 0x55a32169d800 > -4643> 2018-07-11 07:34:10.328956 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:10.364777) v4 -- 0x55a30fa76400 con 0 > -4642> 2018-07-11 07:34:10.328970 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12677 conn(0x55a32169c000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1865 0x55a330935e00 osd_ping(ping e494 stamp 2018-07-11 07:34:10.364777) v4 > -4641> 2018-07-11 07:34:10.328994 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.1 10.8.128.106:0/12677 1865 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:10.364777) v4 ==== 2004+0+0 (3585471036 0 0) 0x55a330935e00 con 0x55a32169c000 > -4640> 2018-07-11 07:34:10.329007 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:10.364777) v4 -- 0x55a332ff2e00 con 0 > -4639> 2018-07-11 07:34:10.541056 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a33de4d8c0 con 0 > -4638> 2018-07-11 07:34:10.563726 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5124 0x55a31367f400 replica scrub(pg: 6.15,from:0'0,to:430'106624,epoch:494/492,start:6:a9ea6c00:::benchmark_data_magna108_41525_object43844:0,end:6:a9eb87e8:::benchmark_data_magna108_35617_object176457:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -4637> 2018-07-11 07:34:10.563755 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5124 ==== replica scrub(pg: 6.15,from:0'0,to:430'106624,epoch:494/492,start:6:a9ea6c00:::benchmark_data_magna108_41525_object43844:0,end:6:a9eb87e8:::benchmark_data_magna108_35617_object176457:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 217+0+0 (3885606817 0 0) 0x55a31367f400 con 0x55a317019000 > -4636> 2018-07-11 07:34:10.828635 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15255 conn(0x55a311b64000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1869 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:34:12.042537) v4 > -4635> 2018-07-11 07:34:10.828635 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15255 conn(0x55a311b65800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1869 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:34:12.042537) v4 > -4634> 2018-07-11 07:34:10.828667 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.0 10.8.128.113:0/15255 1869 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:12.042537) v4 ==== 2004+0+0 (2497176933 0 0) 0x55a332e79a00 con 0x55a311b65800 > -4633> 2018-07-11 07:34:10.828665 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.0 10.8.128.113:0/15255 1869 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:12.042537) v4 ==== 2004+0+0 (2497176933 0 0) 0x55a34de78600 con 0x55a311b64000 > -4632> 2018-07-11 07:34:10.828689 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:12.042537) v4 -- 0x55a34de63a00 con 0 > -4631> 2018-07-11 07:34:10.828723 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:12.042537) v4 -- 0x55a30fa76400 con 0 > -4630> 2018-07-11 07:34:10.877755 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12650 conn(0x55a311b68800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1860 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:34:10.913549) v4 > -4629> 2018-07-11 07:34:10.877782 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.3 10.8.128.106:0/12650 1860 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:10.913549) v4 ==== 2004+0+0 (1194213162 0 0) 0x55a34de6f000 con 0x55a311b68800 > -4628> 2018-07-11 07:34:10.877792 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12650 conn(0x55a311b67000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1860 0x55a31ce7ba00 osd_ping(ping e494 stamp 2018-07-11 07:34:10.913549) v4 > -4627> 2018-07-11 07:34:10.877802 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:10.913549) v4 -- 0x55a30fa76400 con 0 > -4626> 2018-07-11 07:34:10.877809 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.3 10.8.128.106:0/12650 1860 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:10.913549) v4 ==== 2004+0+0 (1194213162 0 0) 0x55a31ce7ba00 con 0x55a311b67000 > -4625> 2018-07-11 07:34:10.877825 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:10.913549) v4 -- 0x55a31ce7ca00 con 0 > -4624> 2018-07-11 07:34:11.157123 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15919 conn(0x55a317074800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1848 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:34:11.780971) v4 > -4623> 2018-07-11 07:34:11.157144 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.11 10.8.128.114:0/15919 1848 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:11.780971) v4 ==== 2004+0+0 (1497649559 0 0) 0x55a324474c00 con 0x55a317074800 > -4622> 2018-07-11 07:34:11.157154 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:11.780971) v4 -- 0x55a31707da00 con 0 > -4621> 2018-07-11 07:34:11.157192 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15919 conn(0x55a3170b4000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1848 0x55a332ff2e00 osd_ping(ping e494 stamp 2018-07-11 07:34:11.780971) v4 > -4620> 2018-07-11 07:34:11.157213 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.11 10.8.128.114:0/15919 1848 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:11.780971) v4 ==== 2004+0+0 (1497649559 0 0) 0x55a332ff2e00 con 0x55a3170b4000 > -4619> 2018-07-11 07:34:11.157225 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:11.780971) v4 -- 0x55a34dea9e00 con 0 > -4618> 2018-07-11 07:34:11.184151 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a33a6ef680 con 0 > -4617> 2018-07-11 07:34:11.197167 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5125 0x55a31367f680 replica scrub(pg: 6.15,from:0'0,to:430'106636,epoch:494/492,start:6:a9eb87e8:::benchmark_data_magna108_35617_object176457:0,end:6:a9ec3fac:::benchmark_data_magna108.ceph.redhat.com_2478_object280060:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -4616> 2018-07-11 07:34:11.197197 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5125 ==== replica scrub(pg: 6.15,from:0'0,to:430'106636,epoch:494/492,start:6:a9eb87e8:::benchmark_data_magna108_35617_object176457:0,end:6:a9ec3fac:::benchmark_data_magna108.ceph.redhat.com_2478_object280060:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (1768881998 0 0) 0x55a31367f680 con 0x55a317019000 > -4615> 2018-07-11 07:34:11.669963 7f3543899700 5 osd.10 494 heartbeat: osd_stat(784 GB used, 147 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,0,1]) > -4614> 2018-07-11 07:34:11.669979 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6802/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:34:11.669977) v4 -- 0x55a33234a600 con 0 > -4613> 2018-07-11 07:34:11.670018 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6803/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:34:11.669977) v4 -- 0x55a3170d0200 con 0 > -4612> 2018-07-11 07:34:11.670034 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6806/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:34:11.669977) v4 -- 0x55a32442b400 con 0 > -4611> 2018-07-11 07:34:11.670052 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6807/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:34:11.669977) v4 -- 0x55a32447a200 con 0 > -4610> 2018-07-11 07:34:11.670071 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6808/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:34:11.669977) v4 -- 0x55a32442a800 con 0 > -4609> 2018-07-11 07:34:11.670093 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6810/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:34:11.669977) v4 -- 0x55a32442b200 con 0 > -4608> 2018-07-11 07:34:11.670120 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6809/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:34:11.669977) v4 -- 0x55a32e457c00 con 0 > -4607> 2018-07-11 07:34:11.670140 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6811/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:34:11.669977) v4 -- 0x55a34de84c00 con 0 > -4606> 2018-07-11 07:34:11.670167 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6810/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:34:11.669977) v4 -- 0x55a34de63400 con 0 > -4605> 2018-07-11 07:34:11.670189 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6811/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:34:11.669977) v4 -- 0x55a34de5a800 con 0 > -4604> 2018-07-11 07:34:11.670208 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6806/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:34:11.669977) v4 -- 0x55a34dea9a00 con 0 > -4603> 2018-07-11 07:34:11.670223 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6807/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:34:11.669977) v4 -- 0x55a31ce7b000 con 0 > -4602> 2018-07-11 07:34:11.670244 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6802/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:34:11.669977) v4 -- 0x55a314e7a600 con 0 > -4601> 2018-07-11 07:34:11.670281 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6803/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:34:11.669977) v4 -- 0x55a32eb28000 con 0 > -4600> 2018-07-11 07:34:11.670298 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6806/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:34:11.669977) v4 -- 0x55a32162a200 con 0 > -4599> 2018-07-11 07:34:11.670317 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6807/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:34:11.669977) v4 -- 0x55a332e85a00 con 0 > -4598> 2018-07-11 07:34:11.670345 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6802/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:34:11.669977) v4 -- 0x55a34de79a00 con 0 > -4597> 2018-07-11 07:34:11.670367 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6803/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:34:11.669977) v4 -- 0x55a32eb29200 con 0 > -4596> 2018-07-11 07:34:11.670389 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6806/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:34:11.669977) v4 -- 0x55a332384800 con 0 > -4595> 2018-07-11 07:34:11.670416 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6807/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:34:11.669977) v4 -- 0x55a317044400 con 0 > -4594> 2018-07-11 07:34:11.670442 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6802/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:34:11.669977) v4 -- 0x55a34dea9800 con 0 > -4593> 2018-07-11 07:34:11.670463 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6803/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:34:11.669977) v4 -- 0x55a34de63800 con 0 > -4592> 2018-07-11 07:34:11.670513 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6803/15255 conn(0x55a314e1d800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1850 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:11.669977) v4 > -4591> 2018-07-11 07:34:11.670511 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6807/12677 conn(0x55a314e85800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1850 0x55a34dea9e00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:11.669977) v4 > -4590> 2018-07-11 07:34:11.670517 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6809/12650 conn(0x55a314e1f000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1850 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:11.669977) v4 > -4589> 2018-07-11 07:34:11.670537 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6803/15255 1850 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:11.669977) v4 ==== 2004+0+0 (4095058680 0 0) 0x55a324474c00 con 0x55a314e1d800 > -4588> 2018-07-11 07:34:11.670537 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6807/12677 1850 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:11.669977) v4 ==== 2004+0+0 (4095058680 0 0) 0x55a34dea9e00 con 0x55a314e85800 > -4587> 2018-07-11 07:34:11.670542 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6809/12650 1850 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:11.669977) v4 ==== 2004+0+0 (4095058680 0 0) 0x55a34de6f000 con 0x55a314e1f000 > -4586> 2018-07-11 07:34:11.670579 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6802/15255 conn(0x55a314e1c000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1850 0x55a34dea9e00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:11.669977) v4 > -4585> 2018-07-11 07:34:11.670597 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6802/15255 1850 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:11.669977) v4 ==== 2004+0+0 (4095058680 0 0) 0x55a34dea9e00 con 0x55a314e1c000 > -4584> 2018-07-11 07:34:11.670634 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6810/12651 conn(0x55a314e97800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1850 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:11.669977) v4 > -4583> 2018-07-11 07:34:11.670651 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6810/12651 1850 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:11.669977) v4 ==== 2004+0+0 (4095058680 0 0) 0x55a324474c00 con 0x55a314e97800 > -4582> 2018-07-11 07:34:11.670650 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6808/12651 conn(0x55a314e96000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1850 0x55a34dea9e00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:11.669977) v4 > -4581> 2018-07-11 07:34:11.670660 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6806/12677 conn(0x55a314e84000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1850 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:11.669977) v4 > -4580> 2018-07-11 07:34:11.670673 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6806/12677 1850 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:11.669977) v4 ==== 2004+0+0 (4095058680 0 0) 0x55a34de78600 con 0x55a314e84000 > -4579> 2018-07-11 07:34:11.670671 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6808/12651 1850 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:11.669977) v4 ==== 2004+0+0 (4095058680 0 0) 0x55a34dea9e00 con 0x55a314e96000 > -4578> 2018-07-11 07:34:11.670684 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6811/15287 conn(0x55a314e04800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1850 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:11.669977) v4 > -4577> 2018-07-11 07:34:11.670696 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6811/15287 1850 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:11.669977) v4 ==== 2004+0+0 (4095058680 0 0) 0x55a324474c00 con 0x55a314e04800 > -4576> 2018-07-11 07:34:11.670713 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6806/15263 conn(0x55a314e99000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1850 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:11.669977) v4 > -4575> 2018-07-11 07:34:11.670721 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6806/12617 conn(0x55a314e26000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1850 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:11.669977) v4 > -4574> 2018-07-11 07:34:11.670729 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6806/12617 1850 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:11.669977) v4 ==== 2004+0+0 (4095058680 0 0) 0x55a324474c00 con 0x55a314e26000 > -4573> 2018-07-11 07:34:11.670727 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6806/15263 1850 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:11.669977) v4 ==== 2004+0+0 (4095058680 0 0) 0x55a34de78600 con 0x55a314e99000 > -4572> 2018-07-11 07:34:11.670731 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6810/15287 conn(0x55a314e03000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1850 0x55a34dea9e00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:11.669977) v4 > -4571> 2018-07-11 07:34:11.670750 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6810/15287 1850 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:11.669977) v4 ==== 2004+0+0 (4095058680 0 0) 0x55a34dea9e00 con 0x55a314e03000 > -4570> 2018-07-11 07:34:11.670760 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6806/15885 conn(0x55a314e00000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1850 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:11.669977) v4 > -4569> 2018-07-11 07:34:11.670769 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6803/13323 conn(0x55a314e16800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1850 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:11.669977) v4 > -4568> 2018-07-11 07:34:11.670785 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6806/15885 1850 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:11.669977) v4 ==== 2004+0+0 (4095058680 0 0) 0x55a324474c00 con 0x55a314e00000 > -4567> 2018-07-11 07:34:11.670789 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6803/13323 1850 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:11.669977) v4 ==== 2004+0+0 (4095058680 0 0) 0x55a34de78600 con 0x55a314e16800 > -4566> 2018-07-11 07:34:11.670788 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6811/12650 conn(0x55a314e20800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1850 0x55a34dea9e00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:11.669977) v4 > -4565> 2018-07-11 07:34:11.670806 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6811/12650 1850 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:11.669977) v4 ==== 2004+0+0 (4095058680 0 0) 0x55a34dea9e00 con 0x55a314e20800 > -4564> 2018-07-11 07:34:11.670821 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6802/13323 conn(0x55a314e15000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1850 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:11.669977) v4 > -4563> 2018-07-11 07:34:11.670816 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6807/12617 conn(0x55a314e27800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1850 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:11.669977) v4 > -4562> 2018-07-11 07:34:11.670836 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6802/13323 1850 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:11.669977) v4 ==== 2004+0+0 (4095058680 0 0) 0x55a324474c00 con 0x55a314e15000 > -4561> 2018-07-11 07:34:11.670838 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6807/15263 conn(0x55a314e9a800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1850 0x55a34dea9e00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:11.669977) v4 > -4560> 2018-07-11 07:34:11.670840 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6807/12617 1850 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:11.669977) v4 ==== 2004+0+0 (4095058680 0 0) 0x55a34de78600 con 0x55a314e27800 > -4559> 2018-07-11 07:34:11.670852 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6807/15263 1850 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:11.669977) v4 ==== 2004+0+0 (4095058680 0 0) 0x55a34dea9e00 con 0x55a314e9a800 > -4558> 2018-07-11 07:34:11.670862 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6802/15919 conn(0x55a314e87000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1850 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:11.669977) v4 > -4557> 2018-07-11 07:34:11.670882 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6807/15885 conn(0x55a314e01800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1850 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:11.669977) v4 > -4556> 2018-07-11 07:34:11.670874 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6802/15919 1850 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:11.669977) v4 ==== 2004+0+0 (4095058680 0 0) 0x55a324474c00 con 0x55a314e87000 > -4555> 2018-07-11 07:34:11.670884 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6803/11967 conn(0x55a314e13800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1850 0x55a34dea9e00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:11.669977) v4 > -4554> 2018-07-11 07:34:11.670898 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6807/15885 1850 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:11.669977) v4 ==== 2004+0+0 (4095058680 0 0) 0x55a34de78600 con 0x55a314e01800 > -4553> 2018-07-11 07:34:11.670908 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6803/11967 1850 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:11.669977) v4 ==== 2004+0+0 (4095058680 0 0) 0x55a34dea9e00 con 0x55a314e13800 > -4552> 2018-07-11 07:34:11.670933 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6802/11967 conn(0x55a314e12000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1850 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:11.669977) v4 > -4551> 2018-07-11 07:34:11.670945 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6802/11967 1850 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:11.669977) v4 ==== 2004+0+0 (4095058680 0 0) 0x55a34de78600 con 0x55a314e12000 > -4550> 2018-07-11 07:34:11.671001 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6803/15919 conn(0x55a314e88800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1850 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:11.669977) v4 > -4549> 2018-07-11 07:34:11.671016 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6803/15919 1850 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:11.669977) v4 ==== 2004+0+0 (4095058680 0 0) 0x55a34de78600 con 0x55a314e88800 > -4548> 2018-07-11 07:34:11.816642 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a34a7a78c0 con 0 > -4547> 2018-07-11 07:34:11.839916 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5126 0x55a31367f900 replica scrub(pg: 6.15,from:0'0,to:461'106915,epoch:494/492,start:6:a9ec3fac:::benchmark_data_magna108.ceph.redhat.com_2478_object280060:0,end:6:a9ed5205:::benchmark_data_magna108_35617_object484059:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -4546> 2018-07-11 07:34:11.839942 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5126 ==== replica scrub(pg: 6.15,from:0'0,to:461'106915,epoch:494/492,start:6:a9ec3fac:::benchmark_data_magna108.ceph.redhat.com_2478_object280060:0,end:6:a9ed5205:::benchmark_data_magna108_35617_object484059:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (554794835 0 0) 0x55a31367f900 con 0x55a317019000 > -4545> 2018-07-11 07:34:12.050658 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.115:0/13323 conn(0x55a321665800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1856 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:34:12.388165) v4 > -4544> 2018-07-11 07:34:12.050682 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.6 10.8.128.115:0/13323 1856 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:12.388165) v4 ==== 2004+0+0 (3916880785 0 0) 0x55a324474c00 con 0x55a321665800 > -4543> 2018-07-11 07:34:12.050697 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:12.388165) v4 -- 0x55a34de5a800 con 0 > -4542> 2018-07-11 07:34:12.050824 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.115:0/13323 conn(0x55a321668800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1856 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:34:12.388165) v4 > -4541> 2018-07-11 07:34:12.050841 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.6 10.8.128.115:0/13323 1856 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:12.388165) v4 ==== 2004+0+0 (3916880785 0 0) 0x55a332e79a00 con 0x55a321668800 > -4540> 2018-07-11 07:34:12.050851 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:12.388165) v4 -- 0x55a34de5a800 con 0 > -4539> 2018-07-11 07:34:12.254600 7f35488a3700 1 -- 10.8.128.107:6800/12665 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55a32a5c5b80 con 0 > -4538> 2018-07-11 07:34:12.254673 7f35488a3700 1 -- 10.8.128.107:6800/12665 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55a32ebb9340 con 0 > -4537> 2018-07-11 07:34:12.840578 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a33b929440 con 0 > -4536> 2018-07-11 07:34:12.847609 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5127 0x55a31367fb80 replica scrub(pg: 6.15,from:0'0,to:430'105954,epoch:494/492,start:6:a9ed5205:::benchmark_data_magna108_35617_object484059:0,end:6:a9eea8ee:::benchmark_data_magna108_35617_object435400:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -4535> 2018-07-11 07:34:12.847641 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5127 ==== replica scrub(pg: 6.15,from:0'0,to:430'105954,epoch:494/492,start:6:a9ed5205:::benchmark_data_magna108_35617_object484059:0,end:6:a9eea8ee:::benchmark_data_magna108_35617_object435400:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (2187028504 0 0) 0x55a31367fb80 con 0x55a317019000 > -4534> 2018-07-11 07:34:12.968409 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15885 conn(0x55a311b76800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1867 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:34:13.592265) v4 > -4533> 2018-07-11 07:34:12.968413 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15885 conn(0x55a3170b5800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1867 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:34:13.592265) v4 > -4532> 2018-07-11 07:34:12.968436 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.7 10.8.128.114:0/15885 1867 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:13.592265) v4 ==== 2004+0+0 (4029792358 0 0) 0x55a34de78600 con 0x55a311b76800 > -4531> 2018-07-11 07:34:12.968441 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.7 10.8.128.114:0/15885 1867 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:13.592265) v4 ==== 2004+0+0 (4029792358 0 0) 0x55a324474c00 con 0x55a3170b5800 > -4530> 2018-07-11 07:34:12.968448 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:13.592265) v4 -- 0x55a32e457c00 con 0 > -4529> 2018-07-11 07:34:12.968475 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:13.592265) v4 -- 0x55a34de5a800 con 0 > -4528> 2018-07-11 07:34:13.029924 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15287 conn(0x55a311b72000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1841 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:34:14.243904) v4 > -4527> 2018-07-11 07:34:13.029948 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.4 10.8.128.113:0/15287 1841 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:14.243904) v4 ==== 2004+0+0 (2071608602 0 0) 0x55a332e79a00 con 0x55a311b72000 > -4526> 2018-07-11 07:34:13.029957 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:14.243904) v4 -- 0x55a34de5a800 con 0 > -4525> 2018-07-11 07:34:13.029966 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15287 conn(0x55a311b75000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1841 0x55a34dea9e00 osd_ping(ping e494 stamp 2018-07-11 07:34:14.243904) v4 > -4524> 2018-07-11 07:34:13.029988 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.4 10.8.128.113:0/15287 1841 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:14.243904) v4 ==== 2004+0+0 (2071608602 0 0) 0x55a34dea9e00 con 0x55a311b75000 > -4523> 2018-07-11 07:34:13.030006 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:14.243904) v4 -- 0x55a32447a200 con 0 > -4522> 2018-07-11 07:34:13.151638 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b7000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1795 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:34:12.902302) v4 > -4521> 2018-07-11 07:34:13.151645 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b8800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1795 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:34:12.902302) v4 > -4520> 2018-07-11 07:34:13.151662 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.8 10.8.128.108:0/11967 1795 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:12.902302) v4 ==== 2004+0+0 (3907695462 0 0) 0x55a324474c00 con 0x55a3170b8800 > -4519> 2018-07-11 07:34:13.151662 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.8 10.8.128.108:0/11967 1795 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:12.902302) v4 ==== 2004+0+0 (3907695462 0 0) 0x55a34de6f000 con 0x55a3170b7000 > -4518> 2018-07-11 07:34:13.151671 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:12.902302) v4 -- 0x55a32442b200 con 0 > -4517> 2018-07-11 07:34:13.151708 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:12.902302) v4 -- 0x55a32442b400 con 0 > -4516> 2018-07-11 07:34:13.319894 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15263 conn(0x55a321664000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1843 0x55a34de84c00 osd_ping(ping e494 stamp 2018-07-11 07:34:14.533748) v4 > -4515> 2018-07-11 07:34:13.319919 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.5 10.8.128.113:0/15263 1843 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:14.533748) v4 ==== 2004+0+0 (4204618261 0 0) 0x55a34de84c00 con 0x55a321664000 > -4514> 2018-07-11 07:34:13.319933 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:14.533748) v4 -- 0x55a33234a600 con 0 > -4513> 2018-07-11 07:34:13.320017 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15263 conn(0x55a321667000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1843 0x55a34dea9e00 osd_ping(ping e494 stamp 2018-07-11 07:34:14.533748) v4 > -4512> 2018-07-11 07:34:13.320037 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.5 10.8.128.113:0/15263 1843 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:14.533748) v4 ==== 2004+0+0 (4204618261 0 0) 0x55a34dea9e00 con 0x55a321667000 > -4511> 2018-07-11 07:34:13.320048 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:14.533748) v4 -- 0x55a32eb3bc00 con 0 > -4510> 2018-07-11 07:34:13.588612 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a324f1c6c0 con 0 > -4509> 2018-07-11 07:34:13.597618 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5128 0x55a34982c000 replica scrub(pg: 6.15,from:0'0,to:430'106351,epoch:494/492,start:6:a9eea8ee:::benchmark_data_magna108_35617_object435400:0,end:6:a9ef7695:::benchmark_data_magna108_35617_object460742:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -4508> 2018-07-11 07:34:13.597647 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5128 ==== replica scrub(pg: 6.15,from:0'0,to:430'106351,epoch:494/492,start:6:a9eea8ee:::benchmark_data_magna108_35617_object435400:0,end:6:a9ef7695:::benchmark_data_magna108_35617_object460742:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (2540634817 0 0) 0x55a34982c000 con 0x55a317019000 > -4507> 2018-07-11 07:34:14.233774 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a324f22480 con 0 > -4506> 2018-07-11 07:34:14.248319 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5129 0x55a34982c280 replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:494/492,start:6:a9ef7695:::benchmark_data_magna108_35617_object460742:0,end:6:a9f067f5:::benchmark_data_magna108.ceph.redhat.com_2478_object384933:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -4505> 2018-07-11 07:34:14.248342 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5129 ==== replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:494/492,start:6:a9ef7695:::benchmark_data_magna108_35617_object460742:0,end:6:a9f067f5:::benchmark_data_magna108.ceph.redhat.com_2478_object384933:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (2841662997 0 0) 0x55a34982c280 con 0x55a317019000 > -4504> 2018-07-11 07:34:14.378217 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12650 conn(0x55a311b67000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1861 0x55a34de84c00 osd_ping(ping e494 stamp 2018-07-11 07:34:14.413996) v4 > -4503> 2018-07-11 07:34:14.378239 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.3 10.8.128.106:0/12650 1861 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:14.413996) v4 ==== 2004+0+0 (2551250718 0 0) 0x55a34de84c00 con 0x55a311b67000 > -4502> 2018-07-11 07:34:14.378229 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12650 conn(0x55a311b68800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1861 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:34:14.413996) v4 > -4501> 2018-07-11 07:34:14.378257 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:14.413996) v4 -- 0x55a32eb20e00 con 0 > -4500> 2018-07-11 07:34:14.378272 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.3 10.8.128.106:0/12650 1861 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:14.413996) v4 ==== 2004+0+0 (2551250718 0 0) 0x55a34de78600 con 0x55a311b68800 > -4499> 2018-07-11 07:34:14.378297 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:14.413996) v4 -- 0x55a32442b400 con 0 > -4498> 2018-07-11 07:34:14.730369 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15287 conn(0x55a311b72000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1842 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:34:15.944377) v4 > -4497> 2018-07-11 07:34:14.730395 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.4 10.8.128.113:0/15287 1842 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:15.944377) v4 ==== 2004+0+0 (1113684758 0 0) 0x55a332e79a00 con 0x55a311b72000 > -4496> 2018-07-11 07:34:14.730406 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:15.944377) v4 -- 0x55a3170d0200 con 0 > -4495> 2018-07-11 07:34:14.730401 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15287 conn(0x55a311b75000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1842 0x55a32eb3bc00 osd_ping(ping e494 stamp 2018-07-11 07:34:15.944377) v4 > -4494> 2018-07-11 07:34:14.730420 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.4 10.8.128.113:0/15287 1842 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:15.944377) v4 ==== 2004+0+0 (1113684758 0 0) 0x55a32eb3bc00 con 0x55a311b75000 > -4493> 2018-07-11 07:34:14.730437 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:15.944377) v4 -- 0x55a332eaca00 con 0 > -4492> 2018-07-11 07:34:14.951131 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.115:0/13323 conn(0x55a321665800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1857 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:34:15.288643) v4 > -4491> 2018-07-11 07:34:14.951158 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.6 10.8.128.115:0/13323 1857 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:15.288643) v4 ==== 2004+0+0 (792593691 0 0) 0x55a324474c00 con 0x55a321665800 > -4490> 2018-07-11 07:34:14.951174 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:15.288643) v4 -- 0x55a31707da00 con 0 > -4489> 2018-07-11 07:34:14.951229 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.115:0/13323 conn(0x55a321668800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1857 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:34:15.288643) v4 > -4488> 2018-07-11 07:34:14.951263 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.6 10.8.128.115:0/13323 1857 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:15.288643) v4 ==== 2004+0+0 (792593691 0 0) 0x55a324474c00 con 0x55a321668800 > -4487> 2018-07-11 07:34:14.951277 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:15.288643) v4 -- 0x55a34dec7a00 con 0 > -4486> 2018-07-11 07:34:15.170654 7f3543899700 5 osd.10 494 heartbeat: osd_stat(784 GB used, 147 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,0,0,1]) > -4485> 2018-07-11 07:34:15.170668 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6802/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:34:15.170667) v4 -- 0x55a34dea9400 con 0 > -4484> 2018-07-11 07:34:15.170703 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6803/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:34:15.170667) v4 -- 0x55a332e77800 con 0 > -4483> 2018-07-11 07:34:15.170718 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6806/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:34:15.170667) v4 -- 0x55a32eb14000 con 0 > -4482> 2018-07-11 07:34:15.170733 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6807/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:34:15.170667) v4 -- 0x55a324474200 con 0 > -4481> 2018-07-11 07:34:15.170757 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6808/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:34:15.170667) v4 -- 0x55a332e76a00 con 0 > -4480> 2018-07-11 07:34:15.170773 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6810/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:34:15.170667) v4 -- 0x55a332eac400 con 0 > -4479> 2018-07-11 07:34:15.170787 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6809/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:34:15.170667) v4 -- 0x55a328a50600 con 0 > -4478> 2018-07-11 07:34:15.170799 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6811/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:34:15.170667) v4 -- 0x55a332e84800 con 0 > -4477> 2018-07-11 07:34:15.170811 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6810/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:34:15.170667) v4 -- 0x55a34de63000 con 0 > -4476> 2018-07-11 07:34:15.170827 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6811/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:34:15.170667) v4 -- 0x55a324474800 con 0 > -4475> 2018-07-11 07:34:15.170846 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6806/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:34:15.170667) v4 -- 0x55a34de63200 con 0 > -4474> 2018-07-11 07:34:15.170861 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6807/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:34:15.170667) v4 -- 0x55a34dec7400 con 0 > -4473> 2018-07-11 07:34:15.170873 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6802/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:34:15.170667) v4 -- 0x55a32442a400 con 0 > -4472> 2018-07-11 07:34:15.170889 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6803/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:34:15.170667) v4 -- 0x55a34dea9c00 con 0 > -4471> 2018-07-11 07:34:15.170904 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6806/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:34:15.170667) v4 -- 0x55a32eb29000 con 0 > -4470> 2018-07-11 07:34:15.170916 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6807/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:34:15.170667) v4 -- 0x55a32eb29e00 con 0 > -4469> 2018-07-11 07:34:15.170928 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6802/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:34:15.170667) v4 -- 0x55a32eb28200 con 0 > -4468> 2018-07-11 07:34:15.170940 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6803/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:34:15.170667) v4 -- 0x55a332e76e00 con 0 > -4467> 2018-07-11 07:34:15.170953 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6806/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:34:15.170667) v4 -- 0x55a332e79400 con 0 > -4466> 2018-07-11 07:34:15.170966 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6807/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:34:15.170667) v4 -- 0x55a34de85400 con 0 > -4465> 2018-07-11 07:34:15.170982 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6802/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:34:15.170667) v4 -- 0x55a34de6e800 con 0 > -4464> 2018-07-11 07:34:15.170994 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6803/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:34:15.170667) v4 -- 0x55a332e76c00 con 0 > -4463> 2018-07-11 07:34:15.171153 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6807/12617 conn(0x55a314e27800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1851 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:15.170667) v4 > -4462> 2018-07-11 07:34:15.171176 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6807/12617 1851 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:15.170667) v4 ==== 2004+0+0 (313116926 0 0) 0x55a34de6f000 con 0x55a314e27800 > -4461> 2018-07-11 07:34:15.171176 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6806/12617 conn(0x55a314e26000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1851 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:15.170667) v4 > -4460> 2018-07-11 07:34:15.171196 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6806/12617 1851 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:15.170667) v4 ==== 2004+0+0 (313116926 0 0) 0x55a324474c00 con 0x55a314e26000 > -4459> 2018-07-11 07:34:15.171236 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6803/15255 conn(0x55a314e1d800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1851 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:15.170667) v4 > -4458> 2018-07-11 07:34:15.171259 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6803/15255 1851 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:15.170667) v4 ==== 2004+0+0 (313116926 0 0) 0x55a324474c00 con 0x55a314e1d800 > -4457> 2018-07-11 07:34:15.171270 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6807/12677 conn(0x55a314e85800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1851 0x55a34de84c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:15.170667) v4 > -4456> 2018-07-11 07:34:15.171290 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6807/12677 1851 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:15.170667) v4 ==== 2004+0+0 (313116926 0 0) 0x55a34de84c00 con 0x55a314e85800 > -4455> 2018-07-11 07:34:15.171291 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6806/12677 conn(0x55a314e84000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1851 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:15.170667) v4 > -4454> 2018-07-11 07:34:15.171306 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6810/12651 conn(0x55a314e97800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1851 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:15.170667) v4 > -4453> 2018-07-11 07:34:15.171318 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6810/12651 1851 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:15.170667) v4 ==== 2004+0+0 (313116926 0 0) 0x55a324474c00 con 0x55a314e97800 > -4452> 2018-07-11 07:34:15.171315 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6806/12677 1851 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:15.170667) v4 ==== 2004+0+0 (313116926 0 0) 0x55a34de6f000 con 0x55a314e84000 > -4451> 2018-07-11 07:34:15.171339 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6802/15255 conn(0x55a314e1c000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1851 0x55a34de84c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:15.170667) v4 > -4450> 2018-07-11 07:34:15.171352 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6802/15255 1851 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:15.170667) v4 ==== 2004+0+0 (313116926 0 0) 0x55a34de84c00 con 0x55a314e1c000 > -4449> 2018-07-11 07:34:15.171356 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6811/15287 conn(0x55a314e04800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1851 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:15.170667) v4 > -4448> 2018-07-11 07:34:15.171375 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6811/15287 1851 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:15.170667) v4 ==== 2004+0+0 (313116926 0 0) 0x55a324474c00 con 0x55a314e04800 > -4447> 2018-07-11 07:34:15.171378 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6808/12651 conn(0x55a314e96000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1851 0x55a34de84c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:15.170667) v4 > -4446> 2018-07-11 07:34:15.171394 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6808/12651 1851 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:15.170667) v4 ==== 2004+0+0 (313116926 0 0) 0x55a34de84c00 con 0x55a314e96000 > -4445> 2018-07-11 07:34:15.171394 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6809/12650 conn(0x55a314e1f000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1851 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:15.170667) v4 > -4444> 2018-07-11 07:34:15.171414 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6809/12650 1851 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:15.170667) v4 ==== 2004+0+0 (313116926 0 0) 0x55a34de6f000 con 0x55a314e1f000 > -4443> 2018-07-11 07:34:15.171416 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6802/13323 conn(0x55a314e15000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1851 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:15.170667) v4 > -4442> 2018-07-11 07:34:15.171427 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6802/13323 1851 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:15.170667) v4 ==== 2004+0+0 (313116926 0 0) 0x55a324474c00 con 0x55a314e15000 > -4441> 2018-07-11 07:34:15.171425 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6811/12650 conn(0x55a314e20800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1851 0x55a34de84c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:15.170667) v4 > -4440> 2018-07-11 07:34:15.171444 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6811/12650 1851 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:15.170667) v4 ==== 2004+0+0 (313116926 0 0) 0x55a34de84c00 con 0x55a314e20800 > -4439> 2018-07-11 07:34:15.171464 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6806/15263 conn(0x55a314e99000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1851 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:15.170667) v4 > -4438> 2018-07-11 07:34:15.171478 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6810/15287 conn(0x55a314e03000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1851 0x55a34de84c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:15.170667) v4 > -4437> 2018-07-11 07:34:15.171482 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6806/15263 1851 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:15.170667) v4 ==== 2004+0+0 (313116926 0 0) 0x55a34de6f000 con 0x55a314e99000 > -4436> 2018-07-11 07:34:15.171491 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6810/15287 1851 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:15.170667) v4 ==== 2004+0+0 (313116926 0 0) 0x55a34de84c00 con 0x55a314e03000 > -4435> 2018-07-11 07:34:15.171514 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6802/15919 conn(0x55a314e87000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1851 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:15.170667) v4 > -4434> 2018-07-11 07:34:15.171525 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6802/11967 conn(0x55a314e12000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1851 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:15.170667) v4 > -4433> 2018-07-11 07:34:15.171529 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6807/15263 conn(0x55a314e9a800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1851 0x55a34de84c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:15.170667) v4 > -4432> 2018-07-11 07:34:15.171532 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6802/15919 1851 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:15.170667) v4 ==== 2004+0+0 (313116926 0 0) 0x55a324474c00 con 0x55a314e87000 > -4431> 2018-07-11 07:34:15.171544 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6802/11967 1851 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:15.170667) v4 ==== 2004+0+0 (313116926 0 0) 0x55a34de6f000 con 0x55a314e12000 > -4430> 2018-07-11 07:34:15.171548 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6807/15263 1851 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:15.170667) v4 ==== 2004+0+0 (313116926 0 0) 0x55a34de84c00 con 0x55a314e9a800 > -4429> 2018-07-11 07:34:15.171566 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6806/15885 conn(0x55a314e00000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1851 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:15.170667) v4 > -4428> 2018-07-11 07:34:15.171578 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6806/15885 1851 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:15.170667) v4 ==== 2004+0+0 (313116926 0 0) 0x55a324474c00 con 0x55a314e00000 > -4427> 2018-07-11 07:34:15.171606 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6803/11967 conn(0x55a314e13800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1851 0x55a34de84c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:15.170667) v4 > -4426> 2018-07-11 07:34:15.171608 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6803/13323 conn(0x55a314e16800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1851 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:15.170667) v4 > -4425> 2018-07-11 07:34:15.171619 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6803/11967 1851 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:15.170667) v4 ==== 2004+0+0 (313116926 0 0) 0x55a34de84c00 con 0x55a314e13800 > -4424> 2018-07-11 07:34:15.171621 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6803/13323 1851 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:15.170667) v4 ==== 2004+0+0 (313116926 0 0) 0x55a34de6f000 con 0x55a314e16800 > -4423> 2018-07-11 07:34:15.171662 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6803/15919 conn(0x55a314e88800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1851 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:15.170667) v4 > -4422> 2018-07-11 07:34:15.171675 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6803/15919 1851 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:15.170667) v4 ==== 2004+0+0 (313116926 0 0) 0x55a34de6f000 con 0x55a314e88800 > -4421> 2018-07-11 07:34:15.171715 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6807/15885 conn(0x55a314e01800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1851 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:15.170667) v4 > -4420> 2018-07-11 07:34:15.171729 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6807/15885 1851 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:15.170667) v4 ==== 2004+0+0 (313116926 0 0) 0x55a34de6f000 con 0x55a314e01800 > -4419> 2018-07-11 07:34:15.265704 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12651 conn(0x55a32169f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1846 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:34:15.301553) v4 > -4418> 2018-07-11 07:34:15.265730 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.2 10.8.128.106:0/12651 1846 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:15.301553) v4 ==== 2004+0+0 (2755466931 0 0) 0x55a324474c00 con 0x55a32169f000 > -4417> 2018-07-11 07:34:15.265740 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:15.301553) v4 -- 0x55a332eac400 con 0 > -4416> 2018-07-11 07:34:15.265727 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12651 conn(0x55a3216a0800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1846 0x55a34de84c00 osd_ping(ping e494 stamp 2018-07-11 07:34:15.301553) v4 > -4415> 2018-07-11 07:34:15.265752 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.2 10.8.128.106:0/12651 1846 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:15.301553) v4 ==== 2004+0+0 (2755466931 0 0) 0x55a34de84c00 con 0x55a3216a0800 > -4414> 2018-07-11 07:34:15.265769 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:15.301553) v4 -- 0x55a324474200 con 0 > -4413> 2018-07-11 07:34:15.268903 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15885 conn(0x55a3170b5800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1868 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:34:15.892726) v4 > -4412> 2018-07-11 07:34:15.268908 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15885 conn(0x55a311b76800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1868 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:34:15.892726) v4 > -4411> 2018-07-11 07:34:15.268925 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.7 10.8.128.114:0/15885 1868 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:15.892726) v4 ==== 2004+0+0 (2981183385 0 0) 0x55a34de6f000 con 0x55a311b76800 > -4410> 2018-07-11 07:34:15.268924 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.7 10.8.128.114:0/15885 1868 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:15.892726) v4 ==== 2004+0+0 (2981183385 0 0) 0x55a332e79a00 con 0x55a3170b5800 > -4409> 2018-07-11 07:34:15.268943 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:15.892726) v4 -- 0x55a32eb14000 con 0 > -4408> 2018-07-11 07:34:15.268964 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:15.892726) v4 -- 0x55a332eac400 con 0 > -4407> 2018-07-11 07:34:15.521885 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 59] Generated table #36583: 4957169 keys, 80348275 bytes > -4406> 2018-07-11 07:34:15.521941 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294455521912, "cf_name": "default", "job": 59, "event": "table_file_creation", "file_number": 36583, "file_size": 80348275, "table_properties": {"data_size": 67109395, "index_size": 689177, "filter_size": 12548713, "raw_key_size": 143207113, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957169, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957169", "kMergeOperands": "0"}} > -4405> 2018-07-11 07:34:15.629332 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12677 conn(0x55a32169d800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1866 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:34:15.665216) v4 > -4404> 2018-07-11 07:34:15.629356 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.1 10.8.128.106:0/12677 1866 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:15.665216) v4 ==== 2004+0+0 (1485289191 0 0) 0x55a34de78600 con 0x55a32169d800 > -4403> 2018-07-11 07:34:15.629369 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:15.665216) v4 -- 0x55a32eb14000 con 0 > -4402> 2018-07-11 07:34:15.629361 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12677 conn(0x55a32169c000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1866 0x55a32eb3bc00 osd_ping(ping e494 stamp 2018-07-11 07:34:15.665216) v4 > -4401> 2018-07-11 07:34:15.629379 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.1 10.8.128.106:0/12677 1866 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:15.665216) v4 ==== 2004+0+0 (1485289191 0 0) 0x55a32eb3bc00 con 0x55a32169c000 > -4400> 2018-07-11 07:34:15.629392 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:15.665216) v4 -- 0x55a324474200 con 0 > -4399> 2018-07-11 07:34:15.844688 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a32eefd8c0 con 0 > -4398> 2018-07-11 07:34:15.857520 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15919 conn(0x55a317074800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1849 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:34:16.481382) v4 > -4397> 2018-07-11 07:34:15.857545 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.11 10.8.128.114:0/15919 1849 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:16.481382) v4 ==== 2004+0+0 (2851034169 0 0) 0x55a324474c00 con 0x55a317074800 > -4396> 2018-07-11 07:34:15.857562 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:16.481382) v4 -- 0x55a332eac400 con 0 > -4395> 2018-07-11 07:34:15.857590 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15919 conn(0x55a3170b4000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1849 0x55a32eb3bc00 osd_ping(ping e494 stamp 2018-07-11 07:34:16.481382) v4 > -4394> 2018-07-11 07:34:15.857616 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.11 10.8.128.114:0/15919 1849 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:16.481382) v4 ==== 2004+0+0 (2851034169 0 0) 0x55a32eb3bc00 con 0x55a3170b4000 > -4393> 2018-07-11 07:34:15.857631 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:16.481382) v4 -- 0x55a324474200 con 0 > -4392> 2018-07-11 07:34:15.869868 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5130 0x55a34982c500 replica scrub(pg: 6.15,from:0'0,to:461'106916,epoch:494/492,start:6:a9f067f5:::benchmark_data_magna108.ceph.redhat.com_2478_object384933:0,end:6:a9f1b428:::benchmark_data_magna108_35617_object475198:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -4391> 2018-07-11 07:34:15.869896 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5130 ==== replica scrub(pg: 6.15,from:0'0,to:461'106916,epoch:494/492,start:6:a9f067f5:::benchmark_data_magna108.ceph.redhat.com_2478_object384933:0,end:6:a9f1b428:::benchmark_data_magna108_35617_object475198:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (4155613625 0 0) 0x55a34982c500 con 0x55a317019000 > -4390> 2018-07-11 07:34:15.879871 7f355d88e700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.107:0/12617 conn(0x55a32447f800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1819 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:34:15.879397) v4 > -4389> 2018-07-11 07:34:15.879882 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.107:0/12617 conn(0x55a32447e000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1819 0x55a32eb3bc00 osd_ping(ping e494 stamp 2018-07-11 07:34:15.879397) v4 > -4388> 2018-07-11 07:34:15.879893 7f355d88e700 1 -- 10.8.128.107:6803/12665 <== osd.9 10.8.128.107:0/12617 1819 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:15.879397) v4 ==== 2004+0+0 (1862892521 0 0) 0x55a34de6f000 con 0x55a32447f800 > -4387> 2018-07-11 07:34:15.879911 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.9 10.8.128.107:0/12617 1819 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:15.879397) v4 ==== 2004+0+0 (1862892521 0 0) 0x55a32eb3bc00 con 0x55a32447e000 > -4386> 2018-07-11 07:34:15.879919 7f355d88e700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:15.879397) v4 -- 0x55a32eb14000 con 0 > -4385> 2018-07-11 07:34:15.880039 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:15.879397) v4 -- 0x55a324474200 con 0 > -4384> 2018-07-11 07:34:16.128915 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15255 conn(0x55a311b65800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1870 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:34:17.342960) v4 > -4383> 2018-07-11 07:34:16.128937 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.0 10.8.128.113:0/15255 1870 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:17.342960) v4 ==== 2004+0+0 (4238829345 0 0) 0x55a332e79a00 con 0x55a311b65800 > -4382> 2018-07-11 07:34:16.128948 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:17.342960) v4 -- 0x55a332eac400 con 0 > -4381> 2018-07-11 07:34:16.128975 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15255 conn(0x55a311b64000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1870 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:34:17.342960) v4 > -4380> 2018-07-11 07:34:16.128998 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.0 10.8.128.113:0/15255 1870 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:17.342960) v4 ==== 2004+0+0 (4238829345 0 0) 0x55a34de78600 con 0x55a311b64000 > -4379> 2018-07-11 07:34:16.129020 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:17.342960) v4 -- 0x55a32eb14000 con 0 > -4378> 2018-07-11 07:34:16.380471 7f355d88e700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.107:0/12617 conn(0x55a32447f800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1820 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:34:16.379985) v4 > -4377> 2018-07-11 07:34:16.380469 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.107:0/12617 conn(0x55a32447e000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1820 0x55a34de84c00 osd_ping(ping e494 stamp 2018-07-11 07:34:16.379985) v4 > -4376> 2018-07-11 07:34:16.380497 7f355d88e700 1 -- 10.8.128.107:6803/12665 <== osd.9 10.8.128.107:0/12617 1820 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:16.379985) v4 ==== 2004+0+0 (2418981101 0 0) 0x55a34de6f000 con 0x55a32447f800 > -4375> 2018-07-11 07:34:16.380502 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.9 10.8.128.107:0/12617 1820 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:16.379985) v4 ==== 2004+0+0 (2418981101 0 0) 0x55a34de84c00 con 0x55a32447e000 > -4374> 2018-07-11 07:34:16.380509 7f355d88e700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:16.379985) v4 -- 0x55a32eb14000 con 0 > -4373> 2018-07-11 07:34:16.380534 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:16.379985) v4 -- 0x55a324474200 con 0 > -4372> 2018-07-11 07:34:16.424179 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 59] Generated table #36584: 717170 keys, 11624640 bytes > -4371> 2018-07-11 07:34:16.424221 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294456424194, "cf_name": "default", "job": 59, "event": "table_file_creation", "file_number": 36584, "file_size": 11624640, "table_properties": {"data_size": 9708922, "index_size": 99269, "filter_size": 1815464, "raw_key_size": 20718243, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 2383, "num_entries": 717170, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "717170", "kMergeOperands": "0"}} > -4370> 2018-07-11 07:34:16.430678 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15287 conn(0x55a311b72000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1843 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:34:17.644753) v4 > -4369> 2018-07-11 07:34:16.430695 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15287 conn(0x55a311b75000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1843 0x55a32eb3bc00 osd_ping(ping e494 stamp 2018-07-11 07:34:17.644753) v4 > -4368> 2018-07-11 07:34:16.430706 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.4 10.8.128.113:0/15287 1843 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:17.644753) v4 ==== 2004+0+0 (3142724335 0 0) 0x55a324474c00 con 0x55a311b72000 > -4367> 2018-07-11 07:34:16.430716 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:17.644753) v4 -- 0x55a332e77800 con 0 > -4366> 2018-07-11 07:34:16.430714 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.4 10.8.128.113:0/15287 1843 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:17.644753) v4 ==== 2004+0+0 (3142724335 0 0) 0x55a32eb3bc00 con 0x55a311b75000 > -4365> 2018-07-11 07:34:16.430744 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:17.644753) v4 -- 0x55a34dea9400 con 0 > -4364> 2018-07-11 07:34:16.651604 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.115:0/13323 conn(0x55a321665800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1858 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:34:16.989112) v4 > -4363> 2018-07-11 07:34:16.651625 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.6 10.8.128.115:0/13323 1858 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:16.989112) v4 ==== 2004+0+0 (734198695 0 0) 0x55a332e79a00 con 0x55a321665800 > -4362> 2018-07-11 07:34:16.651635 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:16.989112) v4 -- 0x55a34dec7a00 con 0 > -4361> 2018-07-11 07:34:16.651672 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.115:0/13323 conn(0x55a321668800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1858 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:34:16.989112) v4 > -4360> 2018-07-11 07:34:16.651685 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.6 10.8.128.115:0/13323 1858 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:16.989112) v4 ==== 2004+0+0 (734198695 0 0) 0x55a332e79a00 con 0x55a321668800 > -4359> 2018-07-11 07:34:16.651694 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:16.989112) v4 -- 0x55a32e456200 con 0 > -4358> 2018-07-11 07:34:16.652180 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b7000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1796 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:34:16.402805) v4 > -4357> 2018-07-11 07:34:16.652192 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b8800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1796 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:34:16.402805) v4 > -4356> 2018-07-11 07:34:16.652200 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.8 10.8.128.108:0/11967 1796 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:16.402805) v4 ==== 2004+0+0 (899909147 0 0) 0x55a34de78600 con 0x55a3170b7000 > -4355> 2018-07-11 07:34:16.652205 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.8 10.8.128.108:0/11967 1796 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:16.402805) v4 ==== 2004+0+0 (899909147 0 0) 0x55a324474c00 con 0x55a3170b8800 > -4354> 2018-07-11 07:34:16.652210 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:16.402805) v4 -- 0x55a32442b400 con 0 > -4353> 2018-07-11 07:34:16.652231 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:16.402805) v4 -- 0x55a34dec7a00 con 0 > -4352> 2018-07-11 07:34:16.744696 7f35498a5700 10 monclient: tick > -4351> 2018-07-11 07:34:16.744710 7f35498a5700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2018-07-11 07:33:46.744708) > -4350> 2018-07-11 07:34:16.813687 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a324f44b40 con 0 > -4349> 2018-07-11 07:34:16.831644 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5131 0x55a34982c780 replica scrub(pg: 6.15,from:0'0,to:430'105840,epoch:494/492,start:6:a9f1b428:::benchmark_data_magna108_35617_object475198:0,end:6:a9f2ae1c:::benchmark_data_magna108.ceph.redhat.com_2478_object247581:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -4348> 2018-07-11 07:34:16.831680 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5131 ==== replica scrub(pg: 6.15,from:0'0,to:430'105840,epoch:494/492,start:6:a9f1b428:::benchmark_data_magna108_35617_object475198:0,end:6:a9f2ae1c:::benchmark_data_magna108.ceph.redhat.com_2478_object247581:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (2719954421 0 0) 0x55a34982c780 con 0x55a317019000 > -4347> 2018-07-11 07:34:16.871133 7f3543899700 5 osd.10 494 heartbeat: osd_stat(784 GB used, 146 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,1]) > -4346> 2018-07-11 07:34:16.871147 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6802/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:34:16.871146) v4 -- 0x55a332fae000 con 0 > -4345> 2018-07-11 07:34:16.871184 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6803/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:34:16.871146) v4 -- 0x55a332ff3400 con 0 > -4344> 2018-07-11 07:34:16.871201 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6806/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:34:16.871146) v4 -- 0x55a332e76400 con 0 > -4343> 2018-07-11 07:34:16.871215 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6807/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:34:16.871146) v4 -- 0x55a34de6e200 con 0 > -4342> 2018-07-11 07:34:16.871228 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6808/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:34:16.871146) v4 -- 0x55a33234ee00 con 0 > -4341> 2018-07-11 07:34:16.871238 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6810/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:34:16.871146) v4 -- 0x55a33532de00 con 0 > -4340> 2018-07-11 07:34:16.871266 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6809/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:34:16.871146) v4 -- 0x55a34de6f600 con 0 > -4339> 2018-07-11 07:34:16.871281 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6811/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:34:16.871146) v4 -- 0x55a34de6fa00 con 0 > -4338> 2018-07-11 07:34:16.871307 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6810/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:34:16.871146) v4 -- 0x55a332fafc00 con 0 > -4337> 2018-07-11 07:34:16.871320 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6811/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:34:16.871146) v4 -- 0x55a32447ae00 con 0 > -4336> 2018-07-11 07:34:16.871336 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6806/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:34:16.871146) v4 -- 0x55a32442aa00 con 0 > -4335> 2018-07-11 07:34:16.871350 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6807/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:34:16.871146) v4 -- 0x55a33235ae00 con 0 > -4334> 2018-07-11 07:34:16.871369 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6802/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:34:16.871146) v4 -- 0x55a32447b400 con 0 > -4333> 2018-07-11 07:34:16.871382 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6803/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:34:16.871146) v4 -- 0x55a332354c00 con 0 > -4332> 2018-07-11 07:34:16.871395 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6806/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:34:16.871146) v4 -- 0x55a34de78e00 con 0 > -4331> 2018-07-11 07:34:16.871409 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6807/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:34:16.871146) v4 -- 0x55a330935600 con 0 > -4330> 2018-07-11 07:34:16.871426 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6802/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:34:16.871146) v4 -- 0x55a32442ac00 con 0 > -4329> 2018-07-11 07:34:16.871438 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6803/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:34:16.871146) v4 -- 0x55a32eb21c00 con 0 > -4328> 2018-07-11 07:34:16.871450 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6806/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:34:16.871146) v4 -- 0x55a34de84600 con 0 > -4327> 2018-07-11 07:34:16.871463 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6807/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:34:16.871146) v4 -- 0x55a30f329600 con 0 > -4326> 2018-07-11 07:34:16.871482 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6802/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:34:16.871146) v4 -- 0x55a32eb21e00 con 0 > -4325> 2018-07-11 07:34:16.871495 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6803/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:34:16.871146) v4 -- 0x55a32aa0fe00 con 0 > -4324> 2018-07-11 07:34:16.871608 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6802/15255 conn(0x55a314e1c000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1852 0x55a32eb3bc00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:16.871146) v4 > -4323> 2018-07-11 07:34:16.871629 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6802/15255 1852 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:16.871146) v4 ==== 2004+0+0 (2871545206 0 0) 0x55a32eb3bc00 con 0x55a314e1c000 > -4322> 2018-07-11 07:34:16.871639 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6806/12617 conn(0x55a314e26000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1852 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:16.871146) v4 > -4321> 2018-07-11 07:34:16.871661 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6806/12617 1852 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:16.871146) v4 ==== 2004+0+0 (2871545206 0 0) 0x55a324474c00 con 0x55a314e26000 > -4320> 2018-07-11 07:34:16.871677 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6807/12617 conn(0x55a314e27800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1852 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:16.871146) v4 > -4319> 2018-07-11 07:34:16.871683 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6807/12677 conn(0x55a314e85800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1852 0x55a32eb3bc00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:16.871146) v4 > -4318> 2018-07-11 07:34:16.871694 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6807/12617 1852 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:16.871146) v4 ==== 2004+0+0 (2871545206 0 0) 0x55a34de6f000 con 0x55a314e27800 > -4317> 2018-07-11 07:34:16.871701 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6807/12677 1852 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:16.871146) v4 ==== 2004+0+0 (2871545206 0 0) 0x55a32eb3bc00 con 0x55a314e85800 > -4316> 2018-07-11 07:34:16.871732 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6809/12650 conn(0x55a314e1f000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1852 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:16.871146) v4 > -4315> 2018-07-11 07:34:16.871720 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6811/15287 conn(0x55a314e04800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1852 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:16.871146) v4 > -4314> 2018-07-11 07:34:16.871746 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6809/12650 1852 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:16.871146) v4 ==== 2004+0+0 (2871545206 0 0) 0x55a34de6f000 con 0x55a314e1f000 > -4313> 2018-07-11 07:34:16.871746 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6808/12651 conn(0x55a314e96000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1852 0x55a32eb3bc00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:16.871146) v4 > -4312> 2018-07-11 07:34:16.871755 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6811/15287 1852 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:16.871146) v4 ==== 2004+0+0 (2871545206 0 0) 0x55a324474c00 con 0x55a314e04800 > -4311> 2018-07-11 07:34:16.871765 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6808/12651 1852 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:16.871146) v4 ==== 2004+0+0 (2871545206 0 0) 0x55a32eb3bc00 con 0x55a314e96000 > -4310> 2018-07-11 07:34:16.871791 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6806/12677 conn(0x55a314e84000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1852 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:16.871146) v4 > -4309> 2018-07-11 07:34:16.871804 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6806/12677 1852 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:16.871146) v4 ==== 2004+0+0 (2871545206 0 0) 0x55a34de6f000 con 0x55a314e84000 > -4308> 2018-07-11 07:34:16.871801 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6803/15255 conn(0x55a314e1d800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1852 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:16.871146) v4 > -4307> 2018-07-11 07:34:16.871819 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6803/15255 1852 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:16.871146) v4 ==== 2004+0+0 (2871545206 0 0) 0x55a324474c00 con 0x55a314e1d800 > -4306> 2018-07-11 07:34:16.871813 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6810/15287 conn(0x55a314e03000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1852 0x55a32eb3bc00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:16.871146) v4 > -4305> 2018-07-11 07:34:16.871829 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6810/15287 1852 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:16.871146) v4 ==== 2004+0+0 (2871545206 0 0) 0x55a32eb3bc00 con 0x55a314e03000 > -4304> 2018-07-11 07:34:16.871849 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6806/15263 conn(0x55a314e99000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1852 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:16.871146) v4 > -4303> 2018-07-11 07:34:16.871852 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6810/12651 conn(0x55a314e97800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1852 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:16.871146) v4 > -4302> 2018-07-11 07:34:16.871860 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6806/15263 1852 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:16.871146) v4 ==== 2004+0+0 (2871545206 0 0) 0x55a34de6f000 con 0x55a314e99000 > -4301> 2018-07-11 07:34:16.871862 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6810/12651 1852 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:16.871146) v4 ==== 2004+0+0 (2871545206 0 0) 0x55a324474c00 con 0x55a314e97800 > -4300> 2018-07-11 07:34:16.871875 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6811/12650 conn(0x55a314e20800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1852 0x55a32eb3bc00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:16.871146) v4 > -4299> 2018-07-11 07:34:16.871888 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6811/12650 1852 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:16.871146) v4 ==== 2004+0+0 (2871545206 0 0) 0x55a32eb3bc00 con 0x55a314e20800 > -4298> 2018-07-11 07:34:16.871892 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6803/13323 conn(0x55a314e16800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1852 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:16.871146) v4 > -4297> 2018-07-11 07:34:16.871901 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6803/13323 1852 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:16.871146) v4 ==== 2004+0+0 (2871545206 0 0) 0x55a34de6f000 con 0x55a314e16800 > -4296> 2018-07-11 07:34:16.871916 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6802/13323 conn(0x55a314e15000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1852 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:16.871146) v4 > -4295> 2018-07-11 07:34:16.871926 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6802/13323 1852 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:16.871146) v4 ==== 2004+0+0 (2871545206 0 0) 0x55a324474c00 con 0x55a314e15000 > -4294> 2018-07-11 07:34:16.871937 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6802/11967 conn(0x55a314e12000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1852 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:16.871146) v4 > -4293> 2018-07-11 07:34:16.871935 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6807/15263 conn(0x55a314e9a800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1852 0x55a32eb3bc00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:16.871146) v4 > -4292> 2018-07-11 07:34:16.871945 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6806/15885 conn(0x55a314e00000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1852 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:16.871146) v4 > -4291> 2018-07-11 07:34:16.871947 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6802/11967 1852 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:16.871146) v4 ==== 2004+0+0 (2871545206 0 0) 0x55a34de6f000 con 0x55a314e12000 > -4290> 2018-07-11 07:34:16.871957 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6807/15263 1852 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:16.871146) v4 ==== 2004+0+0 (2871545206 0 0) 0x55a32eb3bc00 con 0x55a314e9a800 > -4289> 2018-07-11 07:34:16.871963 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6806/15885 1852 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:16.871146) v4 ==== 2004+0+0 (2871545206 0 0) 0x55a324474c00 con 0x55a314e00000 > -4288> 2018-07-11 07:34:16.871994 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6802/15919 conn(0x55a314e87000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1852 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:16.871146) v4 > -4287> 2018-07-11 07:34:16.872006 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6802/15919 1852 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:16.871146) v4 ==== 2004+0+0 (2871545206 0 0) 0x55a324474c00 con 0x55a314e87000 > -4286> 2018-07-11 07:34:16.872002 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6807/15885 conn(0x55a314e01800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1852 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:16.871146) v4 > -4285> 2018-07-11 07:34:16.872011 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6803/11967 conn(0x55a314e13800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1852 0x55a32eb3bc00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:16.871146) v4 > -4284> 2018-07-11 07:34:16.872020 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6807/15885 1852 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:16.871146) v4 ==== 2004+0+0 (2871545206 0 0) 0x55a34de6f000 con 0x55a314e01800 > -4283> 2018-07-11 07:34:16.872032 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6803/11967 1852 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:16.871146) v4 ==== 2004+0+0 (2871545206 0 0) 0x55a32eb3bc00 con 0x55a314e13800 > -4282> 2018-07-11 07:34:16.872142 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6803/15919 conn(0x55a314e88800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1852 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:16.871146) v4 > -4281> 2018-07-11 07:34:16.872159 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6803/15919 1852 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:16.871146) v4 ==== 2004+0+0 (2871545206 0 0) 0x55a34de6f000 con 0x55a314e88800 > -4280> 2018-07-11 07:34:16.880989 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.107:0/12617 conn(0x55a32447e000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1821 0x55a32eb3bc00 osd_ping(ping e494 stamp 2018-07-11 07:34:16.880578) v4 > -4279> 2018-07-11 07:34:16.881005 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.9 10.8.128.107:0/12617 1821 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:16.880578) v4 ==== 2004+0+0 (565211513 0 0) 0x55a32eb3bc00 con 0x55a32447e000 > -4278> 2018-07-11 07:34:16.881020 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:16.880578) v4 -- 0x55a33234ee00 con 0 > -4277> 2018-07-11 07:34:16.881021 7f355d88e700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.107:0/12617 conn(0x55a32447f800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1821 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:34:16.880578) v4 > -4276> 2018-07-11 07:34:16.881035 7f355d88e700 1 -- 10.8.128.107:6803/12665 <== osd.9 10.8.128.107:0/12617 1821 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:16.880578) v4 ==== 2004+0+0 (565211513 0 0) 0x55a34de6f000 con 0x55a32447f800 > -4275> 2018-07-11 07:34:16.881043 7f355d88e700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:16.880578) v4 -- 0x55a32442aa00 con 0 > -4274> 2018-07-11 07:34:16.931035 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15287 conn(0x55a311b75000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1844 0x55a34dea9400 osd_ping(ping e494 stamp 2018-07-11 07:34:18.145121) v4 > -4273> 2018-07-11 07:34:16.931054 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.4 10.8.128.113:0/15287 1844 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:18.145121) v4 ==== 2004+0+0 (185825329 0 0) 0x55a34dea9400 con 0x55a311b75000 > -4272> 2018-07-11 07:34:16.931042 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15287 conn(0x55a311b72000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1844 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:34:18.145121) v4 > -4271> 2018-07-11 07:34:16.931063 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:18.145121) v4 -- 0x55a34de6e200 con 0 > -4270> 2018-07-11 07:34:16.931073 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.4 10.8.128.113:0/15287 1844 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:18.145121) v4 ==== 2004+0+0 (185825329 0 0) 0x55a324474c00 con 0x55a311b72000 > -4269> 2018-07-11 07:34:16.931090 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:18.145121) v4 -- 0x55a33532de00 con 0 > -4268> 2018-07-11 07:34:17.152504 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b7000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1797 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:34:16.903230) v4 > -4267> 2018-07-11 07:34:17.152524 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.8 10.8.128.108:0/11967 1797 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:16.903230) v4 ==== 2004+0+0 (1404689237 0 0) 0x55a34de78600 con 0x55a3170b7000 > -4266> 2018-07-11 07:34:17.152536 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:16.903230) v4 -- 0x55a34de6f600 con 0 > -4265> 2018-07-11 07:34:17.152546 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b8800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1797 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:34:16.903230) v4 > -4264> 2018-07-11 07:34:17.152571 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.8 10.8.128.108:0/11967 1797 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:16.903230) v4 ==== 2004+0+0 (1404689237 0 0) 0x55a332e79a00 con 0x55a3170b8800 > -4263> 2018-07-11 07:34:17.152591 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:16.903230) v4 -- 0x55a332ff3400 con 0 > -4262> 2018-07-11 07:34:17.254870 7f35488a3700 1 -- 10.8.128.107:6800/12665 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55a32a57a580 con 0 > -4261> 2018-07-11 07:34:17.254944 7f35488a3700 1 -- 10.8.128.107:6800/12665 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55a32aa0d600 con 0 > -4260> 2018-07-11 07:34:17.508306 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a324f45200 con 0 > -4259> 2018-07-11 07:34:17.523653 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5132 0x55a34982ca00 replica scrub(pg: 6.15,from:0'0,to:430'105374,epoch:494/492,start:6:a9f2ae1c:::benchmark_data_magna108.ceph.redhat.com_2478_object247581:0,end:6:a9f423e4:::benchmark_data_magna108.ceph.redhat.com_2478_object399550:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -4258> 2018-07-11 07:34:17.523683 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5132 ==== replica scrub(pg: 6.15,from:0'0,to:430'105374,epoch:494/492,start:6:a9f2ae1c:::benchmark_data_magna108.ceph.redhat.com_2478_object247581:0,end:6:a9f423e4:::benchmark_data_magna108.ceph.redhat.com_2478_object399550:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 248+0+0 (1695168365 0 0) 0x55a34982ca00 con 0x55a317019000 > -4257> 2018-07-11 07:34:17.569254 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15885 conn(0x55a3170b5800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1869 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:34:18.193139) v4 > -4256> 2018-07-11 07:34:17.569276 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.7 10.8.128.114:0/15885 1869 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:18.193139) v4 ==== 2004+0+0 (1184609706 0 0) 0x55a324474c00 con 0x55a3170b5800 > -4255> 2018-07-11 07:34:17.569288 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:18.193139) v4 -- 0x55a332ff3400 con 0 > -4254> 2018-07-11 07:34:17.569276 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15885 conn(0x55a311b76800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1869 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:34:18.193139) v4 > -4253> 2018-07-11 07:34:17.569298 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.7 10.8.128.114:0/15885 1869 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:18.193139) v4 ==== 2004+0+0 (1184609706 0 0) 0x55a34de6f000 con 0x55a311b76800 > -4252> 2018-07-11 07:34:17.569316 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:18.193139) v4 -- 0x55a34de6f600 con 0 > -4251> 2018-07-11 07:34:18.153058 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a324f4d8c0 con 0 > -4250> 2018-07-11 07:34:18.182128 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5133 0x55a34982cc80 replica scrub(pg: 6.15,from:0'0,to:430'106193,epoch:494/492,start:6:a9f423e4:::benchmark_data_magna108.ceph.redhat.com_2478_object399550:0,end:6:a9f50d1f:::benchmark_data_magna108_35617_object962157:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -4249> 2018-07-11 07:34:18.182159 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5133 ==== replica scrub(pg: 6.15,from:0'0,to:430'106193,epoch:494/492,start:6:a9f423e4:::benchmark_data_magna108.ceph.redhat.com_2478_object399550:0,end:6:a9f50d1f:::benchmark_data_magna108_35617_object962157:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (786787083 0 0) 0x55a34982cc80 con 0x55a317019000 > -4248> 2018-07-11 07:34:18.818719 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a324f54d80 con 0 > -4247> 2018-07-11 07:34:18.832118 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5134 0x55a34982d180 replica scrub(pg: 6.15,from:0'0,to:430'106171,epoch:494/492,start:6:a9f50d1f:::benchmark_data_magna108_35617_object962157:0,end:6:a9f65598:::benchmark_data_magna108_35617_object379296:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -4246> 2018-07-11 07:34:18.832147 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5134 ==== replica scrub(pg: 6.15,from:0'0,to:430'106171,epoch:494/492,start:6:a9f50d1f:::benchmark_data_magna108_35617_object962157:0,end:6:a9f65598:::benchmark_data_magna108_35617_object379296:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (3115897080 0 0) 0x55a34982d180 con 0x55a317019000 > -4245> 2018-07-11 07:34:19.171687 7f3543899700 5 osd.10 494 heartbeat: osd_stat(784 GB used, 146 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,0,1]) > -4244> 2018-07-11 07:34:19.171703 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6802/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:34:19.171700) v4 -- 0x55a330934000 con 0 > -4243> 2018-07-11 07:34:19.171744 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6803/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:34:19.171700) v4 -- 0x55a31ce7a000 con 0 > -4242> 2018-07-11 07:34:19.171761 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6806/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:34:19.171700) v4 -- 0x55a33532d600 con 0 > -4241> 2018-07-11 07:34:19.171775 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6807/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:34:19.171700) v4 -- 0x55a3314cfc00 con 0 > -4240> 2018-07-11 07:34:19.171799 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6808/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:34:19.171700) v4 -- 0x55a31ce7a800 con 0 > -4239> 2018-07-11 07:34:19.171809 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6810/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:34:19.171700) v4 -- 0x55a34de6fe00 con 0 > -4238> 2018-07-11 07:34:19.171820 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6809/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:34:19.171700) v4 -- 0x55a332ead200 con 0 > -4237> 2018-07-11 07:34:19.171831 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6811/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:34:19.171700) v4 -- 0x55a34de6ea00 con 0 > -4236> 2018-07-11 07:34:19.171843 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6810/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:34:19.171700) v4 -- 0x55a34de85000 con 0 > -4235> 2018-07-11 07:34:19.171859 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6811/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:34:19.171700) v4 -- 0x55a332365400 con 0 > -4234> 2018-07-11 07:34:19.171873 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6806/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:34:19.171700) v4 -- 0x55a33235a400 con 0 > -4233> 2018-07-11 07:34:19.171891 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6807/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:34:19.171700) v4 -- 0x55a34de84200 con 0 > -4232> 2018-07-11 07:34:19.171904 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6802/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:34:19.171700) v4 -- 0x55a33532d200 con 0 > -4231> 2018-07-11 07:34:19.171917 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6803/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:34:19.171700) v4 -- 0x55a34de85e00 con 0 > -4230> 2018-07-11 07:34:19.171930 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6806/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:34:19.171700) v4 -- 0x55a33235a200 con 0 > -4229> 2018-07-11 07:34:19.171942 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6807/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:34:19.171700) v4 -- 0x55a34de79200 con 0 > -4228> 2018-07-11 07:34:19.171957 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6802/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:34:19.171700) v4 -- 0x55a34de78000 con 0 > -4227> 2018-07-11 07:34:19.171972 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6803/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:34:19.171700) v4 -- 0x55a314e7b400 con 0 > -4226> 2018-07-11 07:34:19.171987 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6806/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:34:19.171700) v4 -- 0x55a34de5a200 con 0 > -4225> 2018-07-11 07:34:19.172000 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6807/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:34:19.171700) v4 -- 0x55a3170dfc00 con 0 > -4224> 2018-07-11 07:34:19.172013 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6802/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:34:19.171700) v4 -- 0x55a332ff3800 con 0 > -4223> 2018-07-11 07:34:19.172025 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6803/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:34:19.171700) v4 -- 0x55a332e78400 con 0 > -4222> 2018-07-11 07:34:19.172180 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6806/12617 conn(0x55a314e26000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1853 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.171700) v4 > -4221> 2018-07-11 07:34:19.172186 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6807/12617 conn(0x55a314e27800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1853 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.171700) v4 > -4220> 2018-07-11 07:34:19.172202 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6806/12617 1853 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.171700) v4 ==== 2004+0+0 (772084876 0 0) 0x55a332e79a00 con 0x55a314e26000 > -4219> 2018-07-11 07:34:19.172204 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6807/12617 1853 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.171700) v4 ==== 2004+0+0 (772084876 0 0) 0x55a34de78600 con 0x55a314e27800 > -4218> 2018-07-11 07:34:19.172253 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6803/15255 conn(0x55a314e1d800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1853 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.171700) v4 > -4217> 2018-07-11 07:34:19.172229 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6802/15255 conn(0x55a314e1c000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1853 0x55a32eb3bc00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.171700) v4 > -4216> 2018-07-11 07:34:19.172284 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6803/15255 1853 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.171700) v4 ==== 2004+0+0 (772084876 0 0) 0x55a332e79a00 con 0x55a314e1d800 > -4215> 2018-07-11 07:34:19.172289 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6802/15255 1853 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.171700) v4 ==== 2004+0+0 (772084876 0 0) 0x55a32eb3bc00 con 0x55a314e1c000 > -4214> 2018-07-11 07:34:19.172332 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6810/12651 conn(0x55a314e97800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1853 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.171700) v4 > -4213> 2018-07-11 07:34:19.172346 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6810/12651 1853 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.171700) v4 ==== 2004+0+0 (772084876 0 0) 0x55a332e79a00 con 0x55a314e97800 > -4212> 2018-07-11 07:34:19.172342 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6807/12677 conn(0x55a314e85800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1853 0x55a32eb3bc00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.171700) v4 > -4211> 2018-07-11 07:34:19.172351 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6809/12650 conn(0x55a314e1f000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1853 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.171700) v4 > -4210> 2018-07-11 07:34:19.172359 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6807/12677 1853 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.171700) v4 ==== 2004+0+0 (772084876 0 0) 0x55a32eb3bc00 con 0x55a314e85800 > -4209> 2018-07-11 07:34:19.172369 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6809/12650 1853 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.171700) v4 ==== 2004+0+0 (772084876 0 0) 0x55a34de78600 con 0x55a314e1f000 > -4208> 2018-07-11 07:34:19.172378 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6811/15287 conn(0x55a314e04800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1853 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.171700) v4 > -4207> 2018-07-11 07:34:19.172390 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6811/15287 1853 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.171700) v4 ==== 2004+0+0 (772084876 0 0) 0x55a332e79a00 con 0x55a314e04800 > -4206> 2018-07-11 07:34:19.172405 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6807/15263 conn(0x55a314e9a800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1853 0x55a32eb3bc00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.171700) v4 > -4205> 2018-07-11 07:34:19.172419 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6807/15263 1853 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.171700) v4 ==== 2004+0+0 (772084876 0 0) 0x55a32eb3bc00 con 0x55a314e9a800 > -4204> 2018-07-11 07:34:19.172420 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6806/12677 conn(0x55a314e84000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1853 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.171700) v4 > -4203> 2018-07-11 07:34:19.172433 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6806/12677 1853 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.171700) v4 ==== 2004+0+0 (772084876 0 0) 0x55a34de78600 con 0x55a314e84000 > -4202> 2018-07-11 07:34:19.172459 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6806/15885 conn(0x55a314e00000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1853 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.171700) v4 > -4201> 2018-07-11 07:34:19.172466 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6806/15263 conn(0x55a314e99000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1853 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.171700) v4 > -4200> 2018-07-11 07:34:19.172473 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6806/15885 1853 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.171700) v4 ==== 2004+0+0 (772084876 0 0) 0x55a332e79a00 con 0x55a314e00000 > -4199> 2018-07-11 07:34:19.172476 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6806/15263 1853 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.171700) v4 ==== 2004+0+0 (772084876 0 0) 0x55a34de78600 con 0x55a314e99000 > -4198> 2018-07-11 07:34:19.172469 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6810/15287 conn(0x55a314e03000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1853 0x55a32eb3bc00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.171700) v4 > -4197> 2018-07-11 07:34:19.172486 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6810/15287 1853 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.171700) v4 ==== 2004+0+0 (772084876 0 0) 0x55a32eb3bc00 con 0x55a314e03000 > -4196> 2018-07-11 07:34:19.172516 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6802/13323 conn(0x55a314e15000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1853 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.171700) v4 > -4195> 2018-07-11 07:34:19.172531 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6802/13323 1853 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.171700) v4 ==== 2004+0+0 (772084876 0 0) 0x55a332e79a00 con 0x55a314e15000 > -4194> 2018-07-11 07:34:19.172537 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6803/13323 conn(0x55a314e16800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1853 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.171700) v4 > -4193> 2018-07-11 07:34:19.172534 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6811/12650 conn(0x55a314e20800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1853 0x55a32eb3bc00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.171700) v4 > -4192> 2018-07-11 07:34:19.172555 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6803/13323 1853 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.171700) v4 ==== 2004+0+0 (772084876 0 0) 0x55a34de78600 con 0x55a314e16800 > -4191> 2018-07-11 07:34:19.172558 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6811/12650 1853 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.171700) v4 ==== 2004+0+0 (772084876 0 0) 0x55a32eb3bc00 con 0x55a314e20800 > -4190> 2018-07-11 07:34:19.172584 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6807/15885 conn(0x55a314e01800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1853 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.171700) v4 > -4189> 2018-07-11 07:34:19.172596 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6807/15885 1853 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.171700) v4 ==== 2004+0+0 (772084876 0 0) 0x55a34de78600 con 0x55a314e01800 > -4188> 2018-07-11 07:34:19.172603 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6808/12651 conn(0x55a314e96000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1853 0x55a32eb3bc00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.171700) v4 > -4187> 2018-07-11 07:34:19.172620 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6808/12651 1853 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.171700) v4 ==== 2004+0+0 (772084876 0 0) 0x55a32eb3bc00 con 0x55a314e96000 > -4186> 2018-07-11 07:34:19.172625 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6802/11967 conn(0x55a314e12000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1853 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.171700) v4 > -4185> 2018-07-11 07:34:19.172634 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6802/11967 1853 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.171700) v4 ==== 2004+0+0 (772084876 0 0) 0x55a34de78600 con 0x55a314e12000 > -4184> 2018-07-11 07:34:19.172664 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6803/15919 conn(0x55a314e88800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1853 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.171700) v4 > -4183> 2018-07-11 07:34:19.172664 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6803/11967 conn(0x55a314e13800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1853 0x55a32eb3bc00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.171700) v4 > -4182> 2018-07-11 07:34:19.172678 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6803/15919 1853 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.171700) v4 ==== 2004+0+0 (772084876 0 0) 0x55a34de78600 con 0x55a314e88800 > -4181> 2018-07-11 07:34:19.172682 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6803/11967 1853 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.171700) v4 ==== 2004+0+0 (772084876 0 0) 0x55a32eb3bc00 con 0x55a314e13800 > -4180> 2018-07-11 07:34:19.172731 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6802/15919 conn(0x55a314e87000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1853 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.171700) v4 > -4179> 2018-07-11 07:34:19.172745 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6802/15919 1853 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.171700) v4 ==== 2004+0+0 (772084876 0 0) 0x55a332e79a00 con 0x55a314e87000 > -4178> 2018-07-11 07:34:19.220279 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15263 conn(0x55a321667000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1844 0x55a32eb3bc00 osd_ping(ping e494 stamp 2018-07-11 07:34:20.434360) v4 > -4177> 2018-07-11 07:34:19.220305 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.5 10.8.128.113:0/15263 1844 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:20.434360) v4 ==== 2004+0+0 (1602531636 0 0) 0x55a32eb3bc00 con 0x55a321667000 > -4176> 2018-07-11 07:34:19.220321 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.434360) v4 -- 0x55a34de6ea00 con 0 > -4175> 2018-07-11 07:34:19.220374 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15263 conn(0x55a321664000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1844 0x55a34de6e200 osd_ping(ping e494 stamp 2018-07-11 07:34:20.434360) v4 > -4174> 2018-07-11 07:34:19.220389 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.5 10.8.128.113:0/15263 1844 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:20.434360) v4 ==== 2004+0+0 (1602531636 0 0) 0x55a34de6e200 con 0x55a321664000 > -4173> 2018-07-11 07:34:19.220399 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.434360) v4 -- 0x55a31ce7a800 con 0 > -4172> 2018-07-11 07:34:19.453021 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b7000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1798 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:34:19.203700) v4 > -4171> 2018-07-11 07:34:19.453044 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.8 10.8.128.108:0/11967 1798 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:19.203700) v4 ==== 2004+0+0 (2681496052 0 0) 0x55a34de78600 con 0x55a3170b7000 > -4170> 2018-07-11 07:34:19.453044 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b8800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1798 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:34:19.203700) v4 > -4169> 2018-07-11 07:34:19.453061 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.8 10.8.128.108:0/11967 1798 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:19.203700) v4 ==== 2004+0+0 (2681496052 0 0) 0x55a332e79a00 con 0x55a3170b8800 > -4168> 2018-07-11 07:34:19.453062 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.203700) v4 -- 0x55a33235a400 con 0 > -4167> 2018-07-11 07:34:19.453098 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.203700) v4 -- 0x55a332365400 con 0 > -4166> 2018-07-11 07:34:19.502878 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a324f65200 con 0 > -4165> 2018-07-11 07:34:19.515555 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5135 0x55a34982d680 replica scrub(pg: 6.15,from:0'0,to:430'106330,epoch:494/492,start:6:a9f65598:::benchmark_data_magna108_35617_object379296:0,end:6:a9f7cb48:::benchmark_data_magna108.ceph.redhat.com_2478_object387062:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -4164> 2018-07-11 07:34:19.515586 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5135 ==== replica scrub(pg: 6.15,from:0'0,to:430'106330,epoch:494/492,start:6:a9f65598:::benchmark_data_magna108_35617_object379296:0,end:6:a9f7cb48:::benchmark_data_magna108.ceph.redhat.com_2478_object387062:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (68063541 0 0) 0x55a34982d680 con 0x55a317019000 > -4163> 2018-07-11 07:34:19.552053 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.115:0/13323 conn(0x55a321665800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1859 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:34:19.889612) v4 > -4162> 2018-07-11 07:34:19.552079 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.6 10.8.128.115:0/13323 1859 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:19.889612) v4 ==== 2004+0+0 (2082450062 0 0) 0x55a324474c00 con 0x55a321665800 > -4161> 2018-07-11 07:34:19.552096 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.889612) v4 -- 0x55a332365400 con 0 > -4160> 2018-07-11 07:34:19.552152 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.115:0/13323 conn(0x55a321668800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1859 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:34:19.889612) v4 > -4159> 2018-07-11 07:34:19.552172 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.6 10.8.128.115:0/13323 1859 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:19.889612) v4 ==== 2004+0+0 (2082450062 0 0) 0x55a324474c00 con 0x55a321668800 > -4158> 2018-07-11 07:34:19.552184 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.889612) v4 -- 0x55a34de6fe00 con 0 > -4157> 2018-07-11 07:34:19.629301 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15255 conn(0x55a311b64000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1871 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:34:20.843406) v4 > -4156> 2018-07-11 07:34:19.629327 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.0 10.8.128.113:0/15255 1871 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:20.843406) v4 ==== 2004+0+0 (1718601630 0 0) 0x55a34de6f000 con 0x55a311b64000 > -4155> 2018-07-11 07:34:19.629342 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.843406) v4 -- 0x55a33235a400 con 0 > -4154> 2018-07-11 07:34:19.629331 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15255 conn(0x55a311b65800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1871 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:34:20.843406) v4 > -4153> 2018-07-11 07:34:19.629355 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.0 10.8.128.113:0/15255 1871 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:20.843406) v4 ==== 2004+0+0 (1718601630 0 0) 0x55a332e79a00 con 0x55a311b65800 > -4152> 2018-07-11 07:34:19.629371 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.843406) v4 -- 0x55a34de6fe00 con 0 > -4151> 2018-07-11 07:34:19.672154 7f3543899700 5 osd.10 494 heartbeat: osd_stat(784 GB used, 146 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,1]) > -4150> 2018-07-11 07:34:19.672168 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6802/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:34:19.672166) v4 -- 0x55a332e84e00 con 0 > -4149> 2018-07-11 07:34:19.672191 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6803/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:34:19.672166) v4 -- 0x55a32eb3ba00 con 0 > -4148> 2018-07-11 07:34:19.672206 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6806/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:34:19.672166) v4 -- 0x55a332ff3a00 con 0 > -4147> 2018-07-11 07:34:19.672223 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6807/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:34:19.672166) v4 -- 0x55a32eb3ac00 con 0 > -4146> 2018-07-11 07:34:19.672235 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6808/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:34:19.672166) v4 -- 0x55a32ee76200 con 0 > -4145> 2018-07-11 07:34:19.672260 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6810/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:34:19.672166) v4 -- 0x55a332354000 con 0 > -4144> 2018-07-11 07:34:19.672275 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6809/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:34:19.672166) v4 -- 0x55a32eb3a800 con 0 > -4143> 2018-07-11 07:34:19.672291 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6811/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:34:19.672166) v4 -- 0x55a32ee77000 con 0 > -4142> 2018-07-11 07:34:19.672312 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6810/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:34:19.672166) v4 -- 0x55a332e76600 con 0 > -4141> 2018-07-11 07:34:19.672324 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6811/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:34:19.672166) v4 -- 0x55a332354800 con 0 > -4140> 2018-07-11 07:34:19.672342 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6806/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:34:19.672166) v4 -- 0x55a32eb28a00 con 0 > -4139> 2018-07-11 07:34:19.672357 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6807/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:34:19.672166) v4 -- 0x55a33234b000 con 0 > -4138> 2018-07-11 07:34:19.672370 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6802/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:34:19.672166) v4 -- 0x55a332e99000 con 0 > -4137> 2018-07-11 07:34:19.672382 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6803/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:34:19.672166) v4 -- 0x55a32aa0f600 con 0 > -4136> 2018-07-11 07:34:19.672399 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6806/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:34:19.672166) v4 -- 0x55a32e456800 con 0 > -4135> 2018-07-11 07:34:19.672411 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6807/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:34:19.672166) v4 -- 0x55a34de85200 con 0 > -4134> 2018-07-11 07:34:19.672431 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6802/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:34:19.672166) v4 -- 0x55a33dff1400 con 0 > -4133> 2018-07-11 07:34:19.672444 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6803/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:34:19.672166) v4 -- 0x55a33dff1200 con 0 > -4132> 2018-07-11 07:34:19.672456 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6806/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:34:19.672166) v4 -- 0x55a33dff1000 con 0 > -4131> 2018-07-11 07:34:19.672469 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6807/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:34:19.672166) v4 -- 0x55a33dff0c00 con 0 > -4130> 2018-07-11 07:34:19.672481 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6802/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:34:19.672166) v4 -- 0x55a33dff0a00 con 0 > -4129> 2018-07-11 07:34:19.672496 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6803/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:34:19.672166) v4 -- 0x55a33dff0200 con 0 > -4128> 2018-07-11 07:34:19.672628 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6803/15255 conn(0x55a314e1d800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1854 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.672166) v4 > -4127> 2018-07-11 07:34:19.672650 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6803/15255 1854 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.672166) v4 ==== 2004+0+0 (176497916 0 0) 0x55a332e79a00 con 0x55a314e1d800 > -4126> 2018-07-11 07:34:19.672681 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6806/12617 conn(0x55a314e26000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1854 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.672166) v4 > -4125> 2018-07-11 07:34:19.672695 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6806/12617 1854 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.672166) v4 ==== 2004+0+0 (176497916 0 0) 0x55a332e79a00 con 0x55a314e26000 > -4124> 2018-07-11 07:34:19.672707 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6807/12617 conn(0x55a314e27800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1854 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.672166) v4 > -4123> 2018-07-11 07:34:19.672705 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6810/15287 conn(0x55a314e03000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1854 0x55a31ce7a800 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.672166) v4 > -4122> 2018-07-11 07:34:19.672726 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6807/12617 1854 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.672166) v4 ==== 2004+0+0 (176497916 0 0) 0x55a34de78600 con 0x55a314e27800 > -4121> 2018-07-11 07:34:19.672729 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6810/15287 1854 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.672166) v4 ==== 2004+0+0 (176497916 0 0) 0x55a31ce7a800 con 0x55a314e03000 > -4120> 2018-07-11 07:34:19.672775 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6810/12651 conn(0x55a314e97800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1854 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.672166) v4 > -4119> 2018-07-11 07:34:19.672770 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6806/12677 conn(0x55a314e84000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1854 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.672166) v4 > -4118> 2018-07-11 07:34:19.672790 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6810/12651 1854 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.672166) v4 ==== 2004+0+0 (176497916 0 0) 0x55a332e79a00 con 0x55a314e97800 > -4117> 2018-07-11 07:34:19.672797 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6806/12677 1854 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.672166) v4 ==== 2004+0+0 (176497916 0 0) 0x55a34de78600 con 0x55a314e84000 > -4116> 2018-07-11 07:34:19.672799 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6807/12677 conn(0x55a314e85800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1854 0x55a31ce7a800 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.672166) v4 > -4115> 2018-07-11 07:34:19.672823 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6811/15287 conn(0x55a314e04800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1854 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.672166) v4 > -4114> 2018-07-11 07:34:19.672826 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6807/12677 1854 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.672166) v4 ==== 2004+0+0 (176497916 0 0) 0x55a31ce7a800 con 0x55a314e85800 > -4113> 2018-07-11 07:34:19.672832 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6806/15263 conn(0x55a314e99000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1854 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.672166) v4 > -4112> 2018-07-11 07:34:19.672845 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6806/15263 1854 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.672166) v4 ==== 2004+0+0 (176497916 0 0) 0x55a34de78600 con 0x55a314e99000 > -4111> 2018-07-11 07:34:19.672858 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6811/12650 conn(0x55a314e20800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1854 0x55a31ce7a800 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.672166) v4 > -4110> 2018-07-11 07:34:19.672835 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6811/15287 1854 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.672166) v4 ==== 2004+0+0 (176497916 0 0) 0x55a332e79a00 con 0x55a314e04800 > -4109> 2018-07-11 07:34:19.672873 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6811/12650 1854 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.672166) v4 ==== 2004+0+0 (176497916 0 0) 0x55a31ce7a800 con 0x55a314e20800 > -4108> 2018-07-11 07:34:19.672876 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6809/12650 conn(0x55a314e1f000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1854 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.672166) v4 > -4107> 2018-07-11 07:34:19.672888 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6809/12650 1854 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.672166) v4 ==== 2004+0+0 (176497916 0 0) 0x55a34de78600 con 0x55a314e1f000 > -4106> 2018-07-11 07:34:19.672901 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6802/15255 conn(0x55a314e1c000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1854 0x55a31ce7a800 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.672166) v4 > -4105> 2018-07-11 07:34:19.672906 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6802/13323 conn(0x55a314e15000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1854 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.672166) v4 > -4104> 2018-07-11 07:34:19.672912 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6802/15255 1854 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.672166) v4 ==== 2004+0+0 (176497916 0 0) 0x55a31ce7a800 con 0x55a314e1c000 > -4103> 2018-07-11 07:34:19.672916 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6803/13323 conn(0x55a314e16800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1854 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.672166) v4 > -4102> 2018-07-11 07:34:19.672920 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6802/13323 1854 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.672166) v4 ==== 2004+0+0 (176497916 0 0) 0x55a332e79a00 con 0x55a314e15000 > -4101> 2018-07-11 07:34:19.672930 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6803/13323 1854 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.672166) v4 ==== 2004+0+0 (176497916 0 0) 0x55a34de78600 con 0x55a314e16800 > -4100> 2018-07-11 07:34:19.672942 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6808/12651 conn(0x55a314e96000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1854 0x55a31ce7a800 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.672166) v4 > -4099> 2018-07-11 07:34:19.672954 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6808/12651 1854 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.672166) v4 ==== 2004+0+0 (176497916 0 0) 0x55a31ce7a800 con 0x55a314e96000 > -4098> 2018-07-11 07:34:19.672956 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6806/15885 conn(0x55a314e00000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1854 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.672166) v4 > -4097> 2018-07-11 07:34:19.672960 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6807/15885 conn(0x55a314e01800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1854 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.672166) v4 > -4096> 2018-07-11 07:34:19.672969 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6806/15885 1854 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.672166) v4 ==== 2004+0+0 (176497916 0 0) 0x55a332e79a00 con 0x55a314e00000 > -4095> 2018-07-11 07:34:19.672969 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6807/15885 1854 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.672166) v4 ==== 2004+0+0 (176497916 0 0) 0x55a34de78600 con 0x55a314e01800 > -4094> 2018-07-11 07:34:19.672984 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6807/15263 conn(0x55a314e9a800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1854 0x55a31ce7a800 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.672166) v4 > -4093> 2018-07-11 07:34:19.673010 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6807/15263 1854 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.672166) v4 ==== 2004+0+0 (176497916 0 0) 0x55a31ce7a800 con 0x55a314e9a800 > -4092> 2018-07-11 07:34:19.673015 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6803/15919 conn(0x55a314e88800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1854 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.672166) v4 > -4091> 2018-07-11 07:34:19.673028 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6803/15919 1854 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.672166) v4 ==== 2004+0+0 (176497916 0 0) 0x55a34de78600 con 0x55a314e88800 > -4090> 2018-07-11 07:34:19.673061 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6802/11967 conn(0x55a314e12000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1854 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.672166) v4 > -4089> 2018-07-11 07:34:19.673074 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6802/11967 1854 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.672166) v4 ==== 2004+0+0 (176497916 0 0) 0x55a34de78600 con 0x55a314e12000 > -4088> 2018-07-11 07:34:19.673073 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6802/15919 conn(0x55a314e87000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1854 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.672166) v4 > -4087> 2018-07-11 07:34:19.673076 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6803/11967 conn(0x55a314e13800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1854 0x55a31ce7a800 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.672166) v4 > -4086> 2018-07-11 07:34:19.673090 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6802/15919 1854 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.672166) v4 ==== 2004+0+0 (176497916 0 0) 0x55a332e79a00 con 0x55a314e87000 > -4085> 2018-07-11 07:34:19.673095 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6803/11967 1854 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.672166) v4 ==== 2004+0+0 (176497916 0 0) 0x55a31ce7a800 con 0x55a314e13800 > -4084> 2018-07-11 07:34:19.781632 7f355d88e700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.107:0/12617 conn(0x55a32447f800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1822 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:34:19.781139) v4 > -4083> 2018-07-11 07:34:19.781660 7f355d88e700 1 -- 10.8.128.107:6803/12665 <== osd.9 10.8.128.107:0/12617 1822 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:19.781139) v4 ==== 2004+0+0 (3877229541 0 0) 0x55a34de78600 con 0x55a32447f800 > -4082> 2018-07-11 07:34:19.781672 7f355d88e700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.781139) v4 -- 0x55a33235a400 con 0 > -4081> 2018-07-11 07:34:19.781708 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.107:0/12617 conn(0x55a32447e000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1822 0x55a31ce7a800 osd_ping(ping e494 stamp 2018-07-11 07:34:19.781139) v4 > -4080> 2018-07-11 07:34:19.781729 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.9 10.8.128.107:0/12617 1822 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:19.781139) v4 ==== 2004+0+0 (3877229541 0 0) 0x55a31ce7a800 con 0x55a32447e000 > -4079> 2018-07-11 07:34:19.781740 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:19.781139) v4 -- 0x55a32eb3ac00 con 0 > -4078> 2018-07-11 07:34:20.172652 7f3543899700 5 osd.10 494 heartbeat: osd_stat(784 GB used, 146 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,0,0,1]) > -4077> 2018-07-11 07:34:20.172666 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6802/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:34:20.172665) v4 -- 0x55a33dff0800 con 0 > -4076> 2018-07-11 07:34:20.172702 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6803/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:34:20.172665) v4 -- 0x55a33dff0600 con 0 > -4075> 2018-07-11 07:34:20.172718 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6806/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:34:20.172665) v4 -- 0x55a310611800 con 0 > -4074> 2018-07-11 07:34:20.172732 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6807/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:34:20.172665) v4 -- 0x55a332faf800 con 0 > -4073> 2018-07-11 07:34:20.172744 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6808/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:34:20.172665) v4 -- 0x55a32447b000 con 0 > -4072> 2018-07-11 07:34:20.172758 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6810/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:34:20.172665) v4 -- 0x55a30f58e000 con 0 > -4071> 2018-07-11 07:34:20.172770 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6809/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:34:20.172665) v4 -- 0x55a332eada00 con 0 > -4070> 2018-07-11 07:34:20.172781 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6811/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:34:20.172665) v4 -- 0x55a332ead000 con 0 > -4069> 2018-07-11 07:34:20.172797 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6810/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:34:20.172665) v4 -- 0x55a32eb3b400 con 0 > -4068> 2018-07-11 07:34:20.172817 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6811/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:34:20.172665) v4 -- 0x55a32eb3b600 con 0 > -4067> 2018-07-11 07:34:20.172831 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6806/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:34:20.172665) v4 -- 0x55a32eb14800 con 0 > -4066> 2018-07-11 07:34:20.172843 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6807/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:34:20.172665) v4 -- 0x55a34dec6600 con 0 > -4065> 2018-07-11 07:34:20.172859 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6802/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:34:20.172665) v4 -- 0x55a34dec6000 con 0 > -4064> 2018-07-11 07:34:20.172877 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6803/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:34:20.172665) v4 -- 0x55a332ff3e00 con 0 > -4063> 2018-07-11 07:34:20.172890 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6806/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:34:20.172665) v4 -- 0x55a332385800 con 0 > -4062> 2018-07-11 07:34:20.172901 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6807/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:34:20.172665) v4 -- 0x55a3314cf600 con 0 > -4061> 2018-07-11 07:34:20.172915 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6802/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:34:20.172665) v4 -- 0x55a34de62a00 con 0 > -4060> 2018-07-11 07:34:20.172928 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6803/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:34:20.172665) v4 -- 0x55a327983200 con 0 > -4059> 2018-07-11 07:34:20.172941 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6806/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:34:20.172665) v4 -- 0x55a333f6f000 con 0 > -4058> 2018-07-11 07:34:20.172954 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6807/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:34:20.172665) v4 -- 0x55a32447bc00 con 0 > -4057> 2018-07-11 07:34:20.172967 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6802/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:34:20.172665) v4 -- 0x55a32447a600 con 0 > -4056> 2018-07-11 07:34:20.172978 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6803/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:34:20.172665) v4 -- 0x55a332e78800 con 0 > -4055> 2018-07-11 07:34:20.173148 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6806/12617 conn(0x55a314e26000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1855 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.172665) v4 > -4054> 2018-07-11 07:34:20.173156 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6807/12617 conn(0x55a314e27800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1855 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.172665) v4 > -4053> 2018-07-11 07:34:20.173150 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6802/15255 conn(0x55a314e1c000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1855 0x55a32eb3bc00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.172665) v4 > -4052> 2018-07-11 07:34:20.173173 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6806/12617 1855 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.172665) v4 ==== 2004+0+0 (3310963371 0 0) 0x55a324474c00 con 0x55a314e26000 > -4051> 2018-07-11 07:34:20.173176 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6802/15255 1855 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.172665) v4 ==== 2004+0+0 (3310963371 0 0) 0x55a32eb3bc00 con 0x55a314e1c000 > -4050> 2018-07-11 07:34:20.173174 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6807/12617 1855 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.172665) v4 ==== 2004+0+0 (3310963371 0 0) 0x55a34de6f000 con 0x55a314e27800 > -4049> 2018-07-11 07:34:20.173233 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6803/15255 conn(0x55a314e1d800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1855 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.172665) v4 > -4048> 2018-07-11 07:34:20.173259 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6803/15255 1855 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.172665) v4 ==== 2004+0+0 (3310963371 0 0) 0x55a324474c00 con 0x55a314e1d800 > -4047> 2018-07-11 07:34:20.173264 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6806/12677 conn(0x55a314e84000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1855 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.172665) v4 > -4046> 2018-07-11 07:34:20.173273 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6807/12677 conn(0x55a314e85800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1855 0x55a32eb3bc00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.172665) v4 > -4045> 2018-07-11 07:34:20.173288 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6807/12677 1855 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.172665) v4 ==== 2004+0+0 (3310963371 0 0) 0x55a32eb3bc00 con 0x55a314e85800 > -4044> 2018-07-11 07:34:20.173288 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6806/12677 1855 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.172665) v4 ==== 2004+0+0 (3310963371 0 0) 0x55a34de6f000 con 0x55a314e84000 > -4043> 2018-07-11 07:34:20.173308 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6810/12651 conn(0x55a314e97800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1855 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.172665) v4 > -4042> 2018-07-11 07:34:20.173322 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6810/12651 1855 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.172665) v4 ==== 2004+0+0 (3310963371 0 0) 0x55a324474c00 con 0x55a314e97800 > -4041> 2018-07-11 07:34:20.173343 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6807/15263 conn(0x55a314e9a800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1855 0x55a32eb3bc00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.172665) v4 > -4040> 2018-07-11 07:34:20.173370 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6807/15263 1855 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.172665) v4 ==== 2004+0+0 (3310963371 0 0) 0x55a32eb3bc00 con 0x55a314e9a800 > -4039> 2018-07-11 07:34:20.173365 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6809/12650 conn(0x55a314e1f000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1855 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.172665) v4 > -4038> 2018-07-11 07:34:20.173381 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6809/12650 1855 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.172665) v4 ==== 2004+0+0 (3310963371 0 0) 0x55a34de6f000 con 0x55a314e1f000 > -4037> 2018-07-11 07:34:20.173355 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6811/15287 conn(0x55a314e04800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1855 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.172665) v4 > -4036> 2018-07-11 07:34:20.173405 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6811/12650 conn(0x55a314e20800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1855 0x55a32eb3bc00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.172665) v4 > -4035> 2018-07-11 07:34:20.173407 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6811/15287 1855 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.172665) v4 ==== 2004+0+0 (3310963371 0 0) 0x55a324474c00 con 0x55a314e04800 > -4034> 2018-07-11 07:34:20.173421 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6811/12650 1855 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.172665) v4 ==== 2004+0+0 (3310963371 0 0) 0x55a32eb3bc00 con 0x55a314e20800 > -4033> 2018-07-11 07:34:20.173431 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6802/11967 conn(0x55a314e12000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1855 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.172665) v4 > -4032> 2018-07-11 07:34:20.173451 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6808/12651 conn(0x55a314e96000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1855 0x55a32eb3bc00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.172665) v4 > -4031> 2018-07-11 07:34:20.173451 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6802/11967 1855 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.172665) v4 ==== 2004+0+0 (3310963371 0 0) 0x55a34de6f000 con 0x55a314e12000 > -4030> 2018-07-11 07:34:20.173466 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6808/12651 1855 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.172665) v4 ==== 2004+0+0 (3310963371 0 0) 0x55a32eb3bc00 con 0x55a314e96000 > -4029> 2018-07-11 07:34:20.173451 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6802/13323 conn(0x55a314e15000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1855 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.172665) v4 > -4028> 2018-07-11 07:34:20.173488 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6802/13323 1855 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.172665) v4 ==== 2004+0+0 (3310963371 0 0) 0x55a324474c00 con 0x55a314e15000 > -4027> 2018-07-11 07:34:20.173495 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6810/15287 conn(0x55a314e03000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1855 0x55a32eb3bc00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.172665) v4 > -4026> 2018-07-11 07:34:20.173505 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6810/15287 1855 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.172665) v4 ==== 2004+0+0 (3310963371 0 0) 0x55a32eb3bc00 con 0x55a314e03000 > -4025> 2018-07-11 07:34:20.173511 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6803/13323 conn(0x55a314e16800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1855 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.172665) v4 > -4024> 2018-07-11 07:34:20.173528 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6802/15919 conn(0x55a314e87000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1855 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.172665) v4 > -4023> 2018-07-11 07:34:20.173536 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6803/11967 conn(0x55a314e13800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1855 0x55a32eb3bc00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.172665) v4 > -4022> 2018-07-11 07:34:20.173542 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6802/15919 1855 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.172665) v4 ==== 2004+0+0 (3310963371 0 0) 0x55a324474c00 con 0x55a314e87000 > -4021> 2018-07-11 07:34:20.173528 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6803/13323 1855 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.172665) v4 ==== 2004+0+0 (3310963371 0 0) 0x55a34de6f000 con 0x55a314e16800 > -4020> 2018-07-11 07:34:20.173546 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6803/11967 1855 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.172665) v4 ==== 2004+0+0 (3310963371 0 0) 0x55a32eb3bc00 con 0x55a314e13800 > -4019> 2018-07-11 07:34:20.173586 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6806/15885 conn(0x55a314e00000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1855 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.172665) v4 > -4018> 2018-07-11 07:34:20.173599 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6806/15885 1855 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.172665) v4 ==== 2004+0+0 (3310963371 0 0) 0x55a324474c00 con 0x55a314e00000 > -4017> 2018-07-11 07:34:20.173594 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6806/15263 conn(0x55a314e99000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1855 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.172665) v4 > -4016> 2018-07-11 07:34:20.173610 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6806/15263 1855 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.172665) v4 ==== 2004+0+0 (3310963371 0 0) 0x55a34de6f000 con 0x55a314e99000 > -4015> 2018-07-11 07:34:20.173651 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6807/15885 conn(0x55a314e01800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1855 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.172665) v4 > -4014> 2018-07-11 07:34:20.173664 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6807/15885 1855 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.172665) v4 ==== 2004+0+0 (3310963371 0 0) 0x55a34de6f000 con 0x55a314e01800 > -4013> 2018-07-11 07:34:20.173705 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6803/15919 conn(0x55a314e88800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1855 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.172665) v4 > -4012> 2018-07-11 07:34:20.173718 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6803/15919 1855 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.172665) v4 ==== 2004+0+0 (3310963371 0 0) 0x55a34de6f000 con 0x55a314e88800 > -4011> 2018-07-11 07:34:20.175348 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a31f93b680 con 0 > -4010> 2018-07-11 07:34:20.202827 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5136 0x55a34982d900 replica scrub(pg: 6.15,from:0'0,to:430'105972,epoch:494/492,start:6:a9f7cb48:::benchmark_data_magna108.ceph.redhat.com_2478_object387062:0,end:6:a9f8cd9b:::benchmark_data_magna108_35617_object528263:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -4009> 2018-07-11 07:34:20.202859 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5136 ==== replica scrub(pg: 6.15,from:0'0,to:430'105972,epoch:494/492,start:6:a9f7cb48:::benchmark_data_magna108.ceph.redhat.com_2478_object387062:0,end:6:a9f8cd9b:::benchmark_data_magna108_35617_object528263:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (1186361518 0 0) 0x55a34982d900 con 0x55a317019000 > -4008> 2018-07-11 07:34:20.278624 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12650 conn(0x55a311b67000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1862 0x55a32eb3bc00 osd_ping(ping e494 stamp 2018-07-11 07:34:20.314549) v4 > -4007> 2018-07-11 07:34:20.278643 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.3 10.8.128.106:0/12650 1862 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:20.314549) v4 ==== 2004+0+0 (1788263746 0 0) 0x55a32eb3bc00 con 0x55a311b67000 > -4006> 2018-07-11 07:34:20.278657 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.314549) v4 -- 0x55a327983200 con 0 > -4005> 2018-07-11 07:34:20.278728 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12650 conn(0x55a311b68800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1862 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:34:20.314549) v4 > -4004> 2018-07-11 07:34:20.278745 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.3 10.8.128.106:0/12650 1862 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:20.314549) v4 ==== 2004+0+0 (1788263746 0 0) 0x55a34de6f000 con 0x55a311b68800 > -4003> 2018-07-11 07:34:20.278756 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.314549) v4 -- 0x55a332e78800 con 0 > -4002> 2018-07-11 07:34:20.431440 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15287 conn(0x55a311b72000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1845 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:34:21.645533) v4 > -4001> 2018-07-11 07:34:20.431452 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15287 conn(0x55a311b75000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1845 0x55a31ce7a800 osd_ping(ping e494 stamp 2018-07-11 07:34:21.645533) v4 > -4000> 2018-07-11 07:34:20.431465 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.4 10.8.128.113:0/15287 1845 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:21.645533) v4 ==== 2004+0+0 (2166531095 0 0) 0x55a324474c00 con 0x55a311b72000 > -3999> 2018-07-11 07:34:20.431474 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:21.645533) v4 -- 0x55a32447a600 con 0 > -3998> 2018-07-11 07:34:20.431473 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.4 10.8.128.113:0/15287 1845 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:21.645533) v4 ==== 2004+0+0 (2166531095 0 0) 0x55a31ce7a800 con 0x55a311b75000 > -3997> 2018-07-11 07:34:20.431499 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:21.645533) v4 -- 0x55a327983200 con 0 > -3996> 2018-07-11 07:34:20.557836 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15919 conn(0x55a317074800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1850 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:34:21.181781) v4 > -3995> 2018-07-11 07:34:20.557857 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.11 10.8.128.114:0/15919 1850 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:21.181781) v4 ==== 2004+0+0 (3688324425 0 0) 0x55a332e79a00 con 0x55a317074800 > -3994> 2018-07-11 07:34:20.557867 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:21.181781) v4 -- 0x55a32447a600 con 0 > -3993> 2018-07-11 07:34:20.557923 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15919 conn(0x55a3170b4000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1850 0x55a32eb3bc00 osd_ping(ping e494 stamp 2018-07-11 07:34:21.181781) v4 > -3992> 2018-07-11 07:34:20.557947 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.11 10.8.128.114:0/15919 1850 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:21.181781) v4 ==== 2004+0+0 (3688324425 0 0) 0x55a32eb3bc00 con 0x55a3170b4000 > -3991> 2018-07-11 07:34:20.557959 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:21.181781) v4 -- 0x55a34dec6600 con 0 > -3990> 2018-07-11 07:34:20.673168 7f3543899700 5 osd.10 494 heartbeat: osd_stat(784 GB used, 146 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,0,1]) > -3989> 2018-07-11 07:34:20.673183 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6802/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:34:20.673181) v4 -- 0x55a332ff2600 con 0 > -3988> 2018-07-11 07:34:20.673222 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6803/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:34:20.673181) v4 -- 0x55a332348a00 con 0 > -3987> 2018-07-11 07:34:20.673239 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6806/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:34:20.673181) v4 -- 0x55a332349a00 con 0 > -3986> 2018-07-11 07:34:20.673278 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6807/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:34:20.673181) v4 -- 0x55a332348800 con 0 > -3985> 2018-07-11 07:34:20.673296 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6808/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:34:20.673181) v4 -- 0x55a332348600 con 0 > -3984> 2018-07-11 07:34:20.673316 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6810/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:34:20.673181) v4 -- 0x55a332348000 con 0 > -3983> 2018-07-11 07:34:20.673347 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6809/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:34:20.673181) v4 -- 0x55a332385400 con 0 > -3982> 2018-07-11 07:34:20.673368 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6811/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:34:20.673181) v4 -- 0x55a332384600 con 0 > -3981> 2018-07-11 07:34:20.673395 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6810/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:34:20.673181) v4 -- 0x55a330934600 con 0 > -3980> 2018-07-11 07:34:20.673417 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6811/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:34:20.673181) v4 -- 0x55a332385a00 con 0 > -3979> 2018-07-11 07:34:20.673437 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6806/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:34:20.673181) v4 -- 0x55a330935400 con 0 > -3978> 2018-07-11 07:34:20.673453 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6807/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:34:20.673181) v4 -- 0x55a332384200 con 0 > -3977> 2018-07-11 07:34:20.673476 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6802/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:34:20.673181) v4 -- 0x55a32e457e00 con 0 > -3976> 2018-07-11 07:34:20.673500 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6803/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:34:20.673181) v4 -- 0x55a330934c00 con 0 > -3975> 2018-07-11 07:34:20.673518 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6806/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:34:20.673181) v4 -- 0x55a332385000 con 0 > -3974> 2018-07-11 07:34:20.673539 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6807/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:34:20.673181) v4 -- 0x55a3170d1600 con 0 > -3973> 2018-07-11 07:34:20.673566 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6802/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:34:20.673181) v4 -- 0x55a332385200 con 0 > -3972> 2018-07-11 07:34:20.673589 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6803/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:34:20.673181) v4 -- 0x55a32162b800 con 0 > -3971> 2018-07-11 07:34:20.673609 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6806/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:34:20.673181) v4 -- 0x55a32162b000 con 0 > -3970> 2018-07-11 07:34:20.673632 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6807/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:34:20.673181) v4 -- 0x55a3170d1c00 con 0 > -3969> 2018-07-11 07:34:20.673653 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6802/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:34:20.673181) v4 -- 0x55a332eac600 con 0 > -3968> 2018-07-11 07:34:20.673674 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6803/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:34:20.673181) v4 -- 0x55a332faf200 con 0 > -3967> 2018-07-11 07:34:20.673732 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6803/15255 conn(0x55a314e1d800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1856 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.673181) v4 > -3966> 2018-07-11 07:34:20.673743 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6807/12677 conn(0x55a314e85800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1856 0x55a31ce7a800 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.673181) v4 > -3965> 2018-07-11 07:34:20.673754 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6803/15255 1856 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.673181) v4 ==== 2004+0+0 (2359782481 0 0) 0x55a324474c00 con 0x55a314e1d800 > -3964> 2018-07-11 07:34:20.673771 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6807/12677 1856 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.673181) v4 ==== 2004+0+0 (2359782481 0 0) 0x55a31ce7a800 con 0x55a314e85800 > -3963> 2018-07-11 07:34:20.673804 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6806/12677 conn(0x55a314e84000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1856 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.673181) v4 > -3962> 2018-07-11 07:34:20.673835 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6806/12677 1856 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.673181) v4 ==== 2004+0+0 (2359782481 0 0) 0x55a34de78600 con 0x55a314e84000 > -3961> 2018-07-11 07:34:20.673835 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6808/12651 conn(0x55a314e96000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1856 0x55a31ce7a800 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.673181) v4 > -3960> 2018-07-11 07:34:20.673851 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6808/12651 1856 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.673181) v4 ==== 2004+0+0 (2359782481 0 0) 0x55a31ce7a800 con 0x55a314e96000 > -3959> 2018-07-11 07:34:20.673849 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6806/12617 conn(0x55a314e26000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1856 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.673181) v4 > -3958> 2018-07-11 07:34:20.673862 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6806/12617 1856 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.673181) v4 ==== 2004+0+0 (2359782481 0 0) 0x55a332e79a00 con 0x55a314e26000 > -3957> 2018-07-11 07:34:20.673871 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6807/12617 conn(0x55a314e27800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1856 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.673181) v4 > -3956> 2018-07-11 07:34:20.673873 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6802/15255 conn(0x55a314e1c000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1856 0x55a31ce7a800 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.673181) v4 > -3955> 2018-07-11 07:34:20.673885 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6807/12617 1856 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.673181) v4 ==== 2004+0+0 (2359782481 0 0) 0x55a34de78600 con 0x55a314e27800 > -3954> 2018-07-11 07:34:20.673889 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6802/15255 1856 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.673181) v4 ==== 2004+0+0 (2359782481 0 0) 0x55a31ce7a800 con 0x55a314e1c000 > -3953> 2018-07-11 07:34:20.673897 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6811/15287 conn(0x55a314e04800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1856 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.673181) v4 > -3952> 2018-07-11 07:34:20.673919 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6811/12650 conn(0x55a314e20800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1856 0x55a31ce7a800 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.673181) v4 > -3951> 2018-07-11 07:34:20.673916 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6809/12650 conn(0x55a314e1f000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1856 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.673181) v4 > -3950> 2018-07-11 07:34:20.673909 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6811/15287 1856 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.673181) v4 ==== 2004+0+0 (2359782481 0 0) 0x55a332e79a00 con 0x55a314e04800 > -3949> 2018-07-11 07:34:20.673930 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6811/12650 1856 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.673181) v4 ==== 2004+0+0 (2359782481 0 0) 0x55a31ce7a800 con 0x55a314e20800 > -3948> 2018-07-11 07:34:20.673945 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6809/12650 1856 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.673181) v4 ==== 2004+0+0 (2359782481 0 0) 0x55a34de78600 con 0x55a314e1f000 > -3947> 2018-07-11 07:34:20.673982 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6807/15263 conn(0x55a314e9a800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1856 0x55a31ce7a800 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.673181) v4 > -3946> 2018-07-11 07:34:20.673988 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6803/13323 conn(0x55a314e16800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1856 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.673181) v4 > -3945> 2018-07-11 07:34:20.673970 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6810/12651 conn(0x55a314e97800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1856 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.673181) v4 > -3944> 2018-07-11 07:34:20.673998 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6807/15263 1856 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.673181) v4 ==== 2004+0+0 (2359782481 0 0) 0x55a31ce7a800 con 0x55a314e9a800 > -3943> 2018-07-11 07:34:20.674002 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6810/12651 1856 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.673181) v4 ==== 2004+0+0 (2359782481 0 0) 0x55a332e79a00 con 0x55a314e97800 > -3942> 2018-07-11 07:34:20.674000 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6803/13323 1856 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.673181) v4 ==== 2004+0+0 (2359782481 0 0) 0x55a34de78600 con 0x55a314e16800 > -3941> 2018-07-11 07:34:20.674034 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6810/15287 conn(0x55a314e03000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1856 0x55a31ce7a800 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.673181) v4 > -3940> 2018-07-11 07:34:20.674045 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6810/15287 1856 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.673181) v4 ==== 2004+0+0 (2359782481 0 0) 0x55a31ce7a800 con 0x55a314e03000 > -3939> 2018-07-11 07:34:20.674052 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6806/15263 conn(0x55a314e99000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1856 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.673181) v4 > -3938> 2018-07-11 07:34:20.674065 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6806/15263 1856 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.673181) v4 ==== 2004+0+0 (2359782481 0 0) 0x55a34de78600 con 0x55a314e99000 > -3937> 2018-07-11 07:34:20.674076 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6803/11967 conn(0x55a314e13800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1856 0x55a31ce7a800 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.673181) v4 > -3936> 2018-07-11 07:34:20.674068 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6802/13323 conn(0x55a314e15000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1856 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.673181) v4 > -3935> 2018-07-11 07:34:20.674088 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6803/11967 1856 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.673181) v4 ==== 2004+0+0 (2359782481 0 0) 0x55a31ce7a800 con 0x55a314e13800 > -3934> 2018-07-11 07:34:20.674094 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6807/15885 conn(0x55a314e01800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1856 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.673181) v4 > -3933> 2018-07-11 07:34:20.674098 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6802/13323 1856 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.673181) v4 ==== 2004+0+0 (2359782481 0 0) 0x55a332e79a00 con 0x55a314e15000 > -3932> 2018-07-11 07:34:20.674105 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6807/15885 1856 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.673181) v4 ==== 2004+0+0 (2359782481 0 0) 0x55a34de78600 con 0x55a314e01800 > -3931> 2018-07-11 07:34:20.674128 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6806/15885 conn(0x55a314e00000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1856 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.673181) v4 > -3930> 2018-07-11 07:34:20.674134 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6802/11967 conn(0x55a314e12000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1856 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.673181) v4 > -3929> 2018-07-11 07:34:20.674144 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6806/15885 1856 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.673181) v4 ==== 2004+0+0 (2359782481 0 0) 0x55a332e79a00 con 0x55a314e00000 > -3928> 2018-07-11 07:34:20.674144 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6802/11967 1856 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.673181) v4 ==== 2004+0+0 (2359782481 0 0) 0x55a34de78600 con 0x55a314e12000 > -3927> 2018-07-11 07:34:20.674177 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6802/15919 conn(0x55a314e87000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1856 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.673181) v4 > -3926> 2018-07-11 07:34:20.674190 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6802/15919 1856 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.673181) v4 ==== 2004+0+0 (2359782481 0 0) 0x55a332e79a00 con 0x55a314e87000 > -3925> 2018-07-11 07:34:20.674186 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6803/15919 conn(0x55a314e88800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1856 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.673181) v4 > -3924> 2018-07-11 07:34:20.674204 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6803/15919 1856 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:20.673181) v4 ==== 2004+0+0 (2359782481 0 0) 0x55a34de78600 con 0x55a314e88800 > -3923> 2018-07-11 07:34:20.916921 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a325fbafc0 con 0 > -3922> 2018-07-11 07:34:20.935850 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5137 0x55a34982db80 replica scrub(pg: 6.15,from:0'0,to:430'106684,epoch:494/492,start:6:a9f8cd9b:::benchmark_data_magna108_35617_object528263:0,end:6:a9f9e08d:::benchmark_data_magna108_35617_object887124:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -3921> 2018-07-11 07:34:20.935873 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5137 ==== replica scrub(pg: 6.15,from:0'0,to:430'106684,epoch:494/492,start:6:a9f8cd9b:::benchmark_data_magna108_35617_object528263:0,end:6:a9f9e08d:::benchmark_data_magna108_35617_object887124:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (2227035668 0 0) 0x55a34982db80 con 0x55a317019000 > -3920> 2018-07-11 07:34:21.166075 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12651 conn(0x55a32169f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1847 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:34:21.202003) v4 > -3919> 2018-07-11 07:34:21.166101 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.2 10.8.128.106:0/12651 1847 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:21.202003) v4 ==== 2004+0+0 (688407342 0 0) 0x55a332e79a00 con 0x55a32169f000 > -3918> 2018-07-11 07:34:21.166102 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12651 conn(0x55a3216a0800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1847 0x55a31ce7a800 osd_ping(ping e494 stamp 2018-07-11 07:34:21.202003) v4 > -3917> 2018-07-11 07:34:21.166116 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:21.202003) v4 -- 0x55a332348a00 con 0 > -3916> 2018-07-11 07:34:21.166127 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.2 10.8.128.106:0/12651 1847 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:21.202003) v4 ==== 2004+0+0 (688407342 0 0) 0x55a31ce7a800 con 0x55a3216a0800 > -3915> 2018-07-11 07:34:21.166145 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:21.202003) v4 -- 0x55a34dec6600 con 0 > -3914> 2018-07-11 07:34:21.529749 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12677 conn(0x55a32169d800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1867 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:34:21.565673) v4 > -3913> 2018-07-11 07:34:21.529774 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.1 10.8.128.106:0/12677 1867 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:21.565673) v4 ==== 2004+0+0 (1004147597 0 0) 0x55a34de78600 con 0x55a32169d800 > -3912> 2018-07-11 07:34:21.529777 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12677 conn(0x55a32169c000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1867 0x55a332ff2600 osd_ping(ping e494 stamp 2018-07-11 07:34:21.565673) v4 > -3911> 2018-07-11 07:34:21.529792 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:21.565673) v4 -- 0x55a332e78800 con 0 > -3910> 2018-07-11 07:34:21.529808 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.1 10.8.128.106:0/12677 1867 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:21.565673) v4 ==== 2004+0+0 (1004147597 0 0) 0x55a332ff2600 con 0x55a32169c000 > -3909> 2018-07-11 07:34:21.529825 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:21.565673) v4 -- 0x55a32eb3b400 con 0 > -3908> 2018-07-11 07:34:21.546519 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a33c627b00 con 0 > -3907> 2018-07-11 07:34:21.569795 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5138 0x55a321004000 replica scrub(pg: 6.15,from:0'0,to:461'106918,epoch:494/492,start:6:a9f9e08d:::benchmark_data_magna108_35617_object887124:0,end:6:a9fb15ad:::benchmark_data_magna108_35617_object560869:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -3906> 2018-07-11 07:34:21.569823 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5138 ==== replica scrub(pg: 6.15,from:0'0,to:461'106918,epoch:494/492,start:6:a9f9e08d:::benchmark_data_magna108_35617_object887124:0,end:6:a9fb15ad:::benchmark_data_magna108_35617_object560869:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (2083349648 0 0) 0x55a321004000 con 0x55a317019000 > -3905> 2018-07-11 07:34:22.082231 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.107:0/12617 conn(0x55a32447e000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1823 0x55a31ce7a800 osd_ping(ping e494 stamp 2018-07-11 07:34:22.081769) v4 > -3904> 2018-07-11 07:34:22.082231 7f355d88e700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.107:0/12617 conn(0x55a32447f800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1823 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:34:22.081769) v4 > -3903> 2018-07-11 07:34:22.082268 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.9 10.8.128.107:0/12617 1823 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:22.081769) v4 ==== 2004+0+0 (403553008 0 0) 0x55a31ce7a800 con 0x55a32447e000 > -3902> 2018-07-11 07:34:22.082290 7f355d88e700 1 -- 10.8.128.107:6803/12665 <== osd.9 10.8.128.107:0/12617 1823 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:22.081769) v4 ==== 2004+0+0 (403553008 0 0) 0x55a34de6f000 con 0x55a32447f800 > -3901> 2018-07-11 07:34:22.082293 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:22.081769) v4 -- 0x55a332ead000 con 0 > -3900> 2018-07-11 07:34:22.082324 7f355d88e700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:22.081769) v4 -- 0x55a332e78800 con 0 > -3899> 2018-07-11 07:34:22.255149 7f35488a3700 1 -- 10.8.128.107:6800/12665 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55a3170c4840 con 0 > -3898> 2018-07-11 07:34:22.255223 7f35488a3700 1 -- 10.8.128.107:6800/12665 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55a31708ab00 con 0 > -3897> 2018-07-11 07:34:22.258263 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15919 conn(0x55a317074800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1851 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:34:22.882195) v4 > -3896> 2018-07-11 07:34:22.258287 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.11 10.8.128.114:0/15919 1851 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:22.882195) v4 ==== 2004+0+0 (49661111 0 0) 0x55a324474c00 con 0x55a317074800 > -3895> 2018-07-11 07:34:22.258300 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:22.882195) v4 -- 0x55a332348a00 con 0 > -3894> 2018-07-11 07:34:22.258359 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15919 conn(0x55a3170b4000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1851 0x55a32eb3b400 osd_ping(ping e494 stamp 2018-07-11 07:34:22.882195) v4 > -3893> 2018-07-11 07:34:22.258381 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.11 10.8.128.114:0/15919 1851 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:22.882195) v4 ==== 2004+0+0 (49661111 0 0) 0x55a32eb3b400 con 0x55a3170b4000 > -3892> 2018-07-11 07:34:22.258397 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:22.882195) v4 -- 0x55a32447b000 con 0 > -3891> 2018-07-11 07:34:22.869667 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15885 conn(0x55a3170b5800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1870 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:34:23.493579) v4 > -3890> 2018-07-11 07:34:22.869672 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15885 conn(0x55a311b76800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1870 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:34:23.493579) v4 > -3889> 2018-07-11 07:34:22.869696 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.7 10.8.128.114:0/15885 1870 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:23.493579) v4 ==== 2004+0+0 (1704807114 0 0) 0x55a332e79a00 con 0x55a3170b5800 > -3888> 2018-07-11 07:34:22.869700 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.7 10.8.128.114:0/15885 1870 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:23.493579) v4 ==== 2004+0+0 (1704807114 0 0) 0x55a34de78600 con 0x55a311b76800 > -3887> 2018-07-11 07:34:22.869707 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:23.493579) v4 -- 0x55a32447a600 con 0 > -3886> 2018-07-11 07:34:22.869732 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:23.493579) v4 -- 0x55a32447bc00 con 0 > -3885> 2018-07-11 07:34:22.952699 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a324f8ad80 con 0 > -3884> 2018-07-11 07:34:23.016518 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5139 0x55a321004280 replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:494/492,start:6:a9fb15ad:::benchmark_data_magna108_35617_object560869:0,end:6:a9fc34db:::benchmark_data_magna108_35617_object314058:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -3883> 2018-07-11 07:34:23.016541 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5139 ==== replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:494/492,start:6:a9fb15ad:::benchmark_data_magna108_35617_object560869:0,end:6:a9fc34db:::benchmark_data_magna108_35617_object314058:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (2994832930 0 0) 0x55a321004280 con 0x55a317019000 > -3882> 2018-07-11 07:34:23.320635 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15263 conn(0x55a321664000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1845 0x55a31ce7a800 osd_ping(ping e494 stamp 2018-07-11 07:34:24.534780) v4 > -3881> 2018-07-11 07:34:23.320662 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.5 10.8.128.113:0/15263 1845 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:24.534780) v4 ==== 2004+0+0 (2082672795 0 0) 0x55a31ce7a800 con 0x55a321664000 > -3880> 2018-07-11 07:34:23.320677 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:24.534780) v4 -- 0x55a332faf800 con 0 > -3879> 2018-07-11 07:34:23.320730 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15263 conn(0x55a321667000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1845 0x55a31ce7a800 osd_ping(ping e494 stamp 2018-07-11 07:34:24.534780) v4 > -3878> 2018-07-11 07:34:23.320753 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.5 10.8.128.113:0/15263 1845 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:24.534780) v4 ==== 2004+0+0 (2082672795 0 0) 0x55a31ce7a800 con 0x55a321667000 > -3877> 2018-07-11 07:34:23.320763 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:24.534780) v4 -- 0x55a32eb3ac00 con 0 > -3876> 2018-07-11 07:34:23.718211 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 59] Generated table #36585: 4957167 keys, 80348326 bytes > -3875> 2018-07-11 07:34:23.718273 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294463718238, "cf_name": "default", "job": 59, "event": "table_file_creation", "file_number": 36585, "file_size": 80348326, "table_properties": {"data_size": 67109413, "index_size": 689216, "filter_size": 12548707, "raw_key_size": 143207044, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957167, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957167", "kMergeOperands": "0"}} > -3874> 2018-07-11 07:34:23.830276 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12677 conn(0x55a32169d800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1868 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:34:23.866199) v4 > -3873> 2018-07-11 07:34:23.830302 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.1 10.8.128.106:0/12677 1868 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:23.866199) v4 ==== 2004+0+0 (3974343518 0 0) 0x55a34de6f000 con 0x55a32169d800 > -3872> 2018-07-11 07:34:23.830295 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12677 conn(0x55a32169c000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1868 0x55a32447b000 osd_ping(ping e494 stamp 2018-07-11 07:34:23.866199) v4 > -3871> 2018-07-11 07:34:23.830314 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:23.866199) v4 -- 0x55a34de62a00 con 0 > -3870> 2018-07-11 07:34:23.830314 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.1 10.8.128.106:0/12677 1868 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:23.866199) v4 ==== 2004+0+0 (3974343518 0 0) 0x55a32447b000 con 0x55a32169c000 > -3869> 2018-07-11 07:34:23.830344 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:23.866199) v4 -- 0x55a332faf800 con 0 > -3868> 2018-07-11 07:34:23.970117 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15885 conn(0x55a3170b5800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1871 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:34:24.594002) v4 > -3867> 2018-07-11 07:34:23.970122 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15885 conn(0x55a311b76800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1871 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:34:24.594002) v4 > -3866> 2018-07-11 07:34:23.970139 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.7 10.8.128.114:0/15885 1871 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:24.594002) v4 ==== 2004+0+0 (1559072890 0 0) 0x55a34de78600 con 0x55a311b76800 > -3865> 2018-07-11 07:34:23.970138 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.7 10.8.128.114:0/15885 1871 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:24.594002) v4 ==== 2004+0+0 (1559072890 0 0) 0x55a324474c00 con 0x55a3170b5800 > -3864> 2018-07-11 07:34:23.970149 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:24.594002) v4 -- 0x55a3314cf600 con 0 > -3863> 2018-07-11 07:34:23.970171 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:24.594002) v4 -- 0x55a333f6f000 con 0 > -3862> 2018-07-11 07:34:24.252450 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.115:0/13323 conn(0x55a321665800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1860 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:34:24.590065) v4 > -3861> 2018-07-11 07:34:24.252476 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.6 10.8.128.115:0/13323 1860 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:24.590065) v4 ==== 2004+0+0 (72499479 0 0) 0x55a332e79a00 con 0x55a321665800 > -3860> 2018-07-11 07:34:24.252493 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:24.590065) v4 -- 0x55a332385800 con 0 > -3859> 2018-07-11 07:34:24.252573 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.115:0/13323 conn(0x55a321668800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1860 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:34:24.590065) v4 > -3858> 2018-07-11 07:34:24.252589 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.6 10.8.128.115:0/13323 1860 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:24.590065) v4 ==== 2004+0+0 (72499479 0 0) 0x55a324474c00 con 0x55a321668800 > -3857> 2018-07-11 07:34:24.252599 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:24.590065) v4 -- 0x55a34dec6000 con 0 > -3856> 2018-07-11 07:34:24.267422 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a324fa0b40 con 0 > -3855> 2018-07-11 07:34:24.283023 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5140 0x55a321004500 replica scrub(pg: 6.15,from:0'0,to:430'106448,epoch:494/492,start:6:a9fc34db:::benchmark_data_magna108_35617_object314058:0,end:6:a9fd37ff:::benchmark_data_magna108_35617_object277779:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -3854> 2018-07-11 07:34:24.283052 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5140 ==== replica scrub(pg: 6.15,from:0'0,to:430'106448,epoch:494/492,start:6:a9fc34db:::benchmark_data_magna108_35617_object314058:0,end:6:a9fd37ff:::benchmark_data_magna108_35617_object277779:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (1360988446 0 0) 0x55a321004500 con 0x55a317019000 > -3853> 2018-07-11 07:34:24.470508 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15885 conn(0x55a3170b5800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1872 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:34:25.094433) v4 > -3852> 2018-07-11 07:34:24.470505 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15885 conn(0x55a311b76800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1872 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:34:25.094433) v4 > -3851> 2018-07-11 07:34:24.470534 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.7 10.8.128.114:0/15885 1872 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:25.094433) v4 ==== 2004+0+0 (42552537 0 0) 0x55a332e79a00 con 0x55a3170b5800 > -3850> 2018-07-11 07:34:24.470540 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.7 10.8.128.114:0/15885 1872 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:25.094433) v4 ==== 2004+0+0 (42552537 0 0) 0x55a34de6f000 con 0x55a311b76800 > -3849> 2018-07-11 07:34:24.470551 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:25.094433) v4 -- 0x55a34dec6000 con 0 > -3848> 2018-07-11 07:34:24.470583 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:25.094433) v4 -- 0x55a3314cf600 con 0 > -3847> 2018-07-11 07:34:24.901813 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a331a28000 con 0 > -3846> 2018-07-11 07:34:24.927715 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5141 0x55a321004780 replica scrub(pg: 6.15,from:0'0,to:430'106788,epoch:494/492,start:6:a9fd37ff:::benchmark_data_magna108_35617_object277779:0,end:6:a9fe29e8:::benchmark_data_magna108.ceph.redhat.com_2478_object372856:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -3845> 2018-07-11 07:34:24.927738 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5141 ==== replica scrub(pg: 6.15,from:0'0,to:430'106788,epoch:494/492,start:6:a9fd37ff:::benchmark_data_magna108_35617_object277779:0,end:6:a9fe29e8:::benchmark_data_magna108.ceph.redhat.com_2478_object372856:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (4039679209 0 0) 0x55a321004780 con 0x55a317019000 > -3844> 2018-07-11 07:34:24.929648 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15255 conn(0x55a311b64000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1872 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:34:26.143835) v4 > -3843> 2018-07-11 07:34:24.929671 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.0 10.8.128.113:0/15255 1872 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:26.143835) v4 ==== 2004+0+0 (1080574341 0 0) 0x55a34de78600 con 0x55a311b64000 > -3842> 2018-07-11 07:34:24.929681 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:26.143835) v4 -- 0x55a3314cf600 con 0 > -3841> 2018-07-11 07:34:24.929669 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15255 conn(0x55a311b65800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1872 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:34:26.143835) v4 > -3840> 2018-07-11 07:34:24.929690 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.0 10.8.128.113:0/15255 1872 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:26.143835) v4 ==== 2004+0+0 (1080574341 0 0) 0x55a324474c00 con 0x55a311b65800 > -3839> 2018-07-11 07:34:24.929709 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:26.143835) v4 -- 0x55a34dec6000 con 0 > -3838> 2018-07-11 07:34:24.930660 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12677 conn(0x55a32169d800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1869 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:34:24.966674) v4 > -3837> 2018-07-11 07:34:24.930681 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.1 10.8.128.106:0/12677 1869 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:24.966674) v4 ==== 2004+0+0 (4205496950 0 0) 0x55a34de6f000 con 0x55a32169d800 > -3836> 2018-07-11 07:34:24.930690 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:24.966674) v4 -- 0x55a3314cf600 con 0 > -3835> 2018-07-11 07:34:24.930739 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12677 conn(0x55a32169c000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1869 0x55a31ce7a800 osd_ping(ping e494 stamp 2018-07-11 07:34:24.966674) v4 > -3834> 2018-07-11 07:34:24.930765 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.1 10.8.128.106:0/12677 1869 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:24.966674) v4 ==== 2004+0+0 (4205496950 0 0) 0x55a31ce7a800 con 0x55a32169c000 > -3833> 2018-07-11 07:34:24.930782 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:24.966674) v4 -- 0x55a332faf800 con 0 > -3832> 2018-07-11 07:34:25.131898 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15287 conn(0x55a311b75000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1846 0x55a32447b000 osd_ping(ping e494 stamp 2018-07-11 07:34:26.346024) v4 > -3831> 2018-07-11 07:34:25.131895 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15287 conn(0x55a311b72000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1846 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:34:26.346024) v4 > -3830> 2018-07-11 07:34:25.131927 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.4 10.8.128.113:0/15287 1846 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:26.346024) v4 ==== 2004+0+0 (21309042 0 0) 0x55a332e79a00 con 0x55a311b72000 > -3829> 2018-07-11 07:34:25.131928 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.4 10.8.128.113:0/15287 1846 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:26.346024) v4 ==== 2004+0+0 (21309042 0 0) 0x55a32447b000 con 0x55a311b75000 > -3828> 2018-07-11 07:34:25.131939 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:26.346024) v4 -- 0x55a34dec6000 con 0 > -3827> 2018-07-11 07:34:25.131964 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:26.346024) v4 -- 0x55a332faf800 con 0 > -3826> 2018-07-11 07:34:25.353491 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b7000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1799 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:34:25.104197) v4 > -3825> 2018-07-11 07:34:25.353514 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.8 10.8.128.108:0/11967 1799 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:25.104197) v4 ==== 2004+0+0 (3339828568 0 0) 0x55a34de78600 con 0x55a3170b7000 > -3824> 2018-07-11 07:34:25.353513 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b8800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1799 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:34:25.104197) v4 > -3823> 2018-07-11 07:34:25.353530 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.8 10.8.128.108:0/11967 1799 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:25.104197) v4 ==== 2004+0+0 (3339828568 0 0) 0x55a324474c00 con 0x55a3170b8800 > -3822> 2018-07-11 07:34:25.353529 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:25.104197) v4 -- 0x55a332ff3e00 con 0 > -3821> 2018-07-11 07:34:25.353559 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:25.104197) v4 -- 0x55a32eb3b600 con 0 > -3820> 2018-07-11 07:34:25.431265 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12677 conn(0x55a32169d800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1870 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:34:25.467185) v4 > -3819> 2018-07-11 07:34:25.431286 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.1 10.8.128.106:0/12677 1870 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:25.467185) v4 ==== 2004+0+0 (2767733372 0 0) 0x55a34de6f000 con 0x55a32169d800 > -3818> 2018-07-11 07:34:25.431296 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:25.467185) v4 -- 0x55a32eb14800 con 0 > -3817> 2018-07-11 07:34:25.431337 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12677 conn(0x55a32169c000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1870 0x55a31ce7a800 osd_ping(ping e494 stamp 2018-07-11 07:34:25.467185) v4 > -3816> 2018-07-11 07:34:25.431360 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.1 10.8.128.106:0/12677 1870 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:25.467185) v4 ==== 2004+0+0 (2767733372 0 0) 0x55a31ce7a800 con 0x55a32169c000 > -3815> 2018-07-11 07:34:25.431373 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:25.467185) v4 -- 0x55a332e84e00 con 0 > -3814> 2018-07-11 07:34:25.553785 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a312be7680 con 0 > -3813> 2018-07-11 07:34:25.566821 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5142 0x55a321004a00 replica scrub(pg: 6.15,from:0'0,to:430'106209,epoch:494/492,start:6:a9fe29e8:::benchmark_data_magna108.ceph.redhat.com_2478_object372856:0,end:6:a9ff3870:::benchmark_data_magna108_41525_object54078:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -3812> 2018-07-11 07:34:25.566850 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5142 ==== replica scrub(pg: 6.15,from:0'0,to:430'106209,epoch:494/492,start:6:a9fe29e8:::benchmark_data_magna108.ceph.redhat.com_2478_object372856:0,end:6:a9ff3870:::benchmark_data_magna108_41525_object54078:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 232+0+0 (3749213759 0 0) 0x55a321004a00 con 0x55a317019000 > -3811> 2018-07-11 07:34:25.582871 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.107:0/12617 conn(0x55a32447e000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1824 0x55a32447b000 osd_ping(ping e494 stamp 2018-07-11 07:34:25.582388) v4 > -3810> 2018-07-11 07:34:25.582889 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.9 10.8.128.107:0/12617 1824 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:25.582388) v4 ==== 2004+0+0 (3022383632 0 0) 0x55a32447b000 con 0x55a32447e000 > -3809> 2018-07-11 07:34:25.582883 7f355d88e700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.107:0/12617 conn(0x55a32447f800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1824 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:34:25.582388) v4 > -3808> 2018-07-11 07:34:25.582899 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:25.582388) v4 -- 0x55a3314cfc00 con 0 > -3807> 2018-07-11 07:34:25.582907 7f355d88e700 1 -- 10.8.128.107:6803/12665 <== osd.9 10.8.128.107:0/12617 1824 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:25.582388) v4 ==== 2004+0+0 (3022383632 0 0) 0x55a34de78600 con 0x55a32447f800 > -3806> 2018-07-11 07:34:25.582961 7f355d88e700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:25.582388) v4 -- 0x55a32eb14800 con 0 > -3805> 2018-07-11 07:34:25.632173 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15287 conn(0x55a311b75000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1847 0x55a332e84e00 osd_ping(ping e494 stamp 2018-07-11 07:34:26.846420) v4 > -3804> 2018-07-11 07:34:25.632179 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15287 conn(0x55a311b72000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1847 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:34:26.846420) v4 > -3803> 2018-07-11 07:34:25.632201 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.4 10.8.128.113:0/15287 1847 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:26.846420) v4 ==== 2004+0+0 (562329375 0 0) 0x55a332e84e00 con 0x55a311b75000 > -3802> 2018-07-11 07:34:25.632204 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.4 10.8.128.113:0/15287 1847 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:26.846420) v4 ==== 2004+0+0 (562329375 0 0) 0x55a332e79a00 con 0x55a311b72000 > -3801> 2018-07-11 07:34:25.632212 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:26.846420) v4 -- 0x55a3314cfc00 con 0 > -3800> 2018-07-11 07:34:25.632243 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:26.846420) v4 -- 0x55a32eb3b600 con 0 > -3799> 2018-07-11 07:34:25.758647 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15919 conn(0x55a317074800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1852 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:34:26.382590) v4 > -3798> 2018-07-11 07:34:25.758668 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.11 10.8.128.114:0/15919 1852 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:26.382590) v4 ==== 2004+0+0 (4130282053 0 0) 0x55a324474c00 con 0x55a317074800 > -3797> 2018-07-11 07:34:25.758682 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:26.382590) v4 -- 0x55a32eb3b600 con 0 > -3796> 2018-07-11 07:34:25.758696 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15919 conn(0x55a3170b4000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1852 0x55a32447b000 osd_ping(ping e494 stamp 2018-07-11 07:34:26.382590) v4 > -3795> 2018-07-11 07:34:25.758718 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.11 10.8.128.114:0/15919 1852 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:26.382590) v4 ==== 2004+0+0 (4130282053 0 0) 0x55a32447b000 con 0x55a3170b4000 > -3794> 2018-07-11 07:34:25.758729 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:26.382590) v4 -- 0x55a330934000 con 0 > -3793> 2018-07-11 07:34:26.170903 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15885 conn(0x55a3170b5800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1873 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:34:26.794880) v4 > -3792> 2018-07-11 07:34:26.170925 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.7 10.8.128.114:0/15885 1873 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:26.794880) v4 ==== 2004+0+0 (136020791 0 0) 0x55a332e79a00 con 0x55a3170b5800 > -3791> 2018-07-11 07:34:26.170937 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:26.794880) v4 -- 0x55a30f58e000 con 0 > -3790> 2018-07-11 07:34:26.170926 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15885 conn(0x55a311b76800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1873 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:34:26.794880) v4 > -3789> 2018-07-11 07:34:26.170955 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.7 10.8.128.114:0/15885 1873 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:26.794880) v4 ==== 2004+0+0 (136020791 0 0) 0x55a34de6f000 con 0x55a311b76800 > -3788> 2018-07-11 07:34:26.170967 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:26.794880) v4 -- 0x55a332eada00 con 0 > -3787> 2018-07-11 07:34:26.179107 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12650 conn(0x55a311b67000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1863 0x55a332e84e00 osd_ping(ping e494 stamp 2018-07-11 07:34:26.215086) v4 > -3786> 2018-07-11 07:34:26.179109 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12650 conn(0x55a311b68800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1863 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:34:26.215086) v4 > -3785> 2018-07-11 07:34:26.179138 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.3 10.8.128.106:0/12650 1863 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:26.215086) v4 ==== 2004+0+0 (214845963 0 0) 0x55a34de78600 con 0x55a311b68800 > -3784> 2018-07-11 07:34:26.179135 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.3 10.8.128.106:0/12650 1863 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:26.215086) v4 ==== 2004+0+0 (214845963 0 0) 0x55a332e84e00 con 0x55a311b67000 > -3783> 2018-07-11 07:34:26.179154 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:26.215086) v4 -- 0x55a310611800 con 0 > -3782> 2018-07-11 07:34:26.179182 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:26.215086) v4 -- 0x55a332fae000 con 0 > -3781> 2018-07-11 07:34:26.234705 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a324fc4900 con 0 > -3780> 2018-07-11 07:34:26.250307 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5143 0x55a321004c80 replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:494/492,start:6:a9ff3870:::benchmark_data_magna108_41525_object54078:0,end:6:aa0074b7:::benchmark_data_magna108_35617_object138932:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -3779> 2018-07-11 07:34:26.250330 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5143 ==== replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:494/492,start:6:a9ff3870:::benchmark_data_magna108_41525_object54078:0,end:6:aa0074b7:::benchmark_data_magna108_35617_object138932:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 217+0+0 (2350362785 0 0) 0x55a321004c80 con 0x55a317019000 > -3778> 2018-07-11 07:34:26.573854 7f3543899700 5 osd.10 494 heartbeat: osd_stat(784 GB used, 146 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,0,1]) > -3777> 2018-07-11 07:34:26.573868 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6802/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:34:26.573867) v4 -- 0x55a34de62e00 con 0 > -3776> 2018-07-11 07:34:26.573909 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6803/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:34:26.573867) v4 -- 0x55a332faea00 con 0 > -3775> 2018-07-11 07:34:26.573925 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6806/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:34:26.573867) v4 -- 0x55a34de6e000 con 0 > -3774> 2018-07-11 07:34:26.573940 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6807/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:34:26.573867) v4 -- 0x55a34de62600 con 0 > -3773> 2018-07-11 07:34:26.573962 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6808/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:34:26.573867) v4 -- 0x55a33235a000 con 0 > -3772> 2018-07-11 07:34:26.573984 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6810/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:34:26.573867) v4 -- 0x55a3314ce000 con 0 > -3771> 2018-07-11 07:34:26.574013 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6809/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:34:26.573867) v4 -- 0x55a33235aa00 con 0 > -3770> 2018-07-11 07:34:26.574030 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6811/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:34:26.573867) v4 -- 0x55a32447ba00 con 0 > -3769> 2018-07-11 07:34:26.574060 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6810/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:34:26.573867) v4 -- 0x55a332feaa00 con 0 > -3768> 2018-07-11 07:34:26.574087 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6811/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:34:26.573867) v4 -- 0x55a33532ca00 con 0 > -3767> 2018-07-11 07:34:26.574104 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6806/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:34:26.573867) v4 -- 0x55a332ffbc00 con 0 > -3766> 2018-07-11 07:34:26.574117 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6807/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:34:26.573867) v4 -- 0x55a332fea800 con 0 > -3765> 2018-07-11 07:34:26.574139 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6802/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:34:26.573867) v4 -- 0x55a32fb19400 con 0 > -3764> 2018-07-11 07:34:26.574161 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6803/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:34:26.573867) v4 -- 0x55a332ffb200 con 0 > -3763> 2018-07-11 07:34:26.574177 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6806/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:34:26.573867) v4 -- 0x55a332feb200 con 0 > -3762> 2018-07-11 07:34:26.574198 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6807/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:34:26.573867) v4 -- 0x55a314e7b800 con 0 > -3761> 2018-07-11 07:34:26.574226 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6802/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:34:26.573867) v4 -- 0x55a332fe8c00 con 0 > -3760> 2018-07-11 07:34:26.574266 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6803/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:34:26.573867) v4 -- 0x55a332ffb600 con 0 > -3759> 2018-07-11 07:34:26.574274 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6802/15255 conn(0x55a314e1c000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1857 0x55a32447b000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:26.573867) v4 > -3758> 2018-07-11 07:34:26.574288 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6806/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:34:26.573867) v4 -- 0x55a332ff2c00 con 0 > -3757> 2018-07-11 07:34:26.574291 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6802/15255 1857 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:26.573867) v4 ==== 2004+0+0 (2964230961 0 0) 0x55a32447b000 con 0x55a314e1c000 > -3756> 2018-07-11 07:34:26.574311 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6807/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:34:26.573867) v4 -- 0x55a332349800 con 0 > -3755> 2018-07-11 07:34:26.574327 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6802/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:34:26.573867) v4 -- 0x55a310fe7200 con 0 > -3754> 2018-07-11 07:34:26.574346 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6803/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:34:26.573867) v4 -- 0x55a310fe6200 con 0 > -3753> 2018-07-11 07:34:26.574368 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6803/15255 conn(0x55a314e1d800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1857 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:26.573867) v4 > -3752> 2018-07-11 07:34:26.574389 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6803/15255 1857 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:26.573867) v4 ==== 2004+0+0 (2964230961 0 0) 0x55a324474c00 con 0x55a314e1d800 > -3751> 2018-07-11 07:34:26.574438 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6808/12651 conn(0x55a314e96000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1857 0x55a332fae000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:26.573867) v4 > -3750> 2018-07-11 07:34:26.574462 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6808/12651 1857 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:26.573867) v4 ==== 2004+0+0 (2964230961 0 0) 0x55a332fae000 con 0x55a314e96000 > -3749> 2018-07-11 07:34:26.574492 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6809/12650 conn(0x55a314e1f000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1857 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:26.573867) v4 > -3748> 2018-07-11 07:34:26.574509 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6809/12650 1857 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:26.573867) v4 ==== 2004+0+0 (2964230961 0 0) 0x55a34de6f000 con 0x55a314e1f000 > -3747> 2018-07-11 07:34:26.574509 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6810/15287 conn(0x55a314e03000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1857 0x55a332fae000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:26.573867) v4 > -3746> 2018-07-11 07:34:26.574503 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6811/15287 conn(0x55a314e04800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1857 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:26.573867) v4 > -3745> 2018-07-11 07:34:26.574522 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6810/15287 1857 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:26.573867) v4 ==== 2004+0+0 (2964230961 0 0) 0x55a332fae000 con 0x55a314e03000 > -3744> 2018-07-11 07:34:26.574526 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6811/15287 1857 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:26.573867) v4 ==== 2004+0+0 (2964230961 0 0) 0x55a332e79a00 con 0x55a314e04800 > -3743> 2018-07-11 07:34:26.574545 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6806/12677 conn(0x55a314e84000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1857 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:26.573867) v4 > -3742> 2018-07-11 07:34:26.574559 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6807/12677 conn(0x55a314e85800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1857 0x55a332fae000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:26.573867) v4 > -3741> 2018-07-11 07:34:26.574573 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6807/12677 1857 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:26.573867) v4 ==== 2004+0+0 (2964230961 0 0) 0x55a332fae000 con 0x55a314e85800 > -3740> 2018-07-11 07:34:26.574581 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6806/12677 1857 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:26.573867) v4 ==== 2004+0+0 (2964230961 0 0) 0x55a34de6f000 con 0x55a314e84000 > -3739> 2018-07-11 07:34:26.574593 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6810/12651 conn(0x55a314e97800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1857 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:26.573867) v4 > -3738> 2018-07-11 07:34:26.574607 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6811/12650 conn(0x55a314e20800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1857 0x55a332fae000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:26.573867) v4 > -3737> 2018-07-11 07:34:26.574610 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6810/12651 1857 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:26.573867) v4 ==== 2004+0+0 (2964230961 0 0) 0x55a332e79a00 con 0x55a314e97800 > -3736> 2018-07-11 07:34:26.574612 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6807/12617 conn(0x55a314e27800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1857 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:26.573867) v4 > -3735> 2018-07-11 07:34:26.574619 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6811/12650 1857 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:26.573867) v4 ==== 2004+0+0 (2964230961 0 0) 0x55a332fae000 con 0x55a314e20800 > -3734> 2018-07-11 07:34:26.574624 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6807/12617 1857 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:26.573867) v4 ==== 2004+0+0 (2964230961 0 0) 0x55a34de6f000 con 0x55a314e27800 > -3733> 2018-07-11 07:34:26.574646 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6806/12617 conn(0x55a314e26000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1857 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:26.573867) v4 > -3732> 2018-07-11 07:34:26.574663 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6806/12617 1857 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:26.573867) v4 ==== 2004+0+0 (2964230961 0 0) 0x55a332e79a00 con 0x55a314e26000 > -3731> 2018-07-11 07:34:26.574664 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6806/15263 conn(0x55a314e99000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1857 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:26.573867) v4 > -3730> 2018-07-11 07:34:26.574660 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6807/15263 conn(0x55a314e9a800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1857 0x55a332fae000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:26.573867) v4 > -3729> 2018-07-11 07:34:26.574678 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6806/15263 1857 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:26.573867) v4 ==== 2004+0+0 (2964230961 0 0) 0x55a34de6f000 con 0x55a314e99000 > -3728> 2018-07-11 07:34:26.574685 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6807/15263 1857 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:26.573867) v4 ==== 2004+0+0 (2964230961 0 0) 0x55a332fae000 con 0x55a314e9a800 > -3727> 2018-07-11 07:34:26.574706 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6807/15885 conn(0x55a314e01800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1857 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:26.573867) v4 > -3726> 2018-07-11 07:34:26.574708 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6806/15885 conn(0x55a314e00000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1857 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:26.573867) v4 > -3725> 2018-07-11 07:34:26.574718 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6807/15885 1857 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:26.573867) v4 ==== 2004+0+0 (2964230961 0 0) 0x55a34de6f000 con 0x55a314e01800 > -3724> 2018-07-11 07:34:26.574720 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6806/15885 1857 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:26.573867) v4 ==== 2004+0+0 (2964230961 0 0) 0x55a332e79a00 con 0x55a314e00000 > -3723> 2018-07-11 07:34:26.574725 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6803/11967 conn(0x55a314e13800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1857 0x55a332fae000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:26.573867) v4 > -3722> 2018-07-11 07:34:26.574744 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6803/11967 1857 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:26.573867) v4 ==== 2004+0+0 (2964230961 0 0) 0x55a332fae000 con 0x55a314e13800 > -3721> 2018-07-11 07:34:26.574747 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6803/13323 conn(0x55a314e16800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1857 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:26.573867) v4 > -3720> 2018-07-11 07:34:26.574762 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6803/13323 1857 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:26.573867) v4 ==== 2004+0+0 (2964230961 0 0) 0x55a34de6f000 con 0x55a314e16800 > -3719> 2018-07-11 07:34:26.574775 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6802/13323 conn(0x55a314e15000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1857 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:26.573867) v4 > -3718> 2018-07-11 07:34:26.574790 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6802/13323 1857 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:26.573867) v4 ==== 2004+0+0 (2964230961 0 0) 0x55a332e79a00 con 0x55a314e15000 > -3717> 2018-07-11 07:34:26.574796 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6802/11967 conn(0x55a314e12000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1857 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:26.573867) v4 > -3716> 2018-07-11 07:34:26.574809 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6802/11967 1857 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:26.573867) v4 ==== 2004+0+0 (2964230961 0 0) 0x55a34de6f000 con 0x55a314e12000 > -3715> 2018-07-11 07:34:26.574832 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6802/15919 conn(0x55a314e87000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1857 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:26.573867) v4 > -3714> 2018-07-11 07:34:26.574843 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6803/15919 conn(0x55a314e88800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1857 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:26.573867) v4 > -3713> 2018-07-11 07:34:26.574845 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6802/15919 1857 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:26.573867) v4 ==== 2004+0+0 (2964230961 0 0) 0x55a332e79a00 con 0x55a314e87000 > -3712> 2018-07-11 07:34:26.574857 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6803/15919 1857 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:26.573867) v4 ==== 2004+0+0 (2964230961 0 0) 0x55a34de6f000 con 0x55a314e88800 > -3711> 2018-07-11 07:34:26.630114 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15255 conn(0x55a311b64000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1873 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:34:27.844233) v4 > -3710> 2018-07-11 07:34:26.630133 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.0 10.8.128.113:0/15255 1873 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:27.844233) v4 ==== 2004+0+0 (2525814379 0 0) 0x55a34de6f000 con 0x55a311b64000 > -3709> 2018-07-11 07:34:26.630128 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15255 conn(0x55a311b65800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1873 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:34:27.844233) v4 > -3708> 2018-07-11 07:34:26.630142 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:27.844233) v4 -- 0x55a34de6e000 con 0 > -3707> 2018-07-11 07:34:26.630154 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.0 10.8.128.113:0/15255 1873 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:27.844233) v4 ==== 2004+0+0 (2525814379 0 0) 0x55a332e79a00 con 0x55a311b65800 > -3706> 2018-07-11 07:34:26.630166 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:27.844233) v4 -- 0x55a332faea00 con 0 > -3705> 2018-07-11 07:34:26.679504 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12650 conn(0x55a311b67000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1864 0x55a332fae000 osd_ping(ping e494 stamp 2018-07-11 07:34:26.715509) v4 > -3704> 2018-07-11 07:34:26.679504 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12650 conn(0x55a311b68800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1864 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:34:26.715509) v4 > -3703> 2018-07-11 07:34:26.679533 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.3 10.8.128.106:0/12650 1864 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:26.715509) v4 ==== 2004+0+0 (3044307955 0 0) 0x55a332fae000 con 0x55a311b67000 > -3702> 2018-07-11 07:34:26.679536 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.3 10.8.128.106:0/12650 1864 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:26.715509) v4 ==== 2004+0+0 (3044307955 0 0) 0x55a34de78600 con 0x55a311b68800 > -3701> 2018-07-11 07:34:26.679552 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:26.715509) v4 -- 0x55a34de62e00 con 0 > -3700> 2018-07-11 07:34:26.679582 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:26.715509) v4 -- 0x55a310611800 con 0 > -3699> 2018-07-11 07:34:26.744857 7f35498a5700 10 monclient: tick > -3698> 2018-07-11 07:34:26.744870 7f35498a5700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2018-07-11 07:33:56.744868) > -3697> 2018-07-11 07:34:26.976195 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a324fc5d40 con 0 > -3696> 2018-07-11 07:34:27.002355 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5144 0x55a321004f00 replica scrub(pg: 6.15,from:0'0,to:430'105927,epoch:494/492,start:6:aa0074b7:::benchmark_data_magna108_35617_object138932:0,end:6:aa011510:::benchmark_data_magna108.ceph.redhat.com_2478_object117601:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -3695> 2018-07-11 07:34:27.002385 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5144 ==== replica scrub(pg: 6.15,from:0'0,to:430'105927,epoch:494/492,start:6:aa0074b7:::benchmark_data_magna108_35617_object138932:0,end:6:aa011510:::benchmark_data_magna108.ceph.redhat.com_2478_object117601:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (1216552872 0 0) 0x55a321004f00 con 0x55a317019000 > -3694> 2018-07-11 07:34:27.054158 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b7000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1800 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:34:26.804861) v4 > -3693> 2018-07-11 07:34:27.054172 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b8800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1800 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:34:26.804861) v4 > -3692> 2018-07-11 07:34:27.054183 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.8 10.8.128.108:0/11967 1800 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:26.804861) v4 ==== 2004+0+0 (161416487 0 0) 0x55a34de6f000 con 0x55a3170b7000 > -3691> 2018-07-11 07:34:27.054191 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.8 10.8.128.108:0/11967 1800 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:26.804861) v4 ==== 2004+0+0 (161416487 0 0) 0x55a324474c00 con 0x55a3170b8800 > -3690> 2018-07-11 07:34:27.054205 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:26.804861) v4 -- 0x55a310611800 con 0 > -3689> 2018-07-11 07:34:27.054229 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:26.804861) v4 -- 0x55a332faea00 con 0 > -3688> 2018-07-11 07:34:27.066430 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12651 conn(0x55a3216a0800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1848 0x55a332fae000 osd_ping(ping e494 stamp 2018-07-11 07:34:27.102431) v4 > -3687> 2018-07-11 07:34:27.066455 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.2 10.8.128.106:0/12651 1848 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:27.102431) v4 ==== 2004+0+0 (3158301116 0 0) 0x55a332fae000 con 0x55a3216a0800 > -3686> 2018-07-11 07:34:27.066466 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:27.102431) v4 -- 0x55a34de62e00 con 0 > -3685> 2018-07-11 07:34:27.066506 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12651 conn(0x55a32169f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1848 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:34:27.102431) v4 > -3684> 2018-07-11 07:34:27.066529 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.2 10.8.128.106:0/12651 1848 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:27.102431) v4 ==== 2004+0+0 (3158301116 0 0) 0x55a332e79a00 con 0x55a32169f000 > -3683> 2018-07-11 07:34:27.066545 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:27.102431) v4 -- 0x55a332faea00 con 0 > -3682> 2018-07-11 07:34:27.255426 7f35488a3700 1 -- 10.8.128.107:6800/12665 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55a32c6a22c0 con 0 > -3681> 2018-07-11 07:34:27.255487 7f35488a3700 1 -- 10.8.128.107:6800/12665 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55a32a57a580 con 0 > -3680> 2018-07-11 07:34:27.689796 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a34f39e000 con 0 > -3679> 2018-07-11 07:34:27.708555 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5145 0x55a321005180 replica scrub(pg: 6.15,from:0'0,to:430'106691,epoch:494/492,start:6:aa011510:::benchmark_data_magna108.ceph.redhat.com_2478_object117601:0,end:6:aa01d4af:::benchmark_data_magna108.ceph.redhat.com_2478_object89979:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -3678> 2018-07-11 07:34:27.708589 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5145 ==== replica scrub(pg: 6.15,from:0'0,to:430'106691,epoch:494/492,start:6:aa011510:::benchmark_data_magna108.ceph.redhat.com_2478_object117601:0,end:6:aa01d4af:::benchmark_data_magna108.ceph.redhat.com_2478_object89979:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 247+0+0 (2839804919 0 0) 0x55a321005180 con 0x55a317019000 > -3677> 2018-07-11 07:34:27.730540 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15255 conn(0x55a311b64000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1874 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:34:28.944795) v4 > -3676> 2018-07-11 07:34:27.730566 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.0 10.8.128.113:0/15255 1874 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:28.944795) v4 ==== 2004+0+0 (2727501295 0 0) 0x55a34de78600 con 0x55a311b64000 > -3675> 2018-07-11 07:34:27.730577 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15255 conn(0x55a311b65800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1874 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:34:28.944795) v4 > -3674> 2018-07-11 07:34:27.730584 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:28.944795) v4 -- 0x55a310611800 con 0 > -3673> 2018-07-11 07:34:27.730600 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.0 10.8.128.113:0/15255 1874 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:28.944795) v4 ==== 2004+0+0 (2727501295 0 0) 0x55a324474c00 con 0x55a311b65800 > -3672> 2018-07-11 07:34:27.730623 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:28.944795) v4 -- 0x55a332faea00 con 0 > -3671> 2018-07-11 07:34:28.315732 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a323b28fc0 con 0 > -3670> 2018-07-11 07:34:28.325597 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5146 0x55a321005400 replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:494/492,start:6:aa01d4af:::benchmark_data_magna108.ceph.redhat.com_2478_object89979:0,end:6:aa031ab1:::benchmark_data_magna108_35617_object1009607:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -3669> 2018-07-11 07:34:28.325627 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5146 ==== replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:494/492,start:6:aa01d4af:::benchmark_data_magna108.ceph.redhat.com_2478_object89979:0,end:6:aa031ab1:::benchmark_data_magna108_35617_object1009607:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (2432398190 0 0) 0x55a321005400 con 0x55a317019000 > -3668> 2018-07-11 07:34:28.331660 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12677 conn(0x55a32169d800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1871 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:34:28.367670) v4 > -3667> 2018-07-11 07:34:28.331683 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.1 10.8.128.106:0/12677 1871 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:28.367670) v4 ==== 2004+0+0 (1381936324 0 0) 0x55a34de6f000 con 0x55a32169d800 > -3666> 2018-07-11 07:34:28.331694 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:28.367670) v4 -- 0x55a310611800 con 0 > -3665> 2018-07-11 07:34:28.331708 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12677 conn(0x55a32169c000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1871 0x55a32447b000 osd_ping(ping e494 stamp 2018-07-11 07:34:28.367670) v4 > -3664> 2018-07-11 07:34:28.331731 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.1 10.8.128.106:0/12677 1871 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:28.367670) v4 ==== 2004+0+0 (1381936324 0 0) 0x55a32447b000 con 0x55a32169c000 > -3663> 2018-07-11 07:34:28.331747 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:28.367670) v4 -- 0x55a34de62e00 con 0 > -3662> 2018-07-11 07:34:28.831039 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15255 conn(0x55a311b64000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1875 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:34:30.045229) v4 > -3661> 2018-07-11 07:34:28.831040 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15255 conn(0x55a311b65800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1875 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:34:30.045229) v4 > -3660> 2018-07-11 07:34:28.831064 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.0 10.8.128.113:0/15255 1875 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:30.045229) v4 ==== 2004+0+0 (3432619877 0 0) 0x55a34de78600 con 0x55a311b64000 > -3659> 2018-07-11 07:34:28.831066 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.0 10.8.128.113:0/15255 1875 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:30.045229) v4 ==== 2004+0+0 (3432619877 0 0) 0x55a332e79a00 con 0x55a311b65800 > -3658> 2018-07-11 07:34:28.831083 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:30.045229) v4 -- 0x55a310611800 con 0 > -3657> 2018-07-11 07:34:28.831117 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:30.045229) v4 -- 0x55a332faea00 con 0 > -3656> 2018-07-11 07:34:29.132643 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15287 conn(0x55a311b75000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1848 0x55a332fae000 osd_ping(ping e494 stamp 2018-07-11 07:34:30.346809) v4 > -3655> 2018-07-11 07:34:29.132664 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.4 10.8.128.113:0/15287 1848 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:30.346809) v4 ==== 2004+0+0 (2005451479 0 0) 0x55a332fae000 con 0x55a311b75000 > -3654> 2018-07-11 07:34:29.132673 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:30.346809) v4 -- 0x55a34de62e00 con 0 > -3653> 2018-07-11 07:34:29.132681 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15287 conn(0x55a311b72000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1848 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:34:30.346809) v4 > -3652> 2018-07-11 07:34:29.132704 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.4 10.8.128.113:0/15287 1848 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:30.346809) v4 ==== 2004+0+0 (2005451479 0 0) 0x55a324474c00 con 0x55a311b72000 > -3651> 2018-07-11 07:34:29.132722 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:30.346809) v4 -- 0x55a30f58e000 con 0 > -3650> 2018-07-11 07:34:29.220922 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15263 conn(0x55a321664000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1846 0x55a32447b000 osd_ping(ping e494 stamp 2018-07-11 07:34:30.435181) v4 > -3649> 2018-07-11 07:34:29.220948 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.5 10.8.128.113:0/15263 1846 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:30.435181) v4 ==== 2004+0+0 (2442426117 0 0) 0x55a32447b000 con 0x55a321664000 > -3648> 2018-07-11 07:34:29.220961 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:30.435181) v4 -- 0x55a332eaca00 con 0 > -3647> 2018-07-11 07:34:29.221020 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15263 conn(0x55a321667000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1846 0x55a32447b000 osd_ping(ping e494 stamp 2018-07-11 07:34:30.435181) v4 > -3646> 2018-07-11 07:34:29.221041 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.5 10.8.128.113:0/15263 1846 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:30.435181) v4 ==== 2004+0+0 (2442426117 0 0) 0x55a32447b000 con 0x55a321667000 > -3645> 2018-07-11 07:34:29.221057 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:30.435181) v4 -- 0x55a32eb15c00 con 0 > -3644> 2018-07-11 07:34:29.386818 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 59] Generated table #36586: 4239989 keys, 68723933 bytes > -3643> 2018-07-11 07:34:29.386866 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294469386838, "cf_name": "default", "job": 59, "event": "table_file_creation", "file_number": 36586, "file_size": 68723933, "table_properties": {"data_size": 57400324, "index_size": 589406, "filter_size": 10733214, "raw_key_size": 122488576, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 14087, "num_entries": 4239989, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4239989", "kMergeOperands": "0"}} > -3642> 2018-07-11 07:34:29.386902 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 59] Compacted 1@1 + 7@2 files to L2 => 642786464 bytes > -3641> 2018-07-11 07:34:29.403628 7f354e0ae700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 64993007208 > -3640> 2018-07-11 07:34:29.403754 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:34:29.403592) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 19 83 446 3641 796 0] max score 6.95, MB/sec: 11.3 rd, 11.3 wr, level 2, files in(1, 7) out(9) MB in(76.6, 536.4) out(613.0), read-write-amplify(16.0) write-amplify(8.0) OK, records in: 39657320, records dropped: 0 > > -3639> 2018-07-11 07:34:29.403764 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:34:29.403619) EVENT_LOG_v1 {"time_micros": 1531294469403607, "job": 59, "event": "compaction_finished", "compaction_time_micros": 56889094, "output_level": 2, "num_output_files": 9, "total_output_size": 642786464, "num_input_records": 39657320, "num_output_records": 39657320, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 19, 83, 446, 3641, 796, 0]} > -3638> 2018-07-11 07:34:29.404980 7f354e0ae700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS ------- > -3637> 2018-07-11 07:34:29.404987 7f354e0ae700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] >** DB Stats ** >Uptime(secs): 6023.6 total, 601.8 interval >Cumulative writes: 896 writes, 226M keys, 896 commit groups, 1.0 writes per commit group, ingest: 4.91 GB, 0.84 MB/s >Cumulative WAL: 896 writes, 252 syncs, 3.54 writes per sync, written: 4.91 GB, 0.84 MB/s >Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent >Interval writes: 2 writes, 30M keys, 2 commit groups, 0.7 writes per commit group, ingest: 654.86 MB, 1.09 MB/s >Interval WAL: 2 writes, 1 syncs, 1.00 writes per sync, written: 0.64 MB, 1.09 MB/s >Interval stall: 00:00:0.000 H:M:S, 0.0 percent > >** Compaction Stats [default] ** >Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop >---------------------------------------------------------------------------------------------------------------------------------------------------------- > L0 0/0 0.00 KB 0.0 0.0 0.0 0.0 3.6 3.6 0.0 1.0 0.0 20.5 178 9 19.831 0 0 > L1 19/0 1.11 GB 6.9 7.2 3.6 3.6 7.2 3.6 0.0 2.0 14.9 14.9 491 6 81.821 374M 149 > L2 83/0 5.13 GB 3.0 8.2 2.7 5.6 8.2 2.7 0.0 3.1 11.8 11.8 714 33 21.649 497M 55 > L3 446/0 24.93 GB 1.0 0.1 0.1 0.1 0.1 0.1 0.1 2.0 18.6 18.6 8 1 7.636 5276K 0 > L4 3641/0 250.00 GB 1.0 0.8 0.1 0.6 0.8 0.1 0.1 5.5 21.8 21.8 36 2 17.944 29M 0 > L5 796/0 55.24 GB 0.0 0.0 0.0 0.0 0.0 0.0 0.3 0.0 0.0 0.0 0 0 0.000 0 0 > Sum 4985/0 336.41 GB 0.0 16.3 6.4 9.8 19.9 10.0 0.5 5.5 11.7 14.3 1427 51 27.987 906M 204 > Int 0/0 0.00 KB 0.0 6.9 2.6 4.3 7.4 3.1 0.0 14.9 11.3 12.1 627 17 36.890 413M 1 >Uptime(secs): 6023.6 total, 6023.6 interval >Flush(GB): cumulative 3.581, interval 0.498 >AddFile(GB): cumulative 0.000, interval 0.000 >AddFile(Total Files): cumulative 0, interval 0 >AddFile(L0 Files): cumulative 0, interval 0 >AddFile(Keys): cumulative 0, interval 0 >Cumulative compaction: 19.87 GB write, 3.38 MB/s write, 16.29 GB read, 2.77 MB/s read, 1427.4 seconds >Interval compaction: 7.44 GB write, 1.26 MB/s write, 6.94 GB read, 1.18 MB/s read, 627.1 seconds >Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count > >** File Read Latency Histogram By Level [default] ** > >** Compaction Stats [default] ** >Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop >---------------------------------------------------------------------------------------------------------------------------------------------------------- > L0 0/0 0.00 KB 0.0 0.0 0.0 0.0 3.6 3.6 0.0 1.0 0.0 20.5 178 9 19.831 0 0 > L1 19/0 1.11 GB 6.9 7.2 3.6 3.6 7.2 3.6 0.0 2.0 14.9 14.9 491 6 81.821 374M 149 > L2 83/0 5.13 GB 3.0 8.2 2.7 5.6 8.2 2.7 0.0 3.1 11.8 11.8 714 33 21.649 497M 55 > L3 446/0 24.93 GB 1.0 0.1 0.1 0.1 0.1 0.1 0.1 2.0 18.6 18.6 8 1 7.636 5276K 0 > L4 3641/0 250.00 GB 1.0 0.8 0.1 0.6 0.8 0.1 0.1 5.5 21.8 21.8 36 2 17.944 29M 0 > L5 796/0 55.24 GB 0.0 0.0 0.0 0.0 0.0 0.0 0.3 0.0 0.0 0.0 0 0 0.000 0 0 > Sum 4985/0 336.41 GB 0.0 16.3 6.4 9.8 19.9 10.0 0.5 5.5 11.7 14.3 1427 51 27.987 906M 204 > Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 >Uptime(secs): 6023.6 total, 6023.6 interval >Flush(GB): cumulative 3.581, interval 0.000 >AddFile(GB): cumulative 0.000, interval 0.000 >AddFile(Total Files): cumulative 0, interval 0 >AddFile(L0 Files): cumulative 0, interval 0 >AddFile(Keys): cumulative 0, interval 0 >Cumulative compaction: 19.87 GB write, 3.38 MB/s write, 16.29 GB read, 2.77 MB/s read, 1427.4 seconds >Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds >Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count > >** File Read Latency Histogram By Level [default] ** > > -3636> 2018-07-11 07:34:29.405115 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 60] Compacting 1@1 + 5@2 files to L2, score 6.95 > -3635> 2018-07-11 07:34:29.405128 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 59 Base level 1, inputs: [36561(76MB)], [36504(76MB) 36505(76MB) 36506(33MB) 36507(14MB) 36546(62MB)] > > -3634> 2018-07-11 07:34:29.405143 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294469405131, "job": 60, "event": "compaction_started", "files_L1": [36561], "files_L2": [36504, 36505, 36506, 36507, 36546], "score": 6.94871, "input_data_size": 357728903} > -3633> 2018-07-11 07:34:29.552931 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.115:0/13323 conn(0x55a321665800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1861 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:34:29.890548) v4 > -3632> 2018-07-11 07:34:29.552953 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.6 10.8.128.115:0/13323 1861 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:29.890548) v4 ==== 2004+0+0 (839264398 0 0) 0x55a332e79a00 con 0x55a321665800 > -3631> 2018-07-11 07:34:29.552963 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:29.890548) v4 -- 0x55a33dff0600 con 0 > -3630> 2018-07-11 07:34:29.553002 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.115:0/13323 conn(0x55a321668800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1861 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:34:29.890548) v4 > -3629> 2018-07-11 07:34:29.553016 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.6 10.8.128.115:0/13323 1861 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:29.890548) v4 ==== 2004+0+0 (839264398 0 0) 0x55a332e79a00 con 0x55a321668800 > -3628> 2018-07-11 07:34:29.553026 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:29.890548) v4 -- 0x55a34de6fe00 con 0 > -3627> 2018-07-11 07:34:29.671382 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15885 conn(0x55a311b76800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1874 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:34:30.295291) v4 > -3626> 2018-07-11 07:34:29.671385 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15885 conn(0x55a3170b5800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1874 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:34:30.295291) v4 > -3625> 2018-07-11 07:34:29.671407 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.7 10.8.128.114:0/15885 1874 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:30.295291) v4 ==== 2004+0+0 (991217100 0 0) 0x55a34de6f000 con 0x55a311b76800 > -3624> 2018-07-11 07:34:29.671411 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.7 10.8.128.114:0/15885 1874 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:30.295291) v4 ==== 2004+0+0 (991217100 0 0) 0x55a324474c00 con 0x55a3170b5800 > -3623> 2018-07-11 07:34:29.671421 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:30.295291) v4 -- 0x55a33235a400 con 0 > -3622> 2018-07-11 07:34:29.671445 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:30.295291) v4 -- 0x55a33dff0600 con 0 > -3621> 2018-07-11 07:34:30.074514 7f3543899700 5 osd.10 494 heartbeat: osd_stat(784 GB used, 146 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,0,0,0,1]) > -3620> 2018-07-11 07:34:30.074532 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6802/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:34:30.074530) v4 -- 0x55a310fe6e00 con 0 > -3619> 2018-07-11 07:34:30.074571 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6803/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:34:30.074530) v4 -- 0x55a310fe7600 con 0 > -3618> 2018-07-11 07:34:30.074588 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6806/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:34:30.074530) v4 -- 0x55a332ff2200 con 0 > -3617> 2018-07-11 07:34:30.074615 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6807/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:34:30.074530) v4 -- 0x55a32e456e00 con 0 > -3616> 2018-07-11 07:34:30.074627 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6808/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:34:30.074530) v4 -- 0x55a332ff2800 con 0 > -3615> 2018-07-11 07:34:30.074638 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6810/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:34:30.074530) v4 -- 0x55a33235b200 con 0 > -3614> 2018-07-11 07:34:30.074650 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6809/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:34:30.074530) v4 -- 0x55a32e457400 con 0 > -3613> 2018-07-11 07:34:30.074662 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6811/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:34:30.074530) v4 -- 0x55a332ffa400 con 0 > -3612> 2018-07-11 07:34:30.074676 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6810/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:34:30.074530) v4 -- 0x55a32fb19e00 con 0 > -3611> 2018-07-11 07:34:30.074693 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6811/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:34:30.074530) v4 -- 0x55a32e457200 con 0 > -3610> 2018-07-11 07:34:30.074705 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6806/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:34:30.074530) v4 -- 0x55a332ffba00 con 0 > -3609> 2018-07-11 07:34:30.074724 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6807/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:34:30.074530) v4 -- 0x55a32fb19c00 con 0 > -3608> 2018-07-11 07:34:30.074738 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6802/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:34:30.074530) v4 -- 0x55a332ffb000 con 0 > -3607> 2018-07-11 07:34:30.074750 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6803/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:34:30.074530) v4 -- 0x55a32e456c00 con 0 > -3606> 2018-07-11 07:34:30.074763 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6806/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:34:30.074530) v4 -- 0x55a33234ac00 con 0 > -3605> 2018-07-11 07:34:30.074780 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6807/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:34:30.074530) v4 -- 0x55a332385c00 con 0 > -3604> 2018-07-11 07:34:30.074792 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6802/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:34:30.074530) v4 -- 0x55a332384a00 con 0 > -3603> 2018-07-11 07:34:30.074806 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6803/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:34:30.074530) v4 -- 0x55a310fe6000 con 0 > -3602> 2018-07-11 07:34:30.074821 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6806/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:34:30.074530) v4 -- 0x55a332ffb800 con 0 > -3601> 2018-07-11 07:34:30.074834 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6807/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:34:30.074530) v4 -- 0x55a32ee76e00 con 0 > -3600> 2018-07-11 07:34:30.074850 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6802/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:34:30.074530) v4 -- 0x55a310fe6400 con 0 > -3599> 2018-07-11 07:34:30.074863 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6803/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:34:30.074530) v4 -- 0x55a32ee77600 con 0 > -3598> 2018-07-11 07:34:30.075021 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6807/12617 conn(0x55a314e27800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1858 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:30.074530) v4 > -3597> 2018-07-11 07:34:30.075043 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6807/12617 1858 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:30.074530) v4 ==== 2004+0+0 (3569615935 0 0) 0x55a34de78600 con 0x55a314e27800 > -3596> 2018-07-11 07:34:30.075062 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6807/12677 conn(0x55a314e85800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1858 0x55a332fae000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:30.074530) v4 > -3595> 2018-07-11 07:34:30.075083 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6807/12677 1858 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:30.074530) v4 ==== 2004+0+0 (3569615935 0 0) 0x55a332fae000 con 0x55a314e85800 > -3594> 2018-07-11 07:34:30.075056 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6806/12617 conn(0x55a314e26000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1858 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:30.074530) v4 > -3593> 2018-07-11 07:34:30.075097 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6806/12617 1858 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:30.074530) v4 ==== 2004+0+0 (3569615935 0 0) 0x55a324474c00 con 0x55a314e26000 > -3592> 2018-07-11 07:34:30.075120 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6806/12677 conn(0x55a314e84000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1858 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:30.074530) v4 > -3591> 2018-07-11 07:34:30.075127 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6808/12651 conn(0x55a314e96000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1858 0x55a332fae000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:30.074530) v4 > -3590> 2018-07-11 07:34:30.075136 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6806/12677 1858 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:30.074530) v4 ==== 2004+0+0 (3569615935 0 0) 0x55a34de78600 con 0x55a314e84000 > -3589> 2018-07-11 07:34:30.075138 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6810/12651 conn(0x55a314e97800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1858 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:30.074530) v4 > -3588> 2018-07-11 07:34:30.075138 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6808/12651 1858 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:30.074530) v4 ==== 2004+0+0 (3569615935 0 0) 0x55a332fae000 con 0x55a314e96000 > -3587> 2018-07-11 07:34:30.075151 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6810/12651 1858 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:30.074530) v4 ==== 2004+0+0 (3569615935 0 0) 0x55a324474c00 con 0x55a314e97800 > -3586> 2018-07-11 07:34:30.075179 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6803/15255 conn(0x55a314e1d800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1858 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:30.074530) v4 > -3585> 2018-07-11 07:34:30.075197 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6803/15255 1858 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:30.074530) v4 ==== 2004+0+0 (3569615935 0 0) 0x55a324474c00 con 0x55a314e1d800 > -3584> 2018-07-11 07:34:30.075194 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6809/12650 conn(0x55a314e1f000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1858 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:30.074530) v4 > -3583> 2018-07-11 07:34:30.075206 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6809/12650 1858 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:30.074530) v4 ==== 2004+0+0 (3569615935 0 0) 0x55a34de78600 con 0x55a314e1f000 > -3582> 2018-07-11 07:34:30.075231 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6811/12650 conn(0x55a314e20800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1858 0x55a332fae000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:30.074530) v4 > -3581> 2018-07-11 07:34:30.075235 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6811/15287 conn(0x55a314e04800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1858 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:30.074530) v4 > -3580> 2018-07-11 07:34:30.075264 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6811/12650 1858 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:30.074530) v4 ==== 2004+0+0 (3569615935 0 0) 0x55a332fae000 con 0x55a314e20800 > -3579> 2018-07-11 07:34:30.075271 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6811/15287 1858 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:30.074530) v4 ==== 2004+0+0 (3569615935 0 0) 0x55a324474c00 con 0x55a314e04800 > -3578> 2018-07-11 07:34:30.075272 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6806/15263 conn(0x55a314e99000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1858 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:30.074530) v4 > -3577> 2018-07-11 07:34:30.075294 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6806/15263 1858 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:30.074530) v4 ==== 2004+0+0 (3569615935 0 0) 0x55a34de78600 con 0x55a314e99000 > -3576> 2018-07-11 07:34:30.075304 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6810/15287 conn(0x55a314e03000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1858 0x55a332fae000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:30.074530) v4 > -3575> 2018-07-11 07:34:30.075314 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6810/15287 1858 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:30.074530) v4 ==== 2004+0+0 (3569615935 0 0) 0x55a332fae000 con 0x55a314e03000 > -3574> 2018-07-11 07:34:30.075313 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6802/13323 conn(0x55a314e15000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1858 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:30.074530) v4 > -3573> 2018-07-11 07:34:30.075323 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6802/13323 1858 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:30.074530) v4 ==== 2004+0+0 (3569615935 0 0) 0x55a324474c00 con 0x55a314e15000 > -3572> 2018-07-11 07:34:30.075344 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6807/15263 conn(0x55a314e9a800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1858 0x55a332fae000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:30.074530) v4 > -3571> 2018-07-11 07:34:30.075343 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6803/13323 conn(0x55a314e16800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1858 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:30.074530) v4 > -3570> 2018-07-11 07:34:30.075354 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6806/15885 conn(0x55a314e00000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1858 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:30.074530) v4 > -3569> 2018-07-11 07:34:30.075358 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6807/15263 1858 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:30.074530) v4 ==== 2004+0+0 (3569615935 0 0) 0x55a332fae000 con 0x55a314e9a800 > -3568> 2018-07-11 07:34:30.075363 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6806/15885 1858 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:30.074530) v4 ==== 2004+0+0 (3569615935 0 0) 0x55a324474c00 con 0x55a314e00000 > -3567> 2018-07-11 07:34:30.075364 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6803/13323 1858 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:30.074530) v4 ==== 2004+0+0 (3569615935 0 0) 0x55a34de78600 con 0x55a314e16800 > -3566> 2018-07-11 07:34:30.075374 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6802/15255 conn(0x55a314e1c000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1858 0x55a332fae000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:30.074530) v4 > -3565> 2018-07-11 07:34:30.075384 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6802/15255 1858 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:30.074530) v4 ==== 2004+0+0 (3569615935 0 0) 0x55a332fae000 con 0x55a314e1c000 > -3564> 2018-07-11 07:34:30.075407 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6802/15919 conn(0x55a314e87000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1858 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:30.074530) v4 > -3563> 2018-07-11 07:34:30.075409 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6802/11967 conn(0x55a314e12000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1858 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:30.074530) v4 > -3562> 2018-07-11 07:34:30.075417 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6803/11967 conn(0x55a314e13800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1858 0x55a332fae000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:30.074530) v4 > -3561> 2018-07-11 07:34:30.075423 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6802/11967 1858 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:30.074530) v4 ==== 2004+0+0 (3569615935 0 0) 0x55a34de78600 con 0x55a314e12000 > -3560> 2018-07-11 07:34:30.075421 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6802/15919 1858 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:30.074530) v4 ==== 2004+0+0 (3569615935 0 0) 0x55a324474c00 con 0x55a314e87000 > -3559> 2018-07-11 07:34:30.075432 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6803/11967 1858 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:30.074530) v4 ==== 2004+0+0 (3569615935 0 0) 0x55a332fae000 con 0x55a314e13800 > -3558> 2018-07-11 07:34:30.075455 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6803/15919 conn(0x55a314e88800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1858 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:30.074530) v4 > -3557> 2018-07-11 07:34:30.075471 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6803/15919 1858 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:30.074530) v4 ==== 2004+0+0 (3569615935 0 0) 0x55a34de78600 con 0x55a314e88800 > -3556> 2018-07-11 07:34:30.075502 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6807/15885 conn(0x55a314e01800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1858 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:30.074530) v4 > -3555> 2018-07-11 07:34:30.075514 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6807/15885 1858 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:30.074530) v4 ==== 2004+0+0 (3569615935 0 0) 0x55a34de78600 con 0x55a314e01800 > -3554> 2018-07-11 07:34:30.257093 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a31ff88000 con 0 > -3553> 2018-07-11 07:34:30.277036 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5147 0x55a321005680 replica scrub(pg: 6.15,from:0'0,to:430'105711,epoch:494/492,start:6:aa031ab1:::benchmark_data_magna108_35617_object1009607:0,end:6:aa03d2b5:::benchmark_data_magna108_35617_object813411:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -3552> 2018-07-11 07:34:30.277067 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5147 ==== replica scrub(pg: 6.15,from:0'0,to:430'105711,epoch:494/492,start:6:aa031ab1:::benchmark_data_magna108_35617_object1009607:0,end:6:aa03d2b5:::benchmark_data_magna108_35617_object813411:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 219+0+0 (2588648394 0 0) 0x55a321005680 con 0x55a317019000 > -3551> 2018-07-11 07:34:30.283495 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.107:0/12617 conn(0x55a32447e000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1825 0x55a332fae000 osd_ping(ping e494 stamp 2018-07-11 07:34:30.283003) v4 > -3550> 2018-07-11 07:34:30.283519 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.9 10.8.128.107:0/12617 1825 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:30.283003) v4 ==== 2004+0+0 (1298187333 0 0) 0x55a332fae000 con 0x55a32447e000 > -3549> 2018-07-11 07:34:30.283524 7f355d88e700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.107:0/12617 conn(0x55a32447f800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1825 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:34:30.283003) v4 > -3548> 2018-07-11 07:34:30.283548 7f355d88e700 1 -- 10.8.128.107:6803/12665 <== osd.9 10.8.128.107:0/12617 1825 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:30.283003) v4 ==== 2004+0+0 (1298187333 0 0) 0x55a34de78600 con 0x55a32447f800 > -3547> 2018-07-11 07:34:30.283536 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:30.283003) v4 -- 0x55a32e456e00 con 0 > -3546> 2018-07-11 07:34:30.283589 7f355d88e700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:30.283003) v4 -- 0x55a332ffba00 con 0 > -3545> 2018-07-11 07:34:30.653403 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.115:0/13323 conn(0x55a321665800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1862 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:34:30.991001) v4 > -3544> 2018-07-11 07:34:30.653428 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.6 10.8.128.115:0/13323 1862 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:30.991001) v4 ==== 2004+0+0 (449816003 0 0) 0x55a324474c00 con 0x55a321665800 > -3543> 2018-07-11 07:34:30.653438 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:30.991001) v4 -- 0x55a310fe7600 con 0 > -3542> 2018-07-11 07:34:30.653477 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.115:0/13323 conn(0x55a321668800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1862 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:34:30.991001) v4 > -3541> 2018-07-11 07:34:30.653490 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.6 10.8.128.115:0/13323 1862 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:30.991001) v4 ==== 2004+0+0 (449816003 0 0) 0x55a324474c00 con 0x55a321668800 > -3540> 2018-07-11 07:34:30.653500 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:30.991001) v4 -- 0x55a332365400 con 0 > -3539> 2018-07-11 07:34:30.950451 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a3258b2b40 con 0 > -3538> 2018-07-11 07:34:30.969260 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5148 0x55a321005900 replica scrub(pg: 6.15,from:0'0,to:461'106919,epoch:494/492,start:6:aa03d2b5:::benchmark_data_magna108_35617_object813411:0,end:6:aa04beeb:::benchmark_data_magna108.ceph.redhat.com_2478_object250938:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -3537> 2018-07-11 07:34:30.969283 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5148 ==== replica scrub(pg: 6.15,from:0'0,to:461'106919,epoch:494/492,start:6:aa03d2b5:::benchmark_data_magna108_35617_object813411:0,end:6:aa04beeb:::benchmark_data_magna108.ceph.redhat.com_2478_object250938:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (1683201614 0 0) 0x55a321005900 con 0x55a317019000 > -3536> 2018-07-11 07:34:31.175008 7f3543899700 5 osd.10 494 heartbeat: osd_stat(784 GB used, 146 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,1]) > -3535> 2018-07-11 07:34:31.175023 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6802/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:34:31.175021) v4 -- 0x55a33015c200 con 0 > -3534> 2018-07-11 07:34:31.175062 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6803/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:34:31.175021) v4 -- 0x55a310fe6600 con 0 > -3533> 2018-07-11 07:34:31.175079 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6806/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:34:31.175021) v4 -- 0x55a32eb43000 con 0 > -3532> 2018-07-11 07:34:31.175094 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6807/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:34:31.175021) v4 -- 0x55a310fe6a00 con 0 > -3531> 2018-07-11 07:34:31.175106 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6808/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:34:31.175021) v4 -- 0x55a332348200 con 0 > -3530> 2018-07-11 07:34:31.175116 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6810/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:34:31.175021) v4 -- 0x55a332ff3000 con 0 > -3529> 2018-07-11 07:34:31.175128 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6809/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:34:31.175021) v4 -- 0x55a32442ba00 con 0 > -3528> 2018-07-11 07:34:31.175141 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6811/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:34:31.175021) v4 -- 0x55a3170de400 con 0 > -3527> 2018-07-11 07:34:31.175158 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6810/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:34:31.175021) v4 -- 0x55a332348400 con 0 > -3526> 2018-07-11 07:34:31.175177 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6811/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:34:31.175021) v4 -- 0x55a31707c800 con 0 > -3525> 2018-07-11 07:34:31.175191 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6806/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:34:31.175021) v4 -- 0x55a310fe6800 con 0 > -3524> 2018-07-11 07:34:31.175209 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6807/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:34:31.175021) v4 -- 0x55a310fe6c00 con 0 > -3523> 2018-07-11 07:34:31.175226 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6802/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:34:31.175021) v4 -- 0x55a32442a200 con 0 > -3522> 2018-07-11 07:34:31.175239 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6803/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:34:31.175021) v4 -- 0x55a32e456000 con 0 > -3521> 2018-07-11 07:34:31.175264 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6806/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:34:31.175021) v4 -- 0x55a310fe7000 con 0 > -3520> 2018-07-11 07:34:31.175281 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6807/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:34:31.175021) v4 -- 0x55a32162ac00 con 0 > -3519> 2018-07-11 07:34:31.175299 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6802/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:34:31.175021) v4 -- 0x55a310fe7400 con 0 > -3518> 2018-07-11 07:34:31.175312 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6803/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:34:31.175021) v4 -- 0x55a310fe7800 con 0 > -3517> 2018-07-11 07:34:31.175325 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6806/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:34:31.175021) v4 -- 0x55a310fe7c00 con 0 > -3516> 2018-07-11 07:34:31.175338 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6807/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:34:31.175021) v4 -- 0x55a310fe7e00 con 0 > -3515> 2018-07-11 07:34:31.175351 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6802/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:34:31.175021) v4 -- 0x55a32162a400 con 0 > -3514> 2018-07-11 07:34:31.175363 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6803/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:34:31.175021) v4 -- 0x55a310fe7a00 con 0 > -3513> 2018-07-11 07:34:31.175539 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6803/15255 conn(0x55a314e1d800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1859 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:31.175021) v4 > -3512> 2018-07-11 07:34:31.175555 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6807/12617 conn(0x55a314e27800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1859 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:31.175021) v4 > -3511> 2018-07-11 07:34:31.175555 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6808/12651 conn(0x55a314e96000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1859 0x55a32447b000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:31.175021) v4 > -3510> 2018-07-11 07:34:31.175576 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6803/15255 1859 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:31.175021) v4 ==== 2004+0+0 (4072780281 0 0) 0x55a324474c00 con 0x55a314e1d800 > -3509> 2018-07-11 07:34:31.175577 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6808/12651 1859 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:31.175021) v4 ==== 2004+0+0 (4072780281 0 0) 0x55a32447b000 con 0x55a314e96000 > -3508> 2018-07-11 07:34:31.175577 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6807/12617 1859 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:31.175021) v4 ==== 2004+0+0 (4072780281 0 0) 0x55a34de6f000 con 0x55a314e27800 > -3507> 2018-07-11 07:34:31.175611 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6806/12617 conn(0x55a314e26000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1859 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:31.175021) v4 > -3506> 2018-07-11 07:34:31.175623 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6806/12617 1859 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:31.175021) v4 ==== 2004+0+0 (4072780281 0 0) 0x55a324474c00 con 0x55a314e26000 > -3505> 2018-07-11 07:34:31.175641 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6806/12677 conn(0x55a314e84000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1859 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:31.175021) v4 > -3504> 2018-07-11 07:34:31.175662 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6806/12677 1859 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:31.175021) v4 ==== 2004+0+0 (4072780281 0 0) 0x55a34de6f000 con 0x55a314e84000 > -3503> 2018-07-11 07:34:31.175663 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6802/15255 conn(0x55a314e1c000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1859 0x55a32447b000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:31.175021) v4 > -3502> 2018-07-11 07:34:31.175678 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6802/15255 1859 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:31.175021) v4 ==== 2004+0+0 (4072780281 0 0) 0x55a32447b000 con 0x55a314e1c000 > -3501> 2018-07-11 07:34:31.175684 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6802/13323 conn(0x55a314e15000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1859 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:31.175021) v4 > -3500> 2018-07-11 07:34:31.175699 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6802/13323 1859 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:31.175021) v4 ==== 2004+0+0 (4072780281 0 0) 0x55a324474c00 con 0x55a314e15000 > -3499> 2018-07-11 07:34:31.175710 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6809/12650 conn(0x55a314e1f000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1859 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:31.175021) v4 > -3498> 2018-07-11 07:34:31.175729 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6810/12651 conn(0x55a314e97800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1859 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:31.175021) v4 > -3497> 2018-07-11 07:34:31.175730 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6809/12650 1859 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:31.175021) v4 ==== 2004+0+0 (4072780281 0 0) 0x55a34de6f000 con 0x55a314e1f000 > -3496> 2018-07-11 07:34:31.175741 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6810/12651 1859 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:31.175021) v4 ==== 2004+0+0 (4072780281 0 0) 0x55a324474c00 con 0x55a314e97800 > -3495> 2018-07-11 07:34:31.175754 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6811/12650 conn(0x55a314e20800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1859 0x55a32447b000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:31.175021) v4 > -3494> 2018-07-11 07:34:31.175774 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6811/12650 1859 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:31.175021) v4 ==== 2004+0+0 (4072780281 0 0) 0x55a32447b000 con 0x55a314e20800 > -3493> 2018-07-11 07:34:31.175780 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6803/13323 conn(0x55a314e16800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1859 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:31.175021) v4 > -3492> 2018-07-11 07:34:31.175776 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6811/15287 conn(0x55a314e04800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1859 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:31.175021) v4 > -3491> 2018-07-11 07:34:31.175797 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6803/13323 1859 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:31.175021) v4 ==== 2004+0+0 (4072780281 0 0) 0x55a34de6f000 con 0x55a314e16800 > -3490> 2018-07-11 07:34:31.175802 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6811/15287 1859 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:31.175021) v4 ==== 2004+0+0 (4072780281 0 0) 0x55a324474c00 con 0x55a314e04800 > -3489> 2018-07-11 07:34:31.175855 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6806/15885 conn(0x55a314e00000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1859 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:31.175021) v4 > -3488> 2018-07-11 07:34:31.175856 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6810/15287 conn(0x55a314e03000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1859 0x55a32447b000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:31.175021) v4 > -3487> 2018-07-11 07:34:31.175858 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6806/15263 conn(0x55a314e99000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1859 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:31.175021) v4 > -3486> 2018-07-11 07:34:31.175868 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6806/15885 1859 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:31.175021) v4 ==== 2004+0+0 (4072780281 0 0) 0x55a324474c00 con 0x55a314e00000 > -3485> 2018-07-11 07:34:31.175879 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6810/15287 1859 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:31.175021) v4 ==== 2004+0+0 (4072780281 0 0) 0x55a32447b000 con 0x55a314e03000 > -3484> 2018-07-11 07:34:31.175882 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6806/15263 1859 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:31.175021) v4 ==== 2004+0+0 (4072780281 0 0) 0x55a34de6f000 con 0x55a314e99000 > -3483> 2018-07-11 07:34:31.175909 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6802/15919 conn(0x55a314e87000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1859 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:31.175021) v4 > -3482> 2018-07-11 07:34:31.175922 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6802/15919 1859 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:31.175021) v4 ==== 2004+0+0 (4072780281 0 0) 0x55a324474c00 con 0x55a314e87000 > -3481> 2018-07-11 07:34:31.175939 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6807/12677 conn(0x55a314e85800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1859 0x55a32447b000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:31.175021) v4 > -3480> 2018-07-11 07:34:31.175939 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6803/15919 conn(0x55a314e88800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1859 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:31.175021) v4 > -3479> 2018-07-11 07:34:31.175960 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6807/12677 1859 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:31.175021) v4 ==== 2004+0+0 (4072780281 0 0) 0x55a32447b000 con 0x55a314e85800 > -3478> 2018-07-11 07:34:31.175965 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6803/15919 1859 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:31.175021) v4 ==== 2004+0+0 (4072780281 0 0) 0x55a34de6f000 con 0x55a314e88800 > -3477> 2018-07-11 07:34:31.176008 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6807/15263 conn(0x55a314e9a800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1859 0x55a32447b000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:31.175021) v4 > -3476> 2018-07-11 07:34:31.176008 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6807/15885 conn(0x55a314e01800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1859 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:31.175021) v4 > -3475> 2018-07-11 07:34:31.176029 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6807/15263 1859 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:31.175021) v4 ==== 2004+0+0 (4072780281 0 0) 0x55a32447b000 con 0x55a314e9a800 > -3474> 2018-07-11 07:34:31.176034 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6807/15885 1859 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:31.175021) v4 ==== 2004+0+0 (4072780281 0 0) 0x55a34de6f000 con 0x55a314e01800 > -3473> 2018-07-11 07:34:31.176078 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6803/11967 conn(0x55a314e13800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1859 0x55a32447b000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:31.175021) v4 > -3472> 2018-07-11 07:34:31.176077 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6802/11967 conn(0x55a314e12000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1859 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:31.175021) v4 > -3471> 2018-07-11 07:34:31.176098 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6803/11967 1859 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:31.175021) v4 ==== 2004+0+0 (4072780281 0 0) 0x55a32447b000 con 0x55a314e13800 > -3470> 2018-07-11 07:34:31.176103 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6802/11967 1859 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:31.175021) v4 ==== 2004+0+0 (4072780281 0 0) 0x55a34de6f000 con 0x55a314e12000 > -3469> 2018-07-11 07:34:31.659064 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15919 conn(0x55a317074800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1853 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:34:32.283016) v4 > -3468> 2018-07-11 07:34:31.659087 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.11 10.8.128.114:0/15919 1853 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:32.283016) v4 ==== 2004+0+0 (363551253 0 0) 0x55a324474c00 con 0x55a317074800 > -3467> 2018-07-11 07:34:31.659104 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:32.283016) v4 -- 0x55a332ff3000 con 0 > -3466> 2018-07-11 07:34:31.659197 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15919 conn(0x55a3170b4000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1853 0x55a32447b000 osd_ping(ping e494 stamp 2018-07-11 07:34:32.283016) v4 > -3465> 2018-07-11 07:34:31.659215 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.11 10.8.128.114:0/15919 1853 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:32.283016) v4 ==== 2004+0+0 (363551253 0 0) 0x55a32447b000 con 0x55a3170b4000 > -3464> 2018-07-11 07:34:31.659226 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:32.283016) v4 -- 0x55a332348200 con 0 > -3463> 2018-07-11 07:34:31.754752 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b7000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1801 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:34:31.505505) v4 > -3462> 2018-07-11 07:34:31.754773 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.8 10.8.128.108:0/11967 1801 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:31.505505) v4 ==== 2004+0+0 (233974233 0 0) 0x55a34de6f000 con 0x55a3170b7000 > -3461> 2018-07-11 07:34:31.754768 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b8800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1801 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:34:31.505505) v4 > -3460> 2018-07-11 07:34:31.754792 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.8 10.8.128.108:0/11967 1801 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:31.505505) v4 ==== 2004+0+0 (233974233 0 0) 0x55a332e79a00 con 0x55a3170b8800 > -3459> 2018-07-11 07:34:31.754789 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:31.505505) v4 -- 0x55a310fe6800 con 0 > -3458> 2018-07-11 07:34:31.754841 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:31.505505) v4 -- 0x55a310fe6600 con 0 > -3457> 2018-07-11 07:34:31.766858 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12651 conn(0x55a3216a0800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1849 0x55a32e456e00 osd_ping(ping e494 stamp 2018-07-11 07:34:31.802971) v4 > -3456> 2018-07-11 07:34:31.766879 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.2 10.8.128.106:0/12651 1849 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:31.802971) v4 ==== 2004+0+0 (207930279 0 0) 0x55a32e456e00 con 0x55a3216a0800 > -3455> 2018-07-11 07:34:31.766900 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:31.802971) v4 -- 0x55a310fe6a00 con 0 > -3454> 2018-07-11 07:34:31.766941 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12651 conn(0x55a32169f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1849 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:34:31.802971) v4 > -3453> 2018-07-11 07:34:31.766965 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.2 10.8.128.106:0/12651 1849 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:31.802971) v4 ==== 2004+0+0 (207930279 0 0) 0x55a324474c00 con 0x55a32169f000 > -3452> 2018-07-11 07:34:31.766980 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:31.802971) v4 -- 0x55a332365400 con 0 > -3451> 2018-07-11 07:34:31.832123 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12677 conn(0x55a32169d800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1872 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:34:31.868210) v4 > -3450> 2018-07-11 07:34:31.832144 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.1 10.8.128.106:0/12677 1872 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:31.868210) v4 ==== 2004+0+0 (2199559666 0 0) 0x55a34de78600 con 0x55a32169d800 > -3449> 2018-07-11 07:34:31.832154 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:31.868210) v4 -- 0x55a32442ba00 con 0 > -3448> 2018-07-11 07:34:31.832201 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12677 conn(0x55a32169c000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1872 0x55a32447b000 osd_ping(ping e494 stamp 2018-07-11 07:34:31.868210) v4 > -3447> 2018-07-11 07:34:31.832223 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.1 10.8.128.106:0/12677 1872 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:31.868210) v4 ==== 2004+0+0 (2199559666 0 0) 0x55a32447b000 con 0x55a32169c000 > -3446> 2018-07-11 07:34:31.832235 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:31.868210) v4 -- 0x55a33015c200 con 0 > -3445> 2018-07-11 07:34:31.862022 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a31c2fafc0 con 0 > -3444> 2018-07-11 07:34:31.901125 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5149 0x55a321005b80 replica scrub(pg: 6.15,from:0'0,to:430'106228,epoch:494/492,start:6:aa04beeb:::benchmark_data_magna108.ceph.redhat.com_2478_object250938:0,end:6:aa05997f:::benchmark_data_magna108_35617_object120607:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -3443> 2018-07-11 07:34:31.901156 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5149 ==== replica scrub(pg: 6.15,from:0'0,to:430'106228,epoch:494/492,start:6:aa04beeb:::benchmark_data_magna108.ceph.redhat.com_2478_object250938:0,end:6:aa05997f:::benchmark_data_magna108_35617_object120607:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (2466039687 0 0) 0x55a321005b80 con 0x55a317019000 > -3442> 2018-07-11 07:34:31.979897 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12650 conn(0x55a311b68800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1865 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:34:32.015957) v4 > -3441> 2018-07-11 07:34:31.979920 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.3 10.8.128.106:0/12650 1865 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:32.015957) v4 ==== 2004+0+0 (3935544597 0 0) 0x55a34de6f000 con 0x55a311b68800 > -3440> 2018-07-11 07:34:31.979930 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:32.015957) v4 -- 0x55a32442ba00 con 0 > -3439> 2018-07-11 07:34:31.979985 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12650 conn(0x55a311b67000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1865 0x55a32e456e00 osd_ping(ping e494 stamp 2018-07-11 07:34:32.015957) v4 > -3438> 2018-07-11 07:34:31.980004 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.3 10.8.128.106:0/12650 1865 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:32.015957) v4 ==== 2004+0+0 (3935544597 0 0) 0x55a32e456e00 con 0x55a311b67000 > -3437> 2018-07-11 07:34:31.980015 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:32.015957) v4 -- 0x55a310fe6e00 con 0 > -3436> 2018-07-11 07:34:32.255693 7f35488a3700 1 -- 10.8.128.107:6800/12665 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55a32ebb8000 con 0 > -3435> 2018-07-11 07:34:32.255758 7f35488a3700 1 -- 10.8.128.107:6800/12665 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55a32a5c5b80 con 0 > -3434> 2018-07-11 07:34:32.571732 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15885 conn(0x55a311b76800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1875 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:34:33.195719) v4 > -3433> 2018-07-11 07:34:32.571735 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15885 conn(0x55a3170b5800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1875 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:34:33.195719) v4 > -3432> 2018-07-11 07:34:32.571757 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.7 10.8.128.114:0/15885 1875 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:33.195719) v4 ==== 2004+0+0 (637985523 0 0) 0x55a34de78600 con 0x55a311b76800 > -3431> 2018-07-11 07:34:32.571761 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.7 10.8.128.114:0/15885 1875 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:33.195719) v4 ==== 2004+0+0 (637985523 0 0) 0x55a332e79a00 con 0x55a3170b5800 > -3430> 2018-07-11 07:34:32.571768 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:33.195719) v4 -- 0x55a32442ba00 con 0 > -3429> 2018-07-11 07:34:32.571793 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:33.195719) v4 -- 0x55a332365400 con 0 > -3428> 2018-07-11 07:34:32.575959 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a3258d8240 con 0 > -3427> 2018-07-11 07:34:32.592709 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5150 0x55a311214000 replica scrub(pg: 6.15,from:0'0,to:430'106027,epoch:494/492,start:6:aa05997f:::benchmark_data_magna108_35617_object120607:0,end:6:aa0666bf:::benchmark_data_magna108.ceph.redhat.com_2478_object257487:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -3426> 2018-07-11 07:34:32.592738 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5150 ==== replica scrub(pg: 6.15,from:0'0,to:430'106027,epoch:494/492,start:6:aa05997f:::benchmark_data_magna108_35617_object120607:0,end:6:aa0666bf:::benchmark_data_magna108.ceph.redhat.com_2478_object257487:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (864385637 0 0) 0x55a311214000 con 0x55a317019000 > -3425> 2018-07-11 07:34:32.884686 7f354489b700 5 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:307] [JOB 0] Delete db/036565.sst type=2 #36565 -- OK > > -3424> 2018-07-11 07:34:32.884718 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294472884700, "job": 0, "event": "table_file_deletion", "file_number": 36565} > -3423> 2018-07-11 07:34:32.884733 7f354489b700 5 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:307] [JOB 0] Delete db.slow/036559.sst type=2 #36559 -- OK > > -3422> 2018-07-11 07:34:32.884738 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294472884736, "job": 0, "event": "table_file_deletion", "file_number": 36559} > -3421> 2018-07-11 07:34:32.884746 7f354489b700 5 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:307] [JOB 0] Delete db.slow/036558.sst type=2 #36558 -- OK > > -3420> 2018-07-11 07:34:32.884751 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294472884749, "job": 0, "event": "table_file_deletion", "file_number": 36558} > -3419> 2018-07-11 07:34:32.884762 7f354489b700 5 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:307] [JOB 0] Delete db.slow/036557.sst type=2 #36557 -- OK > > -3418> 2018-07-11 07:34:32.884767 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294472884765, "job": 0, "event": "table_file_deletion", "file_number": 36557} > -3417> 2018-07-11 07:34:32.884777 7f354489b700 5 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:307] [JOB 0] Delete db.slow/036554.sst type=2 #36554 -- OK > > -3416> 2018-07-11 07:34:32.884786 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294472884784, "job": 0, "event": "table_file_deletion", "file_number": 36554} > -3415> 2018-07-11 07:34:32.884796 7f354489b700 5 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:307] [JOB 0] Delete db.slow/036553.sst type=2 #36553 -- OK > > -3414> 2018-07-11 07:34:32.884802 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294472884800, "job": 0, "event": "table_file_deletion", "file_number": 36553} > -3413> 2018-07-11 07:34:32.884810 7f354489b700 5 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:307] [JOB 0] Delete db.slow/036551.sst type=2 #36551 -- OK > > -3412> 2018-07-11 07:34:32.884815 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294472884813, "job": 0, "event": "table_file_deletion", "file_number": 36551} > -3411> 2018-07-11 07:34:32.884825 7f354489b700 5 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:307] [JOB 0] Delete db.slow/036523.sst type=2 #36523 -- OK > > -3410> 2018-07-11 07:34:32.884830 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294472884828, "job": 0, "event": "table_file_deletion", "file_number": 36523} > -3409> 2018-07-11 07:34:32.953959 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.115:0/13323 conn(0x55a321665800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1863 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:34:33.291643) v4 > -3408> 2018-07-11 07:34:32.953981 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.6 10.8.128.115:0/13323 1863 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:33.291643) v4 ==== 2004+0+0 (2498094330 0 0) 0x55a324474c00 con 0x55a321665800 > -3407> 2018-07-11 07:34:32.953997 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:33.291643) v4 -- 0x55a332365400 con 0 > -3406> 2018-07-11 07:34:32.954081 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.115:0/13323 conn(0x55a321668800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1863 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:34:33.291643) v4 > -3405> 2018-07-11 07:34:32.954101 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.6 10.8.128.115:0/13323 1863 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:33.291643) v4 ==== 2004+0+0 (2498094330 0 0) 0x55a332e79a00 con 0x55a321668800 > -3404> 2018-07-11 07:34:32.954111 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:33.291643) v4 -- 0x55a332365400 con 0 > -3403> 2018-07-11 07:34:33.539639 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a31364e240 con 0 > -3402> 2018-07-11 07:34:33.560460 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5151 0x55a311214280 replica scrub(pg: 6.15,from:0'0,to:430'106693,epoch:494/492,start:6:aa0666bf:::benchmark_data_magna108.ceph.redhat.com_2478_object257487:0,end:6:aa0756be:::benchmark_data_magna108_35617_object876483:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -3401> 2018-07-11 07:34:33.560492 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5151 ==== replica scrub(pg: 6.15,from:0'0,to:430'106693,epoch:494/492,start:6:aa0666bf:::benchmark_data_magna108.ceph.redhat.com_2478_object257487:0,end:6:aa0756be:::benchmark_data_magna108_35617_object876483:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (968991822 0 0) 0x55a311214280 con 0x55a317019000 > -3400> 2018-07-11 07:34:33.833037 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15287 conn(0x55a311b75000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1849 0x55a33015c200 osd_ping(ping e494 stamp 2018-07-11 07:34:35.047357) v4 > -3399> 2018-07-11 07:34:33.833063 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.4 10.8.128.113:0/15287 1849 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:35.047357) v4 ==== 2004+0+0 (2810244550 0 0) 0x55a33015c200 con 0x55a311b75000 > -3398> 2018-07-11 07:34:33.833063 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15287 conn(0x55a311b72000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1849 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:34:35.047357) v4 > -3397> 2018-07-11 07:34:33.833073 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:35.047357) v4 -- 0x55a310fe6e00 con 0 > -3396> 2018-07-11 07:34:33.833088 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.4 10.8.128.113:0/15287 1849 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:35.047357) v4 ==== 2004+0+0 (2810244550 0 0) 0x55a324474c00 con 0x55a311b72000 > -3395> 2018-07-11 07:34:33.833107 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:35.047357) v4 -- 0x55a332365400 con 0 > -3394> 2018-07-11 07:34:34.132709 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12677 conn(0x55a32169d800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1873 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:34:34.168739) v4 > -3393> 2018-07-11 07:34:34.132714 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12677 conn(0x55a32169c000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1873 0x55a32e456e00 osd_ping(ping e494 stamp 2018-07-11 07:34:34.168739) v4 > -3392> 2018-07-11 07:34:34.132737 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.1 10.8.128.106:0/12677 1873 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:34.168739) v4 ==== 2004+0+0 (3269236082 0 0) 0x55a34de6f000 con 0x55a32169d800 > -3391> 2018-07-11 07:34:34.132740 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.1 10.8.128.106:0/12677 1873 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:34.168739) v4 ==== 2004+0+0 (3269236082 0 0) 0x55a32e456e00 con 0x55a32169c000 > -3390> 2018-07-11 07:34:34.132747 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:34.168739) v4 -- 0x55a32442ba00 con 0 > -3389> 2018-07-11 07:34:34.132781 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:34.168739) v4 -- 0x55a310fe6e00 con 0 > -3388> 2018-07-11 07:34:34.280291 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12650 conn(0x55a311b67000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1866 0x55a33015c200 osd_ping(ping e494 stamp 2018-07-11 07:34:34.316357) v4 > -3387> 2018-07-11 07:34:34.280291 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12650 conn(0x55a311b68800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1866 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:34:34.316357) v4 > -3386> 2018-07-11 07:34:34.280322 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.3 10.8.128.106:0/12650 1866 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:34.316357) v4 ==== 2004+0+0 (2422223235 0 0) 0x55a34de78600 con 0x55a311b68800 > -3385> 2018-07-11 07:34:34.280319 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.3 10.8.128.106:0/12650 1866 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:34.316357) v4 ==== 2004+0+0 (2422223235 0 0) 0x55a33015c200 con 0x55a311b67000 > -3384> 2018-07-11 07:34:34.280340 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:34.316357) v4 -- 0x55a32442ba00 con 0 > -3383> 2018-07-11 07:34:34.280371 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:34.316357) v4 -- 0x55a310fe6e00 con 0 > -3382> 2018-07-11 07:34:34.312006 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a34a03db00 con 0 > -3381> 2018-07-11 07:34:34.326473 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5152 0x55a311214500 replica scrub(pg: 6.15,from:0'0,to:430'106827,epoch:494/492,start:6:aa0756be:::benchmark_data_magna108_35617_object876483:0,end:6:aa083369:::benchmark_data_magna108_35617_object818431:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -3380> 2018-07-11 07:34:34.326496 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5152 ==== replica scrub(pg: 6.15,from:0'0,to:430'106827,epoch:494/492,start:6:aa0756be:::benchmark_data_magna108_35617_object876483:0,end:6:aa083369:::benchmark_data_magna108_35617_object818431:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (2882739814 0 0) 0x55a311214500 con 0x55a317019000 > -3379> 2018-07-11 07:34:34.521213 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15263 conn(0x55a321664000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1847 0x55a32e456e00 osd_ping(ping e494 stamp 2018-07-11 07:34:35.735617) v4 > -3378> 2018-07-11 07:34:34.521240 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.5 10.8.128.113:0/15263 1847 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:35.735617) v4 ==== 2004+0+0 (1064258414 0 0) 0x55a32e456e00 con 0x55a321664000 > -3377> 2018-07-11 07:34:34.521259 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:35.735617) v4 -- 0x55a332eaca00 con 0 > -3376> 2018-07-11 07:34:34.521324 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15263 conn(0x55a321667000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1847 0x55a310fe6e00 osd_ping(ping e494 stamp 2018-07-11 07:34:35.735617) v4 > -3375> 2018-07-11 07:34:34.521339 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.5 10.8.128.113:0/15263 1847 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:35.735617) v4 ==== 2004+0+0 (1064258414 0 0) 0x55a310fe6e00 con 0x55a321667000 > -3374> 2018-07-11 07:34:34.521348 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:35.735617) v4 -- 0x55a33234f800 con 0 > -3373> 2018-07-11 07:34:34.655195 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b7000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1802 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:34:34.405974) v4 > -3372> 2018-07-11 07:34:34.655216 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.8 10.8.128.108:0/11967 1802 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:34.405974) v4 ==== 2004+0+0 (3351267266 0 0) 0x55a34de6f000 con 0x55a3170b7000 > -3371> 2018-07-11 07:34:34.655226 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:34.405974) v4 -- 0x55a32eb43000 con 0 > -3370> 2018-07-11 07:34:34.655253 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b8800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1802 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:34:34.405974) v4 > -3369> 2018-07-11 07:34:34.655275 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.8 10.8.128.108:0/11967 1802 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:34.405974) v4 ==== 2004+0+0 (3351267266 0 0) 0x55a332e79a00 con 0x55a3170b8800 > -3368> 2018-07-11 07:34:34.655286 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:34.405974) v4 -- 0x55a310fe7600 con 0 > -3367> 2018-07-11 07:34:34.675522 7f3543899700 5 osd.10 494 heartbeat: osd_stat(784 GB used, 146 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,0,1]) > -3366> 2018-07-11 07:34:34.675538 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6802/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:34:34.675535) v4 -- 0x55a32fb18400 con 0 > -3365> 2018-07-11 07:34:34.675560 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6803/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:34:34.675535) v4 -- 0x55a332fe9800 con 0 > -3364> 2018-07-11 07:34:34.675588 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6806/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:34:34.675535) v4 -- 0x55a332fea200 con 0 > -3363> 2018-07-11 07:34:34.675604 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6807/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:34:34.675535) v4 -- 0x55a332fe8400 con 0 > -3362> 2018-07-11 07:34:34.675617 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6808/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:34:34.675535) v4 -- 0x55a332fe8600 con 0 > -3361> 2018-07-11 07:34:34.675628 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6810/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:34:34.675535) v4 -- 0x55a332354400 con 0 > -3360> 2018-07-11 07:34:34.675641 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6809/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:34:34.675535) v4 -- 0x55a332ffae00 con 0 > -3359> 2018-07-11 07:34:34.675656 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6811/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:34:34.675535) v4 -- 0x55a332fe9400 con 0 > -3358> 2018-07-11 07:34:34.675679 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6810/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:34:34.675535) v4 -- 0x55a33532d400 con 0 > -3357> 2018-07-11 07:34:34.675695 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6811/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:34:34.675535) v4 -- 0x55a332fe9c00 con 0 > -3356> 2018-07-11 07:34:34.675708 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6806/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:34:34.675535) v4 -- 0x55a32fb18a00 con 0 > -3355> 2018-07-11 07:34:34.675721 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6807/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:34:34.675535) v4 -- 0x55a332fe9600 con 0 > -3354> 2018-07-11 07:34:34.675734 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6802/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:34:34.675535) v4 -- 0x55a332fe9000 con 0 > -3353> 2018-07-11 07:34:34.675747 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6803/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:34:34.675535) v4 -- 0x55a33532d000 con 0 > -3352> 2018-07-11 07:34:34.675763 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6806/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:34:34.675535) v4 -- 0x55a332fe8a00 con 0 > -3351> 2018-07-11 07:34:34.675781 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6807/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:34:34.675535) v4 -- 0x55a33532c400 con 0 > -3350> 2018-07-11 07:34:34.675794 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6802/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:34:34.675535) v4 -- 0x55a332fe8200 con 0 > -3349> 2018-07-11 07:34:34.675806 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6803/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:34:34.675535) v4 -- 0x55a33234b200 con 0 > -3348> 2018-07-11 07:34:34.675823 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6806/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:34:34.675535) v4 -- 0x55a34de5aa00 con 0 > -3347> 2018-07-11 07:34:34.675836 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6807/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:34:34.675535) v4 -- 0x55a32447a000 con 0 > -3346> 2018-07-11 07:34:34.675850 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6802/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:34:34.675535) v4 -- 0x55a32ee76a00 con 0 > -3345> 2018-07-11 07:34:34.675864 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6803/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:34:34.675535) v4 -- 0x55a332ff3c00 con 0 > -3344> 2018-07-11 07:34:34.676007 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6807/12617 conn(0x55a314e27800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1860 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:34.675535) v4 > -3343> 2018-07-11 07:34:34.676013 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6802/15255 conn(0x55a314e1c000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1860 0x55a32e456e00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:34.675535) v4 > -3342> 2018-07-11 07:34:34.676028 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6807/12617 1860 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:34.675535) v4 ==== 2004+0+0 (2016270206 0 0) 0x55a34de78600 con 0x55a314e27800 > -3341> 2018-07-11 07:34:34.676036 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6803/15255 conn(0x55a314e1d800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1860 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:34.675535) v4 > -3340> 2018-07-11 07:34:34.676043 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6802/15255 1860 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:34.675535) v4 ==== 2004+0+0 (2016270206 0 0) 0x55a32e456e00 con 0x55a314e1c000 > -3339> 2018-07-11 07:34:34.676057 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6803/15255 1860 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:34.675535) v4 ==== 2004+0+0 (2016270206 0 0) 0x55a332e79a00 con 0x55a314e1d800 > -3338> 2018-07-11 07:34:34.676100 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6807/12677 conn(0x55a314e85800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1860 0x55a32e456e00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:34.675535) v4 > -3337> 2018-07-11 07:34:34.676122 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6807/12677 1860 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:34.675535) v4 ==== 2004+0+0 (2016270206 0 0) 0x55a32e456e00 con 0x55a314e85800 > -3336> 2018-07-11 07:34:34.676121 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6806/12617 conn(0x55a314e26000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1860 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:34.675535) v4 > -3335> 2018-07-11 07:34:34.676117 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6806/12677 conn(0x55a314e84000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1860 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:34.675535) v4 > -3334> 2018-07-11 07:34:34.676140 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6806/12617 1860 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:34.675535) v4 ==== 2004+0+0 (2016270206 0 0) 0x55a332e79a00 con 0x55a314e26000 > -3333> 2018-07-11 07:34:34.676149 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6806/12677 1860 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:34.675535) v4 ==== 2004+0+0 (2016270206 0 0) 0x55a34de78600 con 0x55a314e84000 > -3332> 2018-07-11 07:34:34.676174 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6808/12651 conn(0x55a314e96000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1860 0x55a32e456e00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:34.675535) v4 > -3331> 2018-07-11 07:34:34.676195 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6808/12651 1860 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:34.675535) v4 ==== 2004+0+0 (2016270206 0 0) 0x55a32e456e00 con 0x55a314e96000 > -3330> 2018-07-11 07:34:34.676200 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6806/15263 conn(0x55a314e99000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1860 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:34.675535) v4 > -3329> 2018-07-11 07:34:34.676212 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6806/15263 1860 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:34.675535) v4 ==== 2004+0+0 (2016270206 0 0) 0x55a34de78600 con 0x55a314e99000 > -3328> 2018-07-11 07:34:34.676226 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6811/15287 conn(0x55a314e04800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1860 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:34.675535) v4 > -3327> 2018-07-11 07:34:34.676254 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6810/15287 conn(0x55a314e03000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1860 0x55a32e456e00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:34.675535) v4 > -3326> 2018-07-11 07:34:34.676271 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6809/12650 conn(0x55a314e1f000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1860 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:34.675535) v4 > -3325> 2018-07-11 07:34:34.676271 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6810/15287 1860 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:34.675535) v4 ==== 2004+0+0 (2016270206 0 0) 0x55a32e456e00 con 0x55a314e03000 > -3324> 2018-07-11 07:34:34.676244 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6811/15287 1860 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:34.675535) v4 ==== 2004+0+0 (2016270206 0 0) 0x55a332e79a00 con 0x55a314e04800 > -3323> 2018-07-11 07:34:34.676287 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6809/12650 1860 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:34.675535) v4 ==== 2004+0+0 (2016270206 0 0) 0x55a34de78600 con 0x55a314e1f000 > -3322> 2018-07-11 07:34:34.676331 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6802/11967 conn(0x55a314e12000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1860 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:34.675535) v4 > -3321> 2018-07-11 07:34:34.676345 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6802/11967 1860 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:34.675535) v4 ==== 2004+0+0 (2016270206 0 0) 0x55a34de78600 con 0x55a314e12000 > -3320> 2018-07-11 07:34:34.676343 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6811/12650 conn(0x55a314e20800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1860 0x55a32e456e00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:34.675535) v4 > -3319> 2018-07-11 07:34:34.676343 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6810/12651 conn(0x55a314e97800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1860 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:34.675535) v4 > -3318> 2018-07-11 07:34:34.676365 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6811/12650 1860 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:34.675535) v4 ==== 2004+0+0 (2016270206 0 0) 0x55a32e456e00 con 0x55a314e20800 > -3317> 2018-07-11 07:34:34.676370 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6810/12651 1860 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:34.675535) v4 ==== 2004+0+0 (2016270206 0 0) 0x55a332e79a00 con 0x55a314e97800 > -3316> 2018-07-11 07:34:34.676377 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6803/13323 conn(0x55a314e16800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1860 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:34.675535) v4 > -3315> 2018-07-11 07:34:34.676389 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6803/13323 1860 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:34.675535) v4 ==== 2004+0+0 (2016270206 0 0) 0x55a34de78600 con 0x55a314e16800 > -3314> 2018-07-11 07:34:34.676414 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6807/15263 conn(0x55a314e9a800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1860 0x55a32e456e00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:34.675535) v4 > -3313> 2018-07-11 07:34:34.676434 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6807/15263 1860 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:34.675535) v4 ==== 2004+0+0 (2016270206 0 0) 0x55a32e456e00 con 0x55a314e9a800 > -3312> 2018-07-11 07:34:34.676414 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6806/15885 conn(0x55a314e00000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1860 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:34.675535) v4 > -3311> 2018-07-11 07:34:34.676450 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6806/15885 1860 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:34.675535) v4 ==== 2004+0+0 (2016270206 0 0) 0x55a332e79a00 con 0x55a314e00000 > -3310> 2018-07-11 07:34:34.676459 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6807/15885 conn(0x55a314e01800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1860 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:34.675535) v4 > -3309> 2018-07-11 07:34:34.676473 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6807/15885 1860 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:34.675535) v4 ==== 2004+0+0 (2016270206 0 0) 0x55a34de78600 con 0x55a314e01800 > -3308> 2018-07-11 07:34:34.676482 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6803/11967 conn(0x55a314e13800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1860 0x55a32e456e00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:34.675535) v4 > -3307> 2018-07-11 07:34:34.676495 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6802/15919 conn(0x55a314e87000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1860 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:34.675535) v4 > -3306> 2018-07-11 07:34:34.676501 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6803/11967 1860 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:34.675535) v4 ==== 2004+0+0 (2016270206 0 0) 0x55a32e456e00 con 0x55a314e13800 > -3305> 2018-07-11 07:34:34.676511 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6802/15919 1860 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:34.675535) v4 ==== 2004+0+0 (2016270206 0 0) 0x55a332e79a00 con 0x55a314e87000 > -3304> 2018-07-11 07:34:34.676531 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6803/15919 conn(0x55a314e88800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1860 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:34.675535) v4 > -3303> 2018-07-11 07:34:34.676545 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6803/15919 1860 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:34.675535) v4 ==== 2004+0+0 (2016270206 0 0) 0x55a34de78600 con 0x55a314e88800 > -3302> 2018-07-11 07:34:34.676549 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6802/13323 conn(0x55a314e15000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1860 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:34.675535) v4 > -3301> 2018-07-11 07:34:34.676561 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6802/13323 1860 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:34.675535) v4 ==== 2004+0+0 (2016270206 0 0) 0x55a332e79a00 con 0x55a314e15000 > -3300> 2018-07-11 07:34:34.731622 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15255 conn(0x55a311b65800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1876 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:34:35.945787) v4 > -3299> 2018-07-11 07:34:34.731646 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.0 10.8.128.113:0/15255 1876 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:35.945787) v4 ==== 2004+0+0 (4186136847 0 0) 0x55a332e79a00 con 0x55a311b65800 > -3298> 2018-07-11 07:34:34.731659 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:35.945787) v4 -- 0x55a332354400 con 0 > -3297> 2018-07-11 07:34:34.731670 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15255 conn(0x55a311b64000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1876 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:34:35.945787) v4 > -3296> 2018-07-11 07:34:34.731695 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.0 10.8.128.113:0/15255 1876 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:35.945787) v4 ==== 2004+0+0 (4186136847 0 0) 0x55a34de78600 con 0x55a311b64000 > -3295> 2018-07-11 07:34:34.731707 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:35.945787) v4 -- 0x55a332ffae00 con 0 > -3294> 2018-07-11 07:34:34.984045 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.107:0/12617 conn(0x55a32447e000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1826 0x55a32e456e00 osd_ping(ping e494 stamp 2018-07-11 07:34:34.983610) v4 > -3293> 2018-07-11 07:34:34.984067 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.9 10.8.128.107:0/12617 1826 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:34.983610) v4 ==== 2004+0+0 (1135871111 0 0) 0x55a32e456e00 con 0x55a32447e000 > -3292> 2018-07-11 07:34:34.984079 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:34.983610) v4 -- 0x55a332fe8400 con 0 > -3291> 2018-07-11 07:34:34.984085 7f355d88e700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.107:0/12617 conn(0x55a32447f800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1826 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:34:34.983610) v4 > -3290> 2018-07-11 07:34:34.984107 7f355d88e700 1 -- 10.8.128.107:6803/12665 <== osd.9 10.8.128.107:0/12617 1826 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:34.983610) v4 ==== 2004+0+0 (1135871111 0 0) 0x55a34de6f000 con 0x55a32447f800 > -3289> 2018-07-11 07:34:34.984118 7f355d88e700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:34.983610) v4 -- 0x55a332ffae00 con 0 > -3288> 2018-07-11 07:34:35.072980 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a3324d86c0 con 0 > -3287> 2018-07-11 07:34:35.094874 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5153 0x55a311214780 replica scrub(pg: 6.15,from:0'0,to:430'105409,epoch:494/492,start:6:aa083369:::benchmark_data_magna108_35617_object818431:0,end:6:aa0931e4:::benchmark_data_magna108_35617_object584111:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -3286> 2018-07-11 07:34:35.094905 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5153 ==== replica scrub(pg: 6.15,from:0'0,to:430'105409,epoch:494/492,start:6:aa083369:::benchmark_data_magna108_35617_object818431:0,end:6:aa0931e4:::benchmark_data_magna108_35617_object584111:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (1568842011 0 0) 0x55a311214780 con 0x55a317019000 > -3285> 2018-07-11 07:34:35.175998 7f3543899700 5 osd.10 494 heartbeat: osd_stat(784 GB used, 146 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,1]) > -3284> 2018-07-11 07:34:35.176012 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6802/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:34:35.176010) v4 -- 0x55a332ff2400 con 0 > -3283> 2018-07-11 07:34:35.176038 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6803/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:34:35.176010) v4 -- 0x55a32fb18800 con 0 > -3282> 2018-07-11 07:34:35.176054 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6806/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:34:35.176010) v4 -- 0x55a32ee76800 con 0 > -3281> 2018-07-11 07:34:35.176068 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6807/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:34:35.176010) v4 -- 0x55a32fb19a00 con 0 > -3280> 2018-07-11 07:34:35.176081 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6808/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:34:35.176010) v4 -- 0x55a332fea600 con 0 > -3279> 2018-07-11 07:34:35.176093 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6810/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:34:35.176010) v4 -- 0x55a332feb600 con 0 > -3278> 2018-07-11 07:34:35.176109 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6809/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:34:35.176010) v4 -- 0x55a332feba00 con 0 > -3277> 2018-07-11 07:34:35.176134 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6811/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:34:35.176010) v4 -- 0x55a332feac00 con 0 > -3276> 2018-07-11 07:34:35.176149 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6810/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:34:35.176010) v4 -- 0x55a32fb18000 con 0 > -3275> 2018-07-11 07:34:35.176168 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6811/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:34:35.176010) v4 -- 0x55a332feb400 con 0 > -3274> 2018-07-11 07:34:35.176182 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6806/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:34:35.176010) v4 -- 0x55a32ee77400 con 0 > -3273> 2018-07-11 07:34:35.176194 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6807/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:34:35.176010) v4 -- 0x55a332fe9a00 con 0 > -3272> 2018-07-11 07:34:35.176207 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6802/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:34:35.176010) v4 -- 0x55a332fea000 con 0 > -3271> 2018-07-11 07:34:35.176220 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6803/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:34:35.176010) v4 -- 0x55a32ee76400 con 0 > -3270> 2018-07-11 07:34:35.176233 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6806/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:34:35.176010) v4 -- 0x55a32ee77e00 con 0 > -3269> 2018-07-11 07:34:35.176260 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6807/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:34:35.176010) v4 -- 0x55a330934e00 con 0 > -3268> 2018-07-11 07:34:35.176283 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6802/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:34:35.176010) v4 -- 0x55a332e77400 con 0 > -3267> 2018-07-11 07:34:35.176300 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6803/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:34:35.176010) v4 -- 0x55a32fb18600 con 0 > -3266> 2018-07-11 07:34:35.176313 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6806/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:34:35.176010) v4 -- 0x55a332e76800 con 0 > -3265> 2018-07-11 07:34:35.176327 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6807/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:34:35.176010) v4 -- 0x55a32eb20200 con 0 > -3264> 2018-07-11 07:34:35.176340 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6802/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:34:35.176010) v4 -- 0x55a32ee77200 con 0 > -3263> 2018-07-11 07:34:35.176352 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6803/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:34:35.176010) v4 -- 0x55a332e77000 con 0 > -3262> 2018-07-11 07:34:35.176456 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6807/12677 conn(0x55a314e85800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1861 0x55a32e456e00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:35.176010) v4 > -3261> 2018-07-11 07:34:35.176477 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6807/12677 1861 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:35.176010) v4 ==== 2004+0+0 (461805130 0 0) 0x55a32e456e00 con 0x55a314e85800 > -3260> 2018-07-11 07:34:35.176516 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6806/12677 conn(0x55a314e84000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1861 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:35.176010) v4 > -3259> 2018-07-11 07:34:35.176523 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6808/12651 conn(0x55a314e96000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1861 0x55a32e456e00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:35.176010) v4 > -3258> 2018-07-11 07:34:35.176515 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6810/12651 conn(0x55a314e97800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1861 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:35.176010) v4 > -3257> 2018-07-11 07:34:35.176536 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6806/12677 1861 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:35.176010) v4 ==== 2004+0+0 (461805130 0 0) 0x55a34de78600 con 0x55a314e84000 > -3256> 2018-07-11 07:34:35.176537 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6808/12651 1861 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:35.176010) v4 ==== 2004+0+0 (461805130 0 0) 0x55a32e456e00 con 0x55a314e96000 > -3255> 2018-07-11 07:34:35.176540 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6810/12651 1861 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:35.176010) v4 ==== 2004+0+0 (461805130 0 0) 0x55a332e79a00 con 0x55a314e97800 > -3254> 2018-07-11 07:34:35.176571 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6802/15255 conn(0x55a314e1c000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1861 0x55a32e456e00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:35.176010) v4 > -3253> 2018-07-11 07:34:35.176564 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6807/12617 conn(0x55a314e27800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1861 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:35.176010) v4 > -3252> 2018-07-11 07:34:35.176583 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6802/15255 1861 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:35.176010) v4 ==== 2004+0+0 (461805130 0 0) 0x55a32e456e00 con 0x55a314e1c000 > -3251> 2018-07-11 07:34:35.176591 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6807/12617 1861 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:35.176010) v4 ==== 2004+0+0 (461805130 0 0) 0x55a34de78600 con 0x55a314e27800 > -3250> 2018-07-11 07:34:35.176602 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6803/15255 conn(0x55a314e1d800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1861 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:35.176010) v4 > -3249> 2018-07-11 07:34:35.176614 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6811/12650 conn(0x55a314e20800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1861 0x55a32e456e00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:35.176010) v4 > -3248> 2018-07-11 07:34:35.176623 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6811/12650 1861 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:35.176010) v4 ==== 2004+0+0 (461805130 0 0) 0x55a32e456e00 con 0x55a314e20800 > -3247> 2018-07-11 07:34:35.176626 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6809/12650 conn(0x55a314e1f000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1861 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:35.176010) v4 > -3246> 2018-07-11 07:34:35.176622 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6803/15255 1861 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:35.176010) v4 ==== 2004+0+0 (461805130 0 0) 0x55a332e79a00 con 0x55a314e1d800 > -3245> 2018-07-11 07:34:35.176641 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6809/12650 1861 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:35.176010) v4 ==== 2004+0+0 (461805130 0 0) 0x55a34de78600 con 0x55a314e1f000 > -3244> 2018-07-11 07:34:35.176659 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6810/15287 conn(0x55a314e03000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1861 0x55a32e456e00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:35.176010) v4 > -3243> 2018-07-11 07:34:35.176672 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6810/15287 1861 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:35.176010) v4 ==== 2004+0+0 (461805130 0 0) 0x55a32e456e00 con 0x55a314e03000 > -3242> 2018-07-11 07:34:35.176670 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6806/12617 conn(0x55a314e26000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1861 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:35.176010) v4 > -3241> 2018-07-11 07:34:35.176674 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6806/15263 conn(0x55a314e99000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1861 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:35.176010) v4 > -3240> 2018-07-11 07:34:35.176681 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6806/12617 1861 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:35.176010) v4 ==== 2004+0+0 (461805130 0 0) 0x55a332e79a00 con 0x55a314e26000 > -3239> 2018-07-11 07:34:35.176687 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6806/15263 1861 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:35.176010) v4 ==== 2004+0+0 (461805130 0 0) 0x55a34de78600 con 0x55a314e99000 > -3238> 2018-07-11 07:34:35.176700 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6807/15263 conn(0x55a314e9a800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1861 0x55a32e456e00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:35.176010) v4 > -3237> 2018-07-11 07:34:35.176716 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6807/15263 1861 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:35.176010) v4 ==== 2004+0+0 (461805130 0 0) 0x55a32e456e00 con 0x55a314e9a800 > -3236> 2018-07-11 07:34:35.176712 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6811/15287 conn(0x55a314e04800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1861 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:35.176010) v4 > -3235> 2018-07-11 07:34:35.176730 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6803/13323 conn(0x55a314e16800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1861 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:35.176010) v4 > -3234> 2018-07-11 07:34:35.176733 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6811/15287 1861 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:35.176010) v4 ==== 2004+0+0 (461805130 0 0) 0x55a332e79a00 con 0x55a314e04800 > -3233> 2018-07-11 07:34:35.176743 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6803/13323 1861 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:35.176010) v4 ==== 2004+0+0 (461805130 0 0) 0x55a34de78600 con 0x55a314e16800 > -3232> 2018-07-11 07:34:35.176776 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6802/11967 conn(0x55a314e12000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1861 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:35.176010) v4 > -3231> 2018-07-11 07:34:35.176777 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6806/15885 conn(0x55a314e00000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1861 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:35.176010) v4 > -3230> 2018-07-11 07:34:35.176789 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6802/11967 1861 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:35.176010) v4 ==== 2004+0+0 (461805130 0 0) 0x55a34de78600 con 0x55a314e12000 > -3229> 2018-07-11 07:34:35.176802 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6806/15885 1861 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:35.176010) v4 ==== 2004+0+0 (461805130 0 0) 0x55a332e79a00 con 0x55a314e00000 > -3228> 2018-07-11 07:34:35.176800 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6803/11967 conn(0x55a314e13800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1861 0x55a32e456e00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:35.176010) v4 > -3227> 2018-07-11 07:34:35.176820 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6803/11967 1861 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:35.176010) v4 ==== 2004+0+0 (461805130 0 0) 0x55a32e456e00 con 0x55a314e13800 > -3226> 2018-07-11 07:34:35.176832 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6803/15919 conn(0x55a314e88800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1861 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:35.176010) v4 > -3225> 2018-07-11 07:34:35.176845 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6803/15919 1861 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:35.176010) v4 ==== 2004+0+0 (461805130 0 0) 0x55a34de78600 con 0x55a314e88800 > -3224> 2018-07-11 07:34:35.176850 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6802/13323 conn(0x55a314e15000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1861 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:35.176010) v4 > -3223> 2018-07-11 07:34:35.176865 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6802/13323 1861 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:35.176010) v4 ==== 2004+0+0 (461805130 0 0) 0x55a332e79a00 con 0x55a314e15000 > -3222> 2018-07-11 07:34:35.176874 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6807/15885 conn(0x55a314e01800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1861 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:35.176010) v4 > -3221> 2018-07-11 07:34:35.176886 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6807/15885 1861 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:35.176010) v4 ==== 2004+0+0 (461805130 0 0) 0x55a34de78600 con 0x55a314e01800 > -3220> 2018-07-11 07:34:35.176900 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6802/15919 conn(0x55a314e87000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1861 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:35.176010) v4 > -3219> 2018-07-11 07:34:35.176916 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6802/15919 1861 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:35.176010) v4 ==== 2004+0+0 (461805130 0 0) 0x55a332e79a00 con 0x55a314e87000 > -3218> 2018-07-11 07:34:35.484676 7f355d88e700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.107:0/12617 conn(0x55a32447f800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1827 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:34:35.484166) v4 > -3217> 2018-07-11 07:34:35.484698 7f355d88e700 1 -- 10.8.128.107:6803/12665 <== osd.9 10.8.128.107:0/12617 1827 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:35.484166) v4 ==== 2004+0+0 (3610716949 0 0) 0x55a34de78600 con 0x55a32447f800 > -3216> 2018-07-11 07:34:35.484708 7f355d88e700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:35.484166) v4 -- 0x55a32ee76800 con 0 > -3215> 2018-07-11 07:34:35.484715 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.107:0/12617 conn(0x55a32447e000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1827 0x55a32e456e00 osd_ping(ping e494 stamp 2018-07-11 07:34:35.484166) v4 > -3214> 2018-07-11 07:34:35.484737 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.9 10.8.128.107:0/12617 1827 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:35.484166) v4 ==== 2004+0+0 (3610716949 0 0) 0x55a32e456e00 con 0x55a32447e000 > -3213> 2018-07-11 07:34:35.484748 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:35.484166) v4 -- 0x55a332ff2400 con 0 > -3212> 2018-07-11 07:34:36.072093 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15885 conn(0x55a3170b5800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1876 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:34:36.696119) v4 > -3211> 2018-07-11 07:34:36.072119 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.7 10.8.128.114:0/15885 1876 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:36.696119) v4 ==== 2004+0+0 (3121867994 0 0) 0x55a332e79a00 con 0x55a3170b5800 > -3210> 2018-07-11 07:34:36.072130 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:36.696119) v4 -- 0x55a332feb600 con 0 > -3209> 2018-07-11 07:34:36.072118 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15885 conn(0x55a311b76800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1876 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:34:36.696119) v4 > -3208> 2018-07-11 07:34:36.072142 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.7 10.8.128.114:0/15885 1876 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:36.696119) v4 ==== 2004+0+0 (3121867994 0 0) 0x55a34de6f000 con 0x55a311b76800 > -3207> 2018-07-11 07:34:36.072157 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:36.696119) v4 -- 0x55a32ee76800 con 0 > -3206> 2018-07-11 07:34:36.433208 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12677 conn(0x55a32169d800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1874 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:34:36.469244) v4 > -3205> 2018-07-11 07:34:36.433232 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.1 10.8.128.106:0/12677 1874 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:36.469244) v4 ==== 2004+0+0 (2205627563 0 0) 0x55a34de78600 con 0x55a32169d800 > -3204> 2018-07-11 07:34:36.433236 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12677 conn(0x55a32169c000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1874 0x55a310fe6e00 osd_ping(ping e494 stamp 2018-07-11 07:34:36.469244) v4 > -3203> 2018-07-11 07:34:36.433262 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:36.469244) v4 -- 0x55a32ee76800 con 0 > -3202> 2018-07-11 07:34:36.433280 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.1 10.8.128.106:0/12677 1874 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:36.469244) v4 ==== 2004+0+0 (2205627563 0 0) 0x55a310fe6e00 con 0x55a32169c000 > -3201> 2018-07-11 07:34:36.433309 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:36.469244) v4 -- 0x55a332ff2400 con 0 > -3200> 2018-07-11 07:34:36.706823 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a321aa8000 con 0 > -3199> 2018-07-11 07:34:36.718376 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5154 0x55a311214a00 replica scrub(pg: 6.15,from:0'0,to:430'105771,epoch:494/492,start:6:aa0931e4:::benchmark_data_magna108_35617_object584111:0,end:6:aa0a2ec8:::benchmark_data_magna108_35617_object207537:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -3198> 2018-07-11 07:34:36.718397 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5154 ==== replica scrub(pg: 6.15,from:0'0,to:430'105771,epoch:494/492,start:6:aa0931e4:::benchmark_data_magna108_35617_object584111:0,end:6:aa0a2ec8:::benchmark_data_magna108_35617_object207537:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (462184766 0 0) 0x55a311214a00 con 0x55a317019000 > -3197> 2018-07-11 07:34:36.744978 7f35498a5700 10 monclient: tick > -3196> 2018-07-11 07:34:36.744991 7f35498a5700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2018-07-11 07:34:06.744989) > -3195> 2018-07-11 07:34:36.827062 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 60] Generated table #36587: 4957166 keys, 80348282 bytes > -3194> 2018-07-11 07:34:36.827117 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294476827089, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 36587, "file_size": 80348282, "table_properties": {"data_size": 67109360, "index_size": 689228, "filter_size": 12548704, "raw_key_size": 143207024, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957166, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957166", "kMergeOperands": "0"}} > -3193> 2018-07-11 07:34:37.067289 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12651 conn(0x55a32169f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1850 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:34:37.103392) v4 > -3192> 2018-07-11 07:34:37.067312 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.2 10.8.128.106:0/12651 1850 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:37.103392) v4 ==== 2004+0+0 (1749201037 0 0) 0x55a324474c00 con 0x55a32169f000 > -3191> 2018-07-11 07:34:37.067324 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:37.103392) v4 -- 0x55a332feb600 con 0 > -3190> 2018-07-11 07:34:37.067330 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12651 conn(0x55a3216a0800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1850 0x55a310fe6e00 osd_ping(ping e494 stamp 2018-07-11 07:34:37.103392) v4 > -3189> 2018-07-11 07:34:37.067360 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.2 10.8.128.106:0/12651 1850 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:37.103392) v4 ==== 2004+0+0 (1749201037 0 0) 0x55a310fe6e00 con 0x55a3216a0800 > -3188> 2018-07-11 07:34:37.067378 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:37.103392) v4 -- 0x55a332ff2400 con 0 > -3187> 2018-07-11 07:34:37.255965 7f35488a3700 1 -- 10.8.128.107:6800/12665 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55a324472dc0 con 0 > -3186> 2018-07-11 07:34:37.256029 7f35488a3700 1 -- 10.8.128.107:6800/12665 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55a31708a000 con 0 > -3185> 2018-07-11 07:34:37.521515 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a352468480 con 0 > -3184> 2018-07-11 07:34:37.535309 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5155 0x55a311214c80 replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:494/492,start:6:aa0a2ec8:::benchmark_data_magna108_35617_object207537:0,end:6:aa0b063e:::benchmark_data_magna108_35617_object117796:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -3183> 2018-07-11 07:34:37.535340 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5155 ==== replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:494/492,start:6:aa0a2ec8:::benchmark_data_magna108_35617_object207537:0,end:6:aa0b063e:::benchmark_data_magna108_35617_object117796:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (3666620243 0 0) 0x55a311214c80 con 0x55a317019000 > -3182> 2018-07-11 07:34:37.559327 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15919 conn(0x55a3170b4000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1854 0x55a32e456e00 osd_ping(ping e494 stamp 2018-07-11 07:34:38.183402) v4 > -3181> 2018-07-11 07:34:37.559351 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.11 10.8.128.114:0/15919 1854 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:38.183402) v4 ==== 2004+0+0 (1679770602 0 0) 0x55a32e456e00 con 0x55a3170b4000 > -3180> 2018-07-11 07:34:37.559366 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:38.183402) v4 -- 0x55a32fb18400 con 0 > -3179> 2018-07-11 07:34:37.559372 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15919 conn(0x55a317074800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1854 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:34:38.183402) v4 > -3178> 2018-07-11 07:34:37.559397 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.11 10.8.128.114:0/15919 1854 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:38.183402) v4 ==== 2004+0+0 (1679770602 0 0) 0x55a332e79a00 con 0x55a317074800 > -3177> 2018-07-11 07:34:37.559410 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:38.183402) v4 -- 0x55a32fb18800 con 0 > -3176> 2018-07-11 07:34:37.654580 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.115:0/13323 conn(0x55a321665800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1864 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:34:37.992300) v4 > -3175> 2018-07-11 07:34:37.654603 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.6 10.8.128.115:0/13323 1864 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:37.992300) v4 ==== 2004+0+0 (1776543999 0 0) 0x55a324474c00 con 0x55a321665800 > -3174> 2018-07-11 07:34:37.654618 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:37.992300) v4 -- 0x55a332354400 con 0 > -3173> 2018-07-11 07:34:37.654665 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.115:0/13323 conn(0x55a321668800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1864 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:34:37.992300) v4 > -3172> 2018-07-11 07:34:37.654680 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.6 10.8.128.115:0/13323 1864 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:37.992300) v4 ==== 2004+0+0 (1776543999 0 0) 0x55a324474c00 con 0x55a321668800 > -3171> 2018-07-11 07:34:37.654690 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:37.992300) v4 -- 0x55a310fe7600 con 0 > -3170> 2018-07-11 07:34:37.780778 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12650 conn(0x55a311b67000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1867 0x55a332ff2400 osd_ping(ping e494 stamp 2018-07-11 07:34:37.816818) v4 > -3169> 2018-07-11 07:34:37.780816 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.3 10.8.128.106:0/12650 1867 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:37.816818) v4 ==== 2004+0+0 (26507095 0 0) 0x55a332ff2400 con 0x55a311b67000 > -3168> 2018-07-11 07:34:37.780836 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:37.816818) v4 -- 0x55a33234f800 con 0 > -3167> 2018-07-11 07:34:37.780831 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12650 conn(0x55a311b68800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1867 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:34:37.816818) v4 > -3166> 2018-07-11 07:34:37.780855 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.3 10.8.128.106:0/12650 1867 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:37.816818) v4 ==== 2004+0+0 (26507095 0 0) 0x55a34de6f000 con 0x55a311b68800 > -3165> 2018-07-11 07:34:37.780872 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:37.816818) v4 -- 0x55a332ffae00 con 0 > -3164> 2018-07-11 07:34:38.208400 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a329bfc6c0 con 0 > -3163> 2018-07-11 07:34:38.235704 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5156 0x55a311214f00 replica scrub(pg: 6.15,from:0'0,to:430'106353,epoch:494/492,start:6:aa0b063e:::benchmark_data_magna108_35617_object117796:0,end:6:aa0c1f60:::benchmark_data_magna108_35617_object25789:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -3162> 2018-07-11 07:34:38.235734 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5156 ==== replica scrub(pg: 6.15,from:0'0,to:430'106353,epoch:494/492,start:6:aa0b063e:::benchmark_data_magna108_35617_object117796:0,end:6:aa0c1f60:::benchmark_data_magna108_35617_object25789:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 217+0+0 (830688606 0 0) 0x55a311214f00 con 0x55a317019000 > -3161> 2018-07-11 07:34:38.676496 7f3543899700 5 osd.10 494 heartbeat: osd_stat(784 GB used, 146 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,0,1]) > -3160> 2018-07-11 07:34:38.676513 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6802/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:34:38.676510) v4 -- 0x55a32eb43800 con 0 > -3159> 2018-07-11 07:34:38.676540 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6803/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:34:38.676510) v4 -- 0x55a313704400 con 0 > -3158> 2018-07-11 07:34:38.676557 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6806/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:34:38.676510) v4 -- 0x55a33b42d800 con 0 > -3157> 2018-07-11 07:34:38.676573 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6807/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:34:38.676510) v4 -- 0x55a32eb42400 con 0 > -3156> 2018-07-11 07:34:38.676592 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6808/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:34:38.676510) v4 -- 0x55a332e76200 con 0 > -3155> 2018-07-11 07:34:38.676614 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6810/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:34:38.676510) v4 -- 0x55a33b42de00 con 0 > -3154> 2018-07-11 07:34:38.676641 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6809/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:34:38.676510) v4 -- 0x55a332e77e00 con 0 > -3153> 2018-07-11 07:34:38.676663 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6811/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:34:38.676510) v4 -- 0x55a32eb42000 con 0 > -3152> 2018-07-11 07:34:38.676693 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6810/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:34:38.676510) v4 -- 0x55a332e78a00 con 0 > -3151> 2018-07-11 07:34:38.676716 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6811/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:34:38.676510) v4 -- 0x55a32eb42c00 con 0 > -3150> 2018-07-11 07:34:38.676732 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6806/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:34:38.676510) v4 -- 0x55a32eb42600 con 0 > -3149> 2018-07-11 07:34:38.676749 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6807/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:34:38.676510) v4 -- 0x55a32eb43600 con 0 > -3148> 2018-07-11 07:34:38.676771 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6802/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:34:38.676510) v4 -- 0x55a33234f200 con 0 > -3147> 2018-07-11 07:34:38.676796 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6803/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:34:38.676510) v4 -- 0x55a3314cec00 con 0 > -3146> 2018-07-11 07:34:38.676813 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6806/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:34:38.676510) v4 -- 0x55a3314cf200 con 0 > -3145> 2018-07-11 07:34:38.676833 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6807/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:34:38.676510) v4 -- 0x55a32ee76600 con 0 > -3144> 2018-07-11 07:34:38.676861 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6802/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:34:38.676510) v4 -- 0x55a34de5ac00 con 0 > -3143> 2018-07-11 07:34:38.676884 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6803/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:34:38.676510) v4 -- 0x55a3314ce800 con 0 > -3142> 2018-07-11 07:34:38.676905 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6806/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:34:38.676510) v4 -- 0x55a330934a00 con 0 > -3141> 2018-07-11 07:34:38.676929 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6807/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:34:38.676510) v4 -- 0x55a32eb43400 con 0 > -3140> 2018-07-11 07:34:38.676923 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6802/15255 conn(0x55a314e1c000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1862 0x55a32e456e00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:38.676510) v4 > -3139> 2018-07-11 07:34:38.676952 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6802/15255 1862 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:38.676510) v4 ==== 2004+0+0 (3840759412 0 0) 0x55a32e456e00 con 0x55a314e1c000 > -3138> 2018-07-11 07:34:38.676957 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6802/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:34:38.676510) v4 -- 0x55a314e7a800 con 0 > -3137> 2018-07-11 07:34:38.676977 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6803/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:34:38.676510) v4 -- 0x55a332e99a00 con 0 > -3136> 2018-07-11 07:34:38.677006 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6803/15255 conn(0x55a314e1d800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1862 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:38.676510) v4 > -3135> 2018-07-11 07:34:38.677026 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6803/15255 1862 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:38.676510) v4 ==== 2004+0+0 (3840759412 0 0) 0x55a332e79a00 con 0x55a314e1d800 > -3134> 2018-07-11 07:34:38.677113 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6811/15287 conn(0x55a314e04800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1862 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:38.676510) v4 > -3133> 2018-07-11 07:34:38.677123 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6807/12677 conn(0x55a314e85800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1862 0x55a332ff2400 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:38.676510) v4 > -3132> 2018-07-11 07:34:38.677134 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6811/15287 1862 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:38.676510) v4 ==== 2004+0+0 (3840759412 0 0) 0x55a324474c00 con 0x55a314e04800 > -3131> 2018-07-11 07:34:38.677147 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6807/12677 1862 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:38.676510) v4 ==== 2004+0+0 (3840759412 0 0) 0x55a332ff2400 con 0x55a314e85800 > -3130> 2018-07-11 07:34:38.677145 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6806/12677 conn(0x55a314e84000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1862 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:38.676510) v4 > -3129> 2018-07-11 07:34:38.677160 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6806/12677 1862 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:38.676510) v4 ==== 2004+0+0 (3840759412 0 0) 0x55a34de78600 con 0x55a314e84000 > -3128> 2018-07-11 07:34:38.677173 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6806/12617 conn(0x55a314e26000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1862 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:38.676510) v4 > -3127> 2018-07-11 07:34:38.677190 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6806/12617 1862 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:38.676510) v4 ==== 2004+0+0 (3840759412 0 0) 0x55a324474c00 con 0x55a314e26000 > -3126> 2018-07-11 07:34:38.677206 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6811/12650 conn(0x55a314e20800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1862 0x55a332ff2400 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:38.676510) v4 > -3125> 2018-07-11 07:34:38.677223 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6811/12650 1862 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:38.676510) v4 ==== 2004+0+0 (3840759412 0 0) 0x55a332ff2400 con 0x55a314e20800 > -3124> 2018-07-11 07:34:38.677223 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6809/12650 conn(0x55a314e1f000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1862 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:38.676510) v4 > -3123> 2018-07-11 07:34:38.677236 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6810/12651 conn(0x55a314e97800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1862 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:38.676510) v4 > -3122> 2018-07-11 07:34:38.677244 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6809/12650 1862 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:38.676510) v4 ==== 2004+0+0 (3840759412 0 0) 0x55a34de78600 con 0x55a314e1f000 > -3121> 2018-07-11 07:34:38.677257 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6810/12651 1862 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:38.676510) v4 ==== 2004+0+0 (3840759412 0 0) 0x55a324474c00 con 0x55a314e97800 > -3120> 2018-07-11 07:34:38.677290 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6808/12651 conn(0x55a314e96000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1862 0x55a332ff2400 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:38.676510) v4 > -3119> 2018-07-11 07:34:38.677306 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6808/12651 1862 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:38.676510) v4 ==== 2004+0+0 (3840759412 0 0) 0x55a332ff2400 con 0x55a314e96000 > -3118> 2018-07-11 07:34:38.677305 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6806/15885 conn(0x55a314e00000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1862 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:38.676510) v4 > -3117> 2018-07-11 07:34:38.677320 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6806/15263 conn(0x55a314e99000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1862 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:38.676510) v4 > -3116> 2018-07-11 07:34:38.677327 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6806/15885 1862 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:38.676510) v4 ==== 2004+0+0 (3840759412 0 0) 0x55a324474c00 con 0x55a314e00000 > -3115> 2018-07-11 07:34:38.677337 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6806/15263 1862 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:38.676510) v4 ==== 2004+0+0 (3840759412 0 0) 0x55a34de78600 con 0x55a314e99000 > -3114> 2018-07-11 07:34:38.677356 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6810/15287 conn(0x55a314e03000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1862 0x55a332ff2400 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:38.676510) v4 > -3113> 2018-07-11 07:34:38.677368 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6802/13323 conn(0x55a314e15000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1862 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:38.676510) v4 > -3112> 2018-07-11 07:34:38.677368 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6807/12617 conn(0x55a314e27800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1862 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:38.676510) v4 > -3111> 2018-07-11 07:34:38.677386 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6810/15287 1862 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:38.676510) v4 ==== 2004+0+0 (3840759412 0 0) 0x55a332ff2400 con 0x55a314e03000 > -3110> 2018-07-11 07:34:38.677394 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6807/12617 1862 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:38.676510) v4 ==== 2004+0+0 (3840759412 0 0) 0x55a34de78600 con 0x55a314e27800 > -3109> 2018-07-11 07:34:38.677389 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6802/13323 1862 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:38.676510) v4 ==== 2004+0+0 (3840759412 0 0) 0x55a324474c00 con 0x55a314e15000 > -3108> 2018-07-11 07:34:38.677441 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6807/15263 conn(0x55a314e9a800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1862 0x55a332ff2400 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:38.676510) v4 > -3107> 2018-07-11 07:34:38.677439 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6807/15885 conn(0x55a314e01800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1862 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:38.676510) v4 > -3106> 2018-07-11 07:34:38.677439 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6802/15919 conn(0x55a314e87000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1862 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:38.676510) v4 > -3105> 2018-07-11 07:34:38.677457 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6807/15885 1862 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:38.676510) v4 ==== 2004+0+0 (3840759412 0 0) 0x55a34de78600 con 0x55a314e01800 > -3104> 2018-07-11 07:34:38.677460 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6807/15263 1862 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:38.676510) v4 ==== 2004+0+0 (3840759412 0 0) 0x55a332ff2400 con 0x55a314e9a800 > -3103> 2018-07-11 07:34:38.677466 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6802/15919 1862 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:38.676510) v4 ==== 2004+0+0 (3840759412 0 0) 0x55a324474c00 con 0x55a314e87000 > -3102> 2018-07-11 07:34:38.677501 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6803/13323 conn(0x55a314e16800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1862 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:38.676510) v4 > -3101> 2018-07-11 07:34:38.677503 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6803/11967 conn(0x55a314e13800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1862 0x55a332ff2400 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:38.676510) v4 > -3100> 2018-07-11 07:34:38.677516 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6803/13323 1862 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:38.676510) v4 ==== 2004+0+0 (3840759412 0 0) 0x55a34de78600 con 0x55a314e16800 > -3099> 2018-07-11 07:34:38.677517 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6803/11967 1862 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:38.676510) v4 ==== 2004+0+0 (3840759412 0 0) 0x55a332ff2400 con 0x55a314e13800 > -3098> 2018-07-11 07:34:38.677551 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6802/11967 conn(0x55a314e12000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1862 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:38.676510) v4 > -3097> 2018-07-11 07:34:38.677563 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6802/11967 1862 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:38.676510) v4 ==== 2004+0+0 (3840759412 0 0) 0x55a34de78600 con 0x55a314e12000 > -3096> 2018-07-11 07:34:38.677594 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6803/15919 conn(0x55a314e88800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1862 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:38.676510) v4 > -3095> 2018-07-11 07:34:38.677606 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6803/15919 1862 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:38.676510) v4 ==== 2004+0+0 (3840759412 0 0) 0x55a34de78600 con 0x55a314e88800 > -3094> 2018-07-11 07:34:38.755673 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b7000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1803 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:34:38.506467) v4 > -3093> 2018-07-11 07:34:38.755694 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.8 10.8.128.108:0/11967 1803 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:38.506467) v4 ==== 2004+0+0 (1870113590 0 0) 0x55a34de78600 con 0x55a3170b7000 > -3092> 2018-07-11 07:34:38.755690 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b8800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1803 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:34:38.506467) v4 > -3091> 2018-07-11 07:34:38.755705 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:38.506467) v4 -- 0x55a332ffae00 con 0 > -3090> 2018-07-11 07:34:38.755707 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.8 10.8.128.108:0/11967 1803 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:38.506467) v4 ==== 2004+0+0 (1870113590 0 0) 0x55a324474c00 con 0x55a3170b8800 > -3089> 2018-07-11 07:34:38.755730 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:38.506467) v4 -- 0x55a310fe7600 con 0 > -3088> 2018-07-11 07:34:38.767786 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12651 conn(0x55a3216a0800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1851 0x55a332ff2400 osd_ping(ping e494 stamp 2018-07-11 07:34:38.803863) v4 > -3087> 2018-07-11 07:34:38.767811 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.2 10.8.128.106:0/12651 1851 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:38.803863) v4 ==== 2004+0+0 (1426296000 0 0) 0x55a332ff2400 con 0x55a3216a0800 > -3086> 2018-07-11 07:34:38.767820 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:38.803863) v4 -- 0x55a33234f800 con 0 > -3085> 2018-07-11 07:34:38.767873 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12651 conn(0x55a32169f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1851 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:34:38.803863) v4 > -3084> 2018-07-11 07:34:38.767893 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.2 10.8.128.106:0/12651 1851 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:38.803863) v4 ==== 2004+0+0 (1426296000 0 0) 0x55a332e79a00 con 0x55a32169f000 > -3083> 2018-07-11 07:34:38.767904 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:38.803863) v4 -- 0x55a332354400 con 0 > -3082> 2018-07-11 07:34:38.881286 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12650 conn(0x55a311b68800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1868 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:34:38.917400) v4 > -3081> 2018-07-11 07:34:38.881308 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.3 10.8.128.106:0/12650 1868 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:38.917400) v4 ==== 2004+0+0 (885596246 0 0) 0x55a34de6f000 con 0x55a311b68800 > -3080> 2018-07-11 07:34:38.881317 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:38.917400) v4 -- 0x55a332ffae00 con 0 > -3079> 2018-07-11 07:34:38.881306 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12650 conn(0x55a311b67000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1868 0x55a32e456e00 osd_ping(ping e494 stamp 2018-07-11 07:34:38.917400) v4 > -3078> 2018-07-11 07:34:38.881328 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.3 10.8.128.106:0/12650 1868 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:38.917400) v4 ==== 2004+0+0 (885596246 0 0) 0x55a32e456e00 con 0x55a311b67000 > -3077> 2018-07-11 07:34:38.881345 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:38.917400) v4 -- 0x55a33234f800 con 0 > -3076> 2018-07-11 07:34:38.927009 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a321744000 con 0 > -3075> 2018-07-11 07:34:38.953355 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5157 0x55a311215180 replica scrub(pg: 6.15,from:0'0,to:461'106920,epoch:494/492,start:6:aa0c1f60:::benchmark_data_magna108_35617_object25789:0,end:6:aa0d8b5a:::benchmark_data_magna108_41525_object57831:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -3074> 2018-07-11 07:34:38.953379 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5157 ==== replica scrub(pg: 6.15,from:0'0,to:461'106920,epoch:494/492,start:6:aa0c1f60:::benchmark_data_magna108_35617_object25789:0,end:6:aa0d8b5a:::benchmark_data_magna108_41525_object57831:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 216+0+0 (621234071 0 0) 0x55a311215180 con 0x55a317019000 > -3073> 2018-07-11 07:34:39.133421 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15287 conn(0x55a311b75000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1850 0x55a332ff2400 osd_ping(ping e494 stamp 2018-07-11 07:34:40.347774) v4 > -3072> 2018-07-11 07:34:39.133449 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.4 10.8.128.113:0/15287 1850 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:40.347774) v4 ==== 2004+0+0 (1434103171 0 0) 0x55a332ff2400 con 0x55a311b75000 > -3071> 2018-07-11 07:34:39.133459 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:40.347774) v4 -- 0x55a33234f800 con 0 > -3070> 2018-07-11 07:34:39.133453 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15287 conn(0x55a311b72000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1850 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:34:40.347774) v4 > -3069> 2018-07-11 07:34:39.133475 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.4 10.8.128.113:0/15287 1850 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:40.347774) v4 ==== 2004+0+0 (1434103171 0 0) 0x55a324474c00 con 0x55a311b72000 > -3068> 2018-07-11 07:34:39.133493 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:40.347774) v4 -- 0x55a31ce7a000 con 0 > -3067> 2018-07-11 07:34:39.221663 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15263 conn(0x55a321667000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1848 0x55a32e456e00 osd_ping(ping e494 stamp 2018-07-11 07:34:40.436033) v4 > -3066> 2018-07-11 07:34:39.221684 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.5 10.8.128.113:0/15263 1848 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:40.436033) v4 ==== 2004+0+0 (2399803249 0 0) 0x55a32e456e00 con 0x55a321667000 > -3065> 2018-07-11 07:34:39.221698 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:40.436033) v4 -- 0x55a3170de000 con 0 > -3064> 2018-07-11 07:34:39.221750 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15263 conn(0x55a321664000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1848 0x55a32e456e00 osd_ping(ping e494 stamp 2018-07-11 07:34:40.436033) v4 > -3063> 2018-07-11 07:34:39.221765 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.5 10.8.128.113:0/15263 1848 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:40.436033) v4 ==== 2004+0+0 (2399803249 0 0) 0x55a32e456e00 con 0x55a321664000 > -3062> 2018-07-11 07:34:39.221775 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:40.436033) v4 -- 0x55a32447b800 con 0 > -3061> 2018-07-11 07:34:39.432098 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15255 conn(0x55a311b65800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1877 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:34:40.646409) v4 > -3060> 2018-07-11 07:34:39.432119 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.0 10.8.128.113:0/15255 1877 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:40.646409) v4 ==== 2004+0+0 (190394131 0 0) 0x55a332e79a00 con 0x55a311b65800 > -3059> 2018-07-11 07:34:39.432129 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:40.646409) v4 -- 0x55a332ff3400 con 0 > -3058> 2018-07-11 07:34:39.432148 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15255 conn(0x55a311b64000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1877 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:34:40.646409) v4 > -3057> 2018-07-11 07:34:39.432169 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.0 10.8.128.113:0/15255 1877 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:40.646409) v4 ==== 2004+0+0 (190394131 0 0) 0x55a34de78600 con 0x55a311b64000 > -3056> 2018-07-11 07:34:39.432184 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:40.646409) v4 -- 0x55a332fea200 con 0 > -3055> 2018-07-11 07:34:39.859838 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15919 conn(0x55a317074800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1855 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:34:40.483846) v4 > -3054> 2018-07-11 07:34:39.859863 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.11 10.8.128.114:0/15919 1855 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:40.483846) v4 ==== 2004+0+0 (3176357475 0 0) 0x55a324474c00 con 0x55a317074800 > -3053> 2018-07-11 07:34:39.859881 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:40.483846) v4 -- 0x55a34dec7a00 con 0 > -3052> 2018-07-11 07:34:39.859889 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15919 conn(0x55a3170b4000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1855 0x55a332ff2400 osd_ping(ping e494 stamp 2018-07-11 07:34:40.483846) v4 > -3051> 2018-07-11 07:34:39.859914 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.11 10.8.128.114:0/15919 1855 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:40.483846) v4 ==== 2004+0+0 (3176357475 0 0) 0x55a332ff2400 con 0x55a3170b4000 > -3050> 2018-07-11 07:34:39.859929 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:40.483846) v4 -- 0x55a32447b800 con 0 > -3049> 2018-07-11 07:34:39.885228 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a3323cab40 con 0 > -3048> 2018-07-11 07:34:39.919234 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5158 0x55a311215400 replica scrub(pg: 6.15,from:0'0,to:430'105739,epoch:494/492,start:6:aa0d8b5a:::benchmark_data_magna108_41525_object57831:0,end:6:aa0e7065:::benchmark_data_magna108_35617_object435061:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -3047> 2018-07-11 07:34:39.919277 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5158 ==== replica scrub(pg: 6.15,from:0'0,to:430'105739,epoch:494/492,start:6:aa0d8b5a:::benchmark_data_magna108_41525_object57831:0,end:6:aa0e7065:::benchmark_data_magna108_35617_object435061:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 217+0+0 (2654621317 0 0) 0x55a311215400 con 0x55a317019000 > -3046> 2018-07-11 07:34:40.322068 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15263 conn(0x55a321664000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1849 0x55a32e456e00 osd_ping(ping e494 stamp 2018-07-11 07:34:41.536485) v4 > -3045> 2018-07-11 07:34:40.322097 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.5 10.8.128.113:0/15263 1849 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:41.536485) v4 ==== 2004+0+0 (4074667836 0 0) 0x55a32e456e00 con 0x55a321664000 > -3044> 2018-07-11 07:34:40.322110 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:41.536485) v4 -- 0x55a32447b800 con 0 > -3043> 2018-07-11 07:34:40.322166 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15263 conn(0x55a321667000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1849 0x55a332ff2400 osd_ping(ping e494 stamp 2018-07-11 07:34:41.536485) v4 > -3042> 2018-07-11 07:34:40.322182 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.5 10.8.128.113:0/15263 1849 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:41.536485) v4 ==== 2004+0+0 (4074667836 0 0) 0x55a332ff2400 con 0x55a321667000 > -3041> 2018-07-11 07:34:40.322192 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:41.536485) v4 -- 0x55a32447b800 con 0 > -3040> 2018-07-11 07:34:40.456127 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b8800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1804 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:34:40.206908) v4 > -3039> 2018-07-11 07:34:40.456148 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.8 10.8.128.108:0/11967 1804 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:40.206908) v4 ==== 2004+0+0 (3630960437 0 0) 0x55a332e79a00 con 0x55a3170b8800 > -3038> 2018-07-11 07:34:40.456158 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:40.206908) v4 -- 0x55a32e331a00 con 0 > -3037> 2018-07-11 07:34:40.456155 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b7000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1804 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:34:40.206908) v4 > -3036> 2018-07-11 07:34:40.456181 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.8 10.8.128.108:0/11967 1804 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:40.206908) v4 ==== 2004+0+0 (3630960437 0 0) 0x55a34de6f000 con 0x55a3170b7000 > -3035> 2018-07-11 07:34:40.456196 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:40.206908) v4 -- 0x55a32eb43000 con 0 > -3034> 2018-07-11 07:34:40.532589 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15255 conn(0x55a311b64000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1878 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:34:41.746990) v4 > -3033> 2018-07-11 07:34:40.532591 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15255 conn(0x55a311b65800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1878 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:34:41.746990) v4 > -3032> 2018-07-11 07:34:40.532610 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.0 10.8.128.113:0/15255 1878 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:41.746990) v4 ==== 2004+0+0 (2711141484 0 0) 0x55a34de78600 con 0x55a311b64000 > -3031> 2018-07-11 07:34:40.532612 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.0 10.8.128.113:0/15255 1878 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:41.746990) v4 ==== 2004+0+0 (2711141484 0 0) 0x55a324474c00 con 0x55a311b65800 > -3030> 2018-07-11 07:34:40.532620 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:41.746990) v4 -- 0x55a332ffba00 con 0 > -3029> 2018-07-11 07:34:40.532644 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:41.746990) v4 -- 0x55a32e331800 con 0 > -3028> 2018-07-11 07:34:40.533571 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12677 conn(0x55a32169d800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1875 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:34:40.569761) v4 > -3027> 2018-07-11 07:34:40.533588 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.1 10.8.128.106:0/12677 1875 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:40.569761) v4 ==== 2004+0+0 (2066076930 0 0) 0x55a34de6f000 con 0x55a32169d800 > -3026> 2018-07-11 07:34:40.533596 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:40.569761) v4 -- 0x55a32e457400 con 0 > -3025> 2018-07-11 07:34:40.533646 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12677 conn(0x55a32169c000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1875 0x55a32e456e00 osd_ping(ping e494 stamp 2018-07-11 07:34:40.569761) v4 > -3024> 2018-07-11 07:34:40.533669 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.1 10.8.128.106:0/12677 1875 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:40.569761) v4 ==== 2004+0+0 (2066076930 0 0) 0x55a32e456e00 con 0x55a32169c000 > -3023> 2018-07-11 07:34:40.533681 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:40.569761) v4 -- 0x55a3170de000 con 0 > -3022> 2018-07-11 07:34:40.555008 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.115:0/13323 conn(0x55a321668800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1865 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:34:40.892736) v4 > -3021> 2018-07-11 07:34:40.555034 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.6 10.8.128.115:0/13323 1865 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:40.892736) v4 ==== 2004+0+0 (267145060 0 0) 0x55a332e79a00 con 0x55a321668800 > -3020> 2018-07-11 07:34:40.555049 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:40.892736) v4 -- 0x55a32e331200 con 0 > -3019> 2018-07-11 07:34:40.555077 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.115:0/13323 conn(0x55a321665800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1865 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:34:40.892736) v4 > -3018> 2018-07-11 07:34:40.555089 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.6 10.8.128.115:0/13323 1865 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:40.892736) v4 ==== 2004+0+0 (267145060 0 0) 0x55a332e79a00 con 0x55a321665800 > -3017> 2018-07-11 07:34:40.555098 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:40.892736) v4 -- 0x55a33be6e800 con 0 > -3016> 2018-07-11 07:34:40.563515 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a3318c1b00 con 0 > -3015> 2018-07-11 07:34:40.577502 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5159 0x55a311215680 replica scrub(pg: 6.15,from:0'0,to:430'106578,epoch:494/492,start:6:aa0e7065:::benchmark_data_magna108_35617_object435061:0,end:6:aa0f0e85:::benchmark_data_magna108_35617_object855659:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -3014> 2018-07-11 07:34:40.577545 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5159 ==== replica scrub(pg: 6.15,from:0'0,to:430'106578,epoch:494/492,start:6:aa0e7065:::benchmark_data_magna108_35617_object435061:0,end:6:aa0f0e85:::benchmark_data_magna108_35617_object855659:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (3773199076 0 0) 0x55a311215680 con 0x55a317019000 > -3013> 2018-07-11 07:34:40.772557 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15885 conn(0x55a311b76800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1877 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:34:41.396562) v4 > -3012> 2018-07-11 07:34:40.772563 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15885 conn(0x55a3170b5800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1877 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:34:41.396562) v4 > -3011> 2018-07-11 07:34:40.772580 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.7 10.8.128.114:0/15885 1877 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:41.396562) v4 ==== 2004+0+0 (400834051 0 0) 0x55a324474c00 con 0x55a3170b5800 > -3010> 2018-07-11 07:34:40.772580 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.7 10.8.128.114:0/15885 1877 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:41.396562) v4 ==== 2004+0+0 (400834051 0 0) 0x55a34de78600 con 0x55a311b76800 > -3009> 2018-07-11 07:34:40.772590 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:41.396562) v4 -- 0x55a33be6e800 con 0 > -3008> 2018-07-11 07:34:40.772617 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:41.396562) v4 -- 0x55a32e457400 con 0 > -3007> 2018-07-11 07:34:41.385283 7f355d88e700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.107:0/12617 conn(0x55a32447f800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1828 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:34:41.384793) v4 > -3006> 2018-07-11 07:34:41.385306 7f355d88e700 1 -- 10.8.128.107:6803/12665 <== osd.9 10.8.128.107:0/12617 1828 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:41.384793) v4 ==== 2004+0+0 (508224038 0 0) 0x55a34de6f000 con 0x55a32447f800 > -3005> 2018-07-11 07:34:41.385316 7f355d88e700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:41.384793) v4 -- 0x55a32e457400 con 0 > -3004> 2018-07-11 07:34:41.385362 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.107:0/12617 conn(0x55a32447e000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1828 0x55a332ff2400 osd_ping(ping e494 stamp 2018-07-11 07:34:41.384793) v4 > -3003> 2018-07-11 07:34:41.385381 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.9 10.8.128.107:0/12617 1828 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:41.384793) v4 ==== 2004+0+0 (508224038 0 0) 0x55a332ff2400 con 0x55a32447e000 > -3002> 2018-07-11 07:34:41.385392 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:41.384793) v4 -- 0x55a32e457000 con 0 > -3001> 2018-07-11 07:34:41.572322 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a3306aafc0 con 0 > -3000> 2018-07-11 07:34:41.593914 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5160 0x55a311215900 replica scrub(pg: 6.15,from:0'0,to:430'106407,epoch:494/492,start:6:aa0f0e85:::benchmark_data_magna108_35617_object855659:0,end:6:aa102a76:::benchmark_data_magna108.ceph.redhat.com_2478_object433307:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -2999> 2018-07-11 07:34:41.593939 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5160 ==== replica scrub(pg: 6.15,from:0'0,to:430'106407,epoch:494/492,start:6:aa0f0e85:::benchmark_data_magna108_35617_object855659:0,end:6:aa102a76:::benchmark_data_magna108.ceph.redhat.com_2478_object433307:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (3438019150 0 0) 0x55a311215900 con 0x55a317019000 > -2998> 2018-07-11 07:34:41.634154 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12677 conn(0x55a32169d800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1876 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:34:41.670275) v4 > -2997> 2018-07-11 07:34:41.634164 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12677 conn(0x55a32169c000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1876 0x55a32e456e00 osd_ping(ping e494 stamp 2018-07-11 07:34:41.670275) v4 > -2996> 2018-07-11 07:34:41.634185 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.1 10.8.128.106:0/12677 1876 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:41.670275) v4 ==== 2004+0+0 (2753112405 0 0) 0x55a32e456e00 con 0x55a32169c000 > -2995> 2018-07-11 07:34:41.634182 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.1 10.8.128.106:0/12677 1876 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:41.670275) v4 ==== 2004+0+0 (2753112405 0 0) 0x55a34de78600 con 0x55a32169d800 > -2994> 2018-07-11 07:34:41.634197 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:41.670275) v4 -- 0x55a31ce7a200 con 0 > -2993> 2018-07-11 07:34:41.634224 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:41.670275) v4 -- 0x55a32e457400 con 0 > -2992> 2018-07-11 07:34:42.177132 7f3543899700 5 osd.10 494 heartbeat: osd_stat(784 GB used, 146 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,0,0,1]) > -2991> 2018-07-11 07:34:42.177149 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6802/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:34:42.177148) v4 -- 0x55a32fb19000 con 0 > -2990> 2018-07-11 07:34:42.177178 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6803/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:34:42.177148) v4 -- 0x55a332febc00 con 0 > -2989> 2018-07-11 07:34:42.177203 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6806/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:34:42.177148) v4 -- 0x55a332364e00 con 0 > -2988> 2018-07-11 07:34:42.177230 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6807/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:34:42.177148) v4 -- 0x55a33234f600 con 0 > -2987> 2018-07-11 07:34:42.177269 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6808/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:34:42.177148) v4 -- 0x55a34dea8a00 con 0 > -2986> 2018-07-11 07:34:42.177293 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6810/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:34:42.177148) v4 -- 0x55a332e98600 con 0 > -2985> 2018-07-11 07:34:42.177324 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6809/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:34:42.177148) v4 -- 0x55a332ffa600 con 0 > -2984> 2018-07-11 07:34:42.177353 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6811/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:34:42.177148) v4 -- 0x55a31706f800 con 0 > -2983> 2018-07-11 07:34:42.177371 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6810/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:34:42.177148) v4 -- 0x55a34de85c00 con 0 > -2982> 2018-07-11 07:34:42.177390 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6811/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:34:42.177148) v4 -- 0x55a330935800 con 0 > -2981> 2018-07-11 07:34:42.177418 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6806/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:34:42.177148) v4 -- 0x55a32fb18200 con 0 > -2980> 2018-07-11 07:34:42.177443 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6807/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:34:42.177148) v4 -- 0x55a34de79400 con 0 > -2979> 2018-07-11 07:34:42.177459 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6802/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:34:42.177148) v4 -- 0x55a34de79e00 con 0 > -2978> 2018-07-11 07:34:42.177479 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6803/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:34:42.177148) v4 -- 0x55a330935000 con 0 > -2977> 2018-07-11 07:34:42.177503 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6806/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:34:42.177148) v4 -- 0x55a34dea9600 con 0 > -2976> 2018-07-11 07:34:42.177518 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6807/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:34:42.177148) v4 -- 0x55a332354a00 con 0 > -2975> 2018-07-11 07:34:42.177537 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6802/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:34:42.177148) v4 -- 0x55a332e99600 con 0 > -2974> 2018-07-11 07:34:42.177558 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6803/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:34:42.177148) v4 -- 0x55a34dea8000 con 0 > -2973> 2018-07-11 07:34:42.177579 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6806/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:34:42.177148) v4 -- 0x55a34de85a00 con 0 > -2972> 2018-07-11 07:34:42.177601 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6807/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:34:42.177148) v4 -- 0x55a324474400 con 0 > -2971> 2018-07-11 07:34:42.177622 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6802/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:34:42.177148) v4 -- 0x55a324474e00 con 0 > -2970> 2018-07-11 07:34:42.177643 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6802/15255 conn(0x55a314e1c000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1863 0x55a32e456e00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:42.177148) v4 > -2969> 2018-07-11 07:34:42.177650 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6803/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:34:42.177148) v4 -- 0x55a32eb21400 con 0 > -2968> 2018-07-11 07:34:42.177666 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6802/15255 1863 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:42.177148) v4 ==== 2004+0+0 (3734051186 0 0) 0x55a32e456e00 con 0x55a314e1c000 > -2967> 2018-07-11 07:34:42.177706 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6803/15255 conn(0x55a314e1d800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1863 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:42.177148) v4 > -2966> 2018-07-11 07:34:42.177716 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6808/12651 conn(0x55a314e96000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1863 0x55a32e456e00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:42.177148) v4 > -2965> 2018-07-11 07:34:42.177727 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6808/12651 1863 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:42.177148) v4 ==== 2004+0+0 (3734051186 0 0) 0x55a32e456e00 con 0x55a314e96000 > -2964> 2018-07-11 07:34:42.177722 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6809/12650 conn(0x55a314e1f000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1863 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:42.177148) v4 > -2963> 2018-07-11 07:34:42.177727 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6803/15255 1863 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:42.177148) v4 ==== 2004+0+0 (3734051186 0 0) 0x55a332e79a00 con 0x55a314e1d800 > -2962> 2018-07-11 07:34:42.177749 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6809/12650 1863 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:42.177148) v4 ==== 2004+0+0 (3734051186 0 0) 0x55a34de78600 con 0x55a314e1f000 > -2961> 2018-07-11 07:34:42.177777 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6810/15287 conn(0x55a314e03000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1863 0x55a32e456e00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:42.177148) v4 > -2960> 2018-07-11 07:34:42.177790 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6810/15287 1863 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:42.177148) v4 ==== 2004+0+0 (3734051186 0 0) 0x55a32e456e00 con 0x55a314e03000 > -2959> 2018-07-11 07:34:42.177807 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6810/12651 conn(0x55a314e97800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1863 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:42.177148) v4 > -2958> 2018-07-11 07:34:42.177822 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6810/12651 1863 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:42.177148) v4 ==== 2004+0+0 (3734051186 0 0) 0x55a324474c00 con 0x55a314e97800 > -2957> 2018-07-11 07:34:42.177828 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6807/12677 conn(0x55a314e85800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1863 0x55a32e456e00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:42.177148) v4 > -2956> 2018-07-11 07:34:42.177854 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6811/15287 conn(0x55a314e04800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1863 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:42.177148) v4 > -2955> 2018-07-11 07:34:42.177822 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6806/12677 conn(0x55a314e84000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1863 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:42.177148) v4 > -2954> 2018-07-11 07:34:42.177864 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6811/15287 1863 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:42.177148) v4 ==== 2004+0+0 (3734051186 0 0) 0x55a324474c00 con 0x55a314e04800 > -2953> 2018-07-11 07:34:42.177864 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6807/12677 1863 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:42.177148) v4 ==== 2004+0+0 (3734051186 0 0) 0x55a32e456e00 con 0x55a314e85800 > -2952> 2018-07-11 07:34:42.177872 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6806/12677 1863 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:42.177148) v4 ==== 2004+0+0 (3734051186 0 0) 0x55a34de78600 con 0x55a314e84000 > -2951> 2018-07-11 07:34:42.177896 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6811/12650 conn(0x55a314e20800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1863 0x55a32e456e00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:42.177148) v4 > -2950> 2018-07-11 07:34:42.177912 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6811/12650 1863 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:42.177148) v4 ==== 2004+0+0 (3734051186 0 0) 0x55a32e456e00 con 0x55a314e20800 > -2949> 2018-07-11 07:34:42.177885 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6806/12617 conn(0x55a314e26000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1863 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:42.177148) v4 > -2948> 2018-07-11 07:34:42.177933 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6806/12617 1863 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:42.177148) v4 ==== 2004+0+0 (3734051186 0 0) 0x55a324474c00 con 0x55a314e26000 > -2947> 2018-07-11 07:34:42.177945 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6807/15263 conn(0x55a314e9a800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1863 0x55a32e456e00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:42.177148) v4 > -2946> 2018-07-11 07:34:42.177957 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6807/15263 1863 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:42.177148) v4 ==== 2004+0+0 (3734051186 0 0) 0x55a32e456e00 con 0x55a314e9a800 > -2945> 2018-07-11 07:34:42.177956 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6806/15263 conn(0x55a314e99000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1863 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:42.177148) v4 > -2944> 2018-07-11 07:34:42.177965 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6802/13323 conn(0x55a314e15000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1863 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:42.177148) v4 > -2943> 2018-07-11 07:34:42.177978 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6806/15263 1863 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:42.177148) v4 ==== 2004+0+0 (3734051186 0 0) 0x55a34de6f000 con 0x55a314e99000 > -2942> 2018-07-11 07:34:42.177977 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6802/13323 1863 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:42.177148) v4 ==== 2004+0+0 (3734051186 0 0) 0x55a324474c00 con 0x55a314e15000 > -2941> 2018-07-11 07:34:42.178009 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6807/12617 conn(0x55a314e27800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1863 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:42.177148) v4 > -2940> 2018-07-11 07:34:42.178016 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6803/11967 conn(0x55a314e13800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1863 0x55a32e456e00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:42.177148) v4 > -2939> 2018-07-11 07:34:42.178022 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6807/12617 1863 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:42.177148) v4 ==== 2004+0+0 (3734051186 0 0) 0x55a34de6f000 con 0x55a314e27800 > -2938> 2018-07-11 07:34:42.178024 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6806/15885 conn(0x55a314e00000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1863 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:42.177148) v4 > -2937> 2018-07-11 07:34:42.178031 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6803/11967 1863 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:42.177148) v4 ==== 2004+0+0 (3734051186 0 0) 0x55a32e456e00 con 0x55a314e13800 > -2936> 2018-07-11 07:34:42.178036 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6806/15885 1863 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:42.177148) v4 ==== 2004+0+0 (3734051186 0 0) 0x55a324474c00 con 0x55a314e00000 > -2935> 2018-07-11 07:34:42.178075 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6803/13323 conn(0x55a314e16800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1863 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:42.177148) v4 > -2934> 2018-07-11 07:34:42.178089 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6803/13323 1863 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:42.177148) v4 ==== 2004+0+0 (3734051186 0 0) 0x55a34de6f000 con 0x55a314e16800 > -2933> 2018-07-11 07:34:42.178127 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6807/15885 conn(0x55a314e01800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1863 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:42.177148) v4 > -2932> 2018-07-11 07:34:42.178140 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6807/15885 1863 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:42.177148) v4 ==== 2004+0+0 (3734051186 0 0) 0x55a34de6f000 con 0x55a314e01800 > -2931> 2018-07-11 07:34:42.178180 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6802/11967 conn(0x55a314e12000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1863 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:42.177148) v4 > -2930> 2018-07-11 07:34:42.178193 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6802/11967 1863 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:42.177148) v4 ==== 2004+0+0 (3734051186 0 0) 0x55a34de6f000 con 0x55a314e12000 > -2929> 2018-07-11 07:34:42.178208 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6802/15919 conn(0x55a314e87000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1863 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:42.177148) v4 > -2928> 2018-07-11 07:34:42.178223 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6802/15919 1863 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:42.177148) v4 ==== 2004+0+0 (3734051186 0 0) 0x55a324474c00 con 0x55a314e87000 > -2927> 2018-07-11 07:34:42.178237 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6803/15919 conn(0x55a314e88800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1863 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:42.177148) v4 > -2926> 2018-07-11 07:34:42.178260 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6803/15919 1863 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:42.177148) v4 ==== 2004+0+0 (3734051186 0 0) 0x55a34de6f000 con 0x55a314e88800 > -2925> 2018-07-11 07:34:42.256218 7f35488a3700 1 -- 10.8.128.107:6800/12665 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55a3244722c0 con 0 > -2924> 2018-07-11 07:34:42.256295 7f35488a3700 1 -- 10.8.128.107:6800/12665 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55a3309338c0 con 0 > -2923> 2018-07-11 07:34:42.268294 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12651 conn(0x55a3216a0800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1852 0x55a32e456e00 osd_ping(ping e494 stamp 2018-07-11 07:34:42.304432) v4 > -2922> 2018-07-11 07:34:42.268309 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12651 conn(0x55a32169f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1852 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:34:42.304432) v4 > -2921> 2018-07-11 07:34:42.268326 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.2 10.8.128.106:0/12651 1852 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:42.304432) v4 ==== 2004+0+0 (4086363970 0 0) 0x55a324474c00 con 0x55a32169f000 > -2920> 2018-07-11 07:34:42.268323 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.2 10.8.128.106:0/12651 1852 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:42.304432) v4 ==== 2004+0+0 (4086363970 0 0) 0x55a32e456e00 con 0x55a3216a0800 > -2919> 2018-07-11 07:34:42.268343 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:42.304432) v4 -- 0x55a332e98600 con 0 > -2918> 2018-07-11 07:34:42.268365 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:42.304432) v4 -- 0x55a33234f600 con 0 > -2917> 2018-07-11 07:34:42.303399 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a31c788480 con 0 > -2916> 2018-07-11 07:34:42.319511 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5161 0x55a311215b80 replica scrub(pg: 6.15,from:0'0,to:430'106712,epoch:494/492,start:6:aa102a76:::benchmark_data_magna108.ceph.redhat.com_2478_object433307:0,end:6:aa10eb80:::benchmark_data_magna108_35617_object899320:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -2915> 2018-07-11 07:34:42.319541 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5161 ==== replica scrub(pg: 6.15,from:0'0,to:430'106712,epoch:494/492,start:6:aa102a76:::benchmark_data_magna108.ceph.redhat.com_2478_object433307:0,end:6:aa10eb80:::benchmark_data_magna108_35617_object899320:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (2752074830 0 0) 0x55a311215b80 con 0x55a317019000 > -2914> 2018-07-11 07:34:43.233784 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15287 conn(0x55a311b75000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1851 0x55a32e457000 osd_ping(ping e494 stamp 2018-07-11 07:34:44.448233) v4 > -2913> 2018-07-11 07:34:43.233810 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.4 10.8.128.113:0/15287 1851 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:44.448233) v4 ==== 2004+0+0 (1842855980 0 0) 0x55a32e457000 con 0x55a311b75000 > -2912> 2018-07-11 07:34:43.233826 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:44.448233) v4 -- 0x55a32fb19000 con 0 > -2911> 2018-07-11 07:34:43.233823 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15287 conn(0x55a311b72000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1851 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:34:44.448233) v4 > -2910> 2018-07-11 07:34:43.233849 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.4 10.8.128.113:0/15287 1851 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:44.448233) v4 ==== 2004+0+0 (1842855980 0 0) 0x55a332e79a00 con 0x55a311b72000 > -2909> 2018-07-11 07:34:43.233868 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:44.448233) v4 -- 0x55a332e98600 con 0 > -2908> 2018-07-11 07:34:43.368818 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12651 conn(0x55a3216a0800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1853 0x55a33234f600 osd_ping(ping e494 stamp 2018-07-11 07:34:43.404982) v4 > -2907> 2018-07-11 07:34:43.368840 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12651 conn(0x55a32169f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1853 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:34:43.404982) v4 > -2906> 2018-07-11 07:34:43.368846 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.2 10.8.128.106:0/12651 1853 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:43.404982) v4 ==== 2004+0+0 (222687508 0 0) 0x55a33234f600 con 0x55a3216a0800 > -2905> 2018-07-11 07:34:43.368865 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.2 10.8.128.106:0/12651 1853 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:43.404982) v4 ==== 2004+0+0 (222687508 0 0) 0x55a324474c00 con 0x55a32169f000 > -2904> 2018-07-11 07:34:43.368875 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:43.404982) v4 -- 0x55a32fb19000 con 0 > -2903> 2018-07-11 07:34:43.368907 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:43.404982) v4 -- 0x55a332e98600 con 0 > -2902> 2018-07-11 07:34:43.581698 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12650 conn(0x55a311b67000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1869 0x55a32e457000 osd_ping(ping e494 stamp 2018-07-11 07:34:43.617797) v4 > -2901> 2018-07-11 07:34:43.581724 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.3 10.8.128.106:0/12650 1869 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:43.617797) v4 ==== 2004+0+0 (1125695303 0 0) 0x55a32e457000 con 0x55a311b67000 > -2900> 2018-07-11 07:34:43.581735 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12650 conn(0x55a311b68800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1869 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:34:43.617797) v4 > -2899> 2018-07-11 07:34:43.581744 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:43.617797) v4 -- 0x55a32fb19000 con 0 > -2898> 2018-07-11 07:34:43.581752 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.3 10.8.128.106:0/12650 1869 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:43.617797) v4 ==== 2004+0+0 (1125695303 0 0) 0x55a34de6f000 con 0x55a311b68800 > -2897> 2018-07-11 07:34:43.581767 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:43.617797) v4 -- 0x55a330935000 con 0 > -2896> 2018-07-11 07:34:43.781367 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 60] Generated table #36588: 4957160 keys, 80348286 bytes > -2895> 2018-07-11 07:34:43.781405 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294483781389, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 36588, "file_size": 80348286, "table_properties": {"data_size": 67109393, "index_size": 689214, "filter_size": 12548689, "raw_key_size": 143206847, "raw_average_key_size": 28, "raw_value_size": 60, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957160, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957155", "kMergeOperands": "0"}} > -2894> 2018-07-11 07:34:43.822374 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15263 conn(0x55a321664000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1850 0x55a33234f600 osd_ping(ping e494 stamp 2018-07-11 07:34:45.036875) v4 > -2893> 2018-07-11 07:34:43.822401 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.5 10.8.128.113:0/15263 1850 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:45.036875) v4 ==== 2004+0+0 (1391049402 0 0) 0x55a33234f600 con 0x55a321664000 > -2892> 2018-07-11 07:34:43.822412 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:45.036875) v4 -- 0x55a31ce7a200 con 0 > -2891> 2018-07-11 07:34:43.822487 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15263 conn(0x55a321667000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1850 0x55a32e457000 osd_ping(ping e494 stamp 2018-07-11 07:34:45.036875) v4 > -2890> 2018-07-11 07:34:43.822502 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.5 10.8.128.113:0/15263 1850 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:45.036875) v4 ==== 2004+0+0 (1391049402 0 0) 0x55a32e457000 con 0x55a321667000 > -2889> 2018-07-11 07:34:43.822512 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:45.036875) v4 -- 0x55a34dea8800 con 0 > -2888> 2018-07-11 07:34:43.989135 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a3306e06c0 con 0 > -2887> 2018-07-11 07:34:44.003580 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5162 0x55a356208000 replica scrub(pg: 6.15,from:0'0,to:430'105762,epoch:494/492,start:6:aa10eb80:::benchmark_data_magna108_35617_object899320:0,end:6:aa11d5c9:::benchmark_data_magna108_35617_object287145:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -2886> 2018-07-11 07:34:44.003612 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5162 ==== replica scrub(pg: 6.15,from:0'0,to:430'105762,epoch:494/492,start:6:aa10eb80:::benchmark_data_magna108_35617_object899320:0,end:6:aa11d5c9:::benchmark_data_magna108_35617_object287145:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (1150626765 0 0) 0x55a356208000 con 0x55a317019000 > -2885> 2018-07-11 07:34:44.144995 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 60] Generated table #36589: 198890 keys, 3224173 bytes > -2884> 2018-07-11 07:34:44.145034 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294484145009, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 36589, "file_size": 3224173, "table_properties": {"data_size": 2692554, "index_size": 27157, "filter_size": 503477, "raw_key_size": 5745720, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 661, "num_entries": 198890, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "198890", "kMergeOperands": "0"}} > -2883> 2018-07-11 07:34:44.560211 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15919 conn(0x55a3170b4000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1856 0x55a33234f600 osd_ping(ping e494 stamp 2018-07-11 07:34:45.184280) v4 > -2882> 2018-07-11 07:34:44.560230 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15919 conn(0x55a317074800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1856 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:34:45.184280) v4 > -2881> 2018-07-11 07:34:44.560238 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.11 10.8.128.114:0/15919 1856 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:45.184280) v4 ==== 2004+0+0 (2361865984 0 0) 0x55a33234f600 con 0x55a3170b4000 > -2880> 2018-07-11 07:34:44.560260 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.11 10.8.128.114:0/15919 1856 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:45.184280) v4 ==== 2004+0+0 (2361865984 0 0) 0x55a332e79a00 con 0x55a317074800 > -2879> 2018-07-11 07:34:44.560269 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:45.184280) v4 -- 0x55a34de79600 con 0 > -2878> 2018-07-11 07:34:44.560300 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:45.184280) v4 -- 0x55a332e98600 con 0 > -2877> 2018-07-11 07:34:44.632967 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15255 conn(0x55a311b65800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1879 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:34:45.847439) v4 > -2876> 2018-07-11 07:34:44.632980 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15255 conn(0x55a311b64000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1879 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:34:45.847439) v4 > -2875> 2018-07-11 07:34:44.633003 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.0 10.8.128.113:0/15255 1879 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:45.847439) v4 ==== 2004+0+0 (3663616204 0 0) 0x55a324474c00 con 0x55a311b65800 > -2874> 2018-07-11 07:34:44.633008 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.0 10.8.128.113:0/15255 1879 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:45.847439) v4 ==== 2004+0+0 (3663616204 0 0) 0x55a34de78600 con 0x55a311b64000 > -2873> 2018-07-11 07:34:44.633014 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:45.847439) v4 -- 0x55a332e98600 con 0 > -2872> 2018-07-11 07:34:44.633044 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:45.847439) v4 -- 0x55a330935000 con 0 > -2871> 2018-07-11 07:34:44.737580 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a3428ee000 con 0 > -2870> 2018-07-11 07:34:44.752099 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5163 0x55a356208280 replica scrub(pg: 6.15,from:0'0,to:430'106436,epoch:494/492,start:6:aa11d5c9:::benchmark_data_magna108_35617_object287145:0,end:6:aa12f139:::benchmark_data_magna108_35617_object702145:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -2869> 2018-07-11 07:34:44.752130 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5163 ==== replica scrub(pg: 6.15,from:0'0,to:430'106436,epoch:494/492,start:6:aa11d5c9:::benchmark_data_magna108_35617_object287145:0,end:6:aa12f139:::benchmark_data_magna108_35617_object702145:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (3245658 0 0) 0x55a356208280 con 0x55a317019000 > -2868> 2018-07-11 07:34:44.872947 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15885 conn(0x55a3170b5800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1878 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:34:45.497005) v4 > -2867> 2018-07-11 07:34:44.872968 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.7 10.8.128.114:0/15885 1878 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:45.497005) v4 ==== 2004+0+0 (2754196162 0 0) 0x55a332e79a00 con 0x55a3170b5800 > -2866> 2018-07-11 07:34:44.872966 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15885 conn(0x55a311b76800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1878 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:34:45.497005) v4 > -2865> 2018-07-11 07:34:44.872977 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:45.497005) v4 -- 0x55a332febc00 con 0 > -2864> 2018-07-11 07:34:44.872990 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.7 10.8.128.114:0/15885 1878 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:45.497005) v4 ==== 2004+0+0 (2754196162 0 0) 0x55a34de6f000 con 0x55a311b76800 > -2863> 2018-07-11 07:34:44.873013 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:45.497005) v4 -- 0x55a32fb18200 con 0 > -2862> 2018-07-11 07:34:45.133363 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15255 conn(0x55a311b64000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1880 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:34:46.347853) v4 > -2861> 2018-07-11 07:34:45.133390 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.0 10.8.128.113:0/15255 1880 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:46.347853) v4 ==== 2004+0+0 (2799911406 0 0) 0x55a34de78600 con 0x55a311b64000 > -2860> 2018-07-11 07:34:45.133383 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15255 conn(0x55a311b65800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1880 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:34:46.347853) v4 > -2859> 2018-07-11 07:34:45.133402 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:46.347853) v4 -- 0x55a332ffa600 con 0 > -2858> 2018-07-11 07:34:45.133405 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.0 10.8.128.113:0/15255 1880 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:46.347853) v4 ==== 2004+0+0 (2799911406 0 0) 0x55a324474c00 con 0x55a311b65800 > -2857> 2018-07-11 07:34:45.133425 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:46.347853) v4 -- 0x55a33be6e800 con 0 > -2856> 2018-07-11 07:34:45.156569 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b7000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1805 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:34:44.907400) v4 > -2855> 2018-07-11 07:34:45.156589 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.8 10.8.128.108:0/11967 1805 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:44.907400) v4 ==== 2004+0+0 (4108356544 0 0) 0x55a34de6f000 con 0x55a3170b7000 > -2854> 2018-07-11 07:34:45.156599 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:44.907400) v4 -- 0x55a332364e00 con 0 > -2853> 2018-07-11 07:34:45.156646 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b8800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1805 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:34:44.907400) v4 > -2852> 2018-07-11 07:34:45.156671 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.8 10.8.128.108:0/11967 1805 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:44.907400) v4 ==== 2004+0+0 (4108356544 0 0) 0x55a332e79a00 con 0x55a3170b8800 > -2851> 2018-07-11 07:34:45.156688 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:44.907400) v4 -- 0x55a32e331200 con 0 > -2850> 2018-07-11 07:34:45.282227 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12650 conn(0x55a311b67000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1870 0x55a34dea8800 osd_ping(ping e494 stamp 2018-07-11 07:34:45.318353) v4 > -2849> 2018-07-11 07:34:45.282263 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.3 10.8.128.106:0/12650 1870 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:45.318353) v4 ==== 2004+0+0 (390425530 0 0) 0x55a34dea8800 con 0x55a311b67000 > -2848> 2018-07-11 07:34:45.282274 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:45.318353) v4 -- 0x55a34de79800 con 0 > -2847> 2018-07-11 07:34:45.282324 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12650 conn(0x55a311b68800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1870 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:34:45.318353) v4 > -2846> 2018-07-11 07:34:45.282345 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.3 10.8.128.106:0/12650 1870 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:45.318353) v4 ==== 2004+0+0 (390425530 0 0) 0x55a34de78600 con 0x55a311b68800 > -2845> 2018-07-11 07:34:45.282355 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:45.318353) v4 -- 0x55a332364e00 con 0 > -2844> 2018-07-11 07:34:45.433546 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a349f7ab40 con 0 > -2843> 2018-07-11 07:34:45.445406 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5164 0x55a356208500 replica scrub(pg: 6.15,from:0'0,to:430'105406,epoch:494/492,start:6:aa12f139:::benchmark_data_magna108_35617_object702145:0,end:6:aa141a51:::benchmark_data_magna108_35617_object482209:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -2842> 2018-07-11 07:34:45.445429 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5164 ==== replica scrub(pg: 6.15,from:0'0,to:430'105406,epoch:494/492,start:6:aa12f139:::benchmark_data_magna108_35617_object702145:0,end:6:aa141a51:::benchmark_data_magna108_35617_object482209:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (2529258129 0 0) 0x55a356208500 con 0x55a317019000 > -2841> 2018-07-11 07:34:45.522732 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15263 conn(0x55a321664000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1851 0x55a34de79600 osd_ping(ping e494 stamp 2018-07-11 07:34:46.737302) v4 > -2840> 2018-07-11 07:34:45.522758 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.5 10.8.128.113:0/15263 1851 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:46.737302) v4 ==== 2004+0+0 (3846653964 0 0) 0x55a34de79600 con 0x55a321664000 > -2839> 2018-07-11 07:34:45.522771 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:46.737302) v4 -- 0x55a34de79800 con 0 > -2838> 2018-07-11 07:34:45.522853 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15263 conn(0x55a321667000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1851 0x55a34dea8800 osd_ping(ping e494 stamp 2018-07-11 07:34:46.737302) v4 > -2837> 2018-07-11 07:34:45.522870 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.5 10.8.128.113:0/15263 1851 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:46.737302) v4 ==== 2004+0+0 (3846653964 0 0) 0x55a34dea8800 con 0x55a321667000 > -2836> 2018-07-11 07:34:45.522884 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:46.737302) v4 -- 0x55a34de79800 con 0 > -2835> 2018-07-11 07:34:45.633682 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15255 conn(0x55a311b65800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1881 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:34:46.848227) v4 > -2834> 2018-07-11 07:34:45.633703 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.0 10.8.128.113:0/15255 1881 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:46.848227) v4 ==== 2004+0+0 (40422765 0 0) 0x55a324474c00 con 0x55a311b65800 > -2833> 2018-07-11 07:34:45.633711 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15255 conn(0x55a311b64000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1881 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:34:46.848227) v4 > -2832> 2018-07-11 07:34:45.633721 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:46.848227) v4 -- 0x55a31334c600 con 0 > -2831> 2018-07-11 07:34:45.633734 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.0 10.8.128.113:0/15255 1881 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:46.848227) v4 ==== 2004+0+0 (40422765 0 0) 0x55a34de6f000 con 0x55a311b64000 > -2830> 2018-07-11 07:34:45.633753 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:46.848227) v4 -- 0x55a32e457400 con 0 > -2829> 2018-07-11 07:34:46.023230 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15263 conn(0x55a321667000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1852 0x55a34de79600 osd_ping(ping e494 stamp 2018-07-11 07:34:47.237728) v4 > -2828> 2018-07-11 07:34:46.023277 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.5 10.8.128.113:0/15263 1852 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:47.237728) v4 ==== 2004+0+0 (3106119908 0 0) 0x55a34de79600 con 0x55a321667000 > -2827> 2018-07-11 07:34:46.023288 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:47.237728) v4 -- 0x55a332e77600 con 0 > -2826> 2018-07-11 07:34:46.023344 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15263 conn(0x55a321664000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1852 0x55a34dea8800 osd_ping(ping e494 stamp 2018-07-11 07:34:47.237728) v4 > -2825> 2018-07-11 07:34:46.023358 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.5 10.8.128.113:0/15263 1852 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:47.237728) v4 ==== 2004+0+0 (3106119908 0 0) 0x55a34dea8800 con 0x55a321664000 > -2824> 2018-07-11 07:34:46.023369 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:47.237728) v4 -- 0x55a332e77200 con 0 > -2823> 2018-07-11 07:34:46.085873 7f355d88e700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.107:0/12617 conn(0x55a32447f800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1829 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:34:46.085428) v4 > -2822> 2018-07-11 07:34:46.085885 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.107:0/12617 conn(0x55a32447e000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1829 0x55a34de79600 osd_ping(ping e494 stamp 2018-07-11 07:34:46.085428) v4 > -2821> 2018-07-11 07:34:46.085896 7f355d88e700 1 -- 10.8.128.107:6803/12665 <== osd.9 10.8.128.107:0/12617 1829 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:46.085428) v4 ==== 2004+0+0 (2674634797 0 0) 0x55a34de78600 con 0x55a32447f800 > -2820> 2018-07-11 07:34:46.085902 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.9 10.8.128.107:0/12617 1829 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:46.085428) v4 ==== 2004+0+0 (2674634797 0 0) 0x55a34de79600 con 0x55a32447e000 > -2819> 2018-07-11 07:34:46.085907 7f355d88e700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:46.085428) v4 -- 0x55a332ff2200 con 0 > -2818> 2018-07-11 07:34:46.085972 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:46.085428) v4 -- 0x55a332349600 con 0 > -2817> 2018-07-11 07:34:46.105711 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a325956480 con 0 > -2816> 2018-07-11 07:34:46.120563 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5165 0x55a356208780 replica scrub(pg: 6.15,from:0'0,to:430'106221,epoch:494/492,start:6:aa141a51:::benchmark_data_magna108_35617_object482209:0,end:6:aa150ab0:::benchmark_data_magna108_35617_object259728:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -2815> 2018-07-11 07:34:46.120592 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5165 ==== replica scrub(pg: 6.15,from:0'0,to:430'106221,epoch:494/492,start:6:aa141a51:::benchmark_data_magna108_35617_object482209:0,end:6:aa150ab0:::benchmark_data_magna108_35617_object259728:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (1950793210 0 0) 0x55a356208780 con 0x55a317019000 > -2814> 2018-07-11 07:34:46.269293 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12651 conn(0x55a32169f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1854 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:34:46.305488) v4 > -2813> 2018-07-11 07:34:46.269295 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12651 conn(0x55a3216a0800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1854 0x55a34dea8800 osd_ping(ping e494 stamp 2018-07-11 07:34:46.305488) v4 > -2812> 2018-07-11 07:34:46.269319 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.2 10.8.128.106:0/12651 1854 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:46.305488) v4 ==== 2004+0+0 (2700669438 0 0) 0x55a34dea8800 con 0x55a3216a0800 > -2811> 2018-07-11 07:34:46.269318 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.2 10.8.128.106:0/12651 1854 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:46.305488) v4 ==== 2004+0+0 (2700669438 0 0) 0x55a332e79a00 con 0x55a32169f000 > -2810> 2018-07-11 07:34:46.269336 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:46.305488) v4 -- 0x55a34de79000 con 0 > -2809> 2018-07-11 07:34:46.269368 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:46.305488) v4 -- 0x55a31334c600 con 0 > -2808> 2018-07-11 07:34:46.277777 7f3543899700 5 osd.10 494 heartbeat: osd_stat(784 GB used, 146 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,1]) > -2807> 2018-07-11 07:34:46.277791 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6802/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:34:46.277790) v4 -- 0x55a34de84e00 con 0 > -2806> 2018-07-11 07:34:46.277828 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6803/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:34:46.277790) v4 -- 0x55a32ee77a00 con 0 > -2805> 2018-07-11 07:34:46.277843 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6806/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:34:46.277790) v4 -- 0x55a32447a400 con 0 > -2804> 2018-07-11 07:34:46.277858 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6807/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:34:46.277790) v4 -- 0x55a332feae00 con 0 > -2803> 2018-07-11 07:34:46.277878 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6808/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:34:46.277790) v4 -- 0x55a332faf400 con 0 > -2802> 2018-07-11 07:34:46.277897 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6810/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:34:46.277790) v4 -- 0x55a3314cee00 con 0 > -2801> 2018-07-11 07:34:46.277927 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6809/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:34:46.277790) v4 -- 0x55a3170df400 con 0 > -2800> 2018-07-11 07:34:46.277945 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6811/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:34:46.277790) v4 -- 0x55a3314cf800 con 0 > -2799> 2018-07-11 07:34:46.277968 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6810/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:34:46.277790) v4 -- 0x55a332349e00 con 0 > -2798> 2018-07-11 07:34:46.277991 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6811/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:34:46.277790) v4 -- 0x55a33234e800 con 0 > -2797> 2018-07-11 07:34:46.278010 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6806/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:34:46.277790) v4 -- 0x55a34de5b600 con 0 > -2796> 2018-07-11 07:34:46.278028 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6807/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:34:46.277790) v4 -- 0x55a314e7aa00 con 0 > -2795> 2018-07-11 07:34:46.278050 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6802/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:34:46.277790) v4 -- 0x55a332364200 con 0 > -2794> 2018-07-11 07:34:46.278074 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6803/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:34:46.277790) v4 -- 0x55a32162a600 con 0 > -2793> 2018-07-11 07:34:46.278091 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6806/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:34:46.277790) v4 -- 0x55a3177a8a00 con 0 > -2792> 2018-07-11 07:34:46.278115 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6807/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:34:46.277790) v4 -- 0x55a34de5a000 con 0 > -2791> 2018-07-11 07:34:46.278141 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6802/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:34:46.277790) v4 -- 0x55a332ec2a00 con 0 > -2790> 2018-07-11 07:34:46.278163 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6803/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:34:46.277790) v4 -- 0x55a323f54200 con 0 > -2789> 2018-07-11 07:34:46.278185 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6806/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:34:46.277790) v4 -- 0x55a332ec2e00 con 0 > -2788> 2018-07-11 07:34:46.278209 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6807/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:34:46.277790) v4 -- 0x55a332ec2800 con 0 > -2787> 2018-07-11 07:34:46.278205 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6807/12677 conn(0x55a314e85800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1864 0x55a34dea8800 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:46.277790) v4 > -2786> 2018-07-11 07:34:46.278222 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6807/12677 1864 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:46.277790) v4 ==== 2004+0+0 (849623412 0 0) 0x55a34dea8800 con 0x55a314e85800 > -2785> 2018-07-11 07:34:46.278229 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6802/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:34:46.277790) v4 -- 0x55a32eb20a00 con 0 > -2784> 2018-07-11 07:34:46.278264 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6803/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:34:46.277790) v4 -- 0x55a332ec2400 con 0 > -2783> 2018-07-11 07:34:46.278281 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6803/15255 conn(0x55a314e1d800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1864 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:46.277790) v4 > -2782> 2018-07-11 07:34:46.278304 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6803/15255 1864 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:46.277790) v4 ==== 2004+0+0 (849623412 0 0) 0x55a324474c00 con 0x55a314e1d800 > -2781> 2018-07-11 07:34:46.278331 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6806/12677 conn(0x55a314e84000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1864 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:46.277790) v4 > -2780> 2018-07-11 07:34:46.278349 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6806/12677 1864 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:46.277790) v4 ==== 2004+0+0 (849623412 0 0) 0x55a34de78600 con 0x55a314e84000 > -2779> 2018-07-11 07:34:46.278377 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6802/15255 conn(0x55a314e1c000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1864 0x55a332349600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:46.277790) v4 > -2778> 2018-07-11 07:34:46.278394 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6802/15255 1864 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:46.277790) v4 ==== 2004+0+0 (849623412 0 0) 0x55a332349600 con 0x55a314e1c000 > -2777> 2018-07-11 07:34:46.278400 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6810/12651 conn(0x55a314e97800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1864 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:46.277790) v4 > -2776> 2018-07-11 07:34:46.278428 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6811/12650 conn(0x55a314e20800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1864 0x55a332349600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:46.277790) v4 > -2775> 2018-07-11 07:34:46.278428 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6806/15263 conn(0x55a314e99000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1864 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:46.277790) v4 > -2774> 2018-07-11 07:34:46.278440 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6811/12650 1864 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:46.277790) v4 ==== 2004+0+0 (849623412 0 0) 0x55a332349600 con 0x55a314e20800 > -2773> 2018-07-11 07:34:46.278443 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6806/15263 1864 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:46.277790) v4 ==== 2004+0+0 (849623412 0 0) 0x55a34de6f000 con 0x55a314e99000 > -2772> 2018-07-11 07:34:46.278419 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6810/12651 1864 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:46.277790) v4 ==== 2004+0+0 (849623412 0 0) 0x55a332e79a00 con 0x55a314e97800 > -2771> 2018-07-11 07:34:46.278475 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6808/12651 conn(0x55a314e96000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1864 0x55a332349600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:46.277790) v4 > -2770> 2018-07-11 07:34:46.278488 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6808/12651 1864 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:46.277790) v4 ==== 2004+0+0 (849623412 0 0) 0x55a332349600 con 0x55a314e96000 > -2769> 2018-07-11 07:34:46.278484 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6809/12650 conn(0x55a314e1f000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1864 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:46.277790) v4 > -2768> 2018-07-11 07:34:46.278502 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6809/12650 1864 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:46.277790) v4 ==== 2004+0+0 (849623412 0 0) 0x55a34de6f000 con 0x55a314e1f000 > -2767> 2018-07-11 07:34:46.278498 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6811/15287 conn(0x55a314e04800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1864 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:46.277790) v4 > -2766> 2018-07-11 07:34:46.278514 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6811/15287 1864 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:46.277790) v4 ==== 2004+0+0 (849623412 0 0) 0x55a332e79a00 con 0x55a314e04800 > -2765> 2018-07-11 07:34:46.278528 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6807/12617 conn(0x55a314e27800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1864 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:46.277790) v4 > -2764> 2018-07-11 07:34:46.278523 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6810/15287 conn(0x55a314e03000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1864 0x55a332349600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:46.277790) v4 > -2763> 2018-07-11 07:34:46.278542 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6807/12617 1864 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:46.277790) v4 ==== 2004+0+0 (849623412 0 0) 0x55a34de6f000 con 0x55a314e27800 > -2762> 2018-07-11 07:34:46.278548 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6810/15287 1864 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:46.277790) v4 ==== 2004+0+0 (849623412 0 0) 0x55a332349600 con 0x55a314e03000 > -2761> 2018-07-11 07:34:46.278561 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6806/12617 conn(0x55a314e26000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1864 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:46.277790) v4 > -2760> 2018-07-11 07:34:46.278569 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6803/13323 conn(0x55a314e16800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1864 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:46.277790) v4 > -2759> 2018-07-11 07:34:46.278580 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6807/15263 conn(0x55a314e9a800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1864 0x55a332349600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:46.277790) v4 > -2758> 2018-07-11 07:34:46.278585 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6803/13323 1864 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:46.277790) v4 ==== 2004+0+0 (849623412 0 0) 0x55a34de6f000 con 0x55a314e16800 > -2757> 2018-07-11 07:34:46.278579 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6806/12617 1864 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:46.277790) v4 ==== 2004+0+0 (849623412 0 0) 0x55a332e79a00 con 0x55a314e26000 > -2756> 2018-07-11 07:34:46.278592 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6807/15263 1864 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:46.277790) v4 ==== 2004+0+0 (849623412 0 0) 0x55a332349600 con 0x55a314e9a800 > -2755> 2018-07-11 07:34:46.278615 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6807/15885 conn(0x55a314e01800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1864 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:46.277790) v4 > -2754> 2018-07-11 07:34:46.278623 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6803/11967 conn(0x55a314e13800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1864 0x55a332349600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:46.277790) v4 > -2753> 2018-07-11 07:34:46.278627 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6807/15885 1864 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:46.277790) v4 ==== 2004+0+0 (849623412 0 0) 0x55a34de6f000 con 0x55a314e01800 > -2752> 2018-07-11 07:34:46.278637 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6803/11967 1864 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:46.277790) v4 ==== 2004+0+0 (849623412 0 0) 0x55a332349600 con 0x55a314e13800 > -2751> 2018-07-11 07:34:46.278626 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6806/15885 conn(0x55a314e00000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1864 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:46.277790) v4 > -2750> 2018-07-11 07:34:46.278648 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6806/15885 1864 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:46.277790) v4 ==== 2004+0+0 (849623412 0 0) 0x55a332e79a00 con 0x55a314e00000 > -2749> 2018-07-11 07:34:46.278670 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6802/11967 conn(0x55a314e12000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1864 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:46.277790) v4 > -2748> 2018-07-11 07:34:46.278682 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6802/11967 1864 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:46.277790) v4 ==== 2004+0+0 (849623412 0 0) 0x55a34de6f000 con 0x55a314e12000 > -2747> 2018-07-11 07:34:46.278694 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6802/13323 conn(0x55a314e15000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1864 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:46.277790) v4 > -2746> 2018-07-11 07:34:46.278712 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6802/13323 1864 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:46.277790) v4 ==== 2004+0+0 (849623412 0 0) 0x55a332e79a00 con 0x55a314e15000 > -2745> 2018-07-11 07:34:46.278755 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6802/15919 conn(0x55a314e87000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1864 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:46.277790) v4 > -2744> 2018-07-11 07:34:46.278759 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6803/15919 conn(0x55a314e88800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1864 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:46.277790) v4 > -2743> 2018-07-11 07:34:46.278772 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6802/15919 1864 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:46.277790) v4 ==== 2004+0+0 (849623412 0 0) 0x55a332e79a00 con 0x55a314e87000 > -2742> 2018-07-11 07:34:46.278784 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6803/15919 1864 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:46.277790) v4 ==== 2004+0+0 (849623412 0 0) 0x55a34de6f000 con 0x55a314e88800 > -2741> 2018-07-11 07:34:46.382658 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12650 conn(0x55a311b67000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1871 0x55a332349600 osd_ping(ping e494 stamp 2018-07-11 07:34:46.418918) v4 > -2740> 2018-07-11 07:34:46.382682 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.3 10.8.128.106:0/12650 1871 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:46.418918) v4 ==== 2004+0+0 (289166637 0 0) 0x55a332349600 con 0x55a311b67000 > -2739> 2018-07-11 07:34:46.382700 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:46.418918) v4 -- 0x55a34de79000 con 0 > -2738> 2018-07-11 07:34:46.382798 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12650 conn(0x55a311b68800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1871 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:34:46.418918) v4 > -2737> 2018-07-11 07:34:46.382815 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.3 10.8.128.106:0/12650 1871 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:46.418918) v4 ==== 2004+0+0 (289166637 0 0) 0x55a34de6f000 con 0x55a311b68800 > -2736> 2018-07-11 07:34:46.382827 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:46.418918) v4 -- 0x55a33235a400 con 0 > -2735> 2018-07-11 07:34:46.455473 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.115:0/13323 conn(0x55a321668800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1866 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:34:46.793162) v4 > -2734> 2018-07-11 07:34:46.455499 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.6 10.8.128.115:0/13323 1866 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:46.793162) v4 ==== 2004+0+0 (2516781809 0 0) 0x55a332e79a00 con 0x55a321668800 > -2733> 2018-07-11 07:34:46.455513 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:46.793162) v4 -- 0x55a32ee77a00 con 0 > -2732> 2018-07-11 07:34:46.455591 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.115:0/13323 conn(0x55a321665800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1866 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:34:46.793162) v4 > -2731> 2018-07-11 07:34:46.455612 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.6 10.8.128.115:0/13323 1866 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:46.793162) v4 ==== 2004+0+0 (2516781809 0 0) 0x55a324474c00 con 0x55a321665800 > -2730> 2018-07-11 07:34:46.455626 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:46.793162) v4 -- 0x55a32ee77a00 con 0 > -2729> 2018-07-11 07:34:46.745122 7f35498a5700 10 monclient: tick > -2728> 2018-07-11 07:34:46.745135 7f35498a5700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2018-07-11 07:34:16.745134) > -2727> 2018-07-11 07:34:46.833982 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a325960240 con 0 > -2726> 2018-07-11 07:34:46.845771 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5166 0x55a356208a00 replica scrub(pg: 6.15,from:0'0,to:430'106604,epoch:494/492,start:6:aa150ab0:::benchmark_data_magna108_35617_object259728:0,end:6:aa15f2e0:::benchmark_data_magna108_35617_object955355:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -2725> 2018-07-11 07:34:46.845803 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5166 ==== replica scrub(pg: 6.15,from:0'0,to:430'106604,epoch:494/492,start:6:aa150ab0:::benchmark_data_magna108_35617_object259728:0,end:6:aa15f2e0:::benchmark_data_magna108_35617_object955355:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (3942184630 0 0) 0x55a356208a00 con 0x55a317019000 > -2724> 2018-07-11 07:34:47.123654 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15263 conn(0x55a321664000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1853 0x55a332349600 osd_ping(ping e494 stamp 2018-07-11 07:34:48.338124) v4 > -2723> 2018-07-11 07:34:47.123682 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.5 10.8.128.113:0/15263 1853 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:48.338124) v4 ==== 2004+0+0 (570003179 0 0) 0x55a332349600 con 0x55a321664000 > -2722> 2018-07-11 07:34:47.123695 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:48.338124) v4 -- 0x55a34de79000 con 0 > -2721> 2018-07-11 07:34:47.123782 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15263 conn(0x55a321667000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1853 0x55a34dea8800 osd_ping(ping e494 stamp 2018-07-11 07:34:48.338124) v4 > -2720> 2018-07-11 07:34:47.123800 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.5 10.8.128.113:0/15263 1853 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:48.338124) v4 ==== 2004+0+0 (570003179 0 0) 0x55a34dea8800 con 0x55a321667000 > -2719> 2018-07-11 07:34:47.123813 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:48.338124) v4 -- 0x55a34de79000 con 0 > -2718> 2018-07-11 07:34:47.256505 7f35488a3700 1 -- 10.8.128.107:6800/12665 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55a3170c42c0 con 0 > -2717> 2018-07-11 07:34:47.256581 7f35488a3700 1 -- 10.8.128.107:6800/12665 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55a32ec12b00 con 0 > -2716> 2018-07-11 07:34:47.334149 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15287 conn(0x55a311b75000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1852 0x55a332349600 osd_ping(ping e494 stamp 2018-07-11 07:34:48.548655) v4 > -2715> 2018-07-11 07:34:47.334175 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.4 10.8.128.113:0/15287 1852 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:48.548655) v4 ==== 2004+0+0 (1670807981 0 0) 0x55a332349600 con 0x55a311b75000 > -2714> 2018-07-11 07:34:47.334194 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:48.548655) v4 -- 0x55a34de6fc00 con 0 > -2713> 2018-07-11 07:34:47.334243 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15287 conn(0x55a311b72000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1852 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:34:48.548655) v4 > -2712> 2018-07-11 07:34:47.334273 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.4 10.8.128.113:0/15287 1852 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:48.548655) v4 ==== 2004+0+0 (1670807981 0 0) 0x55a332e79a00 con 0x55a311b72000 > -2711> 2018-07-11 07:34:47.334285 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:48.548655) v4 -- 0x55a31334c600 con 0 > -2710> 2018-07-11 07:34:47.500834 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a332edfb00 con 0 > -2709> 2018-07-11 07:34:47.529008 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5167 0x55a356208c80 replica scrub(pg: 6.15,from:0'0,to:430'105459,epoch:494/492,start:6:aa15f2e0:::benchmark_data_magna108_35617_object955355:0,end:6:aa1684ba:::benchmark_data_magna108_41525_object57871:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -2708> 2018-07-11 07:34:47.529037 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5167 ==== replica scrub(pg: 6.15,from:0'0,to:430'105459,epoch:494/492,start:6:aa15f2e0:::benchmark_data_magna108_35617_object955355:0,end:6:aa1684ba:::benchmark_data_magna108_41525_object57871:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 217+0+0 (1168460294 0 0) 0x55a356208c80 con 0x55a317019000 > -2707> 2018-07-11 07:34:47.534558 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12677 conn(0x55a32169d800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1877 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:34:47.570784) v4 > -2706> 2018-07-11 07:34:47.534579 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.1 10.8.128.106:0/12677 1877 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:47.570784) v4 ==== 2004+0+0 (3306258911 0 0) 0x55a34de78600 con 0x55a32169d800 > -2705> 2018-07-11 07:34:47.534588 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:47.570784) v4 -- 0x55a33235a400 con 0 > -2704> 2018-07-11 07:34:47.534610 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12677 conn(0x55a32169c000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1877 0x55a34dea8800 osd_ping(ping e494 stamp 2018-07-11 07:34:47.570784) v4 > -2703> 2018-07-11 07:34:47.534635 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.1 10.8.128.106:0/12677 1877 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:47.570784) v4 ==== 2004+0+0 (3306258911 0 0) 0x55a34dea8800 con 0x55a32169c000 > -2702> 2018-07-11 07:34:47.534649 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:47.570784) v4 -- 0x55a33532c200 con 0 > -2701> 2018-07-11 07:34:47.624110 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15263 conn(0x55a321667000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1854 0x55a34de6fc00 osd_ping(ping e494 stamp 2018-07-11 07:34:48.838715) v4 > -2700> 2018-07-11 07:34:47.624138 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.5 10.8.128.113:0/15263 1854 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:48.838715) v4 ==== 2004+0+0 (1057396756 0 0) 0x55a34de6fc00 con 0x55a321667000 > -2699> 2018-07-11 07:34:47.624152 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:48.838715) v4 -- 0x55a32442a000 con 0 > -2698> 2018-07-11 07:34:47.624198 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15263 conn(0x55a321664000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1854 0x55a34de6fc00 osd_ping(ping e494 stamp 2018-07-11 07:34:48.838715) v4 > -2697> 2018-07-11 07:34:47.624212 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.5 10.8.128.113:0/15263 1854 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:48.838715) v4 ==== 2004+0+0 (1057396756 0 0) 0x55a34de6fc00 con 0x55a321664000 > -2696> 2018-07-11 07:34:47.624222 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:48.838715) v4 -- 0x55a33235a600 con 0 > -2695> 2018-07-11 07:34:47.834755 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15287 conn(0x55a311b75000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1853 0x55a34dea8800 osd_ping(ping e494 stamp 2018-07-11 07:34:49.049265) v4 > -2694> 2018-07-11 07:34:47.834776 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.4 10.8.128.113:0/15287 1853 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:49.049265) v4 ==== 2004+0+0 (1794457270 0 0) 0x55a34dea8800 con 0x55a311b75000 > -2693> 2018-07-11 07:34:47.834788 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:49.049265) v4 -- 0x55a32442a000 con 0 > -2692> 2018-07-11 07:34:47.834941 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15287 conn(0x55a311b72000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1853 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:34:49.049265) v4 > -2691> 2018-07-11 07:34:47.834957 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.4 10.8.128.113:0/15287 1853 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:49.049265) v4 ==== 2004+0+0 (1794457270 0 0) 0x55a324474c00 con 0x55a311b72000 > -2690> 2018-07-11 07:34:47.834967 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:49.049265) v4 -- 0x55a31334da00 con 0 > -2689> 2018-07-11 07:34:48.137603 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a32a6f3440 con 0 > -2688> 2018-07-11 07:34:48.154505 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5168 0x55a356208f00 replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:494/492,start:6:aa1684ba:::benchmark_data_magna108_41525_object57871:0,end:6:aa17140c:::benchmark_data_magna108_35617_object627726:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -2687> 2018-07-11 07:34:48.154529 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5168 ==== replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:494/492,start:6:aa1684ba:::benchmark_data_magna108_41525_object57871:0,end:6:aa17140c:::benchmark_data_magna108_35617_object627726:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 217+0+0 (2534306940 0 0) 0x55a356208f00 con 0x55a317019000 > -2686> 2018-07-11 07:34:48.534100 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15255 conn(0x55a311b65800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1882 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:34:49.748632) v4 > -2685> 2018-07-11 07:34:48.534107 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15255 conn(0x55a311b64000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1882 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:34:49.748632) v4 > -2684> 2018-07-11 07:34:48.534133 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.0 10.8.128.113:0/15255 1882 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:49.748632) v4 ==== 2004+0+0 (3561628768 0 0) 0x55a332e79a00 con 0x55a311b65800 > -2683> 2018-07-11 07:34:48.534137 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.0 10.8.128.113:0/15255 1882 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:49.748632) v4 ==== 2004+0+0 (3561628768 0 0) 0x55a34de6f000 con 0x55a311b64000 > -2682> 2018-07-11 07:34:48.534149 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:49.748632) v4 -- 0x55a3131ba600 con 0 > -2681> 2018-07-11 07:34:48.534177 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:49.748632) v4 -- 0x55a332ead200 con 0 > -2680> 2018-07-11 07:34:48.660699 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15919 conn(0x55a3170b4000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1857 0x55a34de6fc00 osd_ping(ping e494 stamp 2018-07-11 07:34:49.284721) v4 > -2679> 2018-07-11 07:34:48.660721 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.11 10.8.128.114:0/15919 1857 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:49.284721) v4 ==== 2004+0+0 (2481505504 0 0) 0x55a34de6fc00 con 0x55a3170b4000 > -2678> 2018-07-11 07:34:48.660731 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:49.284721) v4 -- 0x55a332ffa000 con 0 > -2677> 2018-07-11 07:34:48.660725 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15919 conn(0x55a317074800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1857 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:34:49.284721) v4 > -2676> 2018-07-11 07:34:48.660742 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.11 10.8.128.114:0/15919 1857 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:49.284721) v4 ==== 2004+0+0 (2481505504 0 0) 0x55a324474c00 con 0x55a317074800 > -2675> 2018-07-11 07:34:48.660751 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:49.284721) v4 -- 0x55a34866ae00 con 0 > -2674> 2018-07-11 07:34:48.973315 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15885 conn(0x55a3170b5800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1879 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:34:49.597395) v4 > -2673> 2018-07-11 07:34:48.973320 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15885 conn(0x55a311b76800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1879 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:34:49.597395) v4 > -2672> 2018-07-11 07:34:48.973338 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.7 10.8.128.114:0/15885 1879 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:49.597395) v4 ==== 2004+0+0 (3174011598 0 0) 0x55a332e79a00 con 0x55a3170b5800 > -2671> 2018-07-11 07:34:48.973341 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.7 10.8.128.114:0/15885 1879 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:49.597395) v4 ==== 2004+0+0 (3174011598 0 0) 0x55a34de78600 con 0x55a311b76800 > -2670> 2018-07-11 07:34:48.973351 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:49.597395) v4 -- 0x55a32f761c00 con 0 > -2669> 2018-07-11 07:34:48.973378 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:49.597395) v4 -- 0x55a33532d600 con 0 > -2668> 2018-07-11 07:34:48.986334 7f355d88e700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.107:0/12617 conn(0x55a32447f800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1830 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:34:48.985981) v4 > -2667> 2018-07-11 07:34:48.986352 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.107:0/12617 conn(0x55a32447e000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1830 0x55a32442a000 osd_ping(ping e494 stamp 2018-07-11 07:34:48.985981) v4 > -2666> 2018-07-11 07:34:48.986378 7f355d88e700 1 -- 10.8.128.107:6803/12665 <== osd.9 10.8.128.107:0/12617 1830 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:48.985981) v4 ==== 2004+0+0 (3049408602 0 0) 0x55a34de6f000 con 0x55a32447f800 > -2665> 2018-07-11 07:34:48.986384 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.9 10.8.128.107:0/12617 1830 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:48.985981) v4 ==== 2004+0+0 (3049408602 0 0) 0x55a32442a000 con 0x55a32447e000 > -2664> 2018-07-11 07:34:48.986392 7f355d88e700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:48.985981) v4 -- 0x55a34de6f600 con 0 > -2663> 2018-07-11 07:34:48.986423 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:48.985981) v4 -- 0x55a33235b000 con 0 > -2662> 2018-07-11 07:34:49.048786 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a32594c900 con 0 > -2661> 2018-07-11 07:34:49.063611 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5169 0x55a356209180 replica scrub(pg: 6.15,from:0'0,to:430'106643,epoch:494/492,start:6:aa17140c:::benchmark_data_magna108_35617_object627726:0,end:6:aa17c172:::benchmark_data_magna108_35617_object286242:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -2660> 2018-07-11 07:34:49.063639 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5169 ==== replica scrub(pg: 6.15,from:0'0,to:430'106643,epoch:494/492,start:6:aa17140c:::benchmark_data_magna108_35617_object627726:0,end:6:aa17c172:::benchmark_data_magna108_35617_object286242:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (214224452 0 0) 0x55a356209180 con 0x55a317019000 > -2659> 2018-07-11 07:34:49.256994 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b7000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1806 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:34:49.007837) v4 > -2658> 2018-07-11 07:34:49.257017 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.8 10.8.128.108:0/11967 1806 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:49.007837) v4 ==== 2004+0+0 (1518695726 0 0) 0x55a34de78600 con 0x55a3170b7000 > -2657> 2018-07-11 07:34:49.257027 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:49.007837) v4 -- 0x55a34de6f600 con 0 > -2656> 2018-07-11 07:34:49.257042 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b8800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1806 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:34:49.007837) v4 > -2655> 2018-07-11 07:34:49.257065 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.8 10.8.128.108:0/11967 1806 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:49.007837) v4 ==== 2004+0+0 (1518695726 0 0) 0x55a324474c00 con 0x55a3170b8800 > -2654> 2018-07-11 07:34:49.257077 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:49.007837) v4 -- 0x55a32f761c00 con 0 > -2653> 2018-07-11 07:34:49.747811 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a322624d80 con 0 > -2652> 2018-07-11 07:34:49.755608 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5170 0x55a356209400 replica scrub(pg: 6.15,from:0'0,to:430'106252,epoch:494/492,start:6:aa17c172:::benchmark_data_magna108_35617_object286242:0,end:6:aa18bf57:::benchmark_data_magna108_35617_object485584:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -2651> 2018-07-11 07:34:49.755637 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5170 ==== replica scrub(pg: 6.15,from:0'0,to:430'106252,epoch:494/492,start:6:aa17c172:::benchmark_data_magna108_35617_object286242:0,end:6:aa18bf57:::benchmark_data_magna108_35617_object485584:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (961362781 0 0) 0x55a356209400 con 0x55a317019000 > -2650> 2018-07-11 07:34:49.760970 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15919 conn(0x55a317074800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1858 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:34:50.385118) v4 > -2649> 2018-07-11 07:34:49.760973 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15919 conn(0x55a3170b4000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1858 0x55a34de6fc00 osd_ping(ping e494 stamp 2018-07-11 07:34:50.385118) v4 > -2648> 2018-07-11 07:34:49.760992 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.11 10.8.128.114:0/15919 1858 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:50.385118) v4 ==== 2004+0+0 (3066875009 0 0) 0x55a332e79a00 con 0x55a317074800 > -2647> 2018-07-11 07:34:49.760994 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.11 10.8.128.114:0/15919 1858 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:50.385118) v4 ==== 2004+0+0 (3066875009 0 0) 0x55a34de6fc00 con 0x55a3170b4000 > -2646> 2018-07-11 07:34:49.761004 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:50.385118) v4 -- 0x55a32f761c00 con 0 > -2645> 2018-07-11 07:34:49.761029 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:50.385118) v4 -- 0x55a33234b400 con 0 > -2644> 2018-07-11 07:34:49.769624 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12651 conn(0x55a3216a0800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1855 0x55a33b42d400 osd_ping(ping e494 stamp 2018-07-11 07:34:49.805901) v4 > -2643> 2018-07-11 07:34:49.769651 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.2 10.8.128.106:0/12651 1855 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:49.805901) v4 ==== 2004+0+0 (2869392550 0 0) 0x55a33b42d400 con 0x55a3216a0800 > -2642> 2018-07-11 07:34:49.769668 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:49.805901) v4 -- 0x55a33234b400 con 0 > -2641> 2018-07-11 07:34:49.769696 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12651 conn(0x55a32169f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1855 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:34:49.805901) v4 > -2640> 2018-07-11 07:34:49.769717 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.2 10.8.128.106:0/12651 1855 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:49.805901) v4 ==== 2004+0+0 (2869392550 0 0) 0x55a324474c00 con 0x55a32169f000 > -2639> 2018-07-11 07:34:49.769733 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:49.805901) v4 -- 0x55a323f54800 con 0 > -2638> 2018-07-11 07:34:49.778422 7f3543899700 5 osd.10 494 heartbeat: osd_stat(784 GB used, 146 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,1]) > -2637> 2018-07-11 07:34:49.778437 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6802/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:34:49.778435) v4 -- 0x55a332ec2200 con 0 > -2636> 2018-07-11 07:34:49.778472 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6803/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:34:49.778435) v4 -- 0x55a332e99e00 con 0 > -2635> 2018-07-11 07:34:49.778488 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6806/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:34:49.778435) v4 -- 0x55a332ec3a00 con 0 > -2634> 2018-07-11 07:34:49.778504 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6807/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:34:49.778435) v4 -- 0x55a332ec3400 con 0 > -2633> 2018-07-11 07:34:49.778517 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6808/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:34:49.778435) v4 -- 0x55a332ec3000 con 0 > -2632> 2018-07-11 07:34:49.778527 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6810/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:34:49.778435) v4 -- 0x55a332ec3600 con 0 > -2631> 2018-07-11 07:34:49.778541 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6809/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:34:49.778435) v4 -- 0x55a332ec3e00 con 0 > -2630> 2018-07-11 07:34:49.778553 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6811/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:34:49.778435) v4 -- 0x55a332ec2600 con 0 > -2629> 2018-07-11 07:34:49.778577 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6810/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:34:49.778435) v4 -- 0x55a332ec3c00 con 0 > -2628> 2018-07-11 07:34:49.778591 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6811/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:34:49.778435) v4 -- 0x55a32f760000 con 0 > -2627> 2018-07-11 07:34:49.778605 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6806/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:34:49.778435) v4 -- 0x55a332ec3800 con 0 > -2626> 2018-07-11 07:34:49.778625 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6807/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:34:49.778435) v4 -- 0x55a31706f600 con 0 > -2625> 2018-07-11 07:34:49.778639 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6802/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:34:49.778435) v4 -- 0x55a34de5b200 con 0 > -2624> 2018-07-11 07:34:49.778651 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6803/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:34:49.778435) v4 -- 0x55a332365a00 con 0 > -2623> 2018-07-11 07:34:49.778663 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6806/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:34:49.778435) v4 -- 0x55a332ec2000 con 0 > -2622> 2018-07-11 07:34:49.778675 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6807/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:34:49.778435) v4 -- 0x55a32eb14400 con 0 > -2621> 2018-07-11 07:34:49.778687 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6802/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:34:49.778435) v4 -- 0x55a32f760c00 con 0 > -2620> 2018-07-11 07:34:49.778705 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6803/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:34:49.778435) v4 -- 0x55a32f760400 con 0 > -2619> 2018-07-11 07:34:49.778721 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6806/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:34:49.778435) v4 -- 0x55a32f761400 con 0 > -2618> 2018-07-11 07:34:49.778734 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6807/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:34:49.778435) v4 -- 0x55a32f761a00 con 0 > -2617> 2018-07-11 07:34:49.778746 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6802/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:34:49.778435) v4 -- 0x55a332e98800 con 0 > -2616> 2018-07-11 07:34:49.778758 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6803/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:34:49.778435) v4 -- 0x55a32f761600 con 0 > -2615> 2018-07-11 07:34:49.778857 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6802/15255 conn(0x55a314e1c000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1865 0x55a34de6fc00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:49.778435) v4 > -2614> 2018-07-11 07:34:49.778884 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6802/15255 1865 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:49.778435) v4 ==== 2004+0+0 (1907713210 0 0) 0x55a34de6fc00 con 0x55a314e1c000 > -2613> 2018-07-11 07:34:49.778917 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6803/15255 conn(0x55a314e1d800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1865 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:49.778435) v4 > -2612> 2018-07-11 07:34:49.778940 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6803/15255 1865 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:49.778435) v4 ==== 2004+0+0 (1907713210 0 0) 0x55a324474c00 con 0x55a314e1d800 > -2611> 2018-07-11 07:34:49.778940 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6808/12651 conn(0x55a314e96000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1865 0x55a34de6fc00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:49.778435) v4 > -2610> 2018-07-11 07:34:49.778956 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6807/12617 conn(0x55a314e27800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1865 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:49.778435) v4 > -2609> 2018-07-11 07:34:49.778960 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6808/12651 1865 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:49.778435) v4 ==== 2004+0+0 (1907713210 0 0) 0x55a34de6fc00 con 0x55a314e96000 > -2608> 2018-07-11 07:34:49.778975 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6807/12617 1865 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:49.778435) v4 ==== 2004+0+0 (1907713210 0 0) 0x55a34de78600 con 0x55a314e27800 > -2607> 2018-07-11 07:34:49.778981 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6806/12617 conn(0x55a314e26000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1865 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:49.778435) v4 > -2606> 2018-07-11 07:34:49.779007 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6807/12677 conn(0x55a314e85800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1865 0x55a34de6fc00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:49.778435) v4 > -2605> 2018-07-11 07:34:49.779017 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6806/12617 1865 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:49.778435) v4 ==== 2004+0+0 (1907713210 0 0) 0x55a324474c00 con 0x55a314e26000 > -2604> 2018-07-11 07:34:49.779021 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6807/12677 1865 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:49.778435) v4 ==== 2004+0+0 (1907713210 0 0) 0x55a34de6fc00 con 0x55a314e85800 > -2603> 2018-07-11 07:34:49.779022 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6806/12677 conn(0x55a314e84000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1865 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:49.778435) v4 > -2602> 2018-07-11 07:34:49.779039 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6806/12677 1865 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:49.778435) v4 ==== 2004+0+0 (1907713210 0 0) 0x55a34de78600 con 0x55a314e84000 > -2601> 2018-07-11 07:34:49.779069 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6810/12651 conn(0x55a314e97800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1865 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:49.778435) v4 > -2600> 2018-07-11 07:34:49.779072 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6810/15287 conn(0x55a314e03000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1865 0x55a34de6fc00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:49.778435) v4 > -2599> 2018-07-11 07:34:49.779084 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6809/12650 conn(0x55a314e1f000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1865 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:49.778435) v4 > -2598> 2018-07-11 07:34:49.779091 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6810/12651 1865 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:49.778435) v4 ==== 2004+0+0 (1907713210 0 0) 0x55a324474c00 con 0x55a314e97800 > -2597> 2018-07-11 07:34:49.779094 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6810/15287 1865 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:49.778435) v4 ==== 2004+0+0 (1907713210 0 0) 0x55a34de6fc00 con 0x55a314e03000 > -2596> 2018-07-11 07:34:49.779101 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6809/12650 1865 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:49.778435) v4 ==== 2004+0+0 (1907713210 0 0) 0x55a34de78600 con 0x55a314e1f000 > -2595> 2018-07-11 07:34:49.779141 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6807/15263 conn(0x55a314e9a800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1865 0x55a34de6fc00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:49.778435) v4 > -2594> 2018-07-11 07:34:49.779159 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6807/15263 1865 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:49.778435) v4 ==== 2004+0+0 (1907713210 0 0) 0x55a34de6fc00 con 0x55a314e9a800 > -2593> 2018-07-11 07:34:49.779192 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6806/15263 conn(0x55a314e99000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1865 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:49.778435) v4 > -2592> 2018-07-11 07:34:49.779212 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6806/15263 1865 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:49.778435) v4 ==== 2004+0+0 (1907713210 0 0) 0x55a34de78600 con 0x55a314e99000 > -2591> 2018-07-11 07:34:49.779210 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6811/12650 conn(0x55a314e20800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1865 0x55a34de6fc00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:49.778435) v4 > -2590> 2018-07-11 07:34:49.779138 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6811/15287 conn(0x55a314e04800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1865 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:49.778435) v4 > -2589> 2018-07-11 07:34:49.779227 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6811/12650 1865 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:49.778435) v4 ==== 2004+0+0 (1907713210 0 0) 0x55a34de6fc00 con 0x55a314e20800 > -2588> 2018-07-11 07:34:49.779231 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6811/15287 1865 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:49.778435) v4 ==== 2004+0+0 (1907713210 0 0) 0x55a324474c00 con 0x55a314e04800 > -2587> 2018-07-11 07:34:49.779288 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6803/13323 conn(0x55a314e16800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1865 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:49.778435) v4 > -2586> 2018-07-11 07:34:49.779292 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6806/15885 conn(0x55a314e00000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1865 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:49.778435) v4 > -2585> 2018-07-11 07:34:49.779303 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6803/13323 1865 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:49.778435) v4 ==== 2004+0+0 (1907713210 0 0) 0x55a34de78600 con 0x55a314e16800 > -2584> 2018-07-11 07:34:49.779292 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6803/11967 conn(0x55a314e13800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1865 0x55a34de6fc00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:49.778435) v4 > -2583> 2018-07-11 07:34:49.779316 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6806/15885 1865 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:49.778435) v4 ==== 2004+0+0 (1907713210 0 0) 0x55a324474c00 con 0x55a314e00000 > -2582> 2018-07-11 07:34:49.779323 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6803/11967 1865 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:49.778435) v4 ==== 2004+0+0 (1907713210 0 0) 0x55a34de6fc00 con 0x55a314e13800 > -2581> 2018-07-11 07:34:49.779347 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6807/15885 conn(0x55a314e01800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1865 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:49.778435) v4 > -2580> 2018-07-11 07:34:49.779361 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6807/15885 1865 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:49.778435) v4 ==== 2004+0+0 (1907713210 0 0) 0x55a34de78600 con 0x55a314e01800 > -2579> 2018-07-11 07:34:49.779366 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6802/13323 conn(0x55a314e15000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1865 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:49.778435) v4 > -2578> 2018-07-11 07:34:49.779381 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6802/13323 1865 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:49.778435) v4 ==== 2004+0+0 (1907713210 0 0) 0x55a324474c00 con 0x55a314e15000 > -2577> 2018-07-11 07:34:49.779403 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6803/15919 conn(0x55a314e88800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1865 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:49.778435) v4 > -2576> 2018-07-11 07:34:49.779415 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6802/15919 conn(0x55a314e87000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1865 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:49.778435) v4 > -2575> 2018-07-11 07:34:49.779416 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6803/15919 1865 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:49.778435) v4 ==== 2004+0+0 (1907713210 0 0) 0x55a34de78600 con 0x55a314e88800 > -2574> 2018-07-11 07:34:49.779430 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6802/15919 1865 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:49.778435) v4 ==== 2004+0+0 (1907713210 0 0) 0x55a324474c00 con 0x55a314e87000 > -2573> 2018-07-11 07:34:49.779453 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6802/11967 conn(0x55a314e12000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1865 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:49.778435) v4 > -2572> 2018-07-11 07:34:49.779465 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6802/11967 1865 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:49.778435) v4 ==== 2004+0+0 (1907713210 0 0) 0x55a34de78600 con 0x55a314e12000 > -2571> 2018-07-11 07:34:50.135282 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15287 conn(0x55a311b72000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1854 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:34:51.349823) v4 > -2570> 2018-07-11 07:34:50.135284 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15287 conn(0x55a311b75000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1854 0x55a34de6fc00 osd_ping(ping e494 stamp 2018-07-11 07:34:51.349823) v4 > -2569> 2018-07-11 07:34:50.135311 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.4 10.8.128.113:0/15287 1854 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:51.349823) v4 ==== 2004+0+0 (3758273024 0 0) 0x55a324474c00 con 0x55a311b72000 > -2568> 2018-07-11 07:34:50.135313 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.4 10.8.128.113:0/15287 1854 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:51.349823) v4 ==== 2004+0+0 (3758273024 0 0) 0x55a34de6fc00 con 0x55a311b75000 > -2567> 2018-07-11 07:34:50.135331 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:51.349823) v4 -- 0x55a323f54800 con 0 > -2566> 2018-07-11 07:34:50.135360 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:51.349823) v4 -- 0x55a33234b400 con 0 > -2565> 2018-07-11 07:34:50.357401 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b7000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1807 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:34:50.108316) v4 > -2564> 2018-07-11 07:34:50.357429 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.8 10.8.128.108:0/11967 1807 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:50.108316) v4 ==== 2004+0+0 (370429413 0 0) 0x55a34de78600 con 0x55a3170b7000 > -2563> 2018-07-11 07:34:50.357441 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:50.108316) v4 -- 0x55a34de6f600 con 0 > -2562> 2018-07-11 07:34:50.357456 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b8800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1807 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:34:50.108316) v4 > -2561> 2018-07-11 07:34:50.357485 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.8 10.8.128.108:0/11967 1807 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:50.108316) v4 ==== 2004+0+0 (370429413 0 0) 0x55a332e79a00 con 0x55a3170b8800 > -2560> 2018-07-11 07:34:50.357508 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:50.108316) v4 -- 0x55a323f54800 con 0 > -2559> 2018-07-11 07:34:50.435053 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12677 conn(0x55a32169d800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1878 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:34:50.471302) v4 > -2558> 2018-07-11 07:34:50.435079 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.1 10.8.128.106:0/12677 1878 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:50.471302) v4 ==== 2004+0+0 (1083625500 0 0) 0x55a34de6f000 con 0x55a32169d800 > -2557> 2018-07-11 07:34:50.435095 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:50.471302) v4 -- 0x55a34de6f600 con 0 > -2556> 2018-07-11 07:34:50.435121 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12677 conn(0x55a32169c000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1878 0x55a33b42d400 osd_ping(ping e494 stamp 2018-07-11 07:34:50.471302) v4 > -2555> 2018-07-11 07:34:50.435142 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.1 10.8.128.106:0/12677 1878 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:50.471302) v4 ==== 2004+0+0 (1083625500 0 0) 0x55a33b42d400 con 0x55a32169c000 > -2554> 2018-07-11 07:34:50.435153 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:50.471302) v4 -- 0x55a33234b400 con 0 > -2553> 2018-07-11 07:34:50.483164 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12650 conn(0x55a311b67000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1872 0x55a34de6fc00 osd_ping(ping e494 stamp 2018-07-11 07:34:50.519441) v4 > -2552> 2018-07-11 07:34:50.483164 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12650 conn(0x55a311b68800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1872 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:34:50.519441) v4 > -2551> 2018-07-11 07:34:50.483192 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.3 10.8.128.106:0/12650 1872 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:50.519441) v4 ==== 2004+0+0 (3161028632 0 0) 0x55a34de6fc00 con 0x55a311b67000 > -2550> 2018-07-11 07:34:50.483195 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.3 10.8.128.106:0/12650 1872 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:50.519441) v4 ==== 2004+0+0 (3161028632 0 0) 0x55a34de78600 con 0x55a311b68800 > -2549> 2018-07-11 07:34:50.483211 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:50.519441) v4 -- 0x55a33234b400 con 0 > -2548> 2018-07-11 07:34:50.483239 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:50.519441) v4 -- 0x55a34de6f600 con 0 > -2547> 2018-07-11 07:34:50.673823 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15885 conn(0x55a3170b5800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1880 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:34:51.297860) v4 > -2546> 2018-07-11 07:34:50.673826 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15885 conn(0x55a311b76800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1880 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:34:51.297860) v4 > -2545> 2018-07-11 07:34:50.673848 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.7 10.8.128.114:0/15885 1880 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:51.297860) v4 ==== 2004+0+0 (3781040401 0 0) 0x55a324474c00 con 0x55a3170b5800 > -2544> 2018-07-11 07:34:50.673853 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.7 10.8.128.114:0/15885 1880 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:51.297860) v4 ==== 2004+0+0 (3781040401 0 0) 0x55a34de6f000 con 0x55a311b76800 > -2543> 2018-07-11 07:34:50.673857 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:51.297860) v4 -- 0x55a323f54800 con 0 > -2542> 2018-07-11 07:34:50.673890 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:51.297860) v4 -- 0x55a34de6f600 con 0 > -2541> 2018-07-11 07:34:51.414610 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 60] Generated table #36590: 4957168 keys, 80348311 bytes > -2540> 2018-07-11 07:34:51.414664 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294491414636, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 36590, "file_size": 80348311, "table_properties": {"data_size": 67109355, "index_size": 689256, "filter_size": 12548710, "raw_key_size": 143207073, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957168, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957168", "kMergeOperands": "0"}} > -2539> 2018-07-11 07:34:51.434474 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15255 conn(0x55a311b64000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1883 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:34:52.649049) v4 > -2538> 2018-07-11 07:34:51.434503 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.0 10.8.128.113:0/15255 1883 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:52.649049) v4 ==== 2004+0+0 (3644588093 0 0) 0x55a34de78600 con 0x55a311b64000 > -2537> 2018-07-11 07:34:51.434515 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:52.649049) v4 -- 0x55a34de6f600 con 0 > -2536> 2018-07-11 07:34:51.434559 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15255 conn(0x55a311b65800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1883 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:34:52.649049) v4 > -2535> 2018-07-11 07:34:51.434584 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.0 10.8.128.113:0/15255 1883 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:52.649049) v4 ==== 2004+0+0 (3644588093 0 0) 0x55a332e79a00 con 0x55a311b65800 > -2534> 2018-07-11 07:34:51.434602 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:52.649049) v4 -- 0x55a323f54800 con 0 > -2533> 2018-07-11 07:34:51.594042 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a3340fbb00 con 0 > -2532> 2018-07-11 07:34:51.622440 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5171 0x55a356209680 replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:494/492,start:6:aa18bf57:::benchmark_data_magna108_35617_object485584:0,end:6:aa198ff8:::benchmark_data_magna108_35617_object703308:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -2531> 2018-07-11 07:34:51.622471 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5171 ==== replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:494/492,start:6:aa18bf57:::benchmark_data_magna108_35617_object485584:0,end:6:aa198ff8:::benchmark_data_magna108_35617_object703308:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (3272323049 0 0) 0x55a356209680 con 0x55a317019000 > -2530> 2018-07-11 07:34:51.724442 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15263 conn(0x55a321664000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1855 0x55a33b42d400 osd_ping(ping e494 stamp 2018-07-11 07:34:52.939107) v4 > -2529> 2018-07-11 07:34:51.724469 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.5 10.8.128.113:0/15263 1855 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:52.939107) v4 ==== 2004+0+0 (184958042 0 0) 0x55a33b42d400 con 0x55a321664000 > -2528> 2018-07-11 07:34:51.724486 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:52.939107) v4 -- 0x55a33234b400 con 0 > -2527> 2018-07-11 07:34:51.724571 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15263 conn(0x55a321667000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1855 0x55a34de6fc00 osd_ping(ping e494 stamp 2018-07-11 07:34:52.939107) v4 > -2526> 2018-07-11 07:34:51.724587 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.5 10.8.128.113:0/15263 1855 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:52.939107) v4 ==== 2004+0+0 (184958042 0 0) 0x55a34de6fc00 con 0x55a321667000 > -2525> 2018-07-11 07:34:51.724598 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:52.939107) v4 -- 0x55a323f54400 con 0 > -2524> 2018-07-11 07:34:51.934881 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15255 conn(0x55a311b64000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1884 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:34:53.149501) v4 > -2523> 2018-07-11 07:34:51.934902 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.0 10.8.128.113:0/15255 1884 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:53.149501) v4 ==== 2004+0+0 (2069700352 0 0) 0x55a34de6f000 con 0x55a311b64000 > -2522> 2018-07-11 07:34:51.934916 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:53.149501) v4 -- 0x55a332e76400 con 0 > -2521> 2018-07-11 07:34:51.934925 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15255 conn(0x55a311b65800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1884 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:34:53.149501) v4 > -2520> 2018-07-11 07:34:51.934943 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.0 10.8.128.113:0/15255 1884 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:53.149501) v4 ==== 2004+0+0 (2069700352 0 0) 0x55a324474c00 con 0x55a311b65800 > -2519> 2018-07-11 07:34:51.934960 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:53.149501) v4 -- 0x55a32f761e00 con 0 > -2518> 2018-07-11 07:34:52.057917 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b8800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1808 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:34:51.808741) v4 > -2517> 2018-07-11 07:34:52.057928 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b7000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1808 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:34:51.808741) v4 > -2516> 2018-07-11 07:34:52.057945 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.8 10.8.128.108:0/11967 1808 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:51.808741) v4 ==== 2004+0+0 (1240545930 0 0) 0x55a34de78600 con 0x55a3170b7000 > -2515> 2018-07-11 07:34:52.057943 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.8 10.8.128.108:0/11967 1808 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:51.808741) v4 ==== 2004+0+0 (1240545930 0 0) 0x55a332e79a00 con 0x55a3170b8800 > -2514> 2018-07-11 07:34:52.057954 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:51.808741) v4 -- 0x55a32442b400 con 0 > -2513> 2018-07-11 07:34:52.057978 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:51.808741) v4 -- 0x55a32f760a00 con 0 > -2512> 2018-07-11 07:34:52.256821 7f35488a3700 1 -- 10.8.128.107:6800/12665 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55a30f75edc0 con 0 > -2511> 2018-07-11 07:34:52.256900 7f35488a3700 1 -- 10.8.128.107:6800/12665 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55a330932580 con 0 > -2510> 2018-07-11 07:34:52.355971 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.115:0/13323 conn(0x55a321665800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1867 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:34:52.693830) v4 > -2509> 2018-07-11 07:34:52.355993 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.6 10.8.128.115:0/13323 1867 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:52.693830) v4 ==== 2004+0+0 (3801226142 0 0) 0x55a324474c00 con 0x55a321665800 > -2508> 2018-07-11 07:34:52.356002 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:52.693830) v4 -- 0x55a32f761200 con 0 > -2507> 2018-07-11 07:34:52.356071 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.115:0/13323 conn(0x55a321668800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1867 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:34:52.693830) v4 > -2506> 2018-07-11 07:34:52.356087 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.6 10.8.128.115:0/13323 1867 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:52.693830) v4 ==== 2004+0+0 (3801226142 0 0) 0x55a332e79a00 con 0x55a321668800 > -2505> 2018-07-11 07:34:52.356097 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:52.693830) v4 -- 0x55a3177a8200 con 0 > -2504> 2018-07-11 07:34:52.381456 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a32fb98fc0 con 0 > -2503> 2018-07-11 07:34:52.402182 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5172 0x55a356209900 replica scrub(pg: 6.15,from:0'0,to:430'106212,epoch:494/492,start:6:aa198ff8:::benchmark_data_magna108_35617_object703308:0,end:6:aa1a467e:::benchmark_data_magna108_35617_object487860:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -2502> 2018-07-11 07:34:52.402212 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5172 ==== replica scrub(pg: 6.15,from:0'0,to:430'106212,epoch:494/492,start:6:aa198ff8:::benchmark_data_magna108_35617_object703308:0,end:6:aa1a467e:::benchmark_data_magna108_35617_object487860:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (3381947473 0 0) 0x55a356209900 con 0x55a317019000 > -2501> 2018-07-11 07:34:52.435286 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15255 conn(0x55a311b64000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1885 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:34:53.649932) v4 > -2500> 2018-07-11 07:34:52.435302 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15255 conn(0x55a311b65800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1885 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:34:53.649932) v4 > -2499> 2018-07-11 07:34:52.435315 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.0 10.8.128.113:0/15255 1885 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:53.649932) v4 ==== 2004+0+0 (2980503586 0 0) 0x55a34de6f000 con 0x55a311b64000 > -2498> 2018-07-11 07:34:52.435320 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.0 10.8.128.113:0/15255 1885 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:53.649932) v4 ==== 2004+0+0 (2980503586 0 0) 0x55a324474c00 con 0x55a311b65800 > -2497> 2018-07-11 07:34:52.435325 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:53.649932) v4 -- 0x55a32442b400 con 0 > -2496> 2018-07-11 07:34:52.435349 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:53.649932) v4 -- 0x55a3177a8200 con 0 > -2495> 2018-07-11 07:34:52.558518 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b8800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1809 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:34:52.309391) v4 > -2494> 2018-07-11 07:34:52.558541 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.8 10.8.128.108:0/11967 1809 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:52.309391) v4 ==== 2004+0+0 (245076681 0 0) 0x55a332e79a00 con 0x55a3170b8800 > -2493> 2018-07-11 07:34:52.558557 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:52.309391) v4 -- 0x55a3177a8200 con 0 > -2492> 2018-07-11 07:34:52.558563 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b7000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1809 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:34:52.309391) v4 > -2491> 2018-07-11 07:34:52.558584 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.8 10.8.128.108:0/11967 1809 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:52.309391) v4 ==== 2004+0+0 (245076681 0 0) 0x55a34de78600 con 0x55a3170b7000 > -2490> 2018-07-11 07:34:52.558595 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:52.309391) v4 -- 0x55a30fa76400 con 0 > -2489> 2018-07-11 07:34:52.670109 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12651 conn(0x55a32169f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1856 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:34:52.706354) v4 > -2488> 2018-07-11 07:34:52.670131 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12651 conn(0x55a3216a0800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1856 0x55a33b42d400 osd_ping(ping e494 stamp 2018-07-11 07:34:52.706354) v4 > -2487> 2018-07-11 07:34:52.670137 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.2 10.8.128.106:0/12651 1856 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:52.706354) v4 ==== 2004+0+0 (936649767 0 0) 0x55a324474c00 con 0x55a32169f000 > -2486> 2018-07-11 07:34:52.670157 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.2 10.8.128.106:0/12651 1856 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:52.706354) v4 ==== 2004+0+0 (936649767 0 0) 0x55a33b42d400 con 0x55a3216a0800 > -2485> 2018-07-11 07:34:52.670164 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:52.706354) v4 -- 0x55a32f760800 con 0 > -2484> 2018-07-11 07:34:52.670194 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:52.706354) v4 -- 0x55a33532c800 con 0 > -2483> 2018-07-11 07:34:52.856615 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.115:0/13323 conn(0x55a321665800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1868 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:34:53.194462) v4 > -2482> 2018-07-11 07:34:52.856640 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.6 10.8.128.115:0/13323 1868 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:53.194462) v4 ==== 2004+0+0 (2650659606 0 0) 0x55a332e79a00 con 0x55a321665800 > -2481> 2018-07-11 07:34:52.856654 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:53.194462) v4 -- 0x55a3177a8800 con 0 > -2480> 2018-07-11 07:34:52.856705 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.115:0/13323 conn(0x55a321668800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1868 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:34:53.194462) v4 > -2479> 2018-07-11 07:34:52.856724 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.6 10.8.128.115:0/13323 1868 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:53.194462) v4 ==== 2004+0+0 (2650659606 0 0) 0x55a332e79a00 con 0x55a321668800 > -2478> 2018-07-11 07:34:52.856735 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:53.194462) v4 -- 0x55a32f761000 con 0 > -2477> 2018-07-11 07:34:53.333537 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 60] Generated table #36591: 1042836 keys, 16903262 bytes > -2476> 2018-07-11 07:34:53.333568 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294493333554, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 36591, "file_size": 16903262, "table_properties": {"data_size": 14117846, "index_size": 144565, "filter_size": 2639864, "raw_key_size": 30126382, "raw_average_key_size": 28, "raw_value_size": 47, "raw_average_value_size": 0, "num_data_blocks": 3465, "num_entries": 1042836, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1042832", "kMergeOperands": "0"}} > -2475> 2018-07-11 07:34:53.376872 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a317a5c240 con 0 > -2474> 2018-07-11 07:34:53.389276 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5173 0x55a356209b80 replica scrub(pg: 6.15,from:0'0,to:430'105794,epoch:494/492,start:6:aa1a467e:::benchmark_data_magna108_35617_object487860:0,end:6:aa1bab3b:::benchmark_data_magna108.ceph.redhat.com_2478_object128604:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -2473> 2018-07-11 07:34:53.389304 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5173 ==== replica scrub(pg: 6.15,from:0'0,to:430'105794,epoch:494/492,start:6:aa1a467e:::benchmark_data_magna108_35617_object487860:0,end:6:aa1bab3b:::benchmark_data_magna108.ceph.redhat.com_2478_object128604:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (1415606906 0 0) 0x55a356209b80 con 0x55a317019000 > -2472> 2018-07-11 07:34:53.424912 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15263 conn(0x55a321664000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1856 0x55a34de6fc00 osd_ping(ping e494 stamp 2018-07-11 07:34:54.639498) v4 > -2471> 2018-07-11 07:34:53.424936 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.5 10.8.128.113:0/15263 1856 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:54.639498) v4 ==== 2004+0+0 (1029692736 0 0) 0x55a34de6fc00 con 0x55a321664000 > -2470> 2018-07-11 07:34:53.424953 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:54.639498) v4 -- 0x55a332354e00 con 0 > -2469> 2018-07-11 07:34:53.425032 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15263 conn(0x55a321667000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1856 0x55a33532c800 osd_ping(ping e494 stamp 2018-07-11 07:34:54.639498) v4 > -2468> 2018-07-11 07:34:53.425051 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.5 10.8.128.113:0/15263 1856 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:54.639498) v4 ==== 2004+0+0 (1029692736 0 0) 0x55a33532c800 con 0x55a321667000 > -2467> 2018-07-11 07:34:53.425066 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:54.639498) v4 -- 0x55a332354e00 con 0 > -2466> 2018-07-11 07:34:53.686913 7f355d88e700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.107:0/12617 conn(0x55a32447f800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1831 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:34:53.686451) v4 > -2465> 2018-07-11 07:34:53.686914 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.107:0/12617 conn(0x55a32447e000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1831 0x55a34de6fc00 osd_ping(ping e494 stamp 2018-07-11 07:34:53.686451) v4 > -2464> 2018-07-11 07:34:53.686937 7f355d88e700 1 -- 10.8.128.107:6803/12665 <== osd.9 10.8.128.107:0/12617 1831 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:53.686451) v4 ==== 2004+0+0 (150548886 0 0) 0x55a34de6f000 con 0x55a32447f800 > -2463> 2018-07-11 07:34:53.686938 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.9 10.8.128.107:0/12617 1831 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:53.686451) v4 ==== 2004+0+0 (150548886 0 0) 0x55a34de6fc00 con 0x55a32447e000 > -2462> 2018-07-11 07:34:53.686949 7f355d88e700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:53.686451) v4 -- 0x55a30fa76400 con 0 > -2461> 2018-07-11 07:34:53.686975 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:53.686451) v4 -- 0x55a332354e00 con 0 > -2460> 2018-07-11 07:34:53.935457 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12677 conn(0x55a32169d800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1879 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:34:53.971795) v4 > -2459> 2018-07-11 07:34:53.935483 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.1 10.8.128.106:0/12677 1879 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:53.971795) v4 ==== 2004+0+0 (345649059 0 0) 0x55a34de78600 con 0x55a32169d800 > -2458> 2018-07-11 07:34:53.935497 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:53.971795) v4 -- 0x55a30fa76400 con 0 > -2457> 2018-07-11 07:34:53.935510 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12677 conn(0x55a32169c000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1879 0x55a33532c800 osd_ping(ping e494 stamp 2018-07-11 07:34:53.971795) v4 > -2456> 2018-07-11 07:34:53.935531 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.1 10.8.128.106:0/12677 1879 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:53.971795) v4 ==== 2004+0+0 (345649059 0 0) 0x55a33532c800 con 0x55a32169c000 > -2455> 2018-07-11 07:34:53.935542 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:53.971795) v4 -- 0x55a30f329a00 con 0 > -2454> 2018-07-11 07:34:53.957108 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.115:0/13323 conn(0x55a321668800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1869 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:34:54.294928) v4 > -2453> 2018-07-11 07:34:53.957134 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.6 10.8.128.115:0/13323 1869 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:54.294928) v4 ==== 2004+0+0 (2232592589 0 0) 0x55a324474c00 con 0x55a321668800 > -2452> 2018-07-11 07:34:53.957152 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:54.294928) v4 -- 0x55a3177a8800 con 0 > -2451> 2018-07-11 07:34:53.957184 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.115:0/13323 conn(0x55a321665800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1869 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:34:54.294928) v4 > -2450> 2018-07-11 07:34:53.957196 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.6 10.8.128.115:0/13323 1869 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:54.294928) v4 ==== 2004+0+0 (2232592589 0 0) 0x55a324474c00 con 0x55a321665800 > -2449> 2018-07-11 07:34:53.957206 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:54.294928) v4 -- 0x55a34dec6c00 con 0 > -2448> 2018-07-11 07:34:53.983577 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12650 conn(0x55a311b67000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1873 0x55a34de6fc00 osd_ping(ping e494 stamp 2018-07-11 07:34:54.019909) v4 > -2447> 2018-07-11 07:34:53.983604 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.3 10.8.128.106:0/12650 1873 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:54.019909) v4 ==== 2004+0+0 (3394276144 0 0) 0x55a34de6fc00 con 0x55a311b67000 > -2446> 2018-07-11 07:34:53.983623 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:54.019909) v4 -- 0x55a332ead400 con 0 > -2445> 2018-07-11 07:34:53.983629 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12650 conn(0x55a311b68800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1873 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:34:54.019909) v4 > -2444> 2018-07-11 07:34:53.983647 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.3 10.8.128.106:0/12650 1873 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:54.019909) v4 ==== 2004+0+0 (3394276144 0 0) 0x55a34de6f000 con 0x55a311b68800 > -2443> 2018-07-11 07:34:53.983663 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:54.019909) v4 -- 0x55a348fbca00 con 0 > -2442> 2018-07-11 07:34:54.174497 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a32fbcc6c0 con 0 > -2441> 2018-07-11 07:34:54.189388 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5174 0x55a3202a0000 replica scrub(pg: 6.15,from:0'0,to:430'106009,epoch:494/492,start:6:aa1bab3b:::benchmark_data_magna108.ceph.redhat.com_2478_object128604:0,end:6:aa1c679e:::benchmark_data_magna108_35617_object907390:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -2440> 2018-07-11 07:34:54.189411 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5174 ==== replica scrub(pg: 6.15,from:0'0,to:430'106009,epoch:494/492,start:6:aa1bab3b:::benchmark_data_magna108.ceph.redhat.com_2478_object128604:0,end:6:aa1c679e:::benchmark_data_magna108_35617_object907390:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (227934682 0 0) 0x55a3202a0000 con 0x55a317019000 > -2439> 2018-07-11 07:34:54.235611 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15287 conn(0x55a311b75000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1855 0x55a33532c800 osd_ping(ping e494 stamp 2018-07-11 07:34:55.450305) v4 > -2438> 2018-07-11 07:34:54.235638 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.4 10.8.128.113:0/15287 1855 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:55.450305) v4 ==== 2004+0+0 (2773192206 0 0) 0x55a33532c800 con 0x55a311b75000 > -2437> 2018-07-11 07:34:54.235650 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:55.450305) v4 -- 0x55a33b42c800 con 0 > -2436> 2018-07-11 07:34:54.235643 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15287 conn(0x55a311b72000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1855 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:34:55.450305) v4 > -2435> 2018-07-11 07:34:54.235660 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.4 10.8.128.113:0/15287 1855 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:55.450305) v4 ==== 2004+0+0 (2773192206 0 0) 0x55a332e79a00 con 0x55a311b72000 > -2434> 2018-07-11 07:34:54.235673 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:55.450305) v4 -- 0x55a34dec6c00 con 0 > -2433> 2018-07-11 07:34:54.436019 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12677 conn(0x55a32169c000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1880 0x55a332ead400 osd_ping(ping e494 stamp 2018-07-11 07:34:54.472311) v4 > -2432> 2018-07-11 07:34:54.436022 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12677 conn(0x55a32169d800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1880 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:34:54.472311) v4 > -2431> 2018-07-11 07:34:54.436041 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.1 10.8.128.106:0/12677 1880 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:54.472311) v4 ==== 2004+0+0 (1216330268 0 0) 0x55a332ead400 con 0x55a32169c000 > -2430> 2018-07-11 07:34:54.436047 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.1 10.8.128.106:0/12677 1880 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:54.472311) v4 ==== 2004+0+0 (1216330268 0 0) 0x55a34de78600 con 0x55a32169d800 > -2429> 2018-07-11 07:34:54.436053 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:54.472311) v4 -- 0x55a33b42c800 con 0 > -2428> 2018-07-11 07:34:54.436074 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:54.472311) v4 -- 0x55a348fbca00 con 0 > -2427> 2018-07-11 07:34:54.970475 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12651 conn(0x55a3216a0800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1857 0x55a33532c800 osd_ping(ping e494 stamp 2018-07-11 07:34:55.006786) v4 > -2426> 2018-07-11 07:34:54.970505 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.2 10.8.128.106:0/12651 1857 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:55.006786) v4 ==== 2004+0+0 (3981846149 0 0) 0x55a33532c800 con 0x55a3216a0800 > -2425> 2018-07-11 07:34:54.970494 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12651 conn(0x55a32169f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1857 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:34:55.006786) v4 > -2424> 2018-07-11 07:34:54.970524 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:55.006786) v4 -- 0x55a33b42c800 con 0 > -2423> 2018-07-11 07:34:54.970522 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.2 10.8.128.106:0/12651 1857 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:55.006786) v4 ==== 2004+0+0 (3981846149 0 0) 0x55a324474c00 con 0x55a32169f000 > -2422> 2018-07-11 07:34:54.970554 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:55.006786) v4 -- 0x55a34dec6c00 con 0 > -2421> 2018-07-11 07:34:55.046993 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a3259aa900 con 0 > -2420> 2018-07-11 07:34:55.069050 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5175 0x55a3202a0280 replica scrub(pg: 6.15,from:0'0,to:430'106759,epoch:494/492,start:6:aa1c679e:::benchmark_data_magna108_35617_object907390:0,end:6:aa1d8e01:::benchmark_data_magna108_35617_object696035:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -2419> 2018-07-11 07:34:55.069080 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5175 ==== replica scrub(pg: 6.15,from:0'0,to:430'106759,epoch:494/492,start:6:aa1c679e:::benchmark_data_magna108_35617_object907390:0,end:6:aa1d8e01:::benchmark_data_magna108_35617_object696035:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (2733752561 0 0) 0x55a3202a0280 con 0x55a317019000 > -2418> 2018-07-11 07:34:55.078901 7f3543899700 5 osd.10 494 heartbeat: osd_stat(784 GB used, 146 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,1]) > -2417> 2018-07-11 07:34:55.078917 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6802/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:34:55.078915) v4 -- 0x55a3177a8000 con 0 > -2416> 2018-07-11 07:34:55.078954 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6803/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:34:55.078915) v4 -- 0x55a33532c000 con 0 > -2415> 2018-07-11 07:34:55.078971 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6806/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:34:55.078915) v4 -- 0x55a332fe8e00 con 0 > -2414> 2018-07-11 07:34:55.078987 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6807/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:34:55.078915) v4 -- 0x55a332fe9e00 con 0 > -2413> 2018-07-11 07:34:55.079005 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6808/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:34:55.078915) v4 -- 0x55a32ee77c00 con 0 > -2412> 2018-07-11 07:34:55.079028 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6810/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:34:55.078915) v4 -- 0x55a332355c00 con 0 > -2411> 2018-07-11 07:34:55.079054 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6809/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:34:55.078915) v4 -- 0x55a332ffbe00 con 0 > -2410> 2018-07-11 07:34:55.079075 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6811/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:34:55.078915) v4 -- 0x55a332348c00 con 0 > -2409> 2018-07-11 07:34:55.079102 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6810/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:34:55.078915) v4 -- 0x55a324474000 con 0 > -2408> 2018-07-11 07:34:55.079124 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6811/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:34:55.078915) v4 -- 0x55a324474a00 con 0 > -2407> 2018-07-11 07:34:55.079143 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6806/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:34:55.078915) v4 -- 0x55a34de84400 con 0 > -2406> 2018-07-11 07:34:55.079158 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6807/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:34:55.078915) v4 -- 0x55a332eac200 con 0 > -2405> 2018-07-11 07:34:55.079178 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6802/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:34:55.078915) v4 -- 0x55a332ffaa00 con 0 > -2404> 2018-07-11 07:34:55.079202 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6803/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:34:55.078915) v4 -- 0x55a310611200 con 0 > -2403> 2018-07-11 07:34:55.079219 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6806/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:34:55.078915) v4 -- 0x55a32e456400 con 0 > -2402> 2018-07-11 07:34:55.079240 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6807/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:34:55.078915) v4 -- 0x55a33235bc00 con 0 > -2401> 2018-07-11 07:34:55.079284 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6802/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:34:55.078915) v4 -- 0x55a33235b600 con 0 > -2400> 2018-07-11 07:34:55.079308 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6803/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:34:55.078915) v4 -- 0x55a32e456600 con 0 > -2399> 2018-07-11 07:34:55.079326 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6806/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:34:55.078915) v4 -- 0x55a330935a00 con 0 > -2398> 2018-07-11 07:34:55.079350 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6807/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:34:55.078915) v4 -- 0x55a332349c00 con 0 > -2397> 2018-07-11 07:34:55.079347 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6807/12677 conn(0x55a314e85800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1866 0x55a332ead400 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:55.078915) v4 > -2396> 2018-07-11 07:34:55.079378 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6807/12677 1866 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:55.078915) v4 ==== 2004+0+0 (3104718 0 0) 0x55a332ead400 con 0x55a314e85800 > -2395> 2018-07-11 07:34:55.079379 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6802/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:34:55.078915) v4 -- 0x55a332ffac00 con 0 > -2394> 2018-07-11 07:34:55.079387 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6806/12677 conn(0x55a314e84000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1866 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:55.078915) v4 > -2393> 2018-07-11 07:34:55.079402 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6806/12677 1866 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:55.078915) v4 ==== 2004+0+0 (3104718 0 0) 0x55a34de6f000 con 0x55a314e84000 > -2392> 2018-07-11 07:34:55.079408 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6803/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:34:55.078915) v4 -- 0x55a3314cf000 con 0 > -2391> 2018-07-11 07:34:55.079417 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6803/15255 conn(0x55a314e1d800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1866 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:55.078915) v4 > -2390> 2018-07-11 07:34:55.079453 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6803/15255 1866 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:55.078915) v4 ==== 2004+0+0 (3104718 0 0) 0x55a332e79a00 con 0x55a314e1d800 > -2389> 2018-07-11 07:34:55.079505 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6810/12651 conn(0x55a314e97800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1866 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:55.078915) v4 > -2388> 2018-07-11 07:34:55.079519 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6810/12651 1866 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:55.078915) v4 ==== 2004+0+0 (3104718 0 0) 0x55a324474c00 con 0x55a314e97800 > -2387> 2018-07-11 07:34:55.079517 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6802/15255 conn(0x55a314e1c000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1866 0x55a33b42c800 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:55.078915) v4 > -2386> 2018-07-11 07:34:55.079539 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6802/15255 1866 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:55.078915) v4 ==== 2004+0+0 (3104718 0 0) 0x55a33b42c800 con 0x55a314e1c000 > -2385> 2018-07-11 07:34:55.079519 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6806/15263 conn(0x55a314e99000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1866 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:55.078915) v4 > -2384> 2018-07-11 07:34:55.079554 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6811/15287 conn(0x55a314e04800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1866 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:55.078915) v4 > -2383> 2018-07-11 07:34:55.079565 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6811/15287 1866 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:55.078915) v4 ==== 2004+0+0 (3104718 0 0) 0x55a324474c00 con 0x55a314e04800 > -2382> 2018-07-11 07:34:55.079566 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6806/15263 1866 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:55.078915) v4 ==== 2004+0+0 (3104718 0 0) 0x55a34de78600 con 0x55a314e99000 > -2381> 2018-07-11 07:34:55.079590 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6806/12617 conn(0x55a314e26000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1866 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:55.078915) v4 > -2380> 2018-07-11 07:34:55.079600 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6806/12617 1866 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:55.078915) v4 ==== 2004+0+0 (3104718 0 0) 0x55a324474c00 con 0x55a314e26000 > -2379> 2018-07-11 07:34:55.079585 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6808/12651 conn(0x55a314e96000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1866 0x55a33b42c800 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:55.078915) v4 > -2378> 2018-07-11 07:34:55.079618 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6809/12650 conn(0x55a314e1f000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1866 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:55.078915) v4 > -2377> 2018-07-11 07:34:55.079630 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6809/12650 1866 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:55.078915) v4 ==== 2004+0+0 (3104718 0 0) 0x55a34de78600 con 0x55a314e1f000 > -2376> 2018-07-11 07:34:55.079631 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6802/13323 conn(0x55a314e15000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1866 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:55.078915) v4 > -2375> 2018-07-11 07:34:55.079629 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6808/12651 1866 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:55.078915) v4 ==== 2004+0+0 (3104718 0 0) 0x55a33b42c800 con 0x55a314e96000 > -2374> 2018-07-11 07:34:55.079640 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6802/13323 1866 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:55.078915) v4 ==== 2004+0+0 (3104718 0 0) 0x55a324474c00 con 0x55a314e15000 > -2373> 2018-07-11 07:34:55.079673 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6807/12617 conn(0x55a314e27800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1866 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:55.078915) v4 > -2372> 2018-07-11 07:34:55.079686 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6807/12617 1866 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:55.078915) v4 ==== 2004+0+0 (3104718 0 0) 0x55a34de78600 con 0x55a314e27800 > -2371> 2018-07-11 07:34:55.079697 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6806/15885 conn(0x55a314e00000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1866 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:55.078915) v4 > -2370> 2018-07-11 07:34:55.079695 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6810/15287 conn(0x55a314e03000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1866 0x55a33b42c800 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:55.078915) v4 > -2369> 2018-07-11 07:34:55.079715 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6803/13323 conn(0x55a314e16800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1866 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:55.078915) v4 > -2368> 2018-07-11 07:34:55.079719 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6810/15287 1866 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:55.078915) v4 ==== 2004+0+0 (3104718 0 0) 0x55a33b42c800 con 0x55a314e03000 > -2367> 2018-07-11 07:34:55.079711 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6806/15885 1866 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:55.078915) v4 ==== 2004+0+0 (3104718 0 0) 0x55a324474c00 con 0x55a314e00000 > -2366> 2018-07-11 07:34:55.079731 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6803/13323 1866 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:55.078915) v4 ==== 2004+0+0 (3104718 0 0) 0x55a34de78600 con 0x55a314e16800 > -2365> 2018-07-11 07:34:55.079765 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6807/15885 conn(0x55a314e01800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1866 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:55.078915) v4 > -2364> 2018-07-11 07:34:55.079767 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6811/12650 conn(0x55a314e20800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1866 0x55a33b42c800 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:55.078915) v4 > -2363> 2018-07-11 07:34:55.079776 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6807/15885 1866 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:55.078915) v4 ==== 2004+0+0 (3104718 0 0) 0x55a34de78600 con 0x55a314e01800 > -2362> 2018-07-11 07:34:55.079787 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6811/12650 1866 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:55.078915) v4 ==== 2004+0+0 (3104718 0 0) 0x55a33b42c800 con 0x55a314e20800 > -2361> 2018-07-11 07:34:55.079814 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6802/11967 conn(0x55a314e12000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1866 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:55.078915) v4 > -2360> 2018-07-11 07:34:55.079826 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6802/11967 1866 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:55.078915) v4 ==== 2004+0+0 (3104718 0 0) 0x55a34de78600 con 0x55a314e12000 > -2359> 2018-07-11 07:34:55.079835 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6807/15263 conn(0x55a314e9a800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1866 0x55a33b42c800 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:55.078915) v4 > -2358> 2018-07-11 07:34:55.079857 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6807/15263 1866 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:55.078915) v4 ==== 2004+0+0 (3104718 0 0) 0x55a33b42c800 con 0x55a314e9a800 > -2357> 2018-07-11 07:34:55.079862 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6802/15919 conn(0x55a314e87000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1866 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:55.078915) v4 > -2356> 2018-07-11 07:34:55.079883 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6802/15919 1866 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:55.078915) v4 ==== 2004+0+0 (3104718 0 0) 0x55a324474c00 con 0x55a314e87000 > -2355> 2018-07-11 07:34:55.079886 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6803/15919 conn(0x55a314e88800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1866 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:55.078915) v4 > -2354> 2018-07-11 07:34:55.079897 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6803/11967 conn(0x55a314e13800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1866 0x55a33b42c800 osd_ping(ping_reply e494 stamp 2018-07-11 07:34:55.078915) v4 > -2353> 2018-07-11 07:34:55.079902 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6803/15919 1866 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:55.078915) v4 ==== 2004+0+0 (3104718 0 0) 0x55a34de78600 con 0x55a314e88800 > -2352> 2018-07-11 07:34:55.079918 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6803/11967 1866 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:34:55.078915) v4 ==== 2004+0+0 (3104718 0 0) 0x55a33b42c800 con 0x55a314e13800 > -2351> 2018-07-11 07:34:55.335964 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15287 conn(0x55a311b72000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1856 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:34:56.550662) v4 > -2350> 2018-07-11 07:34:55.335986 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.4 10.8.128.113:0/15287 1856 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:56.550662) v4 ==== 2004+0+0 (2788868368 0 0) 0x55a324474c00 con 0x55a311b72000 > -2349> 2018-07-11 07:34:55.335996 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:56.550662) v4 -- 0x55a34dec6c00 con 0 > -2348> 2018-07-11 07:34:55.335987 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15287 conn(0x55a311b75000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1856 0x55a33b42c800 osd_ping(ping e494 stamp 2018-07-11 07:34:56.550662) v4 > -2347> 2018-07-11 07:34:55.336015 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.4 10.8.128.113:0/15287 1856 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:56.550662) v4 ==== 2004+0+0 (2788868368 0 0) 0x55a33b42c800 con 0x55a311b75000 > -2346> 2018-07-11 07:34:55.336028 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:56.550662) v4 -- 0x55a33b42c400 con 0 > -2345> 2018-07-11 07:34:55.374159 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15885 conn(0x55a311b76800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1881 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:34:55.998273) v4 > -2344> 2018-07-11 07:34:55.374180 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.7 10.8.128.114:0/15885 1881 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:55.998273) v4 ==== 2004+0+0 (3852100436 0 0) 0x55a34de78600 con 0x55a311b76800 > -2343> 2018-07-11 07:34:55.374173 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15885 conn(0x55a3170b5800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1881 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:34:55.998273) v4 > -2342> 2018-07-11 07:34:55.374192 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:55.998273) v4 -- 0x55a348fbca00 con 0 > -2341> 2018-07-11 07:34:55.374193 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.7 10.8.128.114:0/15885 1881 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:55.998273) v4 ==== 2004+0+0 (3852100436 0 0) 0x55a332e79a00 con 0x55a3170b5800 > -2340> 2018-07-11 07:34:55.374219 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:55.998273) v4 -- 0x55a34dec6c00 con 0 > -2339> 2018-07-11 07:34:55.536372 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12677 conn(0x55a32169d800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1881 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:34:55.572798) v4 > -2338> 2018-07-11 07:34:55.536393 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.1 10.8.128.106:0/12677 1881 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:55.572798) v4 ==== 2004+0+0 (348801231 0 0) 0x55a34de6f000 con 0x55a32169d800 > -2337> 2018-07-11 07:34:55.536407 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:55.572798) v4 -- 0x55a348fbca00 con 0 > -2336> 2018-07-11 07:34:55.536541 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12677 conn(0x55a32169c000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1881 0x55a332ead400 osd_ping(ping e494 stamp 2018-07-11 07:34:55.572798) v4 > -2335> 2018-07-11 07:34:55.536560 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.1 10.8.128.106:0/12677 1881 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:55.572798) v4 ==== 2004+0+0 (348801231 0 0) 0x55a332ead400 con 0x55a32169c000 > -2334> 2018-07-11 07:34:55.536575 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:55.572798) v4 -- 0x55a33b42c400 con 0 > -2333> 2018-07-11 07:34:55.661453 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15919 conn(0x55a3170b4000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1859 0x55a33b42c800 osd_ping(ping e494 stamp 2018-07-11 07:34:56.285533) v4 > -2332> 2018-07-11 07:34:55.661474 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.11 10.8.128.114:0/15919 1859 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:56.285533) v4 ==== 2004+0+0 (2864124189 0 0) 0x55a33b42c800 con 0x55a3170b4000 > -2331> 2018-07-11 07:34:55.661483 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:56.285533) v4 -- 0x55a33b42c400 con 0 > -2330> 2018-07-11 07:34:55.661482 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15919 conn(0x55a317074800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1859 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:34:56.285533) v4 > -2329> 2018-07-11 07:34:55.661501 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.11 10.8.128.114:0/15919 1859 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:56.285533) v4 ==== 2004+0+0 (2864124189 0 0) 0x55a324474c00 con 0x55a317074800 > -2328> 2018-07-11 07:34:55.661515 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:56.285533) v4 -- 0x55a34dec6c00 con 0 > -2327> 2018-07-11 07:34:55.684183 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12650 conn(0x55a311b67000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1874 0x55a332ead400 osd_ping(ping e494 stamp 2018-07-11 07:34:55.720416) v4 > -2326> 2018-07-11 07:34:55.684205 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.3 10.8.128.106:0/12650 1874 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:55.720416) v4 ==== 2004+0+0 (1637678521 0 0) 0x55a332ead400 con 0x55a311b67000 > -2325> 2018-07-11 07:34:55.684215 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:55.720416) v4 -- 0x55a33b42c400 con 0 > -2324> 2018-07-11 07:34:55.684314 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12650 conn(0x55a311b68800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1874 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:34:55.720416) v4 > -2323> 2018-07-11 07:34:55.684334 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.3 10.8.128.106:0/12650 1874 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:55.720416) v4 ==== 2004+0+0 (1637678521 0 0) 0x55a34de78600 con 0x55a311b68800 > -2322> 2018-07-11 07:34:55.684345 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:55.720416) v4 -- 0x55a348fbca00 con 0 > -2321> 2018-07-11 07:34:55.773810 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a33d7ce480 con 0 > -2320> 2018-07-11 07:34:55.789405 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5176 0x55a3202a0500 replica scrub(pg: 6.15,from:0'0,to:430'106739,epoch:494/492,start:6:aa1d8e01:::benchmark_data_magna108_35617_object696035:0,end:6:aa1ee9d9:::benchmark_data_magna108_35617_object390960:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -2319> 2018-07-11 07:34:55.789434 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5176 ==== replica scrub(pg: 6.15,from:0'0,to:430'106739,epoch:494/492,start:6:aa1d8e01:::benchmark_data_magna108_35617_object696035:0,end:6:aa1ee9d9:::benchmark_data_magna108_35617_object390960:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (4032862562 0 0) 0x55a3202a0500 con 0x55a317019000 > -2318> 2018-07-11 07:34:56.530290 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a3493b0480 con 0 > -2317> 2018-07-11 07:34:56.548157 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5177 0x55a3202a0780 replica scrub(pg: 6.15,from:0'0,to:430'105576,epoch:494/492,start:6:aa1ee9d9:::benchmark_data_magna108_35617_object390960:0,end:6:aa1f96e9:::benchmark_data_magna108_35617_object14205:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -2316> 2018-07-11 07:34:56.548185 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5177 ==== replica scrub(pg: 6.15,from:0'0,to:430'105576,epoch:494/492,start:6:aa1ee9d9:::benchmark_data_magna108_35617_object390960:0,end:6:aa1f96e9:::benchmark_data_magna108_35617_object14205:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 217+0+0 (3106939697 0 0) 0x55a3202a0780 con 0x55a317019000 > -2315> 2018-07-11 07:34:56.745208 7f35498a5700 10 monclient: tick > -2314> 2018-07-11 07:34:56.745221 7f35498a5700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2018-07-11 07:34:26.745220) > -2313> 2018-07-11 07:34:56.925477 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15263 conn(0x55a321664000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1857 0x55a332ead400 osd_ping(ping e494 stamp 2018-07-11 07:34:58.140106) v4 > -2312> 2018-07-11 07:34:56.925500 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.5 10.8.128.113:0/15263 1857 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:58.140106) v4 ==== 2004+0+0 (3441352488 0 0) 0x55a332ead400 con 0x55a321664000 > -2311> 2018-07-11 07:34:56.925516 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:58.140106) v4 -- 0x55a33b42c400 con 0 > -2310> 2018-07-11 07:34:56.925572 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15263 conn(0x55a321667000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1857 0x55a332ead400 osd_ping(ping e494 stamp 2018-07-11 07:34:58.140106) v4 > -2309> 2018-07-11 07:34:56.925589 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.5 10.8.128.113:0/15263 1857 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:58.140106) v4 ==== 2004+0+0 (3441352488 0 0) 0x55a332ead400 con 0x55a321667000 > -2308> 2018-07-11 07:34:56.925599 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:58.140106) v4 -- 0x55a332364400 con 0 > -2307> 2018-07-11 07:34:57.135655 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15255 conn(0x55a311b65800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1886 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:34:58.350359) v4 > -2306> 2018-07-11 07:34:57.135676 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.0 10.8.128.113:0/15255 1886 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:58.350359) v4 ==== 2004+0+0 (2745235588 0 0) 0x55a332e79a00 con 0x55a311b65800 > -2305> 2018-07-11 07:34:57.135685 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:58.350359) v4 -- 0x55a3177a8800 con 0 > -2304> 2018-07-11 07:34:57.135707 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15255 conn(0x55a311b64000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1886 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:34:58.350359) v4 > -2303> 2018-07-11 07:34:57.135729 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.0 10.8.128.113:0/15255 1886 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:58.350359) v4 ==== 2004+0+0 (2745235588 0 0) 0x55a34de6f000 con 0x55a311b64000 > -2302> 2018-07-11 07:34:57.135747 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:58.350359) v4 -- 0x55a3203f5e00 con 0 > -2301> 2018-07-11 07:34:57.240795 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a31fb4e900 con 0 > -2300> 2018-07-11 07:34:57.257083 7f35488a3700 1 -- 10.8.128.107:6800/12665 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55a324471080 con 0 > -2299> 2018-07-11 07:34:57.257145 7f35488a3700 1 -- 10.8.128.107:6800/12665 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55a330933b80 con 0 > -2298> 2018-07-11 07:34:57.269215 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5178 0x55a3202a0a00 replica scrub(pg: 6.15,from:0'0,to:430'106600,epoch:494/492,start:6:aa1f96e9:::benchmark_data_magna108_35617_object14205:0,end:6:aa201d2c:::benchmark_data_magna108_35617_object738376:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -2297> 2018-07-11 07:34:57.269238 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5178 ==== replica scrub(pg: 6.15,from:0'0,to:430'106600,epoch:494/492,start:6:aa1f96e9:::benchmark_data_magna108_35617_object14205:0,end:6:aa201d2c:::benchmark_data_magna108_35617_object738376:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 217+0+0 (1611509967 0 0) 0x55a3202a0a00 con 0x55a317019000 > -2296> 2018-07-11 07:34:57.271026 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12651 conn(0x55a32169f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1858 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:34:57.307330) v4 > -2295> 2018-07-11 07:34:57.271022 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12651 conn(0x55a3216a0800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1858 0x55a33b42c800 osd_ping(ping e494 stamp 2018-07-11 07:34:57.307330) v4 > -2294> 2018-07-11 07:34:57.271049 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.2 10.8.128.106:0/12651 1858 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:57.307330) v4 ==== 2004+0+0 (1131820052 0 0) 0x55a324474c00 con 0x55a32169f000 > -2293> 2018-07-11 07:34:57.271059 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:57.307330) v4 -- 0x55a3177a8800 con 0 > -2292> 2018-07-11 07:34:57.271054 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.2 10.8.128.106:0/12651 1858 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:57.307330) v4 ==== 2004+0+0 (1131820052 0 0) 0x55a33b42c800 con 0x55a3216a0800 > -2291> 2018-07-11 07:34:57.271082 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:57.307330) v4 -- 0x55a33b42c400 con 0 > -2290> 2018-07-11 07:34:57.361892 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15919 conn(0x55a317074800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1860 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:34:57.985976) v4 > -2289> 2018-07-11 07:34:57.361893 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15919 conn(0x55a3170b4000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1860 0x55a332364400 osd_ping(ping e494 stamp 2018-07-11 07:34:57.985976) v4 > -2288> 2018-07-11 07:34:57.361922 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.11 10.8.128.114:0/15919 1860 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:57.985976) v4 ==== 2004+0+0 (3695733742 0 0) 0x55a332e79a00 con 0x55a317074800 > -2287> 2018-07-11 07:34:57.361926 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.11 10.8.128.114:0/15919 1860 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:57.985976) v4 ==== 2004+0+0 (3695733742 0 0) 0x55a332364400 con 0x55a3170b4000 > -2286> 2018-07-11 07:34:57.361942 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:57.985976) v4 -- 0x55a3177a8800 con 0 > -2285> 2018-07-11 07:34:57.361970 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:57.985976) v4 -- 0x55a33b42c400 con 0 > -2284> 2018-07-11 07:34:57.636408 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15287 conn(0x55a311b72000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1857 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:34:58.851124) v4 > -2283> 2018-07-11 07:34:57.636429 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.4 10.8.128.113:0/15287 1857 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:58.851124) v4 ==== 2004+0+0 (3800833687 0 0) 0x55a324474c00 con 0x55a311b72000 > -2282> 2018-07-11 07:34:57.636439 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:58.851124) v4 -- 0x55a34de5a600 con 0 > -2281> 2018-07-11 07:34:57.636447 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15287 conn(0x55a311b75000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1857 0x55a33b42c800 osd_ping(ping e494 stamp 2018-07-11 07:34:58.851124) v4 > -2280> 2018-07-11 07:34:57.636468 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.4 10.8.128.113:0/15287 1857 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:58.851124) v4 ==== 2004+0+0 (3800833687 0 0) 0x55a33b42c800 con 0x55a311b75000 > -2279> 2018-07-11 07:34:57.636482 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:58.851124) v4 -- 0x55a30f668000 con 0 > -2278> 2018-07-11 07:34:57.858966 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b7000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1810 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:34:57.609867) v4 > -2277> 2018-07-11 07:34:57.858987 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.8 10.8.128.108:0/11967 1810 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:57.609867) v4 ==== 2004+0+0 (4066294088 0 0) 0x55a34de78600 con 0x55a3170b7000 > -2276> 2018-07-11 07:34:57.858997 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:57.609867) v4 -- 0x55a3203f6000 con 0 > -2275> 2018-07-11 07:34:57.859010 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b8800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1810 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:34:57.609867) v4 > -2274> 2018-07-11 07:34:57.859031 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.8 10.8.128.108:0/11967 1810 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:57.609867) v4 ==== 2004+0+0 (4066294088 0 0) 0x55a332e79a00 con 0x55a3170b8800 > -2273> 2018-07-11 07:34:57.859042 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:57.609867) v4 -- 0x55a332ff2000 con 0 > -2272> 2018-07-11 07:34:58.022018 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a357da6b40 con 0 > -2271> 2018-07-11 07:34:58.040187 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5179 0x55a3202a0c80 replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:494/492,start:6:aa201d2c:::benchmark_data_magna108_35617_object738376:0,end:6:aa213577:::benchmark_data_magna108.ceph.redhat.com_2478_object404059:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -2270> 2018-07-11 07:34:58.040212 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5179 ==== replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:494/492,start:6:aa201d2c:::benchmark_data_magna108_35617_object738376:0,end:6:aa213577:::benchmark_data_magna108.ceph.redhat.com_2478_object404059:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (2045278171 0 0) 0x55a3202a0c80 con 0x55a317019000 > -2269> 2018-07-11 07:34:58.387530 7f355d88e700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.107:0/12617 conn(0x55a32447f800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1832 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:34:58.387011) v4 > -2268> 2018-07-11 07:34:58.387556 7f355d88e700 1 -- 10.8.128.107:6803/12665 <== osd.9 10.8.128.107:0/12617 1832 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:58.387011) v4 ==== 2004+0+0 (1709703122 0 0) 0x55a34de6f000 con 0x55a32447f800 > -2267> 2018-07-11 07:34:58.387569 7f355d88e700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:58.387011) v4 -- 0x55a3203f6000 con 0 > -2266> 2018-07-11 07:34:58.387582 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.107:0/12617 conn(0x55a32447e000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1832 0x55a332364400 osd_ping(ping e494 stamp 2018-07-11 07:34:58.387011) v4 > -2265> 2018-07-11 07:34:58.387601 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.9 10.8.128.107:0/12617 1832 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:58.387011) v4 ==== 2004+0+0 (1709703122 0 0) 0x55a332364400 con 0x55a32447e000 > -2264> 2018-07-11 07:34:58.387612 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:58.387011) v4 -- 0x55a33b42c200 con 0 > -2263> 2018-07-11 07:34:58.657531 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.115:0/13323 conn(0x55a321665800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1870 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:34:58.995426) v4 > -2262> 2018-07-11 07:34:58.657553 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.6 10.8.128.115:0/13323 1870 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:58.995426) v4 ==== 2004+0+0 (1222723441 0 0) 0x55a324474c00 con 0x55a321665800 > -2261> 2018-07-11 07:34:58.657565 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:58.995426) v4 -- 0x55a332ff2000 con 0 > -2260> 2018-07-11 07:34:58.657636 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.115:0/13323 conn(0x55a321668800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1870 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:34:58.995426) v4 > -2259> 2018-07-11 07:34:58.657651 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.6 10.8.128.115:0/13323 1870 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:58.995426) v4 ==== 2004+0+0 (1222723441 0 0) 0x55a332e79a00 con 0x55a321668800 > -2258> 2018-07-11 07:34:58.657659 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:58.995426) v4 -- 0x55a32eb14e00 con 0 > -2257> 2018-07-11 07:34:58.708613 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a3183d8480 con 0 > -2256> 2018-07-11 07:34:58.723531 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5180 0x55a3202a0f00 replica scrub(pg: 6.15,from:0'0,to:430'106248,epoch:494/492,start:6:aa213577:::benchmark_data_magna108.ceph.redhat.com_2478_object404059:0,end:6:aa2209fe:::benchmark_data_magna108_35617_object978351:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -2255> 2018-07-11 07:34:58.723561 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5180 ==== replica scrub(pg: 6.15,from:0'0,to:430'106248,epoch:494/492,start:6:aa213577:::benchmark_data_magna108.ceph.redhat.com_2478_object404059:0,end:6:aa2209fe:::benchmark_data_magna108_35617_object978351:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (296391756 0 0) 0x55a3202a0f00 con 0x55a317019000 > -2254> 2018-07-11 07:34:58.736921 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15287 conn(0x55a311b75000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1858 0x55a30f668000 osd_ping(ping e494 stamp 2018-07-11 07:34:59.951576) v4 > -2253> 2018-07-11 07:34:58.736927 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15287 conn(0x55a311b72000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1858 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:34:59.951576) v4 > -2252> 2018-07-11 07:34:58.736942 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.4 10.8.128.113:0/15287 1858 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:59.951576) v4 ==== 2004+0+0 (20428018 0 0) 0x55a30f668000 con 0x55a311b75000 > -2251> 2018-07-11 07:34:58.736944 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.4 10.8.128.113:0/15287 1858 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:59.951576) v4 ==== 2004+0+0 (20428018 0 0) 0x55a324474c00 con 0x55a311b72000 > -2250> 2018-07-11 07:34:58.736951 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:59.951576) v4 -- 0x55a33b42da00 con 0 > -2249> 2018-07-11 07:34:58.736984 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:59.951576) v4 -- 0x55a32eb14e00 con 0 > -2248> 2018-07-11 07:34:58.888068 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.107:0/12617 conn(0x55a32447e000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1833 0x55a33b42c200 osd_ping(ping e494 stamp 2018-07-11 07:34:58.887614) v4 > -2247> 2018-07-11 07:34:58.888084 7f355d88e700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.107:0/12617 conn(0x55a32447f800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1833 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:34:58.887614) v4 > -2246> 2018-07-11 07:34:58.888096 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.9 10.8.128.107:0/12617 1833 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:58.887614) v4 ==== 2004+0+0 (1970727005 0 0) 0x55a33b42c200 con 0x55a32447e000 > -2245> 2018-07-11 07:34:58.888103 7f355d88e700 1 -- 10.8.128.107:6803/12665 <== osd.9 10.8.128.107:0/12617 1833 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:58.887614) v4 ==== 2004+0+0 (1970727005 0 0) 0x55a34de78600 con 0x55a32447f800 > -2244> 2018-07-11 07:34:58.888119 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:58.887614) v4 -- 0x55a33b42da00 con 0 > -2243> 2018-07-11 07:34:58.888149 7f355d88e700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:58.887614) v4 -- 0x55a3203f6000 con 0 > -2242> 2018-07-11 07:34:59.559465 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b7000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1811 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:34:59.310323) v4 > -2241> 2018-07-11 07:34:59.559481 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b8800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1811 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:34:59.310323) v4 > -2240> 2018-07-11 07:34:59.559494 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.8 10.8.128.108:0/11967 1811 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:59.310323) v4 ==== 2004+0+0 (2696345574 0 0) 0x55a34de6f000 con 0x55a3170b7000 > -2239> 2018-07-11 07:34:59.559499 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.8 10.8.128.108:0/11967 1811 ==== osd_ping(ping e494 stamp 2018-07-11 07:34:59.310323) v4 ==== 2004+0+0 (2696345574 0 0) 0x55a332e79a00 con 0x55a3170b8800 > -2238> 2018-07-11 07:34:59.559504 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:59.310323) v4 -- 0x55a3203f6000 con 0 > -2237> 2018-07-11 07:34:59.559545 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:34:59.310323) v4 -- 0x55a32eb14e00 con 0 > -2236> 2018-07-11 07:34:59.825920 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15263 conn(0x55a321664000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1858 0x55a30f668000 osd_ping(ping e494 stamp 2018-07-11 07:35:01.040562) v4 > -2235> 2018-07-11 07:34:59.825943 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.5 10.8.128.113:0/15263 1858 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:01.040562) v4 ==== 2004+0+0 (3216161108 0 0) 0x55a30f668000 con 0x55a321664000 > -2234> 2018-07-11 07:34:59.825959 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:01.040562) v4 -- 0x55a33b42da00 con 0 > -2233> 2018-07-11 07:34:59.826046 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15263 conn(0x55a321667000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1858 0x55a33b42c200 osd_ping(ping e494 stamp 2018-07-11 07:35:01.040562) v4 > -2232> 2018-07-11 07:34:59.826062 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.5 10.8.128.113:0/15263 1858 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:01.040562) v4 ==== 2004+0+0 (3216161108 0 0) 0x55a33b42c200 con 0x55a321667000 > -2231> 2018-07-11 07:34:59.826070 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:01.040562) v4 -- 0x55a33ede2200 con 0 > -2230> 2018-07-11 07:35:00.074574 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15885 conn(0x55a311b76800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1882 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:35:00.698686) v4 > -2229> 2018-07-11 07:35:00.074595 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.7 10.8.128.114:0/15885 1882 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:00.698686) v4 ==== 2004+0+0 (3766568418 0 0) 0x55a34de78600 con 0x55a311b76800 > -2228> 2018-07-11 07:35:00.074605 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:00.698686) v4 -- 0x55a3203f5c00 con 0 > -2227> 2018-07-11 07:35:00.074598 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15885 conn(0x55a3170b5800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1882 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:35:00.698686) v4 > -2226> 2018-07-11 07:35:00.074615 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.7 10.8.128.114:0/15885 1882 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:00.698686) v4 ==== 2004+0+0 (3766568418 0 0) 0x55a324474c00 con 0x55a3170b5800 > -2225> 2018-07-11 07:35:00.074628 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:00.698686) v4 -- 0x55a332e99200 con 0 > -2224> 2018-07-11 07:35:00.384650 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12650 conn(0x55a311b67000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1875 0x55a30f668000 osd_ping(ping e494 stamp 2018-07-11 07:35:00.420999) v4 > -2223> 2018-07-11 07:35:00.384688 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.3 10.8.128.106:0/12650 1875 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:00.420999) v4 ==== 2004+0+0 (3623259528 0 0) 0x55a30f668000 con 0x55a311b67000 > -2222> 2018-07-11 07:35:00.384704 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:00.420999) v4 -- 0x55a33ede2000 con 0 > -2221> 2018-07-11 07:35:00.384730 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12650 conn(0x55a311b68800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1875 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:35:00.420999) v4 > -2220> 2018-07-11 07:35:00.384751 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.3 10.8.128.106:0/12650 1875 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:00.420999) v4 ==== 2004+0+0 (3623259528 0 0) 0x55a34de6f000 con 0x55a311b68800 > -2219> 2018-07-11 07:35:00.384768 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:00.420999) v4 -- 0x55a33b42d200 con 0 > -2218> 2018-07-11 07:35:00.423813 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a325d45680 con 0 > -2217> 2018-07-11 07:35:00.453024 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5181 0x55a3202a1180 replica scrub(pg: 6.15,from:0'0,to:430'106710,epoch:494/492,start:6:aa2209fe:::benchmark_data_magna108_35617_object978351:0,end:6:aa230886:::benchmark_data_magna108.ceph.redhat.com_2478_object296676:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -2216> 2018-07-11 07:35:00.453047 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5181 ==== replica scrub(pg: 6.15,from:0'0,to:430'106710,epoch:494/492,start:6:aa2209fe:::benchmark_data_magna108_35617_object978351:0,end:6:aa230886:::benchmark_data_magna108.ceph.redhat.com_2478_object296676:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (3912968618 0 0) 0x55a3202a1180 con 0x55a317019000 > -2215> 2018-07-11 07:35:00.588653 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.107:0/12617 conn(0x55a32447e000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1834 0x55a33b42c200 osd_ping(ping e494 stamp 2018-07-11 07:35:00.588186) v4 > -2214> 2018-07-11 07:35:00.588662 7f355d88e700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.107:0/12617 conn(0x55a32447f800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1834 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:35:00.588186) v4 > -2213> 2018-07-11 07:35:00.588674 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.9 10.8.128.107:0/12617 1834 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:00.588186) v4 ==== 2004+0+0 (641377185 0 0) 0x55a33b42c200 con 0x55a32447e000 > -2212> 2018-07-11 07:35:00.588680 7f355d88e700 1 -- 10.8.128.107:6803/12665 <== osd.9 10.8.128.107:0/12617 1834 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:00.588186) v4 ==== 2004+0+0 (641377185 0 0) 0x55a34de78600 con 0x55a32447f800 > -2211> 2018-07-11 07:35:00.588686 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:00.588186) v4 -- 0x55a33b42d000 con 0 > -2210> 2018-07-11 07:35:00.588727 7f355d88e700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:00.588186) v4 -- 0x55a33b42d200 con 0 > -2209> 2018-07-11 07:35:00.643184 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 60] Generated table #36592: 4957167 keys, 80348283 bytes > -2208> 2018-07-11 07:35:00.643234 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294500643218, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 36592, "file_size": 80348283, "table_properties": {"data_size": 67109331, "index_size": 689255, "filter_size": 12548707, "raw_key_size": 143207039, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957167, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957167", "kMergeOperands": "0"}} > -2207> 2018-07-11 07:35:00.771362 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12651 conn(0x55a3216a0800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1859 0x55a33ede2000 osd_ping(ping e494 stamp 2018-07-11 07:35:00.807782) v4 > -2206> 2018-07-11 07:35:00.771393 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.2 10.8.128.106:0/12651 1859 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:00.807782) v4 ==== 2004+0+0 (3483182841 0 0) 0x55a33ede2000 con 0x55a3216a0800 > -2205> 2018-07-11 07:35:00.771409 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:00.807782) v4 -- 0x55a33b42d000 con 0 > -2204> 2018-07-11 07:35:00.771406 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12651 conn(0x55a32169f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1859 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:35:00.807782) v4 > -2203> 2018-07-11 07:35:00.771431 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.2 10.8.128.106:0/12651 1859 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:00.807782) v4 ==== 2004+0+0 (3483182841 0 0) 0x55a332e79a00 con 0x55a32169f000 > -2202> 2018-07-11 07:35:00.771447 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:00.807782) v4 -- 0x55a332e99200 con 0 > -2201> 2018-07-11 07:35:00.979561 7f3543899700 5 osd.10 494 heartbeat: osd_stat(784 GB used, 146 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,0,0,1]) > -2200> 2018-07-11 07:35:00.979577 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6802/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:35:00.979575) v4 -- 0x55a33532cc00 con 0 > -2199> 2018-07-11 07:35:00.979615 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6803/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:35:00.979575) v4 -- 0x55a32fb18c00 con 0 > -2198> 2018-07-11 07:35:00.979631 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6806/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:35:00.979575) v4 -- 0x55a32442b000 con 0 > -2197> 2018-07-11 07:35:00.979646 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6807/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:35:00.979575) v4 -- 0x55a33235b400 con 0 > -2196> 2018-07-11 07:35:00.979665 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6808/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:35:00.979575) v4 -- 0x55a32fb19200 con 0 > -2195> 2018-07-11 07:35:00.979685 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6810/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:35:00.979575) v4 -- 0x55a332faee00 con 0 > -2194> 2018-07-11 07:35:00.979718 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6809/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:35:00.979575) v4 -- 0x55a3314cfe00 con 0 > -2193> 2018-07-11 07:35:00.979736 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6811/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:35:00.979575) v4 -- 0x55a332feb800 con 0 > -2192> 2018-07-11 07:35:00.979762 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6810/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:35:00.979575) v4 -- 0x55a332e79000 con 0 > -2191> 2018-07-11 07:35:00.979785 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6811/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:35:00.979575) v4 -- 0x55a32e457600 con 0 > -2190> 2018-07-11 07:35:00.979807 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6806/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:35:00.979575) v4 -- 0x55a332354600 con 0 > -2189> 2018-07-11 07:35:00.979823 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6807/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:35:00.979575) v4 -- 0x55a33ede2600 con 0 > -2188> 2018-07-11 07:35:00.979846 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6802/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:35:00.979575) v4 -- 0x55a33ede2c00 con 0 > -2187> 2018-07-11 07:35:00.979871 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6803/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:35:00.979575) v4 -- 0x55a323f55200 con 0 > -2186> 2018-07-11 07:35:00.979887 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6806/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:35:00.979575) v4 -- 0x55a323f55400 con 0 > -2185> 2018-07-11 07:35:00.979907 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6807/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:35:00.979575) v4 -- 0x55a327787200 con 0 > -2184> 2018-07-11 07:35:00.979934 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6802/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:35:00.979575) v4 -- 0x55a33bff1c00 con 0 > -2183> 2018-07-11 07:35:00.979957 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6803/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:35:00.979575) v4 -- 0x55a323f55600 con 0 > -2182> 2018-07-11 07:35:00.979980 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6806/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:35:00.979575) v4 -- 0x55a327786a00 con 0 > -2181> 2018-07-11 07:35:00.980006 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6807/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:35:00.979575) v4 -- 0x55a327786c00 con 0 > -2180> 2018-07-11 07:35:00.980025 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6802/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:35:00.979575) v4 -- 0x55a3177a9000 con 0 > -2179> 2018-07-11 07:35:00.980044 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6803/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:35:00.979575) v4 -- 0x55a327787000 con 0 > -2178> 2018-07-11 07:35:00.980065 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6807/12677 conn(0x55a314e85800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1867 0x55a33b42c200 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:00.979575) v4 > -2177> 2018-07-11 07:35:00.980085 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6807/12677 1867 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:00.979575) v4 ==== 2004+0+0 (3520633267 0 0) 0x55a33b42c200 con 0x55a314e85800 > -2176> 2018-07-11 07:35:00.980101 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6810/12651 conn(0x55a314e97800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1867 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:00.979575) v4 > -2175> 2018-07-11 07:35:00.980119 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6808/12651 conn(0x55a314e96000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1867 0x55a33b42c200 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:00.979575) v4 > -2174> 2018-07-11 07:35:00.980135 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6806/12677 conn(0x55a314e84000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1867 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:00.979575) v4 > -2173> 2018-07-11 07:35:00.980156 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6808/12651 1867 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:00.979575) v4 ==== 2004+0+0 (3520633267 0 0) 0x55a33b42c200 con 0x55a314e96000 > -2172> 2018-07-11 07:35:00.980159 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6806/12677 1867 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:00.979575) v4 ==== 2004+0+0 (3520633267 0 0) 0x55a34de6f000 con 0x55a314e84000 > -2171> 2018-07-11 07:35:00.980157 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6810/12651 1867 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:00.979575) v4 ==== 2004+0+0 (3520633267 0 0) 0x55a332e79a00 con 0x55a314e97800 > -2170> 2018-07-11 07:35:00.980185 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6802/15255 conn(0x55a314e1c000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1867 0x55a33b42c200 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:00.979575) v4 > -2169> 2018-07-11 07:35:00.980195 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6802/15255 1867 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:00.979575) v4 ==== 2004+0+0 (3520633267 0 0) 0x55a33b42c200 con 0x55a314e1c000 > -2168> 2018-07-11 07:35:00.980226 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6811/12650 conn(0x55a314e20800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1867 0x55a33b42c200 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:00.979575) v4 > -2167> 2018-07-11 07:35:00.980237 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6811/12650 1867 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:00.979575) v4 ==== 2004+0+0 (3520633267 0 0) 0x55a33b42c200 con 0x55a314e20800 > -2166> 2018-07-11 07:35:00.980230 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6803/15255 conn(0x55a314e1d800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1867 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:00.979575) v4 > -2165> 2018-07-11 07:35:00.980238 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6809/12650 conn(0x55a314e1f000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1867 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:00.979575) v4 > -2164> 2018-07-11 07:35:00.980257 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6803/15255 1867 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:00.979575) v4 ==== 2004+0+0 (3520633267 0 0) 0x55a332e79a00 con 0x55a314e1d800 > -2163> 2018-07-11 07:35:00.980266 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6809/12650 1867 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:00.979575) v4 ==== 2004+0+0 (3520633267 0 0) 0x55a34de6f000 con 0x55a314e1f000 > -2162> 2018-07-11 07:35:00.980279 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6810/15287 conn(0x55a314e03000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1867 0x55a33b42c200 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:00.979575) v4 > -2161> 2018-07-11 07:35:00.980292 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6810/15287 1867 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:00.979575) v4 ==== 2004+0+0 (3520633267 0 0) 0x55a33b42c200 con 0x55a314e03000 > -2160> 2018-07-11 07:35:00.980299 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6807/12617 conn(0x55a314e27800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1867 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:00.979575) v4 > -2159> 2018-07-11 07:35:00.980310 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6807/12617 1867 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:00.979575) v4 ==== 2004+0+0 (3520633267 0 0) 0x55a34de6f000 con 0x55a314e27800 > -2158> 2018-07-11 07:35:00.980307 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6811/15287 conn(0x55a314e04800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1867 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:00.979575) v4 > -2157> 2018-07-11 07:35:00.980329 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6807/15263 conn(0x55a314e9a800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1867 0x55a33b42c200 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:00.979575) v4 > -2156> 2018-07-11 07:35:00.980344 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6811/15287 1867 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:00.979575) v4 ==== 2004+0+0 (3520633267 0 0) 0x55a332e79a00 con 0x55a314e04800 > -2155> 2018-07-11 07:35:00.980345 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6803/13323 conn(0x55a314e16800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1867 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:00.979575) v4 > -2154> 2018-07-11 07:35:00.980348 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6807/15263 1867 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:00.979575) v4 ==== 2004+0+0 (3520633267 0 0) 0x55a33b42c200 con 0x55a314e9a800 > -2153> 2018-07-11 07:35:00.980362 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6803/13323 1867 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:00.979575) v4 ==== 2004+0+0 (3520633267 0 0) 0x55a34de6f000 con 0x55a314e16800 > -2152> 2018-07-11 07:35:00.980381 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6806/12617 conn(0x55a314e26000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1867 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:00.979575) v4 > -2151> 2018-07-11 07:35:00.980392 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6803/11967 conn(0x55a314e13800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1867 0x55a33b42c200 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:00.979575) v4 > -2150> 2018-07-11 07:35:00.980399 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6806/12617 1867 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:00.979575) v4 ==== 2004+0+0 (3520633267 0 0) 0x55a332e79a00 con 0x55a314e26000 > -2149> 2018-07-11 07:35:00.980399 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6806/15263 conn(0x55a314e99000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1867 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:00.979575) v4 > -2148> 2018-07-11 07:35:00.980406 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6803/11967 1867 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:00.979575) v4 ==== 2004+0+0 (3520633267 0 0) 0x55a33b42c200 con 0x55a314e13800 > -2147> 2018-07-11 07:35:00.980412 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6806/15263 1867 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:00.979575) v4 ==== 2004+0+0 (3520633267 0 0) 0x55a34de6f000 con 0x55a314e99000 > -2146> 2018-07-11 07:35:00.980442 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6802/11967 conn(0x55a314e12000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1867 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:00.979575) v4 > -2145> 2018-07-11 07:35:00.980454 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6802/11967 1867 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:00.979575) v4 ==== 2004+0+0 (3520633267 0 0) 0x55a34de6f000 con 0x55a314e12000 > -2144> 2018-07-11 07:35:00.980443 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6806/15885 conn(0x55a314e00000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1867 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:00.979575) v4 > -2143> 2018-07-11 07:35:00.980464 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6806/15885 1867 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:00.979575) v4 ==== 2004+0+0 (3520633267 0 0) 0x55a332e79a00 con 0x55a314e00000 > -2142> 2018-07-11 07:35:00.980482 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6807/15885 conn(0x55a314e01800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1867 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:00.979575) v4 > -2141> 2018-07-11 07:35:00.980493 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6807/15885 1867 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:00.979575) v4 ==== 2004+0+0 (3520633267 0 0) 0x55a34de6f000 con 0x55a314e01800 > -2140> 2018-07-11 07:35:00.980506 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6802/13323 conn(0x55a314e15000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1867 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:00.979575) v4 > -2139> 2018-07-11 07:35:00.980524 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6802/13323 1867 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:00.979575) v4 ==== 2004+0+0 (3520633267 0 0) 0x55a332e79a00 con 0x55a314e15000 > -2138> 2018-07-11 07:35:00.980527 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6803/15919 conn(0x55a314e88800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1867 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:00.979575) v4 > -2137> 2018-07-11 07:35:00.980537 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6803/15919 1867 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:00.979575) v4 ==== 2004+0+0 (3520633267 0 0) 0x55a34de6f000 con 0x55a314e88800 > -2136> 2018-07-11 07:35:00.980568 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6802/15919 conn(0x55a314e87000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1867 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:00.979575) v4 > -2135> 2018-07-11 07:35:00.980585 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6802/15919 1867 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:00.979575) v4 ==== 2004+0+0 (3520633267 0 0) 0x55a332e79a00 con 0x55a314e87000 > -2134> 2018-07-11 07:35:01.340329 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a325d60fc0 con 0 > -2133> 2018-07-11 07:35:01.348776 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5182 0x55a3202a1400 replica scrub(pg: 6.15,from:0'0,to:430'106870,epoch:494/492,start:6:aa230886:::benchmark_data_magna108.ceph.redhat.com_2478_object296676:0,end:6:aa247c15:::benchmark_data_magna108_35617_object1027550:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -2132> 2018-07-11 07:35:01.348807 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5182 ==== replica scrub(pg: 6.15,from:0'0,to:430'106870,epoch:494/492,start:6:aa230886:::benchmark_data_magna108.ceph.redhat.com_2478_object296676:0,end:6:aa247c15:::benchmark_data_magna108_35617_object1027550:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 234+0+0 (1978447897 0 0) 0x55a3202a1400 con 0x55a317019000 > -2131> 2018-07-11 07:35:01.436930 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12677 conn(0x55a32169d800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1882 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:35:01.473316) v4 > -2130> 2018-07-11 07:35:01.436953 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.1 10.8.128.106:0/12677 1882 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:01.473316) v4 ==== 2004+0+0 (1954887298 0 0) 0x55a34de6f000 con 0x55a32169d800 > -2129> 2018-07-11 07:35:01.436965 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:01.473316) v4 -- 0x55a332354600 con 0 > -2128> 2018-07-11 07:35:01.436958 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12677 conn(0x55a32169c000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1882 0x55a33b42c200 osd_ping(ping e494 stamp 2018-07-11 07:35:01.473316) v4 > -2127> 2018-07-11 07:35:01.436984 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.1 10.8.128.106:0/12677 1882 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:01.473316) v4 ==== 2004+0+0 (1954887298 0 0) 0x55a33b42c200 con 0x55a32169c000 > -2126> 2018-07-11 07:35:01.437004 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:01.473316) v4 -- 0x55a33235b400 con 0 > -2125> 2018-07-11 07:35:02.085205 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12650 conn(0x55a311b68800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1876 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:35:02.121543) v4 > -2124> 2018-07-11 07:35:02.085213 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12650 conn(0x55a311b67000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1876 0x55a32fb19200 osd_ping(ping e494 stamp 2018-07-11 07:35:02.121543) v4 > -2123> 2018-07-11 07:35:02.085242 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.3 10.8.128.106:0/12650 1876 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:02.121543) v4 ==== 2004+0+0 (3383165026 0 0) 0x55a32fb19200 con 0x55a311b67000 > -2122> 2018-07-11 07:35:02.085239 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.3 10.8.128.106:0/12650 1876 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:02.121543) v4 ==== 2004+0+0 (3383165026 0 0) 0x55a34de78600 con 0x55a311b68800 > -2121> 2018-07-11 07:35:02.085265 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:02.121543) v4 -- 0x55a33235b400 con 0 > -2120> 2018-07-11 07:35:02.085292 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:02.121543) v4 -- 0x55a332354600 con 0 > -2119> 2018-07-11 07:35:02.180224 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 60] Generated table #36593: 999999 keys, 16208887 bytes > -2118> 2018-07-11 07:35:02.180273 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294502180258, "cf_name": "default", "job": 60, "event": "table_file_creation", "file_number": 36593, "file_size": 16208887, "table_properties": {"data_size": 13537846, "index_size": 138627, "filter_size": 2531427, "raw_key_size": 28888866, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 3323, "num_entries": 999999, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "999999", "kMergeOperands": "0"}} > -2117> 2018-07-11 07:35:02.180304 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 60] Compacted 1@1 + 5@2 files to L2 => 357729484 bytes > -2116> 2018-07-11 07:35:02.194508 7f354e0ae700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 64884155848 > -2115> 2018-07-11 07:35:02.194639 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:35:02.194473) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 18 85 446 3641 796 0] max score 6.35, MB/sec: 10.9 rd, 10.9 wr, level 2, files in(1, 5) out(7) MB in(76.6, 264.5) out(341.2), read-write-amplify(8.9) write-amplify(4.5) OK, records in: 22070386, records dropped: 0 > > -2114> 2018-07-11 07:35:02.194651 7f354e0ae700 4 rocksdb: (Original Log Time 2018/07/11-07:35:02.194499) EVENT_LOG_v1 {"time_micros": 1531294502194488, "job": 60, "event": "compaction_finished", "compaction_time_micros": 32775144, "output_level": 2, "num_output_files": 7, "total_output_size": 357729484, "num_input_records": 22070386, "num_output_records": 22070386, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 18, 85, 446, 3641, 796, 0]} > -2113> 2018-07-11 07:35:02.195681 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 61] Compacting 1@1 + 9@2 files to L2, score 6.35 > -2112> 2018-07-11 07:35:02.195694 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 60 Base level 1, inputs: [36562(76MB)], [36592(76MB) 36593(15MB) 36547(76MB) 36548(69MB) 36549(56MB) 36531(53MB) 36532(70MB) 36542(76MB) 36543(76MB)] > > -2111> 2018-07-11 07:35:02.195720 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294502195697, "job": 61, "event": "compaction_started", "files_L1": [36562], "files_L2": [36592, 36593, 36547, 36548, 36549, 36531, 36532, 36542, 36543], "score": 6.35392, "input_data_size": 679330645} > -2110> 2018-07-11 07:35:02.257390 7f35488a3700 1 -- 10.8.128.107:6800/12665 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55a317057080 con 0 > -2109> 2018-07-11 07:35:02.257461 7f35488a3700 1 -- 10.8.128.107:6800/12665 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55a3170ebb80 con 0 > -2108> 2018-07-11 07:35:02.459920 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b7000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1812 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:35:02.210778) v4 > -2107> 2018-07-11 07:35:02.459948 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.8 10.8.128.108:0/11967 1812 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:02.210778) v4 ==== 2004+0+0 (1350003574 0 0) 0x55a34de6f000 con 0x55a3170b7000 > -2106> 2018-07-11 07:35:02.459941 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b8800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1812 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:35:02.210778) v4 > -2105> 2018-07-11 07:35:02.459959 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:02.210778) v4 -- 0x55a332354600 con 0 > -2104> 2018-07-11 07:35:02.459960 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.8 10.8.128.108:0/11967 1812 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:02.210778) v4 ==== 2004+0+0 (1350003574 0 0) 0x55a324474c00 con 0x55a3170b8800 > -2103> 2018-07-11 07:35:02.459988 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:02.210778) v4 -- 0x55a32e457600 con 0 > -2102> 2018-07-11 07:35:02.543046 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a32cbc8d80 con 0 > -2101> 2018-07-11 07:35:02.569718 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5183 0x55a3202a1680 replica scrub(pg: 6.15,from:0'0,to:430'106855,epoch:494/492,start:6:aa247c15:::benchmark_data_magna108_35617_object1027550:0,end:6:aa255e05:::benchmark_data_magna108_35617_object752807:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -2100> 2018-07-11 07:35:02.569741 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5183 ==== replica scrub(pg: 6.15,from:0'0,to:430'106855,epoch:494/492,start:6:aa247c15:::benchmark_data_magna108_35617_object1027550:0,end:6:aa255e05:::benchmark_data_magna108_35617_object752807:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 219+0+0 (1744595534 0 0) 0x55a3202a1680 con 0x55a317019000 > -2099> 2018-07-11 07:35:02.662349 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15919 conn(0x55a317074800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1861 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:35:03.286524) v4 > -2098> 2018-07-11 07:35:02.662374 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.11 10.8.128.114:0/15919 1861 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:03.286524) v4 ==== 2004+0+0 (4002050068 0 0) 0x55a332e79a00 con 0x55a317074800 > -2097> 2018-07-11 07:35:02.662386 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:03.286524) v4 -- 0x55a332faee00 con 0 > -2096> 2018-07-11 07:35:02.662393 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15919 conn(0x55a3170b4000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1861 0x55a33b42c200 osd_ping(ping e494 stamp 2018-07-11 07:35:03.286524) v4 > -2095> 2018-07-11 07:35:02.662415 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.11 10.8.128.114:0/15919 1861 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:03.286524) v4 ==== 2004+0+0 (4002050068 0 0) 0x55a33b42c200 con 0x55a3170b4000 > -2094> 2018-07-11 07:35:02.662426 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:03.286524) v4 -- 0x55a33b42d000 con 0 > -2093> 2018-07-11 07:35:02.974929 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15885 conn(0x55a311b76800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1883 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:35:03.599163) v4 > -2092> 2018-07-11 07:35:02.974931 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15885 conn(0x55a3170b5800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1883 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:35:03.599163) v4 > -2091> 2018-07-11 07:35:02.974953 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.7 10.8.128.114:0/15885 1883 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:03.599163) v4 ==== 2004+0+0 (3341923121 0 0) 0x55a34de78600 con 0x55a311b76800 > -2090> 2018-07-11 07:35:02.974957 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.7 10.8.128.114:0/15885 1883 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:03.599163) v4 ==== 2004+0+0 (3341923121 0 0) 0x55a324474c00 con 0x55a3170b5800 > -2089> 2018-07-11 07:35:02.974965 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:03.599163) v4 -- 0x55a3314cfe00 con 0 > -2088> 2018-07-11 07:35:02.974988 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:03.599163) v4 -- 0x55a32fb18c00 con 0 > -2087> 2018-07-11 07:35:03.035985 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15255 conn(0x55a311b65800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1887 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:35:04.250771) v4 > -2086> 2018-07-11 07:35:03.036009 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.0 10.8.128.113:0/15255 1887 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:04.250771) v4 ==== 2004+0+0 (2036832253 0 0) 0x55a332e79a00 con 0x55a311b65800 > -2085> 2018-07-11 07:35:03.036018 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:04.250771) v4 -- 0x55a332e99200 con 0 > -2084> 2018-07-11 07:35:03.036009 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15255 conn(0x55a311b64000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1887 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:35:04.250771) v4 > -2083> 2018-07-11 07:35:03.036027 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.0 10.8.128.113:0/15255 1887 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:04.250771) v4 ==== 2004+0+0 (2036832253 0 0) 0x55a34de6f000 con 0x55a311b64000 > -2082> 2018-07-11 07:35:03.036041 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:04.250771) v4 -- 0x55a32442b000 con 0 > -2081> 2018-07-11 07:35:03.071946 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12651 conn(0x55a3216a0800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1860 0x55a33235b400 osd_ping(ping e494 stamp 2018-07-11 07:35:03.108283) v4 > -2080> 2018-07-11 07:35:03.071968 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.2 10.8.128.106:0/12651 1860 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:03.108283) v4 ==== 2004+0+0 (277955109 0 0) 0x55a33235b400 con 0x55a3216a0800 > -2079> 2018-07-11 07:35:03.071977 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:03.108283) v4 -- 0x55a33234be00 con 0 > -2078> 2018-07-11 07:35:03.072045 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12651 conn(0x55a32169f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1860 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:35:03.108283) v4 > -2077> 2018-07-11 07:35:03.072070 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.2 10.8.128.106:0/12651 1860 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:03.108283) v4 ==== 2004+0+0 (277955109 0 0) 0x55a324474c00 con 0x55a32169f000 > -2076> 2018-07-11 07:35:03.072087 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:03.108283) v4 -- 0x55a332ff2a00 con 0 > -2075> 2018-07-11 07:35:03.211366 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a325d65b00 con 0 > -2074> 2018-07-11 07:35:03.224238 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5184 0x55a3202a1900 replica scrub(pg: 6.15,from:0'0,to:430'105999,epoch:494/492,start:6:aa255e05:::benchmark_data_magna108_35617_object752807:0,end:6:aa2673d1:::benchmark_data_magna108_35617_object666137:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -2073> 2018-07-11 07:35:03.224274 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5184 ==== replica scrub(pg: 6.15,from:0'0,to:430'105999,epoch:494/492,start:6:aa255e05:::benchmark_data_magna108_35617_object752807:0,end:6:aa2673d1:::benchmark_data_magna108_35617_object666137:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (1148870570 0 0) 0x55a3202a1900 con 0x55a317019000 > -2072> 2018-07-11 07:35:03.437264 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15287 conn(0x55a311b72000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1859 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:35:04.652017) v4 > -2071> 2018-07-11 07:35:03.437276 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15287 conn(0x55a311b75000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1859 0x55a33b42c200 osd_ping(ping e494 stamp 2018-07-11 07:35:04.652017) v4 > -2070> 2018-07-11 07:35:03.437289 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.4 10.8.128.113:0/15287 1859 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:04.652017) v4 ==== 2004+0+0 (995406610 0 0) 0x55a332e79a00 con 0x55a311b72000 > -2069> 2018-07-11 07:35:03.437293 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.4 10.8.128.113:0/15287 1859 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:04.652017) v4 ==== 2004+0+0 (995406610 0 0) 0x55a33b42c200 con 0x55a311b75000 > -2068> 2018-07-11 07:35:03.437302 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:04.652017) v4 -- 0x55a332ff2a00 con 0 > -2067> 2018-07-11 07:35:03.437327 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:04.652017) v4 -- 0x55a33234be00 con 0 > -2066> 2018-07-11 07:35:03.762849 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15919 conn(0x55a317074800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1862 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:35:04.387001) v4 > -2065> 2018-07-11 07:35:03.762872 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.11 10.8.128.114:0/15919 1862 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:04.387001) v4 ==== 2004+0+0 (3257113445 0 0) 0x55a324474c00 con 0x55a317074800 > -2064> 2018-07-11 07:35:03.762881 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:04.387001) v4 -- 0x55a332ff2a00 con 0 > -2063> 2018-07-11 07:35:03.762989 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15919 conn(0x55a3170b4000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1862 0x55a33235b400 osd_ping(ping e494 stamp 2018-07-11 07:35:04.387001) v4 > -2062> 2018-07-11 07:35:03.763009 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.11 10.8.128.114:0/15919 1862 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:04.387001) v4 ==== 2004+0+0 (3257113445 0 0) 0x55a33235b400 con 0x55a3170b4000 > -2061> 2018-07-11 07:35:03.763021 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:04.387001) v4 -- 0x55a324474600 con 0 > -2060> 2018-07-11 07:35:03.931695 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a325d686c0 con 0 > -2059> 2018-07-11 07:35:03.952132 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5185 0x55a3202a1b80 replica scrub(pg: 6.15,from:0'0,to:430'106100,epoch:494/492,start:6:aa2673d1:::benchmark_data_magna108_35617_object666137:0,end:6:aa2761a3:::benchmark_data_magna108_35617_object789741:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -2058> 2018-07-11 07:35:03.952161 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5185 ==== replica scrub(pg: 6.15,from:0'0,to:430'106100,epoch:494/492,start:6:aa2673d1:::benchmark_data_magna108_35617_object666137:0,end:6:aa2761a3:::benchmark_data_magna108_35617_object789741:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (3342008899 0 0) 0x55a3202a1b80 con 0x55a317019000 > -2057> 2018-07-11 07:35:03.957938 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.115:0/13323 conn(0x55a321665800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1871 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:35:04.295893) v4 > -2056> 2018-07-11 07:35:03.957963 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.6 10.8.128.115:0/13323 1871 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:04.295893) v4 ==== 2004+0+0 (3933745061 0 0) 0x55a332e79a00 con 0x55a321665800 > -2055> 2018-07-11 07:35:03.957981 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:04.295893) v4 -- 0x55a332ff2a00 con 0 > -2054> 2018-07-11 07:35:03.958060 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.115:0/13323 conn(0x55a321668800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1871 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:35:04.295893) v4 > -2053> 2018-07-11 07:35:03.958076 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.6 10.8.128.115:0/13323 1871 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:04.295893) v4 ==== 2004+0+0 (3933745061 0 0) 0x55a324474c00 con 0x55a321668800 > -2052> 2018-07-11 07:35:03.958086 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:04.295893) v4 -- 0x55a332ff2a00 con 0 > -2051> 2018-07-11 07:35:04.385658 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12650 conn(0x55a311b67000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1877 0x55a33b42c200 osd_ping(ping e494 stamp 2018-07-11 07:35:04.422067) v4 > -2050> 2018-07-11 07:35:04.385658 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12650 conn(0x55a311b68800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1877 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:35:04.422067) v4 > -2049> 2018-07-11 07:35:04.385688 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.3 10.8.128.106:0/12650 1877 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:04.422067) v4 ==== 2004+0+0 (3447504213 0 0) 0x55a33b42c200 con 0x55a311b67000 > -2048> 2018-07-11 07:35:04.385690 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.3 10.8.128.106:0/12650 1877 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:04.422067) v4 ==== 2004+0+0 (3447504213 0 0) 0x55a34de78600 con 0x55a311b68800 > -2047> 2018-07-11 07:35:04.385706 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:04.422067) v4 -- 0x55a33bff1800 con 0 > -2046> 2018-07-11 07:35:04.385736 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:04.422067) v4 -- 0x55a33b42d200 con 0 > -2045> 2018-07-11 07:35:04.696041 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a325d746c0 con 0 > -2044> 2018-07-11 07:35:04.707961 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5186 0x55a31c870000 replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:494/492,start:6:aa2761a3:::benchmark_data_magna108_35617_object789741:0,end:6:aa285926:::benchmark_data_magna108_35617_object341583:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -2043> 2018-07-11 07:35:04.707991 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5186 ==== replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:494/492,start:6:aa2761a3:::benchmark_data_magna108_35617_object789741:0,end:6:aa285926:::benchmark_data_magna108_35617_object341583:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (1853529966 0 0) 0x55a31c870000 con 0x55a317019000 > -2042> 2018-07-11 07:35:04.937290 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12677 conn(0x55a32169c000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1883 0x55a324474600 osd_ping(ping e494 stamp 2018-07-11 07:35:04.973756) v4 > -2041> 2018-07-11 07:35:04.937294 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12677 conn(0x55a32169d800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1883 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:35:04.973756) v4 > -2040> 2018-07-11 07:35:04.937312 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.1 10.8.128.106:0/12677 1883 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:04.973756) v4 ==== 2004+0+0 (95181146 0 0) 0x55a324474600 con 0x55a32169c000 > -2039> 2018-07-11 07:35:04.937313 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.1 10.8.128.106:0/12677 1883 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:04.973756) v4 ==== 2004+0+0 (95181146 0 0) 0x55a34de6f000 con 0x55a32169d800 > -2038> 2018-07-11 07:35:04.937324 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:04.973756) v4 -- 0x55a33532dc00 con 0 > -2037> 2018-07-11 07:35:04.937354 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:04.973756) v4 -- 0x55a33b42d200 con 0 > -2036> 2018-07-11 07:35:05.126471 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15263 conn(0x55a321667000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1859 0x55a33bff1800 osd_ping(ping e494 stamp 2018-07-11 07:35:06.341133) v4 > -2035> 2018-07-11 07:35:05.126497 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.5 10.8.128.113:0/15263 1859 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:06.341133) v4 ==== 2004+0+0 (1953262754 0 0) 0x55a33bff1800 con 0x55a321667000 > -2034> 2018-07-11 07:35:05.126518 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:06.341133) v4 -- 0x55a327786600 con 0 > -2033> 2018-07-11 07:35:05.126581 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15263 conn(0x55a321664000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1859 0x55a324474600 osd_ping(ping e494 stamp 2018-07-11 07:35:06.341133) v4 > -2032> 2018-07-11 07:35:05.126595 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.5 10.8.128.113:0/15263 1859 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:06.341133) v4 ==== 2004+0+0 (1953262754 0 0) 0x55a324474600 con 0x55a321664000 > -2031> 2018-07-11 07:35:05.126608 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:06.341133) v4 -- 0x55a32eb3aa00 con 0 > -2030> 2018-07-11 07:35:05.452229 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a34867c000 con 0 > -2029> 2018-07-11 07:35:05.466455 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5187 0x55a31c870280 replica scrub(pg: 6.15,from:0'0,to:430'105800,epoch:494/492,start:6:aa285926:::benchmark_data_magna108_35617_object341583:0,end:6:aa295ff3:::benchmark_data_magna108.ceph.redhat.com_2478_object131760:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -2028> 2018-07-11 07:35:05.466478 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5187 ==== replica scrub(pg: 6.15,from:0'0,to:430'105800,epoch:494/492,start:6:aa285926:::benchmark_data_magna108_35617_object341583:0,end:6:aa295ff3:::benchmark_data_magna108.ceph.redhat.com_2478_object131760:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (1896158619 0 0) 0x55a31c870280 con 0x55a317019000 > -2027> 2018-07-11 07:35:05.889202 7f355d88e700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.107:0/12617 conn(0x55a32447f800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1835 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:35:05.888762) v4 > -2026> 2018-07-11 07:35:05.889213 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.107:0/12617 conn(0x55a32447e000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1835 0x55a33bff1800 osd_ping(ping e494 stamp 2018-07-11 07:35:05.888762) v4 > -2025> 2018-07-11 07:35:05.889229 7f355d88e700 1 -- 10.8.128.107:6803/12665 <== osd.9 10.8.128.107:0/12617 1835 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:05.888762) v4 ==== 2004+0+0 (587107996 0 0) 0x55a34de78600 con 0x55a32447f800 > -2024> 2018-07-11 07:35:05.889236 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.9 10.8.128.107:0/12617 1835 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:05.888762) v4 ==== 2004+0+0 (587107996 0 0) 0x55a33bff1800 con 0x55a32447e000 > -2023> 2018-07-11 07:35:05.889240 7f355d88e700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:05.888762) v4 -- 0x55a33b42d200 con 0 > -2022> 2018-07-11 07:35:05.889298 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:05.888762) v4 -- 0x55a3177a8c00 con 0 > -2021> 2018-07-11 07:35:06.099276 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a32e7f5200 con 0 > -2020> 2018-07-11 07:35:06.116621 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5188 0x55a31c870500 replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:494/492,start:6:aa295ff3:::benchmark_data_magna108.ceph.redhat.com_2478_object131760:0,end:6:aa2a39ac:::benchmark_data_magna108_35617_object788246:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -2019> 2018-07-11 07:35:06.116652 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5188 ==== replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:494/492,start:6:aa295ff3:::benchmark_data_magna108.ceph.redhat.com_2478_object131760:0,end:6:aa2a39ac:::benchmark_data_magna108_35617_object788246:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (3731289906 0 0) 0x55a31c870500 con 0x55a317019000 > -2018> 2018-07-11 07:35:06.280177 7f3543899700 5 osd.10 494 heartbeat: osd_stat(784 GB used, 146 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,1]) > -2017> 2018-07-11 07:35:06.280192 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6802/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:35:06.280190) v4 -- 0x55a327786000 con 0 > -2016> 2018-07-11 07:35:06.280215 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6803/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:35:06.280190) v4 -- 0x55a3177a9400 con 0 > -2015> 2018-07-11 07:35:06.280233 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6806/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:35:06.280190) v4 -- 0x55a327787800 con 0 > -2014> 2018-07-11 07:35:06.280262 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6807/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:35:06.280190) v4 -- 0x55a327787400 con 0 > -2013> 2018-07-11 07:35:06.280286 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6808/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:35:06.280190) v4 -- 0x55a327787600 con 0 > -2012> 2018-07-11 07:35:06.280319 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6810/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:35:06.280190) v4 -- 0x55a327788200 con 0 > -2011> 2018-07-11 07:35:06.280340 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6809/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:35:06.280190) v4 -- 0x55a33ede2e00 con 0 > -2010> 2018-07-11 07:35:06.280357 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6811/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:35:06.280190) v4 -- 0x55a33ede2a00 con 0 > -2009> 2018-07-11 07:35:06.280378 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6810/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:35:06.280190) v4 -- 0x55a349f88600 con 0 > -2008> 2018-07-11 07:35:06.280407 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6811/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:35:06.280190) v4 -- 0x55a33ede3600 con 0 > -2007> 2018-07-11 07:35:06.280425 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6806/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:35:06.280190) v4 -- 0x55a33ede3a00 con 0 > -2006> 2018-07-11 07:35:06.280438 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6807/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:35:06.280190) v4 -- 0x55a33ede3200 con 0 > -2005> 2018-07-11 07:35:06.280459 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6802/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:35:06.280190) v4 -- 0x55a33ede3c00 con 0 > -2004> 2018-07-11 07:35:06.280482 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6803/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:35:06.280190) v4 -- 0x55a327789400 con 0 > -2003> 2018-07-11 07:35:06.280501 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6806/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:35:06.280190) v4 -- 0x55a33ede3400 con 0 > -2002> 2018-07-11 07:35:06.280519 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6807/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:35:06.280190) v4 -- 0x55a332e85e00 con 0 > -2001> 2018-07-11 07:35:06.280547 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6802/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:35:06.280190) v4 -- 0x55a327789a00 con 0 > -2000> 2018-07-11 07:35:06.280570 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6803/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:35:06.280190) v4 -- 0x55a323f54c00 con 0 > -1999> 2018-07-11 07:35:06.280593 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6806/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:35:06.280190) v4 -- 0x55a349f88e00 con 0 > -1998> 2018-07-11 07:35:06.280615 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6807/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:35:06.280190) v4 -- 0x55a349f88c00 con 0 > -1997> 2018-07-11 07:35:06.280635 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6802/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:35:06.280190) v4 -- 0x55a332ec3200 con 0 > -1996> 2018-07-11 07:35:06.280659 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6803/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:35:06.280190) v4 -- 0x55a327789200 con 0 > -1995> 2018-07-11 07:35:06.280741 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6803/15255 conn(0x55a314e1d800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1868 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:06.280190) v4 > -1994> 2018-07-11 07:35:06.280770 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6802/15255 conn(0x55a314e1c000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1868 0x55a33bff1800 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:06.280190) v4 > -1993> 2018-07-11 07:35:06.280761 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6803/15255 1868 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:06.280190) v4 ==== 2004+0+0 (2432292252 0 0) 0x55a324474c00 con 0x55a314e1d800 > -1992> 2018-07-11 07:35:06.280790 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6802/15255 1868 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:06.280190) v4 ==== 2004+0+0 (2432292252 0 0) 0x55a33bff1800 con 0x55a314e1c000 > -1991> 2018-07-11 07:35:06.280813 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6807/12617 conn(0x55a314e27800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1868 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:06.280190) v4 > -1990> 2018-07-11 07:35:06.280821 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6810/12651 conn(0x55a314e97800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1868 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:06.280190) v4 > -1989> 2018-07-11 07:35:06.280831 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6807/12617 1868 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:06.280190) v4 ==== 2004+0+0 (2432292252 0 0) 0x55a34de6f000 con 0x55a314e27800 > -1988> 2018-07-11 07:35:06.280832 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6810/12651 1868 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:06.280190) v4 ==== 2004+0+0 (2432292252 0 0) 0x55a324474c00 con 0x55a314e97800 > -1987> 2018-07-11 07:35:06.280854 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6807/12677 conn(0x55a314e85800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1868 0x55a33bff1800 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:06.280190) v4 > -1986> 2018-07-11 07:35:06.280873 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6807/12677 1868 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:06.280190) v4 ==== 2004+0+0 (2432292252 0 0) 0x55a33bff1800 con 0x55a314e85800 > -1985> 2018-07-11 07:35:06.280883 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6811/15287 conn(0x55a314e04800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1868 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:06.280190) v4 > -1984> 2018-07-11 07:35:06.280874 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6809/12650 conn(0x55a314e1f000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1868 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:06.280190) v4 > -1983> 2018-07-11 07:35:06.280896 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6811/15287 1868 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:06.280190) v4 ==== 2004+0+0 (2432292252 0 0) 0x55a324474c00 con 0x55a314e04800 > -1982> 2018-07-11 07:35:06.280899 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6809/12650 1868 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:06.280190) v4 ==== 2004+0+0 (2432292252 0 0) 0x55a34de6f000 con 0x55a314e1f000 > -1981> 2018-07-11 07:35:06.280902 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6810/15287 conn(0x55a314e03000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1868 0x55a33bff1800 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:06.280190) v4 > -1980> 2018-07-11 07:35:06.280920 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6806/12617 conn(0x55a314e26000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1868 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:06.280190) v4 > -1979> 2018-07-11 07:35:06.280918 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6810/15287 1868 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:06.280190) v4 ==== 2004+0+0 (2432292252 0 0) 0x55a33bff1800 con 0x55a314e03000 > -1978> 2018-07-11 07:35:06.280928 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6806/12617 1868 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:06.280190) v4 ==== 2004+0+0 (2432292252 0 0) 0x55a324474c00 con 0x55a314e26000 > -1977> 2018-07-11 07:35:06.280933 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6806/15263 conn(0x55a314e99000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1868 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:06.280190) v4 > -1976> 2018-07-11 07:35:06.280943 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6806/15263 1868 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:06.280190) v4 ==== 2004+0+0 (2432292252 0 0) 0x55a34de6f000 con 0x55a314e99000 > -1975> 2018-07-11 07:35:06.280954 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6806/15885 conn(0x55a314e00000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1868 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:06.280190) v4 > -1974> 2018-07-11 07:35:06.280963 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6806/15885 1868 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:06.280190) v4 ==== 2004+0+0 (2432292252 0 0) 0x55a324474c00 con 0x55a314e00000 > -1973> 2018-07-11 07:35:06.280962 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6807/15263 conn(0x55a314e9a800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1868 0x55a33bff1800 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:06.280190) v4 > -1972> 2018-07-11 07:35:06.280981 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6807/15263 1868 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:06.280190) v4 ==== 2004+0+0 (2432292252 0 0) 0x55a33bff1800 con 0x55a314e9a800 > -1971> 2018-07-11 07:35:06.280992 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6802/13323 conn(0x55a314e15000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1868 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:06.280190) v4 > -1970> 2018-07-11 07:35:06.280978 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6803/13323 conn(0x55a314e16800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1868 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:06.280190) v4 > -1969> 2018-07-11 07:35:06.281005 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6802/13323 1868 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:06.280190) v4 ==== 2004+0+0 (2432292252 0 0) 0x55a324474c00 con 0x55a314e15000 > -1968> 2018-07-11 07:35:06.281009 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6803/13323 1868 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:06.280190) v4 ==== 2004+0+0 (2432292252 0 0) 0x55a34de6f000 con 0x55a314e16800 > -1967> 2018-07-11 07:35:06.281030 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6811/12650 conn(0x55a314e20800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1868 0x55a33bff1800 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:06.280190) v4 > -1966> 2018-07-11 07:35:06.281043 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6807/15885 conn(0x55a314e01800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1868 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:06.280190) v4 > -1965> 2018-07-11 07:35:06.281044 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6811/12650 1868 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:06.280190) v4 ==== 2004+0+0 (2432292252 0 0) 0x55a33bff1800 con 0x55a314e20800 > -1964> 2018-07-11 07:35:06.281057 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6807/15885 1868 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:06.280190) v4 ==== 2004+0+0 (2432292252 0 0) 0x55a34de6f000 con 0x55a314e01800 > -1963> 2018-07-11 07:35:06.281059 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6802/15919 conn(0x55a314e87000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1868 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:06.280190) v4 > -1962> 2018-07-11 07:35:06.281073 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6802/15919 1868 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:06.280190) v4 ==== 2004+0+0 (2432292252 0 0) 0x55a324474c00 con 0x55a314e87000 > -1961> 2018-07-11 07:35:06.281082 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6806/12677 conn(0x55a314e84000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1868 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:06.280190) v4 > -1960> 2018-07-11 07:35:06.281094 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6806/12677 1868 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:06.280190) v4 ==== 2004+0+0 (2432292252 0 0) 0x55a34de6f000 con 0x55a314e84000 > -1959> 2018-07-11 07:35:06.281097 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6808/12651 conn(0x55a314e96000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1868 0x55a33bff1800 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:06.280190) v4 > -1958> 2018-07-11 07:35:06.281115 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6808/12651 1868 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:06.280190) v4 ==== 2004+0+0 (2432292252 0 0) 0x55a33bff1800 con 0x55a314e96000 > -1957> 2018-07-11 07:35:06.281124 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6802/11967 conn(0x55a314e12000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1868 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:06.280190) v4 > -1956> 2018-07-11 07:35:06.281136 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6802/11967 1868 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:06.280190) v4 ==== 2004+0+0 (2432292252 0 0) 0x55a34de6f000 con 0x55a314e12000 > -1955> 2018-07-11 07:35:06.281165 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6803/15919 conn(0x55a314e88800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1868 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:06.280190) v4 > -1954> 2018-07-11 07:35:06.281176 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6803/15919 1868 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:06.280190) v4 ==== 2004+0+0 (2432292252 0 0) 0x55a34de6f000 con 0x55a314e88800 > -1953> 2018-07-11 07:35:06.281188 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6803/11967 conn(0x55a314e13800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1868 0x55a33bff1800 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:06.280190) v4 > -1952> 2018-07-11 07:35:06.281203 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6803/11967 1868 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:06.280190) v4 ==== 2004+0+0 (2432292252 0 0) 0x55a33bff1800 con 0x55a314e13800 > -1951> 2018-07-11 07:35:06.389728 7f355d88e700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.107:0/12617 conn(0x55a32447f800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1836 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:35:06.389277) v4 > -1950> 2018-07-11 07:35:06.389750 7f355d88e700 1 -- 10.8.128.107:6803/12665 <== osd.9 10.8.128.107:0/12617 1836 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:06.389277) v4 ==== 2004+0+0 (4161410500 0 0) 0x55a34de6f000 con 0x55a32447f800 > -1949> 2018-07-11 07:35:06.389759 7f355d88e700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:06.389277) v4 -- 0x55a33ede2e00 con 0 > -1948> 2018-07-11 07:35:06.389787 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.107:0/12617 conn(0x55a32447e000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1836 0x55a33bff1800 osd_ping(ping e494 stamp 2018-07-11 07:35:06.389277) v4 > -1947> 2018-07-11 07:35:06.389807 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.9 10.8.128.107:0/12617 1836 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:06.389277) v4 ==== 2004+0+0 (4161410500 0 0) 0x55a33bff1800 con 0x55a32447e000 > -1946> 2018-07-11 07:35:06.389818 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:06.389277) v4 -- 0x55a327787400 con 0 > -1945> 2018-07-11 07:35:06.475414 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15885 conn(0x55a3170b5800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1884 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:35:07.099596) v4 > -1944> 2018-07-11 07:35:06.475427 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15885 conn(0x55a311b76800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1884 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:35:07.099596) v4 > -1943> 2018-07-11 07:35:06.475436 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.7 10.8.128.114:0/15885 1884 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:07.099596) v4 ==== 2004+0+0 (1606446306 0 0) 0x55a324474c00 con 0x55a3170b5800 > -1942> 2018-07-11 07:35:06.475445 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.7 10.8.128.114:0/15885 1884 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:07.099596) v4 ==== 2004+0+0 (1606446306 0 0) 0x55a34de78600 con 0x55a311b76800 > -1941> 2018-07-11 07:35:06.475447 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:07.099596) v4 -- 0x55a3177a9400 con 0 > -1940> 2018-07-11 07:35:06.475480 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:07.099596) v4 -- 0x55a327787800 con 0 > -1939> 2018-07-11 07:35:06.560455 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b7000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1813 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:35:06.311450) v4 > -1938> 2018-07-11 07:35:06.560475 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.8 10.8.128.108:0/11967 1813 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:06.311450) v4 ==== 2004+0+0 (1714130081 0 0) 0x55a34de6f000 con 0x55a3170b7000 > -1937> 2018-07-11 07:35:06.560485 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:06.311450) v4 -- 0x55a33b42d200 con 0 > -1936> 2018-07-11 07:35:06.560531 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b8800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1813 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:35:06.311450) v4 > -1935> 2018-07-11 07:35:06.560553 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.8 10.8.128.108:0/11967 1813 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:06.311450) v4 ==== 2004+0+0 (1714130081 0 0) 0x55a332e79a00 con 0x55a3170b8800 > -1934> 2018-07-11 07:35:06.560563 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:06.311450) v4 -- 0x55a332ff2a00 con 0 > -1933> 2018-07-11 07:35:06.572444 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12651 conn(0x55a3216a0800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1861 0x55a32eb3aa00 osd_ping(ping e494 stamp 2018-07-11 07:35:06.608896) v4 > -1932> 2018-07-11 07:35:06.572475 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.2 10.8.128.106:0/12651 1861 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:06.608896) v4 ==== 2004+0+0 (619199962 0 0) 0x55a32eb3aa00 con 0x55a3216a0800 > -1931> 2018-07-11 07:35:06.572484 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:06.608896) v4 -- 0x55a327786000 con 0 > -1930> 2018-07-11 07:35:06.572554 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12651 conn(0x55a32169f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1861 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:35:06.608896) v4 > -1929> 2018-07-11 07:35:06.572580 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.2 10.8.128.106:0/12651 1861 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:06.608896) v4 ==== 2004+0+0 (619199962 0 0) 0x55a324474c00 con 0x55a32169f000 > -1928> 2018-07-11 07:35:06.572590 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:06.608896) v4 -- 0x55a332e98e00 con 0 > -1927> 2018-07-11 07:35:06.685243 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a31bcfefc0 con 0 > -1926> 2018-07-11 07:35:06.699969 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5189 0x55a31c870780 replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:494/492,start:6:aa2a39ac:::benchmark_data_magna108_35617_object788246:0,end:6:aa2b0f97:::benchmark_data_magna108_35617_object873893:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -1925> 2018-07-11 07:35:06.700001 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5189 ==== replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:494/492,start:6:aa2a39ac:::benchmark_data_magna108_35617_object788246:0,end:6:aa2b0f97:::benchmark_data_magna108_35617_object873893:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (263147548 0 0) 0x55a31c870780 con 0x55a317019000 > -1924> 2018-07-11 07:35:06.745324 7f35498a5700 10 monclient: tick > -1923> 2018-07-11 07:35:06.745337 7f35498a5700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2018-07-11 07:34:36.745335) > -1922> 2018-07-11 07:35:07.060978 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b8800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1814 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:35:06.811868) v4 > -1921> 2018-07-11 07:35:07.061000 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.8 10.8.128.108:0/11967 1814 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:06.811868) v4 ==== 2004+0+0 (4092261798 0 0) 0x55a332e79a00 con 0x55a3170b8800 > -1920> 2018-07-11 07:35:07.061009 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:06.811868) v4 -- 0x55a332faec00 con 0 > -1919> 2018-07-11 07:35:07.061028 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b7000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1814 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:35:06.811868) v4 > -1918> 2018-07-11 07:35:07.061047 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.8 10.8.128.108:0/11967 1814 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:06.811868) v4 ==== 2004+0+0 (4092261798 0 0) 0x55a34de78600 con 0x55a3170b7000 > -1917> 2018-07-11 07:35:07.061058 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:06.811868) v4 -- 0x55a332e84600 con 0 > -1916> 2018-07-11 07:35:07.257656 7f35488a3700 1 -- 10.8.128.107:6800/12665 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55a3244722c0 con 0 > -1915> 2018-07-11 07:35:07.257717 7f35488a3700 1 -- 10.8.128.107:6800/12665 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55a324471080 con 0 > -1914> 2018-07-11 07:35:07.313781 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a33a2d5b00 con 0 > -1913> 2018-07-11 07:35:07.325072 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5190 0x55a31c870a00 replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:494/492,start:6:aa2b0f97:::benchmark_data_magna108_35617_object873893:0,end:6:aa2c157c:::benchmark_data_magna108.ceph.redhat.com_2478_object214393:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -1912> 2018-07-11 07:35:07.325105 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5190 ==== replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:494/492,start:6:aa2b0f97:::benchmark_data_magna108_35617_object873893:0,end:6:aa2c157c:::benchmark_data_magna108.ceph.redhat.com_2478_object214393:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (1779319929 0 0) 0x55a31c870a00 con 0x55a317019000 > -1911> 2018-07-11 07:35:07.886058 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12650 conn(0x55a311b67000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1878 0x55a32eb3aa00 osd_ping(ping e494 stamp 2018-07-11 07:35:07.922505) v4 > -1910> 2018-07-11 07:35:07.886080 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.3 10.8.128.106:0/12650 1878 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:07.922505) v4 ==== 2004+0+0 (952530631 0 0) 0x55a32eb3aa00 con 0x55a311b67000 > -1909> 2018-07-11 07:35:07.886080 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12650 conn(0x55a311b68800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1878 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:35:07.922505) v4 > -1908> 2018-07-11 07:35:07.886090 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:07.922505) v4 -- 0x55a3177a8c00 con 0 > -1907> 2018-07-11 07:35:07.886098 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.3 10.8.128.106:0/12650 1878 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:07.922505) v4 ==== 2004+0+0 (952530631 0 0) 0x55a34de6f000 con 0x55a311b68800 > -1906> 2018-07-11 07:35:07.886115 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:07.922505) v4 -- 0x55a332e84600 con 0 > -1905> 2018-07-11 07:35:08.000737 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a325d96000 con 0 > -1904> 2018-07-11 07:35:08.017009 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5191 0x55a31c870c80 replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:494/492,start:6:aa2c157c:::benchmark_data_magna108.ceph.redhat.com_2478_object214393:0,end:6:aa2d1cea:::benchmark_data_magna108_35617_object632010:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -1903> 2018-07-11 07:35:08.017040 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5191 ==== replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:494/492,start:6:aa2c157c:::benchmark_data_magna108.ceph.redhat.com_2478_object214393:0,end:6:aa2d1cea:::benchmark_data_magna108_35617_object632010:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (219387154 0 0) 0x55a31c870c80 con 0x55a317019000 > -1902> 2018-07-11 07:35:08.273002 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12651 conn(0x55a3216a0800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1862 0x55a327786000 osd_ping(ping e494 stamp 2018-07-11 07:35:08.309484) v4 > -1901> 2018-07-11 07:35:08.273025 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.2 10.8.128.106:0/12651 1862 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:08.309484) v4 ==== 2004+0+0 (217298654 0 0) 0x55a327786000 con 0x55a3216a0800 > -1900> 2018-07-11 07:35:08.273026 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12651 conn(0x55a32169f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1862 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:35:08.309484) v4 > -1899> 2018-07-11 07:35:08.273037 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:08.309484) v4 -- 0x55a3177a8c00 con 0 > -1898> 2018-07-11 07:35:08.273049 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.2 10.8.128.106:0/12651 1862 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:08.309484) v4 ==== 2004+0+0 (217298654 0 0) 0x55a324474c00 con 0x55a32169f000 > -1897> 2018-07-11 07:35:08.273068 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:08.309484) v4 -- 0x55a332faec00 con 0 > -1896> 2018-07-11 07:35:08.463178 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15919 conn(0x55a3170b4000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1863 0x55a32eb3aa00 osd_ping(ping e494 stamp 2018-07-11 07:35:09.087398) v4 > -1895> 2018-07-11 07:35:08.463183 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15919 conn(0x55a317074800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1863 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:35:09.087398) v4 > -1894> 2018-07-11 07:35:08.463198 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.11 10.8.128.114:0/15919 1863 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:09.087398) v4 ==== 2004+0+0 (104222970 0 0) 0x55a32eb3aa00 con 0x55a3170b4000 > -1893> 2018-07-11 07:35:08.463208 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:09.087398) v4 -- 0x55a33bff1600 con 0 > -1892> 2018-07-11 07:35:08.463204 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.11 10.8.128.114:0/15919 1863 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:09.087398) v4 ==== 2004+0+0 (104222970 0 0) 0x55a332e79a00 con 0x55a317074800 > -1891> 2018-07-11 07:35:08.463232 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:09.087398) v4 -- 0x55a32eb14c00 con 0 > -1890> 2018-07-11 07:35:08.626967 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15263 conn(0x55a321667000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1860 0x55a327786000 osd_ping(ping e494 stamp 2018-07-11 07:35:09.841803) v4 > -1889> 2018-07-11 07:35:08.626997 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.5 10.8.128.113:0/15263 1860 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:09.841803) v4 ==== 2004+0+0 (458411433 0 0) 0x55a327786000 con 0x55a321667000 > -1888> 2018-07-11 07:35:08.627009 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:09.841803) v4 -- 0x55a32eb28c00 con 0 > -1887> 2018-07-11 07:35:08.627055 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15263 conn(0x55a321664000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1860 0x55a327786000 osd_ping(ping e494 stamp 2018-07-11 07:35:09.841803) v4 > -1886> 2018-07-11 07:35:08.627068 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.5 10.8.128.113:0/15263 1860 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:09.841803) v4 ==== 2004+0+0 (458411433 0 0) 0x55a327786000 con 0x55a321664000 > -1885> 2018-07-11 07:35:08.627078 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:09.841803) v4 -- 0x55a32f761800 con 0 > -1884> 2018-07-11 07:35:08.936306 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15255 conn(0x55a311b65800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1888 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:35:10.151217) v4 > -1883> 2018-07-11 07:35:08.936312 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15255 conn(0x55a311b64000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1888 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:35:10.151217) v4 > -1882> 2018-07-11 07:35:08.936339 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.0 10.8.128.113:0/15255 1888 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:10.151217) v4 ==== 2004+0+0 (68110311 0 0) 0x55a34de78600 con 0x55a311b64000 > -1881> 2018-07-11 07:35:08.936336 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.0 10.8.128.113:0/15255 1888 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:10.151217) v4 ==== 2004+0+0 (68110311 0 0) 0x55a324474c00 con 0x55a311b65800 > -1880> 2018-07-11 07:35:08.936356 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:10.151217) v4 -- 0x55a32eb15400 con 0 > -1879> 2018-07-11 07:35:08.936381 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:10.151217) v4 -- 0x55a332ffa800 con 0 > -1878> 2018-07-11 07:35:08.963620 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15919 conn(0x55a317074800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1864 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:35:09.587835) v4 > -1877> 2018-07-11 07:35:08.963652 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.11 10.8.128.114:0/15919 1864 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:09.587835) v4 ==== 2004+0+0 (3887232456 0 0) 0x55a332e79a00 con 0x55a317074800 > -1876> 2018-07-11 07:35:08.963661 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:09.587835) v4 -- 0x55a332fae200 con 0 > -1875> 2018-07-11 07:35:08.963686 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15919 conn(0x55a3170b4000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1864 0x55a32eb3aa00 osd_ping(ping e494 stamp 2018-07-11 07:35:09.587835) v4 > -1874> 2018-07-11 07:35:08.963708 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.11 10.8.128.114:0/15919 1864 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:09.587835) v4 ==== 2004+0+0 (3887232456 0 0) 0x55a32eb3aa00 con 0x55a3170b4000 > -1873> 2018-07-11 07:35:08.963719 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:09.587835) v4 -- 0x55a32eb28c00 con 0 > -1872> 2018-07-11 07:35:09.037702 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12677 conn(0x55a32169d800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1884 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:35:09.074179) v4 > -1871> 2018-07-11 07:35:09.037704 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12677 conn(0x55a32169c000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1884 0x55a327786000 osd_ping(ping e494 stamp 2018-07-11 07:35:09.074179) v4 > -1870> 2018-07-11 07:35:09.037731 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.1 10.8.128.106:0/12677 1884 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:09.074179) v4 ==== 2004+0+0 (3071282763 0 0) 0x55a34de6f000 con 0x55a32169d800 > -1869> 2018-07-11 07:35:09.037741 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:09.074179) v4 -- 0x55a32eb15400 con 0 > -1868> 2018-07-11 07:35:09.037738 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.1 10.8.128.106:0/12677 1884 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:09.074179) v4 ==== 2004+0+0 (3071282763 0 0) 0x55a327786000 con 0x55a32169c000 > -1867> 2018-07-11 07:35:09.037766 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:09.074179) v4 -- 0x55a32eb28c00 con 0 > -1866> 2018-07-11 07:35:09.337581 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15287 conn(0x55a311b72000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1860 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:35:10.552462) v4 > -1865> 2018-07-11 07:35:09.337602 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.4 10.8.128.113:0/15287 1860 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:10.552462) v4 ==== 2004+0+0 (2682509391 0 0) 0x55a324474c00 con 0x55a311b72000 > -1864> 2018-07-11 07:35:09.337611 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:10.552462) v4 -- 0x55a332fae200 con 0 > -1863> 2018-07-11 07:35:09.337623 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15287 conn(0x55a311b75000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1860 0x55a32eb3aa00 osd_ping(ping e494 stamp 2018-07-11 07:35:10.552462) v4 > -1862> 2018-07-11 07:35:09.337645 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.4 10.8.128.113:0/15287 1860 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:10.552462) v4 ==== 2004+0+0 (2682509391 0 0) 0x55a32eb3aa00 con 0x55a311b75000 > -1861> 2018-07-11 07:35:09.337656 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:10.552462) v4 -- 0x55a32eb28c00 con 0 > -1860> 2018-07-11 07:35:09.375874 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15885 conn(0x55a3170b5800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1885 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:35:10.000016) v4 > -1859> 2018-07-11 07:35:09.375878 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15885 conn(0x55a311b76800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1885 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:35:10.000016) v4 > -1858> 2018-07-11 07:35:09.375896 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.7 10.8.128.114:0/15885 1885 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:10.000016) v4 ==== 2004+0+0 (3777654119 0 0) 0x55a332e79a00 con 0x55a3170b5800 > -1857> 2018-07-11 07:35:09.375897 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.7 10.8.128.114:0/15885 1885 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:10.000016) v4 ==== 2004+0+0 (3777654119 0 0) 0x55a34de78600 con 0x55a311b76800 > -1856> 2018-07-11 07:35:09.375911 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:10.000016) v4 -- 0x55a332fae200 con 0 > -1855> 2018-07-11 07:35:09.375934 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:10.000016) v4 -- 0x55a32eb15400 con 0 > -1854> 2018-07-11 07:35:09.464206 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15919 conn(0x55a317074800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1865 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:35:10.088352) v4 > -1853> 2018-07-11 07:35:09.464227 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.11 10.8.128.114:0/15919 1865 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:10.088352) v4 ==== 2004+0+0 (882648789 0 0) 0x55a324474c00 con 0x55a317074800 > -1852> 2018-07-11 07:35:09.464236 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:10.088352) v4 -- 0x55a32eb20400 con 0 > -1851> 2018-07-11 07:35:09.464223 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15919 conn(0x55a3170b4000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1865 0x55a327786000 osd_ping(ping e494 stamp 2018-07-11 07:35:10.088352) v4 > -1850> 2018-07-11 07:35:09.464257 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.11 10.8.128.114:0/15919 1865 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:10.088352) v4 ==== 2004+0+0 (882648789 0 0) 0x55a327786000 con 0x55a3170b4000 > -1849> 2018-07-11 07:35:09.464275 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:10.088352) v4 -- 0x55a32eb20600 con 0 > -1848> 2018-07-11 07:35:09.682060 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a33cdd8fc0 con 0 > -1847> 2018-07-11 07:35:09.692222 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5192 0x55a31c870f00 replica scrub(pg: 6.15,from:0'0,to:430'105466,epoch:494/492,start:6:aa2d1cea:::benchmark_data_magna108_35617_object632010:0,end:6:aa2e0057:::benchmark_data_magna108_35617_object385662:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -1846> 2018-07-11 07:35:09.692258 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5192 ==== replica scrub(pg: 6.15,from:0'0,to:430'105466,epoch:494/492,start:6:aa2d1cea:::benchmark_data_magna108_35617_object632010:0,end:6:aa2e0057:::benchmark_data_magna108_35617_object385662:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (1236998751 0 0) 0x55a31c870f00 con 0x55a317019000 > -1845> 2018-07-11 07:35:09.845550 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 61] Generated table #36594: 4957167 keys, 80348309 bytes > -1844> 2018-07-11 07:35:09.845602 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294509845573, "cf_name": "default", "job": 61, "event": "table_file_creation", "file_number": 36594, "file_size": 80348309, "table_properties": {"data_size": 67109393, "index_size": 689219, "filter_size": 12548707, "raw_key_size": 143207044, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957167, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957167", "kMergeOperands": "0"}} > -1843> 2018-07-11 07:35:09.858384 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.115:0/13323 conn(0x55a321665800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1872 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:35:10.196370) v4 > -1842> 2018-07-11 07:35:09.858416 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.6 10.8.128.115:0/13323 1872 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:10.196370) v4 ==== 2004+0+0 (3921141077 0 0) 0x55a332e79a00 con 0x55a321665800 > -1841> 2018-07-11 07:35:09.858436 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:10.196370) v4 -- 0x55a32eb20400 con 0 > -1840> 2018-07-11 07:35:09.858527 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.115:0/13323 conn(0x55a321668800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1872 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:35:10.196370) v4 > -1839> 2018-07-11 07:35:09.858543 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.6 10.8.128.115:0/13323 1872 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:10.196370) v4 ==== 2004+0+0 (3921141077 0 0) 0x55a324474c00 con 0x55a321668800 > -1838> 2018-07-11 07:35:09.858553 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:10.196370) v4 -- 0x55a32eb20400 con 0 > -1837> 2018-07-11 07:35:09.890416 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.107:0/12617 conn(0x55a32447e000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1837 0x55a32eb3aa00 osd_ping(ping e494 stamp 2018-07-11 07:35:09.889916) v4 > -1836> 2018-07-11 07:35:09.890438 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.9 10.8.128.107:0/12617 1837 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:09.889916) v4 ==== 2004+0+0 (1575331986 0 0) 0x55a32eb3aa00 con 0x55a32447e000 > -1835> 2018-07-11 07:35:09.890448 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:09.889916) v4 -- 0x55a327789600 con 0 > -1834> 2018-07-11 07:35:09.890435 7f355d88e700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.107:0/12617 conn(0x55a32447f800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1837 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:35:09.889916) v4 > -1833> 2018-07-11 07:35:09.890495 7f355d88e700 1 -- 10.8.128.107:6803/12665 <== osd.9 10.8.128.107:0/12617 1837 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:09.889916) v4 ==== 2004+0+0 (1575331986 0 0) 0x55a34de6f000 con 0x55a32447f800 > -1832> 2018-07-11 07:35:09.890516 7f355d88e700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:09.889916) v4 -- 0x55a330935200 con 0 > -1831> 2018-07-11 07:35:10.503421 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a347b838c0 con 0 > -1830> 2018-07-11 07:35:10.533742 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5193 0x55a31c871180 replica scrub(pg: 6.15,from:0'0,to:430'106647,epoch:494/492,start:6:aa2e0057:::benchmark_data_magna108_35617_object385662:0,end:6:aa2edfcc:::benchmark_data_magna108.ceph.redhat.com_2478_object127856:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -1829> 2018-07-11 07:35:10.533768 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5193 ==== replica scrub(pg: 6.15,from:0'0,to:430'106647,epoch:494/492,start:6:aa2e0057:::benchmark_data_magna108_35617_object385662:0,end:6:aa2edfcc:::benchmark_data_magna108.ceph.redhat.com_2478_object127856:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (123739560 0 0) 0x55a31c871180 con 0x55a317019000 > -1828> 2018-07-11 07:35:10.636810 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15255 conn(0x55a311b65800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1889 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:35:11.851645) v4 > -1827> 2018-07-11 07:35:10.636832 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.0 10.8.128.113:0/15255 1889 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:11.851645) v4 ==== 2004+0+0 (328043375 0 0) 0x55a332e79a00 con 0x55a311b65800 > -1826> 2018-07-11 07:35:10.636844 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:11.851645) v4 -- 0x55a32eb20400 con 0 > -1825> 2018-07-11 07:35:10.636869 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15255 conn(0x55a311b64000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1889 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:35:11.851645) v4 > -1824> 2018-07-11 07:35:10.636890 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.0 10.8.128.113:0/15255 1889 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:11.851645) v4 ==== 2004+0+0 (328043375 0 0) 0x55a34de78600 con 0x55a311b64000 > -1823> 2018-07-11 07:35:10.636905 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:11.851645) v4 -- 0x55a330935200 con 0 > -1822> 2018-07-11 07:35:10.786384 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12650 conn(0x55a311b68800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1879 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:35:10.822892) v4 > -1821> 2018-07-11 07:35:10.786384 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12650 conn(0x55a311b67000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1879 0x55a32eb20600 osd_ping(ping e494 stamp 2018-07-11 07:35:10.822892) v4 > -1820> 2018-07-11 07:35:10.786415 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.3 10.8.128.106:0/12650 1879 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:10.822892) v4 ==== 2004+0+0 (2634260162 0 0) 0x55a34de6f000 con 0x55a311b68800 > -1819> 2018-07-11 07:35:10.786419 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.3 10.8.128.106:0/12650 1879 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:10.822892) v4 ==== 2004+0+0 (2634260162 0 0) 0x55a32eb20600 con 0x55a311b67000 > -1818> 2018-07-11 07:35:10.786435 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:10.822892) v4 -- 0x55a330935200 con 0 > -1817> 2018-07-11 07:35:10.786468 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:10.822892) v4 -- 0x55a327788e00 con 0 > -1816> 2018-07-11 07:35:10.927277 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15263 conn(0x55a321664000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1861 0x55a327789600 osd_ping(ping e494 stamp 2018-07-11 07:35:12.142192) v4 > -1815> 2018-07-11 07:35:10.927299 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.5 10.8.128.113:0/15263 1861 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:12.142192) v4 ==== 2004+0+0 (2282815056 0 0) 0x55a327789600 con 0x55a321664000 > -1814> 2018-07-11 07:35:10.927309 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:12.142192) v4 -- 0x55a327788e00 con 0 > -1813> 2018-07-11 07:35:10.927349 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15263 conn(0x55a321667000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1861 0x55a327789600 osd_ping(ping e494 stamp 2018-07-11 07:35:12.142192) v4 > -1812> 2018-07-11 07:35:10.927363 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.5 10.8.128.113:0/15263 1861 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:12.142192) v4 ==== 2004+0+0 (2282815056 0 0) 0x55a327789600 con 0x55a321667000 > -1811> 2018-07-11 07:35:10.927371 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:12.142192) v4 -- 0x55a30f773a00 con 0 > -1810> 2018-07-11 07:35:10.990996 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.107:0/12617 conn(0x55a32447e000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1838 0x55a32eb20600 osd_ping(ping e494 stamp 2018-07-11 07:35:10.990555) v4 > -1809> 2018-07-11 07:35:10.991003 7f355d88e700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.107:0/12617 conn(0x55a32447f800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1838 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:35:10.990555) v4 > -1808> 2018-07-11 07:35:10.991027 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.9 10.8.128.107:0/12617 1838 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:10.990555) v4 ==== 2004+0+0 (3520840848 0 0) 0x55a32eb20600 con 0x55a32447e000 > -1807> 2018-07-11 07:35:10.991034 7f355d88e700 1 -- 10.8.128.107:6803/12665 <== osd.9 10.8.128.107:0/12617 1838 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:10.990555) v4 ==== 2004+0+0 (3520840848 0 0) 0x55a34de78600 con 0x55a32447f800 > -1806> 2018-07-11 07:35:10.991045 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:10.990555) v4 -- 0x55a327788e00 con 0 > -1805> 2018-07-11 07:35:10.991094 7f355d88e700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:10.990555) v4 -- 0x55a332e85000 con 0 > -1804> 2018-07-11 07:35:11.137286 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15255 conn(0x55a311b65800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1890 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:35:12.352239) v4 > -1803> 2018-07-11 07:35:11.137313 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.0 10.8.128.113:0/15255 1890 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:12.352239) v4 ==== 2004+0+0 (688490045 0 0) 0x55a324474c00 con 0x55a311b65800 > -1802> 2018-07-11 07:35:11.137312 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15255 conn(0x55a311b64000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1890 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:35:12.352239) v4 > -1801> 2018-07-11 07:35:11.137327 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:12.352239) v4 -- 0x55a33234ea00 con 0 > -1800> 2018-07-11 07:35:11.137329 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.0 10.8.128.113:0/15255 1890 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:12.352239) v4 ==== 2004+0+0 (688490045 0 0) 0x55a34de6f000 con 0x55a311b64000 > -1799> 2018-07-11 07:35:11.137352 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:12.352239) v4 -- 0x55a32e331e00 con 0 > -1798> 2018-07-11 07:35:11.360197 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a325da6b40 con 0 > -1797> 2018-07-11 07:35:11.375484 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5194 0x55a31c871400 replica scrub(pg: 6.15,from:0'0,to:430'106662,epoch:494/492,start:6:aa2edfcc:::benchmark_data_magna108.ceph.redhat.com_2478_object127856:0,end:6:aa2fd813:::benchmark_data_magna108_35617_object790256:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -1796> 2018-07-11 07:35:11.375510 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5194 ==== replica scrub(pg: 6.15,from:0'0,to:430'106662,epoch:494/492,start:6:aa2edfcc:::benchmark_data_magna108.ceph.redhat.com_2478_object127856:0,end:6:aa2fd813:::benchmark_data_magna108_35617_object790256:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (1572733738 0 0) 0x55a31c871400 con 0x55a317019000 > -1795> 2018-07-11 07:35:11.580849 7f3543899700 5 osd.10 494 heartbeat: osd_stat(784 GB used, 146 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,1]) > -1794> 2018-07-11 07:35:11.580863 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6802/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:35:11.580861) v4 -- 0x55a349f88200 con 0 > -1793> 2018-07-11 07:35:11.580901 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6803/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:35:11.580861) v4 -- 0x55a32442a600 con 0 > -1792> 2018-07-11 07:35:11.580916 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6806/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:35:11.580861) v4 -- 0x55a349f88400 con 0 > -1791> 2018-07-11 07:35:11.580933 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6807/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:35:11.580861) v4 -- 0x55a332feb000 con 0 > -1790> 2018-07-11 07:35:11.580956 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6808/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:35:11.580861) v4 -- 0x55a332fae600 con 0 > -1789> 2018-07-11 07:35:11.580968 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6810/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:35:11.580861) v4 -- 0x55a30f772000 con 0 > -1788> 2018-07-11 07:35:11.580982 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6809/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:35:11.580861) v4 -- 0x55a34da1b600 con 0 > -1787> 2018-07-11 07:35:11.580996 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6811/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:35:11.580861) v4 -- 0x55a34da1b200 con 0 > -1786> 2018-07-11 07:35:11.581008 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6810/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:35:11.580861) v4 -- 0x55a30f772800 con 0 > -1785> 2018-07-11 07:35:11.581023 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6811/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:35:11.580861) v4 -- 0x55a34da1b400 con 0 > -1784> 2018-07-11 07:35:11.581036 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6806/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:35:11.580861) v4 -- 0x55a332349400 con 0 > -1783> 2018-07-11 07:35:11.581054 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6807/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:35:11.580861) v4 -- 0x55a3314ce200 con 0 > -1782> 2018-07-11 07:35:11.581071 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6802/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:35:11.580861) v4 -- 0x55a331d5c000 con 0 > -1781> 2018-07-11 07:35:11.581084 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6803/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:35:11.580861) v4 -- 0x55a332fe8000 con 0 > -1780> 2018-07-11 07:35:11.581101 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6806/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:35:11.580861) v4 -- 0x55a331d5d400 con 0 > -1779> 2018-07-11 07:35:11.581114 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6807/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:35:11.580861) v4 -- 0x55a327788600 con 0 > -1778> 2018-07-11 07:35:11.581127 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6802/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:35:11.580861) v4 -- 0x55a331d5da00 con 0 > -1777> 2018-07-11 07:35:11.581139 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6803/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:35:11.580861) v4 -- 0x55a34da1a400 con 0 > -1776> 2018-07-11 07:35:11.581152 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6806/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:35:11.580861) v4 -- 0x55a327788400 con 0 > -1775> 2018-07-11 07:35:11.581165 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6807/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:35:11.580861) v4 -- 0x55a323f55a00 con 0 > -1774> 2018-07-11 07:35:11.581178 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6802/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:35:11.580861) v4 -- 0x55a332ff3600 con 0 > -1773> 2018-07-11 07:35:11.581197 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6803/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:35:11.580861) v4 -- 0x55a323f55c00 con 0 > -1772> 2018-07-11 07:35:11.581367 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6803/15255 conn(0x55a314e1d800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1869 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:11.580861) v4 > -1771> 2018-07-11 07:35:11.581389 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6803/15255 1869 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:11.580861) v4 ==== 2004+0+0 (3280728590 0 0) 0x55a324474c00 con 0x55a314e1d800 > -1770> 2018-07-11 07:35:11.581402 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6807/12617 conn(0x55a314e27800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1869 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:11.580861) v4 > -1769> 2018-07-11 07:35:11.581427 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6807/12617 1869 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:11.580861) v4 ==== 2004+0+0 (3280728590 0 0) 0x55a34de78600 con 0x55a314e27800 > -1768> 2018-07-11 07:35:11.581454 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6807/12677 conn(0x55a314e85800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1869 0x55a327789600 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:11.580861) v4 > -1767> 2018-07-11 07:35:11.581454 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6806/12617 conn(0x55a314e26000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1869 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:11.580861) v4 > -1766> 2018-07-11 07:35:11.581477 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6807/12677 1869 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:11.580861) v4 ==== 2004+0+0 (3280728590 0 0) 0x55a327789600 con 0x55a314e85800 > -1765> 2018-07-11 07:35:11.581483 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6806/12617 1869 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:11.580861) v4 ==== 2004+0+0 (3280728590 0 0) 0x55a324474c00 con 0x55a314e26000 > -1764> 2018-07-11 07:35:11.581501 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6806/12677 conn(0x55a314e84000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1869 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:11.580861) v4 > -1763> 2018-07-11 07:35:11.581518 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6806/12677 1869 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:11.580861) v4 ==== 2004+0+0 (3280728590 0 0) 0x55a34de78600 con 0x55a314e84000 > -1762> 2018-07-11 07:35:11.581533 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6808/12651 conn(0x55a314e96000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1869 0x55a327789600 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:11.580861) v4 > -1761> 2018-07-11 07:35:11.581554 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6808/12651 1869 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:11.580861) v4 ==== 2004+0+0 (3280728590 0 0) 0x55a327789600 con 0x55a314e96000 > -1760> 2018-07-11 07:35:11.581559 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6810/12651 conn(0x55a314e97800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1869 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:11.580861) v4 > -1759> 2018-07-11 07:35:11.581572 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6809/12650 conn(0x55a314e1f000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1869 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:11.580861) v4 > -1758> 2018-07-11 07:35:11.581575 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6810/12651 1869 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:11.580861) v4 ==== 2004+0+0 (3280728590 0 0) 0x55a324474c00 con 0x55a314e97800 > -1757> 2018-07-11 07:35:11.581585 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6809/12650 1869 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:11.580861) v4 ==== 2004+0+0 (3280728590 0 0) 0x55a34de78600 con 0x55a314e1f000 > -1756> 2018-07-11 07:35:11.581585 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6802/15255 conn(0x55a314e1c000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1869 0x55a327789600 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:11.580861) v4 > -1755> 2018-07-11 07:35:11.581598 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6802/15255 1869 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:11.580861) v4 ==== 2004+0+0 (3280728590 0 0) 0x55a327789600 con 0x55a314e1c000 > -1754> 2018-07-11 07:35:11.581621 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6806/15263 conn(0x55a314e99000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1869 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:11.580861) v4 > -1753> 2018-07-11 07:35:11.581632 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6806/15263 1869 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:11.580861) v4 ==== 2004+0+0 (3280728590 0 0) 0x55a34de78600 con 0x55a314e99000 > -1752> 2018-07-11 07:35:11.581644 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6807/15263 conn(0x55a314e9a800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1869 0x55a327789600 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:11.580861) v4 > -1751> 2018-07-11 07:35:11.581664 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6803/13323 conn(0x55a314e16800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1869 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:11.580861) v4 > -1750> 2018-07-11 07:35:11.581663 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6807/15263 1869 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:11.580861) v4 ==== 2004+0+0 (3280728590 0 0) 0x55a327789600 con 0x55a314e9a800 > -1749> 2018-07-11 07:35:11.581680 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6803/13323 1869 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:11.580861) v4 ==== 2004+0+0 (3280728590 0 0) 0x55a34de78600 con 0x55a314e16800 > -1748> 2018-07-11 07:35:11.581702 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6811/15287 conn(0x55a314e04800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1869 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:11.580861) v4 > -1747> 2018-07-11 07:35:11.581711 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6802/11967 conn(0x55a314e12000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1869 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:11.580861) v4 > -1746> 2018-07-11 07:35:11.581727 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6811/15287 1869 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:11.580861) v4 ==== 2004+0+0 (3280728590 0 0) 0x55a324474c00 con 0x55a314e04800 > -1745> 2018-07-11 07:35:11.581736 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6810/15287 conn(0x55a314e03000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1869 0x55a327789600 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:11.580861) v4 > -1744> 2018-07-11 07:35:11.581734 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6802/11967 1869 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:11.580861) v4 ==== 2004+0+0 (3280728590 0 0) 0x55a34de78600 con 0x55a314e12000 > -1743> 2018-07-11 07:35:11.581752 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6810/15287 1869 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:11.580861) v4 ==== 2004+0+0 (3280728590 0 0) 0x55a327789600 con 0x55a314e03000 > -1742> 2018-07-11 07:35:11.581779 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6802/15919 conn(0x55a314e87000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1869 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:11.580861) v4 > -1741> 2018-07-11 07:35:11.581791 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6807/15885 conn(0x55a314e01800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1869 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:11.580861) v4 > -1740> 2018-07-11 07:35:11.581802 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6807/15885 1869 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:11.580861) v4 ==== 2004+0+0 (3280728590 0 0) 0x55a34de78600 con 0x55a314e01800 > -1739> 2018-07-11 07:35:11.581799 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6802/15919 1869 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:11.580861) v4 ==== 2004+0+0 (3280728590 0 0) 0x55a324474c00 con 0x55a314e87000 > -1738> 2018-07-11 07:35:11.581803 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6811/12650 conn(0x55a314e20800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1869 0x55a327789600 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:11.580861) v4 > -1737> 2018-07-11 07:35:11.581818 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6811/12650 1869 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:11.580861) v4 ==== 2004+0+0 (3280728590 0 0) 0x55a327789600 con 0x55a314e20800 > -1736> 2018-07-11 07:35:11.581834 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6803/15919 conn(0x55a314e88800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1869 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:11.580861) v4 > -1735> 2018-07-11 07:35:11.581846 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6803/15919 1869 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:11.580861) v4 ==== 2004+0+0 (3280728590 0 0) 0x55a34de78600 con 0x55a314e88800 > -1734> 2018-07-11 07:35:11.581847 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6802/13323 conn(0x55a314e15000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1869 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:11.580861) v4 > -1733> 2018-07-11 07:35:11.581865 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6803/11967 conn(0x55a314e13800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1869 0x55a327789600 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:11.580861) v4 > -1732> 2018-07-11 07:35:11.581873 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6802/13323 1869 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:11.580861) v4 ==== 2004+0+0 (3280728590 0 0) 0x55a324474c00 con 0x55a314e15000 > -1731> 2018-07-11 07:35:11.581882 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6803/11967 1869 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:11.580861) v4 ==== 2004+0+0 (3280728590 0 0) 0x55a327789600 con 0x55a314e13800 > -1730> 2018-07-11 07:35:11.581916 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6806/15885 conn(0x55a314e00000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1869 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:11.580861) v4 > -1729> 2018-07-11 07:35:11.581934 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6806/15885 1869 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:11.580861) v4 ==== 2004+0+0 (3280728590 0 0) 0x55a324474c00 con 0x55a314e00000 > -1728> 2018-07-11 07:35:11.676136 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15885 conn(0x55a311b76800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1886 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:35:12.300416) v4 > -1727> 2018-07-11 07:35:11.676152 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.7 10.8.128.114:0/15885 1886 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:12.300416) v4 ==== 2004+0+0 (1441969098 0 0) 0x55a34de78600 con 0x55a311b76800 > -1726> 2018-07-11 07:35:11.676161 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:12.300416) v4 -- 0x55a349f88400 con 0 > -1725> 2018-07-11 07:35:11.676173 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15885 conn(0x55a3170b5800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1886 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:35:12.300416) v4 > -1724> 2018-07-11 07:35:11.676198 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.7 10.8.128.114:0/15885 1886 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:12.300416) v4 ==== 2004+0+0 (1441969098 0 0) 0x55a324474c00 con 0x55a3170b5800 > -1723> 2018-07-11 07:35:11.676209 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:12.300416) v4 -- 0x55a32442a600 con 0 > -1722> 2018-07-11 07:35:11.792322 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 61] Generated table #36595: 1042833 keys, 16903140 bytes > -1721> 2018-07-11 07:35:11.792352 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294511792337, "cf_name": "default", "job": 61, "event": "table_file_creation", "file_number": 36595, "file_size": 16903140, "table_properties": {"data_size": 14117724, "index_size": 144573, "filter_size": 2639856, "raw_key_size": 30126296, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 3465, "num_entries": 1042833, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1042833", "kMergeOperands": "0"}} > -1720> 2018-07-11 07:35:12.257926 7f35488a3700 1 -- 10.8.128.107:6800/12665 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55a32a5c18c0 con 0 > -1719> 2018-07-11 07:35:12.257991 7f35488a3700 1 -- 10.8.128.107:6800/12665 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55a30f75edc0 con 0 > -1718> 2018-07-11 07:35:12.323563 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a31e493200 con 0 > -1717> 2018-07-11 07:35:12.352691 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5195 0x55a31c871680 replica scrub(pg: 6.15,from:0'0,to:461'106921,epoch:494/492,start:6:aa2fd813:::benchmark_data_magna108_35617_object790256:0,end:6:aa30cb8b:::benchmark_data_magna108_35617_object1029722:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -1716> 2018-07-11 07:35:12.352723 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5195 ==== replica scrub(pg: 6.15,from:0'0,to:461'106921,epoch:494/492,start:6:aa2fd813:::benchmark_data_magna108_35617_object790256:0,end:6:aa30cb8b:::benchmark_data_magna108_35617_object1029722:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 219+0+0 (3379546955 0 0) 0x55a31c871680 con 0x55a317019000 > -1715> 2018-07-11 07:35:12.776556 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15885 conn(0x55a311b76800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1887 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:35:13.400821) v4 > -1714> 2018-07-11 07:35:12.776579 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.7 10.8.128.114:0/15885 1887 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:13.400821) v4 ==== 2004+0+0 (2402775384 0 0) 0x55a34de6f000 con 0x55a311b76800 > -1713> 2018-07-11 07:35:12.776573 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15885 conn(0x55a3170b5800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1887 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:35:13.400821) v4 > -1712> 2018-07-11 07:35:12.776593 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:13.400821) v4 -- 0x55a349f88400 con 0 > -1711> 2018-07-11 07:35:12.776592 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.7 10.8.128.114:0/15885 1887 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:13.400821) v4 ==== 2004+0+0 (2402775384 0 0) 0x55a332e79a00 con 0x55a3170b5800 > -1710> 2018-07-11 07:35:12.776624 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:13.400821) v4 -- 0x55a32442a600 con 0 > -1709> 2018-07-11 07:35:12.961543 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b7000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1815 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:35:12.712491) v4 > -1708> 2018-07-11 07:35:12.961565 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.8 10.8.128.108:0/11967 1815 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:12.712491) v4 ==== 2004+0+0 (2189929101 0 0) 0x55a34de78600 con 0x55a3170b7000 > -1707> 2018-07-11 07:35:12.961565 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b8800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1815 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:35:12.712491) v4 > -1706> 2018-07-11 07:35:12.961578 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:12.712491) v4 -- 0x55a32e331e00 con 0 > -1705> 2018-07-11 07:35:12.961582 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.8 10.8.128.108:0/11967 1815 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:12.712491) v4 ==== 2004+0+0 (2189929101 0 0) 0x55a324474c00 con 0x55a3170b8800 > -1704> 2018-07-11 07:35:12.961596 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:12.712491) v4 -- 0x55a33234ea00 con 0 > -1703> 2018-07-11 07:35:12.973374 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12651 conn(0x55a3216a0800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1863 0x55a327789600 osd_ping(ping e494 stamp 2018-07-11 07:35:13.009903) v4 > -1702> 2018-07-11 07:35:12.973397 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.2 10.8.128.106:0/12651 1863 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:13.009903) v4 ==== 2004+0+0 (2248527596 0 0) 0x55a327789600 con 0x55a3216a0800 > -1701> 2018-07-11 07:35:12.973407 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:13.009903) v4 -- 0x55a30f772e00 con 0 > -1700> 2018-07-11 07:35:12.973500 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12651 conn(0x55a32169f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1863 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:35:13.009903) v4 > -1699> 2018-07-11 07:35:12.973525 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.2 10.8.128.106:0/12651 1863 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:13.009903) v4 ==== 2004+0+0 (2248527596 0 0) 0x55a332e79a00 con 0x55a32169f000 > -1698> 2018-07-11 07:35:12.973541 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:13.009903) v4 -- 0x55a33234fe00 con 0 > -1697> 2018-07-11 07:35:13.151060 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a3333c6b40 con 0 > -1696> 2018-07-11 07:35:13.227691 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15263 conn(0x55a321664000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1862 0x55a327789600 osd_ping(ping e494 stamp 2018-07-11 07:35:14.442648) v4 > -1695> 2018-07-11 07:35:13.227719 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.5 10.8.128.113:0/15263 1862 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:14.442648) v4 ==== 2004+0+0 (2405353121 0 0) 0x55a327789600 con 0x55a321664000 > -1694> 2018-07-11 07:35:13.227734 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:14.442648) v4 -- 0x55a30f772600 con 0 > -1693> 2018-07-11 07:35:13.227802 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15263 conn(0x55a321667000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1862 0x55a332feb000 osd_ping(ping e494 stamp 2018-07-11 07:35:14.442648) v4 > -1692> 2018-07-11 07:35:13.227818 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.5 10.8.128.113:0/15263 1862 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:14.442648) v4 ==== 2004+0+0 (2405353121 0 0) 0x55a332feb000 con 0x55a321667000 > -1691> 2018-07-11 07:35:13.227828 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:14.442648) v4 -- 0x55a34da1b000 con 0 > -1690> 2018-07-11 07:35:13.275960 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5196 0x55a31c871900 replica scrub(pg: 6.15,from:0'0,to:430'106688,epoch:494/492,start:6:aa30cb8b:::benchmark_data_magna108_35617_object1029722:0,end:6:aa31ce3e:::benchmark_data_magna108.ceph.redhat.com_2478_object334929:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -1689> 2018-07-11 07:35:13.275990 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5196 ==== replica scrub(pg: 6.15,from:0'0,to:430'106688,epoch:494/492,start:6:aa30cb8b:::benchmark_data_magna108_35617_object1029722:0,end:6:aa31ce3e:::benchmark_data_magna108.ceph.redhat.com_2478_object334929:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 234+0+0 (2175123600 0 0) 0x55a31c871900 con 0x55a317019000 > -1688> 2018-07-11 07:35:13.291558 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.107:0/12617 conn(0x55a32447e000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1839 0x55a332feb000 osd_ping(ping e494 stamp 2018-07-11 07:35:13.291107) v4 > -1687> 2018-07-11 07:35:13.291581 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.9 10.8.128.107:0/12617 1839 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:13.291107) v4 ==== 2004+0+0 (841837424 0 0) 0x55a332feb000 con 0x55a32447e000 > -1686> 2018-07-11 07:35:13.291598 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:13.291107) v4 -- 0x55a30f773000 con 0 > -1685> 2018-07-11 07:35:13.291598 7f355d88e700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.107:0/12617 conn(0x55a32447f800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1839 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:35:13.291107) v4 > -1684> 2018-07-11 07:35:13.291616 7f355d88e700 1 -- 10.8.128.107:6803/12665 <== osd.9 10.8.128.107:0/12617 1839 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:13.291107) v4 ==== 2004+0+0 (841837424 0 0) 0x55a34de6f000 con 0x55a32447f800 > -1683> 2018-07-11 07:35:13.291628 7f355d88e700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:13.291107) v4 -- 0x55a32e331e00 con 0 > -1682> 2018-07-11 07:35:13.728115 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15263 conn(0x55a321664000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1863 0x55a34da1b000 osd_ping(ping e494 stamp 2018-07-11 07:35:14.943079) v4 > -1681> 2018-07-11 07:35:13.728143 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.5 10.8.128.113:0/15263 1863 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:14.943079) v4 ==== 2004+0+0 (1910576490 0 0) 0x55a34da1b000 con 0x55a321664000 > -1680> 2018-07-11 07:35:13.728158 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:14.943079) v4 -- 0x55a30f773000 con 0 > -1679> 2018-07-11 07:35:13.728241 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15263 conn(0x55a321667000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1863 0x55a332feb000 osd_ping(ping e494 stamp 2018-07-11 07:35:14.943079) v4 > -1678> 2018-07-11 07:35:13.728268 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.5 10.8.128.113:0/15263 1863 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:14.943079) v4 ==== 2004+0+0 (1910576490 0 0) 0x55a332feb000 con 0x55a321667000 > -1677> 2018-07-11 07:35:13.728278 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:14.943079) v4 -- 0x55a30f773000 con 0 > -1676> 2018-07-11 07:35:13.738082 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12677 conn(0x55a32169d800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1885 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:35:13.774608) v4 > -1675> 2018-07-11 07:35:13.738092 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12677 conn(0x55a32169c000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1885 0x55a34da1b000 osd_ping(ping e494 stamp 2018-07-11 07:35:13.774608) v4 > -1674> 2018-07-11 07:35:13.738103 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.1 10.8.128.106:0/12677 1885 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:13.774608) v4 ==== 2004+0+0 (1561140112 0 0) 0x55a34de78600 con 0x55a32169d800 > -1673> 2018-07-11 07:35:13.738115 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.1 10.8.128.106:0/12677 1885 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:13.774608) v4 ==== 2004+0+0 (1561140112 0 0) 0x55a34da1b000 con 0x55a32169c000 > -1672> 2018-07-11 07:35:13.738122 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:13.774608) v4 -- 0x55a32e331c00 con 0 > -1671> 2018-07-11 07:35:13.738146 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:13.774608) v4 -- 0x55a30f773c00 con 0 > -1670> 2018-07-11 07:35:13.927214 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a325db6480 con 0 > -1669> 2018-07-11 07:35:13.942976 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5197 0x55a31c871b80 replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:494/492,start:6:aa31ce3e:::benchmark_data_magna108.ceph.redhat.com_2478_object334929:0,end:6:aa32f194:::benchmark_data_magna108.ceph.redhat.com_2478_object222714:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -1668> 2018-07-11 07:35:13.943005 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5197 ==== replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:494/492,start:6:aa31ce3e:::benchmark_data_magna108.ceph.redhat.com_2478_object334929:0,end:6:aa32f194:::benchmark_data_magna108.ceph.redhat.com_2478_object222714:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 248+0+0 (2043658051 0 0) 0x55a31c871b80 con 0x55a317019000 > -1667> 2018-07-11 07:35:14.037975 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15287 conn(0x55a311b72000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1861 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:35:15.252905) v4 > -1666> 2018-07-11 07:35:14.037970 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15287 conn(0x55a311b75000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1861 0x55a332feb000 osd_ping(ping e494 stamp 2018-07-11 07:35:15.252905) v4 > -1665> 2018-07-11 07:35:14.037998 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.4 10.8.128.113:0/15287 1861 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:15.252905) v4 ==== 2004+0+0 (419987351 0 0) 0x55a324474c00 con 0x55a311b72000 > -1664> 2018-07-11 07:35:14.038003 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.4 10.8.128.113:0/15287 1861 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:15.252905) v4 ==== 2004+0+0 (419987351 0 0) 0x55a332feb000 con 0x55a311b75000 > -1663> 2018-07-11 07:35:14.038011 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:15.252905) v4 -- 0x55a33234fe00 con 0 > -1662> 2018-07-11 07:35:14.038037 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:15.252905) v4 -- 0x55a30f772400 con 0 > -1661> 2018-07-11 07:35:14.073937 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12651 conn(0x55a3216a0800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1864 0x55a30f773c00 osd_ping(ping e494 stamp 2018-07-11 07:35:14.110460) v4 > -1660> 2018-07-11 07:35:14.073962 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.2 10.8.128.106:0/12651 1864 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:14.110460) v4 ==== 2004+0+0 (3725517454 0 0) 0x55a30f773c00 con 0x55a3216a0800 > -1659> 2018-07-11 07:35:14.073974 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:14.110460) v4 -- 0x55a30f772400 con 0 > -1658> 2018-07-11 07:35:14.074001 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12651 conn(0x55a32169f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1864 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:35:14.110460) v4 > -1657> 2018-07-11 07:35:14.074021 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.2 10.8.128.106:0/12651 1864 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:14.110460) v4 ==== 2004+0+0 (3725517454 0 0) 0x55a332e79a00 con 0x55a32169f000 > -1656> 2018-07-11 07:35:14.074032 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:14.110460) v4 -- 0x55a33234fe00 con 0 > -1655> 2018-07-11 07:35:14.164495 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15919 conn(0x55a3170b4000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1866 0x55a332feb000 osd_ping(ping e494 stamp 2018-07-11 07:35:14.788755) v4 > -1654> 2018-07-11 07:35:14.164502 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15919 conn(0x55a317074800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1866 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:35:14.788755) v4 > -1653> 2018-07-11 07:35:14.164519 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.11 10.8.128.114:0/15919 1866 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:14.788755) v4 ==== 2004+0+0 (1044365507 0 0) 0x55a332feb000 con 0x55a3170b4000 > -1652> 2018-07-11 07:35:14.164523 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.11 10.8.128.114:0/15919 1866 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:14.788755) v4 ==== 2004+0+0 (1044365507 0 0) 0x55a324474c00 con 0x55a317074800 > -1651> 2018-07-11 07:35:14.164530 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:14.788755) v4 -- 0x55a30f772400 con 0 > -1650> 2018-07-11 07:35:14.164567 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:14.788755) v4 -- 0x55a33234fe00 con 0 > -1649> 2018-07-11 07:35:14.238498 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12677 conn(0x55a32169d800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1886 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:35:14.275026) v4 > -1648> 2018-07-11 07:35:14.238502 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12677 conn(0x55a32169c000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1886 0x55a30f773c00 osd_ping(ping e494 stamp 2018-07-11 07:35:14.275026) v4 > -1647> 2018-07-11 07:35:14.238520 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.1 10.8.128.106:0/12677 1886 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:14.275026) v4 ==== 2004+0+0 (2120528880 0 0) 0x55a34de6f000 con 0x55a32169d800 > -1646> 2018-07-11 07:35:14.238522 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.1 10.8.128.106:0/12677 1886 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:14.275026) v4 ==== 2004+0+0 (2120528880 0 0) 0x55a30f773c00 con 0x55a32169c000 > -1645> 2018-07-11 07:35:14.238531 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:14.275026) v4 -- 0x55a32e331c00 con 0 > -1644> 2018-07-11 07:35:14.238564 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:14.275026) v4 -- 0x55a30f772400 con 0 > -1643> 2018-07-11 07:35:14.481343 7f3543899700 5 osd.10 494 heartbeat: osd_stat(784 GB used, 146 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,0,0,1]) > -1642> 2018-07-11 07:35:14.481359 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6802/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:35:14.481357) v4 -- 0x55a323f55e00 con 0 > -1641> 2018-07-11 07:35:14.481396 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6803/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:35:14.481357) v4 -- 0x55a327788800 con 0 > -1640> 2018-07-11 07:35:14.481420 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6806/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:35:14.481357) v4 -- 0x55a327787a00 con 0 > -1639> 2018-07-11 07:35:14.481448 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6807/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:35:14.481357) v4 -- 0x55a323f54000 con 0 > -1638> 2018-07-11 07:35:14.481469 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6808/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:35:14.481357) v4 -- 0x55a33234a000 con 0 > -1637> 2018-07-11 07:35:14.481489 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6810/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:35:14.481357) v4 -- 0x55a33bff1a00 con 0 > -1636> 2018-07-11 07:35:14.481512 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6809/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:35:14.481357) v4 -- 0x55a327786200 con 0 > -1635> 2018-07-11 07:35:14.481542 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6811/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:35:14.481357) v4 -- 0x55a332365000 con 0 > -1634> 2018-07-11 07:35:14.481561 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6810/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:35:14.481357) v4 -- 0x55a33bff1e00 con 0 > -1633> 2018-07-11 07:35:14.481580 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6811/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:35:14.481357) v4 -- 0x55a34da1a000 con 0 > -1632> 2018-07-11 07:35:14.481603 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6806/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:35:14.481357) v4 -- 0x55a33532d800 con 0 > -1631> 2018-07-11 07:35:14.481636 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6807/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:35:14.481357) v4 -- 0x55a331d5d200 con 0 > -1630> 2018-07-11 07:35:14.481653 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6802/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:35:14.481357) v4 -- 0x55a32ccf7e00 con 0 > -1629> 2018-07-11 07:35:14.481673 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6803/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:35:14.481357) v4 -- 0x55a34da1ac00 con 0 > -1628> 2018-07-11 07:35:14.481697 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6806/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:35:14.481357) v4 -- 0x55a352f28e00 con 0 > -1627> 2018-07-11 07:35:14.481715 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6807/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:35:14.481357) v4 -- 0x55a352f28000 con 0 > -1626> 2018-07-11 07:35:14.481730 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6802/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:35:14.481357) v4 -- 0x55a352f28800 con 0 > -1625> 2018-07-11 07:35:14.481745 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6803/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:35:14.481357) v4 -- 0x55a32ccf7a00 con 0 > -1624> 2018-07-11 07:35:14.481759 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6806/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:35:14.481357) v4 -- 0x55a352f28600 con 0 > -1623> 2018-07-11 07:35:14.481778 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6807/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:35:14.481357) v4 -- 0x55a352f29200 con 0 > -1622> 2018-07-11 07:35:14.481801 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6802/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:35:14.481357) v4 -- 0x55a352f29a00 con 0 > -1621> 2018-07-11 07:35:14.481822 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6803/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:35:14.481357) v4 -- 0x55a352f28400 con 0 > -1620> 2018-07-11 07:35:14.481834 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6802/15255 conn(0x55a314e1c000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1870 0x55a30f773c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:14.481357) v4 > -1619> 2018-07-11 07:35:14.481852 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6802/15255 1870 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:14.481357) v4 ==== 2004+0+0 (1094933390 0 0) 0x55a30f773c00 con 0x55a314e1c000 > -1618> 2018-07-11 07:35:14.481913 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6810/12651 conn(0x55a314e97800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1870 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:14.481357) v4 > -1617> 2018-07-11 07:35:14.481919 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6807/12677 conn(0x55a314e85800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1870 0x55a30f773c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:14.481357) v4 > -1616> 2018-07-11 07:35:14.481919 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6806/12677 conn(0x55a314e84000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1870 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:14.481357) v4 > -1615> 2018-07-11 07:35:14.481935 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6807/12677 1870 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:14.481357) v4 ==== 2004+0+0 (1094933390 0 0) 0x55a30f773c00 con 0x55a314e85800 > -1614> 2018-07-11 07:35:14.481934 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6810/12651 1870 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:14.481357) v4 ==== 2004+0+0 (1094933390 0 0) 0x55a324474c00 con 0x55a314e97800 > -1613> 2018-07-11 07:35:14.481941 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6806/12677 1870 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:14.481357) v4 ==== 2004+0+0 (1094933390 0 0) 0x55a34de6f000 con 0x55a314e84000 > -1612> 2018-07-11 07:35:14.481962 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6808/12651 conn(0x55a314e96000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1870 0x55a30f773c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:14.481357) v4 > -1611> 2018-07-11 07:35:14.481972 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6808/12651 1870 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:14.481357) v4 ==== 2004+0+0 (1094933390 0 0) 0x55a30f773c00 con 0x55a314e96000 > -1610> 2018-07-11 07:35:14.481990 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6803/15255 conn(0x55a314e1d800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1870 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:14.481357) v4 > -1609> 2018-07-11 07:35:14.481992 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6806/15263 conn(0x55a314e99000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1870 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:14.481357) v4 > -1608> 2018-07-11 07:35:14.482010 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6806/15263 1870 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:14.481357) v4 ==== 2004+0+0 (1094933390 0 0) 0x55a34de6f000 con 0x55a314e99000 > -1607> 2018-07-11 07:35:14.482007 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6803/15255 1870 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:14.481357) v4 ==== 2004+0+0 (1094933390 0 0) 0x55a324474c00 con 0x55a314e1d800 > -1606> 2018-07-11 07:35:14.482024 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6811/12650 conn(0x55a314e20800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1870 0x55a30f773c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:14.481357) v4 > -1605> 2018-07-11 07:35:14.482038 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6811/12650 1870 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:14.481357) v4 ==== 2004+0+0 (1094933390 0 0) 0x55a30f773c00 con 0x55a314e20800 > -1604> 2018-07-11 07:35:14.482060 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6806/12617 conn(0x55a314e26000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1870 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:14.481357) v4 > -1603> 2018-07-11 07:35:14.482068 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6810/15287 conn(0x55a314e03000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1870 0x55a30f773c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:14.481357) v4 > -1602> 2018-07-11 07:35:14.482065 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6809/12650 conn(0x55a314e1f000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1870 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:14.481357) v4 > -1601> 2018-07-11 07:35:14.482077 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6810/15287 1870 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:14.481357) v4 ==== 2004+0+0 (1094933390 0 0) 0x55a30f773c00 con 0x55a314e03000 > -1600> 2018-07-11 07:35:14.482074 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6806/12617 1870 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:14.481357) v4 ==== 2004+0+0 (1094933390 0 0) 0x55a324474c00 con 0x55a314e26000 > -1599> 2018-07-11 07:35:14.482106 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6807/15263 conn(0x55a314e9a800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1870 0x55a30f773c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:14.481357) v4 > -1598> 2018-07-11 07:35:14.482118 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6807/15263 1870 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:14.481357) v4 ==== 2004+0+0 (1094933390 0 0) 0x55a30f773c00 con 0x55a314e9a800 > -1597> 2018-07-11 07:35:14.482116 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6811/15287 conn(0x55a314e04800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1870 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:14.481357) v4 > -1596> 2018-07-11 07:35:14.482084 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6809/12650 1870 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:14.481357) v4 ==== 2004+0+0 (1094933390 0 0) 0x55a34de78600 con 0x55a314e1f000 > -1595> 2018-07-11 07:35:14.482138 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6811/15287 1870 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:14.481357) v4 ==== 2004+0+0 (1094933390 0 0) 0x55a324474c00 con 0x55a314e04800 > -1594> 2018-07-11 07:35:14.482148 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6803/11967 conn(0x55a314e13800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1870 0x55a30f773c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:14.481357) v4 > -1593> 2018-07-11 07:35:14.482160 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6803/11967 1870 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:14.481357) v4 ==== 2004+0+0 (1094933390 0 0) 0x55a30f773c00 con 0x55a314e13800 > -1592> 2018-07-11 07:35:14.482158 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6807/12617 conn(0x55a314e27800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1870 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:14.481357) v4 > -1591> 2018-07-11 07:35:14.482169 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6807/12617 1870 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:14.481357) v4 ==== 2004+0+0 (1094933390 0 0) 0x55a34de78600 con 0x55a314e27800 > -1590> 2018-07-11 07:35:14.482183 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6806/15885 conn(0x55a314e00000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1870 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:14.481357) v4 > -1589> 2018-07-11 07:35:14.482204 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6803/13323 conn(0x55a314e16800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1870 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:14.481357) v4 > -1588> 2018-07-11 07:35:14.482217 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6803/13323 1870 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:14.481357) v4 ==== 2004+0+0 (1094933390 0 0) 0x55a34de78600 con 0x55a314e16800 > -1587> 2018-07-11 07:35:14.482198 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6806/15885 1870 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:14.481357) v4 ==== 2004+0+0 (1094933390 0 0) 0x55a324474c00 con 0x55a314e00000 > -1586> 2018-07-11 07:35:14.482260 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6807/15885 conn(0x55a314e01800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1870 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:14.481357) v4 > -1585> 2018-07-11 07:35:14.482266 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6802/13323 conn(0x55a314e15000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1870 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:14.481357) v4 > -1584> 2018-07-11 07:35:14.482274 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6807/15885 1870 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:14.481357) v4 ==== 2004+0+0 (1094933390 0 0) 0x55a34de78600 con 0x55a314e01800 > -1583> 2018-07-11 07:35:14.482280 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6802/13323 1870 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:14.481357) v4 ==== 2004+0+0 (1094933390 0 0) 0x55a324474c00 con 0x55a314e15000 > -1582> 2018-07-11 07:35:14.482303 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6802/11967 conn(0x55a314e12000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1870 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:14.481357) v4 > -1581> 2018-07-11 07:35:14.482314 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6802/11967 1870 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:14.481357) v4 ==== 2004+0+0 (1094933390 0 0) 0x55a34de78600 con 0x55a314e12000 > -1580> 2018-07-11 07:35:14.482323 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6802/15919 conn(0x55a314e87000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1870 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:14.481357) v4 > -1579> 2018-07-11 07:35:14.482354 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6802/15919 1870 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:14.481357) v4 ==== 2004+0+0 (1094933390 0 0) 0x55a324474c00 con 0x55a314e87000 > -1578> 2018-07-11 07:35:14.482399 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6803/15919 conn(0x55a314e88800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1870 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:14.481357) v4 > -1577> 2018-07-11 07:35:14.482412 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6803/15919 1870 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:14.481357) v4 ==== 2004+0+0 (1094933390 0 0) 0x55a34de78600 con 0x55a314e88800 > -1576> 2018-07-11 07:35:14.660038 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a313fe1440 con 0 > -1575> 2018-07-11 07:35:14.685980 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5198 0x55a35302c280 replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:494/492,start:6:aa32f194:::benchmark_data_magna108.ceph.redhat.com_2478_object222714:0,end:6:aa3408d9:::benchmark_data_magna108_35617_object1060245:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -1574> 2018-07-11 07:35:14.686012 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5198 ==== replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:494/492,start:6:aa32f194:::benchmark_data_magna108.ceph.redhat.com_2478_object222714:0,end:6:aa3408d9:::benchmark_data_magna108_35617_object1060245:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 234+0+0 (1200873543 0 0) 0x55a35302c280 con 0x55a317019000 > -1573> 2018-07-11 07:35:15.273944 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a325dc8d80 con 0 > -1572> 2018-07-11 07:35:15.293410 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5199 0x55a35302c500 replica scrub(pg: 6.15,from:0'0,to:430'106114,epoch:494/492,start:6:aa3408d9:::benchmark_data_magna108_35617_object1060245:0,end:6:aa34ea84:::benchmark_data_magna108_35617_object321599:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -1571> 2018-07-11 07:35:15.293439 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5199 ==== replica scrub(pg: 6.15,from:0'0,to:430'106114,epoch:494/492,start:6:aa3408d9:::benchmark_data_magna108_35617_object1060245:0,end:6:aa34ea84:::benchmark_data_magna108_35617_object321599:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 219+0+0 (902299807 0 0) 0x55a35302c500 con 0x55a317019000 > -1570> 2018-07-11 07:35:15.581987 7f3543899700 5 osd.10 494 heartbeat: osd_stat(784 GB used, 146 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,0,1]) > -1569> 2018-07-11 07:35:15.582003 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6802/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:35:15.582001) v4 -- 0x55a32ccf7c00 con 0 > -1568> 2018-07-11 07:35:15.582044 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6803/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:35:15.582001) v4 -- 0x55a352f28a00 con 0 > -1567> 2018-07-11 07:35:15.582060 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6806/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:35:15.582001) v4 -- 0x55a352f29600 con 0 > -1566> 2018-07-11 07:35:15.582074 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6807/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:35:15.582001) v4 -- 0x55a352f28200 con 0 > -1565> 2018-07-11 07:35:15.582087 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6808/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:35:15.582001) v4 -- 0x55a352f29c00 con 0 > -1564> 2018-07-11 07:35:15.582101 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6810/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:35:15.582001) v4 -- 0x55a352f28c00 con 0 > -1563> 2018-07-11 07:35:15.582113 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6809/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:35:15.582001) v4 -- 0x55a352f29800 con 0 > -1562> 2018-07-11 07:35:15.582124 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6811/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:35:15.582001) v4 -- 0x55a33235b800 con 0 > -1561> 2018-07-11 07:35:15.582140 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6810/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:35:15.582001) v4 -- 0x55a352f29000 con 0 > -1560> 2018-07-11 07:35:15.582164 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6811/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:35:15.582001) v4 -- 0x55a332364000 con 0 > -1559> 2018-07-11 07:35:15.582179 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6806/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:35:15.582001) v4 -- 0x55a352f29e00 con 0 > -1558> 2018-07-11 07:35:15.582196 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6807/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:35:15.582001) v4 -- 0x55a332364a00 con 0 > -1557> 2018-07-11 07:35:15.582209 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6802/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:35:15.582001) v4 -- 0x55a352f29400 con 0 > -1556> 2018-07-11 07:35:15.582224 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6803/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:35:15.582001) v4 -- 0x55a327788000 con 0 > -1555> 2018-07-11 07:35:15.582237 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6806/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:35:15.582001) v4 -- 0x55a327786e00 con 0 > -1554> 2018-07-11 07:35:15.582271 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6807/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:35:15.582001) v4 -- 0x55a331d5d600 con 0 > -1553> 2018-07-11 07:35:15.582288 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6802/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:35:15.582001) v4 -- 0x55a327789000 con 0 > -1552> 2018-07-11 07:35:15.582299 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6803/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:35:15.582001) v4 -- 0x55a331d5de00 con 0 > -1551> 2018-07-11 07:35:15.582312 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6806/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:35:15.582001) v4 -- 0x55a327788a00 con 0 > -1550> 2018-07-11 07:35:15.582325 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6807/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:35:15.582001) v4 -- 0x55a33dff1a00 con 0 > -1549> 2018-07-11 07:35:15.582342 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6802/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:35:15.582001) v4 -- 0x55a3282cc000 con 0 > -1548> 2018-07-11 07:35:15.582361 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6803/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:35:15.582001) v4 -- 0x55a3282cc200 con 0 > -1547> 2018-07-11 07:35:15.582507 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6806/12617 conn(0x55a314e26000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1871 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:15.582001) v4 > -1546> 2018-07-11 07:35:15.582533 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6808/12651 conn(0x55a314e96000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1871 0x55a30f773c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:15.582001) v4 > -1545> 2018-07-11 07:35:15.582528 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6806/12617 1871 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:15.582001) v4 ==== 2004+0+0 (3678724568 0 0) 0x55a332e79a00 con 0x55a314e26000 > -1544> 2018-07-11 07:35:15.582553 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6808/12651 1871 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:15.582001) v4 ==== 2004+0+0 (3678724568 0 0) 0x55a30f773c00 con 0x55a314e96000 > -1543> 2018-07-11 07:35:15.582561 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6807/12617 conn(0x55a314e27800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1871 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:15.582001) v4 > -1542> 2018-07-11 07:35:15.582577 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6807/12617 1871 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:15.582001) v4 ==== 2004+0+0 (3678724568 0 0) 0x55a34de6f000 con 0x55a314e27800 > -1541> 2018-07-11 07:35:15.582583 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6802/15255 conn(0x55a314e1c000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1871 0x55a30f773c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:15.582001) v4 > -1540> 2018-07-11 07:35:15.582583 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6803/15255 conn(0x55a314e1d800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1871 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:15.582001) v4 > -1539> 2018-07-11 07:35:15.582595 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6802/15255 1871 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:15.582001) v4 ==== 2004+0+0 (3678724568 0 0) 0x55a30f773c00 con 0x55a314e1c000 > -1538> 2018-07-11 07:35:15.582595 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6803/15255 1871 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:15.582001) v4 ==== 2004+0+0 (3678724568 0 0) 0x55a332e79a00 con 0x55a314e1d800 > -1537> 2018-07-11 07:35:15.582630 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6806/12677 conn(0x55a314e84000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1871 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:15.582001) v4 > -1536> 2018-07-11 07:35:15.582645 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6806/12677 1871 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:15.582001) v4 ==== 2004+0+0 (3678724568 0 0) 0x55a34de6f000 con 0x55a314e84000 > -1535> 2018-07-11 07:35:15.582643 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6807/12677 conn(0x55a314e85800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1871 0x55a30f773c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:15.582001) v4 > -1534> 2018-07-11 07:35:15.582649 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6811/15287 conn(0x55a314e04800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1871 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:15.582001) v4 > -1533> 2018-07-11 07:35:15.582655 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6807/12677 1871 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:15.582001) v4 ==== 2004+0+0 (3678724568 0 0) 0x55a30f773c00 con 0x55a314e85800 > -1532> 2018-07-11 07:35:15.582666 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6811/15287 1871 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:15.582001) v4 ==== 2004+0+0 (3678724568 0 0) 0x55a332e79a00 con 0x55a314e04800 > -1531> 2018-07-11 07:35:15.582683 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6806/15263 conn(0x55a314e99000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1871 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:15.582001) v4 > -1530> 2018-07-11 07:35:15.582698 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6806/15263 1871 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:15.582001) v4 ==== 2004+0+0 (3678724568 0 0) 0x55a34de6f000 con 0x55a314e99000 > -1529> 2018-07-11 07:35:15.582696 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6810/15287 conn(0x55a314e03000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1871 0x55a30f773c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:15.582001) v4 > -1528> 2018-07-11 07:35:15.582741 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6810/15287 1871 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:15.582001) v4 ==== 2004+0+0 (3678724568 0 0) 0x55a30f773c00 con 0x55a314e03000 > -1527> 2018-07-11 07:35:15.582749 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6809/12650 conn(0x55a314e1f000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1871 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:15.582001) v4 > -1526> 2018-07-11 07:35:15.582749 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6806/15885 conn(0x55a314e00000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1871 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:15.582001) v4 > -1525> 2018-07-11 07:35:15.582771 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6809/12650 1871 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:15.582001) v4 ==== 2004+0+0 (3678724568 0 0) 0x55a34de6f000 con 0x55a314e1f000 > -1524> 2018-07-11 07:35:15.582775 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6806/15885 1871 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:15.582001) v4 ==== 2004+0+0 (3678724568 0 0) 0x55a332e79a00 con 0x55a314e00000 > -1523> 2018-07-11 07:35:15.582778 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6811/12650 conn(0x55a314e20800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1871 0x55a30f773c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:15.582001) v4 > -1522> 2018-07-11 07:35:15.582790 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6811/12650 1871 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:15.582001) v4 ==== 2004+0+0 (3678724568 0 0) 0x55a30f773c00 con 0x55a314e20800 > -1521> 2018-07-11 07:35:15.582815 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6810/12651 conn(0x55a314e97800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1871 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:15.582001) v4 > -1520> 2018-07-11 07:35:15.582826 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6803/13323 conn(0x55a314e16800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1871 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:15.582001) v4 > -1519> 2018-07-11 07:35:15.582839 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6807/15263 conn(0x55a314e9a800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1871 0x55a30f773c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:15.582001) v4 > -1518> 2018-07-11 07:35:15.582835 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6810/12651 1871 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:15.582001) v4 ==== 2004+0+0 (3678724568 0 0) 0x55a332e79a00 con 0x55a314e97800 > -1517> 2018-07-11 07:35:15.582851 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6807/15263 1871 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:15.582001) v4 ==== 2004+0+0 (3678724568 0 0) 0x55a30f773c00 con 0x55a314e9a800 > -1516> 2018-07-11 07:35:15.582849 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6803/13323 1871 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:15.582001) v4 ==== 2004+0+0 (3678724568 0 0) 0x55a34de6f000 con 0x55a314e16800 > -1515> 2018-07-11 07:35:15.582885 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6803/11967 conn(0x55a314e13800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1871 0x55a30f773c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:15.582001) v4 > -1514> 2018-07-11 07:35:15.582889 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6802/13323 conn(0x55a314e15000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1871 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:15.582001) v4 > -1513> 2018-07-11 07:35:15.582902 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6807/15885 conn(0x55a314e01800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1871 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:15.582001) v4 > -1512> 2018-07-11 07:35:15.582901 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6803/11967 1871 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:15.582001) v4 ==== 2004+0+0 (3678724568 0 0) 0x55a30f773c00 con 0x55a314e13800 > -1511> 2018-07-11 07:35:15.582914 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6802/13323 1871 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:15.582001) v4 ==== 2004+0+0 (3678724568 0 0) 0x55a332e79a00 con 0x55a314e15000 > -1510> 2018-07-11 07:35:15.582920 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6807/15885 1871 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:15.582001) v4 ==== 2004+0+0 (3678724568 0 0) 0x55a34de6f000 con 0x55a314e01800 > -1509> 2018-07-11 07:35:15.582969 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6802/15919 conn(0x55a314e87000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1871 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:15.582001) v4 > -1508> 2018-07-11 07:35:15.582969 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6803/15919 conn(0x55a314e88800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1871 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:15.582001) v4 > -1507> 2018-07-11 07:35:15.582991 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6802/15919 1871 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:15.582001) v4 ==== 2004+0+0 (3678724568 0 0) 0x55a332e79a00 con 0x55a314e87000 > -1506> 2018-07-11 07:35:15.582993 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6803/15919 1871 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:15.582001) v4 ==== 2004+0+0 (3678724568 0 0) 0x55a34de6f000 con 0x55a314e88800 > -1505> 2018-07-11 07:35:15.583038 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6802/11967 conn(0x55a314e12000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1871 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:15.582001) v4 > -1504> 2018-07-11 07:35:15.583053 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6802/11967 1871 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:15.582001) v4 ==== 2004+0+0 (3678724568 0 0) 0x55a34de6f000 con 0x55a314e12000 > -1503> 2018-07-11 07:35:15.758845 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.115:0/13323 conn(0x55a321665800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1873 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:35:16.096822) v4 > -1502> 2018-07-11 07:35:15.758869 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.6 10.8.128.115:0/13323 1873 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:16.096822) v4 ==== 2004+0+0 (4097667480 0 0) 0x55a332e79a00 con 0x55a321665800 > -1501> 2018-07-11 07:35:15.758886 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:16.096822) v4 -- 0x55a332364000 con 0 > -1500> 2018-07-11 07:35:15.758965 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.115:0/13323 conn(0x55a321668800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1873 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:35:16.096822) v4 > -1499> 2018-07-11 07:35:15.758982 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.6 10.8.128.115:0/13323 1873 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:16.096822) v4 ==== 2004+0+0 (4097667480 0 0) 0x55a324474c00 con 0x55a321668800 > -1498> 2018-07-11 07:35:15.758992 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:16.096822) v4 -- 0x55a352f28c00 con 0 > -1497> 2018-07-11 07:35:15.837769 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15255 conn(0x55a311b64000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1891 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:35:17.052654) v4 > -1496> 2018-07-11 07:35:15.837790 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.0 10.8.128.113:0/15255 1891 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:17.052654) v4 ==== 2004+0+0 (2835203425 0 0) 0x55a34de6f000 con 0x55a311b64000 > -1495> 2018-07-11 07:35:15.837799 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:17.052654) v4 -- 0x55a352f29800 con 0 > -1494> 2018-07-11 07:35:15.837791 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15255 conn(0x55a311b65800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1891 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:35:17.052654) v4 > -1493> 2018-07-11 07:35:15.837809 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.0 10.8.128.113:0/15255 1891 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:17.052654) v4 ==== 2004+0+0 (2835203425 0 0) 0x55a332e79a00 con 0x55a311b65800 > -1492> 2018-07-11 07:35:15.837824 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:17.052654) v4 -- 0x55a352f28a00 con 0 > -1491> 2018-07-11 07:35:15.864994 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15919 conn(0x55a3170b4000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1867 0x55a30f773c00 osd_ping(ping e494 stamp 2018-07-11 07:35:16.489141) v4 > -1490> 2018-07-11 07:35:15.865016 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.11 10.8.128.114:0/15919 1867 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:16.489141) v4 ==== 2004+0+0 (1601737745 0 0) 0x55a30f773c00 con 0x55a3170b4000 > -1489> 2018-07-11 07:35:15.865025 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:16.489141) v4 -- 0x55a352f29c00 con 0 > -1488> 2018-07-11 07:35:15.865016 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15919 conn(0x55a317074800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1867 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:35:16.489141) v4 > -1487> 2018-07-11 07:35:15.865034 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.11 10.8.128.114:0/15919 1867 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:16.489141) v4 ==== 2004+0+0 (1601737745 0 0) 0x55a324474c00 con 0x55a317074800 > -1486> 2018-07-11 07:35:15.865043 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:16.489141) v4 -- 0x55a327788800 con 0 > -1485> 2018-07-11 07:35:15.955757 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a3333d6000 con 0 > -1484> 2018-07-11 07:35:15.977099 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5200 0x55a35302c780 replica scrub(pg: 6.15,from:0'0,to:430'106267,epoch:494/492,start:6:aa34ea84:::benchmark_data_magna108_35617_object321599:0,end:6:aa35a691:::benchmark_data_magna108_35617_object52979:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -1483> 2018-07-11 07:35:15.977134 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5200 ==== replica scrub(pg: 6.15,from:0'0,to:430'106267,epoch:494/492,start:6:aa34ea84:::benchmark_data_magna108_35617_object321599:0,end:6:aa35a691:::benchmark_data_magna108_35617_object52979:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 217+0+0 (2044908792 0 0) 0x55a35302c780 con 0x55a317019000 > -1482> 2018-07-11 07:35:16.086774 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12650 conn(0x55a311b67000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1880 0x55a323f54000 osd_ping(ping e494 stamp 2018-07-11 07:35:16.123314) v4 > -1481> 2018-07-11 07:35:16.086803 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.3 10.8.128.106:0/12650 1880 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:16.123314) v4 ==== 2004+0+0 (3090397627 0 0) 0x55a323f54000 con 0x55a311b67000 > -1480> 2018-07-11 07:35:16.086822 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:16.123314) v4 -- 0x55a352f28200 con 0 > -1479> 2018-07-11 07:35:16.086818 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12650 conn(0x55a311b68800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1880 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:35:16.123314) v4 > -1478> 2018-07-11 07:35:16.086843 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.3 10.8.128.106:0/12650 1880 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:16.123314) v4 ==== 2004+0+0 (3090397627 0 0) 0x55a34de78600 con 0x55a311b68800 > -1477> 2018-07-11 07:35:16.086865 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:16.123314) v4 -- 0x55a352f29800 con 0 > -1476> 2018-07-11 07:35:16.462189 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b8800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1816 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:35:16.213095) v4 > -1475> 2018-07-11 07:35:16.462212 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.8 10.8.128.108:0/11967 1816 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:16.213095) v4 ==== 2004+0+0 (1351095267 0 0) 0x55a332e79a00 con 0x55a3170b8800 > -1474> 2018-07-11 07:35:16.462210 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b7000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1816 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:35:16.213095) v4 > -1473> 2018-07-11 07:35:16.462223 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:16.213095) v4 -- 0x55a33234fe00 con 0 > -1472> 2018-07-11 07:35:16.462235 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.8 10.8.128.108:0/11967 1816 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:16.213095) v4 ==== 2004+0+0 (1351095267 0 0) 0x55a34de6f000 con 0x55a3170b7000 > -1471> 2018-07-11 07:35:16.462265 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:16.213095) v4 -- 0x55a352f29600 con 0 > -1470> 2018-07-11 07:35:16.582828 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a325dcf680 con 0 > -1469> 2018-07-11 07:35:16.601788 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5201 0x55a35302ca00 replica scrub(pg: 6.15,from:0'0,to:430'106544,epoch:494/492,start:6:aa35a691:::benchmark_data_magna108_35617_object52979:0,end:6:aa369a98:::benchmark_data_magna108_35617_object782295:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -1468> 2018-07-11 07:35:16.601817 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5201 ==== replica scrub(pg: 6.15,from:0'0,to:430'106544,epoch:494/492,start:6:aa35a691:::benchmark_data_magna108_35617_object52979:0,end:6:aa369a98:::benchmark_data_magna108_35617_object782295:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 217+0+0 (2213448460 0 0) 0x55a35302ca00 con 0x55a317019000 > -1467> 2018-07-11 07:35:16.745412 7f35498a5700 10 monclient: tick > -1466> 2018-07-11 07:35:16.745426 7f35498a5700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2018-07-11 07:34:46.745425) > -1465> 2018-07-11 07:35:17.138915 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12677 conn(0x55a32169c000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1887 0x55a352f28200 osd_ping(ping e494 stamp 2018-07-11 07:35:17.175450) v4 > -1464> 2018-07-11 07:35:17.138912 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12677 conn(0x55a32169d800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1887 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:35:17.175450) v4 > -1463> 2018-07-11 07:35:17.138945 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.1 10.8.128.106:0/12677 1887 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:17.175450) v4 ==== 2004+0+0 (2309623682 0 0) 0x55a352f28200 con 0x55a32169c000 > -1462> 2018-07-11 07:35:17.138945 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.1 10.8.128.106:0/12677 1887 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:17.175450) v4 ==== 2004+0+0 (2309623682 0 0) 0x55a34de78600 con 0x55a32169d800 > -1461> 2018-07-11 07:35:17.138956 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:17.175450) v4 -- 0x55a32ccf7c00 con 0 > -1460> 2018-07-11 07:35:17.138988 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:17.175450) v4 -- 0x55a352f29600 con 0 > -1459> 2018-07-11 07:35:17.258212 7f35488a3700 1 -- 10.8.128.107:6800/12665 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55a3170ea840 con 0 > -1458> 2018-07-11 07:35:17.258297 7f35488a3700 1 -- 10.8.128.107:6800/12665 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55a32a57b8c0 con 0 > -1457> 2018-07-11 07:35:17.828536 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15263 conn(0x55a321664000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1864 0x55a352f29c00 osd_ping(ping e494 stamp 2018-07-11 07:35:19.043592) v4 > -1456> 2018-07-11 07:35:17.828564 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.5 10.8.128.113:0/15263 1864 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:19.043592) v4 ==== 2004+0+0 (2433724334 0 0) 0x55a352f29c00 con 0x55a321664000 > -1455> 2018-07-11 07:35:17.828577 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.043592) v4 -- 0x55a32ccf7c00 con 0 > -1454> 2018-07-11 07:35:17.828635 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15263 conn(0x55a321667000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1864 0x55a352f29c00 osd_ping(ping e494 stamp 2018-07-11 07:35:19.043592) v4 > -1453> 2018-07-11 07:35:17.828661 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.5 10.8.128.113:0/15263 1864 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:19.043592) v4 ==== 2004+0+0 (2433724334 0 0) 0x55a352f29c00 con 0x55a321667000 > -1452> 2018-07-11 07:35:17.828673 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.043592) v4 -- 0x55a30f772400 con 0 > -1451> 2018-07-11 07:35:18.239291 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12677 conn(0x55a32169d800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1888 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:35:18.275873) v4 > -1450> 2018-07-11 07:35:18.239312 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.1 10.8.128.106:0/12677 1888 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:18.275873) v4 ==== 2004+0+0 (3595765030 0 0) 0x55a34de6f000 con 0x55a32169d800 > -1449> 2018-07-11 07:35:18.239323 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:18.275873) v4 -- 0x55a327787a00 con 0 > -1448> 2018-07-11 07:35:18.239312 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12677 conn(0x55a32169c000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1888 0x55a352f28200 osd_ping(ping e494 stamp 2018-07-11 07:35:18.275873) v4 > -1447> 2018-07-11 07:35:18.239343 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.1 10.8.128.106:0/12677 1888 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:18.275873) v4 ==== 2004+0+0 (3595765030 0 0) 0x55a352f28200 con 0x55a32169c000 > -1446> 2018-07-11 07:35:18.239355 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:18.275873) v4 -- 0x55a32ccf7c00 con 0 > -1445> 2018-07-11 07:35:18.387309 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12650 conn(0x55a311b68800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1881 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:35:18.423860) v4 > -1444> 2018-07-11 07:35:18.387309 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12650 conn(0x55a311b67000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1881 0x55a352f29c00 osd_ping(ping e494 stamp 2018-07-11 07:35:18.423860) v4 > -1443> 2018-07-11 07:35:18.387338 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.3 10.8.128.106:0/12650 1881 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:18.423860) v4 ==== 2004+0+0 (886810762 0 0) 0x55a34de78600 con 0x55a311b68800 > -1442> 2018-07-11 07:35:18.387341 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.3 10.8.128.106:0/12650 1881 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:18.423860) v4 ==== 2004+0+0 (886810762 0 0) 0x55a352f29c00 con 0x55a311b67000 > -1441> 2018-07-11 07:35:18.387358 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:18.423860) v4 -- 0x55a32e331c00 con 0 > -1440> 2018-07-11 07:35:18.387389 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:18.423860) v4 -- 0x55a30f772a00 con 0 > -1439> 2018-07-11 07:35:18.408386 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 61] Generated table #36596: 4957167 keys, 80348310 bytes > -1438> 2018-07-11 07:35:18.408431 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294518408405, "cf_name": "default", "job": 61, "event": "table_file_creation", "file_number": 36596, "file_size": 80348310, "table_properties": {"data_size": 67109394, "index_size": 689219, "filter_size": 12548707, "raw_key_size": 143207044, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957167, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957167", "kMergeOperands": "0"}} > -1437> 2018-07-11 07:35:18.454701 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a33caa6900 con 0 > -1436> 2018-07-11 07:35:18.468327 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5202 0x55a35302cc80 replica scrub(pg: 6.15,from:0'0,to:430'106649,epoch:494/492,start:6:aa369a98:::benchmark_data_magna108_35617_object782295:0,end:6:aa37d7f8:::benchmark_data_magna108_35617_object1021301:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -1435> 2018-07-11 07:35:18.468351 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5202 ==== replica scrub(pg: 6.15,from:0'0,to:430'106649,epoch:494/492,start:6:aa369a98:::benchmark_data_magna108_35617_object782295:0,end:6:aa37d7f8:::benchmark_data_magna108_35617_object1021301:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 219+0+0 (2307654543 0 0) 0x55a35302cc80 con 0x55a317019000 > -1434> 2018-07-11 07:35:18.676966 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15885 conn(0x55a311b76800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1888 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:35:19.301229) v4 > -1433> 2018-07-11 07:35:18.676969 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15885 conn(0x55a3170b5800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1888 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:35:19.301229) v4 > -1432> 2018-07-11 07:35:18.676991 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.7 10.8.128.114:0/15885 1888 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:19.301229) v4 ==== 2004+0+0 (3645517188 0 0) 0x55a34de6f000 con 0x55a311b76800 > -1431> 2018-07-11 07:35:18.676995 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.7 10.8.128.114:0/15885 1888 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:19.301229) v4 ==== 2004+0+0 (3645517188 0 0) 0x55a324474c00 con 0x55a3170b5800 > -1430> 2018-07-11 07:35:18.677001 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.301229) v4 -- 0x55a32e331c00 con 0 > -1429> 2018-07-11 07:35:18.677026 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.301229) v4 -- 0x55a33234fe00 con 0 > -1428> 2018-07-11 07:35:18.765429 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15919 conn(0x55a3170b4000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1868 0x55a352f28200 osd_ping(ping e494 stamp 2018-07-11 07:35:19.389717) v4 > -1427> 2018-07-11 07:35:18.765450 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.11 10.8.128.114:0/15919 1868 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:19.389717) v4 ==== 2004+0+0 (4173517903 0 0) 0x55a352f28200 con 0x55a3170b4000 > -1426> 2018-07-11 07:35:18.765459 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.389717) v4 -- 0x55a322454200 con 0 > -1425> 2018-07-11 07:35:18.765466 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15919 conn(0x55a317074800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1868 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:35:19.389717) v4 > -1424> 2018-07-11 07:35:18.765488 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.11 10.8.128.114:0/15919 1868 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:19.389717) v4 ==== 2004+0+0 (4173517903 0 0) 0x55a332e79a00 con 0x55a317074800 > -1423> 2018-07-11 07:35:18.765499 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.389717) v4 -- 0x55a332ead600 con 0 > -1422> 2018-07-11 07:35:18.774374 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12651 conn(0x55a3216a0800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1865 0x55a30f772a00 osd_ping(ping e494 stamp 2018-07-11 07:35:18.811005) v4 > -1421> 2018-07-11 07:35:18.774399 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.2 10.8.128.106:0/12651 1865 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:18.811005) v4 ==== 2004+0+0 (462109942 0 0) 0x55a30f772a00 con 0x55a3216a0800 > -1420> 2018-07-11 07:35:18.774409 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:18.811005) v4 -- 0x55a30f773600 con 0 > -1419> 2018-07-11 07:35:18.774435 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12651 conn(0x55a32169f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1865 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:35:18.811005) v4 > -1418> 2018-07-11 07:35:18.774455 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.2 10.8.128.106:0/12651 1865 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:18.811005) v4 ==== 2004+0+0 (462109942 0 0) 0x55a324474c00 con 0x55a32169f000 > -1417> 2018-07-11 07:35:18.774466 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:18.811005) v4 -- 0x55a32eb15200 con 0 > -1416> 2018-07-11 07:35:19.061993 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a325dd9200 con 0 > -1415> 2018-07-11 07:35:19.082508 7f3543899700 5 osd.10 494 heartbeat: osd_stat(785 GB used, 146 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist []) > -1414> 2018-07-11 07:35:19.082536 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6802/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:35:19.082534) v4 -- 0x55a3282cc400 con 0 > -1413> 2018-07-11 07:35:19.082559 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6803/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:35:19.082534) v4 -- 0x55a3282cc600 con 0 > -1412> 2018-07-11 07:35:19.082573 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6806/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:35:19.082534) v4 -- 0x55a3282cc800 con 0 > -1411> 2018-07-11 07:35:19.082599 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6807/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:35:19.082534) v4 -- 0x55a3282cca00 con 0 > -1410> 2018-07-11 07:35:19.082608 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6808/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:35:19.082534) v4 -- 0x55a3282ccc00 con 0 > -1409> 2018-07-11 07:35:19.082618 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6810/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:35:19.082534) v4 -- 0x55a3282cce00 con 0 > -1408> 2018-07-11 07:35:19.082631 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6809/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:35:19.082534) v4 -- 0x55a3282cd000 con 0 > -1407> 2018-07-11 07:35:19.082642 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6811/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:35:19.082534) v4 -- 0x55a3282cd200 con 0 > -1406> 2018-07-11 07:35:19.082653 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6810/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:35:19.082534) v4 -- 0x55a3282cd400 con 0 > -1405> 2018-07-11 07:35:19.082663 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6811/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:35:19.082534) v4 -- 0x55a3282cd600 con 0 > -1404> 2018-07-11 07:35:19.082686 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6806/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:35:19.082534) v4 -- 0x55a3282cd800 con 0 > -1403> 2018-07-11 07:35:19.082704 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6807/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:35:19.082534) v4 -- 0x55a3282cda00 con 0 > -1402> 2018-07-11 07:35:19.082720 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6802/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:35:19.082534) v4 -- 0x55a3282cdc00 con 0 > -1401> 2018-07-11 07:35:19.082734 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6803/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:35:19.082534) v4 -- 0x55a3282cde00 con 0 > -1400> 2018-07-11 07:35:19.082747 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6806/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:35:19.082534) v4 -- 0x55a323f55000 con 0 > -1399> 2018-07-11 07:35:19.082761 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6807/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:35:19.082534) v4 -- 0x55a322455800 con 0 > -1398> 2018-07-11 07:35:19.082775 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6802/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:35:19.082534) v4 -- 0x55a32ccf6c00 con 0 > -1397> 2018-07-11 07:35:19.082792 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6803/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:35:19.082534) v4 -- 0x55a32ccf6e00 con 0 > -1396> 2018-07-11 07:35:19.082805 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6806/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:35:19.082534) v4 -- 0x55a32ccf7200 con 0 > -1395> 2018-07-11 07:35:19.082821 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6807/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:35:19.082534) v4 -- 0x55a32ccf6a00 con 0 > -1394> 2018-07-11 07:35:19.082835 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6802/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:35:19.082534) v4 -- 0x55a332ffa200 con 0 > -1393> 2018-07-11 07:35:19.082846 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6803/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:35:19.082534) v4 -- 0x55a332384400 con 0 > -1392> 2018-07-11 07:35:19.083003 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6806/12617 conn(0x55a314e26000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1872 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.082534) v4 > -1391> 2018-07-11 07:35:19.083029 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6806/12617 1872 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.082534) v4 ==== 2004+0+0 (2946587413 0 0) 0x55a324474c00 con 0x55a314e26000 > -1390> 2018-07-11 07:35:19.083056 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6806/12677 conn(0x55a314e84000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1872 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.082534) v4 > -1389> 2018-07-11 07:35:19.083078 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6806/12677 1872 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.082534) v4 ==== 2004+0+0 (2946587413 0 0) 0x55a34de78600 con 0x55a314e84000 > -1388> 2018-07-11 07:35:19.083083 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6807/12677 conn(0x55a314e85800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1872 0x55a30f772a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.082534) v4 > -1387> 2018-07-11 07:35:19.083086 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6810/12651 conn(0x55a314e97800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1872 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.082534) v4 > -1386> 2018-07-11 07:35:19.083101 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6807/12677 1872 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.082534) v4 ==== 2004+0+0 (2946587413 0 0) 0x55a30f772a00 con 0x55a314e85800 > -1385> 2018-07-11 07:35:19.083102 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6810/12651 1872 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.082534) v4 ==== 2004+0+0 (2946587413 0 0) 0x55a324474c00 con 0x55a314e97800 > -1384> 2018-07-11 07:35:19.083122 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6807/12617 conn(0x55a314e27800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1872 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.082534) v4 > -1383> 2018-07-11 07:35:19.083140 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6807/12617 1872 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.082534) v4 ==== 2004+0+0 (2946587413 0 0) 0x55a34de78600 con 0x55a314e27800 > -1382> 2018-07-11 07:35:19.083146 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6808/12651 conn(0x55a314e96000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1872 0x55a30f772a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.082534) v4 > -1381> 2018-07-11 07:35:19.083157 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6808/12651 1872 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.082534) v4 ==== 2004+0+0 (2946587413 0 0) 0x55a30f772a00 con 0x55a314e96000 > -1380> 2018-07-11 07:35:19.083155 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6803/15255 conn(0x55a314e1d800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1872 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.082534) v4 > -1379> 2018-07-11 07:35:19.083185 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6802/15255 conn(0x55a314e1c000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1872 0x55a30f772a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.082534) v4 > -1378> 2018-07-11 07:35:19.083182 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6803/15255 1872 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.082534) v4 ==== 2004+0+0 (2946587413 0 0) 0x55a324474c00 con 0x55a314e1d800 > -1377> 2018-07-11 07:35:19.083198 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6802/15255 1872 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.082534) v4 ==== 2004+0+0 (2946587413 0 0) 0x55a30f772a00 con 0x55a314e1c000 > -1376> 2018-07-11 07:35:19.083192 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6809/12650 conn(0x55a314e1f000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1872 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.082534) v4 > -1375> 2018-07-11 07:35:19.083207 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6809/12650 1872 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.082534) v4 ==== 2004+0+0 (2946587413 0 0) 0x55a34de78600 con 0x55a314e1f000 > -1374> 2018-07-11 07:35:19.083237 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6811/12650 conn(0x55a314e20800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1872 0x55a30f772a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.082534) v4 > -1373> 2018-07-11 07:35:19.083261 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6811/12650 1872 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.082534) v4 ==== 2004+0+0 (2946587413 0 0) 0x55a30f772a00 con 0x55a314e20800 > -1372> 2018-07-11 07:35:19.083244 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6811/15287 conn(0x55a314e04800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1872 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.082534) v4 > -1371> 2018-07-11 07:35:19.083272 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6811/15287 1872 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.082534) v4 ==== 2004+0+0 (2946587413 0 0) 0x55a324474c00 con 0x55a314e04800 > -1370> 2018-07-11 07:35:19.083276 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6803/13323 conn(0x55a314e16800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1872 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.082534) v4 > -1369> 2018-07-11 07:35:19.083288 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6803/13323 1872 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.082534) v4 ==== 2004+0+0 (2946587413 0 0) 0x55a34de78600 con 0x55a314e16800 > -1368> 2018-07-11 07:35:19.083296 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6810/15287 conn(0x55a314e03000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1872 0x55a30f772a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.082534) v4 > -1367> 2018-07-11 07:35:19.083309 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6810/15287 1872 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.082534) v4 ==== 2004+0+0 (2946587413 0 0) 0x55a30f772a00 con 0x55a314e03000 > -1366> 2018-07-11 07:35:19.083325 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6802/13323 conn(0x55a314e15000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1872 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.082534) v4 > -1365> 2018-07-11 07:35:19.083338 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6807/15263 conn(0x55a314e9a800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1872 0x55a30f772a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.082534) v4 > -1364> 2018-07-11 07:35:19.083350 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6807/15263 1872 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.082534) v4 ==== 2004+0+0 (2946587413 0 0) 0x55a30f772a00 con 0x55a314e9a800 > -1363> 2018-07-11 07:35:19.083350 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6806/15263 conn(0x55a314e99000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1872 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.082534) v4 > -1362> 2018-07-11 07:35:19.083345 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6802/13323 1872 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.082534) v4 ==== 2004+0+0 (2946587413 0 0) 0x55a324474c00 con 0x55a314e15000 > -1361> 2018-07-11 07:35:19.083381 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6803/11967 conn(0x55a314e13800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1872 0x55a30f772a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.082534) v4 > -1360> 2018-07-11 07:35:19.083382 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6806/15263 1872 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.082534) v4 ==== 2004+0+0 (2946587413 0 0) 0x55a34de78600 con 0x55a314e99000 > -1359> 2018-07-11 07:35:19.083393 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6803/11967 1872 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.082534) v4 ==== 2004+0+0 (2946587413 0 0) 0x55a30f772a00 con 0x55a314e13800 > -1358> 2018-07-11 07:35:19.083420 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6806/15885 conn(0x55a314e00000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1872 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.082534) v4 > -1357> 2018-07-11 07:35:19.083427 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6802/11967 conn(0x55a314e12000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1872 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.082534) v4 > -1356> 2018-07-11 07:35:19.083443 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6806/15885 1872 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.082534) v4 ==== 2004+0+0 (2946587413 0 0) 0x55a324474c00 con 0x55a314e00000 > -1355> 2018-07-11 07:35:19.083447 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6802/11967 1872 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.082534) v4 ==== 2004+0+0 (2946587413 0 0) 0x55a34de78600 con 0x55a314e12000 > -1354> 2018-07-11 07:35:19.083496 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6802/15919 conn(0x55a314e87000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1872 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.082534) v4 > -1353> 2018-07-11 07:35:19.083496 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6807/15885 conn(0x55a314e01800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1872 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.082534) v4 > -1352> 2018-07-11 07:35:19.083518 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6802/15919 1872 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.082534) v4 ==== 2004+0+0 (2946587413 0 0) 0x55a324474c00 con 0x55a314e87000 > -1351> 2018-07-11 07:35:19.083522 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6807/15885 1872 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.082534) v4 ==== 2004+0+0 (2946587413 0 0) 0x55a34de78600 con 0x55a314e01800 > -1350> 2018-07-11 07:35:19.083571 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6803/15919 conn(0x55a314e88800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1872 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.082534) v4 > -1349> 2018-07-11 07:35:19.083587 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6803/15919 1872 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.082534) v4 ==== 2004+0+0 (2946587413 0 0) 0x55a34de78600 con 0x55a314e88800 > -1348> 2018-07-11 07:35:19.086271 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5203 0x55a35302cf00 replica scrub(pg: 6.15,from:0'0,to:430'106176,epoch:494/492,start:6:aa37d7f8:::benchmark_data_magna108_35617_object1021301:0,end:6:aa389ed3:::benchmark_data_magna108_35617_object1023535:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -1347> 2018-07-11 07:35:19.086291 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5203 ==== replica scrub(pg: 6.15,from:0'0,to:430'106176,epoch:494/492,start:6:aa37d7f8:::benchmark_data_magna108_35617_object1021301:0,end:6:aa389ed3:::benchmark_data_magna108_35617_object1023535:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 220+0+0 (3484147693 0 0) 0x55a35302cf00 con 0x55a317019000 > -1346> 2018-07-11 07:35:19.192158 7f355d88e700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.107:0/12617 conn(0x55a32447f800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1840 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:35:19.191680) v4 > -1345> 2018-07-11 07:35:19.192196 7f355d88e700 1 -- 10.8.128.107:6803/12665 <== osd.9 10.8.128.107:0/12617 1840 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:19.191680) v4 ==== 2004+0+0 (1661009000 0 0) 0x55a34de78600 con 0x55a32447f800 > -1344> 2018-07-11 07:35:19.192209 7f355d88e700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.191680) v4 -- 0x55a3282cd800 con 0 > -1343> 2018-07-11 07:35:19.192236 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.107:0/12617 conn(0x55a32447e000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1840 0x55a352f28200 osd_ping(ping e494 stamp 2018-07-11 07:35:19.191680) v4 > -1342> 2018-07-11 07:35:19.192265 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.9 10.8.128.107:0/12617 1840 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:19.191680) v4 ==== 2004+0+0 (1661009000 0 0) 0x55a352f28200 con 0x55a32447e000 > -1341> 2018-07-11 07:35:19.192299 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.191680) v4 -- 0x55a3282cca00 con 0 > -1340> 2018-07-11 07:35:19.280238 7f354489b700 5 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:307] [JOB 0] Delete db/036561.sst type=2 #36561 -- OK > > -1339> 2018-07-11 07:35:19.280265 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294519280258, "job": 0, "event": "table_file_deletion", "file_number": 36561} > -1338> 2018-07-11 07:35:19.280293 7f354489b700 5 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:307] [JOB 0] Delete db.slow/036546.sst type=2 #36546 -- OK > > -1337> 2018-07-11 07:35:19.280302 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294519280298, "job": 0, "event": "table_file_deletion", "file_number": 36546} > -1336> 2018-07-11 07:35:19.280313 7f354489b700 5 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:307] [JOB 0] Delete db.slow/036507.sst type=2 #36507 -- OK > > -1335> 2018-07-11 07:35:19.280321 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294519280318, "job": 0, "event": "table_file_deletion", "file_number": 36507} > -1334> 2018-07-11 07:35:19.280331 7f354489b700 5 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:307] [JOB 0] Delete db.slow/036506.sst type=2 #36506 -- OK > > -1333> 2018-07-11 07:35:19.280340 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294519280336, "job": 0, "event": "table_file_deletion", "file_number": 36506} > -1332> 2018-07-11 07:35:19.280354 7f354489b700 5 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:307] [JOB 0] Delete db.slow/036505.sst type=2 #36505 -- OK > > -1331> 2018-07-11 07:35:19.280365 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294519280361, "job": 0, "event": "table_file_deletion", "file_number": 36505} > -1330> 2018-07-11 07:35:19.280376 7f354489b700 5 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:307] [JOB 0] Delete db.slow/036504.sst type=2 #36504 -- OK > > -1329> 2018-07-11 07:35:19.280386 7f354489b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294519280383, "job": 0, "event": "table_file_deletion", "file_number": 36504} > -1328> 2018-07-11 07:35:19.487803 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12650 conn(0x55a311b68800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1882 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:35:19.524408) v4 > -1327> 2018-07-11 07:35:19.487802 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12650 conn(0x55a311b67000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1882 0x55a30f772a00 osd_ping(ping e494 stamp 2018-07-11 07:35:19.524408) v4 > -1326> 2018-07-11 07:35:19.487831 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.3 10.8.128.106:0/12650 1882 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:19.524408) v4 ==== 2004+0+0 (243253940 0 0) 0x55a34de6f000 con 0x55a311b68800 > -1325> 2018-07-11 07:35:19.487834 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.3 10.8.128.106:0/12650 1882 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:19.524408) v4 ==== 2004+0+0 (243253940 0 0) 0x55a30f772a00 con 0x55a311b67000 > -1324> 2018-07-11 07:35:19.487851 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.524408) v4 -- 0x55a3282cd000 con 0 > -1323> 2018-07-11 07:35:19.487882 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.524408) v4 -- 0x55a3282cc400 con 0 > -1322> 2018-07-11 07:35:19.528957 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15263 conn(0x55a321664000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1865 0x55a352f28200 osd_ping(ping e494 stamp 2018-07-11 07:35:20.744029) v4 > -1321> 2018-07-11 07:35:19.528978 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.5 10.8.128.113:0/15263 1865 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:20.744029) v4 ==== 2004+0+0 (1763624980 0 0) 0x55a352f28200 con 0x55a321664000 > -1320> 2018-07-11 07:35:19.528988 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:20.744029) v4 -- 0x55a30f773600 con 0 > -1319> 2018-07-11 07:35:19.529057 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15263 conn(0x55a321667000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1865 0x55a30f772a00 osd_ping(ping e494 stamp 2018-07-11 07:35:20.744029) v4 > -1318> 2018-07-11 07:35:19.529071 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.5 10.8.128.113:0/15263 1865 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:20.744029) v4 ==== 2004+0+0 (1763624980 0 0) 0x55a30f772a00 con 0x55a321667000 > -1317> 2018-07-11 07:35:19.529079 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:20.744029) v4 -- 0x55a30f772c00 con 0 > -1316> 2018-07-11 07:35:19.583016 7f3543899700 5 osd.10 494 heartbeat: osd_stat(785 GB used, 146 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,0,1]) > -1315> 2018-07-11 07:35:19.583030 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6802/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:35:19.583028) v4 -- 0x55a33234bc00 con 0 > -1314> 2018-07-11 07:35:19.583066 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6803/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:35:19.583028) v4 -- 0x55a33234e400 con 0 > -1313> 2018-07-11 07:35:19.583082 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6806/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:35:19.583028) v4 -- 0x55a34da1bc00 con 0 > -1312> 2018-07-11 07:35:19.583096 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6807/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:35:19.583028) v4 -- 0x55a344e85000 con 0 > -1311> 2018-07-11 07:35:19.583125 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6808/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:35:19.583028) v4 -- 0x55a344e84800 con 0 > -1310> 2018-07-11 07:35:19.583146 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6810/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:35:19.583028) v4 -- 0x55a322455a00 con 0 > -1309> 2018-07-11 07:35:19.583175 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6809/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:35:19.583028) v4 -- 0x55a322454800 con 0 > -1308> 2018-07-11 07:35:19.583191 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6811/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:35:19.583028) v4 -- 0x55a33234f400 con 0 > -1307> 2018-07-11 07:35:19.583213 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6810/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:35:19.583028) v4 -- 0x55a322454e00 con 0 > -1306> 2018-07-11 07:35:19.583238 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6811/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:35:19.583028) v4 -- 0x55a322454a00 con 0 > -1305> 2018-07-11 07:35:19.583271 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6806/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:35:19.583028) v4 -- 0x55a34de6e400 con 0 > -1304> 2018-07-11 07:35:19.583286 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6807/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:35:19.583028) v4 -- 0x55a322454c00 con 0 > -1303> 2018-07-11 07:35:19.583307 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6802/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:35:19.583028) v4 -- 0x55a3435ea400 con 0 > -1302> 2018-07-11 07:35:19.583330 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6803/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:35:19.583028) v4 -- 0x55a322455c00 con 0 > -1301> 2018-07-11 07:35:19.583353 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6806/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:35:19.583028) v4 -- 0x55a30f772200 con 0 > -1300> 2018-07-11 07:35:19.583375 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6807/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:35:19.583028) v4 -- 0x55a322455400 con 0 > -1299> 2018-07-11 07:35:19.583398 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6802/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:35:19.583028) v4 -- 0x55a34da1ba00 con 0 > -1298> 2018-07-11 07:35:19.583420 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6803/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:35:19.583028) v4 -- 0x55a32ccf6000 con 0 > -1297> 2018-07-11 07:35:19.583442 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6806/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:35:19.583028) v4 -- 0x55a32ccf6200 con 0 > -1296> 2018-07-11 07:35:19.583466 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6807/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:35:19.583028) v4 -- 0x55a32f760e00 con 0 > -1295> 2018-07-11 07:35:19.583482 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6802/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:35:19.583028) v4 -- 0x55a323f54e00 con 0 > -1294> 2018-07-11 07:35:19.583501 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6803/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:35:19.583028) v4 -- 0x55a332ec2c00 con 0 > -1293> 2018-07-11 07:35:19.583621 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6806/12677 conn(0x55a314e84000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1873 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.583028) v4 > -1292> 2018-07-11 07:35:19.583619 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6807/12677 conn(0x55a314e85800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1873 0x55a352f28200 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.583028) v4 > -1291> 2018-07-11 07:35:19.583617 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6810/12651 conn(0x55a314e97800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1873 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.583028) v4 > -1290> 2018-07-11 07:35:19.583644 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6806/12677 1873 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.583028) v4 ==== 2004+0+0 (576072065 0 0) 0x55a34de78600 con 0x55a314e84000 > -1289> 2018-07-11 07:35:19.583647 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6807/12677 1873 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.583028) v4 ==== 2004+0+0 (576072065 0 0) 0x55a352f28200 con 0x55a314e85800 > -1288> 2018-07-11 07:35:19.583656 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6810/12651 1873 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.583028) v4 ==== 2004+0+0 (576072065 0 0) 0x55a332e79a00 con 0x55a314e97800 > -1287> 2018-07-11 07:35:19.583689 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6809/12650 conn(0x55a314e1f000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1873 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.583028) v4 > -1286> 2018-07-11 07:35:19.583703 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6809/12650 1873 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.583028) v4 ==== 2004+0+0 (576072065 0 0) 0x55a34de78600 con 0x55a314e1f000 > -1285> 2018-07-11 07:35:19.583709 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6808/12651 conn(0x55a314e96000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1873 0x55a352f28200 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.583028) v4 > -1284> 2018-07-11 07:35:19.583716 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6803/15255 conn(0x55a314e1d800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1873 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.583028) v4 > -1283> 2018-07-11 07:35:19.583723 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6808/12651 1873 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.583028) v4 ==== 2004+0+0 (576072065 0 0) 0x55a352f28200 con 0x55a314e96000 > -1282> 2018-07-11 07:35:19.583734 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6803/15255 1873 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.583028) v4 ==== 2004+0+0 (576072065 0 0) 0x55a332e79a00 con 0x55a314e1d800 > -1281> 2018-07-11 07:35:19.583758 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6806/12617 conn(0x55a314e26000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1873 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.583028) v4 > -1280> 2018-07-11 07:35:19.583760 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6807/12617 conn(0x55a314e27800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1873 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.583028) v4 > -1279> 2018-07-11 07:35:19.583769 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6806/12617 1873 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.583028) v4 ==== 2004+0+0 (576072065 0 0) 0x55a332e79a00 con 0x55a314e26000 > -1278> 2018-07-11 07:35:19.583771 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6807/12617 1873 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.583028) v4 ==== 2004+0+0 (576072065 0 0) 0x55a34de78600 con 0x55a314e27800 > -1277> 2018-07-11 07:35:19.583773 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6802/15255 conn(0x55a314e1c000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1873 0x55a352f28200 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.583028) v4 > -1276> 2018-07-11 07:35:19.583785 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6802/15255 1873 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.583028) v4 ==== 2004+0+0 (576072065 0 0) 0x55a352f28200 con 0x55a314e1c000 > -1275> 2018-07-11 07:35:19.583805 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6811/15287 conn(0x55a314e04800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1873 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.583028) v4 > -1274> 2018-07-11 07:35:19.583812 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6806/15263 conn(0x55a314e99000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1873 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.583028) v4 > -1273> 2018-07-11 07:35:19.583815 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6810/15287 conn(0x55a314e03000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1873 0x55a352f28200 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.583028) v4 > -1272> 2018-07-11 07:35:19.583819 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6811/15287 1873 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.583028) v4 ==== 2004+0+0 (576072065 0 0) 0x55a332e79a00 con 0x55a314e04800 > -1271> 2018-07-11 07:35:19.583822 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6806/15263 1873 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.583028) v4 ==== 2004+0+0 (576072065 0 0) 0x55a34de78600 con 0x55a314e99000 > -1270> 2018-07-11 07:35:19.583828 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6810/15287 1873 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.583028) v4 ==== 2004+0+0 (576072065 0 0) 0x55a352f28200 con 0x55a314e03000 > -1269> 2018-07-11 07:35:19.583855 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6806/15885 conn(0x55a314e00000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1873 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.583028) v4 > -1268> 2018-07-11 07:35:19.583861 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6803/13323 conn(0x55a314e16800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1873 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.583028) v4 > -1267> 2018-07-11 07:35:19.583868 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6806/15885 1873 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.583028) v4 ==== 2004+0+0 (576072065 0 0) 0x55a332e79a00 con 0x55a314e00000 > -1266> 2018-07-11 07:35:19.583866 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6811/12650 conn(0x55a314e20800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1873 0x55a352f28200 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.583028) v4 > -1265> 2018-07-11 07:35:19.583879 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6803/13323 1873 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.583028) v4 ==== 2004+0+0 (576072065 0 0) 0x55a34de78600 con 0x55a314e16800 > -1264> 2018-07-11 07:35:19.583884 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6811/12650 1873 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.583028) v4 ==== 2004+0+0 (576072065 0 0) 0x55a352f28200 con 0x55a314e20800 > -1263> 2018-07-11 07:35:19.583912 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6802/15919 conn(0x55a314e87000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1873 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.583028) v4 > -1262> 2018-07-11 07:35:19.583919 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6807/15263 conn(0x55a314e9a800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1873 0x55a352f28200 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.583028) v4 > -1261> 2018-07-11 07:35:19.583915 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6802/11967 conn(0x55a314e12000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1873 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.583028) v4 > -1260> 2018-07-11 07:35:19.583928 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6802/15919 1873 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.583028) v4 ==== 2004+0+0 (576072065 0 0) 0x55a332e79a00 con 0x55a314e87000 > -1259> 2018-07-11 07:35:19.583929 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6807/15263 1873 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.583028) v4 ==== 2004+0+0 (576072065 0 0) 0x55a352f28200 con 0x55a314e9a800 > -1258> 2018-07-11 07:35:19.583933 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6802/11967 1873 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.583028) v4 ==== 2004+0+0 (576072065 0 0) 0x55a34de78600 con 0x55a314e12000 > -1257> 2018-07-11 07:35:19.583959 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6802/13323 conn(0x55a314e15000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1873 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.583028) v4 > -1256> 2018-07-11 07:35:19.583965 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6803/11967 conn(0x55a314e13800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1873 0x55a352f28200 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.583028) v4 > -1255> 2018-07-11 07:35:19.583971 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6802/13323 1873 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.583028) v4 ==== 2004+0+0 (576072065 0 0) 0x55a332e79a00 con 0x55a314e15000 > -1254> 2018-07-11 07:35:19.583977 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6803/11967 1873 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.583028) v4 ==== 2004+0+0 (576072065 0 0) 0x55a352f28200 con 0x55a314e13800 > -1253> 2018-07-11 07:35:19.583977 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6803/15919 conn(0x55a314e88800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1873 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.583028) v4 > -1252> 2018-07-11 07:35:19.583990 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6803/15919 1873 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.583028) v4 ==== 2004+0+0 (576072065 0 0) 0x55a34de78600 con 0x55a314e88800 > -1251> 2018-07-11 07:35:19.584023 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6807/15885 conn(0x55a314e01800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1873 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.583028) v4 > -1250> 2018-07-11 07:35:19.584035 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6807/15885 1873 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:19.583028) v4 ==== 2004+0+0 (576072065 0 0) 0x55a34de78600 con 0x55a314e01800 > -1249> 2018-07-11 07:35:19.859441 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.115:0/13323 conn(0x55a321665800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1874 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:35:20.197505) v4 > -1248> 2018-07-11 07:35:19.859466 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.6 10.8.128.115:0/13323 1874 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:20.197505) v4 ==== 2004+0+0 (2102055327 0 0) 0x55a332e79a00 con 0x55a321665800 > -1247> 2018-07-11 07:35:19.859482 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:20.197505) v4 -- 0x55a3282cc600 con 0 > -1246> 2018-07-11 07:35:19.859535 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.115:0/13323 conn(0x55a321668800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1874 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:35:20.197505) v4 > -1245> 2018-07-11 07:35:19.859551 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.6 10.8.128.115:0/13323 1874 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:20.197505) v4 ==== 2004+0+0 (2102055327 0 0) 0x55a332e79a00 con 0x55a321668800 > -1244> 2018-07-11 07:35:19.859561 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:20.197505) v4 -- 0x55a332e99c00 con 0 > -1243> 2018-07-11 07:35:19.938396 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15287 conn(0x55a311b75000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1862 0x55a352f28200 osd_ping(ping e494 stamp 2018-07-11 07:35:21.153307) v4 > -1242> 2018-07-11 07:35:19.938417 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.4 10.8.128.113:0/15287 1862 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:21.153307) v4 ==== 2004+0+0 (20938750 0 0) 0x55a352f28200 con 0x55a311b75000 > -1241> 2018-07-11 07:35:19.938423 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15255 conn(0x55a311b65800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1892 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:35:21.153265) v4 > -1240> 2018-07-11 07:35:19.938434 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:21.153307) v4 -- 0x55a33234bc00 con 0 > -1239> 2018-07-11 07:35:19.938447 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.0 10.8.128.113:0/15255 1892 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:21.153265) v4 ==== 2004+0+0 (2030433915 0 0) 0x55a324474c00 con 0x55a311b65800 > -1238> 2018-07-11 07:35:19.938471 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:21.153265) v4 -- 0x55a332e99c00 con 0 > -1237> 2018-07-11 07:35:19.938470 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15255 conn(0x55a311b64000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1892 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:35:21.153265) v4 > -1236> 2018-07-11 07:35:19.938493 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.0 10.8.128.113:0/15255 1892 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:21.153265) v4 ==== 2004+0+0 (2030433915 0 0) 0x55a34de78600 con 0x55a311b64000 > -1235> 2018-07-11 07:35:19.938508 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:21.153265) v4 -- 0x55a34da1bc00 con 0 > -1234> 2018-07-11 07:35:19.938541 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15287 conn(0x55a311b72000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1862 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:35:21.153307) v4 > -1233> 2018-07-11 07:35:19.938564 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.4 10.8.128.113:0/15287 1862 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:21.153307) v4 ==== 2004+0+0 (20938750 0 0) 0x55a332e79a00 con 0x55a311b72000 > -1232> 2018-07-11 07:35:19.938574 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:21.153307) v4 -- 0x55a332e99c00 con 0 > -1231> 2018-07-11 07:35:20.140169 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a3148d7440 con 0 > -1230> 2018-07-11 07:35:20.151883 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5204 0x55a35302d180 replica scrub(pg: 6.15,from:0'0,to:430'106476,epoch:494/492,start:6:aa389ed3:::benchmark_data_magna108_35617_object1023535:0,end:6:aa39d67d:::benchmark_data_magna108_35617_object931477:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -1229> 2018-07-11 07:35:20.151907 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5204 ==== replica scrub(pg: 6.15,from:0'0,to:430'106476,epoch:494/492,start:6:aa389ed3:::benchmark_data_magna108_35617_object1023535:0,end:6:aa39d67d:::benchmark_data_magna108_35617_object931477:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 219+0+0 (414330363 0 0) 0x55a35302d180 con 0x55a317019000 > -1228> 2018-07-11 07:35:20.292697 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.107:0/12617 conn(0x55a32447e000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1841 0x55a30f772a00 osd_ping(ping e494 stamp 2018-07-11 07:35:20.292283) v4 > -1227> 2018-07-11 07:35:20.292742 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.9 10.8.128.107:0/12617 1841 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:20.292283) v4 ==== 2004+0+0 (745366790 0 0) 0x55a30f772a00 con 0x55a32447e000 > -1226> 2018-07-11 07:35:20.292752 7f355d88e700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.107:0/12617 conn(0x55a32447f800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1841 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:35:20.292283) v4 > -1225> 2018-07-11 07:35:20.292763 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:20.292283) v4 -- 0x55a33234bc00 con 0 > -1224> 2018-07-11 07:35:20.292779 7f355d88e700 1 -- 10.8.128.107:6803/12665 <== osd.9 10.8.128.107:0/12617 1841 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:20.292283) v4 ==== 2004+0+0 (745366790 0 0) 0x55a34de6f000 con 0x55a32447f800 > -1223> 2018-07-11 07:35:20.292805 7f355d88e700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:20.292283) v4 -- 0x55a34da1bc00 con 0 > -1222> 2018-07-11 07:35:20.377369 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15885 conn(0x55a311b76800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1889 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:35:21.001665) v4 > -1221> 2018-07-11 07:35:20.377368 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15885 conn(0x55a3170b5800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1889 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:35:21.001665) v4 > -1220> 2018-07-11 07:35:20.377391 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.7 10.8.128.114:0/15885 1889 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:21.001665) v4 ==== 2004+0+0 (2444978386 0 0) 0x55a34de78600 con 0x55a311b76800 > -1219> 2018-07-11 07:35:20.377392 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.7 10.8.128.114:0/15885 1889 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:21.001665) v4 ==== 2004+0+0 (2444978386 0 0) 0x55a324474c00 con 0x55a3170b5800 > -1218> 2018-07-11 07:35:20.377401 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:21.001665) v4 -- 0x55a3282cd000 con 0 > -1217> 2018-07-11 07:35:20.377436 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:21.001665) v4 -- 0x55a3282cc600 con 0 > -1216> 2018-07-11 07:35:20.539673 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12677 conn(0x55a32169d800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1889 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:35:20.576266) v4 > -1215> 2018-07-11 07:35:20.539695 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.1 10.8.128.106:0/12677 1889 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:20.576266) v4 ==== 2004+0+0 (887204338 0 0) 0x55a34de6f000 con 0x55a32169d800 > -1214> 2018-07-11 07:35:20.539704 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:20.576266) v4 -- 0x55a3282cc800 con 0 > -1213> 2018-07-11 07:35:20.539779 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12677 conn(0x55a32169c000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1889 0x55a352f28200 osd_ping(ping e494 stamp 2018-07-11 07:35:20.576266) v4 > -1212> 2018-07-11 07:35:20.539801 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.1 10.8.128.106:0/12677 1889 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:20.576266) v4 ==== 2004+0+0 (887204338 0 0) 0x55a352f28200 con 0x55a32169c000 > -1211> 2018-07-11 07:35:20.539816 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:20.576266) v4 -- 0x55a30f772c00 con 0 > -1210> 2018-07-11 07:35:20.562557 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b8800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1817 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:35:20.313588) v4 > -1209> 2018-07-11 07:35:20.562562 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b7000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1817 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:35:20.313588) v4 > -1208> 2018-07-11 07:35:20.562579 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.8 10.8.128.108:0/11967 1817 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:20.313588) v4 ==== 2004+0+0 (4032735843 0 0) 0x55a34de78600 con 0x55a3170b7000 > -1207> 2018-07-11 07:35:20.562579 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.8 10.8.128.108:0/11967 1817 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:20.313588) v4 ==== 2004+0+0 (4032735843 0 0) 0x55a332e79a00 con 0x55a3170b8800 > -1206> 2018-07-11 07:35:20.562590 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:20.313588) v4 -- 0x55a30f58e200 con 0 > -1205> 2018-07-11 07:35:20.562612 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:20.313588) v4 -- 0x55a32e331c00 con 0 > -1204> 2018-07-11 07:35:20.826951 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a325de3d40 con 0 > -1203> 2018-07-11 07:35:20.852125 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5205 0x55a35302d400 replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:494/492,start:6:aa39d67d:::benchmark_data_magna108_35617_object931477:0,end:6:aa3b48d3:::benchmark_data_magna108_35617_object517125:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -1202> 2018-07-11 07:35:20.852149 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5205 ==== replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:494/492,start:6:aa39d67d:::benchmark_data_magna108_35617_object931477:0,end:6:aa3b48d3:::benchmark_data_magna108_35617_object517125:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (4049592744 0 0) 0x55a35302d400 con 0x55a317019000 > -1201> 2018-07-11 07:35:21.685100 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a325de6fc0 con 0 > -1200> 2018-07-11 07:35:21.694074 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5206 0x55a35302d680 replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:494/492,start:6:aa3b48d3:::benchmark_data_magna108_35617_object517125:0,end:6:aa3c3c25:::benchmark_data_magna108_35617_object54080:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -1199> 2018-07-11 07:35:21.694104 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5206 ==== replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:494/492,start:6:aa3b48d3:::benchmark_data_magna108_35617_object517125:0,end:6:aa3c3c25:::benchmark_data_magna108_35617_object54080:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 217+0+0 (2479579525 0 0) 0x55a35302d680 con 0x55a317019000 > -1198> 2018-07-11 07:35:22.258486 7f35488a3700 1 -- 10.8.128.107:6800/12665 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55a32a5c1b80 con 0 > -1197> 2018-07-11 07:35:22.258558 7f35488a3700 1 -- 10.8.128.107:6800/12665 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55a324471600 con 0 > -1196> 2018-07-11 07:35:22.263079 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b8800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1818 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:35:22.014033) v4 > -1195> 2018-07-11 07:35:22.263097 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b7000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1818 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:35:22.014033) v4 > -1194> 2018-07-11 07:35:22.263106 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.8 10.8.128.108:0/11967 1818 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:22.014033) v4 ==== 2004+0+0 (3030769887 0 0) 0x55a324474c00 con 0x55a3170b8800 > -1193> 2018-07-11 07:35:22.263115 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.8 10.8.128.108:0/11967 1818 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:22.014033) v4 ==== 2004+0+0 (3030769887 0 0) 0x55a34de6f000 con 0x55a3170b7000 > -1192> 2018-07-11 07:35:22.263118 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:22.014033) v4 -- 0x55a3170dee00 con 0 > -1191> 2018-07-11 07:35:22.263155 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:22.014033) v4 -- 0x55a331c65c00 con 0 > -1190> 2018-07-11 07:35:22.266037 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15919 conn(0x55a317074800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1869 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:35:22.890297) v4 > -1189> 2018-07-11 07:35:22.266058 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.11 10.8.128.114:0/15919 1869 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:22.890297) v4 ==== 2004+0+0 (3250907901 0 0) 0x55a332e79a00 con 0x55a317074800 > -1188> 2018-07-11 07:35:22.266069 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:22.890297) v4 -- 0x55a33532da00 con 0 > -1187> 2018-07-11 07:35:22.266085 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15919 conn(0x55a3170b4000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1869 0x55a30f772a00 osd_ping(ping e494 stamp 2018-07-11 07:35:22.890297) v4 > -1186> 2018-07-11 07:35:22.266107 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.11 10.8.128.114:0/15919 1869 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:22.890297) v4 ==== 2004+0+0 (3250907901 0 0) 0x55a30f772a00 con 0x55a3170b4000 > -1185> 2018-07-11 07:35:22.266116 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:22.890297) v4 -- 0x55a322454000 con 0 > -1184> 2018-07-11 07:35:22.274951 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12651 conn(0x55a32169f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1866 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:35:22.311540) v4 > -1183> 2018-07-11 07:35:22.274951 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12651 conn(0x55a3216a0800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1866 0x55a30f772c00 osd_ping(ping e494 stamp 2018-07-11 07:35:22.311540) v4 > -1182> 2018-07-11 07:35:22.274971 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.2 10.8.128.106:0/12651 1866 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:22.311540) v4 ==== 2004+0+0 (3178253483 0 0) 0x55a324474c00 con 0x55a32169f000 > -1181> 2018-07-11 07:35:22.274973 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.2 10.8.128.106:0/12651 1866 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:22.311540) v4 ==== 2004+0+0 (3178253483 0 0) 0x55a30f772c00 con 0x55a3216a0800 > -1180> 2018-07-11 07:35:22.274981 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:22.311540) v4 -- 0x55a33234ae00 con 0 > -1179> 2018-07-11 07:35:22.275014 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:22.311540) v4 -- 0x55a30f773200 con 0 > -1178> 2018-07-11 07:35:22.838842 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15255 conn(0x55a311b65800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1893 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:35:24.053926) v4 > -1177> 2018-07-11 07:35:22.838870 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.0 10.8.128.113:0/15255 1893 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:24.053926) v4 ==== 2004+0+0 (426001182 0 0) 0x55a332e79a00 con 0x55a311b65800 > -1176> 2018-07-11 07:35:22.838893 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:24.053926) v4 -- 0x55a324b15e00 con 0 > -1175> 2018-07-11 07:35:22.838891 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15255 conn(0x55a311b64000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1893 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:35:24.053926) v4 > -1174> 2018-07-11 07:35:22.838910 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.0 10.8.128.113:0/15255 1893 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:24.053926) v4 ==== 2004+0+0 (426001182 0 0) 0x55a34de78600 con 0x55a311b64000 > -1173> 2018-07-11 07:35:22.838924 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:24.053926) v4 -- 0x55a331c67e00 con 0 > -1172> 2018-07-11 07:35:23.193392 7f355d88e700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.107:0/12617 conn(0x55a32447f800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1842 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:35:23.192874) v4 > -1171> 2018-07-11 07:35:23.193414 7f355d88e700 1 -- 10.8.128.107:6803/12665 <== osd.9 10.8.128.107:0/12617 1842 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:23.192874) v4 ==== 2004+0+0 (2053741473 0 0) 0x55a34de6f000 con 0x55a32447f800 > -1170> 2018-07-11 07:35:23.193424 7f355d88e700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:23.192874) v4 -- 0x55a331c67e00 con 0 > -1169> 2018-07-11 07:35:23.193452 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.107:0/12617 conn(0x55a32447e000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1842 0x55a30f772a00 osd_ping(ping e494 stamp 2018-07-11 07:35:23.192874) v4 > -1168> 2018-07-11 07:35:23.193501 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.9 10.8.128.107:0/12617 1842 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:23.192874) v4 ==== 2004+0+0 (2053741473 0 0) 0x55a30f772a00 con 0x55a32447e000 > -1167> 2018-07-11 07:35:23.193518 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:23.192874) v4 -- 0x55a30f773200 con 0 > -1166> 2018-07-11 07:35:23.353845 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 61] Generated table #36597: 2680208 keys, 74793582 bytes > -1165> 2018-07-11 07:35:23.353889 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294523353863, "cf_name": "default", "job": 61, "event": "table_file_creation", "file_number": 36597, "file_size": 74793582, "table_properties": {"data_size": 67111025, "index_size": 829324, "filter_size": 6852241, "raw_key_size": 78331641, "raw_average_key_size": 29, "raw_value_size": 30499285, "raw_average_value_size": 11, "num_data_blocks": 16711, "num_entries": 2680208, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "2042833", "kMergeOperands": "0"}} > -1164> 2018-07-11 07:35:23.629342 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15263 conn(0x55a321664000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1866 0x55a30f772c00 osd_ping(ping e494 stamp 2018-07-11 07:35:24.844470) v4 > -1163> 2018-07-11 07:35:23.629371 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.5 10.8.128.113:0/15263 1866 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:24.844470) v4 ==== 2004+0+0 (3545540989 0 0) 0x55a30f772c00 con 0x55a321664000 > -1162> 2018-07-11 07:35:23.629387 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:24.844470) v4 -- 0x55a322454400 con 0 > -1161> 2018-07-11 07:35:23.629473 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15263 conn(0x55a321667000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1866 0x55a30f772a00 osd_ping(ping e494 stamp 2018-07-11 07:35:24.844470) v4 > -1160> 2018-07-11 07:35:23.629489 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.5 10.8.128.113:0/15263 1866 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:24.844470) v4 ==== 2004+0+0 (3545540989 0 0) 0x55a30f772a00 con 0x55a321667000 > -1159> 2018-07-11 07:35:23.629499 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:24.844470) v4 -- 0x55a30f773800 con 0 > -1158> 2018-07-11 07:35:23.683659 7f3543899700 5 osd.10 494 heartbeat: osd_stat(785 GB used, 146 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,0,0,0,1]) > -1157> 2018-07-11 07:35:23.683675 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6802/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:35:23.683673) v4 -- 0x55a331d5d800 con 0 > -1156> 2018-07-11 07:35:23.683699 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6803/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:35:23.683673) v4 -- 0x55a327787e00 con 0 > -1155> 2018-07-11 07:35:23.683716 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6806/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:35:23.683673) v4 -- 0x55a3282bba00 con 0 > -1154> 2018-07-11 07:35:23.683732 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6807/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:35:23.683673) v4 -- 0x55a315fed600 con 0 > -1153> 2018-07-11 07:35:23.683746 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6808/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:35:23.683673) v4 -- 0x55a315fec800 con 0 > -1152> 2018-07-11 07:35:23.683756 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6810/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:35:23.683673) v4 -- 0x55a315feca00 con 0 > -1151> 2018-07-11 07:35:23.683768 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6809/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:35:23.683673) v4 -- 0x55a315fec600 con 0 > -1150> 2018-07-11 07:35:23.683787 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6811/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:35:23.683673) v4 -- 0x55a315fed000 con 0 > -1149> 2018-07-11 07:35:23.683801 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6810/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:35:23.683673) v4 -- 0x55a3170de200 con 0 > -1148> 2018-07-11 07:35:23.683817 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6811/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:35:23.683673) v4 -- 0x55a3435eac00 con 0 > -1147> 2018-07-11 07:35:23.683840 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6806/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:35:23.683673) v4 -- 0x55a344e84e00 con 0 > -1146> 2018-07-11 07:35:23.683855 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6807/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:35:23.683673) v4 -- 0x55a34da1b800 con 0 > -1145> 2018-07-11 07:35:23.683868 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6802/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:35:23.683673) v4 -- 0x55a3282b9400 con 0 > -1144> 2018-07-11 07:35:23.683880 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6803/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:35:23.683673) v4 -- 0x55a3282b9e00 con 0 > -1143> 2018-07-11 07:35:23.683893 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6806/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:35:23.683673) v4 -- 0x55a344e85600 con 0 > -1142> 2018-07-11 07:35:23.683910 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6807/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:35:23.683673) v4 -- 0x55a3435ea600 con 0 > -1141> 2018-07-11 07:35:23.683923 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6802/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:35:23.683673) v4 -- 0x55a344e85400 con 0 > -1140> 2018-07-11 07:35:23.683935 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6803/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:35:23.683673) v4 -- 0x55a3282b8200 con 0 > -1139> 2018-07-11 07:35:23.683951 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6806/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:35:23.683673) v4 -- 0x55a32ccf6400 con 0 > -1138> 2018-07-11 07:35:23.683965 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6807/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:35:23.683673) v4 -- 0x55a3282b8a00 con 0 > -1137> 2018-07-11 07:35:23.683978 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6802/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:35:23.683673) v4 -- 0x55a3282b8600 con 0 > -1136> 2018-07-11 07:35:23.683990 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6803/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:35:23.683673) v4 -- 0x55a3282ba000 con 0 > -1135> 2018-07-11 07:35:23.684060 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6802/15255 conn(0x55a314e1c000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1874 0x55a30f772a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:23.683673) v4 > -1134> 2018-07-11 07:35:23.684086 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6802/15255 1874 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:23.683673) v4 ==== 2004+0+0 (2772656190 0 0) 0x55a30f772a00 con 0x55a314e1c000 > -1133> 2018-07-11 07:35:23.684131 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6803/15255 conn(0x55a314e1d800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1874 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:23.683673) v4 > -1132> 2018-07-11 07:35:23.684152 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6803/15255 1874 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:23.683673) v4 ==== 2004+0+0 (2772656190 0 0) 0x55a332e79a00 con 0x55a314e1d800 > -1131> 2018-07-11 07:35:23.684170 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6807/12617 conn(0x55a314e27800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1874 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:23.683673) v4 > -1130> 2018-07-11 07:35:23.684189 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6807/12617 1874 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:23.683673) v4 ==== 2004+0+0 (2772656190 0 0) 0x55a34de78600 con 0x55a314e27800 > -1129> 2018-07-11 07:35:23.684199 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6810/12651 conn(0x55a314e97800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1874 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:23.683673) v4 > -1128> 2018-07-11 07:35:23.684218 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6810/12651 1874 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:23.683673) v4 ==== 2004+0+0 (2772656190 0 0) 0x55a332e79a00 con 0x55a314e97800 > -1127> 2018-07-11 07:35:23.684221 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6807/12677 conn(0x55a314e85800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1874 0x55a30f772a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:23.683673) v4 > -1126> 2018-07-11 07:35:23.684232 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6806/12677 conn(0x55a314e84000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1874 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:23.683673) v4 > -1125> 2018-07-11 07:35:23.684242 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6807/12677 1874 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:23.683673) v4 ==== 2004+0+0 (2772656190 0 0) 0x55a30f772a00 con 0x55a314e85800 > -1124> 2018-07-11 07:35:23.684255 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6806/12617 conn(0x55a314e26000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1874 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:23.683673) v4 > -1123> 2018-07-11 07:35:23.684259 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6806/12677 1874 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:23.683673) v4 ==== 2004+0+0 (2772656190 0 0) 0x55a34de78600 con 0x55a314e84000 > -1122> 2018-07-11 07:35:23.684266 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6806/12617 1874 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:23.683673) v4 ==== 2004+0+0 (2772656190 0 0) 0x55a332e79a00 con 0x55a314e26000 > -1121> 2018-07-11 07:35:23.684295 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6809/12650 conn(0x55a314e1f000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1874 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:23.683673) v4 > -1120> 2018-07-11 07:35:23.684302 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6807/15263 conn(0x55a314e9a800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1874 0x55a30f772a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:23.683673) v4 > -1119> 2018-07-11 07:35:23.684298 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6811/15287 conn(0x55a314e04800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1874 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:23.683673) v4 > -1118> 2018-07-11 07:35:23.684324 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6807/15263 1874 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:23.683673) v4 ==== 2004+0+0 (2772656190 0 0) 0x55a30f772a00 con 0x55a314e9a800 > -1117> 2018-07-11 07:35:23.684308 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6809/12650 1874 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:23.683673) v4 ==== 2004+0+0 (2772656190 0 0) 0x55a34de78600 con 0x55a314e1f000 > -1116> 2018-07-11 07:35:23.684333 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6811/15287 1874 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:23.683673) v4 ==== 2004+0+0 (2772656190 0 0) 0x55a332e79a00 con 0x55a314e04800 > -1115> 2018-07-11 07:35:23.684358 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6808/12651 conn(0x55a314e96000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1874 0x55a30f772a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:23.683673) v4 > -1114> 2018-07-11 07:35:23.684371 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6808/12651 1874 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:23.683673) v4 ==== 2004+0+0 (2772656190 0 0) 0x55a30f772a00 con 0x55a314e96000 > -1113> 2018-07-11 07:35:23.684372 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6806/15263 conn(0x55a314e99000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1874 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:23.683673) v4 > -1112> 2018-07-11 07:35:23.684387 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6806/15263 1874 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:23.683673) v4 ==== 2004+0+0 (2772656190 0 0) 0x55a34de78600 con 0x55a314e99000 > -1111> 2018-07-11 07:35:23.684384 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6802/13323 conn(0x55a314e15000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1874 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:23.683673) v4 > -1110> 2018-07-11 07:35:23.684398 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6802/13323 1874 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:23.683673) v4 ==== 2004+0+0 (2772656190 0 0) 0x55a332e79a00 con 0x55a314e15000 > -1109> 2018-07-11 07:35:23.684400 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6810/15287 conn(0x55a314e03000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1874 0x55a30f772a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:23.683673) v4 > -1108> 2018-07-11 07:35:23.684417 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6803/13323 conn(0x55a314e16800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1874 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:23.683673) v4 > -1107> 2018-07-11 07:35:23.684422 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6810/15287 1874 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:23.683673) v4 ==== 2004+0+0 (2772656190 0 0) 0x55a30f772a00 con 0x55a314e03000 > -1106> 2018-07-11 07:35:23.684429 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6803/13323 1874 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:23.683673) v4 ==== 2004+0+0 (2772656190 0 0) 0x55a34de78600 con 0x55a314e16800 > -1105> 2018-07-11 07:35:23.684452 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6811/12650 conn(0x55a314e20800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1874 0x55a30f772a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:23.683673) v4 > -1104> 2018-07-11 07:35:23.684463 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6811/12650 1874 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:23.683673) v4 ==== 2004+0+0 (2772656190 0 0) 0x55a30f772a00 con 0x55a314e20800 > -1103> 2018-07-11 07:35:23.684462 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6806/15885 conn(0x55a314e00000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1874 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:23.683673) v4 > -1102> 2018-07-11 07:35:23.684480 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6806/15885 1874 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:23.683673) v4 ==== 2004+0+0 (2772656190 0 0) 0x55a332e79a00 con 0x55a314e00000 > -1101> 2018-07-11 07:35:23.684503 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6802/11967 conn(0x55a314e12000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1874 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:23.683673) v4 > -1100> 2018-07-11 07:35:23.684524 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6802/11967 1874 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:23.683673) v4 ==== 2004+0+0 (2772656190 0 0) 0x55a34de78600 con 0x55a314e12000 > -1099> 2018-07-11 07:35:23.684568 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6803/15919 conn(0x55a314e88800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1874 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:23.683673) v4 > -1098> 2018-07-11 07:35:23.684582 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6803/15919 1874 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:23.683673) v4 ==== 2004+0+0 (2772656190 0 0) 0x55a34de78600 con 0x55a314e88800 > -1097> 2018-07-11 07:35:23.684619 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6802/15919 conn(0x55a314e87000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1874 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:23.683673) v4 > -1096> 2018-07-11 07:35:23.684624 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6807/15885 conn(0x55a314e01800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1874 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:23.683673) v4 > -1095> 2018-07-11 07:35:23.684634 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6802/15919 1874 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:23.683673) v4 ==== 2004+0+0 (2772656190 0 0) 0x55a332e79a00 con 0x55a314e87000 > -1094> 2018-07-11 07:35:23.684632 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6803/11967 conn(0x55a314e13800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1874 0x55a30f772a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:23.683673) v4 > -1093> 2018-07-11 07:35:23.684637 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6807/15885 1874 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:23.683673) v4 ==== 2004+0+0 (2772656190 0 0) 0x55a34de78600 con 0x55a314e01800 > -1092> 2018-07-11 07:35:23.684650 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6803/11967 1874 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:23.683673) v4 ==== 2004+0+0 (2772656190 0 0) 0x55a30f772a00 con 0x55a314e13800 > -1091> 2018-07-11 07:35:23.792898 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a329dd8b40 con 0 > -1090> 2018-07-11 07:35:23.818870 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5207 0x55a35302d900 replica scrub(pg: 6.15,from:0'0,to:430'106749,epoch:494/492,start:6:aa3c3c25:::benchmark_data_magna108_35617_object54080:0,end:6:aa3d541e:::benchmark_data_magna108.ceph.redhat.com_2478_object189088:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -1089> 2018-07-11 07:35:23.818898 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5207 ==== replica scrub(pg: 6.15,from:0'0,to:430'106749,epoch:494/492,start:6:aa3c3c25:::benchmark_data_magna108_35617_object54080:0,end:6:aa3d541e:::benchmark_data_magna108.ceph.redhat.com_2478_object189088:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 232+0+0 (3409217040 0 0) 0x55a35302d900 con 0x55a317019000 > -1088> 2018-07-11 07:35:23.959801 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.115:0/13323 conn(0x55a321665800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1875 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:35:24.297924) v4 > -1087> 2018-07-11 07:35:23.959823 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.6 10.8.128.115:0/13323 1875 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:24.297924) v4 ==== 2004+0+0 (726794234 0 0) 0x55a332e79a00 con 0x55a321665800 > -1086> 2018-07-11 07:35:23.959836 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:24.297924) v4 -- 0x55a3435eac00 con 0 > -1085> 2018-07-11 07:35:23.959915 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.115:0/13323 conn(0x55a321668800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1875 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:35:24.297924) v4 > -1084> 2018-07-11 07:35:23.959932 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.6 10.8.128.115:0/13323 1875 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:24.297924) v4 ==== 2004+0+0 (726794234 0 0) 0x55a324474c00 con 0x55a321668800 > -1083> 2018-07-11 07:35:23.959942 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:24.297924) v4 -- 0x55a3435eac00 con 0 > -1082> 2018-07-11 07:35:24.038838 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15287 conn(0x55a311b75000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1863 0x55a30f772a00 osd_ping(ping e494 stamp 2018-07-11 07:35:25.253885) v4 > -1081> 2018-07-11 07:35:24.038862 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.4 10.8.128.113:0/15287 1863 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:25.253885) v4 ==== 2004+0+0 (2140698296 0 0) 0x55a30f772a00 con 0x55a311b75000 > -1080> 2018-07-11 07:35:24.038872 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:25.253885) v4 -- 0x55a315fed000 con 0 > -1079> 2018-07-11 07:35:24.038876 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15287 conn(0x55a311b72000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1863 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:35:25.253885) v4 > -1078> 2018-07-11 07:35:24.038905 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.4 10.8.128.113:0/15287 1863 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:25.253885) v4 ==== 2004+0+0 (2140698296 0 0) 0x55a332e79a00 con 0x55a311b72000 > -1077> 2018-07-11 07:35:24.038926 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:25.253885) v4 -- 0x55a3435eac00 con 0 > -1076> 2018-07-11 07:35:24.039998 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12677 conn(0x55a32169c000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1890 0x55a30f772c00 osd_ping(ping e494 stamp 2018-07-11 07:35:24.076678) v4 > -1075> 2018-07-11 07:35:24.040018 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.1 10.8.128.106:0/12677 1890 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:24.076678) v4 ==== 2004+0+0 (2516924061 0 0) 0x55a30f772c00 con 0x55a32169c000 > -1074> 2018-07-11 07:35:24.040028 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:24.076678) v4 -- 0x55a315fec800 con 0 > -1073> 2018-07-11 07:35:24.040017 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12677 conn(0x55a32169d800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1890 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:35:24.076678) v4 > -1072> 2018-07-11 07:35:24.040038 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.1 10.8.128.106:0/12677 1890 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:24.076678) v4 ==== 2004+0+0 (2516924061 0 0) 0x55a34de78600 con 0x55a32169d800 > -1071> 2018-07-11 07:35:24.040055 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:24.076678) v4 -- 0x55a3282b9e00 con 0 > -1070> 2018-07-11 07:35:24.129944 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15263 conn(0x55a321667000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1867 0x55a30f772a00 osd_ping(ping e494 stamp 2018-07-11 07:35:25.344920) v4 > -1069> 2018-07-11 07:35:24.129975 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.5 10.8.128.113:0/15263 1867 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:25.344920) v4 ==== 2004+0+0 (399101221 0 0) 0x55a30f772a00 con 0x55a321667000 > -1068> 2018-07-11 07:35:24.129990 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:25.344920) v4 -- 0x55a315fed600 con 0 > -1067> 2018-07-11 07:35:24.130048 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15263 conn(0x55a321664000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1867 0x55a30f772c00 osd_ping(ping e494 stamp 2018-07-11 07:35:25.344920) v4 > -1066> 2018-07-11 07:35:24.130062 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.5 10.8.128.113:0/15263 1867 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:25.344920) v4 ==== 2004+0+0 (399101221 0 0) 0x55a30f772c00 con 0x55a321664000 > -1065> 2018-07-11 07:35:24.130072 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:25.344920) v4 -- 0x55a331d5d800 con 0 > -1064> 2018-07-11 07:35:24.539215 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15255 conn(0x55a311b65800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1894 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:35:25.754324) v4 > -1063> 2018-07-11 07:35:24.539243 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.0 10.8.128.113:0/15255 1894 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:25.754324) v4 ==== 2004+0+0 (1388700036 0 0) 0x55a324474c00 con 0x55a311b65800 > -1062> 2018-07-11 07:35:24.539265 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:25.754324) v4 -- 0x55a315feca00 con 0 > -1061> 2018-07-11 07:35:24.539296 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15255 conn(0x55a311b64000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1894 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:35:25.754324) v4 > -1060> 2018-07-11 07:35:24.539318 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.0 10.8.128.113:0/15255 1894 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:25.754324) v4 ==== 2004+0+0 (1388700036 0 0) 0x55a34de6f000 con 0x55a311b64000 > -1059> 2018-07-11 07:35:24.539331 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:25.754324) v4 -- 0x55a344e84e00 con 0 > -1058> 2018-07-11 07:35:24.566649 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15919 conn(0x55a317074800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1870 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:35:25.190913) v4 > -1057> 2018-07-11 07:35:24.566670 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.11 10.8.128.114:0/15919 1870 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:25.190913) v4 ==== 2004+0+0 (2015542719 0 0) 0x55a332e79a00 con 0x55a317074800 > -1056> 2018-07-11 07:35:24.566679 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:25.190913) v4 -- 0x55a315feca00 con 0 > -1055> 2018-07-11 07:35:24.566681 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15919 conn(0x55a3170b4000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1870 0x55a30f772a00 osd_ping(ping e494 stamp 2018-07-11 07:35:25.190913) v4 > -1054> 2018-07-11 07:35:24.566701 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.11 10.8.128.114:0/15919 1870 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:25.190913) v4 ==== 2004+0+0 (2015542719 0 0) 0x55a30f772a00 con 0x55a3170b4000 > -1053> 2018-07-11 07:35:24.566717 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:25.190913) v4 -- 0x55a331d5d800 con 0 > -1052> 2018-07-11 07:35:24.972791 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a332f71200 con 0 > -1051> 2018-07-11 07:35:24.985938 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5208 0x55a35302db80 replica scrub(pg: 6.15,from:0'0,to:430'106820,epoch:494/492,start:6:aa3d541e:::benchmark_data_magna108.ceph.redhat.com_2478_object189088:0,end:6:aa3e9d83:::benchmark_data_magna108_35617_object454740:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -1050> 2018-07-11 07:35:24.985962 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5208 ==== replica scrub(pg: 6.15,from:0'0,to:430'106820,epoch:494/492,start:6:aa3d541e:::benchmark_data_magna108.ceph.redhat.com_2478_object189088:0,end:6:aa3e9d83:::benchmark_data_magna108_35617_object454740:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (2630280734 0 0) 0x55a35302db80 con 0x55a317019000 > -1049> 2018-07-11 07:35:25.060242 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.115:0/13323 conn(0x55a321665800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1876 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:35:25.398370) v4 > -1048> 2018-07-11 07:35:25.060281 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.6 10.8.128.115:0/13323 1876 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:25.398370) v4 ==== 2004+0+0 (3937509008 0 0) 0x55a324474c00 con 0x55a321665800 > -1047> 2018-07-11 07:35:25.060296 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:25.398370) v4 -- 0x55a315feca00 con 0 > -1046> 2018-07-11 07:35:25.060385 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.115:0/13323 conn(0x55a321668800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1876 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:35:25.398370) v4 > -1045> 2018-07-11 07:35:25.060403 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.6 10.8.128.115:0/13323 1876 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:25.398370) v4 ==== 2004+0+0 (3937509008 0 0) 0x55a332e79a00 con 0x55a321668800 > -1044> 2018-07-11 07:35:25.060414 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:25.398370) v4 -- 0x55a315feca00 con 0 > -1043> 2018-07-11 07:35:25.175387 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12651 conn(0x55a32169f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1867 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:35:25.211996) v4 > -1042> 2018-07-11 07:35:25.175391 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12651 conn(0x55a3216a0800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1867 0x55a30f772c00 osd_ping(ping e494 stamp 2018-07-11 07:35:25.211996) v4 > -1041> 2018-07-11 07:35:25.175409 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.2 10.8.128.106:0/12651 1867 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:25.211996) v4 ==== 2004+0+0 (2098253190 0 0) 0x55a324474c00 con 0x55a32169f000 > -1040> 2018-07-11 07:35:25.175413 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.2 10.8.128.106:0/12651 1867 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:25.211996) v4 ==== 2004+0+0 (2098253190 0 0) 0x55a30f772c00 con 0x55a3216a0800 > -1039> 2018-07-11 07:35:25.175427 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:25.211996) v4 -- 0x55a315feca00 con 0 > -1038> 2018-07-11 07:35:25.175458 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:25.211996) v4 -- 0x55a331d5d800 con 0 > -1037> 2018-07-11 07:35:25.388210 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12650 conn(0x55a311b68800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1883 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:35:25.424854) v4 > -1036> 2018-07-11 07:35:25.388231 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.3 10.8.128.106:0/12650 1883 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:25.424854) v4 ==== 2004+0+0 (1440438278 0 0) 0x55a34de78600 con 0x55a311b68800 > -1035> 2018-07-11 07:35:25.388241 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:25.424854) v4 -- 0x55a344e84e00 con 0 > -1034> 2018-07-11 07:35:25.388232 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12650 conn(0x55a311b67000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1883 0x55a30f772a00 osd_ping(ping e494 stamp 2018-07-11 07:35:25.424854) v4 > -1033> 2018-07-11 07:35:25.388263 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.3 10.8.128.106:0/12650 1883 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:25.424854) v4 ==== 2004+0+0 (1440438278 0 0) 0x55a30f772a00 con 0x55a311b67000 > -1032> 2018-07-11 07:35:25.388281 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:25.424854) v4 -- 0x55a30f773800 con 0 > -1031> 2018-07-11 07:35:25.666501 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a32c0aa900 con 0 > -1030> 2018-07-11 07:35:25.677797 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15885 conn(0x55a311b76800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1890 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:35:26.302098) v4 > -1029> 2018-07-11 07:35:25.677806 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15885 conn(0x55a3170b5800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1890 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:35:26.302098) v4 > -1028> 2018-07-11 07:35:25.677818 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.7 10.8.128.114:0/15885 1890 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:26.302098) v4 ==== 2004+0+0 (3673757485 0 0) 0x55a34de6f000 con 0x55a311b76800 > -1027> 2018-07-11 07:35:25.677824 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.7 10.8.128.114:0/15885 1890 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:26.302098) v4 ==== 2004+0+0 (3673757485 0 0) 0x55a332e79a00 con 0x55a3170b5800 > -1026> 2018-07-11 07:35:25.677828 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:26.302098) v4 -- 0x55a315fec600 con 0 > -1025> 2018-07-11 07:35:25.677863 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:26.302098) v4 -- 0x55a327787e00 con 0 > -1024> 2018-07-11 07:35:25.686259 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5209 0x55a312272000 replica scrub(pg: 6.15,from:0'0,to:430'106396,epoch:494/492,start:6:aa3e9d83:::benchmark_data_magna108_35617_object454740:0,end:6:aa3facc5:::benchmark_data_magna108_35617_object704373:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -1023> 2018-07-11 07:35:25.686282 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5209 ==== replica scrub(pg: 6.15,from:0'0,to:430'106396,epoch:494/492,start:6:aa3e9d83:::benchmark_data_magna108_35617_object454740:0,end:6:aa3facc5:::benchmark_data_magna108_35617_object704373:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (308424556 0 0) 0x55a312272000 con 0x55a317019000 > -1022> 2018-07-11 07:35:25.763677 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b7000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1819 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:35:25.514709) v4 > -1021> 2018-07-11 07:35:25.763698 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.8 10.8.128.108:0/11967 1819 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:25.514709) v4 ==== 2004+0+0 (1137352490 0 0) 0x55a34de78600 con 0x55a3170b7000 > -1020> 2018-07-11 07:35:25.763689 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b8800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1819 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:35:25.514709) v4 > -1019> 2018-07-11 07:35:25.763707 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:25.514709) v4 -- 0x55a315fec600 con 0 > -1018> 2018-07-11 07:35:25.763711 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.8 10.8.128.108:0/11967 1819 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:25.514709) v4 ==== 2004+0+0 (1137352490 0 0) 0x55a324474c00 con 0x55a3170b8800 > -1017> 2018-07-11 07:35:25.763731 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:25.514709) v4 -- 0x55a327787e00 con 0 > -1016> 2018-07-11 07:35:26.426017 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a32a871680 con 0 > -1015> 2018-07-11 07:35:26.448667 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5210 0x55a312272280 replica scrub(pg: 6.15,from:0'0,to:461'106922,epoch:494/492,start:6:aa3facc5:::benchmark_data_magna108_35617_object704373:0,end:6:aa40a54b:::benchmark_data_magna108_35617_object954325:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -1014> 2018-07-11 07:35:26.448696 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5210 ==== replica scrub(pg: 6.15,from:0'0,to:461'106922,epoch:494/492,start:6:aa3facc5:::benchmark_data_magna108_35617_object704373:0,end:6:aa40a54b:::benchmark_data_magna108_35617_object954325:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (3708871493 0 0) 0x55a312272280 con 0x55a317019000 > -1013> 2018-07-11 07:35:26.745553 7f35498a5700 10 monclient: tick > -1012> 2018-07-11 07:35:26.745566 7f35498a5700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2018-07-11 07:34:56.745564) > -1011> 2018-07-11 07:35:27.184170 7f3543899700 5 osd.10 494 heartbeat: osd_stat(785 GB used, 146 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,0,0,1]) > -1010> 2018-07-11 07:35:27.184186 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6802/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:35:27.184184) v4 -- 0x55a3282b9200 con 0 > -1009> 2018-07-11 07:35:27.184225 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6803/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:35:27.184184) v4 -- 0x55a3282b8e00 con 0 > -1008> 2018-07-11 07:35:27.184271 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6806/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:35:27.184184) v4 -- 0x55a3282b8800 con 0 > -1007> 2018-07-11 07:35:27.184290 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6807/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:35:27.184184) v4 -- 0x55a3282b9000 con 0 > -1006> 2018-07-11 07:35:27.184301 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6808/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:35:27.184184) v4 -- 0x55a3282b9800 con 0 > -1005> 2018-07-11 07:35:27.184313 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6810/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:35:27.184184) v4 -- 0x55a3282b9a00 con 0 > -1004> 2018-07-11 07:35:27.184325 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6809/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:35:27.184184) v4 -- 0x55a3282b9600 con 0 > -1003> 2018-07-11 07:35:27.184337 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6811/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:35:27.184184) v4 -- 0x55a3282baa00 con 0 > -1002> 2018-07-11 07:35:27.184348 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6810/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:35:27.184184) v4 -- 0x55a3282ba600 con 0 > -1001> 2018-07-11 07:35:27.184359 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6811/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:35:27.184184) v4 -- 0x55a3282b9c00 con 0 > -1000> 2018-07-11 07:35:27.184384 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6806/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:35:27.184184) v4 -- 0x55a317045400 con 0 > -999> 2018-07-11 07:35:27.184399 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6807/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:35:27.184184) v4 -- 0x55a32ccf6600 con 0 > -998> 2018-07-11 07:35:27.184416 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6802/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:35:27.184184) v4 -- 0x55a317642000 con 0 > -997> 2018-07-11 07:35:27.184433 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6803/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:35:27.184184) v4 -- 0x55a317642200 con 0 > -996> 2018-07-11 07:35:27.184447 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6806/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:35:27.184184) v4 -- 0x55a317642400 con 0 > -995> 2018-07-11 07:35:27.184463 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6807/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:35:27.184184) v4 -- 0x55a317642600 con 0 > -994> 2018-07-11 07:35:27.184477 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6802/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:35:27.184184) v4 -- 0x55a317642800 con 0 > -993> 2018-07-11 07:35:27.184489 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6803/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:35:27.184184) v4 -- 0x55a317642a00 con 0 > -992> 2018-07-11 07:35:27.184502 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6806/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:35:27.184184) v4 -- 0x55a317642c00 con 0 > -991> 2018-07-11 07:35:27.184517 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6807/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:35:27.184184) v4 -- 0x55a317642e00 con 0 > -990> 2018-07-11 07:35:27.184530 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6802/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:35:27.184184) v4 -- 0x55a317643000 con 0 > -989> 2018-07-11 07:35:27.184542 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6803/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:35:27.184184) v4 -- 0x55a317643200 con 0 > -988> 2018-07-11 07:35:27.184729 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6803/15255 conn(0x55a314e1d800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1875 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:27.184184) v4 > -987> 2018-07-11 07:35:27.184754 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6803/15255 1875 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:27.184184) v4 ==== 2004+0+0 (4196894940 0 0) 0x55a324474c00 con 0x55a314e1d800 > -986> 2018-07-11 07:35:27.184760 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6807/12617 conn(0x55a314e27800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1875 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:27.184184) v4 > -985> 2018-07-11 07:35:27.184786 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6810/15287 conn(0x55a314e03000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1875 0x55a30f773e00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:27.184184) v4 > -984> 2018-07-11 07:35:27.184816 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6807/12617 1875 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:27.184184) v4 ==== 2004+0+0 (4196894940 0 0) 0x55a34de6f000 con 0x55a314e27800 > -983> 2018-07-11 07:35:27.184815 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6806/12617 conn(0x55a314e26000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1875 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:27.184184) v4 > -982> 2018-07-11 07:35:27.184838 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6806/12617 1875 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:27.184184) v4 ==== 2004+0+0 (4196894940 0 0) 0x55a324474c00 con 0x55a314e26000 > -981> 2018-07-11 07:35:27.184818 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6810/15287 1875 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:27.184184) v4 ==== 2004+0+0 (4196894940 0 0) 0x55a30f773e00 con 0x55a314e03000 > -980> 2018-07-11 07:35:27.184874 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6806/12677 conn(0x55a314e84000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1875 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:27.184184) v4 > -979> 2018-07-11 07:35:27.184895 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6806/12677 1875 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:27.184184) v4 ==== 2004+0+0 (4196894940 0 0) 0x55a34de6f000 con 0x55a314e84000 > -978> 2018-07-11 07:35:27.184915 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6811/12650 conn(0x55a314e20800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1875 0x55a30f773e00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:27.184184) v4 > -977> 2018-07-11 07:35:27.184931 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6811/12650 1875 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:27.184184) v4 ==== 2004+0+0 (4196894940 0 0) 0x55a30f773e00 con 0x55a314e20800 > -976> 2018-07-11 07:35:27.184928 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6811/15287 conn(0x55a314e04800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1875 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:27.184184) v4 > -975> 2018-07-11 07:35:27.184942 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6809/12650 conn(0x55a314e1f000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1875 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:27.184184) v4 > -974> 2018-07-11 07:35:27.184958 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6809/12650 1875 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:27.184184) v4 ==== 2004+0+0 (4196894940 0 0) 0x55a34de6f000 con 0x55a314e1f000 > -973> 2018-07-11 07:35:27.184945 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6811/15287 1875 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:27.184184) v4 ==== 2004+0+0 (4196894940 0 0) 0x55a324474c00 con 0x55a314e04800 > -972> 2018-07-11 07:35:27.184989 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6808/12651 conn(0x55a314e96000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1875 0x55a30f773e00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:27.184184) v4 > -971> 2018-07-11 07:35:27.185006 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6808/12651 1875 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:27.184184) v4 ==== 2004+0+0 (4196894940 0 0) 0x55a30f773e00 con 0x55a314e96000 > -970> 2018-07-11 07:35:27.185005 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6806/15263 conn(0x55a314e99000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1875 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:27.184184) v4 > -969> 2018-07-11 07:35:27.185019 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6806/15885 conn(0x55a314e00000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1875 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:27.184184) v4 > -968> 2018-07-11 07:35:27.185026 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6806/15263 1875 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:27.184184) v4 ==== 2004+0+0 (4196894940 0 0) 0x55a34de6f000 con 0x55a314e99000 > -967> 2018-07-11 07:35:27.185034 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6806/15885 1875 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:27.184184) v4 ==== 2004+0+0 (4196894940 0 0) 0x55a324474c00 con 0x55a314e00000 > -966> 2018-07-11 07:35:27.185035 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6802/15255 conn(0x55a314e1c000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1875 0x55a30f773e00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:27.184184) v4 > -965> 2018-07-11 07:35:27.185049 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6802/15255 1875 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:27.184184) v4 ==== 2004+0+0 (4196894940 0 0) 0x55a30f773e00 con 0x55a314e1c000 > -964> 2018-07-11 07:35:27.185068 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6803/13323 conn(0x55a314e16800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1875 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:27.184184) v4 > -963> 2018-07-11 07:35:27.185086 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6803/13323 1875 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:27.184184) v4 ==== 2004+0+0 (4196894940 0 0) 0x55a34de6f000 con 0x55a314e16800 > -962> 2018-07-11 07:35:27.185108 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6810/12651 conn(0x55a314e97800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1875 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:27.184184) v4 > -961> 2018-07-11 07:35:27.185118 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6807/12677 conn(0x55a314e85800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1875 0x55a30f773e00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:27.184184) v4 > -960> 2018-07-11 07:35:27.185128 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6810/12651 1875 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:27.184184) v4 ==== 2004+0+0 (4196894940 0 0) 0x55a324474c00 con 0x55a314e97800 > -959> 2018-07-11 07:35:27.185131 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6807/12677 1875 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:27.184184) v4 ==== 2004+0+0 (4196894940 0 0) 0x55a30f773e00 con 0x55a314e85800 > -958> 2018-07-11 07:35:27.185136 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6807/15885 conn(0x55a314e01800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1875 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:27.184184) v4 > -957> 2018-07-11 07:35:27.185150 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6807/15885 1875 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:27.184184) v4 ==== 2004+0+0 (4196894940 0 0) 0x55a34de6f000 con 0x55a314e01800 > -956> 2018-07-11 07:35:27.185182 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6807/15263 conn(0x55a314e9a800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1875 0x55a30f773e00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:27.184184) v4 > -955> 2018-07-11 07:35:27.185179 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6802/13323 conn(0x55a314e15000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1875 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:27.184184) v4 > -954> 2018-07-11 07:35:27.185192 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6803/15919 conn(0x55a314e88800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1875 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:27.184184) v4 > -953> 2018-07-11 07:35:27.185198 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6802/13323 1875 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:27.184184) v4 ==== 2004+0+0 (4196894940 0 0) 0x55a324474c00 con 0x55a314e15000 > -952> 2018-07-11 07:35:27.185197 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6807/15263 1875 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:27.184184) v4 ==== 2004+0+0 (4196894940 0 0) 0x55a30f773e00 con 0x55a314e9a800 > -951> 2018-07-11 07:35:27.185208 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6803/15919 1875 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:27.184184) v4 ==== 2004+0+0 (4196894940 0 0) 0x55a34de6f000 con 0x55a314e88800 > -950> 2018-07-11 07:35:27.185260 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6803/11967 conn(0x55a314e13800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1875 0x55a30f773e00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:27.184184) v4 > -949> 2018-07-11 07:35:27.185258 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6802/15919 conn(0x55a314e87000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1875 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:27.184184) v4 > -948> 2018-07-11 07:35:27.185265 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6802/11967 conn(0x55a314e12000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1875 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:27.184184) v4 > -947> 2018-07-11 07:35:27.185276 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6803/11967 1875 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:27.184184) v4 ==== 2004+0+0 (4196894940 0 0) 0x55a30f773e00 con 0x55a314e13800 > -946> 2018-07-11 07:35:27.185277 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6802/15919 1875 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:27.184184) v4 ==== 2004+0+0 (4196894940 0 0) 0x55a324474c00 con 0x55a314e87000 > -945> 2018-07-11 07:35:27.185277 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6802/11967 1875 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:27.184184) v4 ==== 2004+0+0 (4196894940 0 0) 0x55a34de6f000 con 0x55a314e12000 > -944> 2018-07-11 07:35:27.258739 7f35488a3700 1 -- 10.8.128.107:6800/12665 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55a3170ea2c0 con 0 > -943> 2018-07-11 07:35:27.258807 7f35488a3700 1 -- 10.8.128.107:6800/12665 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55a30fa24dc0 con 0 > -942> 2018-07-11 07:35:27.293967 7f355d88e700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.107:0/12617 conn(0x55a32447f800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1843 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:35:27.293479) v4 > -941> 2018-07-11 07:35:27.293967 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.107:0/12617 conn(0x55a32447e000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1843 0x55a30f773e00 osd_ping(ping e494 stamp 2018-07-11 07:35:27.293479) v4 > -940> 2018-07-11 07:35:27.294016 7f355d88e700 1 -- 10.8.128.107:6803/12665 <== osd.9 10.8.128.107:0/12617 1843 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:27.293479) v4 ==== 2004+0+0 (3386122589 0 0) 0x55a34de6f000 con 0x55a32447f800 > -939> 2018-07-11 07:35:27.294039 7f355d88e700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:27.293479) v4 -- 0x55a3282bba00 con 0 > -938> 2018-07-11 07:35:27.294016 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.9 10.8.128.107:0/12617 1843 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:27.293479) v4 ==== 2004+0+0 (3386122589 0 0) 0x55a30f773e00 con 0x55a32447e000 > -937> 2018-07-11 07:35:27.294079 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:27.293479) v4 -- 0x55a34da1a200 con 0 > -936> 2018-07-11 07:35:27.467045 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15919 conn(0x55a3170b4000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1871 0x55a30f772c00 osd_ping(ping e494 stamp 2018-07-11 07:35:28.091292) v4 > -935> 2018-07-11 07:35:27.467071 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.11 10.8.128.114:0/15919 1871 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:28.091292) v4 ==== 2004+0+0 (957631490 0 0) 0x55a30f772c00 con 0x55a3170b4000 > -934> 2018-07-11 07:35:27.467082 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:28.091292) v4 -- 0x55a322454600 con 0 > -933> 2018-07-11 07:35:27.467077 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15919 conn(0x55a317074800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1871 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:35:28.091292) v4 > -932> 2018-07-11 07:35:27.467100 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.11 10.8.128.114:0/15919 1871 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:28.091292) v4 ==== 2004+0+0 (957631490 0 0) 0x55a324474c00 con 0x55a317074800 > -931> 2018-07-11 07:35:27.467116 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:28.091292) v4 -- 0x55a327787e00 con 0 > -930> 2018-07-11 07:35:27.540450 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12677 conn(0x55a32169d800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1891 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:35:27.577130) v4 > -929> 2018-07-11 07:35:27.540471 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.1 10.8.128.106:0/12677 1891 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:27.577130) v4 ==== 2004+0+0 (1803357110 0 0) 0x55a34de78600 con 0x55a32169d800 > -928> 2018-07-11 07:35:27.540463 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12677 conn(0x55a32169c000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1891 0x55a30f773e00 osd_ping(ping e494 stamp 2018-07-11 07:35:27.577130) v4 > -927> 2018-07-11 07:35:27.540481 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:27.577130) v4 -- 0x55a331c67e00 con 0 > -926> 2018-07-11 07:35:27.540484 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.1 10.8.128.106:0/12677 1891 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:27.577130) v4 ==== 2004+0+0 (1803357110 0 0) 0x55a30f773e00 con 0x55a32169c000 > -925> 2018-07-11 07:35:27.540503 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:27.577130) v4 -- 0x55a327787c00 con 0 > -924> 2018-07-11 07:35:27.978203 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15885 conn(0x55a311b76800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1891 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:35:28.602528) v4 > -923> 2018-07-11 07:35:27.978206 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15885 conn(0x55a3170b5800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1891 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:35:28.602528) v4 > -922> 2018-07-11 07:35:27.978228 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.7 10.8.128.114:0/15885 1891 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:28.602528) v4 ==== 2004+0+0 (4074448302 0 0) 0x55a34de6f000 con 0x55a311b76800 > -921> 2018-07-11 07:35:27.978232 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.7 10.8.128.114:0/15885 1891 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:28.602528) v4 ==== 2004+0+0 (4074448302 0 0) 0x55a332e79a00 con 0x55a3170b5800 > -920> 2018-07-11 07:35:27.978237 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:28.602528) v4 -- 0x55a331c67e00 con 0 > -919> 2018-07-11 07:35:27.978277 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:28.602528) v4 -- 0x55a327787e00 con 0 > -918> 2018-07-11 07:35:28.173950 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a324b8cd80 con 0 > -917> 2018-07-11 07:35:28.186694 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5211 0x55a312272500 replica scrub(pg: 6.15,from:0'0,to:430'105450,epoch:494/492,start:6:aa40a54b:::benchmark_data_magna108_35617_object954325:0,end:6:aa41b2b3:::benchmark_data_magna108.ceph.redhat.com_2478_object279727:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -916> 2018-07-11 07:35:28.186717 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5211 ==== replica scrub(pg: 6.15,from:0'0,to:430'105450,epoch:494/492,start:6:aa40a54b:::benchmark_data_magna108_35617_object954325:0,end:6:aa41b2b3:::benchmark_data_magna108.ceph.redhat.com_2478_object279727:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (1864476335 0 0) 0x55a312272500 con 0x55a317019000 > -915> 2018-07-11 07:35:28.230278 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15263 conn(0x55a321664000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1868 0x55a30f772c00 osd_ping(ping e494 stamp 2018-07-11 07:35:29.445476) v4 > -914> 2018-07-11 07:35:28.230300 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.5 10.8.128.113:0/15263 1868 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:29.445476) v4 ==== 2004+0+0 (1963362624 0 0) 0x55a30f772c00 con 0x55a321664000 > -913> 2018-07-11 07:35:28.230309 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:29.445476) v4 -- 0x55a327787c00 con 0 > -912> 2018-07-11 07:35:28.230345 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15263 conn(0x55a321667000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1868 0x55a30f772c00 osd_ping(ping e494 stamp 2018-07-11 07:35:29.445476) v4 > -911> 2018-07-11 07:35:28.230358 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.5 10.8.128.113:0/15263 1868 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:29.445476) v4 ==== 2004+0+0 (1963362624 0 0) 0x55a30f772c00 con 0x55a321667000 > -910> 2018-07-11 07:35:28.230367 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:29.445476) v4 -- 0x55a327788c00 con 0 > -909> 2018-07-11 07:35:28.394600 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.107:0/12617 conn(0x55a32447e000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1844 0x55a30f773e00 osd_ping(ping e494 stamp 2018-07-11 07:35:28.394136) v4 > -908> 2018-07-11 07:35:28.394642 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.9 10.8.128.107:0/12617 1844 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:28.394136) v4 ==== 2004+0+0 (994651658 0 0) 0x55a30f773e00 con 0x55a32447e000 > -907> 2018-07-11 07:35:28.394641 7f355d88e700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.107:0/12617 conn(0x55a32447f800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1844 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:35:28.394136) v4 > -906> 2018-07-11 07:35:28.394655 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:28.394136) v4 -- 0x55a327788c00 con 0 > -905> 2018-07-11 07:35:28.394666 7f355d88e700 1 -- 10.8.128.107:6803/12665 <== osd.9 10.8.128.107:0/12617 1844 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:28.394136) v4 ==== 2004+0+0 (994651658 0 0) 0x55a34de78600 con 0x55a32447f800 > -904> 2018-07-11 07:35:28.394687 7f355d88e700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:28.394136) v4 -- 0x55a331c67e00 con 0 > -903> 2018-07-11 07:35:28.449107 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 61] Generated table #36598: 2298857 keys, 73864103 bytes > -902> 2018-07-11 07:35:28.449150 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294528449124, "cf_name": "default", "job": 61, "event": "table_file_creation", "file_number": 36598, "file_size": 73864103, "table_properties": {"data_size": 67109070, "index_size": 855721, "filter_size": 5898320, "raw_key_size": 67525849, "raw_average_key_size": 29, "raw_value_size": 35594171, "raw_average_value_size": 15, "num_data_blocks": 16748, "num_entries": 2298857, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1595309", "kMergeOperands": "0"}} > -901> 2018-07-11 07:35:29.063343 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a329dfc000 con 0 > -900> 2018-07-11 07:35:29.078520 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5212 0x55a312272780 replica scrub(pg: 6.15,from:0'0,to:430'105654,epoch:494/492,start:6:aa41b2b3:::benchmark_data_magna108.ceph.redhat.com_2478_object279727:0,end:6:aa42e82c:::benchmark_data_magna108_35617_object194385:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -899> 2018-07-11 07:35:29.078543 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5212 ==== replica scrub(pg: 6.15,from:0'0,to:430'105654,epoch:494/492,start:6:aa41b2b3:::benchmark_data_magna108.ceph.redhat.com_2478_object279727:0,end:6:aa42e82c:::benchmark_data_magna108_35617_object194385:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (232654436 0 0) 0x55a312272780 con 0x55a317019000 > -898> 2018-07-11 07:35:29.275910 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12651 conn(0x55a3216a0800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1868 0x55a30f772c00 osd_ping(ping e494 stamp 2018-07-11 07:35:29.312581) v4 > -897> 2018-07-11 07:35:29.275931 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.2 10.8.128.106:0/12651 1868 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:29.312581) v4 ==== 2004+0+0 (316778895 0 0) 0x55a30f772c00 con 0x55a3216a0800 > -896> 2018-07-11 07:35:29.275941 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:29.312581) v4 -- 0x55a327787c00 con 0 > -895> 2018-07-11 07:35:29.275937 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12651 conn(0x55a32169f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1868 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:35:29.312581) v4 > -894> 2018-07-11 07:35:29.275957 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.2 10.8.128.106:0/12651 1868 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:29.312581) v4 ==== 2004+0+0 (316778895 0 0) 0x55a324474c00 con 0x55a32169f000 > -893> 2018-07-11 07:35:29.275971 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:29.312581) v4 -- 0x55a327787e00 con 0 > -892> 2018-07-11 07:35:29.330721 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15263 conn(0x55a321667000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1869 0x55a327788c00 osd_ping(ping e494 stamp 2018-07-11 07:35:30.545916) v4 > -891> 2018-07-11 07:35:29.330748 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.5 10.8.128.113:0/15263 1869 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:30.545916) v4 ==== 2004+0+0 (1880569347 0 0) 0x55a327788c00 con 0x55a321667000 > -890> 2018-07-11 07:35:29.330757 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:30.545916) v4 -- 0x55a327787c00 con 0 > -889> 2018-07-11 07:35:29.330794 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15263 conn(0x55a321664000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1869 0x55a327788c00 osd_ping(ping e494 stamp 2018-07-11 07:35:30.545916) v4 > -888> 2018-07-11 07:35:29.330807 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.5 10.8.128.113:0/15263 1869 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:30.545916) v4 ==== 2004+0+0 (1880569347 0 0) 0x55a327788c00 con 0x55a321664000 > -887> 2018-07-11 07:35:29.330817 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:30.545916) v4 -- 0x55a331d5cc00 con 0 > -886> 2018-07-11 07:35:29.339235 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15287 conn(0x55a311b75000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1864 0x55a30f772c00 osd_ping(ping e494 stamp 2018-07-11 07:35:30.554411) v4 > -885> 2018-07-11 07:35:29.339266 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.4 10.8.128.113:0/15287 1864 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:30.554411) v4 ==== 2004+0+0 (393499811 0 0) 0x55a30f772c00 con 0x55a311b75000 > -884> 2018-07-11 07:35:29.339277 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:30.554411) v4 -- 0x55a327787c00 con 0 > -883> 2018-07-11 07:35:29.339285 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15287 conn(0x55a311b72000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1864 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:35:30.554411) v4 > -882> 2018-07-11 07:35:29.339306 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.4 10.8.128.113:0/15287 1864 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:30.554411) v4 ==== 2004+0+0 (393499811 0 0) 0x55a332e79a00 con 0x55a311b72000 > -881> 2018-07-11 07:35:29.339317 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:30.554411) v4 -- 0x55a324b15e00 con 0 > -880> 2018-07-11 07:35:29.720276 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a325e26b40 con 0 > -879> 2018-07-11 07:35:29.744768 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5213 0x55a312272a00 replica scrub(pg: 6.15,from:0'0,to:430'106392,epoch:494/492,start:6:aa42e82c:::benchmark_data_magna108_35617_object194385:0,end:6:aa438efa:::benchmark_data_magna108_35617_object660123:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -878> 2018-07-11 07:35:29.744797 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5213 ==== replica scrub(pg: 6.15,from:0'0,to:430'106392,epoch:494/492,start:6:aa42e82c:::benchmark_data_magna108_35617_object194385:0,end:6:aa438efa:::benchmark_data_magna108_35617_object660123:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (21484430 0 0) 0x55a312272a00 con 0x55a317019000 > -877> 2018-07-11 07:35:30.360976 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.115:0/13323 conn(0x55a321668800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1877 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:35:30.699012) v4 > -876> 2018-07-11 07:35:30.361003 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.6 10.8.128.115:0/13323 1877 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:30.699012) v4 ==== 2004+0+0 (884570267 0 0) 0x55a324474c00 con 0x55a321668800 > -875> 2018-07-11 07:35:30.361014 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:30.699012) v4 -- 0x55a324b15e00 con 0 > -874> 2018-07-11 07:35:30.361064 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.115:0/13323 conn(0x55a321665800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1877 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:35:30.699012) v4 > -873> 2018-07-11 07:35:30.361078 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.6 10.8.128.115:0/13323 1877 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:30.699012) v4 ==== 2004+0+0 (884570267 0 0) 0x55a324474c00 con 0x55a321665800 > -872> 2018-07-11 07:35:30.361087 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:30.699012) v4 -- 0x55a33e564000 con 0 > -871> 2018-07-11 07:35:30.439942 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15287 conn(0x55a311b75000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1865 0x55a327788c00 osd_ping(ping e494 stamp 2018-07-11 07:35:31.655067) v4 > -870> 2018-07-11 07:35:30.439954 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15255 conn(0x55a311b65800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1895 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:35:31.654891) v4 > -869> 2018-07-11 07:35:30.439964 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.4 10.8.128.113:0/15287 1865 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:31.655067) v4 ==== 2004+0+0 (916453276 0 0) 0x55a327788c00 con 0x55a311b75000 > -868> 2018-07-11 07:35:30.439970 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.0 10.8.128.113:0/15255 1895 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:31.654891) v4 ==== 2004+0+0 (536099916 0 0) 0x55a332e79a00 con 0x55a311b65800 > -867> 2018-07-11 07:35:30.439979 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:31.654891) v4 -- 0x55a324b15e00 con 0 > -866> 2018-07-11 07:35:30.440025 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:31.655067) v4 -- 0x55a349f88000 con 0 > -865> 2018-07-11 07:35:30.440025 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15287 conn(0x55a311b72000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1865 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:35:31.655067) v4 > -864> 2018-07-11 07:35:30.440046 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.4 10.8.128.113:0/15287 1865 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:31.655067) v4 ==== 2004+0+0 (916453276 0 0) 0x55a332e79a00 con 0x55a311b72000 > -863> 2018-07-11 07:35:30.440057 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:31.655067) v4 -- 0x55a3282bac00 con 0 > -862> 2018-07-11 07:35:30.440051 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15255 conn(0x55a311b64000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1895 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:35:31.654891) v4 > -861> 2018-07-11 07:35:30.440068 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.0 10.8.128.113:0/15255 1895 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:31.654891) v4 ==== 2004+0+0 (536099916 0 0) 0x55a34de6f000 con 0x55a311b64000 > -860> 2018-07-11 07:35:30.440080 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:31.654891) v4 -- 0x55a331c66a00 con 0 > -859> 2018-07-11 07:35:30.460732 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 61] Generated table #36599: 1404691 keys, 22768272 bytes > -858> 2018-07-11 07:35:30.460762 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294530460748, "cf_name": "default", "job": 61, "event": "table_file_creation", "file_number": 36599, "file_size": 22768272, "table_properties": {"data_size": 19016491, "index_size": 194922, "filter_size": 3555872, "raw_key_size": 40579972, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 4667, "num_entries": 1404691, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1404691", "kMergeOperands": "0"}} > -857> 2018-07-11 07:35:30.688522 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12650 conn(0x55a311b68800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1884 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:35:30.725238) v4 > -856> 2018-07-11 07:35:30.688522 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12650 conn(0x55a311b67000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1884 0x55a327787c00 osd_ping(ping e494 stamp 2018-07-11 07:35:30.725238) v4 > -855> 2018-07-11 07:35:30.688543 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.3 10.8.128.106:0/12650 1884 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:30.725238) v4 ==== 2004+0+0 (968796212 0 0) 0x55a34de78600 con 0x55a311b68800 > -854> 2018-07-11 07:35:30.688547 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.3 10.8.128.106:0/12650 1884 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:30.725238) v4 ==== 2004+0+0 (968796212 0 0) 0x55a327787c00 con 0x55a311b67000 > -853> 2018-07-11 07:35:30.688557 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:30.725238) v4 -- 0x55a31334c800 con 0 > -852> 2018-07-11 07:35:30.688589 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:30.725238) v4 -- 0x55a327789c00 con 0 > -851> 2018-07-11 07:35:30.714050 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a32658c000 con 0 > -850> 2018-07-11 07:35:30.728868 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5214 0x55a312272c80 replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:494/492,start:6:aa438efa:::benchmark_data_magna108_35617_object660123:0,end:6:aa4467f2:::benchmark_data_magna108_35617_object644361:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -849> 2018-07-11 07:35:30.728892 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5214 ==== replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:494/492,start:6:aa438efa:::benchmark_data_magna108_35617_object660123:0,end:6:aa4467f2:::benchmark_data_magna108_35617_object644361:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (326651899 0 0) 0x55a312272c80 con 0x55a317019000 > -848> 2018-07-11 07:35:30.940164 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15255 conn(0x55a311b64000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1896 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:35:32.155451) v4 > -847> 2018-07-11 07:35:30.940191 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.0 10.8.128.113:0/15255 1896 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:32.155451) v4 ==== 2004+0+0 (3423021055 0 0) 0x55a34de6f000 con 0x55a311b64000 > -846> 2018-07-11 07:35:30.940187 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15255 conn(0x55a311b65800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1896 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:35:32.155451) v4 > -845> 2018-07-11 07:35:30.940206 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.0 10.8.128.113:0/15255 1896 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:32.155451) v4 ==== 2004+0+0 (3423021055 0 0) 0x55a324474c00 con 0x55a311b65800 > -844> 2018-07-11 07:35:30.940205 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:32.155451) v4 -- 0x55a3203f7200 con 0 > -843> 2018-07-11 07:35:30.940239 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:32.155451) v4 -- 0x55a324b15e00 con 0 > -842> 2018-07-11 07:35:31.358003 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a32c830900 con 0 > -841> 2018-07-11 07:35:31.370092 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5215 0x55a312272f00 replica scrub(pg: 6.15,from:0'0,to:430'106534,epoch:494/492,start:6:aa4467f2:::benchmark_data_magna108_35617_object644361:0,end:6:aa458953:::benchmark_data_magna108_35617_object1038313:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -840> 2018-07-11 07:35:31.370122 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5215 ==== replica scrub(pg: 6.15,from:0'0,to:430'106534,epoch:494/492,start:6:aa4467f2:::benchmark_data_magna108_35617_object644361:0,end:6:aa458953:::benchmark_data_magna108_35617_object1038313:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 219+0+0 (129404777 0 0) 0x55a312272f00 con 0x55a317019000 > -839> 2018-07-11 07:35:31.664193 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b7000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1820 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:35:31.415162) v4 > -838> 2018-07-11 07:35:31.664194 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b8800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1820 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:35:31.415162) v4 > -837> 2018-07-11 07:35:31.664224 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.8 10.8.128.108:0/11967 1820 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:31.415162) v4 ==== 2004+0+0 (3019691264 0 0) 0x55a34de78600 con 0x55a3170b7000 > -836> 2018-07-11 07:35:31.664227 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.8 10.8.128.108:0/11967 1820 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:31.415162) v4 ==== 2004+0+0 (3019691264 0 0) 0x55a332e79a00 con 0x55a3170b8800 > -835> 2018-07-11 07:35:31.664243 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:31.415162) v4 -- 0x55a3203f7200 con 0 > -834> 2018-07-11 07:35:31.664289 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:31.415162) v4 -- 0x55a324b15e00 con 0 > -833> 2018-07-11 07:35:32.058158 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a3560cb440 con 0 > -832> 2018-07-11 07:35:32.077213 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5216 0x55a312273180 replica scrub(pg: 6.15,from:0'0,to:430'106459,epoch:494/492,start:6:aa458953:::benchmark_data_magna108_35617_object1038313:0,end:6:aa4680fb:::benchmark_data_magna108_35617_object832734:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -831> 2018-07-11 07:35:32.077244 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5216 ==== replica scrub(pg: 6.15,from:0'0,to:430'106459,epoch:494/492,start:6:aa458953:::benchmark_data_magna108_35617_object1038313:0,end:6:aa4680fb:::benchmark_data_magna108_35617_object832734:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 219+0+0 (4078828482 0 0) 0x55a312273180 con 0x55a317019000 > -830> 2018-07-11 07:35:32.167487 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15919 conn(0x55a3170b4000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1872 0x55a327788c00 osd_ping(ping e494 stamp 2018-07-11 07:35:32.791833) v4 > -829> 2018-07-11 07:35:32.167509 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.11 10.8.128.114:0/15919 1872 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:32.791833) v4 ==== 2004+0+0 (3140231033 0 0) 0x55a327788c00 con 0x55a3170b4000 > -828> 2018-07-11 07:35:32.167506 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15919 conn(0x55a317074800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1872 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:35:32.791833) v4 > -827> 2018-07-11 07:35:32.167522 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:32.791833) v4 -- 0x55a327789800 con 0 > -826> 2018-07-11 07:35:32.167523 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.11 10.8.128.114:0/15919 1872 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:32.791833) v4 ==== 2004+0+0 (3140231033 0 0) 0x55a324474c00 con 0x55a317074800 > -825> 2018-07-11 07:35:32.167551 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:32.791833) v4 -- 0x55a324b15e00 con 0 > -824> 2018-07-11 07:35:32.259002 7f35488a3700 1 -- 10.8.128.107:6800/12665 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55a32c6a22c0 con 0 > -823> 2018-07-11 07:35:32.259067 7f35488a3700 1 -- 10.8.128.107:6800/12665 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55a32aa0db80 con 0 > -822> 2018-07-11 07:35:32.746496 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a323d85d40 con 0 > -821> 2018-07-11 07:35:32.762725 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5217 0x55a312273400 replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:494/492,start:6:aa4680fb:::benchmark_data_magna108_35617_object832734:0,end:6:aa47f54b:::benchmark_data_magna108_35617_object903801:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -820> 2018-07-11 07:35:32.762748 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5217 ==== replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:494/492,start:6:aa4680fb:::benchmark_data_magna108_35617_object832734:0,end:6:aa47f54b:::benchmark_data_magna108_35617_object903801:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (4088075258 0 0) 0x55a312273400 con 0x55a317019000 > -819> 2018-07-11 07:35:32.776359 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12651 conn(0x55a32169f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1869 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:35:32.813073) v4 > -818> 2018-07-11 07:35:32.776359 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12651 conn(0x55a3216a0800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1869 0x55a32aa0fc00 osd_ping(ping e494 stamp 2018-07-11 07:35:32.813073) v4 > -817> 2018-07-11 07:35:32.776381 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.2 10.8.128.106:0/12651 1869 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:32.813073) v4 ==== 2004+0+0 (2321322079 0 0) 0x55a332e79a00 con 0x55a32169f000 > -816> 2018-07-11 07:35:32.776382 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.2 10.8.128.106:0/12651 1869 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:32.813073) v4 ==== 2004+0+0 (2321322079 0 0) 0x55a32aa0fc00 con 0x55a3216a0800 > -815> 2018-07-11 07:35:32.776393 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:32.813073) v4 -- 0x55a32fb18e00 con 0 > -814> 2018-07-11 07:35:32.776421 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:32.813073) v4 -- 0x55a33e564600 con 0 > -813> 2018-07-11 07:35:32.830922 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15263 conn(0x55a321664000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1870 0x55a327789800 osd_ping(ping e494 stamp 2018-07-11 07:35:34.046317) v4 > -812> 2018-07-11 07:35:32.830948 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.5 10.8.128.113:0/15263 1870 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:34.046317) v4 ==== 2004+0+0 (1054317193 0 0) 0x55a327789800 con 0x55a321664000 > -811> 2018-07-11 07:35:32.830972 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:34.046317) v4 -- 0x55a3131bc200 con 0 > -810> 2018-07-11 07:35:32.831064 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15263 conn(0x55a321667000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1870 0x55a32aa0fc00 osd_ping(ping e494 stamp 2018-07-11 07:35:34.046317) v4 > -809> 2018-07-11 07:35:32.831085 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.5 10.8.128.113:0/15263 1870 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:34.046317) v4 ==== 2004+0+0 (1054317193 0 0) 0x55a32aa0fc00 con 0x55a321667000 > -808> 2018-07-11 07:35:32.831106 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:34.046317) v4 -- 0x55a3131bb400 con 0 > -807> 2018-07-11 07:35:33.084731 7f3543899700 5 osd.10 494 heartbeat: osd_stat(785 GB used, 146 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,0,1]) > -806> 2018-07-11 07:35:33.084748 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6802/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:35:33.084746) v4 -- 0x55a317643400 con 0 > -805> 2018-07-11 07:35:33.084787 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6803/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:35:33.084746) v4 -- 0x55a317643600 con 0 > -804> 2018-07-11 07:35:33.084801 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6806/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:35:33.084746) v4 -- 0x55a317643800 con 0 > -803> 2018-07-11 07:35:33.084816 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6807/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:35:33.084746) v4 -- 0x55a317643a00 con 0 > -802> 2018-07-11 07:35:33.084846 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6808/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:35:33.084746) v4 -- 0x55a317643c00 con 0 > -801> 2018-07-11 07:35:33.084867 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6810/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:35:33.084746) v4 -- 0x55a317643e00 con 0 > -800> 2018-07-11 07:35:33.084897 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6809/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:35:33.084746) v4 -- 0x55a31e39a000 con 0 > -799> 2018-07-11 07:35:33.084923 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6811/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:35:33.084746) v4 -- 0x55a31e39a200 con 0 > -798> 2018-07-11 07:35:33.084951 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6810/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:35:33.084746) v4 -- 0x55a31e39a400 con 0 > -797> 2018-07-11 07:35:33.084975 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6811/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:35:33.084746) v4 -- 0x55a31e39a600 con 0 > -796> 2018-07-11 07:35:33.084990 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6806/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:35:33.084746) v4 -- 0x55a31e39a800 con 0 > -795> 2018-07-11 07:35:33.085003 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6807/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:35:33.084746) v4 -- 0x55a31e39aa00 con 0 > -794> 2018-07-11 07:35:33.085023 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6802/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:35:33.084746) v4 -- 0x55a31e39ac00 con 0 > -793> 2018-07-11 07:35:33.085046 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6803/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:35:33.084746) v4 -- 0x55a31e39ae00 con 0 > -792> 2018-07-11 07:35:33.085066 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6806/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:35:33.084746) v4 -- 0x55a31e39b000 con 0 > -791> 2018-07-11 07:35:33.085088 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6807/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:35:33.084746) v4 -- 0x55a31e39b200 con 0 > -790> 2018-07-11 07:35:33.085114 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6802/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:35:33.084746) v4 -- 0x55a31e39b400 con 0 > -789> 2018-07-11 07:35:33.085126 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6803/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:35:33.084746) v4 -- 0x55a31e39b600 con 0 > -788> 2018-07-11 07:35:33.085139 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6806/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:35:33.084746) v4 -- 0x55a31e39b800 con 0 > -787> 2018-07-11 07:35:33.085153 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6807/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:35:33.084746) v4 -- 0x55a31e39ba00 con 0 > -786> 2018-07-11 07:35:33.085145 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6806/12677 conn(0x55a314e84000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1876 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:33.084746) v4 > -785> 2018-07-11 07:35:33.085164 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6802/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:35:33.084746) v4 -- 0x55a31e39bc00 con 0 > -784> 2018-07-11 07:35:33.085164 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6806/12677 1876 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:33.084746) v4 ==== 2004+0+0 (3432170333 0 0) 0x55a34de6f000 con 0x55a314e84000 > -783> 2018-07-11 07:35:33.085177 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6803/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:35:33.084746) v4 -- 0x55a31e39be00 con 0 > -782> 2018-07-11 07:35:33.085265 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6807/12677 conn(0x55a314e85800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1876 0x55a327789800 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:33.084746) v4 > -781> 2018-07-11 07:35:33.085311 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6807/12677 1876 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:33.084746) v4 ==== 2004+0+0 (3432170333 0 0) 0x55a327789800 con 0x55a314e85800 > -780> 2018-07-11 07:35:33.085318 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6810/12651 conn(0x55a314e97800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1876 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:33.084746) v4 > -779> 2018-07-11 07:35:33.085339 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6810/12651 1876 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:33.084746) v4 ==== 2004+0+0 (3432170333 0 0) 0x55a332e79a00 con 0x55a314e97800 > -778> 2018-07-11 07:35:33.085382 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6806/12617 conn(0x55a314e26000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1876 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:33.084746) v4 > -777> 2018-07-11 07:35:33.085382 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6809/12650 conn(0x55a314e1f000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1876 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:33.084746) v4 > -776> 2018-07-11 07:35:33.085395 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6806/12617 1876 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:33.084746) v4 ==== 2004+0+0 (3432170333 0 0) 0x55a332e79a00 con 0x55a314e26000 > -775> 2018-07-11 07:35:33.085390 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6802/15255 conn(0x55a314e1c000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1876 0x55a327789800 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:33.084746) v4 > -774> 2018-07-11 07:35:33.085401 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6809/12650 1876 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:33.084746) v4 ==== 2004+0+0 (3432170333 0 0) 0x55a34de78600 con 0x55a314e1f000 > -773> 2018-07-11 07:35:33.085406 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6802/15255 1876 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:33.084746) v4 ==== 2004+0+0 (3432170333 0 0) 0x55a327789800 con 0x55a314e1c000 > -772> 2018-07-11 07:35:33.085425 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6803/15255 conn(0x55a314e1d800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1876 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:33.084746) v4 > -771> 2018-07-11 07:35:33.085437 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6803/15255 1876 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:33.084746) v4 ==== 2004+0+0 (3432170333 0 0) 0x55a332e79a00 con 0x55a314e1d800 > -770> 2018-07-11 07:35:33.085475 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6806/15263 conn(0x55a314e99000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1876 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:33.084746) v4 > -769> 2018-07-11 07:35:33.085486 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6811/15287 conn(0x55a314e04800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1876 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:33.084746) v4 > -768> 2018-07-11 07:35:33.085486 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6808/12651 conn(0x55a314e96000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1876 0x55a327789800 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:33.084746) v4 > -767> 2018-07-11 07:35:33.085491 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6806/15263 1876 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:33.084746) v4 ==== 2004+0+0 (3432170333 0 0) 0x55a34de78600 con 0x55a314e99000 > -766> 2018-07-11 07:35:33.085501 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6811/15287 1876 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:33.084746) v4 ==== 2004+0+0 (3432170333 0 0) 0x55a332e79a00 con 0x55a314e04800 > -765> 2018-07-11 07:35:33.085502 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6808/12651 1876 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:33.084746) v4 ==== 2004+0+0 (3432170333 0 0) 0x55a327789800 con 0x55a314e96000 > -764> 2018-07-11 07:35:33.085527 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6807/12617 conn(0x55a314e27800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1876 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:33.084746) v4 > -763> 2018-07-11 07:35:33.085540 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6806/15885 conn(0x55a314e00000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1876 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:33.084746) v4 > -762> 2018-07-11 07:35:33.085540 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6807/12617 1876 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:33.084746) v4 ==== 2004+0+0 (3432170333 0 0) 0x55a34de78600 con 0x55a314e27800 > -761> 2018-07-11 07:35:33.085555 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6806/15885 1876 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:33.084746) v4 ==== 2004+0+0 (3432170333 0 0) 0x55a332e79a00 con 0x55a314e00000 > -760> 2018-07-11 07:35:33.085561 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6811/12650 conn(0x55a314e20800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1876 0x55a327789800 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:33.084746) v4 > -759> 2018-07-11 07:35:33.085575 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6811/12650 1876 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:33.084746) v4 ==== 2004+0+0 (3432170333 0 0) 0x55a327789800 con 0x55a314e20800 > -758> 2018-07-11 07:35:33.085585 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6802/15919 conn(0x55a314e87000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1876 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:33.084746) v4 > -757> 2018-07-11 07:35:33.085597 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6802/15919 1876 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:33.084746) v4 ==== 2004+0+0 (3432170333 0 0) 0x55a332e79a00 con 0x55a314e87000 > -756> 2018-07-11 07:35:33.085603 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6803/13323 conn(0x55a314e16800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1876 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:33.084746) v4 > -755> 2018-07-11 07:35:33.085608 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6807/15263 conn(0x55a314e9a800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1876 0x55a327789800 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:33.084746) v4 > -754> 2018-07-11 07:35:33.085625 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6807/15263 1876 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:33.084746) v4 ==== 2004+0+0 (3432170333 0 0) 0x55a327789800 con 0x55a314e9a800 > -753> 2018-07-11 07:35:33.085625 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6802/13323 conn(0x55a314e15000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1876 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:33.084746) v4 > -752> 2018-07-11 07:35:33.085638 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6802/13323 1876 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:33.084746) v4 ==== 2004+0+0 (3432170333 0 0) 0x55a332e79a00 con 0x55a314e15000 > -751> 2018-07-11 07:35:33.085622 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6803/13323 1876 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:33.084746) v4 ==== 2004+0+0 (3432170333 0 0) 0x55a34de78600 con 0x55a314e16800 > -750> 2018-07-11 07:35:33.085656 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6810/15287 conn(0x55a314e03000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1876 0x55a327789800 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:33.084746) v4 > -749> 2018-07-11 07:35:33.085670 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6810/15287 1876 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:33.084746) v4 ==== 2004+0+0 (3432170333 0 0) 0x55a327789800 con 0x55a314e03000 > -748> 2018-07-11 07:35:33.085698 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6807/15885 conn(0x55a314e01800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1876 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:33.084746) v4 > -747> 2018-07-11 07:35:33.085704 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6803/11967 conn(0x55a314e13800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1876 0x55a327789800 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:33.084746) v4 > -746> 2018-07-11 07:35:33.085714 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6803/11967 1876 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:33.084746) v4 ==== 2004+0+0 (3432170333 0 0) 0x55a327789800 con 0x55a314e13800 > -745> 2018-07-11 07:35:33.085712 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6807/15885 1876 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:33.084746) v4 ==== 2004+0+0 (3432170333 0 0) 0x55a34de78600 con 0x55a314e01800 > -744> 2018-07-11 07:35:33.085756 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6803/15919 conn(0x55a314e88800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1876 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:33.084746) v4 > -743> 2018-07-11 07:35:33.085769 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6803/15919 1876 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:33.084746) v4 ==== 2004+0+0 (3432170333 0 0) 0x55a34de78600 con 0x55a314e88800 > -742> 2018-07-11 07:35:33.085806 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6802/11967 conn(0x55a314e12000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1876 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:33.084746) v4 > -741> 2018-07-11 07:35:33.085820 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6802/11967 1876 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:33.084746) v4 ==== 2004+0+0 (3432170333 0 0) 0x55a34de78600 con 0x55a314e12000 > -740> 2018-07-11 07:35:33.240665 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15255 conn(0x55a311b65800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1897 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:35:34.455862) v4 > -739> 2018-07-11 07:35:33.240691 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.0 10.8.128.113:0/15255 1897 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:34.455862) v4 ==== 2004+0+0 (779599155 0 0) 0x55a332e79a00 con 0x55a311b65800 > -738> 2018-07-11 07:35:33.240703 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:34.455862) v4 -- 0x55a317643e00 con 0 > -737> 2018-07-11 07:35:33.240795 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15255 conn(0x55a311b64000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1897 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:35:34.455862) v4 > -736> 2018-07-11 07:35:33.240813 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.0 10.8.128.113:0/15255 1897 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:34.455862) v4 ==== 2004+0+0 (779599155 0 0) 0x55a34de78600 con 0x55a311b64000 > -735> 2018-07-11 07:35:33.240832 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:34.455862) v4 -- 0x55a31e39a000 con 0 > -734> 2018-07-11 07:35:33.440773 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12677 conn(0x55a32169d800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1892 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:35:33.477556) v4 > -733> 2018-07-11 07:35:33.440795 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.1 10.8.128.106:0/12677 1892 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:33.477556) v4 ==== 2004+0+0 (328763663 0 0) 0x55a34de6f000 con 0x55a32169d800 > -732> 2018-07-11 07:35:33.440804 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:33.477556) v4 -- 0x55a31e39a000 con 0 > -731> 2018-07-11 07:35:33.440826 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12677 conn(0x55a32169c000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1892 0x55a327789800 osd_ping(ping e494 stamp 2018-07-11 07:35:33.477556) v4 > -730> 2018-07-11 07:35:33.440847 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.1 10.8.128.106:0/12677 1892 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:33.477556) v4 ==== 2004+0+0 (328763663 0 0) 0x55a327789800 con 0x55a32169c000 > -729> 2018-07-11 07:35:33.440869 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:33.477556) v4 -- 0x55a317643400 con 0 > -728> 2018-07-11 07:35:33.532296 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a335588fc0 con 0 > -727> 2018-07-11 07:35:33.556715 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5218 0x55a312273680 replica scrub(pg: 6.15,from:0'0,to:461'106923,epoch:494/492,start:6:aa47f54b:::benchmark_data_magna108_35617_object903801:0,end:6:aa48c225:::benchmark_data_magna108.ceph.redhat.com_2478_object288968:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -726> 2018-07-11 07:35:33.556745 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5218 ==== replica scrub(pg: 6.15,from:0'0,to:461'106923,epoch:494/492,start:6:aa47f54b:::benchmark_data_magna108_35617_object903801:0,end:6:aa48c225:::benchmark_data_magna108.ceph.redhat.com_2478_object288968:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (1458574609 0 0) 0x55a312273680 con 0x55a317019000 > -725> 2018-07-11 07:35:33.878524 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15885 conn(0x55a311b76800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1892 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:35:34.502927) v4 > -724> 2018-07-11 07:35:33.878530 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15885 conn(0x55a3170b5800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1892 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:35:34.502927) v4 > -723> 2018-07-11 07:35:33.878553 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.7 10.8.128.114:0/15885 1892 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:34.502927) v4 ==== 2004+0+0 (1817541634 0 0) 0x55a34de78600 con 0x55a311b76800 > -722> 2018-07-11 07:35:33.878557 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.7 10.8.128.114:0/15885 1892 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:34.502927) v4 ==== 2004+0+0 (1817541634 0 0) 0x55a324474c00 con 0x55a3170b5800 > -721> 2018-07-11 07:35:33.878566 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:34.502927) v4 -- 0x55a31e39a000 con 0 > -720> 2018-07-11 07:35:33.878591 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:34.502927) v4 -- 0x55a317643e00 con 0 > -719> 2018-07-11 07:35:34.189016 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12650 conn(0x55a311b67000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1885 0x55a32aa0fc00 osd_ping(ping e494 stamp 2018-07-11 07:35:34.225751) v4 > -718> 2018-07-11 07:35:34.189047 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.3 10.8.128.106:0/12650 1885 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:34.225751) v4 ==== 2004+0+0 (1665060112 0 0) 0x55a32aa0fc00 con 0x55a311b67000 > -717> 2018-07-11 07:35:34.189049 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12650 conn(0x55a311b68800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1885 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:35:34.225751) v4 > -716> 2018-07-11 07:35:34.189071 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:34.225751) v4 -- 0x55a317643400 con 0 > -715> 2018-07-11 07:35:34.189074 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.3 10.8.128.106:0/12650 1885 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:34.225751) v4 ==== 2004+0+0 (1665060112 0 0) 0x55a34de6f000 con 0x55a311b68800 > -714> 2018-07-11 07:35:34.189103 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:34.225751) v4 -- 0x55a31e39a000 con 0 > -713> 2018-07-11 07:35:34.295140 7f355d88e700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.107:0/12617 conn(0x55a32447f800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1845 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:35:34.294762) v4 > -712> 2018-07-11 07:35:34.295140 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.107:0/12617 conn(0x55a32447e000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1845 0x55a327789800 osd_ping(ping e494 stamp 2018-07-11 07:35:34.294762) v4 > -711> 2018-07-11 07:35:34.295173 7f355d88e700 1 -- 10.8.128.107:6803/12665 <== osd.9 10.8.128.107:0/12617 1845 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:34.294762) v4 ==== 2004+0+0 (672698703 0 0) 0x55a34de78600 con 0x55a32447f800 > -710> 2018-07-11 07:35:34.295177 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.9 10.8.128.107:0/12617 1845 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:34.294762) v4 ==== 2004+0+0 (672698703 0 0) 0x55a327789800 con 0x55a32447e000 > -709> 2018-07-11 07:35:34.295191 7f355d88e700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:34.294762) v4 -- 0x55a31e39a000 con 0 > -708> 2018-07-11 07:35:34.295219 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:34.294762) v4 -- 0x55a317643400 con 0 > -707> 2018-07-11 07:35:34.408438 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a314691440 con 0 > -706> 2018-07-11 07:35:34.460799 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5219 0x55a312273900 replica scrub(pg: 6.15,from:0'0,to:430'106621,epoch:494/492,start:6:aa48c225:::benchmark_data_magna108.ceph.redhat.com_2478_object288968:0,end:6:aa49d07a:::benchmark_data_magna108_35617_object690463:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -705> 2018-07-11 07:35:34.460824 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5219 ==== replica scrub(pg: 6.15,from:0'0,to:430'106621,epoch:494/492,start:6:aa48c225:::benchmark_data_magna108.ceph.redhat.com_2478_object288968:0,end:6:aa49d07a:::benchmark_data_magna108_35617_object690463:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (2345855871 0 0) 0x55a312273900 con 0x55a317019000 > -704> 2018-07-11 07:35:34.468078 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15919 conn(0x55a317074800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1873 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:35:35.092398) v4 > -703> 2018-07-11 07:35:34.468099 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.11 10.8.128.114:0/15919 1873 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:35.092398) v4 ==== 2004+0+0 (1444149850 0 0) 0x55a332e79a00 con 0x55a317074800 > -702> 2018-07-11 07:35:34.468108 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:35.092398) v4 -- 0x55a317643e00 con 0 > -701> 2018-07-11 07:35:34.468113 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15919 conn(0x55a3170b4000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1873 0x55a32aa0fc00 osd_ping(ping e494 stamp 2018-07-11 07:35:35.092398) v4 > -700> 2018-07-11 07:35:34.468130 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.11 10.8.128.114:0/15919 1873 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:35.092398) v4 ==== 2004+0+0 (1444149850 0 0) 0x55a32aa0fc00 con 0x55a3170b4000 > -699> 2018-07-11 07:35:34.468139 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:35.092398) v4 -- 0x55a3131bb400 con 0 > -698> 2018-07-11 07:35:34.785350 7f3543899700 5 osd.10 494 heartbeat: osd_stat(785 GB used, 146 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,0,1]) > -697> 2018-07-11 07:35:34.785365 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6802/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:35:34.785363) v4 -- 0x55a333036800 con 0 > -696> 2018-07-11 07:35:34.785404 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6803/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:35:34.785363) v4 -- 0x55a3177a9a00 con 0 > -695> 2018-07-11 07:35:34.785418 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6806/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:35:34.785363) v4 -- 0x55a3177a8e00 con 0 > -694> 2018-07-11 07:35:34.785432 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6807/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:35:34.785363) v4 -- 0x55a3177a9c00 con 0 > -693> 2018-07-11 07:35:34.785460 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6808/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:35:34.785363) v4 -- 0x55a34866b800 con 0 > -692> 2018-07-11 07:35:34.785481 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6810/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:35:34.785363) v4 -- 0x55a322455200 con 0 > -691> 2018-07-11 07:35:34.785512 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6809/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:35:34.785363) v4 -- 0x55a3177a9600 con 0 > -690> 2018-07-11 07:35:34.785530 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6811/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:35:34.785363) v4 -- 0x55a3282bb200 con 0 > -689> 2018-07-11 07:35:34.785558 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6810/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:35:34.785363) v4 -- 0x55a3177a9200 con 0 > -688> 2018-07-11 07:35:34.785582 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6811/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:35:34.785363) v4 -- 0x55a34866ba00 con 0 > -687> 2018-07-11 07:35:34.785598 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6806/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:35:34.785363) v4 -- 0x55a33e564a00 con 0 > -686> 2018-07-11 07:35:34.785611 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6807/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:35:34.785363) v4 -- 0x55a324b14c00 con 0 > -685> 2018-07-11 07:35:34.785631 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6802/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:35:34.785363) v4 -- 0x55a34866b400 con 0 > -684> 2018-07-11 07:35:34.785654 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6803/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:35:34.785363) v4 -- 0x55a3131ba200 con 0 > -683> 2018-07-11 07:35:34.785673 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6806/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:35:34.785363) v4 -- 0x55a33e564800 con 0 > -682> 2018-07-11 07:35:34.785694 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6807/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:35:34.785363) v4 -- 0x55a3131bbc00 con 0 > -681> 2018-07-11 07:35:34.785721 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6802/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:35:34.785363) v4 -- 0x55a33e564c00 con 0 > -680> 2018-07-11 07:35:34.785748 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6803/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:35:34.785363) v4 -- 0x55a324b14600 con 0 > -679> 2018-07-11 07:35:34.785772 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6806/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:35:34.785363) v4 -- 0x55a324b14800 con 0 > -678> 2018-07-11 07:35:34.785794 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6807/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:35:34.785363) v4 -- 0x55a324b14e00 con 0 > -677> 2018-07-11 07:35:34.785813 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6802/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:35:34.785363) v4 -- 0x55a323f54a00 con 0 > -676> 2018-07-11 07:35:34.785832 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6803/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:35:34.785363) v4 -- 0x55a3131bae00 con 0 > -675> 2018-07-11 07:35:34.785854 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6802/15255 conn(0x55a314e1c000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1877 0x55a317643400 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:34.785363) v4 > -674> 2018-07-11 07:35:34.785860 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6803/15255 conn(0x55a314e1d800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1877 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:34.785363) v4 > -673> 2018-07-11 07:35:34.785881 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6802/15255 1877 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:34.785363) v4 ==== 2004+0+0 (1934900029 0 0) 0x55a317643400 con 0x55a314e1c000 > -672> 2018-07-11 07:35:34.785888 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6803/15255 1877 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:34.785363) v4 ==== 2004+0+0 (1934900029 0 0) 0x55a324474c00 con 0x55a314e1d800 > -671> 2018-07-11 07:35:34.785897 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6809/12650 conn(0x55a314e1f000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1877 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:34.785363) v4 > -670> 2018-07-11 07:35:34.785914 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6809/12650 1877 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:34.785363) v4 ==== 2004+0+0 (1934900029 0 0) 0x55a34de78600 con 0x55a314e1f000 > -669> 2018-07-11 07:35:34.785959 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6807/12677 conn(0x55a314e85800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1877 0x55a317643400 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:34.785363) v4 > -668> 2018-07-11 07:35:34.785972 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6811/15287 conn(0x55a314e04800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1877 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:34.785363) v4 > -667> 2018-07-11 07:35:34.785976 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6807/12677 1877 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:34.785363) v4 ==== 2004+0+0 (1934900029 0 0) 0x55a317643400 con 0x55a314e85800 > -666> 2018-07-11 07:35:34.785991 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6811/15287 1877 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:34.785363) v4 ==== 2004+0+0 (1934900029 0 0) 0x55a332e79a00 con 0x55a314e04800 > -665> 2018-07-11 07:35:34.786001 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6806/12677 conn(0x55a314e84000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1877 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:34.785363) v4 > -664> 2018-07-11 07:35:34.786018 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6806/12677 1877 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:34.785363) v4 ==== 2004+0+0 (1934900029 0 0) 0x55a34de6f000 con 0x55a314e84000 > -663> 2018-07-11 07:35:34.786018 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6806/12617 conn(0x55a314e26000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1877 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:34.785363) v4 > -662> 2018-07-11 07:35:34.786019 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6808/12651 conn(0x55a314e96000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1877 0x55a317643400 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:34.785363) v4 > -661> 2018-07-11 07:35:34.786028 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6806/12617 1877 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:34.785363) v4 ==== 2004+0+0 (1934900029 0 0) 0x55a332e79a00 con 0x55a314e26000 > -660> 2018-07-11 07:35:34.786031 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6808/12651 1877 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:34.785363) v4 ==== 2004+0+0 (1934900029 0 0) 0x55a317643400 con 0x55a314e96000 > -659> 2018-07-11 07:35:34.786045 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6807/12617 conn(0x55a314e27800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1877 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:34.785363) v4 > -658> 2018-07-11 07:35:34.786064 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6810/12651 conn(0x55a314e97800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1877 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:34.785363) v4 > -657> 2018-07-11 07:35:34.786078 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6807/12617 1877 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:34.785363) v4 ==== 2004+0+0 (1934900029 0 0) 0x55a34de6f000 con 0x55a314e27800 > -656> 2018-07-11 07:35:34.786066 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6811/12650 conn(0x55a314e20800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1877 0x55a317643400 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:34.785363) v4 > -655> 2018-07-11 07:35:34.786086 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6810/12651 1877 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:34.785363) v4 ==== 2004+0+0 (1934900029 0 0) 0x55a332e79a00 con 0x55a314e97800 > -654> 2018-07-11 07:35:34.786091 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6811/12650 1877 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:34.785363) v4 ==== 2004+0+0 (1934900029 0 0) 0x55a317643400 con 0x55a314e20800 > -653> 2018-07-11 07:35:34.786119 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6803/13323 conn(0x55a314e16800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1877 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:34.785363) v4 > -652> 2018-07-11 07:35:34.786120 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6802/13323 conn(0x55a314e15000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1877 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:34.785363) v4 > -651> 2018-07-11 07:35:34.786122 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6810/15287 conn(0x55a314e03000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1877 0x55a317643400 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:34.785363) v4 > -650> 2018-07-11 07:35:34.786137 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6802/13323 1877 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:34.785363) v4 ==== 2004+0+0 (1934900029 0 0) 0x55a332e79a00 con 0x55a314e15000 > -649> 2018-07-11 07:35:34.786135 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6803/13323 1877 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:34.785363) v4 ==== 2004+0+0 (1934900029 0 0) 0x55a34de6f000 con 0x55a314e16800 > -648> 2018-07-11 07:35:34.786145 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6810/15287 1877 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:34.785363) v4 ==== 2004+0+0 (1934900029 0 0) 0x55a317643400 con 0x55a314e03000 > -647> 2018-07-11 07:35:34.786165 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6806/15885 conn(0x55a314e00000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1877 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:34.785363) v4 > -646> 2018-07-11 07:35:34.786172 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6806/15263 conn(0x55a314e99000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1877 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:34.785363) v4 > -645> 2018-07-11 07:35:34.786177 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6806/15885 1877 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:34.785363) v4 ==== 2004+0+0 (1934900029 0 0) 0x55a332e79a00 con 0x55a314e00000 > -644> 2018-07-11 07:35:34.786183 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6807/15263 conn(0x55a314e9a800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1877 0x55a317643400 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:34.785363) v4 > -643> 2018-07-11 07:35:34.786187 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6806/15263 1877 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:34.785363) v4 ==== 2004+0+0 (1934900029 0 0) 0x55a34de6f000 con 0x55a314e99000 > -642> 2018-07-11 07:35:34.786203 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6807/15263 1877 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:34.785363) v4 ==== 2004+0+0 (1934900029 0 0) 0x55a317643400 con 0x55a314e9a800 > -641> 2018-07-11 07:35:34.786213 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6802/15919 conn(0x55a314e87000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1877 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:34.785363) v4 > -640> 2018-07-11 07:35:34.786225 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6802/15919 1877 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:34.785363) v4 ==== 2004+0+0 (1934900029 0 0) 0x55a332e79a00 con 0x55a314e87000 > -639> 2018-07-11 07:35:34.786233 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6807/15885 conn(0x55a314e01800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1877 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:34.785363) v4 > -638> 2018-07-11 07:35:34.786238 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6803/11967 conn(0x55a314e13800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1877 0x55a317643400 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:34.785363) v4 > -637> 2018-07-11 07:35:34.786257 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6807/15885 1877 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:34.785363) v4 ==== 2004+0+0 (1934900029 0 0) 0x55a34de6f000 con 0x55a314e01800 > -636> 2018-07-11 07:35:34.786264 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6803/11967 1877 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:34.785363) v4 ==== 2004+0+0 (1934900029 0 0) 0x55a317643400 con 0x55a314e13800 > -635> 2018-07-11 07:35:34.786305 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6802/11967 conn(0x55a314e12000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1877 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:34.785363) v4 > -634> 2018-07-11 07:35:34.786319 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6802/11967 1877 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:34.785363) v4 ==== 2004+0+0 (1934900029 0 0) 0x55a34de6f000 con 0x55a314e12000 > -633> 2018-07-11 07:35:34.786354 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6803/15919 conn(0x55a314e88800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1877 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:34.785363) v4 > -632> 2018-07-11 07:35:34.786366 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6803/15919 1877 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:34.785363) v4 ==== 2004+0+0 (1934900029 0 0) 0x55a34de6f000 con 0x55a314e88800 > -631> 2018-07-11 07:35:35.061524 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.115:0/13323 conn(0x55a321668800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1878 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:35:35.399675) v4 > -630> 2018-07-11 07:35:35.061548 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.6 10.8.128.115:0/13323 1878 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:35.399675) v4 ==== 2004+0+0 (3349931423 0 0) 0x55a332e79a00 con 0x55a321668800 > -629> 2018-07-11 07:35:35.061564 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:35.399675) v4 -- 0x55a34866ba00 con 0 > -628> 2018-07-11 07:35:35.061596 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.115:0/13323 conn(0x55a321665800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1878 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:35:35.399675) v4 > -627> 2018-07-11 07:35:35.061610 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.6 10.8.128.115:0/13323 1878 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:35.399675) v4 ==== 2004+0+0 (3349931423 0 0) 0x55a332e79a00 con 0x55a321665800 > -626> 2018-07-11 07:35:35.061620 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:35.399675) v4 -- 0x55a3177a9a00 con 0 > -625> 2018-07-11 07:35:35.091540 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a315592240 con 0 > -624> 2018-07-11 07:35:35.104797 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5220 0x55a312273b80 replica scrub(pg: 6.15,from:0'0,to:430'106593,epoch:494/492,start:6:aa49d07a:::benchmark_data_magna108_35617_object690463:0,end:6:aa4af263:::benchmark_data_magna108_35617_object782329:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -623> 2018-07-11 07:35:35.104820 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5220 ==== replica scrub(pg: 6.15,from:0'0,to:430'106593,epoch:494/492,start:6:aa49d07a:::benchmark_data_magna108_35617_object690463:0,end:6:aa4af263:::benchmark_data_magna108_35617_object782329:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (2668746485 0 0) 0x55a312273b80 con 0x55a317019000 > -622> 2018-07-11 07:35:35.131463 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15263 conn(0x55a321664000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1871 0x55a317643400 osd_ping(ping e494 stamp 2018-07-11 07:35:36.346812) v4 > -621> 2018-07-11 07:35:35.131489 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.5 10.8.128.113:0/15263 1871 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:36.346812) v4 ==== 2004+0+0 (266784994 0 0) 0x55a317643400 con 0x55a321664000 > -620> 2018-07-11 07:35:35.131502 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:36.346812) v4 -- 0x55a34866b800 con 0 > -619> 2018-07-11 07:35:35.131579 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15263 conn(0x55a321667000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1871 0x55a3282bb200 osd_ping(ping e494 stamp 2018-07-11 07:35:36.346812) v4 > -618> 2018-07-11 07:35:35.131595 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.5 10.8.128.113:0/15263 1871 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:36.346812) v4 ==== 2004+0+0 (266784994 0 0) 0x55a3282bb200 con 0x55a321667000 > -617> 2018-07-11 07:35:35.131605 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:36.346812) v4 -- 0x55a34866b800 con 0 > -616> 2018-07-11 07:35:35.141176 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12677 conn(0x55a32169d800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1893 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:35:35.177965) v4 > -615> 2018-07-11 07:35:35.141174 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12677 conn(0x55a32169c000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1893 0x55a317643400 osd_ping(ping e494 stamp 2018-07-11 07:35:35.177965) v4 > -614> 2018-07-11 07:35:35.141200 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.1 10.8.128.106:0/12677 1893 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:35.177965) v4 ==== 2004+0+0 (3084508800 0 0) 0x55a34de6f000 con 0x55a32169d800 > -613> 2018-07-11 07:35:35.141203 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.1 10.8.128.106:0/12677 1893 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:35.177965) v4 ==== 2004+0+0 (3084508800 0 0) 0x55a317643400 con 0x55a32169c000 > -612> 2018-07-11 07:35:35.141212 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:35.177965) v4 -- 0x55a33e564a00 con 0 > -611> 2018-07-11 07:35:35.141234 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:35.177965) v4 -- 0x55a3177a9c00 con 0 > -610> 2018-07-11 07:35:35.562192 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.115:0/13323 conn(0x55a321665800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1879 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:35:35.900343) v4 > -609> 2018-07-11 07:35:35.562215 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.6 10.8.128.115:0/13323 1879 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:35.900343) v4 ==== 2004+0+0 (4258957353 0 0) 0x55a324474c00 con 0x55a321665800 > -608> 2018-07-11 07:35:35.562225 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:35.900343) v4 -- 0x55a34866ba00 con 0 > -607> 2018-07-11 07:35:35.562299 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.115:0/13323 conn(0x55a321668800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1879 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:35:35.900343) v4 > -606> 2018-07-11 07:35:35.562314 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.6 10.8.128.115:0/13323 1879 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:35.900343) v4 ==== 2004+0+0 (4258957353 0 0) 0x55a332e79a00 con 0x55a321668800 > -605> 2018-07-11 07:35:35.562324 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:35.900343) v4 -- 0x55a317643e00 con 0 > -604> 2018-07-11 07:35:35.740237 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15287 conn(0x55a311b75000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1866 0x55a3282bb200 osd_ping(ping e494 stamp 2018-07-11 07:35:36.955513) v4 > -603> 2018-07-11 07:35:35.740275 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.4 10.8.128.113:0/15287 1866 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:36.955513) v4 ==== 2004+0+0 (562214092 0 0) 0x55a3282bb200 con 0x55a311b75000 > -602> 2018-07-11 07:35:35.740291 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:36.955513) v4 -- 0x55a333036800 con 0 > -601> 2018-07-11 07:35:35.740284 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15287 conn(0x55a311b72000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1866 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:35:36.955513) v4 > -600> 2018-07-11 07:35:35.740311 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.4 10.8.128.113:0/15287 1866 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:36.955513) v4 ==== 2004+0+0 (562214092 0 0) 0x55a324474c00 con 0x55a311b72000 > -599> 2018-07-11 07:35:35.740324 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:36.955513) v4 -- 0x55a317643600 con 0 > -598> 2018-07-11 07:35:36.062672 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.115:0/13323 conn(0x55a321665800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1880 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:35:36.400802) v4 > -597> 2018-07-11 07:35:36.062698 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.6 10.8.128.115:0/13323 1880 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:36.400802) v4 ==== 2004+0+0 (41811077 0 0) 0x55a332e79a00 con 0x55a321665800 > -596> 2018-07-11 07:35:36.062711 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:36.400802) v4 -- 0x55a33e564600 con 0 > -595> 2018-07-11 07:35:36.062762 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.115:0/13323 conn(0x55a321668800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1880 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:35:36.400802) v4 > -594> 2018-07-11 07:35:36.062783 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.6 10.8.128.115:0/13323 1880 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:36.400802) v4 ==== 2004+0+0 (41811077 0 0) 0x55a332e79a00 con 0x55a321668800 > -593> 2018-07-11 07:35:36.062797 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:36.400802) v4 -- 0x55a3282b8000 con 0 > -592> 2018-07-11 07:35:36.141211 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15255 conn(0x55a311b65800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1898 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:35:37.356436) v4 > -591> 2018-07-11 07:35:36.141235 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.0 10.8.128.113:0/15255 1898 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:37.356436) v4 ==== 2004+0+0 (2591747039 0 0) 0x55a324474c00 con 0x55a311b65800 > -590> 2018-07-11 07:35:36.141256 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:37.356436) v4 -- 0x55a3282b8000 con 0 > -589> 2018-07-11 07:35:36.141231 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15255 conn(0x55a311b64000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1898 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:35:37.356436) v4 > -588> 2018-07-11 07:35:36.141277 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.0 10.8.128.113:0/15255 1898 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:37.356436) v4 ==== 2004+0+0 (2591747039 0 0) 0x55a34de78600 con 0x55a311b64000 > -587> 2018-07-11 07:35:36.141294 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:37.356436) v4 -- 0x55a33e564a00 con 0 > -586> 2018-07-11 07:35:36.240585 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15287 conn(0x55a311b72000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1867 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:35:37.455949) v4 > -585> 2018-07-11 07:35:36.240625 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.4 10.8.128.113:0/15287 1867 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:37.455949) v4 ==== 2004+0+0 (2855442755 0 0) 0x55a332e79a00 con 0x55a311b72000 > -584> 2018-07-11 07:35:36.240625 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15287 conn(0x55a311b75000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1867 0x55a317643400 osd_ping(ping e494 stamp 2018-07-11 07:35:37.455949) v4 > -583> 2018-07-11 07:35:36.240638 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:37.455949) v4 -- 0x55a33e564600 con 0 > -582> 2018-07-11 07:35:36.240649 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.4 10.8.128.113:0/15287 1867 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:37.455949) v4 ==== 2004+0+0 (2855442755 0 0) 0x55a317643400 con 0x55a311b75000 > -581> 2018-07-11 07:35:36.240671 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:37.455949) v4 -- 0x55a3131bb400 con 0 > -580> 2018-07-11 07:35:36.276793 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12651 conn(0x55a3216a0800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1870 0x55a3282bb200 osd_ping(ping e494 stamp 2018-07-11 07:35:36.313555) v4 > -579> 2018-07-11 07:35:36.276814 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.2 10.8.128.106:0/12651 1870 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:36.313555) v4 ==== 2004+0+0 (3908067485 0 0) 0x55a3282bb200 con 0x55a3216a0800 > -578> 2018-07-11 07:35:36.276803 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12651 conn(0x55a32169f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1870 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:35:36.313555) v4 > -577> 2018-07-11 07:35:36.276823 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:36.313555) v4 -- 0x55a324b15200 con 0 > -576> 2018-07-11 07:35:36.276824 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.2 10.8.128.106:0/12651 1870 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:36.313555) v4 ==== 2004+0+0 (3908067485 0 0) 0x55a324474c00 con 0x55a32169f000 > -575> 2018-07-11 07:35:36.276843 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:36.313555) v4 -- 0x55a344e84200 con 0 > -574> 2018-07-11 07:35:36.489527 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12650 conn(0x55a311b67000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1886 0x55a317643400 osd_ping(ping e494 stamp 2018-07-11 07:35:36.526286) v4 > -573> 2018-07-11 07:35:36.489565 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.3 10.8.128.106:0/12650 1886 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:36.526286) v4 ==== 2004+0+0 (3380311265 0 0) 0x55a317643400 con 0x55a311b67000 > -572> 2018-07-11 07:35:36.489585 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:36.526286) v4 -- 0x55a34866aa00 con 0 > -571> 2018-07-11 07:35:36.489599 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12650 conn(0x55a311b68800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1886 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:35:36.526286) v4 > -570> 2018-07-11 07:35:36.489618 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.3 10.8.128.106:0/12650 1886 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:36.526286) v4 ==== 2004+0+0 (3380311265 0 0) 0x55a34de6f000 con 0x55a311b68800 > -569> 2018-07-11 07:35:36.489639 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:36.526286) v4 -- 0x55a3177a9600 con 0 > -568> 2018-07-11 07:35:36.745695 7f35498a5700 10 monclient: tick > -567> 2018-07-11 07:35:36.745708 7f35498a5700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2018-07-11 07:35:06.745706) > -566> 2018-07-11 07:35:36.758892 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a3146e1440 con 0 > -565> 2018-07-11 07:35:36.772633 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5221 0x55a347e10000 replica scrub(pg: 6.15,from:0'0,to:430'105491,epoch:494/492,start:6:aa4af263:::benchmark_data_magna108_35617_object782329:0,end:6:aa4bfd6c:::benchmark_data_magna108.ceph.redhat.com_2478_object241410:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -564> 2018-07-11 07:35:36.772657 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5221 ==== replica scrub(pg: 6.15,from:0'0,to:430'105491,epoch:494/492,start:6:aa4af263:::benchmark_data_magna108_35617_object782329:0,end:6:aa4bfd6c:::benchmark_data_magna108.ceph.redhat.com_2478_object241410:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (3392590629 0 0) 0x55a347e10000 con 0x55a317019000 > -563> 2018-07-11 07:35:36.916162 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 61] Generated table #36600: 4680131 keys, 79657723 bytes > -562> 2018-07-11 07:35:36.916216 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294536916187, "cf_name": "default", "job": 61, "event": "table_file_creation", "file_number": 36600, "file_size": 79657723, "table_properties": {"data_size": 67112207, "index_size": 689382, "filter_size": 11855141, "raw_key_size": 135203775, "raw_average_key_size": 28, "raw_value_size": 3749621, "raw_average_value_size": 0, "num_data_blocks": 16473, "num_entries": 4680131, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4364742", "kMergeOperands": "0"}} > -561> 2018-07-11 07:35:36.964806 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b8800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1821 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:35:36.715849) v4 > -560> 2018-07-11 07:35:36.964831 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.8 10.8.128.108:0/11967 1821 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:36.715849) v4 ==== 2004+0+0 (2764322946 0 0) 0x55a332e79a00 con 0x55a3170b8800 > -559> 2018-07-11 07:35:36.964823 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b7000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1821 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:35:36.715849) v4 > -558> 2018-07-11 07:35:36.964843 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:36.715849) v4 -- 0x55a344e84200 con 0 > -557> 2018-07-11 07:35:36.964843 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.8 10.8.128.108:0/11967 1821 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:36.715849) v4 ==== 2004+0+0 (2764322946 0 0) 0x55a34de78600 con 0x55a3170b7000 > -556> 2018-07-11 07:35:36.964877 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:36.715849) v4 -- 0x55a3177a9600 con 0 > -555> 2018-07-11 07:35:37.195755 7f355d88e700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.107:0/12617 conn(0x55a32447f800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1846 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:35:37.195239) v4 > -554> 2018-07-11 07:35:37.195777 7f355d88e700 1 -- 10.8.128.107:6803/12665 <== osd.9 10.8.128.107:0/12617 1846 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:37.195239) v4 ==== 2004+0+0 (2763689053 0 0) 0x55a34de6f000 con 0x55a32447f800 > -553> 2018-07-11 07:35:37.195768 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.107:0/12617 conn(0x55a32447e000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1846 0x55a317643400 osd_ping(ping e494 stamp 2018-07-11 07:35:37.195239) v4 > -552> 2018-07-11 07:35:37.195792 7f355d88e700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:37.195239) v4 -- 0x55a3177a9600 con 0 > -551> 2018-07-11 07:35:37.195794 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.9 10.8.128.107:0/12617 1846 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:37.195239) v4 ==== 2004+0+0 (2763689053 0 0) 0x55a317643400 con 0x55a32447e000 > -550> 2018-07-11 07:35:37.195827 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:37.195239) v4 -- 0x55a33e565600 con 0 > -549> 2018-07-11 07:35:37.259279 7f35488a3700 1 -- 10.8.128.107:6800/12665 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55a32ebb8000 con 0 > -548> 2018-07-11 07:35:37.259344 7f35488a3700 1 -- 10.8.128.107:6800/12665 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55a31708a000 con 0 > -547> 2018-07-11 07:35:37.368519 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15919 conn(0x55a3170b4000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1874 0x55a34866aa00 osd_ping(ping e494 stamp 2018-07-11 07:35:37.992936) v4 > -546> 2018-07-11 07:35:37.368544 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.11 10.8.128.114:0/15919 1874 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:37.992936) v4 ==== 2004+0+0 (291331984 0 0) 0x55a34866aa00 con 0x55a3170b4000 > -545> 2018-07-11 07:35:37.368557 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:37.992936) v4 -- 0x55a33e565000 con 0 > -544> 2018-07-11 07:35:37.368563 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15919 conn(0x55a317074800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1874 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:35:37.992936) v4 > -543> 2018-07-11 07:35:37.368584 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.11 10.8.128.114:0/15919 1874 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:37.992936) v4 ==== 2004+0+0 (291331984 0 0) 0x55a324474c00 con 0x55a317074800 > -542> 2018-07-11 07:35:37.368595 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:37.992936) v4 -- 0x55a344e84c00 con 0 > -541> 2018-07-11 07:35:37.687085 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a32c7bbb00 con 0 > -540> 2018-07-11 07:35:37.697704 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5222 0x55a347e10280 replica scrub(pg: 6.15,from:0'0,to:430'106859,epoch:494/492,start:6:aa4bfd6c:::benchmark_data_magna108.ceph.redhat.com_2478_object241410:0,end:6:aa4ce7c7:::benchmark_data_magna108_35617_object672425:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -539> 2018-07-11 07:35:37.697729 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5222 ==== replica scrub(pg: 6.15,from:0'0,to:430'106859,epoch:494/492,start:6:aa4bfd6c:::benchmark_data_magna108.ceph.redhat.com_2478_object241410:0,end:6:aa4ce7c7:::benchmark_data_magna108_35617_object672425:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (2808143529 0 0) 0x55a347e10280 con 0x55a317019000 > -538> 2018-07-11 07:35:37.775896 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 61] Generated table #36601: 635258 keys, 10297011 bytes > -537> 2018-07-11 07:35:37.775939 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294537775911, "cf_name": "default", "job": 61, "event": "table_file_creation", "file_number": 36601, "file_size": 10297011, "table_properties": {"data_size": 8600037, "index_size": 87877, "filter_size": 1608112, "raw_key_size": 18351909, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 2111, "num_entries": 635258, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "635258", "kMergeOperands": "0"}} > -536> 2018-07-11 07:35:37.941101 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15287 conn(0x55a311b75000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1868 0x55a317643400 osd_ping(ping e494 stamp 2018-07-11 07:35:39.156390) v4 > -535> 2018-07-11 07:35:37.941126 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.4 10.8.128.113:0/15287 1868 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:39.156390) v4 ==== 2004+0+0 (329980650 0 0) 0x55a317643400 con 0x55a311b75000 > -534> 2018-07-11 07:35:37.941137 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:39.156390) v4 -- 0x55a33e565800 con 0 > -533> 2018-07-11 07:35:37.941129 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15287 conn(0x55a311b72000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1868 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:35:39.156390) v4 > -532> 2018-07-11 07:35:37.941149 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.4 10.8.128.113:0/15287 1868 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:39.156390) v4 ==== 2004+0+0 (329980650 0 0) 0x55a332e79a00 con 0x55a311b72000 > -531> 2018-07-11 07:35:37.941163 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:39.156390) v4 -- 0x55a344e84c00 con 0 > -530> 2018-07-11 07:35:38.031938 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15263 conn(0x55a321664000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1872 0x55a33e565000 osd_ping(ping e494 stamp 2018-07-11 07:35:39.247344) v4 > -529> 2018-07-11 07:35:38.031960 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.5 10.8.128.113:0/15263 1872 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:39.247344) v4 ==== 2004+0+0 (2146429722 0 0) 0x55a33e565000 con 0x55a321664000 > -528> 2018-07-11 07:35:38.031970 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:39.247344) v4 -- 0x55a33e565800 con 0 > -527> 2018-07-11 07:35:38.032034 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15263 conn(0x55a321667000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1872 0x55a317643400 osd_ping(ping e494 stamp 2018-07-11 07:35:39.247344) v4 > -526> 2018-07-11 07:35:38.032048 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.5 10.8.128.113:0/15263 1872 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:39.247344) v4 ==== 2004+0+0 (2146429722 0 0) 0x55a317643400 con 0x55a321667000 > -525> 2018-07-11 07:35:38.032058 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:39.247344) v4 -- 0x55a33e565800 con 0 > -524> 2018-07-11 07:35:38.402747 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a3266b6d80 con 0 > -523> 2018-07-11 07:35:38.427076 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5223 0x55a347e10500 replica scrub(pg: 6.15,from:0'0,to:461'106924,epoch:494/492,start:6:aa4ce7c7:::benchmark_data_magna108_35617_object672425:0,end:6:aa4dfe96:::benchmark_data_magna108_35617_object196460:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -522> 2018-07-11 07:35:38.427098 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5223 ==== replica scrub(pg: 6.15,from:0'0,to:461'106924,epoch:494/492,start:6:aa4ce7c7:::benchmark_data_magna108_35617_object672425:0,end:6:aa4dfe96:::benchmark_data_magna108_35617_object196460:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (2473639992 0 0) 0x55a347e10500 con 0x55a317019000 > -521> 2018-07-11 07:35:39.179028 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15885 conn(0x55a311b76800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1893 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:35:39.803352) v4 > -520> 2018-07-11 07:35:39.179054 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.7 10.8.128.114:0/15885 1893 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:39.803352) v4 ==== 2004+0+0 (1633258050 0 0) 0x55a34de78600 con 0x55a311b76800 > -519> 2018-07-11 07:35:39.179064 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:39.803352) v4 -- 0x55a3177a8e00 con 0 > -518> 2018-07-11 07:35:39.179136 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15885 conn(0x55a3170b5800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1893 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:35:39.803352) v4 > -517> 2018-07-11 07:35:39.179157 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.7 10.8.128.114:0/15885 1893 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:39.803352) v4 ==== 2004+0+0 (1633258050 0 0) 0x55a324474c00 con 0x55a3170b5800 > -516> 2018-07-11 07:35:39.179169 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:39.803352) v4 -- 0x55a3282b8c00 con 0 > -515> 2018-07-11 07:35:39.286153 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a32988b200 con 0 > -514> 2018-07-11 07:35:39.381259 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5224 0x55a347e10780 replica scrub(pg: 6.15,from:0'0,to:430'106373,epoch:494/492,start:6:aa4dfe96:::benchmark_data_magna108_35617_object196460:0,end:6:aa4f5d72:::benchmark_data_magna108_41525_object32880:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -513> 2018-07-11 07:35:39.381291 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5224 ==== replica scrub(pg: 6.15,from:0'0,to:430'106373,epoch:494/492,start:6:aa4dfe96:::benchmark_data_magna108_35617_object196460:0,end:6:aa4f5d72:::benchmark_data_magna108_41525_object32880:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 217+0+0 (946304535 0 0) 0x55a347e10780 con 0x55a317019000 > -512> 2018-07-11 07:35:39.865214 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b7000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1822 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:35:39.616307) v4 > -511> 2018-07-11 07:35:39.865241 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.8 10.8.128.108:0/11967 1822 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:39.616307) v4 ==== 2004+0+0 (1739573131 0 0) 0x55a34de6f000 con 0x55a3170b7000 > -510> 2018-07-11 07:35:39.865260 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:39.616307) v4 -- 0x55a3177a8e00 con 0 > -509> 2018-07-11 07:35:39.865272 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b8800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1822 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:35:39.616307) v4 > -508> 2018-07-11 07:35:39.865294 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.8 10.8.128.108:0/11967 1822 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:39.616307) v4 ==== 2004+0+0 (1739573131 0 0) 0x55a332e79a00 con 0x55a3170b8800 > -507> 2018-07-11 07:35:39.865306 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:39.616307) v4 -- 0x55a32ccf7000 con 0 > -506> 2018-07-11 07:35:39.986743 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a3266c8900 con 0 > -505> 2018-07-11 07:35:40.006471 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5225 0x55a347e10a00 replica scrub(pg: 6.15,from:0'0,to:430'106580,epoch:494/492,start:6:aa4f5d72:::benchmark_data_magna108_41525_object32880:0,end:6:aa5059c6:::benchmark_data_magna108.ceph.redhat.com_2478_object129199:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -504> 2018-07-11 07:35:40.006501 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5225 ==== replica scrub(pg: 6.15,from:0'0,to:430'106580,epoch:494/492,start:6:aa4f5d72:::benchmark_data_magna108_41525_object32880:0,end:6:aa5059c6:::benchmark_data_magna108.ceph.redhat.com_2478_object129199:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 232+0+0 (1623464945 0 0) 0x55a347e10a00 con 0x55a317019000 > -503> 2018-07-11 07:35:40.085996 7f3543899700 5 osd.10 494 heartbeat: osd_stat(785 GB used, 146 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,1]) > -502> 2018-07-11 07:35:40.086011 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6802/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:35:40.086008) v4 -- 0x55a3435eb400 con 0 > -501> 2018-07-11 07:35:40.086046 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6803/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:35:40.086008) v4 -- 0x55a33e565c00 con 0 > -500> 2018-07-11 07:35:40.086060 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6806/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:35:40.086008) v4 -- 0x55a33e565e00 con 0 > -499> 2018-07-11 07:35:40.086074 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6807/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:35:40.086008) v4 -- 0x55a3131baa00 con 0 > -498> 2018-07-11 07:35:40.086099 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6808/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:35:40.086008) v4 -- 0x55a3131bb800 con 0 > -497> 2018-07-11 07:35:40.086111 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6810/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:35:40.086008) v4 -- 0x55a3131bb200 con 0 > -496> 2018-07-11 07:35:40.086121 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6809/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:35:40.086008) v4 -- 0x55a3131bba00 con 0 > -495> 2018-07-11 07:35:40.086133 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6811/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:35:40.086008) v4 -- 0x55a3131bc600 con 0 > -494> 2018-07-11 07:35:40.086148 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6810/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:35:40.086008) v4 -- 0x55a3131bbe00 con 0 > -493> 2018-07-11 07:35:40.086172 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6811/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:35:40.086008) v4 -- 0x55a3131bac00 con 0 > -492> 2018-07-11 07:35:40.086186 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6806/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:35:40.086008) v4 -- 0x55a32ee76c00 con 0 > -491> 2018-07-11 07:35:40.086209 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6807/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:35:40.086008) v4 -- 0x55a33e564e00 con 0 > -490> 2018-07-11 07:35:40.086224 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6802/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:35:40.086008) v4 -- 0x55a34de62800 con 0 > -489> 2018-07-11 07:35:40.086236 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6803/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:35:40.086008) v4 -- 0x55a33e565400 con 0 > -488> 2018-07-11 07:35:40.086260 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6806/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:35:40.086008) v4 -- 0x55a324b15a00 con 0 > -487> 2018-07-11 07:35:40.086274 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6807/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:35:40.086008) v4 -- 0x55a3435eba00 con 0 > -486> 2018-07-11 07:35:40.086294 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6802/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:35:40.086008) v4 -- 0x55a3435ebe00 con 0 > -485> 2018-07-11 07:35:40.086308 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6803/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:35:40.086008) v4 -- 0x55a3177a9e00 con 0 > -484> 2018-07-11 07:35:40.086322 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6806/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:35:40.086008) v4 -- 0x55a34dec7e00 con 0 > -483> 2018-07-11 07:35:40.086338 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6807/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:35:40.086008) v4 -- 0x55a3435ea200 con 0 > -482> 2018-07-11 07:35:40.086351 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6802/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:35:40.086008) v4 -- 0x55a324b15c00 con 0 > -481> 2018-07-11 07:35:40.086362 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6803/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:35:40.086008) v4 -- 0x55a3435ea800 con 0 > -480> 2018-07-11 07:35:40.086498 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6802/15255 conn(0x55a314e1c000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1878 0x55a33e565000 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:40.086008) v4 > -479> 2018-07-11 07:35:40.086524 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6802/15255 1878 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:40.086008) v4 ==== 2004+0+0 (1993934136 0 0) 0x55a33e565000 con 0x55a314e1c000 > -478> 2018-07-11 07:35:40.086525 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6807/12617 conn(0x55a314e27800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1878 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:40.086008) v4 > -477> 2018-07-11 07:35:40.086562 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6807/12617 1878 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:40.086008) v4 ==== 2004+0+0 (1993934136 0 0) 0x55a34de6f000 con 0x55a314e27800 > -476> 2018-07-11 07:35:40.086581 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6807/12677 conn(0x55a314e85800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1878 0x55a33e565000 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:40.086008) v4 > -475> 2018-07-11 07:35:40.086568 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6806/12617 conn(0x55a314e26000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1878 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:40.086008) v4 > -474> 2018-07-11 07:35:40.086606 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6806/12617 1878 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:40.086008) v4 ==== 2004+0+0 (1993934136 0 0) 0x55a324474c00 con 0x55a314e26000 > -473> 2018-07-11 07:35:40.086602 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6807/12677 1878 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:40.086008) v4 ==== 2004+0+0 (1993934136 0 0) 0x55a33e565000 con 0x55a314e85800 > -472> 2018-07-11 07:35:40.086604 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6809/12650 conn(0x55a314e1f000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1878 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:40.086008) v4 > -471> 2018-07-11 07:35:40.086640 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6803/15255 conn(0x55a314e1d800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1878 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:40.086008) v4 > -470> 2018-07-11 07:35:40.086627 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6809/12650 1878 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:40.086008) v4 ==== 2004+0+0 (1993934136 0 0) 0x55a34de6f000 con 0x55a314e1f000 > -469> 2018-07-11 07:35:40.086651 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6803/15255 1878 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:40.086008) v4 ==== 2004+0+0 (1993934136 0 0) 0x55a324474c00 con 0x55a314e1d800 > -468> 2018-07-11 07:35:40.086668 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6810/15287 conn(0x55a314e03000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1878 0x55a33e565000 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:40.086008) v4 > -467> 2018-07-11 07:35:40.086678 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6806/12677 conn(0x55a314e84000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1878 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:40.086008) v4 > -466> 2018-07-11 07:35:40.086693 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6810/15287 1878 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:40.086008) v4 ==== 2004+0+0 (1993934136 0 0) 0x55a33e565000 con 0x55a314e03000 > -465> 2018-07-11 07:35:40.086697 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6806/12677 1878 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:40.086008) v4 ==== 2004+0+0 (1993934136 0 0) 0x55a34de6f000 con 0x55a314e84000 > -464> 2018-07-11 07:35:40.086696 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6810/12651 conn(0x55a314e97800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1878 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:40.086008) v4 > -463> 2018-07-11 07:35:40.086707 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6810/12651 1878 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:40.086008) v4 ==== 2004+0+0 (1993934136 0 0) 0x55a324474c00 con 0x55a314e97800 > -462> 2018-07-11 07:35:40.086731 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6808/12651 conn(0x55a314e96000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1878 0x55a33e565000 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:40.086008) v4 > -461> 2018-07-11 07:35:40.086743 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6808/12651 1878 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:40.086008) v4 ==== 2004+0+0 (1993934136 0 0) 0x55a33e565000 con 0x55a314e96000 > -460> 2018-07-11 07:35:40.086743 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6806/15263 conn(0x55a314e99000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1878 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:40.086008) v4 > -459> 2018-07-11 07:35:40.086751 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6811/15287 conn(0x55a314e04800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1878 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:40.086008) v4 > -458> 2018-07-11 07:35:40.086756 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6806/15263 1878 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:40.086008) v4 ==== 2004+0+0 (1993934136 0 0) 0x55a34de6f000 con 0x55a314e99000 > -457> 2018-07-11 07:35:40.086766 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6811/15287 1878 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:40.086008) v4 ==== 2004+0+0 (1993934136 0 0) 0x55a324474c00 con 0x55a314e04800 > -456> 2018-07-11 07:35:40.086777 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6811/12650 conn(0x55a314e20800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1878 0x55a33e565000 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:40.086008) v4 > -455> 2018-07-11 07:35:40.086789 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6811/12650 1878 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:40.086008) v4 ==== 2004+0+0 (1993934136 0 0) 0x55a33e565000 con 0x55a314e20800 > -454> 2018-07-11 07:35:40.086796 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6806/15885 conn(0x55a314e00000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1878 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:40.086008) v4 > -453> 2018-07-11 07:35:40.086800 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6803/13323 conn(0x55a314e16800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1878 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:40.086008) v4 > -452> 2018-07-11 07:35:40.086807 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6806/15885 1878 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:40.086008) v4 ==== 2004+0+0 (1993934136 0 0) 0x55a324474c00 con 0x55a314e00000 > -451> 2018-07-11 07:35:40.086810 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6803/13323 1878 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:40.086008) v4 ==== 2004+0+0 (1993934136 0 0) 0x55a34de6f000 con 0x55a314e16800 > -450> 2018-07-11 07:35:40.086819 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6807/15263 conn(0x55a314e9a800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1878 0x55a33e565000 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:40.086008) v4 > -449> 2018-07-11 07:35:40.086830 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6807/15263 1878 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:40.086008) v4 ==== 2004+0+0 (1993934136 0 0) 0x55a33e565000 con 0x55a314e9a800 > -448> 2018-07-11 07:35:40.086840 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6803/15919 conn(0x55a314e88800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1878 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:40.086008) v4 > -447> 2018-07-11 07:35:40.086852 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6803/15919 1878 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:40.086008) v4 ==== 2004+0+0 (1993934136 0 0) 0x55a34de6f000 con 0x55a314e88800 > -446> 2018-07-11 07:35:40.086856 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6802/13323 conn(0x55a314e15000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1878 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:40.086008) v4 > -445> 2018-07-11 07:35:40.086869 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6802/13323 1878 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:40.086008) v4 ==== 2004+0+0 (1993934136 0 0) 0x55a324474c00 con 0x55a314e15000 > -444> 2018-07-11 07:35:40.086881 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6802/11967 conn(0x55a314e12000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1878 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:40.086008) v4 > -443> 2018-07-11 07:35:40.086893 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6802/11967 1878 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:40.086008) v4 ==== 2004+0+0 (1993934136 0 0) 0x55a34de6f000 con 0x55a314e12000 > -442> 2018-07-11 07:35:40.086904 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6802/15919 conn(0x55a314e87000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1878 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:40.086008) v4 > -441> 2018-07-11 07:35:40.086897 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6803/11967 conn(0x55a314e13800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1878 0x55a33e565000 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:40.086008) v4 > -440> 2018-07-11 07:35:40.086918 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6802/15919 1878 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:40.086008) v4 ==== 2004+0+0 (1993934136 0 0) 0x55a324474c00 con 0x55a314e87000 > -439> 2018-07-11 07:35:40.086924 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6803/11967 1878 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:40.086008) v4 ==== 2004+0+0 (1993934136 0 0) 0x55a33e565000 con 0x55a314e13800 > -438> 2018-07-11 07:35:40.086926 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6807/15885 conn(0x55a314e01800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1878 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:40.086008) v4 > -437> 2018-07-11 07:35:40.086937 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6807/15885 1878 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:40.086008) v4 ==== 2004+0+0 (1993934136 0 0) 0x55a34de6f000 con 0x55a314e01800 > -436> 2018-07-11 07:35:40.377287 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12651 conn(0x55a3216a0800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1871 0x55a33e565000 osd_ping(ping e494 stamp 2018-07-11 07:35:40.414079) v4 > -435> 2018-07-11 07:35:40.377297 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12651 conn(0x55a32169f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1871 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:35:40.414079) v4 > -434> 2018-07-11 07:35:40.377312 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.2 10.8.128.106:0/12651 1871 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:40.414079) v4 ==== 2004+0+0 (3888879611 0 0) 0x55a33e565000 con 0x55a3216a0800 > -433> 2018-07-11 07:35:40.377315 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.2 10.8.128.106:0/12651 1871 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:40.414079) v4 ==== 2004+0+0 (3888879611 0 0) 0x55a324474c00 con 0x55a32169f000 > -432> 2018-07-11 07:35:40.377322 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:40.414079) v4 -- 0x55a3131baa00 con 0 > -431> 2018-07-11 07:35:40.377355 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:40.414079) v4 -- 0x55a33e565c00 con 0 > -430> 2018-07-11 07:35:40.589955 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12650 conn(0x55a311b67000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1887 0x55a34866be00 osd_ping(ping e494 stamp 2018-07-11 07:35:40.626820) v4 > -429> 2018-07-11 07:35:40.589977 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.3 10.8.128.106:0/12650 1887 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:40.626820) v4 ==== 2004+0+0 (534341538 0 0) 0x55a34866be00 con 0x55a311b67000 > -428> 2018-07-11 07:35:40.589986 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:40.626820) v4 -- 0x55a3435eb400 con 0 > -427> 2018-07-11 07:35:40.590053 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12650 conn(0x55a311b68800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1887 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:35:40.626820) v4 > -426> 2018-07-11 07:35:40.590071 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.3 10.8.128.106:0/12650 1887 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:40.626820) v4 ==== 2004+0+0 (534341538 0 0) 0x55a34de6f000 con 0x55a311b68800 > -425> 2018-07-11 07:35:40.590081 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:40.626820) v4 -- 0x55a3131bba00 con 0 > -424> 2018-07-11 07:35:40.615753 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a3266e3b00 con 0 > -423> 2018-07-11 07:35:40.635195 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5226 0x55a347e10c80 replica scrub(pg: 6.15,from:0'0,to:430'106416,epoch:494/492,start:6:aa5059c6:::benchmark_data_magna108.ceph.redhat.com_2478_object129199:0,end:6:aa517c48:::benchmark_data_magna108_35617_object1121339:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -422> 2018-07-11 07:35:40.635225 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5226 ==== replica scrub(pg: 6.15,from:0'0,to:430'106416,epoch:494/492,start:6:aa5059c6:::benchmark_data_magna108.ceph.redhat.com_2478_object129199:0,end:6:aa517c48:::benchmark_data_magna108_35617_object1121339:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 234+0+0 (2615275622 0 0) 0x55a347e10c80 con 0x55a317019000 > -421> 2018-07-11 07:35:40.841708 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15255 conn(0x55a311b65800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1899 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:35:42.056982) v4 > -420> 2018-07-11 07:35:40.841722 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15255 conn(0x55a311b64000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1899 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:35:42.056982) v4 > -419> 2018-07-11 07:35:40.841735 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.0 10.8.128.113:0/15255 1899 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:42.056982) v4 ==== 2004+0+0 (3489727047 0 0) 0x55a332e79a00 con 0x55a311b65800 > -418> 2018-07-11 07:35:40.841743 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.0 10.8.128.113:0/15255 1899 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:42.056982) v4 ==== 2004+0+0 (3489727047 0 0) 0x55a34de78600 con 0x55a311b64000 > -417> 2018-07-11 07:35:40.841746 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:42.056982) v4 -- 0x55a33e565c00 con 0 > -416> 2018-07-11 07:35:40.841780 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:42.056982) v4 -- 0x55a3131bba00 con 0 > -415> 2018-07-11 07:35:40.879583 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15885 conn(0x55a311b76800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1894 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:35:41.503992) v4 > -414> 2018-07-11 07:35:40.879606 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.7 10.8.128.114:0/15885 1894 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:41.503992) v4 ==== 2004+0+0 (85649116 0 0) 0x55a34de6f000 con 0x55a311b76800 > -413> 2018-07-11 07:35:40.879615 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:41.503992) v4 -- 0x55a3131bba00 con 0 > -412> 2018-07-11 07:35:40.879640 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15885 conn(0x55a3170b5800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1894 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:35:41.503992) v4 > -411> 2018-07-11 07:35:40.879661 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.7 10.8.128.114:0/15885 1894 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:41.503992) v4 ==== 2004+0+0 (85649116 0 0) 0x55a324474c00 con 0x55a3170b5800 > -410> 2018-07-11 07:35:40.879672 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:41.503992) v4 -- 0x55a33e565c00 con 0 > -409> 2018-07-11 07:35:41.041523 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12677 conn(0x55a32169d800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1894 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:35:41.078368) v4 > -408> 2018-07-11 07:35:41.041544 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.1 10.8.128.106:0/12677 1894 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:41.078368) v4 ==== 2004+0+0 (1837525822 0 0) 0x55a34de78600 con 0x55a32169d800 > -407> 2018-07-11 07:35:41.041555 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:41.078368) v4 -- 0x55a3131bba00 con 0 > -406> 2018-07-11 07:35:41.041571 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12677 conn(0x55a32169c000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1894 0x55a33e565000 osd_ping(ping e494 stamp 2018-07-11 07:35:41.078368) v4 > -405> 2018-07-11 07:35:41.041592 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.1 10.8.128.106:0/12677 1894 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:41.078368) v4 ==== 2004+0+0 (1837525822 0 0) 0x55a33e565000 con 0x55a32169c000 > -404> 2018-07-11 07:35:41.041604 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:41.078368) v4 -- 0x55a34866bc00 con 0 > -403> 2018-07-11 07:35:41.505305 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a345408b40 con 0 > -402> 2018-07-11 07:35:41.515089 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5227 0x55a347e10f00 replica scrub(pg: 6.15,from:0'0,to:430'106611,epoch:494/492,start:6:aa517c48:::benchmark_data_magna108_35617_object1121339:0,end:6:aa524323:::benchmark_data_magna108_35617_object607779:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -401> 2018-07-11 07:35:41.515120 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5227 ==== replica scrub(pg: 6.15,from:0'0,to:430'106611,epoch:494/492,start:6:aa517c48:::benchmark_data_magna108_35617_object1121339:0,end:6:aa524323:::benchmark_data_magna108_35617_object607779:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 219+0+0 (832296717 0 0) 0x55a347e10f00 con 0x55a317019000 > -400> 2018-07-11 07:35:41.532379 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15263 conn(0x55a321664000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1873 0x55a3435eb400 osd_ping(ping e494 stamp 2018-07-11 07:35:42.747841) v4 > -399> 2018-07-11 07:35:41.532401 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.5 10.8.128.113:0/15263 1873 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:42.747841) v4 ==== 2004+0+0 (1333819644 0 0) 0x55a3435eb400 con 0x55a321664000 > -398> 2018-07-11 07:35:41.532413 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:42.747841) v4 -- 0x55a324b14200 con 0 > -397> 2018-07-11 07:35:41.532459 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15263 conn(0x55a321667000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1873 0x55a3435eb400 osd_ping(ping e494 stamp 2018-07-11 07:35:42.747841) v4 > -396> 2018-07-11 07:35:41.532472 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.5 10.8.128.113:0/15263 1873 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:42.747841) v4 ==== 2004+0+0 (1333819644 0 0) 0x55a3435eb400 con 0x55a321667000 > -395> 2018-07-11 07:35:41.532481 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:42.747841) v4 -- 0x55a33e565a00 con 0 > -394> 2018-07-11 07:35:41.896373 7f355d88e700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.107:0/12617 conn(0x55a32447f800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1847 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:35:41.895882) v4 > -393> 2018-07-11 07:35:41.896400 7f355d88e700 1 -- 10.8.128.107:6803/12665 <== osd.9 10.8.128.107:0/12617 1847 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:41.895882) v4 ==== 2004+0+0 (37180133 0 0) 0x55a34de6f000 con 0x55a32447f800 > -392> 2018-07-11 07:35:41.896412 7f355d88e700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:41.895882) v4 -- 0x55a33e565e00 con 0 > -391> 2018-07-11 07:35:41.896417 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.107:0/12617 conn(0x55a32447e000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1847 0x55a34866bc00 osd_ping(ping e494 stamp 2018-07-11 07:35:41.895882) v4 > -390> 2018-07-11 07:35:41.896443 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.9 10.8.128.107:0/12617 1847 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:41.895882) v4 ==== 2004+0+0 (37180133 0 0) 0x55a34866bc00 con 0x55a32447e000 > -389> 2018-07-11 07:35:41.896455 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:41.895882) v4 -- 0x55a324b14200 con 0 > -388> 2018-07-11 07:35:41.963259 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.115:0/13323 conn(0x55a321668800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1881 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:35:42.301454) v4 > -387> 2018-07-11 07:35:41.963281 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.6 10.8.128.115:0/13323 1881 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:42.301454) v4 ==== 2004+0+0 (2419451482 0 0) 0x55a332e79a00 con 0x55a321668800 > -386> 2018-07-11 07:35:41.963292 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:42.301454) v4 -- 0x55a32ccf7000 con 0 > -385> 2018-07-11 07:35:41.963347 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.115:0/13323 conn(0x55a321665800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1881 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:35:42.301454) v4 > -384> 2018-07-11 07:35:41.963361 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.6 10.8.128.115:0/13323 1881 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:42.301454) v4 ==== 2004+0+0 (2419451482 0 0) 0x55a324474c00 con 0x55a321665800 > -383> 2018-07-11 07:35:41.963370 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:42.301454) v4 -- 0x55a3435ea000 con 0 > -382> 2018-07-11 07:35:42.077815 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12651 conn(0x55a3216a0800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1872 0x55a3435eb400 osd_ping(ping e494 stamp 2018-07-11 07:35:42.114616) v4 > -381> 2018-07-11 07:35:42.077823 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12651 conn(0x55a32169f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1872 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:35:42.114616) v4 > -380> 2018-07-11 07:35:42.077837 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.2 10.8.128.106:0/12651 1872 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:42.114616) v4 ==== 2004+0+0 (2458209684 0 0) 0x55a3435eb400 con 0x55a3216a0800 > -379> 2018-07-11 07:35:42.077844 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.2 10.8.128.106:0/12651 1872 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:42.114616) v4 ==== 2004+0+0 (2458209684 0 0) 0x55a332e79a00 con 0x55a32169f000 > -378> 2018-07-11 07:35:42.077846 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:42.114616) v4 -- 0x55a34866ac00 con 0 > -377> 2018-07-11 07:35:42.077889 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:42.114616) v4 -- 0x55a32f760200 con 0 > -376> 2018-07-11 07:35:42.236320 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a3266f2000 con 0 > -375> 2018-07-11 07:35:42.259552 7f35488a3700 1 -- 10.8.128.107:6800/12665 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55a317011080 con 0 > -374> 2018-07-11 07:35:42.259626 7f35488a3700 1 -- 10.8.128.107:6800/12665 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55a3216de580 con 0 > -373> 2018-07-11 07:35:42.260708 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5228 0x55a347e11180 replica scrub(pg: 6.15,from:0'0,to:430'106465,epoch:494/492,start:6:aa524323:::benchmark_data_magna108_35617_object607779:0,end:6:aa5366e2:::benchmark_data_magna108_41525_object15465:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -372> 2018-07-11 07:35:42.260734 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5228 ==== replica scrub(pg: 6.15,from:0'0,to:430'106465,epoch:494/492,start:6:aa524323:::benchmark_data_magna108_35617_object607779:0,end:6:aa5366e2:::benchmark_data_magna108_41525_object15465:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 217+0+0 (2911972108 0 0) 0x55a347e11180 con 0x55a317019000 > -371> 2018-07-11 07:35:42.641406 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15287 conn(0x55a311b72000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1869 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:35:43.856838) v4 > -370> 2018-07-11 07:35:42.641427 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.4 10.8.128.113:0/15287 1869 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:43.856838) v4 ==== 2004+0+0 (897232053 0 0) 0x55a324474c00 con 0x55a311b72000 > -369> 2018-07-11 07:35:42.641437 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:43.856838) v4 -- 0x55a32f760200 con 0 > -368> 2018-07-11 07:35:42.641449 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15287 conn(0x55a311b75000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1869 0x55a34866bc00 osd_ping(ping e494 stamp 2018-07-11 07:35:43.856838) v4 > -367> 2018-07-11 07:35:42.641470 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.4 10.8.128.113:0/15287 1869 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:43.856838) v4 ==== 2004+0+0 (897232053 0 0) 0x55a34866bc00 con 0x55a311b75000 > -366> 2018-07-11 07:35:42.641481 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:43.856838) v4 -- 0x55a324b14000 con 0 > -365> 2018-07-11 07:35:42.669073 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15919 conn(0x55a3170b4000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1875 0x55a34866ac00 osd_ping(ping e494 stamp 2018-07-11 07:35:43.293479) v4 > -364> 2018-07-11 07:35:42.669085 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15919 conn(0x55a317074800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.11 seq 1875 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:35:43.293479) v4 > -363> 2018-07-11 07:35:42.669094 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.11 10.8.128.114:0/15919 1875 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:43.293479) v4 ==== 2004+0+0 (2579961918 0 0) 0x55a34866ac00 con 0x55a3170b4000 > -362> 2018-07-11 07:35:42.669109 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.11 10.8.128.114:0/15919 1875 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:43.293479) v4 ==== 2004+0+0 (2579961918 0 0) 0x55a332e79a00 con 0x55a317074800 > -361> 2018-07-11 07:35:42.669116 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:43.293479) v4 -- 0x55a34866a400 con 0 > -360> 2018-07-11 07:35:42.669141 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:43.293479) v4 -- 0x55a34da1be00 con 0 > -359> 2018-07-11 07:35:42.765735 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b7000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1823 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:35:42.516790) v4 > -358> 2018-07-11 07:35:42.765756 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.8 10.8.128.108:0/11967 1823 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:42.516790) v4 ==== 2004+0+0 (1855063894 0 0) 0x55a34de78600 con 0x55a3170b7000 > -357> 2018-07-11 07:35:42.765754 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b8800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1823 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:35:42.516790) v4 > -356> 2018-07-11 07:35:42.765768 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:42.516790) v4 -- 0x55a3177a8e00 con 0 > -355> 2018-07-11 07:35:42.765776 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.8 10.8.128.108:0/11967 1823 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:42.516790) v4 ==== 2004+0+0 (1855063894 0 0) 0x55a324474c00 con 0x55a3170b8800 > -354> 2018-07-11 07:35:42.765788 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:42.516790) v4 -- 0x55a322455600 con 0 > -353> 2018-07-11 07:35:42.904977 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a3194d8b40 con 0 > -352> 2018-07-11 07:35:42.927305 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5229 0x55a347e11400 replica scrub(pg: 6.15,from:0'0,to:430'106075,epoch:494/492,start:6:aa5366e2:::benchmark_data_magna108_41525_object15465:0,end:6:aa54985a:::benchmark_data_magna108.ceph.redhat.com_2478_object336543:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -351> 2018-07-11 07:35:42.927328 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5229 ==== replica scrub(pg: 6.15,from:0'0,to:430'106075,epoch:494/492,start:6:aa5366e2:::benchmark_data_magna108_41525_object15465:0,end:6:aa54985a:::benchmark_data_magna108.ceph.redhat.com_2478_object336543:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 232+0+0 (2170342996 0 0) 0x55a347e11400 con 0x55a317019000 > -350> 2018-07-11 07:35:42.986505 7f3543899700 5 osd.10 494 heartbeat: osd_stat(785 GB used, 146 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,1]) > -349> 2018-07-11 07:35:42.986520 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6802/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:35:42.986518) v4 -- 0x55a3131bc400 con 0 > -348> 2018-07-11 07:35:42.986556 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6803/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:35:42.986518) v4 -- 0x55a33bff0200 con 0 > -347> 2018-07-11 07:35:42.986572 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6806/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:35:42.986518) v4 -- 0x55a324b15400 con 0 > -346> 2018-07-11 07:35:42.986586 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6807/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:35:42.986518) v4 -- 0x55a3435eae00 con 0 > -345> 2018-07-11 07:35:42.986599 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6808/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:35:42.986518) v4 -- 0x55a324b15000 con 0 > -344> 2018-07-11 07:35:42.986614 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6810/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:35:42.986518) v4 -- 0x55a3131bc000 con 0 > -343> 2018-07-11 07:35:42.986626 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6809/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:35:42.986518) v4 -- 0x55a3435ebc00 con 0 > -342> 2018-07-11 07:35:42.986640 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6811/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:35:42.986518) v4 -- 0x55a33bff0000 con 0 > -341> 2018-07-11 07:35:42.986666 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6810/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:35:42.986518) v4 -- 0x55a33e565200 con 0 > -340> 2018-07-11 07:35:42.986680 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6811/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:35:42.986518) v4 -- 0x55a3435eb200 con 0 > -339> 2018-07-11 07:35:42.986697 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6806/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:35:42.986518) v4 -- 0x55a324b15600 con 0 > -338> 2018-07-11 07:35:42.986714 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6807/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:35:42.986518) v4 -- 0x55a3435eb000 con 0 > -337> 2018-07-11 07:35:42.986727 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6802/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:35:42.986518) v4 -- 0x55a324b15800 con 0 > -336> 2018-07-11 07:35:42.986740 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6803/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:35:42.986518) v4 -- 0x55a343719600 con 0 > -335> 2018-07-11 07:35:42.986752 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6806/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:35:42.986518) v4 -- 0x55a333036000 con 0 > -334> 2018-07-11 07:35:42.986767 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6807/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:35:42.986518) v4 -- 0x55a3435eb800 con 0 > -333> 2018-07-11 07:35:42.986785 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6802/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:35:42.986518) v4 -- 0x55a333036e00 con 0 > -332> 2018-07-11 07:35:42.986797 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6803/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:35:42.986518) v4 -- 0x55a343719e00 con 0 > -331> 2018-07-11 07:35:42.986811 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6806/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:35:42.986518) v4 -- 0x55a33e564200 con 0 > -330> 2018-07-11 07:35:42.986824 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6807/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:35:42.986518) v4 -- 0x55a343719400 con 0 > -329> 2018-07-11 07:35:42.986837 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6802/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:35:42.986518) v4 -- 0x55a315fed800 con 0 > -328> 2018-07-11 07:35:42.986852 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6803/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:35:42.986518) v4 -- 0x55a33bff0e00 con 0 > -327> 2018-07-11 07:35:42.986934 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6803/15255 conn(0x55a314e1d800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1879 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:42.986518) v4 > -326> 2018-07-11 07:35:42.986958 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6803/15255 1879 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:42.986518) v4 ==== 2004+0+0 (3291951796 0 0) 0x55a324474c00 con 0x55a314e1d800 > -325> 2018-07-11 07:35:42.987011 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6807/12617 conn(0x55a314e27800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1879 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:42.986518) v4 > -324> 2018-07-11 07:35:42.987008 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6802/15255 conn(0x55a314e1c000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1879 0x55a34866a400 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:42.986518) v4 > -323> 2018-07-11 07:35:42.987036 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6807/12617 1879 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:42.986518) v4 ==== 2004+0+0 (3291951796 0 0) 0x55a34de6f000 con 0x55a314e27800 > -322> 2018-07-11 07:35:42.987041 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6802/15255 1879 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:42.986518) v4 ==== 2004+0+0 (3291951796 0 0) 0x55a34866a400 con 0x55a314e1c000 > -321> 2018-07-11 07:35:42.987066 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6806/12617 conn(0x55a314e26000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1879 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:42.986518) v4 > -320> 2018-07-11 07:35:42.987076 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6806/12677 conn(0x55a314e84000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1879 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:42.986518) v4 > -319> 2018-07-11 07:35:42.987082 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6806/12617 1879 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:42.986518) v4 ==== 2004+0+0 (3291951796 0 0) 0x55a324474c00 con 0x55a314e26000 > -318> 2018-07-11 07:35:42.987090 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6806/12677 1879 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:42.986518) v4 ==== 2004+0+0 (3291951796 0 0) 0x55a34de6f000 con 0x55a314e84000 > -317> 2018-07-11 07:35:42.987099 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6808/12651 conn(0x55a314e96000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1879 0x55a34866a400 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:42.986518) v4 > -316> 2018-07-11 07:35:42.987122 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6810/12651 conn(0x55a314e97800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1879 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:42.986518) v4 > -315> 2018-07-11 07:35:42.987143 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6808/12651 1879 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:42.986518) v4 ==== 2004+0+0 (3291951796 0 0) 0x55a34866a400 con 0x55a314e96000 > -314> 2018-07-11 07:35:42.987142 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6809/12650 conn(0x55a314e1f000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1879 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:42.986518) v4 > -313> 2018-07-11 07:35:42.987148 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6810/12651 1879 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:42.986518) v4 ==== 2004+0+0 (3291951796 0 0) 0x55a324474c00 con 0x55a314e97800 > -312> 2018-07-11 07:35:42.987154 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6809/12650 1879 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:42.986518) v4 ==== 2004+0+0 (3291951796 0 0) 0x55a34de6f000 con 0x55a314e1f000 > -311> 2018-07-11 07:35:42.987186 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6811/15287 conn(0x55a314e04800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1879 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:42.986518) v4 > -310> 2018-07-11 07:35:42.987188 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6810/15287 conn(0x55a314e03000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1879 0x55a34866a400 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:42.986518) v4 > -309> 2018-07-11 07:35:42.987205 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6811/15287 1879 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:42.986518) v4 ==== 2004+0+0 (3291951796 0 0) 0x55a324474c00 con 0x55a314e04800 > -308> 2018-07-11 07:35:42.987206 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6810/15287 1879 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:42.986518) v4 ==== 2004+0+0 (3291951796 0 0) 0x55a34866a400 con 0x55a314e03000 > -307> 2018-07-11 07:35:42.987244 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6807/15263 conn(0x55a314e9a800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1879 0x55a34866a400 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:42.986518) v4 > -306> 2018-07-11 07:35:42.987277 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6807/15263 1879 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:42.986518) v4 ==== 2004+0+0 (3291951796 0 0) 0x55a34866a400 con 0x55a314e9a800 > -305> 2018-07-11 07:35:42.987244 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6806/15263 conn(0x55a314e99000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1879 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:42.986518) v4 > -304> 2018-07-11 07:35:42.987300 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6806/15263 1879 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:42.986518) v4 ==== 2004+0+0 (3291951796 0 0) 0x55a34de6f000 con 0x55a314e99000 > -303> 2018-07-11 07:35:42.987312 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6802/13323 conn(0x55a314e15000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1879 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:42.986518) v4 > -302> 2018-07-11 07:35:42.987335 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6802/13323 1879 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:42.986518) v4 ==== 2004+0+0 (3291951796 0 0) 0x55a324474c00 con 0x55a314e15000 > -301> 2018-07-11 07:35:42.987332 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6811/12650 conn(0x55a314e20800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1879 0x55a34866a400 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:42.986518) v4 > -300> 2018-07-11 07:35:42.987352 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6811/12650 1879 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:42.986518) v4 ==== 2004+0+0 (3291951796 0 0) 0x55a34866a400 con 0x55a314e20800 > -299> 2018-07-11 07:35:42.987363 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6802/11967 conn(0x55a314e12000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1879 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:42.986518) v4 > -298> 2018-07-11 07:35:42.987384 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6802/11967 1879 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:42.986518) v4 ==== 2004+0+0 (3291951796 0 0) 0x55a34de6f000 con 0x55a314e12000 > -297> 2018-07-11 07:35:42.987388 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6802/15919 conn(0x55a314e87000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1879 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:42.986518) v4 > -296> 2018-07-11 07:35:42.987403 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6807/12677 conn(0x55a314e85800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1879 0x55a34866a400 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:42.986518) v4 > -295> 2018-07-11 07:35:42.987414 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6802/15919 1879 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:42.986518) v4 ==== 2004+0+0 (3291951796 0 0) 0x55a324474c00 con 0x55a314e87000 > -294> 2018-07-11 07:35:42.987421 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6807/12677 1879 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:42.986518) v4 ==== 2004+0+0 (3291951796 0 0) 0x55a34866a400 con 0x55a314e85800 > -293> 2018-07-11 07:35:42.987436 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6803/13323 conn(0x55a314e16800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1879 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:42.986518) v4 > -292> 2018-07-11 07:35:42.987447 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6806/15885 conn(0x55a314e00000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1879 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:42.986518) v4 > -291> 2018-07-11 07:35:42.987453 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6803/13323 1879 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:42.986518) v4 ==== 2004+0+0 (3291951796 0 0) 0x55a34de6f000 con 0x55a314e16800 > -290> 2018-07-11 07:35:42.987463 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6806/15885 1879 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:42.986518) v4 ==== 2004+0+0 (3291951796 0 0) 0x55a324474c00 con 0x55a314e00000 > -289> 2018-07-11 07:35:42.987467 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6803/11967 conn(0x55a314e13800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1879 0x55a34866a400 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:42.986518) v4 > -288> 2018-07-11 07:35:42.987485 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6803/11967 1879 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:42.986518) v4 ==== 2004+0+0 (3291951796 0 0) 0x55a34866a400 con 0x55a314e13800 > -287> 2018-07-11 07:35:42.987494 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6807/15885 conn(0x55a314e01800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1879 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:42.986518) v4 > -286> 2018-07-11 07:35:42.987509 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6807/15885 1879 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:42.986518) v4 ==== 2004+0+0 (3291951796 0 0) 0x55a34de6f000 con 0x55a314e01800 > -285> 2018-07-11 07:35:42.987550 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6803/15919 conn(0x55a314e88800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1879 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:42.986518) v4 > -284> 2018-07-11 07:35:42.987563 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6803/15919 1879 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:42.986518) v4 ==== 2004+0+0 (3291951796 0 0) 0x55a34de6f000 con 0x55a314e88800 > -283> 2018-07-11 07:35:43.663631 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.115:0/13323 conn(0x55a321665800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1882 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:35:44.001904) v4 > -282> 2018-07-11 07:35:43.663656 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.6 10.8.128.115:0/13323 1882 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:44.001904) v4 ==== 2004+0+0 (3807354239 0 0) 0x55a324474c00 con 0x55a321665800 > -281> 2018-07-11 07:35:43.663670 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:44.001904) v4 -- 0x55a324b15800 con 0 > -280> 2018-07-11 07:35:43.663818 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.115:0/13323 conn(0x55a321668800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1882 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:35:44.001904) v4 > -279> 2018-07-11 07:35:43.663835 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.6 10.8.128.115:0/13323 1882 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:44.001904) v4 ==== 2004+0+0 (3807354239 0 0) 0x55a332e79a00 con 0x55a321668800 > -278> 2018-07-11 07:35:43.663846 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:44.001904) v4 -- 0x55a3435eb200 con 0 > -277> 2018-07-11 07:35:43.741740 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15287 conn(0x55a311b75000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1870 0x55a34866a400 osd_ping(ping e494 stamp 2018-07-11 07:35:44.957250) v4 > -276> 2018-07-11 07:35:43.741748 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15287 conn(0x55a311b72000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1870 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:35:44.957250) v4 > -275> 2018-07-11 07:35:43.741763 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.4 10.8.128.113:0/15287 1870 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:44.957250) v4 ==== 2004+0+0 (2016827109 0 0) 0x55a34866a400 con 0x55a311b75000 > -274> 2018-07-11 07:35:43.741766 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.4 10.8.128.113:0/15287 1870 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:44.957250) v4 ==== 2004+0+0 (2016827109 0 0) 0x55a324474c00 con 0x55a311b72000 > -273> 2018-07-11 07:35:43.741773 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:44.957250) v4 -- 0x55a324b15000 con 0 > -272> 2018-07-11 07:35:43.741797 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:44.957250) v4 -- 0x55a3131bc000 con 0 > -271> 2018-07-11 07:35:43.742390 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15255 conn(0x55a311b65800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1900 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:35:44.957574) v4 > -270> 2018-07-11 07:35:43.742407 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.0 10.8.128.113:0/15255 1900 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:44.957574) v4 ==== 2004+0+0 (3861381251 0 0) 0x55a332e79a00 con 0x55a311b65800 > -269> 2018-07-11 07:35:43.742418 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:44.957574) v4 -- 0x55a33bff0200 con 0 > -268> 2018-07-11 07:35:43.742649 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15255 conn(0x55a311b64000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1900 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:35:44.957574) v4 > -267> 2018-07-11 07:35:43.742669 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.0 10.8.128.113:0/15255 1900 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:44.957574) v4 ==== 2004+0+0 (3861381251 0 0) 0x55a34de6f000 con 0x55a311b64000 > -266> 2018-07-11 07:35:43.742680 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:44.957574) v4 -- 0x55a343719600 con 0 > -265> 2018-07-11 07:35:43.866304 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b8800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1824 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:35:43.617422) v4 > -264> 2018-07-11 07:35:43.866325 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.8 10.8.128.108:0/11967 1824 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:43.617422) v4 ==== 2004+0+0 (1234839585 0 0) 0x55a324474c00 con 0x55a3170b8800 > -263> 2018-07-11 07:35:43.866310 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b7000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1824 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:35:43.617422) v4 > -262> 2018-07-11 07:35:43.866334 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:43.617422) v4 -- 0x55a322455600 con 0 > -261> 2018-07-11 07:35:43.866334 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.8 10.8.128.108:0/11967 1824 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:43.617422) v4 ==== 2004+0+0 (1234839585 0 0) 0x55a34de78600 con 0x55a3170b7000 > -260> 2018-07-11 07:35:43.866360 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:43.617422) v4 -- 0x55a324b15600 con 0 > -259> 2018-07-11 07:35:44.378322 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12651 conn(0x55a32169f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1873 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:35:44.415181) v4 > -258> 2018-07-11 07:35:44.378329 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12651 conn(0x55a3216a0800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1873 0x55a34866bc00 osd_ping(ping e494 stamp 2018-07-11 07:35:44.415181) v4 > -257> 2018-07-11 07:35:44.378344 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.2 10.8.128.106:0/12651 1873 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:44.415181) v4 ==== 2004+0+0 (4230900541 0 0) 0x55a332e79a00 con 0x55a32169f000 > -256> 2018-07-11 07:35:44.378352 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.2 10.8.128.106:0/12651 1873 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:44.415181) v4 ==== 2004+0+0 (4230900541 0 0) 0x55a34866bc00 con 0x55a3216a0800 > -255> 2018-07-11 07:35:44.378354 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:44.415181) v4 -- 0x55a3282ba800 con 0 > -254> 2018-07-11 07:35:44.378389 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:44.415181) v4 -- 0x55a3435eae00 con 0 > -253> 2018-07-11 07:35:44.627903 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a3266f3b00 con 0 > -252> 2018-07-11 07:35:44.640457 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5230 0x55a347e11680 replica scrub(pg: 6.15,from:0'0,to:430'106001,epoch:494/492,start:6:aa54985a:::benchmark_data_magna108.ceph.redhat.com_2478_object336543:0,end:6:aa55cb87:::benchmark_data_magna108_35617_object772459:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -251> 2018-07-11 07:35:44.640480 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5230 ==== replica scrub(pg: 6.15,from:0'0,to:430'106001,epoch:494/492,start:6:aa54985a:::benchmark_data_magna108.ceph.redhat.com_2478_object336543:0,end:6:aa55cb87:::benchmark_data_magna108_35617_object772459:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (2989765110 0 0) 0x55a347e11680 con 0x55a317019000 > -250> 2018-07-11 07:35:44.787691 7f354e0ae700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 61] Generated table #36602: 4957168 keys, 80348314 bytes > -249> 2018-07-11 07:35:44.787726 7f354e0ae700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294544787711, "cf_name": "default", "job": 61, "event": "table_file_creation", "file_number": 36602, "file_size": 80348314, "table_properties": {"data_size": 67109372, "index_size": 689242, "filter_size": 12548710, "raw_key_size": 143207073, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957168, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957168", "kMergeOperands": "0"}} > -248> 2018-07-11 07:35:44.842779 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15255 conn(0x55a311b64000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1901 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:35:46.058304) v4 > -247> 2018-07-11 07:35:44.842795 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15255 conn(0x55a311b65800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 1901 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:35:46.058304) v4 > -246> 2018-07-11 07:35:44.842811 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.0 10.8.128.113:0/15255 1901 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:46.058304) v4 ==== 2004+0+0 (1711173656 0 0) 0x55a324474c00 con 0x55a311b65800 > -245> 2018-07-11 07:35:44.842809 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.0 10.8.128.113:0/15255 1901 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:46.058304) v4 ==== 2004+0+0 (1711173656 0 0) 0x55a34de6f000 con 0x55a311b64000 > -244> 2018-07-11 07:35:44.842830 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:46.058304) v4 -- 0x55a3282ba800 con 0 > -243> 2018-07-11 07:35:44.842868 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:46.058304) v4 -- 0x55a324b15600 con 0 > -242> 2018-07-11 07:35:44.966779 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b7000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1825 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:35:44.717914) v4 > -241> 2018-07-11 07:35:44.966800 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.8 10.8.128.108:0/11967 1825 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:44.717914) v4 ==== 2004+0+0 (3227231909 0 0) 0x55a34de78600 con 0x55a3170b7000 > -240> 2018-07-11 07:35:44.966809 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:44.717914) v4 -- 0x55a324b15600 con 0 > -239> 2018-07-11 07:35:44.966809 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b8800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1825 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:35:44.717914) v4 > -238> 2018-07-11 07:35:44.966841 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.8 10.8.128.108:0/11967 1825 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:44.717914) v4 ==== 2004+0+0 (3227231909 0 0) 0x55a332e79a00 con 0x55a3170b8800 > -237> 2018-07-11 07:35:44.966853 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:44.717914) v4 -- 0x55a3282ba800 con 0 > -236> 2018-07-11 07:35:45.290449 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12650 conn(0x55a311b67000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1888 0x55a34866a400 osd_ping(ping e494 stamp 2018-07-11 07:35:45.327357) v4 > -235> 2018-07-11 07:35:45.290471 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.3 10.8.128.106:0/12650 1888 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:45.327357) v4 ==== 2004+0+0 (3290518629 0 0) 0x55a34866a400 con 0x55a311b67000 > -234> 2018-07-11 07:35:45.290482 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:45.327357) v4 -- 0x55a3435eae00 con 0 > -233> 2018-07-11 07:35:45.290524 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12650 conn(0x55a311b68800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1888 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:35:45.327357) v4 > -232> 2018-07-11 07:35:45.290546 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.3 10.8.128.106:0/12650 1888 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:45.327357) v4 ==== 2004+0+0 (3290518629 0 0) 0x55a34de6f000 con 0x55a311b68800 > -231> 2018-07-11 07:35:45.290562 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:45.327357) v4 -- 0x55a3435ebc00 con 0 > -230> 2018-07-11 07:35:45.364104 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.115:0/13323 conn(0x55a321665800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1883 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:35:45.702322) v4 > -229> 2018-07-11 07:35:45.364124 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.6 10.8.128.115:0/13323 1883 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:45.702322) v4 ==== 2004+0+0 (1423106908 0 0) 0x55a324474c00 con 0x55a321665800 > -228> 2018-07-11 07:35:45.364142 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:45.702322) v4 -- 0x55a31706f200 con 0 > -227> 2018-07-11 07:35:45.364189 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.115:0/13323 conn(0x55a321668800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1883 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:35:45.702322) v4 > -226> 2018-07-11 07:35:45.364208 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.6 10.8.128.115:0/13323 1883 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:45.702322) v4 ==== 2004+0+0 (1423106908 0 0) 0x55a324474c00 con 0x55a321668800 > -225> 2018-07-11 07:35:45.364219 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:45.702322) v4 -- 0x55a3282ba400 con 0 > -224> 2018-07-11 07:35:45.396993 7f355d88e700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.107:0/12617 conn(0x55a32447f800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1848 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:35:45.396531) v4 > -223> 2018-07-11 07:35:45.396976 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.107:0/12617 conn(0x55a32447e000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.9 seq 1848 0x55a34866bc00 osd_ping(ping e494 stamp 2018-07-11 07:35:45.396531) v4 > -222> 2018-07-11 07:35:45.397015 7f355d88e700 1 -- 10.8.128.107:6803/12665 <== osd.9 10.8.128.107:0/12617 1848 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:45.396531) v4 ==== 2004+0+0 (1301472730 0 0) 0x55a34de78600 con 0x55a32447f800 > -221> 2018-07-11 07:35:45.397030 7f355d88e700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:45.396531) v4 -- 0x55a324b15400 con 0 > -220> 2018-07-11 07:35:45.397029 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.9 10.8.128.107:0/12617 1848 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:45.396531) v4 ==== 2004+0+0 (1301472730 0 0) 0x55a34866bc00 con 0x55a32447e000 > -219> 2018-07-11 07:35:45.397062 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:45.396531) v4 -- 0x55a3131bc400 con 0 > -218> 2018-07-11 07:35:45.456924 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a3382c46c0 con 0 > -217> 2018-07-11 07:35:45.473872 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5231 0x55a347e11900 replica scrub(pg: 6.15,from:0'0,to:430'106678,epoch:494/492,start:6:aa55cb87:::benchmark_data_magna108_35617_object772459:0,end:6:aa56ceae:::benchmark_data_magna108_35617_object300106:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -216> 2018-07-11 07:35:45.473895 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5231 ==== replica scrub(pg: 6.15,from:0'0,to:430'106678,epoch:494/492,start:6:aa55cb87:::benchmark_data_magna108_35617_object772459:0,end:6:aa56ceae:::benchmark_data_magna108_35617_object300106:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (2744652983 0 0) 0x55a347e11900 con 0x55a317019000 > -215> 2018-07-11 07:35:45.478763 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12651 conn(0x55a3216a0800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1874 0x55a34866a400 osd_ping(ping e494 stamp 2018-07-11 07:35:45.515662) v4 > -214> 2018-07-11 07:35:45.478768 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12651 conn(0x55a32169f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=20 cs=1 l=1). rx osd.2 seq 1874 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:35:45.515662) v4 > -213> 2018-07-11 07:35:45.478791 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.2 10.8.128.106:0/12651 1874 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:45.515662) v4 ==== 2004+0+0 (1943383317 0 0) 0x55a34866a400 con 0x55a3216a0800 > -212> 2018-07-11 07:35:45.478795 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.2 10.8.128.106:0/12651 1874 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:45.515662) v4 ==== 2004+0+0 (1943383317 0 0) 0x55a332e79a00 con 0x55a32169f000 > -211> 2018-07-11 07:35:45.478803 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:45.515662) v4 -- 0x55a34866b200 con 0 > -210> 2018-07-11 07:35:45.478827 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:45.515662) v4 -- 0x55a3282ba400 con 0 > -209> 2018-07-11 07:35:45.580191 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.114:0/15885 conn(0x55a311b76800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1895 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:35:46.204564) v4 > -208> 2018-07-11 07:35:45.580212 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.7 10.8.128.114:0/15885 1895 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:46.204564) v4 ==== 2004+0+0 (135258365 0 0) 0x55a34de6f000 con 0x55a311b76800 > -207> 2018-07-11 07:35:45.580224 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:46.204564) v4 -- 0x55a324b15400 con 0 > -206> 2018-07-11 07:35:45.580242 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.114:0/15885 conn(0x55a3170b5800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.7 seq 1895 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:35:46.204564) v4 > -205> 2018-07-11 07:35:45.580273 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.7 10.8.128.114:0/15885 1895 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:46.204564) v4 ==== 2004+0+0 (135258365 0 0) 0x55a324474c00 con 0x55a3170b5800 > -204> 2018-07-11 07:35:45.580290 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:46.204564) v4 -- 0x55a31706f200 con 0 > -203> 2018-07-11 07:35:45.632887 7f355d08d700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15263 conn(0x55a321664000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1874 0x55a3131bc400 osd_ping(ping e494 stamp 2018-07-11 07:35:46.848291) v4 > -202> 2018-07-11 07:35:45.632910 7f355d08d700 1 -- 10.8.128.107:6802/12665 <== osd.5 10.8.128.113:0/15263 1874 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:46.848291) v4 ==== 2004+0+0 (2212367253 0 0) 0x55a3131bc400 con 0x55a321664000 > -201> 2018-07-11 07:35:45.632927 7f355d08d700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:46.848291) v4 -- 0x55a324b14a00 con 0 > -200> 2018-07-11 07:35:45.633011 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15263 conn(0x55a321667000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.5 seq 1874 0x55a34866a400 osd_ping(ping e494 stamp 2018-07-11 07:35:46.848291) v4 > -199> 2018-07-11 07:35:45.633027 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.5 10.8.128.113:0/15263 1874 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:46.848291) v4 ==== 2004+0+0 (2212367253 0 0) 0x55a34866a400 con 0x55a321667000 > -198> 2018-07-11 07:35:45.633037 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:46.848291) v4 -- 0x55a324b14400 con 0 > -197> 2018-07-11 07:35:46.042177 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.113:0/15287 conn(0x55a311b75000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1871 0x55a3131bc400 osd_ping(ping e494 stamp 2018-07-11 07:35:47.257657) v4 > -196> 2018-07-11 07:35:46.042199 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.4 10.8.128.113:0/15287 1871 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:47.257657) v4 ==== 2004+0+0 (341641160 0 0) 0x55a3131bc400 con 0x55a311b75000 > -195> 2018-07-11 07:35:46.042211 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:47.257657) v4 -- 0x55a315fede00 con 0 > -194> 2018-07-11 07:35:46.042210 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.113:0/15287 conn(0x55a311b72000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1871 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:35:47.257657) v4 > -193> 2018-07-11 07:35:46.042236 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.4 10.8.128.113:0/15287 1871 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:47.257657) v4 ==== 2004+0+0 (341641160 0 0) 0x55a332e79a00 con 0x55a311b72000 > -192> 2018-07-11 07:35:46.042258 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:47.257657) v4 -- 0x55a32ccf7400 con 0 > -191> 2018-07-11 07:35:46.179109 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a3382b4b40 con 0 > -190> 2018-07-11 07:35:46.190947 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5232 0x55a347e11b80 replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:494/492,start:6:aa56ceae:::benchmark_data_magna108_35617_object300106:0,end:6:aa57e543:::benchmark_data_magna108_35617_object1054139:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -189> 2018-07-11 07:35:46.190970 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5232 ==== replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:494/492,start:6:aa56ceae:::benchmark_data_magna108_35617_object300106:0,end:6:aa57e543:::benchmark_data_magna108_35617_object1054139:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 219+0+0 (2288123806 0 0) 0x55a347e11b80 con 0x55a317019000 > -188> 2018-07-11 07:35:46.667263 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b8800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1826 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:35:46.418365) v4 > -187> 2018-07-11 07:35:46.667268 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.108:0/11967 conn(0x55a3170b7000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.8 seq 1826 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:35:46.418365) v4 > -186> 2018-07-11 07:35:46.667287 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.8 10.8.128.108:0/11967 1826 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:46.418365) v4 ==== 2004+0+0 (338179983 0 0) 0x55a324474c00 con 0x55a3170b8800 > -185> 2018-07-11 07:35:46.667288 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.8 10.8.128.108:0/11967 1826 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:46.418365) v4 ==== 2004+0+0 (338179983 0 0) 0x55a34de78600 con 0x55a3170b7000 > -184> 2018-07-11 07:35:46.667298 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:46.418365) v4 -- 0x55a32ccf7400 con 0 > -183> 2018-07-11 07:35:46.667323 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:46.418365) v4 -- 0x55a324b15400 con 0 > -182> 2018-07-11 07:35:46.745825 7f35498a5700 10 monclient: tick > -181> 2018-07-11 07:35:46.745838 7f35498a5700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2018-07-11 07:35:16.745836) > -180> 2018-07-11 07:35:46.941922 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12677 conn(0x55a32169d800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1895 0x55a34de6f000 osd_ping(ping e494 stamp 2018-07-11 07:35:46.978799) v4 > -179> 2018-07-11 07:35:46.941944 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.1 10.8.128.106:0/12677 1895 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:46.978799) v4 ==== 2004+0+0 (3453155393 0 0) 0x55a34de6f000 con 0x55a32169d800 > -178> 2018-07-11 07:35:46.941936 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12677 conn(0x55a32169c000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.1 seq 1895 0x55a315fede00 osd_ping(ping e494 stamp 2018-07-11 07:35:46.978799) v4 > -177> 2018-07-11 07:35:46.941953 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:46.978799) v4 -- 0x55a3177a8e00 con 0 > -176> 2018-07-11 07:35:46.941955 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.1 10.8.128.106:0/12677 1895 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:46.978799) v4 ==== 2004+0+0 (3453155393 0 0) 0x55a315fede00 con 0x55a32169c000 > -175> 2018-07-11 07:35:46.941974 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:46.978799) v4 -- 0x55a3282bb400 con 0 > -174> 2018-07-11 07:35:46.982538 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a326726480 con 0 > -173> 2018-07-11 07:35:47.010928 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5233 0x55a326706000 replica scrub(pg: 6.15,from:0'0,to:430'105963,epoch:494/492,start:6:aa57e543:::benchmark_data_magna108_35617_object1054139:0,end:6:aa58e1c1:::benchmark_data_magna108_35617_object643260:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -172> 2018-07-11 07:35:47.010958 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5233 ==== replica scrub(pg: 6.15,from:0'0,to:430'105963,epoch:494/492,start:6:aa57e543:::benchmark_data_magna108_35617_object1054139:0,end:6:aa58e1c1:::benchmark_data_magna108_35617_object643260:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 219+0+0 (684433461 0 0) 0x55a326706000 con 0x55a317019000 > -171> 2018-07-11 07:35:47.259872 7f35488a3700 1 -- 10.8.128.107:6800/12665 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55a332443b80 con 0 > -170> 2018-07-11 07:35:47.259944 7f35488a3700 1 -- 10.8.128.107:6800/12665 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55a32ebb8000 con 0 > -169> 2018-07-11 07:35:47.590930 7f355d08d700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.106:0/12650 conn(0x55a311b67000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1889 0x55a34866a400 osd_ping(ping e494 stamp 2018-07-11 07:35:47.627798) v4 > -168> 2018-07-11 07:35:47.590961 7f355d08d700 1 -- 10.8.128.107:6803/12665 <== osd.3 10.8.128.106:0/12650 1889 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:47.627798) v4 ==== 2004+0+0 (4199427749 0 0) 0x55a34866a400 con 0x55a311b67000 > -167> 2018-07-11 07:35:47.590982 7f355d08d700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:47.627798) v4 -- 0x55a32f760600 con 0 > -166> 2018-07-11 07:35:47.590990 7f355d88e700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.106:0/12650 conn(0x55a311b68800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1889 0x55a34de78600 osd_ping(ping e494 stamp 2018-07-11 07:35:47.627798) v4 > -165> 2018-07-11 07:35:47.591008 7f355d88e700 1 -- 10.8.128.107:6802/12665 <== osd.3 10.8.128.106:0/12650 1889 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:47.627798) v4 ==== 2004+0+0 (4199427749 0 0) 0x55a34de78600 con 0x55a311b68800 > -164> 2018-07-11 07:35:47.591028 7f355d88e700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:47.627798) v4 -- 0x55a3177a8e00 con 0 > -163> 2018-07-11 07:35:47.663969 7f354689f700 1 -- 10.8.128.107:6801/12665 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e494 from shard 10) v1 -- 0x55a31cd2e240 con 0 > -162> 2018-07-11 07:35:47.682826 7f355d08d700 5 -- 10.8.128.107:6801/12665 >> 10.8.128.113:6805/15263 conn(0x55a317019000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.5 seq 5234 0x55a326706280 replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:494/492,start:6:aa58e1c1:::benchmark_data_magna108_35617_object643260:0,end:6:aa59b6d2:::benchmark_data_magna108.ceph.redhat.com_2478_object119775:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -161> 2018-07-11 07:35:47.682856 7f355d08d700 1 -- 10.8.128.107:6801/12665 <== osd.5 10.8.128.113:6805/15263 5234 ==== replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:494/492,start:6:aa58e1c1:::benchmark_data_magna108_35617_object643260:0,end:6:aa59b6d2:::benchmark_data_magna108.ceph.redhat.com_2478_object119775:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (3097651273 0 0) 0x55a326706280 con 0x55a317019000 > -160> 2018-07-11 07:35:47.686998 7f3543899700 5 osd.10 494 heartbeat: osd_stat(785 GB used, 146 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,1]) > -159> 2018-07-11 07:35:47.687013 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6802/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:35:47.687012) v4 -- 0x55a333037200 con 0 > -158> 2018-07-11 07:35:47.687037 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6803/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:35:47.687012) v4 -- 0x55a34de6f400 con 0 > -157> 2018-07-11 07:35:47.687052 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6806/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:35:47.687012) v4 -- 0x55a33bff0c00 con 0 > -156> 2018-07-11 07:35:47.687066 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6807/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:35:47.687012) v4 -- 0x55a332e78000 con 0 > -155> 2018-07-11 07:35:47.687088 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6808/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:35:47.687012) v4 -- 0x55a33bff0800 con 0 > -154> 2018-07-11 07:35:47.687110 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6810/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:35:47.687012) v4 -- 0x55a333036200 con 0 > -153> 2018-07-11 07:35:47.687139 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6809/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:35:47.687012) v4 -- 0x55a343719a00 con 0 > -152> 2018-07-11 07:35:47.687158 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6811/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:35:47.687012) v4 -- 0x55a333036a00 con 0 > -151> 2018-07-11 07:35:47.687185 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6810/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:35:47.687012) v4 -- 0x55a333036600 con 0 > -150> 2018-07-11 07:35:47.687206 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6811/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:35:47.687012) v4 -- 0x55a343719c00 con 0 > -149> 2018-07-11 07:35:47.687228 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6806/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:35:47.687012) v4 -- 0x55a333036400 con 0 > -148> 2018-07-11 07:35:47.687242 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6807/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:35:47.687012) v4 -- 0x55a333037c00 con 0 > -147> 2018-07-11 07:35:47.687279 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6802/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:35:47.687012) v4 -- 0x55a333037600 con 0 > -146> 2018-07-11 07:35:47.687302 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6803/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:35:47.687012) v4 -- 0x55a333036c00 con 0 > -145> 2018-07-11 07:35:47.687318 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6806/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:35:47.687012) v4 -- 0x55a333037400 con 0 > -144> 2018-07-11 07:35:47.687338 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6807/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:35:47.687012) v4 -- 0x55a33047f600 con 0 > -143> 2018-07-11 07:35:47.687365 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6802/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:35:47.687012) v4 -- 0x55a3131bce00 con 0 > -142> 2018-07-11 07:35:47.687387 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6803/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:35:47.687012) v4 -- 0x55a333037000 con 0 > -141> 2018-07-11 07:35:47.687409 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6806/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:35:47.687012) v4 -- 0x55a3282bb600 con 0 > -140> 2018-07-11 07:35:47.687434 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6807/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:35:47.687012) v4 -- 0x55a33047ee00 con 0 > -139> 2018-07-11 07:35:47.687445 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6806/12677 conn(0x55a314e84000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1880 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:47.687012) v4 > -138> 2018-07-11 07:35:47.687462 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6802/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:35:47.687012) v4 -- 0x55a333037a00 con 0 > -137> 2018-07-11 07:35:47.687467 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6806/12677 1880 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:47.687012) v4 ==== 2004+0+0 (1031161723 0 0) 0x55a34de6f000 con 0x55a314e84000 > -136> 2018-07-11 07:35:47.687482 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6803/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:35:47.687012) v4 -- 0x55a3131ba400 con 0 > -135> 2018-07-11 07:35:47.687533 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6808/12651 conn(0x55a314e96000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1880 0x55a32f760600 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:47.687012) v4 > -134> 2018-07-11 07:35:47.687542 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6803/15255 conn(0x55a314e1d800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1880 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:47.687012) v4 > -133> 2018-07-11 07:35:47.687559 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6808/12651 1880 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:47.687012) v4 ==== 2004+0+0 (1031161723 0 0) 0x55a32f760600 con 0x55a314e96000 > -132> 2018-07-11 07:35:47.687564 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6803/15255 1880 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:47.687012) v4 ==== 2004+0+0 (1031161723 0 0) 0x55a332e79a00 con 0x55a314e1d800 > -131> 2018-07-11 07:35:47.687600 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6802/15255 conn(0x55a314e1c000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1880 0x55a32f760600 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:47.687012) v4 > -130> 2018-07-11 07:35:47.687606 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6809/12650 conn(0x55a314e1f000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1880 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:47.687012) v4 > -129> 2018-07-11 07:35:47.687609 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6810/12651 conn(0x55a314e97800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1880 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:47.687012) v4 > -128> 2018-07-11 07:35:47.687619 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6809/12650 1880 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:47.687012) v4 ==== 2004+0+0 (1031161723 0 0) 0x55a34de78600 con 0x55a314e1f000 > -127> 2018-07-11 07:35:47.687623 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6810/12651 1880 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:47.687012) v4 ==== 2004+0+0 (1031161723 0 0) 0x55a332e79a00 con 0x55a314e97800 > -126> 2018-07-11 07:35:47.687619 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6802/15255 1880 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:47.687012) v4 ==== 2004+0+0 (1031161723 0 0) 0x55a32f760600 con 0x55a314e1c000 > -125> 2018-07-11 07:35:47.687665 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6811/15287 conn(0x55a314e04800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1880 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:47.687012) v4 > -124> 2018-07-11 07:35:47.687678 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6811/15287 1880 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:47.687012) v4 ==== 2004+0+0 (1031161723 0 0) 0x55a332e79a00 con 0x55a314e04800 > -123> 2018-07-11 07:35:47.687669 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6807/12677 conn(0x55a314e85800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1880 0x55a32f760600 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:47.687012) v4 > -122> 2018-07-11 07:35:47.687694 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6807/12617 conn(0x55a314e27800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1880 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:47.687012) v4 > -121> 2018-07-11 07:35:47.687699 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6807/12677 1880 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:47.687012) v4 ==== 2004+0+0 (1031161723 0 0) 0x55a32f760600 con 0x55a314e85800 > -120> 2018-07-11 07:35:47.687709 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6807/12617 1880 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:47.687012) v4 ==== 2004+0+0 (1031161723 0 0) 0x55a34de78600 con 0x55a314e27800 > -119> 2018-07-11 07:35:47.687707 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6806/12617 conn(0x55a314e26000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1880 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:47.687012) v4 > -118> 2018-07-11 07:35:47.687716 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6806/12617 1880 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:47.687012) v4 ==== 2004+0+0 (1031161723 0 0) 0x55a332e79a00 con 0x55a314e26000 > -117> 2018-07-11 07:35:47.687754 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6806/15885 conn(0x55a314e00000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1880 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:47.687012) v4 > -116> 2018-07-11 07:35:47.687756 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6806/15263 conn(0x55a314e99000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1880 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:47.687012) v4 > -115> 2018-07-11 07:35:47.687767 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6806/15263 1880 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:47.687012) v4 ==== 2004+0+0 (1031161723 0 0) 0x55a34de78600 con 0x55a314e99000 > -114> 2018-07-11 07:35:47.687767 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6806/15885 1880 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:47.687012) v4 ==== 2004+0+0 (1031161723 0 0) 0x55a332e79a00 con 0x55a314e00000 > -113> 2018-07-11 07:35:47.687782 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6811/12650 conn(0x55a314e20800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1880 0x55a32f760600 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:47.687012) v4 > -112> 2018-07-11 07:35:47.687796 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6811/12650 1880 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:47.687012) v4 ==== 2004+0+0 (1031161723 0 0) 0x55a32f760600 con 0x55a314e20800 > -111> 2018-07-11 07:35:47.687802 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6807/15885 conn(0x55a314e01800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1880 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:47.687012) v4 > -110> 2018-07-11 07:35:47.687815 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6807/15885 1880 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:47.687012) v4 ==== 2004+0+0 (1031161723 0 0) 0x55a34de78600 con 0x55a314e01800 > -109> 2018-07-11 07:35:47.687822 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6802/13323 conn(0x55a314e15000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1880 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:47.687012) v4 > -108> 2018-07-11 07:35:47.687837 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6802/13323 1880 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:47.687012) v4 ==== 2004+0+0 (1031161723 0 0) 0x55a332e79a00 con 0x55a314e15000 > -107> 2018-07-11 07:35:47.687848 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6803/13323 conn(0x55a314e16800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1880 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:47.687012) v4 > -106> 2018-07-11 07:35:47.687858 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6802/15919 conn(0x55a314e87000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1880 0x55a332e79a00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:47.687012) v4 > -105> 2018-07-11 07:35:47.687848 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6810/15287 conn(0x55a314e03000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1880 0x55a32f760600 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:47.687012) v4 > -104> 2018-07-11 07:35:47.687866 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6802/15919 1880 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:47.687012) v4 ==== 2004+0+0 (1031161723 0 0) 0x55a332e79a00 con 0x55a314e87000 > -103> 2018-07-11 07:35:47.687871 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6810/15287 1880 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:47.687012) v4 ==== 2004+0+0 (1031161723 0 0) 0x55a32f760600 con 0x55a314e03000 > -102> 2018-07-11 07:35:47.687866 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6803/13323 1880 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:47.687012) v4 ==== 2004+0+0 (1031161723 0 0) 0x55a34de78600 con 0x55a314e16800 > -101> 2018-07-11 07:35:47.687906 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6802/11967 conn(0x55a314e12000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1880 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:47.687012) v4 > -100> 2018-07-11 07:35:47.687915 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6807/15263 conn(0x55a314e9a800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1880 0x55a32f760600 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:47.687012) v4 > -99> 2018-07-11 07:35:47.687922 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6802/11967 1880 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:47.687012) v4 ==== 2004+0+0 (1031161723 0 0) 0x55a34de78600 con 0x55a314e12000 > -98> 2018-07-11 07:35:47.687925 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6807/15263 1880 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:47.687012) v4 ==== 2004+0+0 (1031161723 0 0) 0x55a32f760600 con 0x55a314e9a800 > -97> 2018-07-11 07:35:47.687959 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6803/11967 conn(0x55a314e13800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1880 0x55a32f760600 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:47.687012) v4 > -96> 2018-07-11 07:35:47.687971 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6803/11967 1880 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:47.687012) v4 ==== 2004+0+0 (1031161723 0 0) 0x55a32f760600 con 0x55a314e13800 > -95> 2018-07-11 07:35:47.687996 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6803/15919 conn(0x55a314e88800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1880 0x55a34de78600 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:47.687012) v4 > -94> 2018-07-11 07:35:47.688018 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6803/15919 1880 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:47.687012) v4 ==== 2004+0+0 (1031161723 0 0) 0x55a34de78600 con 0x55a314e88800 > -93> 2018-07-11 07:35:48.187604 7f3543899700 5 osd.10 494 heartbeat: osd_stat(785 GB used, 146 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,0,1]) > -92> 2018-07-11 07:35:48.187621 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6802/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:35:48.187619) v4 -- 0x55a333037800 con 0 > -91> 2018-07-11 07:35:48.187649 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6803/15255 -- osd_ping(ping e494 stamp 2018-07-11 07:35:48.187619) v4 -- 0x55a33047f200 con 0 > -90> 2018-07-11 07:35:48.187682 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6806/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:35:48.187619) v4 -- 0x55a315fec200 con 0 > -89> 2018-07-11 07:35:48.187699 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6807/12677 -- osd_ping(ping e494 stamp 2018-07-11 07:35:48.187619) v4 -- 0x55a333037e00 con 0 > -88> 2018-07-11 07:35:48.187716 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6808/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:35:48.187619) v4 -- 0x55a332e78e00 con 0 > -87> 2018-07-11 07:35:48.187733 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6810/12651 -- osd_ping(ping e494 stamp 2018-07-11 07:35:48.187619) v4 -- 0x55a34866a000 con 0 > -86> 2018-07-11 07:35:48.187751 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6809/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:35:48.187619) v4 -- 0x55a3314cfa00 con 0 > -85> 2018-07-11 07:35:48.187777 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.106:6811/12650 -- osd_ping(ping e494 stamp 2018-07-11 07:35:48.187619) v4 -- 0x55a34de85600 con 0 > -84> 2018-07-11 07:35:48.187791 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6810/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:35:48.187619) v4 -- 0x55a32700b200 con 0 > -83> 2018-07-11 07:35:48.187813 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6811/15287 -- osd_ping(ping e494 stamp 2018-07-11 07:35:48.187619) v4 -- 0x55a32700ae00 con 0 > -82> 2018-07-11 07:35:48.187828 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6806/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:35:48.187619) v4 -- 0x55a32700b600 con 0 > -81> 2018-07-11 07:35:48.187841 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.113:6807/15263 -- osd_ping(ping e494 stamp 2018-07-11 07:35:48.187619) v4 -- 0x55a32700be00 con 0 > -80> 2018-07-11 07:35:48.187853 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6802/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:35:48.187619) v4 -- 0x55a33047f800 con 0 > -79> 2018-07-11 07:35:48.187865 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.115:6803/13323 -- osd_ping(ping e494 stamp 2018-07-11 07:35:48.187619) v4 -- 0x55a33bff1000 con 0 > -78> 2018-07-11 07:35:48.187878 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6806/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:35:48.187619) v4 -- 0x55a3282bbe00 con 0 > -77> 2018-07-11 07:35:48.187890 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6807/15885 -- osd_ping(ping e494 stamp 2018-07-11 07:35:48.187619) v4 -- 0x55a3131bc800 con 0 > -76> 2018-07-11 07:35:48.187903 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6802/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:35:48.187619) v4 -- 0x55a327c0f200 con 0 > -75> 2018-07-11 07:35:48.187920 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.108:6803/11967 -- osd_ping(ping e494 stamp 2018-07-11 07:35:48.187619) v4 -- 0x55a35772be00 con 0 > -74> 2018-07-11 07:35:48.187933 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6806/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:35:48.187619) v4 -- 0x55a31ea06c00 con 0 > -73> 2018-07-11 07:35:48.187945 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.107:6807/12617 -- osd_ping(ping e494 stamp 2018-07-11 07:35:48.187619) v4 -- 0x55a327c0ea00 con 0 > -72> 2018-07-11 07:35:48.187958 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6802/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:35:48.187619) v4 -- 0x55a31ea06200 con 0 > -71> 2018-07-11 07:35:48.187969 7f3543899700 1 -- 10.8.128.107:0/12665 --> 10.8.128.114:6803/15919 -- osd_ping(ping e494 stamp 2018-07-11 07:35:48.187619) v4 -- 0x55a327c0ee00 con 0 > -70> 2018-07-11 07:35:48.188074 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6802/15255 conn(0x55a314e1c000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1881 0x55a3282bb400 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:48.187619) v4 > -69> 2018-07-11 07:35:48.188098 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6802/15255 1881 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:48.187619) v4 ==== 2004+0+0 (2649923136 0 0) 0x55a3282bb400 con 0x55a314e1c000 > -68> 2018-07-11 07:35:48.188146 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6803/15255 conn(0x55a314e1d800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.0 seq 1881 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:48.187619) v4 > -67> 2018-07-11 07:35:48.188174 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.0 10.8.128.113:6803/15255 1881 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:48.187619) v4 ==== 2004+0+0 (2649923136 0 0) 0x55a324474c00 con 0x55a314e1d800 > -66> 2018-07-11 07:35:48.188192 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6807/12617 conn(0x55a314e27800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1881 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:48.187619) v4 > -65> 2018-07-11 07:35:48.188194 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6808/12651 conn(0x55a314e96000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1881 0x55a3282bb400 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:48.187619) v4 > -64> 2018-07-11 07:35:48.188211 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.107:6806/12617 conn(0x55a314e26000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.9 seq 1881 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:48.187619) v4 > -63> 2018-07-11 07:35:48.188216 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6807/12617 1881 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:48.187619) v4 ==== 2004+0+0 (2649923136 0 0) 0x55a34de6f000 con 0x55a314e27800 > -62> 2018-07-11 07:35:48.188221 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6808/12651 1881 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:48.187619) v4 ==== 2004+0+0 (2649923136 0 0) 0x55a3282bb400 con 0x55a314e96000 > -61> 2018-07-11 07:35:48.188226 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.9 10.8.128.107:6806/12617 1881 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:48.187619) v4 ==== 2004+0+0 (2649923136 0 0) 0x55a324474c00 con 0x55a314e26000 > -60> 2018-07-11 07:35:48.188282 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6806/12677 conn(0x55a314e84000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1881 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:48.187619) v4 > -59> 2018-07-11 07:35:48.188281 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6810/12651 conn(0x55a314e97800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.2 seq 1881 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:48.187619) v4 > -58> 2018-07-11 07:35:48.188300 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6807/15263 conn(0x55a314e9a800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1881 0x55a3282bb400 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:48.187619) v4 > -57> 2018-07-11 07:35:48.188304 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6806/12677 1881 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:48.187619) v4 ==== 2004+0+0 (2649923136 0 0) 0x55a34de6f000 con 0x55a314e84000 > -56> 2018-07-11 07:35:48.188307 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.2 10.8.128.106:6810/12651 1881 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:48.187619) v4 ==== 2004+0+0 (2649923136 0 0) 0x55a324474c00 con 0x55a314e97800 > -55> 2018-07-11 07:35:48.188315 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6807/15263 1881 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:48.187619) v4 ==== 2004+0+0 (2649923136 0 0) 0x55a3282bb400 con 0x55a314e9a800 > -54> 2018-07-11 07:35:48.188360 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6807/12677 conn(0x55a314e85800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.1 seq 1881 0x55a3282bb400 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:48.187619) v4 > -53> 2018-07-11 07:35:48.188360 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6809/12650 conn(0x55a314e1f000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1881 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:48.187619) v4 > -52> 2018-07-11 07:35:48.188360 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6811/15287 conn(0x55a314e04800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1881 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:48.187619) v4 > -51> 2018-07-11 07:35:48.188373 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.1 10.8.128.106:6807/12677 1881 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:48.187619) v4 ==== 2004+0+0 (2649923136 0 0) 0x55a3282bb400 con 0x55a314e85800 > -50> 2018-07-11 07:35:48.188378 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6809/12650 1881 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:48.187619) v4 ==== 2004+0+0 (2649923136 0 0) 0x55a34de6f000 con 0x55a314e1f000 > -49> 2018-07-11 07:35:48.188378 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6811/15287 1881 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:48.187619) v4 ==== 2004+0+0 (2649923136 0 0) 0x55a324474c00 con 0x55a314e04800 > -48> 2018-07-11 07:35:48.188401 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6810/15287 conn(0x55a314e03000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.4 seq 1881 0x55a3282bb400 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:48.187619) v4 > -47> 2018-07-11 07:35:48.188417 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.4 10.8.128.113:6810/15287 1881 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:48.187619) v4 ==== 2004+0+0 (2649923136 0 0) 0x55a3282bb400 con 0x55a314e03000 > -46> 2018-07-11 07:35:48.188431 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6802/13323 conn(0x55a314e15000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1881 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:48.187619) v4 > -45> 2018-07-11 07:35:48.188451 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6802/13323 1881 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:48.187619) v4 ==== 2004+0+0 (2649923136 0 0) 0x55a324474c00 con 0x55a314e15000 > -44> 2018-07-11 07:35:48.188430 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.113:6806/15263 conn(0x55a314e99000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.5 seq 1881 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:48.187619) v4 > -43> 2018-07-11 07:35:48.188467 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.5 10.8.128.113:6806/15263 1881 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:48.187619) v4 ==== 2004+0+0 (2649923136 0 0) 0x55a34de6f000 con 0x55a314e99000 > -42> 2018-07-11 07:35:48.188469 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.106:6811/12650 conn(0x55a314e20800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.3 seq 1881 0x55a3282bb400 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:48.187619) v4 > -41> 2018-07-11 07:35:48.188484 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.3 10.8.128.106:6811/12650 1881 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:48.187619) v4 ==== 2004+0+0 (2649923136 0 0) 0x55a3282bb400 con 0x55a314e20800 > -40> 2018-07-11 07:35:48.188498 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6806/15885 conn(0x55a314e00000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1881 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:48.187619) v4 > -39> 2018-07-11 07:35:48.188509 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6802/11967 conn(0x55a314e12000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1881 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:48.187619) v4 > -38> 2018-07-11 07:35:48.188513 7f355d08d700 5 -- 10.8.128.107:0/12665 >> 10.8.128.108:6803/11967 conn(0x55a314e13800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.8 seq 1881 0x55a3282bb400 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:48.187619) v4 > -37> 2018-07-11 07:35:48.188518 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6806/15885 1881 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:48.187619) v4 ==== 2004+0+0 (2649923136 0 0) 0x55a324474c00 con 0x55a314e00000 > -36> 2018-07-11 07:35:48.188524 7f355d08d700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6803/11967 1881 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:48.187619) v4 ==== 2004+0+0 (2649923136 0 0) 0x55a3282bb400 con 0x55a314e13800 > -35> 2018-07-11 07:35:48.188523 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.8 10.8.128.108:6802/11967 1881 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:48.187619) v4 ==== 2004+0+0 (2649923136 0 0) 0x55a34de6f000 con 0x55a314e12000 > -34> 2018-07-11 07:35:48.188576 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.115:6803/13323 conn(0x55a314e16800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1881 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:48.187619) v4 > -33> 2018-07-11 07:35:48.188576 7f355c88c700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6802/15919 conn(0x55a314e87000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1881 0x55a324474c00 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:48.187619) v4 > -32> 2018-07-11 07:35:48.188599 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.6 10.8.128.115:6803/13323 1881 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:48.187619) v4 ==== 2004+0+0 (2649923136 0 0) 0x55a34de6f000 con 0x55a314e16800 > -31> 2018-07-11 07:35:48.188601 7f355c88c700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6802/15919 1881 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:48.187619) v4 ==== 2004+0+0 (2649923136 0 0) 0x55a324474c00 con 0x55a314e87000 > -30> 2018-07-11 07:35:48.188648 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6807/15885 conn(0x55a314e01800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.7 seq 1881 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:48.187619) v4 > -29> 2018-07-11 07:35:48.188663 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.7 10.8.128.114:6807/15885 1881 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:48.187619) v4 ==== 2004+0+0 (2649923136 0 0) 0x55a34de6f000 con 0x55a314e01800 > -28> 2018-07-11 07:35:48.188696 7f355d88e700 5 -- 10.8.128.107:0/12665 >> 10.8.128.114:6803/15919 conn(0x55a314e88800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.11 seq 1881 0x55a34de6f000 osd_ping(ping_reply e494 stamp 2018-07-11 07:35:48.187619) v4 > -27> 2018-07-11 07:35:48.188708 7f355d88e700 1 -- 10.8.128.107:0/12665 <== osd.11 10.8.128.114:6803/15919 1881 ==== osd_ping(ping_reply e494 stamp 2018-07-11 07:35:48.187619) v4 ==== 2004+0+0 (2649923136 0 0) 0x55a34de6f000 con 0x55a314e88800 > -26> 2018-07-11 07:35:48.258555 7f354e0ae700 -1 bluefs _allocate failed to allocate 0x3000000 on bdev 2, free 0x1700000 > -25> 2018-07-11 07:35:48.258569 7f354e0ae700 0 stupidalloc 0x0x55a30f764070 dump free bin 0: 0 extents > -24> 2018-07-11 07:35:48.258574 7f354e0ae700 0 stupidalloc 0x0x55a30f764070 dump free bin 1: 0 extents > -23> 2018-07-11 07:35:48.258576 7f354e0ae700 0 stupidalloc 0x0x55a30f764070 dump free bin 2: 0 extents > -22> 2018-07-11 07:35:48.258577 7f354e0ae700 0 stupidalloc 0x0x55a30f764070 dump free bin 3: 0 extents > -21> 2018-07-11 07:35:48.258579 7f354e0ae700 0 stupidalloc 0x0x55a30f764070 dump free bin 4: 0 extents > -20> 2018-07-11 07:35:48.258580 7f354e0ae700 0 stupidalloc 0x0x55a30f764070 dump free bin 5: 0 extents > -19> 2018-07-11 07:35:48.258582 7f354e0ae700 0 stupidalloc 0x0x55a30f764070 dump free bin 6: 0 extents > -18> 2018-07-11 07:35:48.258584 7f354e0ae700 0 stupidalloc 0x0x55a30f764070 dump free bin 7: 0 extents > -17> 2018-07-11 07:35:48.258585 7f354e0ae700 0 stupidalloc 0x0x55a30f764070 dump free bin 8: 0 extents > -16> 2018-07-11 07:35:48.258587 7f354e0ae700 0 stupidalloc 0x0x55a30f764070 dump free bin 9: 14 extents > -15> 2018-07-11 07:35:48.258588 7f354e0ae700 0 stupidalloc 0x0x55a30f764070 dump 0x26fcc00000~100000 > -14> 2018-07-11 07:35:48.258590 7f354e0ae700 0 stupidalloc 0x0x55a30f764070 dump 0x26ff300000~100000 > -13> 2018-07-11 07:35:48.258591 7f354e0ae700 0 stupidalloc 0x0x55a30f764070 dump 0x33ed800000~100000 > -12> 2018-07-11 07:35:48.258592 7f354e0ae700 0 stupidalloc 0x0x55a30f764070 dump 0x7649900000~100000 > -11> 2018-07-11 07:35:48.258594 7f354e0ae700 0 stupidalloc 0x0x55a30f764070 dump 0x9c2d700000~100000 > -10> 2018-07-11 07:35:48.258595 7f354e0ae700 0 stupidalloc 0x0x55a30f764070 dump 0x9c4d200000~100000 > -9> 2018-07-11 07:35:48.258597 7f354e0ae700 0 stupidalloc 0x0x55a30f764070 dump 0x9d69d00000~200000 > -8> 2018-07-11 07:35:48.258598 7f354e0ae700 0 stupidalloc 0x0x55a30f764070 dump 0xa1f8e00000~100000 > -7> 2018-07-11 07:35:48.258599 7f354e0ae700 0 stupidalloc 0x0x55a30f764070 dump 0xa3b9d00000~100000 > -6> 2018-07-11 07:35:48.258601 7f354e0ae700 0 stupidalloc 0x0x55a30f764070 dump 0xc70a100000~500000 > -5> 2018-07-11 07:35:48.258602 7f354e0ae700 0 stupidalloc 0x0x55a30f764070 dump 0xc72bb00000~200000 > -4> 2018-07-11 07:35:48.258603 7f354e0ae700 0 stupidalloc 0x0x55a30f764070 dump 0xc76ae00000~100000 > -3> 2018-07-11 07:35:48.258604 7f354e0ae700 0 stupidalloc 0x0x55a30f764070 dump 0xc781400000~200000 > -2> 2018-07-11 07:35:48.258606 7f354e0ae700 0 stupidalloc 0x0x55a30f764070 dump 0xc98f300000~300000 > -1> 2018-07-11 07:35:48.258608 7f354e0ae700 -1 bluefs _flush_range allocated: 0x0 offset: 0x0 length: 0x2f0901c > 0> 2018-07-11 07:35:48.264086 7f354e0ae700 -1 /builddir/build/BUILD/ceph-12.2.5/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, uint64_t)' thread 7f354e0ae700 time 2018-07-11 07:35:48.258621 >/builddir/build/BUILD/ceph-12.2.5/src/os/bluestore/BlueFS.cc: 1525: FAILED assert(0 == "bluefs enospc") > > ceph version 12.2.5-27.el7cp (6b7d1a8cf2e2376fd0a12424947c8c97aa1c0890) luminous (stable) > 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x110) [0x55a304abd710] > 2: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1b1b) [0x55a304a3c13b] > 3: (BlueFS::_flush(BlueFS::FileWriter*, bool)+0x188) [0x55a304a3c6a8] > 4: (BlueRocksWritableFile::Flush()+0x3c) [0x55a304a5823c] > 5: (rocksdb::WritableFileWriter::Flush()+0xaf) [0x55a304e2755f] > 6: (rocksdb::WritableFileWriter::Sync(bool)+0x33) [0x55a304e27773] > 7: (rocksdb::CompactionJob::FinishCompactionOutputFile(rocksdb::Status const&, rocksdb::CompactionJob::SubcompactionState*, rocksdb::RangeDelAggregator*, CompactionIterationStats*, rocksdb::Slice const*)+0xa1b) [0x55a304eed7ab] > 8: (rocksdb::CompactionJob::ProcessKeyValueCompaction(rocksdb::CompactionJob::SubcompactionState*)+0xea2) [0x55a304ef0222] > 9: (rocksdb::CompactionJob::Run()+0x1de) [0x55a304ef0cde] > 10: (rocksdb::DBImpl::BackgroundCompaction(bool*, rocksdb::JobContext*, rocksdb::LogBuffer*, void*)+0x9c7) [0x55a304e7c687] > 11: (rocksdb::DBImpl::BackgroundCallCompaction(void*)+0x283) [0x55a304e7fc73] > 12: (rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long)+0x271) [0x55a304ee6e31] > 13: (rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*)+0x3d) [0x55a304ee701d] > 14: (()+0xb5070) [0x7f355fafc070] > 15: (()+0x7dd5) [0x7f356016fdd5] > 16: (clone()+0x6d) [0x7f355f260b3d] > NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. > >--- logging levels --- > 0/ 5 none > 0/ 1 lockdep > 0/ 1 context > 1/ 1 crush > 1/ 5 mds > 1/ 5 mds_balancer > 1/ 5 mds_locker > 1/ 5 mds_log > 1/ 5 mds_log_expire > 1/ 5 mds_migrator > 0/ 1 buffer > 0/ 1 timer > 0/ 1 filer > 0/ 1 striper > 0/ 1 objecter > 0/ 5 rados > 0/ 5 rbd > 0/ 5 rbd_mirror > 0/ 5 rbd_replay > 0/ 5 journaler > 0/ 5 objectcacher > 0/ 5 client > 1/ 5 osd > 0/ 5 optracker > 0/ 5 objclass > 1/ 3 filestore > 1/ 3 journal > 0/ 5 ms > 1/ 5 mon > 0/10 monc > 1/ 5 paxos > 0/ 5 tp > 1/ 5 auth > 1/ 5 crypto > 1/ 1 finisher > 1/ 1 reserver > 1/ 5 heartbeatmap > 1/ 5 perfcounter > 1/ 5 rgw > 1/10 civetweb > 1/ 5 javaclient > 1/ 5 asok > 1/ 1 throttle > 0/ 0 refs > 1/ 5 xio > 1/ 5 compressor > 1/ 5 bluestore > 1/ 5 bluefs > 1/ 3 bdev > 1/ 5 kstore > 4/ 5 rocksdb > 4/ 5 leveldb > 4/ 5 memdb > 1/ 5 kinetic > 1/ 5 fuse > 1/ 5 mgr > 1/ 5 mgrc > 1/ 5 dpdk > 1/ 5 eventtrace > -2/-2 (syslog threshold) > -1/-1 (stderr threshold) > max_recent 10000 > max_new 1000 > log_file /var/log/ceph/ceph-osd.10.log >--- end dump of recent events --- >2018-07-11 07:35:48.311552 7f354e0ae700 -1 *** Caught signal (Aborted) ** > in thread 7f354e0ae700 thread_name:rocksdb:bg0 > > ceph version 12.2.5-27.el7cp (6b7d1a8cf2e2376fd0a12424947c8c97aa1c0890) luminous (stable) > 1: (()+0xa42d81) [0x55a304a7ed81] > 2: (()+0xf680) [0x7f3560177680] > 3: (gsignal()+0x37) [0x7f355f198207] > 4: (abort()+0x148) [0x7f355f1998f8] > 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x284) [0x55a304abd884] > 6: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1b1b) [0x55a304a3c13b] > 7: (BlueFS::_flush(BlueFS::FileWriter*, bool)+0x188) [0x55a304a3c6a8] > 8: (BlueRocksWritableFile::Flush()+0x3c) [0x55a304a5823c] > 9: (rocksdb::WritableFileWriter::Flush()+0xaf) [0x55a304e2755f] > 10: (rocksdb::WritableFileWriter::Sync(bool)+0x33) [0x55a304e27773] > 11: (rocksdb::CompactionJob::FinishCompactionOutputFile(rocksdb::Status const&, rocksdb::CompactionJob::SubcompactionState*, rocksdb::RangeDelAggregator*, CompactionIterationStats*, rocksdb::Slice const*)+0xa1b) [0x55a304eed7ab] > 12: (rocksdb::CompactionJob::ProcessKeyValueCompaction(rocksdb::CompactionJob::SubcompactionState*)+0xea2) [0x55a304ef0222] > 13: (rocksdb::CompactionJob::Run()+0x1de) [0x55a304ef0cde] > 14: (rocksdb::DBImpl::BackgroundCompaction(bool*, rocksdb::JobContext*, rocksdb::LogBuffer*, void*)+0x9c7) [0x55a304e7c687] > 15: (rocksdb::DBImpl::BackgroundCallCompaction(void*)+0x283) [0x55a304e7fc73] > 16: (rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long)+0x271) [0x55a304ee6e31] > 17: (rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*)+0x3d) [0x55a304ee701d] > 18: (()+0xb5070) [0x7f355fafc070] > 19: (()+0x7dd5) [0x7f356016fdd5] > 20: (clone()+0x6d) [0x7f355f260b3d] > NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. > >--- begin dump of recent events --- > -6> 2018-07-11 07:35:48.264881 7f355c88c700 5 -- 10.8.128.107:6802/12665 >> 10.8.128.115:0/13323 conn(0x55a321665800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1884 0x55a324474c00 osd_ping(ping e494 stamp 2018-07-11 07:35:48.603013) v4 > -5> 2018-07-11 07:35:48.264903 7f355c88c700 1 -- 10.8.128.107:6802/12665 <== osd.6 10.8.128.115:0/13323 1884 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:48.603013) v4 ==== 2004+0+0 (1313543086 0 0) 0x55a324474c00 con 0x55a321665800 > -4> 2018-07-11 07:35:48.264913 7f355c88c700 1 -- 10.8.128.107:6802/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:48.603013) v4 -- 0x55a34866a000 con 0 > -3> 2018-07-11 07:35:48.264980 7f355c88c700 5 -- 10.8.128.107:6803/12665 >> 10.8.128.115:0/13323 conn(0x55a321668800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=11 cs=1 l=1). rx osd.6 seq 1884 0x55a332e79a00 osd_ping(ping e494 stamp 2018-07-11 07:35:48.603013) v4 > -2> 2018-07-11 07:35:48.264994 7f355c88c700 1 -- 10.8.128.107:6803/12665 <== osd.6 10.8.128.115:0/13323 1884 ==== osd_ping(ping e494 stamp 2018-07-11 07:35:48.603013) v4 ==== 2004+0+0 (1313543086 0 0) 0x55a332e79a00 con 0x55a321668800 > -1> 2018-07-11 07:35:48.265002 7f355c88c700 1 -- 10.8.128.107:6803/12665 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e494 stamp 2018-07-11 07:35:48.603013) v4 -- 0x55a34866a000 con 0 > 0> 2018-07-11 07:35:48.311552 7f354e0ae700 -1 *** Caught signal (Aborted) ** > in thread 7f354e0ae700 thread_name:rocksdb:bg0 > > ceph version 12.2.5-27.el7cp (6b7d1a8cf2e2376fd0a12424947c8c97aa1c0890) luminous (stable) > 1: (()+0xa42d81) [0x55a304a7ed81] > 2: (()+0xf680) [0x7f3560177680] > 3: (gsignal()+0x37) [0x7f355f198207] > 4: (abort()+0x148) [0x7f355f1998f8] > 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x284) [0x55a304abd884] > 6: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1b1b) [0x55a304a3c13b] > 7: (BlueFS::_flush(BlueFS::FileWriter*, bool)+0x188) [0x55a304a3c6a8] > 8: (BlueRocksWritableFile::Flush()+0x3c) [0x55a304a5823c] > 9: (rocksdb::WritableFileWriter::Flush()+0xaf) [0x55a304e2755f] > 10: (rocksdb::WritableFileWriter::Sync(bool)+0x33) [0x55a304e27773] > 11: (rocksdb::CompactionJob::FinishCompactionOutputFile(rocksdb::Status const&, rocksdb::CompactionJob::SubcompactionState*, rocksdb::RangeDelAggregator*, CompactionIterationStats*, rocksdb::Slice const*)+0xa1b) [0x55a304eed7ab] > 12: (rocksdb::CompactionJob::ProcessKeyValueCompaction(rocksdb::CompactionJob::SubcompactionState*)+0xea2) [0x55a304ef0222] > 13: (rocksdb::CompactionJob::Run()+0x1de) [0x55a304ef0cde] > 14: (rocksdb::DBImpl::BackgroundCompaction(bool*, rocksdb::JobContext*, rocksdb::LogBuffer*, void*)+0x9c7) [0x55a304e7c687] > 15: (rocksdb::DBImpl::BackgroundCallCompaction(void*)+0x283) [0x55a304e7fc73] > 16: (rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long)+0x271) [0x55a304ee6e31] > 17: (rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*)+0x3d) [0x55a304ee701d] > 18: (()+0xb5070) [0x7f355fafc070] > 19: (()+0x7dd5) [0x7f356016fdd5] > 20: (clone()+0x6d) [0x7f355f260b3d] > NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. > >--- logging levels --- > 0/ 5 none > 0/ 1 lockdep > 0/ 1 context > 1/ 1 crush > 1/ 5 mds > 1/ 5 mds_balancer > 1/ 5 mds_locker > 1/ 5 mds_log > 1/ 5 mds_log_expire > 1/ 5 mds_migrator > 0/ 1 buffer > 0/ 1 timer > 0/ 1 filer > 0/ 1 striper > 0/ 1 objecter > 0/ 5 rados > 0/ 5 rbd > 0/ 5 rbd_mirror > 0/ 5 rbd_replay > 0/ 5 journaler > 0/ 5 objectcacher > 0/ 5 client > 1/ 5 osd > 0/ 5 optracker > 0/ 5 objclass > 1/ 3 filestore > 1/ 3 journal > 0/ 5 ms > 1/ 5 mon > 0/10 monc > 1/ 5 paxos > 0/ 5 tp > 1/ 5 auth > 1/ 5 crypto > 1/ 1 finisher > 1/ 1 reserver > 1/ 5 heartbeatmap > 1/ 5 perfcounter > 1/ 5 rgw > 1/10 civetweb > 1/ 5 javaclient > 1/ 5 asok > 1/ 1 throttle > 0/ 0 refs > 1/ 5 xio > 1/ 5 compressor > 1/ 5 bluestore > 1/ 5 bluefs > 1/ 3 bdev > 1/ 5 kstore > 4/ 5 rocksdb > 4/ 5 leveldb > 4/ 5 memdb > 1/ 5 kinetic > 1/ 5 fuse > 1/ 5 mgr > 1/ 5 mgrc > 1/ 5 dpdk > 1/ 5 eventtrace > -2/-2 (syslog threshold) > -1/-1 (stderr threshold) > max_recent 10000 > max_new 1000 > log_file /var/log/ceph/ceph-osd.10.log >--- end dump of recent events --- >2018-07-11 07:36:51.134173 7ffad0decd80 0 set uid:gid to 167:167 (ceph:ceph) >2018-07-11 07:36:51.134207 7ffad0decd80 0 ceph version 12.2.5-27.el7cp (6b7d1a8cf2e2376fd0a12424947c8c97aa1c0890) luminous (stable), process (unknown), pid 13927 >2018-07-11 07:36:51.140601 7ffad0decd80 0 pidfile_write: ignore empty --pid-file >2018-07-11 07:36:51.177408 7ffad0decd80 0 load: jerasure load: lrc load: isa >2018-07-11 07:36:51.177509 7ffad0decd80 1 bdev create path /var/lib/ceph/osd/ceph-10/block type kernel >2018-07-11 07:36:51.177520 7ffad0decd80 1 bdev(0x55d97f5acc00 /var/lib/ceph/osd/ceph-10/block) open path /var/lib/ceph/osd/ceph-10/block >2018-07-11 07:36:51.178034 7ffad0decd80 1 bdev(0x55d97f5acc00 /var/lib/ceph/osd/ceph-10/block) open size 1000098959360 (0xe8da8b1000, 931 GB) block_size 4096 (4096 B) rotational >2018-07-11 07:36:51.178292 7ffad0decd80 1 bluestore(/var/lib/ceph/osd/ceph-10) _set_cache_sizes max 0.5 < ratio 0.99 >2018-07-11 07:36:51.178326 7ffad0decd80 1 bluestore(/var/lib/ceph/osd/ceph-10) _set_cache_sizes cache_size 1073741824 meta 0.5 kv 0.5 data 0 >2018-07-11 07:36:51.178334 7ffad0decd80 1 bdev(0x55d97f5acc00 /var/lib/ceph/osd/ceph-10/block) close >2018-07-11 07:36:51.432856 7ffad0decd80 1 bluestore(/var/lib/ceph/osd/ceph-10) _mount path /var/lib/ceph/osd/ceph-10 >2018-07-11 07:36:51.433179 7ffad0decd80 1 bdev create path /var/lib/ceph/osd/ceph-10/block type kernel >2018-07-11 07:36:51.433190 7ffad0decd80 1 bdev(0x55d97f5aea00 /var/lib/ceph/osd/ceph-10/block) open path /var/lib/ceph/osd/ceph-10/block >2018-07-11 07:36:51.433415 7ffad0decd80 1 bdev(0x55d97f5aea00 /var/lib/ceph/osd/ceph-10/block) open size 1000098959360 (0xe8da8b1000, 931 GB) block_size 4096 (4096 B) rotational >2018-07-11 07:36:51.433667 7ffad0decd80 1 bluestore(/var/lib/ceph/osd/ceph-10) _set_cache_sizes max 0.5 < ratio 0.99 >2018-07-11 07:36:51.433679 7ffad0decd80 1 bluestore(/var/lib/ceph/osd/ceph-10) _set_cache_sizes cache_size 1073741824 meta 0.5 kv 0.5 data 0 >2018-07-11 07:36:51.433760 7ffad0decd80 1 bdev create path /var/lib/ceph/osd/ceph-10/block.db type kernel >2018-07-11 07:36:51.433768 7ffad0decd80 1 bdev(0x55d97f4d5800 /var/lib/ceph/osd/ceph-10/block.db) open path /var/lib/ceph/osd/ceph-10/block.db >2018-07-11 07:36:51.433951 7ffad0decd80 1 bdev(0x55d97f4d5800 /var/lib/ceph/osd/ceph-10/block.db) open size 1073741824 (0x40000000, 1024 MB) block_size 4096 (4096 B) rotational >2018-07-11 07:36:51.433962 7ffad0decd80 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-10/block.db size 1024 MB >2018-07-11 07:36:51.434228 7ffad0decd80 1 bdev create path /var/lib/ceph/osd/ceph-10/block type kernel >2018-07-11 07:36:51.434235 7ffad0decd80 1 bdev(0x55d97f5afa00 /var/lib/ceph/osd/ceph-10/block) open path /var/lib/ceph/osd/ceph-10/block >2018-07-11 07:36:51.434414 7ffad0decd80 1 bdev(0x55d97f5afa00 /var/lib/ceph/osd/ceph-10/block) open size 1000098959360 (0xe8da8b1000, 931 GB) block_size 4096 (4096 B) rotational >2018-07-11 07:36:51.434422 7ffad0decd80 1 bluefs add_block_device bdev 2 path /var/lib/ceph/osd/ceph-10/block size 931 GB >2018-07-11 07:36:51.434460 7ffad0decd80 1 bdev create path /var/lib/ceph/osd/ceph-10/block.wal type kernel >2018-07-11 07:36:51.434465 7ffad0decd80 1 bdev(0x55d97f5aee00 /var/lib/ceph/osd/ceph-10/block.wal) open path /var/lib/ceph/osd/ceph-10/block.wal >2018-07-11 07:36:51.434633 7ffad0decd80 1 bdev(0x55d97f5aee00 /var/lib/ceph/osd/ceph-10/block.wal) open size 603979776 (0x24000000, 576 MB) block_size 4096 (4096 B) rotational >2018-07-11 07:36:51.434641 7ffad0decd80 1 bluefs add_block_device bdev 0 path /var/lib/ceph/osd/ceph-10/block.wal size 576 MB >2018-07-11 07:36:51.434869 7ffad0decd80 1 bluefs mount >2018-07-11 07:36:51.542903 7ffad0decd80 0 set rocksdb option compaction_readahead_size = 2097152 >2018-07-11 07:36:51.542919 7ffad0decd80 0 set rocksdb option compression = kNoCompression >2018-07-11 07:36:51.542937 7ffad0decd80 0 set rocksdb option max_write_buffer_number = 4 >2018-07-11 07:36:51.542942 7ffad0decd80 0 set rocksdb option min_write_buffer_number_to_merge = 1 >2018-07-11 07:36:51.542945 7ffad0decd80 0 set rocksdb option recycle_log_file_num = 4 >2018-07-11 07:36:51.542948 7ffad0decd80 0 set rocksdb option writable_file_max_buffer_size = 0 >2018-07-11 07:36:51.542952 7ffad0decd80 0 set rocksdb option write_buffer_size = 268435456 >2018-07-11 07:36:51.542979 7ffad0decd80 0 set rocksdb option compaction_readahead_size = 2097152 >2018-07-11 07:36:51.542983 7ffad0decd80 0 set rocksdb option compression = kNoCompression >2018-07-11 07:36:51.542987 7ffad0decd80 0 set rocksdb option max_write_buffer_number = 4 >2018-07-11 07:36:51.542990 7ffad0decd80 0 set rocksdb option min_write_buffer_number_to_merge = 1 >2018-07-11 07:36:51.542994 7ffad0decd80 0 set rocksdb option recycle_log_file_num = 4 >2018-07-11 07:36:51.542997 7ffad0decd80 0 set rocksdb option writable_file_max_buffer_size = 0 >2018-07-11 07:36:51.543000 7ffad0decd80 0 set rocksdb option write_buffer_size = 268435456 >2018-07-11 07:36:51.543149 7ffad0decd80 4 rocksdb: RocksDB version: 5.4.0 > >2018-07-11 07:36:51.543160 7ffad0decd80 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ >2018-07-11 07:36:51.543162 7ffad0decd80 4 rocksdb: Compile date Jun 28 2018 >2018-07-11 07:36:51.543165 7ffad0decd80 4 rocksdb: DB SUMMARY > >2018-07-11 07:36:51.543186 7ffad0decd80 4 rocksdb: CURRENT file: CURRENT > >2018-07-11 07:36:51.543194 7ffad0decd80 4 rocksdb: IDENTITY file: IDENTITY > >2018-07-11 07:36:51.543202 7ffad0decd80 4 rocksdb: MANIFEST file: MANIFEST-036296 size: 384429 Bytes > >2018-07-11 07:36:51.543207 7ffad0decd80 4 rocksdb: SST files in db dir, Total Num: 18, files: 036405.sst 036503.sst 036560.sst 036562.sst 036563.sst 036564.sst 036566.sst 036567.sst 036568.sst > >2018-07-11 07:36:51.546162 7ffad0decd80 4 rocksdb: SST files in db.slow dir, Total Num: 4977, files: 000549.sst 000550.sst 000565.sst 000566.sst 000605.sst 000617.sst 000620.sst 000622.sst 000634.sst > >2018-07-11 07:36:51.546238 7ffad0decd80 4 rocksdb: Write Ahead Log file in db.wal: 036463.log size: 755588953 ; 036492.log size: 686899020 ; > >2018-07-11 07:36:51.546253 7ffad0decd80 4 rocksdb: Options.error_if_exists: 0 >2018-07-11 07:36:51.546255 7ffad0decd80 4 rocksdb: Options.create_if_missing: 0 >2018-07-11 07:36:51.546256 7ffad0decd80 4 rocksdb: Options.paranoid_checks: 1 >2018-07-11 07:36:51.546257 7ffad0decd80 4 rocksdb: Options.env: 0x55d97f5a7040 >2018-07-11 07:36:51.546258 7ffad0decd80 4 rocksdb: Options.info_log: 0x55d97f9412c0 >2018-07-11 07:36:51.546259 7ffad0decd80 4 rocksdb: Options.max_open_files: -1 >2018-07-11 07:36:51.546260 7ffad0decd80 4 rocksdb: Options.max_file_opening_threads: 16 >2018-07-11 07:36:51.546260 7ffad0decd80 4 rocksdb: Options.use_fsync: 0 >2018-07-11 07:36:51.546261 7ffad0decd80 4 rocksdb: Options.max_log_file_size: 0 >2018-07-11 07:36:51.546262 7ffad0decd80 4 rocksdb: Options.max_manifest_file_size: 18446744073709551615 >2018-07-11 07:36:51.546263 7ffad0decd80 4 rocksdb: Options.log_file_time_to_roll: 0 >2018-07-11 07:36:51.546264 7ffad0decd80 4 rocksdb: Options.keep_log_file_num: 1000 >2018-07-11 07:36:51.546265 7ffad0decd80 4 rocksdb: Options.recycle_log_file_num: 4 >2018-07-11 07:36:51.546265 7ffad0decd80 4 rocksdb: Options.allow_fallocate: 1 >2018-07-11 07:36:51.546266 7ffad0decd80 4 rocksdb: Options.allow_mmap_reads: 0 >2018-07-11 07:36:51.546267 7ffad0decd80 4 rocksdb: Options.allow_mmap_writes: 0 >2018-07-11 07:36:51.546268 7ffad0decd80 4 rocksdb: Options.use_direct_reads: 0 >2018-07-11 07:36:51.546268 7ffad0decd80 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 >2018-07-11 07:36:51.546269 7ffad0decd80 4 rocksdb: Options.create_missing_column_families: 0 >2018-07-11 07:36:51.546270 7ffad0decd80 4 rocksdb: Options.db_log_dir: >2018-07-11 07:36:51.546271 7ffad0decd80 4 rocksdb: Options.wal_dir: db.wal >2018-07-11 07:36:51.546271 7ffad0decd80 4 rocksdb: Options.table_cache_numshardbits: 6 >2018-07-11 07:36:51.546272 7ffad0decd80 4 rocksdb: Options.max_subcompactions: 1 >2018-07-11 07:36:51.546273 7ffad0decd80 4 rocksdb: Options.max_background_flushes: 1 >2018-07-11 07:36:51.546274 7ffad0decd80 4 rocksdb: Options.WAL_ttl_seconds: 0 >2018-07-11 07:36:51.546275 7ffad0decd80 4 rocksdb: Options.WAL_size_limit_MB: 0 >2018-07-11 07:36:51.546276 7ffad0decd80 4 rocksdb: Options.manifest_preallocation_size: 4194304 >2018-07-11 07:36:51.546276 7ffad0decd80 4 rocksdb: Options.is_fd_close_on_exec: 1 >2018-07-11 07:36:51.546277 7ffad0decd80 4 rocksdb: Options.advise_random_on_open: 1 >2018-07-11 07:36:51.546278 7ffad0decd80 4 rocksdb: Options.db_write_buffer_size: 0 >2018-07-11 07:36:51.546278 7ffad0decd80 4 rocksdb: Options.access_hint_on_compaction_start: 1 >2018-07-11 07:36:51.546279 7ffad0decd80 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 1 >2018-07-11 07:36:51.546280 7ffad0decd80 4 rocksdb: Options.compaction_readahead_size: 2097152 >2018-07-11 07:36:51.546281 7ffad0decd80 4 rocksdb: Options.random_access_max_buffer_size: 1048576 >2018-07-11 07:36:51.546281 7ffad0decd80 4 rocksdb: Options.writable_file_max_buffer_size: 0 >2018-07-11 07:36:51.546282 7ffad0decd80 4 rocksdb: Options.use_adaptive_mutex: 0 >2018-07-11 07:36:51.546283 7ffad0decd80 4 rocksdb: Options.rate_limiter: (nil) >2018-07-11 07:36:51.546284 7ffad0decd80 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 >2018-07-11 07:36:51.546284 7ffad0decd80 4 rocksdb: Options.bytes_per_sync: 0 >2018-07-11 07:36:51.546285 7ffad0decd80 4 rocksdb: Options.wal_bytes_per_sync: 0 >2018-07-11 07:36:51.546286 7ffad0decd80 4 rocksdb: Options.wal_recovery_mode: 0 >2018-07-11 07:36:51.546287 7ffad0decd80 4 rocksdb: Options.enable_thread_tracking: 0 >2018-07-11 07:36:51.546287 7ffad0decd80 4 rocksdb: Options.allow_concurrent_memtable_write: 1 >2018-07-11 07:36:51.546288 7ffad0decd80 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 >2018-07-11 07:36:51.546289 7ffad0decd80 4 rocksdb: Options.write_thread_max_yield_usec: 100 >2018-07-11 07:36:51.546289 7ffad0decd80 4 rocksdb: Options.write_thread_slow_yield_usec: 3 >2018-07-11 07:36:51.546290 7ffad0decd80 4 rocksdb: Options.row_cache: None >2018-07-11 07:36:51.546291 7ffad0decd80 4 rocksdb: Options.wal_filter: None >2018-07-11 07:36:51.546292 7ffad0decd80 4 rocksdb: Options.avoid_flush_during_recovery: 0 >2018-07-11 07:36:51.546293 7ffad0decd80 4 rocksdb: Options.base_background_compactions: 1 >2018-07-11 07:36:51.546294 7ffad0decd80 4 rocksdb: Options.max_background_compactions: 1 >2018-07-11 07:36:51.546294 7ffad0decd80 4 rocksdb: Options.avoid_flush_during_shutdown: 0 >2018-07-11 07:36:51.546295 7ffad0decd80 4 rocksdb: Options.delayed_write_rate : 16777216 >2018-07-11 07:36:51.546296 7ffad0decd80 4 rocksdb: Options.max_total_wal_size: 0 >2018-07-11 07:36:51.546297 7ffad0decd80 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 >2018-07-11 07:36:51.546297 7ffad0decd80 4 rocksdb: Options.stats_dump_period_sec: 600 >2018-07-11 07:36:51.546298 7ffad0decd80 4 rocksdb: Compression algorithms supported: >2018-07-11 07:36:51.546301 7ffad0decd80 4 rocksdb: Snappy supported: 0 >2018-07-11 07:36:51.546302 7ffad0decd80 4 rocksdb: Zlib supported: 0 >2018-07-11 07:36:51.546303 7ffad0decd80 4 rocksdb: Bzip supported: 0 >2018-07-11 07:36:51.546304 7ffad0decd80 4 rocksdb: LZ4 supported: 0 >2018-07-11 07:36:51.546305 7ffad0decd80 4 rocksdb: ZSTD supported: 0 >2018-07-11 07:36:51.546306 7ffad0decd80 4 rocksdb: Fast CRC32 supported: 1 >2018-07-11 07:36:51.548985 7ffad0decd80 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/version_set.cc:2609] Recovering from manifest file: MANIFEST-036296 > >2018-07-11 07:36:51.549044 7ffad0decd80 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:407] --------------- Options for column family [default]: > >2018-07-11 07:36:51.549048 7ffad0decd80 4 rocksdb: Options.comparator: leveldb.BytewiseComparator >2018-07-11 07:36:51.549058 7ffad0decd80 4 rocksdb: Options.merge_operator: .T:int64_array.b:bitwise_xor >2018-07-11 07:36:51.549059 7ffad0decd80 4 rocksdb: Options.compaction_filter: None >2018-07-11 07:36:51.549060 7ffad0decd80 4 rocksdb: Options.compaction_filter_factory: None >2018-07-11 07:36:51.549061 7ffad0decd80 4 rocksdb: Options.memtable_factory: SkipListFactory >2018-07-11 07:36:51.549061 7ffad0decd80 4 rocksdb: Options.table_factory: BlockBasedTable >2018-07-11 07:36:51.549081 7ffad0decd80 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55d97f246228) > cache_index_and_filter_blocks: 1 > cache_index_and_filter_blocks_with_high_priority: 1 > pin_l0_filter_and_index_blocks_in_cache: 1 > index_type: 0 > hash_index_allow_collision: 1 > checksum: 1 > no_block_cache: 0 > block_cache: 0x55d97f6aa408 > block_cache_name: LRUCache > block_cache_options: > capacity : 536870912 > num_shard_bits : 4 > strict_capacity_limit : 0 > high_pri_pool_ratio: 0.000 > block_cache_compressed: (nil) > persistent_cache: (nil) > block_size: 4096 > block_size_deviation: 10 > block_restart_interval: 16 > index_block_restart_interval: 1 > filter_policy: rocksdb.BuiltinBloomFilter > whole_key_filtering: 1 > format_version: 2 > >2018-07-11 07:36:51.549085 7ffad0decd80 4 rocksdb: Options.write_buffer_size: 268435456 >2018-07-11 07:36:51.549087 7ffad0decd80 4 rocksdb: Options.max_write_buffer_number: 4 >2018-07-11 07:36:51.549088 7ffad0decd80 4 rocksdb: Options.compression: NoCompression >2018-07-11 07:36:51.549089 7ffad0decd80 4 rocksdb: Options.bottommost_compression: Disabled >2018-07-11 07:36:51.549090 7ffad0decd80 4 rocksdb: Options.prefix_extractor: nullptr >2018-07-11 07:36:51.549091 7ffad0decd80 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr >2018-07-11 07:36:51.549092 7ffad0decd80 4 rocksdb: Options.num_levels: 7 >2018-07-11 07:36:51.549092 7ffad0decd80 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 >2018-07-11 07:36:51.549093 7ffad0decd80 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 >2018-07-11 07:36:51.549094 7ffad0decd80 4 rocksdb: Options.compression_opts.window_bits: -14 >2018-07-11 07:36:51.549095 7ffad0decd80 4 rocksdb: Options.compression_opts.level: -1 >2018-07-11 07:36:51.549096 7ffad0decd80 4 rocksdb: Options.compression_opts.strategy: 0 >2018-07-11 07:36:51.549096 7ffad0decd80 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 >2018-07-11 07:36:51.549097 7ffad0decd80 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 >2018-07-11 07:36:51.549098 7ffad0decd80 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 >2018-07-11 07:36:51.549099 7ffad0decd80 4 rocksdb: Options.level0_stop_writes_trigger: 36 >2018-07-11 07:36:51.549100 7ffad0decd80 4 rocksdb: Options.target_file_size_base: 67108864 >2018-07-11 07:36:51.549101 7ffad0decd80 4 rocksdb: Options.target_file_size_multiplier: 1 >2018-07-11 07:36:51.549102 7ffad0decd80 4 rocksdb: Options.max_bytes_for_level_base: 268435456 >2018-07-11 07:36:51.549103 7ffad0decd80 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0 >2018-07-11 07:36:51.549103 7ffad0decd80 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 >2018-07-11 07:36:51.549106 7ffad0decd80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 >2018-07-11 07:36:51.549107 7ffad0decd80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 >2018-07-11 07:36:51.549108 7ffad0decd80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 >2018-07-11 07:36:51.549109 7ffad0decd80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 >2018-07-11 07:36:51.549110 7ffad0decd80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 >2018-07-11 07:36:51.549111 7ffad0decd80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 >2018-07-11 07:36:51.549111 7ffad0decd80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 >2018-07-11 07:36:51.549112 7ffad0decd80 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 >2018-07-11 07:36:51.549113 7ffad0decd80 4 rocksdb: Options.max_compaction_bytes: 1677721600 >2018-07-11 07:36:51.549114 7ffad0decd80 4 rocksdb: Options.arena_block_size: 33554432 >2018-07-11 07:36:51.549115 7ffad0decd80 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 >2018-07-11 07:36:51.549116 7ffad0decd80 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 >2018-07-11 07:36:51.549117 7ffad0decd80 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 >2018-07-11 07:36:51.549118 7ffad0decd80 4 rocksdb: Options.disable_auto_compactions: 0 >2018-07-11 07:36:51.549119 7ffad0decd80 4 rocksdb: Options.compaction_style: kCompactionStyleLevel >2018-07-11 07:36:51.549120 7ffad0decd80 4 rocksdb: Options.compaction_pri: kByCompensatedSize >2018-07-11 07:36:51.549121 7ffad0decd80 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 >2018-07-11 07:36:51.549122 7ffad0decd80 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 >2018-07-11 07:36:51.549123 7ffad0decd80 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 >2018-07-11 07:36:51.549129 7ffad0decd80 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 >2018-07-11 07:36:51.549130 7ffad0decd80 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 >2018-07-11 07:36:51.549131 7ffad0decd80 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 >2018-07-11 07:36:51.549132 7ffad0decd80 4 rocksdb: Options.table_properties_collectors: >2018-07-11 07:36:51.549133 7ffad0decd80 4 rocksdb: Options.inplace_update_support: 0 >2018-07-11 07:36:51.549134 7ffad0decd80 4 rocksdb: Options.inplace_update_num_locks: 10000 >2018-07-11 07:36:51.549135 7ffad0decd80 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 >2018-07-11 07:36:51.549136 7ffad0decd80 4 rocksdb: Options.memtable_huge_page_size: 0 >2018-07-11 07:36:51.549137 7ffad0decd80 4 rocksdb: Options.bloom_locality: 0 >2018-07-11 07:36:51.549138 7ffad0decd80 4 rocksdb: Options.max_successive_merges: 0 >2018-07-11 07:36:51.549139 7ffad0decd80 4 rocksdb: Options.optimize_filters_for_hits: 0 >2018-07-11 07:36:51.549139 7ffad0decd80 4 rocksdb: Options.paranoid_file_checks: 0 >2018-07-11 07:36:51.549140 7ffad0decd80 4 rocksdb: Options.force_consistency_checks: 0 >2018-07-11 07:36:51.549141 7ffad0decd80 4 rocksdb: Options.report_bg_io_stats: 0 >2018-07-11 07:38:00.542491 7ffad0decd80 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/version_set.cc:2859] Recovered from manifest file:db/MANIFEST-036296 succeeded,manifest_file_number is 36296, next_file_number is 36595, last_sequence is 12881315148, log_number is 0,prev_log_number is 0,max_column_family is 0 > >2018-07-11 07:38:00.542508 7ffad0decd80 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/version_set.cc:2867] Column family [default] (ID 0), log number is 36492 > >2018-07-11 07:38:00.549637 7ffad0decd80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294680549628, "job": 1, "event": "recovery_started", "log_files": [36463, 36492]} >2018-07-11 07:38:00.549645 7ffad0decd80 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_open.cc:482] Recovering log #36463 mode 0 >2018-07-11 07:38:08.555495 7ffad0decd80 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_open.cc:482] Recovering log #36492 mode 0 >2018-07-11 07:38:08.568490 7ffad0decd80 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/version_set.cc:2395] Creating manifest 36595 > >2018-07-11 07:38:08.606393 7ffad0decd80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294688606387, "job": 1, "event": "recovery_finished"} >2018-07-11 07:38:08.606427 7ffad0decd80 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 64884155848 >2018-07-11 07:38:08.608845 7ffad0decd80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294688608840, "job": 2, "event": "table_file_deletion", "file_number": 36602} >2018-07-11 07:38:08.608860 7ffad0decd80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294688608857, "job": 2, "event": "table_file_deletion", "file_number": 36601} >2018-07-11 07:38:08.608873 7ffad0decd80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294688608871, "job": 2, "event": "table_file_deletion", "file_number": 36600} >2018-07-11 07:38:08.608889 7ffad0decd80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294688608887, "job": 2, "event": "table_file_deletion", "file_number": 36599} >2018-07-11 07:38:08.608904 7ffad0decd80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294688608902, "job": 2, "event": "table_file_deletion", "file_number": 36598} >2018-07-11 07:38:08.608920 7ffad0decd80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294688608917, "job": 2, "event": "table_file_deletion", "file_number": 36597} >2018-07-11 07:38:08.608934 7ffad0decd80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294688608932, "job": 2, "event": "table_file_deletion", "file_number": 36596} >2018-07-11 07:38:08.608944 7ffad0decd80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294688608942, "job": 2, "event": "table_file_deletion", "file_number": 36595} >2018-07-11 07:38:08.608959 7ffad0decd80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294688608957, "job": 2, "event": "table_file_deletion", "file_number": 36594} >2018-07-11 07:38:08.631793 7ffad0decd80 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_open.cc:1063] DB pointer 0x55d97f946000 >2018-07-11 07:38:08.632119 7ffabc3f1700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS ------- >2018-07-11 07:38:08.632126 7ffabc3f1700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] >** DB Stats ** >Uptime(secs): 77.1 total, 77.1 interval >Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s >Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s >Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent >Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s >Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s >Interval stall: 00:00:0.000 H:M:S, 0.0 percent > >** Compaction Stats [default] ** >Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop >---------------------------------------------------------------------------------------------------------------------------------------------------------- > L1 18/0 1.04 GB 6.4 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 > L2 85/0 5.20 GB 3.1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 > L3 446/0 24.93 GB 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 > L4 3641/0 250.00 GB 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 > L5 796/0 55.24 GB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 > Sum 4986/0 336.41 GB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 > Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 >Uptime(secs): 77.1 total, 77.1 interval >Flush(GB): cumulative 0.000, interval 0.000 >AddFile(GB): cumulative 0.000, interval 0.000 >AddFile(Total Files): cumulative 0, interval 0 >AddFile(L0 Files): cumulative 0, interval 0 >AddFile(Keys): cumulative 0, interval 0 >Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds >Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds >Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count > >** File Read Latency Histogram By Level [default] ** > >** Compaction Stats [default] ** >Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop >---------------------------------------------------------------------------------------------------------------------------------------------------------- > L1 18/0 1.04 GB 6.4 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 > L2 85/0 5.20 GB 3.1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 > L3 446/0 24.93 GB 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 > L4 3641/0 250.00 GB 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 > L5 796/0 55.24 GB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 > Sum 4986/0 336.41 GB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 > Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 >Uptime(secs): 77.1 total, 77.1 interval >Flush(GB): cumulative 0.000, interval 0.000 >AddFile(GB): cumulative 0.000, interval 0.000 >AddFile(Total Files): cumulative 0, interval 0 >AddFile(L0 Files): cumulative 0, interval 0 >AddFile(Keys): cumulative 0, interval 0 >Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds >Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds >Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count > >** File Read Latency Histogram By Level [default] ** > >2018-07-11 07:38:08.632218 7ffad0decd80 1 bluestore(/var/lib/ceph/osd/ceph-10) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152 >2018-07-11 07:38:08.632343 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 3] Compacting 1@1 + 9@2 files to L2, score 6.35 >2018-07-11 07:38:08.632361 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2 Base level 1, inputs: [36562(76MB)], [36592(76MB) 36593(15MB) 36547(76MB) 36548(69MB) 36549(56MB) 36531(53MB) 36532(70MB) 36542(76MB) 36543(76MB)] > >2018-07-11 07:38:08.632377 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294688632365, "job": 3, "event": "compaction_started", "files_L1": [36562], "files_L2": [36592, 36593, 36547, 36548, 36549, 36531, 36532, 36542, 36543], "score": 6.35392, "input_data_size": 679330645} >2018-07-11 07:38:08.683408 7ffad0decd80 1 freelist init >2018-07-11 07:38:09.211309 7ffad0decd80 1 bluestore(/var/lib/ceph/osd/ceph-10) _open_alloc opening allocation metadata >2018-07-11 07:38:09.475995 7ffad0decd80 1 bluestore(/var/lib/ceph/osd/ceph-10) _open_alloc loaded 495 G in 77601 extents >2018-07-11 07:38:10.157686 7ffad0decd80 0 _get_class not permitted to load kvs >2018-07-11 07:38:10.157712 7ffad0decd80 0 _get_class not permitted to load lua >2018-07-11 07:38:10.160842 7ffad0decd80 0 <cls> /builddir/build/BUILD/ceph-12.2.5/src/cls/cephfs/cls_cephfs.cc:197: loading cephfs >2018-07-11 07:38:10.163179 7ffad0decd80 0 <cls> /builddir/build/BUILD/ceph-12.2.5/src/cls/hello/cls_hello.cc:296: loading cls_hello >2018-07-11 07:38:10.164142 7ffad0decd80 0 _get_class not permitted to load sdk >2018-07-11 07:38:10.164777 7ffad0decd80 0 osd.10 494 crush map has features 288514119978713088, adjusting msgr requires for clients >2018-07-11 07:38:10.164787 7ffad0decd80 0 osd.10 494 crush map has features 288514394856620032 was 8705, adjusting msgr requires for mons >2018-07-11 07:38:10.164805 7ffad0decd80 0 osd.10 494 crush map has features 1009090335235915776, adjusting msgr requires for osds >2018-07-11 07:38:11.901141 7ffad0decd80 0 osd.10 494 load_pgs >2018-07-11 07:38:16.076668 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 3] Generated table #36599: 4957167 keys, 80348309 bytes >2018-07-11 07:38:16.076725 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294696076693, "cf_name": "default", "job": 3, "event": "table_file_creation", "file_number": 36599, "file_size": 80348309, "table_properties": {"data_size": 67109393, "index_size": 689219, "filter_size": 12548707, "raw_key_size": 143207044, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957167, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957167", "kMergeOperands": "0"}} >2018-07-11 07:38:17.671705 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 3] Generated table #36600: 1042833 keys, 16903140 bytes >2018-07-11 07:38:17.671751 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294697671735, "cf_name": "default", "job": 3, "event": "table_file_creation", "file_number": 36600, "file_size": 16903140, "table_properties": {"data_size": 14117724, "index_size": 144573, "filter_size": 2639856, "raw_key_size": 30126296, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 3465, "num_entries": 1042833, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1042833", "kMergeOperands": "0"}} >2018-07-11 07:38:22.291061 7ffad0decd80 0 osd.10 494 load_pgs opened 71 pgs >2018-07-11 07:38:22.291089 7ffad0decd80 0 osd.10 494 using weightedpriority op queue with priority op cut off at 64. >2018-07-11 07:38:22.292284 7ffad0decd80 -1 osd.10 494 log_to_monitors {default=true} >2018-07-11 07:38:22.298176 7ffab63e5700 1 osd.10 pg_epoch: 494 pg[5.37( v 166'68122 (166'66622,166'68122] local-lis/les=492/493 n=68122 ec=71/71 lis/c 492/492 les/c/f 493/493/0 492/492/487) [4,10,8] r=1 lpr=493 crt=166'68122 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.298183 7ffab5be4700 1 osd.10 pg_epoch: 494 pg[5.35( v 166'67898 (166'66348,166'67898] local-lis/les=492/493 n=67898 ec=71/71 lis/c 492/492 les/c/f 493/493/0 492/492/470) [0,11,10] r=2 lpr=493 crt=166'67898 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.298327 7ffab5be4700 1 osd.10 pg_epoch: 494 pg[5.c( v 166'68190 (166'66593,166'68190] local-lis/les=492/493 n=68190 ec=71/71 lis/c 492/492 les/c/f 493/493/0 492/492/470) [1,10,4] r=1 lpr=493 crt=166'68190 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.298338 7ffab63e5700 1 osd.10 pg_epoch: 494 pg[5.33( v 166'68409 (166'66894,166'68409] local-lis/les=492/493 n=68409 ec=71/71 lis/c 492/492 les/c/f 493/493/0 492/492/465) [2,10,8] r=1 lpr=493 crt=166'68409 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.298424 7ffab5be4700 1 osd.10 pg_epoch: 494 pg[16.e( empty local-lis/les=492/493 n=0 ec=442/442 lis/c 492/492 les/c/f 493/493/0 492/492/465) [2,4,10] r=2 lpr=493 crt=0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.298462 7ffab63e5700 1 osd.10 pg_epoch: 494 pg[10.f( v 383'568 (0'0,383'568] local-lis/les=492/493 n=494 ec=193/193 lis/c 492/492 les/c/f 493/493/0 492/492/478) [11,10,2] r=1 lpr=493 crt=383'568 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.298521 7ffab5be4700 1 osd.10 pg_epoch: 494 pg[5.24( v 166'68298 (166'66798,166'68298] local-lis/les=492/493 n=68298 ec=71/71 lis/c 492/492 les/c/f 493/493/0 492/492/492) [10,11,1] r=0 lpr=493 crt=166'68298 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 07:38:22.298573 7ffab63e5700 1 osd.10 pg_epoch: 494 pg[6.3( v 430'104329 (430'102829,430'104329] local-lis/les=492/493 n=25881 ec=76/76 lis/c 492/492 les/c/f 493/493/0 492/492/470) [0,10,11] r=1 lpr=493 crt=430'104329 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.298671 7ffab63e5700 1 osd.10 pg_epoch: 494 pg[6.32( v 430'104474 (430'102974,430'104474] local-lis/les=492/493 n=25649 ec=76/76 lis/c 492/492 les/c/f 493/493/0 492/492/487) [4,2,10] r=2 lpr=493 crt=430'104474 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.298709 7ffab5be4700 1 osd.10 pg_epoch: 494 pg[13.ds1( v 377'10591 (355'1551,377'10591] local-lis/les=492/493 n=2744 ec=345/345 lis/c 492/492 les/c/f 493/493/0 492/492/470) [0,10,1,8,11,6]p0(0) r=1 lpr=493 crt=377'10591 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.298766 7ffab63e5700 1 osd.10 pg_epoch: 494 pg[6.0( v 430'122181 (430'120681,430'122181] local-lis/les=492/493 n=25944 ec=76/76 lis/c 492/492 les/c/f 493/493/0 492/492/492) [10,7,2] r=0 lpr=493 crt=430'122181 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 07:38:22.298840 7ffab5be4700 1 osd.10 pg_epoch: 494 pg[13.cs2( v 377'10679 (371'2117,377'10679] local-lis/les=492/493 n=2785 ec=345/345 lis/c 492/492 les/c/f 493/493/0 492/492/470) [1,11,10,4,8,6]p1(0) r=2 lpr=493 crt=377'10679 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.298892 7ffab63e5700 1 osd.10 pg_epoch: 494 pg[5.10( v 166'68154 (166'66598,166'68154] local-lis/les=492/493 n=68154 ec=71/71 lis/c 492/492 les/c/f 493/493/0 492/492/492) [10,8,4] r=0 lpr=493 crt=166'68154 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 07:38:22.298940 7ffab5be4700 1 osd.10 pg_epoch: 494 pg[16.10( empty local-lis/les=492/493 n=0 ec=442/442 lis/c 492/492 les/c/f 493/493/0 492/492/465) [2,10,7] r=1 lpr=493 crt=0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.299036 7ffab63e5700 1 osd.10 pg_epoch: 494 pg[11.4s2( v 383'60130 (352'56449,383'60130] local-lis/les=492/493 n=60130 ec=267/267 lis/c 492/492 les/c/f 493/493/0 492/492/470) [0,6,10,8,3,11]p0(0) r=2 lpr=493 crt=383'60130 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.299041 7ffab5be4700 1 osd.10 pg_epoch: 494 pg[10.a( v 386'2032 (203'500,386'2032] local-lis/les=492/493 n=516 ec=193/193 lis/c 492/492 les/c/f 493/493/0 492/492/492) [10,2,5] r=0 lpr=493 crt=386'2032 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 07:38:22.299158 7ffab63e5700 1 osd.10 pg_epoch: 494 pg[13.1s1( v 377'9368 (355'1456,377'9368] local-lis/les=492/493 n=2661 ec=345/345 lis/c 492/492 les/c/f 493/493/0 492/492/472) [8,10,0,2,6,7]p8(0) r=1 lpr=493 crt=377'9368 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.299165 7ffab5be4700 1 osd.10 pg_epoch: 494 pg[5.23( v 166'68497 (166'66949,166'68497] local-lis/les=492/493 n=68497 ec=71/71 lis/c 492/492 les/c/f 493/493/0 492/492/472) [8,3,10] r=2 lpr=493 crt=166'68497 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.299296 7ffab5be4700 1 osd.10 pg_epoch: 494 pg[6.1f( v 430'114415 (430'112915,430'114415] local-lis/les=492/493 n=26413 ec=76/76 lis/c 492/492 les/c/f 493/493/0 492/492/484) [3,10,0] r=1 lpr=493 crt=430'114415 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.299301 7ffab63e5700 1 osd.10 pg_epoch: 494 pg[11.3s5( v 383'60232 (352'56579,383'60232] local-lis/les=492/493 n=60232 ec=267/267 lis/c 492/492 les/c/f 493/493/0 492/492/481) [6,8,0,11,2,10]p6(0) r=5 lpr=493 crt=383'60232 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.299402 7ffab5be4700 1 osd.10 pg_epoch: 494 pg[10.13( v 383'839 (0'0,383'839] local-lis/les=492/493 n=523 ec=193/193 lis/c 492/492 les/c/f 493/493/0 492/492/470) [1,10,7] r=1 lpr=493 crt=383'839 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.299424 7ffab63e5700 1 osd.10 pg_epoch: 494 pg[11.6s0( v 383'59732 (352'56233,383'59732] local-lis/les=492/493 n=59732 ec=267/267 lis/c 492/492 les/c/f 493/493/0 492/492/492) [10,1,11,6,5,8]p10(0) r=0 lpr=493 crt=383'59732 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 07:38:22.299504 7ffab5be4700 1 osd.10 pg_epoch: 494 pg[5.9( v 166'68275 (166'66775,166'68275] local-lis/les=492/493 n=68275 ec=71/71 lis/c 492/492 les/c/f 493/493/0 492/492/492) [10,8,7] r=0 lpr=493 crt=166'68275 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 07:38:22.299563 7ffab63e5700 1 osd.10 pg_epoch: 494 pg[6.13( v 430'99276 (430'97776,430'99276] local-lis/les=492/493 n=26101 ec=76/76 lis/c 492/492 les/c/f 493/493/0 492/492/492) [10,7,0] r=0 lpr=493 crt=430'99276 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 07:38:22.299628 7ffab5be4700 1 osd.10 pg_epoch: 494 pg[6.a( v 430'109231 (430'107731,430'109231] local-lis/les=492/493 n=26267 ec=76/76 lis/c 492/492 les/c/f 493/493/0 492/492/467) [5,10,6] r=1 lpr=493 crt=430'109231 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.299690 7ffab63e5700 1 osd.10 pg_epoch: 494 pg[9.1b( v 463'472 (0'0,463'472] local-lis/les=492/493 n=256 ec=190/190 lis/c 492/492 les/c/f 493/493/0 492/492/492) [10,1,5] r=0 lpr=493 crt=463'472 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 07:38:22.299731 7ffab5be4700 1 osd.10 pg_epoch: 494 pg[12.0( v 383'7619 (383'6093,383'7619] local-lis/les=492/493 n=7619 ec=331/331 lis/c 492/492 les/c/f 493/493/0 492/492/487) [4,10,3] r=1 lpr=493 crt=383'7619 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.299813 7ffab63e5700 1 osd.10 pg_epoch: 494 pg[6.14( v 461'122884 (430'121291,461'122884] local-lis/les=492/493 n=25983 ec=76/76 lis/c 492/492 les/c/f 493/493/0 492/492/492) [10,4,3] r=0 lpr=493 crt=461'122884 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 07:38:22.299839 7ffab5be4700 1 osd.10 pg_epoch: 494 pg[5.4( v 166'68216 (166'66716,166'68216] local-lis/les=492/493 n=68216 ec=71/71 lis/c 492/492 les/c/f 493/493/0 492/492/468) [7,2,10] r=2 lpr=493 crt=166'68216 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.299949 7ffab5be4700 1 osd.10 pg_epoch: 494 pg[6.7( v 430'109641 (430'108141,430'109641] local-lis/les=492/493 n=25825 ec=76/76 lis/c 492/492 les/c/f 493/493/0 492/492/492) [10,6,3] r=0 lpr=493 crt=430'109641 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 07:38:22.299954 7ffab63e5700 1 osd.10 pg_epoch: 494 pg[5.14( v 166'68424 (166'66924,166'68424] local-lis/les=492/493 n=68424 ec=71/71 lis/c 492/492 les/c/f 493/493/0 492/492/468) [7,10,2] r=1 lpr=493 crt=166'68424 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.300053 7ffab63e5700 1 osd.10 pg_epoch: 494 pg[6.17( v 430'108120 (430'106620,430'108120] local-lis/les=492/493 n=26407 ec=76/76 lis/c 492/492 les/c/f 493/493/0 492/492/492) [10,2,5] r=0 lpr=493 crt=430'108120 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 07:38:22.300077 7ffab5be4700 1 osd.10 pg_epoch: 494 pg[6.e( v 430'116090 (430'114517,430'116090] local-lis/les=492/493 n=25967 ec=76/76 lis/c 492/492 les/c/f 493/493/0 492/492/487) [4,1,10] r=2 lpr=493 crt=430'116090 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.300175 7ffab63e5700 1 osd.10 pg_epoch: 494 pg[6.21( v 461'128646 (430'127048,461'128646] local-lis/les=492/493 n=25982 ec=76/76 lis/c 492/492 les/c/f 493/493/0 492/492/470) [0,2,10] r=2 lpr=493 crt=461'128646 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.300190 7ffab5be4700 1 osd.10 pg_epoch: 494 pg[11.8s3( v 383'59629 (352'55946,383'59629] local-lis/les=492/493 n=59629 ec=267/267 lis/c 492/492 les/c/f 493/493/0 492/492/468) [7,1,6,10,5,8]p7(0) r=3 lpr=493 crt=383'59629 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.300276 7ffab63e5700 1 osd.10 pg_epoch: 494 pg[10.9( v 390'615 (0'0,390'615] local-lis/les=492/493 n=478 ec=193/193 lis/c 492/492 les/c/f 493/493/0 492/492/470) [0,2,10] r=2 lpr=493 crt=390'615 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.300298 7ffab5be4700 1 osd.10 pg_epoch: 494 pg[13.8s5( v 377'9578 (367'1954,377'9578] local-lis/les=492/493 n=2631 ec=345/345 lis/c 492/492 les/c/f 493/493/0 492/492/465) [2,11,5,8,6,10]p2(0) r=5 lpr=493 crt=377'9578 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.300376 7ffab63e5700 1 osd.10 pg_epoch: 494 pg[5.1c( v 383'68804 (166'67302,383'68804] local-lis/les=492/493 n=68803 ec=71/71 lis/c 492/492 les/c/f 493/493/0 492/492/492) [10,8,2] r=0 lpr=493 crt=383'68804 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 07:38:22.300394 7ffab5be4700 1 osd.10 pg_epoch: 494 pg[6.d( v 461'110631 (430'109050,461'110631] local-lis/les=492/493 n=25732 ec=76/76 lis/c 492/492 les/c/f 493/493/0 492/492/492) [10,7,2] r=0 lpr=493 crt=461'110631 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 07:38:22.300517 7ffab63e5700 1 osd.10 pg_epoch: 494 pg[16.d( empty local-lis/les=492/493 n=0 ec=442/442 lis/c 492/492 les/c/f 493/493/0 492/492/470) [0,3,10] r=2 lpr=493 crt=0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.300517 7ffab5be4700 1 osd.10 pg_epoch: 494 pg[6.2( v 430'110237 (430'108737,430'110237] local-lis/les=492/493 n=25907 ec=76/76 lis/c 492/492 les/c/f 493/493/0 492/492/468) [7,10,2] r=1 lpr=493 crt=430'110237 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.300622 7ffab5be4700 1 osd.10 pg_epoch: 494 pg[11.as3( v 383'60057 (367'56547,383'60057] local-lis/les=492/493 n=60057 ec=267/267 lis/c 492/492 les/c/f 493/493/0 492/492/484) [3,5,8,10,11,6]p3(0) r=3 lpr=493 crt=383'60057 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.300637 7ffab63e5700 1 osd.10 pg_epoch: 494 pg[13.fs4( v 377'8989 (0'0,377'8989] local-lis/les=492/493 n=2712 ec=345/345 lis/c 492/492 les/c/f 493/493/0 492/492/467) [5,7,3,6,10,8]p5(0) r=4 lpr=493 crt=377'8979 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.300737 7ffab5be4700 1 osd.10 pg_epoch: 494 pg[13.bs4( v 377'9782 (355'1491,377'9782] local-lis/les=492/493 n=2694 ec=345/345 lis/c 492/492 les/c/f 493/493/0 492/492/478) [11,8,1,4,10,6]p11(0) r=4 lpr=493 crt=377'9782 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.300750 7ffab63e5700 1 osd.10 pg_epoch: 494 pg[16.1a( empty local-lis/les=492/493 n=0 ec=442/442 lis/c 492/492 les/c/f 493/493/0 492/492/478) [11,10,3] r=1 lpr=493 crt=0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.300851 7ffab5be4700 1 osd.10 pg_epoch: 494 pg[6.26( v 461'123787 (430'122194,461'123787] local-lis/les=492/493 n=26121 ec=76/76 lis/c 492/492 les/c/f 493/493/0 492/492/468) [7,10,8] r=1 lpr=493 crt=461'123787 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.300863 7ffab63e5700 1 osd.10 pg_epoch: 494 pg[6.15( v 463'106940 (430'105357,463'106940] local-lis/les=492/493 n=25757 ec=76/76 lis/c 492/492 les/c/f 493/493/0 492/492/467) [5,10,11] r=1 lpr=493 crt=463'106940 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.300965 7ffab63e5700 1 osd.10 pg_epoch: 494 pg[16.3( empty local-lis/les=492/493 n=0 ec=442/442 lis/c 492/492 les/c/f 493/493/0 492/492/467) [5,2,10] r=2 lpr=493 crt=0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.300965 7ffab5be4700 1 osd.10 pg_epoch: 494 pg[13.9s1( v 377'10260 (352'219,377'10260] local-lis/les=492/493 n=2727 ec=345/345 lis/c 492/492 les/c/f 493/493/0 492/492/467) [5,10,3,6,8,11]p5(0) r=1 lpr=493 crt=377'10254 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.301062 7ffab63e5700 1 osd.10 pg_epoch: 494 pg[13.3s1( v 377'10405 (371'2081,377'10405] local-lis/les=492/493 n=2736 ec=345/345 lis/c 492/492 les/c/f 493/493/0 492/492/470) [1,10,8,6,7,5]p1(0) r=1 lpr=493 crt=377'10405 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.301080 7ffab5be4700 1 osd.10 pg_epoch: 494 pg[11.ds3( v 383'59865 (367'56373,383'59865] local-lis/les=492/493 n=59865 ec=267/267 lis/c 492/492 les/c/f 493/493/0 492/492/465) [2,4,11,10,8,6]p2(0) r=3 lpr=493 crt=383'59865 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.301167 7ffab63e5700 1 osd.10 pg_epoch: 494 pg[11.fs3( v 383'63252 (352'59530,383'63252] local-lis/les=492/493 n=60320 ec=267/267 lis/c 492/492 les/c/f 493/493/0 492/492/470) [0,6,2,10,7,8]p0(0) r=3 lpr=493 crt=383'63252 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.301284 7ffab63e5700 1 osd.10 pg_epoch: 494 pg[11.5s0( v 383'60461 (352'56750,383'60461] local-lis/les=492/493 n=60461 ec=267/267 lis/c 492/492 les/c/f 493/493/0 492/492/492) [10,5,8,6,1,7]p10(0) r=0 lpr=493 crt=383'60461 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 07:38:22.301427 7ffab63e5700 1 osd.10 pg_epoch: 494 pg[11.2s2( v 383'59971 (377'58405,383'59971] local-lis/les=492/493 n=59971 ec=267/267 lis/c 492/492 les/c/f 493/493/0 492/492/465) [2,6,10,0,7,8]p2(0) r=2 lpr=493 crt=383'59970 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.301532 7ffab63e5700 1 osd.10 pg_epoch: 494 pg[13.2s4( v 377'9209 (0'0,377'9209] local-lis/les=492/493 n=2712 ec=345/345 lis/c 492/492 les/c/f 493/493/0 492/492/467) [5,8,6,7,10,3]p5(0) r=4 lpr=493 crt=377'9201 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.301633 7ffab63e5700 1 osd.10 pg_epoch: 494 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=492/493 n=68041 ec=71/71 lis/c 492/492 les/c/f 493/493/0 492/492/492) [10,2,4] r=0 lpr=493 crt=166'68041 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 07:38:22.301759 7ffab63e5700 1 osd.10 pg_epoch: 494 pg[11.1s0( v 383'60378 (352'56674,383'60378] local-lis/les=492/493 n=60378 ec=267/267 lis/c 492/492 les/c/f 493/493/0 492/492/492) [10,5,2,8,6,11]p10(0) r=0 lpr=493 crt=383'60378 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 07:38:22.301921 7ffab63e5700 1 osd.10 pg_epoch: 494 pg[13.4s3( v 377'9519 (355'1503,377'9519] local-lis/les=492/493 n=2672 ec=345/345 lis/c 492/492 les/c/f 493/493/0 492/492/467) [5,1,8,10,11,6]p5(0) r=3 lpr=493 crt=377'9519 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.302019 7ffab63e5700 1 osd.10 pg_epoch: 494 pg[5.2f( v 166'67864 (166'66267,166'67864] local-lis/les=492/493 n=67864 ec=71/71 lis/c 492/492 les/c/f 493/493/0 492/492/472) [8,10,4] r=1 lpr=493 crt=166'67864 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.302121 7ffab63e5700 1 osd.10 pg_epoch: 494 pg[6.2c( v 457'120217 (430'118627,457'120217] local-lis/les=492/493 n=26249 ec=76/76 lis/c 492/492 les/c/f 493/493/0 492/492/487) [4,6,10] r=2 lpr=493 crt=457'120217 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.302217 7ffab63e5700 1 osd.10 pg_epoch: 494 pg[10.7( v 386'575 (0'0,386'575] local-lis/les=492/493 n=507 ec=193/193 lis/c 492/492 les/c/f 493/493/0 492/492/492) [10,6,3] r=0 lpr=493 crt=386'575 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 07:38:22.302310 7ffad0decd80 0 osd.10 494 done with init, starting boot process >2018-07-11 07:38:22.302354 7ffad0decd80 1 osd.10 494 start_boot >2018-07-11 07:38:22.732812 7ffabd3f3700 1 osd.10 498 state: booting -> active >2018-07-11 07:38:22.733031 7ffab5be4700 1 osd.10 pg_epoch: 495 pg[6.32( v 430'104474 (430'102974,430'104474] local-lis/les=492/493 n=25649 ec=76/76 lis/c 492/492 les/c/f 493/493/0 495/495/487) [4,2] r=-1 lpr=495 pi=[492,495)/1 crt=430'104474 lcod 0'0 unknown NOTIFY] start_peering_interval up [4,2,10] -> [4,2], acting [4,2,10] -> [4,2], acting_primary 4 -> 4, up_primary 4 -> 4, role 2 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.733030 7ffab63e5700 1 osd.10 pg_epoch: 495 pg[5.37( v 166'68122 (166'66622,166'68122] local-lis/les=492/493 n=68122 ec=71/71 lis/c 492/492 les/c/f 493/493/0 495/495/487) [4,8] r=-1 lpr=495 pi=[492,495)/1 crt=166'68122 lcod 0'0 unknown NOTIFY] start_peering_interval up [4,10,8] -> [4,8], acting [4,10,8] -> [4,8], acting_primary 4 -> 4, up_primary 4 -> 4, role 1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.733227 7ffab63e5700 1 osd.10 pg_epoch: 498 pg[5.37( v 166'68122 (166'66622,166'68122] local-lis/les=492/493 n=68122 ec=71/71 lis/c 492/492 les/c/f 493/493/0 498/498/487) [4,10,8] r=1 lpr=498 pi=[492,498)/1 crt=166'68122 lcod 0'0 unknown NOTIFY] start_peering_interval up [4,8] -> [4,10,8], acting [4,8] -> [4,10,8], acting_primary 4 -> 4, up_primary 4 -> 4, role -1 -> 1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.733233 7ffab5be4700 1 osd.10 pg_epoch: 498 pg[6.32( v 430'104474 (430'102974,430'104474] local-lis/les=492/493 n=25649 ec=76/76 lis/c 492/492 les/c/f 493/493/0 498/498/487) [4,2,10] r=2 lpr=498 pi=[492,498)/1 crt=430'104474 lcod 0'0 unknown NOTIFY] start_peering_interval up [4,2] -> [4,2,10], acting [4,2] -> [4,2,10], acting_primary 4 -> 4, up_primary 4 -> 4, role -1 -> 2, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.733370 7ffab63e5700 1 osd.10 pg_epoch: 498 pg[5.37( v 166'68122 (166'66622,166'68122] local-lis/les=492/493 n=68122 ec=71/71 lis/c 492/492 les/c/f 493/493/0 498/498/487) [4,10,8] r=1 lpr=498 pi=[492,498)/1 crt=166'68122 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.733376 7ffab5be4700 1 osd.10 pg_epoch: 498 pg[6.32( v 430'104474 (430'102974,430'104474] local-lis/les=492/493 n=25649 ec=76/76 lis/c 492/492 les/c/f 493/493/0 498/498/487) [4,2,10] r=2 lpr=498 pi=[492,498)/1 crt=430'104474 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.733609 7ffab5be4700 1 osd.10 pg_epoch: 495 pg[6.0( v 430'122181 (430'120681,430'122181] local-lis/les=492/493 n=25944 ec=76/76 lis/c 492/492 les/c/f 493/493/0 492/492/492) [10,7,2] r=0 lpr=493 crt=430'122181 lcod 0'0 mlcod 0'0 peering] state<Started/Primary/Peering>: Peering, affected_by_map, going to Reset >2018-07-11 07:38:22.733615 7ffab63e5700 1 osd.10 pg_epoch: 495 pg[5.33( v 166'68409 (166'66894,166'68409] local-lis/les=492/493 n=68409 ec=71/71 lis/c 492/492 les/c/f 493/493/0 495/495/465) [2,8] r=-1 lpr=495 pi=[492,495)/1 crt=166'68409 lcod 0'0 unknown NOTIFY] start_peering_interval up [2,10,8] -> [2,8], acting [2,10,8] -> [2,8], acting_primary 2 -> 2, up_primary 2 -> 2, role 1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.733722 7ffab63e5700 1 osd.10 pg_epoch: 498 pg[5.33( v 166'68409 (166'66894,166'68409] local-lis/les=492/493 n=68409 ec=71/71 lis/c 492/492 les/c/f 493/493/0 498/498/465) [2,10,8] r=1 lpr=498 pi=[492,498)/1 crt=166'68409 lcod 0'0 unknown NOTIFY] start_peering_interval up [2,8] -> [2,10,8], acting [2,8] -> [2,10,8], acting_primary 2 -> 2, up_primary 2 -> 2, role -1 -> 1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.733786 7ffab5be4700 1 osd.10 pg_epoch: 495 pg[6.0( v 430'122181 (430'120681,430'122181] local-lis/les=492/493 n=25944 ec=76/76 lis/c 492/492 les/c/f 493/493/0 495/495/495) [7,2] r=-1 lpr=495 pi=[492,495)/1 crt=430'122181 lcod 0'0 unknown] start_peering_interval up [10,7,2] -> [7,2], acting [10,7,2] -> [7,2], acting_primary 10 -> 7, up_primary 10 -> 7, role 0 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.733838 7ffab63e5700 1 osd.10 pg_epoch: 498 pg[5.33( v 166'68409 (166'66894,166'68409] local-lis/les=492/493 n=68409 ec=71/71 lis/c 492/492 les/c/f 493/493/0 498/498/465) [2,10,8] r=1 lpr=498 pi=[492,498)/1 crt=166'68409 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.733896 7ffab5be4700 1 osd.10 pg_epoch: 498 pg[6.0( v 430'122181 (430'120681,430'122181] local-lis/les=492/493 n=25944 ec=76/76 lis/c 492/492 les/c/f 493/493/0 498/498/498) [10,7,2] r=0 lpr=498 pi=[492,498)/1 crt=430'122181 lcod 0'0 mlcod 0'0 unknown NOTIFY] start_peering_interval up [7,2] -> [10,7,2], acting [7,2] -> [10,7,2], acting_primary 7 -> 10, up_primary 7 -> 10, role -1 -> 0, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.734017 7ffab5be4700 1 osd.10 pg_epoch: 498 pg[6.0( v 430'122181 (430'120681,430'122181] local-lis/les=492/493 n=25944 ec=76/76 lis/c 492/492 les/c/f 493/493/0 498/498/498) [10,7,2] r=0 lpr=498 pi=[492,498)/1 crt=430'122181 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 07:38:22.734082 7ffab63e5700 1 osd.10 pg_epoch: 495 pg[5.35( v 166'67898 (166'66348,166'67898] local-lis/les=492/493 n=67898 ec=71/71 lis/c 492/492 les/c/f 493/493/0 495/495/470) [0,11] r=-1 lpr=495 pi=[492,495)/1 crt=166'67898 lcod 0'0 unknown NOTIFY] start_peering_interval up [0,11,10] -> [0,11], acting [0,11,10] -> [0,11], acting_primary 0 -> 0, up_primary 0 -> 0, role 2 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.734205 7ffab63e5700 1 osd.10 pg_epoch: 498 pg[5.35( v 166'67898 (166'66348,166'67898] local-lis/les=492/493 n=67898 ec=71/71 lis/c 492/492 les/c/f 493/493/0 498/498/470) [0,11,10] r=2 lpr=498 pi=[492,498)/1 crt=166'67898 lcod 0'0 unknown NOTIFY] start_peering_interval up [0,11] -> [0,11,10], acting [0,11] -> [0,11,10], acting_primary 0 -> 0, up_primary 0 -> 0, role -1 -> 2, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.734229 7ffab5be4700 1 osd.10 pg_epoch: 495 pg[5.10( v 166'68154 (166'66598,166'68154] local-lis/les=492/493 n=68154 ec=71/71 lis/c 492/492 les/c/f 493/493/0 492/492/492) [10,8,4] r=0 lpr=493 crt=166'68154 lcod 0'0 mlcod 0'0 peering] state<Started/Primary/Peering>: Peering, affected_by_map, going to Reset >2018-07-11 07:38:22.734338 7ffab63e5700 1 osd.10 pg_epoch: 498 pg[5.35( v 166'67898 (166'66348,166'67898] local-lis/les=492/493 n=67898 ec=71/71 lis/c 492/492 les/c/f 493/493/0 498/498/470) [0,11,10] r=2 lpr=498 pi=[492,498)/1 crt=166'67898 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.734409 7ffab5be4700 1 osd.10 pg_epoch: 495 pg[5.10( v 166'68154 (166'66598,166'68154] local-lis/les=492/493 n=68154 ec=71/71 lis/c 492/492 les/c/f 493/493/0 495/495/495) [8,4] r=-1 lpr=495 pi=[492,495)/1 crt=166'68154 lcod 0'0 unknown] start_peering_interval up [10,8,4] -> [8,4], acting [10,8,4] -> [8,4], acting_primary 10 -> 8, up_primary 10 -> 8, role 0 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.734529 7ffab5be4700 1 osd.10 pg_epoch: 498 pg[5.10( v 166'68154 (166'66598,166'68154] local-lis/les=492/493 n=68154 ec=71/71 lis/c 492/492 les/c/f 493/493/0 498/498/498) [10,8,4] r=0 lpr=498 pi=[492,498)/1 crt=166'68154 lcod 0'0 mlcod 0'0 unknown NOTIFY] start_peering_interval up [8,4] -> [10,8,4], acting [8,4] -> [10,8,4], acting_primary 8 -> 10, up_primary 8 -> 10, role -1 -> 0, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.734564 7ffab63e5700 1 osd.10 pg_epoch: 495 pg[5.c( v 166'68190 (166'66593,166'68190] local-lis/les=492/493 n=68190 ec=71/71 lis/c 492/492 les/c/f 493/493/0 495/495/470) [1,4] r=-1 lpr=495 pi=[492,495)/1 crt=166'68190 lcod 0'0 unknown NOTIFY] start_peering_interval up [1,10,4] -> [1,4], acting [1,10,4] -> [1,4], acting_primary 1 -> 1, up_primary 1 -> 1, role 1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.734647 7ffab5be4700 1 osd.10 pg_epoch: 498 pg[5.10( v 166'68154 (166'66598,166'68154] local-lis/les=492/493 n=68154 ec=71/71 lis/c 492/492 les/c/f 493/493/0 498/498/498) [10,8,4] r=0 lpr=498 pi=[492,498)/1 crt=166'68154 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 07:38:22.734686 7ffab63e5700 1 osd.10 pg_epoch: 498 pg[5.c( v 166'68190 (166'66593,166'68190] local-lis/les=492/493 n=68190 ec=71/71 lis/c 492/492 les/c/f 493/493/0 498/498/470) [1,10,4] r=1 lpr=498 pi=[492,498)/1 crt=166'68190 lcod 0'0 unknown NOTIFY] start_peering_interval up [1,4] -> [1,10,4], acting [1,4] -> [1,10,4], acting_primary 1 -> 1, up_primary 1 -> 1, role -1 -> 1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.734784 7ffab63e5700 1 osd.10 pg_epoch: 498 pg[5.c( v 166'68190 (166'66593,166'68190] local-lis/les=492/493 n=68190 ec=71/71 lis/c 492/492 les/c/f 493/493/0 498/498/470) [1,10,4] r=1 lpr=498 pi=[492,498)/1 crt=166'68190 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.734889 7ffab5be4700 1 osd.10 pg_epoch: 495 pg[11.4s2( v 383'60130 (352'56449,383'60130] local-lis/les=492/493 n=60130 ec=267/267 lis/c 492/492 les/c/f 493/493/0 495/495/470) [0,6,2147483647,8,3,11]p0(0) r=-1 lpr=495 pi=[492,495)/1 crt=383'60130 lcod 0'0 unknown NOTIFY] start_peering_interval up [0,6,10,8,3,11] -> [0,6,2147483647,8,3,11], acting [0,6,10,8,3,11] -> [0,6,2147483647,8,3,11], acting_primary 0(0) -> 0, up_primary 0(0) -> 0, role 2 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.734920 7ffab63e5700 1 osd.10 pg_epoch: 495 pg[16.e( empty local-lis/les=492/493 n=0 ec=442/442 lis/c 492/492 les/c/f 493/493/0 495/495/465) [2,4] r=-1 lpr=495 pi=[492,495)/1 crt=0'0 unknown NOTIFY] start_peering_interval up [2,4,10] -> [2,4], acting [2,4,10] -> [2,4], acting_primary 2 -> 2, up_primary 2 -> 2, role 2 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.734997 7ffab63e5700 1 osd.10 pg_epoch: 498 pg[16.e( empty local-lis/les=492/493 n=0 ec=442/442 lis/c 492/492 les/c/f 493/493/0 498/498/465) [2,4,10] r=2 lpr=498 pi=[492,498)/1 crt=0'0 unknown NOTIFY] start_peering_interval up [2,4] -> [2,4,10], acting [2,4] -> [2,4,10], acting_primary 2 -> 2, up_primary 2 -> 2, role -1 -> 2, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.735019 7ffab5be4700 1 osd.10 pg_epoch: 498 pg[11.4s2( v 383'60130 (352'56449,383'60130] local-lis/les=492/493 n=60130 ec=267/267 lis/c 492/492 les/c/f 493/493/0 498/498/470) [0,6,10,8,3,11]p0(0) r=2 lpr=498 pi=[492,498)/1 crt=383'60130 lcod 0'0 unknown NOTIFY] start_peering_interval up [0,6,2147483647,8,3,11] -> [0,6,10,8,3,11], acting [0,6,2147483647,8,3,11] -> [0,6,10,8,3,11], acting_primary 0(0) -> 0, up_primary 0(0) -> 0, role -1 -> 2, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.735066 7ffab63e5700 1 osd.10 pg_epoch: 498 pg[16.e( empty local-lis/les=492/493 n=0 ec=442/442 lis/c 492/492 les/c/f 493/493/0 498/498/465) [2,4,10] r=2 lpr=498 pi=[492,498)/1 crt=0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.735100 7ffab5be4700 1 osd.10 pg_epoch: 498 pg[11.4s2( v 383'60130 (352'56449,383'60130] local-lis/les=492/493 n=60130 ec=267/267 lis/c 492/492 les/c/f 493/493/0 498/498/470) [0,6,10,8,3,11]p0(0) r=2 lpr=498 pi=[492,498)/1 crt=383'60130 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.735156 7ffab63e5700 1 osd.10 pg_epoch: 495 pg[5.24( v 166'68298 (166'66798,166'68298] local-lis/les=492/493 n=68298 ec=71/71 lis/c 492/492 les/c/f 493/493/0 492/492/492) [10,11,1] r=0 lpr=493 crt=166'68298 lcod 0'0 mlcod 0'0 peering] state<Started/Primary/Peering>: Peering, affected_by_map, going to Reset >2018-07-11 07:38:22.735272 7ffab63e5700 1 osd.10 pg_epoch: 495 pg[5.24( v 166'68298 (166'66798,166'68298] local-lis/les=492/493 n=68298 ec=71/71 lis/c 492/492 les/c/f 493/493/0 495/495/495) [11,1] r=-1 lpr=495 pi=[492,495)/1 crt=166'68298 lcod 0'0 unknown] start_peering_interval up [10,11,1] -> [11,1], acting [10,11,1] -> [11,1], acting_primary 10 -> 11, up_primary 10 -> 11, role 0 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.735289 7ffab5be4700 1 osd.10 pg_epoch: 495 pg[13.1s1( v 377'9368 (355'1456,377'9368] local-lis/les=492/493 n=2661 ec=345/345 lis/c 492/492 les/c/f 493/493/0 495/495/472) [8,2147483647,0,2,6,7]p8(0) r=-1 lpr=495 pi=[492,495)/1 crt=377'9368 lcod 0'0 unknown NOTIFY] start_peering_interval up [8,10,0,2,6,7] -> [8,2147483647,0,2,6,7], acting [8,10,0,2,6,7] -> [8,2147483647,0,2,6,7], acting_primary 8(0) -> 8, up_primary 8(0) -> 8, role 1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.735349 7ffab63e5700 1 osd.10 pg_epoch: 498 pg[5.24( v 166'68298 (166'66798,166'68298] local-lis/les=492/493 n=68298 ec=71/71 lis/c 492/492 les/c/f 493/493/0 498/498/498) [10,11,1] r=0 lpr=498 pi=[492,498)/1 crt=166'68298 lcod 0'0 mlcod 0'0 unknown NOTIFY] start_peering_interval up [11,1] -> [10,11,1], acting [11,1] -> [10,11,1], acting_primary 11 -> 10, up_primary 11 -> 10, role -1 -> 0, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.735403 7ffab5be4700 1 osd.10 pg_epoch: 498 pg[13.1s1( v 377'9368 (355'1456,377'9368] local-lis/les=492/493 n=2661 ec=345/345 lis/c 492/492 les/c/f 493/493/0 498/498/472) [8,10,0,2,6,7]p8(0) r=1 lpr=498 pi=[492,498)/1 crt=377'9368 lcod 0'0 unknown NOTIFY] start_peering_interval up [8,2147483647,0,2,6,7] -> [8,10,0,2,6,7], acting [8,2147483647,0,2,6,7] -> [8,10,0,2,6,7], acting_primary 8(0) -> 8, up_primary 8(0) -> 8, role -1 -> 1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.735427 7ffab63e5700 1 osd.10 pg_epoch: 498 pg[5.24( v 166'68298 (166'66798,166'68298] local-lis/les=492/493 n=68298 ec=71/71 lis/c 492/492 les/c/f 493/493/0 498/498/498) [10,11,1] r=0 lpr=498 pi=[492,498)/1 crt=166'68298 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 07:38:22.735484 7ffab5be4700 1 osd.10 pg_epoch: 498 pg[13.1s1( v 377'9368 (355'1456,377'9368] local-lis/les=492/493 n=2661 ec=345/345 lis/c 492/492 les/c/f 493/493/0 498/498/472) [8,10,0,2,6,7]p8(0) r=1 lpr=498 pi=[492,498)/1 crt=377'9368 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.735635 7ffab63e5700 1 osd.10 pg_epoch: 495 pg[13.ds1( v 377'10591 (355'1551,377'10591] local-lis/les=492/493 n=2744 ec=345/345 lis/c 492/492 les/c/f 493/493/0 495/495/470) [0,2147483647,1,8,11,6]p0(0) r=-1 lpr=495 pi=[492,495)/1 crt=377'10591 lcod 0'0 unknown NOTIFY] start_peering_interval up [0,10,1,8,11,6] -> [0,2147483647,1,8,11,6], acting [0,10,1,8,11,6] -> [0,2147483647,1,8,11,6], acting_primary 0(0) -> 0, up_primary 0(0) -> 0, role 1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.735651 7ffab5be4700 1 osd.10 pg_epoch: 495 pg[11.3s5( v 383'60232 (352'56579,383'60232] local-lis/les=492/493 n=60232 ec=267/267 lis/c 492/492 les/c/f 493/493/0 495/495/481) [6,8,0,11,2,2147483647]p6(0) r=-1 lpr=495 pi=[492,495)/1 crt=383'60232 lcod 0'0 unknown NOTIFY] start_peering_interval up [6,8,0,11,2,10] -> [6,8,0,11,2,2147483647], acting [6,8,0,11,2,10] -> [6,8,0,11,2,2147483647], acting_primary 6(0) -> 6, up_primary 6(0) -> 6, role 5 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.735734 7ffab63e5700 1 osd.10 pg_epoch: 498 pg[13.ds1( v 377'10591 (355'1551,377'10591] local-lis/les=492/493 n=2744 ec=345/345 lis/c 492/492 les/c/f 493/493/0 498/498/470) [0,10,1,8,11,6]p0(0) r=1 lpr=498 pi=[492,498)/1 crt=377'10591 lcod 0'0 unknown NOTIFY] start_peering_interval up [0,2147483647,1,8,11,6] -> [0,10,1,8,11,6], acting [0,2147483647,1,8,11,6] -> [0,10,1,8,11,6], acting_primary 0(0) -> 0, up_primary 0(0) -> 0, role -1 -> 1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.735748 7ffab5be4700 1 osd.10 pg_epoch: 498 pg[11.3s5( v 383'60232 (352'56579,383'60232] local-lis/les=492/493 n=60232 ec=267/267 lis/c 492/492 les/c/f 493/493/0 498/498/481) [6,8,0,11,2,10]p6(0) r=5 lpr=498 pi=[492,498)/1 crt=383'60232 lcod 0'0 unknown NOTIFY] start_peering_interval up [6,8,0,11,2,2147483647] -> [6,8,0,11,2,10], acting [6,8,0,11,2,2147483647] -> [6,8,0,11,2,10], acting_primary 6(0) -> 6, up_primary 6(0) -> 6, role -1 -> 5, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.735818 7ffab63e5700 1 osd.10 pg_epoch: 498 pg[13.ds1( v 377'10591 (355'1551,377'10591] local-lis/les=492/493 n=2744 ec=345/345 lis/c 492/492 les/c/f 493/493/0 498/498/470) [0,10,1,8,11,6]p0(0) r=1 lpr=498 pi=[492,498)/1 crt=377'10591 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.735830 7ffab5be4700 1 osd.10 pg_epoch: 498 pg[11.3s5( v 383'60232 (352'56579,383'60232] local-lis/les=492/493 n=60232 ec=267/267 lis/c 492/492 les/c/f 493/493/0 498/498/481) [6,8,0,11,2,10]p6(0) r=5 lpr=498 pi=[492,498)/1 crt=383'60232 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.735921 7ffab5be4700 1 osd.10 pg_epoch: 495 pg[11.6s0( v 383'59732 (352'56233,383'59732] local-lis/les=492/493 n=59732 ec=267/267 lis/c 492/492 les/c/f 493/493/0 492/492/492) [10,1,11,6,5,8]p10(0) r=0 lpr=493 crt=383'59732 lcod 0'0 mlcod 0'0 peering] state<Started/Primary/Peering>: Peering, affected_by_map, going to Reset >2018-07-11 07:38:22.735985 7ffab63e5700 1 osd.10 pg_epoch: 495 pg[13.cs2( v 377'10679 (371'2117,377'10679] local-lis/les=492/493 n=2785 ec=345/345 lis/c 492/492 les/c/f 493/493/0 495/495/470) [1,11,2147483647,4,8,6]p1(0) r=-1 lpr=495 pi=[492,495)/1 crt=377'10679 lcod 0'0 unknown NOTIFY] start_peering_interval up [1,11,10,4,8,6] -> [1,11,2147483647,4,8,6], acting [1,11,10,4,8,6] -> [1,11,2147483647,4,8,6], acting_primary 1(0) -> 1, up_primary 1(0) -> 1, role 2 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.736040 7ffab5be4700 1 osd.10 pg_epoch: 495 pg[11.6s0( v 383'59732 (352'56233,383'59732] local-lis/les=492/493 n=59732 ec=267/267 lis/c 492/492 les/c/f 493/493/0 495/495/495) [2147483647,1,11,6,5,8]p1(1) r=-1 lpr=495 pi=[492,495)/1 crt=383'59732 lcod 0'0 unknown] start_peering_interval up [10,1,11,6,5,8] -> [2147483647,1,11,6,5,8], acting [10,1,11,6,5,8] -> [2147483647,1,11,6,5,8], acting_primary 10(0) -> 1, up_primary 10(0) -> 1, role 0 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.736072 7ffab63e5700 1 osd.10 pg_epoch: 498 pg[13.cs2( v 377'10679 (371'2117,377'10679] local-lis/les=492/493 n=2785 ec=345/345 lis/c 492/492 les/c/f 493/493/0 498/498/470) [1,11,10,4,8,6]p1(0) r=2 lpr=498 pi=[492,498)/1 crt=377'10679 lcod 0'0 unknown NOTIFY] start_peering_interval up [1,11,2147483647,4,8,6] -> [1,11,10,4,8,6], acting [1,11,2147483647,4,8,6] -> [1,11,10,4,8,6], acting_primary 1(0) -> 1, up_primary 1(0) -> 1, role -1 -> 2, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.736138 7ffab5be4700 1 osd.10 pg_epoch: 498 pg[11.6s0( v 383'59732 (352'56233,383'59732] local-lis/les=492/493 n=59732 ec=267/267 lis/c 492/492 les/c/f 493/493/0 498/498/498) [10,1,11,6,5,8]p10(0) r=0 lpr=498 pi=[492,498)/1 crt=383'59732 lcod 0'0 mlcod 0'0 unknown NOTIFY] start_peering_interval up [2147483647,1,11,6,5,8] -> [10,1,11,6,5,8], acting [2147483647,1,11,6,5,8] -> [10,1,11,6,5,8], acting_primary 1(1) -> 10, up_primary 1(1) -> 10, role -1 -> 0, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.736147 7ffab63e5700 1 osd.10 pg_epoch: 498 pg[13.cs2( v 377'10679 (371'2117,377'10679] local-lis/les=492/493 n=2785 ec=345/345 lis/c 492/492 les/c/f 493/493/0 498/498/470) [1,11,10,4,8,6]p1(0) r=2 lpr=498 pi=[492,498)/1 crt=377'10679 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.736215 7ffab5be4700 1 osd.10 pg_epoch: 498 pg[11.6s0( v 383'59732 (352'56233,383'59732] local-lis/les=492/493 n=59732 ec=267/267 lis/c 492/492 les/c/f 493/493/0 498/498/498) [10,1,11,6,5,8]p10(0) r=0 lpr=498 pi=[492,498)/1 crt=383'59732 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 07:38:22.736296 7ffab63e5700 1 osd.10 pg_epoch: 495 pg[16.10( empty local-lis/les=492/493 n=0 ec=442/442 lis/c 492/492 les/c/f 493/493/0 495/495/465) [2,7] r=-1 lpr=495 pi=[492,495)/1 crt=0'0 unknown NOTIFY] start_peering_interval up [2,10,7] -> [2,7], acting [2,10,7] -> [2,7], acting_primary 2 -> 2, up_primary 2 -> 2, role 1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.736355 7ffab5be4700 1 osd.10 pg_epoch: 495 pg[6.13( v 430'99276 (430'97776,430'99276] local-lis/les=492/493 n=26101 ec=76/76 lis/c 492/492 les/c/f 493/493/0 492/492/492) [10,7,0] r=0 lpr=493 crt=430'99276 lcod 0'0 mlcod 0'0 peering] state<Started/Primary/Peering>: Peering, affected_by_map, going to Reset >2018-07-11 07:38:22.736360 7ffab63e5700 1 osd.10 pg_epoch: 498 pg[16.10( empty local-lis/les=492/493 n=0 ec=442/442 lis/c 492/492 les/c/f 493/493/0 498/498/465) [2,10,7] r=1 lpr=498 pi=[492,498)/1 crt=0'0 unknown NOTIFY] start_peering_interval up [2,7] -> [2,10,7], acting [2,7] -> [2,10,7], acting_primary 2 -> 2, up_primary 2 -> 2, role -1 -> 1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.736423 7ffab63e5700 1 osd.10 pg_epoch: 498 pg[16.10( empty local-lis/les=492/493 n=0 ec=442/442 lis/c 492/492 les/c/f 493/493/0 498/498/465) [2,10,7] r=1 lpr=498 pi=[492,498)/1 crt=0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.736449 7ffab5be4700 1 osd.10 pg_epoch: 495 pg[6.13( v 430'99276 (430'97776,430'99276] local-lis/les=492/493 n=26101 ec=76/76 lis/c 492/492 les/c/f 493/493/0 495/495/495) [7,0] r=-1 lpr=495 pi=[492,495)/1 crt=430'99276 lcod 0'0 unknown] start_peering_interval up [10,7,0] -> [7,0], acting [10,7,0] -> [7,0], acting_primary 10 -> 7, up_primary 10 -> 7, role 0 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.736507 7ffab63e5700 1 osd.10 pg_epoch: 495 pg[10.a( v 386'2032 (203'500,386'2032] local-lis/les=492/493 n=516 ec=193/193 lis/c 492/492 les/c/f 493/493/0 492/492/492) [10,2,5] r=0 lpr=493 crt=386'2032 lcod 0'0 mlcod 0'0 peering] state<Started/Primary/Peering>: Peering, affected_by_map, going to Reset >2018-07-11 07:38:22.736518 7ffab5be4700 1 osd.10 pg_epoch: 498 pg[6.13( v 430'99276 (430'97776,430'99276] local-lis/les=492/493 n=26101 ec=76/76 lis/c 492/492 les/c/f 493/493/0 498/498/498) [10,7,0] r=0 lpr=498 pi=[492,498)/1 crt=430'99276 lcod 0'0 mlcod 0'0 unknown NOTIFY] start_peering_interval up [7,0] -> [10,7,0], acting [7,0] -> [10,7,0], acting_primary 7 -> 10, up_primary 7 -> 10, role -1 -> 0, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.736598 7ffab5be4700 1 osd.10 pg_epoch: 498 pg[6.13( v 430'99276 (430'97776,430'99276] local-lis/les=492/493 n=26101 ec=76/76 lis/c 492/492 les/c/f 493/493/0 498/498/498) [10,7,0] r=0 lpr=498 pi=[492,498)/1 crt=430'99276 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 07:38:22.736613 7ffab63e5700 1 osd.10 pg_epoch: 495 pg[10.a( v 386'2032 (203'500,386'2032] local-lis/les=492/493 n=516 ec=193/193 lis/c 492/492 les/c/f 493/493/0 495/495/495) [2,5] r=-1 lpr=495 pi=[492,495)/1 crt=386'2032 lcod 0'0 unknown] start_peering_interval up [10,2,5] -> [2,5], acting [10,2,5] -> [2,5], acting_primary 10 -> 2, up_primary 10 -> 2, role 0 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.736680 7ffab63e5700 1 osd.10 pg_epoch: 498 pg[10.a( v 386'2032 (203'500,386'2032] local-lis/les=492/493 n=516 ec=193/193 lis/c 492/492 les/c/f 493/493/0 498/498/498) [10,2,5] r=0 lpr=498 pi=[492,498)/1 crt=386'2032 lcod 0'0 mlcod 0'0 unknown NOTIFY] start_peering_interval up [2,5] -> [10,2,5], acting [2,5] -> [10,2,5], acting_primary 2 -> 10, up_primary 2 -> 10, role -1 -> 0, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.736724 7ffab5be4700 1 osd.10 pg_epoch: 495 pg[9.1c( v 494'483 (0'0,494'483] local-lis/les=492/493 n=238 ec=190/190 lis/c 492/492 les/c/f 493/493/0 495/495/470) [1,7] r=-1 lpr=495 pi=[492,495)/1 crt=494'483 lcod 0'0 unknown NOTIFY] start_peering_interval up [1,10,7] -> [1,7], acting [1,10,7] -> [1,7], acting_primary 1 -> 1, up_primary 1 -> 1, role 1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.736750 7ffab63e5700 1 osd.10 pg_epoch: 498 pg[10.a( v 386'2032 (203'500,386'2032] local-lis/les=492/493 n=516 ec=193/193 lis/c 492/492 les/c/f 493/493/0 498/498/498) [10,2,5] r=0 lpr=498 pi=[492,498)/1 crt=386'2032 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 07:38:22.736788 7ffab5be4700 1 osd.10 pg_epoch: 498 pg[9.1c( v 494'483 (0'0,494'483] local-lis/les=492/493 n=238 ec=190/190 lis/c 492/492 les/c/f 493/493/0 498/498/470) [1,10,7] r=1 lpr=498 pi=[492,498)/1 crt=494'483 lcod 0'0 unknown NOTIFY] start_peering_interval up [1,7] -> [1,10,7], acting [1,7] -> [1,10,7], acting_primary 1 -> 1, up_primary 1 -> 1, role -1 -> 1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.736861 7ffab5be4700 1 osd.10 pg_epoch: 498 pg[9.1c( v 494'483 (0'0,494'483] local-lis/les=492/493 n=238 ec=190/190 lis/c 492/492 les/c/f 493/493/0 498/498/470) [1,10,7] r=1 lpr=498 pi=[492,498)/1 crt=494'483 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.736923 7ffab63e5700 1 osd.10 pg_epoch: 495 pg[5.23( v 166'68497 (166'66949,166'68497] local-lis/les=492/493 n=68497 ec=71/71 lis/c 492/492 les/c/f 493/493/0 495/495/472) [8,3] r=-1 lpr=495 pi=[492,495)/1 crt=166'68497 lcod 0'0 unknown NOTIFY] start_peering_interval up [8,3,10] -> [8,3], acting [8,3,10] -> [8,3], acting_primary 8 -> 8, up_primary 8 -> 8, role 2 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.736943 7ffab5be4700 1 osd.10 pg_epoch: 495 pg[9.1b( v 463'472 (0'0,463'472] local-lis/les=492/493 n=256 ec=190/190 lis/c 492/492 les/c/f 493/493/0 492/492/492) [10,1,5] r=0 lpr=493 crt=463'472 lcod 0'0 mlcod 0'0 peering] state<Started/Primary/Peering>: Peering, affected_by_map, going to Reset >2018-07-11 07:38:22.736988 7ffab63e5700 1 osd.10 pg_epoch: 498 pg[5.23( v 166'68497 (166'66949,166'68497] local-lis/les=492/493 n=68497 ec=71/71 lis/c 492/492 les/c/f 493/493/0 498/498/472) [8,3,10] r=2 lpr=498 pi=[492,498)/1 crt=166'68497 lcod 0'0 unknown NOTIFY] start_peering_interval up [8,3] -> [8,3,10], acting [8,3] -> [8,3,10], acting_primary 8 -> 8, up_primary 8 -> 8, role -1 -> 2, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.737035 7ffab5be4700 1 osd.10 pg_epoch: 495 pg[9.1b( v 463'472 (0'0,463'472] local-lis/les=492/493 n=256 ec=190/190 lis/c 492/492 les/c/f 493/493/0 495/495/495) [1,5] r=-1 lpr=495 pi=[492,495)/1 crt=463'472 lcod 0'0 unknown] start_peering_interval up [10,1,5] -> [1,5], acting [10,1,5] -> [1,5], acting_primary 10 -> 1, up_primary 10 -> 1, role 0 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.737057 7ffab63e5700 1 osd.10 pg_epoch: 498 pg[5.23( v 166'68497 (166'66949,166'68497] local-lis/les=492/493 n=68497 ec=71/71 lis/c 492/492 les/c/f 493/493/0 498/498/472) [8,3,10] r=2 lpr=498 pi=[492,498)/1 crt=166'68497 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.737094 7ffab5be4700 1 osd.10 pg_epoch: 498 pg[9.1b( v 463'472 (0'0,463'472] local-lis/les=492/493 n=256 ec=190/190 lis/c 492/492 les/c/f 493/493/0 498/498/498) [10,1,5] r=0 lpr=498 pi=[492,498)/1 crt=463'472 lcod 0'0 mlcod 0'0 unknown NOTIFY] start_peering_interval up [1,5] -> [10,1,5], acting [1,5] -> [10,1,5], acting_primary 1 -> 10, up_primary 1 -> 10, role -1 -> 0, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.737165 7ffab5be4700 1 osd.10 pg_epoch: 498 pg[9.1b( v 463'472 (0'0,463'472] local-lis/les=492/493 n=256 ec=190/190 lis/c 492/492 les/c/f 493/493/0 498/498/498) [10,1,5] r=0 lpr=498 pi=[492,498)/1 crt=463'472 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 07:38:22.737203 7ffab63e5700 1 osd.10 pg_epoch: 495 pg[10.f( v 383'568 (0'0,383'568] local-lis/les=492/493 n=494 ec=193/193 lis/c 492/492 les/c/f 493/493/0 495/495/478) [11,2] r=-1 lpr=495 pi=[492,495)/1 crt=383'568 lcod 0'0 unknown NOTIFY] start_peering_interval up [11,10,2] -> [11,2], acting [11,10,2] -> [11,2], acting_primary 11 -> 11, up_primary 11 -> 11, role 1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.737280 7ffab63e5700 1 osd.10 pg_epoch: 498 pg[10.f( v 383'568 (0'0,383'568] local-lis/les=492/493 n=494 ec=193/193 lis/c 492/492 les/c/f 493/493/0 498/498/478) [11,10,2] r=1 lpr=498 pi=[492,498)/1 crt=383'568 lcod 0'0 unknown NOTIFY] start_peering_interval up [11,2] -> [11,10,2], acting [11,2] -> [11,10,2], acting_primary 11 -> 11, up_primary 11 -> 11, role -1 -> 1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.737293 7ffab5be4700 1 osd.10 pg_epoch: 495 pg[6.14( v 461'122884 (430'121291,461'122884] local-lis/les=492/493 n=25983 ec=76/76 lis/c 492/492 les/c/f 493/493/0 492/492/492) [10,4,3] r=0 lpr=493 crt=461'122884 lcod 0'0 mlcod 0'0 peering] state<Started/Primary/Peering>: Peering, affected_by_map, going to Reset >2018-07-11 07:38:22.737348 7ffab63e5700 1 osd.10 pg_epoch: 498 pg[10.f( v 383'568 (0'0,383'568] local-lis/les=492/493 n=494 ec=193/193 lis/c 492/492 les/c/f 493/493/0 498/498/478) [11,10,2] r=1 lpr=498 pi=[492,498)/1 crt=383'568 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.737388 7ffab5be4700 1 osd.10 pg_epoch: 495 pg[6.14( v 461'122884 (430'121291,461'122884] local-lis/les=492/493 n=25983 ec=76/76 lis/c 492/492 les/c/f 493/493/0 495/495/495) [4,3] r=-1 lpr=495 pi=[492,495)/1 crt=461'122884 lcod 0'0 unknown] start_peering_interval up [10,4,3] -> [4,3], acting [10,4,3] -> [4,3], acting_primary 10 -> 4, up_primary 10 -> 4, role 0 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.737454 7ffab5be4700 1 osd.10 pg_epoch: 498 pg[6.14( v 461'122884 (430'121291,461'122884] local-lis/les=492/493 n=25983 ec=76/76 lis/c 492/492 les/c/f 493/493/0 498/498/498) [10,4,3] r=0 lpr=498 pi=[492,498)/1 crt=461'122884 lcod 0'0 mlcod 0'0 unknown NOTIFY] start_peering_interval up [4,3] -> [10,4,3], acting [4,3] -> [10,4,3], acting_primary 4 -> 10, up_primary 4 -> 10, role -1 -> 0, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.737492 7ffab63e5700 1 osd.10 pg_epoch: 495 pg[6.3( v 430'104329 (430'102829,430'104329] local-lis/les=492/493 n=25881 ec=76/76 lis/c 492/492 les/c/f 493/493/0 495/495/470) [0,11] r=-1 lpr=495 pi=[492,495)/1 crt=430'104329 lcod 0'0 unknown NOTIFY] start_peering_interval up [0,10,11] -> [0,11], acting [0,10,11] -> [0,11], acting_primary 0 -> 0, up_primary 0 -> 0, role 1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.737529 7ffab5be4700 1 osd.10 pg_epoch: 498 pg[6.14( v 461'122884 (430'121291,461'122884] local-lis/les=492/493 n=25983 ec=76/76 lis/c 492/492 les/c/f 493/493/0 498/498/498) [10,4,3] r=0 lpr=498 pi=[492,498)/1 crt=461'122884 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 07:38:22.737561 7ffab63e5700 1 osd.10 pg_epoch: 498 pg[6.3( v 430'104329 (430'102829,430'104329] local-lis/les=492/493 n=25881 ec=76/76 lis/c 492/492 les/c/f 493/493/0 498/498/470) [0,10,11] r=1 lpr=498 pi=[492,498)/1 crt=430'104329 lcod 0'0 unknown NOTIFY] start_peering_interval up [0,11] -> [0,10,11], acting [0,11] -> [0,10,11], acting_primary 0 -> 0, up_primary 0 -> 0, role -1 -> 1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.737629 7ffab63e5700 1 osd.10 pg_epoch: 498 pg[6.3( v 430'104329 (430'102829,430'104329] local-lis/les=492/493 n=25881 ec=76/76 lis/c 492/492 les/c/f 493/493/0 498/498/470) [0,10,11] r=1 lpr=498 pi=[492,498)/1 crt=430'104329 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.737691 7ffab5be4700 1 osd.10 pg_epoch: 495 pg[5.14( v 166'68424 (166'66924,166'68424] local-lis/les=492/493 n=68424 ec=71/71 lis/c 492/492 les/c/f 493/493/0 495/495/468) [7,2] r=-1 lpr=495 pi=[492,495)/1 crt=166'68424 lcod 0'0 unknown NOTIFY] start_peering_interval up [7,10,2] -> [7,2], acting [7,10,2] -> [7,2], acting_primary 7 -> 7, up_primary 7 -> 7, role 1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.737761 7ffab5be4700 1 osd.10 pg_epoch: 498 pg[5.14( v 166'68424 (166'66924,166'68424] local-lis/les=492/493 n=68424 ec=71/71 lis/c 492/492 les/c/f 493/493/0 498/498/468) [7,10,2] r=1 lpr=498 pi=[492,498)/1 crt=166'68424 lcod 0'0 unknown NOTIFY] start_peering_interval up [7,2] -> [7,10,2], acting [7,2] -> [7,10,2], acting_primary 7 -> 7, up_primary 7 -> 7, role -1 -> 1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.737863 7ffab5be4700 1 osd.10 pg_epoch: 498 pg[5.14( v 166'68424 (166'66924,166'68424] local-lis/les=492/493 n=68424 ec=71/71 lis/c 492/492 les/c/f 493/493/0 498/498/468) [7,10,2] r=1 lpr=498 pi=[492,498)/1 crt=166'68424 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.737969 7ffab63e5700 1 osd.10 pg_epoch: 495 pg[5.9( v 166'68275 (166'66775,166'68275] local-lis/les=492/493 n=68275 ec=71/71 lis/c 492/492 les/c/f 493/493/0 492/492/492) [10,8,7] r=0 lpr=493 crt=166'68275 lcod 0'0 mlcod 0'0 peering] state<Started/Primary/Peering>: Peering, affected_by_map, going to Reset >2018-07-11 07:38:22.737978 7ffab5be4700 1 osd.10 pg_epoch: 495 pg[6.17( v 430'108120 (430'106620,430'108120] local-lis/les=492/493 n=26407 ec=76/76 lis/c 492/492 les/c/f 493/493/0 492/492/492) [10,2,5] r=0 lpr=493 crt=430'108120 lcod 0'0 mlcod 0'0 peering] state<Started/Primary/Peering>: Peering, affected_by_map, going to Reset >2018-07-11 07:38:22.738076 7ffab5be4700 1 osd.10 pg_epoch: 495 pg[6.17( v 430'108120 (430'106620,430'108120] local-lis/les=492/493 n=26407 ec=76/76 lis/c 492/492 les/c/f 493/493/0 495/495/495) [2,5] r=-1 lpr=495 pi=[492,495)/1 crt=430'108120 lcod 0'0 unknown] start_peering_interval up [10,2,5] -> [2,5], acting [10,2,5] -> [2,5], acting_primary 10 -> 2, up_primary 10 -> 2, role 0 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.738150 7ffab5be4700 1 osd.10 pg_epoch: 498 pg[6.17( v 430'108120 (430'106620,430'108120] local-lis/les=492/493 n=26407 ec=76/76 lis/c 492/492 les/c/f 493/493/0 498/498/498) [10,2,5] r=0 lpr=498 pi=[492,498)/1 crt=430'108120 lcod 0'0 mlcod 0'0 unknown NOTIFY] start_peering_interval up [2,5] -> [10,2,5], acting [2,5] -> [10,2,5], acting_primary 2 -> 10, up_primary 2 -> 10, role -1 -> 0, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.738157 7ffab63e5700 1 osd.10 pg_epoch: 495 pg[5.9( v 166'68275 (166'66775,166'68275] local-lis/les=492/493 n=68275 ec=71/71 lis/c 492/492 les/c/f 493/493/0 495/495/495) [8,7] r=-1 lpr=495 pi=[492,495)/1 crt=166'68275 lcod 0'0 unknown] start_peering_interval up [10,8,7] -> [8,7], acting [10,8,7] -> [8,7], acting_primary 10 -> 8, up_primary 10 -> 8, role 0 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.738215 7ffab5be4700 1 osd.10 pg_epoch: 498 pg[6.17( v 430'108120 (430'106620,430'108120] local-lis/les=492/493 n=26407 ec=76/76 lis/c 492/492 les/c/f 493/493/0 498/498/498) [10,2,5] r=0 lpr=498 pi=[492,498)/1 crt=430'108120 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 07:38:22.738269 7ffab63e5700 1 osd.10 pg_epoch: 498 pg[5.9( v 166'68275 (166'66775,166'68275] local-lis/les=492/493 n=68275 ec=71/71 lis/c 492/492 les/c/f 493/493/0 498/498/498) [10,8,7] r=0 lpr=498 pi=[492,498)/1 crt=166'68275 lcod 0'0 mlcod 0'0 unknown NOTIFY] start_peering_interval up [8,7] -> [10,8,7], acting [8,7] -> [10,8,7], acting_primary 8 -> 10, up_primary 8 -> 10, role -1 -> 0, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.738380 7ffab63e5700 1 osd.10 pg_epoch: 498 pg[5.9( v 166'68275 (166'66775,166'68275] local-lis/les=492/493 n=68275 ec=71/71 lis/c 492/492 les/c/f 493/493/0 498/498/498) [10,8,7] r=0 lpr=498 pi=[492,498)/1 crt=166'68275 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 07:38:22.738406 7ffab5be4700 1 osd.10 pg_epoch: 495 pg[6.21( v 461'128646 (430'127048,461'128646] local-lis/les=492/493 n=25982 ec=76/76 lis/c 492/492 les/c/f 493/493/0 495/495/470) [0,2] r=-1 lpr=495 pi=[492,495)/1 crt=461'128646 lcod 0'0 unknown NOTIFY] start_peering_interval up [0,2,10] -> [0,2], acting [0,2,10] -> [0,2], acting_primary 0 -> 0, up_primary 0 -> 0, role 2 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.738470 7ffab5be4700 1 osd.10 pg_epoch: 498 pg[6.21( v 461'128646 (430'127048,461'128646] local-lis/les=492/493 n=25982 ec=76/76 lis/c 492/492 les/c/f 493/493/0 498/498/470) [0,2,10] r=2 lpr=498 pi=[492,498)/1 crt=461'128646 lcod 0'0 unknown NOTIFY] start_peering_interval up [0,2] -> [0,2,10], acting [0,2] -> [0,2,10], acting_primary 0 -> 0, up_primary 0 -> 0, role -1 -> 2, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.738539 7ffab5be4700 1 osd.10 pg_epoch: 498 pg[6.21( v 461'128646 (430'127048,461'128646] local-lis/les=492/493 n=25982 ec=76/76 lis/c 492/492 les/c/f 493/493/0 498/498/470) [0,2,10] r=2 lpr=498 pi=[492,498)/1 crt=461'128646 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.738658 7ffab63e5700 1 osd.10 pg_epoch: 495 pg[6.a( v 430'109231 (430'107731,430'109231] local-lis/les=492/493 n=26267 ec=76/76 lis/c 492/492 les/c/f 493/493/0 495/495/467) [5,6] r=-1 lpr=495 pi=[492,495)/1 crt=430'109231 lcod 0'0 unknown NOTIFY] start_peering_interval up [5,10,6] -> [5,6], acting [5,10,6] -> [5,6], acting_primary 5 -> 5, up_primary 5 -> 5, role 1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.738669 7ffab5be4700 1 osd.10 pg_epoch: 495 pg[10.9( v 390'615 (0'0,390'615] local-lis/les=492/493 n=478 ec=193/193 lis/c 492/492 les/c/f 493/493/0 495/495/470) [0,2] r=-1 lpr=495 pi=[492,495)/1 crt=390'615 lcod 0'0 unknown NOTIFY] start_peering_interval up [0,2,10] -> [0,2], acting [0,2,10] -> [0,2], acting_primary 0 -> 0, up_primary 0 -> 0, role 2 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.738746 7ffab5be4700 1 osd.10 pg_epoch: 498 pg[10.9( v 390'615 (0'0,390'615] local-lis/les=492/493 n=478 ec=193/193 lis/c 492/492 les/c/f 493/493/0 498/498/470) [0,2,10] r=2 lpr=498 pi=[492,498)/1 crt=390'615 lcod 0'0 unknown NOTIFY] start_peering_interval up [0,2] -> [0,2,10], acting [0,2] -> [0,2,10], acting_primary 0 -> 0, up_primary 0 -> 0, role -1 -> 2, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.738795 7ffab63e5700 1 osd.10 pg_epoch: 498 pg[6.a( v 430'109231 (430'107731,430'109231] local-lis/les=492/493 n=26267 ec=76/76 lis/c 492/492 les/c/f 493/493/0 498/498/467) [5,10,6] r=1 lpr=498 pi=[492,498)/1 crt=430'109231 lcod 0'0 unknown NOTIFY] start_peering_interval up [5,6] -> [5,10,6], acting [5,6] -> [5,10,6], acting_primary 5 -> 5, up_primary 5 -> 5, role -1 -> 1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.738830 7ffab5be4700 1 osd.10 pg_epoch: 498 pg[10.9( v 390'615 (0'0,390'615] local-lis/les=492/493 n=478 ec=193/193 lis/c 492/492 les/c/f 493/493/0 498/498/470) [0,2,10] r=2 lpr=498 pi=[492,498)/1 crt=390'615 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.738926 7ffab5be4700 1 osd.10 pg_epoch: 495 pg[9.a( v 494'572 (0'0,494'572] local-lis/les=492/493 n=232 ec=190/190 lis/c 492/492 les/c/f 493/493/0 495/495/470) [0,11] r=-1 lpr=495 pi=[492,495)/1 crt=494'572 lcod 0'0 unknown NOTIFY] start_peering_interval up [0,10,11] -> [0,11], acting [0,10,11] -> [0,11], acting_primary 0 -> 0, up_primary 0 -> 0, role 1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.738926 7ffab63e5700 1 osd.10 pg_epoch: 498 pg[6.a( v 430'109231 (430'107731,430'109231] local-lis/les=492/493 n=26267 ec=76/76 lis/c 492/492 les/c/f 493/493/0 498/498/467) [5,10,6] r=1 lpr=498 pi=[492,498)/1 crt=430'109231 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.738994 7ffab5be4700 1 osd.10 pg_epoch: 498 pg[9.a( v 494'572 (0'0,494'572] local-lis/les=492/493 n=232 ec=190/190 lis/c 492/492 les/c/f 493/493/0 498/498/470) [0,10,11] r=1 lpr=498 pi=[492,498)/1 crt=494'572 lcod 0'0 unknown NOTIFY] start_peering_interval up [0,11] -> [0,10,11], acting [0,11] -> [0,10,11], acting_primary 0 -> 0, up_primary 0 -> 0, role -1 -> 1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.739070 7ffab5be4700 1 osd.10 pg_epoch: 498 pg[9.a( v 494'572 (0'0,494'572] local-lis/les=492/493 n=232 ec=190/190 lis/c 492/492 les/c/f 493/493/0 498/498/470) [0,10,11] r=1 lpr=498 pi=[492,498)/1 crt=494'572 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.739158 7ffab5be4700 1 osd.10 pg_epoch: 495 pg[9.7( v 486'503 (0'0,486'503] local-lis/les=492/493 n=248 ec=190/190 lis/c 492/492 les/c/f 493/494/0 495/495/468) [7,2] r=-1 lpr=495 pi=[492,495)/1 crt=486'503 lcod 0'0 unknown NOTIFY] start_peering_interval up [7,10,2] -> [7,2], acting [7,10,2] -> [7,2], acting_primary 7 -> 7, up_primary 7 -> 7, role 1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.739169 7ffab63e5700 1 osd.10 pg_epoch: 495 pg[12.0( v 383'7619 (383'6093,383'7619] local-lis/les=492/493 n=7619 ec=331/331 lis/c 492/492 les/c/f 493/493/0 495/495/487) [4,3] r=-1 lpr=495 pi=[492,495)/1 crt=383'7619 lcod 0'0 unknown NOTIFY] start_peering_interval up [4,10,3] -> [4,3], acting [4,10,3] -> [4,3], acting_primary 4 -> 4, up_primary 4 -> 4, role 1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.739225 7ffab5be4700 1 osd.10 pg_epoch: 498 pg[9.7( v 486'503 (0'0,486'503] local-lis/les=492/493 n=248 ec=190/190 lis/c 492/492 les/c/f 493/494/0 498/498/468) [7,10,2] r=1 lpr=498 pi=[492,498)/1 crt=486'503 lcod 0'0 unknown NOTIFY] start_peering_interval up [7,2] -> [7,10,2], acting [7,2] -> [7,10,2], acting_primary 7 -> 7, up_primary 7 -> 7, role -1 -> 1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.739323 7ffab5be4700 1 osd.10 pg_epoch: 498 pg[9.7( v 486'503 (0'0,486'503] local-lis/les=492/493 n=248 ec=190/190 lis/c 492/492 les/c/f 493/494/0 498/498/468) [7,10,2] r=1 lpr=498 pi=[492,498)/1 crt=486'503 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.739312 7ffab63e5700 1 osd.10 pg_epoch: 498 pg[12.0( v 383'7619 (383'6093,383'7619] local-lis/les=492/493 n=7619 ec=331/331 lis/c 492/492 les/c/f 493/493/0 498/498/487) [4,10,3] r=1 lpr=498 pi=[492,498)/1 crt=383'7619 lcod 0'0 unknown NOTIFY] start_peering_interval up [4,3] -> [4,10,3], acting [4,3] -> [4,10,3], acting_primary 4 -> 4, up_primary 4 -> 4, role -1 -> 1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.739392 7ffab63e5700 1 osd.10 pg_epoch: 498 pg[12.0( v 383'7619 (383'6093,383'7619] local-lis/les=492/493 n=7619 ec=331/331 lis/c 492/492 les/c/f 493/493/0 498/498/487) [4,10,3] r=1 lpr=498 pi=[492,498)/1 crt=383'7619 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.739399 7ffab5be4700 1 osd.10 pg_epoch: 495 pg[5.1c( v 383'68804 (166'67302,383'68804] local-lis/les=492/493 n=68803 ec=71/71 lis/c 492/492 les/c/f 493/493/0 492/492/492) [10,8,2] r=0 lpr=493 crt=383'68804 lcod 0'0 mlcod 0'0 peering] state<Started/Primary/Peering>: Peering, affected_by_map, going to Reset >2018-07-11 07:38:22.739492 7ffab5be4700 1 osd.10 pg_epoch: 495 pg[5.1c( v 383'68804 (166'67302,383'68804] local-lis/les=492/493 n=68803 ec=71/71 lis/c 492/492 les/c/f 493/493/0 495/495/495) [8,2] r=-1 lpr=495 pi=[492,495)/1 crt=383'68804 lcod 0'0 unknown] start_peering_interval up [10,8,2] -> [8,2], acting [10,8,2] -> [8,2], acting_primary 10 -> 8, up_primary 10 -> 8, role 0 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.739554 7ffab5be4700 1 osd.10 pg_epoch: 498 pg[5.1c( v 383'68804 (166'67302,383'68804] local-lis/les=492/493 n=68803 ec=71/71 lis/c 492/492 les/c/f 493/493/0 498/498/498) [10,8,2] r=0 lpr=498 pi=[492,498)/1 crt=383'68804 lcod 0'0 mlcod 0'0 unknown NOTIFY] start_peering_interval up [8,2] -> [10,8,2], acting [8,2] -> [10,8,2], acting_primary 8 -> 10, up_primary 8 -> 10, role -1 -> 0, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.739565 7ffab63e5700 1 osd.10 pg_epoch: 495 pg[5.4( v 166'68216 (166'66716,166'68216] local-lis/les=492/493 n=68216 ec=71/71 lis/c 492/492 les/c/f 493/493/0 495/495/468) [7,2] r=-1 lpr=495 pi=[492,495)/1 crt=166'68216 lcod 0'0 unknown NOTIFY] start_peering_interval up [7,2,10] -> [7,2], acting [7,2,10] -> [7,2], acting_primary 7 -> 7, up_primary 7 -> 7, role 2 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.739623 7ffab5be4700 1 osd.10 pg_epoch: 498 pg[5.1c( v 383'68804 (166'67302,383'68804] local-lis/les=492/493 n=68803 ec=71/71 lis/c 492/492 les/c/f 493/493/0 498/498/498) [10,8,2] r=0 lpr=498 pi=[492,498)/1 crt=383'68804 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 07:38:22.739628 7ffab63e5700 1 osd.10 pg_epoch: 498 pg[5.4( v 166'68216 (166'66716,166'68216] local-lis/les=492/493 n=68216 ec=71/71 lis/c 492/492 les/c/f 493/493/0 498/498/468) [7,2,10] r=2 lpr=498 pi=[492,498)/1 crt=166'68216 lcod 0'0 unknown NOTIFY] start_peering_interval up [7,2] -> [7,2,10], acting [7,2] -> [7,2,10], acting_primary 7 -> 7, up_primary 7 -> 7, role -1 -> 2, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.739690 7ffab63e5700 1 osd.10 pg_epoch: 498 pg[5.4( v 166'68216 (166'66716,166'68216] local-lis/les=492/493 n=68216 ec=71/71 lis/c 492/492 les/c/f 493/493/0 498/498/468) [7,2,10] r=2 lpr=498 pi=[492,498)/1 crt=166'68216 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.739768 7ffab63e5700 1 osd.10 pg_epoch: 495 pg[6.7( v 430'109641 (430'108141,430'109641] local-lis/les=492/493 n=25825 ec=76/76 lis/c 492/492 les/c/f 493/493/0 492/492/492) [10,6,3] r=0 lpr=493 crt=430'109641 lcod 0'0 mlcod 0'0 peering] state<Started/Primary/Peering>: Peering, affected_by_map, going to Reset >2018-07-11 07:38:22.739785 7ffab5be4700 1 osd.10 pg_epoch: 495 pg[6.1f( v 430'114415 (430'112915,430'114415] local-lis/les=492/493 n=26413 ec=76/76 lis/c 492/492 les/c/f 493/493/0 495/495/484) [3,0] r=-1 lpr=495 pi=[492,495)/1 crt=430'114415 lcod 0'0 unknown NOTIFY] start_peering_interval up [3,10,0] -> [3,0], acting [3,10,0] -> [3,0], acting_primary 3 -> 3, up_primary 3 -> 3, role 1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.739847 7ffab5be4700 1 osd.10 pg_epoch: 498 pg[6.1f( v 430'114415 (430'112915,430'114415] local-lis/les=492/493 n=26413 ec=76/76 lis/c 492/492 les/c/f 493/493/0 498/498/484) [3,10,0] r=1 lpr=498 pi=[492,498)/1 crt=430'114415 lcod 0'0 unknown NOTIFY] start_peering_interval up [3,0] -> [3,10,0], acting [3,0] -> [3,10,0], acting_primary 3 -> 3, up_primary 3 -> 3, role -1 -> 1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.739865 7ffab63e5700 1 osd.10 pg_epoch: 495 pg[6.7( v 430'109641 (430'108141,430'109641] local-lis/les=492/493 n=25825 ec=76/76 lis/c 492/492 les/c/f 493/493/0 495/495/495) [6,3] r=-1 lpr=495 pi=[492,495)/1 crt=430'109641 lcod 0'0 unknown] start_peering_interval up [10,6,3] -> [6,3], acting [10,6,3] -> [6,3], acting_primary 10 -> 6, up_primary 10 -> 6, role 0 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.739912 7ffab5be4700 1 osd.10 pg_epoch: 498 pg[6.1f( v 430'114415 (430'112915,430'114415] local-lis/les=492/493 n=26413 ec=76/76 lis/c 492/492 les/c/f 493/493/0 498/498/484) [3,10,0] r=1 lpr=498 pi=[492,498)/1 crt=430'114415 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.739927 7ffab63e5700 1 osd.10 pg_epoch: 498 pg[6.7( v 430'109641 (430'108141,430'109641] local-lis/les=492/493 n=25825 ec=76/76 lis/c 492/492 les/c/f 493/493/0 498/498/498) [10,6,3] r=0 lpr=498 pi=[492,498)/1 crt=430'109641 lcod 0'0 mlcod 0'0 unknown NOTIFY] start_peering_interval up [6,3] -> [10,6,3], acting [6,3] -> [10,6,3], acting_primary 6 -> 10, up_primary 6 -> 10, role -1 -> 0, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.739996 7ffab63e5700 1 osd.10 pg_epoch: 498 pg[6.7( v 430'109641 (430'108141,430'109641] local-lis/les=492/493 n=25825 ec=76/76 lis/c 492/492 les/c/f 493/493/0 498/498/498) [10,6,3] r=0 lpr=498 pi=[492,498)/1 crt=430'109641 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 07:38:22.740007 7ffab5be4700 1 osd.10 pg_epoch: 495 pg[9.10( v 494'394 (0'0,494'394] local-lis/les=492/493 n=251 ec=190/190 lis/c 492/492 les/c/f 493/493/0 495/495/487) [4,11] r=-1 lpr=495 pi=[492,495)/1 crt=494'394 lcod 0'0 unknown NOTIFY] start_peering_interval up [4,11,10] -> [4,11], acting [4,11,10] -> [4,11], acting_primary 4 -> 4, up_primary 4 -> 4, role 2 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.740074 7ffab5be4700 1 osd.10 pg_epoch: 498 pg[9.10( v 494'394 (0'0,494'394] local-lis/les=492/493 n=251 ec=190/190 lis/c 492/492 les/c/f 493/493/0 498/498/487) [4,11,10] r=2 lpr=498 pi=[492,498)/1 crt=494'394 lcod 0'0 unknown NOTIFY] start_peering_interval up [4,11] -> [4,11,10], acting [4,11] -> [4,11,10], acting_primary 4 -> 4, up_primary 4 -> 4, role -1 -> 2, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.740120 7ffab63e5700 1 osd.10 pg_epoch: 495 pg[5.d( v 166'68209 (166'66709,166'68209] local-lis/les=492/493 n=68209 ec=71/71 lis/c 492/492 les/c/f 493/493/0 495/495/495) [7,0] r=-1 lpr=495 pi=[492,495)/1 crt=166'68209 lcod 0'0 unknown] start_peering_interval up [10,7,0] -> [7,0], acting [10,7,0] -> [7,0], acting_primary 10 -> 7, up_primary 10 -> 7, role 0 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.740140 7ffab5be4700 1 osd.10 pg_epoch: 498 pg[9.10( v 494'394 (0'0,494'394] local-lis/les=492/493 n=251 ec=190/190 lis/c 492/492 les/c/f 493/493/0 498/498/487) [4,11,10] r=2 lpr=498 pi=[492,498)/1 crt=494'394 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.740187 7ffab63e5700 1 osd.10 pg_epoch: 498 pg[5.d( v 166'68209 (166'66709,166'68209] local-lis/les=492/493 n=68209 ec=71/71 lis/c 492/492 les/c/f 493/493/0 498/498/498) [10,7,0] r=0 lpr=498 pi=[492,498)/1 crt=166'68209 lcod 0'0 mlcod 0'0 unknown NOTIFY] start_peering_interval up [7,0] -> [10,7,0], acting [7,0] -> [10,7,0], acting_primary 7 -> 10, up_primary 7 -> 10, role -1 -> 0, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.740268 7ffab63e5700 1 osd.10 pg_epoch: 498 pg[5.d( v 166'68209 (166'66709,166'68209] local-lis/les=492/493 n=68209 ec=71/71 lis/c 492/492 les/c/f 493/493/0 498/498/498) [10,7,0] r=0 lpr=498 pi=[492,498)/1 crt=166'68209 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 07:38:22.740289 7ffab5be4700 1 osd.10 pg_epoch: 495 pg[10.13( v 383'839 (0'0,383'839] local-lis/les=492/493 n=523 ec=193/193 lis/c 492/492 les/c/f 493/493/0 495/495/470) [1,7] r=-1 lpr=495 pi=[492,495)/1 crt=383'839 lcod 0'0 unknown NOTIFY] start_peering_interval up [1,10,7] -> [1,7], acting [1,10,7] -> [1,7], acting_primary 1 -> 1, up_primary 1 -> 1, role 1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.740354 7ffab5be4700 1 osd.10 pg_epoch: 498 pg[10.13( v 383'839 (0'0,383'839] local-lis/les=492/493 n=523 ec=193/193 lis/c 492/492 les/c/f 493/493/0 498/498/470) [1,10,7] r=1 lpr=498 pi=[492,498)/1 crt=383'839 lcod 0'0 unknown NOTIFY] start_peering_interval up [1,7] -> [1,10,7], acting [1,7] -> [1,10,7], acting_primary 1 -> 1, up_primary 1 -> 1, role -1 -> 1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.740430 7ffab5be4700 1 osd.10 pg_epoch: 498 pg[10.13( v 383'839 (0'0,383'839] local-lis/les=492/493 n=523 ec=193/193 lis/c 492/492 les/c/f 493/493/0 498/498/470) [1,10,7] r=1 lpr=498 pi=[492,498)/1 crt=383'839 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.740439 7ffab63e5700 1 osd.10 pg_epoch: 495 pg[6.e( v 430'116090 (430'114517,430'116090] local-lis/les=492/493 n=25967 ec=76/76 lis/c 492/492 les/c/f 493/493/0 495/495/487) [4,1] r=-1 lpr=495 pi=[492,495)/1 crt=430'116090 lcod 0'0 unknown NOTIFY] start_peering_interval up [4,1,10] -> [4,1], acting [4,1,10] -> [4,1], acting_primary 4 -> 4, up_primary 4 -> 4, role 2 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.740505 7ffab63e5700 1 osd.10 pg_epoch: 498 pg[6.e( v 430'116090 (430'114517,430'116090] local-lis/les=492/493 n=25967 ec=76/76 lis/c 492/492 les/c/f 493/493/0 498/498/487) [4,1,10] r=2 lpr=498 pi=[492,498)/1 crt=430'116090 lcod 0'0 unknown NOTIFY] start_peering_interval up [4,1] -> [4,1,10], acting [4,1] -> [4,1,10], acting_primary 4 -> 4, up_primary 4 -> 4, role -1 -> 2, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.740584 7ffab63e5700 1 osd.10 pg_epoch: 498 pg[6.e( v 430'116090 (430'114517,430'116090] local-lis/les=492/493 n=25967 ec=76/76 lis/c 492/492 les/c/f 493/493/0 498/498/487) [4,1,10] r=2 lpr=498 pi=[492,498)/1 crt=430'116090 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.740781 7ffab5be4700 1 osd.10 pg_epoch: 495 pg[9.1a( v 494'341 (0'0,494'341] local-lis/les=492/493 n=242 ec=190/190 lis/c 492/492 les/c/f 493/493/0 495/495/468) [7,2] r=-1 lpr=495 pi=[492,495)/1 crt=494'341 lcod 0'0 unknown NOTIFY] start_peering_interval up [7,2,10] -> [7,2], acting [7,2,10] -> [7,2], acting_primary 7 -> 7, up_primary 7 -> 7, role 2 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.740847 7ffab63e5700 1 osd.10 pg_epoch: 495 pg[11.8s3( v 383'59629 (352'55946,383'59629] local-lis/les=492/493 n=59629 ec=267/267 lis/c 492/492 les/c/f 493/493/0 495/495/468) [7,1,6,2147483647,5,8]p7(0) r=-1 lpr=495 pi=[492,495)/1 crt=383'59629 lcod 0'0 unknown NOTIFY] start_peering_interval up [7,1,6,10,5,8] -> [7,1,6,2147483647,5,8], acting [7,1,6,10,5,8] -> [7,1,6,2147483647,5,8], acting_primary 7(0) -> 7, up_primary 7(0) -> 7, role 3 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.740885 7ffab5be4700 1 osd.10 pg_epoch: 498 pg[9.1a( v 494'341 (0'0,494'341] local-lis/les=492/493 n=242 ec=190/190 lis/c 492/492 les/c/f 493/493/0 498/498/468) [7,2,10] r=2 lpr=498 pi=[492,498)/1 crt=494'341 lcod 0'0 unknown NOTIFY] start_peering_interval up [7,2] -> [7,2,10], acting [7,2] -> [7,2,10], acting_primary 7 -> 7, up_primary 7 -> 7, role -1 -> 2, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.740983 7ffab5be4700 1 osd.10 pg_epoch: 498 pg[9.1a( v 494'341 (0'0,494'341] local-lis/les=492/493 n=242 ec=190/190 lis/c 492/492 les/c/f 493/493/0 498/498/468) [7,2,10] r=2 lpr=498 pi=[492,498)/1 crt=494'341 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.741073 7ffab63e5700 1 osd.10 pg_epoch: 498 pg[11.8s3( v 383'59629 (352'55946,383'59629] local-lis/les=492/493 n=59629 ec=267/267 lis/c 492/492 les/c/f 493/493/0 498/498/468) [7,1,6,10,5,8]p7(0) r=3 lpr=498 pi=[492,498)/1 crt=383'59629 lcod 0'0 unknown NOTIFY] start_peering_interval up [7,1,6,2147483647,5,8] -> [7,1,6,10,5,8], acting [7,1,6,2147483647,5,8] -> [7,1,6,10,5,8], acting_primary 7(0) -> 7, up_primary 7(0) -> 7, role -1 -> 3, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.741179 7ffab5be4700 1 osd.10 pg_epoch: 495 pg[6.15( v 463'106940 (430'105357,463'106940] local-lis/les=492/493 n=25757 ec=76/76 lis/c 492/492 les/c/f 493/493/0 495/495/467) [5,11] r=-1 lpr=495 pi=[492,495)/1 crt=463'106940 lcod 0'0 unknown NOTIFY] start_peering_interval up [5,10,11] -> [5,11], acting [5,10,11] -> [5,11], acting_primary 5 -> 5, up_primary 5 -> 5, role 1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.741211 7ffab63e5700 1 osd.10 pg_epoch: 498 pg[11.8s3( v 383'59629 (352'55946,383'59629] local-lis/les=492/493 n=59629 ec=267/267 lis/c 492/492 les/c/f 493/493/0 498/498/468) [7,1,6,10,5,8]p7(0) r=3 lpr=498 pi=[492,498)/1 crt=383'59629 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.741272 7ffab5be4700 1 osd.10 pg_epoch: 498 pg[6.15( v 463'106940 (430'105357,463'106940] local-lis/les=492/493 n=25757 ec=76/76 lis/c 492/492 les/c/f 493/493/0 498/498/467) [5,10,11] r=1 lpr=498 pi=[492,498)/1 crt=463'106940 lcod 0'0 unknown NOTIFY] start_peering_interval up [5,11] -> [5,10,11], acting [5,11] -> [5,10,11], acting_primary 5 -> 5, up_primary 5 -> 5, role -1 -> 1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.741363 7ffab5be4700 1 osd.10 pg_epoch: 498 pg[6.15( v 463'106940 (430'105357,463'106940] local-lis/les=492/493 n=25757 ec=76/76 lis/c 492/492 les/c/f 493/493/0 498/498/467) [5,10,11] r=1 lpr=498 pi=[492,498)/1 crt=463'106940 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.741474 7ffab63e5700 1 osd.10 pg_epoch: 495 pg[13.8s5( v 377'9578 (367'1954,377'9578] local-lis/les=492/493 n=2631 ec=345/345 lis/c 492/492 les/c/f 493/493/0 495/495/465) [2,11,5,8,6,2147483647]p2(0) r=-1 lpr=495 pi=[492,495)/1 crt=377'9578 lcod 0'0 unknown NOTIFY] start_peering_interval up [2,11,5,8,6,10] -> [2,11,5,8,6,2147483647], acting [2,11,5,8,6,10] -> [2,11,5,8,6,2147483647], acting_primary 2(0) -> 2, up_primary 2(0) -> 2, role 5 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.741572 7ffab5be4700 1 osd.10 pg_epoch: 495 pg[16.3( empty local-lis/les=492/493 n=0 ec=442/442 lis/c 492/492 les/c/f 493/493/0 495/495/467) [5,2] r=-1 lpr=495 pi=[492,495)/1 crt=0'0 unknown NOTIFY] start_peering_interval up [5,2,10] -> [5,2], acting [5,2,10] -> [5,2], acting_primary 5 -> 5, up_primary 5 -> 5, role 2 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.741613 7ffab63e5700 1 osd.10 pg_epoch: 498 pg[13.8s5( v 377'9578 (367'1954,377'9578] local-lis/les=492/493 n=2631 ec=345/345 lis/c 492/492 les/c/f 493/493/0 498/498/465) [2,11,5,8,6,10]p2(0) r=5 lpr=498 pi=[492,498)/1 crt=377'9578 lcod 0'0 unknown NOTIFY] start_peering_interval up [2,11,5,8,6,2147483647] -> [2,11,5,8,6,10], acting [2,11,5,8,6,2147483647] -> [2,11,5,8,6,10], acting_primary 2(0) -> 2, up_primary 2(0) -> 2, role -1 -> 5, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.741662 7ffab5be4700 1 osd.10 pg_epoch: 498 pg[16.3( empty local-lis/les=492/493 n=0 ec=442/442 lis/c 492/492 les/c/f 493/493/0 498/498/467) [5,2,10] r=2 lpr=498 pi=[492,498)/1 crt=0'0 unknown NOTIFY] start_peering_interval up [5,2] -> [5,2,10], acting [5,2] -> [5,2,10], acting_primary 5 -> 5, up_primary 5 -> 5, role -1 -> 2, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.741739 7ffab5be4700 1 osd.10 pg_epoch: 498 pg[16.3( empty local-lis/les=492/493 n=0 ec=442/442 lis/c 492/492 les/c/f 493/493/0 498/498/467) [5,2,10] r=2 lpr=498 pi=[492,498)/1 crt=0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.741741 7ffab63e5700 1 osd.10 pg_epoch: 498 pg[13.8s5( v 377'9578 (367'1954,377'9578] local-lis/les=492/493 n=2631 ec=345/345 lis/c 492/492 les/c/f 493/493/0 498/498/465) [2,11,5,8,6,10]p2(0) r=5 lpr=498 pi=[492,498)/1 crt=377'9578 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.741861 7ffab63e5700 1 osd.10 pg_epoch: 495 pg[6.d( v 461'110631 (430'109050,461'110631] local-lis/les=492/493 n=25732 ec=76/76 lis/c 492/492 les/c/f 493/493/0 492/492/492) [10,7,2] r=0 lpr=493 crt=461'110631 lcod 0'0 mlcod 0'0 peering] state<Started/Primary/Peering>: Peering, affected_by_map, going to Reset >2018-07-11 07:38:22.741917 7ffab5be4700 1 osd.10 pg_epoch: 495 pg[13.3s1( v 377'10405 (371'2081,377'10405] local-lis/les=492/493 n=2736 ec=345/345 lis/c 492/492 les/c/f 493/493/0 495/495/470) [1,2147483647,8,6,7,5]p1(0) r=-1 lpr=495 pi=[492,495)/1 crt=377'10405 lcod 0'0 unknown NOTIFY] start_peering_interval up [1,10,8,6,7,5] -> [1,2147483647,8,6,7,5], acting [1,10,8,6,7,5] -> [1,2147483647,8,6,7,5], acting_primary 1(0) -> 1, up_primary 1(0) -> 1, role 1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.741988 7ffab63e5700 1 osd.10 pg_epoch: 495 pg[6.d( v 461'110631 (430'109050,461'110631] local-lis/les=492/493 n=25732 ec=76/76 lis/c 492/492 les/c/f 493/493/0 495/495/495) [7,2] r=-1 lpr=495 pi=[492,495)/1 crt=461'110631 lcod 0'0 unknown] start_peering_interval up [10,7,2] -> [7,2], acting [10,7,2] -> [7,2], acting_primary 10 -> 7, up_primary 10 -> 7, role 0 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.741999 7ffab5be4700 1 osd.10 pg_epoch: 498 pg[13.3s1( v 377'10405 (371'2081,377'10405] local-lis/les=492/493 n=2736 ec=345/345 lis/c 492/492 les/c/f 493/493/0 498/498/470) [1,10,8,6,7,5]p1(0) r=1 lpr=498 pi=[492,498)/1 crt=377'10405 lcod 0'0 unknown NOTIFY] start_peering_interval up [1,2147483647,8,6,7,5] -> [1,10,8,6,7,5], acting [1,2147483647,8,6,7,5] -> [1,10,8,6,7,5], acting_primary 1(0) -> 1, up_primary 1(0) -> 1, role -1 -> 1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.742070 7ffab5be4700 1 osd.10 pg_epoch: 498 pg[13.3s1( v 377'10405 (371'2081,377'10405] local-lis/les=492/493 n=2736 ec=345/345 lis/c 492/492 les/c/f 493/493/0 498/498/470) [1,10,8,6,7,5]p1(0) r=1 lpr=498 pi=[492,498)/1 crt=377'10405 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.742074 7ffab63e5700 1 osd.10 pg_epoch: 498 pg[6.d( v 461'110631 (430'109050,461'110631] local-lis/les=492/493 n=25732 ec=76/76 lis/c 492/492 les/c/f 493/493/0 498/498/498) [10,7,2] r=0 lpr=498 pi=[492,498)/1 crt=461'110631 lcod 0'0 mlcod 0'0 unknown NOTIFY] start_peering_interval up [7,2] -> [10,7,2], acting [7,2] -> [10,7,2], acting_primary 7 -> 10, up_primary 7 -> 10, role -1 -> 0, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.742152 7ffab63e5700 1 osd.10 pg_epoch: 498 pg[6.d( v 461'110631 (430'109050,461'110631] local-lis/les=492/493 n=25732 ec=76/76 lis/c 492/492 les/c/f 493/493/0 498/498/498) [10,7,2] r=0 lpr=498 pi=[492,498)/1 crt=461'110631 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 07:38:22.742230 7ffab5be4700 1 osd.10 pg_epoch: 495 pg[11.fs3( v 383'63252 (352'59530,383'63252] local-lis/les=492/493 n=60320 ec=267/267 lis/c 492/492 les/c/f 493/493/0 495/495/470) [0,6,2,2147483647,7,8]p0(0) r=-1 lpr=495 pi=[492,495)/1 crt=383'63252 lcod 0'0 unknown NOTIFY] start_peering_interval up [0,6,2,10,7,8] -> [0,6,2,2147483647,7,8], acting [0,6,2,10,7,8] -> [0,6,2,2147483647,7,8], acting_primary 0(0) -> 0, up_primary 0(0) -> 0, role 3 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.742301 7ffab63e5700 1 osd.10 pg_epoch: 495 pg[9.d( v 494'450 (0'0,494'450] local-lis/les=492/493 n=228 ec=190/190 lis/c 492/492 les/c/f 493/493/0 495/495/470) [0,6] r=-1 lpr=495 pi=[492,495)/1 crt=494'450 lcod 0'0 unknown NOTIFY] start_peering_interval up [0,6,10] -> [0,6], acting [0,6,10] -> [0,6], acting_primary 0 -> 0, up_primary 0 -> 0, role 2 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.742336 7ffab5be4700 1 osd.10 pg_epoch: 498 pg[11.fs3( v 383'63252 (352'59530,383'63252] local-lis/les=492/493 n=60320 ec=267/267 lis/c 492/492 les/c/f 493/493/0 498/498/470) [0,6,2,10,7,8]p0(0) r=3 lpr=498 pi=[492,498)/1 crt=383'63252 lcod 0'0 unknown NOTIFY] start_peering_interval up [0,6,2,2147483647,7,8] -> [0,6,2,10,7,8], acting [0,6,2,2147483647,7,8] -> [0,6,2,10,7,8], acting_primary 0(0) -> 0, up_primary 0(0) -> 0, role -1 -> 3, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.742369 7ffab63e5700 1 osd.10 pg_epoch: 498 pg[9.d( v 494'450 (0'0,494'450] local-lis/les=492/493 n=228 ec=190/190 lis/c 492/492 les/c/f 493/493/0 498/498/470) [0,6,10] r=2 lpr=498 pi=[492,498)/1 crt=494'450 lcod 0'0 unknown NOTIFY] start_peering_interval up [0,6] -> [0,6,10], acting [0,6] -> [0,6,10], acting_primary 0 -> 0, up_primary 0 -> 0, role -1 -> 2, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.742411 7ffab5be4700 1 osd.10 pg_epoch: 498 pg[11.fs3( v 383'63252 (352'59530,383'63252] local-lis/les=492/493 n=60320 ec=267/267 lis/c 492/492 les/c/f 493/493/0 498/498/470) [0,6,2,10,7,8]p0(0) r=3 lpr=498 pi=[492,498)/1 crt=383'63252 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.742436 7ffab63e5700 1 osd.10 pg_epoch: 498 pg[9.d( v 494'450 (0'0,494'450] local-lis/les=492/493 n=228 ec=190/190 lis/c 492/492 les/c/f 493/493/0 498/498/470) [0,6,10] r=2 lpr=498 pi=[492,498)/1 crt=494'450 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.742507 7ffab5be4700 1 osd.10 pg_epoch: 495 pg[11.5s0( v 383'60461 (352'56750,383'60461] local-lis/les=492/493 n=60461 ec=267/267 lis/c 492/492 les/c/f 493/493/0 492/492/492) [10,5,8,6,1,7]p10(0) r=0 lpr=493 crt=383'60461 lcod 0'0 mlcod 0'0 peering] state<Started/Primary/Peering>: Peering, affected_by_map, going to Reset >2018-07-11 07:38:22.742568 7ffab63e5700 1 osd.10 pg_epoch: 495 pg[6.2( v 430'110237 (430'108737,430'110237] local-lis/les=492/493 n=25907 ec=76/76 lis/c 492/492 les/c/f 493/493/0 495/495/468) [7,2] r=-1 lpr=495 pi=[492,495)/1 crt=430'110237 lcod 0'0 unknown NOTIFY] start_peering_interval up [7,10,2] -> [7,2], acting [7,10,2] -> [7,2], acting_primary 7 -> 7, up_primary 7 -> 7, role 1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.742619 7ffab5be4700 1 osd.10 pg_epoch: 495 pg[11.5s0( v 383'60461 (352'56750,383'60461] local-lis/les=492/493 n=60461 ec=267/267 lis/c 492/492 les/c/f 493/493/0 495/495/495) [2147483647,5,8,6,1,7]p5(1) r=-1 lpr=495 pi=[492,495)/1 crt=383'60461 lcod 0'0 unknown] start_peering_interval up [10,5,8,6,1,7] -> [2147483647,5,8,6,1,7], acting [10,5,8,6,1,7] -> [2147483647,5,8,6,1,7], acting_primary 10(0) -> 5, up_primary 10(0) -> 5, role 0 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.742629 7ffab63e5700 1 osd.10 pg_epoch: 498 pg[6.2( v 430'110237 (430'108737,430'110237] local-lis/les=492/493 n=25907 ec=76/76 lis/c 492/492 les/c/f 493/493/0 498/498/468) [7,10,2] r=1 lpr=498 pi=[492,498)/1 crt=430'110237 lcod 0'0 unknown NOTIFY] start_peering_interval up [7,2] -> [7,10,2], acting [7,2] -> [7,10,2], acting_primary 7 -> 7, up_primary 7 -> 7, role -1 -> 1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.742695 7ffab63e5700 1 osd.10 pg_epoch: 498 pg[6.2( v 430'110237 (430'108737,430'110237] local-lis/les=492/493 n=25907 ec=76/76 lis/c 492/492 les/c/f 493/493/0 498/498/468) [7,10,2] r=1 lpr=498 pi=[492,498)/1 crt=430'110237 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.742718 7ffab5be4700 1 osd.10 pg_epoch: 498 pg[11.5s0( v 383'60461 (352'56750,383'60461] local-lis/les=492/493 n=60461 ec=267/267 lis/c 492/492 les/c/f 493/493/0 498/498/498) [10,5,8,6,1,7]p10(0) r=0 lpr=498 pi=[492,498)/1 crt=383'60461 lcod 0'0 mlcod 0'0 unknown NOTIFY] start_peering_interval up [2147483647,5,8,6,1,7] -> [10,5,8,6,1,7], acting [2147483647,5,8,6,1,7] -> [10,5,8,6,1,7], acting_primary 5(1) -> 10, up_primary 5(1) -> 10, role -1 -> 0, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.742796 7ffab5be4700 1 osd.10 pg_epoch: 498 pg[11.5s0( v 383'60461 (352'56750,383'60461] local-lis/les=492/493 n=60461 ec=267/267 lis/c 492/492 les/c/f 493/493/0 498/498/498) [10,5,8,6,1,7]p10(0) r=0 lpr=498 pi=[492,498)/1 crt=383'60461 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 07:38:22.742847 7ffab63e5700 1 osd.10 pg_epoch: 495 pg[11.as3( v 383'60057 (367'56547,383'60057] local-lis/les=492/493 n=60057 ec=267/267 lis/c 492/492 les/c/f 493/493/0 495/495/484) [3,5,8,2147483647,11,6]p3(0) r=-1 lpr=495 pi=[492,495)/1 crt=383'60057 lcod 0'0 unknown NOTIFY] start_peering_interval up [3,5,8,10,11,6] -> [3,5,8,2147483647,11,6], acting [3,5,8,10,11,6] -> [3,5,8,2147483647,11,6], acting_primary 3(0) -> 3, up_primary 3(0) -> 3, role 3 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.742931 7ffab63e5700 1 osd.10 pg_epoch: 498 pg[11.as3( v 383'60057 (367'56547,383'60057] local-lis/les=492/493 n=60057 ec=267/267 lis/c 492/492 les/c/f 493/493/0 498/498/484) [3,5,8,10,11,6]p3(0) r=3 lpr=498 pi=[492,498)/1 crt=383'60057 lcod 0'0 unknown NOTIFY] start_peering_interval up [3,5,8,2147483647,11,6] -> [3,5,8,10,11,6], acting [3,5,8,2147483647,11,6] -> [3,5,8,10,11,6], acting_primary 3(0) -> 3, up_primary 3(0) -> 3, role -1 -> 3, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.742990 7ffab5be4700 1 osd.10 pg_epoch: 495 pg[11.2s2( v 383'59971 (377'58405,383'59971] local-lis/les=492/493 n=59971 ec=267/267 lis/c 492/492 les/c/f 493/493/0 495/495/465) [2,6,2147483647,0,7,8]p2(0) r=-1 lpr=495 pi=[492,495)/1 crt=383'59970 lcod 0'0 unknown NOTIFY] start_peering_interval up [2,6,10,0,7,8] -> [2,6,2147483647,0,7,8], acting [2,6,10,0,7,8] -> [2,6,2147483647,0,7,8], acting_primary 2(0) -> 2, up_primary 2(0) -> 2, role 2 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.743002 7ffab63e5700 1 osd.10 pg_epoch: 498 pg[11.as3( v 383'60057 (367'56547,383'60057] local-lis/les=492/493 n=60057 ec=267/267 lis/c 492/492 les/c/f 493/493/0 498/498/484) [3,5,8,10,11,6]p3(0) r=3 lpr=498 pi=[492,498)/1 crt=383'60057 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.743084 7ffab5be4700 1 osd.10 pg_epoch: 498 pg[11.2s2( v 383'59971 (377'58405,383'59971] local-lis/les=492/493 n=59971 ec=267/267 lis/c 492/492 les/c/f 493/493/0 498/498/465) [2,6,10,0,7,8]p2(0) r=2 lpr=498 pi=[492,498)/1 crt=383'59970 lcod 0'0 unknown NOTIFY] start_peering_interval up [2,6,2147483647,0,7,8] -> [2,6,10,0,7,8], acting [2,6,2147483647,0,7,8] -> [2,6,10,0,7,8], acting_primary 2(0) -> 2, up_primary 2(0) -> 2, role -1 -> 2, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.743161 7ffab5be4700 1 osd.10 pg_epoch: 498 pg[11.2s2( v 383'59971 (377'58405,383'59971] local-lis/les=492/493 n=59971 ec=267/267 lis/c 492/492 les/c/f 493/493/0 498/498/465) [2,6,10,0,7,8]p2(0) r=2 lpr=498 pi=[492,498)/1 crt=383'59970 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.743159 7ffab63e5700 1 osd.10 pg_epoch: 495 pg[13.bs4( v 377'9782 (355'1491,377'9782] local-lis/les=492/493 n=2694 ec=345/345 lis/c 492/492 les/c/f 493/493/0 495/495/478) [11,8,1,4,2147483647,6]p11(0) r=-1 lpr=495 pi=[492,495)/1 crt=377'9782 lcod 0'0 unknown NOTIFY] start_peering_interval up [11,8,1,4,10,6] -> [11,8,1,4,2147483647,6], acting [11,8,1,4,10,6] -> [11,8,1,4,2147483647,6], acting_primary 11(0) -> 11, up_primary 11(0) -> 11, role 4 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.743237 7ffab63e5700 1 osd.10 pg_epoch: 498 pg[13.bs4( v 377'9782 (355'1491,377'9782] local-lis/les=492/493 n=2694 ec=345/345 lis/c 492/492 les/c/f 493/493/0 498/498/478) [11,8,1,4,10,6]p11(0) r=4 lpr=498 pi=[492,498)/1 crt=377'9782 lcod 0'0 unknown NOTIFY] start_peering_interval up [11,8,1,4,2147483647,6] -> [11,8,1,4,10,6], acting [11,8,1,4,2147483647,6] -> [11,8,1,4,10,6], acting_primary 11(0) -> 11, up_primary 11(0) -> 11, role -1 -> 4, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.743268 7ffab5be4700 1 osd.10 pg_epoch: 495 pg[9.1e( v 494'752 (0'0,494'752] local-lis/les=492/493 n=270 ec=190/190 lis/c 492/492 les/c/f 493/493/0 495/495/465) [2,4] r=-1 lpr=495 pi=[492,495)/1 crt=494'752 lcod 0'0 unknown NOTIFY] start_peering_interval up [2,4,10] -> [2,4], acting [2,4,10] -> [2,4], acting_primary 2 -> 2, up_primary 2 -> 2, role 2 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.743324 7ffab63e5700 1 osd.10 pg_epoch: 498 pg[13.bs4( v 377'9782 (355'1491,377'9782] local-lis/les=492/493 n=2694 ec=345/345 lis/c 492/492 les/c/f 493/493/0 498/498/478) [11,8,1,4,10,6]p11(0) r=4 lpr=498 pi=[492,498)/1 crt=377'9782 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.743336 7ffab5be4700 1 osd.10 pg_epoch: 498 pg[9.1e( v 494'752 (0'0,494'752] local-lis/les=492/493 n=270 ec=190/190 lis/c 492/492 les/c/f 493/493/0 498/498/465) [2,4,10] r=2 lpr=498 pi=[492,498)/1 crt=494'752 lcod 0'0 unknown NOTIFY] start_peering_interval up [2,4] -> [2,4,10], acting [2,4] -> [2,4,10], acting_primary 2 -> 2, up_primary 2 -> 2, role -1 -> 2, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.743402 7ffab5be4700 1 osd.10 pg_epoch: 498 pg[9.1e( v 494'752 (0'0,494'752] local-lis/les=492/493 n=270 ec=190/190 lis/c 492/492 les/c/f 493/493/0 498/498/465) [2,4,10] r=2 lpr=498 pi=[492,498)/1 crt=494'752 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.743461 7ffab63e5700 1 osd.10 pg_epoch: 495 pg[6.26( v 461'123787 (430'122194,461'123787] local-lis/les=492/493 n=26121 ec=76/76 lis/c 492/492 les/c/f 493/493/0 495/495/468) [7,8] r=-1 lpr=495 pi=[492,495)/1 crt=461'123787 lcod 0'0 unknown NOTIFY] start_peering_interval up [7,10,8] -> [7,8], acting [7,10,8] -> [7,8], acting_primary 7 -> 7, up_primary 7 -> 7, role 1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.743524 7ffab63e5700 1 osd.10 pg_epoch: 498 pg[6.26( v 461'123787 (430'122194,461'123787] local-lis/les=492/493 n=26121 ec=76/76 lis/c 492/492 les/c/f 493/493/0 498/498/468) [7,10,8] r=1 lpr=498 pi=[492,498)/1 crt=461'123787 lcod 0'0 unknown NOTIFY] start_peering_interval up [7,8] -> [7,10,8], acting [7,8] -> [7,10,8], acting_primary 7 -> 7, up_primary 7 -> 7, role -1 -> 1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.743553 7ffab5be4700 1 osd.10 pg_epoch: 495 pg[13.2s4( v 377'9209 (0'0,377'9209] local-lis/les=492/493 n=2712 ec=345/345 lis/c 492/492 les/c/f 493/493/0 495/495/467) [5,8,6,7,2147483647,3]p5(0) r=-1 lpr=495 pi=[492,495)/1 crt=377'9201 lcod 0'0 unknown NOTIFY] start_peering_interval up [5,8,6,7,10,3] -> [5,8,6,7,2147483647,3], acting [5,8,6,7,10,3] -> [5,8,6,7,2147483647,3], acting_primary 5(0) -> 5, up_primary 5(0) -> 5, role 4 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.743592 7ffab63e5700 1 osd.10 pg_epoch: 498 pg[6.26( v 461'123787 (430'122194,461'123787] local-lis/les=492/493 n=26121 ec=76/76 lis/c 492/492 les/c/f 493/493/0 498/498/468) [7,10,8] r=1 lpr=498 pi=[492,498)/1 crt=461'123787 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.743646 7ffab5be4700 1 osd.10 pg_epoch: 498 pg[13.2s4( v 377'9209 (0'0,377'9209] local-lis/les=492/493 n=2712 ec=345/345 lis/c 492/492 les/c/f 493/493/0 498/498/467) [5,8,6,7,10,3]p5(0) r=4 lpr=498 pi=[492,498)/1 crt=377'9201 lcod 0'0 unknown NOTIFY] start_peering_interval up [5,8,6,7,2147483647,3] -> [5,8,6,7,10,3], acting [5,8,6,7,2147483647,3] -> [5,8,6,7,10,3], acting_primary 5(0) -> 5, up_primary 5(0) -> 5, role -1 -> 4, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.743720 7ffab5be4700 1 osd.10 pg_epoch: 498 pg[13.2s4( v 377'9209 (0'0,377'9209] local-lis/les=492/493 n=2712 ec=345/345 lis/c 492/492 les/c/f 493/493/0 498/498/467) [5,8,6,7,10,3]p5(0) r=4 lpr=498 pi=[492,498)/1 crt=377'9201 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.743748 7ffab63e5700 1 osd.10 pg_epoch: 495 pg[13.9s1( v 377'10260 (352'219,377'10260] local-lis/les=492/493 n=2727 ec=345/345 lis/c 492/492 les/c/f 493/493/0 495/495/467) [5,2147483647,3,6,8,11]p5(0) r=-1 lpr=495 pi=[492,495)/1 crt=377'10254 lcod 0'0 unknown NOTIFY] start_peering_interval up [5,10,3,6,8,11] -> [5,2147483647,3,6,8,11], acting [5,10,3,6,8,11] -> [5,2147483647,3,6,8,11], acting_primary 5(0) -> 5, up_primary 5(0) -> 5, role 1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.743805 7ffab5be4700 1 osd.10 pg_epoch: 495 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=492/493 n=68041 ec=71/71 lis/c 492/492 les/c/f 493/493/0 492/492/492) [10,2,4] r=0 lpr=493 crt=166'68041 lcod 0'0 mlcod 0'0 peering] state<Started/Primary/Peering>: Peering, affected_by_map, going to Reset >2018-07-11 07:38:22.743829 7ffab63e5700 1 osd.10 pg_epoch: 498 pg[13.9s1( v 377'10260 (352'219,377'10260] local-lis/les=492/493 n=2727 ec=345/345 lis/c 492/492 les/c/f 493/493/0 498/498/467) [5,10,3,6,8,11]p5(0) r=1 lpr=498 pi=[492,498)/1 crt=377'10254 lcod 0'0 unknown NOTIFY] start_peering_interval up [5,2147483647,3,6,8,11] -> [5,10,3,6,8,11], acting [5,2147483647,3,6,8,11] -> [5,10,3,6,8,11], acting_primary 5(0) -> 5, up_primary 5(0) -> 5, role -1 -> 1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.743902 7ffab5be4700 1 osd.10 pg_epoch: 495 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=492/493 n=68041 ec=71/71 lis/c 492/492 les/c/f 493/493/0 495/495/495) [2,4] r=-1 lpr=495 pi=[492,495)/1 crt=166'68041 lcod 0'0 unknown] start_peering_interval up [10,2,4] -> [2,4], acting [10,2,4] -> [2,4], acting_primary 10 -> 2, up_primary 10 -> 2, role 0 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.743908 7ffab63e5700 1 osd.10 pg_epoch: 498 pg[13.9s1( v 377'10260 (352'219,377'10260] local-lis/les=492/493 n=2727 ec=345/345 lis/c 492/492 les/c/f 493/493/0 498/498/467) [5,10,3,6,8,11]p5(0) r=1 lpr=498 pi=[492,498)/1 crt=377'10254 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.743971 7ffab5be4700 1 osd.10 pg_epoch: 498 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=492/493 n=68041 ec=71/71 lis/c 492/492 les/c/f 493/493/0 498/498/498) [10,2,4] r=0 lpr=498 pi=[492,498)/1 crt=166'68041 lcod 0'0 mlcod 0'0 unknown NOTIFY] start_peering_interval up [2,4] -> [10,2,4], acting [2,4] -> [10,2,4], acting_primary 2 -> 10, up_primary 2 -> 10, role -1 -> 0, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.744047 7ffab5be4700 1 osd.10 pg_epoch: 498 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=492/493 n=68041 ec=71/71 lis/c 492/492 les/c/f 493/493/0 498/498/498) [10,2,4] r=0 lpr=498 pi=[492,498)/1 crt=166'68041 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 07:38:22.744076 7ffab63e5700 1 osd.10 pg_epoch: 495 pg[11.ds3( v 383'59865 (367'56373,383'59865] local-lis/les=492/493 n=59865 ec=267/267 lis/c 492/492 les/c/f 493/493/0 495/495/465) [2,4,11,2147483647,8,6]p2(0) r=-1 lpr=495 pi=[492,495)/1 crt=383'59865 lcod 0'0 unknown NOTIFY] start_peering_interval up [2,4,11,10,8,6] -> [2,4,11,2147483647,8,6], acting [2,4,11,10,8,6] -> [2,4,11,2147483647,8,6], acting_primary 2(0) -> 2, up_primary 2(0) -> 2, role 3 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.744169 7ffab5be4700 1 osd.10 pg_epoch: 495 pg[11.1s0( v 383'60378 (352'56674,383'60378] local-lis/les=492/493 n=60378 ec=267/267 lis/c 492/492 les/c/f 493/493/0 492/492/492) [10,5,2,8,6,11]p10(0) r=0 lpr=493 crt=383'60378 lcod 0'0 mlcod 0'0 peering] state<Started/Primary/Peering>: Peering, affected_by_map, going to Reset >2018-07-11 07:38:22.744188 7ffab63e5700 1 osd.10 pg_epoch: 498 pg[11.ds3( v 383'59865 (367'56373,383'59865] local-lis/les=492/493 n=59865 ec=267/267 lis/c 492/492 les/c/f 493/493/0 498/498/465) [2,4,11,10,8,6]p2(0) r=3 lpr=498 pi=[492,498)/1 crt=383'59865 lcod 0'0 unknown NOTIFY] start_peering_interval up [2,4,11,2147483647,8,6] -> [2,4,11,10,8,6], acting [2,4,11,2147483647,8,6] -> [2,4,11,10,8,6], acting_primary 2(0) -> 2, up_primary 2(0) -> 2, role -1 -> 3, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.744277 7ffab63e5700 1 osd.10 pg_epoch: 498 pg[11.ds3( v 383'59865 (367'56373,383'59865] local-lis/les=492/493 n=59865 ec=267/267 lis/c 492/492 les/c/f 493/493/0 498/498/465) [2,4,11,10,8,6]p2(0) r=3 lpr=498 pi=[492,498)/1 crt=383'59865 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.744294 7ffab5be4700 1 osd.10 pg_epoch: 495 pg[11.1s0( v 383'60378 (352'56674,383'60378] local-lis/les=492/493 n=60378 ec=267/267 lis/c 492/492 les/c/f 493/493/0 495/495/495) [2147483647,5,2,8,6,11]p5(1) r=-1 lpr=495 pi=[492,495)/1 crt=383'60378 lcod 0'0 unknown] start_peering_interval up [10,5,2,8,6,11] -> [2147483647,5,2,8,6,11], acting [10,5,2,8,6,11] -> [2147483647,5,2,8,6,11], acting_primary 10(0) -> 5, up_primary 10(0) -> 5, role 0 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.744393 7ffab5be4700 1 osd.10 pg_epoch: 498 pg[11.1s0( v 383'60378 (352'56674,383'60378] local-lis/les=492/493 n=60378 ec=267/267 lis/c 492/492 les/c/f 493/493/0 498/498/498) [10,5,2,8,6,11]p10(0) r=0 lpr=498 pi=[492,498)/1 crt=383'60378 lcod 0'0 mlcod 0'0 unknown NOTIFY] start_peering_interval up [2147483647,5,2,8,6,11] -> [10,5,2,8,6,11], acting [2147483647,5,2,8,6,11] -> [10,5,2,8,6,11], acting_primary 5(1) -> 10, up_primary 5(1) -> 10, role -1 -> 0, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.744411 7ffab63e5700 1 osd.10 pg_epoch: 495 pg[16.d( empty local-lis/les=492/493 n=0 ec=442/442 lis/c 492/492 les/c/f 493/493/0 495/495/470) [0,3] r=-1 lpr=495 pi=[492,495)/1 crt=0'0 unknown NOTIFY] start_peering_interval up [0,3,10] -> [0,3], acting [0,3,10] -> [0,3], acting_primary 0 -> 0, up_primary 0 -> 0, role 2 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.744468 7ffab5be4700 1 osd.10 pg_epoch: 498 pg[11.1s0( v 383'60378 (352'56674,383'60378] local-lis/les=492/493 n=60378 ec=267/267 lis/c 492/492 les/c/f 493/493/0 498/498/498) [10,5,2,8,6,11]p10(0) r=0 lpr=498 pi=[492,498)/1 crt=383'60378 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 07:38:22.744473 7ffab63e5700 1 osd.10 pg_epoch: 498 pg[16.d( empty local-lis/les=492/493 n=0 ec=442/442 lis/c 492/492 les/c/f 493/493/0 498/498/470) [0,3,10] r=2 lpr=498 pi=[492,498)/1 crt=0'0 unknown NOTIFY] start_peering_interval up [0,3] -> [0,3,10], acting [0,3] -> [0,3,10], acting_primary 0 -> 0, up_primary 0 -> 0, role -1 -> 2, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.744537 7ffab63e5700 1 osd.10 pg_epoch: 498 pg[16.d( empty local-lis/les=492/493 n=0 ec=442/442 lis/c 492/492 les/c/f 493/493/0 498/498/470) [0,3,10] r=2 lpr=498 pi=[492,498)/1 crt=0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.744686 7ffab5be4700 1 osd.10 pg_epoch: 495 pg[13.5s2( v 377'9560 (0'0,377'9560] local-lis/les=493/494 n=2691 ec=345/345 lis/c 493/493 les/c/f 494/494/0 495/495/487) [4,3,2147483647,11,2147483647,8]/[4,3,2147483647,11,8,8]p4(0) r=-1 lpr=495 pi=[493,495)/1 crt=377'9552 lcod 0'0 remapped NOTIFY] start_peering_interval up [4,3,10,11,2147483647,8] -> [4,3,2147483647,11,2147483647,8], acting [4,3,10,11,8,8] -> [4,3,2147483647,11,8,8], acting_primary 4(0) -> 4, up_primary 4(0) -> 4, role 2 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.744691 7ffab63e5700 1 osd.10 pg_epoch: 495 pg[13.fs4( v 377'8989 (0'0,377'8989] local-lis/les=492/493 n=2712 ec=345/345 lis/c 492/492 les/c/f 493/493/0 495/495/467) [5,7,3,6,2147483647,8]p5(0) r=-1 lpr=495 pi=[492,495)/1 crt=377'8979 lcod 0'0 unknown NOTIFY] start_peering_interval up [5,7,3,6,10,8] -> [5,7,3,6,2147483647,8], acting [5,7,3,6,10,8] -> [5,7,3,6,2147483647,8], acting_primary 5(0) -> 5, up_primary 5(0) -> 5, role 4 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.744806 7ffab63e5700 1 osd.10 pg_epoch: 498 pg[13.fs4( v 377'8989 (0'0,377'8989] local-lis/les=492/493 n=2712 ec=345/345 lis/c 492/492 les/c/f 493/493/0 498/498/467) [5,7,3,6,10,8]p5(0) r=4 lpr=498 pi=[492,498)/1 crt=377'8979 lcod 0'0 unknown NOTIFY] start_peering_interval up [5,7,3,6,2147483647,8] -> [5,7,3,6,10,8], acting [5,7,3,6,2147483647,8] -> [5,7,3,6,10,8], acting_primary 5(0) -> 5, up_primary 5(0) -> 5, role -1 -> 4, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.744878 7ffab63e5700 1 osd.10 pg_epoch: 498 pg[13.fs4( v 377'8989 (0'0,377'8989] local-lis/les=492/493 n=2712 ec=345/345 lis/c 492/492 les/c/f 493/493/0 498/498/467) [5,7,3,6,10,8]p5(0) r=4 lpr=498 pi=[492,498)/1 crt=377'8979 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.744953 7ffab5be4700 1 osd.10 pg_epoch: 498 pg[13.5s2( v 377'9560 (0'0,377'9560] local-lis/les=493/494 n=2691 ec=345/345 lis/c 493/493 les/c/f 494/494/0 498/498/487) [4,3,10,11,2147483647,8]/[4,3,2147483647,11,8,8]p4(0) r=-1 lpr=498 pi=[493,498)/1 crt=377'9552 lcod 0'0 remapped NOTIFY] start_peering_interval up [4,3,2147483647,11,2147483647,8] -> [4,3,10,11,2147483647,8], acting [4,3,2147483647,11,8,8] -> [4,3,2147483647,11,8,8], acting_primary 4(0) -> 4, up_primary 4(0) -> 4, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.745005 7ffab63e5700 1 osd.10 pg_epoch: 495 pg[16.1a( empty local-lis/les=492/493 n=0 ec=442/442 lis/c 492/492 les/c/f 493/493/0 495/495/478) [11,3] r=-1 lpr=495 pi=[492,495)/1 crt=0'0 unknown NOTIFY] start_peering_interval up [11,10,3] -> [11,3], acting [11,10,3] -> [11,3], acting_primary 11 -> 11, up_primary 11 -> 11, role 1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.745043 7ffab5be4700 1 osd.10 pg_epoch: 498 pg[13.5s2( v 377'9560 (0'0,377'9560] local-lis/les=493/494 n=2691 ec=345/345 lis/c 493/493 les/c/f 494/494/0 498/498/487) [4,3,10,11,2147483647,8]/[4,3,2147483647,11,8,8]p4(0) r=-1 lpr=498 pi=[493,498)/1 crt=377'9552 lcod 0'0 remapped NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.745065 7ffab63e5700 1 osd.10 pg_epoch: 498 pg[16.1a( empty local-lis/les=492/493 n=0 ec=442/442 lis/c 492/492 les/c/f 493/493/0 498/498/478) [11,10,3] r=1 lpr=498 pi=[492,498)/1 crt=0'0 unknown NOTIFY] start_peering_interval up [11,3] -> [11,10,3], acting [11,3] -> [11,10,3], acting_primary 11 -> 11, up_primary 11 -> 11, role -1 -> 1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.745132 7ffab63e5700 1 osd.10 pg_epoch: 498 pg[16.1a( empty local-lis/les=492/493 n=0 ec=442/442 lis/c 492/492 les/c/f 493/493/0 498/498/478) [11,10,3] r=1 lpr=498 pi=[492,498)/1 crt=0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.745204 7ffab5be4700 1 osd.10 pg_epoch: 495 pg[13.4s3( v 377'9519 (355'1503,377'9519] local-lis/les=492/493 n=2672 ec=345/345 lis/c 492/492 les/c/f 493/493/0 495/495/467) [5,1,8,2147483647,11,6]p5(0) r=-1 lpr=495 pi=[492,495)/1 crt=377'9519 lcod 0'0 unknown NOTIFY] start_peering_interval up [5,1,8,10,11,6] -> [5,1,8,2147483647,11,6], acting [5,1,8,10,11,6] -> [5,1,8,2147483647,11,6], acting_primary 5(0) -> 5, up_primary 5(0) -> 5, role 3 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.745329 7ffab5be4700 1 osd.10 pg_epoch: 498 pg[13.4s3( v 377'9519 (355'1503,377'9519] local-lis/les=492/493 n=2672 ec=345/345 lis/c 492/492 les/c/f 493/493/0 498/498/467) [5,1,8,10,11,6]p5(0) r=3 lpr=498 pi=[492,498)/1 crt=377'9519 lcod 0'0 unknown NOTIFY] start_peering_interval up [5,1,8,2147483647,11,6] -> [5,1,8,10,11,6], acting [5,1,8,2147483647,11,6] -> [5,1,8,10,11,6], acting_primary 5(0) -> 5, up_primary 5(0) -> 5, role -1 -> 3, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.745446 7ffab5be4700 1 osd.10 pg_epoch: 498 pg[13.4s3( v 377'9519 (355'1503,377'9519] local-lis/les=492/493 n=2672 ec=345/345 lis/c 492/492 les/c/f 493/493/0 498/498/467) [5,1,8,10,11,6]p5(0) r=3 lpr=498 pi=[492,498)/1 crt=377'9519 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.745663 7ffab5be4700 1 osd.10 pg_epoch: 495 pg[5.2f( v 166'67864 (166'66267,166'67864] local-lis/les=492/493 n=67864 ec=71/71 lis/c 492/492 les/c/f 493/493/0 495/495/472) [8,4] r=-1 lpr=495 pi=[492,495)/1 crt=166'67864 lcod 0'0 unknown NOTIFY] start_peering_interval up [8,10,4] -> [8,4], acting [8,10,4] -> [8,4], acting_primary 8 -> 8, up_primary 8 -> 8, role 1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.745735 7ffab5be4700 1 osd.10 pg_epoch: 498 pg[5.2f( v 166'67864 (166'66267,166'67864] local-lis/les=492/493 n=67864 ec=71/71 lis/c 492/492 les/c/f 493/493/0 498/498/472) [8,10,4] r=1 lpr=498 pi=[492,498)/1 crt=166'67864 lcod 0'0 unknown NOTIFY] start_peering_interval up [8,4] -> [8,10,4], acting [8,4] -> [8,10,4], acting_primary 8 -> 8, up_primary 8 -> 8, role -1 -> 1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.745803 7ffab5be4700 1 osd.10 pg_epoch: 498 pg[5.2f( v 166'67864 (166'66267,166'67864] local-lis/les=492/493 n=67864 ec=71/71 lis/c 492/492 les/c/f 493/493/0 498/498/472) [8,10,4] r=1 lpr=498 pi=[492,498)/1 crt=166'67864 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.745930 7ffab5be4700 1 osd.10 pg_epoch: 495 pg[6.2c( v 457'120217 (430'118627,457'120217] local-lis/les=492/493 n=26249 ec=76/76 lis/c 492/492 les/c/f 493/493/0 495/495/487) [4,6] r=-1 lpr=495 pi=[492,495)/1 crt=457'120217 lcod 0'0 unknown NOTIFY] start_peering_interval up [4,6,10] -> [4,6], acting [4,6,10] -> [4,6], acting_primary 4 -> 4, up_primary 4 -> 4, role 2 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.745991 7ffab5be4700 1 osd.10 pg_epoch: 498 pg[6.2c( v 457'120217 (430'118627,457'120217] local-lis/les=492/493 n=26249 ec=76/76 lis/c 492/492 les/c/f 493/493/0 498/498/487) [4,6,10] r=2 lpr=498 pi=[492,498)/1 crt=457'120217 lcod 0'0 unknown NOTIFY] start_peering_interval up [4,6] -> [4,6,10], acting [4,6] -> [4,6,10], acting_primary 4 -> 4, up_primary 4 -> 4, role -1 -> 2, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.746058 7ffab5be4700 1 osd.10 pg_epoch: 498 pg[6.2c( v 457'120217 (430'118627,457'120217] local-lis/les=492/493 n=26249 ec=76/76 lis/c 492/492 les/c/f 493/493/0 498/498/487) [4,6,10] r=2 lpr=498 pi=[492,498)/1 crt=457'120217 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.746209 7ffab5be4700 1 osd.10 pg_epoch: 498 pg[10.1c( v 386'649 (0'0,386'649] lb MIN (bitwise) local-lis/les=419/420 n=500 ec=193/193 lis/c 419/419 les/c/f 420/420/0 487/487/478) [11,4,8] r=-1 lpr=494 pi=[419,487)/2 crt=386'649 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:22.746270 7ffab5be4700 1 osd.10 pg_epoch: 495 pg[10.7( v 386'575 (0'0,386'575] local-lis/les=492/493 n=507 ec=193/193 lis/c 492/492 les/c/f 493/493/0 492/492/492) [10,6,3] r=0 lpr=493 crt=386'575 lcod 0'0 mlcod 0'0 peering] state<Started/Primary/Peering>: Peering, affected_by_map, going to Reset >2018-07-11 07:38:22.746367 7ffab5be4700 1 osd.10 pg_epoch: 495 pg[10.7( v 386'575 (0'0,386'575] local-lis/les=492/493 n=507 ec=193/193 lis/c 492/492 les/c/f 493/493/0 495/495/495) [6,3] r=-1 lpr=495 pi=[492,495)/1 crt=386'575 lcod 0'0 unknown] start_peering_interval up [10,6,3] -> [6,3], acting [10,6,3] -> [6,3], acting_primary 10 -> 6, up_primary 10 -> 6, role 0 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.746429 7ffab5be4700 1 osd.10 pg_epoch: 498 pg[10.7( v 386'575 (0'0,386'575] local-lis/les=492/493 n=507 ec=193/193 lis/c 492/492 les/c/f 493/493/0 498/498/498) [10,6,3] r=0 lpr=498 pi=[492,498)/1 crt=386'575 lcod 0'0 mlcod 0'0 unknown NOTIFY] start_peering_interval up [6,3] -> [10,6,3], acting [6,3] -> [10,6,3], acting_primary 6 -> 10, up_primary 6 -> 10, role -1 -> 0, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:22.746495 7ffab5be4700 1 osd.10 pg_epoch: 498 pg[10.7( v 386'575 (0'0,386'575] local-lis/les=492/493 n=507 ec=193/193 lis/c 492/492 les/c/f 493/493/0 498/498/498) [10,6,3] r=0 lpr=498 pi=[492,498)/1 crt=386'575 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 07:38:23.484132 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 3] Generated table #36601: 4957167 keys, 80348310 bytes >2018-07-11 07:38:23.484180 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294703484152, "cf_name": "default", "job": 3, "event": "table_file_creation", "file_number": 36601, "file_size": 80348310, "table_properties": {"data_size": 67109394, "index_size": 689219, "filter_size": 12548707, "raw_key_size": 143207044, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957167, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957167", "kMergeOperands": "0"}} >2018-07-11 07:38:24.061113 7ffab5be4700 1 osd.10 pg_epoch: 500 pg[13.5s2( v 377'9560 (0'0,377'9560] local-lis/les=493/494 n=2691 ec=345/345 lis/c 495/493 les/c/f 497/494/0 498/500/487) [4,3,10,11,2147483647,8]/[4,3,10,11,8,8]p4(0) r=2 lpr=500 pi=[493,500)/1 crt=377'9552 lcod 0'0 remapped NOTIFY] start_peering_interval up [4,3,10,11,2147483647,8] -> [4,3,10,11,2147483647,8], acting [4,3,2147483647,11,8,8] -> [4,3,10,11,8,8], acting_primary 4(0) -> 4, up_primary 4(0) -> 4, role -1 -> 2, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:38:24.061208 7ffab5be4700 1 osd.10 pg_epoch: 500 pg[13.5s2( v 377'9560 (0'0,377'9560] local-lis/les=493/494 n=2691 ec=345/345 lis/c 495/493 les/c/f 497/494/0 498/500/487) [4,3,10,11,2147483647,8]/[4,3,10,11,8,8]p4(0) r=2 lpr=500 pi=[493,500)/1 crt=377'9552 lcod 0'0 remapped NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:38:24.421779 7ffacabcf700 0 -- 10.8.128.107:6801/13927 >> 10.8.128.107:6805/12617 conn(0x55d985d48800 :6801 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg: challenging authorizer >2018-07-11 07:38:24.421996 7ffacabcf700 0 -- 10.8.128.107:6801/13927 >> 10.8.128.107:6805/12617 conn(0x55d985d48800 :6801 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg accept connect_seq 0 vs existing csq=0 existing_state=STATE_CONNECTING_WAIT_CONNECT_REPLY >2018-07-11 07:38:24.422329 7ffacabcf700 0 -- 10.8.128.107:6801/13927 >> 10.8.128.107:6805/12617 conn(0x55d985d47000 :-1 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg: challenging authorizer >2018-07-11 07:38:26.893092 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 3] Generated table #36602: 2680208 keys, 74793582 bytes >2018-07-11 07:38:26.893139 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294706893109, "cf_name": "default", "job": 3, "event": "table_file_creation", "file_number": 36602, "file_size": 74793582, "table_properties": {"data_size": 67111025, "index_size": 829324, "filter_size": 6852241, "raw_key_size": 78331641, "raw_average_key_size": 29, "raw_value_size": 30499285, "raw_average_value_size": 11, "num_data_blocks": 16711, "num_entries": 2680208, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "2042833", "kMergeOperands": "0"}} >2018-07-11 07:38:29.966330 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 3] Generated table #36603: 2298857 keys, 73864103 bytes >2018-07-11 07:38:29.966374 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294709966347, "cf_name": "default", "job": 3, "event": "table_file_creation", "file_number": 36603, "file_size": 73864103, "table_properties": {"data_size": 67109070, "index_size": 855721, "filter_size": 5898320, "raw_key_size": 67525849, "raw_average_key_size": 29, "raw_value_size": 35594171, "raw_average_value_size": 15, "num_data_blocks": 16748, "num_entries": 2298857, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1595309", "kMergeOperands": "0"}} >2018-07-11 07:38:31.442488 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 3] Generated table #36604: 1404691 keys, 22768272 bytes >2018-07-11 07:38:31.442531 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294711442503, "cf_name": "default", "job": 3, "event": "table_file_creation", "file_number": 36604, "file_size": 22768272, "table_properties": {"data_size": 19016491, "index_size": 194922, "filter_size": 3555872, "raw_key_size": 40579972, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 4667, "num_entries": 1404691, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1404691", "kMergeOperands": "0"}} >2018-07-11 07:38:36.391302 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 3] Generated table #36605: 4680131 keys, 79657723 bytes >2018-07-11 07:38:36.391342 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294716391325, "cf_name": "default", "job": 3, "event": "table_file_creation", "file_number": 36605, "file_size": 79657723, "table_properties": {"data_size": 67112207, "index_size": 689382, "filter_size": 11855141, "raw_key_size": 135203775, "raw_average_key_size": 28, "raw_value_size": 3749621, "raw_average_value_size": 0, "num_data_blocks": 16473, "num_entries": 4680131, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4364742", "kMergeOperands": "0"}} >2018-07-11 07:38:37.075268 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 3] Generated table #36606: 635258 keys, 10297011 bytes >2018-07-11 07:38:37.075299 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294717075284, "cf_name": "default", "job": 3, "event": "table_file_creation", "file_number": 36606, "file_size": 10297011, "table_properties": {"data_size": 8600037, "index_size": 87877, "filter_size": 1608112, "raw_key_size": 18351909, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 2111, "num_entries": 635258, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "635258", "kMergeOperands": "0"}} >2018-07-11 07:38:42.397603 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 3] Generated table #36607: 4957168 keys, 80348314 bytes >2018-07-11 07:38:42.397659 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294722397629, "cf_name": "default", "job": 3, "event": "table_file_creation", "file_number": 36607, "file_size": 80348314, "table_properties": {"data_size": 67109372, "index_size": 689242, "filter_size": 12548710, "raw_key_size": 143207073, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957168, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957168", "kMergeOperands": "0"}} >2018-07-11 07:38:45.632569 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 3] Generated table #36608: 3042832 keys, 49319964 bytes >2018-07-11 07:38:45.632616 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294725632600, "cf_name": "default", "job": 3, "event": "table_file_creation", "file_number": 36608, "file_size": 49319964, "table_properties": {"data_size": 41193404, "index_size": 422867, "filter_size": 7702704, "raw_key_size": 87904047, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 10110, "num_entries": 3042832, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "3042832", "kMergeOperands": "0"}} >2018-07-11 07:38:50.985289 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 3] Generated table #36609: 4957167 keys, 80348284 bytes >2018-07-11 07:38:50.985365 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294730985348, "cf_name": "default", "job": 3, "event": "table_file_creation", "file_number": 36609, "file_size": 80348284, "table_properties": {"data_size": 67109332, "index_size": 689255, "filter_size": 12548707, "raw_key_size": 143207034, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957167, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957167", "kMergeOperands": "0"}} >2018-07-11 07:38:52.957845 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 3] Generated table #36610: 1871497 keys, 30334468 bytes >2018-07-11 07:38:52.957888 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294732957862, "cf_name": "default", "job": 3, "event": "table_file_creation", "file_number": 36610, "file_size": 30334468, "table_properties": {"data_size": 25336050, "index_size": 259873, "filter_size": 4737558, "raw_key_size": 54065474, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 6218, "num_entries": 1871497, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1871497", "kMergeOperands": "0"}} >2018-07-11 07:38:52.957961 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 3] Compacted 1@1 + 9@2 files to L2 => 679331480 bytes >2018-07-11 07:38:52.976375 7ffabc3f1700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 64758359563 >2018-07-11 07:38:52.976606 7ffabc3f1700 4 rocksdb: (Original Log Time 2018/07/11-07:38:52.976336) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 17 88 446 3641 796 0] max score 5.76, MB/sec: 15.3 rd, 15.3 wr, level 2, files in(1, 9) out(12) MB in(76.6, 571.2) out(647.9), read-write-amplify(16.9) write-amplify(8.5) OK, records in: 37484976, records dropped: 0 > >2018-07-11 07:38:52.976616 7ffabc3f1700 4 rocksdb: (Original Log Time 2018/07/11-07:38:52.976363) EVENT_LOG_v1 {"time_micros": 1531294732976351, "job": 3, "event": "compaction_finished", "compaction_time_micros": 44325554, "output_level": 2, "num_output_files": 12, "total_output_size": 679331480, "num_input_records": 37484976, "num_output_records": 37484976, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 17, 88, 446, 3641, 796, 0]} >2018-07-11 07:38:52.977103 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294732977099, "job": 3, "event": "table_file_deletion", "file_number": 36593} >2018-07-11 07:38:52.977131 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294732977129, "job": 3, "event": "table_file_deletion", "file_number": 36592} >2018-07-11 07:38:52.977145 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294732977142, "job": 3, "event": "table_file_deletion", "file_number": 36562} >2018-07-11 07:38:52.977155 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294732977153, "job": 3, "event": "table_file_deletion", "file_number": 36549} >2018-07-11 07:38:52.977168 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294732977166, "job": 3, "event": "table_file_deletion", "file_number": 36548} >2018-07-11 07:38:52.977182 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294732977180, "job": 3, "event": "table_file_deletion", "file_number": 36547} >2018-07-11 07:38:52.977196 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294732977194, "job": 3, "event": "table_file_deletion", "file_number": 36543} >2018-07-11 07:38:52.977209 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294732977206, "job": 3, "event": "table_file_deletion", "file_number": 36542} >2018-07-11 07:38:52.977235 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294732977232, "job": 3, "event": "table_file_deletion", "file_number": 36532} >2018-07-11 07:38:52.977255 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294732977253, "job": 3, "event": "table_file_deletion", "file_number": 36531} >2018-07-11 07:38:52.977413 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 4] Compacting 1@1 + 8@2 files to L2, score 5.76 >2018-07-11 07:38:52.977428 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 3 Base level 1, inputs: [36563(76MB)], [36609(76MB) 36610(28MB) 36544(1356KB) 36545(28MB) 36535(76MB) 36536(76MB) 36537(43MB) 36538(74MB)] > >2018-07-11 07:38:52.977444 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294732977432, "job": 4, "event": "compaction_started", "files_L1": [36563], "files_L2": [36609, 36610, 36544, 36545, 36535, 36536, 36537, 36538], "score": 5.75913, "input_data_size": 506736418} >2018-07-11 07:38:58.469074 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 4] Generated table #36611: 4957168 keys, 80348309 bytes >2018-07-11 07:38:58.469126 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294738469098, "cf_name": "default", "job": 4, "event": "table_file_creation", "file_number": 36611, "file_size": 80348309, "table_properties": {"data_size": 67109353, "index_size": 689256, "filter_size": 12548710, "raw_key_size": 143207073, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957168, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957168", "kMergeOperands": "0"}} >2018-07-11 07:39:03.700610 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 4] Generated table #36612: 4957165 keys, 80348256 bytes >2018-07-11 07:39:03.700657 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294743700630, "cf_name": "default", "job": 4, "event": "table_file_creation", "file_number": 36612, "file_size": 80348256, "table_properties": {"data_size": 67109345, "index_size": 689220, "filter_size": 12548701, "raw_key_size": 143206995, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957165, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957165", "kMergeOperands": "0"}} >2018-07-11 07:39:08.973530 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 4] Generated table #36613: 4957165 keys, 80348260 bytes >2018-07-11 07:39:08.973572 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294748973555, "cf_name": "default", "job": 4, "event": "table_file_creation", "file_number": 36613, "file_size": 80348260, "table_properties": {"data_size": 67109368, "index_size": 689201, "filter_size": 12548701, "raw_key_size": 143206995, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957165, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957165", "kMergeOperands": "0"}} >2018-07-11 07:39:12.395027 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 4] Generated table #36614: 3128502 keys, 50708500 bytes >2018-07-11 07:39:12.395073 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294752395046, "cf_name": "default", "job": 4, "event": "table_file_creation", "file_number": 36614, "file_size": 50708500, "table_properties": {"data_size": 42353191, "index_size": 434751, "filter_size": 7919569, "raw_key_size": 90378957, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 10394, "num_entries": 3128502, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "3128502", "kMergeOperands": "0"}} >2018-07-11 07:39:17.661548 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 4] Generated table #36615: 4957168 keys, 80348314 bytes >2018-07-11 07:39:17.661600 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294757661572, "cf_name": "default", "job": 4, "event": "table_file_creation", "file_number": 36615, "file_size": 80348314, "table_properties": {"data_size": 67109372, "index_size": 689242, "filter_size": 12548710, "raw_key_size": 143207073, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957168, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957168", "kMergeOperands": "0"}} >2018-07-11 07:39:23.058422 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 4] Generated table #36616: 4957166 keys, 80348273 bytes >2018-07-11 07:39:23.058477 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294763058449, "cf_name": "default", "job": 4, "event": "table_file_creation", "file_number": 36616, "file_size": 80348273, "table_properties": {"data_size": 67109348, "index_size": 689231, "filter_size": 12548704, "raw_key_size": 143207013, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957166, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957166", "kMergeOperands": "0"}} >2018-07-11 07:39:25.987766 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 4] Generated table #36617: 2755657 keys, 54285105 bytes >2018-07-11 07:39:25.987811 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294765987783, "cf_name": "default", "job": 4, "event": "table_file_creation", "file_number": 36617, "file_size": 54285105, "table_properties": {"data_size": 46808069, "index_size": 480734, "filter_size": 6995311, "raw_key_size": 79607880, "raw_average_key_size": 28, "raw_value_size": 9493216, "raw_average_value_size": 3, "num_data_blocks": 11493, "num_entries": 2755657, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1957162", "kMergeOperands": "0"}} >2018-07-11 07:39:25.987844 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 4] Compacted 1@1 + 8@2 files to L2 => 506735017 bytes >2018-07-11 07:39:26.000481 7ffabc3f1700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 64611292743 >2018-07-11 07:39:26.000742 7ffabc3f1700 4 rocksdb: (Original Log Time 2018/07/11-07:39:26.000444) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 16 87 446 3641 796 0] max score 5.16, MB/sec: 15.4 rd, 15.4 wr, level 2, files in(1, 8) out(7) MB in(76.6, 406.6) out(483.3), read-write-amplify(12.6) write-amplify(6.3) OK, records in: 30669991, records dropped: 0 > >2018-07-11 07:39:26.000752 7ffabc3f1700 4 rocksdb: (Original Log Time 2018/07/11-07:39:26.000473) EVENT_LOG_v1 {"time_micros": 1531294766000458, "job": 4, "event": "compaction_finished", "compaction_time_micros": 33010382, "output_level": 2, "num_output_files": 7, "total_output_size": 506735017, "num_input_records": 30669991, "num_output_records": 30669991, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 16, 87, 446, 3641, 796, 0]} >2018-07-11 07:39:26.001290 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294766001284, "job": 4, "event": "table_file_deletion", "file_number": 36610} >2018-07-11 07:39:26.001315 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294766001312, "job": 4, "event": "table_file_deletion", "file_number": 36609} >2018-07-11 07:39:26.001347 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294766001341, "job": 4, "event": "table_file_deletion", "file_number": 36563} >2018-07-11 07:39:26.001371 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294766001368, "job": 4, "event": "table_file_deletion", "file_number": 36545} >2018-07-11 07:39:26.001393 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294766001390, "job": 4, "event": "table_file_deletion", "file_number": 36544} >2018-07-11 07:39:26.001420 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294766001416, "job": 4, "event": "table_file_deletion", "file_number": 36538} >2018-07-11 07:39:26.001443 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294766001439, "job": 4, "event": "table_file_deletion", "file_number": 36537} >2018-07-11 07:39:26.001468 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294766001464, "job": 4, "event": "table_file_deletion", "file_number": 36536} >2018-07-11 07:39:26.001491 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294766001487, "job": 4, "event": "table_file_deletion", "file_number": 36535} >2018-07-11 07:39:26.001664 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 5] Compacting 1@1 + 6@2 files to L2, score 5.16 >2018-07-11 07:39:26.001680 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 4 Base level 1, inputs: [36568(76MB)], [36511(76MB) 36512(76MB) 36513(73MB) 36514(76MB) 36515(76MB) 36516(67MB)] > >2018-07-11 07:39:26.001699 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294766001685, "job": 5, "event": "compaction_started", "files_L1": [36568], "files_L2": [36511, 36512, 36513, 36514, 36515, 36516], "score": 5.16434, "input_data_size": 549734018} >2018-07-11 07:39:31.719802 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 5] Generated table #36618: 4957166 keys, 80348309 bytes >2018-07-11 07:39:31.719865 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294771719833, "cf_name": "default", "job": 5, "event": "table_file_creation", "file_number": 36618, "file_size": 80348309, "table_properties": {"data_size": 67109463, "index_size": 689152, "filter_size": 12548704, "raw_key_size": 143207027, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957166, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957166", "kMergeOperands": "0"}} >2018-07-11 07:39:37.333624 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 5] Generated table #36619: 4957165 keys, 80348251 bytes >2018-07-11 07:39:37.333681 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294777333663, "cf_name": "default", "job": 5, "event": "table_file_creation", "file_number": 36619, "file_size": 80348251, "table_properties": {"data_size": 67109324, "index_size": 689236, "filter_size": 12548701, "raw_key_size": 143206997, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957165, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957165", "kMergeOperands": "0"}} >2018-07-11 07:39:41.842419 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 5] Generated table #36620: 3652559 keys, 77083366 bytes >2018-07-11 07:39:41.842469 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294781842441, "cf_name": "default", "job": 5, "event": "table_file_creation", "file_number": 36620, "file_size": 77083366, "table_properties": {"data_size": 67109912, "index_size": 689894, "filter_size": 9282567, "raw_key_size": 105518376, "raw_average_key_size": 28, "raw_value_size": 17645051, "raw_average_value_size": 4, "num_data_blocks": 16481, "num_entries": 3652559, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "2168396", "kMergeOperands": "0"}} >2018-07-11 07:39:47.253276 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 5] Generated table #36621: 4957165 keys, 80348278 bytes >2018-07-11 07:39:47.253312 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294787253296, "cf_name": "default", "job": 5, "event": "table_file_creation", "file_number": 36621, "file_size": 80348278, "table_properties": {"data_size": 67109386, "index_size": 689201, "filter_size": 12548701, "raw_key_size": 143206997, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957165, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957165", "kMergeOperands": "0"}} >2018-07-11 07:39:52.582476 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 5] Generated table #36622: 4957165 keys, 80348252 bytes >2018-07-11 07:39:52.582531 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294792582500, "cf_name": "default", "job": 5, "event": "table_file_creation", "file_number": 36622, "file_size": 80348252, "table_properties": {"data_size": 67109362, "index_size": 689199, "filter_size": 12548701, "raw_key_size": 143206993, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957165, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957165", "kMergeOperands": "0"}} >2018-07-11 07:39:58.117351 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 5] Generated table #36623: 4957165 keys, 80348268 bytes >2018-07-11 07:39:58.117404 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294798117376, "cf_name": "default", "job": 5, "event": "table_file_creation", "file_number": 36623, "file_size": 80348268, "table_properties": {"data_size": 67109371, "index_size": 689206, "filter_size": 12548701, "raw_key_size": 143206992, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957165, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957165", "kMergeOperands": "0"}} >2018-07-11 07:40:02.715083 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 5] Generated table #36624: 4374832 keys, 70909536 bytes >2018-07-11 07:40:02.715122 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294802715106, "cf_name": "default", "job": 5, "event": "table_file_creation", "file_number": 36624, "file_size": 70909536, "table_properties": {"data_size": 59225770, "index_size": 608215, "filter_size": 11074562, "raw_key_size": 126384029, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 14535, "num_entries": 4374832, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4374832", "kMergeOperands": "0"}} >2018-07-11 07:40:02.715166 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 5] Compacted 1@1 + 6@2 files to L2 => 549734260 bytes >2018-07-11 07:40:02.724687 7ffabc3f1700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 64437132305 >2018-07-11 07:40:02.725075 7ffabc3f1700 4 rocksdb: (Original Log Time 2018/07/11-07:40:02.724649) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 15 88 446 3641 796 0] max score 4.57, MB/sec: 15.0 rd, 15.0 wr, level 2, files in(1, 6) out(7) MB in(76.6, 447.6) out(524.3), read-write-amplify(13.7) write-amplify(6.8) OK, records in: 32813217, records dropped: 0 > >2018-07-11 07:40:02.725085 7ffabc3f1700 4 rocksdb: (Original Log Time 2018/07/11-07:40:02.724680) EVENT_LOG_v1 {"time_micros": 1531294802724664, "job": 5, "event": "compaction_finished", "compaction_time_micros": 36713450, "output_level": 2, "num_output_files": 7, "total_output_size": 549734260, "num_input_records": 32813217, "num_output_records": 32813217, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 15, 88, 446, 3641, 796, 0]} >2018-07-11 07:40:02.725608 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294802725604, "job": 5, "event": "table_file_deletion", "file_number": 36568} >2018-07-11 07:40:02.725623 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294802725621, "job": 5, "event": "table_file_deletion", "file_number": 36516} >2018-07-11 07:40:02.725635 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294802725633, "job": 5, "event": "table_file_deletion", "file_number": 36515} >2018-07-11 07:40:02.725647 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294802725645, "job": 5, "event": "table_file_deletion", "file_number": 36514} >2018-07-11 07:40:02.725657 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294802725655, "job": 5, "event": "table_file_deletion", "file_number": 36513} >2018-07-11 07:40:02.725668 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294802725665, "job": 5, "event": "table_file_deletion", "file_number": 36512} >2018-07-11 07:40:02.725681 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294802725678, "job": 5, "event": "table_file_deletion", "file_number": 36511} >2018-07-11 07:40:02.725848 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 6] Compacting 1@1 + 3@2 files to L2, score 4.57 >2018-07-11 07:40:02.725860 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 5 Base level 1, inputs: [36566(76MB)], [36585(76MB) 36586(65MB) 36555(76MB)] > >2018-07-11 07:40:02.725873 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294802725863, "job": 6, "event": "compaction_started", "files_L1": [36566], "files_L2": [36585, 36586, 36555], "score": 4.56955, "input_data_size": 309768793} >2018-07-11 07:40:08.345973 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 6] Generated table #36625: 4957166 keys, 80348298 bytes >2018-07-11 07:40:08.346017 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294808346000, "cf_name": "default", "job": 6, "event": "table_file_creation", "file_number": 36625, "file_size": 80348298, "table_properties": {"data_size": 67109382, "index_size": 689222, "filter_size": 12548704, "raw_key_size": 143207018, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957166, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957166", "kMergeOperands": "0"}} >2018-07-11 07:40:13.839206 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 6] Generated table #36626: 4957165 keys, 80348244 bytes >2018-07-11 07:40:13.839268 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294813839234, "cf_name": "default", "job": 6, "event": "table_file_creation", "file_number": 36626, "file_size": 80348244, "table_properties": {"data_size": 67109319, "index_size": 689234, "filter_size": 12548701, "raw_key_size": 143206997, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957165, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957165", "kMergeOperands": "0"}} >2018-07-11 07:40:19.073586 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 6] Generated table #36627: 4957165 keys, 80348239 bytes >2018-07-11 07:40:19.073641 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294819073611, "cf_name": "default", "job": 6, "event": "table_file_creation", "file_number": 36627, "file_size": 80348239, "table_properties": {"data_size": 67109320, "index_size": 689228, "filter_size": 12548701, "raw_key_size": 143206993, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957165, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957165", "kMergeOperands": "0"}} >2018-07-11 07:40:23.655422 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 6] Generated table #36628: 4239992 keys, 68723984 bytes >2018-07-11 07:40:23.655473 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294823655445, "cf_name": "default", "job": 6, "event": "table_file_creation", "file_number": 36628, "file_size": 68723984, "table_properties": {"data_size": 57400338, "index_size": 589436, "filter_size": 10733221, "raw_key_size": 122488651, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 14087, "num_entries": 4239992, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4239992", "kMergeOperands": "0"}} >2018-07-11 07:40:23.655508 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 6] Compacted 1@1 + 3@2 files to L2 => 309768765 bytes >2018-07-11 07:40:23.666039 7ffabc3f1700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 64227590661 >2018-07-11 07:40:23.666388 7ffabc3f1700 4 rocksdb: (Original Log Time 2018/07/11-07:40:23.666003) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 14 89 446 3641 796 0] max score 3.97, MB/sec: 14.8 rd, 14.8 wr, level 2, files in(1, 3) out(4) MB in(76.6, 218.8) out(295.4), read-write-amplify(7.7) write-amplify(3.9) OK, records in: 19111488, records dropped: 0 > >2018-07-11 07:40:23.666398 7ffabc3f1700 4 rocksdb: (Original Log Time 2018/07/11-07:40:23.666031) EVENT_LOG_v1 {"time_micros": 1531294823666017, "job": 6, "event": "compaction_finished", "compaction_time_micros": 20929619, "output_level": 2, "num_output_files": 4, "total_output_size": 309768765, "num_input_records": 19111488, "num_output_records": 19111488, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 14, 89, 446, 3641, 796, 0]} >2018-07-11 07:40:23.666918 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294823666913, "job": 6, "event": "table_file_deletion", "file_number": 36586} >2018-07-11 07:40:23.666945 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294823666943, "job": 6, "event": "table_file_deletion", "file_number": 36585} >2018-07-11 07:40:23.666957 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294823666955, "job": 6, "event": "table_file_deletion", "file_number": 36566} >2018-07-11 07:40:23.666970 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294823666968, "job": 6, "event": "table_file_deletion", "file_number": 36555} >2018-07-11 07:40:23.667130 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 7] Compacting 1@1 + 8@2 files to L2, score 3.97 >2018-07-11 07:40:23.667143 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 6 Base level 1, inputs: [36567(76MB)], [36628(65MB) 36556(10MB) 36508(76MB) 36526(76MB) 36527(76MB) 36528(76MB) 36529(38MB) 36618(76MB)] > >2018-07-11 07:40:23.667157 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294823667146, "job": 7, "event": "compaction_started", "files_L1": [36567], "files_L2": [36628, 36556, 36508, 36526, 36527, 36528, 36529, 36618], "score": 3.97476, "input_data_size": 601482747} >2018-07-11 07:40:29.172801 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 7] Generated table #36629: 4957165 keys, 80348267 bytes >2018-07-11 07:40:29.172857 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294829172828, "cf_name": "default", "job": 7, "event": "table_file_creation", "file_number": 36629, "file_size": 80348267, "table_properties": {"data_size": 67109329, "index_size": 689247, "filter_size": 12548701, "raw_key_size": 143206998, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957165, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957165", "kMergeOperands": "0"}} >2018-07-11 07:40:34.706660 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 7] Generated table #36630: 4957165 keys, 80348264 bytes >2018-07-11 07:40:34.706713 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294834706685, "cf_name": "default", "job": 7, "event": "table_file_creation", "file_number": 36630, "file_size": 80348264, "table_properties": {"data_size": 67109338, "index_size": 689235, "filter_size": 12548701, "raw_key_size": 143206993, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957165, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957165", "kMergeOperands": "0"}} >2018-07-11 07:40:39.995937 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 7] Generated table #36631: 4957169 keys, 80348334 bytes >2018-07-11 07:40:39.995994 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294839995962, "cf_name": "default", "job": 7, "event": "table_file_creation", "file_number": 36631, "file_size": 80348334, "table_properties": {"data_size": 67109390, "index_size": 689241, "filter_size": 12548713, "raw_key_size": 143207110, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957169, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957169", "kMergeOperands": "0"}} >2018-07-11 07:40:45.494945 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 7] Generated table #36632: 4957165 keys, 80348261 bytes >2018-07-11 07:40:45.495001 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294845494971, "cf_name": "default", "job": 7, "event": "table_file_creation", "file_number": 36632, "file_size": 80348261, "table_properties": {"data_size": 67109364, "index_size": 689206, "filter_size": 12548701, "raw_key_size": 143206996, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957165, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957165", "kMergeOperands": "0"}} >2018-07-11 07:40:50.762601 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 7] Generated table #36633: 4957165 keys, 80348273 bytes >2018-07-11 07:40:50.762644 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294850762627, "cf_name": "default", "job": 7, "event": "table_file_creation", "file_number": 36633, "file_size": 80348273, "table_properties": {"data_size": 67109386, "index_size": 689196, "filter_size": 12548701, "raw_key_size": 143206991, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957165, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957165", "kMergeOperands": "0"}} >2018-07-11 07:40:56.456638 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 7] Generated table #36634: 4957166 keys, 80348258 bytes >2018-07-11 07:40:56.456695 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294856456678, "cf_name": "default", "job": 7, "event": "table_file_creation", "file_number": 36634, "file_size": 80348258, "table_properties": {"data_size": 67109381, "index_size": 689183, "filter_size": 12548704, "raw_key_size": 143207029, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957166, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957166", "kMergeOperands": "0"}} >2018-07-11 07:41:01.928749 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 7] Generated table #36635: 4957165 keys, 80348230 bytes >2018-07-11 07:41:01.928805 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294861928774, "cf_name": "default", "job": 7, "event": "table_file_creation", "file_number": 36635, "file_size": 80348230, "table_properties": {"data_size": 67109361, "index_size": 689178, "filter_size": 12548701, "raw_key_size": 143206995, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957165, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957165", "kMergeOperands": "0"}} >2018-07-11 07:41:04.487267 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 7] Generated table #36636: 2408866 keys, 39044325 bytes >2018-07-11 07:41:04.487301 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294864487285, "cf_name": "default", "job": 7, "event": "table_file_creation", "file_number": 36636, "file_size": 39044325, "table_properties": {"data_size": 32610852, "index_size": 334619, "filter_size": 6097865, "raw_key_size": 69589459, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 8003, "num_entries": 2408866, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "2408866", "kMergeOperands": "0"}} >2018-07-11 07:41:04.487336 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 7] Compacted 1@1 + 8@2 files to L2 => 601482212 bytes >2018-07-11 07:41:04.498380 7ffabc3f1700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 63970666831 >2018-07-11 07:41:04.498785 7ffabc3f1700 4 rocksdb: (Original Log Time 2018/07/11-07:41:04.498344) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 13 89 446 3641 796 0] max score 3.40, MB/sec: 14.7 rd, 14.7 wr, level 2, files in(1, 8) out(8) MB in(76.6, 497.0) out(573.6), read-write-amplify(15.0) write-amplify(7.5) OK, records in: 37109026, records dropped: 0 > >2018-07-11 07:41:04.498795 7ffabc3f1700 4 rocksdb: (Original Log Time 2018/07/11-07:41:04.498372) EVENT_LOG_v1 {"time_micros": 1531294864498358, "job": 7, "event": "compaction_finished", "compaction_time_micros": 40820163, "output_level": 2, "num_output_files": 8, "total_output_size": 601482212, "num_input_records": 37109026, "num_output_records": 37109026, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 13, 89, 446, 3641, 796, 0]} >2018-07-11 07:41:04.499339 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294864499334, "job": 7, "event": "table_file_deletion", "file_number": 36628} >2018-07-11 07:41:04.499355 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294864499353, "job": 7, "event": "table_file_deletion", "file_number": 36618} >2018-07-11 07:41:04.499370 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294864499368, "job": 7, "event": "table_file_deletion", "file_number": 36567} >2018-07-11 07:41:04.499386 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294864499383, "job": 7, "event": "table_file_deletion", "file_number": 36556} >2018-07-11 07:41:04.499399 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294864499397, "job": 7, "event": "table_file_deletion", "file_number": 36529} >2018-07-11 07:41:04.499412 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294864499410, "job": 7, "event": "table_file_deletion", "file_number": 36528} >2018-07-11 07:41:04.499425 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294864499423, "job": 7, "event": "table_file_deletion", "file_number": 36527} >2018-07-11 07:41:04.499439 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294864499435, "job": 7, "event": "table_file_deletion", "file_number": 36526} >2018-07-11 07:41:04.499453 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294864499450, "job": 7, "event": "table_file_deletion", "file_number": 36508} >2018-07-11 07:41:04.499623 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 8] Compacting 1@2 + 2@3 files to L3, score 3.40 >2018-07-11 07:41:04.499635 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 7 Base level 2, inputs: [36631(76MB)], [29890(31MB) 33941(39MB)] > >2018-07-11 07:41:04.499649 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294864499638, "job": 8, "event": "compaction_started", "files_L2": [36631], "files_L3": [29890, 33941], "score": 3.40137, "input_data_size": 154895178} >2018-07-11 07:41:09.565401 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 8] Generated table #36637: 4901134 keys, 80210241 bytes >2018-07-11 07:41:09.565444 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294869565426, "cf_name": "default", "job": 8, "event": "table_file_creation", "file_number": 36637, "file_size": 80210241, "table_properties": {"data_size": 67111545, "index_size": 689273, "filter_size": 12408431, "raw_key_size": 141588326, "raw_average_key_size": 28, "raw_value_size": 759979, "raw_average_value_size": 0, "num_data_blocks": 16471, "num_entries": 4901134, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4837210", "kMergeOperands": "0"}} >2018-07-11 07:41:12.822820 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 8] Generated table #36638: 2694269 keys, 74684263 bytes >2018-07-11 07:41:12.822853 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294872822838, "cf_name": "default", "job": 8, "event": "table_file_creation", "file_number": 36638, "file_size": 74684263, "table_properties": {"data_size": 67109910, "index_size": 689913, "filter_size": 6883448, "raw_key_size": 77834429, "raw_average_key_size": 28, "raw_value_size": 30605771, "raw_average_value_size": 11, "num_data_blocks": 16489, "num_entries": 2694269, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "119952", "kMergeOperands": "0"}} >2018-07-11 07:41:12.865802 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 8] Generated table #36639: 7 keys, 1136 bytes >2018-07-11 07:41:12.865838 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294872865813, "cf_name": "default", "job": 8, "event": "table_file_creation", "file_number": 36639, "file_size": 1136, "table_properties": {"data_size": 117, "index_size": 27, "filter_size": 28, "raw_key_size": 202, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 1, "num_entries": 7, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "7", "kMergeOperands": "0"}} >2018-07-11 07:41:12.865863 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 8] Compacted 1@2 + 2@3 files to L3 => 154895640 bytes >2018-07-11 07:41:12.881647 7ffabc3f1700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 63745240211 >2018-07-11 07:41:12.882020 7ffabc3f1700 4 rocksdb: (Original Log Time 2018/07/11-07:41:12.881614) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 13 88 447 3641 796 0] max score 3.38, MB/sec: 18.5 rd, 18.5 wr, level 3, files in(1, 2) out(3) MB in(76.6, 71.1) out(147.7), read-write-amplify(3.9) write-amplify(1.9) OK, records in: 7595410, records dropped: 0 > >2018-07-11 07:41:12.882029 7ffabc3f1700 4 rocksdb: (Original Log Time 2018/07/11-07:41:12.881639) EVENT_LOG_v1 {"time_micros": 1531294872881628, "job": 8, "event": "compaction_finished", "compaction_time_micros": 8366202, "output_level": 3, "num_output_files": 3, "total_output_size": 154895640, "num_input_records": 7595410, "num_output_records": 7595410, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 13, 88, 447, 3641, 796, 0]} >2018-07-11 07:41:12.882588 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294872882584, "job": 8, "event": "table_file_deletion", "file_number": 36631} >2018-07-11 07:41:12.882616 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294872882614, "job": 8, "event": "table_file_deletion", "file_number": 33941} >2018-07-11 07:41:12.882629 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294872882627, "job": 8, "event": "table_file_deletion", "file_number": 29890} >2018-07-11 07:41:12.882788 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 9] Compacting 1@1 + 1@2 files to L2, score 3.38 >2018-07-11 07:41:12.882799 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 8 Base level 1, inputs: [36569(76MB)], [36624(67MB)] > >2018-07-11 07:41:12.882811 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294872882802, "job": 9, "event": "compaction_started", "files_L1": [36569], "files_L2": [36624], "score": 3.37997, "input_data_size": 151257768} >2018-07-11 07:41:18.185419 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 9] Generated table #36640: 4957167 keys, 80348259 bytes >2018-07-11 07:41:18.185475 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294878185446, "cf_name": "default", "job": 9, "event": "table_file_creation", "file_number": 36640, "file_size": 80348259, "table_properties": {"data_size": 67109325, "index_size": 689237, "filter_size": 12548707, "raw_key_size": 143207052, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 16470, "num_entries": 4957167, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4957167", "kMergeOperands": "0"}} >2018-07-11 07:41:22.540034 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 9] Generated table #36641: 4374830 keys, 70909519 bytes >2018-07-11 07:41:22.540084 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294882540056, "cf_name": "default", "job": 9, "event": "table_file_creation", "file_number": 36641, "file_size": 70909519, "table_properties": {"data_size": 59225843, "index_size": 608130, "filter_size": 11074557, "raw_key_size": 126383986, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 14535, "num_entries": 4374830, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4374830", "kMergeOperands": "0"}} >2018-07-11 07:41:22.540115 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 9] Compacted 1@1 + 1@2 files to L2 => 151257778 bytes >2018-07-11 07:41:22.556396 7ffabc3f1700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 63426701429 >2018-07-11 07:41:22.556604 7ffabc3f1700 4 rocksdb: (Original Log Time 2018/07/11-07:41:22.556362) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 12 89 447 3641 796 0] max score 3.40, MB/sec: 15.7 rd, 15.7 wr, level 2, files in(1, 1) out(2) MB in(76.6, 67.6) out(144.3), read-write-amplify(3.8) write-amplify(1.9) OK, records in: 9331997, records dropped: 0 > >2018-07-11 07:41:22.556614 7ffabc3f1700 4 rocksdb: (Original Log Time 2018/07/11-07:41:22.556388) EVENT_LOG_v1 {"time_micros": 1531294882556377, "job": 9, "event": "compaction_finished", "compaction_time_micros": 9657292, "output_level": 2, "num_output_files": 2, "total_output_size": 151257778, "num_input_records": 9331997, "num_output_records": 9331997, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 12, 89, 447, 3641, 796, 0]} >2018-07-11 07:41:22.557205 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294882557189, "job": 9, "event": "table_file_deletion", "file_number": 36624} >2018-07-11 07:41:22.557221 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294882557219, "job": 9, "event": "table_file_deletion", "file_number": 36569} >2018-07-11 07:41:22.557405 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 10] Compacting 1@2 + 8@3 files to L3, score 3.40 >2018-07-11 07:41:22.557419 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 9 Base level 2, inputs: [36615(76MB)], [16814(70MB) 16815(71MB) 16816(71MB) 16817(71MB) 16820(824KB) 19088(71MB) 19090(71MB) 19091(70MB)] > >2018-07-11 07:41:22.557435 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294882557422, "job": 10, "event": "compaction_started", "files_L2": [36615], "files_L3": [16814, 16815, 16816, 16817, 16820, 19088, 19090, 19091], "score": 3.40137, "input_data_size": 601252623} >2018-07-11 07:41:26.595988 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 10] Generated table #36642: 3106038 keys, 75716796 bytes >2018-07-11 07:41:26.596033 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294886596007, "cf_name": "default", "job": 10, "event": "table_file_creation", "file_number": 36642, "file_size": 75716796, "table_properties": {"data_size": 67111674, "index_size": 689797, "filter_size": 7914333, "raw_key_size": 89729992, "raw_average_key_size": 28, "raw_value_size": 25038456, "raw_average_value_size": 8, "num_data_blocks": 16486, "num_entries": 3106038, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1000000", "kMergeOperands": "0"}} >2018-07-11 07:41:27.378949 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 10] Generated table #36643: 502370 keys, 14195387 bytes >2018-07-11 07:41:27.378991 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294887378976, "cf_name": "default", "job": 10, "event": "table_file_creation", "file_number": 36643, "file_size": 14195387, "table_properties": {"data_size": 12779417, "index_size": 130957, "filter_size": 1284027, "raw_key_size": 14512909, "raw_average_key_size": 28, "raw_value_size": 5972619, "raw_average_value_size": 11, "num_data_blocks": 3140, "num_entries": 502370, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:41:31.237726 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 10] Generated table #36644: 3106038 keys, 75717300 bytes >2018-07-11 07:41:31.237772 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294891237744, "cf_name": "default", "job": 10, "event": "table_file_creation", "file_number": 36644, "file_size": 75717300, "table_properties": {"data_size": 67111710, "index_size": 690265, "filter_size": 7914333, "raw_key_size": 89729994, "raw_average_key_size": 28, "raw_value_size": 25038458, "raw_average_value_size": 8, "num_data_blocks": 16486, "num_entries": 3106038, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1000000", "kMergeOperands": "0"}} >2018-07-11 07:41:34.522763 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 10] Generated table #36645: 2638239 keys, 74546612 bytes >2018-07-11 07:41:34.522810 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294894522782, "cf_name": "default", "job": 10, "event": "table_file_creation", "file_number": 36645, "file_size": 74546612, "table_properties": {"data_size": 67112156, "index_size": 690286, "filter_size": 6743180, "raw_key_size": 76215790, "raw_average_key_size": 28, "raw_value_size": 31365727, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638239, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:41:37.718572 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 10] Generated table #36646: 2638241 keys, 74546532 bytes >2018-07-11 07:41:37.718617 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294897718589, "cf_name": "default", "job": 10, "event": "table_file_creation", "file_number": 36646, "file_size": 74546532, "table_properties": {"data_size": 67112174, "index_size": 690184, "filter_size": 6743184, "raw_key_size": 76215849, "raw_average_key_size": 28, "raw_value_size": 31365752, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:41:40.585258 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 10] Generated table #36647: 2562036 keys, 48298160 bytes >2018-07-11 07:41:40.585303 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294900585275, "cf_name": "default", "job": 10, "event": "table_file_creation", "file_number": 36647, "file_size": 48298160, "table_properties": {"data_size": 41372917, "index_size": 424868, "filter_size": 6499384, "raw_key_size": 74014378, "raw_average_key_size": 28, "raw_value_size": 6681986, "raw_average_value_size": 2, "num_data_blocks": 10158, "num_entries": 2562036, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "2000000", "kMergeOperands": "0"}} >2018-07-11 07:41:43.807520 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 10] Generated table #36648: 2638240 keys, 74545876 bytes >2018-07-11 07:41:43.807564 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294903807536, "cf_name": "default", "job": 10, "event": "table_file_creation", "file_number": 36648, "file_size": 74545876, "table_properties": {"data_size": 67112110, "index_size": 689595, "filter_size": 6743181, "raw_key_size": 76215818, "raw_average_key_size": 28, "raw_value_size": 31365738, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:41:46.988113 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 10] Generated table #36649: 2638241 keys, 74546035 bytes >2018-07-11 07:41:46.988159 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294906988130, "cf_name": "default", "job": 10, "event": "table_file_creation", "file_number": 36649, "file_size": 74546035, "table_properties": {"data_size": 67112157, "index_size": 689704, "filter_size": 6743184, "raw_key_size": 76215861, "raw_average_key_size": 28, "raw_value_size": 31365764, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:41:49.591169 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 10] Generated table #36650: 1648523 keys, 46580917 bytes >2018-07-11 07:41:49.591212 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294909591185, "cf_name": "default", "job": 10, "event": "table_file_creation", "file_number": 36650, "file_size": 46580917, "table_properties": {"data_size": 41935471, "index_size": 430937, "filter_size": 4213521, "raw_key_size": 47624002, "raw_average_key_size": 28, "raw_value_size": 19599111, "raw_average_value_size": 11, "num_data_blocks": 10304, "num_entries": 1648523, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:41:49.591251 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 10] Compacted 1@2 + 8@3 files to L3 => 558693615 bytes >2018-07-11 07:41:49.605948 7ffabc3f1700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 62752292078 >2018-07-11 07:41:49.606135 7ffabc3f1700 4 rocksdb: (Original Log Time 2018/07/11-07:41:49.605913) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 12 88 448 3641 796 0] max score 3.34, MB/sec: 22.2 rd, 20.7 wr, level 3, files in(1, 8) out(9) MB in(76.6, 496.8) out(532.8), read-write-amplify(14.4) write-amplify(7.0) OK, records in: 23392302, records dropped: 1914336 > >2018-07-11 07:41:49.606145 7ffabc3f1700 4 rocksdb: (Original Log Time 2018/07/11-07:41:49.605939) EVENT_LOG_v1 {"time_micros": 1531294909605928, "job": 10, "event": "compaction_finished", "compaction_time_micros": 27033794, "output_level": 3, "num_output_files": 9, "total_output_size": 558693615, "num_input_records": 23392302, "num_output_records": 21477966, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 12, 88, 448, 3641, 796, 0]} >2018-07-11 07:41:49.606743 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294909606739, "job": 10, "event": "table_file_deletion", "file_number": 36615} >2018-07-11 07:41:49.606773 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294909606770, "job": 10, "event": "table_file_deletion", "file_number": 19091} >2018-07-11 07:41:49.606784 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294909606782, "job": 10, "event": "table_file_deletion", "file_number": 19090} >2018-07-11 07:41:49.606798 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294909606796, "job": 10, "event": "table_file_deletion", "file_number": 19088} >2018-07-11 07:41:49.606810 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294909606808, "job": 10, "event": "table_file_deletion", "file_number": 16820} >2018-07-11 07:41:49.606820 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294909606818, "job": 10, "event": "table_file_deletion", "file_number": 16817} >2018-07-11 07:41:49.606834 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294909606831, "job": 10, "event": "table_file_deletion", "file_number": 16816} >2018-07-11 07:41:49.606847 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294909606844, "job": 10, "event": "table_file_deletion", "file_number": 16815} >2018-07-11 07:41:49.606860 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294909606858, "job": 10, "event": "table_file_deletion", "file_number": 16814} >2018-07-11 07:41:49.607044 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 11] Compacting 1@2 + 15@3 files to L3, score 3.34 >2018-07-11 07:41:49.607061 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 10 Base level 2, inputs: [36607(76MB)], [8474(71MB) 8479(1088B) 8663(71MB) 8664(30MB) 8669(40MB) 11314(71MB) 9148(71MB) 9150(30MB) 9068(71MB) 9072(1113B) 11315(71MB) 9078(71MB) 9198(71MB) 9073(71MB) 9394(71MB)] > >2018-07-11 07:41:49.607078 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294909607066, "job": 11, "event": "compaction_started", "files_L2": [36607], "files_L3": [8474, 8479, 8663, 8664, 8669, 11314, 9148, 9150, 9068, 9072, 11315, 9078, 9198, 9073, 9394], "score": 3.34189, "input_data_size": 932428643} >2018-07-11 07:41:53.392922 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 11] Generated table #36651: 3106037 keys, 75717117 bytes >2018-07-11 07:41:53.392967 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294913392941, "cf_name": "default", "job": 11, "event": "table_file_creation", "file_number": 36651, "file_size": 75717117, "table_properties": {"data_size": 67111700, "index_size": 690094, "filter_size": 7914331, "raw_key_size": 89729963, "raw_average_key_size": 28, "raw_value_size": 25038444, "raw_average_value_size": 8, "num_data_blocks": 16486, "num_entries": 3106037, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1000000", "kMergeOperands": "0"}} >2018-07-11 07:41:55.158445 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 11] Generated table #36652: 1532205 keys, 31246768 bytes >2018-07-11 07:41:55.158488 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294915158460, "cf_name": "default", "job": 11, "event": "table_file_creation", "file_number": 36652, "file_size": 31246768, "table_properties": {"data_size": 27076242, "index_size": 277828, "filter_size": 3891708, "raw_key_size": 44263707, "raw_average_key_size": 28, "raw_value_size": 6327332, "raw_average_value_size": 4, "num_data_blocks": 6649, "num_entries": 1532205, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1000000", "kMergeOperands": "0"}} >2018-07-11 07:41:58.389628 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 11] Generated table #36653: 2638240 keys, 74546009 bytes >2018-07-11 07:41:58.389673 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294918389646, "cf_name": "default", "job": 11, "event": "table_file_creation", "file_number": 36653, "file_size": 74546009, "table_properties": {"data_size": 67112108, "index_size": 689730, "filter_size": 6743181, "raw_key_size": 76215829, "raw_average_key_size": 28, "raw_value_size": 31365749, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:42:01.642628 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 11] Generated table #36654: 2638240 keys, 74546077 bytes >2018-07-11 07:42:01.642673 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294921642646, "cf_name": "default", "job": 11, "event": "table_file_creation", "file_number": 36654, "file_size": 74546077, "table_properties": {"data_size": 67112114, "index_size": 689792, "filter_size": 6743181, "raw_key_size": 76215826, "raw_average_key_size": 28, "raw_value_size": 31365746, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:42:04.885560 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 11] Generated table #36655: 2638240 keys, 74546090 bytes >2018-07-11 07:42:04.885606 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294924885578, "cf_name": "default", "job": 11, "event": "table_file_creation", "file_number": 36655, "file_size": 74546090, "table_properties": {"data_size": 67112102, "index_size": 689817, "filter_size": 6743181, "raw_key_size": 76215819, "raw_average_key_size": 28, "raw_value_size": 31365739, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:42:06.045580 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 11] Generated table #36656: 1000005 keys, 16209050 bytes >2018-07-11 07:42:06.045621 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294926045595, "cf_name": "default", "job": 11, "event": "table_file_creation", "file_number": 36656, "file_size": 16209050, "table_properties": {"data_size": 13537978, "index_size": 138643, "filter_size": 2531442, "raw_key_size": 28889035, "raw_average_key_size": 28, "raw_value_size": 60, "raw_average_value_size": 0, "num_data_blocks": 3323, "num_entries": 1000005, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1000000", "kMergeOperands": "0"}} >2018-07-11 07:42:09.270659 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 11] Generated table #36657: 2638241 keys, 74545843 bytes >2018-07-11 07:42:09.270706 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294929270678, "cf_name": "default", "job": 11, "event": "table_file_creation", "file_number": 36657, "file_size": 74545843, "table_properties": {"data_size": 67112089, "index_size": 689581, "filter_size": 6743183, "raw_key_size": 76215848, "raw_average_key_size": 28, "raw_value_size": 31365751, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:42:12.531670 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 11] Generated table #36658: 2638241 keys, 74546804 bytes >2018-07-11 07:42:12.531713 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294932531687, "cf_name": "default", "job": 11, "event": "table_file_creation", "file_number": 36658, "file_size": 74546804, "table_properties": {"data_size": 67112154, "index_size": 690476, "filter_size": 6743184, "raw_key_size": 76215845, "raw_average_key_size": 28, "raw_value_size": 31365748, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:42:15.728871 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 11] Generated table #36659: 2638239 keys, 74546212 bytes >2018-07-11 07:42:15.728916 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294935728889, "cf_name": "default", "job": 11, "event": "table_file_creation", "file_number": 36659, "file_size": 74546212, "table_properties": {"data_size": 67112126, "index_size": 689917, "filter_size": 6743179, "raw_key_size": 76215804, "raw_average_key_size": 28, "raw_value_size": 31365741, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638239, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:42:18.938200 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 11] Generated table #36660: 2638239 keys, 74546553 bytes >2018-07-11 07:42:18.938249 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294938938218, "cf_name": "default", "job": 11, "event": "table_file_creation", "file_number": 36660, "file_size": 74546553, "table_properties": {"data_size": 67112121, "index_size": 690263, "filter_size": 6743179, "raw_key_size": 76215790, "raw_average_key_size": 28, "raw_value_size": 31365727, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638239, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:42:20.818970 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 11] Generated table #36661: 1482240 keys, 41882465 bytes >2018-07-11 07:42:20.819010 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294940818985, "cf_name": "default", "job": 11, "event": "table_file_creation", "file_number": 36661, "file_size": 41882465, "table_properties": {"data_size": 37705589, "index_size": 387367, "filter_size": 3788521, "raw_key_size": 42820262, "raw_average_key_size": 28, "raw_value_size": 17622182, "raw_average_value_size": 11, "num_data_blocks": 9265, "num_entries": 1482240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:42:24.372297 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 11] Generated table #36662: 2943327 keys, 75308790 bytes >2018-07-11 07:42:24.372343 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294944372316, "cf_name": "default", "job": 11, "event": "table_file_creation", "file_number": 36662, "file_size": 75308790, "table_properties": {"data_size": 67110292, "index_size": 690531, "filter_size": 7506975, "raw_key_size": 85029443, "raw_average_key_size": 28, "raw_value_size": 27237649, "raw_average_value_size": 9, "num_data_blocks": 16487, "num_entries": 2943327, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "652310", "kMergeOperands": "0"}} >2018-07-11 07:42:27.591554 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 11] Generated table #36663: 2800791 keys, 74950646 bytes >2018-07-11 07:42:27.591591 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294947591575, "cf_name": "default", "job": 11, "event": "table_file_creation", "file_number": 36663, "file_size": 74950646, "table_properties": {"data_size": 67109498, "index_size": 690026, "filter_size": 7150130, "raw_key_size": 80911764, "raw_average_key_size": 28, "raw_value_size": 29164662, "raw_average_value_size": 10, "num_data_blocks": 16488, "num_entries": 2800791, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "347690", "kMergeOperands": "0"}} >2018-07-11 07:42:30.528693 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 11] Generated table #36664: 1866212 keys, 52732218 bytes >2018-07-11 07:42:30.528736 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294950528709, "cf_name": "default", "job": 11, "event": "table_file_creation", "file_number": 36664, "file_size": 52732218, "table_properties": {"data_size": 47473156, "index_size": 488153, "filter_size": 4769921, "raw_key_size": 53912801, "raw_average_key_size": 28, "raw_value_size": 22187197, "raw_average_value_size": 11, "num_data_blocks": 11664, "num_entries": 1866212, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:42:30.528776 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 11] Compacted 1@2 + 15@3 files to L3 => 889870642 bytes >2018-07-11 07:42:30.538352 7ffabc3f1700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 62072769235 >2018-07-11 07:42:30.538563 7ffabc3f1700 4 rocksdb: (Original Log Time 2018/07/11-07:42:30.538320) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 12 87 447 3641 796 0] max score 3.28, MB/sec: 22.8 rd, 21.7 wr, level 3, files in(1, 15) out(14) MB in(76.6, 812.6) out(848.6), read-write-amplify(22.7) write-amplify(11.1) OK, records in: 35112833, records dropped: 1914336 > >2018-07-11 07:42:30.538572 7ffabc3f1700 4 rocksdb: (Original Log Time 2018/07/11-07:42:30.538344) EVENT_LOG_v1 {"time_micros": 1531294950538333, "job": 11, "event": "compaction_finished", "compaction_time_micros": 40921675, "output_level": 3, "num_output_files": 14, "total_output_size": 889870642, "num_input_records": 35112833, "num_output_records": 33198497, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 12, 87, 447, 3641, 796, 0]} >2018-07-11 07:42:30.539138 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294950539134, "job": 11, "event": "table_file_deletion", "file_number": 36607} >2018-07-11 07:42:30.539169 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294950539165, "job": 11, "event": "table_file_deletion", "file_number": 11315} >2018-07-11 07:42:30.539180 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294950539178, "job": 11, "event": "table_file_deletion", "file_number": 11314} >2018-07-11 07:42:30.539192 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294950539190, "job": 11, "event": "table_file_deletion", "file_number": 9394} >2018-07-11 07:42:30.539203 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294950539201, "job": 11, "event": "table_file_deletion", "file_number": 9198} >2018-07-11 07:42:30.539220 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294950539218, "job": 11, "event": "table_file_deletion", "file_number": 9150} >2018-07-11 07:42:30.539233 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294950539231, "job": 11, "event": "table_file_deletion", "file_number": 9148} >2018-07-11 07:42:30.539253 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294950539244, "job": 11, "event": "table_file_deletion", "file_number": 9078} >2018-07-11 07:42:30.539267 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294950539265, "job": 11, "event": "table_file_deletion", "file_number": 9073} >2018-07-11 07:42:30.539280 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294950539278, "job": 11, "event": "table_file_deletion", "file_number": 9072} >2018-07-11 07:42:30.539292 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294950539290, "job": 11, "event": "table_file_deletion", "file_number": 9068} >2018-07-11 07:42:30.539306 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294950539304, "job": 11, "event": "table_file_deletion", "file_number": 8669} >2018-07-11 07:42:30.539316 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294950539314, "job": 11, "event": "table_file_deletion", "file_number": 8664} >2018-07-11 07:42:30.539326 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294950539324, "job": 11, "event": "table_file_deletion", "file_number": 8663} >2018-07-11 07:42:30.539339 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294950539337, "job": 11, "event": "table_file_deletion", "file_number": 8479} >2018-07-11 07:42:30.539353 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294950539351, "job": 11, "event": "table_file_deletion", "file_number": 8474} >2018-07-11 07:42:30.539522 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 12] Compacting 1@2 + 2@3 files to L3, score 3.28 >2018-07-11 07:42:30.539534 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 11 Base level 2, inputs: [36590(76MB)], [20434(39MB) 20435(71MB)] > >2018-07-11 07:42:30.539547 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294950539537, "job": 12, "event": "compaction_started", "files_L2": [36590], "files_L3": [20434, 20435], "score": 3.28241, "input_data_size": 196772126} >2018-07-11 07:42:34.604355 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 12] Generated table #36665: 3106038 keys, 75717272 bytes >2018-07-11 07:42:34.604402 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294954604374, "cf_name": "default", "job": 12, "event": "table_file_creation", "file_number": 36665, "file_size": 75717272, "table_properties": {"data_size": 67111709, "index_size": 690238, "filter_size": 7914333, "raw_key_size": 89729986, "raw_average_key_size": 28, "raw_value_size": 25038450, "raw_average_value_size": 8, "num_data_blocks": 16486, "num_entries": 3106038, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1000000", "kMergeOperands": "0"}} >2018-07-11 07:42:38.649037 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 12] Generated table #36666: 3573838 keys, 76887560 bytes >2018-07-11 07:42:38.649072 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294958649056, "cf_name": "default", "job": 12, "event": "table_file_creation", "file_number": 36666, "file_size": 76887560, "table_properties": {"data_size": 67111372, "index_size": 689706, "filter_size": 9085490, "raw_key_size": 103244217, "raw_average_key_size": 28, "raw_value_size": 18711191, "raw_average_value_size": 5, "num_data_blocks": 16482, "num_entries": 3573838, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "2000000", "kMergeOperands": "0"}} >2018-07-11 07:42:41.191638 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 12] Generated table #36667: 2397614 keys, 44168242 bytes >2018-07-11 07:42:41.191682 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294961191655, "cf_name": "default", "job": 12, "event": "table_file_creation", "file_number": 36667, "file_size": 44168242, "table_properties": {"data_size": 37700077, "index_size": 386988, "filter_size": 6080186, "raw_key_size": 69264401, "raw_average_key_size": 28, "raw_value_size": 5236416, "raw_average_value_size": 2, "num_data_blocks": 9256, "num_entries": 2397614, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1957168", "kMergeOperands": "0"}} >2018-07-11 07:42:41.191711 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 12] Compacted 1@2 + 2@3 files to L3 => 196773074 bytes >2018-07-11 07:42:41.204920 7ffabc3f1700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 61837675227 >2018-07-11 07:42:41.205128 7ffabc3f1700 4 rocksdb: (Original Log Time 2018/07/11-07:42:41.204886) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 12 86 448 3641 796 0] max score 3.22, MB/sec: 18.5 rd, 18.5 wr, level 3, files in(1, 2) out(3) MB in(76.6, 111.0) out(187.7), read-write-amplify(4.9) write-amplify(2.4) OK, records in: 9077490, records dropped: 0 > >2018-07-11 07:42:41.205137 7ffabc3f1700 4 rocksdb: (Original Log Time 2018/07/11-07:42:41.204912) EVENT_LOG_v1 {"time_micros": 1531294961204901, "job": 12, "event": "compaction_finished", "compaction_time_micros": 10652151, "output_level": 3, "num_output_files": 3, "total_output_size": 196773074, "num_input_records": 9077490, "num_output_records": 9077490, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 12, 86, 448, 3641, 796, 0]} >2018-07-11 07:42:41.205744 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294961205738, "job": 12, "event": "table_file_deletion", "file_number": 36590} >2018-07-11 07:42:41.205773 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294961205771, "job": 12, "event": "table_file_deletion", "file_number": 20435} >2018-07-11 07:42:41.205784 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294961205782, "job": 12, "event": "table_file_deletion", "file_number": 20434} >2018-07-11 07:42:41.205950 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 13] Compacting 1@2 + 7@3 files to L3, score 3.22 >2018-07-11 07:42:41.205963 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 12 Base level 2, inputs: [36611(76MB)], [12990(39MB) 12994(71MB) 12997(329KB) 15164(70MB) 15166(71MB) 15167(18KB) 15169(71MB)] > >2018-07-11 07:42:41.205977 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294961205967, "job": 13, "event": "compaction_started", "files_L2": [36611], "files_L3": [12990, 12994, 12997, 15164, 15166, 15167, 15169], "score": 3.22293, "input_data_size": 420154912} >2018-07-11 07:42:45.603559 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 13] Generated table #36668: 3481924 keys, 74290560 bytes >2018-07-11 07:42:45.603607 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294965603579, "cf_name": "default", "job": 13, "event": "table_file_creation", "file_number": 36668, "file_size": 74290560, "table_properties": {"data_size": 64773201, "index_size": 665816, "filter_size": 8850552, "raw_key_size": 100588928, "raw_average_key_size": 28, "raw_value_size": 17618440, "raw_average_value_size": 5, "num_data_blocks": 15907, "num_entries": 3481924, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "2000000", "kMergeOperands": "0"}} >2018-07-11 07:42:49.129336 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 13] Generated table #36669: 3106037 keys, 75717153 bytes >2018-07-11 07:42:49.129382 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294969129355, "cf_name": "default", "job": 13, "event": "table_file_creation", "file_number": 36669, "file_size": 75717153, "table_properties": {"data_size": 67111706, "index_size": 690124, "filter_size": 7914331, "raw_key_size": 89729964, "raw_average_key_size": 28, "raw_value_size": 25038445, "raw_average_value_size": 8, "num_data_blocks": 16486, "num_entries": 3106037, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1000000", "kMergeOperands": "0"}} >2018-07-11 07:42:50.999355 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 13] Generated table #36670: 1482238 keys, 41882633 bytes >2018-07-11 07:42:50.999395 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294970999370, "cf_name": "default", "job": 13, "event": "table_file_creation", "file_number": 36670, "file_size": 41882633, "table_properties": {"data_size": 37705537, "index_size": 387592, "filter_size": 3788516, "raw_key_size": 42820206, "raw_average_key_size": 28, "raw_value_size": 17622160, "raw_average_value_size": 11, "num_data_blocks": 9265, "num_entries": 1482238, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:42:53.081563 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 13] Generated table #36671: 1678620 keys, 47431273 bytes >2018-07-11 07:42:53.081609 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294973081582, "cf_name": "default", "job": 13, "event": "table_file_creation", "file_number": 36671, "file_size": 47431273, "table_properties": {"data_size": 42701094, "index_size": 438744, "filter_size": 4290447, "raw_key_size": 48493469, "raw_average_key_size": 28, "raw_value_size": 19956929, "raw_average_value_size": 11, "num_data_blocks": 10492, "num_entries": 1678620, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:42:56.550848 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 13] Generated table #36672: 3106037 keys, 75717352 bytes >2018-07-11 07:42:56.550896 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294976550868, "cf_name": "default", "job": 13, "event": "table_file_creation", "file_number": 36672, "file_size": 75717352, "table_properties": {"data_size": 67111719, "index_size": 690310, "filter_size": 7914331, "raw_key_size": 89729963, "raw_average_key_size": 28, "raw_value_size": 25038444, "raw_average_value_size": 8, "num_data_blocks": 16486, "num_entries": 3106037, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1000000", "kMergeOperands": "0"}} >2018-07-11 07:42:59.763863 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 13] Generated table #36673: 2638241 keys, 74546379 bytes >2018-07-11 07:42:59.763909 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294979763883, "cf_name": "default", "job": 13, "event": "table_file_creation", "file_number": 36673, "file_size": 74546379, "table_properties": {"data_size": 67112176, "index_size": 690030, "filter_size": 6743183, "raw_key_size": 76215849, "raw_average_key_size": 28, "raw_value_size": 31365752, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:43:00.434842 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 13] Generated table #36674: 532824 keys, 15055956 bytes >2018-07-11 07:43:00.434882 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294980434856, "cf_name": "default", "job": 13, "event": "table_file_creation", "file_number": 36674, "file_size": 15055956, "table_properties": {"data_size": 13554089, "index_size": 139013, "filter_size": 1361868, "raw_key_size": 15392695, "raw_average_key_size": 28, "raw_value_size": 6334687, "raw_average_value_size": 11, "num_data_blocks": 3331, "num_entries": 532824, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:43:01.453318 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 13] Generated table #36675: 957168 keys, 15514645 bytes >2018-07-11 07:43:01.453359 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294981453333, "cf_name": "default", "job": 13, "event": "table_file_creation", "file_number": 36675, "file_size": 15514645, "table_properties": {"data_size": 12958005, "index_size": 132650, "filter_size": 2423003, "raw_key_size": 27651513, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 3180, "num_entries": 957168, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "957168", "kMergeOperands": "0"}} >2018-07-11 07:43:01.453388 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 13] Compacted 1@2 + 7@3 files to L3 => 420155951 bytes >2018-07-11 07:43:01.462922 7ffabc3f1700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 61597544288 >2018-07-11 07:43:01.463295 7ffabc3f1700 4 rocksdb: (Original Log Time 2018/07/11-07:43:01.462887) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 12 85 449 3641 796 0] max score 3.16, MB/sec: 20.8 rd, 20.8 wr, level 3, files in(1, 7) out(8) MB in(76.6, 324.1) out(400.7), read-write-amplify(10.5) write-amplify(5.2) OK, records in: 16983089, records dropped: 0 > >2018-07-11 07:43:01.463306 7ffabc3f1700 4 rocksdb: (Original Log Time 2018/07/11-07:43:01.462914) EVENT_LOG_v1 {"time_micros": 1531294981462902, "job": 13, "event": "compaction_finished", "compaction_time_micros": 20247396, "output_level": 3, "num_output_files": 8, "total_output_size": 420155951, "num_input_records": 16983089, "num_output_records": 16983089, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 12, 85, 449, 3641, 796, 0]} >2018-07-11 07:43:01.463840 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294981463834, "job": 13, "event": "table_file_deletion", "file_number": 36611} >2018-07-11 07:43:01.463870 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294981463868, "job": 13, "event": "table_file_deletion", "file_number": 15169} >2018-07-11 07:43:01.463881 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294981463879, "job": 13, "event": "table_file_deletion", "file_number": 15167} >2018-07-11 07:43:01.463891 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294981463889, "job": 13, "event": "table_file_deletion", "file_number": 15166} >2018-07-11 07:43:01.463901 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294981463900, "job": 13, "event": "table_file_deletion", "file_number": 15164} >2018-07-11 07:43:01.463913 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294981463911, "job": 13, "event": "table_file_deletion", "file_number": 12997} >2018-07-11 07:43:01.463932 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294981463930, "job": 13, "event": "table_file_deletion", "file_number": 12994} >2018-07-11 07:43:01.463946 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294981463942, "job": 13, "event": "table_file_deletion", "file_number": 12990} >2018-07-11 07:43:01.464117 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 14] Compacting 1@2 + 9@3 files to L3, score 3.16 >2018-07-11 07:43:01.464131 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 13 Base level 2, inputs: [36601(76MB)], [5055(71MB) 5060(71MB) 5062(1064B) 6095(71MB) 6098(71MB) 6100(22MB) 6103(71MB) 6108(37MB) 7374(39MB)] > >2018-07-11 07:43:01.464145 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294981464134, "job": 14, "event": "compaction_started", "files_L2": [36601], "files_L3": [5055, 5060, 5062, 6095, 6098, 6100, 6103, 6108, 7374], "score": 3.16345, "input_data_size": 557071488} >2018-07-11 07:43:05.302409 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 14] Generated table #36676: 3106038 keys, 75717438 bytes >2018-07-11 07:43:05.302457 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294985302429, "cf_name": "default", "job": 14, "event": "table_file_creation", "file_number": 36676, "file_size": 75717438, "table_properties": {"data_size": 67111729, "index_size": 690384, "filter_size": 7914333, "raw_key_size": 89729987, "raw_average_key_size": 28, "raw_value_size": 25038451, "raw_average_value_size": 8, "num_data_blocks": 16486, "num_entries": 3106038, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1000000", "kMergeOperands": "0"}} >2018-07-11 07:43:08.435068 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 14] Generated table #36677: 2638240 keys, 74546377 bytes >2018-07-11 07:43:08.435114 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294988435085, "cf_name": "default", "job": 14, "event": "table_file_creation", "file_number": 36677, "file_size": 74546377, "table_properties": {"data_size": 67112123, "index_size": 690083, "filter_size": 6743181, "raw_key_size": 76215828, "raw_average_key_size": 28, "raw_value_size": 31365748, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:43:09.225758 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 14] Generated table #36678: 532205 keys, 15038507 bytes >2018-07-11 07:43:09.225800 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294989225774, "cf_name": "default", "job": 14, "event": "table_file_creation", "file_number": 36678, "file_size": 15038507, "table_properties": {"data_size": 13538366, "index_size": 138870, "filter_size": 1360285, "raw_key_size": 15374808, "raw_average_key_size": 28, "raw_value_size": 6327323, "raw_average_value_size": 11, "num_data_blocks": 3327, "num_entries": 532205, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:43:11.269229 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 14] Generated table #36679: 2000000 keys, 32417287 bytes >2018-07-11 07:43:11.269276 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294991269250, "cf_name": "default", "job": 14, "event": "table_file_creation", "file_number": 36679, "file_size": 32417287, "table_properties": {"data_size": 27075674, "index_size": 277774, "filter_size": 5062852, "raw_key_size": 57777780, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 6645, "num_entries": 2000000, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "2000000", "kMergeOperands": "0"}} >2018-07-11 07:43:14.522580 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 14] Generated table #36680: 2638241 keys, 74545923 bytes >2018-07-11 07:43:14.522627 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294994522599, "cf_name": "default", "job": 14, "event": "table_file_creation", "file_number": 36680, "file_size": 74545923, "table_properties": {"data_size": 67112182, "index_size": 689568, "filter_size": 6743183, "raw_key_size": 76215864, "raw_average_key_size": 28, "raw_value_size": 31365767, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:43:18.512775 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 14] Generated table #36681: 2638239 keys, 74546111 bytes >2018-07-11 07:43:18.512818 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531294998512791, "cf_name": "default", "job": 14, "event": "table_file_creation", "file_number": 36681, "file_size": 74546111, "table_properties": {"data_size": 67112120, "index_size": 689822, "filter_size": 6743179, "raw_key_size": 76215802, "raw_average_key_size": 28, "raw_value_size": 31365739, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638239, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:43:21.728064 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 14] Generated table #36682: 2638240 keys, 74546194 bytes >2018-07-11 07:43:21.728107 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295001728081, "cf_name": "default", "job": 14, "event": "table_file_creation", "file_number": 36682, "file_size": 74546194, "table_properties": {"data_size": 67112149, "index_size": 689874, "filter_size": 6743181, "raw_key_size": 76215822, "raw_average_key_size": 28, "raw_value_size": 31365742, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:43:23.274969 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 14] Generated table #36683: 1198233 keys, 33857421 bytes >2018-07-11 07:43:23.275010 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295003274984, "cf_name": "default", "job": 14, "event": "table_file_creation", "file_number": 36683, "file_size": 33857421, "table_properties": {"data_size": 30480906, "index_size": 312918, "filter_size": 3062609, "raw_key_size": 34615618, "raw_average_key_size": 28, "raw_value_size": 14245657, "raw_average_value_size": 11, "num_data_blocks": 7489, "num_entries": 1198233, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:43:25.961203 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 14] Generated table #36684: 2439247 keys, 57392025 bytes >2018-07-11 07:43:25.961254 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295005961221, "cf_name": "default", "job": 14, "event": "table_file_creation", "file_number": 36684, "file_size": 57392025, "table_properties": {"data_size": 50659463, "index_size": 520471, "filter_size": 6211100, "raw_key_size": 70467125, "raw_average_key_size": 28, "raw_value_size": 17620281, "raw_average_value_size": 7, "num_data_blocks": 12443, "num_entries": 2439247, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "957167", "kMergeOperands": "0"}} >2018-07-11 07:43:25.961290 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 14] Compacted 1@2 + 9@3 files to L3 => 512607283 bytes >2018-07-11 07:43:25.979203 7ffabc3f1700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 60882811104 >2018-07-11 07:43:25.979467 7ffabc3f1700 4 rocksdb: (Original Log Time 2018/07/11-07:43:25.979169) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 12 84 449 3641 796 0] max score 3.10, MB/sec: 22.7 rd, 20.9 wr, level 3, files in(1, 9) out(9) MB in(76.6, 454.6) out(488.9), read-write-amplify(13.3) write-amplify(6.4) OK, records in: 21828683, records dropped: 2000000 > >2018-07-11 07:43:25.979477 7ffabc3f1700 4 rocksdb: (Original Log Time 2018/07/11-07:43:25.979195) EVENT_LOG_v1 {"time_micros": 1531295005979183, "job": 14, "event": "compaction_finished", "compaction_time_micros": 24497126, "output_level": 3, "num_output_files": 9, "total_output_size": 512607283, "num_input_records": 21828683, "num_output_records": 19828683, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 12, 84, 449, 3641, 796, 0]} >2018-07-11 07:43:25.980069 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295005980065, "job": 14, "event": "table_file_deletion", "file_number": 36601} >2018-07-11 07:43:25.980089 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295005980087, "job": 14, "event": "table_file_deletion", "file_number": 7374} >2018-07-11 07:43:25.980107 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295005980105, "job": 14, "event": "table_file_deletion", "file_number": 6108} >2018-07-11 07:43:25.980121 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295005980119, "job": 14, "event": "table_file_deletion", "file_number": 6103} >2018-07-11 07:43:25.980142 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295005980140, "job": 14, "event": "table_file_deletion", "file_number": 6100} >2018-07-11 07:43:25.980158 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295005980155, "job": 14, "event": "table_file_deletion", "file_number": 6098} >2018-07-11 07:43:25.980170 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295005980168, "job": 14, "event": "table_file_deletion", "file_number": 6095} >2018-07-11 07:43:25.980185 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295005980183, "job": 14, "event": "table_file_deletion", "file_number": 5062} >2018-07-11 07:43:25.980198 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295005980196, "job": 14, "event": "table_file_deletion", "file_number": 5060} >2018-07-11 07:43:25.980211 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295005980208, "job": 14, "event": "table_file_deletion", "file_number": 5055} >2018-07-11 07:43:25.980392 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 15] Compacting 1@2 + 23@3 files to L3, score 3.10 >2018-07-11 07:43:25.980411 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 14 Base level 2, inputs: [36599(76MB)], [3442(71MB) 3462(71MB) 3476(71MB) 3496(71MB) 3513(71MB) 3547(71MB) 3566(71MB) 3603(71MB) 3637(71MB) 3654(71MB) 3712(71MB) 3743(71MB) 3758(71MB) 3794(71MB) 3809(71MB) 4824(1064B) 4652(71MB) 4655(71MB) 4661(71MB) 4662(1170B) 4973(71MB) 4975(23MB) 4976(71MB)] > >2018-07-11 07:43:25.980428 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295005980417, "job": 15, "event": "compaction_started", "files_L2": [36599], "files_L3": [3442, 3462, 3476, 3496, 3513, 3547, 3566, 3603, 3637, 3654, 3712, 3743, 3758, 3794, 3809, 4824, 4652, 4655, 4661, 4662, 4973, 4975, 4976], "score": 3.10397, "input_data_size": 1596392354} >2018-07-11 07:43:29.748324 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 15] Generated table #36685: 3106038 keys, 75717299 bytes >2018-07-11 07:43:29.748359 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295009748344, "cf_name": "default", "job": 15, "event": "table_file_creation", "file_number": 36685, "file_size": 75717299, "table_properties": {"data_size": 67111714, "index_size": 690260, "filter_size": 7914333, "raw_key_size": 89729986, "raw_average_key_size": 28, "raw_value_size": 25038450, "raw_average_value_size": 8, "num_data_blocks": 16486, "num_entries": 3106038, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1000000", "kMergeOperands": "0"}} >2018-07-11 07:43:32.900554 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 15] Generated table #36686: 2638240 keys, 74546308 bytes >2018-07-11 07:43:32.900599 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295012900572, "cf_name": "default", "job": 15, "event": "table_file_creation", "file_number": 36686, "file_size": 74546308, "table_properties": {"data_size": 67112086, "index_size": 690051, "filter_size": 6743181, "raw_key_size": 76215831, "raw_average_key_size": 28, "raw_value_size": 31365751, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:43:36.142936 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 15] Generated table #36687: 2638240 keys, 74546480 bytes >2018-07-11 07:43:36.142981 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295016142954, "cf_name": "default", "job": 15, "event": "table_file_creation", "file_number": 36687, "file_size": 74546480, "table_properties": {"data_size": 67112151, "index_size": 690158, "filter_size": 6743181, "raw_key_size": 76215826, "raw_average_key_size": 28, "raw_value_size": 31365746, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:43:39.395343 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 15] Generated table #36688: 2638240 keys, 74546183 bytes >2018-07-11 07:43:39.395392 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295019395376, "cf_name": "default", "job": 15, "event": "table_file_creation", "file_number": 36688, "file_size": 74546183, "table_properties": {"data_size": 67112087, "index_size": 689925, "filter_size": 6743181, "raw_key_size": 76215831, "raw_average_key_size": 28, "raw_value_size": 31365751, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:43:43.768164 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 15] Generated table #36689: 3049277 keys, 75573156 bytes >2018-07-11 07:43:43.768197 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295023768182, "cf_name": "default", "job": 15, "event": "table_file_creation", "file_number": 36689, "file_size": 75573156, "table_properties": {"data_size": 67109837, "index_size": 690102, "filter_size": 7772225, "raw_key_size": 88090226, "raw_average_key_size": 28, "raw_value_size": 25804210, "raw_average_value_size": 8, "num_data_blocks": 16486, "num_entries": 3049277, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "878830", "kMergeOperands": "0"}} >2018-07-11 07:43:46.923867 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 15] Generated table #36690: 2694841 keys, 74685764 bytes >2018-07-11 07:43:46.923913 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295026923886, "cf_name": "default", "job": 15, "event": "table_file_creation", "file_number": 36690, "file_size": 74685764, "table_properties": {"data_size": 67110008, "index_size": 689885, "filter_size": 6884879, "raw_key_size": 77850967, "raw_average_key_size": 28, "raw_value_size": 30598087, "raw_average_value_size": 11, "num_data_blocks": 16489, "num_entries": 2694841, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "121170", "kMergeOperands": "0"}} >2018-07-11 07:43:49.207577 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 15] Generated table #36691: 1950038 keys, 43053175 bytes >2018-07-11 07:43:49.207620 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295029207594, "cf_name": "default", "job": 15, "event": "table_file_creation", "file_number": 36691, "file_size": 43053175, "table_properties": {"data_size": 37705125, "index_size": 387388, "filter_size": 4959672, "raw_key_size": 56334435, "raw_average_key_size": 28, "raw_value_size": 11294899, "raw_average_value_size": 5, "num_data_blocks": 9261, "num_entries": 1950038, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1000000", "kMergeOperands": "0"}} >2018-07-11 07:43:52.349692 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 15] Generated table #36692: 2638240 keys, 74546562 bytes >2018-07-11 07:43:52.349735 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295032349708, "cf_name": "default", "job": 15, "event": "table_file_creation", "file_number": 36692, "file_size": 74546562, "table_properties": {"data_size": 67112128, "index_size": 690263, "filter_size": 6743181, "raw_key_size": 76215818, "raw_average_key_size": 28, "raw_value_size": 31365738, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:43:55.522000 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 15] Generated table #36693: 2638241 keys, 74546141 bytes >2018-07-11 07:43:55.522034 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295035522019, "cf_name": "default", "job": 15, "event": "table_file_creation", "file_number": 36693, "file_size": 74546141, "table_properties": {"data_size": 67112194, "index_size": 689773, "filter_size": 6743184, "raw_key_size": 76215855, "raw_average_key_size": 28, "raw_value_size": 31365758, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:43:58.730850 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 15] Generated table #36694: 2638240 keys, 74546125 bytes >2018-07-11 07:43:58.730894 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295038730866, "cf_name": "default", "job": 15, "event": "table_file_creation", "file_number": 36694, "file_size": 74546125, "table_properties": {"data_size": 67112134, "index_size": 689820, "filter_size": 6743181, "raw_key_size": 76215822, "raw_average_key_size": 28, "raw_value_size": 31365742, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:44:02.003000 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 15] Generated table #36695: 2638240 keys, 74546618 bytes >2018-07-11 07:44:02.003046 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295042003031, "cf_name": "default", "job": 15, "event": "table_file_creation", "file_number": 36695, "file_size": 74546618, "table_properties": {"data_size": 67112107, "index_size": 690340, "filter_size": 6743181, "raw_key_size": 76215826, "raw_average_key_size": 28, "raw_value_size": 31365746, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:44:05.280364 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 15] Generated table #36696: 2638241 keys, 74546529 bytes >2018-07-11 07:44:05.280410 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295045280383, "cf_name": "default", "job": 15, "event": "table_file_creation", "file_number": 36696, "file_size": 74546529, "table_properties": {"data_size": 67112188, "index_size": 690167, "filter_size": 6743184, "raw_key_size": 76215848, "raw_average_key_size": 28, "raw_value_size": 31365751, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:44:08.476258 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 15] Generated table #36697: 2638240 keys, 74546321 bytes >2018-07-11 07:44:08.476290 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295048476276, "cf_name": "default", "job": 15, "event": "table_file_creation", "file_number": 36697, "file_size": 74546321, "table_properties": {"data_size": 67112136, "index_size": 690014, "filter_size": 6743181, "raw_key_size": 76215823, "raw_average_key_size": 28, "raw_value_size": 31365743, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:44:10.655873 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 15] Generated table #36698: 1752777 keys, 49527135 bytes >2018-07-11 07:44:10.655919 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295050655891, "cf_name": "default", "job": 15, "event": "table_file_creation", "file_number": 36698, "file_size": 49527135, "table_properties": {"data_size": 44587491, "index_size": 458669, "filter_size": 4479987, "raw_key_size": 50635787, "raw_average_key_size": 28, "raw_value_size": 20838578, "raw_average_value_size": 11, "num_data_blocks": 10955, "num_entries": 1752777, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:44:13.827793 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 15] Generated table #36699: 2638241 keys, 74545846 bytes >2018-07-11 07:44:13.827838 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295053827811, "cf_name": "default", "job": 15, "event": "table_file_creation", "file_number": 36699, "file_size": 74545846, "table_properties": {"data_size": 67112147, "index_size": 689526, "filter_size": 6743183, "raw_key_size": 76215861, "raw_average_key_size": 28, "raw_value_size": 31365764, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:44:16.964746 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 15] Generated table #36700: 2638241 keys, 74545923 bytes >2018-07-11 07:44:16.964792 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295056964764, "cf_name": "default", "job": 15, "event": "table_file_creation", "file_number": 36700, "file_size": 74545923, "table_properties": {"data_size": 67112090, "index_size": 689660, "filter_size": 6743183, "raw_key_size": 76215847, "raw_average_key_size": 28, "raw_value_size": 31365750, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:44:20.198738 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 15] Generated table #36701: 2638240 keys, 74546314 bytes >2018-07-11 07:44:20.198784 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295060198756, "cf_name": "default", "job": 15, "event": "table_file_creation", "file_number": 36701, "file_size": 74546314, "table_properties": {"data_size": 67112094, "index_size": 690049, "filter_size": 6743181, "raw_key_size": 76215827, "raw_average_key_size": 28, "raw_value_size": 31365747, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:44:23.470545 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 15] Generated table #36702: 2638240 keys, 74546267 bytes >2018-07-11 07:44:23.470589 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295063470562, "cf_name": "default", "job": 15, "event": "table_file_creation", "file_number": 36702, "file_size": 74546267, "table_properties": {"data_size": 67112163, "index_size": 689933, "filter_size": 6743181, "raw_key_size": 76215821, "raw_average_key_size": 28, "raw_value_size": 31365741, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:44:26.698678 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 15] Generated table #36703: 2638240 keys, 74546977 bytes >2018-07-11 07:44:26.698709 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295066698694, "cf_name": "default", "job": 15, "event": "table_file_creation", "file_number": 36703, "file_size": 74546977, "table_properties": {"data_size": 67112135, "index_size": 690671, "filter_size": 6743181, "raw_key_size": 76215819, "raw_average_key_size": 28, "raw_value_size": 31365739, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:44:26.845512 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 15] Generated table #36704: 8 keys, 1247 bytes >2018-07-11 07:44:26.845551 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295066845539, "cf_name": "default", "job": 15, "event": "table_file_creation", "file_number": 36704, "file_size": 1247, "table_properties": {"data_size": 223, "index_size": 28, "filter_size": 30, "raw_key_size": 231, "raw_average_key_size": 28, "raw_value_size": 95, "raw_average_value_size": 11, "num_data_blocks": 1, "num_entries": 8, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:44:29.984415 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 15] Generated table #36705: 2638240 keys, 74545957 bytes >2018-07-11 07:44:29.984459 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295069984433, "cf_name": "default", "job": 15, "event": "table_file_creation", "file_number": 36705, "file_size": 74545957, "table_properties": {"data_size": 67112160, "index_size": 689626, "filter_size": 6743181, "raw_key_size": 76215829, "raw_average_key_size": 28, "raw_value_size": 31365749, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:44:33.141198 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 15] Generated table #36706: 2638240 keys, 74546326 bytes >2018-07-11 07:44:33.141244 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295073141216, "cf_name": "default", "job": 15, "event": "table_file_creation", "file_number": 36706, "file_size": 74546326, "table_properties": {"data_size": 67112139, "index_size": 690016, "filter_size": 6743181, "raw_key_size": 76215825, "raw_average_key_size": 28, "raw_value_size": 31365745, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:44:35.206132 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 15] Generated table #36707: 1846310 keys, 40638009 bytes >2018-07-11 07:44:35.206174 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295075206148, "cf_name": "default", "job": 15, "event": "table_file_creation", "file_number": 36707, "file_size": 40638009, "table_properties": {"data_size": 35576213, "index_size": 365214, "filter_size": 4695592, "raw_key_size": 53337837, "raw_average_key_size": 28, "raw_value_size": 10570922, "raw_average_value_size": 5, "num_data_blocks": 8738, "num_entries": 1846310, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "957167", "kMergeOperands": "0"}} >2018-07-11 07:44:35.206215 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 15] Compacted 1@2 + 23@3 files to L3 => 1551936662 bytes >2018-07-11 07:44:35.219517 7ffabc3f1700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 60162241779 >2018-07-11 07:44:35.219996 7ffabc3f1700 4 rocksdb: (Original Log Time 2018/07/11-07:44:35.219481) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 12 83 449 3641 796 0] max score 3.04, MB/sec: 23.1 rd, 22.4 wr, level 3, files in(1, 23) out(23) MB in(76.6, 1445.8) out(1480.0), read-write-amplify(39.2) write-amplify(19.3) OK, records in: 58611133, records dropped: 2000000 > >2018-07-11 07:44:35.220005 7ffabc3f1700 4 rocksdb: (Original Log Time 2018/07/11-07:44:35.219509) EVENT_LOG_v1 {"time_micros": 1531295075219496, "job": 15, "event": "compaction_finished", "compaction_time_micros": 69225761, "output_level": 3, "num_output_files": 23, "total_output_size": 1551936662, "num_input_records": 58611133, "num_output_records": 56611133, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 12, 83, 449, 3641, 796, 0]} >2018-07-11 07:44:35.220592 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295075220588, "job": 15, "event": "table_file_deletion", "file_number": 36599} >2018-07-11 07:44:35.220611 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295075220607, "job": 15, "event": "table_file_deletion", "file_number": 4976} >2018-07-11 07:44:35.220625 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295075220623, "job": 15, "event": "table_file_deletion", "file_number": 4975} >2018-07-11 07:44:35.220639 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295075220637, "job": 15, "event": "table_file_deletion", "file_number": 4973} >2018-07-11 07:44:35.220652 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295075220650, "job": 15, "event": "table_file_deletion", "file_number": 4824} >2018-07-11 07:44:35.220667 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295075220664, "job": 15, "event": "table_file_deletion", "file_number": 4662} >2018-07-11 07:44:35.220679 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295075220677, "job": 15, "event": "table_file_deletion", "file_number": 4661} >2018-07-11 07:44:35.220692 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295075220690, "job": 15, "event": "table_file_deletion", "file_number": 4655} >2018-07-11 07:44:35.220704 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295075220702, "job": 15, "event": "table_file_deletion", "file_number": 4652} >2018-07-11 07:44:35.220719 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295075220715, "job": 15, "event": "table_file_deletion", "file_number": 3809} >2018-07-11 07:44:35.220731 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295075220729, "job": 15, "event": "table_file_deletion", "file_number": 3794} >2018-07-11 07:44:35.220744 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295075220742, "job": 15, "event": "table_file_deletion", "file_number": 3758} >2018-07-11 07:44:35.220757 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295075220755, "job": 15, "event": "table_file_deletion", "file_number": 3743} >2018-07-11 07:44:35.220773 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295075220771, "job": 15, "event": "table_file_deletion", "file_number": 3712} >2018-07-11 07:44:35.220786 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295075220784, "job": 15, "event": "table_file_deletion", "file_number": 3654} >2018-07-11 07:44:35.220798 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295075220796, "job": 15, "event": "table_file_deletion", "file_number": 3637} >2018-07-11 07:44:35.220811 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295075220809, "job": 15, "event": "table_file_deletion", "file_number": 3603} >2018-07-11 07:44:35.220824 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295075220822, "job": 15, "event": "table_file_deletion", "file_number": 3566} >2018-07-11 07:44:35.220836 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295075220834, "job": 15, "event": "table_file_deletion", "file_number": 3547} >2018-07-11 07:44:35.220849 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295075220847, "job": 15, "event": "table_file_deletion", "file_number": 3513} >2018-07-11 07:44:35.220862 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295075220859, "job": 15, "event": "table_file_deletion", "file_number": 3496} >2018-07-11 07:44:35.220874 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295075220872, "job": 15, "event": "table_file_deletion", "file_number": 3476} >2018-07-11 07:44:35.220887 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295075220885, "job": 15, "event": "table_file_deletion", "file_number": 3462} >2018-07-11 07:44:35.220897 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295075220895, "job": 15, "event": "table_file_deletion", "file_number": 3442} >2018-07-11 07:44:35.221073 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 16] Compacting 1@2 + 7@3 files to L3, score 3.04 >2018-07-11 07:44:35.221085 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 15 Base level 2, inputs: [36583(76MB)], [30998(71MB) 30999(71MB) 31000(64MB) 31001(71MB) 31002(71MB) 31003(6752KB) 25651(71MB)] > >2018-07-11 07:44:35.221099 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295075221089, "job": 16, "event": "compaction_started", "files_L2": [36583], "files_L3": [30998, 30999, 31000, 31001, 31002, 31003, 25651], "score": 3.04449, "input_data_size": 527583416} >2018-07-11 07:44:38.749696 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 16] Generated table #36708: 2853527 keys, 75084049 bytes >2018-07-11 07:44:38.749743 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295078749726, "cf_name": "default", "job": 16, "event": "table_file_creation", "file_number": 36708, "file_size": 75084049, "table_properties": {"data_size": 67111277, "index_size": 689620, "filter_size": 7282160, "raw_key_size": 82435226, "raw_average_key_size": 28, "raw_value_size": 28453169, "raw_average_value_size": 9, "num_data_blocks": 16488, "num_entries": 2853527, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "460270", "kMergeOperands": "0"}} >2018-07-11 07:44:42.765986 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 16] Generated table #36709: 3497690 keys, 76698625 bytes >2018-07-11 07:44:42.766038 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295082766010, "cf_name": "default", "job": 16, "event": "table_file_creation", "file_number": 36709, "file_size": 76698625, "table_properties": {"data_size": 67112859, "index_size": 689923, "filter_size": 8894851, "raw_key_size": 101044391, "raw_average_key_size": 28, "raw_value_size": 19742510, "raw_average_value_size": 5, "num_data_blocks": 16483, "num_entries": 3497690, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1837105", "kMergeOperands": "0"}} >2018-07-11 07:44:45.686423 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 16] Generated table #36710: 2638240 keys, 74546005 bytes >2018-07-11 07:44:45.686467 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295085686440, "cf_name": "default", "job": 16, "event": "table_file_creation", "file_number": 36710, "file_size": 74546005, "table_properties": {"data_size": 67112108, "index_size": 689726, "filter_size": 6743181, "raw_key_size": 76215823, "raw_average_key_size": 28, "raw_value_size": 31365743, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:44:48.872538 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 16] Generated table #36711: 2638240 keys, 74546317 bytes >2018-07-11 07:44:48.872571 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295088872556, "cf_name": "default", "job": 16, "event": "table_file_creation", "file_number": 36711, "file_size": 74546317, "table_properties": {"data_size": 67112148, "index_size": 689998, "filter_size": 6743181, "raw_key_size": 76215827, "raw_average_key_size": 28, "raw_value_size": 31365747, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:44:52.097498 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 16] Generated table #36712: 2638240 keys, 74546264 bytes >2018-07-11 07:44:52.097529 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295092097514, "cf_name": "default", "job": 16, "event": "table_file_creation", "file_number": 36712, "file_size": 74546264, "table_properties": {"data_size": 67112103, "index_size": 689990, "filter_size": 6743181, "raw_key_size": 76215819, "raw_average_key_size": 28, "raw_value_size": 31365739, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:44:54.341090 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 16] Generated table #36713: 1486281 keys, 34376068 bytes >2018-07-11 07:44:54.341141 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295094341124, "cf_name": "default", "job": 16, "event": "table_file_creation", "file_number": 36713, "file_size": 34376068, "table_properties": {"data_size": 30280807, "index_size": 310935, "filter_size": 3783336, "raw_key_size": 42937036, "raw_average_key_size": 28, "raw_value_size": 10149916, "raw_average_value_size": 6, "num_data_blocks": 7438, "num_entries": 1486281, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "632551", "kMergeOperands": "0"}} >2018-07-11 07:44:57.503662 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 16] Generated table #36714: 2853277 keys, 75083382 bytes >2018-07-11 07:44:57.503709 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295097503681, "cf_name": "default", "job": 16, "event": "table_file_creation", "file_number": 36714, "file_size": 75083382, "table_properties": {"data_size": 67111302, "index_size": 689553, "filter_size": 7281535, "raw_key_size": 82427992, "raw_average_key_size": 28, "raw_value_size": 28456567, "raw_average_value_size": 9, "num_data_blocks": 16488, "num_entries": 2853277, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "459734", "kMergeOperands": "0"}} >2018-07-11 07:44:58.969916 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 16] Generated table #36715: 1444757 keys, 26365714 bytes >2018-07-11 07:44:58.969960 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295098969932, "cf_name": "default", "job": 16, "event": "table_file_creation", "file_number": 36715, "file_size": 26365714, "table_properties": {"data_size": 22470871, "index_size": 230557, "filter_size": 3663296, "raw_key_size": 41737428, "raw_average_key_size": 28, "raw_value_size": 2909174, "raw_average_value_size": 2, "num_data_blocks": 5517, "num_entries": 1444757, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1200060", "kMergeOperands": "0"}} >2018-07-11 07:44:58.969994 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 16] Compacted 1@2 + 7@3 files to L3 => 511246424 bytes >2018-07-11 07:44:58.985674 7ffabc3f1700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 59733406692 >2018-07-11 07:44:58.985855 7ffabc3f1700 4 rocksdb: (Original Log Time 2018/07/11-07:44:58.985637) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 12 82 450 3641 796 0] max score 2.99, MB/sec: 22.2 rd, 21.5 wr, level 3, files in(1, 7) out(8) MB in(76.6, 426.5) out(487.6), read-write-amplify(12.9) write-amplify(6.4) OK, records in: 20785150, records dropped: 734898 > >2018-07-11 07:44:58.985865 7ffabc3f1700 4 rocksdb: (Original Log Time 2018/07/11-07:44:58.985665) EVENT_LOG_v1 {"time_micros": 1531295098985651, "job": 16, "event": "compaction_finished", "compaction_time_micros": 23748878, "output_level": 3, "num_output_files": 8, "total_output_size": 511246424, "num_input_records": 20785150, "num_output_records": 20050252, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 12, 82, 450, 3641, 796, 0]} >2018-07-11 07:44:58.986498 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295098986494, "job": 16, "event": "table_file_deletion", "file_number": 36583} >2018-07-11 07:44:58.986516 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295098986514, "job": 16, "event": "table_file_deletion", "file_number": 31003} >2018-07-11 07:44:58.986527 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295098986525, "job": 16, "event": "table_file_deletion", "file_number": 31002} >2018-07-11 07:44:58.986540 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295098986538, "job": 16, "event": "table_file_deletion", "file_number": 31001} >2018-07-11 07:44:58.986553 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295098986551, "job": 16, "event": "table_file_deletion", "file_number": 31000} >2018-07-11 07:44:58.986566 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295098986564, "job": 16, "event": "table_file_deletion", "file_number": 30999} >2018-07-11 07:44:58.986579 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295098986577, "job": 16, "event": "table_file_deletion", "file_number": 30998} >2018-07-11 07:44:58.986594 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295098986591, "job": 16, "event": "table_file_deletion", "file_number": 25651} >2018-07-11 07:44:58.986772 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 17] Compacting 1@2 + 1@3 files to L3, score 2.99 >2018-07-11 07:44:58.986783 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 16 Base level 2, inputs: [36625(76MB)], [25646(71MB)] > >2018-07-11 07:44:58.986796 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295098986786, "job": 17, "event": "compaction_started", "files_L2": [36625], "files_L3": [25646], "score": 2.98502, "input_data_size": 154894241} >2018-07-11 07:45:03.431657 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 17] Generated table #36716: 3865211 keys, 77614786 bytes >2018-07-11 07:45:03.431694 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295103431679, "cf_name": "default", "job": 17, "event": "table_file_creation", "file_number": 36716, "file_size": 77614786, "table_properties": {"data_size": 67109034, "index_size": 689809, "filter_size": 9814950, "raw_key_size": 111661671, "raw_average_key_size": 28, "raw_value_size": 14768088, "raw_average_value_size": 3, "num_data_blocks": 16479, "num_entries": 3865211, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "2623036", "kMergeOperands": "0"}} >2018-07-11 07:45:07.022002 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 17] Generated table #36717: 3240797 keys, 66399639 bytes >2018-07-11 07:45:07.022050 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295107022022, "cf_name": "default", "job": 17, "event": "table_file_creation", "file_number": 36717, "file_size": 66399639, "table_properties": {"data_size": 57575059, "index_size": 591535, "filter_size": 8232054, "raw_key_size": 93623007, "raw_average_key_size": 28, "raw_value_size": 13688461, "raw_average_value_size": 4, "num_data_blocks": 14138, "num_entries": 3240797, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "2089431", "kMergeOperands": "0"}} >2018-07-11 07:45:07.022080 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 17] Compacted 1@2 + 1@3 files to L3 => 144014425 bytes >2018-07-11 07:45:07.035609 7ffabc3f1700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 59356510041 >2018-07-11 07:45:07.035804 7ffabc3f1700 4 rocksdb: (Original Log Time 2018/07/11-07:45:07.035575) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 12 81 451 3641 796 0] max score 2.93, MB/sec: 19.3 rd, 17.9 wr, level 3, files in(1, 1) out(2) MB in(76.6, 71.1) out(137.3), read-write-amplify(3.7) write-amplify(1.8) OK, records in: 7595406, records dropped: 489398 > >2018-07-11 07:45:07.035813 7ffabc3f1700 4 rocksdb: (Original Log Time 2018/07/11-07:45:07.035601) EVENT_LOG_v1 {"time_micros": 1531295107035590, "job": 17, "event": "compaction_finished", "compaction_time_micros": 8035271, "output_level": 3, "num_output_files": 2, "total_output_size": 144014425, "num_input_records": 7595406, "num_output_records": 7106008, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 12, 81, 451, 3641, 796, 0]} >2018-07-11 07:45:07.036454 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295107036450, "job": 17, "event": "table_file_deletion", "file_number": 36625} >2018-07-11 07:45:07.036475 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295107036470, "job": 17, "event": "table_file_deletion", "file_number": 25646} >2018-07-11 07:45:07.036663 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 18] Compacting 1@2 + 11@3 files to L3, score 2.93 >2018-07-11 07:45:07.036677 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 17 Base level 2, inputs: [36587(76MB)], [1635(71MB) 1654(71MB) 1670(71MB) 1692(71MB) 1717(71MB) 30058(1168B) 30059(71MB) 30060(71MB) 30063(71MB) 30064(71MB) 30065(71MB)] > >2018-07-11 07:45:07.036692 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295107036681, "job": 18, "event": "compaction_started", "files_L2": [36587], "files_L3": [1635, 1654, 1670, 1692, 1717, 30058, 30059, 30060, 30063, 30064, 30065], "score": 2.92554, "input_data_size": 825808761} >2018-07-11 07:45:10.425302 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 18] Generated table #36718: 2691113 keys, 74676449 bytes >2018-07-11 07:45:10.425351 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295110425321, "cf_name": "default", "job": 18, "event": "table_file_creation", "file_number": 36718, "file_size": 74676449, "table_properties": {"data_size": 67109845, "index_size": 690066, "filter_size": 6875546, "raw_key_size": 77743278, "raw_average_key_size": 28, "raw_value_size": 30648398, "raw_average_value_size": 11, "num_data_blocks": 16489, "num_entries": 2691113, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "113211", "kMergeOperands": "0"}} >2018-07-11 07:45:13.627892 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 18] Generated table #36719: 2638241 keys, 74546128 bytes >2018-07-11 07:45:13.627924 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295113627909, "cf_name": "default", "job": 18, "event": "table_file_creation", "file_number": 36719, "file_size": 74546128, "table_properties": {"data_size": 67112184, "index_size": 689771, "filter_size": 6743183, "raw_key_size": 76215856, "raw_average_key_size": 28, "raw_value_size": 31365759, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:45:15.420194 7ffab53e3700 0 log_channel(cluster) log [DBG] : 11.5 scrub starts >2018-07-11 07:45:15.421474 7ffabdbf4700 1 bluestore(/var/lib/ceph/osd/ceph-10) _balance_bluefs_freespace gifting 0xd2d9f00000~80000000 to bluefs >2018-07-11 07:45:15.435195 7ffabdbf4700 1 bluefs add_block_extent bdev 2 0xd2d9f00000~80000000 >2018-07-11 07:45:16.897304 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 18] Generated table #36720: 2638240 keys, 74546359 bytes >2018-07-11 07:45:16.897334 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295116897320, "cf_name": "default", "job": 18, "event": "table_file_creation", "file_number": 36720, "file_size": 74546359, "table_properties": {"data_size": 67112153, "index_size": 690035, "filter_size": 6743181, "raw_key_size": 76215825, "raw_average_key_size": 28, "raw_value_size": 31365745, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:45:20.493323 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 18] Generated table #36721: 2638240 keys, 74546712 bytes >2018-07-11 07:45:20.493355 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295120493340, "cf_name": "default", "job": 18, "event": "table_file_creation", "file_number": 36721, "file_size": 74546712, "table_properties": {"data_size": 67112139, "index_size": 690402, "filter_size": 6743181, "raw_key_size": 76215831, "raw_average_key_size": 28, "raw_value_size": 31365751, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:45:24.049457 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 18] Generated table #36722: 2638240 keys, 74546043 bytes >2018-07-11 07:45:24.049487 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295124049472, "cf_name": "default", "job": 18, "event": "table_file_creation", "file_number": 36722, "file_size": 74546043, "table_properties": {"data_size": 67112142, "index_size": 689730, "filter_size": 6743181, "raw_key_size": 76215825, "raw_average_key_size": 28, "raw_value_size": 31365745, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:45:25.352960 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 18] Generated table #36723: 1060342 keys, 17913943 bytes >2018-07-11 07:45:25.352990 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295125352976, "cf_name": "default", "job": 18, "event": "table_file_creation", "file_number": 36723, "file_size": 17913943, "table_properties": {"data_size": 15072873, "index_size": 154424, "filter_size": 2685657, "raw_key_size": 30632100, "raw_average_key_size": 28, "raw_value_size": 717396, "raw_average_value_size": 0, "num_data_blocks": 3700, "num_entries": 1060342, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1000000", "kMergeOperands": "0"}} >2018-07-11 07:45:29.336437 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 18] Generated table #36724: 3106038 keys, 75716991 bytes >2018-07-11 07:45:29.336468 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295129336454, "cf_name": "default", "job": 18, "event": "table_file_creation", "file_number": 36724, "file_size": 75716991, "table_properties": {"data_size": 67111722, "index_size": 689944, "filter_size": 7914333, "raw_key_size": 89729990, "raw_average_key_size": 28, "raw_value_size": 25038454, "raw_average_value_size": 8, "num_data_blocks": 16486, "num_entries": 3106038, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1000000", "kMergeOperands": "0"}} >2018-07-11 07:45:34.148724 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 18] Generated table #36725: 3106038 keys, 75716966 bytes >2018-07-11 07:45:34.148769 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295134148754, "cf_name": "default", "job": 18, "event": "table_file_creation", "file_number": 36725, "file_size": 75716966, "table_properties": {"data_size": 67111744, "index_size": 689898, "filter_size": 7914332, "raw_key_size": 89729989, "raw_average_key_size": 28, "raw_value_size": 25038453, "raw_average_value_size": 8, "num_data_blocks": 16486, "num_entries": 3106038, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1000000", "kMergeOperands": "0"}} >2018-07-11 07:45:37.675697 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 18] Generated table #36726: 2638240 keys, 74546901 bytes >2018-07-11 07:45:37.675739 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295137675712, "cf_name": "default", "job": 18, "event": "table_file_creation", "file_number": 36726, "file_size": 74546901, "table_properties": {"data_size": 67112145, "index_size": 690585, "filter_size": 6743181, "raw_key_size": 76215825, "raw_average_key_size": 28, "raw_value_size": 31365745, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:45:38.100714 7ffab53e3700 0 log_channel(cluster) log [DBG] : 11.5 scrub ok >2018-07-11 07:45:38.100838 7ffabdbf4700 1 bluestore(/var/lib/ceph/osd/ceph-10) _balance_bluefs_freespace gifting 0xd359f00000~80000000 to bluefs >2018-07-11 07:45:38.118107 7ffabdbf4700 1 bluefs add_block_extent bdev 2 0xd359f00000~80000000 >2018-07-11 07:45:38.460730 7ffab4be2700 0 log_channel(cluster) log [DBG] : 5.2e deep-scrub starts >2018-07-11 07:45:42.741508 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 18] Generated table #36727: 2638241 keys, 74546899 bytes >2018-07-11 07:45:42.741551 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295142741524, "cf_name": "default", "job": 18, "event": "table_file_creation", "file_number": 36727, "file_size": 74546899, "table_properties": {"data_size": 67112148, "index_size": 690577, "filter_size": 6743184, "raw_key_size": 76215855, "raw_average_key_size": 28, "raw_value_size": 31365758, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:45:43.247222 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 18] Generated table #36728: 64412 keys, 1820617 bytes >2018-07-11 07:45:43.247267 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295143247235, "cf_name": "default", "job": 18, "event": "table_file_creation", "file_number": 36728, "file_size": 1820617, "table_properties": {"data_size": 1638539, "index_size": 16459, "filter_size": 164638, "raw_key_size": 1860793, "raw_average_key_size": 28, "raw_value_size": 765789, "raw_average_value_size": 11, "num_data_blocks": 403, "num_entries": 64412, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:45:48.535949 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 18] Generated table #36729: 2638241 keys, 74545909 bytes >2018-07-11 07:45:48.535992 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295148535965, "cf_name": "default", "job": 18, "event": "table_file_creation", "file_number": 36729, "file_size": 74545909, "table_properties": {"data_size": 67112165, "index_size": 689571, "filter_size": 6743183, "raw_key_size": 76215855, "raw_average_key_size": 28, "raw_value_size": 31365758, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:45:50.207350 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 18] Generated table #36730: 843955 keys, 13679631 bytes >2018-07-11 07:45:50.207391 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295150207365, "cf_name": "default", "job": 18, "event": "table_file_creation", "file_number": 36730, "file_size": 13679631, "table_properties": {"data_size": 11425336, "index_size": 116898, "filter_size": 2136410, "raw_key_size": 24380918, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 2804, "num_entries": 843955, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "843955", "kMergeOperands": "0"}} >2018-07-11 07:45:50.207422 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 18] Compacted 1@2 + 11@3 files to L3 => 781349548 bytes >2018-07-11 07:45:50.226408 7ffabc3f1700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 58617841082 >2018-07-11 07:45:50.226690 7ffabc3f1700 4 rocksdb: (Original Log Time 2018/07/11-07:45:50.226373) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 12 80 453 3641 796 0] max score 2.87, MB/sec: 19.1 rd, 18.1 wr, level 3, files in(1, 11) out(13) MB in(76.6, 710.9) out(745.2), read-write-amplify(20.0) write-amplify(9.7) OK, records in: 31339581, records dropped: 2000000 > >2018-07-11 07:45:50.226699 7ffabc3f1700 4 rocksdb: (Original Log Time 2018/07/11-07:45:50.226400) EVENT_LOG_v1 {"time_micros": 1531295150226387, "job": 18, "event": "compaction_finished", "compaction_time_micros": 43170709, "output_level": 3, "num_output_files": 13, "total_output_size": 781349548, "num_input_records": 31339581, "num_output_records": 29339581, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 12, 80, 453, 3641, 796, 0]} >2018-07-11 07:45:50.227338 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295150227333, "job": 18, "event": "table_file_deletion", "file_number": 36587} >2018-07-11 07:45:50.227353 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295150227351, "job": 18, "event": "table_file_deletion", "file_number": 30065} >2018-07-11 07:45:50.227365 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295150227363, "job": 18, "event": "table_file_deletion", "file_number": 30064} >2018-07-11 07:45:50.227380 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295150227377, "job": 18, "event": "table_file_deletion", "file_number": 30063} >2018-07-11 07:45:50.227393 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295150227391, "job": 18, "event": "table_file_deletion", "file_number": 30060} >2018-07-11 07:45:50.227406 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295150227404, "job": 18, "event": "table_file_deletion", "file_number": 30059} >2018-07-11 07:45:50.227419 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295150227417, "job": 18, "event": "table_file_deletion", "file_number": 30058} >2018-07-11 07:45:50.227434 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295150227432, "job": 18, "event": "table_file_deletion", "file_number": 1717} >2018-07-11 07:45:50.227448 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295150227446, "job": 18, "event": "table_file_deletion", "file_number": 1692} >2018-07-11 07:45:50.227461 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295150227458, "job": 18, "event": "table_file_deletion", "file_number": 1670} >2018-07-11 07:45:50.227474 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295150227471, "job": 18, "event": "table_file_deletion", "file_number": 1654} >2018-07-11 07:45:50.227488 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295150227484, "job": 18, "event": "table_file_deletion", "file_number": 1635} >2018-07-11 07:45:50.227680 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 19] Compacting 1@2 + 3@3 files to L3, score 2.87 >2018-07-11 07:45:50.227691 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 18 Base level 2, inputs: [36640(76MB)], [31422(39MB) 31424(71MB) 31426(39MB)] > >2018-07-11 07:45:50.227704 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295150227694, "job": 19, "event": "compaction_started", "files_L2": [36640], "files_L3": [31422, 31424, 31426], "score": 2.86606, "input_data_size": 238649821} >2018-07-11 07:46:00.692310 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 19] Generated table #36731: 3573834 keys, 76887235 bytes >2018-07-11 07:46:00.692357 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295160692329, "cf_name": "default", "job": 19, "event": "table_file_creation", "file_number": 36731, "file_size": 76887235, "table_properties": {"data_size": 67111272, "index_size": 689490, "filter_size": 9085481, "raw_key_size": 103244096, "raw_average_key_size": 28, "raw_value_size": 18711138, "raw_average_value_size": 5, "num_data_blocks": 16482, "num_entries": 3573834, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "2000000", "kMergeOperands": "0"}} >2018-07-11 07:46:07.273348 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 19] Generated table #36732: 3154535 keys, 75835881 bytes >2018-07-11 07:46:07.273393 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295167273367, "cf_name": "default", "job": 19, "event": "table_file_creation", "file_number": 36732, "file_size": 75835881, "table_properties": {"data_size": 67109312, "index_size": 689834, "filter_size": 8035743, "raw_key_size": 91131009, "raw_average_key_size": 28, "raw_value_size": 24380158, "raw_average_value_size": 7, "num_data_blocks": 16485, "num_entries": 3154535, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1103868", "kMergeOperands": "0"}} >2018-07-11 07:46:07.592016 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 19] Generated table #36733: 124600 keys, 3521147 bytes >2018-07-11 07:46:07.592054 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295167592029, "cf_name": "default", "job": 19, "event": "table_file_creation", "file_number": 36733, "file_size": 3521147, "table_properties": {"data_size": 3169610, "index_size": 32080, "filter_size": 318472, "raw_key_size": 3599555, "raw_average_key_size": 28, "raw_value_size": 1481355, "raw_average_value_size": 11, "num_data_blocks": 779, "num_entries": 124600, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:46:07.592076 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 19] Compacted 1@2 + 3@3 files to L3 => 156244263 bytes >2018-07-11 07:46:07.609371 7ffabc3f1700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 57469048567 >2018-07-11 07:46:07.609570 7ffabc3f1700 4 rocksdb: (Original Log Time 2018/07/11-07:46:07.609338) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 12 79 453 3641 796 0] max score 2.81, MB/sec: 13.7 rd, 9.0 wr, level 3, files in(1, 3) out(3) MB in(76.6, 151.0) out(149.0), read-write-amplify(4.9) write-amplify(1.9) OK, records in: 10559567, records dropped: 3706598 > >2018-07-11 07:46:07.609579 7ffabc3f1700 4 rocksdb: (Original Log Time 2018/07/11-07:46:07.609364) EVENT_LOG_v1 {"time_micros": 1531295167609353, "job": 19, "event": "compaction_finished", "compaction_time_micros": 17364361, "output_level": 3, "num_output_files": 3, "total_output_size": 156244263, "num_input_records": 10559567, "num_output_records": 6852969, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 12, 79, 453, 3641, 796, 0]} >2018-07-11 07:46:07.610187 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295167610183, "job": 19, "event": "table_file_deletion", "file_number": 36640} >2018-07-11 07:46:07.610219 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295167610216, "job": 19, "event": "table_file_deletion", "file_number": 31426} >2018-07-11 07:46:07.610230 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295167610228, "job": 19, "event": "table_file_deletion", "file_number": 31424} >2018-07-11 07:46:07.610241 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295167610239, "job": 19, "event": "table_file_deletion", "file_number": 31422} >2018-07-11 07:46:07.610428 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 20] Compacting 1@2 + 13@3 files to L3, score 2.81 >2018-07-11 07:46:07.610444 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 19 Base level 2, inputs: [36616(76MB)], [36650(44MB) 19092(71MB) 19096(71MB) 19098(71MB) 19100(71MB) 19102(39MB) 19104(31MB) 19105(71MB) 19106(71MB) 19109(1064B) 19113(71MB) 19114(899KB) 19580(71MB)] > >2018-07-11 07:46:07.610459 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295167610448, "job": 20, "event": "compaction_started", "files_L2": [36616], "files_L3": [36650, 19092, 19096, 19098, 19100, 19102, 19104, 19105, 19106, 19109, 19113, 19114, 19580], "score": 2.80658, "input_data_size": 798732997} >2018-07-11 07:46:15.470045 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 20] Generated table #36734: 3459235 keys, 76601186 bytes >2018-07-11 07:46:15.470097 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295175470080, "cf_name": "default", "job": 20, "event": "table_file_creation", "file_number": 36734, "file_size": 76601186, "table_properties": {"data_size": 67111490, "index_size": 690128, "filter_size": 8798576, "raw_key_size": 99933454, "raw_average_key_size": 28, "raw_value_size": 20261297, "raw_average_value_size": 5, "num_data_blocks": 16483, "num_entries": 3459235, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1755014", "kMergeOperands": "0"}} >2018-07-11 07:46:19.443673 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 20] Generated table #36735: 1784697 keys, 47477177 bytes >2018-07-11 07:46:19.443716 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295179443690, "cf_name": "default", "job": 20, "event": "table_file_creation", "file_number": 36735, "file_size": 47477177, "table_properties": {"data_size": 42484104, "index_size": 436514, "filter_size": 4555569, "raw_key_size": 51557914, "raw_average_key_size": 28, "raw_value_size": 18305447, "raw_average_value_size": 10, "num_data_blocks": 10438, "num_entries": 1784697, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "244986", "kMergeOperands": "0"}} >2018-07-11 07:46:25.298305 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 20] Generated table #36736: 2638240 keys, 74545946 bytes >2018-07-11 07:46:25.298349 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295185298322, "cf_name": "default", "job": 20, "event": "table_file_creation", "file_number": 36736, "file_size": 74545946, "table_properties": {"data_size": 67112096, "index_size": 689679, "filter_size": 6743181, "raw_key_size": 76215833, "raw_average_key_size": 28, "raw_value_size": 31365753, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:46:30.665699 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 20] Generated table #36737: 2637103 keys, 74514208 bytes >2018-07-11 07:46:30.665745 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295190665729, "cf_name": "default", "job": 20, "event": "table_file_creation", "file_number": 36737, "file_size": 74514208, "table_properties": {"data_size": 67083175, "index_size": 689778, "filter_size": 6740265, "raw_key_size": 76182974, "raw_average_key_size": 28, "raw_value_size": 31352223, "raw_average_value_size": 11, "num_data_blocks": 16482, "num_entries": 2637103, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:46:36.016993 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 20] Generated table #36738: 2638241 keys, 74545953 bytes >2018-07-11 07:46:36.017035 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295196017008, "cf_name": "default", "job": 20, "event": "table_file_creation", "file_number": 36738, "file_size": 74545953, "table_properties": {"data_size": 67112128, "index_size": 689652, "filter_size": 6743183, "raw_key_size": 76215852, "raw_average_key_size": 28, "raw_value_size": 31365755, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:46:41.941134 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 20] Generated table #36739: 2638239 keys, 74546188 bytes >2018-07-11 07:46:41.941169 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295201941154, "cf_name": "default", "job": 20, "event": "table_file_creation", "file_number": 36739, "file_size": 74546188, "table_properties": {"data_size": 67112110, "index_size": 689909, "filter_size": 6743179, "raw_key_size": 76215802, "raw_average_key_size": 28, "raw_value_size": 31365739, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638239, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:46:42.195289 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 20] Generated table #36740: 2 keys, 1097 bytes >2018-07-11 07:46:42.195328 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295202195315, "cf_name": "default", "job": 20, "event": "table_file_creation", "file_number": 36740, "file_size": 1097, "table_properties": {"data_size": 90, "index_size": 27, "filter_size": 18, "raw_key_size": 57, "raw_average_key_size": 28, "raw_value_size": 23, "raw_average_value_size": 11, "num_data_blocks": 1, "num_entries": 2, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:46:48.043367 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 20] Generated table #36741: 2638241 keys, 74546022 bytes >2018-07-11 07:46:48.043402 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295208043384, "cf_name": "default", "job": 20, "event": "table_file_creation", "file_number": 36741, "file_size": 74546022, "table_properties": {"data_size": 67112149, "index_size": 689700, "filter_size": 6743183, "raw_key_size": 76215847, "raw_average_key_size": 28, "raw_value_size": 31365750, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:46:53.805807 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 20] Generated table #36742: 2638241 keys, 74545921 bytes >2018-07-11 07:46:53.805852 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295213805824, "cf_name": "default", "job": 20, "event": "table_file_creation", "file_number": 36742, "file_size": 74545921, "table_properties": {"data_size": 67112164, "index_size": 689584, "filter_size": 6743183, "raw_key_size": 76215855, "raw_average_key_size": 28, "raw_value_size": 31365758, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:46:53.870104 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 20] Generated table #36743: 1 keys, 1047 bytes >2018-07-11 07:46:53.870140 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295213870116, "cf_name": "default", "job": 20, "event": "table_file_creation", "file_number": 36743, "file_size": 1047, "table_properties": {"data_size": 40, "index_size": 27, "filter_size": 18, "raw_key_size": 24, "raw_average_key_size": 24, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 1, "num_entries": 1, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1", "kMergeOperands": "0"}} >2018-07-11 07:47:00.051014 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 20] Generated table #36744: 3106036 keys, 75716667 bytes >2018-07-11 07:47:00.051061 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295220051046, "cf_name": "default", "job": 20, "event": "table_file_creation", "file_number": 36744, "file_size": 75716667, "table_properties": {"data_size": 67111735, "index_size": 689612, "filter_size": 7914328, "raw_key_size": 89729941, "raw_average_key_size": 28, "raw_value_size": 25038446, "raw_average_value_size": 8, "num_data_blocks": 16486, "num_entries": 3106036, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "999999", "kMergeOperands": "0"}} >2018-07-11 07:47:02.874752 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 20] Generated table #36745: 1436253 keys, 30083957 bytes >2018-07-11 07:47:02.874780 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295222874766, "cf_name": "default", "job": 20, "event": "table_file_creation", "file_number": 36745, "file_size": 30083957, "table_properties": {"data_size": 26164617, "index_size": 268735, "filter_size": 3649615, "raw_key_size": 41491753, "raw_average_key_size": 28, "raw_value_size": 6714339, "raw_average_value_size": 4, "num_data_blocks": 6426, "num_entries": 1436253, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "871496", "kMergeOperands": "0"}} >2018-07-11 07:47:08.608969 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 20] Generated table #36746: 2664657 keys, 74608643 bytes >2018-07-11 07:47:08.609013 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295228608986, "cf_name": "default", "job": 20, "event": "table_file_creation", "file_number": 36746, "file_size": 74608643, "table_properties": {"data_size": 67108938, "index_size": 689401, "filter_size": 6809312, "raw_key_size": 76978972, "raw_average_key_size": 28, "raw_value_size": 31005289, "raw_average_value_size": 11, "num_data_blocks": 16489, "num_entries": 2664657, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "56736", "kMergeOperands": "0"}} >2018-07-11 07:47:10.492549 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 20] Generated table #36747: 1016421 keys, 16840358 bytes >2018-07-11 07:47:10.492591 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295230492564, "cf_name": "default", "job": 20, "event": "table_file_creation", "file_number": 36747, "file_size": 16840358, "table_properties": {"data_size": 14121003, "index_size": 144625, "filter_size": 2573741, "raw_key_size": 29363278, "raw_average_key_size": 28, "raw_value_size": 360460, "raw_average_value_size": 0, "num_data_blocks": 3466, "num_entries": 1016421, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "986102", "kMergeOperands": "0"}} >2018-07-11 07:47:10.492623 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 20] Compacted 1@2 + 13@3 files to L3 => 768574370 bytes >2018-07-11 07:47:10.508012 7ffabc3f1700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 56868126488 >2018-07-11 07:47:10.508131 7ffabc3f1700 4 rocksdb: (Original Log Time 2018/07/11-07:47:10.507977) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 12 78 454 3641 796 0] max score 2.79, MB/sec: 12.7 rd, 12.2 wr, level 3, files in(1, 13) out(14) MB in(76.6, 685.1) out(733.0), read-write-amplify(19.5) write-amplify(9.6) OK, records in: 30381271, records dropped: 1085664 > >2018-07-11 07:47:10.508139 7ffabc3f1700 4 rocksdb: (Original Log Time 2018/07/11-07:47:10.508004) EVENT_LOG_v1 {"time_micros": 1531295230507992, "job": 20, "event": "compaction_finished", "compaction_time_micros": 62882143, "output_level": 3, "num_output_files": 14, "total_output_size": 768574370, "num_input_records": 30381271, "num_output_records": 29295607, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 12, 78, 454, 3641, 796, 0]} >2018-07-11 07:47:10.508947 7ffab4be2700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295230508930, "job": 0, "event": "table_file_deletion", "file_number": 36650} >2018-07-11 07:47:10.508963 7ffab4be2700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295230508961, "job": 0, "event": "table_file_deletion", "file_number": 36616} >2018-07-11 07:47:10.508977 7ffab4be2700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295230508975, "job": 0, "event": "table_file_deletion", "file_number": 19580} >2018-07-11 07:47:10.508990 7ffab4be2700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295230508988, "job": 0, "event": "table_file_deletion", "file_number": 19114} >2018-07-11 07:47:10.509002 7ffab4be2700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295230508999, "job": 0, "event": "table_file_deletion", "file_number": 19113} >2018-07-11 07:47:10.509016 7ffab4be2700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295230509014, "job": 0, "event": "table_file_deletion", "file_number": 19109} >2018-07-11 07:47:10.509030 7ffab4be2700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295230509027, "job": 0, "event": "table_file_deletion", "file_number": 19106} >2018-07-11 07:47:10.509043 7ffab4be2700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295230509041, "job": 0, "event": "table_file_deletion", "file_number": 19105} >2018-07-11 07:47:10.509057 7ffab4be2700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295230509055, "job": 0, "event": "table_file_deletion", "file_number": 19104} >2018-07-11 07:47:10.509069 7ffab4be2700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295230509067, "job": 0, "event": "table_file_deletion", "file_number": 19102} >2018-07-11 07:47:10.509080 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 21] Compacting 1@1 + 2@2 files to L2, score 2.79 >2018-07-11 07:47:10.509087 7ffab4be2700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295230509085, "job": 0, "event": "table_file_deletion", "file_number": 19100} >2018-07-11 07:47:10.509093 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 20 Base level 1, inputs: [36560(76MB)], [32683(65MB) 32684(1064B)] > >2018-07-11 07:47:10.509099 7ffab4be2700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295230509097, "job": 0, "event": "table_file_deletion", "file_number": 19098} >2018-07-11 07:47:10.509107 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295230509097, "job": 21, "event": "compaction_started", "files_L1": [36560], "files_L2": [32683, 32684], "score": 2.78518, "input_data_size": 148905219} >2018-07-11 07:47:10.509109 7ffab4be2700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295230509107, "job": 0, "event": "table_file_deletion", "file_number": 19096} >2018-07-11 07:47:10.509121 7ffab4be2700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295230509119, "job": 0, "event": "table_file_deletion", "file_number": 19092} >2018-07-11 07:47:16.135951 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 21] Generated table #36748: 2037370 keys, 73169276 bytes >2018-07-11 07:47:16.135994 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295236135967, "cf_name": "default", "job": 21, "event": "table_file_creation", "file_number": 36748, "file_size": 73169276, "table_properties": {"data_size": 67112887, "index_size": 811857, "filter_size": 5243541, "raw_key_size": 61985138, "raw_average_key_size": 30, "raw_value_size": 38957188, "raw_average_value_size": 19, "num_data_blocks": 16492, "num_entries": 2037370, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "357341", "kMergeOperands": "0"}} >2018-07-11 07:47:21.743409 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 21] Generated table #36749: 3722572 keys, 61300207 bytes >2018-07-11 07:47:21.743458 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295241743431, "cf_name": "default", "job": 21, "event": "table_file_creation", "file_number": 36749, "file_size": 61300207, "table_properties": {"data_size": 51346669, "index_size": 527176, "filter_size": 9425371, "raw_key_size": 107540985, "raw_average_key_size": 28, "raw_value_size": 950075, "raw_average_value_size": 0, "num_data_blocks": 12602, "num_entries": 3722572, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "3642659", "kMergeOperands": "0"}} >2018-07-11 07:47:23.038077 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 21] Generated table #36750: 886790 keys, 14373964 bytes >2018-07-11 07:47:23.038118 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295243038091, "cf_name": "default", "job": 21, "event": "table_file_creation", "file_number": 36750, "file_size": 14373964, "table_properties": {"data_size": 12005269, "index_size": 122860, "filter_size": 2244848, "raw_key_size": 25618373, "raw_average_key_size": 28, "raw_value_size": 12, "raw_average_value_size": 0, "num_data_blocks": 2947, "num_entries": 886790, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "886789", "kMergeOperands": "0"}} >2018-07-11 07:47:23.038141 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 21] Compacted 1@1 + 2@2 files to L2 => 148843447 bytes >2018-07-11 07:47:23.057847 7ffabc3f1700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 56511516969 >2018-07-11 07:47:23.058057 7ffabc3f1700 4 rocksdb: (Original Log Time 2018/07/11-07:47:23.057814) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 11 79 454 3641 796 0] max score 2.80, MB/sec: 11.9 rd, 11.9 wr, level 2, files in(1, 2) out(3) MB in(76.5, 65.5) out(141.9), read-write-amplify(3.7) write-amplify(1.9) OK, records in: 6646937, records dropped: 205 > >2018-07-11 07:47:23.058068 7ffabc3f1700 4 rocksdb: (Original Log Time 2018/07/11-07:47:23.057839) EVENT_LOG_v1 {"time_micros": 1531295243057828, "job": 21, "event": "compaction_finished", "compaction_time_micros": 12529022, "output_level": 2, "num_output_files": 3, "total_output_size": 148843447, "num_input_records": 6646937, "num_output_records": 6646732, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 11, 79, 454, 3641, 796, 0]} >2018-07-11 07:47:23.058686 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295243058681, "job": 21, "event": "table_file_deletion", "file_number": 36560} >2018-07-11 07:47:23.058713 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295243058710, "job": 21, "event": "table_file_deletion", "file_number": 32684} >2018-07-11 07:47:23.058735 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295243058732, "job": 21, "event": "table_file_deletion", "file_number": 32683} >2018-07-11 07:47:23.058916 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 22] Compacting 1@2 + 6@3 files to L3, score 2.80 >2018-07-11 07:47:23.058930 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 21 Base level 2, inputs: [36621(76MB)], [28425(71MB) 28482(71MB) 28523(71MB) 28566(71MB) 28626(71MB) 28673(71MB)] > >2018-07-11 07:47:23.058950 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295243058935, "job": 22, "event": "compaction_started", "files_L2": [36621], "files_L3": [28425, 28482, 28523, 28566, 28626, 28673], "score": 2.80346, "input_data_size": 527623152} >2018-07-11 07:47:30.370704 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 22] Generated table #36751: 3092599 keys, 75683284 bytes >2018-07-11 07:47:30.370751 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295250370722, "cf_name": "default", "job": 22, "event": "table_file_creation", "file_number": 36751, "file_size": 75683284, "table_properties": {"data_size": 67111234, "index_size": 690371, "filter_size": 7880687, "raw_key_size": 89341734, "raw_average_key_size": 28, "raw_value_size": 25219748, "raw_average_value_size": 8, "num_data_blocks": 16486, "num_entries": 3092599, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "971311", "kMergeOperands": "0"}} >2018-07-11 07:47:36.423846 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 22] Generated table #36752: 2651679 keys, 74580156 bytes >2018-07-11 07:47:36.423878 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295256423864, "cf_name": "default", "job": 22, "event": "table_file_creation", "file_number": 36752, "file_size": 74580156, "table_properties": {"data_size": 67112584, "index_size": 689754, "filter_size": 6776826, "raw_key_size": 76604065, "raw_average_key_size": 28, "raw_value_size": 31184435, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2651679, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "28689", "kMergeOperands": "0"}} >2018-07-11 07:47:42.596732 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 22] Generated table #36753: 2638240 keys, 74546913 bytes >2018-07-11 07:47:42.596780 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295262596749, "cf_name": "default", "job": 22, "event": "table_file_creation", "file_number": 36753, "file_size": 74546913, "table_properties": {"data_size": 67112109, "index_size": 690633, "filter_size": 6743181, "raw_key_size": 76215825, "raw_average_key_size": 28, "raw_value_size": 31365745, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:47:49.244193 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 22] Generated table #36754: 3106039 keys, 75716860 bytes >2018-07-11 07:47:49.244239 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295269244212, "cf_name": "default", "job": 22, "event": "table_file_creation", "file_number": 36754, "file_size": 75716860, "table_properties": {"data_size": 67111773, "index_size": 689760, "filter_size": 7914335, "raw_key_size": 89730024, "raw_average_key_size": 28, "raw_value_size": 25038471, "raw_average_value_size": 8, "num_data_blocks": 16486, "num_entries": 3106039, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1000000", "kMergeOperands": "0"}} >2018-07-11 07:47:56.722332 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 22] Generated table #36755: 2802576 keys, 74955016 bytes >2018-07-11 07:47:56.722376 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295276722349, "cf_name": "default", "job": 22, "event": "table_file_creation", "file_number": 36755, "file_size": 74955016, "table_properties": {"data_size": 67109585, "index_size": 689840, "filter_size": 7154599, "raw_key_size": 80963278, "raw_average_key_size": 28, "raw_value_size": 29140573, "raw_average_value_size": 10, "num_data_blocks": 16488, "num_entries": 2802576, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "351499", "kMergeOperands": "0"}} >2018-07-11 07:48:03.051827 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 22] Generated table #36756: 3065697 keys, 75614541 bytes >2018-07-11 07:48:03.051873 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295283051846, "cf_name": "default", "job": 22, "event": "table_file_creation", "file_number": 36756, "file_size": 75614541, "table_properties": {"data_size": 67110392, "index_size": 689824, "filter_size": 7813333, "raw_key_size": 88564593, "raw_average_key_size": 28, "raw_value_size": 25582691, "raw_average_value_size": 8, "num_data_blocks": 16486, "num_entries": 3065697, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "913882", "kMergeOperands": "0"}} >2018-07-11 07:48:04.777642 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 22] Generated table #36757: 926638 keys, 20880165 bytes >2018-07-11 07:48:04.777684 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295284777657, "cf_name": "default", "job": 22, "event": "table_file_creation", "file_number": 36757, "file_size": 20880165, "table_properties": {"data_size": 18333343, "index_size": 188194, "filter_size": 2357638, "raw_key_size": 26769539, "raw_average_key_size": 28, "raw_value_size": 5783068, "raw_average_value_size": 6, "num_data_blocks": 4503, "num_entries": 926638, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "440212", "kMergeOperands": "0"}} >2018-07-11 07:48:04.777709 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 22] Compacted 1@2 + 6@3 files to L3 => 471976935 bytes >2018-07-11 07:48:04.790218 7ffabc3f1700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 55636812087 >2018-07-11 07:48:04.790465 7ffabc3f1700 4 rocksdb: (Original Log Time 2018/07/11-07:48:04.790182) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 11 78 455 3641 796 0] max score 2.74, MB/sec: 12.6 rd, 11.3 wr, level 3, files in(1, 6) out(7) MB in(76.6, 426.6) out(450.1), read-write-amplify(12.4) write-amplify(5.9) OK, records in: 20786612, records dropped: 2503144 > >2018-07-11 07:48:04.790475 7ffabc3f1700 4 rocksdb: (Original Log Time 2018/07/11-07:48:04.790210) EVENT_LOG_v1 {"time_micros": 1531295284790196, "job": 22, "event": "compaction_finished", "compaction_time_micros": 41718743, "output_level": 3, "num_output_files": 7, "total_output_size": 471976935, "num_input_records": 20786612, "num_output_records": 18283468, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 11, 78, 455, 3641, 796, 0]} >2018-07-11 07:48:04.791082 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295284791078, "job": 22, "event": "table_file_deletion", "file_number": 36621} >2018-07-11 07:48:04.791110 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295284791108, "job": 22, "event": "table_file_deletion", "file_number": 28673} >2018-07-11 07:48:04.791122 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295284791120, "job": 22, "event": "table_file_deletion", "file_number": 28626} >2018-07-11 07:48:04.791133 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295284791131, "job": 22, "event": "table_file_deletion", "file_number": 28566} >2018-07-11 07:48:04.791143 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295284791141, "job": 22, "event": "table_file_deletion", "file_number": 28523} >2018-07-11 07:48:04.791154 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295284791152, "job": 22, "event": "table_file_deletion", "file_number": 28482} >2018-07-11 07:48:04.791165 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295284791163, "job": 22, "event": "table_file_deletion", "file_number": 28425} >2018-07-11 07:48:04.791342 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 23] Compacting 1@2 + 1@3 files to L3, score 2.74 >2018-07-11 07:48:04.791353 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 22 Base level 2, inputs: [36634(76MB)], [33946(71MB)] > >2018-07-11 07:48:04.791366 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295284791356, "job": 23, "event": "compaction_started", "files_L2": [36634], "files_L3": [33946], "score": 2.74399, "input_data_size": 154894291} >2018-07-11 07:48:11.539639 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 23] Generated table #36758: 3557774 keys, 76846577 bytes >2018-07-11 07:48:11.539686 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295291539658, "cf_name": "default", "job": 23, "event": "table_file_creation", "file_number": 36758, "file_size": 76846577, "table_properties": {"data_size": 67110743, "index_size": 689574, "filter_size": 9045268, "raw_key_size": 102780161, "raw_average_key_size": 28, "raw_value_size": 18927868, "raw_average_value_size": 5, "num_data_blocks": 16482, "num_entries": 3557774, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1965711", "kMergeOperands": "0"}} >2018-07-11 07:48:18.956205 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 23] Generated table #36759: 4037630 keys, 78047844 bytes >2018-07-11 07:48:18.956258 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295298956225, "cf_name": "default", "job": 23, "event": "table_file_creation", "file_number": 36759, "file_size": 78047844, "table_properties": {"data_size": 67110728, "index_size": 689515, "filter_size": 10246608, "raw_key_size": 116642634, "raw_average_key_size": 28, "raw_value_size": 12437888, "raw_average_value_size": 3, "num_data_blocks": 16478, "num_entries": 4037630, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "2991452", "kMergeOperands": "0"}} >2018-07-11 07:48:19.002764 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 23] Generated table #36760: 3 keys, 1076 bytes >2018-07-11 07:48:19.002787 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295299002776, "cf_name": "default", "job": 23, "event": "table_file_creation", "file_number": 36760, "file_size": 1076, "table_properties": {"data_size": 69, "index_size": 27, "filter_size": 18, "raw_key_size": 87, "raw_average_key_size": 29, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 1, "num_entries": 3, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "3", "kMergeOperands": "0"}} >2018-07-11 07:48:19.002818 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 23] Compacted 1@2 + 1@3 files to L3 => 154895497 bytes >2018-07-11 07:48:19.014915 7ffabc3f1700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 55348093193 >2018-07-11 07:48:19.015121 7ffabc3f1700 4 rocksdb: (Original Log Time 2018/07/11-07:48:19.014881) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 11 77 457 3641 796 0] max score 2.68, MB/sec: 10.9 rd, 10.9 wr, level 3, files in(1, 1) out(3) MB in(76.6, 71.1) out(147.7), read-write-amplify(3.9) write-amplify(1.9) OK, records in: 7595407, records dropped: 0 > >2018-07-11 07:48:19.015131 7ffabc3f1700 4 rocksdb: (Original Log Time 2018/07/11-07:48:19.014907) EVENT_LOG_v1 {"time_micros": 1531295299014896, "job": 23, "event": "compaction_finished", "compaction_time_micros": 14211427, "output_level": 3, "num_output_files": 3, "total_output_size": 154895497, "num_input_records": 7595407, "num_output_records": 7595407, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 11, 77, 457, 3641, 796, 0]} >2018-07-11 07:48:19.015749 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295299015745, "job": 23, "event": "table_file_deletion", "file_number": 36634} >2018-07-11 07:48:19.015776 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295299015774, "job": 23, "event": "table_file_deletion", "file_number": 33946} >2018-07-11 07:48:19.016133 7ffabc3f1700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS ------- >2018-07-11 07:48:19.016139 7ffabc3f1700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] >** DB Stats ** >Uptime(secs): 687.5 total, 610.4 interval >Cumulative writes: 597 writes, 916 keys, 597 commit groups, 1.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s >Cumulative WAL: 597 writes, 122 syncs, 4.85 writes per sync, written: 0.00 GB, 0.00 MB/s >Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent >Interval writes: 597 writes, 916 keys, 597 commit groups, 1.0 writes per commit group, ingest: 0.49 MB, 0.00 MB/s >Interval WAL: 597 writes, 122 syncs, 4.85 writes per sync, written: 0.00 MB, 0.00 MB/s >Interval stall: 00:00:0.000 H:M:S, 0.0 percent > >** Compaction Stats [default] ** >Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop >---------------------------------------------------------------------------------------------------------------------------------------------------------- > L1 11/0 524.75 MB 2.2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 > L2 77/0 4.68 GB 2.7 2.7 0.5 2.2 2.7 0.5 0.0 5.2 14.2 14.2 198 7 28.284 173M 205 > L3 457/0 25.60 GB 1.0 7.2 1.0 6.1 6.8 0.7 0.0 6.5 17.8 16.8 412 14 29.434 301M 18M > L4 3641/0 250.00 GB 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 > L5 796/0 55.24 GB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 > Sum 4982/0 336.03 GB 0.0 9.9 1.6 8.3 9.5 1.2 0.0 10220387288.0 16.6 16.0 610 21 29.051 474M 18M > Int 0/0 0.00 KB 0.0 9.9 1.6 8.3 9.5 1.2 0.0 10220387288.0 16.6 16.0 610 21 29.051 474M 18M >Uptime(secs): 687.5 total, 687.5 interval >Flush(GB): cumulative 0.000, interval 0.000 >AddFile(GB): cumulative 0.000, interval 0.000 >AddFile(Total Files): cumulative 0, interval 0 >AddFile(L0 Files): cumulative 0, interval 0 >AddFile(Keys): cumulative 0, interval 0 >Cumulative compaction: 9.52 GB write, 14.18 MB/s write, 9.90 GB read, 14.75 MB/s read, 610.1 seconds >Interval compaction: 9.52 GB write, 14.18 MB/s write, 9.90 GB read, 14.75 MB/s read, 610.1 seconds >Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count > >** File Read Latency Histogram By Level [default] ** > >** Compaction Stats [default] ** >Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop >---------------------------------------------------------------------------------------------------------------------------------------------------------- > L1 11/0 524.75 MB 2.2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 > L2 77/0 4.68 GB 2.7 2.7 0.5 2.2 2.7 0.5 0.0 5.2 14.2 14.2 198 7 28.284 173M 205 > L3 457/0 25.60 GB 1.0 7.2 1.0 6.1 6.8 0.7 0.0 6.5 17.8 16.8 412 14 29.434 301M 18M > L4 3641/0 250.00 GB 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 > L5 796/0 55.24 GB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 > Sum 4982/0 336.03 GB 0.0 9.9 1.6 8.3 9.5 1.2 0.0 10220387288.0 16.6 16.0 610 21 29.051 474M 18M > Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 >Uptime(secs): 687.5 total, 687.5 interval >Flush(GB): cumulative 0.000, interval 0.000 >AddFile(GB): cumulative 0.000, interval 0.000 >AddFile(Total Files): cumulative 0, interval 0 >AddFile(L0 Files): cumulative 0, interval 0 >AddFile(Keys): cumulative 0, interval 0 >Cumulative compaction: 9.52 GB write, 14.18 MB/s write, 9.90 GB read, 14.75 MB/s read, 610.1 seconds >Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds >Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count > >** File Read Latency Histogram By Level [default] ** > >2018-07-11 07:48:19.016256 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 24] Compacting 1@2 + 4@3 files to L3, score 2.68 >2018-07-11 07:48:19.016267 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 23 Base level 2, inputs: [36633(76MB)], [33942(71MB) 33943(71MB) 33944(71MB) 33945(71MB)] > >2018-07-11 07:48:19.016280 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295299016270, "job": 24, "event": "compaction_started", "files_L2": [36633], "files_L3": [33942, 33943, 33944, 33945], "score": 2.68451, "input_data_size": 378513330} >2018-07-11 07:48:26.209085 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 24] Generated table #36761: 3866624 keys, 77618051 bytes >2018-07-11 07:48:26.209136 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295306209118, "cf_name": "default", "job": 24, "event": "table_file_creation", "file_number": 36761, "file_size": 77618051, "table_properties": {"data_size": 67108994, "index_size": 689577, "filter_size": 9818487, "raw_key_size": 111702463, "raw_average_key_size": 28, "raw_value_size": 14748999, "raw_average_value_size": 3, "num_data_blocks": 16479, "num_entries": 3866624, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "2626053", "kMergeOperands": "0"}} >2018-07-11 07:48:32.160989 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 24] Generated table #36762: 2638240 keys, 74546429 bytes >2018-07-11 07:48:32.161032 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295312161006, "cf_name": "default", "job": 24, "event": "table_file_creation", "file_number": 36762, "file_size": 74546429, "table_properties": {"data_size": 67112149, "index_size": 690109, "filter_size": 6743181, "raw_key_size": 76215830, "raw_average_key_size": 28, "raw_value_size": 31365750, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:48:34.755687 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 24] Generated table #36763: 1100848 keys, 31106103 bytes >2018-07-11 07:48:34.755730 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295314755703, "cf_name": "default", "job": 24, "event": "table_file_creation", "file_number": 36763, "file_size": 31106103, "table_properties": {"data_size": 28003604, "index_size": 287809, "filter_size": 2813702, "raw_key_size": 31802278, "raw_average_key_size": 28, "raw_value_size": 13087862, "raw_average_value_size": 11, "num_data_blocks": 6881, "num_entries": 1100848, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:48:41.046586 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 24] Generated table #36764: 2638241 keys, 74545893 bytes >2018-07-11 07:48:41.046630 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295321046603, "cf_name": "default", "job": 24, "event": "table_file_creation", "file_number": 36764, "file_size": 74545893, "table_properties": {"data_size": 67112146, "index_size": 689574, "filter_size": 6743183, "raw_key_size": 76215848, "raw_average_key_size": 28, "raw_value_size": 31365751, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:48:46.448091 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 24] Generated table #36765: 2638848 keys, 74548429 bytes >2018-07-11 07:48:46.448135 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295326448108, "cf_name": "default", "job": 24, "event": "table_file_creation", "file_number": 36765, "file_size": 74548429, "table_properties": {"data_size": 67112161, "index_size": 690574, "filter_size": 6744703, "raw_key_size": 76233375, "raw_average_key_size": 28, "raw_value_size": 31357556, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638848, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1297", "kMergeOperands": "0"}} >2018-07-11 07:48:49.654135 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 24] Generated table #36766: 2032992 keys, 32952073 bytes >2018-07-11 07:48:49.654177 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295329654151, "cf_name": "default", "job": 24, "event": "table_file_creation", "file_number": 36766, "file_size": 32952073, "table_properties": {"data_size": 27522344, "index_size": 282369, "filter_size": 5146373, "raw_key_size": 58730886, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 6755, "num_entries": 2032992, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "2032992", "kMergeOperands": "0"}} >2018-07-11 07:48:49.654208 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 24] Compacted 1@2 + 4@3 files to L3 => 365316978 bytes >2018-07-11 07:48:49.672548 7ffabc3f1700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 54911116026 >2018-07-11 07:48:49.672712 7ffabc3f1700 4 rocksdb: (Original Log Time 2018/07/11-07:48:49.672515) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 11 76 459 3641 796 0] max score 2.63, MB/sec: 12.4 rd, 11.9 wr, level 3, files in(1, 4) out(6) MB in(76.6, 284.4) out(348.4), read-write-amplify(9.3) write-amplify(4.5) OK, records in: 15509439, records dropped: 593646 > >2018-07-11 07:48:49.672721 7ffabc3f1700 4 rocksdb: (Original Log Time 2018/07/11-07:48:49.672541) EVENT_LOG_v1 {"time_micros": 1531295329672529, "job": 24, "event": "compaction_finished", "compaction_time_micros": 30637911, "output_level": 3, "num_output_files": 6, "total_output_size": 365316978, "num_input_records": 15509439, "num_output_records": 14915793, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 11, 76, 459, 3641, 796, 0]} >2018-07-11 07:48:49.673345 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295329673340, "job": 24, "event": "table_file_deletion", "file_number": 36633} >2018-07-11 07:48:49.673361 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295329673359, "job": 24, "event": "table_file_deletion", "file_number": 33945} >2018-07-11 07:48:49.673376 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295329673374, "job": 24, "event": "table_file_deletion", "file_number": 33944} >2018-07-11 07:48:49.673389 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295329673387, "job": 24, "event": "table_file_deletion", "file_number": 33943} >2018-07-11 07:48:49.673402 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295329673400, "job": 24, "event": "table_file_deletion", "file_number": 33942} >2018-07-11 07:48:49.673598 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 25] Compacting 1@2 + 5@3 files to L3, score 2.63 >2018-07-11 07:48:49.673610 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 24 Base level 2, inputs: [36578(76MB)], [24723(71MB) 24726(49MB) 24727(11MB) 24729(59MB) 24730(71MB)] > >2018-07-11 07:48:49.673623 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295329673614, "job": 25, "event": "compaction_started", "files_L2": [36578], "files_L3": [24723, 24726, 24727, 24729, 24730], "score": 2.62503, "input_data_size": 356115702} >2018-07-11 07:48:56.329994 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 25] Generated table #36767: 2677053 keys, 74640362 bytes >2018-07-11 07:48:56.330039 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295336330012, "cf_name": "default", "job": 25, "event": "table_file_creation", "file_number": 36767, "file_size": 74640362, "table_properties": {"data_size": 67109375, "index_size": 689649, "filter_size": 6840346, "raw_key_size": 77337096, "raw_average_key_size": 28, "raw_value_size": 30838029, "raw_average_value_size": 11, "num_data_blocks": 16489, "num_entries": 2677053, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "83200", "kMergeOperands": "0"}} >2018-07-11 07:49:02.961510 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 25] Generated table #36768: 3294649 keys, 76187807 bytes >2018-07-11 07:49:02.961554 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295342961527, "cf_name": "default", "job": 25, "event": "table_file_creation", "file_number": 36768, "file_size": 76187807, "table_properties": {"data_size": 67109993, "index_size": 690298, "filter_size": 8386524, "raw_key_size": 95178741, "raw_average_key_size": 28, "raw_value_size": 22485763, "raw_average_value_size": 6, "num_data_blocks": 16484, "num_entries": 3294649, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1403323", "kMergeOperands": "0"}} >2018-07-11 07:49:05.598843 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 25] Generated table #36769: 1596677 keys, 25880069 bytes >2018-07-11 07:49:05.598884 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295345598858, "cf_name": "default", "job": 25, "event": "table_file_creation", "file_number": 36769, "file_size": 25880069, "table_properties": {"data_size": 21615574, "index_size": 221637, "filter_size": 4041871, "raw_key_size": 46126234, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 5305, "num_entries": 1596677, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1596677", "kMergeOperands": "0"}} >2018-07-11 07:49:06.617795 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 25] Generated table #36770: 419793 keys, 11862075 bytes >2018-07-11 07:49:06.617837 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295346617811, "cf_name": "default", "job": 25, "event": "table_file_creation", "file_number": 36770, "file_size": 11862075, "table_properties": {"data_size": 10678797, "index_size": 109324, "filter_size": 1072968, "raw_key_size": 12127354, "raw_average_key_size": 28, "raw_value_size": 4990873, "raw_average_value_size": 11, "num_data_blocks": 2624, "num_entries": 419793, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:49:12.283457 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 25] Generated table #36771: 2638240 keys, 74546904 bytes >2018-07-11 07:49:12.283503 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295352283476, "cf_name": "default", "job": 25, "event": "table_file_creation", "file_number": 36771, "file_size": 74546904, "table_properties": {"data_size": 67112092, "index_size": 690641, "filter_size": 6743181, "raw_key_size": 76215825, "raw_average_key_size": 28, "raw_value_size": 31365745, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:49:18.253710 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 25] Generated table #36772: 3009877 keys, 75476969 bytes >2018-07-11 07:49:18.253746 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295358253730, "cf_name": "default", "job": 25, "event": "table_file_creation", "file_number": 36772, "file_size": 75476969, "table_properties": {"data_size": 67112576, "index_size": 689811, "filter_size": 7673590, "raw_key_size": 86951998, "raw_average_key_size": 28, "raw_value_size": 26339837, "raw_average_value_size": 8, "num_data_blocks": 16487, "num_entries": 3009877, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "794377", "kMergeOperands": "0"}} >2018-07-11 07:49:18.940444 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 25] Generated table #36773: 204758 keys, 3319292 bytes >2018-07-11 07:49:18.940484 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295358940457, "cf_name": "default", "job": 25, "event": "table_file_creation", "file_number": 36773, "file_size": 3319292, "table_properties": {"data_size": 2771976, "index_size": 27998, "filter_size": 518333, "raw_key_size": 5915241, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 681, "num_entries": 204758, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "204758", "kMergeOperands": "0"}} >2018-07-11 07:49:20.360333 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 25] Generated table #36774: 873965 keys, 14166071 bytes >2018-07-11 07:49:20.360362 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295360360348, "cf_name": "default", "job": 25, "event": "table_file_creation", "file_number": 36774, "file_size": 14166071, "table_properties": {"data_size": 11831616, "index_size": 121087, "filter_size": 2212381, "raw_key_size": 25247874, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 2904, "num_entries": 873965, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "873965", "kMergeOperands": "0"}} >2018-07-11 07:49:20.360392 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 25] Compacted 1@2 + 5@3 files to L3 => 356079549 bytes >2018-07-11 07:49:20.380282 7ffabc3f1700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 54606516650 >2018-07-11 07:49:20.380505 7ffabc3f1700 4 rocksdb: (Original Log Time 2018/07/11-07:49:20.380225) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 11 75 462 3641 796 0] max score 2.57, MB/sec: 11.6 rd, 11.6 wr, level 3, files in(1, 5) out(8) MB in(76.6, 263.0) out(339.6), read-write-amplify(8.9) write-amplify(4.4) OK, records in: 14716742, records dropped: 1730 > >2018-07-11 07:49:20.380515 7ffabc3f1700 4 rocksdb: (Original Log Time 2018/07/11-07:49:20.380273) EVENT_LOG_v1 {"time_micros": 1531295360380261, "job": 25, "event": "compaction_finished", "compaction_time_micros": 30686751, "output_level": 3, "num_output_files": 8, "total_output_size": 356079549, "num_input_records": 14716742, "num_output_records": 14715012, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 11, 75, 462, 3641, 796, 0]} >2018-07-11 07:49:20.381126 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295360381120, "job": 25, "event": "table_file_deletion", "file_number": 36578} >2018-07-11 07:49:20.381155 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295360381153, "job": 25, "event": "table_file_deletion", "file_number": 24730} >2018-07-11 07:49:20.381166 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295360381164, "job": 25, "event": "table_file_deletion", "file_number": 24729} >2018-07-11 07:49:20.381176 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295360381174, "job": 25, "event": "table_file_deletion", "file_number": 24727} >2018-07-11 07:49:20.381186 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295360381184, "job": 25, "event": "table_file_deletion", "file_number": 24726} >2018-07-11 07:49:20.381209 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295360381206, "job": 25, "event": "table_file_deletion", "file_number": 24723} >2018-07-11 07:49:20.381399 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 26] Compacting 1@2 + 11@3 files to L3, score 2.57 >2018-07-11 07:49:20.381416 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 25 Base level 2, inputs: [36580(76MB)], [24865(34MB) 24866(71MB) 24867(1064B) 24868(56MB) 24871(14MB) 24872(71MB) 24875(31MB) 25477(64MB) 25478(71MB) 25479(6752KB) 27359(71MB)] > >2018-07-11 07:49:20.381430 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295360381419, "job": 26, "event": "compaction_started", "files_L2": [36580], "files_L3": [24865, 24866, 24867, 24868, 24871, 24872, 24875, 25477, 25478, 25479, 27359], "score": 2.56555, "input_data_size": 596875439} >2018-07-11 07:49:26.980065 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 26] Generated table #36775: 2717184 keys, 74742589 bytes >2018-07-11 07:49:26.980109 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295366980082, "cf_name": "default", "job": 26, "event": "table_file_creation", "file_number": 36775, "file_size": 74742589, "table_properties": {"data_size": 67110719, "index_size": 690061, "filter_size": 6940817, "raw_key_size": 78496446, "raw_average_key_size": 28, "raw_value_size": 30296632, "raw_average_value_size": 11, "num_data_blocks": 16489, "num_entries": 2717184, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "168870", "kMergeOperands": "0"}} >2018-07-11 07:49:30.124767 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 26] Generated table #36776: 1370167 keys, 38715738 bytes >2018-07-11 07:49:30.124807 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295370124781, "cf_name": "default", "job": 26, "event": "table_file_creation", "file_number": 36776, "file_size": 38715738, "table_properties": {"data_size": 34854617, "index_size": 358074, "filter_size": 3502059, "raw_key_size": 39582604, "raw_average_key_size": 28, "raw_value_size": 16289765, "raw_average_value_size": 11, "num_data_blocks": 8564, "num_entries": 1370167, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:49:36.642720 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 26] Generated table #36777: 3122567 keys, 75758873 bytes >2018-07-11 07:49:36.642766 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295376642740, "cf_name": "default", "job": 26, "event": "table_file_creation", "file_number": 36777, "file_size": 75758873, "table_properties": {"data_size": 67112266, "index_size": 689901, "filter_size": 7955714, "raw_key_size": 90207476, "raw_average_key_size": 28, "raw_value_size": 24815449, "raw_average_value_size": 7, "num_data_blocks": 16486, "num_entries": 3122567, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1035286", "kMergeOperands": "0"}} >2018-07-11 07:49:38.276317 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 26] Generated table #36778: 964714 keys, 15636986 bytes >2018-07-11 07:49:38.276359 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295378276332, "cf_name": "default", "job": 26, "event": "table_file_creation", "file_number": 36778, "file_size": 15636986, "table_properties": {"data_size": 13060153, "index_size": 133739, "filter_size": 2442107, "raw_key_size": 27869530, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 3206, "num_entries": 964714, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "964714", "kMergeOperands": "0"}} >2018-07-11 07:49:43.786007 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 26] Generated table #36779: 2638239 keys, 74546865 bytes >2018-07-11 07:49:43.786057 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295383786026, "cf_name": "default", "job": 26, "event": "table_file_creation", "file_number": 36779, "file_size": 74546865, "table_properties": {"data_size": 67112075, "index_size": 690621, "filter_size": 6743179, "raw_key_size": 76215801, "raw_average_key_size": 28, "raw_value_size": 31365738, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638239, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:49:50.228888 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 26] Generated table #36780: 2638239 keys, 74546198 bytes >2018-07-11 07:49:50.228934 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295390228919, "cf_name": "default", "job": 26, "event": "table_file_creation", "file_number": 36780, "file_size": 74546198, "table_properties": {"data_size": 67112127, "index_size": 689902, "filter_size": 6743179, "raw_key_size": 76215797, "raw_average_key_size": 28, "raw_value_size": 31365734, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638239, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:49:50.848192 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 26] Generated table #36781: 196358 keys, 5548792 bytes >2018-07-11 07:49:50.848218 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295390848206, "cf_name": "default", "job": 26, "event": "table_file_creation", "file_number": 36781, "file_size": 5548792, "table_properties": {"data_size": 4995018, "index_size": 50904, "filter_size": 501884, "raw_key_size": 5672566, "raw_average_key_size": 28, "raw_value_size": 2334480, "raw_average_value_size": 11, "num_data_blocks": 1228, "num_entries": 196358, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:49:56.853485 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 26] Generated table #36782: 3106037 keys, 75716992 bytes >2018-07-11 07:49:56.853532 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295396853515, "cf_name": "default", "job": 26, "event": "table_file_creation", "file_number": 36782, "file_size": 75716992, "table_properties": {"data_size": 67111687, "index_size": 689982, "filter_size": 7914331, "raw_key_size": 89729974, "raw_average_key_size": 28, "raw_value_size": 25038455, "raw_average_value_size": 8, "num_data_blocks": 16486, "num_entries": 3106037, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1000000", "kMergeOperands": "0"}} >2018-07-11 07:50:00.421270 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 26] Generated table #36783: 1950038 keys, 43053093 bytes >2018-07-11 07:50:00.421312 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295400421285, "cf_name": "default", "job": 26, "event": "table_file_creation", "file_number": 36783, "file_size": 43053093, "table_properties": {"data_size": 37705147, "index_size": 387284, "filter_size": 4959672, "raw_key_size": 56334427, "raw_average_key_size": 28, "raw_value_size": 11294891, "raw_average_value_size": 5, "num_data_blocks": 9261, "num_entries": 1950038, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1000000", "kMergeOperands": "0"}} >2018-07-11 07:50:06.438467 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 26] Generated table #36784: 2638240 keys, 74546549 bytes >2018-07-11 07:50:06.438497 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295406438483, "cf_name": "default", "job": 26, "event": "table_file_creation", "file_number": 36784, "file_size": 74546549, "table_properties": {"data_size": 67112140, "index_size": 690237, "filter_size": 6743182, "raw_key_size": 76215825, "raw_average_key_size": 28, "raw_value_size": 31365745, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:50:09.728357 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 26] Generated table #36785: 1427789 keys, 33664720 bytes >2018-07-11 07:50:09.728398 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295409728372, "cf_name": "default", "job": 26, "event": "table_file_creation", "file_number": 36785, "file_size": 33664720, "table_properties": {"data_size": 29722792, "index_size": 305186, "filter_size": 3635752, "raw_key_size": 41247257, "raw_average_key_size": 28, "raw_value_size": 10383543, "raw_average_value_size": 7, "num_data_blocks": 7301, "num_entries": 1427789, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "554408", "kMergeOperands": "0"}} >2018-07-11 07:50:09.728431 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 26] Compacted 1@2 + 11@3 files to L3 => 586477395 bytes >2018-07-11 07:50:09.745874 7ffabc3f1700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 54183249299 >2018-07-11 07:50:09.746149 7ffabc3f1700 4 rocksdb: (Original Log Time 2018/07/11-07:50:09.745840) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 11 74 462 3641 796 0] max score 2.51, MB/sec: 12.1 rd, 11.9 wr, level 3, files in(1, 11) out(11) MB in(76.6, 492.6) out(559.3), read-write-amplify(14.7) write-amplify(7.3) OK, records in: 23237346, records dropped: 467774 > >2018-07-11 07:50:09.746159 7ffabc3f1700 4 rocksdb: (Original Log Time 2018/07/11-07:50:09.745866) EVENT_LOG_v1 {"time_micros": 1531295409745855, "job": 26, "event": "compaction_finished", "compaction_time_micros": 49346982, "output_level": 3, "num_output_files": 11, "total_output_size": 586477395, "num_input_records": 23237346, "num_output_records": 22769572, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 11, 74, 462, 3641, 796, 0]} >2018-07-11 07:50:09.746750 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295409746746, "job": 26, "event": "table_file_deletion", "file_number": 36580} >2018-07-11 07:50:09.746779 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295409746777, "job": 26, "event": "table_file_deletion", "file_number": 27359} >2018-07-11 07:50:09.746792 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295409746790, "job": 26, "event": "table_file_deletion", "file_number": 25479} >2018-07-11 07:50:09.746802 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295409746800, "job": 26, "event": "table_file_deletion", "file_number": 25478} >2018-07-11 07:50:09.746815 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295409746813, "job": 26, "event": "table_file_deletion", "file_number": 25477} >2018-07-11 07:50:09.746827 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295409746825, "job": 26, "event": "table_file_deletion", "file_number": 24875} >2018-07-11 07:50:09.746841 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295409746839, "job": 26, "event": "table_file_deletion", "file_number": 24872} >2018-07-11 07:50:09.746856 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295409746854, "job": 26, "event": "table_file_deletion", "file_number": 24871} >2018-07-11 07:50:09.746869 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295409746867, "job": 26, "event": "table_file_deletion", "file_number": 24868} >2018-07-11 07:50:09.746881 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295409746879, "job": 26, "event": "table_file_deletion", "file_number": 24867} >2018-07-11 07:50:09.746895 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295409746891, "job": 26, "event": "table_file_deletion", "file_number": 24866} >2018-07-11 07:50:09.746906 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295409746904, "job": 26, "event": "table_file_deletion", "file_number": 24865} >2018-07-11 07:50:09.747072 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 27] Compacting 1@2 + 1@3 files to L3, score 2.51 >2018-07-11 07:50:09.747085 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 26 Base level 2, inputs: [36623(76MB)], [36731(73MB)] > >2018-07-11 07:50:09.747111 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295409747100, "job": 27, "event": "compaction_started", "files_L2": [36623], "files_L3": [36731], "score": 2.50607, "input_data_size": 157235503} >2018-07-11 07:50:18.687381 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 27] Generated table #36786: 4853762 keys, 80090008 bytes >2018-07-11 07:50:18.687437 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295418687408, "cf_name": "default", "job": 27, "event": "table_file_creation", "file_number": 36786, "file_size": 80090008, "table_properties": {"data_size": 67109943, "index_size": 689246, "filter_size": 12289827, "raw_key_size": 140219775, "raw_average_key_size": 28, "raw_value_size": 1399048, "raw_average_value_size": 0, "num_data_blocks": 16471, "num_entries": 4853762, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4736084", "kMergeOperands": "0"}} >2018-07-11 07:50:21.494520 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 27] Generated table #36787: 1654033 keys, 34689040 bytes >2018-07-11 07:50:21.494561 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295421494535, "cf_name": "default", "job": 27, "event": "table_file_creation", "file_number": 36787, "file_size": 34689040, "table_properties": {"data_size": 30175324, "index_size": 309631, "filter_size": 4203095, "raw_key_size": 47783179, "raw_average_key_size": 28, "raw_value_size": 7775728, "raw_average_value_size": 4, "num_data_blocks": 7410, "num_entries": 1654033, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1000000", "kMergeOperands": "0"}} >2018-07-11 07:50:24.530357 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 27] Generated table #36788: 1581042 keys, 32626635 bytes >2018-07-11 07:50:24.530397 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295424530371, "cf_name": "default", "job": 27, "event": "table_file_creation", "file_number": 36788, "file_size": 32626635, "table_properties": {"data_size": 28318518, "index_size": 290593, "filter_size": 4016534, "raw_key_size": 45674552, "raw_average_key_size": 28, "raw_value_size": 6907948, "raw_average_value_size": 4, "num_data_blocks": 6954, "num_entries": 1581042, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1000000", "kMergeOperands": "0"}} >2018-07-11 07:50:24.530424 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 27] Compacted 1@2 + 1@3 files to L3 => 147405683 bytes >2018-07-11 07:50:24.545539 7ffabc3f1700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 53757399101 >2018-07-11 07:50:24.545709 7ffabc3f1700 4 rocksdb: (Original Log Time 2018/07/11-07:50:24.545506) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 11 73 464 3641 796 0] max score 2.45, MB/sec: 10.6 rd, 10.0 wr, level 3, files in(1, 1) out(3) MB in(76.6, 73.3) out(140.6), read-write-amplify(3.8) write-amplify(1.8) OK, records in: 8530999, records dropped: 442162 > >2018-07-11 07:50:24.545719 7ffabc3f1700 4 rocksdb: (Original Log Time 2018/07/11-07:50:24.545531) EVENT_LOG_v1 {"time_micros": 1531295424545520, "job": 27, "event": "compaction_finished", "compaction_time_micros": 14783300, "output_level": 3, "num_output_files": 3, "total_output_size": 147405683, "num_input_records": 8530999, "num_output_records": 8088837, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 11, 73, 464, 3641, 796, 0]} >2018-07-11 07:50:24.546335 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295424546332, "job": 27, "event": "table_file_deletion", "file_number": 36731} >2018-07-11 07:50:24.546356 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295424546353, "job": 27, "event": "table_file_deletion", "file_number": 36623} >2018-07-11 07:50:24.562192 7ffabc3f1700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 53427079270 >2018-07-11 07:50:24.562327 7ffabc3f1700 4 rocksdb: (Original Log Time 2018/07/11-07:50:24.546537) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #36629 to level-3 80348267 bytes > >2018-07-11 07:50:24.562336 7ffabc3f1700 4 rocksdb: (Original Log Time 2018/07/11-07:50:24.562235) EVENT_LOG_v1 {"time_micros": 1531295424562208, "job": 28, "event": "trivial_move", "destination_level": 3, "files": 1, "total_files_size": 80348267} >2018-07-11 07:50:24.562340 7ffabc3f1700 4 rocksdb: (Original Log Time 2018/07/11-07:50:24.562241) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-3 80348267 bytes OK: base level 1 max bytes base 268435456 files[0 11 72 465 3641 796 0] max score 2.39 > >2018-07-11 07:50:24.562921 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 29] Compacting 1@2 + 2@3 files to L3, score 2.39 >2018-07-11 07:50:24.562933 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 28 Base level 2, inputs: [36630(76MB)], [29885(71MB) 29888(39MB)] > >2018-07-11 07:50:24.562946 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295424562936, "job": 29, "event": "compaction_started", "files_L2": [36630], "files_L3": [29885, 29888], "score": 2.38711, "input_data_size": 196831602} >2018-07-11 07:50:32.707587 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 29] Generated table #36789: 3945470 keys, 77818049 bytes >2018-07-11 07:50:32.707637 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295432707621, "cf_name": "default", "job": 29, "event": "table_file_creation", "file_number": 36789, "file_size": 77818049, "table_properties": {"data_size": 67111657, "index_size": 689512, "filter_size": 10015887, "raw_key_size": 113980270, "raw_average_key_size": 28, "raw_value_size": 13685253, "raw_average_value_size": 3, "num_data_blocks": 16479, "num_entries": 3945470, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "2794375", "kMergeOperands": "0"}} >2018-07-11 07:50:38.098943 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 29] Generated table #36790: 2922060 keys, 58471085 bytes >2018-07-11 07:50:38.098990 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295438098962, "cf_name": "default", "job": 29, "event": "table_file_creation", "file_number": 36790, "file_size": 58471085, "table_properties": {"data_size": 50531265, "index_size": 519252, "filter_size": 7419577, "raw_key_size": 84415070, "raw_average_key_size": 28, "raw_value_size": 10962270, "raw_average_value_size": 3, "num_data_blocks": 12408, "num_entries": 2922060, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "2000000", "kMergeOperands": "0"}} >2018-07-11 07:50:41.518699 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 29] Generated table #36791: 1646954 keys, 44575869 bytes >2018-07-11 07:50:41.518742 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295441518715, "cf_name": "default", "job": 29, "event": "table_file_creation", "file_number": 36791, "file_size": 44575869, "table_properties": {"data_size": 39958355, "index_size": 411007, "filter_size": 4205517, "raw_key_size": 47578671, "raw_average_key_size": 28, "raw_value_size": 17645072, "raw_average_value_size": 10, "num_data_blocks": 9817, "num_entries": 1646954, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "162790", "kMergeOperands": "0"}} >2018-07-11 07:50:41.518769 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 29] Compacted 1@2 + 2@3 files to L3 => 180865003 bytes >2018-07-11 07:50:41.536850 7ffabc3f1700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 52917360083 >2018-07-11 07:50:41.537061 7ffabc3f1700 4 rocksdb: (Original Log Time 2018/07/11-07:50:41.536813) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 11 71 466 3641 796 0] max score 2.33, MB/sec: 11.6 rd, 10.7 wr, level 3, files in(1, 2) out(3) MB in(76.6, 111.1) out(172.5), read-write-amplify(4.7) write-amplify(2.3) OK, records in: 9079568, records dropped: 565084 > >2018-07-11 07:50:41.537072 7ffabc3f1700 4 rocksdb: (Original Log Time 2018/07/11-07:50:41.536842) EVENT_LOG_v1 {"time_micros": 1531295441536828, "job": 29, "event": "compaction_finished", "compaction_time_micros": 16955808, "output_level": 3, "num_output_files": 3, "total_output_size": 180865003, "num_input_records": 9079568, "num_output_records": 8514484, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 11, 71, 466, 3641, 796, 0]} >2018-07-11 07:50:41.537669 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295441537665, "job": 29, "event": "table_file_deletion", "file_number": 36630} >2018-07-11 07:50:41.537697 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295441537695, "job": 29, "event": "table_file_deletion", "file_number": 29888} >2018-07-11 07:50:41.537708 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295441537706, "job": 29, "event": "table_file_deletion", "file_number": 29885} >2018-07-11 07:50:41.553501 7ffabc3f1700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 52567428968 >2018-07-11 07:50:41.553624 7ffabc3f1700 4 rocksdb: (Original Log Time 2018/07/11-07:50:41.537866) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #36632 to level-3 80348261 bytes > >2018-07-11 07:50:41.553646 7ffabc3f1700 4 rocksdb: (Original Log Time 2018/07/11-07:50:41.553535) EVENT_LOG_v1 {"time_micros": 1531295441553517, "job": 30, "event": "trivial_move", "destination_level": 3, "files": 1, "total_files_size": 80348261} >2018-07-11 07:50:41.553650 7ffabc3f1700 4 rocksdb: (Original Log Time 2018/07/11-07:50:41.553541) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-3 80348261 bytes OK: base level 1 max bytes base 268435456 files[0 11 70 467 3641 796 0] max score 2.27 > >2018-07-11 07:50:41.554206 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 31] Compacting 1@2 + 10@3 files to L3, score 2.27 >2018-07-11 07:50:41.554220 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 30 Base level 2, inputs: [36613(76MB)], [15298(670KB) 16595(70MB) 16598(71MB) 16599(71MB) 16604(71MB) 16605(71MB) 16606(71MB) 16607(39MB) 16609(71MB) 16613(1169B)] > >2018-07-11 07:50:41.554264 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295441554223, "job": 31, "event": "compaction_started", "files_L2": [36613], "files_L3": [15298, 16595, 16598, 16599, 16604, 16605, 16606, 16607, 16609, 16613], "score": 2.26815, "input_data_size": 643980321} >2018-07-11 07:50:47.732232 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 31] Generated table #36792: 2678207 keys, 74643591 bytes >2018-07-11 07:50:47.732281 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295447732254, "cf_name": "default", "job": 31, "event": "table_file_creation", "file_number": 36792, "file_size": 74643591, "table_properties": {"data_size": 67109414, "index_size": 689949, "filter_size": 6843236, "raw_key_size": 77370428, "raw_average_key_size": 28, "raw_value_size": 30822416, "raw_average_value_size": 11, "num_data_blocks": 16489, "num_entries": 2678207, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "85667", "kMergeOperands": "0"}} >2018-07-11 07:50:48.193329 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 31] Generated table #36793: 44107 keys, 1247013 bytes >2018-07-11 07:50:48.193367 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295448193342, "cf_name": "default", "job": 31, "event": "table_file_creation", "file_number": 36793, "file_size": 1247013, "table_properties": {"data_size": 1122028, "index_size": 11268, "filter_size": 112737, "raw_key_size": 1274204, "raw_average_key_size": 28, "raw_value_size": 524385, "raw_average_value_size": 11, "num_data_blocks": 276, "num_entries": 44107, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:50:53.497234 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 31] Generated table #36794: 2638240 keys, 74546899 bytes >2018-07-11 07:50:53.497284 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295453497256, "cf_name": "default", "job": 31, "event": "table_file_creation", "file_number": 36794, "file_size": 74546899, "table_properties": {"data_size": 67112119, "index_size": 690609, "filter_size": 6743181, "raw_key_size": 76215821, "raw_average_key_size": 28, "raw_value_size": 31365741, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:50:58.895624 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 31] Generated table #36795: 2638893 keys, 74548725 bytes >2018-07-11 07:50:58.895673 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295458895641, "cf_name": "default", "job": 31, "event": "table_file_creation", "file_number": 36795, "file_size": 74548725, "table_properties": {"data_size": 67112170, "index_size": 690748, "filter_size": 6744816, "raw_key_size": 76234681, "raw_average_key_size": 28, "raw_value_size": 31356908, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638893, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1397", "kMergeOperands": "0"}} >2018-07-11 07:51:04.277973 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 31] Generated table #36796: 3105384 keys, 75715616 bytes >2018-07-11 07:51:04.278019 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295464277991, "cf_name": "default", "job": 31, "event": "table_file_creation", "file_number": 36796, "file_size": 75715616, "table_properties": {"data_size": 67111682, "index_size": 690247, "filter_size": 7912695, "raw_key_size": 89711112, "raw_average_key_size": 28, "raw_value_size": 25047286, "raw_average_value_size": 8, "num_data_blocks": 16486, "num_entries": 3105384, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "998603", "kMergeOperands": "0"}} >2018-07-11 07:51:06.944218 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 31] Generated table #36797: 1531281 keys, 31220681 bytes >2018-07-11 07:51:06.944256 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295466944233, "cf_name": "default", "job": 31, "event": "table_file_creation", "file_number": 36797, "file_size": 31220681, "table_properties": {"data_size": 27052724, "index_size": 277619, "filter_size": 3889348, "raw_key_size": 44237012, "raw_average_key_size": 28, "raw_value_size": 6316345, "raw_average_value_size": 4, "num_data_blocks": 6643, "num_entries": 1531281, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1000000", "kMergeOperands": "0"}} >2018-07-11 07:51:12.613701 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 31] Generated table #36798: 2638240 keys, 74546031 bytes >2018-07-11 07:51:12.613746 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295472613718, "cf_name": "default", "job": 31, "event": "table_file_creation", "file_number": 36798, "file_size": 74546031, "table_properties": {"data_size": 67112153, "index_size": 689707, "filter_size": 6743181, "raw_key_size": 76215830, "raw_average_key_size": 28, "raw_value_size": 31365750, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:51:20.994143 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 31] Generated table #36799: 2638240 keys, 74546282 bytes >2018-07-11 07:51:20.994192 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295480994162, "cf_name": "default", "job": 31, "event": "table_file_creation", "file_number": 36799, "file_size": 74546282, "table_properties": {"data_size": 67112108, "index_size": 690003, "filter_size": 6743181, "raw_key_size": 76215828, "raw_average_key_size": 28, "raw_value_size": 31365748, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:51:22.132395 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 31] Generated table #36800: 482081 keys, 13622058 bytes >2018-07-11 07:51:22.132438 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295482132422, "cf_name": "default", "job": 31, "event": "table_file_creation", "file_number": 36800, "file_size": 13622058, "table_properties": {"data_size": 12263292, "index_size": 125601, "filter_size": 1232179, "raw_key_size": 13926785, "raw_average_key_size": 28, "raw_value_size": 5731408, "raw_average_value_size": 11, "num_data_blocks": 3014, "num_entries": 482081, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:51:27.289403 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 31] Generated table #36801: 3106037 keys, 75716797 bytes >2018-07-11 07:51:27.289450 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295487289423, "cf_name": "default", "job": 31, "event": "table_file_creation", "file_number": 36801, "file_size": 75716797, "table_properties": {"data_size": 67111720, "index_size": 689754, "filter_size": 7914331, "raw_key_size": 89729961, "raw_average_key_size": 28, "raw_value_size": 25038442, "raw_average_value_size": 8, "num_data_blocks": 16486, "num_entries": 3106037, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1000000", "kMergeOperands": "0"}} >2018-07-11 07:51:28.361577 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 31] Generated table #36802: 532207 keys, 15038478 bytes >2018-07-11 07:51:28.361617 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295488361591, "cf_name": "default", "job": 31, "event": "table_file_creation", "file_number": 36802, "file_size": 15038478, "table_properties": {"data_size": 13538404, "index_size": 138798, "filter_size": 1360290, "raw_key_size": 15374875, "raw_average_key_size": 28, "raw_value_size": 6327356, "raw_average_value_size": 11, "num_data_blocks": 3327, "num_entries": 532207, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:51:29.730412 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 31] Generated table #36803: 871498 keys, 14126086 bytes >2018-07-11 07:51:29.730453 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295489730438, "cf_name": "default", "job": 31, "event": "table_file_creation", "file_number": 36803, "file_size": 14126086, "table_properties": {"data_size": 11798209, "index_size": 120752, "filter_size": 2206138, "raw_key_size": 25176603, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 2896, "num_entries": 871498, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "871498", "kMergeOperands": "0"}} >2018-07-11 07:51:29.730485 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 31] Compacted 1@2 + 10@3 files to L3 => 599518257 bytes >2018-07-11 07:51:29.744224 7ffabc3f1700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 51734252169 >2018-07-11 07:51:29.744498 7ffabc3f1700 4 rocksdb: (Original Log Time 2018/07/11-07:51:29.744190) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 11 69 469 3641 796 0] max score 2.21, MB/sec: 13.4 rd, 12.4 wr, level 3, files in(1, 10) out(12) MB in(76.6, 537.5) out(571.7), read-write-amplify(15.5) write-amplify(7.5) OK, records in: 24904415, records dropped: 2000000 > >2018-07-11 07:51:29.744506 7ffabc3f1700 4 rocksdb: (Original Log Time 2018/07/11-07:51:29.744216) EVENT_LOG_v1 {"time_micros": 1531295489744205, "job": 31, "event": "compaction_finished", "compaction_time_micros": 48176202, "output_level": 3, "num_output_files": 12, "total_output_size": 599518257, "num_input_records": 24904415, "num_output_records": 22904415, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 11, 69, 469, 3641, 796, 0]} >2018-07-11 07:51:29.745127 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295489745110, "job": 31, "event": "table_file_deletion", "file_number": 36613} >2018-07-11 07:51:29.745145 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295489745142, "job": 31, "event": "table_file_deletion", "file_number": 16613} >2018-07-11 07:51:29.745156 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295489745154, "job": 31, "event": "table_file_deletion", "file_number": 16609} >2018-07-11 07:51:29.745167 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295489745165, "job": 31, "event": "table_file_deletion", "file_number": 16607} >2018-07-11 07:51:29.745178 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295489745176, "job": 31, "event": "table_file_deletion", "file_number": 16606} >2018-07-11 07:51:29.745190 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295489745188, "job": 31, "event": "table_file_deletion", "file_number": 16605} >2018-07-11 07:51:29.745203 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295489745201, "job": 31, "event": "table_file_deletion", "file_number": 16604} >2018-07-11 07:51:29.745217 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295489745215, "job": 31, "event": "table_file_deletion", "file_number": 16599} >2018-07-11 07:51:29.745230 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295489745228, "job": 31, "event": "table_file_deletion", "file_number": 16598} >2018-07-11 07:51:29.745241 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295489745239, "job": 31, "event": "table_file_deletion", "file_number": 16595} >2018-07-11 07:51:29.745262 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295489745260, "job": 31, "event": "table_file_deletion", "file_number": 15298} >2018-07-11 07:51:29.745441 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 32] Compacting 1@2 + 2@3 files to L3, score 2.21 >2018-07-11 07:51:29.745453 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 31 Base level 2, inputs: [36582(76MB)], [26977(71MB) 36708(71MB)] > >2018-07-11 07:51:29.745466 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295489745456, "job": 32, "event": "compaction_started", "files_L2": [36582], "files_L3": [26977, 36708], "score": 2.20867, "input_data_size": 229978276} >2018-07-11 07:51:35.618526 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 32] Generated table #36804: 2757316 keys, 74844399 bytes >2018-07-11 07:51:35.618572 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295495618544, "cf_name": "default", "job": 32, "event": "table_file_creation", "file_number": 36804, "file_size": 74844399, "table_properties": {"data_size": 67112064, "index_size": 690050, "filter_size": 7041293, "raw_key_size": 79655814, "raw_average_key_size": 28, "raw_value_size": 29755235, "raw_average_value_size": 10, "num_data_blocks": 16489, "num_entries": 2757316, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "254540", "kMergeOperands": "0"}} >2018-07-11 07:51:42.048127 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 32] Generated table #36805: 4509432 keys, 79228604 bytes >2018-07-11 07:51:42.048180 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295502048152, "cf_name": "default", "job": 32, "event": "table_file_creation", "file_number": 36805, "file_size": 79228604, "table_properties": {"data_size": 67110483, "index_size": 689344, "filter_size": 11427784, "raw_key_size": 130272483, "raw_average_key_size": 28, "raw_value_size": 6056579, "raw_average_value_size": 1, "num_data_blocks": 16474, "num_entries": 4509432, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4000000", "kMergeOperands": "0"}} >2018-07-11 07:51:48.409955 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 32] Generated table #36806: 3182184 keys, 75905673 bytes >2018-07-11 07:51:48.409989 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295508409974, "cf_name": "default", "job": 32, "event": "table_file_creation", "file_number": 36806, "file_size": 75905673, "table_properties": {"data_size": 67110118, "index_size": 689611, "filter_size": 8104952, "raw_key_size": 91929756, "raw_average_key_size": 28, "raw_value_size": 24007109, "raw_average_value_size": 7, "num_data_blocks": 16484, "num_entries": 3182184, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1162895", "kMergeOperands": "0"}} >2018-07-11 07:51:48.410018 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 32] Compacted 1@2 + 2@3 files to L3 => 229978676 bytes >2018-07-11 07:51:48.426988 7ffabc3f1700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 51362925809 >2018-07-11 07:51:48.427200 7ffabc3f1700 4 rocksdb: (Original Log Time 2018/07/11-07:51:48.426957) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 11 68 470 3641 796 0] max score 2.20, MB/sec: 12.3 rd, 12.3 wr, level 3, files in(1, 2) out(3) MB in(76.6, 142.7) out(219.3), read-write-amplify(5.7) write-amplify(2.9) OK, records in: 10448932, records dropped: 0 > >2018-07-11 07:51:48.427210 7ffabc3f1700 4 rocksdb: (Original Log Time 2018/07/11-07:51:48.426981) EVENT_LOG_v1 {"time_micros": 1531295508426971, "job": 32, "event": "compaction_finished", "compaction_time_micros": 18664538, "output_level": 3, "num_output_files": 3, "total_output_size": 229978676, "num_input_records": 10448932, "num_output_records": 10448932, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 11, 68, 470, 3641, 796, 0]} >2018-07-11 07:51:48.427825 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295508427821, "job": 32, "event": "table_file_deletion", "file_number": 36708} >2018-07-11 07:51:48.427855 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295508427853, "job": 32, "event": "table_file_deletion", "file_number": 36582} >2018-07-11 07:51:48.427869 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295508427867, "job": 32, "event": "table_file_deletion", "file_number": 26977} >2018-07-11 07:51:48.427961 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_picker.cc:461] [default] Expanding@1 1+1(74546956+68114896 bytes) to 4+1 (298185581+68114896bytes) > >2018-07-11 07:51:48.428046 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 33] Compacting 4@1 + 1@2 files to L2, score 2.20 >2018-07-11 07:51:48.428057 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 32 Base level 1, inputs: [36572(71MB) 36573(71MB) 36574(71MB) 36575(71MB)], [36290(64MB)] > >2018-07-11 07:51:48.428071 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295508428061, "job": 33, "event": "compaction_started", "files_L1": [36572, 36573, 36574, 36575], "files_L2": [36290], "score": 2.19535, "input_data_size": 366300477} >2018-07-11 07:51:54.478371 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 33] Generated table #36807: 2638240 keys, 74546968 bytes >2018-07-11 07:51:54.478418 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295514478402, "cf_name": "default", "job": 33, "event": "table_file_creation", "file_number": 36807, "file_size": 74546968, "table_properties": {"data_size": 67112152, "index_size": 690645, "filter_size": 6743181, "raw_key_size": 76215836, "raw_average_key_size": 28, "raw_value_size": 31365756, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:52:00.259916 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 33] Generated table #36808: 2638241 keys, 74546238 bytes >2018-07-11 07:52:00.259959 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295520259932, "cf_name": "default", "job": 33, "event": "table_file_creation", "file_number": 36808, "file_size": 74546238, "table_properties": {"data_size": 67112128, "index_size": 689937, "filter_size": 6743183, "raw_key_size": 76215852, "raw_average_key_size": 28, "raw_value_size": 31365755, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:52:05.857258 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 33] Generated table #36809: 2638242 keys, 74546979 bytes >2018-07-11 07:52:05.857299 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295525857273, "cf_name": "default", "job": 33, "event": "table_file_creation", "file_number": 36809, "file_size": 74546979, "table_properties": {"data_size": 67112152, "index_size": 690651, "filter_size": 6743186, "raw_key_size": 76215888, "raw_average_key_size": 28, "raw_value_size": 31365774, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638242, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:52:11.713466 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 33] Generated table #36810: 2638241 keys, 74546373 bytes >2018-07-11 07:52:11.713498 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295531713484, "cf_name": "default", "job": 33, "event": "table_file_creation", "file_number": 36810, "file_size": 74546373, "table_properties": {"data_size": 67112143, "index_size": 690057, "filter_size": 6743183, "raw_key_size": 76215847, "raw_average_key_size": 28, "raw_value_size": 31365750, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:52:17.036705 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 33] Generated table #36811: 2406128 keys, 68114287 bytes >2018-07-11 07:52:17.036749 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295537036722, "cf_name": "default", "job": 33, "event": "table_file_creation", "file_number": 36811, "file_size": 68114287, "table_properties": {"data_size": 61332638, "index_size": 630459, "filter_size": 6150201, "raw_key_size": 69526219, "raw_average_key_size": 28, "raw_value_size": 28711960, "raw_average_value_size": 11, "num_data_blocks": 15071, "num_entries": 2406128, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "11", "kMergeOperands": "0"}} >2018-07-11 07:52:17.036786 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 33] Compacted 4@1 + 1@2 files to L2 => 366300845 bytes >2018-07-11 07:52:17.051339 7ffabc3f1700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 49121545030 >2018-07-11 07:52:17.051576 7ffabc3f1700 4 rocksdb: (Original Log Time 2018/07/11-07:52:17.051307) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 7 72 470 3641 796 0] max score 2.26, MB/sec: 12.8 rd, 12.8 wr, level 2, files in(4, 1) out(5) MB in(284.4, 65.0) out(349.3), read-write-amplify(2.5) write-amplify(1.2) OK, records in: 12959092, records dropped: 0 > >2018-07-11 07:52:17.051585 7ffabc3f1700 4 rocksdb: (Original Log Time 2018/07/11-07:52:17.051332) EVENT_LOG_v1 {"time_micros": 1531295537051321, "job": 33, "event": "compaction_finished", "compaction_time_micros": 28608701, "output_level": 2, "num_output_files": 5, "total_output_size": 366300845, "num_input_records": 12959092, "num_output_records": 12959092, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 7, 72, 470, 3641, 796, 0]} >2018-07-11 07:52:17.052219 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295537052203, "job": 33, "event": "table_file_deletion", "file_number": 36575} >2018-07-11 07:52:17.052235 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295537052233, "job": 33, "event": "table_file_deletion", "file_number": 36574} >2018-07-11 07:52:17.052255 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295537052253, "job": 33, "event": "table_file_deletion", "file_number": 36573} >2018-07-11 07:52:17.052271 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295537052268, "job": 33, "event": "table_file_deletion", "file_number": 36572} >2018-07-11 07:52:17.052289 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295537052287, "job": 33, "event": "table_file_deletion", "file_number": 36290} >2018-07-11 07:52:17.052468 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 34] Compacting 1@2 + 8@3 files to L3, score 2.26 >2018-07-11 07:52:17.052483 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 33 Base level 2, inputs: [36612(76MB)], [15174(71MB) 15177(577KB) 15288(70MB) 15289(71MB) 15290(71MB) 15291(71MB) 15292(71MB) 15293(1249B)] > >2018-07-11 07:52:17.052497 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295537052486, "job": 34, "event": "compaction_started", "files_L2": [36612], "files_L3": [15174, 15177, 15288, 15289, 15290, 15291, 15292, 15293], "score": 2.26028, "input_data_size": 527530353} >2018-07-11 07:52:22.862199 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 34] Generated table #36812: 2658304 keys, 74596892 bytes >2018-07-11 07:52:22.862241 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295542862215, "cf_name": "default", "job": 34, "event": "table_file_creation", "file_number": 36812, "file_size": 74596892, "table_properties": {"data_size": 67112830, "index_size": 689658, "filter_size": 6793412, "raw_key_size": 76795453, "raw_average_key_size": 28, "raw_value_size": 31095052, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2658304, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "42832", "kMergeOperands": "0"}} >2018-07-11 07:52:28.267153 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 34] Generated table #36813: 2638237 keys, 74546406 bytes >2018-07-11 07:52:28.267197 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295548267171, "cf_name": "default", "job": 34, "event": "table_file_creation", "file_number": 36813, "file_size": 74546406, "table_properties": {"data_size": 67112112, "index_size": 690129, "filter_size": 6743175, "raw_key_size": 76215749, "raw_average_key_size": 28, "raw_value_size": 31365720, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638237, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:52:28.630885 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 34] Generated table #36814: 19370 keys, 548228 bytes >2018-07-11 07:52:28.630924 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295548630910, "cf_name": "default", "job": 34, "event": "table_file_creation", "file_number": 36814, "file_size": 548228, "table_properties": {"data_size": 492756, "index_size": 4977, "filter_size": 49516, "raw_key_size": 559577, "raw_average_key_size": 28, "raw_value_size": 230287, "raw_average_value_size": 11, "num_data_blocks": 122, "num_entries": 19370, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:52:36.237080 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 34] Generated table #36815: 3672198 keys, 77132968 bytes >2018-07-11 07:52:36.237128 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295556237100, "cf_name": "default", "job": 34, "event": "table_file_creation", "file_number": 36815, "file_size": 77132968, "table_properties": {"data_size": 67110610, "index_size": 689631, "filter_size": 9331734, "raw_key_size": 106085701, "raw_average_key_size": 28, "raw_value_size": 17380103, "raw_average_value_size": 4, "num_data_blocks": 16481, "num_entries": 3672198, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "2210319", "kMergeOperands": "0"}} >2018-07-11 07:52:40.598802 7ffabc3f1700 -1 bluefs _allocate failed to allocate 0x4800000 on bdev 2, free 0x0 >2018-07-11 07:52:40.598814 7ffabc3f1700 0 stupidalloc 0x0x55d97f6aa070 dump free bin 0: 0 extents >2018-07-11 07:52:40.598816 7ffabc3f1700 0 stupidalloc 0x0x55d97f6aa070 dump free bin 1: 0 extents >2018-07-11 07:52:40.598830 7ffabc3f1700 0 stupidalloc 0x0x55d97f6aa070 dump free bin 2: 0 extents >2018-07-11 07:52:40.598832 7ffabc3f1700 0 stupidalloc 0x0x55d97f6aa070 dump free bin 3: 0 extents >2018-07-11 07:52:40.598833 7ffabc3f1700 0 stupidalloc 0x0x55d97f6aa070 dump free bin 4: 0 extents >2018-07-11 07:52:40.598834 7ffabc3f1700 0 stupidalloc 0x0x55d97f6aa070 dump free bin 5: 0 extents >2018-07-11 07:52:40.598835 7ffabc3f1700 0 stupidalloc 0x0x55d97f6aa070 dump free bin 6: 0 extents >2018-07-11 07:52:40.598836 7ffabc3f1700 0 stupidalloc 0x0x55d97f6aa070 dump free bin 7: 0 extents >2018-07-11 07:52:40.598837 7ffabc3f1700 0 stupidalloc 0x0x55d97f6aa070 dump free bin 8: 0 extents >2018-07-11 07:52:40.598838 7ffabc3f1700 0 stupidalloc 0x0x55d97f6aa070 dump free bin 9: 0 extents >2018-07-11 07:52:40.598839 7ffabc3f1700 -1 bluefs _flush_range allocated: 0x0 offset: 0x0 length: 0x4717d2d >2018-07-11 07:52:40.604936 7ffabc3f1700 -1 /builddir/build/BUILD/ceph-12.2.5/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, uint64_t)' thread 7ffabc3f1700 time 2018-07-11 07:52:40.598851 >/builddir/build/BUILD/ceph-12.2.5/src/os/bluestore/BlueFS.cc: 1525: FAILED assert(0 == "bluefs enospc") > > ceph version 12.2.5-27.el7cp (6b7d1a8cf2e2376fd0a12424947c8c97aa1c0890) luminous (stable) > 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x110) [0x55d97539a710] > 2: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1b1b) [0x55d97531913b] > 3: (BlueFS::_flush(BlueFS::FileWriter*, bool)+0x188) [0x55d9753196a8] > 4: (BlueRocksWritableFile::Flush()+0x3c) [0x55d97533523c] > 5: (rocksdb::WritableFileWriter::Flush()+0xaf) [0x55d97570455f] > 6: (rocksdb::WritableFileWriter::Sync(bool)+0x33) [0x55d975704773] > 7: (rocksdb::CompactionJob::FinishCompactionOutputFile(rocksdb::Status const&, rocksdb::CompactionJob::SubcompactionState*, rocksdb::RangeDelAggregator*, CompactionIterationStats*, rocksdb::Slice const*)+0xa1b) [0x55d9757ca7ab] > 8: (rocksdb::CompactionJob::ProcessKeyValueCompaction(rocksdb::CompactionJob::SubcompactionState*)+0xea2) [0x55d9757cd222] > 9: (rocksdb::CompactionJob::Run()+0x1de) [0x55d9757cdcde] > 10: (rocksdb::DBImpl::BackgroundCompaction(bool*, rocksdb::JobContext*, rocksdb::LogBuffer*, void*)+0x9c7) [0x55d975759687] > 11: (rocksdb::DBImpl::BackgroundCallCompaction(void*)+0x283) [0x55d97575cc73] > 12: (rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long)+0x271) [0x55d9757c3e31] > 13: (rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*)+0x3d) [0x55d9757c401d] > 14: (()+0xb5070) [0x7ffacde3f070] > 15: (()+0x7dd5) [0x7fface4b2dd5] > 16: (clone()+0x6d) [0x7ffacd5a3b3d] > NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. > >--- begin dump of recent events --- >-10000> 2018-07-11 07:50:23.216825 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:750efa74:::benchmark_data_magna115_10867_object786990:0,end:5:750f7031:::benchmark_data_magna115_10867_object2917475:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98696ef00 con 0 > -9999> 2018-07-11 07:50:23.216870 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:750efa74:::benchmark_data_magna115_10867_object786990:0,end:5:750f7031:::benchmark_data_magna115_10867_object2917475:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98696f900 con 0 > -9998> 2018-07-11 07:50:23.257803 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15255 conn(0x55d986c82800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.0 seq 231 0x55d9c9122a00 osd_ping(ping e501 stamp 2018-07-11 07:50:24.487700) v4 > -9997> 2018-07-11 07:50:23.257824 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.0 10.8.128.113:0/15255 231 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:24.487700) v4 ==== 2004+0+0 (620523567 0 0) 0x55d9c9122a00 con 0x55d986c82800 > -9996> 2018-07-11 07:50:23.257833 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.487700) v4 -- 0x55d9b9ac3000 con 0 > -9995> 2018-07-11 07:50:23.257825 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15255 conn(0x55d986d20000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.0 seq 231 0x55d9869be400 osd_ping(ping e501 stamp 2018-07-11 07:50:24.487700) v4 > -9994> 2018-07-11 07:50:23.257848 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.0 10.8.128.113:0/15255 231 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:24.487700) v4 ==== 2004+0+0 (620523567 0 0) 0x55d9869be400 con 0x55d986d20000 > -9993> 2018-07-11 07:50:23.257862 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.487700) v4 -- 0x55d9c4c1a400 con 0 > -9992> 2018-07-11 07:50:23.413424 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 767 0x55d9c82cbb00 rep_scrubmap(5.2e e501 from shard 2) v1 > -9991> 2018-07-11 07:50:23.413445 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 767 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12468 (1269249638 0 1783063182) 0x55d9c82cbb00 con 0x55d986bf0800 > -9990> 2018-07-11 07:50:23.424026 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 761 0x55d9c82cbd40 rep_scrubmap(5.2e e501 from shard 4) v1 > -9989> 2018-07-11 07:50:23.424045 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 761 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12468 (3686815630 0 1783063182) 0x55d9c82cbd40 con 0x55d986bec000 > -9988> 2018-07-11 07:50:23.455744 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12677 conn(0x55d986db4800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.1 seq 233 0x55d9829e5000 osd_ping(ping e501 stamp 2018-07-11 07:50:23.502576) v4 > -9987> 2018-07-11 07:50:23.455753 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12677 conn(0x55d986df2000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.1 seq 233 0x55d986cd4c00 osd_ping(ping e501 stamp 2018-07-11 07:50:23.502576) v4 > -9986> 2018-07-11 07:50:23.455771 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.1 10.8.128.106:0/12677 233 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:23.502576) v4 ==== 2004+0+0 (257096371 0 0) 0x55d9829e5000 con 0x55d986db4800 > -9985> 2018-07-11 07:50:23.455772 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.1 10.8.128.106:0/12677 233 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:23.502576) v4 ==== 2004+0+0 (257096371 0 0) 0x55d986cd4c00 con 0x55d986df2000 > -9984> 2018-07-11 07:50:23.455786 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:23.502576) v4 -- 0x55d9b9ac3000 con 0 > -9983> 2018-07-11 07:50:23.455811 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:23.502576) v4 -- 0x55d9c4c1a400 con 0 > -9982> 2018-07-11 07:50:23.520332 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -9981> 2018-07-11 07:50:23.520355 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -9980> 2018-07-11 07:50:23.520409 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -9979> 2018-07-11 07:50:23.520894 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -9978> 2018-07-11 07:50:23.574318 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15287 conn(0x55d981240800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 240 0x55d9869be400 osd_ping(ping e501 stamp 2018-07-11 07:50:24.804163) v4 > -9977> 2018-07-11 07:50:23.574342 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.4 10.8.128.113:0/15287 240 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:24.804163) v4 ==== 2004+0+0 (364861313 0 0) 0x55d9869be400 con 0x55d981240800 > -9976> 2018-07-11 07:50:23.574339 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15287 conn(0x55d98123f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 240 0x55d9c9122a00 osd_ping(ping e501 stamp 2018-07-11 07:50:24.804163) v4 > -9975> 2018-07-11 07:50:23.574351 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.804163) v4 -- 0x55d9c4c1a400 con 0 > -9974> 2018-07-11 07:50:23.574366 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.4 10.8.128.113:0/15287 240 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:24.804163) v4 ==== 2004+0+0 (364861313 0 0) 0x55d9c9122a00 con 0x55d98123f000 > -9973> 2018-07-11 07:50:23.574391 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.804163) v4 -- 0x55d9b9ac3000 con 0 > -9972> 2018-07-11 07:50:23.577898 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67367,epoch:501/498,start:5:750f7031:::benchmark_data_magna115_10867_object2917475:0,end:5:750ff02b:::benchmark_data_magna115_10867_object1126773:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98696f180 con 0 > -9971> 2018-07-11 07:50:23.577948 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67367,epoch:501/498,start:5:750f7031:::benchmark_data_magna115_10867_object2917475:0,end:5:750ff02b:::benchmark_data_magna115_10867_object1126773:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98696e000 con 0 > -9970> 2018-07-11 07:50:23.705748 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12650 conn(0x55d986d23000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 221 0x55d9829e5000 osd_ping(ping e501 stamp 2018-07-11 07:50:23.752622) v4 > -9969> 2018-07-11 07:50:23.705769 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.3 10.8.128.106:0/12650 221 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:23.752622) v4 ==== 2004+0+0 (2883258492 0 0) 0x55d9829e5000 con 0x55d986d23000 > -9968> 2018-07-11 07:50:23.705778 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:23.752622) v4 -- 0x55d9b9ac3000 con 0 > -9967> 2018-07-11 07:50:23.705769 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12650 conn(0x55d986d21800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 221 0x55d9bfbde200 osd_ping(ping e501 stamp 2018-07-11 07:50:23.752622) v4 > -9966> 2018-07-11 07:50:23.705786 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.3 10.8.128.106:0/12650 221 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:23.752622) v4 ==== 2004+0+0 (2883258492 0 0) 0x55d9bfbde200 con 0x55d986d21800 > -9965> 2018-07-11 07:50:23.705795 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:23.752622) v4 -- 0x55d980f5a200 con 0 > -9964> 2018-07-11 07:50:23.801422 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 768 0x55d9c5bb0000 rep_scrubmap(5.2e e501 from shard 2) v1 > -9963> 2018-07-11 07:50:23.801443 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 768 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12476 (1269249638 0 1311475831) 0x55d9c5bb0000 con 0x55d986bf0800 > -9962> 2018-07-11 07:50:23.806174 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 762 0x55d9c5bb0240 rep_scrubmap(5.2e e501 from shard 4) v1 > -9961> 2018-07-11 07:50:23.806193 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 762 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12476 (3686815630 0 1311475831) 0x55d9c5bb0240 con 0x55d986bec000 > -9960> 2018-07-11 07:50:23.827509 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 2954 0x55d982cdee00 MOSDScrubReserve(11.3s5 REQUEST e501) v1 > -9959> 2018-07-11 07:50:23.827528 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 2954 ==== MOSDScrubReserve(11.3s5 REQUEST e501) v1 ==== 43+0+0 (3260170403 0 0) 0x55d982cdee00 con 0x55d986c7e000 > -9958> 2018-07-11 07:50:23.827602 7ffab3be0700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e501) v1 -- 0x55d9bcd78600 con 0 > -9957> 2018-07-11 07:50:23.827954 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 2955 0x55d9bcd78600 MOSDScrubReserve(11.3s5 RELEASE e501) v1 > -9956> 2018-07-11 07:50:23.827968 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 2955 ==== MOSDScrubReserve(11.3s5 RELEASE e501) v1 ==== 43+0+0 (223328929 0 0) 0x55d9bcd78600 con 0x55d986c7e000 > -9955> 2018-07-11 07:50:23.953061 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6801/15919 conn(0x55d986c3f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=29 cs=1 l=0). rx osd.11 seq 198 0x55d9bfbde200 MOSDScrubReserve(10.f REQUEST e501) v1 > -9954> 2018-07-11 07:50:23.953079 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.11 10.8.128.114:6801/15919 198 ==== MOSDScrubReserve(10.f REQUEST e501) v1 ==== 43+0+0 (2892966580 0 0) 0x55d9bfbde200 con 0x55d986c3f800 > -9953> 2018-07-11 07:50:23.953131 7ffab53e3700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.114:6801/15919 -- MOSDScrubReserve(10.f REJECT e501) v1 -- 0x55d9c34cc600 con 0 > -9952> 2018-07-11 07:50:23.953409 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6801/15919 conn(0x55d986c3f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=29 cs=1 l=0). rx osd.11 seq 199 0x55d9869bea00 MOSDScrubReserve(10.f RELEASE e501) v1 > -9951> 2018-07-11 07:50:23.953423 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.11 10.8.128.114:6801/15919 199 ==== MOSDScrubReserve(10.f RELEASE e501) v1 ==== 43+0+0 (1668468918 0 0) 0x55d9869bea00 con 0x55d986c3f800 > -9950> 2018-07-11 07:50:24.208222 7ffab1bdc700 5 osd.10 501 heartbeat: osd_stat(787 GB used, 143 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,0,2]) > -9949> 2018-07-11 07:50:24.208237 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6802/15255 -- osd_ping(ping e501 stamp 2018-07-11 07:50:24.208235) v4 -- 0x55d9c3383600 con 0 > -9948> 2018-07-11 07:50:24.208287 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6803/15255 -- osd_ping(ping e501 stamp 2018-07-11 07:50:24.208235) v4 -- 0x55d9829e4600 con 0 > -9947> 2018-07-11 07:50:24.208303 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6806/12677 -- osd_ping(ping e501 stamp 2018-07-11 07:50:24.208235) v4 -- 0x55d986d97200 con 0 > -9946> 2018-07-11 07:50:24.208330 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6807/12677 -- osd_ping(ping e501 stamp 2018-07-11 07:50:24.208235) v4 -- 0x55d9869bf200 con 0 > -9945> 2018-07-11 07:50:24.208342 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6808/12651 -- osd_ping(ping e501 stamp 2018-07-11 07:50:24.208235) v4 -- 0x55d9c9122000 con 0 > -9944> 2018-07-11 07:50:24.208354 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6810/12651 -- osd_ping(ping e501 stamp 2018-07-11 07:50:24.208235) v4 -- 0x55d9b437a200 con 0 > -9943> 2018-07-11 07:50:24.208366 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6809/12650 -- osd_ping(ping e501 stamp 2018-07-11 07:50:24.208235) v4 -- 0x55d9b67d9e00 con 0 > -9942> 2018-07-11 07:50:24.208380 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6811/12650 -- osd_ping(ping e501 stamp 2018-07-11 07:50:24.208235) v4 -- 0x55d9b60df800 con 0 > -9941> 2018-07-11 07:50:24.208397 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6810/15287 -- osd_ping(ping e501 stamp 2018-07-11 07:50:24.208235) v4 -- 0x55d986d96600 con 0 > -9940> 2018-07-11 07:50:24.208408 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6811/15287 -- osd_ping(ping e501 stamp 2018-07-11 07:50:24.208235) v4 -- 0x55d9c3382200 con 0 > -9939> 2018-07-11 07:50:24.208421 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6806/15263 -- osd_ping(ping e501 stamp 2018-07-11 07:50:24.208235) v4 -- 0x55d9bdba5200 con 0 > -9938> 2018-07-11 07:50:24.208435 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6807/15263 -- osd_ping(ping e501 stamp 2018-07-11 07:50:24.208235) v4 -- 0x55d9c3383200 con 0 > -9937> 2018-07-11 07:50:24.208447 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.115:6802/13323 -- osd_ping(ping e501 stamp 2018-07-11 07:50:24.208235) v4 -- 0x55d9c3382e00 con 0 > -9936> 2018-07-11 07:50:24.208459 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.115:6803/13323 -- osd_ping(ping e501 stamp 2018-07-11 07:50:24.208235) v4 -- 0x55d9869bec00 con 0 > -9935> 2018-07-11 07:50:24.208472 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6806/15885 -- osd_ping(ping e501 stamp 2018-07-11 07:50:24.208235) v4 -- 0x55d9b9ac2a00 con 0 > -9934> 2018-07-11 07:50:24.208484 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6807/15885 -- osd_ping(ping e501 stamp 2018-07-11 07:50:24.208235) v4 -- 0x55d986cd7600 con 0 > -9933> 2018-07-11 07:50:24.208496 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.108:6802/11967 -- osd_ping(ping e501 stamp 2018-07-11 07:50:24.208235) v4 -- 0x55d980f5be00 con 0 > -9932> 2018-07-11 07:50:24.208508 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.108:6803/11967 -- osd_ping(ping e501 stamp 2018-07-11 07:50:24.208235) v4 -- 0x55d9c908ec00 con 0 > -9931> 2018-07-11 07:50:24.208520 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.107:6806/12617 -- osd_ping(ping e501 stamp 2018-07-11 07:50:24.208235) v4 -- 0x55d986d96c00 con 0 > -9930> 2018-07-11 07:50:24.208534 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.107:6807/12617 -- osd_ping(ping e501 stamp 2018-07-11 07:50:24.208235) v4 -- 0x55d986b6d400 con 0 > -9929> 2018-07-11 07:50:24.208546 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6802/15919 -- osd_ping(ping e501 stamp 2018-07-11 07:50:24.208235) v4 -- 0x55d982cde600 con 0 > -9928> 2018-07-11 07:50:24.208561 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6803/15919 -- osd_ping(ping e501 stamp 2018-07-11 07:50:24.208235) v4 -- 0x55d9bdba5600 con 0 > -9927> 2018-07-11 07:50:24.208695 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.107:6806/12617 conn(0x55d986bd2000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.9 seq 222 0x55d9829e5000 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.208235) v4 > -9926> 2018-07-11 07:50:24.208722 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.9 10.8.128.107:6806/12617 222 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.208235) v4 ==== 2004+0+0 (27127593 0 0) 0x55d9829e5000 con 0x55d986bd2000 > -9925> 2018-07-11 07:50:24.208721 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.107:6807/12617 conn(0x55d986bd3800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.9 seq 222 0x55d980f5a200 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.208235) v4 > -9924> 2018-07-11 07:50:24.208745 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.9 10.8.128.107:6807/12617 222 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.208235) v4 ==== 2004+0+0 (27127593 0 0) 0x55d980f5a200 con 0x55d986bd3800 > -9923> 2018-07-11 07:50:24.208842 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6806/12677 conn(0x55d986b58000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.1 seq 222 0x55d980f5a200 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.208235) v4 > -9922> 2018-07-11 07:50:24.208841 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6810/12651 conn(0x55d986b9b800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.2 seq 222 0x55d9829e5000 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.208235) v4 > -9921> 2018-07-11 07:50:24.208858 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.1 10.8.128.106:6806/12677 222 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.208235) v4 ==== 2004+0+0 (27127593 0 0) 0x55d980f5a200 con 0x55d986b58000 > -9920> 2018-07-11 07:50:24.208861 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.2 10.8.128.106:6810/12651 222 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.208235) v4 ==== 2004+0+0 (27127593 0 0) 0x55d9829e5000 con 0x55d986b9b800 > -9919> 2018-07-11 07:50:24.208907 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6809/12650 conn(0x55d986bcb000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 222 0x55d980f5a200 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.208235) v4 > -9918> 2018-07-11 07:50:24.208908 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6802/15255 conn(0x55d986bc8000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.0 seq 222 0x55d9869be400 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.208235) v4 > -9917> 2018-07-11 07:50:24.208922 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.3 10.8.128.106:6809/12650 222 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.208235) v4 ==== 2004+0+0 (27127593 0 0) 0x55d980f5a200 con 0x55d986bcb000 > -9916> 2018-07-11 07:50:24.208926 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.0 10.8.128.113:6802/15255 222 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.208235) v4 ==== 2004+0+0 (27127593 0 0) 0x55d9869be400 con 0x55d986bc8000 > -9915> 2018-07-11 07:50:24.208955 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6806/15263 conn(0x55d986b9d000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.5 seq 222 0x55d980f5a200 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.208235) v4 > -9914> 2018-07-11 07:50:24.208950 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6803/15255 conn(0x55d986bc9800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.0 seq 222 0x55d9829e5000 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.208235) v4 > -9913> 2018-07-11 07:50:24.208962 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6808/12651 conn(0x55d986b9a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.2 seq 222 0x55d9869be400 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.208235) v4 > -9912> 2018-07-11 07:50:24.208968 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.5 10.8.128.113:6806/15263 222 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.208235) v4 ==== 2004+0+0 (27127593 0 0) 0x55d980f5a200 con 0x55d986b9d000 > -9911> 2018-07-11 07:50:24.208969 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.0 10.8.128.113:6803/15255 222 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.208235) v4 ==== 2004+0+0 (27127593 0 0) 0x55d9829e5000 con 0x55d986bc9800 > -9910> 2018-07-11 07:50:24.208973 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.2 10.8.128.106:6808/12651 222 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.208235) v4 ==== 2004+0+0 (27127593 0 0) 0x55d9869be400 con 0x55d986b9a000 > -9909> 2018-07-11 07:50:24.209019 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6811/15287 conn(0x55d986bb0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 222 0x55d9829e5000 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.208235) v4 > -9908> 2018-07-11 07:50:24.209033 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.4 10.8.128.113:6811/15287 222 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.208235) v4 ==== 2004+0+0 (27127593 0 0) 0x55d9829e5000 con 0x55d986bb0800 > -9907> 2018-07-11 07:50:24.209030 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6807/12677 conn(0x55d986b59800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.1 seq 222 0x55d9869be400 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.208235) v4 > -9906> 2018-07-11 07:50:24.209043 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.1 10.8.128.106:6807/12677 222 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.208235) v4 ==== 2004+0+0 (27127593 0 0) 0x55d9869be400 con 0x55d986b59800 > -9905> 2018-07-11 07:50:24.209067 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.115:6802/13323 conn(0x55d986bc1000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 222 0x55d9829e5000 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.208235) v4 > -9904> 2018-07-11 07:50:24.209073 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6807/15263 conn(0x55d986b9e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.5 seq 222 0x55d9869be400 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.208235) v4 > -9903> 2018-07-11 07:50:24.209081 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.6 10.8.128.115:6802/13323 222 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.208235) v4 ==== 2004+0+0 (27127593 0 0) 0x55d9829e5000 con 0x55d986bc1000 > -9902> 2018-07-11 07:50:24.209065 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.108:6802/11967 conn(0x55d986bbe000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.8 seq 222 0x55d980f5a200 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.208235) v4 > -9901> 2018-07-11 07:50:24.209087 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.5 10.8.128.113:6807/15263 222 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.208235) v4 ==== 2004+0+0 (27127593 0 0) 0x55d9869be400 con 0x55d986b9e800 > -9900> 2018-07-11 07:50:24.209091 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.8 10.8.128.108:6802/11967 222 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.208235) v4 ==== 2004+0+0 (27127593 0 0) 0x55d980f5a200 con 0x55d986bbe000 > -9899> 2018-07-11 07:50:24.209111 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6806/15885 conn(0x55d986bac000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.7 seq 222 0x55d9829e5000 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.208235) v4 > -9898> 2018-07-11 07:50:24.209123 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.7 10.8.128.114:6806/15885 222 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.208235) v4 ==== 2004+0+0 (27127593 0 0) 0x55d9829e5000 con 0x55d986bac000 > -9897> 2018-07-11 07:50:24.209127 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6807/15885 conn(0x55d986bad800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.7 seq 222 0x55d980f5a200 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.208235) v4 > -9896> 2018-07-11 07:50:24.209132 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6810/15287 conn(0x55d986baf000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 222 0x55d9869be400 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.208235) v4 > -9895> 2018-07-11 07:50:24.209140 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.7 10.8.128.114:6807/15885 222 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.208235) v4 ==== 2004+0+0 (27127593 0 0) 0x55d980f5a200 con 0x55d986bad800 > -9894> 2018-07-11 07:50:24.209146 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.4 10.8.128.113:6810/15287 222 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.208235) v4 ==== 2004+0+0 (27127593 0 0) 0x55d9869be400 con 0x55d986baf000 > -9893> 2018-07-11 07:50:24.209166 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6811/12650 conn(0x55d986bcc800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 222 0x55d9869be400 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.208235) v4 > -9892> 2018-07-11 07:50:24.209169 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.115:6803/13323 conn(0x55d986bc2800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 222 0x55d980f5a200 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.208235) v4 > -9891> 2018-07-11 07:50:24.209175 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.3 10.8.128.106:6811/12650 222 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.208235) v4 ==== 2004+0+0 (27127593 0 0) 0x55d9869be400 con 0x55d986bcc800 > -9890> 2018-07-11 07:50:24.209178 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.6 10.8.128.115:6803/13323 222 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.208235) v4 ==== 2004+0+0 (27127593 0 0) 0x55d980f5a200 con 0x55d986bc2800 > -9889> 2018-07-11 07:50:24.209209 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.108:6803/11967 conn(0x55d986bbf800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.8 seq 222 0x55d9869be400 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.208235) v4 > -9888> 2018-07-11 07:50:24.209214 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6803/15919 conn(0x55d986b5c800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.11 seq 222 0x55d980f5a200 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.208235) v4 > -9887> 2018-07-11 07:50:24.209221 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.8 10.8.128.108:6803/11967 222 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.208235) v4 ==== 2004+0+0 (27127593 0 0) 0x55d9869be400 con 0x55d986bbf800 > -9886> 2018-07-11 07:50:24.209224 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.11 10.8.128.114:6803/15919 222 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.208235) v4 ==== 2004+0+0 (27127593 0 0) 0x55d980f5a200 con 0x55d986b5c800 > -9885> 2018-07-11 07:50:24.209298 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6802/15919 conn(0x55d986b5b000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.11 seq 222 0x55d9829e5000 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.208235) v4 > -9884> 2018-07-11 07:50:24.209313 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.11 10.8.128.114:6802/15919 222 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.208235) v4 ==== 2004+0+0 (27127593 0 0) 0x55d9829e5000 con 0x55d986b5b000 > -9883> 2018-07-11 07:50:24.442698 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -9882> 2018-07-11 07:50:24.442737 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -9881> 2018-07-11 07:50:24.442783 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -9880> 2018-07-11 07:50:24.443221 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -9879> 2018-07-11 07:50:24.478077 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:750ff02b:::benchmark_data_magna115_10867_object1126773:0,end:5:75104c7c:::benchmark_data_magna115_10867_object2087273:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498780 con 0 > -9878> 2018-07-11 07:50:24.478114 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:750ff02b:::benchmark_data_magna115_10867_object1126773:0,end:5:75104c7c:::benchmark_data_magna115_10867_object2087273:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b14f0500 con 0 > -9877> 2018-07-11 07:50:24.530357 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 27] Generated table #36788: 1581042 keys, 32626635 bytes > -9876> 2018-07-11 07:50:24.530397 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295424530371, "cf_name": "default", "job": 27, "event": "table_file_creation", "file_number": 36788, "file_size": 32626635, "table_properties": {"data_size": 28318518, "index_size": 290593, "filter_size": 4016534, "raw_key_size": 45674552, "raw_average_key_size": 28, "raw_value_size": 6907948, "raw_average_value_size": 4, "num_data_blocks": 6954, "num_entries": 1581042, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1000000", "kMergeOperands": "0"}} > -9875> 2018-07-11 07:50:24.530424 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 27] Compacted 1@2 + 1@3 files to L3 => 147405683 bytes > -9874> 2018-07-11 07:50:24.545539 7ffabc3f1700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 53757399101 > -9873> 2018-07-11 07:50:24.545709 7ffabc3f1700 4 rocksdb: (Original Log Time 2018/07/11-07:50:24.545506) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 11 73 464 3641 796 0] max score 2.45, MB/sec: 10.6 rd, 10.0 wr, level 3, files in(1, 1) out(3) MB in(76.6, 73.3) out(140.6), read-write-amplify(3.8) write-amplify(1.8) OK, records in: 8530999, records dropped: 442162 > > -9872> 2018-07-11 07:50:24.545719 7ffabc3f1700 4 rocksdb: (Original Log Time 2018/07/11-07:50:24.545531) EVENT_LOG_v1 {"time_micros": 1531295424545520, "job": 27, "event": "compaction_finished", "compaction_time_micros": 14783300, "output_level": 3, "num_output_files": 3, "total_output_size": 147405683, "num_input_records": 8530999, "num_output_records": 8088837, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 11, 73, 464, 3641, 796, 0]} > -9871> 2018-07-11 07:50:24.546322 7ffabc3f1700 5 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:307] [JOB 27] Delete db.slow/036731.sst type=2 #36731 -- OK > > -9870> 2018-07-11 07:50:24.546335 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295424546332, "job": 27, "event": "table_file_deletion", "file_number": 36731} > -9869> 2018-07-11 07:50:24.546350 7ffabc3f1700 5 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:307] [JOB 27] Delete db.slow/036623.sst type=2 #36623 -- OK > > -9868> 2018-07-11 07:50:24.546356 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295424546353, "job": 27, "event": "table_file_deletion", "file_number": 36623} > -9867> 2018-07-11 07:50:24.562192 7ffabc3f1700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 53427079270 > -9866> 2018-07-11 07:50:24.562327 7ffabc3f1700 4 rocksdb: (Original Log Time 2018/07/11-07:50:24.546537) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #36629 to level-3 80348267 bytes > > -9865> 2018-07-11 07:50:24.562336 7ffabc3f1700 4 rocksdb: (Original Log Time 2018/07/11-07:50:24.562235) EVENT_LOG_v1 {"time_micros": 1531295424562208, "job": 28, "event": "trivial_move", "destination_level": 3, "files": 1, "total_files_size": 80348267} > -9864> 2018-07-11 07:50:24.562340 7ffabc3f1700 4 rocksdb: (Original Log Time 2018/07/11-07:50:24.562241) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-3 80348267 bytes OK: base level 1 max bytes base 268435456 files[0 11 72 465 3641 796 0] max score 2.39 > > -9863> 2018-07-11 07:50:24.562921 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 29] Compacting 1@2 + 2@3 files to L3, score 2.39 > -9862> 2018-07-11 07:50:24.562933 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 28 Base level 2, inputs: [36630(76MB)], [29885(71MB) 29888(39MB)] > > -9861> 2018-07-11 07:50:24.562946 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295424562936, "job": 29, "event": "compaction_started", "files_L2": [36630], "files_L3": [29885, 29888], "score": 2.38711, "input_data_size": 196831602} > -9860> 2018-07-11 07:50:24.662871 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 769 0x55d9c5bb0480 rep_scrubmap(5.2e e501 from shard 2) v1 > -9859> 2018-07-11 07:50:24.662892 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 769 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12472 (1269249638 0 1407644610) 0x55d9c5bb0480 con 0x55d986bf0800 > -9858> 2018-07-11 07:50:24.693624 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 763 0x55d9c5bb06c0 rep_scrubmap(5.2e e501 from shard 4) v1 > -9857> 2018-07-11 07:50:24.693646 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 763 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12472 (3686815630 0 1407644610) 0x55d9c5bb06c0 con 0x55d986bec000 > -9856> 2018-07-11 07:50:24.708734 7ffab1bdc700 5 osd.10 501 heartbeat: osd_stat(787 GB used, 143 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,1,0,1]) > -9855> 2018-07-11 07:50:24.708747 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6802/15255 -- osd_ping(ping e501 stamp 2018-07-11 07:50:24.708745) v4 -- 0x55d9b4388200 con 0 > -9854> 2018-07-11 07:50:24.708782 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6803/15255 -- osd_ping(ping e501 stamp 2018-07-11 07:50:24.708745) v4 -- 0x55d9b40a4c00 con 0 > -9853> 2018-07-11 07:50:24.708796 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6806/12677 -- osd_ping(ping e501 stamp 2018-07-11 07:50:24.708745) v4 -- 0x55d9bdba5e00 con 0 > -9852> 2018-07-11 07:50:24.708810 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6807/12677 -- osd_ping(ping e501 stamp 2018-07-11 07:50:24.708745) v4 -- 0x55d9bfbde400 con 0 > -9851> 2018-07-11 07:50:24.708834 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6808/12651 -- osd_ping(ping e501 stamp 2018-07-11 07:50:24.708745) v4 -- 0x55d986c23000 con 0 > -9850> 2018-07-11 07:50:24.708848 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6810/12651 -- osd_ping(ping e501 stamp 2018-07-11 07:50:24.708745) v4 -- 0x55d9b4389a00 con 0 > -9849> 2018-07-11 07:50:24.708858 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6809/12650 -- osd_ping(ping e501 stamp 2018-07-11 07:50:24.708745) v4 -- 0x55d9c34cc000 con 0 > -9848> 2018-07-11 07:50:24.708870 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6811/12650 -- osd_ping(ping e501 stamp 2018-07-11 07:50:24.708745) v4 -- 0x55d980f5aa00 con 0 > -9847> 2018-07-11 07:50:24.708888 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6810/15287 -- osd_ping(ping e501 stamp 2018-07-11 07:50:24.708745) v4 -- 0x55d9bcd73600 con 0 > -9846> 2018-07-11 07:50:24.708898 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6811/15287 -- osd_ping(ping e501 stamp 2018-07-11 07:50:24.708745) v4 -- 0x55d985074400 con 0 > -9845> 2018-07-11 07:50:24.708908 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6806/15263 -- osd_ping(ping e501 stamp 2018-07-11 07:50:24.708745) v4 -- 0x55d986d96200 con 0 > -9844> 2018-07-11 07:50:24.708920 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6807/15263 -- osd_ping(ping e501 stamp 2018-07-11 07:50:24.708745) v4 -- 0x55d9b9ac2400 con 0 > -9843> 2018-07-11 07:50:24.708931 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.115:6802/13323 -- osd_ping(ping e501 stamp 2018-07-11 07:50:24.708745) v4 -- 0x55d9b67d9c00 con 0 > -9842> 2018-07-11 07:50:24.708945 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.115:6803/13323 -- osd_ping(ping e501 stamp 2018-07-11 07:50:24.708745) v4 -- 0x55d9b67d8200 con 0 > -9841> 2018-07-11 07:50:24.708958 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6806/15885 -- osd_ping(ping e501 stamp 2018-07-11 07:50:24.708745) v4 -- 0x55d986d97400 con 0 > -9840> 2018-07-11 07:50:24.708970 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6807/15885 -- osd_ping(ping e501 stamp 2018-07-11 07:50:24.708745) v4 -- 0x55d9869bfc00 con 0 > -9839> 2018-07-11 07:50:24.708985 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.108:6802/11967 -- osd_ping(ping e501 stamp 2018-07-11 07:50:24.708745) v4 -- 0x55d9829e4400 con 0 > -9838> 2018-07-11 07:50:24.708997 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.108:6803/11967 -- osd_ping(ping e501 stamp 2018-07-11 07:50:24.708745) v4 -- 0x55d9c9123a00 con 0 > -9837> 2018-07-11 07:50:24.709010 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.107:6806/12617 -- osd_ping(ping e501 stamp 2018-07-11 07:50:24.708745) v4 -- 0x55d9bdba4e00 con 0 > -9836> 2018-07-11 07:50:24.709022 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.107:6807/12617 -- osd_ping(ping e501 stamp 2018-07-11 07:50:24.708745) v4 -- 0x55d9869bf800 con 0 > -9835> 2018-07-11 07:50:24.709035 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6802/15919 -- osd_ping(ping e501 stamp 2018-07-11 07:50:24.708745) v4 -- 0x55d9ba818800 con 0 > -9834> 2018-07-11 07:50:24.709046 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6803/15919 -- osd_ping(ping e501 stamp 2018-07-11 07:50:24.708745) v4 -- 0x55d986d33e00 con 0 > -9833> 2018-07-11 07:50:24.709217 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.107:6806/12617 conn(0x55d986bd2000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.9 seq 223 0x55d9829e5000 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.708745) v4 > -9832> 2018-07-11 07:50:24.709237 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.107:6807/12617 conn(0x55d986bd3800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.9 seq 223 0x55d980f5a200 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.708745) v4 > -9831> 2018-07-11 07:50:24.709238 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.9 10.8.128.107:6806/12617 223 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.708745) v4 ==== 2004+0+0 (3825547719 0 0) 0x55d9829e5000 con 0x55d986bd2000 > -9830> 2018-07-11 07:50:24.709269 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.9 10.8.128.107:6807/12617 223 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.708745) v4 ==== 2004+0+0 (3825547719 0 0) 0x55d980f5a200 con 0x55d986bd3800 > -9829> 2018-07-11 07:50:24.709300 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6803/15255 conn(0x55d986bc9800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.0 seq 223 0x55d9829e5000 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.708745) v4 > -9828> 2018-07-11 07:50:24.709314 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.0 10.8.128.113:6803/15255 223 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.708745) v4 ==== 2004+0+0 (3825547719 0 0) 0x55d9829e5000 con 0x55d986bc9800 > -9827> 2018-07-11 07:50:24.709354 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6802/15255 conn(0x55d986bc8000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.0 seq 223 0x55d9869be400 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.708745) v4 > -9826> 2018-07-11 07:50:24.709366 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6810/12651 conn(0x55d986b9b800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.2 seq 223 0x55d9829e5000 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.708745) v4 > -9825> 2018-07-11 07:50:24.709371 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.0 10.8.128.113:6802/15255 223 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.708745) v4 ==== 2004+0+0 (3825547719 0 0) 0x55d9869be400 con 0x55d986bc8000 > -9824> 2018-07-11 07:50:24.709383 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.2 10.8.128.106:6810/12651 223 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.708745) v4 ==== 2004+0+0 (3825547719 0 0) 0x55d9829e5000 con 0x55d986b9b800 > -9823> 2018-07-11 07:50:24.709379 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6806/15263 conn(0x55d986b9d000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.5 seq 223 0x55d980f5a200 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.708745) v4 > -9822> 2018-07-11 07:50:24.709397 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.5 10.8.128.113:6806/15263 223 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.708745) v4 ==== 2004+0+0 (3825547719 0 0) 0x55d980f5a200 con 0x55d986b9d000 > -9821> 2018-07-11 07:50:24.709419 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6808/12651 conn(0x55d986b9a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.2 seq 223 0x55d9869be400 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.708745) v4 > -9820> 2018-07-11 07:50:24.709426 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6811/15287 conn(0x55d986bb0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 223 0x55d9829e5000 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.708745) v4 > -9819> 2018-07-11 07:50:24.709433 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.2 10.8.128.106:6808/12651 223 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.708745) v4 ==== 2004+0+0 (3825547719 0 0) 0x55d9869be400 con 0x55d986b9a000 > -9818> 2018-07-11 07:50:24.709446 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6806/12677 conn(0x55d986b58000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.1 seq 223 0x55d980f5a200 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.708745) v4 > -9817> 2018-07-11 07:50:24.709455 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.4 10.8.128.113:6811/15287 223 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.708745) v4 ==== 2004+0+0 (3825547719 0 0) 0x55d9829e5000 con 0x55d986bb0800 > -9816> 2018-07-11 07:50:24.709460 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.1 10.8.128.106:6806/12677 223 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.708745) v4 ==== 2004+0+0 (3825547719 0 0) 0x55d980f5a200 con 0x55d986b58000 > -9815> 2018-07-11 07:50:24.709475 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6807/15263 conn(0x55d986b9e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.5 seq 223 0x55d9869be400 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.708745) v4 > -9814> 2018-07-11 07:50:24.709488 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.5 10.8.128.113:6807/15263 223 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.708745) v4 ==== 2004+0+0 (3825547719 0 0) 0x55d9869be400 con 0x55d986b9e800 > -9813> 2018-07-11 07:50:24.709509 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6809/12650 conn(0x55d986bcb000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 223 0x55d980f5a200 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.708745) v4 > -9812> 2018-07-11 07:50:24.709518 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6807/12677 conn(0x55d986b59800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.1 seq 223 0x55d9869be400 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.708745) v4 > -9811> 2018-07-11 07:50:24.709523 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.3 10.8.128.106:6809/12650 223 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.708745) v4 ==== 2004+0+0 (3825547719 0 0) 0x55d980f5a200 con 0x55d986bcb000 > -9810> 2018-07-11 07:50:24.709533 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.1 10.8.128.106:6807/12677 223 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.708745) v4 ==== 2004+0+0 (3825547719 0 0) 0x55d9869be400 con 0x55d986b59800 > -9809> 2018-07-11 07:50:24.709564 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.115:6802/13323 conn(0x55d986bc1000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 223 0x55d9829e5000 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.708745) v4 > -9808> 2018-07-11 07:50:24.709573 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.115:6803/13323 conn(0x55d986bc2800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 223 0x55d980f5a200 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.708745) v4 > -9807> 2018-07-11 07:50:24.709583 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6810/15287 conn(0x55d986baf000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 223 0x55d9869be400 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.708745) v4 > -9806> 2018-07-11 07:50:24.709586 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.6 10.8.128.115:6802/13323 223 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.708745) v4 ==== 2004+0+0 (3825547719 0 0) 0x55d9829e5000 con 0x55d986bc1000 > -9805> 2018-07-11 07:50:24.709596 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.4 10.8.128.113:6810/15287 223 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.708745) v4 ==== 2004+0+0 (3825547719 0 0) 0x55d9869be400 con 0x55d986baf000 > -9804> 2018-07-11 07:50:24.709585 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.6 10.8.128.115:6803/13323 223 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.708745) v4 ==== 2004+0+0 (3825547719 0 0) 0x55d980f5a200 con 0x55d986bc2800 > -9803> 2018-07-11 07:50:24.709621 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.108:6802/11967 conn(0x55d986bbe000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.8 seq 223 0x55d980f5a200 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.708745) v4 > -9802> 2018-07-11 07:50:24.709632 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.8 10.8.128.108:6802/11967 223 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.708745) v4 ==== 2004+0+0 (3825547719 0 0) 0x55d980f5a200 con 0x55d986bbe000 > -9801> 2018-07-11 07:50:24.709633 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6811/12650 conn(0x55d986bcc800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 223 0x55d9869be400 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.708745) v4 > -9800> 2018-07-11 07:50:24.709650 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.3 10.8.128.106:6811/12650 223 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.708745) v4 ==== 2004+0+0 (3825547719 0 0) 0x55d9869be400 con 0x55d986bcc800 > -9799> 2018-07-11 07:50:24.709668 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6803/15919 conn(0x55d986b5c800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.11 seq 223 0x55d980f5a200 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.708745) v4 > -9798> 2018-07-11 07:50:24.709668 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6802/15919 conn(0x55d986b5b000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.11 seq 223 0x55d9829e5000 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.708745) v4 > -9797> 2018-07-11 07:50:24.709680 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.11 10.8.128.114:6803/15919 223 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.708745) v4 ==== 2004+0+0 (3825547719 0 0) 0x55d980f5a200 con 0x55d986b5c800 > -9796> 2018-07-11 07:50:24.709683 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.108:6803/11967 conn(0x55d986bbf800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.8 seq 223 0x55d9869be400 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.708745) v4 > -9795> 2018-07-11 07:50:24.709681 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.11 10.8.128.114:6802/15919 223 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.708745) v4 ==== 2004+0+0 (3825547719 0 0) 0x55d9829e5000 con 0x55d986b5b000 > -9794> 2018-07-11 07:50:24.709696 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.8 10.8.128.108:6803/11967 223 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.708745) v4 ==== 2004+0+0 (3825547719 0 0) 0x55d9869be400 con 0x55d986bbf800 > -9793> 2018-07-11 07:50:24.709718 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6806/15885 conn(0x55d986bac000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.7 seq 223 0x55d9829e5000 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.708745) v4 > -9792> 2018-07-11 07:50:24.709720 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6807/15885 conn(0x55d986bad800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.7 seq 223 0x55d980f5a200 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.708745) v4 > -9791> 2018-07-11 07:50:24.709731 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.7 10.8.128.114:6807/15885 223 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.708745) v4 ==== 2004+0+0 (3825547719 0 0) 0x55d980f5a200 con 0x55d986bad800 > -9790> 2018-07-11 07:50:24.709731 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.7 10.8.128.114:6806/15885 223 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:24.708745) v4 ==== 2004+0+0 (3825547719 0 0) 0x55d9829e5000 con 0x55d986bac000 > -9789> 2018-07-11 07:50:24.827916 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 2956 0x55d980f5a200 MOSDScrubReserve(11.3s5 REQUEST e501) v1 > -9788> 2018-07-11 07:50:24.827949 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 2956 ==== MOSDScrubReserve(11.3s5 REQUEST e501) v1 ==== 43+0+0 (3260170403 0 0) 0x55d980f5a200 con 0x55d986c7e000 > -9787> 2018-07-11 07:50:24.828075 7ffab3be0700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e501) v1 -- 0x55d9bcd78600 con 0 > -9786> 2018-07-11 07:50:24.828276 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 2957 0x55d986d33e00 MOSDScrubReserve(11.3s5 RELEASE e501) v1 > -9785> 2018-07-11 07:50:24.828290 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 2957 ==== MOSDScrubReserve(11.3s5 RELEASE e501) v1 ==== 43+0+0 (223328929 0 0) 0x55d986d33e00 con 0x55d986c7e000 > -9784> 2018-07-11 07:50:24.952300 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6805/15263 conn(0x55d986c41000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=31 cs=1 l=0). rx osd.5 seq 2778 0x55d9869be400 MOSDScrubReserve(6.15 REQUEST e501) v1 > -9783> 2018-07-11 07:50:24.952321 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.5 10.8.128.113:6805/15263 2778 ==== MOSDScrubReserve(6.15 REQUEST e501) v1 ==== 43+0+0 (3942742026 0 0) 0x55d9869be400 con 0x55d986c41000 > -9782> 2018-07-11 07:50:24.991833 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.115:0/13323 conn(0x55d986d7e000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 230 0x55d9bcd78600 osd_ping(ping e501 stamp 2018-07-11 07:50:25.337015) v4 > -9781> 2018-07-11 07:50:24.991857 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.6 10.8.128.115:0/13323 230 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:25.337015) v4 ==== 2004+0+0 (669313455 0 0) 0x55d9bcd78600 con 0x55d986d7e000 > -9780> 2018-07-11 07:50:24.991867 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:25.337015) v4 -- 0x55d9c34cc600 con 0 > -9779> 2018-07-11 07:50:24.991894 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.115:0/13323 conn(0x55d986d24800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 230 0x55d986cd4c00 osd_ping(ping e501 stamp 2018-07-11 07:50:25.337015) v4 > -9778> 2018-07-11 07:50:24.991920 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.6 10.8.128.115:0/13323 230 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:25.337015) v4 ==== 2004+0+0 (669313455 0 0) 0x55d986cd4c00 con 0x55d986d24800 > -9777> 2018-07-11 07:50:24.991931 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:25.337015) v4 -- 0x55d9b9ac2400 con 0 > -9776> 2018-07-11 07:50:25.156098 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12677 conn(0x55d986db4800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.1 seq 234 0x55d9829e5000 osd_ping(ping e501 stamp 2018-07-11 07:50:25.202973) v4 > -9775> 2018-07-11 07:50:25.156103 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12677 conn(0x55d986df2000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.1 seq 234 0x55d9c9123a00 osd_ping(ping e501 stamp 2018-07-11 07:50:25.202973) v4 > -9774> 2018-07-11 07:50:25.156132 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.1 10.8.128.106:0/12677 234 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:25.202973) v4 ==== 2004+0+0 (2285821645 0 0) 0x55d9829e5000 con 0x55d986db4800 > -9773> 2018-07-11 07:50:25.156136 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.1 10.8.128.106:0/12677 234 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:25.202973) v4 ==== 2004+0+0 (2285821645 0 0) 0x55d9c9123a00 con 0x55d986df2000 > -9772> 2018-07-11 07:50:25.156144 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:25.202973) v4 -- 0x55d9ba818800 con 0 > -9771> 2018-07-11 07:50:25.156178 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:25.202973) v4 -- 0x55d9b9ac2400 con 0 > -9770> 2018-07-11 07:50:25.229026 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6805/15263 -- MOSDScrubReserve(6.15 REJECT e501) v1 -- 0x55d9829e4200 con 0 > -9769> 2018-07-11 07:50:25.229088 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -9768> 2018-07-11 07:50:25.229114 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -9767> 2018-07-11 07:50:25.229163 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -9766> 2018-07-11 07:50:25.229540 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6805/15263 conn(0x55d986c41000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=31 cs=1 l=0). rx osd.5 seq 2779 0x55d9829e4200 MOSDScrubReserve(6.15 RELEASE e501) v1 > -9765> 2018-07-11 07:50:25.229559 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.5 10.8.128.113:6805/15263 2779 ==== MOSDScrubReserve(6.15 RELEASE e501) v1 ==== 43+0+0 (605848072 0 0) 0x55d9829e4200 con 0x55d986c41000 > -9764> 2018-07-11 07:50:25.229640 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -9763> 2018-07-11 07:50:25.244670 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75104c7c:::benchmark_data_magna115_10867_object2087273:0,end:5:7510b344:::benchmark_data_magna115_10867_object3544942:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455900 con 0 > -9762> 2018-07-11 07:50:25.244721 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75104c7c:::benchmark_data_magna115_10867_object2087273:0,end:5:7510b344:::benchmark_data_magna115_10867_object3544942:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455b80 con 0 > -9761> 2018-07-11 07:50:25.406144 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12650 conn(0x55d986d23000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 222 0x55d9c9122a00 osd_ping(ping e501 stamp 2018-07-11 07:50:25.453040) v4 > -9760> 2018-07-11 07:50:25.406157 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12650 conn(0x55d986d21800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 222 0x55d9869bf800 osd_ping(ping e501 stamp 2018-07-11 07:50:25.453040) v4 > -9759> 2018-07-11 07:50:25.406166 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.3 10.8.128.106:0/12650 222 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:25.453040) v4 ==== 2004+0+0 (1941356273 0 0) 0x55d9c9122a00 con 0x55d986d23000 > -9758> 2018-07-11 07:50:25.406176 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.3 10.8.128.106:0/12650 222 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:25.453040) v4 ==== 2004+0+0 (1941356273 0 0) 0x55d9869bf800 con 0x55d986d21800 > -9757> 2018-07-11 07:50:25.406177 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:25.453040) v4 -- 0x55d9ba818800 con 0 > -9756> 2018-07-11 07:50:25.406199 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:25.453040) v4 -- 0x55d9c34cc600 con 0 > -9755> 2018-07-11 07:50:25.413420 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12651 conn(0x55d986db3000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.2 seq 231 0x55d9bcd78600 osd_ping(ping e501 stamp 2018-07-11 07:50:25.460135) v4 > -9754> 2018-07-11 07:50:25.413444 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.2 10.8.128.106:0/12651 231 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:25.460135) v4 ==== 2004+0+0 (4262673342 0 0) 0x55d9bcd78600 con 0x55d986db3000 > -9753> 2018-07-11 07:50:25.413453 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:25.460135) v4 -- 0x55d9c34cc600 con 0 > -9752> 2018-07-11 07:50:25.413486 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12651 conn(0x55d986db1800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.2 seq 231 0x55d9c9123a00 osd_ping(ping e501 stamp 2018-07-11 07:50:25.460135) v4 > -9751> 2018-07-11 07:50:25.413505 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.2 10.8.128.106:0/12651 231 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:25.460135) v4 ==== 2004+0+0 (4262673342 0 0) 0x55d9c9123a00 con 0x55d986db1800 > -9750> 2018-07-11 07:50:25.413516 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:25.460135) v4 -- 0x55d9b9ac2400 con 0 > -9749> 2018-07-11 07:50:25.465499 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 764 0x55d9c5bb0900 rep_scrubmap(5.2e e501 from shard 4) v1 > -9748> 2018-07-11 07:50:25.465519 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 764 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12476 (3686815630 0 3500694502) 0x55d9c5bb0900 con 0x55d986bec000 > -9747> 2018-07-11 07:50:25.470566 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 770 0x55d9c5bb0b40 rep_scrubmap(5.2e e501 from shard 2) v1 > -9746> 2018-07-11 07:50:25.470585 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 770 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12476 (1269249638 0 3500694502) 0x55d9c5bb0b40 con 0x55d986bf0800 > -9745> 2018-07-11 07:50:25.489863 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -9744> 2018-07-11 07:50:25.489886 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -9743> 2018-07-11 07:50:25.489943 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -9742> 2018-07-11 07:50:25.490400 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -9741> 2018-07-11 07:50:25.534948 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7510b344:::benchmark_data_magna115_10867_object3544942:0,end:5:7511316d:::benchmark_data_magna115_10867_object1074043:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498000 con 0 > -9740> 2018-07-11 07:50:25.535000 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7510b344:::benchmark_data_magna115_10867_object3544942:0,end:5:7511316d:::benchmark_data_magna115_10867_object1074043:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455680 con 0 > -9739> 2018-07-11 07:50:25.746397 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 771 0x55d9c5bb0d80 rep_scrubmap(5.2e e501 from shard 2) v1 > -9738> 2018-07-11 07:50:25.746419 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 771 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12476 (1269249638 0 1649014149) 0x55d9c5bb0d80 con 0x55d986bf0800 > -9737> 2018-07-11 07:50:25.769814 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 765 0x55d9c5bb0fc0 rep_scrubmap(5.2e e501 from shard 4) v1 > -9736> 2018-07-11 07:50:25.769834 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 765 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12476 (3686815630 0 1649014149) 0x55d9c5bb0fc0 con 0x55d986bec000 > -9735> 2018-07-11 07:50:25.806696 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -9734> 2018-07-11 07:50:25.806720 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -9733> 2018-07-11 07:50:25.806783 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -9732> 2018-07-11 07:50:25.807268 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -9731> 2018-07-11 07:50:25.828091 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 2958 0x55d9869bf800 MOSDScrubReserve(11.3s5 REQUEST e501) v1 > -9730> 2018-07-11 07:50:25.828110 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 2958 ==== MOSDScrubReserve(11.3s5 REQUEST e501) v1 ==== 43+0+0 (3260170403 0 0) 0x55d9869bf800 con 0x55d986c7e000 > -9729> 2018-07-11 07:50:25.828233 7ffab3be0700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e501) v1 -- 0x55d986d33e00 con 0 > -9728> 2018-07-11 07:50:25.828573 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 2959 0x55d9c34cc600 MOSDScrubReserve(11.3s5 RELEASE e501) v1 > -9727> 2018-07-11 07:50:25.828587 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 2959 ==== MOSDScrubReserve(11.3s5 RELEASE e501) v1 ==== 43+0+0 (223328929 0 0) 0x55d9c34cc600 con 0x55d986c7e000 > -9726> 2018-07-11 07:50:25.853231 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7511316d:::benchmark_data_magna115_10867_object1074043:0,end:5:7511806e:::benchmark_data_magna115_10867_object2059287:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498280 con 0 > -9725> 2018-07-11 07:50:25.853282 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7511316d:::benchmark_data_magna115_10867_object1074043:0,end:5:7511806e:::benchmark_data_magna115_10867_object2059287:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454f00 con 0 > -9724> 2018-07-11 07:50:25.874716 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15287 conn(0x55d981240800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 241 0x55d9c9123a00 osd_ping(ping e501 stamp 2018-07-11 07:50:27.104612) v4 > -9723> 2018-07-11 07:50:25.874741 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.4 10.8.128.113:0/15287 241 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:27.104612) v4 ==== 2004+0+0 (2876499405 0 0) 0x55d9c9123a00 con 0x55d981240800 > -9722> 2018-07-11 07:50:25.874751 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:27.104612) v4 -- 0x55d9b9ac2400 con 0 > -9721> 2018-07-11 07:50:25.874798 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15287 conn(0x55d98123f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 241 0x55d9829e5000 osd_ping(ping e501 stamp 2018-07-11 07:50:27.104612) v4 > -9720> 2018-07-11 07:50:25.874824 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.4 10.8.128.113:0/15287 241 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:27.104612) v4 ==== 2004+0+0 (2876499405 0 0) 0x55d9829e5000 con 0x55d98123f000 > -9719> 2018-07-11 07:50:25.874838 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:27.104612) v4 -- 0x55d9ba818800 con 0 > -9718> 2018-07-11 07:50:25.906619 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12650 conn(0x55d986d23000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 223 0x55d9c9122a00 osd_ping(ping e501 stamp 2018-07-11 07:50:25.953504) v4 > -9717> 2018-07-11 07:50:25.906619 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12650 conn(0x55d986d21800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 223 0x55d9bcd78600 osd_ping(ping e501 stamp 2018-07-11 07:50:25.953504) v4 > -9716> 2018-07-11 07:50:25.906649 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.3 10.8.128.106:0/12650 223 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:25.953504) v4 ==== 2004+0+0 (613340403 0 0) 0x55d9bcd78600 con 0x55d986d21800 > -9715> 2018-07-11 07:50:25.906647 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.3 10.8.128.106:0/12650 223 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:25.953504) v4 ==== 2004+0+0 (613340403 0 0) 0x55d9c9122a00 con 0x55d986d23000 > -9714> 2018-07-11 07:50:25.906659 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:25.953504) v4 -- 0x55d9829e4400 con 0 > -9713> 2018-07-11 07:50:25.906682 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:25.953504) v4 -- 0x55d9ba818800 con 0 > -9712> 2018-07-11 07:50:25.953435 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6801/15919 conn(0x55d986c3f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=29 cs=1 l=0). rx osd.11 seq 200 0x55d986d33e00 MOSDScrubReserve(10.f REQUEST e501) v1 > -9711> 2018-07-11 07:50:25.953453 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.11 10.8.128.114:6801/15919 200 ==== MOSDScrubReserve(10.f REQUEST e501) v1 ==== 43+0+0 (2892966580 0 0) 0x55d986d33e00 con 0x55d986c3f800 > -9710> 2018-07-11 07:50:25.953524 7ffab53e3700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.114:6801/15919 -- MOSDScrubReserve(10.f REJECT e501) v1 -- 0x55d9869bea00 con 0 > -9709> 2018-07-11 07:50:25.953823 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6801/15919 conn(0x55d986c3f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=29 cs=1 l=0). rx osd.11 seq 201 0x55d9869bea00 MOSDScrubReserve(10.f RELEASE e501) v1 > -9708> 2018-07-11 07:50:25.953837 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.11 10.8.128.114:6801/15919 201 ==== MOSDScrubReserve(10.f RELEASE e501) v1 ==== 43+0+0 (1668468918 0 0) 0x55d9869bea00 con 0x55d986c3f800 > -9707> 2018-07-11 07:50:26.068956 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 766 0x55d9c5bb1200 rep_scrubmap(5.2e e501 from shard 4) v1 > -9706> 2018-07-11 07:50:26.068976 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 766 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12484 (3686815630 0 1037333777) 0x55d9c5bb1200 con 0x55d986bec000 > -9705> 2018-07-11 07:50:26.072890 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 772 0x55d9c5bb1440 rep_scrubmap(5.2e e501 from shard 2) v1 > -9704> 2018-07-11 07:50:26.072910 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 772 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12484 (1269249638 0 1037333777) 0x55d9c5bb1440 con 0x55d986bf0800 > -9703> 2018-07-11 07:50:26.110274 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.114:0/15885 conn(0x55d986d7f800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.7 seq 236 0x55d9829e5000 osd_ping(ping e501 stamp 2018-07-11 07:50:26.740069) v4 > -9702> 2018-07-11 07:50:26.110292 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.114:0/15885 conn(0x55d986d81000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.7 seq 236 0x55d986cd4c00 osd_ping(ping e501 stamp 2018-07-11 07:50:26.740069) v4 > -9701> 2018-07-11 07:50:26.110300 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.7 10.8.128.114:0/15885 236 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:26.740069) v4 ==== 2004+0+0 (1031354402 0 0) 0x55d9829e5000 con 0x55d986d7f800 > -9700> 2018-07-11 07:50:26.110310 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.7 10.8.128.114:0/15885 236 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:26.740069) v4 ==== 2004+0+0 (1031354402 0 0) 0x55d986cd4c00 con 0x55d986d81000 > -9699> 2018-07-11 07:50:26.110315 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:26.740069) v4 -- 0x55d9ba818800 con 0 > -9698> 2018-07-11 07:50:26.110344 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:26.740069) v4 -- 0x55d9b9ac2400 con 0 > -9697> 2018-07-11 07:50:26.131259 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -9696> 2018-07-11 07:50:26.131283 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -9695> 2018-07-11 07:50:26.131346 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -9694> 2018-07-11 07:50:26.131814 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -9693> 2018-07-11 07:50:26.133240 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7511806e:::benchmark_data_magna115_10867_object2059287:0,end:5:7511d20b:::benchmark_data_magna115_11707_object491718:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455400 con 0 > -9692> 2018-07-11 07:50:26.133296 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7511806e:::benchmark_data_magna115_10867_object2059287:0,end:5:7511d20b:::benchmark_data_magna115_11707_object491718:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455180 con 0 > -9691> 2018-07-11 07:50:26.256482 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12677 conn(0x55d986df2000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.1 seq 235 0x55d986cd4c00 osd_ping(ping e501 stamp 2018-07-11 07:50:26.303390) v4 > -9690> 2018-07-11 07:50:26.256485 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12677 conn(0x55d986db4800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.1 seq 235 0x55d9c9122a00 osd_ping(ping e501 stamp 2018-07-11 07:50:26.303390) v4 > -9689> 2018-07-11 07:50:26.256509 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.1 10.8.128.106:0/12677 235 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:26.303390) v4 ==== 2004+0+0 (3781229286 0 0) 0x55d986cd4c00 con 0x55d986df2000 > -9688> 2018-07-11 07:50:26.256515 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.1 10.8.128.106:0/12677 235 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:26.303390) v4 ==== 2004+0+0 (3781229286 0 0) 0x55d9c9122a00 con 0x55d986db4800 > -9687> 2018-07-11 07:50:26.256519 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:26.303390) v4 -- 0x55d9b9ac2400 con 0 > -9686> 2018-07-11 07:50:26.256543 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:26.303390) v4 -- 0x55d9ba818800 con 0 > -9685> 2018-07-11 07:50:26.340862 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 773 0x55d9c5bb1680 rep_scrubmap(5.2e e501 from shard 2) v1 > -9684> 2018-07-11 07:50:26.340881 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 773 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12474 (1269249638 0 2534947048) 0x55d9c5bb1680 con 0x55d986bf0800 > -9683> 2018-07-11 07:50:26.352334 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 767 0x55d9c5bb18c0 rep_scrubmap(5.2e e501 from shard 4) v1 > -9682> 2018-07-11 07:50:26.352353 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 767 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12474 (3686815630 0 2534947048) 0x55d9c5bb18c0 con 0x55d986bec000 > -9681> 2018-07-11 07:50:26.406979 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12650 conn(0x55d986d21800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 224 0x55d9bcd78600 osd_ping(ping e501 stamp 2018-07-11 07:50:26.453900) v4 > -9680> 2018-07-11 07:50:26.406985 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12650 conn(0x55d986d23000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 224 0x55d9829e5000 osd_ping(ping e501 stamp 2018-07-11 07:50:26.453900) v4 > -9679> 2018-07-11 07:50:26.407002 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.3 10.8.128.106:0/12650 224 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:26.453900) v4 ==== 2004+0+0 (314603168 0 0) 0x55d9bcd78600 con 0x55d986d21800 > -9678> 2018-07-11 07:50:26.407007 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.3 10.8.128.106:0/12650 224 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:26.453900) v4 ==== 2004+0+0 (314603168 0 0) 0x55d9829e5000 con 0x55d986d23000 > -9677> 2018-07-11 07:50:26.407018 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:26.453900) v4 -- 0x55d9869bfc00 con 0 > -9676> 2018-07-11 07:50:26.407048 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:26.453900) v4 -- 0x55d9ba818800 con 0 > -9675> 2018-07-11 07:50:26.439237 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -9674> 2018-07-11 07:50:26.439274 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -9673> 2018-07-11 07:50:26.439324 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -9672> 2018-07-11 07:50:26.439788 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -9671> 2018-07-11 07:50:26.461930 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67907,epoch:501/498,start:5:7511d20b:::benchmark_data_magna115_11707_object491718:0,end:5:75122eb9:::benchmark_data_magna115_11707_object44140:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454780 con 0 > -9670> 2018-07-11 07:50:26.461979 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67907,epoch:501/498,start:5:7511d20b:::benchmark_data_magna115_11707_object491718:0,end:5:75122eb9:::benchmark_data_magna115_11707_object44140:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454a00 con 0 > -9669> 2018-07-11 07:50:26.584478 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.114:0/15919 conn(0x55d986df5000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.11 seq 233 0x55d9c9122a00 osd_ping(ping e501 stamp 2018-07-11 07:50:27.214324) v4 > -9668> 2018-07-11 07:50:26.584505 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.11 10.8.128.114:0/15919 233 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:27.214324) v4 ==== 2004+0+0 (272727211 0 0) 0x55d9c9122a00 con 0x55d986df5000 > -9667> 2018-07-11 07:50:26.584522 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:27.214324) v4 -- 0x55d9ba818800 con 0 > -9666> 2018-07-11 07:50:26.584510 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.114:0/15919 conn(0x55d986df3800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.11 seq 233 0x55d9829e4400 osd_ping(ping e501 stamp 2018-07-11 07:50:27.214324) v4 > -9665> 2018-07-11 07:50:26.584545 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.11 10.8.128.114:0/15919 233 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:27.214324) v4 ==== 2004+0+0 (272727211 0 0) 0x55d9829e4400 con 0x55d986df3800 > -9664> 2018-07-11 07:50:26.584568 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:27.214324) v4 -- 0x55d9869bfc00 con 0 > -9663> 2018-07-11 07:50:26.690096 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 768 0x55d9c5bb1b00 rep_scrubmap(5.2e e501 from shard 4) v1 > -9662> 2018-07-11 07:50:26.690115 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 768 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12472 (3686815630 0 2822589500) 0x55d9c5bb1b00 con 0x55d986bec000 > -9661> 2018-07-11 07:50:26.699076 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 774 0x55d9c5bb1d40 rep_scrubmap(5.2e e501 from shard 2) v1 > -9660> 2018-07-11 07:50:26.699096 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 774 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12472 (1269249638 0 2822589500) 0x55d9c5bb1d40 con 0x55d986bf0800 > -9659> 2018-07-11 07:50:26.787931 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -9658> 2018-07-11 07:50:26.787957 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -9657> 2018-07-11 07:50:26.788018 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -9656> 2018-07-11 07:50:26.788498 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -9655> 2018-07-11 07:50:26.789551 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67074,epoch:501/498,start:5:75122eb9:::benchmark_data_magna115_11707_object44140:0,end:5:751287cd:::benchmark_data_magna115_10867_object1843275:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454c80 con 0 > -9654> 2018-07-11 07:50:26.789598 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67074,epoch:501/498,start:5:75122eb9:::benchmark_data_magna115_11707_object44140:0,end:5:751287cd:::benchmark_data_magna115_10867_object1843275:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454500 con 0 > -9653> 2018-07-11 07:50:26.828313 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 2960 0x55d9bcd78600 MOSDScrubReserve(11.3s5 REQUEST e501) v1 > -9652> 2018-07-11 07:50:26.828331 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 2960 ==== MOSDScrubReserve(11.3s5 REQUEST e501) v1 ==== 43+0+0 (3260170403 0 0) 0x55d9bcd78600 con 0x55d986c7e000 > -9651> 2018-07-11 07:50:26.828412 7ffab3be0700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e501) v1 -- 0x55d9c34cc600 con 0 > -9650> 2018-07-11 07:50:26.828968 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 2961 0x55d9869bfc00 MOSDScrubReserve(11.3s5 RELEASE e501) v1 > -9649> 2018-07-11 07:50:26.828988 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 2961 ==== MOSDScrubReserve(11.3s5 RELEASE e501) v1 ==== 43+0+0 (223328929 0 0) 0x55d9869bfc00 con 0x55d986c7e000 > -9648> 2018-07-11 07:50:26.953640 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6801/15919 conn(0x55d986c3f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=29 cs=1 l=0). rx osd.11 seq 202 0x55d9829e4400 MOSDScrubReserve(10.f REQUEST e501) v1 > -9647> 2018-07-11 07:50:26.953658 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.11 10.8.128.114:6801/15919 202 ==== MOSDScrubReserve(10.f REQUEST e501) v1 ==== 43+0+0 (2892966580 0 0) 0x55d9829e4400 con 0x55d986c3f800 > -9646> 2018-07-11 07:50:26.953729 7ffab53e3700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.114:6801/15919 -- MOSDScrubReserve(10.f REJECT e501) v1 -- 0x55d9869bea00 con 0 > -9645> 2018-07-11 07:50:26.954029 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6801/15919 conn(0x55d986c3f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=29 cs=1 l=0). rx osd.11 seq 203 0x55d9869bea00 MOSDScrubReserve(10.f RELEASE e501) v1 > -9644> 2018-07-11 07:50:26.954044 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.11 10.8.128.114:6801/15919 203 ==== MOSDScrubReserve(10.f RELEASE e501) v1 ==== 43+0+0 (1668468918 0 0) 0x55d9869bea00 con 0x55d986c3f800 > -9643> 2018-07-11 07:50:26.982166 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 775 0x55d99e3ce000 rep_scrubmap(5.2e e501 from shard 2) v1 > -9642> 2018-07-11 07:50:26.982187 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 775 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12468 (1269249638 0 1098396112) 0x55d99e3ce000 con 0x55d986bf0800 > -9641> 2018-07-11 07:50:27.009138 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 769 0x55d99e3ce240 rep_scrubmap(5.2e e501 from shard 4) v1 > -9640> 2018-07-11 07:50:27.009157 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 769 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12468 (3686815630 0 1098396112) 0x55d99e3ce240 con 0x55d986bec000 > -9639> 2018-07-11 07:50:27.043168 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -9638> 2018-07-11 07:50:27.043192 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -9637> 2018-07-11 07:50:27.043242 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -9636> 2018-07-11 07:50:27.043701 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -9635> 2018-07-11 07:50:27.095856 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'66973,epoch:501/498,start:5:751287cd:::benchmark_data_magna115_10867_object1843275:0,end:5:7512ead7:::benchmark_data_magna115_10867_object2489741:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98944b900 con 0 > -9634> 2018-07-11 07:50:27.095908 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'66973,epoch:501/498,start:5:751287cd:::benchmark_data_magna115_10867_object1843275:0,end:5:7512ead7:::benchmark_data_magna115_10867_object2489741:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98944bb80 con 0 > -9633> 2018-07-11 07:50:27.113877 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12651 conn(0x55d986db3000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.2 seq 232 0x55d9c34cc600 osd_ping(ping e501 stamp 2018-07-11 07:50:27.160721) v4 > -9632> 2018-07-11 07:50:27.113904 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.2 10.8.128.106:0/12651 232 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:27.160721) v4 ==== 2004+0+0 (3499912969 0 0) 0x55d9c34cc600 con 0x55d986db3000 > -9631> 2018-07-11 07:50:27.113920 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:27.160721) v4 -- 0x55d986d96200 con 0 > -9630> 2018-07-11 07:50:27.113965 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12651 conn(0x55d986db1800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.2 seq 232 0x55d986cd4c00 osd_ping(ping e501 stamp 2018-07-11 07:50:27.160721) v4 > -9629> 2018-07-11 07:50:27.113984 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.2 10.8.128.106:0/12651 232 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:27.160721) v4 ==== 2004+0+0 (3499912969 0 0) 0x55d986cd4c00 con 0x55d986db1800 > -9628> 2018-07-11 07:50:27.113995 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:27.160721) v4 -- 0x55d9b9ac2400 con 0 > -9627> 2018-07-11 07:50:27.280611 7ffacbbd1700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.108:0/11967 conn(0x55d986d82800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.8 seq 214 0x55d9b67d8200 osd_ping(ping e501 stamp 2018-07-11 07:50:27.035935) v4 > -9626> 2018-07-11 07:50:27.280632 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 <== osd.8 10.8.128.108:0/11967 214 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:27.035935) v4 ==== 2004+0+0 (4261978565 0 0) 0x55d9b67d8200 con 0x55d986d82800 > -9625> 2018-07-11 07:50:27.280627 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.108:0/11967 conn(0x55d986db0000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.8 seq 214 0x55d9829e5000 osd_ping(ping e501 stamp 2018-07-11 07:50:27.035935) v4 > -9624> 2018-07-11 07:50:27.280642 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:27.035935) v4 -- 0x55d986d96200 con 0 > -9623> 2018-07-11 07:50:27.280656 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.8 10.8.128.108:0/11967 214 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:27.035935) v4 ==== 2004+0+0 (4261978565 0 0) 0x55d9829e5000 con 0x55d986db0000 > -9622> 2018-07-11 07:50:27.280673 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:27.035935) v4 -- 0x55d9ba818800 con 0 > -9621> 2018-07-11 07:50:27.292274 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.115:0/13323 conn(0x55d986d7e000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 231 0x55d9c34cc600 osd_ping(ping e501 stamp 2018-07-11 07:50:27.637469) v4 > -9620> 2018-07-11 07:50:27.292295 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.6 10.8.128.115:0/13323 231 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:27.637469) v4 ==== 2004+0+0 (2488505686 0 0) 0x55d9c34cc600 con 0x55d986d7e000 > -9619> 2018-07-11 07:50:27.292305 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:27.637469) v4 -- 0x55d986d96200 con 0 > -9618> 2018-07-11 07:50:27.292331 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.115:0/13323 conn(0x55d986d24800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 231 0x55d9c9123a00 osd_ping(ping e501 stamp 2018-07-11 07:50:27.637469) v4 > -9617> 2018-07-11 07:50:27.292356 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.6 10.8.128.115:0/13323 231 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:27.637469) v4 ==== 2004+0+0 (2488505686 0 0) 0x55d9c9123a00 con 0x55d986d24800 > -9616> 2018-07-11 07:50:27.292367 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:27.637469) v4 -- 0x55d9b9ac2400 con 0 > -9615> 2018-07-11 07:50:27.316432 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 776 0x55d99e3ce480 rep_scrubmap(5.2e e501 from shard 2) v1 > -9614> 2018-07-11 07:50:27.316452 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 776 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12472 (1269249638 0 3423817316) 0x55d99e3ce480 con 0x55d986bf0800 > -9613> 2018-07-11 07:50:27.346488 7ffab6be6700 1 -- 10.8.128.107:6800/13927 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55d985053b80 con 0 > -9612> 2018-07-11 07:50:27.346559 7ffab6be6700 1 -- 10.8.128.107:6800/13927 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55d9ba81b600 con 0 > -9611> 2018-07-11 07:50:27.346988 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 770 0x55d99e3ce6c0 rep_scrubmap(5.2e e501 from shard 4) v1 > -9610> 2018-07-11 07:50:27.347008 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 770 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12472 (3686815630 0 3423817316) 0x55d99e3ce6c0 con 0x55d986bec000 > -9609> 2018-07-11 07:50:27.352439 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -9608> 2018-07-11 07:50:27.352466 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -9607> 2018-07-11 07:50:27.352517 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -9606> 2018-07-11 07:50:27.352974 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -9605> 2018-07-11 07:50:27.353845 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7512ead7:::benchmark_data_magna115_10867_object2489741:0,end:5:75134436:::benchmark_data_magna115_10867_object3756202:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb772280 con 0 > -9604> 2018-07-11 07:50:27.353892 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7512ead7:::benchmark_data_magna115_10867_object2489741:0,end:5:75134436:::benchmark_data_magna115_10867_object3756202:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98944b400 con 0 > -9603> 2018-07-11 07:50:27.439656 7ffacbbd1700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15263 conn(0x55d985d44000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.5 seq 214 0x55d9b67d8200 osd_ping(ping e501 stamp 2018-07-11 07:50:28.669553) v4 > -9602> 2018-07-11 07:50:27.439678 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 <== osd.5 10.8.128.113:0/15263 214 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:28.669553) v4 ==== 2004+0+0 (3151410085 0 0) 0x55d9b67d8200 con 0x55d985d44000 > -9601> 2018-07-11 07:50:27.439688 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:28.669553) v4 -- 0x55d986d96200 con 0 > -9600> 2018-07-11 07:50:27.439684 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15263 conn(0x55d985d45800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.5 seq 214 0x55d9c9122a00 osd_ping(ping e501 stamp 2018-07-11 07:50:28.669553) v4 > -9599> 2018-07-11 07:50:27.439701 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.5 10.8.128.113:0/15263 214 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:28.669553) v4 ==== 2004+0+0 (3151410085 0 0) 0x55d9c9122a00 con 0x55d985d45800 > -9598> 2018-07-11 07:50:27.439709 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:28.669553) v4 -- 0x55d9ba818800 con 0 > -9597> 2018-07-11 07:50:27.560184 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 771 0x55d99e3ce900 rep_scrubmap(5.2e e501 from shard 4) v1 > -9596> 2018-07-11 07:50:27.560204 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 771 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12472 (3686815630 0 3159269457) 0x55d99e3ce900 con 0x55d986bec000 > -9595> 2018-07-11 07:50:27.566691 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 777 0x55d99e3ceb40 rep_scrubmap(5.2e e501 from shard 2) v1 > -9594> 2018-07-11 07:50:27.566710 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 777 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12472 (1269249638 0 3159269457) 0x55d99e3ceb40 con 0x55d986bf0800 > -9593> 2018-07-11 07:50:27.609235 7ffab1bdc700 5 osd.10 501 heartbeat: osd_stat(787 GB used, 143 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,2]) > -9592> 2018-07-11 07:50:27.609271 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6802/15255 -- osd_ping(ping e501 stamp 2018-07-11 07:50:27.609257) v4 -- 0x55d986cd6800 con 0 > -9591> 2018-07-11 07:50:27.609297 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6803/15255 -- osd_ping(ping e501 stamp 2018-07-11 07:50:27.609257) v4 -- 0x55d980f5b200 con 0 > -9590> 2018-07-11 07:50:27.609313 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6806/12677 -- osd_ping(ping e501 stamp 2018-07-11 07:50:27.609257) v4 -- 0x55d986cd5400 con 0 > -9589> 2018-07-11 07:50:27.609327 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6807/12677 -- osd_ping(ping e501 stamp 2018-07-11 07:50:27.609257) v4 -- 0x55d986c84000 con 0 > -9588> 2018-07-11 07:50:27.609340 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6808/12651 -- osd_ping(ping e501 stamp 2018-07-11 07:50:27.609257) v4 -- 0x55d9bcd79400 con 0 > -9587> 2018-07-11 07:50:27.609351 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6810/12651 -- osd_ping(ping e501 stamp 2018-07-11 07:50:27.609257) v4 -- 0x55d982cdf600 con 0 > -9586> 2018-07-11 07:50:27.609363 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6809/12650 -- osd_ping(ping e501 stamp 2018-07-11 07:50:27.609257) v4 -- 0x55d9bf804400 con 0 > -9585> 2018-07-11 07:50:27.609377 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6811/12650 -- osd_ping(ping e501 stamp 2018-07-11 07:50:27.609257) v4 -- 0x55d9b67d8e00 con 0 > -9584> 2018-07-11 07:50:27.609395 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6810/15287 -- osd_ping(ping e501 stamp 2018-07-11 07:50:27.609257) v4 -- 0x55d9bdba5000 con 0 > -9583> 2018-07-11 07:50:27.609408 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6811/15287 -- osd_ping(ping e501 stamp 2018-07-11 07:50:27.609257) v4 -- 0x55d9b437b200 con 0 > -9582> 2018-07-11 07:50:27.609420 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6806/15263 -- osd_ping(ping e501 stamp 2018-07-11 07:50:27.609257) v4 -- 0x55d9bdba4200 con 0 > -9581> 2018-07-11 07:50:27.609434 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6807/15263 -- osd_ping(ping e501 stamp 2018-07-11 07:50:27.609257) v4 -- 0x55d9c74e1800 con 0 > -9580> 2018-07-11 07:50:27.609447 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.115:6802/13323 -- osd_ping(ping e501 stamp 2018-07-11 07:50:27.609257) v4 -- 0x55d9bdba4c00 con 0 > -9579> 2018-07-11 07:50:27.609459 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.115:6803/13323 -- osd_ping(ping e501 stamp 2018-07-11 07:50:27.609257) v4 -- 0x55d986c22000 con 0 > -9578> 2018-07-11 07:50:27.609472 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6806/15885 -- osd_ping(ping e501 stamp 2018-07-11 07:50:27.609257) v4 -- 0x55d9b9ac2c00 con 0 > -9577> 2018-07-11 07:50:27.609486 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6807/15885 -- osd_ping(ping e501 stamp 2018-07-11 07:50:27.609257) v4 -- 0x55d986c23000 con 0 > -9576> 2018-07-11 07:50:27.609498 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.108:6802/11967 -- osd_ping(ping e501 stamp 2018-07-11 07:50:27.609257) v4 -- 0x55d9bfbde400 con 0 > -9575> 2018-07-11 07:50:27.609509 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.108:6803/11967 -- osd_ping(ping e501 stamp 2018-07-11 07:50:27.609257) v4 -- 0x55d9b4388200 con 0 > -9574> 2018-07-11 07:50:27.609522 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.107:6806/12617 -- osd_ping(ping e501 stamp 2018-07-11 07:50:27.609257) v4 -- 0x55d9c908ec00 con 0 > -9573> 2018-07-11 07:50:27.609535 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.107:6807/12617 -- osd_ping(ping e501 stamp 2018-07-11 07:50:27.609257) v4 -- 0x55d9c3383200 con 0 > -9572> 2018-07-11 07:50:27.609551 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6802/15919 -- osd_ping(ping e501 stamp 2018-07-11 07:50:27.609257) v4 -- 0x55d986d96600 con 0 > -9571> 2018-07-11 07:50:27.609538 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -9570> 2018-07-11 07:50:27.609564 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6803/15919 -- osd_ping(ping e501 stamp 2018-07-11 07:50:27.609257) v4 -- 0x55d9b60df800 con 0 > -9569> 2018-07-11 07:50:27.609562 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -9568> 2018-07-11 07:50:27.609624 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -9567> 2018-07-11 07:50:27.609710 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.107:6806/12617 conn(0x55d986bd2000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.9 seq 224 0x55d9c9122a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:27.609257) v4 > -9566> 2018-07-11 07:50:27.609731 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.9 10.8.128.107:6806/12617 224 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:27.609257) v4 ==== 2004+0+0 (2686425764 0 0) 0x55d9c9122a00 con 0x55d986bd2000 > -9565> 2018-07-11 07:50:27.609765 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.107:6807/12617 conn(0x55d986bd3800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.9 seq 224 0x55d9c34cc600 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:27.609257) v4 > -9564> 2018-07-11 07:50:27.609788 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.9 10.8.128.107:6807/12617 224 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:27.609257) v4 ==== 2004+0+0 (2686425764 0 0) 0x55d9c34cc600 con 0x55d986bd3800 > -9563> 2018-07-11 07:50:27.609817 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6809/12650 conn(0x55d986bcb000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 224 0x55d9c34cc600 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:27.609257) v4 > -9562> 2018-07-11 07:50:27.609821 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6808/12651 conn(0x55d986b9a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.2 seq 224 0x55d9c9123a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:27.609257) v4 > -9561> 2018-07-11 07:50:27.609829 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.3 10.8.128.106:6809/12650 224 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:27.609257) v4 ==== 2004+0+0 (2686425764 0 0) 0x55d9c34cc600 con 0x55d986bcb000 > -9560> 2018-07-11 07:50:27.609840 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6810/12651 conn(0x55d986b9b800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.2 seq 224 0x55d9c9122a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:27.609257) v4 > -9559> 2018-07-11 07:50:27.609839 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.2 10.8.128.106:6808/12651 224 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:27.609257) v4 ==== 2004+0+0 (2686425764 0 0) 0x55d9c9123a00 con 0x55d986b9a000 > -9558> 2018-07-11 07:50:27.609865 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.2 10.8.128.106:6810/12651 224 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:27.609257) v4 ==== 2004+0+0 (2686425764 0 0) 0x55d9c9122a00 con 0x55d986b9b800 > -9557> 2018-07-11 07:50:27.609871 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6806/12677 conn(0x55d986b58000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.1 seq 224 0x55d9c34cc600 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:27.609257) v4 > -9556> 2018-07-11 07:50:27.609887 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.1 10.8.128.106:6806/12677 224 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:27.609257) v4 ==== 2004+0+0 (2686425764 0 0) 0x55d9c34cc600 con 0x55d986b58000 > -9555> 2018-07-11 07:50:27.609909 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6807/12677 conn(0x55d986b59800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.1 seq 224 0x55d9c9123a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:27.609257) v4 > -9554> 2018-07-11 07:50:27.609924 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.1 10.8.128.106:6807/12677 224 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:27.609257) v4 ==== 2004+0+0 (2686425764 0 0) 0x55d9c9123a00 con 0x55d986b59800 > -9553> 2018-07-11 07:50:27.609932 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6806/15263 conn(0x55d986b9d000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.5 seq 224 0x55d9c34cc600 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:27.609257) v4 > -9552> 2018-07-11 07:50:27.609921 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6803/15255 conn(0x55d986bc9800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.0 seq 224 0x55d9c9122a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:27.609257) v4 > -9551> 2018-07-11 07:50:27.609947 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.5 10.8.128.113:6806/15263 224 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:27.609257) v4 ==== 2004+0+0 (2686425764 0 0) 0x55d9c34cc600 con 0x55d986b9d000 > -9550> 2018-07-11 07:50:27.609953 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.0 10.8.128.113:6803/15255 224 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:27.609257) v4 ==== 2004+0+0 (2686425764 0 0) 0x55d9c9122a00 con 0x55d986bc9800 > -9549> 2018-07-11 07:50:27.609955 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6802/15255 conn(0x55d986bc8000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.0 seq 224 0x55d9c9123a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:27.609257) v4 > -9548> 2018-07-11 07:50:27.609964 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.0 10.8.128.113:6802/15255 224 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:27.609257) v4 ==== 2004+0+0 (2686425764 0 0) 0x55d9c9123a00 con 0x55d986bc8000 > -9547> 2018-07-11 07:50:27.609995 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6811/15287 conn(0x55d986bb0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 224 0x55d9c9122a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:27.609257) v4 > -9546> 2018-07-11 07:50:27.610018 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6811/12650 conn(0x55d986bcc800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 224 0x55d9c9123a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:27.609257) v4 > -9545> 2018-07-11 07:50:27.610031 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.3 10.8.128.106:6811/12650 224 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:27.609257) v4 ==== 2004+0+0 (2686425764 0 0) 0x55d9c9123a00 con 0x55d986bcc800 > -9544> 2018-07-11 07:50:27.610028 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.115:6803/13323 conn(0x55d986bc2800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 224 0x55d9c34cc600 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:27.609257) v4 > -9543> 2018-07-11 07:50:27.610039 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.6 10.8.128.115:6803/13323 224 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:27.609257) v4 ==== 2004+0+0 (2686425764 0 0) 0x55d9c34cc600 con 0x55d986bc2800 > -9542> 2018-07-11 07:50:27.610013 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.4 10.8.128.113:6811/15287 224 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:27.609257) v4 ==== 2004+0+0 (2686425764 0 0) 0x55d9c9122a00 con 0x55d986bb0800 > -9541> 2018-07-11 07:50:27.610060 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6807/15263 conn(0x55d986b9e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.5 seq 224 0x55d9c9123a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:27.609257) v4 > -9540> 2018-07-11 07:50:27.610072 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.5 10.8.128.113:6807/15263 224 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:27.609257) v4 ==== 2004+0+0 (2686425764 0 0) 0x55d9c9123a00 con 0x55d986b9e800 > -9539> 2018-07-11 07:50:27.610079 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.108:6802/11967 conn(0x55d986bbe000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.8 seq 224 0x55d9c34cc600 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:27.609257) v4 > -9538> 2018-07-11 07:50:27.610096 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.8 10.8.128.108:6802/11967 224 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:27.609257) v4 ==== 2004+0+0 (2686425764 0 0) 0x55d9c34cc600 con 0x55d986bbe000 > -9537> 2018-07-11 07:50:27.610102 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.108:6803/11967 conn(0x55d986bbf800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.8 seq 224 0x55d9c9123a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:27.609257) v4 > -9536> 2018-07-11 07:50:27.610097 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.115:6802/13323 conn(0x55d986bc1000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 224 0x55d9c9122a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:27.609257) v4 > -9535> 2018-07-11 07:50:27.610117 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.8 10.8.128.108:6803/11967 224 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:27.609257) v4 ==== 2004+0+0 (2686425764 0 0) 0x55d9c9123a00 con 0x55d986bbf800 > -9534> 2018-07-11 07:50:27.610124 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.6 10.8.128.115:6802/13323 224 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:27.609257) v4 ==== 2004+0+0 (2686425764 0 0) 0x55d9c9122a00 con 0x55d986bc1000 > -9533> 2018-07-11 07:50:27.610144 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6810/15287 conn(0x55d986baf000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 224 0x55d9c9123a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:27.609257) v4 > -9532> 2018-07-11 07:50:27.610156 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.4 10.8.128.113:6810/15287 224 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:27.609257) v4 ==== 2004+0+0 (2686425764 0 0) 0x55d9c9123a00 con 0x55d986baf000 > -9531> 2018-07-11 07:50:27.610174 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6803/15919 conn(0x55d986b5c800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.11 seq 224 0x55d9c34cc600 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:27.609257) v4 > -9530> 2018-07-11 07:50:27.610182 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6802/15919 conn(0x55d986b5b000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.11 seq 224 0x55d9c9122a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:27.609257) v4 > -9529> 2018-07-11 07:50:27.610198 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.11 10.8.128.114:6803/15919 224 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:27.609257) v4 ==== 2004+0+0 (2686425764 0 0) 0x55d9c34cc600 con 0x55d986b5c800 > -9528> 2018-07-11 07:50:27.610200 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.11 10.8.128.114:6802/15919 224 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:27.609257) v4 ==== 2004+0+0 (2686425764 0 0) 0x55d9c9122a00 con 0x55d986b5b000 > -9527> 2018-07-11 07:50:27.610226 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6807/15885 conn(0x55d986bad800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.7 seq 224 0x55d9c34cc600 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:27.609257) v4 > -9526> 2018-07-11 07:50:27.610237 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.7 10.8.128.114:6807/15885 224 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:27.609257) v4 ==== 2004+0+0 (2686425764 0 0) 0x55d9c34cc600 con 0x55d986bad800 > -9525> 2018-07-11 07:50:27.610265 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6806/15885 conn(0x55d986bac000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.7 seq 224 0x55d9c9122a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:27.609257) v4 > -9524> 2018-07-11 07:50:27.610284 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.7 10.8.128.114:6806/15885 224 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:27.609257) v4 ==== 2004+0+0 (2686425764 0 0) 0x55d9c9122a00 con 0x55d986bac000 > -9523> 2018-07-11 07:50:27.610373 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -9522> 2018-07-11 07:50:27.655305 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75134436:::benchmark_data_magna115_10867_object3756202:0,end:5:75139cc4:::benchmark_data_magna115_10867_object3216818:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb772000 con 0 > -9521> 2018-07-11 07:50:27.655343 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75134436:::benchmark_data_magna115_10867_object3756202:0,end:5:75139cc4:::benchmark_data_magna115_10867_object3216818:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb772500 con 0 > -9520> 2018-07-11 07:50:27.828541 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 2962 0x55d9c34cc600 MOSDScrubReserve(11.3s5 REQUEST e501) v1 > -9519> 2018-07-11 07:50:27.828572 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 2962 ==== MOSDScrubReserve(11.3s5 REQUEST e501) v1 ==== 43+0+0 (3260170403 0 0) 0x55d9c34cc600 con 0x55d986c7e000 > -9518> 2018-07-11 07:50:27.828650 7ffab3be0700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e501) v1 -- 0x55d9869bfc00 con 0 > -9517> 2018-07-11 07:50:27.829030 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 2963 0x55d9b60df800 MOSDScrubReserve(11.3s5 RELEASE e501) v1 > -9516> 2018-07-11 07:50:27.829061 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 2963 ==== MOSDScrubReserve(11.3s5 RELEASE e501) v1 ==== 43+0+0 (223328929 0 0) 0x55d9b60df800 con 0x55d986c7e000 > -9515> 2018-07-11 07:50:27.861915 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 778 0x55d99e3ced80 rep_scrubmap(5.2e e501 from shard 2) v1 > -9514> 2018-07-11 07:50:27.861934 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 778 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12484 (1269249638 0 500733922) 0x55d99e3ced80 con 0x55d986bf0800 > -9513> 2018-07-11 07:50:27.891668 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 772 0x55d99e3cefc0 rep_scrubmap(5.2e e501 from shard 4) v1 > -9512> 2018-07-11 07:50:27.891688 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 772 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12484 (3686815630 0 500733922) 0x55d99e3cefc0 con 0x55d986bec000 > -9511> 2018-07-11 07:50:27.919456 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -9510> 2018-07-11 07:50:27.919482 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -9509> 2018-07-11 07:50:27.919530 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -9508> 2018-07-11 07:50:27.919997 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -9507> 2018-07-11 07:50:27.920863 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75139cc4:::benchmark_data_magna115_10867_object3216818:0,end:5:7513e11b:::benchmark_data_magna115_10867_object73318:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb772a00 con 0 > -9506> 2018-07-11 07:50:27.920909 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75139cc4:::benchmark_data_magna115_10867_object3216818:0,end:5:7513e11b:::benchmark_data_magna115_10867_object73318:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb772c80 con 0 > -9505> 2018-07-11 07:50:28.095997 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 779 0x55d99e3cf200 rep_scrubmap(5.2e e501 from shard 2) v1 > -9504> 2018-07-11 07:50:28.096017 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 779 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12470 (1269249638 0 3822973892) 0x55d99e3cf200 con 0x55d986bf0800 > -9503> 2018-07-11 07:50:28.135682 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 773 0x55d99e3cf440 rep_scrubmap(5.2e e501 from shard 4) v1 > -9502> 2018-07-11 07:50:28.135701 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 773 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12470 (3686815630 0 3822973892) 0x55d99e3cf440 con 0x55d986bec000 > -9501> 2018-07-11 07:50:28.175123 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15287 conn(0x55d981240800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 242 0x55d986cd4c00 osd_ping(ping e501 stamp 2018-07-11 07:50:29.405050) v4 > -9500> 2018-07-11 07:50:28.175146 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.4 10.8.128.113:0/15287 242 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:29.405050) v4 ==== 2004+0+0 (3867002952 0 0) 0x55d986cd4c00 con 0x55d981240800 > -9499> 2018-07-11 07:50:28.175148 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15287 conn(0x55d98123f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 242 0x55d9c9122a00 osd_ping(ping e501 stamp 2018-07-11 07:50:29.405050) v4 > -9498> 2018-07-11 07:50:28.175160 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:29.405050) v4 -- 0x55d9b4388200 con 0 > -9497> 2018-07-11 07:50:28.175171 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.4 10.8.128.113:0/15287 242 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:29.405050) v4 ==== 2004+0+0 (3867002952 0 0) 0x55d9c9122a00 con 0x55d98123f000 > -9496> 2018-07-11 07:50:28.175190 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:29.405050) v4 -- 0x55d986d96600 con 0 > -9495> 2018-07-11 07:50:28.226800 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -9494> 2018-07-11 07:50:28.226825 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -9493> 2018-07-11 07:50:28.226886 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -9492> 2018-07-11 07:50:28.227384 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -9491> 2018-07-11 07:50:28.235254 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.107:0/12617 conn(0x55d98123c000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 216 0x55d9c9123a00 osd_ping(ping e501 stamp 2018-07-11 07:50:28.234750) v4 > -9490> 2018-07-11 07:50:28.235267 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.107:0/12617 conn(0x55d98123d800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 216 0x55d9b67d8200 osd_ping(ping e501 stamp 2018-07-11 07:50:28.234750) v4 > -9489> 2018-07-11 07:50:28.235281 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.9 10.8.128.107:0/12617 216 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:28.234750) v4 ==== 2004+0+0 (1728004087 0 0) 0x55d9c9123a00 con 0x55d98123c000 > -9488> 2018-07-11 07:50:28.235286 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.9 10.8.128.107:0/12617 216 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:28.234750) v4 ==== 2004+0+0 (1728004087 0 0) 0x55d9b67d8200 con 0x55d98123d800 > -9487> 2018-07-11 07:50:28.235295 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:28.234750) v4 -- 0x55d9b4388200 con 0 > -9486> 2018-07-11 07:50:28.235317 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:28.234750) v4 -- 0x55d9c3383200 con 0 > -9485> 2018-07-11 07:50:28.247163 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67974,epoch:501/498,start:5:7513e11b:::benchmark_data_magna115_10867_object73318:0,end:5:75142ea7:::benchmark_data_magna115_10867_object1633266:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb772780 con 0 > -9484> 2018-07-11 07:50:28.247195 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67974,epoch:501/498,start:5:7513e11b:::benchmark_data_magna115_10867_object73318:0,end:5:75142ea7:::benchmark_data_magna115_10867_object1633266:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb772f00 con 0 > -9483> 2018-07-11 07:50:28.410653 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.114:0/15885 conn(0x55d986d81000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.7 seq 237 0x55d9c9123a00 osd_ping(ping e501 stamp 2018-07-11 07:50:29.040496) v4 > -9482> 2018-07-11 07:50:28.410658 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.114:0/15885 conn(0x55d986d7f800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.7 seq 237 0x55d9829e5000 osd_ping(ping e501 stamp 2018-07-11 07:50:29.040496) v4 > -9481> 2018-07-11 07:50:28.410675 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.7 10.8.128.114:0/15885 237 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:29.040496) v4 ==== 2004+0+0 (4102563539 0 0) 0x55d9829e5000 con 0x55d986d7f800 > -9480> 2018-07-11 07:50:28.410675 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.7 10.8.128.114:0/15885 237 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:29.040496) v4 ==== 2004+0+0 (4102563539 0 0) 0x55d9c9123a00 con 0x55d986d81000 > -9479> 2018-07-11 07:50:28.410694 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:29.040496) v4 -- 0x55d986d96600 con 0 > -9478> 2018-07-11 07:50:28.410722 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:29.040496) v4 -- 0x55d9b4388200 con 0 > -9477> 2018-07-11 07:50:28.462640 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 780 0x55d99e3cf680 rep_scrubmap(5.2e e501 from shard 2) v1 > -9476> 2018-07-11 07:50:28.462660 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 780 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12476 (1269249638 0 506903125) 0x55d99e3cf680 con 0x55d986bf0800 > -9475> 2018-07-11 07:50:28.500074 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 774 0x55d99e3cf8c0 rep_scrubmap(5.2e e501 from shard 4) v1 > -9474> 2018-07-11 07:50:28.500095 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 774 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12476 (3686815630 0 506903125) 0x55d99e3cf8c0 con 0x55d986bec000 > -9473> 2018-07-11 07:50:28.558163 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15255 conn(0x55d986c82800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.0 seq 232 0x55d9c9122a00 osd_ping(ping e501 stamp 2018-07-11 07:50:29.788106) v4 > -9472> 2018-07-11 07:50:28.558190 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.0 10.8.128.113:0/15255 232 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:29.788106) v4 ==== 2004+0+0 (3983631600 0 0) 0x55d9c9122a00 con 0x55d986c82800 > -9471> 2018-07-11 07:50:28.558207 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:29.788106) v4 -- 0x55d986d96600 con 0 > -9470> 2018-07-11 07:50:28.558309 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15255 conn(0x55d986d20000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.0 seq 232 0x55d986cd4c00 osd_ping(ping e501 stamp 2018-07-11 07:50:29.788106) v4 > -9469> 2018-07-11 07:50:28.558331 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.0 10.8.128.113:0/15255 232 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:29.788106) v4 ==== 2004+0+0 (3983631600 0 0) 0x55d986cd4c00 con 0x55d986d20000 > -9468> 2018-07-11 07:50:28.558342 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:29.788106) v4 -- 0x55d9b4388200 con 0 > -9467> 2018-07-11 07:50:28.573714 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -9466> 2018-07-11 07:50:28.573740 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -9465> 2018-07-11 07:50:28.573791 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -9464> 2018-07-11 07:50:28.574284 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -9463> 2018-07-11 07:50:28.575184 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75142ea7:::benchmark_data_magna115_10867_object1633266:0,end:5:75146e30:::benchmark_data_magna115_10867_object1631062:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb773400 con 0 > -9462> 2018-07-11 07:50:28.575230 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75142ea7:::benchmark_data_magna115_10867_object1633266:0,end:5:75146e30:::benchmark_data_magna115_10867_object1631062:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb773900 con 0 > -9461> 2018-07-11 07:50:28.766301 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 781 0x55d99e3cfb00 rep_scrubmap(5.2e e501 from shard 2) v1 > -9460> 2018-07-11 07:50:28.766322 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 781 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12468 (1269249638 0 2409417671) 0x55d99e3cfb00 con 0x55d986bf0800 > -9459> 2018-07-11 07:50:28.814552 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 775 0x55d99e3cfd40 rep_scrubmap(5.2e e501 from shard 4) v1 > -9458> 2018-07-11 07:50:28.814572 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 775 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12468 (3686815630 0 2409417671) 0x55d99e3cfd40 con 0x55d986bec000 > -9457> 2018-07-11 07:50:28.814734 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -9456> 2018-07-11 07:50:28.814763 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -9455> 2018-07-11 07:50:28.814841 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -9454> 2018-07-11 07:50:28.815642 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -9453> 2018-07-11 07:50:28.828733 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 2964 0x55d9869bfc00 MOSDScrubReserve(11.3s5 REQUEST e501) v1 > -9452> 2018-07-11 07:50:28.828752 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 2964 ==== MOSDScrubReserve(11.3s5 REQUEST e501) v1 ==== 43+0+0 (3260170403 0 0) 0x55d9869bfc00 con 0x55d986c7e000 > -9451> 2018-07-11 07:50:28.828822 7ffab3be0700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e501) v1 -- 0x55d9b60df800 con 0 > -9450> 2018-07-11 07:50:28.829224 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 2965 0x55d9b60df800 MOSDScrubReserve(11.3s5 RELEASE e501) v1 > -9449> 2018-07-11 07:50:28.829266 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 2965 ==== MOSDScrubReserve(11.3s5 RELEASE e501) v1 ==== 43+0+0 (223328929 0 0) 0x55d9b60df800 con 0x55d986c7e000 > -9448> 2018-07-11 07:50:28.830789 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'66679,epoch:501/498,start:5:75146e30:::benchmark_data_magna115_10867_object1631062:0,end:5:7514b902:::benchmark_data_magna115_10867_object2827070:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb773180 con 0 > -9447> 2018-07-11 07:50:28.830838 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'66679,epoch:501/498,start:5:75146e30:::benchmark_data_magna115_10867_object1631062:0,end:5:7514b902:::benchmark_data_magna115_10867_object2827070:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454280 con 0 > -9446> 2018-07-11 07:50:28.953941 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6801/15919 conn(0x55d986c3f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=29 cs=1 l=0). rx osd.11 seq 204 0x55d9b67d8200 MOSDScrubReserve(10.f REQUEST e501) v1 > -9445> 2018-07-11 07:50:28.953959 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.11 10.8.128.114:6801/15919 204 ==== MOSDScrubReserve(10.f REQUEST e501) v1 ==== 43+0+0 (2892966580 0 0) 0x55d9b67d8200 con 0x55d986c3f800 > -9444> 2018-07-11 07:50:28.954034 7ffab53e3700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.114:6801/15919 -- MOSDScrubReserve(10.f REJECT e501) v1 -- 0x55d9869bea00 con 0 > -9443> 2018-07-11 07:50:28.954358 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6801/15919 conn(0x55d986c3f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=29 cs=1 l=0). rx osd.11 seq 205 0x55d9869bea00 MOSDScrubReserve(10.f RELEASE e501) v1 > -9442> 2018-07-11 07:50:28.954373 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.11 10.8.128.114:6801/15919 205 ==== MOSDScrubReserve(10.f RELEASE e501) v1 ==== 43+0+0 (1668468918 0 0) 0x55d9869bea00 con 0x55d986c3f800 > -9441> 2018-07-11 07:50:29.067275 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 776 0x55d9bba52000 rep_scrubmap(5.2e e501 from shard 4) v1 > -9440> 2018-07-11 07:50:29.067296 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 776 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12472 (3686815630 0 3693614380) 0x55d9bba52000 con 0x55d986bec000 > -9439> 2018-07-11 07:50:29.074172 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 782 0x55d9bba52240 rep_scrubmap(5.2e e501 from shard 2) v1 > -9438> 2018-07-11 07:50:29.074192 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 782 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12472 (1269249638 0 3693614380) 0x55d9bba52240 con 0x55d986bf0800 > -9437> 2018-07-11 07:50:29.074349 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -9436> 2018-07-11 07:50:29.074377 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -9435> 2018-07-11 07:50:29.074455 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -9434> 2018-07-11 07:50:29.075183 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -9433> 2018-07-11 07:50:29.089151 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7514b902:::benchmark_data_magna115_10867_object2827070:0,end:5:7514ef24:::benchmark_data_magna115_10867_object1035826:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb773680 con 0 > -9432> 2018-07-11 07:50:29.089200 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7514b902:::benchmark_data_magna115_10867_object2827070:0,end:5:7514ef24:::benchmark_data_magna115_10867_object1035826:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454000 con 0 > -9431> 2018-07-11 07:50:29.156889 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12677 conn(0x55d986df2000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.1 seq 236 0x55d9c9123a00 osd_ping(ping e501 stamp 2018-07-11 07:50:29.203818) v4 > -9430> 2018-07-11 07:50:29.156896 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12677 conn(0x55d986db4800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.1 seq 236 0x55d9829e5000 osd_ping(ping e501 stamp 2018-07-11 07:50:29.203818) v4 > -9429> 2018-07-11 07:50:29.156911 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.1 10.8.128.106:0/12677 236 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:29.203818) v4 ==== 2004+0+0 (25114563 0 0) 0x55d9c9123a00 con 0x55d986df2000 > -9428> 2018-07-11 07:50:29.156916 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.1 10.8.128.106:0/12677 236 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:29.203818) v4 ==== 2004+0+0 (25114563 0 0) 0x55d9829e5000 con 0x55d986db4800 > -9427> 2018-07-11 07:50:29.156922 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:29.203818) v4 -- 0x55d9b4388200 con 0 > -9426> 2018-07-11 07:50:29.156956 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:29.203818) v4 -- 0x55d986d96600 con 0 > -9425> 2018-07-11 07:50:29.312948 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 777 0x55d9bba52480 rep_scrubmap(5.2e e501 from shard 4) v1 > -9424> 2018-07-11 07:50:29.312969 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 777 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12476 (3686815630 0 1832334637) 0x55d9bba52480 con 0x55d986bec000 > -9423> 2018-07-11 07:50:29.317348 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 783 0x55d9bba526c0 rep_scrubmap(5.2e e501 from shard 2) v1 > -9422> 2018-07-11 07:50:29.317368 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 783 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12476 (1269249638 0 1832334637) 0x55d9bba526c0 con 0x55d986bf0800 > -9421> 2018-07-11 07:50:29.335733 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.107:0/12617 conn(0x55d98123d800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 217 0x55d9c3383200 osd_ping(ping e501 stamp 2018-07-11 07:50:29.335412) v4 > -9420> 2018-07-11 07:50:29.335752 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.9 10.8.128.107:0/12617 217 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:29.335412) v4 ==== 2004+0+0 (2269018384 0 0) 0x55d9c3383200 con 0x55d98123d800 > -9419> 2018-07-11 07:50:29.335751 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.107:0/12617 conn(0x55d98123c000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 217 0x55d986cd4c00 osd_ping(ping e501 stamp 2018-07-11 07:50:29.335412) v4 > -9418> 2018-07-11 07:50:29.335763 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:29.335412) v4 -- 0x55d986c23000 con 0 > -9417> 2018-07-11 07:50:29.335774 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.9 10.8.128.107:0/12617 217 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:29.335412) v4 ==== 2004+0+0 (2269018384 0 0) 0x55d986cd4c00 con 0x55d98123c000 > -9416> 2018-07-11 07:50:29.335793 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:29.335412) v4 -- 0x55d9b4388200 con 0 > -9415> 2018-07-11 07:50:29.403151 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -9414> 2018-07-11 07:50:29.403176 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -9413> 2018-07-11 07:50:29.403227 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -9412> 2018-07-11 07:50:29.403696 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -9411> 2018-07-11 07:50:29.455825 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67713,epoch:501/498,start:5:7514ef24:::benchmark_data_magna115_10867_object1035826:0,end:5:751536e5:::benchmark_data_magna115_10867_object2907387:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb773b80 con 0 > -9410> 2018-07-11 07:50:29.455862 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67713,epoch:501/498,start:5:7514ef24:::benchmark_data_magna115_10867_object1035826:0,end:5:751536e5:::benchmark_data_magna115_10867_object2907387:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98944b680 con 0 > -9409> 2018-07-11 07:50:29.692394 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 784 0x55d9bba52900 rep_scrubmap(5.2e e501 from shard 2) v1 > -9408> 2018-07-11 07:50:29.692414 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 784 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12468 (1269249638 0 1949325865) 0x55d9bba52900 con 0x55d986bf0800 > -9407> 2018-07-11 07:50:29.722306 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 778 0x55d9bba52b40 rep_scrubmap(5.2e e501 from shard 4) v1 > -9406> 2018-07-11 07:50:29.722326 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 778 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12468 (3686815630 0 1949325865) 0x55d9bba52b40 con 0x55d986bec000 > -9405> 2018-07-11 07:50:29.722455 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -9404> 2018-07-11 07:50:29.722479 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -9403> 2018-07-11 07:50:29.722530 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -9402> 2018-07-11 07:50:29.722991 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -9401> 2018-07-11 07:50:29.724016 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:751536e5:::benchmark_data_magna115_10867_object2907387:0,end:5:75159267:::benchmark_data_magna115_10867_object1664419:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454c80 con 0 > -9400> 2018-07-11 07:50:29.724061 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:751536e5:::benchmark_data_magna115_10867_object2907387:0,end:5:75159267:::benchmark_data_magna115_10867_object1664419:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454a00 con 0 > -9399> 2018-07-11 07:50:29.740084 7ffacbbd1700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15263 conn(0x55d985d44000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.5 seq 215 0x55d9bfbde400 osd_ping(ping e501 stamp 2018-07-11 07:50:30.970030) v4 > -9398> 2018-07-11 07:50:29.740084 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15263 conn(0x55d985d45800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.5 seq 215 0x55d9c9122a00 osd_ping(ping e501 stamp 2018-07-11 07:50:30.970030) v4 > -9397> 2018-07-11 07:50:29.740108 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 <== osd.5 10.8.128.113:0/15263 215 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:30.970030) v4 ==== 2004+0+0 (4185552420 0 0) 0x55d9bfbde400 con 0x55d985d44000 > -9396> 2018-07-11 07:50:29.740111 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.5 10.8.128.113:0/15263 215 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:30.970030) v4 ==== 2004+0+0 (4185552420 0 0) 0x55d9c9122a00 con 0x55d985d45800 > -9395> 2018-07-11 07:50:29.740117 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:30.970030) v4 -- 0x55d986c23000 con 0 > -9394> 2018-07-11 07:50:29.740137 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:30.970030) v4 -- 0x55d986d96600 con 0 > -9393> 2018-07-11 07:50:29.828993 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 2966 0x55d9c3383200 MOSDScrubReserve(11.3s5 REQUEST e501) v1 > -9392> 2018-07-11 07:50:29.829012 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 2966 ==== MOSDScrubReserve(11.3s5 REQUEST e501) v1 ==== 43+0+0 (3260170403 0 0) 0x55d9c3383200 con 0x55d986c7e000 > -9391> 2018-07-11 07:50:29.829083 7ffab3be0700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e501) v1 -- 0x55d9b60df800 con 0 > -9390> 2018-07-11 07:50:29.829569 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 2967 0x55d9b60df800 MOSDScrubReserve(11.3s5 RELEASE e501) v1 > -9389> 2018-07-11 07:50:29.829587 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 2967 ==== MOSDScrubReserve(11.3s5 RELEASE e501) v1 ==== 43+0+0 (223328929 0 0) 0x55d9b60df800 con 0x55d986c7e000 > -9388> 2018-07-11 07:50:29.907461 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12650 conn(0x55d986d23000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 225 0x55d9829e5000 osd_ping(ping e501 stamp 2018-07-11 07:50:29.954349) v4 > -9387> 2018-07-11 07:50:29.907483 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.3 10.8.128.106:0/12650 225 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:29.954349) v4 ==== 2004+0+0 (879583971 0 0) 0x55d9829e5000 con 0x55d986d23000 > -9386> 2018-07-11 07:50:29.907492 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:29.954349) v4 -- 0x55d986d96600 con 0 > -9385> 2018-07-11 07:50:29.907567 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12650 conn(0x55d986d21800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 225 0x55d9bfbde400 osd_ping(ping e501 stamp 2018-07-11 07:50:29.954349) v4 > -9384> 2018-07-11 07:50:29.907587 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.3 10.8.128.106:0/12650 225 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:29.954349) v4 ==== 2004+0+0 (879583971 0 0) 0x55d9bfbde400 con 0x55d986d21800 > -9383> 2018-07-11 07:50:29.907598 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:29.954349) v4 -- 0x55d986c22000 con 0 > -9382> 2018-07-11 07:50:29.927870 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 785 0x55d9bba52d80 rep_scrubmap(5.2e e501 from shard 2) v1 > -9381> 2018-07-11 07:50:29.927890 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 785 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12466 (1269249638 0 1308212800) 0x55d9bba52d80 con 0x55d986bf0800 > -9380> 2018-07-11 07:50:29.941849 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 779 0x55d9bba52fc0 rep_scrubmap(5.2e e501 from shard 4) v1 > -9379> 2018-07-11 07:50:29.941869 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 779 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12466 (3686815630 0 1308212800) 0x55d9bba52fc0 con 0x55d986bec000 > -9378> 2018-07-11 07:50:30.084998 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.114:0/15919 conn(0x55d986df5000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.11 seq 234 0x55d9c9122a00 osd_ping(ping e501 stamp 2018-07-11 07:50:30.714760) v4 > -9377> 2018-07-11 07:50:30.084998 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.114:0/15919 conn(0x55d986df3800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.11 seq 234 0x55d986c23000 osd_ping(ping e501 stamp 2018-07-11 07:50:30.714760) v4 > -9376> 2018-07-11 07:50:30.085030 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.11 10.8.128.114:0/15919 234 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:30.714760) v4 ==== 2004+0+0 (279896728 0 0) 0x55d986c23000 con 0x55d986df3800 > -9375> 2018-07-11 07:50:30.085028 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.11 10.8.128.114:0/15919 234 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:30.714760) v4 ==== 2004+0+0 (279896728 0 0) 0x55d9c9122a00 con 0x55d986df5000 > -9374> 2018-07-11 07:50:30.085053 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:30.714760) v4 -- 0x55d986c22000 con 0 > -9373> 2018-07-11 07:50:30.085079 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:30.714760) v4 -- 0x55d986d96600 con 0 > -9372> 2018-07-11 07:50:30.092141 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -9371> 2018-07-11 07:50:30.092164 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -9370> 2018-07-11 07:50:30.092223 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -9369> 2018-07-11 07:50:30.092679 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -9368> 2018-07-11 07:50:30.139440 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75159267:::benchmark_data_magna115_10867_object1664419:0,end:5:7515f61c:::benchmark_data_magna115_10867_object1855999:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454780 con 0 > -9367> 2018-07-11 07:50:30.139491 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75159267:::benchmark_data_magna115_10867_object1664419:0,end:5:7515f61c:::benchmark_data_magna115_10867_object1855999:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455180 con 0 > -9366> 2018-07-11 07:50:30.240480 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15263 conn(0x55d985d45800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.5 seq 216 0x55d9829e5000 osd_ping(ping e501 stamp 2018-07-11 07:50:31.470406) v4 > -9365> 2018-07-11 07:50:30.240513 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.5 10.8.128.113:0/15263 216 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:31.470406) v4 ==== 2004+0+0 (2717744560 0 0) 0x55d9829e5000 con 0x55d985d45800 > -9364> 2018-07-11 07:50:30.240524 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:31.470406) v4 -- 0x55d986d96600 con 0 > -9363> 2018-07-11 07:50:30.240577 7ffacbbd1700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15263 conn(0x55d985d44000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.5 seq 216 0x55d9bfbde400 osd_ping(ping e501 stamp 2018-07-11 07:50:31.470406) v4 > -9362> 2018-07-11 07:50:30.240598 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 <== osd.5 10.8.128.113:0/15263 216 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:31.470406) v4 ==== 2004+0+0 (2717744560 0 0) 0x55d9bfbde400 con 0x55d985d44000 > -9361> 2018-07-11 07:50:30.240610 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:31.470406) v4 -- 0x55d986c22000 con 0 > -9360> 2018-07-11 07:50:30.257281 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12677 conn(0x55d986df2000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.1 seq 237 0x55d986cd4c00 osd_ping(ping e501 stamp 2018-07-11 07:50:30.304215) v4 > -9359> 2018-07-11 07:50:30.257285 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12677 conn(0x55d986db4800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.1 seq 237 0x55d9c9122a00 osd_ping(ping e501 stamp 2018-07-11 07:50:30.304215) v4 > -9358> 2018-07-11 07:50:30.257307 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.1 10.8.128.106:0/12677 237 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:30.304215) v4 ==== 2004+0+0 (538695246 0 0) 0x55d986cd4c00 con 0x55d986df2000 > -9357> 2018-07-11 07:50:30.257319 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.1 10.8.128.106:0/12677 237 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:30.304215) v4 ==== 2004+0+0 (538695246 0 0) 0x55d9c9122a00 con 0x55d986db4800 > -9356> 2018-07-11 07:50:30.257326 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:30.304215) v4 -- 0x55d9b4388200 con 0 > -9355> 2018-07-11 07:50:30.257349 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:30.304215) v4 -- 0x55d986d96600 con 0 > -9354> 2018-07-11 07:50:30.358450 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 786 0x55d9bba53200 rep_scrubmap(5.2e e501 from shard 2) v1 > -9353> 2018-07-11 07:50:30.358469 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 786 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12478 (1269249638 0 3152465271) 0x55d9bba53200 con 0x55d986bf0800 > -9352> 2018-07-11 07:50:30.372733 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 780 0x55d9bba53440 rep_scrubmap(5.2e e501 from shard 4) v1 > -9351> 2018-07-11 07:50:30.372753 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 780 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12478 (3686815630 0 3152465271) 0x55d9bba53440 con 0x55d986bec000 > -9350> 2018-07-11 07:50:30.436155 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.107:0/12617 conn(0x55d98123c000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 218 0x55d9c9123a00 osd_ping(ping e501 stamp 2018-07-11 07:50:30.435829) v4 > -9349> 2018-07-11 07:50:30.436165 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.107:0/12617 conn(0x55d98123d800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 218 0x55d986c23000 osd_ping(ping e501 stamp 2018-07-11 07:50:30.435829) v4 > -9348> 2018-07-11 07:50:30.436176 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.9 10.8.128.107:0/12617 218 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:30.435829) v4 ==== 2004+0+0 (393164068 0 0) 0x55d9c9123a00 con 0x55d98123c000 > -9347> 2018-07-11 07:50:30.436182 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.9 10.8.128.107:0/12617 218 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:30.435829) v4 ==== 2004+0+0 (393164068 0 0) 0x55d986c23000 con 0x55d98123d800 > -9346> 2018-07-11 07:50:30.436190 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:30.435829) v4 -- 0x55d9b4388200 con 0 > -9345> 2018-07-11 07:50:30.436219 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:30.435829) v4 -- 0x55d986c22000 con 0 > -9344> 2018-07-11 07:50:30.468513 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -9343> 2018-07-11 07:50:30.468543 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -9342> 2018-07-11 07:50:30.468593 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -9341> 2018-07-11 07:50:30.469059 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -9340> 2018-07-11 07:50:30.470153 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7515f61c:::benchmark_data_magna115_10867_object1855999:0,end:5:75165480:::benchmark_data_magna115_10867_object1681434:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455400 con 0 > -9339> 2018-07-11 07:50:30.470189 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7515f61c:::benchmark_data_magna115_10867_object1855999:0,end:5:75165480:::benchmark_data_magna115_10867_object1681434:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454f00 con 0 > -9338> 2018-07-11 07:50:30.673123 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 787 0x55d9bba53680 rep_scrubmap(5.2e e501 from shard 2) v1 > -9337> 2018-07-11 07:50:30.673142 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 787 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12474 (1269249638 0 3964018653) 0x55d9bba53680 con 0x55d986bf0800 > -9336> 2018-07-11 07:50:30.688258 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 781 0x55d9bba538c0 rep_scrubmap(5.2e e501 from shard 4) v1 > -9335> 2018-07-11 07:50:30.688277 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 781 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12474 (3686815630 0 3964018653) 0x55d9bba538c0 con 0x55d986bec000 > -9334> 2018-07-11 07:50:30.688431 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -9333> 2018-07-11 07:50:30.688458 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -9332> 2018-07-11 07:50:30.688510 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -9331> 2018-07-11 07:50:30.688973 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -9330> 2018-07-11 07:50:30.723000 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75165480:::benchmark_data_magna115_10867_object1681434:0,end:5:75169bcc:::benchmark_data_magna115_10867_object1582074:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498280 con 0 > -9329> 2018-07-11 07:50:30.723050 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75165480:::benchmark_data_magna115_10867_object1681434:0,end:5:75169bcc:::benchmark_data_magna115_10867_object1582074:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455680 con 0 > -9328> 2018-07-11 07:50:30.858597 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15255 conn(0x55d986c82800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.0 seq 233 0x55d9829e5000 osd_ping(ping e501 stamp 2018-07-11 07:50:32.088555) v4 > -9327> 2018-07-11 07:50:30.858603 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15255 conn(0x55d986d20000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.0 seq 233 0x55d986cd4c00 osd_ping(ping e501 stamp 2018-07-11 07:50:32.088555) v4 > -9326> 2018-07-11 07:50:30.858619 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.0 10.8.128.113:0/15255 233 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:32.088555) v4 ==== 2004+0+0 (336853658 0 0) 0x55d9829e5000 con 0x55d986c82800 > -9325> 2018-07-11 07:50:30.858626 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.0 10.8.128.113:0/15255 233 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:32.088555) v4 ==== 2004+0+0 (336853658 0 0) 0x55d986cd4c00 con 0x55d986d20000 > -9324> 2018-07-11 07:50:30.858633 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:32.088555) v4 -- 0x55d986d96600 con 0 > -9323> 2018-07-11 07:50:30.858664 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:32.088555) v4 -- 0x55d9b4388200 con 0 > -9322> 2018-07-11 07:50:30.932254 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 788 0x55d9bba53b00 rep_scrubmap(5.2e e501 from shard 2) v1 > -9321> 2018-07-11 07:50:30.932275 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 788 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12476 (1269249638 0 2054040113) 0x55d9bba53b00 con 0x55d986bf0800 > -9320> 2018-07-11 07:50:30.952893 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6805/15263 conn(0x55d986c41000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=31 cs=1 l=0). rx osd.5 seq 2780 0x55d986cd4c00 MOSDScrubReserve(6.15 REQUEST e501) v1 > -9319> 2018-07-11 07:50:30.952911 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.5 10.8.128.113:6805/15263 2780 ==== MOSDScrubReserve(6.15 REQUEST e501) v1 ==== 43+0+0 (3942742026 0 0) 0x55d986cd4c00 con 0x55d986c41000 > -9318> 2018-07-11 07:50:30.963208 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 782 0x55d9bba53d40 rep_scrubmap(5.2e e501 from shard 4) v1 > -9317> 2018-07-11 07:50:30.963228 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 782 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12476 (3686815630 0 2054040113) 0x55d9bba53d40 con 0x55d986bec000 > -9316> 2018-07-11 07:50:30.987915 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6805/15263 -- MOSDScrubReserve(6.15 REJECT e501) v1 -- 0x55d9829e4200 con 0 > -9315> 2018-07-11 07:50:30.987958 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -9314> 2018-07-11 07:50:30.987988 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -9313> 2018-07-11 07:50:30.988040 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -9312> 2018-07-11 07:50:30.988346 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6805/15263 conn(0x55d986c41000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=31 cs=1 l=0). rx osd.5 seq 2781 0x55d9829e4200 MOSDScrubReserve(6.15 RELEASE e501) v1 > -9311> 2018-07-11 07:50:30.988364 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.5 10.8.128.113:6805/15263 2781 ==== MOSDScrubReserve(6.15 RELEASE e501) v1 ==== 43+0+0 (605848072 0 0) 0x55d9829e4200 con 0x55d986c41000 > -9310> 2018-07-11 07:50:30.988513 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -9309> 2018-07-11 07:50:30.989568 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75169bcc:::benchmark_data_magna115_10867_object1582074:0,end:5:7516f1d4:::benchmark_data_magna115_10867_object3797886:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498000 con 0 > -9308> 2018-07-11 07:50:30.989613 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75169bcc:::benchmark_data_magna115_10867_object1582074:0,end:5:7516f1d4:::benchmark_data_magna115_10867_object3797886:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455b80 con 0 > -9307> 2018-07-11 07:50:31.189330 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 789 0x55d99e408000 rep_scrubmap(5.2e e501 from shard 2) v1 > -9306> 2018-07-11 07:50:31.189350 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 789 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12456 (1269249638 0 3595914464) 0x55d99e408000 con 0x55d986bf0800 > -9305> 2018-07-11 07:50:31.211417 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 783 0x55d99e408240 rep_scrubmap(5.2e e501 from shard 4) v1 > -9304> 2018-07-11 07:50:31.211437 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 783 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12456 (3686815630 0 3595914464) 0x55d99e408240 con 0x55d986bec000 > -9303> 2018-07-11 07:50:31.311040 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.114:0/15885 conn(0x55d986d7f800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.7 seq 238 0x55d9c9122a00 osd_ping(ping e501 stamp 2018-07-11 07:50:31.940902) v4 > -9302> 2018-07-11 07:50:31.311064 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.7 10.8.128.114:0/15885 238 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:31.940902) v4 ==== 2004+0+0 (1662347952 0 0) 0x55d9c9122a00 con 0x55d986d7f800 > -9301> 2018-07-11 07:50:31.311061 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.114:0/15885 conn(0x55d986d81000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.7 seq 238 0x55d9c9123a00 osd_ping(ping e501 stamp 2018-07-11 07:50:31.940902) v4 > -9300> 2018-07-11 07:50:31.311086 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:31.940902) v4 -- 0x55d986d96600 con 0 > -9299> 2018-07-11 07:50:31.311091 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.7 10.8.128.114:0/15885 238 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:31.940902) v4 ==== 2004+0+0 (1662347952 0 0) 0x55d9c9123a00 con 0x55d986d81000 > -9298> 2018-07-11 07:50:31.311122 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:31.940902) v4 -- 0x55d9c74e1800 con 0 > -9297> 2018-07-11 07:50:31.352382 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -9296> 2018-07-11 07:50:31.352405 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -9295> 2018-07-11 07:50:31.352463 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -9294> 2018-07-11 07:50:31.352912 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -9293> 2018-07-11 07:50:31.357668 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12677 conn(0x55d986db4800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.1 seq 238 0x55d9829e5000 osd_ping(ping e501 stamp 2018-07-11 07:50:31.404628) v4 > -9292> 2018-07-11 07:50:31.357689 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.1 10.8.128.106:0/12677 238 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:31.404628) v4 ==== 2004+0+0 (2381160901 0 0) 0x55d9829e5000 con 0x55d986db4800 > -9291> 2018-07-11 07:50:31.357683 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12677 conn(0x55d986df2000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.1 seq 238 0x55d9b4388200 osd_ping(ping e501 stamp 2018-07-11 07:50:31.404628) v4 > -9290> 2018-07-11 07:50:31.357709 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:31.404628) v4 -- 0x55d986d96600 con 0 > -9289> 2018-07-11 07:50:31.357715 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.1 10.8.128.106:0/12677 238 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:31.404628) v4 ==== 2004+0+0 (2381160901 0 0) 0x55d9b4388200 con 0x55d986df2000 > -9288> 2018-07-11 07:50:31.357727 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:31.404628) v4 -- 0x55d9c74e1800 con 0 > -9287> 2018-07-11 07:50:31.381051 7ffacbbd1700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.108:0/11967 conn(0x55d986d82800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.8 seq 215 0x55d9bfbde400 osd_ping(ping e501 stamp 2018-07-11 07:50:31.136380) v4 > -9286> 2018-07-11 07:50:31.381052 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.108:0/11967 conn(0x55d986db0000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.8 seq 215 0x55d9c9122a00 osd_ping(ping e501 stamp 2018-07-11 07:50:31.136380) v4 > -9285> 2018-07-11 07:50:31.381077 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 <== osd.8 10.8.128.108:0/11967 215 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:31.136380) v4 ==== 2004+0+0 (3584781579 0 0) 0x55d9bfbde400 con 0x55d986d82800 > -9284> 2018-07-11 07:50:31.381086 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:31.136380) v4 -- 0x55d986c22000 con 0 > -9283> 2018-07-11 07:50:31.381081 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.8 10.8.128.108:0/11967 215 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:31.136380) v4 ==== 2004+0+0 (3584781579 0 0) 0x55d9c9122a00 con 0x55d986db0000 > -9282> 2018-07-11 07:50:31.381106 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:31.136380) v4 -- 0x55d986d96600 con 0 > -9281> 2018-07-11 07:50:31.398428 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7516f1d4:::benchmark_data_magna115_10867_object3797886:0,end:5:75175c3f:::benchmark_data_magna115_10867_object3246374:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455900 con 0 > -9280> 2018-07-11 07:50:31.398467 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7516f1d4:::benchmark_data_magna115_10867_object3797886:0,end:5:75175c3f:::benchmark_data_magna115_10867_object3246374:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498780 con 0 > -9279> 2018-07-11 07:50:31.555070 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6805/15885 conn(0x55d986c7f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.7 seq 2756 0x55d9b4388200 MOSDScrubReserve(5.4 REQUEST e501) v1 > -9278> 2018-07-11 07:50:31.555100 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.7 10.8.128.114:6805/15885 2756 ==== MOSDScrubReserve(5.4 REQUEST e501) v1 ==== 43+0+0 (1753221512 0 0) 0x55d9b4388200 con 0x55d986c7f800 > -9277> 2018-07-11 07:50:31.555224 7ffab33df700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.114:6805/15885 -- MOSDScrubReserve(5.4 REJECT e501) v1 -- 0x55d986ca9a00 con 0 > -9276> 2018-07-11 07:50:31.555486 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6805/15885 conn(0x55d986c7f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.7 seq 2757 0x55d9c74e1800 MOSDScrubReserve(5.4 RELEASE e501) v1 > -9275> 2018-07-11 07:50:31.555501 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.7 10.8.128.114:6805/15885 2757 ==== MOSDScrubReserve(5.4 RELEASE e501) v1 ==== 43+0+0 (2812146570 0 0) 0x55d9c74e1800 con 0x55d986c7f800 > -9274> 2018-07-11 07:50:31.612828 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 790 0x55d99e408480 rep_scrubmap(5.2e e501 from shard 2) v1 > -9273> 2018-07-11 07:50:31.612850 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 790 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12478 (1269249638 0 4073947297) 0x55d99e408480 con 0x55d986bf0800 > -9272> 2018-07-11 07:50:31.615327 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 784 0x55d99e4086c0 rep_scrubmap(5.2e e501 from shard 4) v1 > -9271> 2018-07-11 07:50:31.615346 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 784 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12478 (3686815630 0 4073947297) 0x55d99e4086c0 con 0x55d986bec000 > -9270> 2018-07-11 07:50:32.301121 7ffab7be8700 10 monclient: tick > -9269> 2018-07-11 07:50:32.301147 7ffab7be8700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2018-07-11 07:50:02.301145) > -9268> 2018-07-11 07:50:32.346810 7ffab6be6700 1 -- 10.8.128.107:6800/13927 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55d9b3fdf8c0 con 0 > -9267> 2018-07-11 07:50:32.346879 7ffab6be6700 1 -- 10.8.128.107:6800/13927 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55d9bd9cc580 con 0 > -9266> 2018-07-11 07:50:32.541012 7ffacbbd1700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15263 conn(0x55d985d44000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.5 seq 217 0x55d986c23000 osd_ping(ping e501 stamp 2018-07-11 07:50:33.770965) v4 > -9265> 2018-07-11 07:50:32.541036 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 <== osd.5 10.8.128.113:0/15263 217 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:33.770965) v4 ==== 2004+0+0 (1769849811 0 0) 0x55d986c23000 con 0x55d985d44000 > -9264> 2018-07-11 07:50:32.541047 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:33.770965) v4 -- 0x55d986c22000 con 0 > -9263> 2018-07-11 07:50:32.541037 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15263 conn(0x55d985d45800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.5 seq 217 0x55d9829e5000 osd_ping(ping e501 stamp 2018-07-11 07:50:33.770965) v4 > -9262> 2018-07-11 07:50:32.541057 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.5 10.8.128.113:0/15263 217 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:33.770965) v4 ==== 2004+0+0 (1769849811 0 0) 0x55d9829e5000 con 0x55d985d45800 > -9261> 2018-07-11 07:50:32.541071 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:33.770965) v4 -- 0x55d986d96600 con 0 > -9260> 2018-07-11 07:50:32.707587 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 29] Generated table #36789: 3945470 keys, 77818049 bytes > -9259> 2018-07-11 07:50:32.707637 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295432707621, "cf_name": "default", "job": 29, "event": "table_file_creation", "file_number": 36789, "file_size": 77818049, "table_properties": {"data_size": 67111657, "index_size": 689512, "filter_size": 10015887, "raw_key_size": 113980270, "raw_average_key_size": 28, "raw_value_size": 13685253, "raw_average_value_size": 3, "num_data_blocks": 16479, "num_entries": 3945470, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "2794375", "kMergeOperands": "0"}} > -9258> 2018-07-11 07:50:32.815410 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -9257> 2018-07-11 07:50:32.815440 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -9256> 2018-07-11 07:50:32.815503 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -9255> 2018-07-11 07:50:32.815969 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -9254> 2018-07-11 07:50:32.831778 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67010,epoch:501/498,start:5:75175c3f:::benchmark_data_magna115_10867_object3246374:0,end:5:7517ad4e:::benchmark_data_magna115_10867_object3380567:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b14f0500 con 0 > -9253> 2018-07-11 07:50:32.831828 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67010,epoch:501/498,start:5:75175c3f:::benchmark_data_magna115_10867_object3246374:0,end:5:7517ad4e:::benchmark_data_magna115_10867_object3380567:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98696f180 con 0 > -9252> 2018-07-11 07:50:32.875462 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15287 conn(0x55d981240800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 243 0x55d9c9123a00 osd_ping(ping e501 stamp 2018-07-11 07:50:34.105505) v4 > -9251> 2018-07-11 07:50:32.875487 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.4 10.8.128.113:0/15287 243 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:34.105505) v4 ==== 2004+0+0 (34227141 0 0) 0x55d9c9123a00 con 0x55d981240800 > -9250> 2018-07-11 07:50:32.875484 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15287 conn(0x55d98123f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 243 0x55d9c9122a00 osd_ping(ping e501 stamp 2018-07-11 07:50:34.105505) v4 > -9249> 2018-07-11 07:50:32.875498 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:34.105505) v4 -- 0x55d9bdba5000 con 0 > -9248> 2018-07-11 07:50:32.875503 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.4 10.8.128.113:0/15287 243 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:34.105505) v4 ==== 2004+0+0 (34227141 0 0) 0x55d9c9122a00 con 0x55d98123f000 > -9247> 2018-07-11 07:50:32.875520 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:34.105505) v4 -- 0x55d986d96600 con 0 > -9246> 2018-07-11 07:50:33.014436 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12651 conn(0x55d986db3000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.2 seq 233 0x55d9bfbde400 osd_ping(ping e501 stamp 2018-07-11 07:50:33.061338) v4 > -9245> 2018-07-11 07:50:33.014459 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.2 10.8.128.106:0/12651 233 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:33.061338) v4 ==== 2004+0+0 (2581209754 0 0) 0x55d9bfbde400 con 0x55d986db3000 > -9244> 2018-07-11 07:50:33.014468 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:33.061338) v4 -- 0x55d986c22000 con 0 > -9243> 2018-07-11 07:50:33.014556 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12651 conn(0x55d986db1800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.2 seq 233 0x55d986ca9a00 osd_ping(ping e501 stamp 2018-07-11 07:50:33.061338) v4 > -9242> 2018-07-11 07:50:33.014575 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.2 10.8.128.106:0/12651 233 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:33.061338) v4 ==== 2004+0+0 (2581209754 0 0) 0x55d986ca9a00 con 0x55d986db1800 > -9241> 2018-07-11 07:50:33.014586 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:33.061338) v4 -- 0x55d9bdba5000 con 0 > -9240> 2018-07-11 07:50:33.031847 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 791 0x55d99e408900 rep_scrubmap(5.2e e501 from shard 2) v1 > -9239> 2018-07-11 07:50:33.031866 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 791 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12474 (1269249638 0 4195118787) 0x55d99e408900 con 0x55d986bf0800 > -9238> 2018-07-11 07:50:33.044284 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 785 0x55d99e408b40 rep_scrubmap(5.2e e501 from shard 4) v1 > -9237> 2018-07-11 07:50:33.044302 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 785 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12474 (3686815630 0 4195118787) 0x55d99e408b40 con 0x55d986bec000 > -9236> 2018-07-11 07:50:33.075015 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -9235> 2018-07-11 07:50:33.075038 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -9234> 2018-07-11 07:50:33.075096 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -9233> 2018-07-11 07:50:33.075556 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -9232> 2018-07-11 07:50:33.123589 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67893,epoch:501/498,start:5:7517ad4e:::benchmark_data_magna115_10867_object3380567:0,end:5:751818cc:::benchmark_data_magna115_10867_object450514:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98696e000 con 0 > -9231> 2018-07-11 07:50:33.123640 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67893,epoch:501/498,start:5:7517ad4e:::benchmark_data_magna115_10867_object3380567:0,end:5:751818cc:::benchmark_data_magna115_10867_object450514:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98696f900 con 0 > -9230> 2018-07-11 07:50:33.192652 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.115:0/13323 conn(0x55d986d7e000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 232 0x55d986c23000 osd_ping(ping e501 stamp 2018-07-11 07:50:33.537904) v4 > -9229> 2018-07-11 07:50:33.192674 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.6 10.8.128.115:0/13323 232 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:33.537904) v4 ==== 2004+0+0 (177612979 0 0) 0x55d986c23000 con 0x55d986d7e000 > -9228> 2018-07-11 07:50:33.192683 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:33.537904) v4 -- 0x55d986c22000 con 0 > -9227> 2018-07-11 07:50:33.192720 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.115:0/13323 conn(0x55d986d24800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 232 0x55d986ca9a00 osd_ping(ping e501 stamp 2018-07-11 07:50:33.537904) v4 > -9226> 2018-07-11 07:50:33.192746 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.6 10.8.128.115:0/13323 232 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:33.537904) v4 ==== 2004+0+0 (177612979 0 0) 0x55d986ca9a00 con 0x55d986d24800 > -9225> 2018-07-11 07:50:33.192757 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:33.537904) v4 -- 0x55d9bdba5000 con 0 > -9224> 2018-07-11 07:50:33.347864 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 792 0x55d99e408d80 rep_scrubmap(5.2e e501 from shard 2) v1 > -9223> 2018-07-11 07:50:33.347885 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 792 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12476 (1269249638 0 1895077917) 0x55d99e408d80 con 0x55d986bf0800 > -9222> 2018-07-11 07:50:33.366495 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 786 0x55d99e408fc0 rep_scrubmap(5.2e e501 from shard 4) v1 > -9221> 2018-07-11 07:50:33.366515 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 786 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12476 (3686815630 0 1895077917) 0x55d99e408fc0 con 0x55d986bec000 > -9220> 2018-07-11 07:50:33.452619 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -9219> 2018-07-11 07:50:33.452644 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -9218> 2018-07-11 07:50:33.452706 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -9217> 2018-07-11 07:50:33.453166 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -9216> 2018-07-11 07:50:33.495760 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:751818cc:::benchmark_data_magna115_10867_object450514:0,end:5:751867e9:::benchmark_data_magna115_10867_object3337367:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98696ef00 con 0 > -9215> 2018-07-11 07:50:33.495812 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:751818cc:::benchmark_data_magna115_10867_object450514:0,end:5:751867e9:::benchmark_data_magna115_10867_object3337367:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b14f0a00 con 0 > -9214> 2018-07-11 07:50:33.509707 7ffab1bdc700 5 osd.10 501 heartbeat: osd_stat(787 GB used, 143 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist []) > -9213> 2018-07-11 07:50:33.509721 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6802/15255 -- osd_ping(ping e501 stamp 2018-07-11 07:50:33.509719) v4 -- 0x55d9c9122000 con 0 > -9212> 2018-07-11 07:50:33.509758 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6803/15255 -- osd_ping(ping e501 stamp 2018-07-11 07:50:33.509719) v4 -- 0x55d9869bf200 con 0 > -9211> 2018-07-11 07:50:33.509775 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6806/12677 -- osd_ping(ping e501 stamp 2018-07-11 07:50:33.509719) v4 -- 0x55d9c3383600 con 0 > -9210> 2018-07-11 07:50:33.509789 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6807/12677 -- osd_ping(ping e501 stamp 2018-07-11 07:50:33.509719) v4 -- 0x55d9c4c1a400 con 0 > -9209> 2018-07-11 07:50:33.509803 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6808/12651 -- osd_ping(ping e501 stamp 2018-07-11 07:50:33.509719) v4 -- 0x55d9b4389400 con 0 > -9208> 2018-07-11 07:50:33.509814 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6810/12651 -- osd_ping(ping e501 stamp 2018-07-11 07:50:33.509719) v4 -- 0x55d9b9ac3c00 con 0 > -9207> 2018-07-11 07:50:33.509827 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6809/12650 -- osd_ping(ping e501 stamp 2018-07-11 07:50:33.509719) v4 -- 0x55d9b3db2800 con 0 > -9206> 2018-07-11 07:50:33.509844 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6811/12650 -- osd_ping(ping e501 stamp 2018-07-11 07:50:33.509719) v4 -- 0x55d9c908e200 con 0 > -9205> 2018-07-11 07:50:33.509857 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6810/15287 -- osd_ping(ping e501 stamp 2018-07-11 07:50:33.509719) v4 -- 0x55d9b3db3a00 con 0 > -9204> 2018-07-11 07:50:33.509871 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6811/15287 -- osd_ping(ping e501 stamp 2018-07-11 07:50:33.509719) v4 -- 0x55d986c22600 con 0 > -9203> 2018-07-11 07:50:33.509884 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6806/15263 -- osd_ping(ping e501 stamp 2018-07-11 07:50:33.509719) v4 -- 0x55d9c3382600 con 0 > -9202> 2018-07-11 07:50:33.509900 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6807/15263 -- osd_ping(ping e501 stamp 2018-07-11 07:50:33.509719) v4 -- 0x55d9c4c1a200 con 0 > -9201> 2018-07-11 07:50:33.509913 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.115:6802/13323 -- osd_ping(ping e501 stamp 2018-07-11 07:50:33.509719) v4 -- 0x55d986c23e00 con 0 > -9200> 2018-07-11 07:50:33.509926 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.115:6803/13323 -- osd_ping(ping e501 stamp 2018-07-11 07:50:33.509719) v4 -- 0x55d9829e5e00 con 0 > -9199> 2018-07-11 07:50:33.509942 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6806/15885 -- osd_ping(ping e501 stamp 2018-07-11 07:50:33.509719) v4 -- 0x55d9b2783e00 con 0 > -9198> 2018-07-11 07:50:33.509954 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6807/15885 -- osd_ping(ping e501 stamp 2018-07-11 07:50:33.509719) v4 -- 0x55d986d40000 con 0 > -9197> 2018-07-11 07:50:33.509967 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.108:6802/11967 -- osd_ping(ping e501 stamp 2018-07-11 07:50:33.509719) v4 -- 0x55d9b2783600 con 0 > -9196> 2018-07-11 07:50:33.509979 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.108:6803/11967 -- osd_ping(ping e501 stamp 2018-07-11 07:50:33.509719) v4 -- 0x55d9b40a5600 con 0 > -9195> 2018-07-11 07:50:33.509991 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.107:6806/12617 -- osd_ping(ping e501 stamp 2018-07-11 07:50:33.509719) v4 -- 0x55d9b60dea00 con 0 > -9194> 2018-07-11 07:50:33.510004 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.107:6807/12617 -- osd_ping(ping e501 stamp 2018-07-11 07:50:33.509719) v4 -- 0x55d982cdfc00 con 0 > -9193> 2018-07-11 07:50:33.510017 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6802/15919 -- osd_ping(ping e501 stamp 2018-07-11 07:50:33.509719) v4 -- 0x55d9c74e0200 con 0 > -9192> 2018-07-11 07:50:33.510029 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6803/15919 -- osd_ping(ping e501 stamp 2018-07-11 07:50:33.509719) v4 -- 0x55d9869d9600 con 0 > -9191> 2018-07-11 07:50:33.510181 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6807/12677 conn(0x55d986b59800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.1 seq 225 0x55d9c9123a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:33.509719) v4 > -9190> 2018-07-11 07:50:33.510201 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.1 10.8.128.106:6807/12677 225 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:33.509719) v4 ==== 2004+0+0 (4287292055 0 0) 0x55d9c9123a00 con 0x55d986b59800 > -9189> 2018-07-11 07:50:33.510194 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.107:6807/12617 conn(0x55d986bd3800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.9 seq 225 0x55d9bfbde400 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:33.509719) v4 > -9188> 2018-07-11 07:50:33.510212 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.9 10.8.128.107:6807/12617 225 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:33.509719) v4 ==== 2004+0+0 (4287292055 0 0) 0x55d9bfbde400 con 0x55d986bd3800 > -9187> 2018-07-11 07:50:33.510223 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.107:6806/12617 conn(0x55d986bd2000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.9 seq 225 0x55d9bcd72c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:33.509719) v4 > -9186> 2018-07-11 07:50:33.510244 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.9 10.8.128.107:6806/12617 225 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:33.509719) v4 ==== 2004+0+0 (4287292055 0 0) 0x55d9bcd72c00 con 0x55d986bd2000 > -9185> 2018-07-11 07:50:33.510275 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6809/12650 conn(0x55d986bcb000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 225 0x55d9bfbde400 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:33.509719) v4 > -9184> 2018-07-11 07:50:33.510290 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.3 10.8.128.106:6809/12650 225 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:33.509719) v4 ==== 2004+0+0 (4287292055 0 0) 0x55d9bfbde400 con 0x55d986bcb000 > -9183> 2018-07-11 07:50:33.510291 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6802/15255 conn(0x55d986bc8000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.0 seq 225 0x55d9c9123a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:33.509719) v4 > -9182> 2018-07-11 07:50:33.510308 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.0 10.8.128.113:6802/15255 225 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:33.509719) v4 ==== 2004+0+0 (4287292055 0 0) 0x55d9c9123a00 con 0x55d986bc8000 > -9181> 2018-07-11 07:50:33.510343 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6808/12651 conn(0x55d986b9a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.2 seq 225 0x55d9c9123a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:33.509719) v4 > -9180> 2018-07-11 07:50:33.510360 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.2 10.8.128.106:6808/12651 225 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:33.509719) v4 ==== 2004+0+0 (4287292055 0 0) 0x55d9c9123a00 con 0x55d986b9a000 > -9179> 2018-07-11 07:50:33.510352 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6803/15255 conn(0x55d986bc9800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.0 seq 225 0x55d9bcd72c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:33.509719) v4 > -9178> 2018-07-11 07:50:33.510359 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6806/12677 conn(0x55d986b58000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.1 seq 225 0x55d9bfbde400 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:33.509719) v4 > -9177> 2018-07-11 07:50:33.510378 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.0 10.8.128.113:6803/15255 225 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:33.509719) v4 ==== 2004+0+0 (4287292055 0 0) 0x55d9bcd72c00 con 0x55d986bc9800 > -9176> 2018-07-11 07:50:33.510393 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6810/15287 conn(0x55d986baf000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 225 0x55d9c9123a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:33.509719) v4 > -9175> 2018-07-11 07:50:33.510383 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.1 10.8.128.106:6806/12677 225 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:33.509719) v4 ==== 2004+0+0 (4287292055 0 0) 0x55d9bfbde400 con 0x55d986b58000 > -9174> 2018-07-11 07:50:33.510408 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.4 10.8.128.113:6810/15287 225 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:33.509719) v4 ==== 2004+0+0 (4287292055 0 0) 0x55d9c9123a00 con 0x55d986baf000 > -9173> 2018-07-11 07:50:33.510433 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6806/15263 conn(0x55d986b9d000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.5 seq 225 0x55d9bfbde400 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:33.509719) v4 > -9172> 2018-07-11 07:50:33.510432 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6810/12651 conn(0x55d986b9b800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.2 seq 225 0x55d9bcd72c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:33.509719) v4 > -9171> 2018-07-11 07:50:33.510445 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.5 10.8.128.113:6806/15263 225 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:33.509719) v4 ==== 2004+0+0 (4287292055 0 0) 0x55d9bfbde400 con 0x55d986b9d000 > -9170> 2018-07-11 07:50:33.510447 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.2 10.8.128.106:6810/12651 225 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:33.509719) v4 ==== 2004+0+0 (4287292055 0 0) 0x55d9bcd72c00 con 0x55d986b9b800 > -9169> 2018-07-11 07:50:33.510458 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6811/12650 conn(0x55d986bcc800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 225 0x55d9c9123a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:33.509719) v4 > -9168> 2018-07-11 07:50:33.510471 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.3 10.8.128.106:6811/12650 225 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:33.509719) v4 ==== 2004+0+0 (4287292055 0 0) 0x55d9c9123a00 con 0x55d986bcc800 > -9167> 2018-07-11 07:50:33.510479 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.115:6803/13323 conn(0x55d986bc2800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 225 0x55d9bfbde400 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:33.509719) v4 > -9166> 2018-07-11 07:50:33.510491 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.6 10.8.128.115:6803/13323 225 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:33.509719) v4 ==== 2004+0+0 (4287292055 0 0) 0x55d9bfbde400 con 0x55d986bc2800 > -9165> 2018-07-11 07:50:33.510490 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6811/15287 conn(0x55d986bb0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 225 0x55d9bcd72c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:33.509719) v4 > -9164> 2018-07-11 07:50:33.510505 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6807/15263 conn(0x55d986b9e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.5 seq 225 0x55d9c9123a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:33.509719) v4 > -9163> 2018-07-11 07:50:33.510508 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.4 10.8.128.113:6811/15287 225 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:33.509719) v4 ==== 2004+0+0 (4287292055 0 0) 0x55d9bcd72c00 con 0x55d986bb0800 > -9162> 2018-07-11 07:50:33.510517 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.5 10.8.128.113:6807/15263 225 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:33.509719) v4 ==== 2004+0+0 (4287292055 0 0) 0x55d9c9123a00 con 0x55d986b9e800 > -9161> 2018-07-11 07:50:33.510553 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.115:6802/13323 conn(0x55d986bc1000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 225 0x55d9bcd72c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:33.509719) v4 > -9160> 2018-07-11 07:50:33.510570 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.6 10.8.128.115:6802/13323 225 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:33.509719) v4 ==== 2004+0+0 (4287292055 0 0) 0x55d9bcd72c00 con 0x55d986bc1000 > -9159> 2018-07-11 07:50:33.510584 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6807/15885 conn(0x55d986bad800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.7 seq 225 0x55d9bfbde400 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:33.509719) v4 > -9158> 2018-07-11 07:50:33.510606 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.7 10.8.128.114:6807/15885 225 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:33.509719) v4 ==== 2004+0+0 (4287292055 0 0) 0x55d9bfbde400 con 0x55d986bad800 > -9157> 2018-07-11 07:50:33.510615 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.108:6803/11967 conn(0x55d986bbf800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.8 seq 225 0x55d9c9123a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:33.509719) v4 > -9156> 2018-07-11 07:50:33.510631 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.8 10.8.128.108:6803/11967 225 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:33.509719) v4 ==== 2004+0+0 (4287292055 0 0) 0x55d9c9123a00 con 0x55d986bbf800 > -9155> 2018-07-11 07:50:33.510627 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6806/15885 conn(0x55d986bac000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.7 seq 225 0x55d9bcd72c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:33.509719) v4 > -9154> 2018-07-11 07:50:33.510644 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.7 10.8.128.114:6806/15885 225 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:33.509719) v4 ==== 2004+0+0 (4287292055 0 0) 0x55d9bcd72c00 con 0x55d986bac000 > -9153> 2018-07-11 07:50:33.510663 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6803/15919 conn(0x55d986b5c800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.11 seq 225 0x55d9bfbde400 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:33.509719) v4 > -9152> 2018-07-11 07:50:33.510678 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.11 10.8.128.114:6803/15919 225 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:33.509719) v4 ==== 2004+0+0 (4287292055 0 0) 0x55d9bfbde400 con 0x55d986b5c800 > -9151> 2018-07-11 07:50:33.510697 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6802/15919 conn(0x55d986b5b000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.11 seq 225 0x55d9bcd72c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:33.509719) v4 > -9150> 2018-07-11 07:50:33.510714 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.11 10.8.128.114:6802/15919 225 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:33.509719) v4 ==== 2004+0+0 (4287292055 0 0) 0x55d9bcd72c00 con 0x55d986b5b000 > -9149> 2018-07-11 07:50:33.510717 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.108:6802/11967 conn(0x55d986bbe000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.8 seq 225 0x55d9bfbde400 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:33.509719) v4 > -9148> 2018-07-11 07:50:33.510733 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.8 10.8.128.108:6802/11967 225 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:33.509719) v4 ==== 2004+0+0 (4287292055 0 0) 0x55d9bfbde400 con 0x55d986bbe000 > -9147> 2018-07-11 07:50:33.641494 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15263 conn(0x55d985d45800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.5 seq 218 0x55d9bcd72c00 osd_ping(ping e501 stamp 2018-07-11 07:50:34.871488) v4 > -9146> 2018-07-11 07:50:33.641515 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.5 10.8.128.113:0/15263 218 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:34.871488) v4 ==== 2004+0+0 (2538305298 0 0) 0x55d9bcd72c00 con 0x55d985d45800 > -9145> 2018-07-11 07:50:33.641525 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:34.871488) v4 -- 0x55d9c9122a00 con 0 > -9144> 2018-07-11 07:50:33.641516 7ffacbbd1700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15263 conn(0x55d985d44000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.5 seq 218 0x55d9bfbde400 osd_ping(ping e501 stamp 2018-07-11 07:50:34.871488) v4 > -9143> 2018-07-11 07:50:33.641536 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 <== osd.5 10.8.128.113:0/15263 218 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:34.871488) v4 ==== 2004+0+0 (2538305298 0 0) 0x55d9bfbde400 con 0x55d985d44000 > -9142> 2018-07-11 07:50:33.641548 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:34.871488) v4 -- 0x55d9869d9600 con 0 > -9141> 2018-07-11 07:50:33.711541 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 793 0x55d99e409200 rep_scrubmap(5.2e e501 from shard 2) v1 > -9140> 2018-07-11 07:50:33.711560 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 793 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12458 (1269249638 0 1226034553) 0x55d99e409200 con 0x55d986bf0800 > -9139> 2018-07-11 07:50:33.715302 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 787 0x55d99e409440 rep_scrubmap(5.2e e501 from shard 4) v1 > -9138> 2018-07-11 07:50:33.715321 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 787 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12458 (3686815630 0 1226034553) 0x55d99e409440 con 0x55d986bec000 > -9137> 2018-07-11 07:50:33.759107 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15255 conn(0x55d986d20000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.0 seq 234 0x55d9c9123a00 osd_ping(ping e501 stamp 2018-07-11 07:50:34.988991) v4 > -9136> 2018-07-11 07:50:33.759110 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15255 conn(0x55d986c82800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.0 seq 234 0x55d9829e5000 osd_ping(ping e501 stamp 2018-07-11 07:50:34.988991) v4 > -9135> 2018-07-11 07:50:33.759129 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.0 10.8.128.113:0/15255 234 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:34.988991) v4 ==== 2004+0+0 (3700531135 0 0) 0x55d9c9123a00 con 0x55d986d20000 > -9134> 2018-07-11 07:50:33.759130 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.0 10.8.128.113:0/15255 234 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:34.988991) v4 ==== 2004+0+0 (3700531135 0 0) 0x55d9829e5000 con 0x55d986c82800 > -9133> 2018-07-11 07:50:33.759138 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:34.988991) v4 -- 0x55d9b40a5600 con 0 > -9132> 2018-07-11 07:50:33.759161 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:34.988991) v4 -- 0x55d9c9122a00 con 0 > -9131> 2018-07-11 07:50:33.812668 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -9130> 2018-07-11 07:50:33.812694 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -9129> 2018-07-11 07:50:33.812745 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -9128> 2018-07-11 07:50:33.813203 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -9127> 2018-07-11 07:50:33.832255 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:751867e9:::benchmark_data_magna115_10867_object3337367:0,end:5:7518bff9:::benchmark_data_magna115_10867_object1322051:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98696f680 con 0 > -9126> 2018-07-11 07:50:33.832294 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:751867e9:::benchmark_data_magna115_10867_object3337367:0,end:5:7518bff9:::benchmark_data_magna115_10867_object1322051:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498a00 con 0 > -9125> 2018-07-11 07:50:33.954583 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6801/15919 conn(0x55d986c3f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=29 cs=1 l=0). rx osd.11 seq 206 0x55d986c23000 MOSDScrubReserve(10.f REQUEST e501) v1 > -9124> 2018-07-11 07:50:33.954601 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.11 10.8.128.114:6801/15919 206 ==== MOSDScrubReserve(10.f REQUEST e501) v1 ==== 43+0+0 (2892966580 0 0) 0x55d986c23000 con 0x55d986c3f800 > -9123> 2018-07-11 07:50:33.954677 7ffab53e3700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.114:6801/15919 -- MOSDScrubReserve(10.f REJECT e501) v1 -- 0x55d9bf805800 con 0 > -9122> 2018-07-11 07:50:33.954998 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6801/15919 conn(0x55d986c3f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=29 cs=1 l=0). rx osd.11 seq 207 0x55d9bf805800 MOSDScrubReserve(10.f RELEASE e501) v1 > -9121> 2018-07-11 07:50:33.955013 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.11 10.8.128.114:6801/15919 207 ==== MOSDScrubReserve(10.f RELEASE e501) v1 ==== 43+0+0 (1668468918 0 0) 0x55d9bf805800 con 0x55d986c3f800 > -9120> 2018-07-11 07:50:34.010231 7ffab1bdc700 5 osd.10 501 heartbeat: osd_stat(787 GB used, 143 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist []) > -9119> 2018-07-11 07:50:34.010243 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6802/15255 -- osd_ping(ping e501 stamp 2018-07-11 07:50:34.010242) v4 -- 0x55d986d41800 con 0 > -9118> 2018-07-11 07:50:34.010288 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6803/15255 -- osd_ping(ping e501 stamp 2018-07-11 07:50:34.010242) v4 -- 0x55d9869bfa00 con 0 > -9117> 2018-07-11 07:50:34.010302 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6806/12677 -- osd_ping(ping e501 stamp 2018-07-11 07:50:34.010242) v4 -- 0x55d986d96400 con 0 > -9116> 2018-07-11 07:50:34.010314 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6807/12677 -- osd_ping(ping e501 stamp 2018-07-11 07:50:34.010242) v4 -- 0x55d9869bea00 con 0 > -9115> 2018-07-11 07:50:34.010338 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6808/12651 -- osd_ping(ping e501 stamp 2018-07-11 07:50:34.010242) v4 -- 0x55d9b67d8200 con 0 > -9114> 2018-07-11 07:50:34.010347 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6810/12651 -- osd_ping(ping e501 stamp 2018-07-11 07:50:34.010242) v4 -- 0x55d9829e4400 con 0 > -9113> 2018-07-11 07:50:34.010359 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6809/12650 -- osd_ping(ping e501 stamp 2018-07-11 07:50:34.010242) v4 -- 0x55d986d33e00 con 0 > -9112> 2018-07-11 07:50:34.010371 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6811/12650 -- osd_ping(ping e501 stamp 2018-07-11 07:50:34.010242) v4 -- 0x55d9bfbde200 con 0 > -9111> 2018-07-11 07:50:34.010383 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6810/15287 -- osd_ping(ping e501 stamp 2018-07-11 07:50:34.010242) v4 -- 0x55d986d33000 con 0 > -9110> 2018-07-11 07:50:34.010400 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6811/15287 -- osd_ping(ping e501 stamp 2018-07-11 07:50:34.010242) v4 -- 0x55d982cdea00 con 0 > -9109> 2018-07-11 07:50:34.010412 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6806/15263 -- osd_ping(ping e501 stamp 2018-07-11 07:50:34.010242) v4 -- 0x55d986cd5c00 con 0 > -9108> 2018-07-11 07:50:34.010425 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6807/15263 -- osd_ping(ping e501 stamp 2018-07-11 07:50:34.010242) v4 -- 0x55d986d4fe00 con 0 > -9107> 2018-07-11 07:50:34.010437 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.115:6802/13323 -- osd_ping(ping e501 stamp 2018-07-11 07:50:34.010242) v4 -- 0x55d986d4e600 con 0 > -9106> 2018-07-11 07:50:34.010454 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.115:6803/13323 -- osd_ping(ping e501 stamp 2018-07-11 07:50:34.010242) v4 -- 0x55d986cd6a00 con 0 > -9105> 2018-07-11 07:50:34.010468 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6806/15885 -- osd_ping(ping e501 stamp 2018-07-11 07:50:34.010242) v4 -- 0x55d980457a00 con 0 > -9104> 2018-07-11 07:50:34.010481 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6807/15885 -- osd_ping(ping e501 stamp 2018-07-11 07:50:34.010242) v4 -- 0x55d986b6dc00 con 0 > -9103> 2018-07-11 07:50:34.010495 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.108:6802/11967 -- osd_ping(ping e501 stamp 2018-07-11 07:50:34.010242) v4 -- 0x55d986cd5200 con 0 > -9102> 2018-07-11 07:50:34.010505 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.108:6803/11967 -- osd_ping(ping e501 stamp 2018-07-11 07:50:34.010242) v4 -- 0x55d9869bf400 con 0 > -9101> 2018-07-11 07:50:34.010518 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.107:6806/12617 -- osd_ping(ping e501 stamp 2018-07-11 07:50:34.010242) v4 -- 0x55d9b67d9600 con 0 > -9100> 2018-07-11 07:50:34.010531 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.107:6807/12617 -- osd_ping(ping e501 stamp 2018-07-11 07:50:34.010242) v4 -- 0x55d986d97e00 con 0 > -9099> 2018-07-11 07:50:34.010542 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6802/15919 -- osd_ping(ping e501 stamp 2018-07-11 07:50:34.010242) v4 -- 0x55d986d97000 con 0 > -9098> 2018-07-11 07:50:34.010554 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6803/15919 -- osd_ping(ping e501 stamp 2018-07-11 07:50:34.010242) v4 -- 0x55d9c3383c00 con 0 > -9097> 2018-07-11 07:50:34.010724 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.107:6806/12617 conn(0x55d986bd2000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.9 seq 226 0x55d9829e5000 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:34.010242) v4 > -9096> 2018-07-11 07:50:34.010746 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.9 10.8.128.107:6806/12617 226 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:34.010242) v4 ==== 2004+0+0 (3807434656 0 0) 0x55d9829e5000 con 0x55d986bd2000 > -9095> 2018-07-11 07:50:34.010740 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.107:6807/12617 conn(0x55d986bd3800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.9 seq 226 0x55d9869d9600 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:34.010242) v4 > -9094> 2018-07-11 07:50:34.010774 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6808/12651 conn(0x55d986b9a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.2 seq 226 0x55d986ca9a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:34.010242) v4 > -9093> 2018-07-11 07:50:34.010784 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.9 10.8.128.107:6807/12617 226 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:34.010242) v4 ==== 2004+0+0 (3807434656 0 0) 0x55d9869d9600 con 0x55d986bd3800 > -9092> 2018-07-11 07:50:34.010795 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.2 10.8.128.106:6808/12651 226 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:34.010242) v4 ==== 2004+0+0 (3807434656 0 0) 0x55d986ca9a00 con 0x55d986b9a000 > -9091> 2018-07-11 07:50:34.010837 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6809/12650 conn(0x55d986bcb000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 226 0x55d9869d9600 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:34.010242) v4 > -9090> 2018-07-11 07:50:34.010844 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6810/12651 conn(0x55d986b9b800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.2 seq 226 0x55d9829e5000 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:34.010242) v4 > -9089> 2018-07-11 07:50:34.010863 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.2 10.8.128.106:6810/12651 226 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:34.010242) v4 ==== 2004+0+0 (3807434656 0 0) 0x55d9829e5000 con 0x55d986b9b800 > -9088> 2018-07-11 07:50:34.010858 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6807/12677 conn(0x55d986b59800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.1 seq 226 0x55d986ca9a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:34.010242) v4 > -9087> 2018-07-11 07:50:34.010877 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.1 10.8.128.106:6807/12677 226 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:34.010242) v4 ==== 2004+0+0 (3807434656 0 0) 0x55d986ca9a00 con 0x55d986b59800 > -9086> 2018-07-11 07:50:34.010882 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.3 10.8.128.106:6809/12650 226 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:34.010242) v4 ==== 2004+0+0 (3807434656 0 0) 0x55d9869d9600 con 0x55d986bcb000 > -9085> 2018-07-11 07:50:34.010914 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6802/15255 conn(0x55d986bc8000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.0 seq 226 0x55d986ca9a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:34.010242) v4 > -9084> 2018-07-11 07:50:34.010928 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6806/12677 conn(0x55d986b58000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.1 seq 226 0x55d9869d9600 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:34.010242) v4 > -9083> 2018-07-11 07:50:34.010927 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.0 10.8.128.113:6802/15255 226 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:34.010242) v4 ==== 2004+0+0 (3807434656 0 0) 0x55d986ca9a00 con 0x55d986bc8000 > -9082> 2018-07-11 07:50:34.010967 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.1 10.8.128.106:6806/12677 226 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:34.010242) v4 ==== 2004+0+0 (3807434656 0 0) 0x55d9869d9600 con 0x55d986b58000 > -9081> 2018-07-11 07:50:34.010970 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6803/15255 conn(0x55d986bc9800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.0 seq 226 0x55d9829e5000 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:34.010242) v4 > -9080> 2018-07-11 07:50:34.010990 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.0 10.8.128.113:6803/15255 226 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:34.010242) v4 ==== 2004+0+0 (3807434656 0 0) 0x55d9829e5000 con 0x55d986bc9800 > -9079> 2018-07-11 07:50:34.011004 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6810/15287 conn(0x55d986baf000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 226 0x55d986ca9a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:34.010242) v4 > -9078> 2018-07-11 07:50:34.011017 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.4 10.8.128.113:6810/15287 226 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:34.010242) v4 ==== 2004+0+0 (3807434656 0 0) 0x55d986ca9a00 con 0x55d986baf000 > -9077> 2018-07-11 07:50:34.011023 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6811/15287 conn(0x55d986bb0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 226 0x55d9829e5000 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:34.010242) v4 > -9076> 2018-07-11 07:50:34.011026 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.115:6803/13323 conn(0x55d986bc2800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 226 0x55d9869d9600 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:34.010242) v4 > -9075> 2018-07-11 07:50:34.011044 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.6 10.8.128.115:6803/13323 226 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:34.010242) v4 ==== 2004+0+0 (3807434656 0 0) 0x55d9869d9600 con 0x55d986bc2800 > -9074> 2018-07-11 07:50:34.011046 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6811/12650 conn(0x55d986bcc800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 226 0x55d986ca9a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:34.010242) v4 > -9073> 2018-07-11 07:50:34.011035 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.4 10.8.128.113:6811/15287 226 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:34.010242) v4 ==== 2004+0+0 (3807434656 0 0) 0x55d9829e5000 con 0x55d986bb0800 > -9072> 2018-07-11 07:50:34.011058 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.3 10.8.128.106:6811/12650 226 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:34.010242) v4 ==== 2004+0+0 (3807434656 0 0) 0x55d986ca9a00 con 0x55d986bcc800 > -9071> 2018-07-11 07:50:34.011087 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6807/15263 conn(0x55d986b9e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.5 seq 226 0x55d986ca9a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:34.010242) v4 > -9070> 2018-07-11 07:50:34.011092 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6806/15263 conn(0x55d986b9d000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.5 seq 226 0x55d9869d9600 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:34.010242) v4 > -9069> 2018-07-11 07:50:34.011094 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.115:6802/13323 conn(0x55d986bc1000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 226 0x55d9829e5000 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:34.010242) v4 > -9068> 2018-07-11 07:50:34.011098 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.5 10.8.128.113:6807/15263 226 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:34.010242) v4 ==== 2004+0+0 (3807434656 0 0) 0x55d986ca9a00 con 0x55d986b9e800 > -9067> 2018-07-11 07:50:34.011114 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.6 10.8.128.115:6802/13323 226 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:34.010242) v4 ==== 2004+0+0 (3807434656 0 0) 0x55d9829e5000 con 0x55d986bc1000 > -9066> 2018-07-11 07:50:34.011112 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.5 10.8.128.113:6806/15263 226 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:34.010242) v4 ==== 2004+0+0 (3807434656 0 0) 0x55d9869d9600 con 0x55d986b9d000 > -9065> 2018-07-11 07:50:34.011165 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6806/15885 conn(0x55d986bac000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.7 seq 226 0x55d9829e5000 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:34.010242) v4 > -9064> 2018-07-11 07:50:34.011178 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.7 10.8.128.114:6806/15885 226 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:34.010242) v4 ==== 2004+0+0 (3807434656 0 0) 0x55d9829e5000 con 0x55d986bac000 > -9063> 2018-07-11 07:50:34.011174 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.108:6802/11967 conn(0x55d986bbe000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.8 seq 226 0x55d9869d9600 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:34.010242) v4 > -9062> 2018-07-11 07:50:34.011197 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.8 10.8.128.108:6802/11967 226 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:34.010242) v4 ==== 2004+0+0 (3807434656 0 0) 0x55d9869d9600 con 0x55d986bbe000 > -9061> 2018-07-11 07:50:34.011214 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6802/15919 conn(0x55d986b5b000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.11 seq 226 0x55d9829e5000 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:34.010242) v4 > -9060> 2018-07-11 07:50:34.011214 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.108:6803/11967 conn(0x55d986bbf800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.8 seq 226 0x55d986ca9a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:34.010242) v4 > -9059> 2018-07-11 07:50:34.011229 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.11 10.8.128.114:6802/15919 226 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:34.010242) v4 ==== 2004+0+0 (3807434656 0 0) 0x55d9829e5000 con 0x55d986b5b000 > -9058> 2018-07-11 07:50:34.011232 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.8 10.8.128.108:6803/11967 226 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:34.010242) v4 ==== 2004+0+0 (3807434656 0 0) 0x55d986ca9a00 con 0x55d986bbf800 > -9057> 2018-07-11 07:50:34.011238 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6807/15885 conn(0x55d986bad800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.7 seq 226 0x55d9869d9600 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:34.010242) v4 > -9056> 2018-07-11 07:50:34.011262 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.7 10.8.128.114:6807/15885 226 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:34.010242) v4 ==== 2004+0+0 (3807434656 0 0) 0x55d9869d9600 con 0x55d986bad800 > -9055> 2018-07-11 07:50:34.011306 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6803/15919 conn(0x55d986b5c800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.11 seq 226 0x55d9869d9600 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:34.010242) v4 > -9054> 2018-07-11 07:50:34.011320 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.11 10.8.128.114:6803/15919 226 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:34.010242) v4 ==== 2004+0+0 (3807434656 0 0) 0x55d9869d9600 con 0x55d986b5c800 > -9053> 2018-07-11 07:50:34.037899 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 794 0x55d99e409680 rep_scrubmap(5.2e e501 from shard 2) v1 > -9052> 2018-07-11 07:50:34.037918 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 794 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12476 (1269249638 0 3231929692) 0x55d99e409680 con 0x55d986bf0800 > -9051> 2018-07-11 07:50:34.069153 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 788 0x55d99e4098c0 rep_scrubmap(5.2e e501 from shard 4) v1 > -9050> 2018-07-11 07:50:34.069173 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 788 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12476 (3686815630 0 3231929692) 0x55d99e4098c0 con 0x55d986bec000 > -9049> 2018-07-11 07:50:34.099871 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -9048> 2018-07-11 07:50:34.099898 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -9047> 2018-07-11 07:50:34.099980 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -9046> 2018-07-11 07:50:34.100718 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -9045> 2018-07-11 07:50:34.139099 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7518bff9:::benchmark_data_magna115_10867_object1322051:0,end:5:7518fafd:::benchmark_data_magna115_10867_object2958320:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98696ec80 con 0 > -9044> 2018-07-11 07:50:34.139151 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7518bff9:::benchmark_data_magna115_10867_object1322051:0,end:5:7518fafd:::benchmark_data_magna115_10867_object2958320:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498f00 con 0 > -9043> 2018-07-11 07:50:34.339030 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 795 0x55d99e409b00 rep_scrubmap(5.2e e501 from shard 2) v1 > -9042> 2018-07-11 07:50:34.339052 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 795 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12476 (1269249638 0 3048438802) 0x55d99e409b00 con 0x55d986bf0800 > -9041> 2018-07-11 07:50:34.344239 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 789 0x55d99e409d40 rep_scrubmap(5.2e e501 from shard 4) v1 > -9040> 2018-07-11 07:50:34.344272 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 789 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12476 (3686815630 0 3048438802) 0x55d99e409d40 con 0x55d986bec000 > -9039> 2018-07-11 07:50:34.355977 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -9038> 2018-07-11 07:50:34.356003 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -9037> 2018-07-11 07:50:34.356085 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -9036> 2018-07-11 07:50:34.356844 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -9035> 2018-07-11 07:50:34.415771 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7518fafd:::benchmark_data_magna115_10867_object2958320:0,end:5:751951f0:::benchmark_data_magna115_10867_object2635569:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1499400 con 0 > -9034> 2018-07-11 07:50:34.415823 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7518fafd:::benchmark_data_magna115_10867_object2958320:0,end:5:751951f0:::benchmark_data_magna115_10867_object2635569:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1499180 con 0 > -9033> 2018-07-11 07:50:34.536624 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.107:0/12617 conn(0x55d98123d800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 219 0x55d9869d9600 osd_ping(ping e501 stamp 2018-07-11 07:50:34.536285) v4 > -9032> 2018-07-11 07:50:34.536625 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.107:0/12617 conn(0x55d98123c000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 219 0x55d986ca9a00 osd_ping(ping e501 stamp 2018-07-11 07:50:34.536285) v4 > -9031> 2018-07-11 07:50:34.536643 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.9 10.8.128.107:0/12617 219 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:34.536285) v4 ==== 2004+0+0 (860063880 0 0) 0x55d9869d9600 con 0x55d98123d800 > -9030> 2018-07-11 07:50:34.536650 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.9 10.8.128.107:0/12617 219 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:34.536285) v4 ==== 2004+0+0 (860063880 0 0) 0x55d986ca9a00 con 0x55d98123c000 > -9029> 2018-07-11 07:50:34.536662 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:34.536285) v4 -- 0x55d9c3383c00 con 0 > -9028> 2018-07-11 07:50:34.536698 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:34.536285) v4 -- 0x55d9869bf400 con 0 > -9027> 2018-07-11 07:50:34.555406 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6805/15885 conn(0x55d986c7f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.7 seq 2758 0x55d9c9123a00 MOSDScrubReserve(5.4 REQUEST e501) v1 > -9026> 2018-07-11 07:50:34.555423 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.7 10.8.128.114:6805/15885 2758 ==== MOSDScrubReserve(5.4 REQUEST e501) v1 ==== 43+0+0 (1753221512 0 0) 0x55d9c9123a00 con 0x55d986c7f800 > -9025> 2018-07-11 07:50:34.555544 7ffab33df700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.114:6805/15885 -- MOSDScrubReserve(5.4 REJECT e501) v1 -- 0x55d9c74e1800 con 0 > -9024> 2018-07-11 07:50:34.555835 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6805/15885 conn(0x55d986c7f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.7 seq 2759 0x55d9c74e1800 MOSDScrubReserve(5.4 RELEASE e501) v1 > -9023> 2018-07-11 07:50:34.555850 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.7 10.8.128.114:6805/15885 2759 ==== MOSDScrubReserve(5.4 RELEASE e501) v1 ==== 43+0+0 (2812146570 0 0) 0x55d9c74e1800 con 0x55d986c7f800 > -9022> 2018-07-11 07:50:34.644534 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 790 0x55d989370000 rep_scrubmap(5.2e e501 from shard 4) v1 > -9021> 2018-07-11 07:50:34.644553 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 790 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12472 (3686815630 0 1179649770) 0x55d989370000 con 0x55d986bec000 > -9020> 2018-07-11 07:50:34.649873 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 796 0x55d989370240 rep_scrubmap(5.2e e501 from shard 2) v1 > -9019> 2018-07-11 07:50:34.649893 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 796 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12472 (1269249638 0 1179649770) 0x55d989370240 con 0x55d986bf0800 > -9018> 2018-07-11 07:50:34.672136 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -9017> 2018-07-11 07:50:34.672160 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -9016> 2018-07-11 07:50:34.672207 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -9015> 2018-07-11 07:50:34.672661 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -9014> 2018-07-11 07:50:34.700531 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67444,epoch:501/498,start:5:751951f0:::benchmark_data_magna115_10867_object2635569:0,end:5:7519a731:::benchmark_data_magna115_10867_object3602026:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498c80 con 0 > -9013> 2018-07-11 07:50:34.700565 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67444,epoch:501/498,start:5:751951f0:::benchmark_data_magna115_10867_object2635569:0,end:5:7519a731:::benchmark_data_magna115_10867_object3602026:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498500 con 0 > -9012> 2018-07-11 07:50:34.912968 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 797 0x55d989370480 rep_scrubmap(5.2e e501 from shard 2) v1 > -9011> 2018-07-11 07:50:34.912993 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 797 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12478 (1269249638 0 136907432) 0x55d989370480 con 0x55d986bf0800 > -9010> 2018-07-11 07:50:34.924327 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 791 0x55d9893706c0 rep_scrubmap(5.2e e501 from shard 4) v1 > -9009> 2018-07-11 07:50:34.924346 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 791 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12478 (3686815630 0 136907432) 0x55d9893706c0 con 0x55d986bec000 > -9008> 2018-07-11 07:50:34.953426 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6805/15263 conn(0x55d986c41000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=31 cs=1 l=0). rx osd.5 seq 2782 0x55d9869bf400 MOSDScrubReserve(6.15 REQUEST e501) v1 > -9007> 2018-07-11 07:50:34.953444 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.5 10.8.128.113:6805/15263 2782 ==== MOSDScrubReserve(6.15 REQUEST e501) v1 ==== 43+0+0 (3942742026 0 0) 0x55d9869bf400 con 0x55d986c41000 > -9006> 2018-07-11 07:50:35.050150 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6805/15263 -- MOSDScrubReserve(6.15 REJECT e501) v1 -- 0x55d9829e4200 con 0 > -9005> 2018-07-11 07:50:35.050214 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -9004> 2018-07-11 07:50:35.050238 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -9003> 2018-07-11 07:50:35.050339 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -9002> 2018-07-11 07:50:35.050738 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6805/15263 conn(0x55d986c41000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=31 cs=1 l=0). rx osd.5 seq 2783 0x55d9829e4200 MOSDScrubReserve(6.15 RELEASE e501) v1 > -9001> 2018-07-11 07:50:35.050759 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.5 10.8.128.113:6805/15263 2783 ==== MOSDScrubReserve(6.15 RELEASE e501) v1 ==== 43+0+0 (605848072 0 0) 0x55d9829e4200 con 0x55d986c41000 > -9000> 2018-07-11 07:50:35.051191 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -8999> 2018-07-11 07:50:35.074311 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7519a731:::benchmark_data_magna115_10867_object3602026:0,end:5:751a176e:::benchmark_data_magna115_10867_object3051900:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1499680 con 0 > -8998> 2018-07-11 07:50:35.074351 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7519a731:::benchmark_data_magna115_10867_object3602026:0,end:5:751a176e:::benchmark_data_magna115_10867_object3051900:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1499b80 con 0 > -8997> 2018-07-11 07:50:35.280396 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 798 0x55d989370900 rep_scrubmap(5.2e e501 from shard 2) v1 > -8996> 2018-07-11 07:50:35.280415 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 798 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12478 (1269249638 0 3485719584) 0x55d989370900 con 0x55d986bf0800 > -8995> 2018-07-11 07:50:35.296077 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 792 0x55d989370b40 rep_scrubmap(5.2e e501 from shard 4) v1 > -8994> 2018-07-11 07:50:35.296095 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 792 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12478 (3686815630 0 3485719584) 0x55d989370b40 con 0x55d986bec000 > -8993> 2018-07-11 07:50:35.315077 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12651 conn(0x55d986db3000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.2 seq 234 0x55d986c23400 osd_ping(ping e501 stamp 2018-07-11 07:50:35.361917) v4 > -8992> 2018-07-11 07:50:35.315101 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.2 10.8.128.106:0/12651 234 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:35.361917) v4 ==== 2004+0+0 (2888761078 0 0) 0x55d986c23400 con 0x55d986db3000 > -8991> 2018-07-11 07:50:35.315112 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:35.361917) v4 -- 0x55d9c3383c00 con 0 > -8990> 2018-07-11 07:50:35.315214 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12651 conn(0x55d986db1800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.2 seq 234 0x55d986d4fe00 osd_ping(ping e501 stamp 2018-07-11 07:50:35.361917) v4 > -8989> 2018-07-11 07:50:35.315233 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.2 10.8.128.106:0/12651 234 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:35.361917) v4 ==== 2004+0+0 (2888761078 0 0) 0x55d986d4fe00 con 0x55d986db1800 > -8988> 2018-07-11 07:50:35.315244 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:35.361917) v4 -- 0x55d986d33000 con 0 > -8987> 2018-07-11 07:50:35.418180 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -8986> 2018-07-11 07:50:35.418205 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -8985> 2018-07-11 07:50:35.418277 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -8984> 2018-07-11 07:50:35.418741 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -8983> 2018-07-11 07:50:35.419787 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:751a176e:::benchmark_data_magna115_10867_object3051900:0,end:5:751a81fd:::benchmark_data_magna115_10867_object1361509:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1499900 con 0 > -8982> 2018-07-11 07:50:35.419834 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:751a176e:::benchmark_data_magna115_10867_object3051900:0,end:5:751a81fd:::benchmark_data_magna115_10867_object1361509:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b14f0280 con 0 > -8981> 2018-07-11 07:50:35.493107 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.115:0/13323 conn(0x55d986d24800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 233 0x55d986d4fe00 osd_ping(ping e501 stamp 2018-07-11 07:50:35.838336) v4 > -8980> 2018-07-11 07:50:35.493114 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.115:0/13323 conn(0x55d986d7e000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 233 0x55d9869d9600 osd_ping(ping e501 stamp 2018-07-11 07:50:35.838336) v4 > -8979> 2018-07-11 07:50:35.493137 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.6 10.8.128.115:0/13323 233 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:35.838336) v4 ==== 2004+0+0 (2183960074 0 0) 0x55d986d4fe00 con 0x55d986d24800 > -8978> 2018-07-11 07:50:35.493142 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.6 10.8.128.115:0/13323 233 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:35.838336) v4 ==== 2004+0+0 (2183960074 0 0) 0x55d9869d9600 con 0x55d986d7e000 > -8977> 2018-07-11 07:50:35.493149 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:35.838336) v4 -- 0x55d986d33000 con 0 > -8976> 2018-07-11 07:50:35.493172 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:35.838336) v4 -- 0x55d9c3383c00 con 0 > -8975> 2018-07-11 07:50:35.615658 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 799 0x55d989370d80 rep_scrubmap(5.2e e501 from shard 2) v1 > -8974> 2018-07-11 07:50:35.615678 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 799 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12486 (1269249638 0 1972947606) 0x55d989370d80 con 0x55d986bf0800 > -8973> 2018-07-11 07:50:35.640334 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 793 0x55d989370fc0 rep_scrubmap(5.2e e501 from shard 4) v1 > -8972> 2018-07-11 07:50:35.640355 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 793 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12486 (3686815630 0 1972947606) 0x55d989370fc0 con 0x55d986bec000 > -8971> 2018-07-11 07:50:35.688871 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -8970> 2018-07-11 07:50:35.688896 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -8969> 2018-07-11 07:50:35.688957 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -8968> 2018-07-11 07:50:35.689430 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -8967> 2018-07-11 07:50:35.707721 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67011,epoch:501/498,start:5:751a81fd:::benchmark_data_magna115_10867_object1361509:0,end:5:751ad3d9:::benchmark_data_magna115_10867_object3353623:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98696f400 con 0 > -8966> 2018-07-11 07:50:35.707770 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67011,epoch:501/498,start:5:751a81fd:::benchmark_data_magna115_10867_object1361509:0,end:5:751ad3d9:::benchmark_data_magna115_10867_object3353623:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b14f0000 con 0 > -8965> 2018-07-11 07:50:35.775967 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15287 conn(0x55d981240800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 244 0x55d986ca9a00 osd_ping(ping e501 stamp 2018-07-11 07:50:37.005996) v4 > -8964> 2018-07-11 07:50:35.775996 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.4 10.8.128.113:0/15287 244 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:37.005996) v4 ==== 2004+0+0 (3206071076 0 0) 0x55d986ca9a00 con 0x55d981240800 > -8963> 2018-07-11 07:50:35.776015 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:37.005996) v4 -- 0x55d986d33000 con 0 > -8962> 2018-07-11 07:50:35.776034 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15287 conn(0x55d98123f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 244 0x55d9829e5000 osd_ping(ping e501 stamp 2018-07-11 07:50:37.005996) v4 > -8961> 2018-07-11 07:50:35.776055 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.4 10.8.128.113:0/15287 244 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:37.005996) v4 ==== 2004+0+0 (3206071076 0 0) 0x55d9829e5000 con 0x55d98123f000 > -8960> 2018-07-11 07:50:35.776066 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:37.005996) v4 -- 0x55d986d97000 con 0 > -8959> 2018-07-11 07:50:35.807928 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12650 conn(0x55d986d21800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 226 0x55d986c23400 osd_ping(ping e501 stamp 2018-07-11 07:50:35.854943) v4 > -8958> 2018-07-11 07:50:35.807952 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.3 10.8.128.106:0/12650 226 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:35.854943) v4 ==== 2004+0+0 (2819105817 0 0) 0x55d986c23400 con 0x55d986d21800 > -8957> 2018-07-11 07:50:35.807952 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12650 conn(0x55d986d23000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 226 0x55d9bcd72c00 osd_ping(ping e501 stamp 2018-07-11 07:50:35.854943) v4 > -8956> 2018-07-11 07:50:35.807964 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:35.854943) v4 -- 0x55d9c3383c00 con 0 > -8955> 2018-07-11 07:50:35.807969 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.3 10.8.128.106:0/12650 226 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:35.854943) v4 ==== 2004+0+0 (2819105817 0 0) 0x55d9bcd72c00 con 0x55d986d23000 > -8954> 2018-07-11 07:50:35.807981 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:35.854943) v4 -- 0x55d986d97000 con 0 > -8953> 2018-07-11 07:50:35.829849 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 2968 0x55d9869d9600 MOSDScrubReserve(11.3s5 REQUEST e501) v1 > -8952> 2018-07-11 07:50:35.829879 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 2968 ==== MOSDScrubReserve(11.3s5 REQUEST e501) v1 ==== 43+0+0 (3260170403 0 0) 0x55d9869d9600 con 0x55d986c7e000 > -8951> 2018-07-11 07:50:35.829915 7ffab3be0700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e501) v1 -- 0x55d9b60df800 con 0 > -8950> 2018-07-11 07:50:35.830271 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 2969 0x55d9b60df800 MOSDScrubReserve(11.3s5 RELEASE e501) v1 > -8949> 2018-07-11 07:50:35.830289 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 2969 ==== MOSDScrubReserve(11.3s5 RELEASE e501) v1 ==== 43+0+0 (223328929 0 0) 0x55d9b60df800 con 0x55d986c7e000 > -8948> 2018-07-11 07:50:35.943499 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 800 0x55d989371200 rep_scrubmap(5.2e e501 from shard 2) v1 > -8947> 2018-07-11 07:50:35.943520 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 800 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12474 (1269249638 0 998326791) 0x55d989371200 con 0x55d986bf0800 > -8946> 2018-07-11 07:50:35.953559 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6805/15263 conn(0x55d986c41000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=31 cs=1 l=0). rx osd.5 seq 2784 0x55d986ca9a00 MOSDScrubReserve(6.15 REQUEST e501) v1 > -8945> 2018-07-11 07:50:35.953578 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.5 10.8.128.113:6805/15263 2784 ==== MOSDScrubReserve(6.15 REQUEST e501) v1 ==== 43+0+0 (3942742026 0 0) 0x55d986ca9a00 con 0x55d986c41000 > -8944> 2018-07-11 07:50:35.960231 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 794 0x55d989371440 rep_scrubmap(5.2e e501 from shard 4) v1 > -8943> 2018-07-11 07:50:35.960259 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 794 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12474 (3686815630 0 998326791) 0x55d989371440 con 0x55d986bec000 > -8942> 2018-07-11 07:50:35.985241 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.114:0/15919 conn(0x55d986df3800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.11 seq 235 0x55d986c23400 osd_ping(ping e501 stamp 2018-07-11 07:50:36.615176) v4 > -8941> 2018-07-11 07:50:35.985289 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.11 10.8.128.114:0/15919 235 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:36.615176) v4 ==== 2004+0+0 (1022821149 0 0) 0x55d986c23400 con 0x55d986df3800 > -8940> 2018-07-11 07:50:35.985303 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:36.615176) v4 -- 0x55d9bf805a00 con 0 > -8939> 2018-07-11 07:50:35.985323 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.114:0/15919 conn(0x55d986df5000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.11 seq 235 0x55d9829e5000 osd_ping(ping e501 stamp 2018-07-11 07:50:36.615176) v4 > -8938> 2018-07-11 07:50:35.985344 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.11 10.8.128.114:0/15919 235 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:36.615176) v4 ==== 2004+0+0 (1022821149 0 0) 0x55d9829e5000 con 0x55d986df5000 > -8937> 2018-07-11 07:50:35.985355 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:36.615176) v4 -- 0x55d986d97000 con 0 > -8936> 2018-07-11 07:50:36.040073 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6805/15263 -- MOSDScrubReserve(6.15 REJECT e501) v1 -- 0x55d9829e4200 con 0 > -8935> 2018-07-11 07:50:36.040127 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -8934> 2018-07-11 07:50:36.040146 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -8933> 2018-07-11 07:50:36.040205 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -8932> 2018-07-11 07:50:36.040630 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6805/15263 conn(0x55d986c41000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=31 cs=1 l=0). rx osd.5 seq 2785 0x55d9829e4200 MOSDScrubReserve(6.15 RELEASE e501) v1 > -8931> 2018-07-11 07:50:36.040651 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.5 10.8.128.113:6805/15263 2785 ==== MOSDScrubReserve(6.15 RELEASE e501) v1 ==== 43+0+0 (605848072 0 0) 0x55d9829e4200 con 0x55d986c41000 > -8930> 2018-07-11 07:50:36.040698 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -8929> 2018-07-11 07:50:36.081474 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.108:0/11967 conn(0x55d986db0000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.8 seq 216 0x55d9bcd72c00 osd_ping(ping e501 stamp 2018-07-11 07:50:35.836830) v4 > -8928> 2018-07-11 07:50:36.081496 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.8 10.8.128.108:0/11967 216 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:35.836830) v4 ==== 2004+0+0 (1702441256 0 0) 0x55d9bcd72c00 con 0x55d986db0000 > -8927> 2018-07-11 07:50:36.081491 7ffacbbd1700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.108:0/11967 conn(0x55d986d82800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.8 seq 216 0x55d9c3383c00 osd_ping(ping e501 stamp 2018-07-11 07:50:35.836830) v4 > -8926> 2018-07-11 07:50:36.081505 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:35.836830) v4 -- 0x55d986d97000 con 0 > -8925> 2018-07-11 07:50:36.081510 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 <== osd.8 10.8.128.108:0/11967 216 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:35.836830) v4 ==== 2004+0+0 (1702441256 0 0) 0x55d9c3383c00 con 0x55d986d82800 > -8924> 2018-07-11 07:50:36.081528 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:35.836830) v4 -- 0x55d9bf805a00 con 0 > -8923> 2018-07-11 07:50:36.099736 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:751ad3d9:::benchmark_data_magna115_10867_object3353623:0,end:5:751b3fc1:::benchmark_data_magna115_10867_object146874:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b14f0500 con 0 > -8922> 2018-07-11 07:50:36.099787 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:751ad3d9:::benchmark_data_magna115_10867_object3353623:0,end:5:751b3fc1:::benchmark_data_magna115_10867_object146874:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455900 con 0 > -8921> 2018-07-11 07:50:36.319617 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 801 0x55d989371680 rep_scrubmap(5.2e e501 from shard 2) v1 > -8920> 2018-07-11 07:50:36.319638 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 801 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12478 (1269249638 0 3162602836) 0x55d989371680 con 0x55d986bf0800 > -8919> 2018-07-11 07:50:36.321387 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 795 0x55d9893718c0 rep_scrubmap(5.2e e501 from shard 4) v1 > -8918> 2018-07-11 07:50:36.321407 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 795 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12478 (3686815630 0 3162602836) 0x55d9893718c0 con 0x55d986bec000 > -8917> 2018-07-11 07:50:36.373469 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -8916> 2018-07-11 07:50:36.373497 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -8915> 2018-07-11 07:50:36.373544 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -8914> 2018-07-11 07:50:36.374006 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -8913> 2018-07-11 07:50:36.374935 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:751b3fc1:::benchmark_data_magna115_10867_object146874:0,end:5:751bacbf:::benchmark_data_magna115_11707_object242726:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498780 con 0 > -8912> 2018-07-11 07:50:36.374982 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:751b3fc1:::benchmark_data_magna115_10867_object146874:0,end:5:751bacbf:::benchmark_data_magna115_11707_object242726:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455b80 con 0 > -8911> 2018-07-11 07:50:36.542002 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15263 conn(0x55d985d45800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.5 seq 219 0x55d9829e5000 osd_ping(ping e501 stamp 2018-07-11 07:50:37.772036) v4 > -8910> 2018-07-11 07:50:36.542025 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.5 10.8.128.113:0/15263 219 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:37.772036) v4 ==== 2004+0+0 (2003489502 0 0) 0x55d9829e5000 con 0x55d985d45800 > -8909> 2018-07-11 07:50:36.542036 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:37.772036) v4 -- 0x55d986d97000 con 0 > -8908> 2018-07-11 07:50:36.542033 7ffacbbd1700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15263 conn(0x55d985d44000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.5 seq 219 0x55d986c23400 osd_ping(ping e501 stamp 2018-07-11 07:50:37.772036) v4 > -8907> 2018-07-11 07:50:36.542056 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 <== osd.5 10.8.128.113:0/15263 219 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:37.772036) v4 ==== 2004+0+0 (2003489502 0 0) 0x55d986c23400 con 0x55d985d44000 > -8906> 2018-07-11 07:50:36.542067 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:37.772036) v4 -- 0x55d9bf805a00 con 0 > -8905> 2018-07-11 07:50:36.579372 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 802 0x55d989371b00 rep_scrubmap(5.2e e501 from shard 2) v1 > -8904> 2018-07-11 07:50:36.579391 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 802 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12474 (1269249638 0 779252924) 0x55d989371b00 con 0x55d986bf0800 > -8903> 2018-07-11 07:50:36.609232 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 796 0x55d989371d40 rep_scrubmap(5.2e e501 from shard 4) v1 > -8902> 2018-07-11 07:50:36.609258 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 796 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12474 (3686815630 0 779252924) 0x55d989371d40 con 0x55d986bec000 > -8901> 2018-07-11 07:50:36.611470 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.114:0/15885 conn(0x55d986d7f800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.7 seq 239 0x55d9bcd72c00 osd_ping(ping e501 stamp 2018-07-11 07:50:37.241308) v4 > -8900> 2018-07-11 07:50:36.611488 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.114:0/15885 conn(0x55d986d81000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.7 seq 239 0x55d986d33000 osd_ping(ping e501 stamp 2018-07-11 07:50:37.241308) v4 > -8899> 2018-07-11 07:50:36.611497 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.7 10.8.128.114:0/15885 239 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:37.241308) v4 ==== 2004+0+0 (2250501240 0 0) 0x55d9bcd72c00 con 0x55d986d7f800 > -8898> 2018-07-11 07:50:36.611516 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.7 10.8.128.114:0/15885 239 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:37.241308) v4 ==== 2004+0+0 (2250501240 0 0) 0x55d986d33000 con 0x55d986d81000 > -8897> 2018-07-11 07:50:36.611524 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:37.241308) v4 -- 0x55d986d97000 con 0 > -8896> 2018-07-11 07:50:36.611554 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:37.241308) v4 -- 0x55d9bfbde200 con 0 > -8895> 2018-07-11 07:50:36.698490 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -8894> 2018-07-11 07:50:36.698515 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -8893> 2018-07-11 07:50:36.698576 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -8892> 2018-07-11 07:50:36.699040 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -8891> 2018-07-11 07:50:36.716655 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:751bacbf:::benchmark_data_magna115_11707_object242726:0,end:5:751c0874:::benchmark_data_magna115_11707_object11040:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498000 con 0 > -8890> 2018-07-11 07:50:36.716705 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:751bacbf:::benchmark_data_magna115_11707_object242726:0,end:5:751c0874:::benchmark_data_magna115_11707_object11040:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455680 con 0 > -8889> 2018-07-11 07:50:36.876386 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15287 conn(0x55d981240800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 245 0x55d986d4fe00 osd_ping(ping e501 stamp 2018-07-11 07:50:38.106458) v4 > -8888> 2018-07-11 07:50:36.876413 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.4 10.8.128.113:0/15287 245 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:38.106458) v4 ==== 2004+0+0 (872306920 0 0) 0x55d986d4fe00 con 0x55d981240800 > -8887> 2018-07-11 07:50:36.876428 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:38.106458) v4 -- 0x55d9bfbde200 con 0 > -8886> 2018-07-11 07:50:36.876425 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15287 conn(0x55d98123f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 245 0x55d9829e5000 osd_ping(ping e501 stamp 2018-07-11 07:50:38.106458) v4 > -8885> 2018-07-11 07:50:36.876442 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.4 10.8.128.113:0/15287 245 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:38.106458) v4 ==== 2004+0+0 (872306920 0 0) 0x55d9829e5000 con 0x55d98123f000 > -8884> 2018-07-11 07:50:36.876458 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:38.106458) v4 -- 0x55d986d97000 con 0 > -8883> 2018-07-11 07:50:36.912930 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 797 0x55d99e220000 rep_scrubmap(5.2e e501 from shard 4) v1 > -8882> 2018-07-11 07:50:36.912951 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 797 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12472 (3686815630 0 540967002) 0x55d99e220000 con 0x55d986bec000 > -8881> 2018-07-11 07:50:36.931612 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 803 0x55d99e220240 rep_scrubmap(5.2e e501 from shard 2) v1 > -8880> 2018-07-11 07:50:36.931632 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 803 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12472 (1269249638 0 540967002) 0x55d99e220240 con 0x55d986bf0800 > -8879> 2018-07-11 07:50:36.953753 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6805/15263 conn(0x55d986c41000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=31 cs=1 l=0). rx osd.5 seq 2786 0x55d986d33000 MOSDScrubReserve(6.15 REQUEST e501) v1 > -8878> 2018-07-11 07:50:36.953770 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.5 10.8.128.113:6805/15263 2786 ==== MOSDScrubReserve(6.15 REQUEST e501) v1 ==== 43+0+0 (3942742026 0 0) 0x55d986d33000 con 0x55d986c41000 > -8877> 2018-07-11 07:50:36.966674 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6805/15263 -- MOSDScrubReserve(6.15 REJECT e501) v1 -- 0x55d9829e4200 con 0 > -8876> 2018-07-11 07:50:36.966800 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -8875> 2018-07-11 07:50:36.966822 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -8874> 2018-07-11 07:50:36.966870 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -8873> 2018-07-11 07:50:36.967102 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6805/15263 conn(0x55d986c41000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=31 cs=1 l=0). rx osd.5 seq 2787 0x55d9829e4200 MOSDScrubReserve(6.15 RELEASE e501) v1 > -8872> 2018-07-11 07:50:36.967122 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.5 10.8.128.113:6805/15263 2787 ==== MOSDScrubReserve(6.15 RELEASE e501) v1 ==== 43+0+0 (605848072 0 0) 0x55d9829e4200 con 0x55d986c41000 > -8871> 2018-07-11 07:50:36.967336 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -8870> 2018-07-11 07:50:37.012743 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'66962,epoch:501/498,start:5:751c0874:::benchmark_data_magna115_11707_object11040:0,end:5:751c72c5:::benchmark_data_magna115_10867_object1989556:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498280 con 0 > -8869> 2018-07-11 07:50:37.012794 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'66962,epoch:501/498,start:5:751c0874:::benchmark_data_magna115_11707_object11040:0,end:5:751c72c5:::benchmark_data_magna115_10867_object1989556:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455400 con 0 > -8868> 2018-07-11 07:50:37.015468 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12651 conn(0x55d986db3000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.2 seq 235 0x55d9c3383c00 osd_ping(ping e501 stamp 2018-07-11 07:50:37.062494) v4 > -8867> 2018-07-11 07:50:37.015494 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.2 10.8.128.106:0/12651 235 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:37.062494) v4 ==== 2004+0+0 (293006550 0 0) 0x55d9c3383c00 con 0x55d986db3000 > -8866> 2018-07-11 07:50:37.015506 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:37.062494) v4 -- 0x55d9bf805a00 con 0 > -8865> 2018-07-11 07:50:37.015513 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12651 conn(0x55d986db1800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.2 seq 235 0x55d9bfbde200 osd_ping(ping e501 stamp 2018-07-11 07:50:37.062494) v4 > -8864> 2018-07-11 07:50:37.015538 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.2 10.8.128.106:0/12651 235 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:37.062494) v4 ==== 2004+0+0 (293006550 0 0) 0x55d9bfbde200 con 0x55d986db1800 > -8863> 2018-07-11 07:50:37.015555 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:37.062494) v4 -- 0x55d9b67d8200 con 0 > -8862> 2018-07-11 07:50:37.085753 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.114:0/15919 conn(0x55d986df3800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.11 seq 236 0x55d986c23400 osd_ping(ping e501 stamp 2018-07-11 07:50:37.715602) v4 > -8861> 2018-07-11 07:50:37.085777 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.11 10.8.128.114:0/15919 236 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:37.715602) v4 ==== 2004+0+0 (2569035898 0 0) 0x55d986c23400 con 0x55d986df3800 > -8860> 2018-07-11 07:50:37.085794 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:37.715602) v4 -- 0x55d9bf805a00 con 0 > -8859> 2018-07-11 07:50:37.085807 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.114:0/15919 conn(0x55d986df5000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.11 seq 236 0x55d9bcd72c00 osd_ping(ping e501 stamp 2018-07-11 07:50:37.715602) v4 > -8858> 2018-07-11 07:50:37.085829 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.11 10.8.128.114:0/15919 236 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:37.715602) v4 ==== 2004+0+0 (2569035898 0 0) 0x55d9bcd72c00 con 0x55d986df5000 > -8857> 2018-07-11 07:50:37.085842 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:37.715602) v4 -- 0x55d986d97000 con 0 > -8856> 2018-07-11 07:50:37.247712 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 798 0x55d99e220480 rep_scrubmap(5.2e e501 from shard 4) v1 > -8855> 2018-07-11 07:50:37.247738 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 798 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12478 (3686815630 0 2761088825) 0x55d99e220480 con 0x55d986bec000 > -8854> 2018-07-11 07:50:37.258021 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12677 conn(0x55d986db4800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.1 seq 239 0x55d9829e5000 osd_ping(ping e501 stamp 2018-07-11 07:50:37.305035) v4 > -8853> 2018-07-11 07:50:37.258029 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12677 conn(0x55d986df2000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.1 seq 239 0x55d9bfbde200 osd_ping(ping e501 stamp 2018-07-11 07:50:37.305035) v4 > -8852> 2018-07-11 07:50:37.258047 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.1 10.8.128.106:0/12677 239 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:37.305035) v4 ==== 2004+0+0 (1481595422 0 0) 0x55d9bfbde200 con 0x55d986df2000 > -8851> 2018-07-11 07:50:37.258043 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.1 10.8.128.106:0/12677 239 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:37.305035) v4 ==== 2004+0+0 (1481595422 0 0) 0x55d9829e5000 con 0x55d986db4800 > -8850> 2018-07-11 07:50:37.258057 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:37.305035) v4 -- 0x55d9b67d8200 con 0 > -8849> 2018-07-11 07:50:37.258081 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:37.305035) v4 -- 0x55d986d97000 con 0 > -8848> 2018-07-11 07:50:37.260671 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 804 0x55d99e2206c0 rep_scrubmap(5.2e e501 from shard 2) v1 > -8847> 2018-07-11 07:50:37.260691 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 804 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12478 (1269249638 0 2761088825) 0x55d99e2206c0 con 0x55d986bf0800 > -8846> 2018-07-11 07:50:37.347108 7ffab6be6700 1 -- 10.8.128.107:6800/13927 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55d9ba81a000 con 0 > -8845> 2018-07-11 07:50:37.347178 7ffab6be6700 1 -- 10.8.128.107:6800/13927 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55d9ba81b340 con 0 > -8844> 2018-07-11 07:50:37.508411 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12650 conn(0x55d986d23000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 227 0x55d9bcd72c00 osd_ping(ping e501 stamp 2018-07-11 07:50:37.555361) v4 > -8843> 2018-07-11 07:50:37.508444 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.3 10.8.128.106:0/12650 227 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:37.555361) v4 ==== 2004+0+0 (3229093742 0 0) 0x55d9bcd72c00 con 0x55d986d23000 > -8842> 2018-07-11 07:50:37.508453 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:37.555361) v4 -- 0x55d986d97000 con 0 > -8841> 2018-07-11 07:50:37.508510 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12650 conn(0x55d986d21800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 227 0x55d9c3383c00 osd_ping(ping e501 stamp 2018-07-11 07:50:37.555361) v4 > -8840> 2018-07-11 07:50:37.508533 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.3 10.8.128.106:0/12650 227 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:37.555361) v4 ==== 2004+0+0 (3229093742 0 0) 0x55d9c3383c00 con 0x55d986d21800 > -8839> 2018-07-11 07:50:37.508544 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:37.555361) v4 -- 0x55d9bf805a00 con 0 > -8838> 2018-07-11 07:50:37.781922 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.108:0/11967 conn(0x55d986db0000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.8 seq 217 0x55d9829e5000 osd_ping(ping e501 stamp 2018-07-11 07:50:37.537308) v4 > -8837> 2018-07-11 07:50:37.781944 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.8 10.8.128.108:0/11967 217 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:37.537308) v4 ==== 2004+0+0 (2411490934 0 0) 0x55d9829e5000 con 0x55d986db0000 > -8836> 2018-07-11 07:50:37.781945 7ffacbbd1700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.108:0/11967 conn(0x55d986d82800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.8 seq 217 0x55d986c23400 osd_ping(ping e501 stamp 2018-07-11 07:50:37.537308) v4 > -8835> 2018-07-11 07:50:37.781960 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:37.537308) v4 -- 0x55d986d97000 con 0 > -8834> 2018-07-11 07:50:37.781963 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 <== osd.8 10.8.128.108:0/11967 217 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:37.537308) v4 ==== 2004+0+0 (2411490934 0 0) 0x55d986c23400 con 0x55d986d82800 > -8833> 2018-07-11 07:50:37.781995 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:37.537308) v4 -- 0x55d9bf805a00 con 0 > -8832> 2018-07-11 07:50:37.830267 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 2970 0x55d9c3383c00 MOSDScrubReserve(11.3s5 REQUEST e501) v1 > -8831> 2018-07-11 07:50:37.830284 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 2970 ==== MOSDScrubReserve(11.3s5 REQUEST e501) v1 ==== 43+0+0 (3260170403 0 0) 0x55d9c3383c00 con 0x55d986c7e000 > -8830> 2018-07-11 07:50:37.830420 7ffab3be0700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e501) v1 -- 0x55d9b60df800 con 0 > -8829> 2018-07-11 07:50:37.830688 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 2971 0x55d9bf805a00 MOSDScrubReserve(11.3s5 RELEASE e501) v1 > -8828> 2018-07-11 07:50:37.830702 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 2971 ==== MOSDScrubReserve(11.3s5 RELEASE e501) v1 ==== 43+0+0 (223328929 0 0) 0x55d9bf805a00 con 0x55d986c7e000 > -8827> 2018-07-11 07:50:37.955262 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6801/15919 conn(0x55d986c3f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=29 cs=1 l=0). rx osd.11 seq 208 0x55d986c23400 MOSDScrubReserve(10.f REQUEST e501) v1 > -8826> 2018-07-11 07:50:37.955279 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.11 10.8.128.114:6801/15919 208 ==== MOSDScrubReserve(10.f REQUEST e501) v1 ==== 43+0+0 (2892966580 0 0) 0x55d986c23400 con 0x55d986c3f800 > -8825> 2018-07-11 07:50:37.955323 7ffab53e3700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.114:6801/15919 -- MOSDScrubReserve(10.f REJECT e501) v1 -- 0x55d9bf805800 con 0 > -8824> 2018-07-11 07:50:37.955626 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6801/15919 conn(0x55d986c3f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=29 cs=1 l=0). rx osd.11 seq 209 0x55d9bf805800 MOSDScrubReserve(10.f RELEASE e501) v1 > -8823> 2018-07-11 07:50:37.955653 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.11 10.8.128.114:6801/15919 209 ==== MOSDScrubReserve(10.f RELEASE e501) v1 ==== 43+0+0 (1668468918 0 0) 0x55d9bf805800 con 0x55d986c3f800 > -8822> 2018-07-11 07:50:38.098943 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 29] Generated table #36790: 2922060 keys, 58471085 bytes > -8821> 2018-07-11 07:50:38.098990 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295438098962, "cf_name": "default", "job": 29, "event": "table_file_creation", "file_number": 36790, "file_size": 58471085, "table_properties": {"data_size": 50531265, "index_size": 519252, "filter_size": 7419577, "raw_key_size": 84415070, "raw_average_key_size": 28, "raw_value_size": 10962270, "raw_average_value_size": 3, "num_data_blocks": 12408, "num_entries": 2922060, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "2000000", "kMergeOperands": "0"}} > -8820> 2018-07-11 07:50:38.110690 7ffab1bdc700 5 osd.10 501 heartbeat: osd_stat(788 GB used, 143 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,0,0,2]) > -8819> 2018-07-11 07:50:38.110706 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6802/15255 -- osd_ping(ping e501 stamp 2018-07-11 07:50:38.110704) v4 -- 0x55d986c23800 con 0 > -8818> 2018-07-11 07:50:38.110747 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6803/15255 -- osd_ping(ping e501 stamp 2018-07-11 07:50:38.110704) v4 -- 0x55d9b67d8a00 con 0 > -8817> 2018-07-11 07:50:38.110763 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6806/12677 -- osd_ping(ping e501 stamp 2018-07-11 07:50:38.110704) v4 -- 0x55d9bf805400 con 0 > -8816> 2018-07-11 07:50:38.110778 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6807/12677 -- osd_ping(ping e501 stamp 2018-07-11 07:50:38.110704) v4 -- 0x55d986cd5600 con 0 > -8815> 2018-07-11 07:50:38.110791 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6808/12651 -- osd_ping(ping e501 stamp 2018-07-11 07:50:38.110704) v4 -- 0x55d9bdba4800 con 0 > -8814> 2018-07-11 07:50:38.110805 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6810/12651 -- osd_ping(ping e501 stamp 2018-07-11 07:50:38.110704) v4 -- 0x55d9c3382a00 con 0 > -8813> 2018-07-11 07:50:38.110817 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6809/12650 -- osd_ping(ping e501 stamp 2018-07-11 07:50:38.110704) v4 -- 0x55d982cdf200 con 0 > -8812> 2018-07-11 07:50:38.110837 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6811/12650 -- osd_ping(ping e501 stamp 2018-07-11 07:50:38.110704) v4 -- 0x55d9bf805000 con 0 > -8811> 2018-07-11 07:50:38.110849 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6810/15287 -- osd_ping(ping e501 stamp 2018-07-11 07:50:38.110704) v4 -- 0x55d9b4388400 con 0 > -8810> 2018-07-11 07:50:38.110875 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6811/15287 -- osd_ping(ping e501 stamp 2018-07-11 07:50:38.110704) v4 -- 0x55d986d4e400 con 0 > -8809> 2018-07-11 07:50:38.110891 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6806/15263 -- osd_ping(ping e501 stamp 2018-07-11 07:50:38.110704) v4 -- 0x55d9869be600 con 0 > -8808> 2018-07-11 07:50:38.110904 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6807/15263 -- osd_ping(ping e501 stamp 2018-07-11 07:50:38.110704) v4 -- 0x55d9ba819a00 con 0 > -8807> 2018-07-11 07:50:38.110920 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.115:6802/13323 -- osd_ping(ping e501 stamp 2018-07-11 07:50:38.110704) v4 -- 0x55d986c22400 con 0 > -8806> 2018-07-11 07:50:38.110935 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.115:6803/13323 -- osd_ping(ping e501 stamp 2018-07-11 07:50:38.110704) v4 -- 0x55d9bfbde400 con 0 > -8805> 2018-07-11 07:50:38.110948 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6806/15885 -- osd_ping(ping e501 stamp 2018-07-11 07:50:38.110704) v4 -- 0x55d986d97e00 con 0 > -8804> 2018-07-11 07:50:38.110960 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6807/15885 -- osd_ping(ping e501 stamp 2018-07-11 07:50:38.110704) v4 -- 0x55d986cd5200 con 0 > -8803> 2018-07-11 07:50:38.110976 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.108:6802/11967 -- osd_ping(ping e501 stamp 2018-07-11 07:50:38.110704) v4 -- 0x55d986b6dc00 con 0 > -8802> 2018-07-11 07:50:38.110989 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.108:6803/11967 -- osd_ping(ping e501 stamp 2018-07-11 07:50:38.110704) v4 -- 0x55d986cd6a00 con 0 > -8801> 2018-07-11 07:50:38.111002 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.107:6806/12617 -- osd_ping(ping e501 stamp 2018-07-11 07:50:38.110704) v4 -- 0x55d986cd5c00 con 0 > -8800> 2018-07-11 07:50:38.111015 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.107:6807/12617 -- osd_ping(ping e501 stamp 2018-07-11 07:50:38.110704) v4 -- 0x55d986d33e00 con 0 > -8799> 2018-07-11 07:50:38.111027 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6802/15919 -- osd_ping(ping e501 stamp 2018-07-11 07:50:38.110704) v4 -- 0x55d986d96400 con 0 > -8798> 2018-07-11 07:50:38.111040 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6803/15919 -- osd_ping(ping e501 stamp 2018-07-11 07:50:38.110704) v4 -- 0x55d982cdfc00 con 0 > -8797> 2018-07-11 07:50:38.111200 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.107:6807/12617 conn(0x55d986bd3800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.9 seq 227 0x55d9b60df800 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:38.110704) v4 > -8796> 2018-07-11 07:50:38.111222 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.9 10.8.128.107:6807/12617 227 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:38.110704) v4 ==== 2004+0+0 (3449045118 0 0) 0x55d9b60df800 con 0x55d986bd3800 > -8795> 2018-07-11 07:50:38.111243 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.107:6806/12617 conn(0x55d986bd2000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.9 seq 227 0x55d9829e5000 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:38.110704) v4 > -8794> 2018-07-11 07:50:38.111270 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.9 10.8.128.107:6806/12617 227 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:38.110704) v4 ==== 2004+0+0 (3449045118 0 0) 0x55d9829e5000 con 0x55d986bd2000 > -8793> 2018-07-11 07:50:38.111315 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6803/15255 conn(0x55d986bc9800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.0 seq 227 0x55d9829e5000 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:38.110704) v4 > -8792> 2018-07-11 07:50:38.111316 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6806/12677 conn(0x55d986b58000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.1 seq 227 0x55d9b60df800 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:38.110704) v4 > -8791> 2018-07-11 07:50:38.111319 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6807/12677 conn(0x55d986b59800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.1 seq 227 0x55d9bfbde200 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:38.110704) v4 > -8790> 2018-07-11 07:50:38.111333 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.0 10.8.128.113:6803/15255 227 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:38.110704) v4 ==== 2004+0+0 (3449045118 0 0) 0x55d9829e5000 con 0x55d986bc9800 > -8789> 2018-07-11 07:50:38.111338 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.1 10.8.128.106:6807/12677 227 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:38.110704) v4 ==== 2004+0+0 (3449045118 0 0) 0x55d9bfbde200 con 0x55d986b59800 > -8788> 2018-07-11 07:50:38.111347 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.1 10.8.128.106:6806/12677 227 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:38.110704) v4 ==== 2004+0+0 (3449045118 0 0) 0x55d9b60df800 con 0x55d986b58000 > -8787> 2018-07-11 07:50:38.111378 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6802/15255 conn(0x55d986bc8000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.0 seq 227 0x55d9bfbde200 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:38.110704) v4 > -8786> 2018-07-11 07:50:38.111386 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6809/12650 conn(0x55d986bcb000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 227 0x55d9b60df800 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:38.110704) v4 > -8785> 2018-07-11 07:50:38.111397 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.3 10.8.128.106:6809/12650 227 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:38.110704) v4 ==== 2004+0+0 (3449045118 0 0) 0x55d9b60df800 con 0x55d986bcb000 > -8784> 2018-07-11 07:50:38.111396 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.0 10.8.128.113:6802/15255 227 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:38.110704) v4 ==== 2004+0+0 (3449045118 0 0) 0x55d9bfbde200 con 0x55d986bc8000 > -8783> 2018-07-11 07:50:38.111401 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6810/12651 conn(0x55d986b9b800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.2 seq 227 0x55d9829e5000 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:38.110704) v4 > -8782> 2018-07-11 07:50:38.111417 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.2 10.8.128.106:6810/12651 227 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:38.110704) v4 ==== 2004+0+0 (3449045118 0 0) 0x55d9829e5000 con 0x55d986b9b800 > -8781> 2018-07-11 07:50:38.111442 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6808/12651 conn(0x55d986b9a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.2 seq 227 0x55d9bfbde200 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:38.110704) v4 > -8780> 2018-07-11 07:50:38.111453 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6806/15263 conn(0x55d986b9d000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.5 seq 227 0x55d9b60df800 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:38.110704) v4 > -8779> 2018-07-11 07:50:38.111455 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.2 10.8.128.106:6808/12651 227 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:38.110704) v4 ==== 2004+0+0 (3449045118 0 0) 0x55d9bfbde200 con 0x55d986b9a000 > -8778> 2018-07-11 07:50:38.111471 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6811/15287 conn(0x55d986bb0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 227 0x55d9829e5000 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:38.110704) v4 > -8777> 2018-07-11 07:50:38.111471 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.5 10.8.128.113:6806/15263 227 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:38.110704) v4 ==== 2004+0+0 (3449045118 0 0) 0x55d9b60df800 con 0x55d986b9d000 > -8776> 2018-07-11 07:50:38.111484 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.4 10.8.128.113:6811/15287 227 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:38.110704) v4 ==== 2004+0+0 (3449045118 0 0) 0x55d9829e5000 con 0x55d986bb0800 > -8775> 2018-07-11 07:50:38.111508 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6807/15263 conn(0x55d986b9e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.5 seq 227 0x55d9bfbde200 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:38.110704) v4 > -8774> 2018-07-11 07:50:38.111512 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.115:6802/13323 conn(0x55d986bc1000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 227 0x55d9829e5000 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:38.110704) v4 > -8773> 2018-07-11 07:50:38.111512 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.108:6802/11967 conn(0x55d986bbe000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.8 seq 227 0x55d9b60df800 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:38.110704) v4 > -8772> 2018-07-11 07:50:38.111523 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.6 10.8.128.115:6802/13323 227 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:38.110704) v4 ==== 2004+0+0 (3449045118 0 0) 0x55d9829e5000 con 0x55d986bc1000 > -8771> 2018-07-11 07:50:38.111522 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.5 10.8.128.113:6807/15263 227 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:38.110704) v4 ==== 2004+0+0 (3449045118 0 0) 0x55d9bfbde200 con 0x55d986b9e800 > -8770> 2018-07-11 07:50:38.111524 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.8 10.8.128.108:6802/11967 227 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:38.110704) v4 ==== 2004+0+0 (3449045118 0 0) 0x55d9b60df800 con 0x55d986bbe000 > -8769> 2018-07-11 07:50:38.111573 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.115:6803/13323 conn(0x55d986bc2800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 227 0x55d9b60df800 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:38.110704) v4 > -8768> 2018-07-11 07:50:38.111586 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.6 10.8.128.115:6803/13323 227 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:38.110704) v4 ==== 2004+0+0 (3449045118 0 0) 0x55d9b60df800 con 0x55d986bc2800 > -8767> 2018-07-11 07:50:38.111588 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6811/12650 conn(0x55d986bcc800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 227 0x55d9bfbde200 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:38.110704) v4 > -8766> 2018-07-11 07:50:38.111585 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6802/15919 conn(0x55d986b5b000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.11 seq 227 0x55d9829e5000 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:38.110704) v4 > -8765> 2018-07-11 07:50:38.111607 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.3 10.8.128.106:6811/12650 227 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:38.110704) v4 ==== 2004+0+0 (3449045118 0 0) 0x55d9bfbde200 con 0x55d986bcc800 > -8764> 2018-07-11 07:50:38.111614 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6803/15919 conn(0x55d986b5c800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.11 seq 227 0x55d9b60df800 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:38.110704) v4 > -8763> 2018-07-11 07:50:38.111612 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.11 10.8.128.114:6802/15919 227 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:38.110704) v4 ==== 2004+0+0 (3449045118 0 0) 0x55d9829e5000 con 0x55d986b5b000 > -8762> 2018-07-11 07:50:38.111622 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.11 10.8.128.114:6803/15919 227 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:38.110704) v4 ==== 2004+0+0 (3449045118 0 0) 0x55d9b60df800 con 0x55d986b5c800 > -8761> 2018-07-11 07:50:38.111658 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6810/15287 conn(0x55d986baf000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 227 0x55d9bfbde200 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:38.110704) v4 > -8760> 2018-07-11 07:50:38.111671 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.4 10.8.128.113:6810/15287 227 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:38.110704) v4 ==== 2004+0+0 (3449045118 0 0) 0x55d9bfbde200 con 0x55d986baf000 > -8759> 2018-07-11 07:50:38.111662 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6807/15885 conn(0x55d986bad800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.7 seq 227 0x55d9b60df800 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:38.110704) v4 > -8758> 2018-07-11 07:50:38.111696 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.7 10.8.128.114:6807/15885 227 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:38.110704) v4 ==== 2004+0+0 (3449045118 0 0) 0x55d9b60df800 con 0x55d986bad800 > -8757> 2018-07-11 07:50:38.111707 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6806/15885 conn(0x55d986bac000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.7 seq 227 0x55d9829e5000 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:38.110704) v4 > -8756> 2018-07-11 07:50:38.111717 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.108:6803/11967 conn(0x55d986bbf800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.8 seq 227 0x55d9bfbde200 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:38.110704) v4 > -8755> 2018-07-11 07:50:38.111725 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.7 10.8.128.114:6806/15885 227 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:38.110704) v4 ==== 2004+0+0 (3449045118 0 0) 0x55d9829e5000 con 0x55d986bac000 > -8754> 2018-07-11 07:50:38.111736 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.8 10.8.128.108:6803/11967 227 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:38.110704) v4 ==== 2004+0+0 (3449045118 0 0) 0x55d9bfbde200 con 0x55d986bbf800 > -8753> 2018-07-11 07:50:38.156091 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -8752> 2018-07-11 07:50:38.156122 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -8751> 2018-07-11 07:50:38.156169 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -8750> 2018-07-11 07:50:38.156644 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -8749> 2018-07-11 07:50:38.175053 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67735,epoch:501/498,start:5:751c72c5:::benchmark_data_magna115_10867_object1989556:0,end:5:751cfcd6:::benchmark_data_magna115_10867_object2361686:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454f00 con 0 > -8748> 2018-07-11 07:50:38.175094 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67735,epoch:501/498,start:5:751c72c5:::benchmark_data_magna115_10867_object1989556:0,end:5:751cfcd6:::benchmark_data_magna115_10867_object2361686:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455180 con 0 > -8747> 2018-07-11 07:50:38.384773 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 805 0x55d99e220900 rep_scrubmap(5.2e e501 from shard 2) v1 > -8746> 2018-07-11 07:50:38.384793 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 805 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12462 (1269249638 0 336169808) 0x55d99e220900 con 0x55d986bf0800 > -8745> 2018-07-11 07:50:38.393734 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.115:0/13323 conn(0x55d986d24800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 234 0x55d986d4fe00 osd_ping(ping e501 stamp 2018-07-11 07:50:38.738975) v4 > -8744> 2018-07-11 07:50:38.393742 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.115:0/13323 conn(0x55d986d7e000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 234 0x55d9b60df800 osd_ping(ping e501 stamp 2018-07-11 07:50:38.738975) v4 > -8743> 2018-07-11 07:50:38.393756 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.6 10.8.128.115:0/13323 234 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:38.738975) v4 ==== 2004+0+0 (2743320842 0 0) 0x55d986d4fe00 con 0x55d986d24800 > -8742> 2018-07-11 07:50:38.393760 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.6 10.8.128.115:0/13323 234 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:38.738975) v4 ==== 2004+0+0 (2743320842 0 0) 0x55d9b60df800 con 0x55d986d7e000 > -8741> 2018-07-11 07:50:38.393766 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:38.738975) v4 -- 0x55d986cd6a00 con 0 > -8740> 2018-07-11 07:50:38.393788 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:38.738975) v4 -- 0x55d982cdfc00 con 0 > -8739> 2018-07-11 07:50:38.404424 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 799 0x55d99e220b40 rep_scrubmap(5.2e e501 from shard 4) v1 > -8738> 2018-07-11 07:50:38.404445 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 799 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12462 (3686815630 0 336169808) 0x55d99e220b40 con 0x55d986bec000 > -8737> 2018-07-11 07:50:38.459486 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15255 conn(0x55d986c82800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.0 seq 235 0x55d9829e5000 osd_ping(ping e501 stamp 2018-07-11 07:50:39.689572) v4 > -8736> 2018-07-11 07:50:38.459510 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.0 10.8.128.113:0/15255 235 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:39.689572) v4 ==== 2004+0+0 (3918144391 0 0) 0x55d9829e5000 con 0x55d986c82800 > -8735> 2018-07-11 07:50:38.459503 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15255 conn(0x55d986d20000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.0 seq 235 0x55d986d4fe00 osd_ping(ping e501 stamp 2018-07-11 07:50:39.689572) v4 > -8734> 2018-07-11 07:50:38.459525 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.0 10.8.128.113:0/15255 235 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:39.689572) v4 ==== 2004+0+0 (3918144391 0 0) 0x55d986d4fe00 con 0x55d986d20000 > -8733> 2018-07-11 07:50:38.459525 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:39.689572) v4 -- 0x55d986d96400 con 0 > -8732> 2018-07-11 07:50:38.459557 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:39.689572) v4 -- 0x55d986cd6a00 con 0 > -8731> 2018-07-11 07:50:38.566317 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -8730> 2018-07-11 07:50:38.566343 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -8729> 2018-07-11 07:50:38.566390 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -8728> 2018-07-11 07:50:38.566854 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -8727> 2018-07-11 07:50:38.583609 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67802,epoch:501/498,start:5:751cfcd6:::benchmark_data_magna115_10867_object2361686:0,end:5:751d63f0:::benchmark_data_magna115_10867_object767254:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454780 con 0 > -8726> 2018-07-11 07:50:38.583659 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67802,epoch:501/498,start:5:751cfcd6:::benchmark_data_magna115_10867_object2361686:0,end:5:751d63f0:::benchmark_data_magna115_10867_object767254:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454a00 con 0 > -8725> 2018-07-11 07:50:38.715953 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12651 conn(0x55d986db3000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.2 seq 236 0x55d9829e4800 osd_ping(ping e501 stamp 2018-07-11 07:50:38.762986) v4 > -8724> 2018-07-11 07:50:38.715979 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.2 10.8.128.106:0/12651 236 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:38.762986) v4 ==== 2004+0+0 (841916706 0 0) 0x55d9829e4800 con 0x55d986db3000 > -8723> 2018-07-11 07:50:38.715995 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:38.762986) v4 -- 0x55d982cdfc00 con 0 > -8722> 2018-07-11 07:50:38.716014 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12651 conn(0x55d986db1800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.2 seq 236 0x55d986d4fe00 osd_ping(ping e501 stamp 2018-07-11 07:50:38.762986) v4 > -8721> 2018-07-11 07:50:38.716035 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.2 10.8.128.106:0/12651 236 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:38.762986) v4 ==== 2004+0+0 (841916706 0 0) 0x55d986d4fe00 con 0x55d986db1800 > -8720> 2018-07-11 07:50:38.716046 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:38.762986) v4 -- 0x55d986cd6a00 con 0 > -8719> 2018-07-11 07:50:38.787035 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 806 0x55d99e220d80 rep_scrubmap(5.2e e501 from shard 2) v1 > -8718> 2018-07-11 07:50:38.787055 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 806 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12470 (1269249638 0 1975120355) 0x55d99e220d80 con 0x55d986bf0800 > -8717> 2018-07-11 07:50:38.814120 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 800 0x55d99e220fc0 rep_scrubmap(5.2e e501 from shard 4) v1 > -8716> 2018-07-11 07:50:38.814138 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 800 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12470 (3686815630 0 1975120355) 0x55d99e220fc0 con 0x55d986bec000 > -8715> 2018-07-11 07:50:38.855882 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -8714> 2018-07-11 07:50:38.855906 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -8713> 2018-07-11 07:50:38.855966 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -8712> 2018-07-11 07:50:38.856459 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -8711> 2018-07-11 07:50:38.896549 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:751d63f0:::benchmark_data_magna115_10867_object767254:0,end:5:751dc996:::benchmark_data_magna115_10867_object2964034:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454c80 con 0 > -8710> 2018-07-11 07:50:38.896600 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:751d63f0:::benchmark_data_magna115_10867_object767254:0,end:5:751dc996:::benchmark_data_magna115_10867_object2964034:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98944b680 con 0 > -8709> 2018-07-11 07:50:39.089155 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 807 0x55d99e221200 rep_scrubmap(5.2e e501 from shard 2) v1 > -8708> 2018-07-11 07:50:39.089177 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 807 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12472 (1269249638 0 2041705618) 0x55d99e221200 con 0x55d986bf0800 > -8707> 2018-07-11 07:50:39.107074 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 801 0x55d99e221440 rep_scrubmap(5.2e e501 from shard 4) v1 > -8706> 2018-07-11 07:50:39.107094 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 801 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12472 (3686815630 0 2041705618) 0x55d99e221440 con 0x55d986bec000 > -8705> 2018-07-11 07:50:39.193580 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -8704> 2018-07-11 07:50:39.193604 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -8703> 2018-07-11 07:50:39.193664 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -8702> 2018-07-11 07:50:39.194125 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -8701> 2018-07-11 07:50:39.242260 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:751dc996:::benchmark_data_magna115_10867_object2964034:0,end:5:751e09c1:::benchmark_data_magna115_10867_object3360227:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb773b80 con 0 > -8700> 2018-07-11 07:50:39.242300 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:751dc996:::benchmark_data_magna115_10867_object2964034:0,end:5:751e09c1:::benchmark_data_magna115_10867_object3360227:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454000 con 0 > -8699> 2018-07-11 07:50:39.466952 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 808 0x55d99e221680 rep_scrubmap(5.2e e501 from shard 2) v1 > -8698> 2018-07-11 07:50:39.466982 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 808 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12480 (1269249638 0 2895116) 0x55d99e221680 con 0x55d986bf0800 > -8697> 2018-07-11 07:50:39.477489 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 802 0x55d99e2218c0 rep_scrubmap(5.2e e501 from shard 4) v1 > -8696> 2018-07-11 07:50:39.477508 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 802 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12480 (3686815630 0 2895116) 0x55d99e2218c0 con 0x55d986bec000 > -8695> 2018-07-11 07:50:39.550267 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -8694> 2018-07-11 07:50:39.550297 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -8693> 2018-07-11 07:50:39.550346 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -8692> 2018-07-11 07:50:39.550810 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -8691> 2018-07-11 07:50:39.551880 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:751e09c1:::benchmark_data_magna115_10867_object3360227:0,end:5:751e7344:::benchmark_data_magna115_10867_object2804137:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb773680 con 0 > -8690> 2018-07-11 07:50:39.551930 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:751e09c1:::benchmark_data_magna115_10867_object3360227:0,end:5:751e7344:::benchmark_data_magna115_10867_object2804137:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454280 con 0 > -8689> 2018-07-11 07:50:39.556044 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6805/15885 conn(0x55d986c7f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.7 seq 2760 0x55d986d4fe00 MOSDScrubReserve(5.4 REQUEST e501) v1 > -8688> 2018-07-11 07:50:39.556063 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.7 10.8.128.114:6805/15885 2760 ==== MOSDScrubReserve(5.4 REQUEST e501) v1 ==== 43+0+0 (1753221512 0 0) 0x55d986d4fe00 con 0x55d986c7f800 > -8687> 2018-07-11 07:50:39.556137 7ffab33df700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.114:6805/15885 -- MOSDScrubReserve(5.4 REJECT e501) v1 -- 0x55d9c74e1800 con 0 > -8686> 2018-07-11 07:50:39.556456 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6805/15885 conn(0x55d986c7f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.7 seq 2761 0x55d9c74e1800 MOSDScrubReserve(5.4 RELEASE e501) v1 > -8685> 2018-07-11 07:50:39.556471 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.7 10.8.128.114:6805/15885 2761 ==== MOSDScrubReserve(5.4 RELEASE e501) v1 ==== 43+0+0 (2812146570 0 0) 0x55d9c74e1800 con 0x55d986c7f800 > -8684> 2018-07-11 07:50:39.559898 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15255 conn(0x55d986d20000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.0 seq 236 0x55d9bfbde200 osd_ping(ping e501 stamp 2018-07-11 07:50:40.789952) v4 > -8683> 2018-07-11 07:50:39.559925 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.0 10.8.128.113:0/15255 236 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:40.789952) v4 ==== 2004+0+0 (1362386169 0 0) 0x55d9bfbde200 con 0x55d986d20000 > -8682> 2018-07-11 07:50:39.559938 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:40.789952) v4 -- 0x55d9ba819a00 con 0 > -8681> 2018-07-11 07:50:39.559958 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15255 conn(0x55d986c82800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.0 seq 236 0x55d9bcd72c00 osd_ping(ping e501 stamp 2018-07-11 07:50:40.789952) v4 > -8680> 2018-07-11 07:50:39.559980 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.0 10.8.128.113:0/15255 236 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:40.789952) v4 ==== 2004+0+0 (1362386169 0 0) 0x55d9bcd72c00 con 0x55d986c82800 > -8679> 2018-07-11 07:50:39.559997 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:40.789952) v4 -- 0x55d986d96400 con 0 > -8678> 2018-07-11 07:50:39.770273 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 809 0x55d99e221b00 rep_scrubmap(5.2e e501 from shard 2) v1 > -8677> 2018-07-11 07:50:39.770292 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 809 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12462 (1269249638 0 51622582) 0x55d99e221b00 con 0x55d986bf0800 > -8676> 2018-07-11 07:50:39.780347 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 803 0x55d99e221d40 rep_scrubmap(5.2e e501 from shard 4) v1 > -8675> 2018-07-11 07:50:39.780367 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 803 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12462 (3686815630 0 51622582) 0x55d99e221d40 con 0x55d986bec000 > -8674> 2018-07-11 07:50:39.805415 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -8673> 2018-07-11 07:50:39.805438 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -8672> 2018-07-11 07:50:39.805498 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -8671> 2018-07-11 07:50:39.805955 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -8670> 2018-07-11 07:50:39.808920 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12650 conn(0x55d986d23000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 228 0x55d9829e5000 osd_ping(ping e501 stamp 2018-07-11 07:50:39.855922) v4 > -8669> 2018-07-11 07:50:39.808932 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12650 conn(0x55d986d21800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 228 0x55d9b60df800 osd_ping(ping e501 stamp 2018-07-11 07:50:39.855922) v4 > -8668> 2018-07-11 07:50:39.808946 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.3 10.8.128.106:0/12650 228 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:39.855922) v4 ==== 2004+0+0 (2535501339 0 0) 0x55d9829e5000 con 0x55d986d23000 > -8667> 2018-07-11 07:50:39.808949 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.3 10.8.128.106:0/12650 228 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:39.855922) v4 ==== 2004+0+0 (2535501339 0 0) 0x55d9b60df800 con 0x55d986d21800 > -8666> 2018-07-11 07:50:39.808965 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:39.855922) v4 -- 0x55d986d96400 con 0 > -8665> 2018-07-11 07:50:39.808994 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:39.855922) v4 -- 0x55d982cdfc00 con 0 > -8664> 2018-07-11 07:50:39.837112 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.107:0/12617 conn(0x55d98123d800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 220 0x55d9829e4800 osd_ping(ping e501 stamp 2018-07-11 07:50:39.836783) v4 > -8663> 2018-07-11 07:50:39.837115 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.107:0/12617 conn(0x55d98123c000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 220 0x55d986cd6a00 osd_ping(ping e501 stamp 2018-07-11 07:50:39.836783) v4 > -8662> 2018-07-11 07:50:39.837136 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.9 10.8.128.107:0/12617 220 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:39.836783) v4 ==== 2004+0+0 (813778548 0 0) 0x55d9829e4800 con 0x55d98123d800 > -8661> 2018-07-11 07:50:39.837141 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.9 10.8.128.107:0/12617 220 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:39.836783) v4 ==== 2004+0+0 (813778548 0 0) 0x55d986cd6a00 con 0x55d98123c000 > -8660> 2018-07-11 07:50:39.837148 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:39.836783) v4 -- 0x55d982cdfc00 con 0 > -8659> 2018-07-11 07:50:39.837179 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:39.836783) v4 -- 0x55d9ba819a00 con 0 > -8658> 2018-07-11 07:50:39.859095 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:751e7344:::benchmark_data_magna115_10867_object2804137:0,end:5:751edd0a:::benchmark_data_magna115_10867_object1652923:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb773180 con 0 > -8657> 2018-07-11 07:50:39.859144 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:751e7344:::benchmark_data_magna115_10867_object2804137:0,end:5:751edd0a:::benchmark_data_magna115_10867_object1652923:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb773400 con 0 > -8656> 2018-07-11 07:50:39.954204 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6805/15263 conn(0x55d986c41000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=31 cs=1 l=0). rx osd.5 seq 2788 0x55d986cd6a00 MOSDScrubReserve(6.15 REQUEST e501) v1 > -8655> 2018-07-11 07:50:39.954221 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.5 10.8.128.113:6805/15263 2788 ==== MOSDScrubReserve(6.15 REQUEST e501) v1 ==== 43+0+0 (3942742026 0 0) 0x55d986cd6a00 con 0x55d986c41000 > -8654> 2018-07-11 07:50:40.098115 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 804 0x55d9c2b20000 rep_scrubmap(5.2e e501 from shard 4) v1 > -8653> 2018-07-11 07:50:40.098136 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 804 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12478 (3686815630 0 1289664292) 0x55d9c2b20000 con 0x55d986bec000 > -8652> 2018-07-11 07:50:40.103206 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 810 0x55d9c2b20240 rep_scrubmap(5.2e e501 from shard 2) v1 > -8651> 2018-07-11 07:50:40.103227 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 810 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12478 (1269249638 0 1289664292) 0x55d9c2b20240 con 0x55d986bf0800 > -8650> 2018-07-11 07:50:40.158420 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12677 conn(0x55d986db4800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.1 seq 240 0x55d9bcd72c00 osd_ping(ping e501 stamp 2018-07-11 07:50:40.205448) v4 > -8649> 2018-07-11 07:50:40.158428 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12677 conn(0x55d986df2000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.1 seq 240 0x55d9bfbde200 osd_ping(ping e501 stamp 2018-07-11 07:50:40.205448) v4 > -8648> 2018-07-11 07:50:40.158444 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.1 10.8.128.106:0/12677 240 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:40.205448) v4 ==== 2004+0+0 (1839678588 0 0) 0x55d9bcd72c00 con 0x55d986db4800 > -8647> 2018-07-11 07:50:40.158446 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.1 10.8.128.106:0/12677 240 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:40.205448) v4 ==== 2004+0+0 (1839678588 0 0) 0x55d9bfbde200 con 0x55d986df2000 > -8646> 2018-07-11 07:50:40.158453 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:40.205448) v4 -- 0x55d986d96400 con 0 > -8645> 2018-07-11 07:50:40.158480 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:40.205448) v4 -- 0x55d9b4388400 con 0 > -8644> 2018-07-11 07:50:40.206470 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6805/15263 -- MOSDScrubReserve(6.15 REJECT e501) v1 -- 0x55d9829e4200 con 0 > -8643> 2018-07-11 07:50:40.206678 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -8642> 2018-07-11 07:50:40.206703 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -8641> 2018-07-11 07:50:40.206752 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -8640> 2018-07-11 07:50:40.206939 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6805/15263 conn(0x55d986c41000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=31 cs=1 l=0). rx osd.5 seq 2789 0x55d9829e4200 MOSDScrubReserve(6.15 RELEASE e501) v1 > -8639> 2018-07-11 07:50:40.206958 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.5 10.8.128.113:6805/15263 2789 ==== MOSDScrubReserve(6.15 RELEASE e501) v1 ==== 43+0+0 (605848072 0 0) 0x55d9829e4200 con 0x55d986c41000 > -8638> 2018-07-11 07:50:40.207209 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -8637> 2018-07-11 07:50:40.208324 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:751edd0a:::benchmark_data_magna115_10867_object1652923:0,end:5:751f4e59:::benchmark_data_magna115_10867_object2206780:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb773900 con 0 > -8636> 2018-07-11 07:50:40.208357 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:751edd0a:::benchmark_data_magna115_10867_object1652923:0,end:5:751f4e59:::benchmark_data_magna115_10867_object2206780:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb772f00 con 0 > -8635> 2018-07-11 07:50:40.405798 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 805 0x55d9c2b20480 rep_scrubmap(5.2e e501 from shard 4) v1 > -8634> 2018-07-11 07:50:40.405818 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 805 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12470 (3686815630 0 2940574055) 0x55d9c2b20480 con 0x55d986bec000 > -8633> 2018-07-11 07:50:40.416398 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12651 conn(0x55d986db1800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.2 seq 237 0x55d9ba819a00 osd_ping(ping e501 stamp 2018-07-11 07:50:40.463478) v4 > -8632> 2018-07-11 07:50:40.416422 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.2 10.8.128.106:0/12651 237 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:40.463478) v4 ==== 2004+0+0 (3038478811 0 0) 0x55d9ba819a00 con 0x55d986db1800 > -8631> 2018-07-11 07:50:40.416436 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:40.463478) v4 -- 0x55d9b4388400 con 0 > -8630> 2018-07-11 07:50:40.416443 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12651 conn(0x55d986db3000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.2 seq 237 0x55d9b60df800 osd_ping(ping e501 stamp 2018-07-11 07:50:40.463478) v4 > -8629> 2018-07-11 07:50:40.416466 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.2 10.8.128.106:0/12651 237 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:40.463478) v4 ==== 2004+0+0 (3038478811 0 0) 0x55d9b60df800 con 0x55d986db3000 > -8628> 2018-07-11 07:50:40.416478 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:40.463478) v4 -- 0x55d982cdfc00 con 0 > -8627> 2018-07-11 07:50:40.418299 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 811 0x55d9c2b206c0 rep_scrubmap(5.2e e501 from shard 2) v1 > -8626> 2018-07-11 07:50:40.418320 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 811 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12470 (1269249638 0 2940574055) 0x55d9c2b206c0 con 0x55d986bf0800 > -8625> 2018-07-11 07:50:40.451868 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -8624> 2018-07-11 07:50:40.451891 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -8623> 2018-07-11 07:50:40.451952 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -8622> 2018-07-11 07:50:40.452433 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -8621> 2018-07-11 07:50:40.500870 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:751f4e59:::benchmark_data_magna115_10867_object2206780:0,end:5:751fb3df:::benchmark_data_magna115_10867_object1337429:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb772780 con 0 > -8620> 2018-07-11 07:50:40.500916 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:751f4e59:::benchmark_data_magna115_10867_object2206780:0,end:5:751fb3df:::benchmark_data_magna115_10867_object1337429:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb772c80 con 0 > -8619> 2018-07-11 07:50:40.556303 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6805/15885 conn(0x55d986c7f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.7 seq 2762 0x55d9bfbde200 MOSDScrubReserve(5.4 REQUEST e501) v1 > -8618> 2018-07-11 07:50:40.556322 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.7 10.8.128.114:6805/15885 2762 ==== MOSDScrubReserve(5.4 REQUEST e501) v1 ==== 43+0+0 (1753221512 0 0) 0x55d9bfbde200 con 0x55d986c7f800 > -8617> 2018-07-11 07:50:40.556393 7ffab33df700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.114:6805/15885 -- MOSDScrubReserve(5.4 REJECT e501) v1 -- 0x55d9c74e1800 con 0 > -8616> 2018-07-11 07:50:40.556726 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6805/15885 conn(0x55d986c7f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.7 seq 2763 0x55d9c74e1800 MOSDScrubReserve(5.4 RELEASE e501) v1 > -8615> 2018-07-11 07:50:40.556740 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.7 10.8.128.114:6805/15885 2763 ==== MOSDScrubReserve(5.4 RELEASE e501) v1 ==== 43+0+0 (2812146570 0 0) 0x55d9c74e1800 con 0x55d986c7f800 > -8614> 2018-07-11 07:50:40.586071 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.114:0/15919 conn(0x55d986df5000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.11 seq 237 0x55d9829e5000 osd_ping(ping e501 stamp 2018-07-11 07:50:41.215981) v4 > -8613> 2018-07-11 07:50:40.586092 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.11 10.8.128.114:0/15919 237 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:41.215981) v4 ==== 2004+0+0 (1040140279 0 0) 0x55d9829e5000 con 0x55d986df5000 > -8612> 2018-07-11 07:50:40.586102 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:41.215981) v4 -- 0x55d986d96400 con 0 > -8611> 2018-07-11 07:50:40.586095 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.114:0/15919 conn(0x55d986df3800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.11 seq 237 0x55d9829e4800 osd_ping(ping e501 stamp 2018-07-11 07:50:41.215981) v4 > -8610> 2018-07-11 07:50:40.586122 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.11 10.8.128.114:0/15919 237 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:41.215981) v4 ==== 2004+0+0 (1040140279 0 0) 0x55d9829e4800 con 0x55d986df3800 > -8609> 2018-07-11 07:50:40.586139 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:41.215981) v4 -- 0x55d982cdfc00 con 0 > -8608> 2018-07-11 07:50:40.694360 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.115:0/13323 conn(0x55d986d7e000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 235 0x55d9b60df800 osd_ping(ping e501 stamp 2018-07-11 07:50:41.039638) v4 > -8607> 2018-07-11 07:50:40.694382 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.6 10.8.128.115:0/13323 235 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:41.039638) v4 ==== 2004+0+0 (1463611852 0 0) 0x55d9b60df800 con 0x55d986d7e000 > -8606> 2018-07-11 07:50:40.694391 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:41.039638) v4 -- 0x55d982cdfc00 con 0 > -8605> 2018-07-11 07:50:40.694396 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.115:0/13323 conn(0x55d986d24800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 235 0x55d9ba819a00 osd_ping(ping e501 stamp 2018-07-11 07:50:41.039638) v4 > -8604> 2018-07-11 07:50:40.694422 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.6 10.8.128.115:0/13323 235 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:41.039638) v4 ==== 2004+0+0 (1463611852 0 0) 0x55d9ba819a00 con 0x55d986d24800 > -8603> 2018-07-11 07:50:40.694433 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:41.039638) v4 -- 0x55d9bf805000 con 0 > -8602> 2018-07-11 07:50:40.730851 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 812 0x55d9c2b20900 rep_scrubmap(5.2e e501 from shard 2) v1 > -8601> 2018-07-11 07:50:40.730870 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 812 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12472 (1269249638 0 18673187) 0x55d9c2b20900 con 0x55d986bf0800 > -8600> 2018-07-11 07:50:40.731722 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 806 0x55d9c2b20b40 rep_scrubmap(5.2e e501 from shard 4) v1 > -8599> 2018-07-11 07:50:40.731742 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 806 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12472 (3686815630 0 18673187) 0x55d9c2b20b40 con 0x55d986bec000 > -8598> 2018-07-11 07:50:40.830801 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 2972 0x55d9829e4800 MOSDScrubReserve(11.3s5 REQUEST e501) v1 > -8597> 2018-07-11 07:50:40.830821 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 2972 ==== MOSDScrubReserve(11.3s5 REQUEST e501) v1 ==== 43+0+0 (3260170403 0 0) 0x55d9829e4800 con 0x55d986c7e000 > -8596> 2018-07-11 07:50:40.830899 7ffab3be0700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e501) v1 -- 0x55d9bf805a00 con 0 > -8595> 2018-07-11 07:50:40.831166 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 2973 0x55d982cdfc00 MOSDScrubReserve(11.3s5 RELEASE e501) v1 > -8594> 2018-07-11 07:50:40.831198 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 2973 ==== MOSDScrubReserve(11.3s5 RELEASE e501) v1 ==== 43+0+0 (223328929 0 0) 0x55d982cdfc00 con 0x55d986c7e000 > -8593> 2018-07-11 07:50:40.954393 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6805/15263 conn(0x55d986c41000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=31 cs=1 l=0). rx osd.5 seq 2790 0x55d9b4388400 MOSDScrubReserve(6.15 REQUEST e501) v1 > -8592> 2018-07-11 07:50:40.954411 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.5 10.8.128.113:6805/15263 2790 ==== MOSDScrubReserve(6.15 REQUEST e501) v1 ==== 43+0+0 (3942742026 0 0) 0x55d9b4388400 con 0x55d986c41000 > -8591> 2018-07-11 07:50:41.118163 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6805/15263 -- MOSDScrubReserve(6.15 REJECT e501) v1 -- 0x55d9829e4200 con 0 > -8590> 2018-07-11 07:50:41.118218 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -8589> 2018-07-11 07:50:41.118236 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -8588> 2018-07-11 07:50:41.118307 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -8587> 2018-07-11 07:50:41.118673 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6805/15263 conn(0x55d986c41000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=31 cs=1 l=0). rx osd.5 seq 2791 0x55d9829e4200 MOSDScrubReserve(6.15 RELEASE e501) v1 > -8586> 2018-07-11 07:50:41.118706 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.5 10.8.128.113:6805/15263 2791 ==== MOSDScrubReserve(6.15 RELEASE e501) v1 ==== 43+0+0 (605848072 0 0) 0x55d9829e4200 con 0x55d986c41000 > -8585> 2018-07-11 07:50:41.118788 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -8584> 2018-07-11 07:50:41.242411 7ffacbbd1700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15263 conn(0x55d985d44000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.5 seq 220 0x55d9b60df800 osd_ping(ping e501 stamp 2018-07-11 07:50:42.472531) v4 > -8583> 2018-07-11 07:50:41.242434 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 <== osd.5 10.8.128.113:0/15263 220 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:42.472531) v4 ==== 2004+0+0 (1090744269 0 0) 0x55d9b60df800 con 0x55d985d44000 > -8582> 2018-07-11 07:50:41.242426 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15263 conn(0x55d985d45800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.5 seq 220 0x55d9bcd72c00 osd_ping(ping e501 stamp 2018-07-11 07:50:42.472531) v4 > -8581> 2018-07-11 07:50:41.242444 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:42.472531) v4 -- 0x55d986d33e00 con 0 > -8580> 2018-07-11 07:50:41.242444 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.5 10.8.128.113:0/15263 220 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:42.472531) v4 ==== 2004+0+0 (1090744269 0 0) 0x55d9bcd72c00 con 0x55d985d45800 > -8579> 2018-07-11 07:50:41.242468 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:42.472531) v4 -- 0x55d986d96400 con 0 > -8578> 2018-07-11 07:50:41.282394 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.108:0/11967 conn(0x55d986db0000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.8 seq 218 0x55d9829e5000 osd_ping(ping e501 stamp 2018-07-11 07:50:41.037755) v4 > -8577> 2018-07-11 07:50:41.282396 7ffacbbd1700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.108:0/11967 conn(0x55d986d82800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.8 seq 218 0x55d9bf805a00 osd_ping(ping e501 stamp 2018-07-11 07:50:41.037755) v4 > -8576> 2018-07-11 07:50:41.282419 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.8 10.8.128.108:0/11967 218 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:41.037755) v4 ==== 2004+0+0 (3861141574 0 0) 0x55d9829e5000 con 0x55d986db0000 > -8575> 2018-07-11 07:50:41.282423 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 <== osd.8 10.8.128.108:0/11967 218 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:41.037755) v4 ==== 2004+0+0 (3861141574 0 0) 0x55d9bf805a00 con 0x55d986d82800 > -8574> 2018-07-11 07:50:41.282430 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:41.037755) v4 -- 0x55d986d96400 con 0 > -8573> 2018-07-11 07:50:41.282455 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:41.037755) v4 -- 0x55d986d33e00 con 0 > -8572> 2018-07-11 07:50:41.381206 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67489,epoch:501/498,start:5:751fb3df:::benchmark_data_magna115_10867_object1337429:0,end:5:75200abe:::benchmark_data_magna115_10867_object569032:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb772a00 con 0 > -8571> 2018-07-11 07:50:41.381242 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67489,epoch:501/498,start:5:751fb3df:::benchmark_data_magna115_10867_object1337429:0,end:5:75200abe:::benchmark_data_magna115_10867_object569032:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb772500 con 0 > -8570> 2018-07-11 07:50:41.518699 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 29] Generated table #36791: 1646954 keys, 44575869 bytes > -8569> 2018-07-11 07:50:41.518742 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295441518715, "cf_name": "default", "job": 29, "event": "table_file_creation", "file_number": 36791, "file_size": 44575869, "table_properties": {"data_size": 39958355, "index_size": 411007, "filter_size": 4205517, "raw_key_size": 47578671, "raw_average_key_size": 28, "raw_value_size": 17645072, "raw_average_value_size": 10, "num_data_blocks": 9817, "num_entries": 1646954, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "162790", "kMergeOperands": "0"}} > -8568> 2018-07-11 07:50:41.518769 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 29] Compacted 1@2 + 2@3 files to L3 => 180865003 bytes > -8567> 2018-07-11 07:50:41.536850 7ffabc3f1700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 52917360083 > -8566> 2018-07-11 07:50:41.537061 7ffabc3f1700 4 rocksdb: (Original Log Time 2018/07/11-07:50:41.536813) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 11 71 466 3641 796 0] max score 2.33, MB/sec: 11.6 rd, 10.7 wr, level 3, files in(1, 2) out(3) MB in(76.6, 111.1) out(172.5), read-write-amplify(4.7) write-amplify(2.3) OK, records in: 9079568, records dropped: 565084 > > -8565> 2018-07-11 07:50:41.537072 7ffabc3f1700 4 rocksdb: (Original Log Time 2018/07/11-07:50:41.536842) EVENT_LOG_v1 {"time_micros": 1531295441536828, "job": 29, "event": "compaction_finished", "compaction_time_micros": 16955808, "output_level": 3, "num_output_files": 3, "total_output_size": 180865003, "num_input_records": 9079568, "num_output_records": 8514484, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 11, 71, 466, 3641, 796, 0]} > -8564> 2018-07-11 07:50:41.537657 7ffabc3f1700 5 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:307] [JOB 29] Delete db.slow/036630.sst type=2 #36630 -- OK > > -8563> 2018-07-11 07:50:41.537669 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295441537665, "job": 29, "event": "table_file_deletion", "file_number": 36630} > -8562> 2018-07-11 07:50:41.537691 7ffabc3f1700 5 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:307] [JOB 29] Delete db.slow/029888.sst type=2 #29888 -- OK > > -8561> 2018-07-11 07:50:41.537697 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295441537695, "job": 29, "event": "table_file_deletion", "file_number": 29888} > -8560> 2018-07-11 07:50:41.537703 7ffabc3f1700 5 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:307] [JOB 29] Delete db.slow/029885.sst type=2 #29885 -- OK > > -8559> 2018-07-11 07:50:41.537708 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295441537706, "job": 29, "event": "table_file_deletion", "file_number": 29885} > -8558> 2018-07-11 07:50:41.553501 7ffabc3f1700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 52567428968 > -8557> 2018-07-11 07:50:41.553624 7ffabc3f1700 4 rocksdb: (Original Log Time 2018/07/11-07:50:41.537866) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #36632 to level-3 80348261 bytes > > -8556> 2018-07-11 07:50:41.553646 7ffabc3f1700 4 rocksdb: (Original Log Time 2018/07/11-07:50:41.553535) EVENT_LOG_v1 {"time_micros": 1531295441553517, "job": 30, "event": "trivial_move", "destination_level": 3, "files": 1, "total_files_size": 80348261} > -8555> 2018-07-11 07:50:41.553650 7ffabc3f1700 4 rocksdb: (Original Log Time 2018/07/11-07:50:41.553541) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-3 80348261 bytes OK: base level 1 max bytes base 268435456 files[0 11 70 467 3641 796 0] max score 2.27 > > -8554> 2018-07-11 07:50:41.554206 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 31] Compacting 1@2 + 10@3 files to L3, score 2.27 > -8553> 2018-07-11 07:50:41.554220 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 30 Base level 2, inputs: [36613(76MB)], [15298(670KB) 16595(70MB) 16598(71MB) 16599(71MB) 16604(71MB) 16605(71MB) 16606(71MB) 16607(39MB) 16609(71MB) 16613(1169B)] > > -8552> 2018-07-11 07:50:41.554264 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295441554223, "job": 31, "event": "compaction_started", "files_L2": [36613], "files_L3": [15298, 16595, 16598, 16599, 16604, 16605, 16606, 16607, 16609, 16613], "score": 2.26815, "input_data_size": 643980321} > -8551> 2018-07-11 07:50:41.593594 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 813 0x55d9c2b20d80 rep_scrubmap(5.2e e501 from shard 2) v1 > -8550> 2018-07-11 07:50:41.593615 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 813 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12470 (1269249638 0 2583274135) 0x55d9c2b20d80 con 0x55d986bf0800 > -8549> 2018-07-11 07:50:41.638189 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 807 0x55d9c2b20fc0 rep_scrubmap(5.2e e501 from shard 4) v1 > -8548> 2018-07-11 07:50:41.638209 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 807 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12470 (3686815630 0 2583274135) 0x55d9c2b20fc0 con 0x55d986bec000 > -8547> 2018-07-11 07:50:41.718722 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -8546> 2018-07-11 07:50:41.718747 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -8545> 2018-07-11 07:50:41.718805 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -8544> 2018-07-11 07:50:41.719280 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -8543> 2018-07-11 07:50:41.776256 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75200abe:::benchmark_data_magna115_10867_object569032:0,end:5:75206eb2:::benchmark_data_magna115_10867_object2503456:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb772000 con 0 > -8542> 2018-07-11 07:50:41.776306 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75200abe:::benchmark_data_magna115_10867_object569032:0,end:5:75206eb2:::benchmark_data_magna115_10867_object2503456:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98944b400 con 0 > -8541> 2018-07-11 07:50:41.994361 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 814 0x55d9c2b21200 rep_scrubmap(5.2e e501 from shard 2) v1 > -8540> 2018-07-11 07:50:41.994393 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 814 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12476 (1269249638 0 1469734350) 0x55d9c2b21200 con 0x55d986bf0800 > -8539> 2018-07-11 07:50:41.998662 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 808 0x55d9c2b21440 rep_scrubmap(5.2e e501 from shard 4) v1 > -8538> 2018-07-11 07:50:41.998682 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 808 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12476 (3686815630 0 1469734350) 0x55d9c2b21440 con 0x55d986bec000 > -8537> 2018-07-11 07:50:42.137776 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.107:0/12617 conn(0x55d98123c000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 221 0x55d9bf805000 osd_ping(ping e501 stamp 2018-07-11 07:50:42.137292) v4 > -8536> 2018-07-11 07:50:42.137801 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.9 10.8.128.107:0/12617 221 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:42.137292) v4 ==== 2004+0+0 (1475010724 0 0) 0x55d9bf805000 con 0x55d98123c000 > -8535> 2018-07-11 07:50:42.137815 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:42.137292) v4 -- 0x55d9bdba4800 con 0 > -8534> 2018-07-11 07:50:42.137807 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.107:0/12617 conn(0x55d98123d800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 221 0x55d9b60df800 osd_ping(ping e501 stamp 2018-07-11 07:50:42.137292) v4 > -8533> 2018-07-11 07:50:42.137825 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.9 10.8.128.107:0/12617 221 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:42.137292) v4 ==== 2004+0+0 (1475010724 0 0) 0x55d9b60df800 con 0x55d98123d800 > -8532> 2018-07-11 07:50:42.137839 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:42.137292) v4 -- 0x55d986d33e00 con 0 > -8531> 2018-07-11 07:50:42.211204 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -8530> 2018-07-11 07:50:42.211233 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -8529> 2018-07-11 07:50:42.211290 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -8528> 2018-07-11 07:50:42.211754 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -8527> 2018-07-11 07:50:42.243048 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67282,epoch:501/498,start:5:75206eb2:::benchmark_data_magna115_10867_object2503456:0,end:5:7520deac:::benchmark_data_magna115_10867_object324864:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb772280 con 0 > -8526> 2018-07-11 07:50:42.243086 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67282,epoch:501/498,start:5:75206eb2:::benchmark_data_magna115_10867_object2503456:0,end:5:7520deac:::benchmark_data_magna115_10867_object324864:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98944bb80 con 0 > -8525> 2018-07-11 07:50:42.301244 7ffab7be8700 10 monclient: tick > -8524> 2018-07-11 07:50:42.301317 7ffab7be8700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2018-07-11 07:50:12.301315) > -8523> 2018-07-11 07:50:42.347420 7ffab6be6700 1 -- 10.8.128.107:6800/13927 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55d9bd9ccdc0 con 0 > -8522> 2018-07-11 07:50:42.347489 7ffab6be6700 1 -- 10.8.128.107:6800/13927 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55d9869ca2c0 con 0 > -8521> 2018-07-11 07:50:42.425850 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 815 0x55d9c2b21680 rep_scrubmap(5.2e e501 from shard 2) v1 > -8520> 2018-07-11 07:50:42.425874 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 815 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12468 (1269249638 0 484489144) 0x55d9c2b21680 con 0x55d986bf0800 > -8519> 2018-07-11 07:50:42.437938 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 809 0x55d9c2b218c0 rep_scrubmap(5.2e e501 from shard 4) v1 > -8518> 2018-07-11 07:50:42.437959 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 809 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12468 (3686815630 0 484489144) 0x55d9c2b218c0 con 0x55d986bec000 > -8517> 2018-07-11 07:50:42.511868 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.114:0/15885 conn(0x55d986d7f800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.7 seq 240 0x55d9829e5000 osd_ping(ping e501 stamp 2018-07-11 07:50:43.141740) v4 > -8516> 2018-07-11 07:50:42.511893 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.7 10.8.128.114:0/15885 240 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:43.141740) v4 ==== 2004+0+0 (2694482588 0 0) 0x55d9829e5000 con 0x55d986d7f800 > -8515> 2018-07-11 07:50:42.511891 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.114:0/15885 conn(0x55d986d81000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.7 seq 240 0x55d9ba819a00 osd_ping(ping e501 stamp 2018-07-11 07:50:43.141740) v4 > -8514> 2018-07-11 07:50:42.511909 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:43.141740) v4 -- 0x55d986d96400 con 0 > -8513> 2018-07-11 07:50:42.511912 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.7 10.8.128.114:0/15885 240 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:43.141740) v4 ==== 2004+0+0 (2694482588 0 0) 0x55d9ba819a00 con 0x55d986d81000 > -8512> 2018-07-11 07:50:42.511938 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:43.141740) v4 -- 0x55d9bdba4800 con 0 > -8511> 2018-07-11 07:50:42.556674 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6805/15885 conn(0x55d986c7f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.7 seq 2764 0x55d9bf805000 MOSDScrubReserve(5.4 REQUEST e501) v1 > -8510> 2018-07-11 07:50:42.556692 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.7 10.8.128.114:6805/15885 2764 ==== MOSDScrubReserve(5.4 REQUEST e501) v1 ==== 43+0+0 (1753221512 0 0) 0x55d9bf805000 con 0x55d986c7f800 > -8509> 2018-07-11 07:50:42.556819 7ffab33df700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.114:6805/15885 -- MOSDScrubReserve(5.4 REJECT e501) v1 -- 0x55d9c74e1800 con 0 > -8508> 2018-07-11 07:50:42.557073 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6805/15885 conn(0x55d986c7f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.7 seq 2765 0x55d9bdba4800 MOSDScrubReserve(5.4 RELEASE e501) v1 > -8507> 2018-07-11 07:50:42.557088 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.7 10.8.128.114:6805/15885 2765 ==== MOSDScrubReserve(5.4 RELEASE e501) v1 ==== 43+0+0 (2812146570 0 0) 0x55d9bdba4800 con 0x55d986c7f800 > -8506> 2018-07-11 07:50:42.637239 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -8505> 2018-07-11 07:50:42.637277 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -8504> 2018-07-11 07:50:42.637337 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -8503> 2018-07-11 07:50:42.637804 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -8502> 2018-07-11 07:50:42.693435 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7520deac:::benchmark_data_magna115_10867_object324864:0,end:5:7521437b:::benchmark_data_magna115_10867_object956414:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98944b900 con 0 > -8501> 2018-07-11 07:50:42.693476 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7520deac:::benchmark_data_magna115_10867_object324864:0,end:5:7521437b:::benchmark_data_magna115_10867_object956414:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454500 con 0 > -8500> 2018-07-11 07:50:42.709324 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12650 conn(0x55d986d23000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 229 0x55d9bcd72c00 osd_ping(ping e501 stamp 2018-07-11 07:50:42.756372) v4 > -8499> 2018-07-11 07:50:42.709333 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12650 conn(0x55d986d21800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 229 0x55d9bf805a00 osd_ping(ping e501 stamp 2018-07-11 07:50:42.756372) v4 > -8498> 2018-07-11 07:50:42.709345 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.3 10.8.128.106:0/12650 229 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:42.756372) v4 ==== 2004+0+0 (1343908424 0 0) 0x55d9bcd72c00 con 0x55d986d23000 > -8497> 2018-07-11 07:50:42.709350 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.3 10.8.128.106:0/12650 229 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:42.756372) v4 ==== 2004+0+0 (1343908424 0 0) 0x55d9bf805a00 con 0x55d986d21800 > -8496> 2018-07-11 07:50:42.709355 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:42.756372) v4 -- 0x55d986d96400 con 0 > -8495> 2018-07-11 07:50:42.709378 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:42.756372) v4 -- 0x55d986d33e00 con 0 > -8494> 2018-07-11 07:50:42.776912 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15287 conn(0x55d981240800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 246 0x55d9c74e1800 osd_ping(ping e501 stamp 2018-07-11 07:50:44.006862) v4 > -8493> 2018-07-11 07:50:42.776933 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.4 10.8.128.113:0/15287 246 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:44.006862) v4 ==== 2004+0+0 (2865778753 0 0) 0x55d9c74e1800 con 0x55d981240800 > -8492> 2018-07-11 07:50:42.776926 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15287 conn(0x55d98123f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 246 0x55d9829e5000 osd_ping(ping e501 stamp 2018-07-11 07:50:44.006862) v4 > -8491> 2018-07-11 07:50:42.776942 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:44.006862) v4 -- 0x55d986cd5600 con 0 > -8490> 2018-07-11 07:50:42.776942 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.4 10.8.128.113:0/15287 246 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:44.006862) v4 ==== 2004+0+0 (2865778753 0 0) 0x55d9829e5000 con 0x55d98123f000 > -8489> 2018-07-11 07:50:42.776964 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:44.006862) v4 -- 0x55d986d96400 con 0 > -8488> 2018-07-11 07:50:42.831031 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 2974 0x55d9b60df800 MOSDScrubReserve(11.3s5 REQUEST e501) v1 > -8487> 2018-07-11 07:50:42.831049 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 2974 ==== MOSDScrubReserve(11.3s5 REQUEST e501) v1 ==== 43+0+0 (3260170403 0 0) 0x55d9b60df800 con 0x55d986c7e000 > -8486> 2018-07-11 07:50:42.831123 7ffab3be0700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e501) v1 -- 0x55d982cdfc00 con 0 > -8485> 2018-07-11 07:50:42.831503 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 2975 0x55d982cdfc00 MOSDScrubReserve(11.3s5 RELEASE e501) v1 > -8484> 2018-07-11 07:50:42.831534 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 2975 ==== MOSDScrubReserve(11.3s5 RELEASE e501) v1 ==== 43+0+0 (223328929 0 0) 0x55d982cdfc00 con 0x55d986c7e000 > -8483> 2018-07-11 07:50:42.915301 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 816 0x55d9c2b21b00 rep_scrubmap(5.2e e501 from shard 2) v1 > -8482> 2018-07-11 07:50:42.915321 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 816 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12472 (1269249638 0 1852370331) 0x55d9c2b21b00 con 0x55d986bf0800 > -8481> 2018-07-11 07:50:42.935855 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 810 0x55d9c2b21d40 rep_scrubmap(5.2e e501 from shard 4) v1 > -8480> 2018-07-11 07:50:42.935874 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 810 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12472 (3686815630 0 1852370331) 0x55d9c2b21d40 con 0x55d986bec000 > -8479> 2018-07-11 07:50:42.950418 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -8478> 2018-07-11 07:50:42.950442 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -8477> 2018-07-11 07:50:42.950501 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -8476> 2018-07-11 07:50:42.950947 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -8475> 2018-07-11 07:50:42.993261 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7521437b:::benchmark_data_magna115_10867_object956414:0,end:5:7521b6ca:::benchmark_data_magna115_10867_object1110212:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454c80 con 0 > -8474> 2018-07-11 07:50:42.993312 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7521437b:::benchmark_data_magna115_10867_object956414:0,end:5:7521b6ca:::benchmark_data_magna115_10867_object1110212:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454a00 con 0 > -8473> 2018-07-11 07:50:43.058799 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12677 conn(0x55d986df2000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.1 seq 241 0x55d9c74e1800 osd_ping(ping e501 stamp 2018-07-11 07:50:43.105871) v4 > -8472> 2018-07-11 07:50:43.058801 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12677 conn(0x55d986db4800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.1 seq 241 0x55d9bcd72c00 osd_ping(ping e501 stamp 2018-07-11 07:50:43.105871) v4 > -8471> 2018-07-11 07:50:43.058825 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.1 10.8.128.106:0/12677 241 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:43.105871) v4 ==== 2004+0+0 (880275689 0 0) 0x55d9c74e1800 con 0x55d986df2000 > -8470> 2018-07-11 07:50:43.058829 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.1 10.8.128.106:0/12677 241 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:43.105871) v4 ==== 2004+0+0 (880275689 0 0) 0x55d9bcd72c00 con 0x55d986db4800 > -8469> 2018-07-11 07:50:43.058835 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:43.105871) v4 -- 0x55d986cd5600 con 0 > -8468> 2018-07-11 07:50:43.058862 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:43.105871) v4 -- 0x55d986d96400 con 0 > -8467> 2018-07-11 07:50:43.193429 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 817 0x55d9c87bc000 rep_scrubmap(5.2e e501 from shard 2) v1 > -8466> 2018-07-11 07:50:43.193449 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 817 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12474 (1269249638 0 2745703042) 0x55d9c87bc000 con 0x55d986bf0800 > -8465> 2018-07-11 07:50:43.209775 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12650 conn(0x55d986d23000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 230 0x55d9829e5000 osd_ping(ping e501 stamp 2018-07-11 07:50:43.256799) v4 > -8464> 2018-07-11 07:50:43.209809 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.3 10.8.128.106:0/12650 230 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:43.256799) v4 ==== 2004+0+0 (1223987282 0 0) 0x55d9829e5000 con 0x55d986d23000 > -8463> 2018-07-11 07:50:43.209818 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:43.256799) v4 -- 0x55d986d96400 con 0 > -8462> 2018-07-11 07:50:43.209871 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12650 conn(0x55d986d21800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 230 0x55d9bf805a00 osd_ping(ping e501 stamp 2018-07-11 07:50:43.256799) v4 > -8461> 2018-07-11 07:50:43.209893 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.3 10.8.128.106:0/12650 230 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:43.256799) v4 ==== 2004+0+0 (1223987282 0 0) 0x55d9bf805a00 con 0x55d986d21800 > -8460> 2018-07-11 07:50:43.209904 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:43.256799) v4 -- 0x55d986b6dc00 con 0 > -8459> 2018-07-11 07:50:43.226165 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 811 0x55d9c87bc240 rep_scrubmap(5.2e e501 from shard 4) v1 > -8458> 2018-07-11 07:50:43.226185 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 811 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12474 (3686815630 0 2745703042) 0x55d9c87bc240 con 0x55d986bec000 > -8457> 2018-07-11 07:50:43.271567 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -8456> 2018-07-11 07:50:43.271604 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -8455> 2018-07-11 07:50:43.271662 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -8454> 2018-07-11 07:50:43.272107 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -8453> 2018-07-11 07:50:43.306798 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67680,epoch:501/498,start:5:7521b6ca:::benchmark_data_magna115_10867_object1110212:0,end:5:752209de:::benchmark_data_magna115_10867_object633459:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454780 con 0 > -8452> 2018-07-11 07:50:43.306849 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67680,epoch:501/498,start:5:7521b6ca:::benchmark_data_magna115_10867_object1110212:0,end:5:752209de:::benchmark_data_magna115_10867_object633459:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454f00 con 0 > -8451> 2018-07-11 07:50:43.411201 7ffab1bdc700 5 osd.10 501 heartbeat: osd_stat(788 GB used, 143 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist []) > -8450> 2018-07-11 07:50:43.411215 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6802/15255 -- osd_ping(ping e501 stamp 2018-07-11 07:50:43.411213) v4 -- 0x55d9b2783600 con 0 > -8449> 2018-07-11 07:50:43.411265 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6803/15255 -- osd_ping(ping e501 stamp 2018-07-11 07:50:43.411213) v4 -- 0x55d986d40000 con 0 > -8448> 2018-07-11 07:50:43.411285 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6806/12677 -- osd_ping(ping e501 stamp 2018-07-11 07:50:43.411213) v4 -- 0x55d9829e5e00 con 0 > -8447> 2018-07-11 07:50:43.411299 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6807/12677 -- osd_ping(ping e501 stamp 2018-07-11 07:50:43.411213) v4 -- 0x55d9c3382600 con 0 > -8446> 2018-07-11 07:50:43.411311 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6808/12651 -- osd_ping(ping e501 stamp 2018-07-11 07:50:43.411213) v4 -- 0x55d9b3db2800 con 0 > -8445> 2018-07-11 07:50:43.411322 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6810/12651 -- osd_ping(ping e501 stamp 2018-07-11 07:50:43.411213) v4 -- 0x55d9c3383600 con 0 > -8444> 2018-07-11 07:50:43.411334 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6809/12650 -- osd_ping(ping e501 stamp 2018-07-11 07:50:43.411213) v4 -- 0x55d986c22000 con 0 > -8443> 2018-07-11 07:50:43.411348 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6811/12650 -- osd_ping(ping e501 stamp 2018-07-11 07:50:43.411213) v4 -- 0x55d9bdba4200 con 0 > -8442> 2018-07-11 07:50:43.411369 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6810/15287 -- osd_ping(ping e501 stamp 2018-07-11 07:50:43.411213) v4 -- 0x55d9bf804400 con 0 > -8441> 2018-07-11 07:50:43.411381 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6811/15287 -- osd_ping(ping e501 stamp 2018-07-11 07:50:43.411213) v4 -- 0x55d986cd5400 con 0 > -8440> 2018-07-11 07:50:43.411395 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6806/15263 -- osd_ping(ping e501 stamp 2018-07-11 07:50:43.411213) v4 -- 0x55d986d96200 con 0 > -8439> 2018-07-11 07:50:43.411410 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6807/15263 -- osd_ping(ping e501 stamp 2018-07-11 07:50:43.411213) v4 -- 0x55d9c34cc000 con 0 > -8438> 2018-07-11 07:50:43.411423 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.115:6802/13323 -- osd_ping(ping e501 stamp 2018-07-11 07:50:43.411213) v4 -- 0x55d9bdba5e00 con 0 > -8437> 2018-07-11 07:50:43.411436 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.115:6803/13323 -- osd_ping(ping e501 stamp 2018-07-11 07:50:43.411213) v4 -- 0x55d9bdba5600 con 0 > -8436> 2018-07-11 07:50:43.411448 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6806/15885 -- osd_ping(ping e501 stamp 2018-07-11 07:50:43.411213) v4 -- 0x55d986b6d400 con 0 > -8435> 2018-07-11 07:50:43.411463 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6807/15885 -- osd_ping(ping e501 stamp 2018-07-11 07:50:43.411213) v4 -- 0x55d980f5be00 con 0 > -8434> 2018-07-11 07:50:43.411475 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.108:6802/11967 -- osd_ping(ping e501 stamp 2018-07-11 07:50:43.411213) v4 -- 0x55d986cd7600 con 0 > -8433> 2018-07-11 07:50:43.411486 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.108:6803/11967 -- osd_ping(ping e501 stamp 2018-07-11 07:50:43.411213) v4 -- 0x55d9869bec00 con 0 > -8432> 2018-07-11 07:50:43.411499 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.107:6806/12617 -- osd_ping(ping e501 stamp 2018-07-11 07:50:43.411213) v4 -- 0x55d9bdba5200 con 0 > -8431> 2018-07-11 07:50:43.411512 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.107:6807/12617 -- osd_ping(ping e501 stamp 2018-07-11 07:50:43.411213) v4 -- 0x55d9b67d9e00 con 0 > -8430> 2018-07-11 07:50:43.411525 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6802/15919 -- osd_ping(ping e501 stamp 2018-07-11 07:50:43.411213) v4 -- 0x55d986d97200 con 0 > -8429> 2018-07-11 07:50:43.411536 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6803/15919 -- osd_ping(ping e501 stamp 2018-07-11 07:50:43.411213) v4 -- 0x55d9c908fe00 con 0 > -8428> 2018-07-11 07:50:43.411686 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.107:6806/12617 conn(0x55d986bd2000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.9 seq 228 0x55d9829e5000 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:43.411213) v4 > -8427> 2018-07-11 07:50:43.411708 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.9 10.8.128.107:6806/12617 228 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:43.411213) v4 ==== 2004+0+0 (1170461600 0 0) 0x55d9829e5000 con 0x55d986bd2000 > -8426> 2018-07-11 07:50:43.411720 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.107:6807/12617 conn(0x55d986bd3800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.9 seq 228 0x55d986d33e00 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:43.411213) v4 > -8425> 2018-07-11 07:50:43.411741 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.9 10.8.128.107:6807/12617 228 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:43.411213) v4 ==== 2004+0+0 (1170461600 0 0) 0x55d986d33e00 con 0x55d986bd3800 > -8424> 2018-07-11 07:50:43.411798 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6806/12677 conn(0x55d986b58000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.1 seq 228 0x55d986d33e00 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:43.411213) v4 > -8423> 2018-07-11 07:50:43.411812 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.1 10.8.128.106:6806/12677 228 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:43.411213) v4 ==== 2004+0+0 (1170461600 0 0) 0x55d986d33e00 con 0x55d986b58000 > -8422> 2018-07-11 07:50:43.411798 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6808/12651 conn(0x55d986b9a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.2 seq 228 0x55d9c74e1800 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:43.411213) v4 > -8421> 2018-07-11 07:50:43.411824 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6810/12651 conn(0x55d986b9b800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.2 seq 228 0x55d9829e5000 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:43.411213) v4 > -8420> 2018-07-11 07:50:43.411838 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.2 10.8.128.106:6808/12651 228 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:43.411213) v4 ==== 2004+0+0 (1170461600 0 0) 0x55d9c74e1800 con 0x55d986b9a000 > -8419> 2018-07-11 07:50:43.411839 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.2 10.8.128.106:6810/12651 228 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:43.411213) v4 ==== 2004+0+0 (1170461600 0 0) 0x55d9829e5000 con 0x55d986b9b800 > -8418> 2018-07-11 07:50:43.411845 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6809/12650 conn(0x55d986bcb000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 228 0x55d986d33e00 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:43.411213) v4 > -8417> 2018-07-11 07:50:43.411876 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6803/15255 conn(0x55d986bc9800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.0 seq 228 0x55d9829e5000 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:43.411213) v4 > -8416> 2018-07-11 07:50:43.411855 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.3 10.8.128.106:6809/12650 228 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:43.411213) v4 ==== 2004+0+0 (1170461600 0 0) 0x55d986d33e00 con 0x55d986bcb000 > -8415> 2018-07-11 07:50:43.411880 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6802/15255 conn(0x55d986bc8000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.0 seq 228 0x55d9c74e1800 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:43.411213) v4 > -8414> 2018-07-11 07:50:43.411906 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.0 10.8.128.113:6803/15255 228 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:43.411213) v4 ==== 2004+0+0 (1170461600 0 0) 0x55d9829e5000 con 0x55d986bc9800 > -8413> 2018-07-11 07:50:43.411917 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.0 10.8.128.113:6802/15255 228 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:43.411213) v4 ==== 2004+0+0 (1170461600 0 0) 0x55d9c74e1800 con 0x55d986bc8000 > -8412> 2018-07-11 07:50:43.411943 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6811/15287 conn(0x55d986bb0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 228 0x55d9829e5000 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:43.411213) v4 > -8411> 2018-07-11 07:50:43.411951 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6810/15287 conn(0x55d986baf000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 228 0x55d9c74e1800 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:43.411213) v4 > -8410> 2018-07-11 07:50:43.411956 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.4 10.8.128.113:6811/15287 228 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:43.411213) v4 ==== 2004+0+0 (1170461600 0 0) 0x55d9829e5000 con 0x55d986bb0800 > -8409> 2018-07-11 07:50:43.411966 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.4 10.8.128.113:6810/15287 228 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:43.411213) v4 ==== 2004+0+0 (1170461600 0 0) 0x55d9c74e1800 con 0x55d986baf000 > -8408> 2018-07-11 07:50:43.411966 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6806/15263 conn(0x55d986b9d000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.5 seq 228 0x55d986d33e00 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:43.411213) v4 > -8407> 2018-07-11 07:50:43.411993 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.115:6802/13323 conn(0x55d986bc1000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 228 0x55d9829e5000 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:43.411213) v4 > -8406> 2018-07-11 07:50:43.411997 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6807/12677 conn(0x55d986b59800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.1 seq 228 0x55d9c74e1800 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:43.411213) v4 > -8405> 2018-07-11 07:50:43.412006 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.6 10.8.128.115:6802/13323 228 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:43.411213) v4 ==== 2004+0+0 (1170461600 0 0) 0x55d9829e5000 con 0x55d986bc1000 > -8404> 2018-07-11 07:50:43.412000 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.5 10.8.128.113:6806/15263 228 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:43.411213) v4 ==== 2004+0+0 (1170461600 0 0) 0x55d986d33e00 con 0x55d986b9d000 > -8403> 2018-07-11 07:50:43.412015 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.1 10.8.128.106:6807/12677 228 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:43.411213) v4 ==== 2004+0+0 (1170461600 0 0) 0x55d9c74e1800 con 0x55d986b59800 > -8402> 2018-07-11 07:50:43.412048 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6807/15263 conn(0x55d986b9e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.5 seq 228 0x55d9c74e1800 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:43.411213) v4 > -8401> 2018-07-11 07:50:43.412065 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.5 10.8.128.113:6807/15263 228 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:43.411213) v4 ==== 2004+0+0 (1170461600 0 0) 0x55d9c74e1800 con 0x55d986b9e800 > -8400> 2018-07-11 07:50:43.412076 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.115:6803/13323 conn(0x55d986bc2800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 228 0x55d986d33e00 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:43.411213) v4 > -8399> 2018-07-11 07:50:43.412092 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.6 10.8.128.115:6803/13323 228 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:43.411213) v4 ==== 2004+0+0 (1170461600 0 0) 0x55d986d33e00 con 0x55d986bc2800 > -8398> 2018-07-11 07:50:43.412104 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6811/12650 conn(0x55d986bcc800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 228 0x55d9c74e1800 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:43.411213) v4 > -8397> 2018-07-11 07:50:43.412117 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.3 10.8.128.106:6811/12650 228 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:43.411213) v4 ==== 2004+0+0 (1170461600 0 0) 0x55d9c74e1800 con 0x55d986bcc800 > -8396> 2018-07-11 07:50:43.412130 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6807/15885 conn(0x55d986bad800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.7 seq 228 0x55d986d33e00 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:43.411213) v4 > -8395> 2018-07-11 07:50:43.412129 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6802/15919 conn(0x55d986b5b000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.11 seq 228 0x55d9829e5000 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:43.411213) v4 > -8394> 2018-07-11 07:50:43.412143 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.7 10.8.128.114:6807/15885 228 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:43.411213) v4 ==== 2004+0+0 (1170461600 0 0) 0x55d986d33e00 con 0x55d986bad800 > -8393> 2018-07-11 07:50:43.412148 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.11 10.8.128.114:6802/15919 228 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:43.411213) v4 ==== 2004+0+0 (1170461600 0 0) 0x55d9829e5000 con 0x55d986b5b000 > -8392> 2018-07-11 07:50:43.412146 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.108:6803/11967 conn(0x55d986bbf800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.8 seq 228 0x55d9c74e1800 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:43.411213) v4 > -8391> 2018-07-11 07:50:43.412158 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.8 10.8.128.108:6803/11967 228 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:43.411213) v4 ==== 2004+0+0 (1170461600 0 0) 0x55d9c74e1800 con 0x55d986bbf800 > -8390> 2018-07-11 07:50:43.412173 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6803/15919 conn(0x55d986b5c800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.11 seq 228 0x55d986d33e00 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:43.411213) v4 > -8389> 2018-07-11 07:50:43.412173 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6806/15885 conn(0x55d986bac000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.7 seq 228 0x55d9829e5000 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:43.411213) v4 > -8388> 2018-07-11 07:50:43.412186 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.11 10.8.128.114:6803/15919 228 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:43.411213) v4 ==== 2004+0+0 (1170461600 0 0) 0x55d986d33e00 con 0x55d986b5c800 > -8387> 2018-07-11 07:50:43.412191 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.7 10.8.128.114:6806/15885 228 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:43.411213) v4 ==== 2004+0+0 (1170461600 0 0) 0x55d9829e5000 con 0x55d986bac000 > -8386> 2018-07-11 07:50:43.412213 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.108:6802/11967 conn(0x55d986bbe000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.8 seq 228 0x55d986d33e00 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:43.411213) v4 > -8385> 2018-07-11 07:50:43.412224 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.8 10.8.128.108:6802/11967 228 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:43.411213) v4 ==== 2004+0+0 (1170461600 0 0) 0x55d986d33e00 con 0x55d986bbe000 > -8384> 2018-07-11 07:50:43.508815 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 818 0x55d9c87bc480 rep_scrubmap(5.2e e501 from shard 2) v1 > -8383> 2018-07-11 07:50:43.508835 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 818 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12474 (1269249638 0 1540687417) 0x55d9c87bc480 con 0x55d986bf0800 > -8382> 2018-07-11 07:50:43.528530 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 812 0x55d9c87bc6c0 rep_scrubmap(5.2e e501 from shard 4) v1 > -8381> 2018-07-11 07:50:43.528547 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 812 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12474 (3686815630 0 1540687417) 0x55d9c87bc6c0 con 0x55d986bec000 > -8380> 2018-07-11 07:50:43.564025 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -8379> 2018-07-11 07:50:43.564049 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -8378> 2018-07-11 07:50:43.564108 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -8377> 2018-07-11 07:50:43.564570 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -8376> 2018-07-11 07:50:43.609992 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'66992,epoch:501/498,start:5:752209de:::benchmark_data_magna115_10867_object633459:0,end:5:75226850:::benchmark_data_magna115_10867_object1235034:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455180 con 0 > -8375> 2018-07-11 07:50:43.610044 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'66992,epoch:501/498,start:5:752209de:::benchmark_data_magna115_10867_object633459:0,end:5:75226850:::benchmark_data_magna115_10867_object1235034:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455400 con 0 > -8374> 2018-07-11 07:50:43.824480 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 813 0x55d9c87bc900 rep_scrubmap(5.2e e501 from shard 4) v1 > -8373> 2018-07-11 07:50:43.824503 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 813 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12466 (3686815630 0 2499009549) 0x55d9c87bc900 con 0x55d986bec000 > -8372> 2018-07-11 07:50:43.826235 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 819 0x55d9c87bcb40 rep_scrubmap(5.2e e501 from shard 2) v1 > -8371> 2018-07-11 07:50:43.826266 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 819 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12466 (1269249638 0 2499009549) 0x55d9c87bcb40 con 0x55d986bf0800 > -8370> 2018-07-11 07:50:43.950064 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -8369> 2018-07-11 07:50:43.950091 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -8368> 2018-07-11 07:50:43.950153 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -8367> 2018-07-11 07:50:43.950632 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -8366> 2018-07-11 07:50:43.968589 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'68015,epoch:501/498,start:5:75226850:::benchmark_data_magna115_10867_object1235034:0,end:5:7522c73f:::benchmark_data_magna115_10867_object2052381:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498280 con 0 > -8365> 2018-07-11 07:50:43.968641 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'68015,epoch:501/498,start:5:75226850:::benchmark_data_magna115_10867_object1235034:0,end:5:7522c73f:::benchmark_data_magna115_10867_object2052381:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455680 con 0 > -8364> 2018-07-11 07:50:44.171358 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 820 0x55d9c87bcd80 rep_scrubmap(5.2e e501 from shard 2) v1 > -8363> 2018-07-11 07:50:44.171378 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 820 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12476 (1269249638 0 3338303312) 0x55d9c87bcd80 con 0x55d986bf0800 > -8362> 2018-07-11 07:50:44.194775 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.115:0/13323 conn(0x55d986d7e000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 236 0x55d986d33e00 osd_ping(ping e501 stamp 2018-07-11 07:50:44.540088) v4 > -8361> 2018-07-11 07:50:44.194797 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.6 10.8.128.115:0/13323 236 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:44.540088) v4 ==== 2004+0+0 (3467467814 0 0) 0x55d986d33e00 con 0x55d986d7e000 > -8360> 2018-07-11 07:50:44.194806 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:44.540088) v4 -- 0x55d9c908fe00 con 0 > -8359> 2018-07-11 07:50:44.194835 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.115:0/13323 conn(0x55d986d24800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 236 0x55d9ba819a00 osd_ping(ping e501 stamp 2018-07-11 07:50:44.540088) v4 > -8358> 2018-07-11 07:50:44.194856 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.6 10.8.128.115:0/13323 236 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:44.540088) v4 ==== 2004+0+0 (3467467814 0 0) 0x55d9ba819a00 con 0x55d986d24800 > -8357> 2018-07-11 07:50:44.194867 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:44.540088) v4 -- 0x55d9869bec00 con 0 > -8356> 2018-07-11 07:50:44.199332 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 814 0x55d9c87bcfc0 rep_scrubmap(5.2e e501 from shard 4) v1 > -8355> 2018-07-11 07:50:44.199351 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 814 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12476 (3686815630 0 3338303312) 0x55d9c87bcfc0 con 0x55d986bec000 > -8354> 2018-07-11 07:50:44.271531 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -8353> 2018-07-11 07:50:44.271555 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -8352> 2018-07-11 07:50:44.271616 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -8351> 2018-07-11 07:50:44.272074 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -8350> 2018-07-11 07:50:44.304734 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7522c73f:::benchmark_data_magna115_10867_object2052381:0,end:5:75231490:::benchmark_data_magna115_10867_object1929343:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498000 con 0 > -8349> 2018-07-11 07:50:44.304784 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7522c73f:::benchmark_data_magna115_10867_object2052381:0,end:5:75231490:::benchmark_data_magna115_10867_object1929343:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455b80 con 0 > -8348> 2018-07-11 07:50:44.488413 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 821 0x55d9c87bd200 rep_scrubmap(5.2e e501 from shard 2) v1 > -8347> 2018-07-11 07:50:44.488434 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 821 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12470 (1269249638 0 2670678990) 0x55d9c87bd200 con 0x55d986bf0800 > -8346> 2018-07-11 07:50:44.524699 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 815 0x55d9c87bd440 rep_scrubmap(5.2e e501 from shard 4) v1 > -8345> 2018-07-11 07:50:44.524718 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 815 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12470 (3686815630 0 2670678990) 0x55d9c87bd440 con 0x55d986bec000 > -8344> 2018-07-11 07:50:44.562080 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -8343> 2018-07-11 07:50:44.562104 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -8342> 2018-07-11 07:50:44.562164 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -8341> 2018-07-11 07:50:44.562655 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -8340> 2018-07-11 07:50:44.602189 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75231490:::benchmark_data_magna115_10867_object1929343:0,end:5:752363d3:::benchmark_data_magna115_10867_object625883:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498780 con 0 > -8339> 2018-07-11 07:50:44.602241 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75231490:::benchmark_data_magna115_10867_object1929343:0,end:5:752363d3:::benchmark_data_magna115_10867_object625883:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455900 con 0 > -8338> 2018-07-11 07:50:44.810501 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 822 0x55d9c87bd680 rep_scrubmap(5.2e e501 from shard 2) v1 > -8337> 2018-07-11 07:50:44.810520 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 822 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12474 (1269249638 0 3317734851) 0x55d9c87bd680 con 0x55d986bf0800 > -8336> 2018-07-11 07:50:44.854882 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 816 0x55d9c87bd8c0 rep_scrubmap(5.2e e501 from shard 4) v1 > -8335> 2018-07-11 07:50:44.854903 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 816 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12474 (3686815630 0 3317734851) 0x55d9c87bd8c0 con 0x55d986bec000 > -8334> 2018-07-11 07:50:44.860431 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15255 conn(0x55d986c82800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.0 seq 237 0x55d9829e5000 osd_ping(ping e501 stamp 2018-07-11 07:50:46.090525) v4 > -8333> 2018-07-11 07:50:44.860432 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15255 conn(0x55d986d20000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.0 seq 237 0x55d9ba819a00 osd_ping(ping e501 stamp 2018-07-11 07:50:46.090525) v4 > -8332> 2018-07-11 07:50:44.860454 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.0 10.8.128.113:0/15255 237 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:46.090525) v4 ==== 2004+0+0 (2483429485 0 0) 0x55d9829e5000 con 0x55d986c82800 > -8331> 2018-07-11 07:50:44.860458 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.0 10.8.128.113:0/15255 237 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:46.090525) v4 ==== 2004+0+0 (2483429485 0 0) 0x55d9ba819a00 con 0x55d986d20000 > -8330> 2018-07-11 07:50:44.860464 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:46.090525) v4 -- 0x55d986d97200 con 0 > -8329> 2018-07-11 07:50:44.860489 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:46.090525) v4 -- 0x55d9869bec00 con 0 > -8328> 2018-07-11 07:50:44.903409 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -8327> 2018-07-11 07:50:44.903434 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -8326> 2018-07-11 07:50:44.903480 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -8325> 2018-07-11 07:50:44.903940 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -8324> 2018-07-11 07:50:44.904806 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'66902,epoch:501/498,start:5:752363d3:::benchmark_data_magna115_10867_object625883:0,end:5:7523e8c8:::benchmark_data_magna115_10867_object376999:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b14f0500 con 0 > -8323> 2018-07-11 07:50:44.904852 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'66902,epoch:501/498,start:5:752363d3:::benchmark_data_magna115_10867_object625883:0,end:5:7523e8c8:::benchmark_data_magna115_10867_object376999:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98696f400 con 0 > -8322> 2018-07-11 07:50:44.910290 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12650 conn(0x55d986d23000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 231 0x55d9bcd72c00 osd_ping(ping e501 stamp 2018-07-11 07:50:44.957371) v4 > -8321> 2018-07-11 07:50:44.910300 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12650 conn(0x55d986d21800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 231 0x55d9bf805a00 osd_ping(ping e501 stamp 2018-07-11 07:50:44.957371) v4 > -8320> 2018-07-11 07:50:44.910316 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.3 10.8.128.106:0/12650 231 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:44.957371) v4 ==== 2004+0+0 (310198475 0 0) 0x55d9bcd72c00 con 0x55d986d23000 > -8319> 2018-07-11 07:50:44.910319 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.3 10.8.128.106:0/12650 231 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:44.957371) v4 ==== 2004+0+0 (310198475 0 0) 0x55d9bf805a00 con 0x55d986d21800 > -8318> 2018-07-11 07:50:44.910333 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:44.957371) v4 -- 0x55d986d97200 con 0 > -8317> 2018-07-11 07:50:44.910364 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:44.957371) v4 -- 0x55d9c908fe00 con 0 > -8316> 2018-07-11 07:50:45.106446 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 817 0x55d9c87bdb00 rep_scrubmap(5.2e e501 from shard 4) v1 > -8315> 2018-07-11 07:50:45.106467 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 817 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12466 (3686815630 0 4029788050) 0x55d9c87bdb00 con 0x55d986bec000 > -8314> 2018-07-11 07:50:45.121101 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 823 0x55d9c87bdd40 rep_scrubmap(5.2e e501 from shard 2) v1 > -8313> 2018-07-11 07:50:45.121120 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 823 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12466 (1269249638 0 4029788050) 0x55d9c87bdd40 con 0x55d986bf0800 > -8312> 2018-07-11 07:50:45.199194 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -8311> 2018-07-11 07:50:45.199221 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -8310> 2018-07-11 07:50:45.199289 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -8309> 2018-07-11 07:50:45.199747 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -8308> 2018-07-11 07:50:45.219221 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7523e8c8:::benchmark_data_magna115_10867_object376999:0,end:5:75242c90:::benchmark_data_magna115_10867_object645783:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b14f0000 con 0 > -8307> 2018-07-11 07:50:45.219278 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7523e8c8:::benchmark_data_magna115_10867_object376999:0,end:5:75242c90:::benchmark_data_magna115_10867_object645783:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b14f0280 con 0 > -8306> 2018-07-11 07:50:45.382780 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.108:0/11967 conn(0x55d986db0000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.8 seq 219 0x55d9829e5000 osd_ping(ping e501 stamp 2018-07-11 07:50:45.138199) v4 > -8305> 2018-07-11 07:50:45.382803 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.8 10.8.128.108:0/11967 219 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:45.138199) v4 ==== 2004+0+0 (783819851 0 0) 0x55d9829e5000 con 0x55d986db0000 > -8304> 2018-07-11 07:50:45.382813 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:45.138199) v4 -- 0x55d986d97200 con 0 > -8303> 2018-07-11 07:50:45.382814 7ffacbbd1700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.108:0/11967 conn(0x55d986d82800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.8 seq 219 0x55d986d33e00 osd_ping(ping e501 stamp 2018-07-11 07:50:45.138199) v4 > -8302> 2018-07-11 07:50:45.382831 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 <== osd.8 10.8.128.108:0/11967 219 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:45.138199) v4 ==== 2004+0+0 (783819851 0 0) 0x55d986d33e00 con 0x55d986d82800 > -8301> 2018-07-11 07:50:45.382854 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:45.138199) v4 -- 0x55d9c908fe00 con 0 > -8300> 2018-07-11 07:50:45.421384 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 824 0x55d9be37c000 rep_scrubmap(5.2e e501 from shard 2) v1 > -8299> 2018-07-11 07:50:45.421403 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 824 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12470 (1269249638 0 1540780601) 0x55d9be37c000 con 0x55d986bf0800 > -8298> 2018-07-11 07:50:45.446741 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 818 0x55d9be37c240 rep_scrubmap(5.2e e501 from shard 4) v1 > -8297> 2018-07-11 07:50:45.446761 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 818 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12470 (3686815630 0 1540780601) 0x55d9be37c240 con 0x55d986bec000 > -8296> 2018-07-11 07:50:45.557114 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6805/15885 conn(0x55d986c7f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.7 seq 2766 0x55d9c74e1800 MOSDScrubReserve(5.4 REQUEST e501) v1 > -8295> 2018-07-11 07:50:45.557132 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.7 10.8.128.114:6805/15885 2766 ==== MOSDScrubReserve(5.4 REQUEST e501) v1 ==== 43+0+0 (1753221512 0 0) 0x55d9c74e1800 con 0x55d986c7f800 > -8294> 2018-07-11 07:50:45.557214 7ffab33df700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.114:6805/15885 -- MOSDScrubReserve(5.4 REJECT e501) v1 -- 0x55d9bdba4800 con 0 > -8293> 2018-07-11 07:50:45.557562 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6805/15885 conn(0x55d986c7f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.7 seq 2767 0x55d9bdba4800 MOSDScrubReserve(5.4 RELEASE e501) v1 > -8292> 2018-07-11 07:50:45.557576 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.7 10.8.128.114:6805/15885 2767 ==== MOSDScrubReserve(5.4 RELEASE e501) v1 ==== 43+0+0 (2812146570 0 0) 0x55d9bdba4800 con 0x55d986c7f800 > -8291> 2018-07-11 07:50:45.558900 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -8290> 2018-07-11 07:50:45.558927 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -8289> 2018-07-11 07:50:45.558987 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -8288> 2018-07-11 07:50:45.559466 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -8287> 2018-07-11 07:50:45.602512 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67805,epoch:501/498,start:5:75242c90:::benchmark_data_magna115_10867_object645783:0,end:5:7524897f:::benchmark_data_magna115_10867_object3128452:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1499900 con 0 > -8286> 2018-07-11 07:50:45.602552 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67805,epoch:501/498,start:5:75242c90:::benchmark_data_magna115_10867_object645783:0,end:5:7524897f:::benchmark_data_magna115_10867_object3128452:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1499680 con 0 > -8285> 2018-07-11 07:50:45.638324 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.107:0/12617 conn(0x55d98123c000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 222 0x55d9ba819a00 osd_ping(ping e501 stamp 2018-07-11 07:50:45.637980) v4 > -8284> 2018-07-11 07:50:45.638347 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.9 10.8.128.107:0/12617 222 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:45.637980) v4 ==== 2004+0+0 (768500 0 0) 0x55d9ba819a00 con 0x55d98123c000 > -8283> 2018-07-11 07:50:45.638356 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:45.637980) v4 -- 0x55d9c34cc000 con 0 > -8282> 2018-07-11 07:50:45.638357 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.107:0/12617 conn(0x55d98123d800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 222 0x55d9bf805a00 osd_ping(ping e501 stamp 2018-07-11 07:50:45.637980) v4 > -8281> 2018-07-11 07:50:45.638384 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.9 10.8.128.107:0/12617 222 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:45.637980) v4 ==== 2004+0+0 (768500 0 0) 0x55d9bf805a00 con 0x55d98123d800 > -8280> 2018-07-11 07:50:45.638403 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:45.637980) v4 -- 0x55d9c908fe00 con 0 > -8279> 2018-07-11 07:50:45.716882 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12651 conn(0x55d986db1800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.2 seq 238 0x55d9869bec00 osd_ping(ping e501 stamp 2018-07-11 07:50:45.763954) v4 > -8278> 2018-07-11 07:50:45.716887 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12651 conn(0x55d986db3000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.2 seq 238 0x55d986d33e00 osd_ping(ping e501 stamp 2018-07-11 07:50:45.763954) v4 > -8277> 2018-07-11 07:50:45.716910 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.2 10.8.128.106:0/12651 238 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:45.763954) v4 ==== 2004+0+0 (1576831513 0 0) 0x55d9869bec00 con 0x55d986db1800 > -8276> 2018-07-11 07:50:45.716914 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.2 10.8.128.106:0/12651 238 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:45.763954) v4 ==== 2004+0+0 (1576831513 0 0) 0x55d986d33e00 con 0x55d986db3000 > -8275> 2018-07-11 07:50:45.716928 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:45.763954) v4 -- 0x55d9c34cc000 con 0 > -8274> 2018-07-11 07:50:45.716956 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:45.763954) v4 -- 0x55d9c908fe00 con 0 > -8273> 2018-07-11 07:50:45.822500 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 825 0x55d9be37c480 rep_scrubmap(5.2e e501 from shard 2) v1 > -8272> 2018-07-11 07:50:45.822519 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 825 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12462 (1269249638 0 4211412117) 0x55d9be37c480 con 0x55d986bf0800 > -8271> 2018-07-11 07:50:45.836201 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 819 0x55d9be37c6c0 rep_scrubmap(5.2e e501 from shard 4) v1 > -8270> 2018-07-11 07:50:45.836219 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 819 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12462 (3686815630 0 4211412117) 0x55d9be37c6c0 con 0x55d986bec000 > -8269> 2018-07-11 07:50:45.886461 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.114:0/15919 conn(0x55d986df5000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.11 seq 238 0x55d9bcd72c00 osd_ping(ping e501 stamp 2018-07-11 07:50:46.516387) v4 > -8268> 2018-07-11 07:50:45.886484 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.11 10.8.128.114:0/15919 238 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:46.516387) v4 ==== 2004+0+0 (3175047950 0 0) 0x55d9bcd72c00 con 0x55d986df5000 > -8267> 2018-07-11 07:50:45.886495 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:46.516387) v4 -- 0x55d986d97200 con 0 > -8266> 2018-07-11 07:50:45.886497 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.114:0/15919 conn(0x55d986df3800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.11 seq 238 0x55d9bf805a00 osd_ping(ping e501 stamp 2018-07-11 07:50:46.516387) v4 > -8265> 2018-07-11 07:50:45.886522 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.11 10.8.128.114:0/15919 238 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:46.516387) v4 ==== 2004+0+0 (3175047950 0 0) 0x55d9bf805a00 con 0x55d986df3800 > -8264> 2018-07-11 07:50:45.886539 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:46.516387) v4 -- 0x55d9c908fe00 con 0 > -8263> 2018-07-11 07:50:45.895202 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.115:0/13323 conn(0x55d986d7e000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 237 0x55d986d33e00 osd_ping(ping e501 stamp 2018-07-11 07:50:46.240571) v4 > -8262> 2018-07-11 07:50:45.895223 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.6 10.8.128.115:0/13323 237 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:46.240571) v4 ==== 2004+0+0 (3556017606 0 0) 0x55d986d33e00 con 0x55d986d7e000 > -8261> 2018-07-11 07:50:45.895234 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:46.240571) v4 -- 0x55d9c908fe00 con 0 > -8260> 2018-07-11 07:50:45.895285 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.115:0/13323 conn(0x55d986d24800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 237 0x55d9ba819a00 osd_ping(ping e501 stamp 2018-07-11 07:50:46.240571) v4 > -8259> 2018-07-11 07:50:45.895305 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.6 10.8.128.115:0/13323 237 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:46.240571) v4 ==== 2004+0+0 (3556017606 0 0) 0x55d9ba819a00 con 0x55d986d24800 > -8258> 2018-07-11 07:50:45.895316 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:46.240571) v4 -- 0x55d9c34cc000 con 0 > -8257> 2018-07-11 07:50:45.896929 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -8256> 2018-07-11 07:50:45.896956 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -8255> 2018-07-11 07:50:45.897017 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -8254> 2018-07-11 07:50:45.897491 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -8253> 2018-07-11 07:50:45.898603 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67873,epoch:501/498,start:5:7524897f:::benchmark_data_magna115_10867_object3128452:0,end:5:752501d6:::benchmark_data_magna115_10867_object1312693:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1499b80 con 0 > -8252> 2018-07-11 07:50:45.898647 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67873,epoch:501/498,start:5:7524897f:::benchmark_data_magna115_10867_object3128452:0,end:5:752501d6:::benchmark_data_magna115_10867_object1312693:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498500 con 0 > -8251> 2018-07-11 07:50:45.956008 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6801/15919 conn(0x55d986c3f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=29 cs=1 l=0). rx osd.11 seq 210 0x55d9bf805a00 MOSDScrubReserve(10.f REQUEST e501) v1 > -8250> 2018-07-11 07:50:45.956028 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.11 10.8.128.114:6801/15919 210 ==== MOSDScrubReserve(10.f REQUEST e501) v1 ==== 43+0+0 (2892966580 0 0) 0x55d9bf805a00 con 0x55d986c3f800 > -8249> 2018-07-11 07:50:45.956097 7ffab53e3700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.114:6801/15919 -- MOSDScrubReserve(10.f REJECT e501) v1 -- 0x55d9bf805800 con 0 > -8248> 2018-07-11 07:50:45.956386 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6801/15919 conn(0x55d986c3f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=29 cs=1 l=0). rx osd.11 seq 211 0x55d9bf805800 MOSDScrubReserve(10.f RELEASE e501) v1 > -8247> 2018-07-11 07:50:45.956401 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.11 10.8.128.114:6801/15919 211 ==== MOSDScrubReserve(10.f RELEASE e501) v1 ==== 43+0+0 (1668468918 0 0) 0x55d9bf805800 con 0x55d986c3f800 > -8246> 2018-07-11 07:50:46.124523 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 826 0x55d9be37c900 rep_scrubmap(5.2e e501 from shard 2) v1 > -8245> 2018-07-11 07:50:46.124542 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 826 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12468 (1269249638 0 2863420082) 0x55d9be37c900 con 0x55d986bf0800 > -8244> 2018-07-11 07:50:46.132398 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 820 0x55d9be37cb40 rep_scrubmap(5.2e e501 from shard 4) v1 > -8243> 2018-07-11 07:50:46.132417 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 820 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12468 (3686815630 0 2863420082) 0x55d9be37cb40 con 0x55d986bec000 > -8242> 2018-07-11 07:50:46.138945 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.107:0/12617 conn(0x55d98123c000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 223 0x55d9ba819a00 osd_ping(ping e501 stamp 2018-07-11 07:50:46.138497) v4 > -8241> 2018-07-11 07:50:46.138964 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.107:0/12617 conn(0x55d98123d800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 223 0x55d986d33e00 osd_ping(ping e501 stamp 2018-07-11 07:50:46.138497) v4 > -8240> 2018-07-11 07:50:46.138969 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.9 10.8.128.107:0/12617 223 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:46.138497) v4 ==== 2004+0+0 (2323541050 0 0) 0x55d9ba819a00 con 0x55d98123c000 > -8239> 2018-07-11 07:50:46.138988 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.9 10.8.128.107:0/12617 223 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:46.138497) v4 ==== 2004+0+0 (2323541050 0 0) 0x55d986d33e00 con 0x55d98123d800 > -8238> 2018-07-11 07:50:46.138996 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:46.138497) v4 -- 0x55d9c34cc000 con 0 > -8237> 2018-07-11 07:50:46.139046 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:46.138497) v4 -- 0x55d9b67d9e00 con 0 > -8236> 2018-07-11 07:50:46.210365 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -8235> 2018-07-11 07:50:46.210392 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -8234> 2018-07-11 07:50:46.210439 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -8233> 2018-07-11 07:50:46.210898 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -8232> 2018-07-11 07:50:46.227774 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67256,epoch:501/498,start:5:752501d6:::benchmark_data_magna115_10867_object1312693:0,end:5:75255a8d:::benchmark_data_magna115_10867_object332334:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498c80 con 0 > -8231> 2018-07-11 07:50:46.227824 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67256,epoch:501/498,start:5:752501d6:::benchmark_data_magna115_10867_object1312693:0,end:5:75255a8d:::benchmark_data_magna115_10867_object332334:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1499400 con 0 > -8230> 2018-07-11 07:50:46.395684 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.115:0/13323 conn(0x55d986d7e000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 238 0x55d9c908fe00 osd_ping(ping e501 stamp 2018-07-11 07:50:46.741013) v4 > -8229> 2018-07-11 07:50:46.395706 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.6 10.8.128.115:0/13323 238 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:46.741013) v4 ==== 2004+0+0 (4139757362 0 0) 0x55d9c908fe00 con 0x55d986d7e000 > -8228> 2018-07-11 07:50:46.395715 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:46.741013) v4 -- 0x55d9b67d9e00 con 0 > -8227> 2018-07-11 07:50:46.395717 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.115:0/13323 conn(0x55d986d24800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 238 0x55d9ba819a00 osd_ping(ping e501 stamp 2018-07-11 07:50:46.741013) v4 > -8226> 2018-07-11 07:50:46.395742 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.6 10.8.128.115:0/13323 238 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:46.741013) v4 ==== 2004+0+0 (4139757362 0 0) 0x55d9ba819a00 con 0x55d986d24800 > -8225> 2018-07-11 07:50:46.395754 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:46.741013) v4 -- 0x55d9c34cc000 con 0 > -8224> 2018-07-11 07:50:46.422861 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 827 0x55d9be37cd80 rep_scrubmap(5.2e e501 from shard 2) v1 > -8223> 2018-07-11 07:50:46.422880 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 827 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12480 (1269249638 0 1790906431) 0x55d9be37cd80 con 0x55d986bf0800 > -8222> 2018-07-11 07:50:46.447000 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 821 0x55d9be37cfc0 rep_scrubmap(5.2e e501 from shard 4) v1 > -8221> 2018-07-11 07:50:46.447019 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 821 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12480 (3686815630 0 1790906431) 0x55d9be37cfc0 con 0x55d986bec000 > -8220> 2018-07-11 07:50:46.542769 7ffacbbd1700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15263 conn(0x55d985d44000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.5 seq 221 0x55d986d33e00 osd_ping(ping e501 stamp 2018-07-11 07:50:47.772966) v4 > -8219> 2018-07-11 07:50:46.542777 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15263 conn(0x55d985d45800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.5 seq 221 0x55d9829e5000 osd_ping(ping e501 stamp 2018-07-11 07:50:47.772966) v4 > -8218> 2018-07-11 07:50:46.542800 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 <== osd.5 10.8.128.113:0/15263 221 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:47.772966) v4 ==== 2004+0+0 (3531062615 0 0) 0x55d986d33e00 con 0x55d985d44000 > -8217> 2018-07-11 07:50:46.542804 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.5 10.8.128.113:0/15263 221 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:47.772966) v4 ==== 2004+0+0 (3531062615 0 0) 0x55d9829e5000 con 0x55d985d45800 > -8216> 2018-07-11 07:50:46.542812 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:47.772966) v4 -- 0x55d9b67d9e00 con 0 > -8215> 2018-07-11 07:50:46.542838 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:47.772966) v4 -- 0x55d986d97200 con 0 > -8214> 2018-07-11 07:50:46.557226 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6805/15885 conn(0x55d986c7f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.7 seq 2768 0x55d9869bec00 MOSDScrubReserve(5.4 REQUEST e501) v1 > -8213> 2018-07-11 07:50:46.557244 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.7 10.8.128.114:6805/15885 2768 ==== MOSDScrubReserve(5.4 REQUEST e501) v1 ==== 43+0+0 (1753221512 0 0) 0x55d9869bec00 con 0x55d986c7f800 > -8212> 2018-07-11 07:50:46.557308 7ffab33df700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.114:6805/15885 -- MOSDScrubReserve(5.4 REJECT e501) v1 -- 0x55d9bdba4800 con 0 > -8211> 2018-07-11 07:50:46.557639 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6805/15885 conn(0x55d986c7f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.7 seq 2769 0x55d9bdba4800 MOSDScrubReserve(5.4 RELEASE e501) v1 > -8210> 2018-07-11 07:50:46.557654 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.7 10.8.128.114:6805/15885 2769 ==== MOSDScrubReserve(5.4 RELEASE e501) v1 ==== 43+0+0 (2812146570 0 0) 0x55d9bdba4800 con 0x55d986c7f800 > -8209> 2018-07-11 07:50:46.955084 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6805/15263 conn(0x55d986c41000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=31 cs=1 l=0). rx osd.5 seq 2792 0x55d9ba819a00 MOSDScrubReserve(6.15 REQUEST e501) v1 > -8208> 2018-07-11 07:50:46.955102 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.5 10.8.128.113:6805/15263 2792 ==== MOSDScrubReserve(6.15 REQUEST e501) v1 ==== 43+0+0 (3942742026 0 0) 0x55d9ba819a00 con 0x55d986c41000 > -8207> 2018-07-11 07:50:47.347674 7ffab6be6700 1 -- 10.8.128.107:6800/13927 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55d9bd9cd600 con 0 > -8206> 2018-07-11 07:50:47.347743 7ffab6be6700 1 -- 10.8.128.107:6800/13927 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55d9b3fdf340 con 0 > -8205> 2018-07-11 07:50:47.586926 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.114:0/15919 conn(0x55d986df3800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.11 seq 239 0x55d9c908fe00 osd_ping(ping e501 stamp 2018-07-11 07:50:48.216848) v4 > -8204> 2018-07-11 07:50:47.586948 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.11 10.8.128.114:0/15919 239 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:48.216848) v4 ==== 2004+0+0 (3025653260 0 0) 0x55d9c908fe00 con 0x55d986df3800 > -8203> 2018-07-11 07:50:47.586973 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:48.216848) v4 -- 0x55d9b67d9e00 con 0 > -8202> 2018-07-11 07:50:47.586968 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.114:0/15919 conn(0x55d986df5000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.11 seq 239 0x55d9bcd72c00 osd_ping(ping e501 stamp 2018-07-11 07:50:48.216848) v4 > -8201> 2018-07-11 07:50:47.586986 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.11 10.8.128.114:0/15919 239 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:48.216848) v4 ==== 2004+0+0 (3025653260 0 0) 0x55d9bcd72c00 con 0x55d986df5000 > -8200> 2018-07-11 07:50:47.586999 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:48.216848) v4 -- 0x55d986d97200 con 0 > -8199> 2018-07-11 07:50:47.622507 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6805/15263 -- MOSDScrubReserve(6.15 REJECT e501) v1 -- 0x55d9829e4200 con 0 > -8198> 2018-07-11 07:50:47.622566 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -8197> 2018-07-11 07:50:47.622591 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -8196> 2018-07-11 07:50:47.622640 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -8195> 2018-07-11 07:50:47.622976 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6805/15263 conn(0x55d986c41000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=31 cs=1 l=0). rx osd.5 seq 2793 0x55d9829e4200 MOSDScrubReserve(6.15 RELEASE e501) v1 > -8194> 2018-07-11 07:50:47.623010 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.5 10.8.128.113:6805/15263 2793 ==== MOSDScrubReserve(6.15 RELEASE e501) v1 ==== 43+0+0 (605848072 0 0) 0x55d9829e4200 con 0x55d986c41000 > -8193> 2018-07-11 07:50:47.623123 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -8192> 2018-07-11 07:50:47.732232 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 31] Generated table #36792: 2678207 keys, 74643591 bytes > -8191> 2018-07-11 07:50:47.732281 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295447732254, "cf_name": "default", "job": 31, "event": "table_file_creation", "file_number": 36792, "file_size": 74643591, "table_properties": {"data_size": 67109414, "index_size": 689949, "filter_size": 6843236, "raw_key_size": 77370428, "raw_average_key_size": 28, "raw_value_size": 30822416, "raw_average_value_size": 11, "num_data_blocks": 16489, "num_entries": 2678207, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "85667", "kMergeOperands": "0"}} > -8190> 2018-07-11 07:50:47.736094 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75255a8d:::benchmark_data_magna115_10867_object332334:0,end:5:7525cdca:::benchmark_data_magna115_11707_object7927:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1499180 con 0 > -8189> 2018-07-11 07:50:47.736151 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75255a8d:::benchmark_data_magna115_10867_object332334:0,end:5:7525cdca:::benchmark_data_magna115_11707_object7927:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498f00 con 0 > -8188> 2018-07-11 07:50:47.759173 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12677 conn(0x55d986df2000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.1 seq 242 0x55d9bf804400 osd_ping(ping e501 stamp 2018-07-11 07:50:47.806281) v4 > -8187> 2018-07-11 07:50:47.759194 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.1 10.8.128.106:0/12677 242 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:47.806281) v4 ==== 2004+0+0 (4208119011 0 0) 0x55d9bf804400 con 0x55d986df2000 > -8186> 2018-07-11 07:50:47.759186 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12677 conn(0x55d986db4800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.1 seq 242 0x55d9829e5000 osd_ping(ping e501 stamp 2018-07-11 07:50:47.806281) v4 > -8185> 2018-07-11 07:50:47.759210 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:47.806281) v4 -- 0x55d9bdba4200 con 0 > -8184> 2018-07-11 07:50:47.759217 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.1 10.8.128.106:0/12677 242 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:47.806281) v4 ==== 2004+0+0 (4208119011 0 0) 0x55d9829e5000 con 0x55d986db4800 > -8183> 2018-07-11 07:50:47.759231 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:47.806281) v4 -- 0x55d986d97200 con 0 > -8182> 2018-07-11 07:50:47.812194 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.114:0/15885 conn(0x55d986d81000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.7 seq 241 0x55d9c34cc000 osd_ping(ping e501 stamp 2018-07-11 07:50:48.442147) v4 > -8181> 2018-07-11 07:50:47.812215 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.7 10.8.128.114:0/15885 241 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:48.442147) v4 ==== 2004+0+0 (675215578 0 0) 0x55d9c34cc000 con 0x55d986d81000 > -8180> 2018-07-11 07:50:47.812225 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:48.442147) v4 -- 0x55d9bdba4200 con 0 > -8179> 2018-07-11 07:50:47.812237 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.114:0/15885 conn(0x55d986d7f800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.7 seq 241 0x55d9bcd72c00 osd_ping(ping e501 stamp 2018-07-11 07:50:48.442147) v4 > -8178> 2018-07-11 07:50:47.812267 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.7 10.8.128.114:0/15885 241 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:48.442147) v4 ==== 2004+0+0 (675215578 0 0) 0x55d9bcd72c00 con 0x55d986d7f800 > -8177> 2018-07-11 07:50:47.812283 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:48.442147) v4 -- 0x55d986d97200 con 0 > -8176> 2018-07-11 07:50:47.831638 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 2976 0x55d986d33e00 MOSDScrubReserve(11.3s5 REQUEST e501) v1 > -8175> 2018-07-11 07:50:47.831657 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 2976 ==== MOSDScrubReserve(11.3s5 REQUEST e501) v1 ==== 43+0+0 (3260170403 0 0) 0x55d986d33e00 con 0x55d986c7e000 > -8174> 2018-07-11 07:50:47.831728 7ffab3be0700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e501) v1 -- 0x55d982cdfc00 con 0 > -8173> 2018-07-11 07:50:47.832092 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 2977 0x55d982cdfc00 MOSDScrubReserve(11.3s5 RELEASE e501) v1 > -8172> 2018-07-11 07:50:47.832109 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 2977 ==== MOSDScrubReserve(11.3s5 RELEASE e501) v1 ==== 43+0+0 (223328929 0 0) 0x55d982cdfc00 con 0x55d986c7e000 > -8171> 2018-07-11 07:50:47.955330 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6805/15263 conn(0x55d986c41000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=31 cs=1 l=0). rx osd.5 seq 2794 0x55d9bf804400 MOSDScrubReserve(6.15 REQUEST e501) v1 > -8170> 2018-07-11 07:50:47.955347 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.5 10.8.128.113:6805/15263 2794 ==== MOSDScrubReserve(6.15 REQUEST e501) v1 ==== 43+0+0 (3942742026 0 0) 0x55d9bf804400 con 0x55d986c41000 > -8169> 2018-07-11 07:50:47.965861 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 822 0x55d9be37d200 rep_scrubmap(5.2e e501 from shard 4) v1 > -8168> 2018-07-11 07:50:47.965881 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 822 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12464 (3686815630 0 2872542442) 0x55d9be37d200 con 0x55d986bec000 > -8167> 2018-07-11 07:50:47.969381 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 828 0x55d9be37d440 rep_scrubmap(5.2e e501 from shard 2) v1 > -8166> 2018-07-11 07:50:47.969400 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 828 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12464 (1269249638 0 2872542442) 0x55d9be37d440 con 0x55d986bf0800 > -8165> 2018-07-11 07:50:48.172891 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6805/15263 -- MOSDScrubReserve(6.15 REJECT e501) v1 -- 0x55d9829e4200 con 0 > -8164> 2018-07-11 07:50:48.173126 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -8163> 2018-07-11 07:50:48.173153 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -8162> 2018-07-11 07:50:48.173203 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -8161> 2018-07-11 07:50:48.173488 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6805/15263 conn(0x55d986c41000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=31 cs=1 l=0). rx osd.5 seq 2795 0x55d9829e4200 MOSDScrubReserve(6.15 RELEASE e501) v1 > -8160> 2018-07-11 07:50:48.173508 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.5 10.8.128.113:6805/15263 2795 ==== MOSDScrubReserve(6.15 RELEASE e501) v1 ==== 43+0+0 (605848072 0 0) 0x55d9829e4200 con 0x55d986c41000 > -8159> 2018-07-11 07:50:48.173676 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -8158> 2018-07-11 07:50:48.193329 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 31] Generated table #36793: 44107 keys, 1247013 bytes > -8157> 2018-07-11 07:50:48.193367 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295448193342, "cf_name": "default", "job": 31, "event": "table_file_creation", "file_number": 36793, "file_size": 1247013, "table_properties": {"data_size": 1122028, "index_size": 11268, "filter_size": 112737, "raw_key_size": 1274204, "raw_average_key_size": 28, "raw_value_size": 524385, "raw_average_value_size": 11, "num_data_blocks": 276, "num_entries": 44107, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} > -8156> 2018-07-11 07:50:48.203013 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67165,epoch:501/498,start:5:7525cdca:::benchmark_data_magna115_11707_object7927:0,end:5:75263546:::benchmark_data_magna115_10867_object2544438:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98696ec80 con 0 > -8155> 2018-07-11 07:50:48.203065 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67165,epoch:501/498,start:5:7525cdca:::benchmark_data_magna115_11707_object7927:0,end:5:75263546:::benchmark_data_magna115_10867_object2544438:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98696f680 con 0 > -8154> 2018-07-11 07:50:48.417882 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 823 0x55d9be37d680 rep_scrubmap(5.2e e501 from shard 4) v1 > -8153> 2018-07-11 07:50:48.417903 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 823 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12468 (3686815630 0 2029420248) 0x55d9be37d680 con 0x55d986bec000 > -8152> 2018-07-11 07:50:48.418109 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 829 0x55d9be37d8c0 rep_scrubmap(5.2e e501 from shard 2) v1 > -8151> 2018-07-11 07:50:48.418128 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 829 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12468 (1269249638 0 2029420248) 0x55d9be37d8c0 con 0x55d986bf0800 > -8150> 2018-07-11 07:50:48.439496 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.107:0/12617 conn(0x55d98123c000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 224 0x55d9bdba4200 osd_ping(ping e501 stamp 2018-07-11 07:50:48.439128) v4 > -8149> 2018-07-11 07:50:48.439501 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.107:0/12617 conn(0x55d98123d800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 224 0x55d9c908fe00 osd_ping(ping e501 stamp 2018-07-11 07:50:48.439128) v4 > -8148> 2018-07-11 07:50:48.439521 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.9 10.8.128.107:0/12617 224 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:48.439128) v4 ==== 2004+0+0 (1330026318 0 0) 0x55d9bdba4200 con 0x55d98123c000 > -8147> 2018-07-11 07:50:48.439525 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.9 10.8.128.107:0/12617 224 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:48.439128) v4 ==== 2004+0+0 (1330026318 0 0) 0x55d9c908fe00 con 0x55d98123d800 > -8146> 2018-07-11 07:50:48.439533 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:48.439128) v4 -- 0x55d9b3db2800 con 0 > -8145> 2018-07-11 07:50:48.439557 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:48.439128) v4 -- 0x55d986cd7600 con 0 > -8144> 2018-07-11 07:50:48.476882 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -8143> 2018-07-11 07:50:48.476908 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -8142> 2018-07-11 07:50:48.476968 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -8141> 2018-07-11 07:50:48.477450 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -8140> 2018-07-11 07:50:48.527951 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'66807,epoch:501/498,start:5:75263546:::benchmark_data_magna115_10867_object2544438:0,end:5:75268c72:::benchmark_data_magna115_10867_object2673186:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498a00 con 0 > -8139> 2018-07-11 07:50:48.528002 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'66807,epoch:501/498,start:5:75263546:::benchmark_data_magna115_10867_object2544438:0,end:5:75268c72:::benchmark_data_magna115_10867_object2673186:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b14f0a00 con 0 > -8138> 2018-07-11 07:50:48.677310 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15287 conn(0x55d981240800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 247 0x55d9bdba4200 osd_ping(ping e501 stamp 2018-07-11 07:50:49.907469) v4 > -8137> 2018-07-11 07:50:48.677331 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.4 10.8.128.113:0/15287 247 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:49.907469) v4 ==== 2004+0+0 (3437070183 0 0) 0x55d9bdba4200 con 0x55d981240800 > -8136> 2018-07-11 07:50:48.677340 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:49.907469) v4 -- 0x55d9b3db2800 con 0 > -8135> 2018-07-11 07:50:48.677334 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15287 conn(0x55d98123f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 247 0x55d9829e5000 osd_ping(ping e501 stamp 2018-07-11 07:50:49.907469) v4 > -8134> 2018-07-11 07:50:48.677352 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.4 10.8.128.113:0/15287 247 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:49.907469) v4 ==== 2004+0+0 (3437070183 0 0) 0x55d9829e5000 con 0x55d98123f000 > -8133> 2018-07-11 07:50:48.677364 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:49.907469) v4 -- 0x55d986d97200 con 0 > -8132> 2018-07-11 07:50:48.696062 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.115:0/13323 conn(0x55d986d7e000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 239 0x55d9b67d9e00 osd_ping(ping e501 stamp 2018-07-11 07:50:49.041462) v4 > -8131> 2018-07-11 07:50:48.696083 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.6 10.8.128.115:0/13323 239 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:49.041462) v4 ==== 2004+0+0 (3131489130 0 0) 0x55d9b67d9e00 con 0x55d986d7e000 > -8130> 2018-07-11 07:50:48.696099 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:49.041462) v4 -- 0x55d986cd7600 con 0 > -8129> 2018-07-11 07:50:48.696153 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.115:0/13323 conn(0x55d986d24800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 239 0x55d9c34cc000 osd_ping(ping e501 stamp 2018-07-11 07:50:49.041462) v4 > -8128> 2018-07-11 07:50:48.696178 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.6 10.8.128.115:0/13323 239 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:49.041462) v4 ==== 2004+0+0 (3131489130 0 0) 0x55d9c34cc000 con 0x55d986d24800 > -8127> 2018-07-11 07:50:48.696189 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:49.041462) v4 -- 0x55d9b3db2800 con 0 > -8126> 2018-07-11 07:50:48.748357 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 830 0x55d9be37db00 rep_scrubmap(5.2e e501 from shard 2) v1 > -8125> 2018-07-11 07:50:48.748377 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 830 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12478 (1269249638 0 872999878) 0x55d9be37db00 con 0x55d986bf0800 > -8124> 2018-07-11 07:50:48.754096 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 824 0x55d9be37dd40 rep_scrubmap(5.2e e501 from shard 4) v1 > -8123> 2018-07-11 07:50:48.754116 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 824 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12478 (3686815630 0 872999878) 0x55d9be37dd40 con 0x55d986bec000 > -8122> 2018-07-11 07:50:48.831843 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 2978 0x55d9c908fe00 MOSDScrubReserve(11.3s5 REQUEST e501) v1 > -8121> 2018-07-11 07:50:48.831862 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 2978 ==== MOSDScrubReserve(11.3s5 REQUEST e501) v1 ==== 43+0+0 (3260170403 0 0) 0x55d9c908fe00 con 0x55d986c7e000 > -8120> 2018-07-11 07:50:48.831935 7ffab3be0700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e501) v1 -- 0x55d982cdfc00 con 0 > -8119> 2018-07-11 07:50:48.832344 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 2979 0x55d982cdfc00 MOSDScrubReserve(11.3s5 RELEASE e501) v1 > -8118> 2018-07-11 07:50:48.832362 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 2979 ==== MOSDScrubReserve(11.3s5 RELEASE e501) v1 ==== 43+0+0 (223328929 0 0) 0x55d982cdfc00 con 0x55d986c7e000 > -8117> 2018-07-11 07:50:48.860488 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -8116> 2018-07-11 07:50:48.860522 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -8115> 2018-07-11 07:50:48.860601 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -8114> 2018-07-11 07:50:48.861064 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -8113> 2018-07-11 07:50:48.911540 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67691,epoch:501/498,start:5:75268c72:::benchmark_data_magna115_10867_object2673186:0,end:5:7526e005:::benchmark_data_magna115_10867_object2825798:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98696ef00 con 0 > -8112> 2018-07-11 07:50:48.911590 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67691,epoch:501/498,start:5:75268c72:::benchmark_data_magna115_10867_object2673186:0,end:5:7526e005:::benchmark_data_magna115_10867_object2825798:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98696f900 con 0 > -8111> 2018-07-11 07:50:48.912574 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.114:0/15885 conn(0x55d986d81000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.7 seq 242 0x55d9c34cc000 osd_ping(ping e501 stamp 2018-07-11 07:50:49.542565) v4 > -8110> 2018-07-11 07:50:48.912600 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.7 10.8.128.114:0/15885 242 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:49.542565) v4 ==== 2004+0+0 (52713147 0 0) 0x55d9c34cc000 con 0x55d986d81000 > -8109> 2018-07-11 07:50:48.912610 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.114:0/15885 conn(0x55d986d7f800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.7 seq 242 0x55d9bcd72c00 osd_ping(ping e501 stamp 2018-07-11 07:50:49.542565) v4 > -8108> 2018-07-11 07:50:48.912618 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:49.542565) v4 -- 0x55d9b3db2800 con 0 > -8107> 2018-07-11 07:50:48.912633 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.7 10.8.128.114:0/15885 242 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:49.542565) v4 ==== 2004+0+0 (52713147 0 0) 0x55d9bcd72c00 con 0x55d986d7f800 > -8106> 2018-07-11 07:50:48.912650 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:49.542565) v4 -- 0x55d986d97200 con 0 > -8105> 2018-07-11 07:50:48.940082 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.107:0/12617 conn(0x55d98123c000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 225 0x55d9bdba4200 osd_ping(ping e501 stamp 2018-07-11 07:50:48.939606) v4 > -8104> 2018-07-11 07:50:48.940089 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.107:0/12617 conn(0x55d98123d800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 225 0x55d9b67d9e00 osd_ping(ping e501 stamp 2018-07-11 07:50:48.939606) v4 > -8103> 2018-07-11 07:50:48.940104 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.9 10.8.128.107:0/12617 225 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:48.939606) v4 ==== 2004+0+0 (866326813 0 0) 0x55d9bdba4200 con 0x55d98123c000 > -8102> 2018-07-11 07:50:48.940109 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.9 10.8.128.107:0/12617 225 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:48.939606) v4 ==== 2004+0+0 (866326813 0 0) 0x55d9b67d9e00 con 0x55d98123d800 > -8101> 2018-07-11 07:50:48.940140 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:48.939606) v4 -- 0x55d9b3db2800 con 0 > -8100> 2018-07-11 07:50:48.940173 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:48.939606) v4 -- 0x55d980f5be00 con 0 > -8099> 2018-07-11 07:50:48.956366 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6801/15919 conn(0x55d986c3f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=29 cs=1 l=0). rx osd.11 seq 212 0x55d986cd7600 MOSDScrubReserve(10.f REQUEST e501) v1 > -8098> 2018-07-11 07:50:48.956384 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.11 10.8.128.114:6801/15919 212 ==== MOSDScrubReserve(10.f REQUEST e501) v1 ==== 43+0+0 (2892966580 0 0) 0x55d986cd7600 con 0x55d986c3f800 > -8097> 2018-07-11 07:50:48.956419 7ffab53e3700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.114:6801/15919 -- MOSDScrubReserve(10.f REJECT e501) v1 -- 0x55d9bf805800 con 0 > -8096> 2018-07-11 07:50:48.956762 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6801/15919 conn(0x55d986c3f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=29 cs=1 l=0). rx osd.11 seq 213 0x55d9bf805800 MOSDScrubReserve(10.f RELEASE e501) v1 > -8095> 2018-07-11 07:50:48.956776 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.11 10.8.128.114:6801/15919 213 ==== MOSDScrubReserve(10.f RELEASE e501) v1 ==== 43+0+0 (1668468918 0 0) 0x55d9bf805800 con 0x55d986c3f800 > -8094> 2018-07-11 07:50:49.117752 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 831 0x55d9b33bc000 rep_scrubmap(5.2e e501 from shard 2) v1 > -8093> 2018-07-11 07:50:49.117771 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 831 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12482 (1269249638 0 3168546020) 0x55d9b33bc000 con 0x55d986bf0800 > -8092> 2018-07-11 07:50:49.153991 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 825 0x55d9b33bc240 rep_scrubmap(5.2e e501 from shard 4) v1 > -8091> 2018-07-11 07:50:49.154013 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 825 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12482 (3686815630 0 3168546020) 0x55d9b33bc240 con 0x55d986bec000 > -8090> 2018-07-11 07:50:49.159642 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -8089> 2018-07-11 07:50:49.159665 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -8088> 2018-07-11 07:50:49.159725 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -8087> 2018-07-11 07:50:49.160189 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -8086> 2018-07-11 07:50:49.177771 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15287 conn(0x55d981240800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 248 0x55d9c34cc000 osd_ping(ping e501 stamp 2018-07-11 07:50:50.408065) v4 > -8085> 2018-07-11 07:50:49.177796 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.4 10.8.128.113:0/15287 248 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:50.408065) v4 ==== 2004+0+0 (3178200373 0 0) 0x55d9c34cc000 con 0x55d981240800 > -8084> 2018-07-11 07:50:49.177810 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:50.408065) v4 -- 0x55d9b3db2800 con 0 > -8083> 2018-07-11 07:50:49.177849 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15287 conn(0x55d98123f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 248 0x55d9829e5000 osd_ping(ping e501 stamp 2018-07-11 07:50:50.408065) v4 > -8082> 2018-07-11 07:50:49.177870 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.4 10.8.128.113:0/15287 248 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:50.408065) v4 ==== 2004+0+0 (3178200373 0 0) 0x55d9829e5000 con 0x55d98123f000 > -8081> 2018-07-11 07:50:49.177882 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:50.408065) v4 -- 0x55d986d97200 con 0 > -8080> 2018-07-11 07:50:49.194968 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7526e005:::benchmark_data_magna115_10867_object2825798:0,end:5:7527337f:::benchmark_data_magna115_10867_object1298717:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98696e000 con 0 > -8079> 2018-07-11 07:50:49.195017 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7526e005:::benchmark_data_magna115_10867_object2825798:0,end:5:7527337f:::benchmark_data_magna115_10867_object1298717:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98696f180 con 0 > -8078> 2018-07-11 07:50:49.311691 7ffab1bdc700 5 osd.10 501 heartbeat: osd_stat(788 GB used, 143 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist []) > -8077> 2018-07-11 07:50:49.311704 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6802/15255 -- osd_ping(ping e501 stamp 2018-07-11 07:50:49.311703) v4 -- 0x55d982cde400 con 0 > -8076> 2018-07-11 07:50:49.311742 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6803/15255 -- osd_ping(ping e501 stamp 2018-07-11 07:50:49.311703) v4 -- 0x55d986c85000 con 0 > -8075> 2018-07-11 07:50:49.311757 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6806/12677 -- osd_ping(ping e501 stamp 2018-07-11 07:50:49.311703) v4 -- 0x55d9b2528000 con 0 > -8074> 2018-07-11 07:50:49.311771 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6807/12677 -- osd_ping(ping e501 stamp 2018-07-11 07:50:49.311703) v4 -- 0x55d985054c00 con 0 > -8073> 2018-07-11 07:50:49.311783 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6808/12651 -- osd_ping(ping e501 stamp 2018-07-11 07:50:49.311703) v4 -- 0x55d986c61000 con 0 > -8072> 2018-07-11 07:50:49.311793 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6810/12651 -- osd_ping(ping e501 stamp 2018-07-11 07:50:49.311703) v4 -- 0x55d980f5a000 con 0 > -8071> 2018-07-11 07:50:49.311805 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6809/12650 -- osd_ping(ping e501 stamp 2018-07-11 07:50:49.311703) v4 -- 0x55d986c61600 con 0 > -8070> 2018-07-11 07:50:49.311818 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6811/12650 -- osd_ping(ping e501 stamp 2018-07-11 07:50:49.311703) v4 -- 0x55d9c34cde00 con 0 > -8069> 2018-07-11 07:50:49.311836 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6810/15287 -- osd_ping(ping e501 stamp 2018-07-11 07:50:49.311703) v4 -- 0x55d986d32400 con 0 > -8068> 2018-07-11 07:50:49.311847 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6811/15287 -- osd_ping(ping e501 stamp 2018-07-11 07:50:49.311703) v4 -- 0x55d9bf805200 con 0 > -8067> 2018-07-11 07:50:49.311860 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6806/15263 -- osd_ping(ping e501 stamp 2018-07-11 07:50:49.311703) v4 -- 0x55d9b60df200 con 0 > -8066> 2018-07-11 07:50:49.311875 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6807/15263 -- osd_ping(ping e501 stamp 2018-07-11 07:50:49.311703) v4 -- 0x55d986d4e200 con 0 > -8065> 2018-07-11 07:50:49.311889 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.115:6802/13323 -- osd_ping(ping e501 stamp 2018-07-11 07:50:49.311703) v4 -- 0x55d982cdf800 con 0 > -8064> 2018-07-11 07:50:49.311902 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.115:6803/13323 -- osd_ping(ping e501 stamp 2018-07-11 07:50:49.311703) v4 -- 0x55d985054a00 con 0 > -8063> 2018-07-11 07:50:49.311915 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6806/15885 -- osd_ping(ping e501 stamp 2018-07-11 07:50:49.311703) v4 -- 0x55d9b9ac2000 con 0 > -8062> 2018-07-11 07:50:49.311926 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6807/15885 -- osd_ping(ping e501 stamp 2018-07-11 07:50:49.311703) v4 -- 0x55d9b9ac3a00 con 0 > -8061> 2018-07-11 07:50:49.311942 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.108:6802/11967 -- osd_ping(ping e501 stamp 2018-07-11 07:50:49.311703) v4 -- 0x55d9bdba4400 con 0 > -8060> 2018-07-11 07:50:49.311954 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.108:6803/11967 -- osd_ping(ping e501 stamp 2018-07-11 07:50:49.311703) v4 -- 0x55d9b67d8400 con 0 > -8059> 2018-07-11 07:50:49.311967 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.107:6806/12617 -- osd_ping(ping e501 stamp 2018-07-11 07:50:49.311703) v4 -- 0x55d9b60df000 con 0 > -8058> 2018-07-11 07:50:49.311980 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.107:6807/12617 -- osd_ping(ping e501 stamp 2018-07-11 07:50:49.311703) v4 -- 0x55d9b9ac3200 con 0 > -8057> 2018-07-11 07:50:49.311992 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6802/15919 -- osd_ping(ping e501 stamp 2018-07-11 07:50:49.311703) v4 -- 0x55d9c4c1ae00 con 0 > -8056> 2018-07-11 07:50:49.312006 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6803/15919 -- osd_ping(ping e501 stamp 2018-07-11 07:50:49.311703) v4 -- 0x55d986d96e00 con 0 > -8055> 2018-07-11 07:50:49.312160 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6802/15255 conn(0x55d986bc8000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.0 seq 229 0x55d9bdba4200 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:49.311703) v4 > -8054> 2018-07-11 07:50:49.312164 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.107:6807/12617 conn(0x55d986bd3800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.9 seq 229 0x55d9b67d9e00 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:49.311703) v4 > -8053> 2018-07-11 07:50:49.312176 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.0 10.8.128.113:6802/15255 229 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:49.311703) v4 ==== 2004+0+0 (3363675034 0 0) 0x55d9bdba4200 con 0x55d986bc8000 > -8052> 2018-07-11 07:50:49.312183 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.9 10.8.128.107:6807/12617 229 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:49.311703) v4 ==== 2004+0+0 (3363675034 0 0) 0x55d9b67d9e00 con 0x55d986bd3800 > -8051> 2018-07-11 07:50:49.312216 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6803/15255 conn(0x55d986bc9800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.0 seq 229 0x55d9829e5000 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:49.311703) v4 > -8050> 2018-07-11 07:50:49.312259 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6808/12651 conn(0x55d986b9a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.2 seq 229 0x55d9bdba4200 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:49.311703) v4 > -8049> 2018-07-11 07:50:49.312237 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.0 10.8.128.113:6803/15255 229 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:49.311703) v4 ==== 2004+0+0 (3363675034 0 0) 0x55d9829e5000 con 0x55d986bc9800 > -8048> 2018-07-11 07:50:49.312277 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.2 10.8.128.106:6808/12651 229 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:49.311703) v4 ==== 2004+0+0 (3363675034 0 0) 0x55d9bdba4200 con 0x55d986b9a000 > -8047> 2018-07-11 07:50:49.312273 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6809/12650 conn(0x55d986bcb000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 229 0x55d9b67d9e00 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:49.311703) v4 > -8046> 2018-07-11 07:50:49.312295 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.3 10.8.128.106:6809/12650 229 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:49.311703) v4 ==== 2004+0+0 (3363675034 0 0) 0x55d9b67d9e00 con 0x55d986bcb000 > -8045> 2018-07-11 07:50:49.312314 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.107:6806/12617 conn(0x55d986bd2000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.9 seq 229 0x55d9829e5000 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:49.311703) v4 > -8044> 2018-07-11 07:50:49.312329 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.9 10.8.128.107:6806/12617 229 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:49.311703) v4 ==== 2004+0+0 (3363675034 0 0) 0x55d9829e5000 con 0x55d986bd2000 > -8043> 2018-07-11 07:50:49.312338 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6806/15263 conn(0x55d986b9d000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.5 seq 229 0x55d9b67d9e00 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:49.311703) v4 > -8042> 2018-07-11 07:50:49.312365 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6811/12650 conn(0x55d986bcc800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 229 0x55d9bdba4200 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:49.311703) v4 > -8041> 2018-07-11 07:50:49.312371 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.5 10.8.128.113:6806/15263 229 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:49.311703) v4 ==== 2004+0+0 (3363675034 0 0) 0x55d9b67d9e00 con 0x55d986b9d000 > -8040> 2018-07-11 07:50:49.312384 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.3 10.8.128.106:6811/12650 229 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:49.311703) v4 ==== 2004+0+0 (3363675034 0 0) 0x55d9bdba4200 con 0x55d986bcc800 > -8039> 2018-07-11 07:50:49.312409 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6810/12651 conn(0x55d986b9b800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.2 seq 229 0x55d9829e5000 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:49.311703) v4 > -8038> 2018-07-11 07:50:49.312424 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.2 10.8.128.106:6810/12651 229 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:49.311703) v4 ==== 2004+0+0 (3363675034 0 0) 0x55d9829e5000 con 0x55d986b9b800 > -8037> 2018-07-11 07:50:49.312424 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6806/12677 conn(0x55d986b58000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.1 seq 229 0x55d9b67d9e00 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:49.311703) v4 > -8036> 2018-07-11 07:50:49.312430 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6807/12677 conn(0x55d986b59800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.1 seq 229 0x55d9bdba4200 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:49.311703) v4 > -8035> 2018-07-11 07:50:49.312438 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.1 10.8.128.106:6806/12677 229 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:49.311703) v4 ==== 2004+0+0 (3363675034 0 0) 0x55d9b67d9e00 con 0x55d986b58000 > -8034> 2018-07-11 07:50:49.312442 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.1 10.8.128.106:6807/12677 229 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:49.311703) v4 ==== 2004+0+0 (3363675034 0 0) 0x55d9bdba4200 con 0x55d986b59800 > -8033> 2018-07-11 07:50:49.312470 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6811/15287 conn(0x55d986bb0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 229 0x55d9829e5000 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:49.311703) v4 > -8032> 2018-07-11 07:50:49.312485 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.115:6803/13323 conn(0x55d986bc2800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 229 0x55d9b67d9e00 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:49.311703) v4 > -8031> 2018-07-11 07:50:49.312486 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6807/15263 conn(0x55d986b9e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.5 seq 229 0x55d9bdba4200 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:49.311703) v4 > -8030> 2018-07-11 07:50:49.312501 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.6 10.8.128.115:6803/13323 229 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:49.311703) v4 ==== 2004+0+0 (3363675034 0 0) 0x55d9b67d9e00 con 0x55d986bc2800 > -8029> 2018-07-11 07:50:49.312506 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.5 10.8.128.113:6807/15263 229 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:49.311703) v4 ==== 2004+0+0 (3363675034 0 0) 0x55d9bdba4200 con 0x55d986b9e800 > -8028> 2018-07-11 07:50:49.312484 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.4 10.8.128.113:6811/15287 229 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:49.311703) v4 ==== 2004+0+0 (3363675034 0 0) 0x55d9829e5000 con 0x55d986bb0800 > -8027> 2018-07-11 07:50:49.312536 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6810/15287 conn(0x55d986baf000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 229 0x55d9bdba4200 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:49.311703) v4 > -8026> 2018-07-11 07:50:49.312548 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.4 10.8.128.113:6810/15287 229 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:49.311703) v4 ==== 2004+0+0 (3363675034 0 0) 0x55d9bdba4200 con 0x55d986baf000 > -8025> 2018-07-11 07:50:49.312546 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.115:6802/13323 conn(0x55d986bc1000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 229 0x55d9829e5000 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:49.311703) v4 > -8024> 2018-07-11 07:50:49.312578 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.6 10.8.128.115:6802/13323 229 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:49.311703) v4 ==== 2004+0+0 (3363675034 0 0) 0x55d9829e5000 con 0x55d986bc1000 > -8023> 2018-07-11 07:50:49.312592 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.108:6803/11967 conn(0x55d986bbf800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.8 seq 229 0x55d9bdba4200 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:49.311703) v4 > -8022> 2018-07-11 07:50:49.312606 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.8 10.8.128.108:6803/11967 229 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:49.311703) v4 ==== 2004+0+0 (3363675034 0 0) 0x55d9bdba4200 con 0x55d986bbf800 > -8021> 2018-07-11 07:50:49.312601 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.108:6802/11967 conn(0x55d986bbe000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.8 seq 229 0x55d9b67d9e00 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:49.311703) v4 > -8020> 2018-07-11 07:50:49.312620 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.8 10.8.128.108:6802/11967 229 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:49.311703) v4 ==== 2004+0+0 (3363675034 0 0) 0x55d9b67d9e00 con 0x55d986bbe000 > -8019> 2018-07-11 07:50:49.312629 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6806/15885 conn(0x55d986bac000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.7 seq 229 0x55d9829e5000 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:49.311703) v4 > -8018> 2018-07-11 07:50:49.312646 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.7 10.8.128.114:6806/15885 229 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:49.311703) v4 ==== 2004+0+0 (3363675034 0 0) 0x55d9829e5000 con 0x55d986bac000 > -8017> 2018-07-11 07:50:49.312669 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6803/15919 conn(0x55d986b5c800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.11 seq 229 0x55d9b67d9e00 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:49.311703) v4 > -8016> 2018-07-11 07:50:49.312686 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6802/15919 conn(0x55d986b5b000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.11 seq 229 0x55d9829e5000 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:49.311703) v4 > -8015> 2018-07-11 07:50:49.312694 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.11 10.8.128.114:6803/15919 229 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:49.311703) v4 ==== 2004+0+0 (3363675034 0 0) 0x55d9b67d9e00 con 0x55d986b5c800 > -8014> 2018-07-11 07:50:49.312704 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.11 10.8.128.114:6802/15919 229 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:49.311703) v4 ==== 2004+0+0 (3363675034 0 0) 0x55d9829e5000 con 0x55d986b5b000 > -8013> 2018-07-11 07:50:49.312725 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6807/15885 conn(0x55d986bad800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.7 seq 229 0x55d9b67d9e00 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:49.311703) v4 > -8012> 2018-07-11 07:50:49.312740 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.7 10.8.128.114:6807/15885 229 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:49.311703) v4 ==== 2004+0+0 (3363675034 0 0) 0x55d9b67d9e00 con 0x55d986bad800 > -8011> 2018-07-11 07:50:49.416946 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 832 0x55d9b33bc480 rep_scrubmap(5.2e e501 from shard 2) v1 > -8010> 2018-07-11 07:50:49.416965 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 832 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12470 (1269249638 0 3788044416) 0x55d9b33bc480 con 0x55d986bf0800 > -8009> 2018-07-11 07:50:49.448341 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 826 0x55d9b33bc6c0 rep_scrubmap(5.2e e501 from shard 4) v1 > -8008> 2018-07-11 07:50:49.448359 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 826 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12470 (3686815630 0 3788044416) 0x55d9b33bc6c0 con 0x55d986bec000 > -8007> 2018-07-11 07:50:49.501817 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -8006> 2018-07-11 07:50:49.501855 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -8005> 2018-07-11 07:50:49.501915 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -8004> 2018-07-11 07:50:49.502390 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -8003> 2018-07-11 07:50:49.520162 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67053,epoch:501/498,start:5:7527337f:::benchmark_data_magna115_10867_object1298717:0,end:5:7527a1b4:::benchmark_data_magna115_10867_object482148:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b14f0500 con 0 > -8002> 2018-07-11 07:50:49.520210 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67053,epoch:501/498,start:5:7527337f:::benchmark_data_magna115_10867_object1298717:0,end:5:7527a1b4:::benchmark_data_magna115_10867_object482148:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498780 con 0 > -8001> 2018-07-11 07:50:49.557695 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6805/15885 conn(0x55d986c7f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.7 seq 2770 0x55d9c34cc000 MOSDScrubReserve(5.4 REQUEST e501) v1 > -8000> 2018-07-11 07:50:49.557714 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.7 10.8.128.114:6805/15885 2770 ==== MOSDScrubReserve(5.4 REQUEST e501) v1 ==== 43+0+0 (1753221512 0 0) 0x55d9c34cc000 con 0x55d986c7f800 > -7999> 2018-07-11 07:50:49.557785 7ffab33df700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.114:6805/15885 -- MOSDScrubReserve(5.4 REJECT e501) v1 -- 0x55d9bdba4800 con 0 > -7998> 2018-07-11 07:50:49.558126 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6805/15885 conn(0x55d986c7f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.7 seq 2771 0x55d9bdba4800 MOSDScrubReserve(5.4 RELEASE e501) v1 > -7997> 2018-07-11 07:50:49.558141 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.7 10.8.128.114:6805/15885 2771 ==== MOSDScrubReserve(5.4 RELEASE e501) v1 ==== 43+0+0 (2812146570 0 0) 0x55d9bdba4800 con 0x55d986c7f800 > -7996> 2018-07-11 07:50:49.560885 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15255 conn(0x55d986d20000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.0 seq 238 0x55d9bdba4200 osd_ping(ping e501 stamp 2018-07-11 07:50:50.791099) v4 > -7995> 2018-07-11 07:50:49.560909 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.0 10.8.128.113:0/15255 238 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:50.791099) v4 ==== 2004+0+0 (3543591286 0 0) 0x55d9bdba4200 con 0x55d986d20000 > -7994> 2018-07-11 07:50:49.560920 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:50.791099) v4 -- 0x55d986d4e200 con 0 > -7993> 2018-07-11 07:50:49.560959 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15255 conn(0x55d986c82800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.0 seq 238 0x55d9829e5000 osd_ping(ping e501 stamp 2018-07-11 07:50:50.791099) v4 > -7992> 2018-07-11 07:50:49.560992 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.0 10.8.128.113:0/15255 238 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:50.791099) v4 ==== 2004+0+0 (3543591286 0 0) 0x55d9829e5000 con 0x55d986c82800 > -7991> 2018-07-11 07:50:49.561009 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:50.791099) v4 -- 0x55d9c4c1ae00 con 0 > -7990> 2018-07-11 07:50:49.724855 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 833 0x55d9b33bc900 rep_scrubmap(5.2e e501 from shard 2) v1 > -7989> 2018-07-11 07:50:49.724874 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 833 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12470 (1269249638 0 1331896226) 0x55d9b33bc900 con 0x55d986bf0800 > -7988> 2018-07-11 07:50:49.737022 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 827 0x55d9b33bcb40 rep_scrubmap(5.2e e501 from shard 4) v1 > -7987> 2018-07-11 07:50:49.737041 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 827 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12470 (3686815630 0 1331896226) 0x55d9b33bcb40 con 0x55d986bec000 > -7986> 2018-07-11 07:50:49.832166 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 2980 0x55d9b67d9e00 MOSDScrubReserve(11.3s5 REQUEST e501) v1 > -7985> 2018-07-11 07:50:49.832183 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 2980 ==== MOSDScrubReserve(11.3s5 REQUEST e501) v1 ==== 43+0+0 (3260170403 0 0) 0x55d9b67d9e00 con 0x55d986c7e000 > -7984> 2018-07-11 07:50:49.832307 7ffab3be0700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e501) v1 -- 0x55d982cdfc00 con 0 > -7983> 2018-07-11 07:50:49.832634 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 2981 0x55d986d96e00 MOSDScrubReserve(11.3s5 RELEASE e501) v1 > -7982> 2018-07-11 07:50:49.832659 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 2981 ==== MOSDScrubReserve(11.3s5 RELEASE e501) v1 ==== 43+0+0 (223328929 0 0) 0x55d986d96e00 con 0x55d986c7e000 > -7981> 2018-07-11 07:50:49.904774 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -7980> 2018-07-11 07:50:49.904802 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -7979> 2018-07-11 07:50:49.904861 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -7978> 2018-07-11 07:50:49.905357 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -7977> 2018-07-11 07:50:49.920216 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67652,epoch:501/498,start:5:7527a1b4:::benchmark_data_magna115_10867_object482148:0,end:5:752806ec:::benchmark_data_magna115_11707_object114289:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455900 con 0 > -7976> 2018-07-11 07:50:49.920275 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67652,epoch:501/498,start:5:7527a1b4:::benchmark_data_magna115_10867_object482148:0,end:5:752806ec:::benchmark_data_magna115_11707_object114289:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455b80 con 0 > -7975> 2018-07-11 07:50:50.043102 7ffacbbd1700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15263 conn(0x55d985d44000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.5 seq 222 0x55d980f5be00 osd_ping(ping e501 stamp 2018-07-11 07:50:51.273405) v4 > -7974> 2018-07-11 07:50:50.043127 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 <== osd.5 10.8.128.113:0/15263 222 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:51.273405) v4 ==== 2004+0+0 (1747773963 0 0) 0x55d980f5be00 con 0x55d985d44000 > -7973> 2018-07-11 07:50:50.043138 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:51.273405) v4 -- 0x55d9b9ac3200 con 0 > -7972> 2018-07-11 07:50:50.043169 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15263 conn(0x55d985d45800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.5 seq 222 0x55d9bcd72c00 osd_ping(ping e501 stamp 2018-07-11 07:50:51.273405) v4 > -7971> 2018-07-11 07:50:50.043189 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.5 10.8.128.113:0/15263 222 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:51.273405) v4 ==== 2004+0+0 (1747773963 0 0) 0x55d9bcd72c00 con 0x55d985d45800 > -7970> 2018-07-11 07:50:50.043205 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:51.273405) v4 -- 0x55d9c4c1ae00 con 0 > -7969> 2018-07-11 07:50:50.109914 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 834 0x55d9b33bcd80 rep_scrubmap(5.2e e501 from shard 2) v1 > -7968> 2018-07-11 07:50:50.109933 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 834 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12480 (1269249638 0 2825705256) 0x55d9b33bcd80 con 0x55d986bf0800 > -7967> 2018-07-11 07:50:50.170076 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 828 0x55d9b33bcfc0 rep_scrubmap(5.2e e501 from shard 4) v1 > -7966> 2018-07-11 07:50:50.170095 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 828 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12480 (3686815630 0 2825705256) 0x55d9b33bcfc0 con 0x55d986bec000 > -7965> 2018-07-11 07:50:50.286940 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -7964> 2018-07-11 07:50:50.286978 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -7963> 2018-07-11 07:50:50.287034 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -7962> 2018-07-11 07:50:50.287505 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -7961> 2018-07-11 07:50:50.336841 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67543,epoch:501/498,start:5:752806ec:::benchmark_data_magna115_11707_object114289:0,end:5:75284ef4:::benchmark_data_magna115_10867_object2825555:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498000 con 0 > -7960> 2018-07-11 07:50:50.336893 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67543,epoch:501/498,start:5:752806ec:::benchmark_data_magna115_11707_object114289:0,end:5:75284ef4:::benchmark_data_magna115_10867_object2825555:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455680 con 0 > -7959> 2018-07-11 07:50:50.520705 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 835 0x55d9b33bd200 rep_scrubmap(5.2e e501 from shard 2) v1 > -7958> 2018-07-11 07:50:50.520726 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 835 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12468 (1269249638 0 2124892731) 0x55d9b33bd200 con 0x55d986bf0800 > -7957> 2018-07-11 07:50:50.525216 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 829 0x55d9b33bd440 rep_scrubmap(5.2e e501 from shard 4) v1 > -7956> 2018-07-11 07:50:50.525236 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 829 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12468 (3686815630 0 2124892731) 0x55d9b33bd440 con 0x55d986bec000 > -7955> 2018-07-11 07:50:50.688891 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -7954> 2018-07-11 07:50:50.688931 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -7953> 2018-07-11 07:50:50.688979 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -7952> 2018-07-11 07:50:50.689452 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -7951> 2018-07-11 07:50:50.703952 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75284ef4:::benchmark_data_magna115_10867_object2825555:0,end:5:7528cad5:::benchmark_data_magna115_10867_object1951120:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498280 con 0 > -7950> 2018-07-11 07:50:50.704002 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75284ef4:::benchmark_data_magna115_10867_object2825555:0,end:5:7528cad5:::benchmark_data_magna115_10867_object1951120:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455400 con 0 > -7949> 2018-07-11 07:50:50.810661 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12650 conn(0x55d986d23000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 232 0x55d9829e5000 osd_ping(ping e501 stamp 2018-07-11 07:50:50.857812) v4 > -7948> 2018-07-11 07:50:50.810685 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.3 10.8.128.106:0/12650 232 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:50.857812) v4 ==== 2004+0+0 (232224013 0 0) 0x55d9829e5000 con 0x55d986d23000 > -7947> 2018-07-11 07:50:50.810695 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:50.857812) v4 -- 0x55d9c4c1ae00 con 0 > -7946> 2018-07-11 07:50:50.810702 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12650 conn(0x55d986d21800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 232 0x55d982cdfc00 osd_ping(ping e501 stamp 2018-07-11 07:50:50.857812) v4 > -7945> 2018-07-11 07:50:50.810725 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.3 10.8.128.106:0/12650 232 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:50.857812) v4 ==== 2004+0+0 (232224013 0 0) 0x55d982cdfc00 con 0x55d986d21800 > -7944> 2018-07-11 07:50:50.810741 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:50.857812) v4 -- 0x55d9b9ac3200 con 0 > -7943> 2018-07-11 07:50:50.916388 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 836 0x55d9b33bd680 rep_scrubmap(5.2e e501 from shard 2) v1 > -7942> 2018-07-11 07:50:50.916410 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 836 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12472 (1269249638 0 3726125099) 0x55d9b33bd680 con 0x55d986bf0800 > -7941> 2018-07-11 07:50:50.956735 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6801/15919 conn(0x55d986c3f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=29 cs=1 l=0). rx osd.11 seq 214 0x55d980f5be00 MOSDScrubReserve(10.f REQUEST e501) v1 > -7940> 2018-07-11 07:50:50.956752 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.11 10.8.128.114:6801/15919 214 ==== MOSDScrubReserve(10.f REQUEST e501) v1 ==== 43+0+0 (2892966580 0 0) 0x55d980f5be00 con 0x55d986c3f800 > -7939> 2018-07-11 07:50:50.956814 7ffab53e3700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.114:6801/15919 -- MOSDScrubReserve(10.f REJECT e501) v1 -- 0x55d9bf805800 con 0 > -7938> 2018-07-11 07:50:50.957182 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6801/15919 conn(0x55d986c3f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=29 cs=1 l=0). rx osd.11 seq 215 0x55d9bf805800 MOSDScrubReserve(10.f RELEASE e501) v1 > -7937> 2018-07-11 07:50:50.957197 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.11 10.8.128.114:6801/15919 215 ==== MOSDScrubReserve(10.f RELEASE e501) v1 ==== 43+0+0 (1668468918 0 0) 0x55d9bf805800 con 0x55d986c3f800 > -7936> 2018-07-11 07:50:50.961597 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 830 0x55d9b33bd8c0 rep_scrubmap(5.2e e501 from shard 4) v1 > -7935> 2018-07-11 07:50:50.961619 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 830 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12472 (3686815630 0 3726125099) 0x55d9b33bd8c0 con 0x55d986bec000 > -7934> 2018-07-11 07:50:51.041396 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -7933> 2018-07-11 07:50:51.041420 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -7932> 2018-07-11 07:50:51.041479 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -7931> 2018-07-11 07:50:51.041940 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -7930> 2018-07-11 07:50:51.087133 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7528cad5:::benchmark_data_magna115_10867_object1951120:0,end:5:7529101a:::benchmark_data_magna115_11707_object420228:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455180 con 0 > -7929> 2018-07-11 07:50:51.087186 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7528cad5:::benchmark_data_magna115_10867_object1951120:0,end:5:7529101a:::benchmark_data_magna115_11707_object420228:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454f00 con 0 > -7928> 2018-07-11 07:50:51.087304 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.114:0/15919 conn(0x55d986df5000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.11 seq 240 0x55d9bcd72c00 osd_ping(ping e501 stamp 2018-07-11 07:50:51.717263) v4 > -7927> 2018-07-11 07:50:51.087308 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.114:0/15919 conn(0x55d986df3800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.11 seq 240 0x55d982cdfc00 osd_ping(ping e501 stamp 2018-07-11 07:50:51.717263) v4 > -7926> 2018-07-11 07:50:51.087327 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.11 10.8.128.114:0/15919 240 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:51.717263) v4 ==== 2004+0+0 (3593324180 0 0) 0x55d9bcd72c00 con 0x55d986df5000 > -7925> 2018-07-11 07:50:51.087334 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.11 10.8.128.114:0/15919 240 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:51.717263) v4 ==== 2004+0+0 (3593324180 0 0) 0x55d982cdfc00 con 0x55d986df3800 > -7924> 2018-07-11 07:50:51.087350 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:51.717263) v4 -- 0x55d9c4c1ae00 con 0 > -7923> 2018-07-11 07:50:51.087383 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:51.717263) v4 -- 0x55d9bdba4400 con 0 > -7922> 2018-07-11 07:50:51.283233 7ffacbbd1700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.108:0/11967 conn(0x55d986d82800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.8 seq 220 0x55d9b9ac3200 osd_ping(ping e501 stamp 2018-07-11 07:50:51.038660) v4 > -7921> 2018-07-11 07:50:51.283237 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.108:0/11967 conn(0x55d986db0000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.8 seq 220 0x55d9829e5000 osd_ping(ping e501 stamp 2018-07-11 07:50:51.038660) v4 > -7920> 2018-07-11 07:50:51.283265 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 <== osd.8 10.8.128.108:0/11967 220 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:51.038660) v4 ==== 2004+0+0 (892082228 0 0) 0x55d9b9ac3200 con 0x55d986d82800 > -7919> 2018-07-11 07:50:51.283267 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.8 10.8.128.108:0/11967 220 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:51.038660) v4 ==== 2004+0+0 (892082228 0 0) 0x55d9829e5000 con 0x55d986db0000 > -7918> 2018-07-11 07:50:51.283275 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:51.038660) v4 -- 0x55d9bdba4400 con 0 > -7917> 2018-07-11 07:50:51.283299 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:51.038660) v4 -- 0x55d9c4c1ae00 con 0 > -7916> 2018-07-11 07:50:51.290135 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 831 0x55d9b33bdb00 rep_scrubmap(5.2e e501 from shard 4) v1 > -7915> 2018-07-11 07:50:51.290155 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 831 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12476 (3686815630 0 363487105) 0x55d9b33bdb00 con 0x55d986bec000 > -7914> 2018-07-11 07:50:51.293736 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 837 0x55d9b33bdd40 rep_scrubmap(5.2e e501 from shard 2) v1 > -7913> 2018-07-11 07:50:51.293756 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 837 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12476 (1269249638 0 363487105) 0x55d9b33bdd40 con 0x55d986bf0800 > -7912> 2018-07-11 07:50:51.388705 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -7911> 2018-07-11 07:50:51.388729 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -7910> 2018-07-11 07:50:51.388787 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -7909> 2018-07-11 07:50:51.389230 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -7908> 2018-07-11 07:50:51.426483 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'66822,epoch:501/498,start:5:7529101a:::benchmark_data_magna115_11707_object420228:0,end:5:75297bc2:::benchmark_data_magna115_11707_object211424:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454780 con 0 > -7907> 2018-07-11 07:50:51.426535 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'66822,epoch:501/498,start:5:7529101a:::benchmark_data_magna115_11707_object420228:0,end:5:75297bc2:::benchmark_data_magna115_11707_object211424:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454c80 con 0 > -7906> 2018-07-11 07:50:51.557975 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6805/15885 conn(0x55d986c7f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.7 seq 2772 0x55d9bdba4200 MOSDScrubReserve(5.4 REQUEST e501) v1 > -7905> 2018-07-11 07:50:51.557996 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.7 10.8.128.114:6805/15885 2772 ==== MOSDScrubReserve(5.4 REQUEST e501) v1 ==== 43+0+0 (1753221512 0 0) 0x55d9bdba4200 con 0x55d986c7f800 > -7904> 2018-07-11 07:50:51.558083 7ffab33df700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.114:6805/15885 -- MOSDScrubReserve(5.4 REJECT e501) v1 -- 0x55d9bdba4800 con 0 > -7903> 2018-07-11 07:50:51.558371 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6805/15885 conn(0x55d986c7f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.7 seq 2773 0x55d9bdba4800 MOSDScrubReserve(5.4 RELEASE e501) v1 > -7902> 2018-07-11 07:50:51.558386 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.7 10.8.128.114:6805/15885 2773 ==== MOSDScrubReserve(5.4 RELEASE e501) v1 ==== 43+0+0 (2812146570 0 0) 0x55d9bdba4800 con 0x55d986c7f800 > -7901> 2018-07-11 07:50:51.617284 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12651 conn(0x55d986db3000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.2 seq 239 0x55d982cdfc00 osd_ping(ping e501 stamp 2018-07-11 07:50:51.664424) v4 > -7900> 2018-07-11 07:50:51.617288 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12651 conn(0x55d986db1800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.2 seq 239 0x55d9b67d8400 osd_ping(ping e501 stamp 2018-07-11 07:50:51.664424) v4 > -7899> 2018-07-11 07:50:51.617306 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.2 10.8.128.106:0/12651 239 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:51.664424) v4 ==== 2004+0+0 (2243796592 0 0) 0x55d982cdfc00 con 0x55d986db3000 > -7898> 2018-07-11 07:50:51.617306 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.2 10.8.128.106:0/12651 239 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:51.664424) v4 ==== 2004+0+0 (2243796592 0 0) 0x55d9b67d8400 con 0x55d986db1800 > -7897> 2018-07-11 07:50:51.617316 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:51.664424) v4 -- 0x55d9bdba4400 con 0 > -7896> 2018-07-11 07:50:51.617341 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:51.664424) v4 -- 0x55d986d32400 con 0 > -7895> 2018-07-11 07:50:51.645194 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 832 0x55d99a412000 rep_scrubmap(5.2e e501 from shard 4) v1 > -7894> 2018-07-11 07:50:51.645213 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 832 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12472 (3686815630 0 2007096420) 0x55d99a412000 con 0x55d986bec000 > -7893> 2018-07-11 07:50:51.668560 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 838 0x55d99a412240 rep_scrubmap(5.2e e501 from shard 2) v1 > -7892> 2018-07-11 07:50:51.668580 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 838 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12472 (1269249638 0 2007096420) 0x55d99a412240 con 0x55d986bf0800 > -7891> 2018-07-11 07:50:51.668720 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -7890> 2018-07-11 07:50:51.668745 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -7889> 2018-07-11 07:50:51.668794 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -7888> 2018-07-11 07:50:51.669285 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -7887> 2018-07-11 07:50:51.704427 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'66600,epoch:501/498,start:5:75297bc2:::benchmark_data_magna115_11707_object211424:0,end:5:7529d081:::benchmark_data_magna115_10867_object1135760:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454a00 con 0 > -7886> 2018-07-11 07:50:51.704465 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'66600,epoch:501/498,start:5:75297bc2:::benchmark_data_magna115_11707_object211424:0,end:5:7529d081:::benchmark_data_magna115_10867_object1135760:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454500 con 0 > -7885> 2018-07-11 07:50:51.832536 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 2982 0x55d9b9ac3200 MOSDScrubReserve(11.3s5 REQUEST e501) v1 > -7884> 2018-07-11 07:50:51.832555 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 2982 ==== MOSDScrubReserve(11.3s5 REQUEST e501) v1 ==== 43+0+0 (3260170403 0 0) 0x55d9b9ac3200 con 0x55d986c7e000 > -7883> 2018-07-11 07:50:51.832590 7ffab3be0700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e501) v1 -- 0x55d986d96e00 con 0 > -7882> 2018-07-11 07:50:51.832970 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 2983 0x55d9bdba4400 MOSDScrubReserve(11.3s5 RELEASE e501) v1 > -7881> 2018-07-11 07:50:51.832985 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 2983 ==== MOSDScrubReserve(11.3s5 RELEASE e501) v1 ==== 43+0+0 (223328929 0 0) 0x55d9bdba4400 con 0x55d986c7e000 > -7880> 2018-07-11 07:50:51.895630 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 839 0x55d99a412480 rep_scrubmap(5.2e e501 from shard 2) v1 > -7879> 2018-07-11 07:50:51.895651 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 839 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12472 (1269249638 0 1455120199) 0x55d99a412480 con 0x55d986bf0800 > -7878> 2018-07-11 07:50:51.913002 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 833 0x55d99a4126c0 rep_scrubmap(5.2e e501 from shard 4) v1 > -7877> 2018-07-11 07:50:51.913022 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 833 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12472 (3686815630 0 1455120199) 0x55d99a4126c0 con 0x55d986bec000 > -7876> 2018-07-11 07:50:52.036759 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -7875> 2018-07-11 07:50:52.036784 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -7874> 2018-07-11 07:50:52.036845 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -7873> 2018-07-11 07:50:52.037334 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -7872> 2018-07-11 07:50:52.063117 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67440,epoch:501/498,start:5:7529d081:::benchmark_data_magna115_10867_object1135760:0,end:5:752a59e2:::benchmark_data_magna115_11707_object127844:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98944b900 con 0 > -7871> 2018-07-11 07:50:52.063164 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67440,epoch:501/498,start:5:7529d081:::benchmark_data_magna115_10867_object1135760:0,end:5:752a59e2:::benchmark_data_magna115_11707_object127844:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98944bb80 con 0 > -7870> 2018-07-11 07:50:52.284477 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 840 0x55d99a412900 rep_scrubmap(5.2e e501 from shard 2) v1 > -7869> 2018-07-11 07:50:52.284510 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 840 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12478 (1269249638 0 3522152385) 0x55d99a412900 con 0x55d986bf0800 > -7868> 2018-07-11 07:50:52.301452 7ffab7be8700 10 monclient: tick > -7867> 2018-07-11 07:50:52.301465 7ffab7be8700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2018-07-11 07:50:22.301463) > -7866> 2018-07-11 07:50:52.304526 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 834 0x55d99a412b40 rep_scrubmap(5.2e e501 from shard 4) v1 > -7865> 2018-07-11 07:50:52.304545 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 834 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12478 (3686815630 0 3522152385) 0x55d99a412b40 con 0x55d986bec000 > -7864> 2018-07-11 07:50:52.347976 7ffab6be6700 1 -- 10.8.128.107:6800/13927 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55d9b3fdedc0 con 0 > -7863> 2018-07-11 07:50:52.348046 7ffab6be6700 1 -- 10.8.128.107:6800/13927 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55d9bd9cd340 con 0 > -7862> 2018-07-11 07:50:52.383677 7ffacbbd1700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.108:0/11967 conn(0x55d986d82800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.8 seq 221 0x55d982cdfc00 osd_ping(ping e501 stamp 2018-07-11 07:50:52.139093) v4 > -7861> 2018-07-11 07:50:52.383676 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.108:0/11967 conn(0x55d986db0000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.8 seq 221 0x55d9bcd72c00 osd_ping(ping e501 stamp 2018-07-11 07:50:52.139093) v4 > -7860> 2018-07-11 07:50:52.383703 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 <== osd.8 10.8.128.108:0/11967 221 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:52.139093) v4 ==== 2004+0+0 (1101208392 0 0) 0x55d982cdfc00 con 0x55d986d82800 > -7859> 2018-07-11 07:50:52.383707 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.8 10.8.128.108:0/11967 221 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:52.139093) v4 ==== 2004+0+0 (1101208392 0 0) 0x55d9bcd72c00 con 0x55d986db0000 > -7858> 2018-07-11 07:50:52.383715 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:52.139093) v4 -- 0x55d9b9ac3a00 con 0 > -7857> 2018-07-11 07:50:52.383751 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:52.139093) v4 -- 0x55d9c4c1ae00 con 0 > -7856> 2018-07-11 07:50:52.461447 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15255 conn(0x55d986c82800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.0 seq 239 0x55d9829e5000 osd_ping(ping e501 stamp 2018-07-11 07:50:53.691673) v4 > -7855> 2018-07-11 07:50:52.461447 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15255 conn(0x55d986d20000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.0 seq 239 0x55d9b67d8400 osd_ping(ping e501 stamp 2018-07-11 07:50:53.691673) v4 > -7854> 2018-07-11 07:50:52.461469 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.0 10.8.128.113:0/15255 239 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:53.691673) v4 ==== 2004+0+0 (3735196285 0 0) 0x55d9829e5000 con 0x55d986c82800 > -7853> 2018-07-11 07:50:52.461471 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.0 10.8.128.113:0/15255 239 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:53.691673) v4 ==== 2004+0+0 (3735196285 0 0) 0x55d9b67d8400 con 0x55d986d20000 > -7852> 2018-07-11 07:50:52.461478 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:53.691673) v4 -- 0x55d9c4c1ae00 con 0 > -7851> 2018-07-11 07:50:52.461503 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:53.691673) v4 -- 0x55d986d32400 con 0 > -7850> 2018-07-11 07:50:52.832844 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 2984 0x55d986d96e00 MOSDScrubReserve(11.3s5 REQUEST e501) v1 > -7849> 2018-07-11 07:50:52.832862 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 2984 ==== MOSDScrubReserve(11.3s5 REQUEST e501) v1 ==== 43+0+0 (3260170403 0 0) 0x55d986d96e00 con 0x55d986c7e000 > -7848> 2018-07-11 07:50:52.832940 7ffab3be0700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e501) v1 -- 0x55d9bdba4400 con 0 > -7847> 2018-07-11 07:50:52.833243 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 2985 0x55d9bdba4400 MOSDScrubReserve(11.3s5 RELEASE e501) v1 > -7846> 2018-07-11 07:50:52.833273 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 2985 ==== MOSDScrubReserve(11.3s5 RELEASE e501) v1 ==== 43+0+0 (223328929 0 0) 0x55d9bdba4400 con 0x55d986c7e000 > -7845> 2018-07-11 07:50:53.013068 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.114:0/15885 conn(0x55d986d7f800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.7 seq 243 0x55d9bcd72c00 osd_ping(ping e501 stamp 2018-07-11 07:50:53.643013) v4 > -7844> 2018-07-11 07:50:53.013095 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.7 10.8.128.114:0/15885 243 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:53.643013) v4 ==== 2004+0+0 (2398216933 0 0) 0x55d9bcd72c00 con 0x55d986d7f800 > -7843> 2018-07-11 07:50:53.013109 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:53.643013) v4 -- 0x55d9c4c1ae00 con 0 > -7842> 2018-07-11 07:50:53.013104 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.114:0/15885 conn(0x55d986d81000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.7 seq 243 0x55d986d4e200 osd_ping(ping e501 stamp 2018-07-11 07:50:53.643013) v4 > -7841> 2018-07-11 07:50:53.013131 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.7 10.8.128.114:0/15885 243 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:53.643013) v4 ==== 2004+0+0 (2398216933 0 0) 0x55d986d4e200 con 0x55d986d81000 > -7840> 2018-07-11 07:50:53.013142 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:53.643013) v4 -- 0x55d986d32400 con 0 > -7839> 2018-07-11 07:50:53.497234 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 31] Generated table #36794: 2638240 keys, 74546899 bytes > -7838> 2018-07-11 07:50:53.497284 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295453497256, "cf_name": "default", "job": 31, "event": "table_file_creation", "file_number": 36794, "file_size": 74546899, "table_properties": {"data_size": 67112119, "index_size": 690609, "filter_size": 6743181, "raw_key_size": 76215821, "raw_average_key_size": 28, "raw_value_size": 31365741, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} > -7837> 2018-07-11 07:50:53.659525 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12677 conn(0x55d986db4800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.1 seq 243 0x55d9829e5000 osd_ping(ping e501 stamp 2018-07-11 07:50:53.706714) v4 > -7836> 2018-07-11 07:50:53.659525 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12677 conn(0x55d986df2000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.1 seq 243 0x55d9b67d8400 osd_ping(ping e501 stamp 2018-07-11 07:50:53.706714) v4 > -7835> 2018-07-11 07:50:53.659560 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.1 10.8.128.106:0/12677 243 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:53.706714) v4 ==== 2004+0+0 (2483495206 0 0) 0x55d9b67d8400 con 0x55d986df2000 > -7834> 2018-07-11 07:50:53.659557 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.1 10.8.128.106:0/12677 243 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:53.706714) v4 ==== 2004+0+0 (2483495206 0 0) 0x55d9829e5000 con 0x55d986db4800 > -7833> 2018-07-11 07:50:53.659574 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:53.706714) v4 -- 0x55d986d32400 con 0 > -7832> 2018-07-11 07:50:53.659604 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:53.706714) v4 -- 0x55d9c4c1ae00 con 0 > -7831> 2018-07-11 07:50:53.690944 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -7830> 2018-07-11 07:50:53.690971 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -7829> 2018-07-11 07:50:53.691030 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -7828> 2018-07-11 07:50:53.691486 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -7827> 2018-07-11 07:50:53.746418 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67237,epoch:501/498,start:5:752a59e2:::benchmark_data_magna115_11707_object127844:0,end:5:752ac33c:::benchmark_data_magna115_10867_object1844981:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb772280 con 0 > -7826> 2018-07-11 07:50:53.746460 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67237,epoch:501/498,start:5:752a59e2:::benchmark_data_magna115_11707_object127844:0,end:5:752ac33c:::benchmark_data_magna115_10867_object1844981:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98944b400 con 0 > -7825> 2018-07-11 07:50:53.955787 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 841 0x55d99a412d80 rep_scrubmap(5.2e e501 from shard 2) v1 > -7824> 2018-07-11 07:50:53.955810 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 841 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12468 (1269249638 0 757150833) 0x55d99a412d80 con 0x55d986bf0800 > -7823> 2018-07-11 07:50:53.955923 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6805/15263 conn(0x55d986c41000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=31 cs=1 l=0). rx osd.5 seq 2796 0x55d986d4e200 MOSDScrubReserve(6.15 REQUEST e501) v1 > -7822> 2018-07-11 07:50:53.955938 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.5 10.8.128.113:6805/15263 2796 ==== MOSDScrubReserve(6.15 REQUEST e501) v1 ==== 43+0+0 (3942742026 0 0) 0x55d986d4e200 con 0x55d986c41000 > -7821> 2018-07-11 07:50:53.967890 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 835 0x55d99a412fc0 rep_scrubmap(5.2e e501 from shard 4) v1 > -7820> 2018-07-11 07:50:53.967911 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 835 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12468 (3686815630 0 757150833) 0x55d99a412fc0 con 0x55d986bec000 > -7819> 2018-07-11 07:50:53.996468 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.115:0/13323 conn(0x55d986d7e000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 240 0x55d982cdfc00 osd_ping(ping e501 stamp 2018-07-11 07:50:54.341921) v4 > -7818> 2018-07-11 07:50:53.996491 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.6 10.8.128.115:0/13323 240 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:54.341921) v4 ==== 2004+0+0 (3760272926 0 0) 0x55d982cdfc00 con 0x55d986d7e000 > -7817> 2018-07-11 07:50:53.996501 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:54.341921) v4 -- 0x55d985054a00 con 0 > -7816> 2018-07-11 07:50:53.996583 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.115:0/13323 conn(0x55d986d24800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 240 0x55d986d32400 osd_ping(ping e501 stamp 2018-07-11 07:50:54.341921) v4 > -7815> 2018-07-11 07:50:53.996608 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.6 10.8.128.115:0/13323 240 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:54.341921) v4 ==== 2004+0+0 (3760272926 0 0) 0x55d986d32400 con 0x55d986d24800 > -7814> 2018-07-11 07:50:53.996619 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:54.341921) v4 -- 0x55d9c34cde00 con 0 > -7813> 2018-07-11 07:50:54.104984 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6805/15263 -- MOSDScrubReserve(6.15 REJECT e501) v1 -- 0x55d9829e4200 con 0 > -7812> 2018-07-11 07:50:54.105043 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -7811> 2018-07-11 07:50:54.105067 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -7810> 2018-07-11 07:50:54.105115 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -7809> 2018-07-11 07:50:54.105484 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6805/15263 conn(0x55d986c41000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=31 cs=1 l=0). rx osd.5 seq 2797 0x55d9829e4200 MOSDScrubReserve(6.15 RELEASE e501) v1 > -7808> 2018-07-11 07:50:54.105503 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.5 10.8.128.113:6805/15263 2797 ==== MOSDScrubReserve(6.15 RELEASE e501) v1 ==== 43+0+0 (605848072 0 0) 0x55d9829e4200 con 0x55d986c41000 > -7807> 2018-07-11 07:50:54.105583 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -7806> 2018-07-11 07:50:54.121368 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67954,epoch:501/498,start:5:752ac33c:::benchmark_data_magna115_10867_object1844981:0,end:5:752b26cd:::benchmark_data_magna115_11707_object432310:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb772000 con 0 > -7805> 2018-07-11 07:50:54.121416 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67954,epoch:501/498,start:5:752ac33c:::benchmark_data_magna115_10867_object1844981:0,end:5:752b26cd:::benchmark_data_magna115_11707_object432310:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb772500 con 0 > -7804> 2018-07-11 07:50:54.240864 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.107:0/12617 conn(0x55d98123d800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 226 0x55d9b9ac3a00 osd_ping(ping e501 stamp 2018-07-11 07:50:54.240284) v4 > -7803> 2018-07-11 07:50:54.240847 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.107:0/12617 conn(0x55d98123c000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 226 0x55d986d32400 osd_ping(ping e501 stamp 2018-07-11 07:50:54.240284) v4 > -7802> 2018-07-11 07:50:54.240892 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.9 10.8.128.107:0/12617 226 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:54.240284) v4 ==== 2004+0+0 (627019997 0 0) 0x55d9b9ac3a00 con 0x55d98123d800 > -7801> 2018-07-11 07:50:54.240895 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.9 10.8.128.107:0/12617 226 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:54.240284) v4 ==== 2004+0+0 (627019997 0 0) 0x55d986d32400 con 0x55d98123c000 > -7800> 2018-07-11 07:50:54.240912 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:54.240284) v4 -- 0x55d985054a00 con 0 > -7799> 2018-07-11 07:50:54.240992 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:54.240284) v4 -- 0x55d9c34cde00 con 0 > -7798> 2018-07-11 07:50:54.335415 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 842 0x55d99a413200 rep_scrubmap(5.2e e501 from shard 2) v1 > -7797> 2018-07-11 07:50:54.335435 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 842 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12474 (1269249638 0 291996384) 0x55d99a413200 con 0x55d986bf0800 > -7796> 2018-07-11 07:50:54.341421 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 836 0x55d99a413440 rep_scrubmap(5.2e e501 from shard 4) v1 > -7795> 2018-07-11 07:50:54.341441 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 836 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12474 (3686815630 0 291996384) 0x55d99a413440 con 0x55d986bec000 > -7794> 2018-07-11 07:50:54.416085 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -7793> 2018-07-11 07:50:54.416124 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -7792> 2018-07-11 07:50:54.416183 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -7791> 2018-07-11 07:50:54.416657 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -7790> 2018-07-11 07:50:54.438201 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67842,epoch:501/498,start:5:752b26cd:::benchmark_data_magna115_11707_object432310:0,end:5:752b8ea7:::benchmark_data_magna115_10867_object3094031:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb772a00 con 0 > -7789> 2018-07-11 07:50:54.438258 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67842,epoch:501/498,start:5:752b26cd:::benchmark_data_magna115_11707_object432310:0,end:5:752b8ea7:::benchmark_data_magna115_10867_object3094031:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb772780 con 0 > -7788> 2018-07-11 07:50:54.478198 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15287 conn(0x55d98123f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 249 0x55d9bcd72c00 osd_ping(ping e501 stamp 2018-07-11 07:50:55.708542) v4 > -7787> 2018-07-11 07:50:54.478197 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15287 conn(0x55d981240800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 249 0x55d986d32400 osd_ping(ping e501 stamp 2018-07-11 07:50:55.708542) v4 > -7786> 2018-07-11 07:50:54.478227 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.4 10.8.128.113:0/15287 249 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:55.708542) v4 ==== 2004+0+0 (378192231 0 0) 0x55d9bcd72c00 con 0x55d98123f000 > -7785> 2018-07-11 07:50:54.478229 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.4 10.8.128.113:0/15287 249 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:55.708542) v4 ==== 2004+0+0 (378192231 0 0) 0x55d986d32400 con 0x55d981240800 > -7784> 2018-07-11 07:50:54.478242 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.708542) v4 -- 0x55d9c34cde00 con 0 > -7783> 2018-07-11 07:50:54.478301 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.708542) v4 -- 0x55d9c4c1ae00 con 0 > -7782> 2018-07-11 07:50:54.587845 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.114:0/15919 conn(0x55d986df3800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.11 seq 241 0x55d982cdfc00 osd_ping(ping e501 stamp 2018-07-11 07:50:55.217853) v4 > -7781> 2018-07-11 07:50:54.587866 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.11 10.8.128.114:0/15919 241 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:55.217853) v4 ==== 2004+0+0 (516776557 0 0) 0x55d982cdfc00 con 0x55d986df3800 > -7780> 2018-07-11 07:50:54.587875 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.217853) v4 -- 0x55d985054a00 con 0 > -7779> 2018-07-11 07:50:54.587898 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.114:0/15919 conn(0x55d986df5000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.11 seq 241 0x55d9829e5000 osd_ping(ping e501 stamp 2018-07-11 07:50:55.217853) v4 > -7778> 2018-07-11 07:50:54.587920 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.11 10.8.128.114:0/15919 241 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:55.217853) v4 ==== 2004+0+0 (516776557 0 0) 0x55d9829e5000 con 0x55d986df5000 > -7777> 2018-07-11 07:50:54.587931 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.217853) v4 -- 0x55d9c4c1ae00 con 0 > -7776> 2018-07-11 07:50:54.648510 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 843 0x55d99a413680 rep_scrubmap(5.2e e501 from shard 2) v1 > -7775> 2018-07-11 07:50:54.648530 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 843 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12474 (1269249638 0 2027383426) 0x55d99a413680 con 0x55d986bf0800 > -7774> 2018-07-11 07:50:54.664038 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 837 0x55d99a4138c0 rep_scrubmap(5.2e e501 from shard 4) v1 > -7773> 2018-07-11 07:50:54.664058 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 837 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12474 (3686815630 0 2027383426) 0x55d99a4138c0 con 0x55d986bec000 > -7772> 2018-07-11 07:50:54.743462 7ffacbbd1700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15263 conn(0x55d985d44000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.5 seq 223 0x55d9b9ac3a00 osd_ping(ping e501 stamp 2018-07-11 07:50:55.973823) v4 > -7771> 2018-07-11 07:50:54.743484 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 <== osd.5 10.8.128.113:0/15263 223 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:55.973823) v4 ==== 2004+0+0 (108390259 0 0) 0x55d9b9ac3a00 con 0x55d985d44000 > -7770> 2018-07-11 07:50:54.743493 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.973823) v4 -- 0x55d985054a00 con 0 > -7769> 2018-07-11 07:50:54.743498 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15263 conn(0x55d985d45800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.5 seq 223 0x55d9bcd72c00 osd_ping(ping e501 stamp 2018-07-11 07:50:55.973823) v4 > -7768> 2018-07-11 07:50:54.743521 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.5 10.8.128.113:0/15263 223 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:55.973823) v4 ==== 2004+0+0 (108390259 0 0) 0x55d9bcd72c00 con 0x55d985d45800 > -7767> 2018-07-11 07:50:54.743537 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.973823) v4 -- 0x55d9c4c1ae00 con 0 > -7766> 2018-07-11 07:50:54.759848 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12677 conn(0x55d986db4800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.1 seq 244 0x55d9829e5000 osd_ping(ping e501 stamp 2018-07-11 07:50:54.807088) v4 > -7765> 2018-07-11 07:50:54.759873 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.1 10.8.128.106:0/12677 244 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:54.807088) v4 ==== 2004+0+0 (2028423481 0 0) 0x55d9829e5000 con 0x55d986db4800 > -7764> 2018-07-11 07:50:54.759870 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12677 conn(0x55d986df2000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.1 seq 244 0x55d9b67d8400 osd_ping(ping e501 stamp 2018-07-11 07:50:54.807088) v4 > -7763> 2018-07-11 07:50:54.759889 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.1 10.8.128.106:0/12677 244 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:54.807088) v4 ==== 2004+0+0 (2028423481 0 0) 0x55d9b67d8400 con 0x55d986df2000 > -7762> 2018-07-11 07:50:54.759890 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:54.807088) v4 -- 0x55d9c4c1ae00 con 0 > -7761> 2018-07-11 07:50:54.759924 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:54.807088) v4 -- 0x55d9c34cde00 con 0 > -7760> 2018-07-11 07:50:54.772160 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -7759> 2018-07-11 07:50:54.772189 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -7758> 2018-07-11 07:50:54.772259 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -7757> 2018-07-11 07:50:54.772745 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -7756> 2018-07-11 07:50:54.789182 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:752b8ea7:::benchmark_data_magna115_10867_object3094031:0,end:5:752beb79:::benchmark_data_magna115_10867_object3130554:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb772c80 con 0 > -7755> 2018-07-11 07:50:54.789233 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:752b8ea7:::benchmark_data_magna115_10867_object3094031:0,end:5:752beb79:::benchmark_data_magna115_10867_object3130554:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb772f00 con 0 > -7754> 2018-07-11 07:50:54.833205 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 2986 0x55d982cdfc00 MOSDScrubReserve(11.3s5 REQUEST e501) v1 > -7753> 2018-07-11 07:50:54.833223 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 2986 ==== MOSDScrubReserve(11.3s5 REQUEST e501) v1 ==== 43+0+0 (3260170403 0 0) 0x55d982cdfc00 con 0x55d986c7e000 > -7752> 2018-07-11 07:50:54.833344 7ffab3be0700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e501) v1 -- 0x55d9bdba4400 con 0 > -7751> 2018-07-11 07:50:54.833676 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 2987 0x55d985054a00 MOSDScrubReserve(11.3s5 RELEASE e501) v1 > -7750> 2018-07-11 07:50:54.833701 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 2987 ==== MOSDScrubReserve(11.3s5 RELEASE e501) v1 ==== 43+0+0 (223328929 0 0) 0x55d985054a00 con 0x55d986c7e000 > -7749> 2018-07-11 07:50:54.978640 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15287 conn(0x55d981240800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 250 0x55d9b67d8400 osd_ping(ping e501 stamp 2018-07-11 07:50:56.209017) v4 > -7748> 2018-07-11 07:50:54.978663 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.4 10.8.128.113:0/15287 250 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:56.209017) v4 ==== 2004+0+0 (1399640360 0 0) 0x55d9b67d8400 con 0x55d981240800 > -7747> 2018-07-11 07:50:54.978680 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:56.209017) v4 -- 0x55d9c34cde00 con 0 > -7746> 2018-07-11 07:50:54.978668 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15287 conn(0x55d98123f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 250 0x55d9bcd72c00 osd_ping(ping e501 stamp 2018-07-11 07:50:56.209017) v4 > -7745> 2018-07-11 07:50:54.978693 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.4 10.8.128.113:0/15287 250 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:56.209017) v4 ==== 2004+0+0 (1399640360 0 0) 0x55d9bcd72c00 con 0x55d98123f000 > -7744> 2018-07-11 07:50:54.978710 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:56.209017) v4 -- 0x55d9c4c1ae00 con 0 > -7743> 2018-07-11 07:50:54.999532 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 838 0x55d99a413b00 rep_scrubmap(5.2e e501 from shard 4) v1 > -7742> 2018-07-11 07:50:54.999552 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 838 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12474 (3686815630 0 646836372) 0x55d99a413b00 con 0x55d986bec000 > -7741> 2018-07-11 07:50:55.032474 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 844 0x55d99a413d40 rep_scrubmap(5.2e e501 from shard 2) v1 > -7740> 2018-07-11 07:50:55.032494 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 844 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12474 (1269249638 0 646836372) 0x55d99a413d40 con 0x55d986bf0800 > -7739> 2018-07-11 07:50:55.101585 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -7738> 2018-07-11 07:50:55.101609 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -7737> 2018-07-11 07:50:55.101671 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -7736> 2018-07-11 07:50:55.102126 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -7735> 2018-07-11 07:50:55.147787 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:752beb79:::benchmark_data_magna115_10867_object3130554:0,end:5:752c52a4:::benchmark_data_magna115_11707_object176972:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb773900 con 0 > -7734> 2018-07-11 07:50:55.147838 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:752beb79:::benchmark_data_magna115_10867_object3130554:0,end:5:752c52a4:::benchmark_data_magna115_11707_object176972:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb773400 con 0 > -7733> 2018-07-11 07:50:55.212195 7ffab1bdc700 5 osd.10 501 heartbeat: osd_stat(788 GB used, 143 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist []) > -7732> 2018-07-11 07:50:55.212208 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6802/15255 -- osd_ping(ping e501 stamp 2018-07-11 07:50:55.212206) v4 -- 0x55d9b40a4e00 con 0 > -7731> 2018-07-11 07:50:55.212244 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6803/15255 -- osd_ping(ping e501 stamp 2018-07-11 07:50:55.212206) v4 -- 0x55d9c3383a00 con 0 > -7730> 2018-07-11 07:50:55.212270 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6806/12677 -- osd_ping(ping e501 stamp 2018-07-11 07:50:55.212206) v4 -- 0x55d9b437ac00 con 0 > -7729> 2018-07-11 07:50:55.212297 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6807/12677 -- osd_ping(ping e501 stamp 2018-07-11 07:50:55.212206) v4 -- 0x55d985055000 con 0 > -7728> 2018-07-11 07:50:55.212309 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6808/12651 -- osd_ping(ping e501 stamp 2018-07-11 07:50:55.212206) v4 -- 0x55d9c908e800 con 0 > -7727> 2018-07-11 07:50:55.212321 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6810/12651 -- osd_ping(ping e501 stamp 2018-07-11 07:50:55.212206) v4 -- 0x55d9c4c1b600 con 0 > -7726> 2018-07-11 07:50:55.212338 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6809/12650 -- osd_ping(ping e501 stamp 2018-07-11 07:50:55.212206) v4 -- 0x55d986d33200 con 0 > -7725> 2018-07-11 07:50:55.212351 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6811/12650 -- osd_ping(ping e501 stamp 2018-07-11 07:50:55.212206) v4 -- 0x55d986c23c00 con 0 > -7724> 2018-07-11 07:50:55.212361 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6810/15287 -- osd_ping(ping e501 stamp 2018-07-11 07:50:55.212206) v4 -- 0x55d9b60dee00 con 0 > -7723> 2018-07-11 07:50:55.212371 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6811/15287 -- osd_ping(ping e501 stamp 2018-07-11 07:50:55.212206) v4 -- 0x55d9bf804800 con 0 > -7722> 2018-07-11 07:50:55.212386 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6806/15263 -- osd_ping(ping e501 stamp 2018-07-11 07:50:55.212206) v4 -- 0x55d9829e4a00 con 0 > -7721> 2018-07-11 07:50:55.212400 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6807/15263 -- osd_ping(ping e501 stamp 2018-07-11 07:50:55.212206) v4 -- 0x55d9b2783400 con 0 > -7720> 2018-07-11 07:50:55.212412 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.115:6802/13323 -- osd_ping(ping e501 stamp 2018-07-11 07:50:55.212206) v4 -- 0x55d9c4c1be00 con 0 > -7719> 2018-07-11 07:50:55.212424 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.115:6803/13323 -- osd_ping(ping e501 stamp 2018-07-11 07:50:55.212206) v4 -- 0x55d986d40600 con 0 > -7718> 2018-07-11 07:50:55.212437 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6806/15885 -- osd_ping(ping e501 stamp 2018-07-11 07:50:55.212206) v4 -- 0x55d9829e5600 con 0 > -7717> 2018-07-11 07:50:55.212449 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6807/15885 -- osd_ping(ping e501 stamp 2018-07-11 07:50:55.212206) v4 -- 0x55d9b40a4000 con 0 > -7716> 2018-07-11 07:50:55.212465 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.108:6802/11967 -- osd_ping(ping e501 stamp 2018-07-11 07:50:55.212206) v4 -- 0x55d980f5a800 con 0 > -7715> 2018-07-11 07:50:55.212478 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.108:6803/11967 -- osd_ping(ping e501 stamp 2018-07-11 07:50:55.212206) v4 -- 0x55d980f5b400 con 0 > -7714> 2018-07-11 07:50:55.212491 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.107:6806/12617 -- osd_ping(ping e501 stamp 2018-07-11 07:50:55.212206) v4 -- 0x55d9829e5200 con 0 > -7713> 2018-07-11 07:50:55.212506 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.107:6807/12617 -- osd_ping(ping e501 stamp 2018-07-11 07:50:55.212206) v4 -- 0x55d9b40a5800 con 0 > -7712> 2018-07-11 07:50:55.212518 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6802/15919 -- osd_ping(ping e501 stamp 2018-07-11 07:50:55.212206) v4 -- 0x55d9bf804600 con 0 > -7711> 2018-07-11 07:50:55.212530 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6803/15919 -- osd_ping(ping e501 stamp 2018-07-11 07:50:55.212206) v4 -- 0x55d986d41e00 con 0 > -7710> 2018-07-11 07:50:55.212718 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.107:6806/12617 conn(0x55d986bd2000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.9 seq 230 0x55d9bcd72c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.212206) v4 > -7709> 2018-07-11 07:50:55.212741 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.9 10.8.128.107:6806/12617 230 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.212206) v4 ==== 2004+0+0 (1428929704 0 0) 0x55d9bcd72c00 con 0x55d986bd2000 > -7708> 2018-07-11 07:50:55.212736 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.107:6807/12617 conn(0x55d986bd3800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.9 seq 230 0x55d9b9ac3a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.212206) v4 > -7707> 2018-07-11 07:50:55.212780 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.9 10.8.128.107:6807/12617 230 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.212206) v4 ==== 2004+0+0 (1428929704 0 0) 0x55d9b9ac3a00 con 0x55d986bd3800 > -7706> 2018-07-11 07:50:55.212778 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6802/15255 conn(0x55d986bc8000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.0 seq 230 0x55d9b67d8400 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.212206) v4 > -7705> 2018-07-11 07:50:55.212806 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.0 10.8.128.113:6802/15255 230 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.212206) v4 ==== 2004+0+0 (1428929704 0 0) 0x55d9b67d8400 con 0x55d986bc8000 > -7704> 2018-07-11 07:50:55.212820 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6809/12650 conn(0x55d986bcb000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 230 0x55d9b9ac3a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.212206) v4 > -7703> 2018-07-11 07:50:55.212842 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6803/15255 conn(0x55d986bc9800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.0 seq 230 0x55d9bcd72c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.212206) v4 > -7702> 2018-07-11 07:50:55.212833 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.3 10.8.128.106:6809/12650 230 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.212206) v4 ==== 2004+0+0 (1428929704 0 0) 0x55d9b9ac3a00 con 0x55d986bcb000 > -7701> 2018-07-11 07:50:55.212846 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6808/12651 conn(0x55d986b9a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.2 seq 230 0x55d9b67d8400 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.212206) v4 > -7700> 2018-07-11 07:50:55.212864 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.2 10.8.128.106:6808/12651 230 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.212206) v4 ==== 2004+0+0 (1428929704 0 0) 0x55d9b67d8400 con 0x55d986b9a000 > -7699> 2018-07-11 07:50:55.212861 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.0 10.8.128.113:6803/15255 230 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.212206) v4 ==== 2004+0+0 (1428929704 0 0) 0x55d9bcd72c00 con 0x55d986bc9800 > -7698> 2018-07-11 07:50:55.212888 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6806/15263 conn(0x55d986b9d000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.5 seq 230 0x55d9b9ac3a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.212206) v4 > -7697> 2018-07-11 07:50:55.212901 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.5 10.8.128.113:6806/15263 230 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.212206) v4 ==== 2004+0+0 (1428929704 0 0) 0x55d9b9ac3a00 con 0x55d986b9d000 > -7696> 2018-07-11 07:50:55.212901 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6807/15263 conn(0x55d986b9e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.5 seq 230 0x55d9b67d8400 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.212206) v4 > -7695> 2018-07-11 07:50:55.212911 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.5 10.8.128.113:6807/15263 230 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.212206) v4 ==== 2004+0+0 (1428929704 0 0) 0x55d9b67d8400 con 0x55d986b9e800 > -7694> 2018-07-11 07:50:55.212941 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6810/15287 conn(0x55d986baf000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 230 0x55d9b67d8400 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.212206) v4 > -7693> 2018-07-11 07:50:55.212946 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6806/12677 conn(0x55d986b58000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.1 seq 230 0x55d9b9ac3a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.212206) v4 > -7692> 2018-07-11 07:50:55.212944 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6810/12651 conn(0x55d986b9b800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.2 seq 230 0x55d9bcd72c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.212206) v4 > -7691> 2018-07-11 07:50:55.212952 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.4 10.8.128.113:6810/15287 230 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.212206) v4 ==== 2004+0+0 (1428929704 0 0) 0x55d9b67d8400 con 0x55d986baf000 > -7690> 2018-07-11 07:50:55.212960 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.1 10.8.128.106:6806/12677 230 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.212206) v4 ==== 2004+0+0 (1428929704 0 0) 0x55d9b9ac3a00 con 0x55d986b58000 > -7689> 2018-07-11 07:50:55.212963 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.2 10.8.128.106:6810/12651 230 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.212206) v4 ==== 2004+0+0 (1428929704 0 0) 0x55d9bcd72c00 con 0x55d986b9b800 > -7688> 2018-07-11 07:50:55.212982 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6811/12650 conn(0x55d986bcc800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 230 0x55d9b67d8400 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.212206) v4 > -7687> 2018-07-11 07:50:55.213010 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6811/15287 conn(0x55d986bb0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 230 0x55d9bcd72c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.212206) v4 > -7686> 2018-07-11 07:50:55.213013 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.3 10.8.128.106:6811/12650 230 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.212206) v4 ==== 2004+0+0 (1428929704 0 0) 0x55d9b67d8400 con 0x55d986bcc800 > -7685> 2018-07-11 07:50:55.213026 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.4 10.8.128.113:6811/15287 230 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.212206) v4 ==== 2004+0+0 (1428929704 0 0) 0x55d9bcd72c00 con 0x55d986bb0800 > -7684> 2018-07-11 07:50:55.213060 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.108:6802/11967 conn(0x55d986bbe000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.8 seq 230 0x55d9b9ac3a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.212206) v4 > -7683> 2018-07-11 07:50:55.213070 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6807/12677 conn(0x55d986b59800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.1 seq 230 0x55d9b67d8400 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.212206) v4 > -7682> 2018-07-11 07:50:55.213078 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.8 10.8.128.108:6802/11967 230 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.212206) v4 ==== 2004+0+0 (1428929704 0 0) 0x55d9b9ac3a00 con 0x55d986bbe000 > -7681> 2018-07-11 07:50:55.213081 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.1 10.8.128.106:6807/12677 230 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.212206) v4 ==== 2004+0+0 (1428929704 0 0) 0x55d9b67d8400 con 0x55d986b59800 > -7680> 2018-07-11 07:50:55.213066 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.115:6802/13323 conn(0x55d986bc1000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 230 0x55d9bcd72c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.212206) v4 > -7679> 2018-07-11 07:50:55.213109 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.115:6803/13323 conn(0x55d986bc2800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 230 0x55d9b9ac3a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.212206) v4 > -7678> 2018-07-11 07:50:55.213117 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.6 10.8.128.115:6803/13323 230 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.212206) v4 ==== 2004+0+0 (1428929704 0 0) 0x55d9b9ac3a00 con 0x55d986bc2800 > -7677> 2018-07-11 07:50:55.213114 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.6 10.8.128.115:6802/13323 230 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.212206) v4 ==== 2004+0+0 (1428929704 0 0) 0x55d9bcd72c00 con 0x55d986bc1000 > -7676> 2018-07-11 07:50:55.213125 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.108:6803/11967 conn(0x55d986bbf800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.8 seq 230 0x55d9b67d8400 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.212206) v4 > -7675> 2018-07-11 07:50:55.213138 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.8 10.8.128.108:6803/11967 230 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.212206) v4 ==== 2004+0+0 (1428929704 0 0) 0x55d9b67d8400 con 0x55d986bbf800 > -7674> 2018-07-11 07:50:55.213148 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6807/15885 conn(0x55d986bad800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.7 seq 230 0x55d9b9ac3a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.212206) v4 > -7673> 2018-07-11 07:50:55.213158 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.7 10.8.128.114:6807/15885 230 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.212206) v4 ==== 2004+0+0 (1428929704 0 0) 0x55d9b9ac3a00 con 0x55d986bad800 > -7672> 2018-07-11 07:50:55.213178 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6802/15919 conn(0x55d986b5b000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.11 seq 230 0x55d9bcd72c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.212206) v4 > -7671> 2018-07-11 07:50:55.213193 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6803/15919 conn(0x55d986b5c800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.11 seq 230 0x55d9b9ac3a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.212206) v4 > -7670> 2018-07-11 07:50:55.213196 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.11 10.8.128.114:6802/15919 230 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.212206) v4 ==== 2004+0+0 (1428929704 0 0) 0x55d9bcd72c00 con 0x55d986b5b000 > -7669> 2018-07-11 07:50:55.213205 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.11 10.8.128.114:6803/15919 230 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.212206) v4 ==== 2004+0+0 (1428929704 0 0) 0x55d9b9ac3a00 con 0x55d986b5c800 > -7668> 2018-07-11 07:50:55.213237 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6806/15885 conn(0x55d986bac000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.7 seq 230 0x55d9bcd72c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.212206) v4 > -7667> 2018-07-11 07:50:55.213262 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.7 10.8.128.114:6806/15885 230 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.212206) v4 ==== 2004+0+0 (1428929704 0 0) 0x55d9bcd72c00 con 0x55d986bac000 > -7666> 2018-07-11 07:50:55.313433 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.114:0/15885 conn(0x55d986d81000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.7 seq 244 0x55d9b67d8400 osd_ping(ping e501 stamp 2018-07-11 07:50:55.943413) v4 > -7665> 2018-07-11 07:50:55.313436 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.114:0/15885 conn(0x55d986d7f800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.7 seq 244 0x55d9bcd72c00 osd_ping(ping e501 stamp 2018-07-11 07:50:55.943413) v4 > -7664> 2018-07-11 07:50:55.313456 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.7 10.8.128.114:0/15885 244 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:55.943413) v4 ==== 2004+0+0 (775450838 0 0) 0x55d9b67d8400 con 0x55d986d81000 > -7663> 2018-07-11 07:50:55.313460 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.7 10.8.128.114:0/15885 244 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:55.943413) v4 ==== 2004+0+0 (775450838 0 0) 0x55d9bcd72c00 con 0x55d986d7f800 > -7662> 2018-07-11 07:50:55.313468 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.943413) v4 -- 0x55d980f5b400 con 0 > -7661> 2018-07-11 07:50:55.313503 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.943413) v4 -- 0x55d9bf804600 con 0 > -7660> 2018-07-11 07:50:55.351261 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 845 0x55d9c369a000 rep_scrubmap(5.2e e501 from shard 2) v1 > -7659> 2018-07-11 07:50:55.351280 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 845 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12472 (1269249638 0 1402167514) 0x55d9c369a000 con 0x55d986bf0800 > -7658> 2018-07-11 07:50:55.380617 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 839 0x55d9c369a240 rep_scrubmap(5.2e e501 from shard 4) v1 > -7657> 2018-07-11 07:50:55.380637 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 839 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12472 (3686815630 0 1402167514) 0x55d9c369a240 con 0x55d986bec000 > -7656> 2018-07-11 07:50:55.405001 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -7655> 2018-07-11 07:50:55.405039 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -7654> 2018-07-11 07:50:55.405098 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -7653> 2018-07-11 07:50:55.405581 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -7652> 2018-07-11 07:50:55.406648 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:752c52a4:::benchmark_data_magna115_11707_object176972:0,end:5:752cec38:::benchmark_data_magna115_10867_object3781344:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb773180 con 0 > -7651> 2018-07-11 07:50:55.406694 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:752c52a4:::benchmark_data_magna115_11707_object176972:0,end:5:752cec38:::benchmark_data_magna115_10867_object3781344:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454280 con 0 > -7650> 2018-07-11 07:50:55.558524 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6805/15885 conn(0x55d986c7f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.7 seq 2774 0x55d9b67d8400 MOSDScrubReserve(5.4 REQUEST e501) v1 > -7649> 2018-07-11 07:50:55.558554 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.7 10.8.128.114:6805/15885 2774 ==== MOSDScrubReserve(5.4 REQUEST e501) v1 ==== 43+0+0 (1753221512 0 0) 0x55d9b67d8400 con 0x55d986c7f800 > -7648> 2018-07-11 07:50:55.558638 7ffab33df700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.114:6805/15885 -- MOSDScrubReserve(5.4 REJECT e501) v1 -- 0x55d9bdba4800 con 0 > -7647> 2018-07-11 07:50:55.558965 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6805/15885 conn(0x55d986c7f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.7 seq 2775 0x55d9bdba4800 MOSDScrubReserve(5.4 RELEASE e501) v1 > -7646> 2018-07-11 07:50:55.558979 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.7 10.8.128.114:6805/15885 2775 ==== MOSDScrubReserve(5.4 RELEASE e501) v1 ==== 43+0+0 (2812146570 0 0) 0x55d9bdba4800 con 0x55d986c7f800 > -7645> 2018-07-11 07:50:55.611386 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 846 0x55d9c369a480 rep_scrubmap(5.2e e501 from shard 2) v1 > -7644> 2018-07-11 07:50:55.611406 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 846 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12466 (1269249638 0 3824574633) 0x55d9c369a480 con 0x55d986bf0800 > -7643> 2018-07-11 07:50:55.624836 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 840 0x55d9c369a6c0 rep_scrubmap(5.2e e501 from shard 4) v1 > -7642> 2018-07-11 07:50:55.624856 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 840 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12466 (3686815630 0 3824574633) 0x55d9c369a6c0 con 0x55d986bec000 > -7641> 2018-07-11 07:50:55.676564 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -7640> 2018-07-11 07:50:55.676588 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -7639> 2018-07-11 07:50:55.676644 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -7638> 2018-07-11 07:50:55.677107 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -7637> 2018-07-11 07:50:55.712643 7ffab1bdc700 5 osd.10 501 heartbeat: osd_stat(788 GB used, 143 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist []) > -7636> 2018-07-11 07:50:55.712655 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6802/15255 -- osd_ping(ping e501 stamp 2018-07-11 07:50:55.712654) v4 -- 0x55d9829e5800 con 0 > -7635> 2018-07-11 07:50:55.712691 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6803/15255 -- osd_ping(ping e501 stamp 2018-07-11 07:50:55.712654) v4 -- 0x55d982d30600 con 0 > -7634> 2018-07-11 07:50:55.712706 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6806/12677 -- osd_ping(ping e501 stamp 2018-07-11 07:50:55.712654) v4 -- 0x55d9b60df400 con 0 > -7633> 2018-07-11 07:50:55.712719 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6807/12677 -- osd_ping(ping e501 stamp 2018-07-11 07:50:55.712654) v4 -- 0x55d9b9ac2200 con 0 > -7632> 2018-07-11 07:50:55.712730 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6808/12651 -- osd_ping(ping e501 stamp 2018-07-11 07:50:55.712654) v4 -- 0x55d982cde200 con 0 > -7631> 2018-07-11 07:50:55.712741 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6810/12651 -- osd_ping(ping e501 stamp 2018-07-11 07:50:55.712654) v4 -- 0x55d9c908f600 con 0 > -7630> 2018-07-11 07:50:55.712753 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6809/12650 -- osd_ping(ping e501 stamp 2018-07-11 07:50:55.712654) v4 -- 0x55d9b3db3200 con 0 > -7629> 2018-07-11 07:50:55.712771 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6811/12650 -- osd_ping(ping e501 stamp 2018-07-11 07:50:55.712654) v4 -- 0x55d982d31600 con 0 > -7628> 2018-07-11 07:50:55.712782 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6810/15287 -- osd_ping(ping e501 stamp 2018-07-11 07:50:55.712654) v4 -- 0x55d97f5adc00 con 0 > -7627> 2018-07-11 07:50:55.712794 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6811/15287 -- osd_ping(ping e501 stamp 2018-07-11 07:50:55.712654) v4 -- 0x55d9b60df600 con 0 > -7626> 2018-07-11 07:50:55.712807 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6806/15263 -- osd_ping(ping e501 stamp 2018-07-11 07:50:55.712654) v4 -- 0x55d982d31e00 con 0 > -7625> 2018-07-11 07:50:55.712819 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6807/15263 -- osd_ping(ping e501 stamp 2018-07-11 07:50:55.712654) v4 -- 0x55d9b9ac3600 con 0 > -7624> 2018-07-11 07:50:55.712832 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.115:6802/13323 -- osd_ping(ping e501 stamp 2018-07-11 07:50:55.712654) v4 -- 0x55d9bcd72800 con 0 > -7623> 2018-07-11 07:50:55.712845 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.115:6803/13323 -- osd_ping(ping e501 stamp 2018-07-11 07:50:55.712654) v4 -- 0x55d9c908ee00 con 0 > -7622> 2018-07-11 07:50:55.712857 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6806/15885 -- osd_ping(ping e501 stamp 2018-07-11 07:50:55.712654) v4 -- 0x55d9b3db3600 con 0 > -7621> 2018-07-11 07:50:55.712869 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6807/15885 -- osd_ping(ping e501 stamp 2018-07-11 07:50:55.712654) v4 -- 0x55d9b3db2600 con 0 > -7620> 2018-07-11 07:50:55.712882 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.108:6802/11967 -- osd_ping(ping e501 stamp 2018-07-11 07:50:55.712654) v4 -- 0x55d9b3db3c00 con 0 > -7619> 2018-07-11 07:50:55.712894 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.108:6803/11967 -- osd_ping(ping e501 stamp 2018-07-11 07:50:55.712654) v4 -- 0x55d982cdfa00 con 0 > -7618> 2018-07-11 07:50:55.712906 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.107:6806/12617 -- osd_ping(ping e501 stamp 2018-07-11 07:50:55.712654) v4 -- 0x55d982d31c00 con 0 > -7617> 2018-07-11 07:50:55.712921 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.107:6807/12617 -- osd_ping(ping e501 stamp 2018-07-11 07:50:55.712654) v4 -- 0x55d9b9ac3400 con 0 > -7616> 2018-07-11 07:50:55.712934 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6802/15919 -- osd_ping(ping e501 stamp 2018-07-11 07:50:55.712654) v4 -- 0x55d982cdf400 con 0 > -7615> 2018-07-11 07:50:55.712946 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6803/15919 -- osd_ping(ping e501 stamp 2018-07-11 07:50:55.712654) v4 -- 0x55d9bdba4800 con 0 > -7614> 2018-07-11 07:50:55.713102 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.107:6806/12617 conn(0x55d986bd2000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.9 seq 231 0x55d9bcd72c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.712654) v4 > -7613> 2018-07-11 07:50:55.713108 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.107:6807/12617 conn(0x55d986bd3800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.9 seq 231 0x55d9b9ac3a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.712654) v4 > -7612> 2018-07-11 07:50:55.713123 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.9 10.8.128.107:6806/12617 231 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.712654) v4 ==== 2004+0+0 (2004154156 0 0) 0x55d9bcd72c00 con 0x55d986bd2000 > -7611> 2018-07-11 07:50:55.713125 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.9 10.8.128.107:6807/12617 231 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.712654) v4 ==== 2004+0+0 (2004154156 0 0) 0x55d9b9ac3a00 con 0x55d986bd3800 > -7610> 2018-07-11 07:50:55.713196 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6802/15255 conn(0x55d986bc8000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.0 seq 231 0x55d986d32400 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.712654) v4 > -7609> 2018-07-11 07:50:55.713231 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.0 10.8.128.113:6802/15255 231 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.712654) v4 ==== 2004+0+0 (2004154156 0 0) 0x55d986d32400 con 0x55d986bc8000 > -7608> 2018-07-11 07:50:55.713237 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6810/12651 conn(0x55d986b9b800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.2 seq 231 0x55d9bcd72c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.712654) v4 > -7607> 2018-07-11 07:50:55.713269 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.2 10.8.128.106:6810/12651 231 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.712654) v4 ==== 2004+0+0 (2004154156 0 0) 0x55d9bcd72c00 con 0x55d986b9b800 > -7606> 2018-07-11 07:50:55.713295 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6807/12677 conn(0x55d986b59800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.1 seq 231 0x55d986d32400 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.712654) v4 > -7605> 2018-07-11 07:50:55.713309 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.1 10.8.128.106:6807/12677 231 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.712654) v4 ==== 2004+0+0 (2004154156 0 0) 0x55d986d32400 con 0x55d986b59800 > -7604> 2018-07-11 07:50:55.713309 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6803/15255 conn(0x55d986bc9800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.0 seq 231 0x55d9bcd72c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.712654) v4 > -7603> 2018-07-11 07:50:55.713319 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.0 10.8.128.113:6803/15255 231 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.712654) v4 ==== 2004+0+0 (2004154156 0 0) 0x55d9bcd72c00 con 0x55d986bc9800 > -7602> 2018-07-11 07:50:55.713309 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6806/15263 conn(0x55d986b9d000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.5 seq 231 0x55d9b9ac3a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.712654) v4 > -7601> 2018-07-11 07:50:55.713346 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.5 10.8.128.113:6806/15263 231 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.712654) v4 ==== 2004+0+0 (2004154156 0 0) 0x55d9b9ac3a00 con 0x55d986b9d000 > -7600> 2018-07-11 07:50:55.713338 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6808/12651 conn(0x55d986b9a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.2 seq 231 0x55d986d32400 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.712654) v4 > -7599> 2018-07-11 07:50:55.713364 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.2 10.8.128.106:6808/12651 231 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.712654) v4 ==== 2004+0+0 (2004154156 0 0) 0x55d986d32400 con 0x55d986b9a000 > -7598> 2018-07-11 07:50:55.713372 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6811/15287 conn(0x55d986bb0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 231 0x55d9bcd72c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.712654) v4 > -7597> 2018-07-11 07:50:55.713387 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.4 10.8.128.113:6811/15287 231 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.712654) v4 ==== 2004+0+0 (2004154156 0 0) 0x55d9bcd72c00 con 0x55d986bb0800 > -7596> 2018-07-11 07:50:55.713396 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6811/12650 conn(0x55d986bcc800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 231 0x55d986d32400 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.712654) v4 > -7595> 2018-07-11 07:50:55.713399 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6809/12650 conn(0x55d986bcb000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 231 0x55d9b9ac3a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.712654) v4 > -7594> 2018-07-11 07:50:55.713408 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.3 10.8.128.106:6811/12650 231 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.712654) v4 ==== 2004+0+0 (2004154156 0 0) 0x55d986d32400 con 0x55d986bcc800 > -7593> 2018-07-11 07:50:55.713437 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.3 10.8.128.106:6809/12650 231 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.712654) v4 ==== 2004+0+0 (2004154156 0 0) 0x55d9b9ac3a00 con 0x55d986bcb000 > -7592> 2018-07-11 07:50:55.713461 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6807/15263 conn(0x55d986b9e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.5 seq 231 0x55d986d32400 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.712654) v4 > -7591> 2018-07-11 07:50:55.713462 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.115:6802/13323 conn(0x55d986bc1000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 231 0x55d9bcd72c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.712654) v4 > -7590> 2018-07-11 07:50:55.713474 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.5 10.8.128.113:6807/15263 231 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.712654) v4 ==== 2004+0+0 (2004154156 0 0) 0x55d986d32400 con 0x55d986b9e800 > -7589> 2018-07-11 07:50:55.713479 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.6 10.8.128.115:6802/13323 231 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.712654) v4 ==== 2004+0+0 (2004154156 0 0) 0x55d9bcd72c00 con 0x55d986bc1000 > -7588> 2018-07-11 07:50:55.713480 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6806/12677 conn(0x55d986b58000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.1 seq 231 0x55d9b9ac3a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.712654) v4 > -7587> 2018-07-11 07:50:55.713506 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6810/15287 conn(0x55d986baf000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 231 0x55d986d32400 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.712654) v4 > -7586> 2018-07-11 07:50:55.713526 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.4 10.8.128.113:6810/15287 231 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.712654) v4 ==== 2004+0+0 (2004154156 0 0) 0x55d986d32400 con 0x55d986baf000 > -7585> 2018-07-11 07:50:55.713522 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.1 10.8.128.106:6806/12677 231 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.712654) v4 ==== 2004+0+0 (2004154156 0 0) 0x55d9b9ac3a00 con 0x55d986b58000 > -7584> 2018-07-11 07:50:55.713536 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6806/15885 conn(0x55d986bac000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.7 seq 231 0x55d9bcd72c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.712654) v4 > -7583> 2018-07-11 07:50:55.713551 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.7 10.8.128.114:6806/15885 231 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.712654) v4 ==== 2004+0+0 (2004154156 0 0) 0x55d9bcd72c00 con 0x55d986bac000 > -7582> 2018-07-11 07:50:55.713557 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.108:6803/11967 conn(0x55d986bbf800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.8 seq 231 0x55d986d32400 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.712654) v4 > -7581> 2018-07-11 07:50:55.713568 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.8 10.8.128.108:6803/11967 231 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.712654) v4 ==== 2004+0+0 (2004154156 0 0) 0x55d986d32400 con 0x55d986bbf800 > -7580> 2018-07-11 07:50:55.713579 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.115:6803/13323 conn(0x55d986bc2800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 231 0x55d9b9ac3a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.712654) v4 > -7579> 2018-07-11 07:50:55.713595 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.6 10.8.128.115:6803/13323 231 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.712654) v4 ==== 2004+0+0 (2004154156 0 0) 0x55d9b9ac3a00 con 0x55d986bc2800 > -7578> 2018-07-11 07:50:55.713636 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6803/15919 conn(0x55d986b5c800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.11 seq 231 0x55d9b9ac3a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.712654) v4 > -7577> 2018-07-11 07:50:55.713650 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.11 10.8.128.114:6803/15919 231 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.712654) v4 ==== 2004+0+0 (2004154156 0 0) 0x55d9b9ac3a00 con 0x55d986b5c800 > -7576> 2018-07-11 07:50:55.713687 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.108:6802/11967 conn(0x55d986bbe000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.8 seq 231 0x55d9b9ac3a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.712654) v4 > -7575> 2018-07-11 07:50:55.713700 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.8 10.8.128.108:6802/11967 231 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.712654) v4 ==== 2004+0+0 (2004154156 0 0) 0x55d9b9ac3a00 con 0x55d986bbe000 > -7574> 2018-07-11 07:50:55.713697 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6802/15919 conn(0x55d986b5b000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.11 seq 231 0x55d9bcd72c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.712654) v4 > -7573> 2018-07-11 07:50:55.713713 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.11 10.8.128.114:6802/15919 231 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.712654) v4 ==== 2004+0+0 (2004154156 0 0) 0x55d9bcd72c00 con 0x55d986b5b000 > -7572> 2018-07-11 07:50:55.713738 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6807/15885 conn(0x55d986bad800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.7 seq 231 0x55d9b9ac3a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.712654) v4 > -7571> 2018-07-11 07:50:55.713751 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.7 10.8.128.114:6807/15885 231 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:50:55.712654) v4 ==== 2004+0+0 (2004154156 0 0) 0x55d9b9ac3a00 con 0x55d986bad800 > -7570> 2018-07-11 07:50:55.731153 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:752cec38:::benchmark_data_magna115_10867_object3781344:0,end:5:752d4eec:::benchmark_data_magna115_10867_object2468505:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb773680 con 0 > -7569> 2018-07-11 07:50:55.731187 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:752cec38:::benchmark_data_magna115_10867_object3781344:0,end:5:752d4eec:::benchmark_data_magna115_10867_object2468505:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454000 con 0 > -7568> 2018-07-11 07:50:55.813804 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.114:0/15885 conn(0x55d986d7f800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.7 seq 245 0x55d9bcd72c00 osd_ping(ping e501 stamp 2018-07-11 07:50:56.443810) v4 > -7567> 2018-07-11 07:50:55.813830 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.7 10.8.128.114:0/15885 245 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:56.443810) v4 ==== 2004+0+0 (517153329 0 0) 0x55d9bcd72c00 con 0x55d986d7f800 > -7566> 2018-07-11 07:50:55.813833 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.114:0/15885 conn(0x55d986d81000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.7 seq 245 0x55d980f5b400 osd_ping(ping e501 stamp 2018-07-11 07:50:56.443810) v4 > -7565> 2018-07-11 07:50:55.813847 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:56.443810) v4 -- 0x55d982cdf400 con 0 > -7564> 2018-07-11 07:50:55.813858 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.7 10.8.128.114:0/15885 245 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:56.443810) v4 ==== 2004+0+0 (517153329 0 0) 0x55d980f5b400 con 0x55d986d81000 > -7563> 2018-07-11 07:50:55.813876 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:56.443810) v4 -- 0x55d982cdfa00 con 0 > -7562> 2018-07-11 07:50:55.940769 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 847 0x55d9c369a900 rep_scrubmap(5.2e e501 from shard 2) v1 > -7561> 2018-07-11 07:50:55.940789 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 847 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12480 (1269249638 0 903393374) 0x55d9c369a900 con 0x55d986bf0800 > -7560> 2018-07-11 07:50:55.957386 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6801/15919 conn(0x55d986c3f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=29 cs=1 l=0). rx osd.11 seq 216 0x55d9b9ac3a00 MOSDScrubReserve(10.f REQUEST e501) v1 > -7559> 2018-07-11 07:50:55.957404 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.11 10.8.128.114:6801/15919 216 ==== MOSDScrubReserve(10.f REQUEST e501) v1 ==== 43+0+0 (2892966580 0 0) 0x55d9b9ac3a00 con 0x55d986c3f800 > -7558> 2018-07-11 07:50:55.957460 7ffab53e3700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.114:6801/15919 -- MOSDScrubReserve(10.f REJECT e501) v1 -- 0x55d9bf805800 con 0 > -7557> 2018-07-11 07:50:55.957710 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 841 0x55d9c369ab40 rep_scrubmap(5.2e e501 from shard 4) v1 > -7556> 2018-07-11 07:50:55.957729 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 841 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12480 (3686815630 0 903393374) 0x55d9c369ab40 con 0x55d986bec000 > -7555> 2018-07-11 07:50:55.957780 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6801/15919 conn(0x55d986c3f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=29 cs=1 l=0). rx osd.11 seq 217 0x55d9bf805800 MOSDScrubReserve(10.f RELEASE e501) v1 > -7554> 2018-07-11 07:50:55.957795 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.11 10.8.128.114:6801/15919 217 ==== MOSDScrubReserve(10.f RELEASE e501) v1 ==== 43+0+0 (1668468918 0 0) 0x55d9bf805800 con 0x55d986c3f800 > -7553> 2018-07-11 07:50:55.961957 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15255 conn(0x55d986c82800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.0 seq 240 0x55d9869bfa00 osd_ping(ping e501 stamp 2018-07-11 07:50:57.192268) v4 > -7552> 2018-07-11 07:50:55.961965 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15255 conn(0x55d986d20000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.0 seq 240 0x55d986d32400 osd_ping(ping e501 stamp 2018-07-11 07:50:57.192268) v4 > -7551> 2018-07-11 07:50:55.961991 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.0 10.8.128.113:0/15255 240 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:57.192268) v4 ==== 2004+0+0 (3940458769 0 0) 0x55d9869bfa00 con 0x55d986c82800 > -7550> 2018-07-11 07:50:55.961995 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.0 10.8.128.113:0/15255 240 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:57.192268) v4 ==== 2004+0+0 (3940458769 0 0) 0x55d986d32400 con 0x55d986d20000 > -7549> 2018-07-11 07:50:55.962004 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:57.192268) v4 -- 0x55d982cdf400 con 0 > -7548> 2018-07-11 07:50:55.962027 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:57.192268) v4 -- 0x55d982cdfa00 con 0 > -7547> 2018-07-11 07:50:56.026835 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -7546> 2018-07-11 07:50:56.026861 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -7545> 2018-07-11 07:50:56.026917 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -7544> 2018-07-11 07:50:56.027387 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -7543> 2018-07-11 07:50:56.028422 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:752d4eec:::benchmark_data_magna115_10867_object2468505:0,end:5:752d957f:::benchmark_data_magna115_10867_object3657212:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb773b80 con 0 > -7542> 2018-07-11 07:50:56.028457 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:752d4eec:::benchmark_data_magna115_10867_object2468505:0,end:5:752d957f:::benchmark_data_magna115_10867_object3657212:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98944b680 con 0 > -7541> 2018-07-11 07:50:56.079079 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15287 conn(0x55d981240800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 251 0x55d986d32400 osd_ping(ping e501 stamp 2018-07-11 07:50:57.309464) v4 > -7540> 2018-07-11 07:50:56.079101 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.4 10.8.128.113:0/15287 251 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:57.309464) v4 ==== 2004+0+0 (213742363 0 0) 0x55d986d32400 con 0x55d981240800 > -7539> 2018-07-11 07:50:56.079110 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:57.309464) v4 -- 0x55d982cdfa00 con 0 > -7538> 2018-07-11 07:50:56.079103 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15287 conn(0x55d98123f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 251 0x55d9bcd72c00 osd_ping(ping e501 stamp 2018-07-11 07:50:57.309464) v4 > -7537> 2018-07-11 07:50:56.079120 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.4 10.8.128.113:0/15287 251 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:57.309464) v4 ==== 2004+0+0 (213742363 0 0) 0x55d9bcd72c00 con 0x55d98123f000 > -7536> 2018-07-11 07:50:56.079129 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:57.309464) v4 -- 0x55d982cdf400 con 0 > -7535> 2018-07-11 07:50:56.240968 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 848 0x55d9c369ad80 rep_scrubmap(5.2e e501 from shard 2) v1 > -7534> 2018-07-11 07:50:56.240987 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 848 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12474 (1269249638 0 2649779552) 0x55d9c369ad80 con 0x55d986bf0800 > -7533> 2018-07-11 07:50:56.258521 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 842 0x55d9c369afc0 rep_scrubmap(5.2e e501 from shard 4) v1 > -7532> 2018-07-11 07:50:56.258541 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 842 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12474 (3686815630 0 2649779552) 0x55d9c369afc0 con 0x55d986bec000 > -7531> 2018-07-11 07:50:56.288237 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.114:0/15919 conn(0x55d986df3800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.11 seq 242 0x55d9bdba4400 osd_ping(ping e501 stamp 2018-07-11 07:50:56.918274) v4 > -7530> 2018-07-11 07:50:56.288268 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.11 10.8.128.114:0/15919 242 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:56.918274) v4 ==== 2004+0+0 (626982204 0 0) 0x55d9bdba4400 con 0x55d986df3800 > -7529> 2018-07-11 07:50:56.288288 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:56.918274) v4 -- 0x55d9b9ac3400 con 0 > -7528> 2018-07-11 07:50:56.288291 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.114:0/15919 conn(0x55d986df5000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.11 seq 242 0x55d9869bfa00 osd_ping(ping e501 stamp 2018-07-11 07:50:56.918274) v4 > -7527> 2018-07-11 07:50:56.288313 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.11 10.8.128.114:0/15919 242 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:56.918274) v4 ==== 2004+0+0 (626982204 0 0) 0x55d9869bfa00 con 0x55d986df5000 > -7526> 2018-07-11 07:50:56.288324 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:56.918274) v4 -- 0x55d982cdf400 con 0 > -7525> 2018-07-11 07:50:56.317698 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12651 conn(0x55d986db3000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.2 seq 240 0x55d9bdba4800 osd_ping(ping e501 stamp 2018-07-11 07:50:56.364920) v4 > -7524> 2018-07-11 07:50:56.317704 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12651 conn(0x55d986db1800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.2 seq 240 0x55d980f5b400 osd_ping(ping e501 stamp 2018-07-11 07:50:56.364920) v4 > -7523> 2018-07-11 07:50:56.317718 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.2 10.8.128.106:0/12651 240 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:56.364920) v4 ==== 2004+0+0 (404428493 0 0) 0x55d9bdba4800 con 0x55d986db3000 > -7522> 2018-07-11 07:50:56.317727 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:56.364920) v4 -- 0x55d9b9ac3400 con 0 > -7521> 2018-07-11 07:50:56.317725 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.2 10.8.128.106:0/12651 240 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:56.364920) v4 ==== 2004+0+0 (404428493 0 0) 0x55d980f5b400 con 0x55d986db1800 > -7520> 2018-07-11 07:50:56.317755 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:56.364920) v4 -- 0x55d982cdfa00 con 0 > -7519> 2018-07-11 07:50:56.318501 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -7518> 2018-07-11 07:50:56.318537 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -7517> 2018-07-11 07:50:56.318591 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -7516> 2018-07-11 07:50:56.319034 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -7515> 2018-07-11 07:50:56.364724 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67634,epoch:501/498,start:5:752d957f:::benchmark_data_magna115_10867_object3657212:0,end:5:752dfa16:::benchmark_data_magna115_10867_object168334:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454500 con 0 > -7514> 2018-07-11 07:50:56.364775 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67634,epoch:501/498,start:5:752d957f:::benchmark_data_magna115_10867_object3657212:0,end:5:752dfa16:::benchmark_data_magna115_10867_object168334:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454c80 con 0 > -7513> 2018-07-11 07:50:56.462400 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15255 conn(0x55d986c82800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.0 seq 241 0x55d9bcd72c00 osd_ping(ping e501 stamp 2018-07-11 07:50:57.692798) v4 > -7512> 2018-07-11 07:50:56.462405 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15255 conn(0x55d986d20000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.0 seq 241 0x55d980f5b400 osd_ping(ping e501 stamp 2018-07-11 07:50:57.692798) v4 > -7511> 2018-07-11 07:50:56.462435 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.0 10.8.128.113:0/15255 241 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:57.692798) v4 ==== 2004+0+0 (2225574260 0 0) 0x55d980f5b400 con 0x55d986d20000 > -7510> 2018-07-11 07:50:56.462421 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.0 10.8.128.113:0/15255 241 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:57.692798) v4 ==== 2004+0+0 (2225574260 0 0) 0x55d9bcd72c00 con 0x55d986c82800 > -7509> 2018-07-11 07:50:56.462444 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:57.692798) v4 -- 0x55d982cdfa00 con 0 > -7508> 2018-07-11 07:50:56.462468 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:57.692798) v4 -- 0x55d982cdf400 con 0 > -7507> 2018-07-11 07:50:56.556952 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 843 0x55d9c369b200 rep_scrubmap(5.2e e501 from shard 4) v1 > -7506> 2018-07-11 07:50:56.556971 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 843 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12474 (3686815630 0 178297228) 0x55d9c369b200 con 0x55d986bec000 > -7505> 2018-07-11 07:50:56.558801 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6805/15885 conn(0x55d986c7f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.7 seq 2776 0x55d980f5b400 MOSDScrubReserve(5.4 REQUEST e501) v1 > -7504> 2018-07-11 07:50:56.558818 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.7 10.8.128.114:6805/15885 2776 ==== MOSDScrubReserve(5.4 REQUEST e501) v1 ==== 43+0+0 (1753221512 0 0) 0x55d980f5b400 con 0x55d986c7f800 > -7503> 2018-07-11 07:50:56.558943 7ffab33df700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.114:6805/15885 -- MOSDScrubReserve(5.4 REJECT e501) v1 -- 0x55d9c4c1a600 con 0 > -7502> 2018-07-11 07:50:56.559256 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6805/15885 conn(0x55d986c7f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.7 seq 2777 0x55d9c4c1a600 MOSDScrubReserve(5.4 RELEASE e501) v1 > -7501> 2018-07-11 07:50:56.559271 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.7 10.8.128.114:6805/15885 2777 ==== MOSDScrubReserve(5.4 RELEASE e501) v1 ==== 43+0+0 (2812146570 0 0) 0x55d9c4c1a600 con 0x55d986c7f800 > -7500> 2018-07-11 07:50:56.560345 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 849 0x55d9c369b440 rep_scrubmap(5.2e e501 from shard 2) v1 > -7499> 2018-07-11 07:50:56.560365 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 849 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12474 (1269249638 0 178297228) 0x55d9c369b440 con 0x55d986bf0800 > -7498> 2018-07-11 07:50:56.612375 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -7497> 2018-07-11 07:50:56.612398 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -7496> 2018-07-11 07:50:56.612451 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -7495> 2018-07-11 07:50:56.612894 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -7494> 2018-07-11 07:50:56.647135 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'66984,epoch:501/498,start:5:752dfa16:::benchmark_data_magna115_10867_object168334:0,end:5:752e63f6:::benchmark_data_magna115_10867_object645952:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454780 con 0 > -7493> 2018-07-11 07:50:56.647186 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'66984,epoch:501/498,start:5:752dfa16:::benchmark_data_magna115_10867_object168334:0,end:5:752e63f6:::benchmark_data_magna115_10867_object645952:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455180 con 0 > -7492> 2018-07-11 07:50:56.711141 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12650 conn(0x55d986d21800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 233 0x55d9bdba4400 osd_ping(ping e501 stamp 2018-07-11 07:50:56.758325) v4 > -7491> 2018-07-11 07:50:56.711144 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12650 conn(0x55d986d23000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 233 0x55d9869bfa00 osd_ping(ping e501 stamp 2018-07-11 07:50:56.758325) v4 > -7490> 2018-07-11 07:50:56.711168 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.3 10.8.128.106:0/12650 233 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:56.758325) v4 ==== 2004+0+0 (218055353 0 0) 0x55d9bdba4400 con 0x55d986d21800 > -7489> 2018-07-11 07:50:56.711177 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:56.758325) v4 -- 0x55d9b9ac3400 con 0 > -7488> 2018-07-11 07:50:56.711174 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.3 10.8.128.106:0/12650 233 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:56.758325) v4 ==== 2004+0+0 (218055353 0 0) 0x55d9869bfa00 con 0x55d986d23000 > -7487> 2018-07-11 07:50:56.711201 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:56.758325) v4 -- 0x55d982cdf400 con 0 > -7486> 2018-07-11 07:50:56.847966 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 850 0x55d9c369b680 rep_scrubmap(5.2e e501 from shard 2) v1 > -7485> 2018-07-11 07:50:56.847986 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 850 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12466 (1269249638 0 1580585169) 0x55d9c369b680 con 0x55d986bf0800 > -7484> 2018-07-11 07:50:56.885715 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 844 0x55d9c369b8c0 rep_scrubmap(5.2e e501 from shard 4) v1 > -7483> 2018-07-11 07:50:56.885735 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 844 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12466 (3686815630 0 1580585169) 0x55d9c369b8c0 con 0x55d986bec000 > -7482> 2018-07-11 07:50:56.992494 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -7481> 2018-07-11 07:50:56.992533 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -7480> 2018-07-11 07:50:56.992589 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -7479> 2018-07-11 07:50:56.993051 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -7478> 2018-07-11 07:50:57.015256 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:752e63f6:::benchmark_data_magna115_10867_object645952:0,end:5:752ec836:::benchmark_data_magna115_10867_object616700:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454f00 con 0 > -7477> 2018-07-11 07:50:57.015293 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:752e63f6:::benchmark_data_magna115_10867_object645952:0,end:5:752ec836:::benchmark_data_magna115_10867_object616700:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498280 con 0 > -7476> 2018-07-11 07:50:57.209501 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 851 0x55d9c369bb00 rep_scrubmap(5.2e e501 from shard 2) v1 > -7475> 2018-07-11 07:50:57.209523 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 851 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12476 (1269249638 0 819017175) 0x55d9c369bb00 con 0x55d986bf0800 > -7474> 2018-07-11 07:50:57.216979 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 845 0x55d9c369bd40 rep_scrubmap(5.2e e501 from shard 4) v1 > -7473> 2018-07-11 07:50:57.216999 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 845 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12476 (3686815630 0 819017175) 0x55d9c369bd40 con 0x55d986bec000 > -7472> 2018-07-11 07:50:57.347506 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -7471> 2018-07-11 07:50:57.347530 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -7470> 2018-07-11 07:50:57.347583 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -7469> 2018-07-11 07:50:57.348025 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -7468> 2018-07-11 07:50:57.348323 7ffab6be6700 1 -- 10.8.128.107:6800/13927 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55d985053b80 con 0 > -7467> 2018-07-11 07:50:57.348384 7ffab6be6700 1 -- 10.8.128.107:6800/13927 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55d986cff600 con 0 > -7466> 2018-07-11 07:50:57.398628 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'66961,epoch:501/498,start:5:752ec836:::benchmark_data_magna115_10867_object616700:0,end:5:752f2860:::benchmark_data_magna115_10867_object3455441:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455400 con 0 > -7465> 2018-07-11 07:50:57.398679 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'66961,epoch:501/498,start:5:752ec836:::benchmark_data_magna115_10867_object616700:0,end:5:752f2860:::benchmark_data_magna115_10867_object3455441:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455680 con 0 > -7464> 2018-07-11 07:50:57.496901 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.115:0/13323 conn(0x55d986d7e000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 241 0x55d9bdba4800 osd_ping(ping e501 stamp 2018-07-11 07:50:57.842370) v4 > -7463> 2018-07-11 07:50:57.496926 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.6 10.8.128.115:0/13323 241 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:57.842370) v4 ==== 2004+0+0 (2959735082 0 0) 0x55d9bdba4800 con 0x55d986d7e000 > -7462> 2018-07-11 07:50:57.496942 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:57.842370) v4 -- 0x55d9b9ac3400 con 0 > -7461> 2018-07-11 07:50:57.496999 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.115:0/13323 conn(0x55d986d24800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 241 0x55d986d32400 osd_ping(ping e501 stamp 2018-07-11 07:50:57.842370) v4 > -7460> 2018-07-11 07:50:57.497024 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.6 10.8.128.115:0/13323 241 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:57.842370) v4 ==== 2004+0+0 (2959735082 0 0) 0x55d986d32400 con 0x55d986d24800 > -7459> 2018-07-11 07:50:57.497035 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:57.842370) v4 -- 0x55d9b9ac3600 con 0 > -7458> 2018-07-11 07:50:57.616217 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 846 0x55d9b1728000 rep_scrubmap(5.2e e501 from shard 4) v1 > -7457> 2018-07-11 07:50:57.616235 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 846 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12474 (3686815630 0 1248955792) 0x55d9b1728000 con 0x55d986bec000 > -7456> 2018-07-11 07:50:57.623023 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 852 0x55d9b1728240 rep_scrubmap(5.2e e501 from shard 2) v1 > -7455> 2018-07-11 07:50:57.623042 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 852 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12474 (1269249638 0 1248955792) 0x55d9b1728240 con 0x55d986bf0800 > -7454> 2018-07-11 07:50:57.768224 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -7453> 2018-07-11 07:50:57.768264 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -7452> 2018-07-11 07:50:57.768324 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -7451> 2018-07-11 07:50:57.768793 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -7450> 2018-07-11 07:50:57.957766 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6801/15919 conn(0x55d986c3f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=29 cs=1 l=0). rx osd.11 seq 218 0x55d9bdba4400 MOSDScrubReserve(10.f REQUEST e501) v1 > -7449> 2018-07-11 07:50:57.957785 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.11 10.8.128.114:6801/15919 218 ==== MOSDScrubReserve(10.f REQUEST e501) v1 ==== 43+0+0 (2892966580 0 0) 0x55d9bdba4400 con 0x55d986c3f800 > -7448> 2018-07-11 07:50:57.957848 7ffab53e3700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.114:6801/15919 -- MOSDScrubReserve(10.f REJECT e501) v1 -- 0x55d9bf805800 con 0 > -7447> 2018-07-11 07:50:57.958163 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6801/15919 conn(0x55d986c3f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=29 cs=1 l=0). rx osd.11 seq 219 0x55d9bf805800 MOSDScrubReserve(10.f RELEASE e501) v1 > -7446> 2018-07-11 07:50:57.958190 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.11 10.8.128.114:6801/15919 219 ==== MOSDScrubReserve(10.f RELEASE e501) v1 ==== 43+0+0 (1668468918 0 0) 0x55d9bf805800 con 0x55d986c3f800 > -7445> 2018-07-11 07:50:57.997392 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.115:0/13323 conn(0x55d986d24800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 242 0x55d982cdfa00 osd_ping(ping e501 stamp 2018-07-11 07:50:58.342809) v4 > -7444> 2018-07-11 07:50:57.997414 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.6 10.8.128.115:0/13323 242 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:58.342809) v4 ==== 2004+0+0 (2216269038 0 0) 0x55d982cdfa00 con 0x55d986d24800 > -7443> 2018-07-11 07:50:57.997424 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:58.342809) v4 -- 0x55d9b9ac3600 con 0 > -7442> 2018-07-11 07:50:57.997438 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.115:0/13323 conn(0x55d986d7e000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 242 0x55d9bdba4800 osd_ping(ping e501 stamp 2018-07-11 07:50:58.342809) v4 > -7441> 2018-07-11 07:50:57.997460 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.6 10.8.128.115:0/13323 242 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:58.342809) v4 ==== 2004+0+0 (2216269038 0 0) 0x55d9bdba4800 con 0x55d986d7e000 > -7440> 2018-07-11 07:50:57.997471 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:58.342809) v4 -- 0x55d9b3db3c00 con 0 > -7439> 2018-07-11 07:50:58.162870 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15255 conn(0x55d986d20000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.0 seq 242 0x55d986d32400 osd_ping(ping e501 stamp 2018-07-11 07:50:59.393255) v4 > -7438> 2018-07-11 07:50:58.162869 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15255 conn(0x55d986c82800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.0 seq 242 0x55d9bcd72c00 osd_ping(ping e501 stamp 2018-07-11 07:50:59.393255) v4 > -7437> 2018-07-11 07:50:58.162897 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.0 10.8.128.113:0/15255 242 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:59.393255) v4 ==== 2004+0+0 (2543659006 0 0) 0x55d986d32400 con 0x55d986d20000 > -7436> 2018-07-11 07:50:58.162908 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:59.393255) v4 -- 0x55d9b9ac3600 con 0 > -7435> 2018-07-11 07:50:58.162902 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.0 10.8.128.113:0/15255 242 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:59.393255) v4 ==== 2004+0+0 (2543659006 0 0) 0x55d9bcd72c00 con 0x55d986c82800 > -7434> 2018-07-11 07:50:58.162934 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:59.393255) v4 -- 0x55d982cdf400 con 0 > -7433> 2018-07-11 07:50:58.284073 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.108:0/11967 conn(0x55d986db0000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.8 seq 222 0x55d9869bfa00 osd_ping(ping e501 stamp 2018-07-11 07:50:58.039557) v4 > -7432> 2018-07-11 07:50:58.284093 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.8 10.8.128.108:0/11967 222 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:58.039557) v4 ==== 2004+0+0 (97612368 0 0) 0x55d9869bfa00 con 0x55d986db0000 > -7431> 2018-07-11 07:50:58.284102 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:58.039557) v4 -- 0x55d982cdf400 con 0 > -7430> 2018-07-11 07:50:58.284157 7ffacbbd1700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.108:0/11967 conn(0x55d986d82800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.8 seq 222 0x55d9b9ac3400 osd_ping(ping e501 stamp 2018-07-11 07:50:58.039557) v4 > -7429> 2018-07-11 07:50:58.284178 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 <== osd.8 10.8.128.108:0/11967 222 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:58.039557) v4 ==== 2004+0+0 (97612368 0 0) 0x55d9b9ac3400 con 0x55d986d82800 > -7428> 2018-07-11 07:50:58.284190 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:58.039557) v4 -- 0x55d9b3db3c00 con 0 > -7427> 2018-07-11 07:50:58.410814 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'66580,epoch:501/498,start:5:752f2860:::benchmark_data_magna115_10867_object3455441:0,end:5:752f99d8:::benchmark_data_magna115_10867_object2682975:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498000 con 0 > -7426> 2018-07-11 07:50:58.410865 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'66580,epoch:501/498,start:5:752f2860:::benchmark_data_magna115_10867_object3455441:0,end:5:752f99d8:::benchmark_data_magna115_10867_object2682975:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455b80 con 0 > -7425> 2018-07-11 07:50:58.606354 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 853 0x55d9b1728480 rep_scrubmap(5.2e e501 from shard 2) v1 > -7424> 2018-07-11 07:50:58.606374 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 853 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12480 (1269249638 0 4151724892) 0x55d9b1728480 con 0x55d986bf0800 > -7423> 2018-07-11 07:50:58.631752 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 847 0x55d9b17286c0 rep_scrubmap(5.2e e501 from shard 4) v1 > -7422> 2018-07-11 07:50:58.631771 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 847 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12480 (3686815630 0 4151724892) 0x55d9b17286c0 con 0x55d986bec000 > -7421> 2018-07-11 07:50:58.714238 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.114:0/15885 conn(0x55d986d7f800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.7 seq 246 0x55d9bcd72c00 osd_ping(ping e501 stamp 2018-07-11 07:50:59.344214) v4 > -7420> 2018-07-11 07:50:58.714280 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.7 10.8.128.114:0/15885 246 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:59.344214) v4 ==== 2004+0+0 (427440837 0 0) 0x55d9bcd72c00 con 0x55d986d7f800 > -7419> 2018-07-11 07:50:58.714284 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.114:0/15885 conn(0x55d986d81000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.7 seq 246 0x55d986d32400 osd_ping(ping e501 stamp 2018-07-11 07:50:59.344214) v4 > -7418> 2018-07-11 07:50:58.714297 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:59.344214) v4 -- 0x55d982cdf400 con 0 > -7417> 2018-07-11 07:50:58.714305 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.7 10.8.128.114:0/15885 246 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:59.344214) v4 ==== 2004+0+0 (427440837 0 0) 0x55d986d32400 con 0x55d986d81000 > -7416> 2018-07-11 07:50:58.714326 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:59.344214) v4 -- 0x55d9b9ac3600 con 0 > -7415> 2018-07-11 07:50:58.895624 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 31] Generated table #36795: 2638893 keys, 74548725 bytes > -7414> 2018-07-11 07:50:58.895673 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295458895641, "cf_name": "default", "job": 31, "event": "table_file_creation", "file_number": 36795, "file_size": 74548725, "table_properties": {"data_size": 67112170, "index_size": 690748, "filter_size": 6744816, "raw_key_size": 76234681, "raw_average_key_size": 28, "raw_value_size": 31356908, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638893, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1397", "kMergeOperands": "0"}} > -7413> 2018-07-11 07:50:58.979483 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15287 conn(0x55d981240800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 252 0x55d982cdfa00 osd_ping(ping e501 stamp 2018-07-11 07:51:00.209878) v4 > -7412> 2018-07-11 07:50:58.979487 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15287 conn(0x55d98123f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 252 0x55d9869bfa00 osd_ping(ping e501 stamp 2018-07-11 07:51:00.209878) v4 > -7411> 2018-07-11 07:50:58.979514 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.4 10.8.128.113:0/15287 252 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:00.209878) v4 ==== 2004+0+0 (761706294 0 0) 0x55d982cdfa00 con 0x55d981240800 > -7410> 2018-07-11 07:50:58.979518 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.4 10.8.128.113:0/15287 252 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:00.209878) v4 ==== 2004+0+0 (761706294 0 0) 0x55d9869bfa00 con 0x55d98123f000 > -7409> 2018-07-11 07:50:58.979530 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:00.209878) v4 -- 0x55d9b9ac3600 con 0 > -7408> 2018-07-11 07:50:58.979559 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:00.209878) v4 -- 0x55d982cdf400 con 0 > -7407> 2018-07-11 07:50:59.051784 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -7406> 2018-07-11 07:50:59.051811 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -7405> 2018-07-11 07:50:59.051867 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -7404> 2018-07-11 07:50:59.052344 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -7403> 2018-07-11 07:50:59.107031 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:752f99d8:::benchmark_data_magna115_10867_object2682975:0,end:5:75300f6c:::benchmark_data_magna115_10867_object2824257:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455900 con 0 > -7402> 2018-07-11 07:50:59.107084 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:752f99d8:::benchmark_data_magna115_10867_object2682975:0,end:5:75300f6c:::benchmark_data_magna115_10867_object2824257:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498780 con 0 > -7401> 2018-07-11 07:50:59.326169 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 854 0x55d9b1728900 rep_scrubmap(5.2e e501 from shard 2) v1 > -7400> 2018-07-11 07:50:59.326192 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 854 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12476 (1269249638 0 553804566) 0x55d9b1728900 con 0x55d986bf0800 > -7399> 2018-07-11 07:50:59.337264 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 848 0x55d9b1728b40 rep_scrubmap(5.2e e501 from shard 4) v1 > -7398> 2018-07-11 07:50:59.337285 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 848 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12476 (3686815630 0 553804566) 0x55d9b1728b40 con 0x55d986bec000 > -7397> 2018-07-11 07:50:59.396843 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -7396> 2018-07-11 07:50:59.396867 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -7395> 2018-07-11 07:50:59.396928 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -7394> 2018-07-11 07:50:59.397410 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -7393> 2018-07-11 07:50:59.443897 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15263 conn(0x55d985d45800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.5 seq 224 0x55d9bcd72c00 osd_ping(ping e501 stamp 2018-07-11 07:51:00.674306) v4 > -7392> 2018-07-11 07:50:59.443897 7ffacbbd1700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15263 conn(0x55d985d44000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.5 seq 224 0x55d9bdba4800 osd_ping(ping e501 stamp 2018-07-11 07:51:00.674306) v4 > -7391> 2018-07-11 07:50:59.443926 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.5 10.8.128.113:0/15263 224 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:00.674306) v4 ==== 2004+0+0 (909433371 0 0) 0x55d9bcd72c00 con 0x55d985d45800 > -7390> 2018-07-11 07:50:59.443930 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 <== osd.5 10.8.128.113:0/15263 224 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:00.674306) v4 ==== 2004+0+0 (909433371 0 0) 0x55d9bdba4800 con 0x55d985d44000 > -7389> 2018-07-11 07:50:59.443945 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:00.674306) v4 -- 0x55d982cdf400 con 0 > -7388> 2018-07-11 07:50:59.443975 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:00.674306) v4 -- 0x55d9b3db3c00 con 0 > -7387> 2018-07-11 07:50:59.448943 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67093,epoch:501/498,start:5:75300f6c:::benchmark_data_magna115_10867_object2824257:0,end:5:75307ac7:::benchmark_data_magna115_10867_object1883107:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b14f0500 con 0 > -7386> 2018-07-11 07:50:59.448994 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67093,epoch:501/498,start:5:75300f6c:::benchmark_data_magna115_10867_object2824257:0,end:5:75307ac7:::benchmark_data_magna115_10867_object1883107:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98696e000 con 0 > -7385> 2018-07-11 07:50:59.479766 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15287 conn(0x55d98123f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 253 0x55d9869bfa00 osd_ping(ping e501 stamp 2018-07-11 07:51:00.710237) v4 > -7384> 2018-07-11 07:50:59.479797 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.4 10.8.128.113:0/15287 253 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:00.710237) v4 ==== 2004+0+0 (962920656 0 0) 0x55d9869bfa00 con 0x55d98123f000 > -7383> 2018-07-11 07:50:59.479802 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15287 conn(0x55d981240800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 253 0x55d986d32400 osd_ping(ping e501 stamp 2018-07-11 07:51:00.710237) v4 > -7382> 2018-07-11 07:50:59.479814 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:00.710237) v4 -- 0x55d982cdf400 con 0 > -7381> 2018-07-11 07:50:59.479820 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.4 10.8.128.113:0/15287 253 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:00.710237) v4 ==== 2004+0+0 (962920656 0 0) 0x55d986d32400 con 0x55d981240800 > -7380> 2018-07-11 07:50:59.479847 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:00.710237) v4 -- 0x55d9b9ac3600 con 0 > -7379> 2018-07-11 07:50:59.541232 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.107:0/12617 conn(0x55d98123d800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 227 0x55d9b9ac3400 osd_ping(ping e501 stamp 2018-07-11 07:50:59.540895) v4 > -7378> 2018-07-11 07:50:59.541262 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.9 10.8.128.107:0/12617 227 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:59.540895) v4 ==== 2004+0+0 (2507244648 0 0) 0x55d9b9ac3400 con 0x55d98123d800 > -7377> 2018-07-11 07:50:59.541257 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.107:0/12617 conn(0x55d98123c000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 227 0x55d982cdfa00 osd_ping(ping e501 stamp 2018-07-11 07:50:59.540895) v4 > -7376> 2018-07-11 07:50:59.541272 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:59.540895) v4 -- 0x55d9b3db3c00 con 0 > -7375> 2018-07-11 07:50:59.541285 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.9 10.8.128.107:0/12617 227 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:59.540895) v4 ==== 2004+0+0 (2507244648 0 0) 0x55d982cdfa00 con 0x55d98123c000 > -7374> 2018-07-11 07:50:59.541300 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:59.540895) v4 -- 0x55d9b9ac3600 con 0 > -7373> 2018-07-11 07:50:59.611481 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12650 conn(0x55d986d23000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 234 0x55d9bcd72c00 osd_ping(ping e501 stamp 2018-07-11 07:50:59.658752) v4 > -7372> 2018-07-11 07:50:59.611505 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.3 10.8.128.106:0/12650 234 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:59.658752) v4 ==== 2004+0+0 (1676605627 0 0) 0x55d9bcd72c00 con 0x55d986d23000 > -7371> 2018-07-11 07:50:59.611515 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:59.658752) v4 -- 0x55d982cdf400 con 0 > -7370> 2018-07-11 07:50:59.611611 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12650 conn(0x55d986d21800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 234 0x55d9bdba4800 osd_ping(ping e501 stamp 2018-07-11 07:50:59.658752) v4 > -7369> 2018-07-11 07:50:59.611635 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.3 10.8.128.106:0/12650 234 ==== osd_ping(ping e501 stamp 2018-07-11 07:50:59.658752) v4 ==== 2004+0+0 (1676605627 0 0) 0x55d9bdba4800 con 0x55d986d21800 > -7368> 2018-07-11 07:50:59.611646 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:50:59.658752) v4 -- 0x55d9b3db3c00 con 0 > -7367> 2018-07-11 07:50:59.644926 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 849 0x55d9b1728d80 rep_scrubmap(5.2e e501 from shard 4) v1 > -7366> 2018-07-11 07:50:59.644943 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 849 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12468 (3686815630 0 4291954797) 0x55d9b1728d80 con 0x55d986bec000 > -7365> 2018-07-11 07:50:59.652814 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 855 0x55d9b1728fc0 rep_scrubmap(5.2e e501 from shard 2) v1 > -7364> 2018-07-11 07:50:59.652834 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 855 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12468 (1269249638 0 4291954797) 0x55d9b1728fc0 con 0x55d986bf0800 > -7363> 2018-07-11 07:50:59.669558 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -7362> 2018-07-11 07:50:59.669582 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -7361> 2018-07-11 07:50:59.669642 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -7360> 2018-07-11 07:50:59.670103 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -7359> 2018-07-11 07:50:59.708208 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75307ac7:::benchmark_data_magna115_10867_object1883107:0,end:5:7530c250:::benchmark_data_magna115_10867_object3765774:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98696f180 con 0 > -7358> 2018-07-11 07:50:59.708255 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75307ac7:::benchmark_data_magna115_10867_object1883107:0,end:5:7530c250:::benchmark_data_magna115_10867_object3765774:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98696f900 con 0 > -7357> 2018-07-11 07:50:59.833860 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 2988 0x55d9b9ac3400 MOSDScrubReserve(11.3s5 REQUEST e501) v1 > -7356> 2018-07-11 07:50:59.833878 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 2988 ==== MOSDScrubReserve(11.3s5 REQUEST e501) v1 ==== 43+0+0 (3260170403 0 0) 0x55d9b9ac3400 con 0x55d986c7e000 > -7355> 2018-07-11 07:50:59.833997 7ffab3be0700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e501) v1 -- 0x55d985054a00 con 0 > -7354> 2018-07-11 07:50:59.834201 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 2989 0x55d9b3db3c00 MOSDScrubReserve(11.3s5 RELEASE e501) v1 > -7353> 2018-07-11 07:50:59.834215 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 2989 ==== MOSDScrubReserve(11.3s5 RELEASE e501) v1 ==== 43+0+0 (223328929 0 0) 0x55d9b3db3c00 con 0x55d986c7e000 > -7352> 2018-07-11 07:50:59.921623 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 850 0x55d9b1729200 rep_scrubmap(5.2e e501 from shard 4) v1 > -7351> 2018-07-11 07:50:59.921644 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 850 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12474 (3686815630 0 2047443405) 0x55d9b1729200 con 0x55d986bec000 > -7350> 2018-07-11 07:50:59.923578 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 856 0x55d9b1729440 rep_scrubmap(5.2e e501 from shard 2) v1 > -7349> 2018-07-11 07:50:59.923598 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 856 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12474 (1269249638 0 2047443405) 0x55d9b1729440 con 0x55d986bf0800 > -7348> 2018-07-11 07:50:59.959390 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -7347> 2018-07-11 07:50:59.959416 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -7346> 2018-07-11 07:50:59.959464 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -7345> 2018-07-11 07:50:59.959927 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -7344> 2018-07-11 07:50:59.982534 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7530c250:::benchmark_data_magna115_10867_object3765774:0,end:5:75311c79:::benchmark_data_magna115_11707_object235954:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98696ef00 con 0 > -7343> 2018-07-11 07:50:59.982571 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7530c250:::benchmark_data_magna115_10867_object3765774:0,end:5:75311c79:::benchmark_data_magna115_11707_object235954:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498a00 con 0 > -7342> 2018-07-11 07:51:00.180339 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 851 0x55d9b1729680 rep_scrubmap(5.2e e501 from shard 4) v1 > -7341> 2018-07-11 07:51:00.180359 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 851 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12472 (3686815630 0 2852160090) 0x55d9b1729680 con 0x55d986bec000 > -7340> 2018-07-11 07:51:00.193438 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 857 0x55d9b17298c0 rep_scrubmap(5.2e e501 from shard 2) v1 > -7339> 2018-07-11 07:51:00.193458 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 857 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12472 (1269249638 0 2852160090) 0x55d9b17298c0 con 0x55d986bf0800 > -7338> 2018-07-11 07:51:00.297983 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.115:0/13323 conn(0x55d986d7e000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 243 0x55d985054a00 osd_ping(ping e501 stamp 2018-07-11 07:51:00.643444) v4 > -7337> 2018-07-11 07:51:00.298006 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.6 10.8.128.115:0/13323 243 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:00.643444) v4 ==== 2004+0+0 (3633757456 0 0) 0x55d985054a00 con 0x55d986d7e000 > -7336> 2018-07-11 07:51:00.298022 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:00.643444) v4 -- 0x55d9bdba4800 con 0 > -7335> 2018-07-11 07:51:00.298050 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.115:0/13323 conn(0x55d986d24800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 243 0x55d982cdfa00 osd_ping(ping e501 stamp 2018-07-11 07:51:00.643444) v4 > -7334> 2018-07-11 07:51:00.298071 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.6 10.8.128.115:0/13323 243 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:00.643444) v4 ==== 2004+0+0 (3633757456 0 0) 0x55d982cdfa00 con 0x55d986d24800 > -7333> 2018-07-11 07:51:00.298083 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:00.643444) v4 -- 0x55d9b9ac3600 con 0 > -7332> 2018-07-11 07:51:00.309931 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -7331> 2018-07-11 07:51:00.309971 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -7330> 2018-07-11 07:51:00.310015 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -7329> 2018-07-11 07:51:00.310495 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -7328> 2018-07-11 07:51:00.357440 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67537,epoch:501/498,start:5:75311c79:::benchmark_data_magna115_11707_object235954:0,end:5:75316497:::benchmark_data_magna115_10867_object3815686:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b14f0a00 con 0 > -7327> 2018-07-11 07:51:00.357479 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67537,epoch:501/498,start:5:75311c79:::benchmark_data_magna115_11707_object235954:0,end:5:75316497:::benchmark_data_magna115_10867_object3815686:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98696f680 con 0 > -7326> 2018-07-11 07:51:00.388685 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.114:0/15919 conn(0x55d986df3800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.11 seq 243 0x55d9b3db2600 osd_ping(ping e501 stamp 2018-07-11 07:51:01.018722) v4 > -7325> 2018-07-11 07:51:00.388705 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.114:0/15919 conn(0x55d986df5000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.11 seq 243 0x55d9869bfa00 osd_ping(ping e501 stamp 2018-07-11 07:51:01.018722) v4 > -7324> 2018-07-11 07:51:00.388715 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.11 10.8.128.114:0/15919 243 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:01.018722) v4 ==== 2004+0+0 (3714607263 0 0) 0x55d9b3db2600 con 0x55d986df3800 > -7323> 2018-07-11 07:51:00.388729 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.11 10.8.128.114:0/15919 243 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:01.018722) v4 ==== 2004+0+0 (3714607263 0 0) 0x55d9869bfa00 con 0x55d986df5000 > -7322> 2018-07-11 07:51:00.388734 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:01.018722) v4 -- 0x55d9bdba4800 con 0 > -7321> 2018-07-11 07:51:00.388766 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:01.018722) v4 -- 0x55d982cdf400 con 0 > -7320> 2018-07-11 07:51:00.418096 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12651 conn(0x55d986db3000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.2 seq 241 0x55d985054a00 osd_ping(ping e501 stamp 2018-07-11 07:51:00.465349) v4 > -7319> 2018-07-11 07:51:00.418129 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.2 10.8.128.106:0/12651 241 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:00.465349) v4 ==== 2004+0+0 (3888286915 0 0) 0x55d985054a00 con 0x55d986db3000 > -7318> 2018-07-11 07:51:00.418139 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:00.465349) v4 -- 0x55d9bdba4800 con 0 > -7317> 2018-07-11 07:51:00.418142 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12651 conn(0x55d986db1800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.2 seq 241 0x55d982cdfa00 osd_ping(ping e501 stamp 2018-07-11 07:51:00.465349) v4 > -7316> 2018-07-11 07:51:00.418163 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.2 10.8.128.106:0/12651 241 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:00.465349) v4 ==== 2004+0+0 (3888286915 0 0) 0x55d982cdfa00 con 0x55d986db1800 > -7315> 2018-07-11 07:51:00.418174 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:00.465349) v4 -- 0x55d9b9ac3600 con 0 > -7314> 2018-07-11 07:51:00.559430 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6805/15885 conn(0x55d986c7f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.7 seq 2778 0x55d986d32400 MOSDScrubReserve(5.4 REQUEST e501) v1 > -7313> 2018-07-11 07:51:00.559449 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.7 10.8.128.114:6805/15885 2778 ==== MOSDScrubReserve(5.4 REQUEST e501) v1 ==== 43+0+0 (1753221512 0 0) 0x55d986d32400 con 0x55d986c7f800 > -7312> 2018-07-11 07:51:00.559495 7ffab33df700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.114:6805/15885 -- MOSDScrubReserve(5.4 REJECT e501) v1 -- 0x55d9c4c1a600 con 0 > -7311> 2018-07-11 07:51:00.559745 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6805/15885 conn(0x55d986c7f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.7 seq 2779 0x55d9b9ac3600 MOSDScrubReserve(5.4 RELEASE e501) v1 > -7310> 2018-07-11 07:51:00.559777 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.7 10.8.128.114:6805/15885 2779 ==== MOSDScrubReserve(5.4 RELEASE e501) v1 ==== 43+0+0 (2812146570 0 0) 0x55d9b9ac3600 con 0x55d986c7f800 > -7309> 2018-07-11 07:51:00.587491 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 852 0x55d9b1729b00 rep_scrubmap(5.2e e501 from shard 4) v1 > -7308> 2018-07-11 07:51:00.587510 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 852 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12462 (3686815630 0 1333529673) 0x55d9b1729b00 con 0x55d986bec000 > -7307> 2018-07-11 07:51:00.596370 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 858 0x55d9b1729d40 rep_scrubmap(5.2e e501 from shard 2) v1 > -7306> 2018-07-11 07:51:00.596391 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 858 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12462 (1269249638 0 1333529673) 0x55d9b1729d40 con 0x55d986bf0800 > -7305> 2018-07-11 07:51:00.659114 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -7304> 2018-07-11 07:51:00.659138 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -7303> 2018-07-11 07:51:00.659196 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -7302> 2018-07-11 07:51:00.659699 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -7301> 2018-07-11 07:51:00.660222 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12677 conn(0x55d986db4800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.1 seq 245 0x55d9bcd72c00 osd_ping(ping e501 stamp 2018-07-11 07:51:00.707504) v4 > -7300> 2018-07-11 07:51:00.660255 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.1 10.8.128.106:0/12677 245 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:00.707504) v4 ==== 2004+0+0 (3798645861 0 0) 0x55d9bcd72c00 con 0x55d986db4800 > -7299> 2018-07-11 07:51:00.660266 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:00.707504) v4 -- 0x55d982cdf400 con 0 > -7298> 2018-07-11 07:51:00.660222 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12677 conn(0x55d986df2000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.1 seq 245 0x55d982cdfa00 osd_ping(ping e501 stamp 2018-07-11 07:51:00.707504) v4 > -7297> 2018-07-11 07:51:00.660358 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.1 10.8.128.106:0/12677 245 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:00.707504) v4 ==== 2004+0+0 (3798645861 0 0) 0x55d982cdfa00 con 0x55d986df2000 > -7296> 2018-07-11 07:51:00.660375 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:00.707504) v4 -- 0x55d97f5adc00 con 0 > -7295> 2018-07-11 07:51:00.707801 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75316497:::benchmark_data_magna115_10867_object3815686:0,end:5:7531ad5e:::benchmark_data_magna115_10867_object2666779:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98696ec80 con 0 > -7294> 2018-07-11 07:51:00.707851 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75316497:::benchmark_data_magna115_10867_object3815686:0,end:5:7531ad5e:::benchmark_data_magna115_10867_object2666779:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498f00 con 0 > -7293> 2018-07-11 07:51:00.892045 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 859 0x55d983e52000 rep_scrubmap(5.2e e501 from shard 2) v1 > -7292> 2018-07-11 07:51:00.892065 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 859 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12480 (1269249638 0 3813165098) 0x55d983e52000 con 0x55d986bf0800 > -7291> 2018-07-11 07:51:00.906624 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 853 0x55d983e52240 rep_scrubmap(5.2e e501 from shard 4) v1 > -7290> 2018-07-11 07:51:00.906645 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 853 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12480 (3686815630 0 3813165098) 0x55d983e52240 con 0x55d986bec000 > -7289> 2018-07-11 07:51:01.107787 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -7288> 2018-07-11 07:51:01.107817 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -7287> 2018-07-11 07:51:01.107876 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -7286> 2018-07-11 07:51:01.108369 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -7285> 2018-07-11 07:51:01.124516 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67898,epoch:501/498,start:5:7531ad5e:::benchmark_data_magna115_10867_object2666779:0,end:5:753205c2:::benchmark_data_magna115_10867_object2833326:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1499180 con 0 > -7284> 2018-07-11 07:51:01.124568 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67898,epoch:501/498,start:5:7531ad5e:::benchmark_data_magna115_10867_object2666779:0,end:5:753205c2:::benchmark_data_magna115_10867_object2833326:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1499400 con 0 > -7283> 2018-07-11 07:51:01.241692 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.107:0/12617 conn(0x55d98123c000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 228 0x55d9c4c1a600 osd_ping(ping e501 stamp 2018-07-11 07:51:01.241354) v4 > -7282> 2018-07-11 07:51:01.241699 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.107:0/12617 conn(0x55d98123d800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 228 0x55d9b3db2600 osd_ping(ping e501 stamp 2018-07-11 07:51:01.241354) v4 > -7281> 2018-07-11 07:51:01.241716 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.9 10.8.128.107:0/12617 228 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:01.241354) v4 ==== 2004+0+0 (372422524 0 0) 0x55d9b3db2600 con 0x55d98123d800 > -7280> 2018-07-11 07:51:01.241715 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.9 10.8.128.107:0/12617 228 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:01.241354) v4 ==== 2004+0+0 (372422524 0 0) 0x55d9c4c1a600 con 0x55d98123c000 > -7279> 2018-07-11 07:51:01.241732 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:01.241354) v4 -- 0x55d9bdba4800 con 0 > -7278> 2018-07-11 07:51:01.241759 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:01.241354) v4 -- 0x55d97f5adc00 con 0 > -7277> 2018-07-11 07:51:01.350753 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 860 0x55d983e52480 rep_scrubmap(5.2e e501 from shard 2) v1 > -7276> 2018-07-11 07:51:01.350774 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 860 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12462 (1269249638 0 930973333) 0x55d983e52480 con 0x55d986bf0800 > -7275> 2018-07-11 07:51:01.376837 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 854 0x55d983e526c0 rep_scrubmap(5.2e e501 from shard 4) v1 > -7274> 2018-07-11 07:51:01.376857 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 854 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12462 (3686815630 0 930973333) 0x55d983e526c0 con 0x55d986bec000 > -7273> 2018-07-11 07:51:01.440564 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -7272> 2018-07-11 07:51:01.440591 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -7271> 2018-07-11 07:51:01.440648 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -7270> 2018-07-11 07:51:01.441114 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -7269> 2018-07-11 07:51:01.442039 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:753205c2:::benchmark_data_magna115_10867_object2833326:0,end:5:7532728b:::benchmark_data_magna115_10867_object557495:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498c80 con 0 > -7268> 2018-07-11 07:51:01.442076 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:753205c2:::benchmark_data_magna115_10867_object2833326:0,end:5:7532728b:::benchmark_data_magna115_10867_object557495:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1499b80 con 0 > -7267> 2018-07-11 07:51:01.559569 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6805/15885 conn(0x55d986c7f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.7 seq 2780 0x55d982cdfa00 MOSDScrubReserve(5.4 REQUEST e501) v1 > -7266> 2018-07-11 07:51:01.559586 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.7 10.8.128.114:6805/15885 2780 ==== MOSDScrubReserve(5.4 REQUEST e501) v1 ==== 43+0+0 (1753221512 0 0) 0x55d982cdfa00 con 0x55d986c7f800 > -7265> 2018-07-11 07:51:01.559705 7ffab33df700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.114:6805/15885 -- MOSDScrubReserve(5.4 REJECT e501) v1 -- 0x55d9b9ac3600 con 0 > -7264> 2018-07-11 07:51:01.560041 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6805/15885 conn(0x55d986c7f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.7 seq 2781 0x55d9b9ac3600 MOSDScrubReserve(5.4 RELEASE e501) v1 > -7263> 2018-07-11 07:51:01.560068 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.7 10.8.128.114:6805/15885 2781 ==== MOSDScrubReserve(5.4 RELEASE e501) v1 ==== 43+0+0 (2812146570 0 0) 0x55d9b9ac3600 con 0x55d986c7f800 > -7262> 2018-07-11 07:51:01.613072 7ffab1bdc700 5 osd.10 501 heartbeat: osd_stat(788 GB used, 143 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist []) > -7261> 2018-07-11 07:51:01.613085 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6802/15255 -- osd_ping(ping e501 stamp 2018-07-11 07:51:01.613083) v4 -- 0x55d9b67d8400 con 0 > -7260> 2018-07-11 07:51:01.613121 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6803/15255 -- osd_ping(ping e501 stamp 2018-07-11 07:51:01.613083) v4 -- 0x55d9bdba4200 con 0 > -7259> 2018-07-11 07:51:01.613137 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6806/12677 -- osd_ping(ping e501 stamp 2018-07-11 07:51:01.613083) v4 -- 0x55d9c34cc000 con 0 > -7258> 2018-07-11 07:51:01.613151 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6807/12677 -- osd_ping(ping e501 stamp 2018-07-11 07:51:01.613083) v4 -- 0x55d9869bec00 con 0 > -7257> 2018-07-11 07:51:01.613164 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6808/12651 -- osd_ping(ping e501 stamp 2018-07-11 07:51:01.613083) v4 -- 0x55d9c74e1800 con 0 > -7256> 2018-07-11 07:51:01.613176 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6810/12651 -- osd_ping(ping e501 stamp 2018-07-11 07:51:01.613083) v4 -- 0x55d9bf805000 con 0 > -7255> 2018-07-11 07:51:01.613189 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6809/12650 -- osd_ping(ping e501 stamp 2018-07-11 07:51:01.613083) v4 -- 0x55d9bfbde200 con 0 > -7254> 2018-07-11 07:51:01.613203 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6811/12650 -- osd_ping(ping e501 stamp 2018-07-11 07:51:01.613083) v4 -- 0x55d986d4fe00 con 0 > -7253> 2018-07-11 07:51:01.613224 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6810/15287 -- osd_ping(ping e501 stamp 2018-07-11 07:51:01.613083) v4 -- 0x55d9c9123a00 con 0 > -7252> 2018-07-11 07:51:01.613236 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6811/15287 -- osd_ping(ping e501 stamp 2018-07-11 07:51:01.613083) v4 -- 0x55d9b4388200 con 0 > -7251> 2018-07-11 07:51:01.613263 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6806/15263 -- osd_ping(ping e501 stamp 2018-07-11 07:51:01.613083) v4 -- 0x55d9c3383400 con 0 > -7250> 2018-07-11 07:51:01.613279 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6807/15263 -- osd_ping(ping e501 stamp 2018-07-11 07:51:01.613083) v4 -- 0x55d980f5a400 con 0 > -7249> 2018-07-11 07:51:01.613293 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.115:6802/13323 -- osd_ping(ping e501 stamp 2018-07-11 07:51:01.613083) v4 -- 0x55d982d30400 con 0 > -7248> 2018-07-11 07:51:01.613306 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.115:6803/13323 -- osd_ping(ping e501 stamp 2018-07-11 07:51:01.613083) v4 -- 0x55d9c9123000 con 0 > -7247> 2018-07-11 07:51:01.613324 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6806/15885 -- osd_ping(ping e501 stamp 2018-07-11 07:51:01.613083) v4 -- 0x55d986d41600 con 0 > -7246> 2018-07-11 07:51:01.613337 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6807/15885 -- osd_ping(ping e501 stamp 2018-07-11 07:51:01.613083) v4 -- 0x55d986d41000 con 0 > -7245> 2018-07-11 07:51:01.613349 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.108:6802/11967 -- osd_ping(ping e501 stamp 2018-07-11 07:51:01.613083) v4 -- 0x55d9b67d8c00 con 0 > -7244> 2018-07-11 07:51:01.613359 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.108:6803/11967 -- osd_ping(ping e501 stamp 2018-07-11 07:51:01.613083) v4 -- 0x55d986d32200 con 0 > -7243> 2018-07-11 07:51:01.613372 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.107:6806/12617 -- osd_ping(ping e501 stamp 2018-07-11 07:51:01.613083) v4 -- 0x55d9c3382800 con 0 > -7242> 2018-07-11 07:51:01.613386 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.107:6807/12617 -- osd_ping(ping e501 stamp 2018-07-11 07:51:01.613083) v4 -- 0x55d9869be000 con 0 > -7241> 2018-07-11 07:51:01.613399 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6802/15919 -- osd_ping(ping e501 stamp 2018-07-11 07:51:01.613083) v4 -- 0x55d9829e5a00 con 0 > -7240> 2018-07-11 07:51:01.613433 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6803/15919 -- osd_ping(ping e501 stamp 2018-07-11 07:51:01.613083) v4 -- 0x55d9869bf600 con 0 > -7239> 2018-07-11 07:51:01.613521 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6803/15255 conn(0x55d986bc9800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.0 seq 232 0x55d9bcd72c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:01.613083) v4 > -7238> 2018-07-11 07:51:01.613551 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6802/15255 conn(0x55d986bc8000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.0 seq 232 0x55d9c4c1a600 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:01.613083) v4 > -7237> 2018-07-11 07:51:01.613563 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.0 10.8.128.113:6803/15255 232 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:01.613083) v4 ==== 2004+0+0 (3511827592 0 0) 0x55d9bcd72c00 con 0x55d986bc9800 > -7236> 2018-07-11 07:51:01.613565 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.0 10.8.128.113:6802/15255 232 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:01.613083) v4 ==== 2004+0+0 (3511827592 0 0) 0x55d9c4c1a600 con 0x55d986bc8000 > -7235> 2018-07-11 07:51:01.613599 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.107:6806/12617 conn(0x55d986bd2000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.9 seq 232 0x55d9bcd72c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:01.613083) v4 > -7234> 2018-07-11 07:51:01.613612 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.9 10.8.128.107:6806/12617 232 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:01.613083) v4 ==== 2004+0+0 (3511827592 0 0) 0x55d9bcd72c00 con 0x55d986bd2000 > -7233> 2018-07-11 07:51:01.613625 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6810/15287 conn(0x55d986baf000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 232 0x55d9c4c1a600 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:01.613083) v4 > -7232> 2018-07-11 07:51:01.613615 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.107:6807/12617 conn(0x55d986bd3800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.9 seq 232 0x55d9869be600 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:01.613083) v4 > -7231> 2018-07-11 07:51:01.613646 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.9 10.8.128.107:6807/12617 232 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:01.613083) v4 ==== 2004+0+0 (3511827592 0 0) 0x55d9869be600 con 0x55d986bd3800 > -7230> 2018-07-11 07:51:01.613638 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.4 10.8.128.113:6810/15287 232 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:01.613083) v4 ==== 2004+0+0 (3511827592 0 0) 0x55d9c4c1a600 con 0x55d986baf000 > -7229> 2018-07-11 07:51:01.613678 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6811/15287 conn(0x55d986bb0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 232 0x55d9bcd72c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:01.613083) v4 > -7228> 2018-07-11 07:51:01.613698 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.4 10.8.128.113:6811/15287 232 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:01.613083) v4 ==== 2004+0+0 (3511827592 0 0) 0x55d9bcd72c00 con 0x55d986bb0800 > -7227> 2018-07-11 07:51:01.613703 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6808/12651 conn(0x55d986b9a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.2 seq 232 0x55d9c4c1a600 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:01.613083) v4 > -7226> 2018-07-11 07:51:01.613715 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.2 10.8.128.106:6808/12651 232 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:01.613083) v4 ==== 2004+0+0 (3511827592 0 0) 0x55d9c4c1a600 con 0x55d986b9a000 > -7225> 2018-07-11 07:51:01.613725 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6806/15263 conn(0x55d986b9d000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.5 seq 232 0x55d9869be600 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:01.613083) v4 > -7224> 2018-07-11 07:51:01.613742 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6810/12651 conn(0x55d986b9b800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.2 seq 232 0x55d9bcd72c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:01.613083) v4 > -7223> 2018-07-11 07:51:01.613739 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6807/15263 conn(0x55d986b9e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.5 seq 232 0x55d9c4c1a600 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:01.613083) v4 > -7222> 2018-07-11 07:51:01.613752 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.2 10.8.128.106:6810/12651 232 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:01.613083) v4 ==== 2004+0+0 (3511827592 0 0) 0x55d9bcd72c00 con 0x55d986b9b800 > -7221> 2018-07-11 07:51:01.613755 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.5 10.8.128.113:6807/15263 232 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:01.613083) v4 ==== 2004+0+0 (3511827592 0 0) 0x55d9c4c1a600 con 0x55d986b9e800 > -7220> 2018-07-11 07:51:01.613753 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.5 10.8.128.113:6806/15263 232 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:01.613083) v4 ==== 2004+0+0 (3511827592 0 0) 0x55d9869be600 con 0x55d986b9d000 > -7219> 2018-07-11 07:51:01.613808 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6807/12677 conn(0x55d986b59800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.1 seq 232 0x55d9c4c1a600 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:01.613083) v4 > -7218> 2018-07-11 07:51:01.613822 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.1 10.8.128.106:6807/12677 232 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:01.613083) v4 ==== 2004+0+0 (3511827592 0 0) 0x55d9c4c1a600 con 0x55d986b59800 > -7217> 2018-07-11 07:51:01.613799 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6809/12650 conn(0x55d986bcb000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 232 0x55d9869be600 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:01.613083) v4 > -7216> 2018-07-11 07:51:01.613832 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.3 10.8.128.106:6809/12650 232 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:01.613083) v4 ==== 2004+0+0 (3511827592 0 0) 0x55d9869be600 con 0x55d986bcb000 > -7215> 2018-07-11 07:51:01.613843 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6806/15885 conn(0x55d986bac000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.7 seq 232 0x55d9bcd72c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:01.613083) v4 > -7214> 2018-07-11 07:51:01.613862 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.7 10.8.128.114:6806/15885 232 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:01.613083) v4 ==== 2004+0+0 (3511827592 0 0) 0x55d9bcd72c00 con 0x55d986bac000 > -7213> 2018-07-11 07:51:01.613851 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6811/12650 conn(0x55d986bcc800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 232 0x55d9c4c1a600 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:01.613083) v4 > -7212> 2018-07-11 07:51:01.613876 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.3 10.8.128.106:6811/12650 232 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:01.613083) v4 ==== 2004+0+0 (3511827592 0 0) 0x55d9c4c1a600 con 0x55d986bcc800 > -7211> 2018-07-11 07:51:01.613896 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.115:6802/13323 conn(0x55d986bc1000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 232 0x55d9bcd72c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:01.613083) v4 > -7210> 2018-07-11 07:51:01.613908 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.6 10.8.128.115:6802/13323 232 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:01.613083) v4 ==== 2004+0+0 (3511827592 0 0) 0x55d9bcd72c00 con 0x55d986bc1000 > -7209> 2018-07-11 07:51:01.613876 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6806/12677 conn(0x55d986b58000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.1 seq 232 0x55d9869be600 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:01.613083) v4 > -7208> 2018-07-11 07:51:01.613919 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.1 10.8.128.106:6806/12677 232 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:01.613083) v4 ==== 2004+0+0 (3511827592 0 0) 0x55d9869be600 con 0x55d986b58000 > -7207> 2018-07-11 07:51:01.613919 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.108:6803/11967 conn(0x55d986bbf800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.8 seq 232 0x55d9c4c1a600 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:01.613083) v4 > -7206> 2018-07-11 07:51:01.613936 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.8 10.8.128.108:6803/11967 232 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:01.613083) v4 ==== 2004+0+0 (3511827592 0 0) 0x55d9c4c1a600 con 0x55d986bbf800 > -7205> 2018-07-11 07:51:01.613941 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6802/15919 conn(0x55d986b5b000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.11 seq 232 0x55d9bcd72c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:01.613083) v4 > -7204> 2018-07-11 07:51:01.613954 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.11 10.8.128.114:6802/15919 232 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:01.613083) v4 ==== 2004+0+0 (3511827592 0 0) 0x55d9bcd72c00 con 0x55d986b5b000 > -7203> 2018-07-11 07:51:01.613975 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6807/15885 conn(0x55d986bad800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.7 seq 232 0x55d9869be600 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:01.613083) v4 > -7202> 2018-07-11 07:51:01.613989 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.7 10.8.128.114:6807/15885 232 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:01.613083) v4 ==== 2004+0+0 (3511827592 0 0) 0x55d9869be600 con 0x55d986bad800 > -7201> 2018-07-11 07:51:01.614029 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.108:6802/11967 conn(0x55d986bbe000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.8 seq 232 0x55d9869be600 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:01.613083) v4 > -7200> 2018-07-11 07:51:01.614043 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.8 10.8.128.108:6802/11967 232 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:01.613083) v4 ==== 2004+0+0 (3511827592 0 0) 0x55d9869be600 con 0x55d986bbe000 > -7199> 2018-07-11 07:51:01.614079 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.115:6803/13323 conn(0x55d986bc2800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 232 0x55d9869be600 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:01.613083) v4 > -7198> 2018-07-11 07:51:01.614093 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.6 10.8.128.115:6803/13323 232 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:01.613083) v4 ==== 2004+0+0 (3511827592 0 0) 0x55d9869be600 con 0x55d986bc2800 > -7197> 2018-07-11 07:51:01.614133 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6803/15919 conn(0x55d986b5c800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.11 seq 232 0x55d9869be600 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:01.613083) v4 > -7196> 2018-07-11 07:51:01.614147 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.11 10.8.128.114:6803/15919 232 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:01.613083) v4 ==== 2004+0+0 (3511827592 0 0) 0x55d9869be600 con 0x55d986b5c800 > -7195> 2018-07-11 07:51:01.614489 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.114:0/15885 conn(0x55d986d81000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.7 seq 247 0x55d9c4c1a600 osd_ping(ping e501 stamp 2018-07-11 07:51:02.244603) v4 > -7194> 2018-07-11 07:51:01.614503 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.7 10.8.128.114:0/15885 247 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:02.244603) v4 ==== 2004+0+0 (3614099329 0 0) 0x55d9c4c1a600 con 0x55d986d81000 > -7193> 2018-07-11 07:51:01.614516 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:02.244603) v4 -- 0x55d986d32200 con 0 > -7192> 2018-07-11 07:51:01.614530 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.114:0/15885 conn(0x55d986d7f800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.7 seq 247 0x55d9bcd72c00 osd_ping(ping e501 stamp 2018-07-11 07:51:02.244603) v4 > -7191> 2018-07-11 07:51:01.614549 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.7 10.8.128.114:0/15885 247 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:02.244603) v4 ==== 2004+0+0 (3614099329 0 0) 0x55d9bcd72c00 con 0x55d986d7f800 > -7190> 2018-07-11 07:51:01.614567 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:02.244603) v4 -- 0x55d9829e5a00 con 0 > -7189> 2018-07-11 07:51:01.630868 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 855 0x55d983e52900 rep_scrubmap(5.2e e501 from shard 4) v1 > -7188> 2018-07-11 07:51:01.630888 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 855 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12472 (3686815630 0 9716575) 0x55d983e52900 con 0x55d986bec000 > -7187> 2018-07-11 07:51:01.636651 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 861 0x55d983e52b40 rep_scrubmap(5.2e e501 from shard 2) v1 > -7186> 2018-07-11 07:51:01.636670 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 861 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12472 (1269249638 0 9716575) 0x55d983e52b40 con 0x55d986bf0800 > -7185> 2018-07-11 07:51:01.741813 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -7184> 2018-07-11 07:51:01.741852 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -7183> 2018-07-11 07:51:01.741908 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -7182> 2018-07-11 07:51:01.742377 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -7181> 2018-07-11 07:51:01.758167 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67587,epoch:501/498,start:5:7532728b:::benchmark_data_magna115_10867_object557495:0,end:5:7532bed5:::benchmark_data_magna115_10867_object2564586:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498500 con 0 > -7180> 2018-07-11 07:51:01.758215 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67587,epoch:501/498,start:5:7532728b:::benchmark_data_magna115_10867_object557495:0,end:5:7532bed5:::benchmark_data_magna115_10867_object2564586:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1499680 con 0 > -7179> 2018-07-11 07:51:01.760706 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12677 conn(0x55d986db4800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.1 seq 246 0x55d9869bfa00 osd_ping(ping e501 stamp 2018-07-11 07:51:01.807936) v4 > -7178> 2018-07-11 07:51:01.760725 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12677 conn(0x55d986df2000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.1 seq 246 0x55d9c4c1a600 osd_ping(ping e501 stamp 2018-07-11 07:51:01.807936) v4 > -7177> 2018-07-11 07:51:01.760733 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.1 10.8.128.106:0/12677 246 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:01.807936) v4 ==== 2004+0+0 (648232102 0 0) 0x55d9869bfa00 con 0x55d986db4800 > -7176> 2018-07-11 07:51:01.760740 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.1 10.8.128.106:0/12677 246 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:01.807936) v4 ==== 2004+0+0 (648232102 0 0) 0x55d9c4c1a600 con 0x55d986df2000 > -7175> 2018-07-11 07:51:01.760750 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:01.807936) v4 -- 0x55d9829e5a00 con 0 > -7174> 2018-07-11 07:51:01.760779 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:01.807936) v4 -- 0x55d986d32200 con 0 > -7173> 2018-07-11 07:51:01.834208 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 2990 0x55d9869be600 MOSDScrubReserve(11.3s5 REQUEST e501) v1 > -7172> 2018-07-11 07:51:01.834227 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 2990 ==== MOSDScrubReserve(11.3s5 REQUEST e501) v1 ==== 43+0+0 (3260170403 0 0) 0x55d9869be600 con 0x55d986c7e000 > -7171> 2018-07-11 07:51:01.834309 7ffab3be0700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e501) v1 -- 0x55d9b3db3c00 con 0 > -7170> 2018-07-11 07:51:01.834666 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 2991 0x55d9b3db3c00 MOSDScrubReserve(11.3s5 RELEASE e501) v1 > -7169> 2018-07-11 07:51:01.834680 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 2991 ==== MOSDScrubReserve(11.3s5 RELEASE e501) v1 ==== 43+0+0 (223328929 0 0) 0x55d9b3db3c00 con 0x55d986c7e000 > -7168> 2018-07-11 07:51:01.967667 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 862 0x55d983e52d80 rep_scrubmap(5.2e e501 from shard 2) v1 > -7167> 2018-07-11 07:51:01.967688 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 862 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12474 (1269249638 0 3826034731) 0x55d983e52d80 con 0x55d986bf0800 > -7166> 2018-07-11 07:51:01.999791 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 856 0x55d983e52fc0 rep_scrubmap(5.2e e501 from shard 4) v1 > -7165> 2018-07-11 07:51:01.999812 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 856 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12474 (3686815630 0 3826034731) 0x55d983e52fc0 con 0x55d986bec000 > -7164> 2018-07-11 07:51:02.087930 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -7163> 2018-07-11 07:51:02.087969 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -7162> 2018-07-11 07:51:02.088026 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -7161> 2018-07-11 07:51:02.088494 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -7160> 2018-07-11 07:51:02.089340 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67342,epoch:501/498,start:5:7532bed5:::benchmark_data_magna115_10867_object2564586:0,end:5:75332f54:::benchmark_data_magna115_10867_object2551838:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1499900 con 0 > -7159> 2018-07-11 07:51:02.089375 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67342,epoch:501/498,start:5:7532bed5:::benchmark_data_magna115_10867_object2564586:0,end:5:75332f54:::benchmark_data_magna115_10867_object2551838:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b14f0000 con 0 > -7158> 2018-07-11 07:51:02.297043 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 863 0x55d983e53200 rep_scrubmap(5.2e e501 from shard 2) v1 > -7157> 2018-07-11 07:51:02.297064 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 863 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12476 (1269249638 0 613852960) 0x55d983e53200 con 0x55d986bf0800 > -7156> 2018-07-11 07:51:02.301564 7ffab7be8700 10 monclient: tick > -7155> 2018-07-11 07:51:02.301578 7ffab7be8700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2018-07-11 07:50:32.301576) > -7154> 2018-07-11 07:51:02.309483 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 857 0x55d983e53440 rep_scrubmap(5.2e e501 from shard 4) v1 > -7153> 2018-07-11 07:51:02.309503 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 857 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12476 (3686815630 0 613852960) 0x55d983e53440 con 0x55d986bec000 > -7152> 2018-07-11 07:51:02.344288 7ffacbbd1700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15263 conn(0x55d985d44000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.5 seq 225 0x55d985054a00 osd_ping(ping e501 stamp 2018-07-11 07:51:03.574783) v4 > -7151> 2018-07-11 07:51:02.344310 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 <== osd.5 10.8.128.113:0/15263 225 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:03.574783) v4 ==== 2004+0+0 (1768840296 0 0) 0x55d985054a00 con 0x55d985d44000 > -7150> 2018-07-11 07:51:02.344320 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:03.574783) v4 -- 0x55d9bf805400 con 0 > -7149> 2018-07-11 07:51:02.344311 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15263 conn(0x55d985d45800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.5 seq 225 0x55d9bcd72c00 osd_ping(ping e501 stamp 2018-07-11 07:51:03.574783) v4 > -7148> 2018-07-11 07:51:02.344329 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.5 10.8.128.113:0/15263 225 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:03.574783) v4 ==== 2004+0+0 (1768840296 0 0) 0x55d9bcd72c00 con 0x55d985d45800 > -7147> 2018-07-11 07:51:02.344343 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:03.574783) v4 -- 0x55d9829e5a00 con 0 > -7146> 2018-07-11 07:51:02.348625 7ffab6be6700 1 -- 10.8.128.107:6800/13927 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55d9b3fdf8c0 con 0 > -7145> 2018-07-11 07:51:02.348692 7ffab6be6700 1 -- 10.8.128.107:6800/13927 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55d9ba81b8c0 con 0 > -7144> 2018-07-11 07:51:02.384439 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.108:0/11967 conn(0x55d986db0000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.8 seq 223 0x55d9bcd72c00 osd_ping(ping e501 stamp 2018-07-11 07:51:02.140005) v4 > -7143> 2018-07-11 07:51:02.384477 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.8 10.8.128.108:0/11967 223 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:02.140005) v4 ==== 2004+0+0 (2002688468 0 0) 0x55d9bcd72c00 con 0x55d986db0000 > -7142> 2018-07-11 07:51:02.384496 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:02.140005) v4 -- 0x55d9829e5a00 con 0 > -7141> 2018-07-11 07:51:02.384539 7ffacbbd1700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.108:0/11967 conn(0x55d986d82800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.8 seq 223 0x55d982cdf200 osd_ping(ping e501 stamp 2018-07-11 07:51:02.140005) v4 > -7140> 2018-07-11 07:51:02.384564 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 <== osd.8 10.8.128.108:0/11967 223 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:02.140005) v4 ==== 2004+0+0 (2002688468 0 0) 0x55d982cdf200 con 0x55d986d82800 > -7139> 2018-07-11 07:51:02.384575 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:02.140005) v4 -- 0x55d9bf805400 con 0 > -7138> 2018-07-11 07:51:02.406108 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -7137> 2018-07-11 07:51:02.406133 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -7136> 2018-07-11 07:51:02.406190 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -7135> 2018-07-11 07:51:02.406659 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -7134> 2018-07-11 07:51:02.425118 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75332f54:::benchmark_data_magna115_10867_object2551838:0,end:5:75339d48:::benchmark_data_magna115_11707_object236556:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b14f0280 con 0 > -7133> 2018-07-11 07:51:02.425166 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75332f54:::benchmark_data_magna115_10867_object2551838:0,end:5:75339d48:::benchmark_data_magna115_11707_object236556:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98696f400 con 0 > -7132> 2018-07-11 07:51:02.649243 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 858 0x55d983e53680 rep_scrubmap(5.2e e501 from shard 4) v1 > -7131> 2018-07-11 07:51:02.649283 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 858 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12476 (3686815630 0 959576729) 0x55d983e53680 con 0x55d986bec000 > -7130> 2018-07-11 07:51:02.660276 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 864 0x55d983e538c0 rep_scrubmap(5.2e e501 from shard 2) v1 > -7129> 2018-07-11 07:51:02.660295 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 864 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12476 (1269249638 0 959576729) 0x55d983e538c0 con 0x55d986bf0800 > -7128> 2018-07-11 07:51:02.750643 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -7127> 2018-07-11 07:51:02.750667 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -7126> 2018-07-11 07:51:02.750723 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -7125> 2018-07-11 07:51:02.751170 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -7124> 2018-07-11 07:51:02.791615 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67422,epoch:501/498,start:5:75339d48:::benchmark_data_magna115_11707_object236556:0,end:5:7533eceb:::benchmark_data_magna115_10867_object3687699:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b14f0500 con 0 > -7123> 2018-07-11 07:51:02.791667 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67422,epoch:501/498,start:5:75339d48:::benchmark_data_magna115_11707_object236556:0,end:5:7533eceb:::benchmark_data_magna115_10867_object3687699:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455900 con 0 > -7122> 2018-07-11 07:51:02.863314 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15255 conn(0x55d986d20000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.0 seq 243 0x55d97f5adc00 osd_ping(ping e501 stamp 2018-07-11 07:51:04.093682) v4 > -7121> 2018-07-11 07:51:02.863341 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.0 10.8.128.113:0/15255 243 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:04.093682) v4 ==== 2004+0+0 (3497069146 0 0) 0x55d97f5adc00 con 0x55d986d20000 > -7120> 2018-07-11 07:51:02.863340 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15255 conn(0x55d986c82800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.0 seq 243 0x55d9869bfa00 osd_ping(ping e501 stamp 2018-07-11 07:51:04.093682) v4 > -7119> 2018-07-11 07:51:02.863351 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:04.093682) v4 -- 0x55d986d32200 con 0 > -7118> 2018-07-11 07:51:02.863363 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.0 10.8.128.113:0/15255 243 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:04.093682) v4 ==== 2004+0+0 (3497069146 0 0) 0x55d9869bfa00 con 0x55d986c82800 > -7117> 2018-07-11 07:51:02.863382 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:04.093682) v4 -- 0x55d9829e5a00 con 0 > -7116> 2018-07-11 07:51:02.958432 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6801/15919 conn(0x55d986c3f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=29 cs=1 l=0). rx osd.11 seq 220 0x55d985054a00 MOSDScrubReserve(10.f REQUEST e501) v1 > -7115> 2018-07-11 07:51:02.958450 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.11 10.8.128.114:6801/15919 220 ==== MOSDScrubReserve(10.f REQUEST e501) v1 ==== 43+0+0 (2892966580 0 0) 0x55d985054a00 con 0x55d986c3f800 > -7114> 2018-07-11 07:51:02.958570 7ffab53e3700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.114:6801/15919 -- MOSDScrubReserve(10.f REJECT e501) v1 -- 0x55d9bf805800 con 0 > -7113> 2018-07-11 07:51:02.958805 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6801/15919 conn(0x55d986c3f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=29 cs=1 l=0). rx osd.11 seq 221 0x55d9bf805400 MOSDScrubReserve(10.f RELEASE e501) v1 > -7112> 2018-07-11 07:51:02.958834 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.11 10.8.128.114:6801/15919 221 ==== MOSDScrubReserve(10.f RELEASE e501) v1 ==== 43+0+0 (1668468918 0 0) 0x55d9bf805400 con 0x55d986c3f800 > -7111> 2018-07-11 07:51:02.988888 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 859 0x55d983e53b00 rep_scrubmap(5.2e e501 from shard 4) v1 > -7110> 2018-07-11 07:51:02.988908 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 859 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12474 (3686815630 0 2835369448) 0x55d983e53b00 con 0x55d986bec000 > -7109> 2018-07-11 07:51:03.004883 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 865 0x55d983e53d40 rep_scrubmap(5.2e e501 from shard 2) v1 > -7108> 2018-07-11 07:51:03.004902 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 865 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12474 (1269249638 0 2835369448) 0x55d983e53d40 con 0x55d986bf0800 > -7107> 2018-07-11 07:51:03.580196 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15287 conn(0x55d981240800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 254 0x55d9c4c1a600 osd_ping(ping e501 stamp 2018-07-11 07:51:04.810638) v4 > -7106> 2018-07-11 07:51:03.580232 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.4 10.8.128.113:0/15287 254 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:04.810638) v4 ==== 2004+0+0 (3617259190 0 0) 0x55d9c4c1a600 con 0x55d981240800 > -7105> 2018-07-11 07:51:03.580229 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15287 conn(0x55d98123f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 254 0x55d9bcd72c00 osd_ping(ping e501 stamp 2018-07-11 07:51:04.810638) v4 > -7104> 2018-07-11 07:51:03.580244 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:04.810638) v4 -- 0x55d986d32200 con 0 > -7103> 2018-07-11 07:51:03.580264 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.4 10.8.128.113:0/15287 254 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:04.810638) v4 ==== 2004+0+0 (3617259190 0 0) 0x55d9bcd72c00 con 0x55d98123f000 > -7102> 2018-07-11 07:51:03.580285 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:04.810638) v4 -- 0x55d9829e5a00 con 0 > -7101> 2018-07-11 07:51:03.958676 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6801/15919 conn(0x55d986c3f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=29 cs=1 l=0). rx osd.11 seq 222 0x55d982cdf200 MOSDScrubReserve(10.f REQUEST e501) v1 > -7100> 2018-07-11 07:51:03.958694 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.11 10.8.128.114:6801/15919 222 ==== MOSDScrubReserve(10.f REQUEST e501) v1 ==== 43+0+0 (2892966580 0 0) 0x55d982cdf200 con 0x55d986c3f800 > -7099> 2018-07-11 07:51:03.958821 7ffab53e3700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.114:6801/15919 -- MOSDScrubReserve(10.f REJECT e501) v1 -- 0x55d9bf805400 con 0 > -7098> 2018-07-11 07:51:03.959157 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6801/15919 conn(0x55d986c3f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=29 cs=1 l=0). rx osd.11 seq 223 0x55d9bf805400 MOSDScrubReserve(10.f RELEASE e501) v1 > -7097> 2018-07-11 07:51:03.959171 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.11 10.8.128.114:6801/15919 223 ==== MOSDScrubReserve(10.f RELEASE e501) v1 ==== 43+0+0 (1668468918 0 0) 0x55d9bf805400 con 0x55d986c3f800 > -7096> 2018-07-11 07:51:04.108243 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -7095> 2018-07-11 07:51:04.108280 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -7094> 2018-07-11 07:51:04.108326 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -7093> 2018-07-11 07:51:04.108786 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -7092> 2018-07-11 07:51:04.125820 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7533eceb:::benchmark_data_magna115_10867_object3687699:0,end:5:753457ae:::benchmark_data_magna115_10867_object1168654:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498780 con 0 > -7091> 2018-07-11 07:51:04.125868 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7533eceb:::benchmark_data_magna115_10867_object3687699:0,end:5:753457ae:::benchmark_data_magna115_10867_object1168654:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455b80 con 0 > -7090> 2018-07-11 07:51:04.277973 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 31] Generated table #36796: 3105384 keys, 75715616 bytes > -7089> 2018-07-11 07:51:04.278019 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295464277991, "cf_name": "default", "job": 31, "event": "table_file_creation", "file_number": 36796, "file_size": 75715616, "table_properties": {"data_size": 67111682, "index_size": 690247, "filter_size": 7912695, "raw_key_size": 89711112, "raw_average_key_size": 28, "raw_value_size": 25047286, "raw_average_value_size": 8, "num_data_blocks": 16486, "num_entries": 3105384, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "998603", "kMergeOperands": "0"}} > -7088> 2018-07-11 07:51:04.357758 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 860 0x55d990386000 rep_scrubmap(5.2e e501 from shard 4) v1 > -7087> 2018-07-11 07:51:04.357779 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 860 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12476 (3686815630 0 1646865774) 0x55d990386000 con 0x55d986bec000 > -7086> 2018-07-11 07:51:04.365037 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 866 0x55d990386240 rep_scrubmap(5.2e e501 from shard 2) v1 > -7085> 2018-07-11 07:51:04.365058 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 866 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12476 (1269249638 0 1646865774) 0x55d990386240 con 0x55d986bf0800 > -7084> 2018-07-11 07:51:04.576866 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -7083> 2018-07-11 07:51:04.576895 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -7082> 2018-07-11 07:51:04.576953 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -7081> 2018-07-11 07:51:04.577425 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -7080> 2018-07-11 07:51:04.592198 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:753457ae:::benchmark_data_magna115_10867_object1168654:0,end:5:7534d386:::benchmark_data_magna115_10867_object2206661:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498000 con 0 > -7079> 2018-07-11 07:51:04.592256 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:753457ae:::benchmark_data_magna115_10867_object1168654:0,end:5:7534d386:::benchmark_data_magna115_10867_object2206661:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455680 con 0 > -7078> 2018-07-11 07:51:04.742222 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.107:0/12617 conn(0x55d98123d800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 229 0x55d9bf805800 osd_ping(ping e501 stamp 2018-07-11 07:51:04.741861) v4 > -7077> 2018-07-11 07:51:04.742222 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.107:0/12617 conn(0x55d98123c000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 229 0x55d9c4c1a600 osd_ping(ping e501 stamp 2018-07-11 07:51:04.741861) v4 > -7076> 2018-07-11 07:51:04.742253 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.9 10.8.128.107:0/12617 229 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:04.741861) v4 ==== 2004+0+0 (3881460134 0 0) 0x55d9c4c1a600 con 0x55d98123c000 > -7075> 2018-07-11 07:51:04.742264 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:04.741861) v4 -- 0x55d986d32200 con 0 > -7074> 2018-07-11 07:51:04.742260 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.9 10.8.128.107:0/12617 229 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:04.741861) v4 ==== 2004+0+0 (3881460134 0 0) 0x55d9bf805800 con 0x55d98123d800 > -7073> 2018-07-11 07:51:04.742313 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:04.741861) v4 -- 0x55d9c4c1aa00 con 0 > -7072> 2018-07-11 07:51:04.809184 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 867 0x55d990386480 rep_scrubmap(5.2e e501 from shard 2) v1 > -7071> 2018-07-11 07:51:04.809204 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 867 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12476 (1269249638 0 1297481314) 0x55d990386480 con 0x55d986bf0800 > -7070> 2018-07-11 07:51:04.813511 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 861 0x55d9903866c0 rep_scrubmap(5.2e e501 from shard 4) v1 > -7069> 2018-07-11 07:51:04.813532 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 861 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12476 (3686815630 0 1297481314) 0x55d9903866c0 con 0x55d986bec000 > -7068> 2018-07-11 07:51:04.897347 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -7067> 2018-07-11 07:51:04.897373 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -7066> 2018-07-11 07:51:04.897420 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -7065> 2018-07-11 07:51:04.897883 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -7064> 2018-07-11 07:51:04.917016 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7534d386:::benchmark_data_magna115_10867_object2206661:0,end:5:75354776:::benchmark_data_magna115_10867_object1354219:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455400 con 0 > -7063> 2018-07-11 07:51:04.917067 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7534d386:::benchmark_data_magna115_10867_object2206661:0,end:5:75354776:::benchmark_data_magna115_10867_object1354219:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498280 con 0 > -7062> 2018-07-11 07:51:04.958864 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6801/15919 conn(0x55d986c3f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=29 cs=1 l=0). rx osd.11 seq 224 0x55d9bdba5400 MOSDScrubReserve(10.f REQUEST e501) v1 > -7061> 2018-07-11 07:51:04.958881 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.11 10.8.128.114:6801/15919 224 ==== MOSDScrubReserve(10.f REQUEST e501) v1 ==== 43+0+0 (2892966580 0 0) 0x55d9bdba5400 con 0x55d986c3f800 > -7060> 2018-07-11 07:51:04.959002 7ffab53e3700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.114:6801/15919 -- MOSDScrubReserve(10.f REJECT e501) v1 -- 0x55d9bf805400 con 0 > -7059> 2018-07-11 07:51:04.959231 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6801/15919 conn(0x55d986c3f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=29 cs=1 l=0). rx osd.11 seq 225 0x55d9c4c1aa00 MOSDScrubReserve(10.f RELEASE e501) v1 > -7058> 2018-07-11 07:51:04.959265 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.11 10.8.128.114:6801/15919 225 ==== MOSDScrubReserve(10.f RELEASE e501) v1 ==== 43+0+0 (1668468918 0 0) 0x55d9c4c1aa00 con 0x55d986c3f800 > -7057> 2018-07-11 07:51:05.114958 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.114:0/15885 conn(0x55d986d7f800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.7 seq 248 0x55d9869bfa00 osd_ping(ping e501 stamp 2018-07-11 07:51:05.745001) v4 > -7056> 2018-07-11 07:51:05.114985 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.7 10.8.128.114:0/15885 248 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:05.745001) v4 ==== 2004+0+0 (1114108804 0 0) 0x55d9869bfa00 con 0x55d986d7f800 > -7055> 2018-07-11 07:51:05.115004 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:05.745001) v4 -- 0x55d9829e5a00 con 0 > -7054> 2018-07-11 07:51:05.115003 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.114:0/15885 conn(0x55d986d81000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.7 seq 248 0x55d97f5adc00 osd_ping(ping e501 stamp 2018-07-11 07:51:05.745001) v4 > -7053> 2018-07-11 07:51:05.115029 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.7 10.8.128.114:0/15885 248 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:05.745001) v4 ==== 2004+0+0 (1114108804 0 0) 0x55d97f5adc00 con 0x55d986d81000 > -7052> 2018-07-11 07:51:05.115045 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:05.745001) v4 -- 0x55d986d32200 con 0 > -7051> 2018-07-11 07:51:05.148174 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 868 0x55d990386900 rep_scrubmap(5.2e e501 from shard 2) v1 > -7050> 2018-07-11 07:51:05.148195 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 868 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12474 (1269249638 0 2039205942) 0x55d990386900 con 0x55d986bf0800 > -7049> 2018-07-11 07:51:05.164343 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 862 0x55d990386b40 rep_scrubmap(5.2e e501 from shard 4) v1 > -7048> 2018-07-11 07:51:05.164362 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 862 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12474 (3686815630 0 2039205942) 0x55d990386b40 con 0x55d986bec000 > -7047> 2018-07-11 07:51:05.261027 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12677 conn(0x55d986df2000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.1 seq 247 0x55d9c4c1a600 osd_ping(ping e501 stamp 2018-07-11 07:51:05.308346) v4 > -7046> 2018-07-11 07:51:05.261026 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12677 conn(0x55d986db4800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.1 seq 247 0x55d9bcd72c00 osd_ping(ping e501 stamp 2018-07-11 07:51:05.308346) v4 > -7045> 2018-07-11 07:51:05.261053 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.1 10.8.128.106:0/12677 247 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:05.308346) v4 ==== 2004+0+0 (490695799 0 0) 0x55d9c4c1a600 con 0x55d986df2000 > -7044> 2018-07-11 07:51:05.261056 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.1 10.8.128.106:0/12677 247 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:05.308346) v4 ==== 2004+0+0 (490695799 0 0) 0x55d9bcd72c00 con 0x55d986db4800 > -7043> 2018-07-11 07:51:05.261063 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:05.308346) v4 -- 0x55d986d32200 con 0 > -7042> 2018-07-11 07:51:05.261085 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:05.308346) v4 -- 0x55d9829e5a00 con 0 > -7041> 2018-07-11 07:51:05.284916 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.108:0/11967 conn(0x55d986db0000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.8 seq 224 0x55d9869bfa00 osd_ping(ping e501 stamp 2018-07-11 07:51:05.040460) v4 > -7040> 2018-07-11 07:51:05.284938 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.8 10.8.128.108:0/11967 224 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:05.040460) v4 ==== 2004+0+0 (3588705466 0 0) 0x55d9869bfa00 con 0x55d986db0000 > -7039> 2018-07-11 07:51:05.284954 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:05.040460) v4 -- 0x55d9829e5a00 con 0 > -7038> 2018-07-11 07:51:05.284971 7ffacbbd1700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.108:0/11967 conn(0x55d986d82800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.8 seq 224 0x55d9bf805800 osd_ping(ping e501 stamp 2018-07-11 07:51:05.040460) v4 > -7037> 2018-07-11 07:51:05.284992 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 <== osd.8 10.8.128.108:0/11967 224 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:05.040460) v4 ==== 2004+0+0 (3588705466 0 0) 0x55d9bf805800 con 0x55d986d82800 > -7036> 2018-07-11 07:51:05.285003 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:05.040460) v4 -- 0x55d9c34cca00 con 0 > -7035> 2018-07-11 07:51:05.342912 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -7034> 2018-07-11 07:51:05.342941 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -7033> 2018-07-11 07:51:05.343003 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -7032> 2018-07-11 07:51:05.343479 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -7031> 2018-07-11 07:51:05.359062 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75354776:::benchmark_data_magna115_10867_object1354219:0,end:5:75358e12:::benchmark_data_magna115_10867_object733655:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454f00 con 0 > -7030> 2018-07-11 07:51:05.359111 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75354776:::benchmark_data_magna115_10867_object1354219:0,end:5:75358e12:::benchmark_data_magna115_10867_object733655:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454780 con 0 > -7029> 2018-07-11 07:51:05.512019 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12650 conn(0x55d986d23000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 235 0x55d9bcd72c00 osd_ping(ping e501 stamp 2018-07-11 07:51:05.559318) v4 > -7028> 2018-07-11 07:51:05.512028 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12650 conn(0x55d986d21800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 235 0x55d9bf805400 osd_ping(ping e501 stamp 2018-07-11 07:51:05.559318) v4 > -7027> 2018-07-11 07:51:05.512041 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.3 10.8.128.106:0/12650 235 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:05.559318) v4 ==== 2004+0+0 (1850826166 0 0) 0x55d9bcd72c00 con 0x55d986d23000 > -7026> 2018-07-11 07:51:05.512046 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.3 10.8.128.106:0/12650 235 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:05.559318) v4 ==== 2004+0+0 (1850826166 0 0) 0x55d9bf805400 con 0x55d986d21800 > -7025> 2018-07-11 07:51:05.512055 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:05.559318) v4 -- 0x55d9829e5a00 con 0 > -7024> 2018-07-11 07:51:05.512087 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:05.559318) v4 -- 0x55d9c34cca00 con 0 > -7023> 2018-07-11 07:51:05.560107 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6805/15885 conn(0x55d986c7f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.7 seq 2782 0x55d9c4c1a600 MOSDScrubReserve(5.4 REQUEST e501) v1 > -7022> 2018-07-11 07:51:05.560124 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.7 10.8.128.114:6805/15885 2782 ==== MOSDScrubReserve(5.4 REQUEST e501) v1 ==== 43+0+0 (1753221512 0 0) 0x55d9c4c1a600 con 0x55d986c7f800 > -7021> 2018-07-11 07:51:05.560255 7ffab33df700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.114:6805/15885 -- MOSDScrubReserve(5.4 REJECT e501) v1 -- 0x55d9b9ac3600 con 0 > -7020> 2018-07-11 07:51:05.560498 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6805/15885 conn(0x55d986c7f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.7 seq 2783 0x55d986d32200 MOSDScrubReserve(5.4 RELEASE e501) v1 > -7019> 2018-07-11 07:51:05.560513 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.7 10.8.128.114:6805/15885 2783 ==== MOSDScrubReserve(5.4 RELEASE e501) v1 ==== 43+0+0 (2812146570 0 0) 0x55d986d32200 con 0x55d986c7f800 > -7018> 2018-07-11 07:51:05.564321 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 869 0x55d990386d80 rep_scrubmap(5.2e e501 from shard 2) v1 > -7017> 2018-07-11 07:51:05.564341 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 869 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12468 (1269249638 0 479619358) 0x55d990386d80 con 0x55d986bf0800 > -7016> 2018-07-11 07:51:05.586375 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 863 0x55d990386fc0 rep_scrubmap(5.2e e501 from shard 4) v1 > -7015> 2018-07-11 07:51:05.586394 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 863 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12468 (3686815630 0 479619358) 0x55d990386fc0 con 0x55d986bec000 > -7014> 2018-07-11 07:51:05.598351 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.115:0/13323 conn(0x55d986d7e000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 244 0x55d9bf805800 osd_ping(ping e501 stamp 2018-07-11 07:51:05.943888) v4 > -7013> 2018-07-11 07:51:05.598373 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.6 10.8.128.115:0/13323 244 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:05.943888) v4 ==== 2004+0+0 (2197488971 0 0) 0x55d9bf805800 con 0x55d986d7e000 > -7012> 2018-07-11 07:51:05.598383 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:05.943888) v4 -- 0x55d9c34cca00 con 0 > -7011> 2018-07-11 07:51:05.598447 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.115:0/13323 conn(0x55d986d24800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 244 0x55d97f5adc00 osd_ping(ping e501 stamp 2018-07-11 07:51:05.943888) v4 > -7010> 2018-07-11 07:51:05.598468 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.6 10.8.128.115:0/13323 244 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:05.943888) v4 ==== 2004+0+0 (2197488971 0 0) 0x55d97f5adc00 con 0x55d986d24800 > -7009> 2018-07-11 07:51:05.598479 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:05.943888) v4 -- 0x55d980f5a400 con 0 > -7008> 2018-07-11 07:51:05.674451 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -7007> 2018-07-11 07:51:05.674478 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -7006> 2018-07-11 07:51:05.674525 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -7005> 2018-07-11 07:51:05.674988 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -7004> 2018-07-11 07:51:05.689039 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.114:0/15919 conn(0x55d986df3800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.11 seq 244 0x55d9bf805400 osd_ping(ping e501 stamp 2018-07-11 07:51:06.319094) v4 > -7003> 2018-07-11 07:51:05.689065 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.11 10.8.128.114:0/15919 244 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:06.319094) v4 ==== 2004+0+0 (2931722659 0 0) 0x55d9bf805400 con 0x55d986df3800 > -7002> 2018-07-11 07:51:05.689079 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:06.319094) v4 -- 0x55d9c34cca00 con 0 > -7001> 2018-07-11 07:51:05.689088 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.114:0/15919 conn(0x55d986df5000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.11 seq 244 0x55d9869bfa00 osd_ping(ping e501 stamp 2018-07-11 07:51:06.319094) v4 > -7000> 2018-07-11 07:51:05.689114 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.11 10.8.128.114:0/15919 244 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:06.319094) v4 ==== 2004+0+0 (2931722659 0 0) 0x55d9869bfa00 con 0x55d986df5000 > -6999> 2018-07-11 07:51:05.689133 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:06.319094) v4 -- 0x55d9829e5a00 con 0 > -6998> 2018-07-11 07:51:05.692212 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75358e12:::benchmark_data_magna115_10867_object733655:0,end:5:7535e41c:::benchmark_data_magna115_10867_object3814285:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455180 con 0 > -6997> 2018-07-11 07:51:05.692271 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75358e12:::benchmark_data_magna115_10867_object733655:0,end:5:7535e41c:::benchmark_data_magna115_10867_object3814285:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454500 con 0 > -6996> 2018-07-11 07:51:05.718606 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12651 conn(0x55d986db3000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.2 seq 242 0x55d9bf805800 osd_ping(ping e501 stamp 2018-07-11 07:51:05.765904) v4 > -6995> 2018-07-11 07:51:05.718626 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.2 10.8.128.106:0/12651 242 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:05.765904) v4 ==== 2004+0+0 (578962426 0 0) 0x55d9bf805800 con 0x55d986db3000 > -6994> 2018-07-11 07:51:05.718635 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:05.765904) v4 -- 0x55d9c34cca00 con 0 > -6993> 2018-07-11 07:51:05.718628 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12651 conn(0x55d986db1800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.2 seq 242 0x55d97f5adc00 osd_ping(ping e501 stamp 2018-07-11 07:51:05.765904) v4 > -6992> 2018-07-11 07:51:05.718649 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.2 10.8.128.106:0/12651 242 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:05.765904) v4 ==== 2004+0+0 (578962426 0 0) 0x55d97f5adc00 con 0x55d986db1800 > -6991> 2018-07-11 07:51:05.718668 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:05.765904) v4 -- 0x55d980f5a400 con 0 > -6990> 2018-07-11 07:51:05.842786 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.107:0/12617 conn(0x55d98123c000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 230 0x55d9b9ac3600 osd_ping(ping e501 stamp 2018-07-11 07:51:05.842336) v4 > -6989> 2018-07-11 07:51:05.842813 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.9 10.8.128.107:0/12617 230 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:05.842336) v4 ==== 2004+0+0 (189566475 0 0) 0x55d9b9ac3600 con 0x55d98123c000 > -6988> 2018-07-11 07:51:05.842821 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.107:0/12617 conn(0x55d98123d800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 230 0x55d9bf805400 osd_ping(ping e501 stamp 2018-07-11 07:51:05.842336) v4 > -6987> 2018-07-11 07:51:05.842831 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:05.842336) v4 -- 0x55d980f5a400 con 0 > -6986> 2018-07-11 07:51:05.842848 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.9 10.8.128.107:0/12617 230 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:05.842336) v4 ==== 2004+0+0 (189566475 0 0) 0x55d9bf805400 con 0x55d98123d800 > -6985> 2018-07-11 07:51:05.842872 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:05.842336) v4 -- 0x55d9c34cca00 con 0 > -6984> 2018-07-11 07:51:05.844638 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15263 conn(0x55d985d45800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.5 seq 226 0x55d9bcd72c00 osd_ping(ping e501 stamp 2018-07-11 07:51:07.075205) v4 > -6983> 2018-07-11 07:51:05.844657 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.5 10.8.128.113:0/15263 226 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:07.075205) v4 ==== 2004+0+0 (775866544 0 0) 0x55d9bcd72c00 con 0x55d985d45800 > -6982> 2018-07-11 07:51:05.844673 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:07.075205) v4 -- 0x55d9829e5a00 con 0 > -6981> 2018-07-11 07:51:05.844677 7ffacbbd1700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15263 conn(0x55d985d44000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.5 seq 226 0x55d9bf805800 osd_ping(ping e501 stamp 2018-07-11 07:51:07.075205) v4 > -6980> 2018-07-11 07:51:05.844699 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 <== osd.5 10.8.128.113:0/15263 226 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:07.075205) v4 ==== 2004+0+0 (775866544 0 0) 0x55d9bf805800 con 0x55d985d44000 > -6979> 2018-07-11 07:51:05.844710 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:07.075205) v4 -- 0x55d9c34cca00 con 0 > -6978> 2018-07-11 07:51:05.901441 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 870 0x55d990387200 rep_scrubmap(5.2e e501 from shard 2) v1 > -6977> 2018-07-11 07:51:05.901461 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 870 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12476 (1269249638 0 3895487477) 0x55d990387200 con 0x55d986bf0800 > -6976> 2018-07-11 07:51:05.923062 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 864 0x55d990387440 rep_scrubmap(5.2e e501 from shard 4) v1 > -6975> 2018-07-11 07:51:05.923081 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 864 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12476 (3686815630 0 3895487477) 0x55d990387440 con 0x55d986bec000 > -6974> 2018-07-11 07:51:05.997732 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -6973> 2018-07-11 07:51:05.997770 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -6972> 2018-07-11 07:51:05.997829 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -6971> 2018-07-11 07:51:05.998325 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -6970> 2018-07-11 07:51:05.999372 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7535e41c:::benchmark_data_magna115_10867_object3814285:0,end:5:753637da:::benchmark_data_magna115_10867_object1338554:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454c80 con 0 > -6969> 2018-07-11 07:51:05.999407 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7535e41c:::benchmark_data_magna115_10867_object3814285:0,end:5:753637da:::benchmark_data_magna115_10867_object1338554:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98944b680 con 0 > -6968> 2018-07-11 07:51:06.098805 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.115:0/13323 conn(0x55d986d7e000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 245 0x55d9bf805400 osd_ping(ping e501 stamp 2018-07-11 07:51:06.444331) v4 > -6967> 2018-07-11 07:51:06.098809 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.115:0/13323 conn(0x55d986d24800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 245 0x55d9b9ac3600 osd_ping(ping e501 stamp 2018-07-11 07:51:06.444331) v4 > -6966> 2018-07-11 07:51:06.098826 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.6 10.8.128.115:0/13323 245 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:06.444331) v4 ==== 2004+0+0 (4213437786 0 0) 0x55d9bf805400 con 0x55d986d7e000 > -6965> 2018-07-11 07:51:06.098831 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.6 10.8.128.115:0/13323 245 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:06.444331) v4 ==== 2004+0+0 (4213437786 0 0) 0x55d9b9ac3600 con 0x55d986d24800 > -6964> 2018-07-11 07:51:06.098839 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:06.444331) v4 -- 0x55d9c34cca00 con 0 > -6963> 2018-07-11 07:51:06.098870 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:06.444331) v4 -- 0x55d980f5a400 con 0 > -6962> 2018-07-11 07:51:06.191433 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 871 0x55d990387680 rep_scrubmap(5.2e e501 from shard 2) v1 > -6961> 2018-07-11 07:51:06.191453 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 871 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12474 (1269249638 0 1178559059) 0x55d990387680 con 0x55d986bf0800 > -6960> 2018-07-11 07:51:06.215370 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.114:0/15885 conn(0x55d986d7f800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.7 seq 249 0x55d9869bfa00 osd_ping(ping e501 stamp 2018-07-11 07:51:06.845403) v4 > -6959> 2018-07-11 07:51:06.215394 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.7 10.8.128.114:0/15885 249 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:06.845403) v4 ==== 2004+0+0 (2047236050 0 0) 0x55d9869bfa00 con 0x55d986d7f800 > -6958> 2018-07-11 07:51:06.215403 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.114:0/15885 conn(0x55d986d81000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.7 seq 249 0x55d9b9ac3600 osd_ping(ping e501 stamp 2018-07-11 07:51:06.845403) v4 > -6957> 2018-07-11 07:51:06.215413 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:06.845403) v4 -- 0x55d9829e5a00 con 0 > -6956> 2018-07-11 07:51:06.215424 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.7 10.8.128.114:0/15885 249 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:06.845403) v4 ==== 2004+0+0 (2047236050 0 0) 0x55d9b9ac3600 con 0x55d986d81000 > -6955> 2018-07-11 07:51:06.215437 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:06.845403) v4 -- 0x55d980f5a400 con 0 > -6954> 2018-07-11 07:51:06.219745 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 865 0x55d9903878c0 rep_scrubmap(5.2e e501 from shard 4) v1 > -6953> 2018-07-11 07:51:06.219765 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 865 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12474 (3686815630 0 1178559059) 0x55d9903878c0 con 0x55d986bec000 > -6952> 2018-07-11 07:51:06.266514 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -6951> 2018-07-11 07:51:06.266539 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -6950> 2018-07-11 07:51:06.266598 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -6949> 2018-07-11 07:51:06.267055 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -6948> 2018-07-11 07:51:06.313572 7ffab1bdc700 5 osd.10 501 heartbeat: osd_stat(788 GB used, 143 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist []) > -6947> 2018-07-11 07:51:06.313585 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6802/15255 -- osd_ping(ping e501 stamp 2018-07-11 07:51:06.313584) v4 -- 0x55d986c22c00 con 0 > -6946> 2018-07-11 07:51:06.313621 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6803/15255 -- osd_ping(ping e501 stamp 2018-07-11 07:51:06.313584) v4 -- 0x55d986d40a00 con 0 > -6945> 2018-07-11 07:51:06.313636 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6806/12677 -- osd_ping(ping e501 stamp 2018-07-11 07:51:06.313584) v4 -- 0x55d9bf804200 con 0 > -6944> 2018-07-11 07:51:06.313650 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6807/12677 -- osd_ping(ping e501 stamp 2018-07-11 07:51:06.313584) v4 -- 0x55d9b60dfa00 con 0 > -6943> 2018-07-11 07:51:06.313663 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6808/12651 -- osd_ping(ping e501 stamp 2018-07-11 07:51:06.313584) v4 -- 0x55d986d4f800 con 0 > -6942> 2018-07-11 07:51:06.313673 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6810/12651 -- osd_ping(ping e501 stamp 2018-07-11 07:51:06.313584) v4 -- 0x55d9b60de600 con 0 > -6941> 2018-07-11 07:51:06.313687 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6809/12650 -- osd_ping(ping e501 stamp 2018-07-11 07:51:06.313584) v4 -- 0x55d986d32c00 con 0 > -6940> 2018-07-11 07:51:06.313699 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6811/12650 -- osd_ping(ping e501 stamp 2018-07-11 07:51:06.313584) v4 -- 0x55d9829e4e00 con 0 > -6939> 2018-07-11 07:51:06.313717 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6810/15287 -- osd_ping(ping e501 stamp 2018-07-11 07:51:06.313584) v4 -- 0x55d9829e4c00 con 0 > -6938> 2018-07-11 07:51:06.313729 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6811/15287 -- osd_ping(ping e501 stamp 2018-07-11 07:51:06.313584) v4 -- 0x55d9b3db2600 con 0 > -6937> 2018-07-11 07:51:06.313742 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6806/15263 -- osd_ping(ping e501 stamp 2018-07-11 07:51:06.313584) v4 -- 0x55d9869bf600 con 0 > -6936> 2018-07-11 07:51:06.313754 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6807/15263 -- osd_ping(ping e501 stamp 2018-07-11 07:51:06.313584) v4 -- 0x55d9869be000 con 0 > -6935> 2018-07-11 07:51:06.313768 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.115:6802/13323 -- osd_ping(ping e501 stamp 2018-07-11 07:51:06.313584) v4 -- 0x55d9b67d8c00 con 0 > -6934> 2018-07-11 07:51:06.313783 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.115:6803/13323 -- osd_ping(ping e501 stamp 2018-07-11 07:51:06.313584) v4 -- 0x55d986d41000 con 0 > -6933> 2018-07-11 07:51:06.313796 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6806/15885 -- osd_ping(ping e501 stamp 2018-07-11 07:51:06.313584) v4 -- 0x55d9c9123000 con 0 > -6932> 2018-07-11 07:51:06.313808 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6807/15885 -- osd_ping(ping e501 stamp 2018-07-11 07:51:06.313584) v4 -- 0x55d9c3383400 con 0 > -6931> 2018-07-11 07:51:06.313819 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.108:6802/11967 -- osd_ping(ping e501 stamp 2018-07-11 07:51:06.313584) v4 -- 0x55d9bfbde200 con 0 > -6930> 2018-07-11 07:51:06.313834 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.108:6803/11967 -- osd_ping(ping e501 stamp 2018-07-11 07:51:06.313584) v4 -- 0x55d9c34cc000 con 0 > -6929> 2018-07-11 07:51:06.313847 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.107:6806/12617 -- osd_ping(ping e501 stamp 2018-07-11 07:51:06.313584) v4 -- 0x55d9bdba4800 con 0 > -6928> 2018-07-11 07:51:06.313859 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.107:6807/12617 -- osd_ping(ping e501 stamp 2018-07-11 07:51:06.313584) v4 -- 0x55d9c908ee00 con 0 > -6927> 2018-07-11 07:51:06.313871 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6802/15919 -- osd_ping(ping e501 stamp 2018-07-11 07:51:06.313584) v4 -- 0x55d982d31e00 con 0 > -6926> 2018-07-11 07:51:06.313883 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6803/15919 -- osd_ping(ping e501 stamp 2018-07-11 07:51:06.313584) v4 -- 0x55d9b3db3200 con 0 > -6925> 2018-07-11 07:51:06.314048 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.107:6806/12617 conn(0x55d986bd2000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.9 seq 233 0x55d9869bfa00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:06.313584) v4 > -6924> 2018-07-11 07:51:06.314069 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.9 10.8.128.107:6806/12617 233 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:06.313584) v4 ==== 2004+0+0 (3609353185 0 0) 0x55d9869bfa00 con 0x55d986bd2000 > -6923> 2018-07-11 07:51:06.314070 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.107:6807/12617 conn(0x55d986bd3800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.9 seq 233 0x55d9bf805800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:06.313584) v4 > -6922> 2018-07-11 07:51:06.314099 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.9 10.8.128.107:6807/12617 233 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:06.313584) v4 ==== 2004+0+0 (3609353185 0 0) 0x55d9bf805800 con 0x55d986bd3800 > -6921> 2018-07-11 07:51:06.314164 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6809/12650 conn(0x55d986bcb000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 233 0x55d9bf805800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:06.313584) v4 > -6920> 2018-07-11 07:51:06.314173 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6802/15255 conn(0x55d986bc8000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.0 seq 233 0x55d97f5adc00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:06.313584) v4 > -6919> 2018-07-11 07:51:06.314169 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6810/12651 conn(0x55d986b9b800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.2 seq 233 0x55d9869bfa00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:06.313584) v4 > -6918> 2018-07-11 07:51:06.314184 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.3 10.8.128.106:6809/12650 233 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:06.313584) v4 ==== 2004+0+0 (3609353185 0 0) 0x55d9bf805800 con 0x55d986bcb000 > -6917> 2018-07-11 07:51:06.314187 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.0 10.8.128.113:6802/15255 233 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:06.313584) v4 ==== 2004+0+0 (3609353185 0 0) 0x55d97f5adc00 con 0x55d986bc8000 > -6916> 2018-07-11 07:51:06.314199 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.2 10.8.128.106:6810/12651 233 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:06.313584) v4 ==== 2004+0+0 (3609353185 0 0) 0x55d9869bfa00 con 0x55d986b9b800 > -6915> 2018-07-11 07:51:06.314241 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6806/15263 conn(0x55d986b9d000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.5 seq 233 0x55d9bf805800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:06.313584) v4 > -6914> 2018-07-11 07:51:06.314259 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6808/12651 conn(0x55d986b9a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.2 seq 233 0x55d97f5adc00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:06.313584) v4 > -6913> 2018-07-11 07:51:06.314270 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.2 10.8.128.106:6808/12651 233 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:06.313584) v4 ==== 2004+0+0 (3609353185 0 0) 0x55d97f5adc00 con 0x55d986b9a000 > -6912> 2018-07-11 07:51:06.314270 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.5 10.8.128.113:6806/15263 233 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:06.313584) v4 ==== 2004+0+0 (3609353185 0 0) 0x55d9bf805800 con 0x55d986b9d000 > -6911> 2018-07-11 07:51:06.314278 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6803/15255 conn(0x55d986bc9800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.0 seq 233 0x55d9869bfa00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:06.313584) v4 > -6910> 2018-07-11 07:51:06.314298 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.0 10.8.128.113:6803/15255 233 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:06.313584) v4 ==== 2004+0+0 (3609353185 0 0) 0x55d9869bfa00 con 0x55d986bc9800 > -6909> 2018-07-11 07:51:06.314322 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6807/15263 conn(0x55d986b9e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.5 seq 233 0x55d97f5adc00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:06.313584) v4 > -6908> 2018-07-11 07:51:06.314325 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6806/12677 conn(0x55d986b58000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.1 seq 233 0x55d9bf805800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:06.313584) v4 > -6907> 2018-07-11 07:51:06.314336 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.5 10.8.128.113:6807/15263 233 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:06.313584) v4 ==== 2004+0+0 (3609353185 0 0) 0x55d97f5adc00 con 0x55d986b9e800 > -6906> 2018-07-11 07:51:06.314337 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.1 10.8.128.106:6806/12677 233 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:06.313584) v4 ==== 2004+0+0 (3609353185 0 0) 0x55d9bf805800 con 0x55d986b58000 > -6905> 2018-07-11 07:51:06.314369 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6811/12650 conn(0x55d986bcc800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 233 0x55d97f5adc00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:06.313584) v4 > -6904> 2018-07-11 07:51:06.314343 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.115:6802/13323 conn(0x55d986bc1000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 233 0x55d9869bfa00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:06.313584) v4 > -6903> 2018-07-11 07:51:06.314381 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.3 10.8.128.106:6811/12650 233 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:06.313584) v4 ==== 2004+0+0 (3609353185 0 0) 0x55d97f5adc00 con 0x55d986bcc800 > -6902> 2018-07-11 07:51:06.314379 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.115:6803/13323 conn(0x55d986bc2800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 233 0x55d9bf805800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:06.313584) v4 > -6901> 2018-07-11 07:51:06.314382 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.6 10.8.128.115:6802/13323 233 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:06.313584) v4 ==== 2004+0+0 (3609353185 0 0) 0x55d9869bfa00 con 0x55d986bc1000 > -6900> 2018-07-11 07:51:06.314389 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.6 10.8.128.115:6803/13323 233 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:06.313584) v4 ==== 2004+0+0 (3609353185 0 0) 0x55d9bf805800 con 0x55d986bc2800 > -6899> 2018-07-11 07:51:06.314414 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6807/12677 conn(0x55d986b59800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.1 seq 233 0x55d97f5adc00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:06.313584) v4 > -6898> 2018-07-11 07:51:06.314426 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.1 10.8.128.106:6807/12677 233 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:06.313584) v4 ==== 2004+0+0 (3609353185 0 0) 0x55d97f5adc00 con 0x55d986b59800 > -6897> 2018-07-11 07:51:06.314447 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6811/15287 conn(0x55d986bb0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 233 0x55d9869bfa00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:06.313584) v4 > -6896> 2018-07-11 07:51:06.314458 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.108:6803/11967 conn(0x55d986bbf800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.8 seq 233 0x55d97f5adc00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:06.313584) v4 > -6895> 2018-07-11 07:51:06.314461 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.108:6802/11967 conn(0x55d986bbe000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.8 seq 233 0x55d9bf805800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:06.313584) v4 > -6894> 2018-07-11 07:51:06.314465 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.4 10.8.128.113:6811/15287 233 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:06.313584) v4 ==== 2004+0+0 (3609353185 0 0) 0x55d9869bfa00 con 0x55d986bb0800 > -6893> 2018-07-11 07:51:06.314468 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.8 10.8.128.108:6803/11967 233 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:06.313584) v4 ==== 2004+0+0 (3609353185 0 0) 0x55d97f5adc00 con 0x55d986bbf800 > -6892> 2018-07-11 07:51:06.314478 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.8 10.8.128.108:6802/11967 233 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:06.313584) v4 ==== 2004+0+0 (3609353185 0 0) 0x55d9bf805800 con 0x55d986bbe000 > -6891> 2018-07-11 07:51:06.314506 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6810/15287 conn(0x55d986baf000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 233 0x55d97f5adc00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:06.313584) v4 > -6890> 2018-07-11 07:51:06.314511 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6802/15919 conn(0x55d986b5b000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.11 seq 233 0x55d9869bfa00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:06.313584) v4 > -6889> 2018-07-11 07:51:06.314519 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.4 10.8.128.113:6810/15287 233 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:06.313584) v4 ==== 2004+0+0 (3609353185 0 0) 0x55d97f5adc00 con 0x55d986baf000 > -6888> 2018-07-11 07:51:06.314517 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6803/15919 conn(0x55d986b5c800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.11 seq 233 0x55d9bf805800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:06.313584) v4 > -6887> 2018-07-11 07:51:06.314531 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.11 10.8.128.114:6802/15919 233 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:06.313584) v4 ==== 2004+0+0 (3609353185 0 0) 0x55d9869bfa00 con 0x55d986b5b000 > -6886> 2018-07-11 07:51:06.314538 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.11 10.8.128.114:6803/15919 233 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:06.313584) v4 ==== 2004+0+0 (3609353185 0 0) 0x55d9bf805800 con 0x55d986b5c800 > -6885> 2018-07-11 07:51:06.314579 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6806/15885 conn(0x55d986bac000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.7 seq 233 0x55d9869bfa00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:06.313584) v4 > -6884> 2018-07-11 07:51:06.314596 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.7 10.8.128.114:6806/15885 233 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:06.313584) v4 ==== 2004+0+0 (3609353185 0 0) 0x55d9869bfa00 con 0x55d986bac000 > -6883> 2018-07-11 07:51:06.314656 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6807/15885 conn(0x55d986bad800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.7 seq 233 0x55d9bf805800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:06.313584) v4 > -6882> 2018-07-11 07:51:06.314672 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.7 10.8.128.114:6807/15885 233 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:06.313584) v4 ==== 2004+0+0 (3609353185 0 0) 0x55d9bf805800 con 0x55d986bad800 > -6881> 2018-07-11 07:51:06.317395 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67098,epoch:501/498,start:5:753637da:::benchmark_data_magna115_10867_object1338554:0,end:5:75368d4b:::benchmark_data_magna115_10867_object3650728:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb773b80 con 0 > -6880> 2018-07-11 07:51:06.317431 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67098,epoch:501/498,start:5:753637da:::benchmark_data_magna115_10867_object1338554:0,end:5:75368d4b:::benchmark_data_magna115_10867_object3650728:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454000 con 0 > -6879> 2018-07-11 07:51:06.344980 7ffacbbd1700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15263 conn(0x55d985d44000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.5 seq 227 0x55d9bf805800 osd_ping(ping e501 stamp 2018-07-11 07:51:07.575593) v4 > -6878> 2018-07-11 07:51:06.345000 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 <== osd.5 10.8.128.113:0/15263 227 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:07.575593) v4 ==== 2004+0+0 (1390901780 0 0) 0x55d9bf805800 con 0x55d985d44000 > -6877> 2018-07-11 07:51:06.344989 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15263 conn(0x55d985d45800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.5 seq 227 0x55d9869bfa00 osd_ping(ping e501 stamp 2018-07-11 07:51:07.575593) v4 > -6876> 2018-07-11 07:51:06.345009 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:07.575593) v4 -- 0x55d9b3db3200 con 0 > -6875> 2018-07-11 07:51:06.345012 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.5 10.8.128.113:0/15263 227 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:07.575593) v4 ==== 2004+0+0 (1390901780 0 0) 0x55d9869bfa00 con 0x55d985d45800 > -6874> 2018-07-11 07:51:06.345030 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:07.575593) v4 -- 0x55d982d31e00 con 0 > -6873> 2018-07-11 07:51:06.363674 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15255 conn(0x55d986c82800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.0 seq 244 0x55d9bcd72c00 osd_ping(ping e501 stamp 2018-07-11 07:51:07.594265) v4 > -6872> 2018-07-11 07:51:06.363697 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.0 10.8.128.113:0/15255 244 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:07.594265) v4 ==== 2004+0+0 (2963745833 0 0) 0x55d9bcd72c00 con 0x55d986c82800 > -6871> 2018-07-11 07:51:06.363715 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:07.594265) v4 -- 0x55d982d31e00 con 0 > -6870> 2018-07-11 07:51:06.363740 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15255 conn(0x55d986d20000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.0 seq 244 0x55d97f5adc00 osd_ping(ping e501 stamp 2018-07-11 07:51:07.594265) v4 > -6869> 2018-07-11 07:51:06.363766 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.0 10.8.128.113:0/15255 244 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:07.594265) v4 ==== 2004+0+0 (2963745833 0 0) 0x55d97f5adc00 con 0x55d986d20000 > -6868> 2018-07-11 07:51:06.363777 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:07.594265) v4 -- 0x55d9c34cc000 con 0 > -6867> 2018-07-11 07:51:06.534174 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 872 0x55d990387b00 rep_scrubmap(5.2e e501 from shard 2) v1 > -6866> 2018-07-11 07:51:06.534194 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 872 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12474 (1269249638 0 811535721) 0x55d990387b00 con 0x55d986bf0800 > -6865> 2018-07-11 07:51:06.537481 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 866 0x55d990387d40 rep_scrubmap(5.2e e501 from shard 4) v1 > -6864> 2018-07-11 07:51:06.537500 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 866 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12474 (3686815630 0 811535721) 0x55d990387d40 con 0x55d986bec000 > -6863> 2018-07-11 07:51:06.943288 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.107:0/12617 conn(0x55d98123c000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 231 0x55d9b9ac3600 osd_ping(ping e501 stamp 2018-07-11 07:51:06.942938) v4 > -6862> 2018-07-11 07:51:06.943298 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.107:0/12617 conn(0x55d98123d800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 231 0x55d9bf805400 osd_ping(ping e501 stamp 2018-07-11 07:51:06.942938) v4 > -6861> 2018-07-11 07:51:06.943316 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.9 10.8.128.107:0/12617 231 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:06.942938) v4 ==== 2004+0+0 (4231097126 0 0) 0x55d9bf805400 con 0x55d98123d800 > -6860> 2018-07-11 07:51:06.943315 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.9 10.8.128.107:0/12617 231 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:06.942938) v4 ==== 2004+0+0 (4231097126 0 0) 0x55d9b9ac3600 con 0x55d98123c000 > -6859> 2018-07-11 07:51:06.943329 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:06.942938) v4 -- 0x55d9b3db3200 con 0 > -6858> 2018-07-11 07:51:06.943365 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:06.942938) v4 -- 0x55d9c34cc000 con 0 > -6857> 2018-07-11 07:51:06.944218 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 31] Generated table #36797: 1531281 keys, 31220681 bytes > -6856> 2018-07-11 07:51:06.944256 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295466944233, "cf_name": "default", "job": 31, "event": "table_file_creation", "file_number": 36797, "file_size": 31220681, "table_properties": {"data_size": 27052724, "index_size": 277619, "filter_size": 3889348, "raw_key_size": 44237012, "raw_average_key_size": 28, "raw_value_size": 6316345, "raw_average_value_size": 4, "num_data_blocks": 6643, "num_entries": 1531281, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1000000", "kMergeOperands": "0"}} > -6855> 2018-07-11 07:51:06.957456 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6805/15263 conn(0x55d986c41000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=31 cs=1 l=0). rx osd.5 seq 2798 0x55d97f5adc00 MOSDScrubReserve(6.15 REQUEST e501) v1 > -6854> 2018-07-11 07:51:06.957474 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.5 10.8.128.113:6805/15263 2798 ==== MOSDScrubReserve(6.15 REQUEST e501) v1 ==== 43+0+0 (3942742026 0 0) 0x55d97f5adc00 con 0x55d986c41000 > -6853> 2018-07-11 07:51:07.036256 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6805/15263 -- MOSDScrubReserve(6.15 REJECT e501) v1 -- 0x55d9829e4200 con 0 > -6852> 2018-07-11 07:51:07.036314 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -6851> 2018-07-11 07:51:07.036334 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -6850> 2018-07-11 07:51:07.036392 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -6849> 2018-07-11 07:51:07.036723 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6805/15263 conn(0x55d986c41000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=31 cs=1 l=0). rx osd.5 seq 2799 0x55d9829e4200 MOSDScrubReserve(6.15 RELEASE e501) v1 > -6848> 2018-07-11 07:51:07.036743 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.5 10.8.128.113:6805/15263 2799 ==== MOSDScrubReserve(6.15 RELEASE e501) v1 ==== 43+0+0 (605848072 0 0) 0x55d9829e4200 con 0x55d986c41000 > -6847> 2018-07-11 07:51:07.036851 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -6846> 2018-07-11 07:51:07.084446 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75368d4b:::benchmark_data_magna115_10867_object3650728:0,end:5:7536f305:::benchmark_data_magna115_10867_object2804500:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb773680 con 0 > -6845> 2018-07-11 07:51:07.084487 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75368d4b:::benchmark_data_magna115_10867_object3650728:0,end:5:7536f305:::benchmark_data_magna115_10867_object2804500:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454280 con 0 > -6844> 2018-07-11 07:51:07.294426 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 873 0x55d9c7f58000 rep_scrubmap(5.2e e501 from shard 2) v1 > -6843> 2018-07-11 07:51:07.294447 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 873 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12462 (1269249638 0 505228096) 0x55d9c7f58000 con 0x55d986bf0800 > -6842> 2018-07-11 07:51:07.313297 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 867 0x55d9c7f58240 rep_scrubmap(5.2e e501 from shard 4) v1 > -6841> 2018-07-11 07:51:07.313316 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 867 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12462 (3686815630 0 505228096) 0x55d9c7f58240 con 0x55d986bec000 > -6840> 2018-07-11 07:51:07.348889 7ffab6be6700 1 -- 10.8.128.107:6800/13927 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55d9ba81ab00 con 0 > -6839> 2018-07-11 07:51:07.348963 7ffab6be6700 1 -- 10.8.128.107:6800/13927 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55d9bd9cdb80 con 0 > -6838> 2018-07-11 07:51:07.389049 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -6837> 2018-07-11 07:51:07.389073 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -6836> 2018-07-11 07:51:07.389131 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -6835> 2018-07-11 07:51:07.389629 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -6834> 2018-07-11 07:51:07.414018 7ffab1bdc700 5 osd.10 501 heartbeat: osd_stat(788 GB used, 143 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist []) > -6833> 2018-07-11 07:51:07.414030 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6802/15255 -- osd_ping(ping e501 stamp 2018-07-11 07:51:07.414028) v4 -- 0x55d9b60df400 con 0 > -6832> 2018-07-11 07:51:07.414052 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6803/15255 -- osd_ping(ping e501 stamp 2018-07-11 07:51:07.414028) v4 -- 0x55d986d41e00 con 0 > -6831> 2018-07-11 07:51:07.414077 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6806/12677 -- osd_ping(ping e501 stamp 2018-07-11 07:51:07.414028) v4 -- 0x55d9b40a5800 con 0 > -6830> 2018-07-11 07:51:07.414091 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6807/12677 -- osd_ping(ping e501 stamp 2018-07-11 07:51:07.414028) v4 -- 0x55d980f5a800 con 0 > -6829> 2018-07-11 07:51:07.414099 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6808/12651 -- osd_ping(ping e501 stamp 2018-07-11 07:51:07.414028) v4 -- 0x55d9b40a4000 con 0 > -6828> 2018-07-11 07:51:07.414106 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6810/12651 -- osd_ping(ping e501 stamp 2018-07-11 07:51:07.414028) v4 -- 0x55d986d40600 con 0 > -6827> 2018-07-11 07:51:07.414113 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6809/12650 -- osd_ping(ping e501 stamp 2018-07-11 07:51:07.414028) v4 -- 0x55d9829e4a00 con 0 > -6826> 2018-07-11 07:51:07.414135 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6811/12650 -- osd_ping(ping e501 stamp 2018-07-11 07:51:07.414028) v4 -- 0x55d986d33200 con 0 > -6825> 2018-07-11 07:51:07.414143 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6810/15287 -- osd_ping(ping e501 stamp 2018-07-11 07:51:07.414028) v4 -- 0x55d9b437ac00 con 0 > -6824> 2018-07-11 07:51:07.414151 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6811/15287 -- osd_ping(ping e501 stamp 2018-07-11 07:51:07.414028) v4 -- 0x55d9b60df200 con 0 > -6823> 2018-07-11 07:51:07.414163 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6806/15263 -- osd_ping(ping e501 stamp 2018-07-11 07:51:07.414028) v4 -- 0x55d986c61600 con 0 > -6822> 2018-07-11 07:51:07.414175 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6807/15263 -- osd_ping(ping e501 stamp 2018-07-11 07:51:07.414028) v4 -- 0x55d9b2528000 con 0 > -6821> 2018-07-11 07:51:07.414193 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.115:6802/13323 -- osd_ping(ping e501 stamp 2018-07-11 07:51:07.414028) v4 -- 0x55d9bdba5600 con 0 > -6820> 2018-07-11 07:51:07.414204 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.115:6803/13323 -- osd_ping(ping e501 stamp 2018-07-11 07:51:07.414028) v4 -- 0x55d986d96200 con 0 > -6819> 2018-07-11 07:51:07.414214 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6806/15885 -- osd_ping(ping e501 stamp 2018-07-11 07:51:07.414028) v4 -- 0x55d986c22000 con 0 > -6818> 2018-07-11 07:51:07.414226 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6807/15885 -- osd_ping(ping e501 stamp 2018-07-11 07:51:07.414028) v4 -- 0x55d9829e5e00 con 0 > -6817> 2018-07-11 07:51:07.414238 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.108:6802/11967 -- osd_ping(ping e501 stamp 2018-07-11 07:51:07.414028) v4 -- 0x55d986b6dc00 con 0 > -6816> 2018-07-11 07:51:07.414269 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.108:6803/11967 -- osd_ping(ping e501 stamp 2018-07-11 07:51:07.414028) v4 -- 0x55d986cd5200 con 0 > -6815> 2018-07-11 07:51:07.414283 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.107:6806/12617 -- osd_ping(ping e501 stamp 2018-07-11 07:51:07.414028) v4 -- 0x55d9bfbde400 con 0 > -6814> 2018-07-11 07:51:07.414297 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.107:6807/12617 -- osd_ping(ping e501 stamp 2018-07-11 07:51:07.414028) v4 -- 0x55d9829e5000 con 0 > -6813> 2018-07-11 07:51:07.414309 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6802/15919 -- osd_ping(ping e501 stamp 2018-07-11 07:51:07.414028) v4 -- 0x55d9c4c1be00 con 0 > -6812> 2018-07-11 07:51:07.414321 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6803/15919 -- osd_ping(ping e501 stamp 2018-07-11 07:51:07.414028) v4 -- 0x55d9bf804800 con 0 > -6811> 2018-07-11 07:51:07.414470 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.107:6806/12617 conn(0x55d986bd2000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.9 seq 234 0x55d9bcd72c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:07.414028) v4 > -6810> 2018-07-11 07:51:07.414492 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.9 10.8.128.107:6806/12617 234 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:07.414028) v4 ==== 2004+0+0 (1008588533 0 0) 0x55d9bcd72c00 con 0x55d986bd2000 > -6809> 2018-07-11 07:51:07.414517 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.107:6807/12617 conn(0x55d986bd3800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.9 seq 234 0x55d9bf805800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:07.414028) v4 > -6808> 2018-07-11 07:51:07.414553 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.9 10.8.128.107:6807/12617 234 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:07.414028) v4 ==== 2004+0+0 (1008588533 0 0) 0x55d9bf805800 con 0x55d986bd3800 > -6807> 2018-07-11 07:51:07.414666 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6803/15255 conn(0x55d986bc9800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.0 seq 234 0x55d9bcd72c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:07.414028) v4 > -6806> 2018-07-11 07:51:07.414663 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6806/15263 conn(0x55d986b9d000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.5 seq 234 0x55d9bf805800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:07.414028) v4 > -6805> 2018-07-11 07:51:07.414681 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.0 10.8.128.113:6803/15255 234 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:07.414028) v4 ==== 2004+0+0 (1008588533 0 0) 0x55d9bcd72c00 con 0x55d986bc9800 > -6804> 2018-07-11 07:51:07.414686 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.5 10.8.128.113:6806/15263 234 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:07.414028) v4 ==== 2004+0+0 (1008588533 0 0) 0x55d9bf805800 con 0x55d986b9d000 > -6803> 2018-07-11 07:51:07.414682 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6802/15255 conn(0x55d986bc8000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.0 seq 234 0x55d9b9ac3600 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:07.414028) v4 > -6802> 2018-07-11 07:51:07.414699 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.0 10.8.128.113:6802/15255 234 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:07.414028) v4 ==== 2004+0+0 (1008588533 0 0) 0x55d9b9ac3600 con 0x55d986bc8000 > -6801> 2018-07-11 07:51:07.414756 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6811/15287 conn(0x55d986bb0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 234 0x55d9bcd72c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:07.414028) v4 > -6800> 2018-07-11 07:51:07.414771 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.4 10.8.128.113:6811/15287 234 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:07.414028) v4 ==== 2004+0+0 (1008588533 0 0) 0x55d9bcd72c00 con 0x55d986bb0800 > -6799> 2018-07-11 07:51:07.414778 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6808/12651 conn(0x55d986b9a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.2 seq 234 0x55d9b9ac3600 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:07.414028) v4 > -6798> 2018-07-11 07:51:07.414777 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6809/12650 conn(0x55d986bcb000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 234 0x55d9bf805800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:07.414028) v4 > -6797> 2018-07-11 07:51:07.414794 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.2 10.8.128.106:6808/12651 234 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:07.414028) v4 ==== 2004+0+0 (1008588533 0 0) 0x55d9b9ac3600 con 0x55d986b9a000 > -6796> 2018-07-11 07:51:07.414801 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.3 10.8.128.106:6809/12650 234 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:07.414028) v4 ==== 2004+0+0 (1008588533 0 0) 0x55d9bf805800 con 0x55d986bcb000 > -6795> 2018-07-11 07:51:07.414826 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6806/12677 conn(0x55d986b58000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.1 seq 234 0x55d9bf805800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:07.414028) v4 > -6794> 2018-07-11 07:51:07.414837 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.1 10.8.128.106:6806/12677 234 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:07.414028) v4 ==== 2004+0+0 (1008588533 0 0) 0x55d9bf805800 con 0x55d986b58000 > -6793> 2018-07-11 07:51:07.414803 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6810/12651 conn(0x55d986b9b800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.2 seq 234 0x55d9bcd72c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:07.414028) v4 > -6792> 2018-07-11 07:51:07.414845 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6810/15287 conn(0x55d986baf000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 234 0x55d9b9ac3600 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:07.414028) v4 > -6791> 2018-07-11 07:51:07.414848 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.2 10.8.128.106:6810/12651 234 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:07.414028) v4 ==== 2004+0+0 (1008588533 0 0) 0x55d9bcd72c00 con 0x55d986b9b800 > -6790> 2018-07-11 07:51:07.414862 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.4 10.8.128.113:6810/15287 234 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:07.414028) v4 ==== 2004+0+0 (1008588533 0 0) 0x55d9b9ac3600 con 0x55d986baf000 > -6789> 2018-07-11 07:51:07.414875 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.115:6803/13323 conn(0x55d986bc2800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 234 0x55d9bf805800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:07.414028) v4 > -6788> 2018-07-11 07:51:07.414891 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.6 10.8.128.115:6803/13323 234 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:07.414028) v4 ==== 2004+0+0 (1008588533 0 0) 0x55d9bf805800 con 0x55d986bc2800 > -6787> 2018-07-11 07:51:07.414893 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6811/12650 conn(0x55d986bcc800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 234 0x55d9b9ac3600 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:07.414028) v4 > -6786> 2018-07-11 07:51:07.414903 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.3 10.8.128.106:6811/12650 234 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:07.414028) v4 ==== 2004+0+0 (1008588533 0 0) 0x55d9b9ac3600 con 0x55d986bcc800 > -6785> 2018-07-11 07:51:07.414897 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6806/15885 conn(0x55d986bac000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.7 seq 234 0x55d9bcd72c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:07.414028) v4 > -6784> 2018-07-11 07:51:07.414929 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6807/15885 conn(0x55d986bad800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.7 seq 234 0x55d9bf805800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:07.414028) v4 > -6783> 2018-07-11 07:51:07.414932 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.7 10.8.128.114:6806/15885 234 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:07.414028) v4 ==== 2004+0+0 (1008588533 0 0) 0x55d9bcd72c00 con 0x55d986bac000 > -6782> 2018-07-11 07:51:07.414936 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6807/12677 conn(0x55d986b59800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.1 seq 234 0x55d9b9ac3600 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:07.414028) v4 > -6781> 2018-07-11 07:51:07.414942 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.7 10.8.128.114:6807/15885 234 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:07.414028) v4 ==== 2004+0+0 (1008588533 0 0) 0x55d9bf805800 con 0x55d986bad800 > -6780> 2018-07-11 07:51:07.414946 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.1 10.8.128.106:6807/12677 234 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:07.414028) v4 ==== 2004+0+0 (1008588533 0 0) 0x55d9b9ac3600 con 0x55d986b59800 > -6779> 2018-07-11 07:51:07.414982 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6803/15919 conn(0x55d986b5c800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.11 seq 234 0x55d9bf805800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:07.414028) v4 > -6778> 2018-07-11 07:51:07.414982 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.115:6802/13323 conn(0x55d986bc1000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 234 0x55d9bcd72c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:07.414028) v4 > -6777> 2018-07-11 07:51:07.414991 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6807/15263 conn(0x55d986b9e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.5 seq 234 0x55d9b9ac3600 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:07.414028) v4 > -6776> 2018-07-11 07:51:07.414997 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.11 10.8.128.114:6803/15919 234 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:07.414028) v4 ==== 2004+0+0 (1008588533 0 0) 0x55d9bf805800 con 0x55d986b5c800 > -6775> 2018-07-11 07:51:07.415001 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.5 10.8.128.113:6807/15263 234 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:07.414028) v4 ==== 2004+0+0 (1008588533 0 0) 0x55d9b9ac3600 con 0x55d986b9e800 > -6774> 2018-07-11 07:51:07.415001 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.6 10.8.128.115:6802/13323 234 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:07.414028) v4 ==== 2004+0+0 (1008588533 0 0) 0x55d9bcd72c00 con 0x55d986bc1000 > -6773> 2018-07-11 07:51:07.415029 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.108:6803/11967 conn(0x55d986bbf800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.8 seq 234 0x55d9b9ac3600 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:07.414028) v4 > -6772> 2018-07-11 07:51:07.415030 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.108:6802/11967 conn(0x55d986bbe000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.8 seq 234 0x55d9bf805800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:07.414028) v4 > -6771> 2018-07-11 07:51:07.415041 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.8 10.8.128.108:6803/11967 234 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:07.414028) v4 ==== 2004+0+0 (1008588533 0 0) 0x55d9b9ac3600 con 0x55d986bbf800 > -6770> 2018-07-11 07:51:07.415044 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.8 10.8.128.108:6802/11967 234 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:07.414028) v4 ==== 2004+0+0 (1008588533 0 0) 0x55d9bf805800 con 0x55d986bbe000 > -6769> 2018-07-11 07:51:07.415042 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6802/15919 conn(0x55d986b5b000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.11 seq 234 0x55d9bcd72c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:07.414028) v4 > -6768> 2018-07-11 07:51:07.415059 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.11 10.8.128.114:6802/15919 234 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:07.414028) v4 ==== 2004+0+0 (1008588533 0 0) 0x55d9bcd72c00 con 0x55d986b5b000 > -6767> 2018-07-11 07:51:07.434506 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7536f305:::benchmark_data_magna115_10867_object2804500:0,end:5:75372ffe:::benchmark_data_magna115_10867_object1288292:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb773180 con 0 > -6766> 2018-07-11 07:51:07.434542 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7536f305:::benchmark_data_magna115_10867_object2804500:0,end:5:75372ffe:::benchmark_data_magna115_10867_object1288292:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb773400 con 0 > -6765> 2018-07-11 07:51:07.655847 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 874 0x55d9c7f58480 rep_scrubmap(5.2e e501 from shard 2) v1 > -6764> 2018-07-11 07:51:07.655866 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 874 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12484 (1269249638 0 3639588181) 0x55d9c7f58480 con 0x55d986bf0800 > -6763> 2018-07-11 07:51:07.679512 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 868 0x55d9c7f586c0 rep_scrubmap(5.2e e501 from shard 4) v1 > -6762> 2018-07-11 07:51:07.679530 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 868 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12484 (3686815630 0 3639588181) 0x55d9c7f586c0 con 0x55d986bec000 > -6761> 2018-07-11 07:51:07.679687 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -6760> 2018-07-11 07:51:07.679712 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -6759> 2018-07-11 07:51:07.679760 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -6758> 2018-07-11 07:51:07.680214 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -6757> 2018-07-11 07:51:07.683226 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75372ffe:::benchmark_data_magna115_10867_object1288292:0,end:5:753766a6:::benchmark_data_magna115_10867_object943342:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb773900 con 0 > -6756> 2018-07-11 07:51:07.683272 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75372ffe:::benchmark_data_magna115_10867_object1288292:0,end:5:753766a6:::benchmark_data_magna115_10867_object943342:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb772f00 con 0 > -6755> 2018-07-11 07:51:07.889607 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 875 0x55d9c7f58900 rep_scrubmap(5.2e e501 from shard 2) v1 > -6754> 2018-07-11 07:51:07.889626 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 875 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12466 (1269249638 0 1206073135) 0x55d9c7f58900 con 0x55d986bf0800 > -6753> 2018-07-11 07:51:07.911949 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 869 0x55d9c7f58b40 rep_scrubmap(5.2e e501 from shard 4) v1 > -6752> 2018-07-11 07:51:07.911969 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 869 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12466 (3686815630 0 1206073135) 0x55d9c7f58b40 con 0x55d986bec000 > -6751> 2018-07-11 07:51:07.957718 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6805/15263 conn(0x55d986c41000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=31 cs=1 l=0). rx osd.5 seq 2800 0x55d9c34cc000 MOSDScrubReserve(6.15 REQUEST e501) v1 > -6750> 2018-07-11 07:51:07.957737 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.5 10.8.128.113:6805/15263 2800 ==== MOSDScrubReserve(6.15 REQUEST e501) v1 ==== 43+0+0 (3942742026 0 0) 0x55d9c34cc000 con 0x55d986c41000 > -6749> 2018-07-11 07:51:08.053524 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6805/15263 -- MOSDScrubReserve(6.15 REJECT e501) v1 -- 0x55d9829e4200 con 0 > -6748> 2018-07-11 07:51:08.053581 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -6747> 2018-07-11 07:51:08.053612 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -6746> 2018-07-11 07:51:08.053656 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -6745> 2018-07-11 07:51:08.054068 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6805/15263 conn(0x55d986c41000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=31 cs=1 l=0). rx osd.5 seq 2801 0x55d9829e4200 MOSDScrubReserve(6.15 RELEASE e501) v1 > -6744> 2018-07-11 07:51:08.054087 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.5 10.8.128.113:6805/15263 2801 ==== MOSDScrubReserve(6.15 RELEASE e501) v1 ==== 43+0+0 (605848072 0 0) 0x55d9829e4200 con 0x55d986c41000 > -6743> 2018-07-11 07:51:08.054115 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -6742> 2018-07-11 07:51:08.076442 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:753766a6:::benchmark_data_magna115_10867_object943342:0,end:5:7537bad3:::benchmark_data_magna115_10867_object3208929:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb772c80 con 0 > -6741> 2018-07-11 07:51:08.076479 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:753766a6:::benchmark_data_magna115_10867_object943342:0,end:5:7537bad3:::benchmark_data_magna115_10867_object3208929:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb772780 con 0 > -6740> 2018-07-11 07:51:08.280534 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15287 conn(0x55d981240800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 255 0x55d986cd5200 osd_ping(ping e501 stamp 2018-07-11 07:51:09.511078) v4 > -6739> 2018-07-11 07:51:08.280560 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.4 10.8.128.113:0/15287 255 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:09.511078) v4 ==== 2004+0+0 (2708988 0 0) 0x55d986cd5200 con 0x55d981240800 > -6738> 2018-07-11 07:51:08.280571 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:09.511078) v4 -- 0x55d9b2528000 con 0 > -6737> 2018-07-11 07:51:08.280588 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15287 conn(0x55d98123f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 255 0x55d9bcd72c00 osd_ping(ping e501 stamp 2018-07-11 07:51:09.511078) v4 > -6736> 2018-07-11 07:51:08.280609 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.4 10.8.128.113:0/15287 255 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:09.511078) v4 ==== 2004+0+0 (2708988 0 0) 0x55d9bcd72c00 con 0x55d98123f000 > -6735> 2018-07-11 07:51:08.280626 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:09.511078) v4 -- 0x55d9c4c1be00 con 0 > -6734> 2018-07-11 07:51:08.305500 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 876 0x55d9c7f58d80 rep_scrubmap(5.2e e501 from shard 2) v1 > -6733> 2018-07-11 07:51:08.305521 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 876 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12474 (1269249638 0 2441515249) 0x55d9c7f58d80 con 0x55d986bf0800 > -6732> 2018-07-11 07:51:08.321987 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 870 0x55d9c7f58fc0 rep_scrubmap(5.2e e501 from shard 4) v1 > -6731> 2018-07-11 07:51:08.322007 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 870 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12474 (3686815630 0 2441515249) 0x55d9c7f58fc0 con 0x55d986bec000 > -6730> 2018-07-11 07:51:08.343534 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -6729> 2018-07-11 07:51:08.343559 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -6728> 2018-07-11 07:51:08.343616 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -6727> 2018-07-11 07:51:08.344077 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -6726> 2018-07-11 07:51:08.344952 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'66671,epoch:501/498,start:5:7537bad3:::benchmark_data_magna115_10867_object3208929:0,end:5:75381d77:::benchmark_data_magna115_10867_object1269125:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb772a00 con 0 > -6725> 2018-07-11 07:51:08.344998 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'66671,epoch:501/498,start:5:7537bad3:::benchmark_data_magna115_10867_object3208929:0,end:5:75381d77:::benchmark_data_magna115_10867_object1269125:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb772500 con 0 > -6724> 2018-07-11 07:51:08.412456 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12650 conn(0x55d986d21800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 236 0x55d9bf805800 osd_ping(ping e501 stamp 2018-07-11 07:51:08.459812) v4 > -6723> 2018-07-11 07:51:08.412456 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12650 conn(0x55d986d23000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 236 0x55d9869bfa00 osd_ping(ping e501 stamp 2018-07-11 07:51:08.459812) v4 > -6722> 2018-07-11 07:51:08.412480 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.3 10.8.128.106:0/12650 236 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:08.459812) v4 ==== 2004+0+0 (1627833729 0 0) 0x55d9bf805800 con 0x55d986d21800 > -6721> 2018-07-11 07:51:08.412485 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.3 10.8.128.106:0/12650 236 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:08.459812) v4 ==== 2004+0+0 (1627833729 0 0) 0x55d9869bfa00 con 0x55d986d23000 > -6720> 2018-07-11 07:51:08.412493 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:08.459812) v4 -- 0x55d9bf804800 con 0 > -6719> 2018-07-11 07:51:08.412524 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:08.459812) v4 -- 0x55d9c4c1be00 con 0 > -6718> 2018-07-11 07:51:08.544495 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 877 0x55d9c7f59200 rep_scrubmap(5.2e e501 from shard 2) v1 > -6717> 2018-07-11 07:51:08.544515 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 877 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12480 (1269249638 0 3190402002) 0x55d9c7f59200 con 0x55d986bf0800 > -6716> 2018-07-11 07:51:08.560492 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6805/15885 conn(0x55d986c7f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.7 seq 2784 0x55d9b9ac3600 MOSDScrubReserve(5.4 REQUEST e501) v1 > -6715> 2018-07-11 07:51:08.560510 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.7 10.8.128.114:6805/15885 2784 ==== MOSDScrubReserve(5.4 REQUEST e501) v1 ==== 43+0+0 (1753221512 0 0) 0x55d9b9ac3600 con 0x55d986c7f800 > -6714> 2018-07-11 07:51:08.560582 7ffab33df700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.114:6805/15885 -- MOSDScrubReserve(5.4 REJECT e501) v1 -- 0x55d986d32200 con 0 > -6713> 2018-07-11 07:51:08.560922 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6805/15885 conn(0x55d986c7f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.7 seq 2785 0x55d986d32200 MOSDScrubReserve(5.4 RELEASE e501) v1 > -6712> 2018-07-11 07:51:08.560937 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.7 10.8.128.114:6805/15885 2785 ==== MOSDScrubReserve(5.4 RELEASE e501) v1 ==== 43+0+0 (2812146570 0 0) 0x55d986d32200 con 0x55d986c7f800 > -6711> 2018-07-11 07:51:08.571421 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 871 0x55d9c7f59440 rep_scrubmap(5.2e e501 from shard 4) v1 > -6710> 2018-07-11 07:51:08.571440 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 871 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12480 (3686815630 0 3190402002) 0x55d9c7f59440 con 0x55d986bec000 > -6709> 2018-07-11 07:51:08.626056 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -6708> 2018-07-11 07:51:08.626080 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -6707> 2018-07-11 07:51:08.626139 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -6706> 2018-07-11 07:51:08.626633 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -6705> 2018-07-11 07:51:08.676647 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67014,epoch:501/498,start:5:75381d77:::benchmark_data_magna115_10867_object1269125:0,end:5:753874bc:::benchmark_data_magna115_10867_object2313165:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb772000 con 0 > -6704> 2018-07-11 07:51:08.676697 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67014,epoch:501/498,start:5:75381d77:::benchmark_data_magna115_10867_object1269125:0,end:5:753874bc:::benchmark_data_magna115_10867_object2313165:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98944b400 con 0 > -6703> 2018-07-11 07:51:08.898876 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 878 0x55d9c7f59680 rep_scrubmap(5.2e e501 from shard 2) v1 > -6702> 2018-07-11 07:51:08.898895 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 878 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12470 (1269249638 0 2592219235) 0x55d9c7f59680 con 0x55d986bf0800 > -6701> 2018-07-11 07:51:08.917074 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 872 0x55d9c7f598c0 rep_scrubmap(5.2e e501 from shard 4) v1 > -6700> 2018-07-11 07:51:08.917094 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 872 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12470 (3686815630 0 2592219235) 0x55d9c7f598c0 con 0x55d986bec000 > -6699> 2018-07-11 07:51:08.957890 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6805/15263 conn(0x55d986c41000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=31 cs=1 l=0). rx osd.5 seq 2802 0x55d986cd5200 MOSDScrubReserve(6.15 REQUEST e501) v1 > -6698> 2018-07-11 07:51:08.957908 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.5 10.8.128.113:6805/15263 2802 ==== MOSDScrubReserve(6.15 REQUEST e501) v1 ==== 43+0+0 (3942742026 0 0) 0x55d986cd5200 con 0x55d986c41000 > -6697> 2018-07-11 07:51:08.966079 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6805/15263 -- MOSDScrubReserve(6.15 REJECT e501) v1 -- 0x55d9829e4200 con 0 > -6696> 2018-07-11 07:51:08.966136 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -6695> 2018-07-11 07:51:08.966166 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -6694> 2018-07-11 07:51:08.966213 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -6693> 2018-07-11 07:51:08.966581 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6805/15263 conn(0x55d986c41000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=31 cs=1 l=0). rx osd.5 seq 2803 0x55d9829e4200 MOSDScrubReserve(6.15 RELEASE e501) v1 > -6692> 2018-07-11 07:51:08.966599 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.5 10.8.128.113:6805/15263 2803 ==== MOSDScrubReserve(6.15 RELEASE e501) v1 ==== 43+0+0 (605848072 0 0) 0x55d9829e4200 con 0x55d986c41000 > -6691> 2018-07-11 07:51:08.966687 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -6690> 2018-07-11 07:51:08.967586 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:753874bc:::benchmark_data_magna115_10867_object2313165:0,end:5:75390645:::benchmark_data_magna115_10867_object798170:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb772280 con 0 > -6689> 2018-07-11 07:51:08.967632 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:753874bc:::benchmark_data_magna115_10867_object2313165:0,end:5:75390645:::benchmark_data_magna115_10867_object798170:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98944bb80 con 0 > -6688> 2018-07-11 07:51:09.183532 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 879 0x55d9c7f59b00 rep_scrubmap(5.2e e501 from shard 2) v1 > -6687> 2018-07-11 07:51:09.183564 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 879 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12476 (1269249638 0 3159908434) 0x55d9c7f59b00 con 0x55d986bf0800 > -6686> 2018-07-11 07:51:09.185859 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 873 0x55d9c7f59d40 rep_scrubmap(5.2e e501 from shard 4) v1 > -6685> 2018-07-11 07:51:09.185878 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 873 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12476 (3686815630 0 3159908434) 0x55d9c7f59d40 con 0x55d986bec000 > -6684> 2018-07-11 07:51:09.215540 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -6683> 2018-07-11 07:51:09.215578 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -6682> 2018-07-11 07:51:09.215619 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -6681> 2018-07-11 07:51:09.216062 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -6680> 2018-07-11 07:51:09.245404 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15263 conn(0x55d985d45800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.5 seq 228 0x55d9bcd72c00 osd_ping(ping e501 stamp 2018-07-11 07:51:10.476022) v4 > -6679> 2018-07-11 07:51:09.245405 7ffacbbd1700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15263 conn(0x55d985d44000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.5 seq 228 0x55d9bf805400 osd_ping(ping e501 stamp 2018-07-11 07:51:10.476022) v4 > -6678> 2018-07-11 07:51:09.245435 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 <== osd.5 10.8.128.113:0/15263 228 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:10.476022) v4 ==== 2004+0+0 (2786023789 0 0) 0x55d9bf805400 con 0x55d985d44000 > -6677> 2018-07-11 07:51:09.245431 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.5 10.8.128.113:0/15263 228 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:10.476022) v4 ==== 2004+0+0 (2786023789 0 0) 0x55d9bcd72c00 con 0x55d985d45800 > -6676> 2018-07-11 07:51:09.245452 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:10.476022) v4 -- 0x55d9bf804800 con 0 > -6675> 2018-07-11 07:51:09.245477 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:10.476022) v4 -- 0x55d9c4c1be00 con 0 > -6674> 2018-07-11 07:51:09.260160 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67756,epoch:501/498,start:5:75390645:::benchmark_data_magna115_10867_object798170:0,end:5:753954fd:::benchmark_data_magna115_11707_object259728:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98944b900 con 0 > -6673> 2018-07-11 07:51:09.260197 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67756,epoch:501/498,start:5:75390645:::benchmark_data_magna115_10867_object798170:0,end:5:753954fd:::benchmark_data_magna115_11707_object259728:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454a00 con 0 > -6672> 2018-07-11 07:51:09.463504 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 880 0x55d9c5d3c000 rep_scrubmap(5.2e e501 from shard 2) v1 > -6671> 2018-07-11 07:51:09.463525 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 880 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12476 (1269249638 0 568705206) 0x55d9c5d3c000 con 0x55d986bf0800 > -6670> 2018-07-11 07:51:09.500119 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 874 0x55d9c5d3c240 rep_scrubmap(5.2e e501 from shard 4) v1 > -6669> 2018-07-11 07:51:09.500139 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 874 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12476 (3686815630 0 568705206) 0x55d9c5d3c240 con 0x55d986bec000 > -6668> 2018-07-11 07:51:09.556947 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -6667> 2018-07-11 07:51:09.556971 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -6666> 2018-07-11 07:51:09.557026 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -6665> 2018-07-11 07:51:09.557473 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -6664> 2018-07-11 07:51:09.586310 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:753954fd:::benchmark_data_magna115_11707_object259728:0,end:5:7539a2f4:::benchmark_data_magna115_10867_object1116057:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98944b680 con 0 > -6663> 2018-07-11 07:51:09.586350 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:753954fd:::benchmark_data_magna115_11707_object259728:0,end:5:7539a2f4:::benchmark_data_magna115_10867_object1116057:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454500 con 0 > -6662> 2018-07-11 07:51:09.789417 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.114:0/15919 conn(0x55d986df3800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.11 seq 245 0x55d9bf805800 osd_ping(ping e501 stamp 2018-07-11 07:51:10.419518) v4 > -6661> 2018-07-11 07:51:09.789443 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.11 10.8.128.114:0/15919 245 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:10.419518) v4 ==== 2004+0+0 (2236940614 0 0) 0x55d9bf805800 con 0x55d986df3800 > -6660> 2018-07-11 07:51:09.789462 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:10.419518) v4 -- 0x55d9bf804800 con 0 > -6659> 2018-07-11 07:51:09.789483 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.114:0/15919 conn(0x55d986df5000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.11 seq 245 0x55d9869bfa00 osd_ping(ping e501 stamp 2018-07-11 07:51:10.419518) v4 > -6658> 2018-07-11 07:51:09.789506 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.11 10.8.128.114:0/15919 245 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:10.419518) v4 ==== 2004+0+0 (2236940614 0 0) 0x55d9869bfa00 con 0x55d986df5000 > -6657> 2018-07-11 07:51:09.789524 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:10.419518) v4 -- 0x55d9c4c1be00 con 0 > -6656> 2018-07-11 07:51:09.803273 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 881 0x55d9c5d3c480 rep_scrubmap(5.2e e501 from shard 2) v1 > -6655> 2018-07-11 07:51:09.803294 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 881 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12468 (1269249638 0 3702450342) 0x55d9c5d3c480 con 0x55d986bf0800 > -6654> 2018-07-11 07:51:09.803361 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 875 0x55d9c5d3c6c0 rep_scrubmap(5.2e e501 from shard 4) v1 > -6653> 2018-07-11 07:51:09.803375 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 875 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12468 (3686815630 0 3702450342) 0x55d9c5d3c6c0 con 0x55d986bec000 > -6652> 2018-07-11 07:51:09.843081 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -6651> 2018-07-11 07:51:09.843107 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -6650> 2018-07-11 07:51:09.843163 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -6649> 2018-07-11 07:51:09.843641 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -6648> 2018-07-11 07:51:09.860395 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7539a2f4:::benchmark_data_magna115_10867_object1116057:0,end:5:7539f63a:::benchmark_data_magna115_10867_object2117035:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455180 con 0 > -6647> 2018-07-11 07:51:09.860431 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7539a2f4:::benchmark_data_magna115_10867_object1116057:0,end:5:7539f63a:::benchmark_data_magna115_10867_object2117035:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454780 con 0 > -6646> 2018-07-11 07:51:09.958133 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6805/15263 conn(0x55d986c41000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=31 cs=1 l=0). rx osd.5 seq 2804 0x55d9b437ac00 MOSDScrubReserve(6.15 REQUEST e501) v1 > -6645> 2018-07-11 07:51:09.958150 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.5 10.8.128.113:6805/15263 2804 ==== MOSDScrubReserve(6.15 REQUEST e501) v1 ==== 43+0+0 (3942742026 0 0) 0x55d9b437ac00 con 0x55d986c41000 > -6644> 2018-07-11 07:51:10.081815 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 882 0x55d9c5d3c900 rep_scrubmap(5.2e e501 from shard 2) v1 > -6643> 2018-07-11 07:51:10.081835 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 882 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12476 (1269249638 0 626381869) 0x55d9c5d3c900 con 0x55d986bf0800 > -6642> 2018-07-11 07:51:10.100590 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 876 0x55d9c5d3cb40 rep_scrubmap(5.2e e501 from shard 4) v1 > -6641> 2018-07-11 07:51:10.100610 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 876 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12476 (3686815630 0 626381869) 0x55d9c5d3cb40 con 0x55d986bec000 > -6640> 2018-07-11 07:51:10.148375 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6805/15263 -- MOSDScrubReserve(6.15 REJECT e501) v1 -- 0x55d9829e4200 con 0 > -6639> 2018-07-11 07:51:10.148572 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -6638> 2018-07-11 07:51:10.148594 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -6637> 2018-07-11 07:51:10.148641 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -6636> 2018-07-11 07:51:10.148912 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6805/15263 conn(0x55d986c41000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=31 cs=1 l=0). rx osd.5 seq 2805 0x55d9829e4200 MOSDScrubReserve(6.15 RELEASE e501) v1 > -6635> 2018-07-11 07:51:10.148932 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.5 10.8.128.113:6805/15263 2805 ==== MOSDScrubReserve(6.15 RELEASE e501) v1 ==== 43+0+0 (605848072 0 0) 0x55d9829e4200 con 0x55d986c41000 > -6634> 2018-07-11 07:51:10.149096 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -6633> 2018-07-11 07:51:10.182768 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7539f63a:::benchmark_data_magna115_10867_object2117035:0,end:5:753a5661:::benchmark_data_magna115_10867_object2753367:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454f00 con 0 > -6632> 2018-07-11 07:51:10.182819 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7539f63a:::benchmark_data_magna115_10867_object2117035:0,end:5:753a5661:::benchmark_data_magna115_10867_object2753367:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498280 con 0 > -6631> 2018-07-11 07:51:10.377931 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 877 0x55d9c5d3cd80 rep_scrubmap(5.2e e501 from shard 4) v1 > -6630> 2018-07-11 07:51:10.377950 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 877 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12482 (3686815630 0 93539259) 0x55d9c5d3cd80 con 0x55d986bec000 > -6629> 2018-07-11 07:51:10.395475 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 883 0x55d9c5d3cfc0 rep_scrubmap(5.2e e501 from shard 2) v1 > -6628> 2018-07-11 07:51:10.395494 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 883 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12482 (1269249638 0 93539259) 0x55d9c5d3cfc0 con 0x55d986bf0800 > -6627> 2018-07-11 07:51:10.443870 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.107:0/12617 conn(0x55d98123d800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 232 0x55d9bf805400 osd_ping(ping e501 stamp 2018-07-11 07:51:10.443385) v4 > -6626> 2018-07-11 07:51:10.443886 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.107:0/12617 conn(0x55d98123c000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 232 0x55d986d33200 osd_ping(ping e501 stamp 2018-07-11 07:51:10.443385) v4 > -6625> 2018-07-11 07:51:10.443904 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.9 10.8.128.107:0/12617 232 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:10.443385) v4 ==== 2004+0+0 (765167722 0 0) 0x55d986d33200 con 0x55d98123c000 > -6624> 2018-07-11 07:51:10.443897 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.9 10.8.128.107:0/12617 232 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:10.443385) v4 ==== 2004+0+0 (765167722 0 0) 0x55d9bf805400 con 0x55d98123d800 > -6623> 2018-07-11 07:51:10.443917 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:10.443385) v4 -- 0x55d9b40a4000 con 0 > -6622> 2018-07-11 07:51:10.443942 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:10.443385) v4 -- 0x55d9bf804800 con 0 > -6621> 2018-07-11 07:51:10.484855 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -6620> 2018-07-11 07:51:10.484892 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -6619> 2018-07-11 07:51:10.484933 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -6618> 2018-07-11 07:51:10.485385 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -6617> 2018-07-11 07:51:10.527258 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:753a5661:::benchmark_data_magna115_10867_object2753367:0,end:5:753accfd:::benchmark_data_magna115_10867_object383126:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455400 con 0 > -6616> 2018-07-11 07:51:10.527297 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:753a5661:::benchmark_data_magna115_10867_object2753367:0,end:5:753accfd:::benchmark_data_magna115_10867_object383126:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455680 con 0 > -6615> 2018-07-11 07:51:10.561384 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12677 conn(0x55d986db4800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.1 seq 248 0x55d9bcd72c00 osd_ping(ping e501 stamp 2018-07-11 07:51:10.608738) v4 > -6614> 2018-07-11 07:51:10.561405 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.1 10.8.128.106:0/12677 248 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:10.608738) v4 ==== 2004+0+0 (1372597039 0 0) 0x55d9bcd72c00 con 0x55d986db4800 > -6613> 2018-07-11 07:51:10.561409 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12677 conn(0x55d986df2000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.1 seq 248 0x55d9b2528000 osd_ping(ping e501 stamp 2018-07-11 07:51:10.608738) v4 > -6612> 2018-07-11 07:51:10.561421 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:10.608738) v4 -- 0x55d9c4c1be00 con 0 > -6611> 2018-07-11 07:51:10.561426 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.1 10.8.128.106:0/12677 248 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:10.608738) v4 ==== 2004+0+0 (1372597039 0 0) 0x55d9b2528000 con 0x55d986df2000 > -6610> 2018-07-11 07:51:10.561457 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:10.608738) v4 -- 0x55d9b40a4000 con 0 > -6609> 2018-07-11 07:51:10.735148 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 878 0x55d9c5d3d200 rep_scrubmap(5.2e e501 from shard 4) v1 > -6608> 2018-07-11 07:51:10.735168 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 878 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12472 (3686815630 0 2836965175) 0x55d9c5d3d200 con 0x55d986bec000 > -6607> 2018-07-11 07:51:10.740078 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 884 0x55d9c5d3d440 rep_scrubmap(5.2e e501 from shard 2) v1 > -6606> 2018-07-11 07:51:10.740097 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 884 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12472 (1269249638 0 2836965175) 0x55d9c5d3d440 con 0x55d986bf0800 > -6605> 2018-07-11 07:51:10.812107 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -6604> 2018-07-11 07:51:10.812133 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -6603> 2018-07-11 07:51:10.812188 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -6602> 2018-07-11 07:51:10.812674 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -6601> 2018-07-11 07:51:10.813750 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:753accfd:::benchmark_data_magna115_10867_object383126:0,end:5:753b34f1:::benchmark_data_magna115_10867_object2408234:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498000 con 0 > -6600> 2018-07-11 07:51:10.813798 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:753accfd:::benchmark_data_magna115_10867_object383126:0,end:5:753b34f1:::benchmark_data_magna115_10867_object2408234:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455b80 con 0 > -6599> 2018-07-11 07:51:10.914448 7ffab1bdc700 5 osd.10 501 heartbeat: osd_stat(788 GB used, 143 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist []) > -6598> 2018-07-11 07:51:10.914462 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6802/15255 -- osd_ping(ping e501 stamp 2018-07-11 07:51:10.914461) v4 -- 0x55d9c4c1b600 con 0 > -6597> 2018-07-11 07:51:10.914488 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6803/15255 -- osd_ping(ping e501 stamp 2018-07-11 07:51:10.914461) v4 -- 0x55d9c3383a00 con 0 > -6596> 2018-07-11 07:51:10.914503 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6806/12677 -- osd_ping(ping e501 stamp 2018-07-11 07:51:10.914461) v4 -- 0x55d9c4c1ae00 con 0 > -6595> 2018-07-11 07:51:10.914518 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6807/12677 -- osd_ping(ping e501 stamp 2018-07-11 07:51:10.914461) v4 -- 0x55d9b60df000 con 0 > -6594> 2018-07-11 07:51:10.914531 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6808/12651 -- osd_ping(ping e501 stamp 2018-07-11 07:51:10.914461) v4 -- 0x55d9b9ac2000 con 0 > -6593> 2018-07-11 07:51:10.914542 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6810/12651 -- osd_ping(ping e501 stamp 2018-07-11 07:51:10.914461) v4 -- 0x55d982cdf800 con 0 > -6592> 2018-07-11 07:51:10.914554 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6809/12650 -- osd_ping(ping e501 stamp 2018-07-11 07:51:10.914461) v4 -- 0x55d9bf805200 con 0 > -6591> 2018-07-11 07:51:10.914572 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6811/12650 -- osd_ping(ping e501 stamp 2018-07-11 07:51:10.914461) v4 -- 0x55d980f5a000 con 0 > -6590> 2018-07-11 07:51:10.914585 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6810/15287 -- osd_ping(ping e501 stamp 2018-07-11 07:51:10.914461) v4 -- 0x55d986c85000 con 0 > -6589> 2018-07-11 07:51:10.914597 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6811/15287 -- osd_ping(ping e501 stamp 2018-07-11 07:51:10.914461) v4 -- 0x55d986d97200 con 0 > -6588> 2018-07-11 07:51:10.914613 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6806/15263 -- osd_ping(ping e501 stamp 2018-07-11 07:51:10.914461) v4 -- 0x55d9bdba5200 con 0 > -6587> 2018-07-11 07:51:10.914626 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6807/15263 -- osd_ping(ping e501 stamp 2018-07-11 07:51:10.914461) v4 -- 0x55d986b6d400 con 0 > -6586> 2018-07-11 07:51:10.914639 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.115:6802/13323 -- osd_ping(ping e501 stamp 2018-07-11 07:51:10.914461) v4 -- 0x55d9bdba5e00 con 0 > -6585> 2018-07-11 07:51:10.914652 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.115:6803/13323 -- osd_ping(ping e501 stamp 2018-07-11 07:51:10.914461) v4 -- 0x55d986cd5400 con 0 > -6584> 2018-07-11 07:51:10.914666 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6806/15885 -- osd_ping(ping e501 stamp 2018-07-11 07:51:10.914461) v4 -- 0x55d9c3383600 con 0 > -6583> 2018-07-11 07:51:10.914680 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6807/15885 -- osd_ping(ping e501 stamp 2018-07-11 07:51:10.914461) v4 -- 0x55d986d40000 con 0 > -6582> 2018-07-11 07:51:10.914693 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.108:6802/11967 -- osd_ping(ping e501 stamp 2018-07-11 07:51:10.914461) v4 -- 0x55d986d96400 con 0 > -6581> 2018-07-11 07:51:10.914706 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.108:6803/11967 -- osd_ping(ping e501 stamp 2018-07-11 07:51:10.914461) v4 -- 0x55d986cd5c00 con 0 > -6580> 2018-07-11 07:51:10.914718 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.107:6806/12617 -- osd_ping(ping e501 stamp 2018-07-11 07:51:10.914461) v4 -- 0x55d986d97e00 con 0 > -6579> 2018-07-11 07:51:10.914731 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.107:6807/12617 -- osd_ping(ping e501 stamp 2018-07-11 07:51:10.914461) v4 -- 0x55d986c22400 con 0 > -6578> 2018-07-11 07:51:10.914742 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6802/15919 -- osd_ping(ping e501 stamp 2018-07-11 07:51:10.914461) v4 -- 0x55d986d4e400 con 0 > -6577> 2018-07-11 07:51:10.914754 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6803/15919 -- osd_ping(ping e501 stamp 2018-07-11 07:51:10.914461) v4 -- 0x55d9c3382a00 con 0 > -6576> 2018-07-11 07:51:10.914934 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.107:6806/12617 conn(0x55d986bd2000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.9 seq 235 0x55d9829e5600 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:10.914461) v4 > -6575> 2018-07-11 07:51:10.914956 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.9 10.8.128.107:6806/12617 235 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:10.914461) v4 ==== 2004+0+0 (4058726518 0 0) 0x55d9829e5600 con 0x55d986bd2000 > -6574> 2018-07-11 07:51:10.914961 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6807/12677 conn(0x55d986b59800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.1 seq 235 0x55d9b2528000 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:10.914461) v4 > -6573> 2018-07-11 07:51:10.914982 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.1 10.8.128.106:6807/12677 235 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:10.914461) v4 ==== 2004+0+0 (4058726518 0 0) 0x55d9b2528000 con 0x55d986b59800 > -6572> 2018-07-11 07:51:10.914980 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.107:6807/12617 conn(0x55d986bd3800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.9 seq 235 0x55d9bf805800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:10.914461) v4 > -6571> 2018-07-11 07:51:10.915014 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.9 10.8.128.107:6807/12617 235 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:10.914461) v4 ==== 2004+0+0 (4058726518 0 0) 0x55d9bf805800 con 0x55d986bd3800 > -6570> 2018-07-11 07:51:10.915039 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6810/12651 conn(0x55d986b9b800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.2 seq 235 0x55d9829e5600 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:10.914461) v4 > -6569> 2018-07-11 07:51:10.915044 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6808/12651 conn(0x55d986b9a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.2 seq 235 0x55d9b2528000 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:10.914461) v4 > -6568> 2018-07-11 07:51:10.915055 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.2 10.8.128.106:6808/12651 235 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:10.914461) v4 ==== 2004+0+0 (4058726518 0 0) 0x55d9b2528000 con 0x55d986b9a000 > -6567> 2018-07-11 07:51:10.915054 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.2 10.8.128.106:6810/12651 235 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:10.914461) v4 ==== 2004+0+0 (4058726518 0 0) 0x55d9829e5600 con 0x55d986b9b800 > -6566> 2018-07-11 07:51:10.915078 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6806/12677 conn(0x55d986b58000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.1 seq 235 0x55d9bf805800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:10.914461) v4 > -6565> 2018-07-11 07:51:10.915098 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.1 10.8.128.106:6806/12677 235 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:10.914461) v4 ==== 2004+0+0 (4058726518 0 0) 0x55d9bf805800 con 0x55d986b58000 > -6564> 2018-07-11 07:51:10.915102 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6802/15255 conn(0x55d986bc8000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.0 seq 235 0x55d9b2528000 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:10.914461) v4 > -6563> 2018-07-11 07:51:10.915112 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.0 10.8.128.113:6802/15255 235 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:10.914461) v4 ==== 2004+0+0 (4058726518 0 0) 0x55d9b2528000 con 0x55d986bc8000 > -6562> 2018-07-11 07:51:10.915121 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6803/15255 conn(0x55d986bc9800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.0 seq 235 0x55d9829e5600 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:10.914461) v4 > -6561> 2018-07-11 07:51:10.915143 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6807/15263 conn(0x55d986b9e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.5 seq 235 0x55d9b2528000 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:10.914461) v4 > -6560> 2018-07-11 07:51:10.915153 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.0 10.8.128.113:6803/15255 235 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:10.914461) v4 ==== 2004+0+0 (4058726518 0 0) 0x55d9829e5600 con 0x55d986bc9800 > -6559> 2018-07-11 07:51:10.915158 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6806/15263 conn(0x55d986b9d000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.5 seq 235 0x55d9bf805800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:10.914461) v4 > -6558> 2018-07-11 07:51:10.915159 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.5 10.8.128.113:6807/15263 235 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:10.914461) v4 ==== 2004+0+0 (4058726518 0 0) 0x55d9b2528000 con 0x55d986b9e800 > -6557> 2018-07-11 07:51:10.915179 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.5 10.8.128.113:6806/15263 235 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:10.914461) v4 ==== 2004+0+0 (4058726518 0 0) 0x55d9bf805800 con 0x55d986b9d000 > -6556> 2018-07-11 07:51:10.915206 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6811/15287 conn(0x55d986bb0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 235 0x55d9829e5600 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:10.914461) v4 > -6555> 2018-07-11 07:51:10.915211 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6810/15287 conn(0x55d986baf000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 235 0x55d9b2528000 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:10.914461) v4 > -6554> 2018-07-11 07:51:10.915222 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.4 10.8.128.113:6810/15287 235 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:10.914461) v4 ==== 2004+0+0 (4058726518 0 0) 0x55d9b2528000 con 0x55d986baf000 > -6553> 2018-07-11 07:51:10.915220 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.4 10.8.128.113:6811/15287 235 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:10.914461) v4 ==== 2004+0+0 (4058726518 0 0) 0x55d9829e5600 con 0x55d986bb0800 > -6552> 2018-07-11 07:51:10.915240 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6809/12650 conn(0x55d986bcb000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 235 0x55d9bf805800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:10.914461) v4 > -6551> 2018-07-11 07:51:10.915268 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.3 10.8.128.106:6809/12650 235 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:10.914461) v4 ==== 2004+0+0 (4058726518 0 0) 0x55d9bf805800 con 0x55d986bcb000 > -6550> 2018-07-11 07:51:10.915283 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6811/12650 conn(0x55d986bcc800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 235 0x55d9b2528000 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:10.914461) v4 > -6549> 2018-07-11 07:51:10.915303 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.115:6802/13323 conn(0x55d986bc1000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 235 0x55d9829e5600 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:10.914461) v4 > -6548> 2018-07-11 07:51:10.915311 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.3 10.8.128.106:6811/12650 235 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:10.914461) v4 ==== 2004+0+0 (4058726518 0 0) 0x55d9b2528000 con 0x55d986bcc800 > -6547> 2018-07-11 07:51:10.915317 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.108:6802/11967 conn(0x55d986bbe000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.8 seq 235 0x55d9bf805800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:10.914461) v4 > -6546> 2018-07-11 07:51:10.915325 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.6 10.8.128.115:6802/13323 235 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:10.914461) v4 ==== 2004+0+0 (4058726518 0 0) 0x55d9829e5600 con 0x55d986bc1000 > -6545> 2018-07-11 07:51:10.915332 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.8 10.8.128.108:6802/11967 235 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:10.914461) v4 ==== 2004+0+0 (4058726518 0 0) 0x55d9bf805800 con 0x55d986bbe000 > -6544> 2018-07-11 07:51:10.915355 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.108:6803/11967 conn(0x55d986bbf800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.8 seq 235 0x55d9b2528000 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:10.914461) v4 > -6543> 2018-07-11 07:51:10.915369 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.8 10.8.128.108:6803/11967 235 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:10.914461) v4 ==== 2004+0+0 (4058726518 0 0) 0x55d9b2528000 con 0x55d986bbf800 > -6542> 2018-07-11 07:51:10.915377 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6802/15919 conn(0x55d986b5b000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.11 seq 235 0x55d9829e5600 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:10.914461) v4 > -6541> 2018-07-11 07:51:10.915379 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.115:6803/13323 conn(0x55d986bc2800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 235 0x55d9bf805800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:10.914461) v4 > -6540> 2018-07-11 07:51:10.915394 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.11 10.8.128.114:6802/15919 235 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:10.914461) v4 ==== 2004+0+0 (4058726518 0 0) 0x55d9829e5600 con 0x55d986b5b000 > -6539> 2018-07-11 07:51:10.915395 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.6 10.8.128.115:6803/13323 235 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:10.914461) v4 ==== 2004+0+0 (4058726518 0 0) 0x55d9bf805800 con 0x55d986bc2800 > -6538> 2018-07-11 07:51:10.915436 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6806/15885 conn(0x55d986bac000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.7 seq 235 0x55d9829e5600 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:10.914461) v4 > -6537> 2018-07-11 07:51:10.915449 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.7 10.8.128.114:6806/15885 235 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:10.914461) v4 ==== 2004+0+0 (4058726518 0 0) 0x55d9829e5600 con 0x55d986bac000 > -6536> 2018-07-11 07:51:10.915452 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6803/15919 conn(0x55d986b5c800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.11 seq 235 0x55d9bf805800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:10.914461) v4 > -6535> 2018-07-11 07:51:10.915471 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.11 10.8.128.114:6803/15919 235 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:10.914461) v4 ==== 2004+0+0 (4058726518 0 0) 0x55d9bf805800 con 0x55d986b5c800 > -6534> 2018-07-11 07:51:10.915512 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6807/15885 conn(0x55d986bad800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.7 seq 235 0x55d9bf805800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:10.914461) v4 > -6533> 2018-07-11 07:51:10.915529 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.7 10.8.128.114:6807/15885 235 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:10.914461) v4 ==== 2004+0+0 (4058726518 0 0) 0x55d9bf805800 con 0x55d986bad800 > -6532> 2018-07-11 07:51:10.958182 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6805/15263 conn(0x55d986c41000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=31 cs=1 l=0). rx osd.5 seq 2806 0x55d9b2528000 MOSDScrubReserve(6.15 REQUEST e501) v1 > -6531> 2018-07-11 07:51:10.958200 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.5 10.8.128.113:6805/15263 2806 ==== MOSDScrubReserve(6.15 REQUEST e501) v1 ==== 43+0+0 (3942742026 0 0) 0x55d9b2528000 con 0x55d986c41000 > -6530> 2018-07-11 07:51:11.023194 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 885 0x55d9c5d3d680 rep_scrubmap(5.2e e501 from shard 2) v1 > -6529> 2018-07-11 07:51:11.023214 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 885 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12482 (1269249638 0 1393464584) 0x55d9c5d3d680 con 0x55d986bf0800 > -6528> 2018-07-11 07:51:11.039104 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 879 0x55d9c5d3d8c0 rep_scrubmap(5.2e e501 from shard 4) v1 > -6527> 2018-07-11 07:51:11.039122 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 879 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12482 (3686815630 0 1393464584) 0x55d9c5d3d8c0 con 0x55d986bec000 > -6526> 2018-07-11 07:51:11.061903 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12677 conn(0x55d986df2000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.1 seq 249 0x55d986d33200 osd_ping(ping e501 stamp 2018-07-11 07:51:11.109266) v4 > -6525> 2018-07-11 07:51:11.061905 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12677 conn(0x55d986db4800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.1 seq 249 0x55d9829e5600 osd_ping(ping e501 stamp 2018-07-11 07:51:11.109266) v4 > -6524> 2018-07-11 07:51:11.061924 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.1 10.8.128.106:0/12677 249 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:11.109266) v4 ==== 2004+0+0 (107586252 0 0) 0x55d986d33200 con 0x55d986df2000 > -6523> 2018-07-11 07:51:11.061928 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.1 10.8.128.106:0/12677 249 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:11.109266) v4 ==== 2004+0+0 (107586252 0 0) 0x55d9829e5600 con 0x55d986db4800 > -6522> 2018-07-11 07:51:11.061935 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:11.109266) v4 -- 0x55d986b6d400 con 0 > -6521> 2018-07-11 07:51:11.061965 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:11.109266) v4 -- 0x55d986d4e400 con 0 > -6520> 2018-07-11 07:51:11.064053 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15255 conn(0x55d986c82800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.0 seq 245 0x55d9869bfa00 osd_ping(ping e501 stamp 2018-07-11 07:51:12.294710) v4 > -6519> 2018-07-11 07:51:11.064078 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.0 10.8.128.113:0/15255 245 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:12.294710) v4 ==== 2004+0+0 (3430204825 0 0) 0x55d9869bfa00 con 0x55d986c82800 > -6518> 2018-07-11 07:51:11.064095 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:12.294710) v4 -- 0x55d986d4e400 con 0 > -6517> 2018-07-11 07:51:11.064100 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15255 conn(0x55d986d20000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.0 seq 245 0x55d986cd5c00 osd_ping(ping e501 stamp 2018-07-11 07:51:12.294710) v4 > -6516> 2018-07-11 07:51:11.064124 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.0 10.8.128.113:0/15255 245 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:12.294710) v4 ==== 2004+0+0 (3430204825 0 0) 0x55d986cd5c00 con 0x55d986d20000 > -6515> 2018-07-11 07:51:11.064136 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:12.294710) v4 -- 0x55d986b6d400 con 0 > -6514> 2018-07-11 07:51:11.072532 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6805/15263 -- MOSDScrubReserve(6.15 REJECT e501) v1 -- 0x55d9829e4200 con 0 > -6513> 2018-07-11 07:51:11.072717 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -6512> 2018-07-11 07:51:11.072740 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -6511> 2018-07-11 07:51:11.072787 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -6510> 2018-07-11 07:51:11.072923 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6805/15263 conn(0x55d986c41000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=31 cs=1 l=0). rx osd.5 seq 2807 0x55d9829e4200 MOSDScrubReserve(6.15 RELEASE e501) v1 > -6509> 2018-07-11 07:51:11.072941 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.5 10.8.128.113:6805/15263 2807 ==== MOSDScrubReserve(6.15 RELEASE e501) v1 ==== 43+0+0 (605848072 0 0) 0x55d9829e4200 con 0x55d986c41000 > -6508> 2018-07-11 07:51:11.073244 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -6507> 2018-07-11 07:51:11.094081 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67423,epoch:501/498,start:5:753b34f1:::benchmark_data_magna115_10867_object2408234:0,end:5:753b99bc:::benchmark_data_magna115_10867_object696436:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498780 con 0 > -6506> 2018-07-11 07:51:11.094131 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67423,epoch:501/498,start:5:753b34f1:::benchmark_data_magna115_10867_object2408234:0,end:5:753b99bc:::benchmark_data_magna115_10867_object696436:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455900 con 0 > -6505> 2018-07-11 07:51:11.185387 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.108:0/11967 conn(0x55d986db0000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.8 seq 225 0x55d9829e5600 osd_ping(ping e501 stamp 2018-07-11 07:51:10.940914) v4 > -6504> 2018-07-11 07:51:11.185399 7ffacbbd1700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.108:0/11967 conn(0x55d986d82800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.8 seq 225 0x55d9bf805800 osd_ping(ping e501 stamp 2018-07-11 07:51:10.940914) v4 > -6503> 2018-07-11 07:51:11.185409 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.8 10.8.128.108:0/11967 225 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:10.940914) v4 ==== 2004+0+0 (2371563584 0 0) 0x55d9829e5600 con 0x55d986db0000 > -6502> 2018-07-11 07:51:11.185417 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 <== osd.8 10.8.128.108:0/11967 225 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:10.940914) v4 ==== 2004+0+0 (2371563584 0 0) 0x55d9bf805800 con 0x55d986d82800 > -6501> 2018-07-11 07:51:11.185427 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:10.940914) v4 -- 0x55d986d4e400 con 0 > -6500> 2018-07-11 07:51:11.185459 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:10.940914) v4 -- 0x55d9c3382a00 con 0 > -6499> 2018-07-11 07:51:11.333084 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 886 0x55d9c5d3db00 rep_scrubmap(5.2e e501 from shard 2) v1 > -6498> 2018-07-11 07:51:11.333103 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 886 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12464 (1269249638 0 2714946510) 0x55d9c5d3db00 con 0x55d986bf0800 > -6497> 2018-07-11 07:51:11.342540 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 880 0x55d9c5d3dd40 rep_scrubmap(5.2e e501 from shard 4) v1 > -6496> 2018-07-11 07:51:11.342559 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 880 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12464 (3686815630 0 2714946510) 0x55d9c5d3dd40 con 0x55d986bec000 > -6495> 2018-07-11 07:51:11.619101 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12651 conn(0x55d986db3000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.2 seq 243 0x55d9bf805400 osd_ping(ping e501 stamp 2018-07-11 07:51:11.666441) v4 > -6494> 2018-07-11 07:51:11.619123 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12651 conn(0x55d986db1800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.2 seq 243 0x55d986cd5c00 osd_ping(ping e501 stamp 2018-07-11 07:51:11.666441) v4 > -6493> 2018-07-11 07:51:11.619138 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.2 10.8.128.106:0/12651 243 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:11.666441) v4 ==== 2004+0+0 (3600452157 0 0) 0x55d9bf805400 con 0x55d986db3000 > -6492> 2018-07-11 07:51:11.619143 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.2 10.8.128.106:0/12651 243 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:11.666441) v4 ==== 2004+0+0 (3600452157 0 0) 0x55d986cd5c00 con 0x55d986db1800 > -6491> 2018-07-11 07:51:11.619150 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:11.666441) v4 -- 0x55d9c3382a00 con 0 > -6490> 2018-07-11 07:51:11.619179 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:11.666441) v4 -- 0x55d986b6d400 con 0 > -6489> 2018-07-11 07:51:11.958303 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6805/15263 conn(0x55d986c41000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=31 cs=1 l=0). rx osd.5 seq 2808 0x55d986d33200 MOSDScrubReserve(6.15 REQUEST e501) v1 > -6488> 2018-07-11 07:51:11.958321 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.5 10.8.128.113:6805/15263 2808 ==== MOSDScrubReserve(6.15 REQUEST e501) v1 ==== 43+0+0 (3942742026 0 0) 0x55d986d33200 con 0x55d986c41000 > -6487> 2018-07-11 07:51:11.999224 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.115:0/13323 conn(0x55d986d7e000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 246 0x55d9bf805800 osd_ping(ping e501 stamp 2018-07-11 07:51:12.344767) v4 > -6486> 2018-07-11 07:51:11.999259 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.6 10.8.128.115:0/13323 246 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:12.344767) v4 ==== 2004+0+0 (2681034713 0 0) 0x55d9bf805800 con 0x55d986d7e000 > -6485> 2018-07-11 07:51:11.999258 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.115:0/13323 conn(0x55d986d24800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 246 0x55d986cd5c00 osd_ping(ping e501 stamp 2018-07-11 07:51:12.344767) v4 > -6484> 2018-07-11 07:51:11.999278 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.6 10.8.128.115:0/13323 246 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:12.344767) v4 ==== 2004+0+0 (2681034713 0 0) 0x55d986cd5c00 con 0x55d986d24800 > -6483> 2018-07-11 07:51:11.999277 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:12.344767) v4 -- 0x55d9c3382a00 con 0 > -6482> 2018-07-11 07:51:11.999318 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:12.344767) v4 -- 0x55d986c85000 con 0 > -6481> 2018-07-11 07:51:12.115683 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.114:0/15885 conn(0x55d986d7f800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.7 seq 250 0x55d9869bfa00 osd_ping(ping e501 stamp 2018-07-11 07:51:12.745803) v4 > -6480> 2018-07-11 07:51:12.115706 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.7 10.8.128.114:0/15885 250 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:12.745803) v4 ==== 2004+0+0 (2499093734 0 0) 0x55d9869bfa00 con 0x55d986d7f800 > -6479> 2018-07-11 07:51:12.115718 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:12.745803) v4 -- 0x55d986d4e400 con 0 > -6478> 2018-07-11 07:51:12.115729 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.114:0/15885 conn(0x55d986d81000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.7 seq 250 0x55d986b6d400 osd_ping(ping e501 stamp 2018-07-11 07:51:12.745803) v4 > -6477> 2018-07-11 07:51:12.115750 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.7 10.8.128.114:0/15885 250 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:12.745803) v4 ==== 2004+0+0 (2499093734 0 0) 0x55d986b6d400 con 0x55d986d81000 > -6476> 2018-07-11 07:51:12.115762 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:12.745803) v4 -- 0x55d986c85000 con 0 > -6475> 2018-07-11 07:51:12.164485 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15255 conn(0x55d986d20000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.0 seq 246 0x55d986cd5c00 osd_ping(ping e501 stamp 2018-07-11 07:51:13.395120) v4 > -6474> 2018-07-11 07:51:12.164506 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.0 10.8.128.113:0/15255 246 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:13.395120) v4 ==== 2004+0+0 (4051376306 0 0) 0x55d986cd5c00 con 0x55d986d20000 > -6473> 2018-07-11 07:51:12.164516 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:13.395120) v4 -- 0x55d986c85000 con 0 > -6472> 2018-07-11 07:51:12.164514 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15255 conn(0x55d986c82800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.0 seq 246 0x55d9829e5600 osd_ping(ping e501 stamp 2018-07-11 07:51:13.395120) v4 > -6471> 2018-07-11 07:51:12.164532 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.0 10.8.128.113:0/15255 246 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:13.395120) v4 ==== 2004+0+0 (4051376306 0 0) 0x55d9829e5600 con 0x55d986c82800 > -6470> 2018-07-11 07:51:12.164545 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:13.395120) v4 -- 0x55d986d4e400 con 0 > -6469> 2018-07-11 07:51:12.301700 7ffab7be8700 10 monclient: tick > -6468> 2018-07-11 07:51:12.301713 7ffab7be8700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2018-07-11 07:50:42.301712) > -6467> 2018-07-11 07:51:12.349202 7ffab6be6700 1 -- 10.8.128.107:6800/13927 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55d9ba81a840 con 0 > -6466> 2018-07-11 07:51:12.349282 7ffab6be6700 1 -- 10.8.128.107:6800/13927 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55d9c1fbd080 con 0 > -6465> 2018-07-11 07:51:12.530823 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6805/15263 -- MOSDScrubReserve(6.15 REJECT e501) v1 -- 0x55d9829e4200 con 0 > -6464> 2018-07-11 07:51:12.530878 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -6463> 2018-07-11 07:51:12.530903 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -6462> 2018-07-11 07:51:12.530950 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -6461> 2018-07-11 07:51:12.531303 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6805/15263 conn(0x55d986c41000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=31 cs=1 l=0). rx osd.5 seq 2809 0x55d9829e4200 MOSDScrubReserve(6.15 RELEASE e501) v1 > -6460> 2018-07-11 07:51:12.531323 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.5 10.8.128.113:6805/15263 2809 ==== MOSDScrubReserve(6.15 RELEASE e501) v1 ==== 43+0+0 (605848072 0 0) 0x55d9829e4200 con 0x55d986c41000 > -6459> 2018-07-11 07:51:12.531420 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -6458> 2018-07-11 07:51:12.561072 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6805/15885 conn(0x55d986c7f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.7 seq 2786 0x55d986cd5c00 MOSDScrubReserve(5.4 REQUEST e501) v1 > -6457> 2018-07-11 07:51:12.561103 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.7 10.8.128.114:6805/15885 2786 ==== MOSDScrubReserve(5.4 REQUEST e501) v1 ==== 43+0+0 (1753221512 0 0) 0x55d986cd5c00 con 0x55d986c7f800 > -6456> 2018-07-11 07:51:12.561129 7ffab33df700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.114:6805/15885 -- MOSDScrubReserve(5.4 REJECT e501) v1 -- 0x55d986d32200 con 0 > -6455> 2018-07-11 07:51:12.561451 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6805/15885 conn(0x55d986c7f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.7 seq 2787 0x55d986d32200 MOSDScrubReserve(5.4 RELEASE e501) v1 > -6454> 2018-07-11 07:51:12.561467 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.7 10.8.128.114:6805/15885 2787 ==== MOSDScrubReserve(5.4 RELEASE e501) v1 ==== 43+0+0 (2812146570 0 0) 0x55d986d32200 con 0x55d986c7f800 > -6453> 2018-07-11 07:51:12.613701 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 31] Generated table #36798: 2638240 keys, 74546031 bytes > -6452> 2018-07-11 07:51:12.613746 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295472613718, "cf_name": "default", "job": 31, "event": "table_file_creation", "file_number": 36798, "file_size": 74546031, "table_properties": {"data_size": 67112153, "index_size": 689707, "filter_size": 6743181, "raw_key_size": 76215830, "raw_average_key_size": 28, "raw_value_size": 31365750, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} > -6451> 2018-07-11 07:51:12.614878 7ffab1bdc700 5 osd.10 501 heartbeat: osd_stat(788 GB used, 142 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist []) > -6450> 2018-07-11 07:51:12.614892 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6802/15255 -- osd_ping(ping e501 stamp 2018-07-11 07:51:12.614891) v4 -- 0x55d9b67d8a00 con 0 > -6449> 2018-07-11 07:51:12.614931 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6803/15255 -- osd_ping(ping e501 stamp 2018-07-11 07:51:12.614891) v4 -- 0x55d986d97000 con 0 > -6448> 2018-07-11 07:51:12.614947 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6806/12677 -- osd_ping(ping e501 stamp 2018-07-11 07:51:12.614891) v4 -- 0x55d9b67d9600 con 0 > -6447> 2018-07-11 07:51:12.614962 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6807/12677 -- osd_ping(ping e501 stamp 2018-07-11 07:51:12.614891) v4 -- 0x55d980457a00 con 0 > -6446> 2018-07-11 07:51:12.614974 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6808/12651 -- osd_ping(ping e501 stamp 2018-07-11 07:51:12.614891) v4 -- 0x55d986d4e600 con 0 > -6445> 2018-07-11 07:51:12.614984 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6810/12651 -- osd_ping(ping e501 stamp 2018-07-11 07:51:12.614891) v4 -- 0x55d982cdea00 con 0 > -6444> 2018-07-11 07:51:12.614996 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6809/12650 -- osd_ping(ping e501 stamp 2018-07-11 07:51:12.614891) v4 -- 0x55d9829e4400 con 0 > -6443> 2018-07-11 07:51:12.615015 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6811/12650 -- osd_ping(ping e501 stamp 2018-07-11 07:51:12.614891) v4 -- 0x55d9bcd72c00 con 0 > -6442> 2018-07-11 07:51:12.615028 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6810/15287 -- osd_ping(ping e501 stamp 2018-07-11 07:51:12.614891) v4 -- 0x55d9c3383a00 con 0 > -6441> 2018-07-11 07:51:12.615039 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6811/15287 -- osd_ping(ping e501 stamp 2018-07-11 07:51:12.614891) v4 -- 0x55d9c4c1be00 con 0 > -6440> 2018-07-11 07:51:12.615051 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6806/15263 -- osd_ping(ping e501 stamp 2018-07-11 07:51:12.614891) v4 -- 0x55d9bfbde400 con 0 > -6439> 2018-07-11 07:51:12.615067 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6807/15263 -- osd_ping(ping e501 stamp 2018-07-11 07:51:12.614891) v4 -- 0x55d986c22000 con 0 > -6438> 2018-07-11 07:51:12.615079 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.115:6802/13323 -- osd_ping(ping e501 stamp 2018-07-11 07:51:12.614891) v4 -- 0x55d9bdba5600 con 0 > -6437> 2018-07-11 07:51:12.615091 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.115:6803/13323 -- osd_ping(ping e501 stamp 2018-07-11 07:51:12.614891) v4 -- 0x55d9b60df200 con 0 > -6436> 2018-07-11 07:51:12.615103 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6806/15885 -- osd_ping(ping e501 stamp 2018-07-11 07:51:12.614891) v4 -- 0x55d986d40600 con 0 > -6435> 2018-07-11 07:51:12.615118 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6807/15885 -- osd_ping(ping e501 stamp 2018-07-11 07:51:12.614891) v4 -- 0x55d986d41e00 con 0 > -6434> 2018-07-11 07:51:12.615131 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.108:6802/11967 -- osd_ping(ping e501 stamp 2018-07-11 07:51:12.614891) v4 -- 0x55d982d31e00 con 0 > -6433> 2018-07-11 07:51:12.615143 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.108:6803/11967 -- osd_ping(ping e501 stamp 2018-07-11 07:51:12.614891) v4 -- 0x55d9bdba4800 con 0 > -6432> 2018-07-11 07:51:12.615155 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.107:6806/12617 -- osd_ping(ping e501 stamp 2018-07-11 07:51:12.614891) v4 -- 0x55d9c9123000 con 0 > -6431> 2018-07-11 07:51:12.615168 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.107:6807/12617 -- osd_ping(ping e501 stamp 2018-07-11 07:51:12.614891) v4 -- 0x55d9b67d8c00 con 0 > -6430> 2018-07-11 07:51:12.615180 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6802/15919 -- osd_ping(ping e501 stamp 2018-07-11 07:51:12.614891) v4 -- 0x55d9b3db2600 con 0 > -6429> 2018-07-11 07:51:12.615192 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6803/15919 -- osd_ping(ping e501 stamp 2018-07-11 07:51:12.614891) v4 -- 0x55d9b60de600 con 0 > -6428> 2018-07-11 07:51:12.615378 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.107:6806/12617 conn(0x55d986bd2000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.9 seq 236 0x55d9829e5600 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:12.614891) v4 > -6427> 2018-07-11 07:51:12.615404 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6809/12650 conn(0x55d986bcb000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 236 0x55d9bf805400 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:12.614891) v4 > -6426> 2018-07-11 07:51:12.615400 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.9 10.8.128.107:6806/12617 236 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:12.614891) v4 ==== 2004+0+0 (3895956623 0 0) 0x55d9829e5600 con 0x55d986bd2000 > -6425> 2018-07-11 07:51:12.615426 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.3 10.8.128.106:6809/12650 236 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:12.614891) v4 ==== 2004+0+0 (3895956623 0 0) 0x55d9bf805400 con 0x55d986bcb000 > -6424> 2018-07-11 07:51:12.615478 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6811/15287 conn(0x55d986bb0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 236 0x55d9829e5600 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:12.614891) v4 > -6423> 2018-07-11 07:51:12.615475 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6808/12651 conn(0x55d986b9a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.2 seq 236 0x55d9c9123200 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:12.614891) v4 > -6422> 2018-07-11 07:51:12.615504 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.2 10.8.128.106:6808/12651 236 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:12.614891) v4 ==== 2004+0+0 (3895956623 0 0) 0x55d9c9123200 con 0x55d986b9a000 > -6421> 2018-07-11 07:51:12.615492 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.4 10.8.128.113:6811/15287 236 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:12.614891) v4 ==== 2004+0+0 (3895956623 0 0) 0x55d9829e5600 con 0x55d986bb0800 > -6420> 2018-07-11 07:51:12.615504 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.107:6807/12617 conn(0x55d986bd3800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.9 seq 236 0x55d9bf805400 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:12.614891) v4 > -6419> 2018-07-11 07:51:12.615517 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.9 10.8.128.107:6807/12617 236 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:12.614891) v4 ==== 2004+0+0 (3895956623 0 0) 0x55d9bf805400 con 0x55d986bd3800 > -6418> 2018-07-11 07:51:12.615540 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6803/15255 conn(0x55d986bc9800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.0 seq 236 0x55d9829e5600 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:12.614891) v4 > -6417> 2018-07-11 07:51:12.615553 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.0 10.8.128.113:6803/15255 236 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:12.614891) v4 ==== 2004+0+0 (3895956623 0 0) 0x55d9829e5600 con 0x55d986bc9800 > -6416> 2018-07-11 07:51:12.615551 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6802/15255 conn(0x55d986bc8000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.0 seq 236 0x55d9bcd79000 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:12.614891) v4 > -6415> 2018-07-11 07:51:12.615563 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.0 10.8.128.113:6802/15255 236 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:12.614891) v4 ==== 2004+0+0 (3895956623 0 0) 0x55d9bcd79000 con 0x55d986bc8000 > -6414> 2018-07-11 07:51:12.615552 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6806/15263 conn(0x55d986b9d000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.5 seq 236 0x55d9bf805400 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:12.614891) v4 > -6413> 2018-07-11 07:51:12.615584 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6810/12651 conn(0x55d986b9b800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.2 seq 236 0x55d9829e5600 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:12.614891) v4 > -6412> 2018-07-11 07:51:12.615589 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.5 10.8.128.113:6806/15263 236 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:12.614891) v4 ==== 2004+0+0 (3895956623 0 0) 0x55d9bf805400 con 0x55d986b9d000 > -6411> 2018-07-11 07:51:12.615596 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.2 10.8.128.106:6810/12651 236 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:12.614891) v4 ==== 2004+0+0 (3895956623 0 0) 0x55d9829e5600 con 0x55d986b9b800 > -6410> 2018-07-11 07:51:12.615597 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6811/12650 conn(0x55d986bcc800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 236 0x55d9bcd79000 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:12.614891) v4 > -6409> 2018-07-11 07:51:12.615607 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.3 10.8.128.106:6811/12650 236 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:12.614891) v4 ==== 2004+0+0 (3895956623 0 0) 0x55d9bcd79000 con 0x55d986bcc800 > -6408> 2018-07-11 07:51:12.615622 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6806/12677 conn(0x55d986b58000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.1 seq 236 0x55d9bf805400 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:12.614891) v4 > -6407> 2018-07-11 07:51:12.615634 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.1 10.8.128.106:6806/12677 236 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:12.614891) v4 ==== 2004+0+0 (3895956623 0 0) 0x55d9bf805400 con 0x55d986b58000 > -6406> 2018-07-11 07:51:12.615637 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6807/15263 conn(0x55d986b9e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.5 seq 236 0x55d9bcd79000 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:12.614891) v4 > -6405> 2018-07-11 07:51:12.615647 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.5 10.8.128.113:6807/15263 236 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:12.614891) v4 ==== 2004+0+0 (3895956623 0 0) 0x55d9bcd79000 con 0x55d986b9e800 > -6404> 2018-07-11 07:51:12.615695 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.115:6802/13323 conn(0x55d986bc1000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 236 0x55d9829e5600 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:12.614891) v4 > -6403> 2018-07-11 07:51:12.615678 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6807/12677 conn(0x55d986b59800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.1 seq 236 0x55d9bcd79000 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:12.614891) v4 > -6402> 2018-07-11 07:51:12.615720 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.1 10.8.128.106:6807/12677 236 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:12.614891) v4 ==== 2004+0+0 (3895956623 0 0) 0x55d9bcd79000 con 0x55d986b59800 > -6401> 2018-07-11 07:51:12.615724 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.115:6803/13323 conn(0x55d986bc2800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 236 0x55d9bf805400 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:12.614891) v4 > -6400> 2018-07-11 07:51:12.615716 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.6 10.8.128.115:6802/13323 236 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:12.614891) v4 ==== 2004+0+0 (3895956623 0 0) 0x55d9829e5600 con 0x55d986bc1000 > -6399> 2018-07-11 07:51:12.615737 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.6 10.8.128.115:6803/13323 236 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:12.614891) v4 ==== 2004+0+0 (3895956623 0 0) 0x55d9bf805400 con 0x55d986bc2800 > -6398> 2018-07-11 07:51:12.615754 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6810/15287 conn(0x55d986baf000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 236 0x55d9bcd79000 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:12.614891) v4 > -6397> 2018-07-11 07:51:12.615766 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.4 10.8.128.113:6810/15287 236 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:12.614891) v4 ==== 2004+0+0 (3895956623 0 0) 0x55d9bcd79000 con 0x55d986baf000 > -6396> 2018-07-11 07:51:12.615791 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.108:6802/11967 conn(0x55d986bbe000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.8 seq 236 0x55d9bf805400 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:12.614891) v4 > -6395> 2018-07-11 07:51:12.615797 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.108:6803/11967 conn(0x55d986bbf800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.8 seq 236 0x55d9bcd79000 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:12.614891) v4 > -6394> 2018-07-11 07:51:12.615804 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.8 10.8.128.108:6802/11967 236 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:12.614891) v4 ==== 2004+0+0 (3895956623 0 0) 0x55d9bf805400 con 0x55d986bbe000 > -6393> 2018-07-11 07:51:12.615806 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.8 10.8.128.108:6803/11967 236 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:12.614891) v4 ==== 2004+0+0 (3895956623 0 0) 0x55d9bcd79000 con 0x55d986bbf800 > -6392> 2018-07-11 07:51:12.615815 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6802/15919 conn(0x55d986b5b000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.11 seq 236 0x55d9829e5600 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:12.614891) v4 > -6391> 2018-07-11 07:51:12.615835 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.11 10.8.128.114:6802/15919 236 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:12.614891) v4 ==== 2004+0+0 (3895956623 0 0) 0x55d9829e5600 con 0x55d986b5b000 > -6390> 2018-07-11 07:51:12.615841 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6807/15885 conn(0x55d986bad800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.7 seq 236 0x55d9bf805400 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:12.614891) v4 > -6389> 2018-07-11 07:51:12.615851 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.7 10.8.128.114:6807/15885 236 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:12.614891) v4 ==== 2004+0+0 (3895956623 0 0) 0x55d9bf805400 con 0x55d986bad800 > -6388> 2018-07-11 07:51:12.615876 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6806/15885 conn(0x55d986bac000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.7 seq 236 0x55d9829e5600 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:12.614891) v4 > -6387> 2018-07-11 07:51:12.615881 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6803/15919 conn(0x55d986b5c800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.11 seq 236 0x55d9bf805400 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:12.614891) v4 > -6386> 2018-07-11 07:51:12.615891 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.11 10.8.128.114:6803/15919 236 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:12.614891) v4 ==== 2004+0+0 (3895956623 0 0) 0x55d9bf805400 con 0x55d986b5c800 > -6385> 2018-07-11 07:51:12.615890 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.7 10.8.128.114:6806/15885 236 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:12.614891) v4 ==== 2004+0+0 (3895956623 0 0) 0x55d9829e5600 con 0x55d986bac000 > -6384> 2018-07-11 07:51:12.619365 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:753b99bc:::benchmark_data_magna115_10867_object696436:0,end:5:753bd033:::benchmark_data_magna115_10867_object744535:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b14f0500 con 0 > -6383> 2018-07-11 07:51:12.619401 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:753b99bc:::benchmark_data_magna115_10867_object696436:0,end:5:753bd033:::benchmark_data_magna115_10867_object744535:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98696f400 con 0 > -6382> 2018-07-11 07:51:12.745811 7ffacbbd1700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15263 conn(0x55d985d44000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.5 seq 229 0x55d9bf805400 osd_ping(ping e501 stamp 2018-07-11 07:51:13.976495) v4 > -6381> 2018-07-11 07:51:12.745817 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15263 conn(0x55d985d45800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.5 seq 229 0x55d9829e5600 osd_ping(ping e501 stamp 2018-07-11 07:51:13.976495) v4 > -6380> 2018-07-11 07:51:12.745834 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 <== osd.5 10.8.128.113:0/15263 229 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:13.976495) v4 ==== 2004+0+0 (216817506 0 0) 0x55d9bf805400 con 0x55d985d44000 > -6379> 2018-07-11 07:51:12.745839 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.5 10.8.128.113:0/15263 229 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:13.976495) v4 ==== 2004+0+0 (216817506 0 0) 0x55d9829e5600 con 0x55d985d45800 > -6378> 2018-07-11 07:51:12.745844 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:13.976495) v4 -- 0x55d9b60de600 con 0 > -6377> 2018-07-11 07:51:12.745874 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:13.976495) v4 -- 0x55d9b3db2600 con 0 > -6376> 2018-07-11 07:51:12.794356 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 887 0x55d9b4a94000 rep_scrubmap(5.2e e501 from shard 2) v1 > -6375> 2018-07-11 07:51:12.794378 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 887 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12480 (1269249638 0 947014476) 0x55d9b4a94000 con 0x55d986bf0800 > -6374> 2018-07-11 07:51:12.843491 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 881 0x55d9b4a94240 rep_scrubmap(5.2e e501 from shard 4) v1 > -6373> 2018-07-11 07:51:12.843510 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 881 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12480 (3686815630 0 947014476) 0x55d9b4a94240 con 0x55d986bec000 > -6372> 2018-07-11 07:51:12.885834 7ffacbbd1700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.108:0/11967 conn(0x55d986d82800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.8 seq 226 0x55d9bf805800 osd_ping(ping e501 stamp 2018-07-11 07:51:12.641371) v4 > -6371> 2018-07-11 07:51:12.885840 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.108:0/11967 conn(0x55d986db0000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.8 seq 226 0x55d9869bfa00 osd_ping(ping e501 stamp 2018-07-11 07:51:12.641371) v4 > -6370> 2018-07-11 07:51:12.885855 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 <== osd.8 10.8.128.108:0/11967 226 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:12.641371) v4 ==== 2004+0+0 (2517613141 0 0) 0x55d9bf805800 con 0x55d986d82800 > -6369> 2018-07-11 07:51:12.885856 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.8 10.8.128.108:0/11967 226 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:12.641371) v4 ==== 2004+0+0 (2517613141 0 0) 0x55d9869bfa00 con 0x55d986db0000 > -6368> 2018-07-11 07:51:12.885865 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:12.641371) v4 -- 0x55d9b60de600 con 0 > -6367> 2018-07-11 07:51:12.885897 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:12.641371) v4 -- 0x55d9b3db2600 con 0 > -6366> 2018-07-11 07:51:12.931026 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -6365> 2018-07-11 07:51:12.931053 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -6364> 2018-07-11 07:51:12.931113 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -6363> 2018-07-11 07:51:12.931609 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -6362> 2018-07-11 07:51:12.959686 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6801/15919 conn(0x55d986c3f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=29 cs=1 l=0). rx osd.11 seq 226 0x55d9bf805400 MOSDScrubReserve(10.f REQUEST e501) v1 > -6361> 2018-07-11 07:51:12.959706 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.11 10.8.128.114:6801/15919 226 ==== MOSDScrubReserve(10.f REQUEST e501) v1 ==== 43+0+0 (2892966580 0 0) 0x55d9bf805400 con 0x55d986c3f800 > -6360> 2018-07-11 07:51:12.959747 7ffab53e3700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.114:6801/15919 -- MOSDScrubReserve(10.f REJECT e501) v1 -- 0x55d9c4c1aa00 con 0 > -6359> 2018-07-11 07:51:12.959993 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6801/15919 conn(0x55d986c3f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=29 cs=1 l=0). rx osd.11 seq 227 0x55d9b60de600 MOSDScrubReserve(10.f RELEASE e501) v1 > -6358> 2018-07-11 07:51:12.960008 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.11 10.8.128.114:6801/15919 227 ==== MOSDScrubReserve(10.f RELEASE e501) v1 ==== 43+0+0 (1668468918 0 0) 0x55d9b60de600 con 0x55d986c3f800 > -6357> 2018-07-11 07:51:12.977843 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:753bd033:::benchmark_data_magna115_10867_object744535:0,end:5:753c2080:::benchmark_data_magna115_10867_object2774516:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b14f0280 con 0 > -6356> 2018-07-11 07:51:12.977893 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:753bd033:::benchmark_data_magna115_10867_object744535:0,end:5:753c2080:::benchmark_data_magna115_10867_object2774516:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b14f0000 con 0 > -6355> 2018-07-11 07:51:12.980980 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15287 conn(0x55d981240800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 256 0x55d986b6d400 osd_ping(ping e501 stamp 2018-07-11 07:51:14.211590) v4 > -6354> 2018-07-11 07:51:12.981001 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.4 10.8.128.113:0/15287 256 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:14.211590) v4 ==== 2004+0+0 (2920980734 0 0) 0x55d986b6d400 con 0x55d981240800 > -6353> 2018-07-11 07:51:12.981013 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:14.211590) v4 -- 0x55d9c9123200 con 0 > -6352> 2018-07-11 07:51:12.981012 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15287 conn(0x55d98123f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 256 0x55d9829e5600 osd_ping(ping e501 stamp 2018-07-11 07:51:14.211590) v4 > -6351> 2018-07-11 07:51:12.981033 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.4 10.8.128.113:0/15287 256 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:14.211590) v4 ==== 2004+0+0 (2920980734 0 0) 0x55d9829e5600 con 0x55d98123f000 > -6350> 2018-07-11 07:51:12.981047 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:14.211590) v4 -- 0x55d9b3db2600 con 0 > -6349> 2018-07-11 07:51:13.198875 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 888 0x55d9b4a94480 rep_scrubmap(5.2e e501 from shard 2) v1 > -6348> 2018-07-11 07:51:13.198895 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 888 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12472 (1269249638 0 3784602362) 0x55d9b4a94480 con 0x55d986bf0800 > -6347> 2018-07-11 07:51:13.209993 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 882 0x55d9b4a946c0 rep_scrubmap(5.2e e501 from shard 4) v1 > -6346> 2018-07-11 07:51:13.210013 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 882 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12472 (3686815630 0 3784602362) 0x55d9b4a946c0 con 0x55d986bec000 > -6345> 2018-07-11 07:51:13.288698 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -6344> 2018-07-11 07:51:13.288724 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -6343> 2018-07-11 07:51:13.288779 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -6342> 2018-07-11 07:51:13.289272 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -6341> 2018-07-11 07:51:13.311124 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'66755,epoch:501/498,start:5:753c2080:::benchmark_data_magna115_10867_object2774516:0,end:5:753c8f2f:::benchmark_data_magna115_10867_object1062243:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1499900 con 0 > -6340> 2018-07-11 07:51:13.311174 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'66755,epoch:501/498,start:5:753c2080:::benchmark_data_magna115_10867_object2774516:0,end:5:753c8f2f:::benchmark_data_magna115_10867_object1062243:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1499680 con 0 > -6339> 2018-07-11 07:51:13.481447 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15287 conn(0x55d98123f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 257 0x55d9869bfa00 osd_ping(ping e501 stamp 2018-07-11 07:51:14.712031) v4 > -6338> 2018-07-11 07:51:13.481470 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.4 10.8.128.113:0/15287 257 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:14.712031) v4 ==== 2004+0+0 (2837708350 0 0) 0x55d9869bfa00 con 0x55d98123f000 > -6337> 2018-07-11 07:51:13.481480 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:14.712031) v4 -- 0x55d9b3db2600 con 0 > -6336> 2018-07-11 07:51:13.481472 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15287 conn(0x55d981240800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 257 0x55d986b6d400 osd_ping(ping e501 stamp 2018-07-11 07:51:14.712031) v4 > -6335> 2018-07-11 07:51:13.481502 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.4 10.8.128.113:0/15287 257 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:14.712031) v4 ==== 2004+0+0 (2837708350 0 0) 0x55d986b6d400 con 0x55d981240800 > -6334> 2018-07-11 07:51:13.481514 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:14.712031) v4 -- 0x55d9c9123200 con 0 > -6333> 2018-07-11 07:51:13.521366 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 889 0x55d9b4a94900 rep_scrubmap(5.2e e501 from shard 2) v1 > -6332> 2018-07-11 07:51:13.521386 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 889 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12468 (1269249638 0 2087541392) 0x55d9b4a94900 con 0x55d986bf0800 > -6331> 2018-07-11 07:51:13.537634 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 883 0x55d9b4a94b40 rep_scrubmap(5.2e e501 from shard 4) v1 > -6330> 2018-07-11 07:51:13.537654 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 883 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12468 (3686815630 0 2087541392) 0x55d9b4a94b40 con 0x55d986bec000 > -6329> 2018-07-11 07:51:13.654012 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -6328> 2018-07-11 07:51:13.654037 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -6327> 2018-07-11 07:51:13.654094 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -6326> 2018-07-11 07:51:13.654597 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -6325> 2018-07-11 07:51:13.703074 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67071,epoch:501/498,start:5:753c8f2f:::benchmark_data_magna115_10867_object1062243:0,end:5:753cf5dc:::benchmark_data_magna115_10867_object978711:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498500 con 0 > -6324> 2018-07-11 07:51:13.703126 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67071,epoch:501/498,start:5:753c8f2f:::benchmark_data_magna115_10867_object1062243:0,end:5:753cf5dc:::benchmark_data_magna115_10867_object978711:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498c80 con 0 > -6323> 2018-07-11 07:51:13.712941 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12650 conn(0x55d986d21800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 237 0x55d9bf805800 osd_ping(ping e501 stamp 2018-07-11 07:51:13.760334) v4 > -6322> 2018-07-11 07:51:13.712963 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.3 10.8.128.106:0/12650 237 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:13.760334) v4 ==== 2004+0+0 (2403886048 0 0) 0x55d9bf805800 con 0x55d986d21800 > -6321> 2018-07-11 07:51:13.712973 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:13.760334) v4 -- 0x55d9b67d8c00 con 0 > -6320> 2018-07-11 07:51:13.712969 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12650 conn(0x55d986d23000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 237 0x55d9829e5600 osd_ping(ping e501 stamp 2018-07-11 07:51:13.760334) v4 > -6319> 2018-07-11 07:51:13.712991 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.3 10.8.128.106:0/12650 237 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:13.760334) v4 ==== 2004+0+0 (2403886048 0 0) 0x55d9829e5600 con 0x55d986d23000 > -6318> 2018-07-11 07:51:13.713002 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:13.760334) v4 -- 0x55d9b3db2600 con 0 > -6317> 2018-07-11 07:51:13.715384 7ffab1bdc700 5 osd.10 501 heartbeat: osd_stat(788 GB used, 142 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist []) > -6316> 2018-07-11 07:51:13.715397 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6802/15255 -- osd_ping(ping e501 stamp 2018-07-11 07:51:13.715396) v4 -- 0x55d986d40a00 con 0 > -6315> 2018-07-11 07:51:13.715432 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6803/15255 -- osd_ping(ping e501 stamp 2018-07-11 07:51:13.715396) v4 -- 0x55d9829e5a00 con 0 > -6314> 2018-07-11 07:51:13.715446 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6806/12677 -- osd_ping(ping e501 stamp 2018-07-11 07:51:13.715396) v4 -- 0x55d9c3382800 con 0 > -6313> 2018-07-11 07:51:13.715459 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6807/12677 -- osd_ping(ping e501 stamp 2018-07-11 07:51:13.715396) v4 -- 0x55d986d41600 con 0 > -6312> 2018-07-11 07:51:13.715482 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6808/12651 -- osd_ping(ping e501 stamp 2018-07-11 07:51:13.715396) v4 -- 0x55d982d30400 con 0 > -6311> 2018-07-11 07:51:13.715494 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6810/12651 -- osd_ping(ping e501 stamp 2018-07-11 07:51:13.715396) v4 -- 0x55d9b4388200 con 0 > -6310> 2018-07-11 07:51:13.715504 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6809/12650 -- osd_ping(ping e501 stamp 2018-07-11 07:51:13.715396) v4 -- 0x55d9bf805000 con 0 > -6309> 2018-07-11 07:51:13.715516 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6811/12650 -- osd_ping(ping e501 stamp 2018-07-11 07:51:13.715396) v4 -- 0x55d9bdba4200 con 0 > -6308> 2018-07-11 07:51:13.715528 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6810/15287 -- osd_ping(ping e501 stamp 2018-07-11 07:51:13.715396) v4 -- 0x55d982cdf400 con 0 > -6307> 2018-07-11 07:51:13.715539 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6811/15287 -- osd_ping(ping e501 stamp 2018-07-11 07:51:13.715396) v4 -- 0x55d982d31c00 con 0 > -6306> 2018-07-11 07:51:13.715557 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6806/15263 -- osd_ping(ping e501 stamp 2018-07-11 07:51:13.715396) v4 -- 0x55d9b3db3600 con 0 > -6305> 2018-07-11 07:51:13.715570 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6807/15263 -- osd_ping(ping e501 stamp 2018-07-11 07:51:13.715396) v4 -- 0x55d9bcd72800 con 0 > -6304> 2018-07-11 07:51:13.715582 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.115:6802/13323 -- osd_ping(ping e501 stamp 2018-07-11 07:51:13.715396) v4 -- 0x55d9b60df600 con 0 > -6303> 2018-07-11 07:51:13.715598 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.115:6803/13323 -- osd_ping(ping e501 stamp 2018-07-11 07:51:13.715396) v4 -- 0x55d9c908f600 con 0 > -6302> 2018-07-11 07:51:13.715613 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6806/15885 -- osd_ping(ping e501 stamp 2018-07-11 07:51:13.715396) v4 -- 0x55d982d30600 con 0 > -6301> 2018-07-11 07:51:13.715626 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6807/15885 -- osd_ping(ping e501 stamp 2018-07-11 07:51:13.715396) v4 -- 0x55d9bf804600 con 0 > -6300> 2018-07-11 07:51:13.715638 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.108:6802/11967 -- osd_ping(ping e501 stamp 2018-07-11 07:51:13.715396) v4 -- 0x55d9829e5200 con 0 > -6299> 2018-07-11 07:51:13.715654 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.108:6803/11967 -- osd_ping(ping e501 stamp 2018-07-11 07:51:13.715396) v4 -- 0x55d986c85000 con 0 > -6298> 2018-07-11 07:51:13.715666 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.107:6806/12617 -- osd_ping(ping e501 stamp 2018-07-11 07:51:13.715396) v4 -- 0x55d9bdba4800 con 0 > -6297> 2018-07-11 07:51:13.715679 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.107:6807/12617 -- osd_ping(ping e501 stamp 2018-07-11 07:51:13.715396) v4 -- 0x55d986c22000 con 0 > -6296> 2018-07-11 07:51:13.715691 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6802/15919 -- osd_ping(ping e501 stamp 2018-07-11 07:51:13.715396) v4 -- 0x55d9c3383a00 con 0 > -6295> 2018-07-11 07:51:13.715703 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6803/15919 -- osd_ping(ping e501 stamp 2018-07-11 07:51:13.715396) v4 -- 0x55d9bcd72c00 con 0 > -6294> 2018-07-11 07:51:13.715860 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.107:6806/12617 conn(0x55d986bd2000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.9 seq 237 0x55d9869bfa00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:13.715396) v4 > -6293> 2018-07-11 07:51:13.715882 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.9 10.8.128.107:6806/12617 237 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:13.715396) v4 ==== 2004+0+0 (3513732205 0 0) 0x55d9869bfa00 con 0x55d986bd2000 > -6292> 2018-07-11 07:51:13.715889 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.107:6807/12617 conn(0x55d986bd3800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.9 seq 237 0x55d9c4c1aa00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:13.715396) v4 > -6291> 2018-07-11 07:51:13.715909 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.9 10.8.128.107:6807/12617 237 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:13.715396) v4 ==== 2004+0+0 (3513732205 0 0) 0x55d9c4c1aa00 con 0x55d986bd3800 > -6290> 2018-07-11 07:51:13.715949 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6810/12651 conn(0x55d986b9b800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.2 seq 237 0x55d9869bfa00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:13.715396) v4 > -6289> 2018-07-11 07:51:13.715967 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.2 10.8.128.106:6810/12651 237 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:13.715396) v4 ==== 2004+0+0 (3513732205 0 0) 0x55d9869bfa00 con 0x55d986b9b800 > -6288> 2018-07-11 07:51:13.715958 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6807/12677 conn(0x55d986b59800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.1 seq 237 0x55d986b6d400 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:13.715396) v4 > -6287> 2018-07-11 07:51:13.715979 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6806/12677 conn(0x55d986b58000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.1 seq 237 0x55d9c4c1aa00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:13.715396) v4 > -6286> 2018-07-11 07:51:13.715998 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.1 10.8.128.106:6806/12677 237 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:13.715396) v4 ==== 2004+0+0 (3513732205 0 0) 0x55d9c4c1aa00 con 0x55d986b58000 > -6285> 2018-07-11 07:51:13.716001 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6803/15255 conn(0x55d986bc9800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.0 seq 237 0x55d9869bfa00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:13.715396) v4 > -6284> 2018-07-11 07:51:13.715983 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.1 10.8.128.106:6807/12677 237 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:13.715396) v4 ==== 2004+0+0 (3513732205 0 0) 0x55d986b6d400 con 0x55d986b59800 > -6283> 2018-07-11 07:51:13.716011 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.0 10.8.128.113:6803/15255 237 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:13.715396) v4 ==== 2004+0+0 (3513732205 0 0) 0x55d9869bfa00 con 0x55d986bc9800 > -6282> 2018-07-11 07:51:13.716022 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6809/12650 conn(0x55d986bcb000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 237 0x55d9c4c1aa00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:13.715396) v4 > -6281> 2018-07-11 07:51:13.716033 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.3 10.8.128.106:6809/12650 237 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:13.715396) v4 ==== 2004+0+0 (3513732205 0 0) 0x55d9c4c1aa00 con 0x55d986bcb000 > -6280> 2018-07-11 07:51:13.716043 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6811/15287 conn(0x55d986bb0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 237 0x55d9869bfa00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:13.715396) v4 > -6279> 2018-07-11 07:51:13.716069 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.4 10.8.128.113:6811/15287 237 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:13.715396) v4 ==== 2004+0+0 (3513732205 0 0) 0x55d9869bfa00 con 0x55d986bb0800 > -6278> 2018-07-11 07:51:13.716066 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6811/12650 conn(0x55d986bcc800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 237 0x55d986b6d400 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:13.715396) v4 > -6277> 2018-07-11 07:51:13.716072 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6806/15263 conn(0x55d986b9d000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.5 seq 237 0x55d9c4c1aa00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:13.715396) v4 > -6276> 2018-07-11 07:51:13.716084 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.5 10.8.128.113:6806/15263 237 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:13.715396) v4 ==== 2004+0+0 (3513732205 0 0) 0x55d9c4c1aa00 con 0x55d986b9d000 > -6275> 2018-07-11 07:51:13.716084 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.3 10.8.128.106:6811/12650 237 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:13.715396) v4 ==== 2004+0+0 (3513732205 0 0) 0x55d986b6d400 con 0x55d986bcc800 > -6274> 2018-07-11 07:51:13.716116 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6802/15255 conn(0x55d986bc8000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.0 seq 237 0x55d986b6d400 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:13.715396) v4 > -6273> 2018-07-11 07:51:13.716129 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.0 10.8.128.113:6802/15255 237 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:13.715396) v4 ==== 2004+0+0 (3513732205 0 0) 0x55d986b6d400 con 0x55d986bc8000 > -6272> 2018-07-11 07:51:13.716174 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6807/15263 conn(0x55d986b9e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.5 seq 237 0x55d986b6d400 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:13.715396) v4 > -6271> 2018-07-11 07:51:13.716192 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.5 10.8.128.113:6807/15263 237 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:13.715396) v4 ==== 2004+0+0 (3513732205 0 0) 0x55d986b6d400 con 0x55d986b9e800 > -6270> 2018-07-11 07:51:13.716200 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.115:6802/13323 conn(0x55d986bc1000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 237 0x55d9869bfa00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:13.715396) v4 > -6269> 2018-07-11 07:51:13.716207 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6807/15885 conn(0x55d986bad800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.7 seq 237 0x55d9c4c1aa00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:13.715396) v4 > -6268> 2018-07-11 07:51:13.716224 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.6 10.8.128.115:6802/13323 237 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:13.715396) v4 ==== 2004+0+0 (3513732205 0 0) 0x55d9869bfa00 con 0x55d986bc1000 > -6267> 2018-07-11 07:51:13.716226 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.7 10.8.128.114:6807/15885 237 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:13.715396) v4 ==== 2004+0+0 (3513732205 0 0) 0x55d9c4c1aa00 con 0x55d986bad800 > -6266> 2018-07-11 07:51:13.716236 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6810/15287 conn(0x55d986baf000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 237 0x55d986b6d400 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:13.715396) v4 > -6265> 2018-07-11 07:51:13.716264 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.4 10.8.128.113:6810/15287 237 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:13.715396) v4 ==== 2004+0+0 (3513732205 0 0) 0x55d986b6d400 con 0x55d986baf000 > -6264> 2018-07-11 07:51:13.716283 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6806/15885 conn(0x55d986bac000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.7 seq 237 0x55d9869bfa00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:13.715396) v4 > -6263> 2018-07-11 07:51:13.716283 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.108:6802/11967 conn(0x55d986bbe000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.8 seq 237 0x55d9c4c1aa00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:13.715396) v4 > -6262> 2018-07-11 07:51:13.716307 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.7 10.8.128.114:6806/15885 237 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:13.715396) v4 ==== 2004+0+0 (3513732205 0 0) 0x55d9869bfa00 con 0x55d986bac000 > -6261> 2018-07-11 07:51:13.716313 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.8 10.8.128.108:6802/11967 237 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:13.715396) v4 ==== 2004+0+0 (3513732205 0 0) 0x55d9c4c1aa00 con 0x55d986bbe000 > -6260> 2018-07-11 07:51:13.716315 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6808/12651 conn(0x55d986b9a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.2 seq 237 0x55d986b6d400 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:13.715396) v4 > -6259> 2018-07-11 07:51:13.716335 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.2 10.8.128.106:6808/12651 237 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:13.715396) v4 ==== 2004+0+0 (3513732205 0 0) 0x55d986b6d400 con 0x55d986b9a000 > -6258> 2018-07-11 07:51:13.716358 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.115:6803/13323 conn(0x55d986bc2800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 237 0x55d9c4c1aa00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:13.715396) v4 > -6257> 2018-07-11 07:51:13.716358 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6802/15919 conn(0x55d986b5b000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.11 seq 237 0x55d9869bfa00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:13.715396) v4 > -6256> 2018-07-11 07:51:13.716376 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.108:6803/11967 conn(0x55d986bbf800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.8 seq 237 0x55d986b6d400 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:13.715396) v4 > -6255> 2018-07-11 07:51:13.716385 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.6 10.8.128.115:6803/13323 237 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:13.715396) v4 ==== 2004+0+0 (3513732205 0 0) 0x55d9c4c1aa00 con 0x55d986bc2800 > -6254> 2018-07-11 07:51:13.716389 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.11 10.8.128.114:6802/15919 237 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:13.715396) v4 ==== 2004+0+0 (3513732205 0 0) 0x55d9869bfa00 con 0x55d986b5b000 > -6253> 2018-07-11 07:51:13.716392 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.8 10.8.128.108:6803/11967 237 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:13.715396) v4 ==== 2004+0+0 (3513732205 0 0) 0x55d986b6d400 con 0x55d986bbf800 > -6252> 2018-07-11 07:51:13.716437 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6803/15919 conn(0x55d986b5c800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.11 seq 237 0x55d9c4c1aa00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:13.715396) v4 > -6251> 2018-07-11 07:51:13.716452 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.11 10.8.128.114:6803/15919 237 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:13.715396) v4 ==== 2004+0+0 (3513732205 0 0) 0x55d9c4c1aa00 con 0x55d986b5c800 > -6250> 2018-07-11 07:51:13.945436 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 890 0x55d9b4a94d80 rep_scrubmap(5.2e e501 from shard 2) v1 > -6249> 2018-07-11 07:51:13.945454 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 890 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12468 (1269249638 0 1223486699) 0x55d9b4a94d80 con 0x55d986bf0800 > -6248> 2018-07-11 07:51:13.946138 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 884 0x55d9b4a94fc0 rep_scrubmap(5.2e e501 from shard 4) v1 > -6247> 2018-07-11 07:51:13.946156 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 884 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12468 (3686815630 0 1223486699) 0x55d9b4a94fc0 con 0x55d986bec000 > -6246> 2018-07-11 07:51:14.017033 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -6245> 2018-07-11 07:51:14.017056 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -6244> 2018-07-11 07:51:14.017111 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -6243> 2018-07-11 07:51:14.017570 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -6242> 2018-07-11 07:51:14.069659 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67848,epoch:501/498,start:5:753cf5dc:::benchmark_data_magna115_10867_object978711:0,end:5:753d54fb:::benchmark_data_magna115_10867_object1519513:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1499b80 con 0 > -6241> 2018-07-11 07:51:14.069711 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67848,epoch:501/498,start:5:753cf5dc:::benchmark_data_magna115_10867_object978711:0,end:5:753d54fb:::benchmark_data_magna115_10867_object1519513:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1499180 con 0 > -6240> 2018-07-11 07:51:14.279281 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 891 0x55d9b4a95200 rep_scrubmap(5.2e e501 from shard 2) v1 > -6239> 2018-07-11 07:51:14.279301 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 891 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12458 (1269249638 0 379038092) 0x55d9b4a95200 con 0x55d986bf0800 > -6238> 2018-07-11 07:51:14.294682 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 885 0x55d9b4a95440 rep_scrubmap(5.2e e501 from shard 4) v1 > -6237> 2018-07-11 07:51:14.294702 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 885 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12458 (3686815630 0 379038092) 0x55d9b4a95440 con 0x55d986bec000 > -6236> 2018-07-11 07:51:14.489862 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.114:0/15919 conn(0x55d986df5000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.11 seq 246 0x55d9869bfa00 osd_ping(ping e501 stamp 2018-07-11 07:51:15.119950) v4 > -6235> 2018-07-11 07:51:14.489862 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.114:0/15919 conn(0x55d986df3800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.11 seq 246 0x55d9c4c1aa00 osd_ping(ping e501 stamp 2018-07-11 07:51:15.119950) v4 > -6234> 2018-07-11 07:51:14.489892 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.11 10.8.128.114:0/15919 246 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:15.119950) v4 ==== 2004+0+0 (601998507 0 0) 0x55d9869bfa00 con 0x55d986df5000 > -6233> 2018-07-11 07:51:14.489894 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.11 10.8.128.114:0/15919 246 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:15.119950) v4 ==== 2004+0+0 (601998507 0 0) 0x55d9c4c1aa00 con 0x55d986df3800 > -6232> 2018-07-11 07:51:14.489909 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:15.119950) v4 -- 0x55d9c3383a00 con 0 > -6231> 2018-07-11 07:51:14.489937 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:15.119950) v4 -- 0x55d9bcd72c00 con 0 > -6230> 2018-07-11 07:51:14.506449 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -6229> 2018-07-11 07:51:14.506475 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -6228> 2018-07-11 07:51:14.506531 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -6227> 2018-07-11 07:51:14.506994 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -6226> 2018-07-11 07:51:14.528293 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'66849,epoch:501/498,start:5:753d54fb:::benchmark_data_magna115_10867_object1519513:0,end:5:753db365:::benchmark_data_magna115_10867_object3540628:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1499400 con 0 > -6225> 2018-07-11 07:51:14.528340 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'66849,epoch:501/498,start:5:753d54fb:::benchmark_data_magna115_10867_object1519513:0,end:5:753db365:::benchmark_data_magna115_10867_object3540628:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498f00 con 0 > -6224> 2018-07-11 07:51:14.731055 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 892 0x55d9b4a95680 rep_scrubmap(5.2e e501 from shard 2) v1 > -6223> 2018-07-11 07:51:14.731074 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 892 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12470 (1269249638 0 2442684041) 0x55d9b4a95680 con 0x55d986bf0800 > -6222> 2018-07-11 07:51:14.767638 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 886 0x55d9b4a958c0 rep_scrubmap(5.2e e501 from shard 4) v1 > -6221> 2018-07-11 07:51:14.767657 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 886 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12470 (3686815630 0 2442684041) 0x55d9b4a958c0 con 0x55d986bec000 > -6220> 2018-07-11 07:51:14.958818 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6805/15263 conn(0x55d986c41000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=31 cs=1 l=0). rx osd.5 seq 2810 0x55d986b6d400 MOSDScrubReserve(6.15 REQUEST e501) v1 > -6219> 2018-07-11 07:51:14.958836 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.5 10.8.128.113:6805/15263 2810 ==== MOSDScrubReserve(6.15 REQUEST e501) v1 ==== 43+0+0 (3942742026 0 0) 0x55d986b6d400 con 0x55d986c41000 > -6218> 2018-07-11 07:51:14.969099 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6805/15263 -- MOSDScrubReserve(6.15 REJECT e501) v1 -- 0x55d9829e4200 con 0 > -6217> 2018-07-11 07:51:14.969152 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -6216> 2018-07-11 07:51:14.969172 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -6215> 2018-07-11 07:51:14.969218 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -6214> 2018-07-11 07:51:14.969592 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6805/15263 conn(0x55d986c41000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=31 cs=1 l=0). rx osd.5 seq 2811 0x55d9829e4200 MOSDScrubReserve(6.15 RELEASE e501) v1 > -6213> 2018-07-11 07:51:14.969613 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.5 10.8.128.113:6805/15263 2811 ==== MOSDScrubReserve(6.15 RELEASE e501) v1 ==== 43+0+0 (605848072 0 0) 0x55d9829e4200 con 0x55d986c41000 > -6212> 2018-07-11 07:51:14.969707 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -6211> 2018-07-11 07:51:15.011579 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:753db365:::benchmark_data_magna115_10867_object3540628:0,end:5:753e1c25:::benchmark_data_magna115_10867_object1842596:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98696ec80 con 0 > -6210> 2018-07-11 07:51:15.011638 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:753db365:::benchmark_data_magna115_10867_object3540628:0,end:5:753e1c25:::benchmark_data_magna115_10867_object1842596:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98696f680 con 0 > -6209> 2018-07-11 07:51:15.144339 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.107:0/12617 conn(0x55d98123c000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 233 0x55d986c85000 osd_ping(ping e501 stamp 2018-07-11 07:51:15.143988) v4 > -6208> 2018-07-11 07:51:15.144360 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.107:0/12617 conn(0x55d98123d800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 233 0x55d9c3383400 osd_ping(ping e501 stamp 2018-07-11 07:51:15.143988) v4 > -6207> 2018-07-11 07:51:15.144372 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.9 10.8.128.107:0/12617 233 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:15.143988) v4 ==== 2004+0+0 (624879247 0 0) 0x55d986c85000 con 0x55d98123c000 > -6206> 2018-07-11 07:51:15.144373 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.9 10.8.128.107:0/12617 233 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:15.143988) v4 ==== 2004+0+0 (624879247 0 0) 0x55d9c3383400 con 0x55d98123d800 > -6205> 2018-07-11 07:51:15.144391 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:15.143988) v4 -- 0x55d9bcd72800 con 0 > -6204> 2018-07-11 07:51:15.144420 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:15.143988) v4 -- 0x55d9bcd72c00 con 0 > -6203> 2018-07-11 07:51:15.211217 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 893 0x55d9b4a95b00 rep_scrubmap(5.2e e501 from shard 2) v1 > -6202> 2018-07-11 07:51:15.211238 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 893 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12472 (1269249638 0 4042538038) 0x55d9b4a95b00 con 0x55d986bf0800 > -6201> 2018-07-11 07:51:15.243552 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 887 0x55d9b4a95d40 rep_scrubmap(5.2e e501 from shard 4) v1 > -6200> 2018-07-11 07:51:15.243573 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 887 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12472 (3686815630 0 4042538038) 0x55d9b4a95d40 con 0x55d986bec000 > -6199> 2018-07-11 07:51:15.358297 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -6198> 2018-07-11 07:51:15.358320 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -6197> 2018-07-11 07:51:15.358377 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -6196> 2018-07-11 07:51:15.358837 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -6195> 2018-07-11 07:51:15.411873 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:753e1c25:::benchmark_data_magna115_10867_object1842596:0,end:5:753e6fca:::benchmark_data_magna115_11707_object302694:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b14f0a00 con 0 > -6194> 2018-07-11 07:51:15.411924 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:753e1c25:::benchmark_data_magna115_10867_object1842596:0,end:5:753e6fca:::benchmark_data_magna115_11707_object302694:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98696ef00 con 0 > -6193> 2018-07-11 07:51:15.413320 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12650 conn(0x55d986d23000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 238 0x55d9829e5600 osd_ping(ping e501 stamp 2018-07-11 07:51:15.460762) v4 > -6192> 2018-07-11 07:51:15.413344 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.3 10.8.128.106:0/12650 238 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:15.460762) v4 ==== 2004+0+0 (2482988528 0 0) 0x55d9829e5600 con 0x55d986d23000 > -6191> 2018-07-11 07:51:15.413353 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:15.460762) v4 -- 0x55d9c3383a00 con 0 > -6190> 2018-07-11 07:51:15.413426 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12650 conn(0x55d986d21800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 238 0x55d9c4c1aa00 osd_ping(ping e501 stamp 2018-07-11 07:51:15.460762) v4 > -6189> 2018-07-11 07:51:15.413446 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.3 10.8.128.106:0/12650 238 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:15.460762) v4 ==== 2004+0+0 (2482988528 0 0) 0x55d9c4c1aa00 con 0x55d986d21800 > -6188> 2018-07-11 07:51:15.413463 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:15.460762) v4 -- 0x55d9bcd72c00 con 0 > -6187> 2018-07-11 07:51:15.640398 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 894 0x55d995e98000 rep_scrubmap(5.2e e501 from shard 2) v1 > -6186> 2018-07-11 07:51:15.640421 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 894 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12474 (1269249638 0 298471392) 0x55d995e98000 con 0x55d986bf0800 > -6185> 2018-07-11 07:51:15.679633 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 888 0x55d995e98240 rep_scrubmap(5.2e e501 from shard 4) v1 > -6184> 2018-07-11 07:51:15.679653 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 888 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12474 (3686815630 0 298471392) 0x55d995e98240 con 0x55d986bec000 > -6183> 2018-07-11 07:51:15.679796 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -6182> 2018-07-11 07:51:15.679822 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -6181> 2018-07-11 07:51:15.679867 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -6180> 2018-07-11 07:51:15.680349 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -6179> 2018-07-11 07:51:15.707664 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:753e6fca:::benchmark_data_magna115_11707_object302694:0,end:5:753ef624:::benchmark_data_magna115_10867_object2770332:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498a00 con 0 > -6178> 2018-07-11 07:51:15.707714 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:753e6fca:::benchmark_data_magna115_11707_object302694:0,end:5:753ef624:::benchmark_data_magna115_10867_object2770332:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98696f900 con 0 > -6177> 2018-07-11 07:51:15.931746 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 889 0x55d995e98480 rep_scrubmap(5.2e e501 from shard 4) v1 > -6176> 2018-07-11 07:51:15.931765 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 889 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12460 (3686815630 0 241563193) 0x55d995e98480 con 0x55d986bec000 > -6175> 2018-07-11 07:51:15.945327 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 895 0x55d995e986c0 rep_scrubmap(5.2e e501 from shard 2) v1 > -6174> 2018-07-11 07:51:15.945346 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 895 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12460 (1269249638 0 241563193) 0x55d995e986c0 con 0x55d986bf0800 > -6173> 2018-07-11 07:51:15.960048 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6801/15919 conn(0x55d986c3f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=29 cs=1 l=0). rx osd.11 seq 228 0x55d9c3383400 MOSDScrubReserve(10.f REQUEST e501) v1 > -6172> 2018-07-11 07:51:15.960066 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.11 10.8.128.114:6801/15919 228 ==== MOSDScrubReserve(10.f REQUEST e501) v1 ==== 43+0+0 (2892966580 0 0) 0x55d9c3383400 con 0x55d986c3f800 > -6171> 2018-07-11 07:51:15.960151 7ffab53e3700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.114:6801/15919 -- MOSDScrubReserve(10.f REJECT e501) v1 -- 0x55d9b60de600 con 0 > -6170> 2018-07-11 07:51:15.960464 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6801/15919 conn(0x55d986c3f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=29 cs=1 l=0). rx osd.11 seq 229 0x55d9b60de600 MOSDScrubReserve(10.f RELEASE e501) v1 > -6169> 2018-07-11 07:51:15.960480 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.11 10.8.128.114:6801/15919 229 ==== MOSDScrubReserve(10.f RELEASE e501) v1 ==== 43+0+0 (1668468918 0 0) 0x55d9b60de600 con 0x55d986c3f800 > -6168> 2018-07-11 07:51:16.015850 7ffab1bdc700 5 osd.10 501 heartbeat: osd_stat(788 GB used, 142 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,2]) > -6167> 2018-07-11 07:51:16.015864 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6802/15255 -- osd_ping(ping e501 stamp 2018-07-11 07:51:16.015862) v4 -- 0x55d986d4e600 con 0 > -6166> 2018-07-11 07:51:16.015904 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6803/15255 -- osd_ping(ping e501 stamp 2018-07-11 07:51:16.015862) v4 -- 0x55d980457a00 con 0 > -6165> 2018-07-11 07:51:16.015920 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6806/12677 -- osd_ping(ping e501 stamp 2018-07-11 07:51:16.015862) v4 -- 0x55d9b67d8a00 con 0 > -6164> 2018-07-11 07:51:16.015935 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6807/12677 -- osd_ping(ping e501 stamp 2018-07-11 07:51:16.015862) v4 -- 0x55d980f5a000 con 0 > -6163> 2018-07-11 07:51:16.015948 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6808/12651 -- osd_ping(ping e501 stamp 2018-07-11 07:51:16.015862) v4 -- 0x55d9b9ac2000 con 0 > -6162> 2018-07-11 07:51:16.015958 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6810/12651 -- osd_ping(ping e501 stamp 2018-07-11 07:51:16.015862) v4 -- 0x55d9b60df000 con 0 > -6161> 2018-07-11 07:51:16.015970 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6809/12650 -- osd_ping(ping e501 stamp 2018-07-11 07:51:16.015862) v4 -- 0x55d9c4c1b600 con 0 > -6160> 2018-07-11 07:51:16.015983 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6811/12650 -- osd_ping(ping e501 stamp 2018-07-11 07:51:16.015862) v4 -- 0x55d9b40a4000 con 0 > -6159> 2018-07-11 07:51:16.016002 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6810/15287 -- osd_ping(ping e501 stamp 2018-07-11 07:51:16.015862) v4 -- 0x55d980f5a800 con 0 > -6158> 2018-07-11 07:51:16.016015 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6811/15287 -- osd_ping(ping e501 stamp 2018-07-11 07:51:16.015862) v4 -- 0x55d9b60df400 con 0 > -6157> 2018-07-11 07:51:16.016029 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6806/15263 -- osd_ping(ping e501 stamp 2018-07-11 07:51:16.015862) v4 -- 0x55d9869be000 con 0 > -6156> 2018-07-11 07:51:16.016043 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6807/15263 -- osd_ping(ping e501 stamp 2018-07-11 07:51:16.015862) v4 -- 0x55d9829e4c00 con 0 > -6155> 2018-07-11 07:51:16.016056 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.115:6802/13323 -- osd_ping(ping e501 stamp 2018-07-11 07:51:16.015862) v4 -- 0x55d9829e4e00 con 0 > -6154> 2018-07-11 07:51:16.016069 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.115:6803/13323 -- osd_ping(ping e501 stamp 2018-07-11 07:51:16.015862) v4 -- 0x55d986d4f800 con 0 > -6153> 2018-07-11 07:51:16.016082 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6806/15885 -- osd_ping(ping e501 stamp 2018-07-11 07:51:16.015862) v4 -- 0x55d9b60dfa00 con 0 > -6152> 2018-07-11 07:51:16.016095 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6807/15885 -- osd_ping(ping e501 stamp 2018-07-11 07:51:16.015862) v4 -- 0x55d986c22c00 con 0 > -6151> 2018-07-11 07:51:16.016107 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.108:6802/11967 -- osd_ping(ping e501 stamp 2018-07-11 07:51:16.015862) v4 -- 0x55d980f5a400 con 0 > -6150> 2018-07-11 07:51:16.016117 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.108:6803/11967 -- osd_ping(ping e501 stamp 2018-07-11 07:51:16.015862) v4 -- 0x55d9c9123a00 con 0 > -6149> 2018-07-11 07:51:16.016129 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.107:6806/12617 -- osd_ping(ping e501 stamp 2018-07-11 07:51:16.015862) v4 -- 0x55d986d4fe00 con 0 > -6148> 2018-07-11 07:51:16.016142 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.107:6807/12617 -- osd_ping(ping e501 stamp 2018-07-11 07:51:16.015862) v4 -- 0x55d9c74e1800 con 0 > -6147> 2018-07-11 07:51:16.016153 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6802/15919 -- osd_ping(ping e501 stamp 2018-07-11 07:51:16.015862) v4 -- 0x55d9869bec00 con 0 > -6146> 2018-07-11 07:51:16.016190 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6803/15919 -- osd_ping(ping e501 stamp 2018-07-11 07:51:16.015862) v4 -- 0x55d9b67d8400 con 0 > -6145> 2018-07-11 07:51:16.016376 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.107:6806/12617 conn(0x55d986bd2000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.9 seq 238 0x55d9829e5600 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:16.015862) v4 > -6144> 2018-07-11 07:51:16.016397 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.9 10.8.128.107:6806/12617 238 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:16.015862) v4 ==== 2004+0+0 (2514281481 0 0) 0x55d9829e5600 con 0x55d986bd2000 > -6143> 2018-07-11 07:51:16.016384 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6806/12677 conn(0x55d986b58000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.1 seq 238 0x55d9bcd72c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:16.015862) v4 > -6142> 2018-07-11 07:51:16.016418 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6808/12651 conn(0x55d986b9a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.2 seq 238 0x55d9bcd79000 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:16.015862) v4 > -6141> 2018-07-11 07:51:16.016434 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.2 10.8.128.106:6808/12651 238 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:16.015862) v4 ==== 2004+0+0 (2514281481 0 0) 0x55d9bcd79000 con 0x55d986b9a000 > -6140> 2018-07-11 07:51:16.016430 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.1 10.8.128.106:6806/12677 238 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:16.015862) v4 ==== 2004+0+0 (2514281481 0 0) 0x55d9bcd72c00 con 0x55d986b58000 > -6139> 2018-07-11 07:51:16.016435 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6803/15255 conn(0x55d986bc9800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.0 seq 238 0x55d9829e5600 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:16.015862) v4 > -6138> 2018-07-11 07:51:16.016463 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.0 10.8.128.113:6803/15255 238 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:16.015862) v4 ==== 2004+0+0 (2514281481 0 0) 0x55d9829e5600 con 0x55d986bc9800 > -6137> 2018-07-11 07:51:16.016478 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6802/15255 conn(0x55d986bc8000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.0 seq 238 0x55d9bcd79000 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:16.015862) v4 > -6136> 2018-07-11 07:51:16.016492 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.0 10.8.128.113:6802/15255 238 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:16.015862) v4 ==== 2004+0+0 (2514281481 0 0) 0x55d9bcd79000 con 0x55d986bc8000 > -6135> 2018-07-11 07:51:16.016497 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6810/12651 conn(0x55d986b9b800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.2 seq 238 0x55d9829e5600 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:16.015862) v4 > -6134> 2018-07-11 07:51:16.016509 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.2 10.8.128.106:6810/12651 238 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:16.015862) v4 ==== 2004+0+0 (2514281481 0 0) 0x55d9829e5600 con 0x55d986b9b800 > -6133> 2018-07-11 07:51:16.016482 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.107:6807/12617 conn(0x55d986bd3800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.9 seq 238 0x55d9bcd72c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:16.015862) v4 > -6132> 2018-07-11 07:51:16.016525 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6811/12650 conn(0x55d986bcc800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 238 0x55d9bcd79000 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:16.015862) v4 > -6131> 2018-07-11 07:51:16.016535 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.3 10.8.128.106:6811/12650 238 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:16.015862) v4 ==== 2004+0+0 (2514281481 0 0) 0x55d9bcd79000 con 0x55d986bcc800 > -6130> 2018-07-11 07:51:16.016533 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.9 10.8.128.107:6807/12617 238 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:16.015862) v4 ==== 2004+0+0 (2514281481 0 0) 0x55d9bcd72c00 con 0x55d986bd3800 > -6129> 2018-07-11 07:51:16.016550 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6811/15287 conn(0x55d986bb0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 238 0x55d9829e5600 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:16.015862) v4 > -6128> 2018-07-11 07:51:16.016564 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.4 10.8.128.113:6811/15287 238 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:16.015862) v4 ==== 2004+0+0 (2514281481 0 0) 0x55d9829e5600 con 0x55d986bb0800 > -6127> 2018-07-11 07:51:16.016565 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6810/15287 conn(0x55d986baf000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 238 0x55d9bcd79000 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:16.015862) v4 > -6126> 2018-07-11 07:51:16.016575 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.4 10.8.128.113:6810/15287 238 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:16.015862) v4 ==== 2004+0+0 (2514281481 0 0) 0x55d9bcd79000 con 0x55d986baf000 > -6125> 2018-07-11 07:51:16.016579 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6806/15263 conn(0x55d986b9d000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.5 seq 238 0x55d9bcd72c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:16.015862) v4 > -6124> 2018-07-11 07:51:16.016600 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.5 10.8.128.113:6806/15263 238 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:16.015862) v4 ==== 2004+0+0 (2514281481 0 0) 0x55d9bcd72c00 con 0x55d986b9d000 > -6123> 2018-07-11 07:51:16.016604 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6807/12677 conn(0x55d986b59800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.1 seq 238 0x55d9bcd79000 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:16.015862) v4 > -6122> 2018-07-11 07:51:16.016613 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.1 10.8.128.106:6807/12677 238 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:16.015862) v4 ==== 2004+0+0 (2514281481 0 0) 0x55d9bcd79000 con 0x55d986b59800 > -6121> 2018-07-11 07:51:16.016643 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6807/15263 conn(0x55d986b9e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.5 seq 238 0x55d9bcd79000 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:16.015862) v4 > -6120> 2018-07-11 07:51:16.016647 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6809/12650 conn(0x55d986bcb000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 238 0x55d9bcd72c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:16.015862) v4 > -6119> 2018-07-11 07:51:16.016655 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.5 10.8.128.113:6807/15263 238 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:16.015862) v4 ==== 2004+0+0 (2514281481 0 0) 0x55d9bcd79000 con 0x55d986b9e800 > -6118> 2018-07-11 07:51:16.016660 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.3 10.8.128.106:6809/12650 238 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:16.015862) v4 ==== 2004+0+0 (2514281481 0 0) 0x55d9bcd72c00 con 0x55d986bcb000 > -6117> 2018-07-11 07:51:16.016735 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.108:6802/11967 conn(0x55d986bbe000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.8 seq 238 0x55d9bcd72c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:16.015862) v4 > -6116> 2018-07-11 07:51:16.016729 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.115:6802/13323 conn(0x55d986bc1000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 238 0x55d9829e5600 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:16.015862) v4 > -6115> 2018-07-11 07:51:16.016772 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.6 10.8.128.115:6802/13323 238 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:16.015862) v4 ==== 2004+0+0 (2514281481 0 0) 0x55d9829e5600 con 0x55d986bc1000 > -6114> 2018-07-11 07:51:16.016770 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.8 10.8.128.108:6802/11967 238 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:16.015862) v4 ==== 2004+0+0 (2514281481 0 0) 0x55d9bcd72c00 con 0x55d986bbe000 > -6113> 2018-07-11 07:51:16.016780 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.108:6803/11967 conn(0x55d986bbf800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.8 seq 238 0x55d9bcd79000 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:16.015862) v4 > -6112> 2018-07-11 07:51:16.016799 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.8 10.8.128.108:6803/11967 238 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:16.015862) v4 ==== 2004+0+0 (2514281481 0 0) 0x55d9bcd79000 con 0x55d986bbf800 > -6111> 2018-07-11 07:51:16.016816 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6802/15919 conn(0x55d986b5b000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.11 seq 238 0x55d9829e5600 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:16.015862) v4 > -6110> 2018-07-11 07:51:16.016829 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.11 10.8.128.114:6802/15919 238 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:16.015862) v4 ==== 2004+0+0 (2514281481 0 0) 0x55d9829e5600 con 0x55d986b5b000 > -6109> 2018-07-11 07:51:16.016825 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.115:6803/13323 conn(0x55d986bc2800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 238 0x55d9bcd72c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:16.015862) v4 > -6108> 2018-07-11 07:51:16.016840 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.6 10.8.128.115:6803/13323 238 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:16.015862) v4 ==== 2004+0+0 (2514281481 0 0) 0x55d9bcd72c00 con 0x55d986bc2800 > -6107> 2018-07-11 07:51:16.016850 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6806/15885 conn(0x55d986bac000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.7 seq 238 0x55d9829e5600 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:16.015862) v4 > -6106> 2018-07-11 07:51:16.016860 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.7 10.8.128.114:6806/15885 238 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:16.015862) v4 ==== 2004+0+0 (2514281481 0 0) 0x55d9829e5600 con 0x55d986bac000 > -6105> 2018-07-11 07:51:16.016886 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6807/15885 conn(0x55d986bad800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.7 seq 238 0x55d9bcd72c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:16.015862) v4 > -6104> 2018-07-11 07:51:16.016900 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.7 10.8.128.114:6807/15885 238 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:16.015862) v4 ==== 2004+0+0 (2514281481 0 0) 0x55d9bcd72c00 con 0x55d986bad800 > -6103> 2018-07-11 07:51:16.016940 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6803/15919 conn(0x55d986b5c800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.11 seq 238 0x55d9bcd72c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:16.015862) v4 > -6102> 2018-07-11 07:51:16.016954 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.11 10.8.128.114:6803/15919 238 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:16.015862) v4 ==== 2004+0+0 (2514281481 0 0) 0x55d9bcd72c00 con 0x55d986b5c800 > -6101> 2018-07-11 07:51:16.018936 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -6100> 2018-07-11 07:51:16.018960 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -6099> 2018-07-11 07:51:16.019007 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -6098> 2018-07-11 07:51:16.019480 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -6097> 2018-07-11 07:51:16.036849 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67458,epoch:501/498,start:5:753ef624:::benchmark_data_magna115_10867_object2770332:0,end:5:753f5c72:::benchmark_data_magna115_10867_object3730615:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98696f180 con 0 > -6096> 2018-07-11 07:51:16.036886 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67458,epoch:501/498,start:5:753ef624:::benchmark_data_magna115_10867_object2770332:0,end:5:753f5c72:::benchmark_data_magna115_10867_object3730615:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98696e000 con 0 > -6095> 2018-07-11 07:51:16.099622 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.115:0/13323 conn(0x55d986d7e000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 247 0x55d9bcd72c00 osd_ping(ping e501 stamp 2018-07-11 07:51:16.445212) v4 > -6094> 2018-07-11 07:51:16.099643 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.6 10.8.128.115:0/13323 247 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:16.445212) v4 ==== 2004+0+0 (160801700 0 0) 0x55d9bcd72c00 con 0x55d986d7e000 > -6093> 2018-07-11 07:51:16.099653 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:16.445212) v4 -- 0x55d9b67d8400 con 0 > -6092> 2018-07-11 07:51:16.099663 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.115:0/13323 conn(0x55d986d24800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 247 0x55d986c85000 osd_ping(ping e501 stamp 2018-07-11 07:51:16.445212) v4 > -6091> 2018-07-11 07:51:16.099688 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.6 10.8.128.115:0/13323 247 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:16.445212) v4 ==== 2004+0+0 (160801700 0 0) 0x55d986c85000 con 0x55d986d24800 > -6090> 2018-07-11 07:51:16.099699 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:16.445212) v4 -- 0x55d9c9123a00 con 0 > -6089> 2018-07-11 07:51:16.262951 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 896 0x55d995e98900 rep_scrubmap(5.2e e501 from shard 2) v1 > -6088> 2018-07-11 07:51:16.262971 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 896 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12464 (1269249638 0 2151866525) 0x55d995e98900 con 0x55d986bf0800 > -6087> 2018-07-11 07:51:16.267119 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 890 0x55d995e98b40 rep_scrubmap(5.2e e501 from shard 4) v1 > -6086> 2018-07-11 07:51:16.267138 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 890 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12464 (3686815630 0 2151866525) 0x55d995e98b40 con 0x55d986bec000 > -6085> 2018-07-11 07:51:16.282015 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -6084> 2018-07-11 07:51:16.282037 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -6083> 2018-07-11 07:51:16.282092 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -6082> 2018-07-11 07:51:16.282556 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -6081> 2018-07-11 07:51:16.317799 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:753f5c72:::benchmark_data_magna115_10867_object3730615:0,end:5:753fc729:::benchmark_data_magna115_10867_object2915536:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b14f0500 con 0 > -6080> 2018-07-11 07:51:16.317850 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:753f5c72:::benchmark_data_magna115_10867_object3730615:0,end:5:753fc729:::benchmark_data_magna115_10867_object2915536:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498780 con 0 > -6079> 2018-07-11 07:51:16.319726 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12651 conn(0x55d986db3000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.2 seq 244 0x55d9c4c1aa00 osd_ping(ping e501 stamp 2018-07-11 07:51:16.366991) v4 > -6078> 2018-07-11 07:51:16.319753 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.2 10.8.128.106:0/12651 244 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:16.366991) v4 ==== 2004+0+0 (2062535127 0 0) 0x55d9c4c1aa00 con 0x55d986db3000 > -6077> 2018-07-11 07:51:16.319748 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12651 conn(0x55d986db1800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.2 seq 244 0x55d986c85000 osd_ping(ping e501 stamp 2018-07-11 07:51:16.366991) v4 > -6076> 2018-07-11 07:51:16.319767 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:16.366991) v4 -- 0x55d9b67d8400 con 0 > -6075> 2018-07-11 07:51:16.319769 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.2 10.8.128.106:0/12651 244 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:16.366991) v4 ==== 2004+0+0 (2062535127 0 0) 0x55d986c85000 con 0x55d986db1800 > -6074> 2018-07-11 07:51:16.319799 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:16.366991) v4 -- 0x55d9c9123a00 con 0 > -6073> 2018-07-11 07:51:16.386292 7ffacbbd1700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.108:0/11967 conn(0x55d986d82800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.8 seq 227 0x55d9bcd72c00 osd_ping(ping e501 stamp 2018-07-11 07:51:16.141823) v4 > -6072> 2018-07-11 07:51:16.386313 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 <== osd.8 10.8.128.108:0/11967 227 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:16.141823) v4 ==== 2004+0+0 (3175205672 0 0) 0x55d9bcd72c00 con 0x55d986d82800 > -6071> 2018-07-11 07:51:16.386303 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.108:0/11967 conn(0x55d986db0000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.8 seq 227 0x55d9829e5600 osd_ping(ping e501 stamp 2018-07-11 07:51:16.141823) v4 > -6070> 2018-07-11 07:51:16.386323 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:16.141823) v4 -- 0x55d9b67d8400 con 0 > -6069> 2018-07-11 07:51:16.386324 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.8 10.8.128.108:0/11967 227 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:16.141823) v4 ==== 2004+0+0 (3175205672 0 0) 0x55d9829e5600 con 0x55d986db0000 > -6068> 2018-07-11 07:51:16.386349 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:16.141823) v4 -- 0x55d9869bec00 con 0 > -6067> 2018-07-11 07:51:16.529209 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 897 0x55d995e98d80 rep_scrubmap(5.2e e501 from shard 2) v1 > -6066> 2018-07-11 07:51:16.529228 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 897 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12470 (1269249638 0 1615162115) 0x55d995e98d80 con 0x55d986bf0800 > -6065> 2018-07-11 07:51:16.551070 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 891 0x55d995e98fc0 rep_scrubmap(5.2e e501 from shard 4) v1 > -6064> 2018-07-11 07:51:16.551090 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 891 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12470 (3686815630 0 1615162115) 0x55d995e98fc0 con 0x55d986bec000 > -6063> 2018-07-11 07:51:16.551231 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -6062> 2018-07-11 07:51:16.551265 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -6061> 2018-07-11 07:51:16.551313 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -6060> 2018-07-11 07:51:16.551786 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -6059> 2018-07-11 07:51:16.578818 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67281,epoch:501/498,start:5:753fc729:::benchmark_data_magna115_10867_object2915536:0,end:5:75402fb4:::benchmark_data_magna115_11707_object239279:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455900 con 0 > -6058> 2018-07-11 07:51:16.578868 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67281,epoch:501/498,start:5:753fc729:::benchmark_data_magna115_10867_object2915536:0,end:5:75402fb4:::benchmark_data_magna115_11707_object239279:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455b80 con 0 > -6057> 2018-07-11 07:51:16.803721 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 892 0x55d995e99200 rep_scrubmap(5.2e e501 from shard 4) v1 > -6056> 2018-07-11 07:51:16.803742 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 892 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12476 (3686815630 0 4059659953) 0x55d995e99200 con 0x55d986bec000 > -6055> 2018-07-11 07:51:16.811592 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 898 0x55d995e99440 rep_scrubmap(5.2e e501 from shard 2) v1 > -6054> 2018-07-11 07:51:16.811612 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 898 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12476 (1269249638 0 4059659953) 0x55d995e99440 con 0x55d986bf0800 > -6053> 2018-07-11 07:51:16.835983 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 2992 0x55d9c4c1aa00 MOSDScrubReserve(11.3s5 REQUEST e501) v1 > -6052> 2018-07-11 07:51:16.836002 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 2992 ==== MOSDScrubReserve(11.3s5 REQUEST e501) v1 ==== 43+0+0 (3260170403 0 0) 0x55d9c4c1aa00 con 0x55d986c7e000 > -6051> 2018-07-11 07:51:16.836075 7ffab3be0700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e501) v1 -- 0x55d9b3db3c00 con 0 > -6050> 2018-07-11 07:51:16.836416 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 2993 0x55d9b3db3c00 MOSDScrubReserve(11.3s5 RELEASE e501) v1 > -6049> 2018-07-11 07:51:16.836435 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 2993 ==== MOSDScrubReserve(11.3s5 RELEASE e501) v1 ==== 43+0+0 (223328929 0 0) 0x55d9b3db3c00 con 0x55d986c7e000 > -6048> 2018-07-11 07:51:16.864923 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15255 conn(0x55d986d20000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.0 seq 247 0x55d986c85000 osd_ping(ping e501 stamp 2018-07-11 07:51:18.095570) v4 > -6047> 2018-07-11 07:51:16.864924 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15255 conn(0x55d986c82800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.0 seq 247 0x55d9869bfa00 osd_ping(ping e501 stamp 2018-07-11 07:51:18.095570) v4 > -6046> 2018-07-11 07:51:16.864947 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.0 10.8.128.113:0/15255 247 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:18.095570) v4 ==== 2004+0+0 (1865129709 0 0) 0x55d986c85000 con 0x55d986d20000 > -6045> 2018-07-11 07:51:16.864951 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.0 10.8.128.113:0/15255 247 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:18.095570) v4 ==== 2004+0+0 (1865129709 0 0) 0x55d9869bfa00 con 0x55d986c82800 > -6044> 2018-07-11 07:51:16.864957 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:18.095570) v4 -- 0x55d9c9123a00 con 0 > -6043> 2018-07-11 07:51:16.864978 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:18.095570) v4 -- 0x55d9869bec00 con 0 > -6042> 2018-07-11 07:51:16.885500 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -6041> 2018-07-11 07:51:16.885524 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -6040> 2018-07-11 07:51:16.885578 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -6039> 2018-07-11 07:51:16.886023 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -6038> 2018-07-11 07:51:16.922133 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'66989,epoch:501/498,start:5:75402fb4:::benchmark_data_magna115_11707_object239279:0,end:5:7540b797:::benchmark_data_magna115_10867_object759526:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498000 con 0 > -6037> 2018-07-11 07:51:16.922185 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'66989,epoch:501/498,start:5:75402fb4:::benchmark_data_magna115_11707_object239279:0,end:5:7540b797:::benchmark_data_magna115_10867_object759526:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455680 con 0 > -6036> 2018-07-11 07:51:16.962352 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12677 conn(0x55d986db4800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.1 seq 250 0x55d9829e5600 osd_ping(ping e501 stamp 2018-07-11 07:51:17.009822) v4 > -6035> 2018-07-11 07:51:16.962355 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12677 conn(0x55d986df2000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.1 seq 250 0x55d986c85000 osd_ping(ping e501 stamp 2018-07-11 07:51:17.009822) v4 > -6034> 2018-07-11 07:51:16.962377 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.1 10.8.128.106:0/12677 250 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:17.009822) v4 ==== 2004+0+0 (215419497 0 0) 0x55d9829e5600 con 0x55d986db4800 > -6033> 2018-07-11 07:51:16.962381 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.1 10.8.128.106:0/12677 250 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:17.009822) v4 ==== 2004+0+0 (215419497 0 0) 0x55d986c85000 con 0x55d986df2000 > -6032> 2018-07-11 07:51:16.962387 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:17.009822) v4 -- 0x55d9869bec00 con 0 > -6031> 2018-07-11 07:51:16.962414 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:17.009822) v4 -- 0x55d9c9123a00 con 0 > -6030> 2018-07-11 07:51:16.981859 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15287 conn(0x55d981240800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 258 0x55d9bcd79000 osd_ping(ping e501 stamp 2018-07-11 07:51:18.212572) v4 > -6029> 2018-07-11 07:51:16.981883 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.4 10.8.128.113:0/15287 258 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:18.212572) v4 ==== 2004+0+0 (380290289 0 0) 0x55d9bcd79000 con 0x55d981240800 > -6028> 2018-07-11 07:51:16.981892 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:18.212572) v4 -- 0x55d9c9123a00 con 0 > -6027> 2018-07-11 07:51:16.981905 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15287 conn(0x55d98123f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 258 0x55d9869bfa00 osd_ping(ping e501 stamp 2018-07-11 07:51:18.212572) v4 > -6026> 2018-07-11 07:51:16.981926 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.4 10.8.128.113:0/15287 258 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:18.212572) v4 ==== 2004+0+0 (380290289 0 0) 0x55d9869bfa00 con 0x55d98123f000 > -6025> 2018-07-11 07:51:16.981937 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:18.212572) v4 -- 0x55d9869bec00 con 0 > -6024> 2018-07-11 07:51:17.113918 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12650 conn(0x55d986d21800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 239 0x55d9bcd72c00 osd_ping(ping e501 stamp 2018-07-11 07:51:17.161336) v4 > -6023> 2018-07-11 07:51:17.113939 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.3 10.8.128.106:0/12650 239 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:17.161336) v4 ==== 2004+0+0 (2654786459 0 0) 0x55d9bcd72c00 con 0x55d986d21800 > -6022> 2018-07-11 07:51:17.113948 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:17.161336) v4 -- 0x55d9c74e1800 con 0 > -6021> 2018-07-11 07:51:17.113944 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12650 conn(0x55d986d23000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 239 0x55d9829e5600 osd_ping(ping e501 stamp 2018-07-11 07:51:17.161336) v4 > -6020> 2018-07-11 07:51:17.113962 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.3 10.8.128.106:0/12650 239 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:17.161336) v4 ==== 2004+0+0 (2654786459 0 0) 0x55d9829e5600 con 0x55d986d23000 > -6019> 2018-07-11 07:51:17.113975 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:17.161336) v4 -- 0x55d9869bec00 con 0 > -6018> 2018-07-11 07:51:17.121233 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 893 0x55d995e99680 rep_scrubmap(5.2e e501 from shard 4) v1 > -6017> 2018-07-11 07:51:17.121260 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 893 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12478 (3686815630 0 2791678132) 0x55d995e99680 con 0x55d986bec000 > -6016> 2018-07-11 07:51:17.135734 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 899 0x55d995e998c0 rep_scrubmap(5.2e e501 from shard 2) v1 > -6015> 2018-07-11 07:51:17.135754 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 899 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12478 (1269249638 0 2791678132) 0x55d995e998c0 con 0x55d986bf0800 > -6014> 2018-07-11 07:51:17.189220 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -6013> 2018-07-11 07:51:17.189266 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -6012> 2018-07-11 07:51:17.189312 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -6011> 2018-07-11 07:51:17.189777 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -6010> 2018-07-11 07:51:17.220837 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7540b797:::benchmark_data_magna115_10867_object759526:0,end:5:754134c0:::benchmark_data_magna115_10867_object2970007:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455400 con 0 > -6009> 2018-07-11 07:51:17.220884 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7540b797:::benchmark_data_magna115_10867_object759526:0,end:5:754134c0:::benchmark_data_magna115_10867_object2970007:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498280 con 0 > -6008> 2018-07-11 07:51:17.349495 7ffab6be6700 1 -- 10.8.128.107:6800/13927 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55d9b3fdf8c0 con 0 > -6007> 2018-07-11 07:51:17.349554 7ffab6be6700 1 -- 10.8.128.107:6800/13927 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55d986cff600 con 0 > -6006> 2018-07-11 07:51:17.416101 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.114:0/15885 conn(0x55d986d81000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.7 seq 251 0x55d9bcd79000 osd_ping(ping e501 stamp 2018-07-11 07:51:18.046209) v4 > -6005> 2018-07-11 07:51:17.416123 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.7 10.8.128.114:0/15885 251 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:18.046209) v4 ==== 2004+0+0 (2339869175 0 0) 0x55d9bcd79000 con 0x55d986d81000 > -6004> 2018-07-11 07:51:17.416123 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.114:0/15885 conn(0x55d986d7f800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.7 seq 251 0x55d9869bfa00 osd_ping(ping e501 stamp 2018-07-11 07:51:18.046209) v4 > -6003> 2018-07-11 07:51:17.416139 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:18.046209) v4 -- 0x55d9c9123a00 con 0 > -6002> 2018-07-11 07:51:17.416146 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.7 10.8.128.114:0/15885 251 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:18.046209) v4 ==== 2004+0+0 (2339869175 0 0) 0x55d9869bfa00 con 0x55d986d7f800 > -6001> 2018-07-11 07:51:17.416169 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:18.046209) v4 -- 0x55d9869bec00 con 0 > -6000> 2018-07-11 07:51:17.439426 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 894 0x55d995e99b00 rep_scrubmap(5.2e e501 from shard 4) v1 > -5999> 2018-07-11 07:51:17.439445 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 894 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12472 (3686815630 0 1896185156) 0x55d995e99b00 con 0x55d986bec000 > -5998> 2018-07-11 07:51:17.444851 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.107:0/12617 conn(0x55d98123c000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 234 0x55d9bcd79000 osd_ping(ping e501 stamp 2018-07-11 07:51:17.444515) v4 > -5997> 2018-07-11 07:51:17.444873 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.9 10.8.128.107:0/12617 234 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:17.444515) v4 ==== 2004+0+0 (3246726086 0 0) 0x55d9bcd79000 con 0x55d98123c000 > -5996> 2018-07-11 07:51:17.444869 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.107:0/12617 conn(0x55d98123d800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 234 0x55d9b67d8400 osd_ping(ping e501 stamp 2018-07-11 07:51:17.444515) v4 > -5995> 2018-07-11 07:51:17.444892 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:17.444515) v4 -- 0x55d9c9123a00 con 0 > -5994> 2018-07-11 07:51:17.444897 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.9 10.8.128.107:0/12617 234 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:17.444515) v4 ==== 2004+0+0 (3246726086 0 0) 0x55d9b67d8400 con 0x55d98123d800 > -5993> 2018-07-11 07:51:17.444924 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:17.444515) v4 -- 0x55d9c74e1800 con 0 > -5992> 2018-07-11 07:51:17.446152 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15263 conn(0x55d985d45800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.5 seq 230 0x55d9829e5600 osd_ping(ping e501 stamp 2018-07-11 07:51:18.676919) v4 > -5991> 2018-07-11 07:51:17.446173 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.5 10.8.128.113:0/15263 230 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:18.676919) v4 ==== 2004+0+0 (2305179435 0 0) 0x55d9829e5600 con 0x55d985d45800 > -5990> 2018-07-11 07:51:17.446184 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:18.676919) v4 -- 0x55d9869bec00 con 0 > -5989> 2018-07-11 07:51:17.446235 7ffacbbd1700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15263 conn(0x55d985d44000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.5 seq 230 0x55d9bcd72c00 osd_ping(ping e501 stamp 2018-07-11 07:51:18.676919) v4 > -5988> 2018-07-11 07:51:17.446260 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 <== osd.5 10.8.128.113:0/15263 230 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:18.676919) v4 ==== 2004+0+0 (2305179435 0 0) 0x55d9bcd72c00 con 0x55d985d44000 > -5987> 2018-07-11 07:51:17.446269 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:18.676919) v4 -- 0x55d9c74e1800 con 0 > -5986> 2018-07-11 07:51:17.467821 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 900 0x55d995e99d40 rep_scrubmap(5.2e e501 from shard 2) v1 > -5985> 2018-07-11 07:51:17.467841 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 900 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12472 (1269249638 0 1896185156) 0x55d995e99d40 con 0x55d986bf0800 > -5984> 2018-07-11 07:51:17.535680 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -5983> 2018-07-11 07:51:17.535706 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -5982> 2018-07-11 07:51:17.535762 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -5981> 2018-07-11 07:51:17.536220 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -5980> 2018-07-11 07:51:17.561749 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6805/15885 conn(0x55d986c7f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.7 seq 2788 0x55d9bcd79000 MOSDScrubReserve(5.4 REQUEST e501) v1 > -5979> 2018-07-11 07:51:17.561770 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.7 10.8.128.114:6805/15885 2788 ==== MOSDScrubReserve(5.4 REQUEST e501) v1 ==== 43+0+0 (1753221512 0 0) 0x55d9bcd79000 con 0x55d986c7f800 > -5978> 2018-07-11 07:51:17.561831 7ffab33df700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.114:6805/15885 -- MOSDScrubReserve(5.4 REJECT e501) v1 -- 0x55d986d32200 con 0 > -5977> 2018-07-11 07:51:17.562147 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6805/15885 conn(0x55d986c7f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.7 seq 2789 0x55d986d32200 MOSDScrubReserve(5.4 RELEASE e501) v1 > -5976> 2018-07-11 07:51:17.562178 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.7 10.8.128.114:6805/15885 2789 ==== MOSDScrubReserve(5.4 RELEASE e501) v1 ==== 43+0+0 (2812146570 0 0) 0x55d986d32200 con 0x55d986c7f800 > -5975> 2018-07-11 07:51:17.579342 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67115,epoch:501/498,start:5:754134c0:::benchmark_data_magna115_10867_object2970007:0,end:5:7541a728:::benchmark_data_magna115_10867_object3273451:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454f00 con 0 > -5974> 2018-07-11 07:51:17.579381 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67115,epoch:501/498,start:5:754134c0:::benchmark_data_magna115_10867_object2970007:0,end:5:7541a728:::benchmark_data_magna115_10867_object3273451:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454780 con 0 > -5973> 2018-07-11 07:51:17.789700 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 901 0x55d9b24f0000 rep_scrubmap(5.2e e501 from shard 2) v1 > -5972> 2018-07-11 07:51:17.789721 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 901 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12478 (1269249638 0 109323192) 0x55d9b24f0000 con 0x55d986bf0800 > -5971> 2018-07-11 07:51:17.801040 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 895 0x55d9b24f0240 rep_scrubmap(5.2e e501 from shard 4) v1 > -5970> 2018-07-11 07:51:17.801061 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 895 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12478 (3686815630 0 109323192) 0x55d9b24f0240 con 0x55d986bec000 > -5969> 2018-07-11 07:51:17.801170 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -5968> 2018-07-11 07:51:17.801193 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -5967> 2018-07-11 07:51:17.801238 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -5966> 2018-07-11 07:51:17.801715 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -5965> 2018-07-11 07:51:17.815612 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67964,epoch:501/498,start:5:7541a728:::benchmark_data_magna115_10867_object3273451:0,end:5:7542042c:::benchmark_data_magna115_10867_object2778337:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455180 con 0 > -5964> 2018-07-11 07:51:17.815661 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67964,epoch:501/498,start:5:7541a728:::benchmark_data_magna115_10867_object3273451:0,end:5:7542042c:::benchmark_data_magna115_10867_object2778337:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98944b680 con 0 > -5963> 2018-07-11 07:51:18.033968 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 902 0x55d9b24f0480 rep_scrubmap(5.2e e501 from shard 2) v1 > -5962> 2018-07-11 07:51:18.033988 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 902 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12478 (1269249638 0 4177095606) 0x55d9b24f0480 con 0x55d986bf0800 > -5961> 2018-07-11 07:51:18.059309 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 896 0x55d9b24f06c0 rep_scrubmap(5.2e e501 from shard 4) v1 > -5960> 2018-07-11 07:51:18.059330 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 896 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12478 (3686815630 0 4177095606) 0x55d9b24f06c0 con 0x55d986bec000 > -5959> 2018-07-11 07:51:18.059444 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -5958> 2018-07-11 07:51:18.059466 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -5957> 2018-07-11 07:51:18.059513 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -5956> 2018-07-11 07:51:18.059975 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -5955> 2018-07-11 07:51:18.079475 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7542042c:::benchmark_data_magna115_10867_object2778337:0,end:5:754277dc:::benchmark_data_magna115_11707_object283688:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454500 con 0 > -5954> 2018-07-11 07:51:18.079513 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7542042c:::benchmark_data_magna115_10867_object2778337:0,end:5:754277dc:::benchmark_data_magna115_11707_object283688:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98944b900 con 0 > -5953> 2018-07-11 07:51:18.282669 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 903 0x55d9b24f0900 rep_scrubmap(5.2e e501 from shard 2) v1 > -5952> 2018-07-11 07:51:18.282690 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 903 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12482 (1269249638 0 3241784850) 0x55d9b24f0900 con 0x55d986bf0800 > -5951> 2018-07-11 07:51:18.295846 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 897 0x55d9b24f0b40 rep_scrubmap(5.2e e501 from shard 4) v1 > -5950> 2018-07-11 07:51:18.295866 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 897 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12482 (3686815630 0 3241784850) 0x55d9b24f0b40 con 0x55d986bec000 > -5949> 2018-07-11 07:51:18.295976 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -5948> 2018-07-11 07:51:18.295998 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -5947> 2018-07-11 07:51:18.296045 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -5946> 2018-07-11 07:51:18.296509 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -5945> 2018-07-11 07:51:18.330649 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67886,epoch:501/498,start:5:754277dc:::benchmark_data_magna115_11707_object283688:0,end:5:7542e94d:::benchmark_data_magna115_10867_object137178:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454a00 con 0 > -5944> 2018-07-11 07:51:18.330698 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67886,epoch:501/498,start:5:754277dc:::benchmark_data_magna115_11707_object283688:0,end:5:7542e94d:::benchmark_data_magna115_10867_object137178:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98944bb80 con 0 > -5943> 2018-07-11 07:51:18.549847 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 904 0x55d9b24f0d80 rep_scrubmap(5.2e e501 from shard 2) v1 > -5942> 2018-07-11 07:51:18.549868 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 904 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12462 (1269249638 0 68361663) 0x55d9b24f0d80 con 0x55d986bf0800 > -5941> 2018-07-11 07:51:18.553040 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 898 0x55d9b24f0fc0 rep_scrubmap(5.2e e501 from shard 4) v1 > -5940> 2018-07-11 07:51:18.553061 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 898 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12462 (3686815630 0 68361663) 0x55d9b24f0fc0 con 0x55d986bec000 > -5939> 2018-07-11 07:51:18.553169 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -5938> 2018-07-11 07:51:18.553191 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -5937> 2018-07-11 07:51:18.553237 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -5936> 2018-07-11 07:51:18.553707 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -5935> 2018-07-11 07:51:18.561849 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6805/15885 conn(0x55d986c7f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.7 seq 2790 0x55d9c9123a00 MOSDScrubReserve(5.4 REQUEST e501) v1 > -5934> 2018-07-11 07:51:18.561868 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.7 10.8.128.114:6805/15885 2790 ==== MOSDScrubReserve(5.4 REQUEST e501) v1 ==== 43+0+0 (1753221512 0 0) 0x55d9c9123a00 con 0x55d986c7f800 > -5933> 2018-07-11 07:51:18.561938 7ffab33df700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.114:6805/15885 -- MOSDScrubReserve(5.4 REJECT e501) v1 -- 0x55d986d32200 con 0 > -5932> 2018-07-11 07:51:18.562267 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6805/15885 conn(0x55d986c7f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.7 seq 2791 0x55d986d32200 MOSDScrubReserve(5.4 RELEASE e501) v1 > -5931> 2018-07-11 07:51:18.562282 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.7 10.8.128.114:6805/15885 2791 ==== MOSDScrubReserve(5.4 RELEASE e501) v1 ==== 43+0+0 (2812146570 0 0) 0x55d986d32200 con 0x55d986c7f800 > -5930> 2018-07-11 07:51:18.571327 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7542e94d:::benchmark_data_magna115_10867_object137178:0,end:5:754341bc:::benchmark_data_magna115_10867_object1945945:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb772280 con 0 > -5929> 2018-07-11 07:51:18.571365 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7542e94d:::benchmark_data_magna115_10867_object137178:0,end:5:754341bc:::benchmark_data_magna115_10867_object1945945:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98944b400 con 0 > -5928> 2018-07-11 07:51:18.775276 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 905 0x55d9b24f1200 rep_scrubmap(5.2e e501 from shard 2) v1 > -5927> 2018-07-11 07:51:18.775296 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 905 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12484 (1269249638 0 2012226788) 0x55d9b24f1200 con 0x55d986bf0800 > -5926> 2018-07-11 07:51:18.803194 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 899 0x55d9b24f1440 rep_scrubmap(5.2e e501 from shard 4) v1 > -5925> 2018-07-11 07:51:18.803214 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 899 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12484 (3686815630 0 2012226788) 0x55d9b24f1440 con 0x55d986bec000 > -5924> 2018-07-11 07:51:18.803326 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -5923> 2018-07-11 07:51:18.803348 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -5922> 2018-07-11 07:51:18.803395 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -5921> 2018-07-11 07:51:18.803858 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -5920> 2018-07-11 07:51:18.812406 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67003,epoch:501/498,start:5:754341bc:::benchmark_data_magna115_10867_object1945945:0,end:5:7543a9e3:::benchmark_data_magna115_10867_object3278858:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb772000 con 0 > -5919> 2018-07-11 07:51:18.812456 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67003,epoch:501/498,start:5:754341bc:::benchmark_data_magna115_10867_object1945945:0,end:5:7543a9e3:::benchmark_data_magna115_10867_object3278858:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb772a00 con 0 > -5918> 2018-07-11 07:51:18.836292 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 2994 0x55d9b67d8400 MOSDScrubReserve(11.3s5 REQUEST e501) v1 > -5917> 2018-07-11 07:51:18.836310 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 2994 ==== MOSDScrubReserve(11.3s5 REQUEST e501) v1 ==== 43+0+0 (3260170403 0 0) 0x55d9b67d8400 con 0x55d986c7e000 > -5916> 2018-07-11 07:51:18.836382 7ffab3be0700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e501) v1 -- 0x55d9b3db3c00 con 0 > -5915> 2018-07-11 07:51:18.836788 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 2995 0x55d9b3db3c00 MOSDScrubReserve(11.3s5 RELEASE e501) v1 > -5914> 2018-07-11 07:51:18.836806 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 2995 ==== MOSDScrubReserve(11.3s5 RELEASE e501) v1 ==== 43+0+0 (223328929 0 0) 0x55d9b3db3c00 con 0x55d986c7e000 > -5913> 2018-07-11 07:51:19.045440 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 900 0x55d9b24f1680 rep_scrubmap(5.2e e501 from shard 4) v1 > -5912> 2018-07-11 07:51:19.045459 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 900 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12472 (3686815630 0 3877655510) 0x55d9b24f1680 con 0x55d986bec000 > -5911> 2018-07-11 07:51:19.048333 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 906 0x55d9b24f18c0 rep_scrubmap(5.2e e501 from shard 2) v1 > -5910> 2018-07-11 07:51:19.048351 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 906 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12472 (1269249638 0 3877655510) 0x55d9b24f18c0 con 0x55d986bf0800 > -5909> 2018-07-11 07:51:19.048504 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -5908> 2018-07-11 07:51:19.048529 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -5907> 2018-07-11 07:51:19.048576 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -5906> 2018-07-11 07:51:19.049039 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -5905> 2018-07-11 07:51:19.080147 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7543a9e3:::benchmark_data_magna115_10867_object3278858:0,end:5:7543fde1:::benchmark_data_magna115_10867_object1299431:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb772500 con 0 > -5904> 2018-07-11 07:51:19.080197 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7543a9e3:::benchmark_data_magna115_10867_object3278858:0,end:5:7543fde1:::benchmark_data_magna115_10867_object1299431:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb772780 con 0 > -5903> 2018-07-11 07:51:19.220203 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12651 conn(0x55d986db3000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.2 seq 245 0x55d9bcd72c00 osd_ping(ping e501 stamp 2018-07-11 07:51:19.267632) v4 > -5902> 2018-07-11 07:51:19.220232 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.2 10.8.128.106:0/12651 245 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:19.267632) v4 ==== 2004+0+0 (3520944671 0 0) 0x55d9bcd72c00 con 0x55d986db3000 > -5901> 2018-07-11 07:51:19.220275 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:19.267632) v4 -- 0x55d980f5a400 con 0 > -5900> 2018-07-11 07:51:19.220296 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12651 conn(0x55d986db1800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.2 seq 245 0x55d986c85000 osd_ping(ping e501 stamp 2018-07-11 07:51:19.267632) v4 > -5899> 2018-07-11 07:51:19.220320 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.2 10.8.128.106:0/12651 245 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:19.267632) v4 ==== 2004+0+0 (3520944671 0 0) 0x55d986c85000 con 0x55d986db1800 > -5898> 2018-07-11 07:51:19.220338 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:19.267632) v4 -- 0x55d980f5a800 con 0 > -5897> 2018-07-11 07:51:19.268337 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 907 0x55d9b24f1b00 rep_scrubmap(5.2e e501 from shard 2) v1 > -5896> 2018-07-11 07:51:19.268355 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 907 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12480 (1269249638 0 2162334346) 0x55d9b24f1b00 con 0x55d986bf0800 > -5895> 2018-07-11 07:51:19.293919 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 901 0x55d9b24f1d40 rep_scrubmap(5.2e e501 from shard 4) v1 > -5894> 2018-07-11 07:51:19.293938 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 901 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12480 (3686815630 0 2162334346) 0x55d9b24f1d40 con 0x55d986bec000 > -5893> 2018-07-11 07:51:19.358237 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -5892> 2018-07-11 07:51:19.358269 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -5891> 2018-07-11 07:51:19.358314 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -5890> 2018-07-11 07:51:19.358749 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -5889> 2018-07-11 07:51:19.397746 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'68004,epoch:501/498,start:5:7543fde1:::benchmark_data_magna115_10867_object1299431:0,end:5:754469d7:::benchmark_data_magna115_10867_object1267307:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb772c80 con 0 > -5888> 2018-07-11 07:51:19.397784 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'68004,epoch:501/498,start:5:7543fde1:::benchmark_data_magna115_10867_object1299431:0,end:5:754469d7:::benchmark_data_magna115_10867_object1267307:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb772f00 con 0 > -5887> 2018-07-11 07:51:19.630265 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 902 0x55d9b40fa000 rep_scrubmap(5.2e e501 from shard 4) v1 > -5886> 2018-07-11 07:51:19.630284 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 902 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12476 (3686815630 0 3565606626) 0x55d9b40fa000 con 0x55d986bec000 > -5885> 2018-07-11 07:51:19.636083 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 908 0x55d9b40fa240 rep_scrubmap(5.2e e501 from shard 2) v1 > -5884> 2018-07-11 07:51:19.636102 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 908 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12476 (1269249638 0 3565606626) 0x55d9b40fa240 con 0x55d986bf0800 > -5883> 2018-07-11 07:51:19.692387 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -5882> 2018-07-11 07:51:19.692413 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -5881> 2018-07-11 07:51:19.692459 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -5880> 2018-07-11 07:51:19.692920 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -5879> 2018-07-11 07:51:19.714095 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'66683,epoch:501/498,start:5:754469d7:::benchmark_data_magna115_10867_object1267307:0,end:5:7544cdf7:::benchmark_data_magna115_10867_object3584507:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb773900 con 0 > -5878> 2018-07-11 07:51:19.714145 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'66683,epoch:501/498,start:5:754469d7:::benchmark_data_magna115_10867_object1267307:0,end:5:7544cdf7:::benchmark_data_magna115_10867_object3584507:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb773400 con 0 > -5877> 2018-07-11 07:51:19.716516 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.114:0/15885 conn(0x55d986d7f800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.7 seq 252 0x55d9869bfa00 osd_ping(ping e501 stamp 2018-07-11 07:51:20.346649) v4 > -5876> 2018-07-11 07:51:19.716543 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.7 10.8.128.114:0/15885 252 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:20.346649) v4 ==== 2004+0+0 (106919558 0 0) 0x55d9869bfa00 con 0x55d986d7f800 > -5875> 2018-07-11 07:51:19.716538 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.114:0/15885 conn(0x55d986d81000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.7 seq 252 0x55d9829e4c00 osd_ping(ping e501 stamp 2018-07-11 07:51:20.346649) v4 > -5874> 2018-07-11 07:51:19.716563 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:20.346649) v4 -- 0x55d9869bec00 con 0 > -5873> 2018-07-11 07:51:19.716568 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.7 10.8.128.114:0/15885 252 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:20.346649) v4 ==== 2004+0+0 (106919558 0 0) 0x55d9829e4c00 con 0x55d986d81000 > -5872> 2018-07-11 07:51:19.716592 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:20.346649) v4 -- 0x55d980f5a800 con 0 > -5871> 2018-07-11 07:51:19.746608 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15263 conn(0x55d985d45800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.5 seq 231 0x55d9829e5600 osd_ping(ping e501 stamp 2018-07-11 07:51:20.977387) v4 > -5870> 2018-07-11 07:51:19.746629 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.5 10.8.128.113:0/15263 231 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:20.977387) v4 ==== 2004+0+0 (411062647 0 0) 0x55d9829e5600 con 0x55d985d45800 > -5869> 2018-07-11 07:51:19.746638 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:20.977387) v4 -- 0x55d9869bec00 con 0 > -5868> 2018-07-11 07:51:19.746629 7ffacbbd1700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15263 conn(0x55d985d44000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.5 seq 231 0x55d9c74e1800 osd_ping(ping e501 stamp 2018-07-11 07:51:20.977387) v4 > -5867> 2018-07-11 07:51:19.746649 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 <== osd.5 10.8.128.113:0/15263 231 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:20.977387) v4 ==== 2004+0+0 (411062647 0 0) 0x55d9c74e1800 con 0x55d985d44000 > -5866> 2018-07-11 07:51:19.746661 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:20.977387) v4 -- 0x55d980f5a400 con 0 > -5865> 2018-07-11 07:51:19.909420 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 903 0x55d9b40fa480 rep_scrubmap(5.2e e501 from shard 4) v1 > -5864> 2018-07-11 07:51:19.909442 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 903 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12482 (3686815630 0 1829399851) 0x55d9b40fa480 con 0x55d986bec000 > -5863> 2018-07-11 07:51:19.911232 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 909 0x55d9b40fa6c0 rep_scrubmap(5.2e e501 from shard 2) v1 > -5862> 2018-07-11 07:51:19.911266 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 909 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12482 (1269249638 0 1829399851) 0x55d9b40fa6c0 con 0x55d986bf0800 > -5861> 2018-07-11 07:51:19.960639 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6801/15919 conn(0x55d986c3f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=29 cs=1 l=0). rx osd.11 seq 230 0x55d9bcd72c00 MOSDScrubReserve(10.f REQUEST e501) v1 > -5860> 2018-07-11 07:51:19.960669 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.11 10.8.128.114:6801/15919 230 ==== MOSDScrubReserve(10.f REQUEST e501) v1 ==== 43+0+0 (2892966580 0 0) 0x55d9bcd72c00 con 0x55d986c3f800 > -5859> 2018-07-11 07:51:19.960697 7ffab53e3700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.114:6801/15919 -- MOSDScrubReserve(10.f REJECT e501) v1 -- 0x55d9b60de600 con 0 > -5858> 2018-07-11 07:51:19.961058 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6801/15919 conn(0x55d986c3f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=29 cs=1 l=0). rx osd.11 seq 231 0x55d9b60de600 MOSDScrubReserve(10.f RELEASE e501) v1 > -5857> 2018-07-11 07:51:19.961072 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.11 10.8.128.114:6801/15919 231 ==== MOSDScrubReserve(10.f RELEASE e501) v1 ==== 43+0+0 (1668468918 0 0) 0x55d9b60de600 con 0x55d986c3f800 > -5856> 2018-07-11 07:51:20.014321 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12650 conn(0x55d986d23000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 240 0x55d9869bfa00 osd_ping(ping e501 stamp 2018-07-11 07:51:20.061775) v4 > -5855> 2018-07-11 07:51:20.014332 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12650 conn(0x55d986d21800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 240 0x55d9c74e1800 osd_ping(ping e501 stamp 2018-07-11 07:51:20.061775) v4 > -5854> 2018-07-11 07:51:20.014345 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.3 10.8.128.106:0/12650 240 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:20.061775) v4 ==== 2004+0+0 (1956720595 0 0) 0x55d9869bfa00 con 0x55d986d23000 > -5853> 2018-07-11 07:51:20.014350 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.3 10.8.128.106:0/12650 240 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:20.061775) v4 ==== 2004+0+0 (1956720595 0 0) 0x55d9c74e1800 con 0x55d986d21800 > -5852> 2018-07-11 07:51:20.014366 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:20.061775) v4 -- 0x55d9869bec00 con 0 > -5851> 2018-07-11 07:51:20.014400 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:20.061775) v4 -- 0x55d986c22c00 con 0 > -5850> 2018-07-11 07:51:20.247138 7ffacbbd1700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15263 conn(0x55d985d44000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.5 seq 232 0x55d980f5a400 osd_ping(ping e501 stamp 2018-07-11 07:51:21.477897) v4 > -5849> 2018-07-11 07:51:20.247148 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15263 conn(0x55d985d45800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.5 seq 232 0x55d9829e5600 osd_ping(ping e501 stamp 2018-07-11 07:51:21.477897) v4 > -5848> 2018-07-11 07:51:20.247159 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 <== osd.5 10.8.128.113:0/15263 232 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:21.477897) v4 ==== 2004+0+0 (2204912290 0 0) 0x55d980f5a400 con 0x55d985d44000 > -5847> 2018-07-11 07:51:20.247171 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:21.477897) v4 -- 0x55d986c22c00 con 0 > -5846> 2018-07-11 07:51:20.247173 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.5 10.8.128.113:0/15263 232 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:21.477897) v4 ==== 2004+0+0 (2204912290 0 0) 0x55d9829e5600 con 0x55d985d45800 > -5845> 2018-07-11 07:51:20.247187 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:21.477897) v4 -- 0x55d9869bec00 con 0 > -5844> 2018-07-11 07:51:20.345310 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.107:0/12617 conn(0x55d98123c000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 235 0x55d986c85000 osd_ping(ping e501 stamp 2018-07-11 07:51:20.344976) v4 > -5843> 2018-07-11 07:51:20.345330 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.9 10.8.128.107:0/12617 235 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:20.344976) v4 ==== 2004+0+0 (1192276395 0 0) 0x55d986c85000 con 0x55d98123c000 > -5842> 2018-07-11 07:51:20.345341 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:20.344976) v4 -- 0x55d980f5a800 con 0 > -5841> 2018-07-11 07:51:20.345344 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.107:0/12617 conn(0x55d98123d800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 235 0x55d9c74e1800 osd_ping(ping e501 stamp 2018-07-11 07:51:20.344976) v4 > -5840> 2018-07-11 07:51:20.345362 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.9 10.8.128.107:0/12617 235 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:20.344976) v4 ==== 2004+0+0 (1192276395 0 0) 0x55d9c74e1800 con 0x55d98123d800 > -5839> 2018-07-11 07:51:20.345373 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:20.344976) v4 -- 0x55d986c22c00 con 0 > -5838> 2018-07-11 07:51:20.365347 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15255 conn(0x55d986c82800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.0 seq 248 0x55d9869bfa00 osd_ping(ping e501 stamp 2018-07-11 07:51:21.596098) v4 > -5837> 2018-07-11 07:51:20.365368 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.0 10.8.128.113:0/15255 248 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:21.596098) v4 ==== 2004+0+0 (755296858 0 0) 0x55d9869bfa00 con 0x55d986c82800 > -5836> 2018-07-11 07:51:20.365369 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15255 conn(0x55d986d20000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.0 seq 248 0x55d9829e4c00 osd_ping(ping e501 stamp 2018-07-11 07:51:21.596098) v4 > -5835> 2018-07-11 07:51:20.365380 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:21.596098) v4 -- 0x55d9869bec00 con 0 > -5834> 2018-07-11 07:51:20.365386 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.0 10.8.128.113:0/15255 248 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:21.596098) v4 ==== 2004+0+0 (755296858 0 0) 0x55d9829e4c00 con 0x55d986d20000 > -5833> 2018-07-11 07:51:20.365401 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:21.596098) v4 -- 0x55d980f5a800 con 0 > -5832> 2018-07-11 07:51:20.390285 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.114:0/15919 conn(0x55d986df5000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.11 seq 247 0x55d9829e5600 osd_ping(ping e501 stamp 2018-07-11 07:51:21.020382) v4 > -5831> 2018-07-11 07:51:20.390307 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.11 10.8.128.114:0/15919 247 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:21.020382) v4 ==== 2004+0+0 (1149141301 0 0) 0x55d9829e5600 con 0x55d986df5000 > -5830> 2018-07-11 07:51:20.390299 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.114:0/15919 conn(0x55d986df3800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.11 seq 247 0x55d980f5a400 osd_ping(ping e501 stamp 2018-07-11 07:51:21.020382) v4 > -5829> 2018-07-11 07:51:20.390316 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.11 10.8.128.114:0/15919 247 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:21.020382) v4 ==== 2004+0+0 (1149141301 0 0) 0x55d980f5a400 con 0x55d986df3800 > -5828> 2018-07-11 07:51:20.390316 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:21.020382) v4 -- 0x55d9869bec00 con 0 > -5827> 2018-07-11 07:51:20.390341 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:21.020382) v4 -- 0x55d986c22c00 con 0 > -5826> 2018-07-11 07:51:20.486712 7ffacbbd1700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.108:0/11967 conn(0x55d986d82800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.8 seq 228 0x55d9c74e1800 osd_ping(ping e501 stamp 2018-07-11 07:51:20.242263) v4 > -5825> 2018-07-11 07:51:20.486721 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.108:0/11967 conn(0x55d986db0000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.8 seq 228 0x55d9869bfa00 osd_ping(ping e501 stamp 2018-07-11 07:51:20.242263) v4 > -5824> 2018-07-11 07:51:20.486734 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 <== osd.8 10.8.128.108:0/11967 228 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:20.242263) v4 ==== 2004+0+0 (2770198475 0 0) 0x55d9c74e1800 con 0x55d986d82800 > -5823> 2018-07-11 07:51:20.486738 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.8 10.8.128.108:0/11967 228 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:20.242263) v4 ==== 2004+0+0 (2770198475 0 0) 0x55d9869bfa00 con 0x55d986db0000 > -5822> 2018-07-11 07:51:20.486743 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:20.242263) v4 -- 0x55d986c22c00 con 0 > -5821> 2018-07-11 07:51:20.486764 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:20.242263) v4 -- 0x55d9869bec00 con 0 > -5820> 2018-07-11 07:51:20.800071 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.115:0/13323 conn(0x55d986d7e000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 248 0x55d980f5a400 osd_ping(ping e501 stamp 2018-07-11 07:51:21.145699) v4 > -5819> 2018-07-11 07:51:20.800092 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.6 10.8.128.115:0/13323 248 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:21.145699) v4 ==== 2004+0+0 (3510414327 0 0) 0x55d980f5a400 con 0x55d986d7e000 > -5818> 2018-07-11 07:51:20.800101 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:21.145699) v4 -- 0x55d986c22c00 con 0 > -5817> 2018-07-11 07:51:20.800102 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.115:0/13323 conn(0x55d986d24800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 248 0x55d986c85000 osd_ping(ping e501 stamp 2018-07-11 07:51:21.145699) v4 > -5816> 2018-07-11 07:51:20.800128 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.6 10.8.128.115:0/13323 248 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:21.145699) v4 ==== 2004+0+0 (3510414327 0 0) 0x55d986c85000 con 0x55d986d24800 > -5815> 2018-07-11 07:51:20.800139 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:21.145699) v4 -- 0x55d980f5a800 con 0 > -5814> 2018-07-11 07:51:20.920815 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12651 conn(0x55d986db3000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.2 seq 246 0x55d9c74e1800 osd_ping(ping e501 stamp 2018-07-11 07:51:20.968223) v4 > -5813> 2018-07-11 07:51:20.920848 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.2 10.8.128.106:0/12651 246 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:20.968223) v4 ==== 2004+0+0 (627854824 0 0) 0x55d9c74e1800 con 0x55d986db3000 > -5812> 2018-07-11 07:51:20.920861 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:20.968223) v4 -- 0x55d986c22c00 con 0 > -5811> 2018-07-11 07:51:20.920923 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12651 conn(0x55d986db1800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.2 seq 246 0x55d9829e4c00 osd_ping(ping e501 stamp 2018-07-11 07:51:20.968223) v4 > -5810> 2018-07-11 07:51:20.920944 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.2 10.8.128.106:0/12651 246 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:20.968223) v4 ==== 2004+0+0 (627854824 0 0) 0x55d9829e4c00 con 0x55d986db1800 > -5809> 2018-07-11 07:51:20.920955 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:20.968223) v4 -- 0x55d980f5a800 con 0 > -5808> 2018-07-11 07:51:20.987037 7ffacbbd1700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.108:0/11967 conn(0x55d986d82800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.8 seq 229 0x55d980f5a400 osd_ping(ping e501 stamp 2018-07-11 07:51:20.742669) v4 > -5807> 2018-07-11 07:51:20.987034 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.108:0/11967 conn(0x55d986db0000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.8 seq 229 0x55d9829e5600 osd_ping(ping e501 stamp 2018-07-11 07:51:20.742669) v4 > -5806> 2018-07-11 07:51:20.987062 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 <== osd.8 10.8.128.108:0/11967 229 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:20.742669) v4 ==== 2004+0+0 (2159448913 0 0) 0x55d980f5a400 con 0x55d986d82800 > -5805> 2018-07-11 07:51:20.987068 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.8 10.8.128.108:0/11967 229 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:20.742669) v4 ==== 2004+0+0 (2159448913 0 0) 0x55d9829e5600 con 0x55d986db0000 > -5804> 2018-07-11 07:51:20.987072 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:20.742669) v4 -- 0x55d986c22c00 con 0 > -5803> 2018-07-11 07:51:20.987105 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:20.742669) v4 -- 0x55d9869bec00 con 0 > -5802> 2018-07-11 07:51:20.994143 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 31] Generated table #36799: 2638240 keys, 74546282 bytes > -5801> 2018-07-11 07:51:20.994192 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295480994162, "cf_name": "default", "job": 31, "event": "table_file_creation", "file_number": 36799, "file_size": 74546282, "table_properties": {"data_size": 67112108, "index_size": 690003, "filter_size": 6743181, "raw_key_size": 76215828, "raw_average_key_size": 28, "raw_value_size": 31365748, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} > -5800> 2018-07-11 07:51:21.024560 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -5799> 2018-07-11 07:51:21.024590 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -5798> 2018-07-11 07:51:21.024649 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -5797> 2018-07-11 07:51:21.025117 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -5796> 2018-07-11 07:51:21.048010 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67906,epoch:501/498,start:5:7544cdf7:::benchmark_data_magna115_10867_object3584507:0,end:5:754533d6:::benchmark_data_magna115_10867_object3586147:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb773180 con 0 > -5795> 2018-07-11 07:51:21.048062 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67906,epoch:501/498,start:5:7544cdf7:::benchmark_data_magna115_10867_object3584507:0,end:5:754533d6:::benchmark_data_magna115_10867_object3586147:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb773680 con 0 > -5794> 2018-07-11 07:51:21.289299 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 910 0x55d9b40fa900 rep_scrubmap(5.2e e501 from shard 2) v1 > -5793> 2018-07-11 07:51:21.289321 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 910 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12470 (1269249638 0 483875280) 0x55d9b40fa900 con 0x55d986bf0800 > -5792> 2018-07-11 07:51:21.303512 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 904 0x55d9b40fab40 rep_scrubmap(5.2e e501 from shard 4) v1 > -5791> 2018-07-11 07:51:21.303544 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 904 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12470 (3686815630 0 483875280) 0x55d9b40fab40 con 0x55d986bec000 > -5790> 2018-07-11 07:51:21.316372 7ffab1bdc700 5 osd.10 501 heartbeat: osd_stat(788 GB used, 142 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,1,1]) > -5789> 2018-07-11 07:51:21.316385 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6802/15255 -- osd_ping(ping e501 stamp 2018-07-11 07:51:21.316384) v4 -- 0x55d982d31600 con 0 > -5788> 2018-07-11 07:51:21.316422 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6803/15255 -- osd_ping(ping e501 stamp 2018-07-11 07:51:21.316384) v4 -- 0x55d982cde200 con 0 > -5787> 2018-07-11 07:51:21.316437 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6806/12677 -- osd_ping(ping e501 stamp 2018-07-11 07:51:21.316384) v4 -- 0x55d9b9ac2200 con 0 > -5786> 2018-07-11 07:51:21.316453 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6807/12677 -- osd_ping(ping e501 stamp 2018-07-11 07:51:21.316384) v4 -- 0x55d9829e5800 con 0 > -5785> 2018-07-11 07:51:21.316477 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6808/12651 -- osd_ping(ping e501 stamp 2018-07-11 07:51:21.316384) v4 -- 0x55d9b2783400 con 0 > -5784> 2018-07-11 07:51:21.316487 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6810/12651 -- osd_ping(ping e501 stamp 2018-07-11 07:51:21.316384) v4 -- 0x55d9b3db3c00 con 0 > -5783> 2018-07-11 07:51:21.316495 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6809/12650 -- osd_ping(ping e501 stamp 2018-07-11 07:51:21.316384) v4 -- 0x55d9b67d8400 con 0 > -5782> 2018-07-11 07:51:21.316505 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6811/12650 -- osd_ping(ping e501 stamp 2018-07-11 07:51:21.316384) v4 -- 0x55d9c4c1aa00 con 0 > -5781> 2018-07-11 07:51:21.316517 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6810/15287 -- osd_ping(ping e501 stamp 2018-07-11 07:51:21.316384) v4 -- 0x55d9869be600 con 0 > -5780> 2018-07-11 07:51:21.316527 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6811/15287 -- osd_ping(ping e501 stamp 2018-07-11 07:51:21.316384) v4 -- 0x55d9b9ac3400 con 0 > -5779> 2018-07-11 07:51:21.316544 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6806/15263 -- osd_ping(ping e501 stamp 2018-07-11 07:51:21.316384) v4 -- 0x55d982cdfc00 con 0 > -5778> 2018-07-11 07:51:21.316558 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6807/15263 -- osd_ping(ping e501 stamp 2018-07-11 07:51:21.316384) v4 -- 0x55d986d96e00 con 0 > -5777> 2018-07-11 07:51:21.316569 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.115:6802/13323 -- osd_ping(ping e501 stamp 2018-07-11 07:51:21.316384) v4 -- 0x55d9b9ac3200 con 0 > -5776> 2018-07-11 07:51:21.316581 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.115:6803/13323 -- osd_ping(ping e501 stamp 2018-07-11 07:51:21.316384) v4 -- 0x55d9b67d9e00 con 0 > -5775> 2018-07-11 07:51:21.316595 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6806/15885 -- osd_ping(ping e501 stamp 2018-07-11 07:51:21.316384) v4 -- 0x55d9c908fe00 con 0 > -5774> 2018-07-11 07:51:21.316609 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6807/15885 -- osd_ping(ping e501 stamp 2018-07-11 07:51:21.316384) v4 -- 0x55d986d33e00 con 0 > -5773> 2018-07-11 07:51:21.316622 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.108:6802/11967 -- osd_ping(ping e501 stamp 2018-07-11 07:51:21.316384) v4 -- 0x55d9b60df800 con 0 > -5772> 2018-07-11 07:51:21.316634 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.108:6803/11967 -- osd_ping(ping e501 stamp 2018-07-11 07:51:21.316384) v4 -- 0x55d9829e4800 con 0 > -5771> 2018-07-11 07:51:21.316647 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.107:6806/12617 -- osd_ping(ping e501 stamp 2018-07-11 07:51:21.316384) v4 -- 0x55d9c3383c00 con 0 > -5770> 2018-07-11 07:51:21.316659 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.107:6807/12617 -- osd_ping(ping e501 stamp 2018-07-11 07:51:21.316384) v4 -- 0x55d9869d9600 con 0 > -5769> 2018-07-11 07:51:21.316673 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6802/15919 -- osd_ping(ping e501 stamp 2018-07-11 07:51:21.316384) v4 -- 0x55d9c3383200 con 0 > -5768> 2018-07-11 07:51:21.316685 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6803/15919 -- osd_ping(ping e501 stamp 2018-07-11 07:51:21.316384) v4 -- 0x55d9869bfc00 con 0 > -5767> 2018-07-11 07:51:21.316856 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.107:6807/12617 conn(0x55d986bd3800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.9 seq 239 0x55d9c74e1800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:21.316384) v4 > -5766> 2018-07-11 07:51:21.316893 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.107:6806/12617 conn(0x55d986bd2000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.9 seq 239 0x55d982cdf800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:21.316384) v4 > -5765> 2018-07-11 07:51:21.316906 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.9 10.8.128.107:6807/12617 239 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:21.316384) v4 ==== 2004+0+0 (2085071852 0 0) 0x55d9c74e1800 con 0x55d986bd3800 > -5764> 2018-07-11 07:51:21.316915 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.9 10.8.128.107:6806/12617 239 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:21.316384) v4 ==== 2004+0+0 (2085071852 0 0) 0x55d982cdf800 con 0x55d986bd2000 > -5763> 2018-07-11 07:51:21.316941 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6806/12677 conn(0x55d986b58000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.1 seq 239 0x55d9c74e1800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:21.316384) v4 > -5762> 2018-07-11 07:51:21.316947 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6808/12651 conn(0x55d986b9a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.2 seq 239 0x55d986c85000 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:21.316384) v4 > -5761> 2018-07-11 07:51:21.316967 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.2 10.8.128.106:6808/12651 239 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:21.316384) v4 ==== 2004+0+0 (2085071852 0 0) 0x55d986c85000 con 0x55d986b9a000 > -5760> 2018-07-11 07:51:21.316959 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6810/12651 conn(0x55d986b9b800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.2 seq 239 0x55d982cdf800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:21.316384) v4 > -5759> 2018-07-11 07:51:21.316954 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.1 10.8.128.106:6806/12677 239 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:21.316384) v4 ==== 2004+0+0 (2085071852 0 0) 0x55d9c74e1800 con 0x55d986b58000 > -5758> 2018-07-11 07:51:21.316986 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.2 10.8.128.106:6810/12651 239 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:21.316384) v4 ==== 2004+0+0 (2085071852 0 0) 0x55d982cdf800 con 0x55d986b9b800 > -5757> 2018-07-11 07:51:21.317012 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6809/12650 conn(0x55d986bcb000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 239 0x55d9c74e1800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:21.316384) v4 > -5756> 2018-07-11 07:51:21.317025 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.3 10.8.128.106:6809/12650 239 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:21.316384) v4 ==== 2004+0+0 (2085071852 0 0) 0x55d9c74e1800 con 0x55d986bcb000 > -5755> 2018-07-11 07:51:21.317029 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6807/12677 conn(0x55d986b59800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.1 seq 239 0x55d986c85000 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:21.316384) v4 > -5754> 2018-07-11 07:51:21.317032 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6803/15255 conn(0x55d986bc9800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.0 seq 239 0x55d982cdf800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:21.316384) v4 > -5753> 2018-07-11 07:51:21.317044 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.1 10.8.128.106:6807/12677 239 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:21.316384) v4 ==== 2004+0+0 (2085071852 0 0) 0x55d986c85000 con 0x55d986b59800 > -5752> 2018-07-11 07:51:21.317045 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.0 10.8.128.113:6803/15255 239 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:21.316384) v4 ==== 2004+0+0 (2085071852 0 0) 0x55d982cdf800 con 0x55d986bc9800 > -5751> 2018-07-11 07:51:21.317054 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6806/15263 conn(0x55d986b9d000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.5 seq 239 0x55d9c74e1800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:21.316384) v4 > -5750> 2018-07-11 07:51:21.317077 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6802/15255 conn(0x55d986bc8000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.0 seq 239 0x55d986c85000 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:21.316384) v4 > -5749> 2018-07-11 07:51:21.317080 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.5 10.8.128.113:6806/15263 239 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:21.316384) v4 ==== 2004+0+0 (2085071852 0 0) 0x55d9c74e1800 con 0x55d986b9d000 > -5748> 2018-07-11 07:51:21.317096 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.0 10.8.128.113:6802/15255 239 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:21.316384) v4 ==== 2004+0+0 (2085071852 0 0) 0x55d986c85000 con 0x55d986bc8000 > -5747> 2018-07-11 07:51:21.317110 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6811/15287 conn(0x55d986bb0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 239 0x55d982cdf800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:21.316384) v4 > -5746> 2018-07-11 07:51:21.317127 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.4 10.8.128.113:6811/15287 239 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:21.316384) v4 ==== 2004+0+0 (2085071852 0 0) 0x55d982cdf800 con 0x55d986bb0800 > -5745> 2018-07-11 07:51:21.317129 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6807/15263 conn(0x55d986b9e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.5 seq 239 0x55d986c85000 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:21.316384) v4 > -5744> 2018-07-11 07:51:21.317138 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.5 10.8.128.113:6807/15263 239 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:21.316384) v4 ==== 2004+0+0 (2085071852 0 0) 0x55d986c85000 con 0x55d986b9e800 > -5743> 2018-07-11 07:51:21.317167 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6811/12650 conn(0x55d986bcc800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 239 0x55d986c85000 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:21.316384) v4 > -5742> 2018-07-11 07:51:21.317179 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.3 10.8.128.106:6811/12650 239 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:21.316384) v4 ==== 2004+0+0 (2085071852 0 0) 0x55d986c85000 con 0x55d986bcc800 > -5741> 2018-07-11 07:51:21.317211 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.115:6802/13323 conn(0x55d986bc1000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 239 0x55d982cdf800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:21.316384) v4 > -5740> 2018-07-11 07:51:21.317229 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.6 10.8.128.115:6802/13323 239 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:21.316384) v4 ==== 2004+0+0 (2085071852 0 0) 0x55d982cdf800 con 0x55d986bc1000 > -5739> 2018-07-11 07:51:21.317232 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6810/15287 conn(0x55d986baf000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 239 0x55d986c85000 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:21.316384) v4 > -5738> 2018-07-11 07:51:21.317260 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.4 10.8.128.113:6810/15287 239 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:21.316384) v4 ==== 2004+0+0 (2085071852 0 0) 0x55d986c85000 con 0x55d986baf000 > -5737> 2018-07-11 07:51:21.317217 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.115:6803/13323 conn(0x55d986bc2800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 239 0x55d9c74e1800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:21.316384) v4 > -5736> 2018-07-11 07:51:21.317271 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.6 10.8.128.115:6803/13323 239 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:21.316384) v4 ==== 2004+0+0 (2085071852 0 0) 0x55d9c74e1800 con 0x55d986bc2800 > -5735> 2018-07-11 07:51:21.317282 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.108:6803/11967 conn(0x55d986bbf800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.8 seq 239 0x55d986c85000 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:21.316384) v4 > -5734> 2018-07-11 07:51:21.317293 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.8 10.8.128.108:6803/11967 239 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:21.316384) v4 ==== 2004+0+0 (2085071852 0 0) 0x55d986c85000 con 0x55d986bbf800 > -5733> 2018-07-11 07:51:21.317292 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6806/15885 conn(0x55d986bac000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.7 seq 239 0x55d982cdf800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:21.316384) v4 > -5732> 2018-07-11 07:51:21.317307 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.7 10.8.128.114:6806/15885 239 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:21.316384) v4 ==== 2004+0+0 (2085071852 0 0) 0x55d982cdf800 con 0x55d986bac000 > -5731> 2018-07-11 07:51:21.317310 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.108:6802/11967 conn(0x55d986bbe000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.8 seq 239 0x55d9c74e1800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:21.316384) v4 > -5730> 2018-07-11 07:51:21.317337 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.8 10.8.128.108:6802/11967 239 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:21.316384) v4 ==== 2004+0+0 (2085071852 0 0) 0x55d9c74e1800 con 0x55d986bbe000 > -5729> 2018-07-11 07:51:21.317341 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6802/15919 conn(0x55d986b5b000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.11 seq 239 0x55d982cdf800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:21.316384) v4 > -5728> 2018-07-11 07:51:21.317354 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.11 10.8.128.114:6802/15919 239 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:21.316384) v4 ==== 2004+0+0 (2085071852 0 0) 0x55d982cdf800 con 0x55d986b5b000 > -5727> 2018-07-11 07:51:21.317367 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6807/15885 conn(0x55d986bad800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.7 seq 239 0x55d9c74e1800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:21.316384) v4 > -5726> 2018-07-11 07:51:21.317379 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.7 10.8.128.114:6807/15885 239 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:21.316384) v4 ==== 2004+0+0 (2085071852 0 0) 0x55d9c74e1800 con 0x55d986bad800 > -5725> 2018-07-11 07:51:21.317409 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6803/15919 conn(0x55d986b5c800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.11 seq 239 0x55d9c74e1800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:21.316384) v4 > -5724> 2018-07-11 07:51:21.317421 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.11 10.8.128.114:6803/15919 239 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:21.316384) v4 ==== 2004+0+0 (2085071852 0 0) 0x55d9c74e1800 con 0x55d986b5c800 > -5723> 2018-07-11 07:51:21.349464 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -5722> 2018-07-11 07:51:21.349490 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -5721> 2018-07-11 07:51:21.349535 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -5720> 2018-07-11 07:51:21.349992 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -5719> 2018-07-11 07:51:21.398461 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67085,epoch:501/498,start:5:754533d6:::benchmark_data_magna115_10867_object3586147:0,end:5:75458d9d:::benchmark_data_magna115_10867_object718856:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454280 con 0 > -5718> 2018-07-11 07:51:21.398500 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67085,epoch:501/498,start:5:754533d6:::benchmark_data_magna115_10867_object3586147:0,end:5:75458d9d:::benchmark_data_magna115_10867_object718856:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454000 con 0 > -5717> 2018-07-11 07:51:21.416955 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.114:0/15885 conn(0x55d986d7f800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.7 seq 253 0x55d982cdf800 osd_ping(ping e501 stamp 2018-07-11 07:51:22.047091) v4 > -5716> 2018-07-11 07:51:21.416972 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.114:0/15885 conn(0x55d986d81000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.7 seq 253 0x55d9829e4c00 osd_ping(ping e501 stamp 2018-07-11 07:51:22.047091) v4 > -5715> 2018-07-11 07:51:21.416980 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.7 10.8.128.114:0/15885 253 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:22.047091) v4 ==== 2004+0+0 (1089768924 0 0) 0x55d982cdf800 con 0x55d986d7f800 > -5714> 2018-07-11 07:51:21.416998 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.7 10.8.128.114:0/15885 253 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:22.047091) v4 ==== 2004+0+0 (1089768924 0 0) 0x55d9829e4c00 con 0x55d986d81000 > -5713> 2018-07-11 07:51:21.417008 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:22.047091) v4 -- 0x55d9c3383200 con 0 > -5712> 2018-07-11 07:51:21.417037 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:22.047091) v4 -- 0x55d9829e4800 con 0 > -5711> 2018-07-11 07:51:21.487411 7ffacbbd1700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.108:0/11967 conn(0x55d986d82800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.8 seq 230 0x55d9c74e1800 osd_ping(ping e501 stamp 2018-07-11 07:51:21.243067) v4 > -5710> 2018-07-11 07:51:21.487421 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.108:0/11967 conn(0x55d986db0000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.8 seq 230 0x55d9829e5600 osd_ping(ping e501 stamp 2018-07-11 07:51:21.243067) v4 > -5709> 2018-07-11 07:51:21.487433 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 <== osd.8 10.8.128.108:0/11967 230 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:21.243067) v4 ==== 2004+0+0 (766367394 0 0) 0x55d9c74e1800 con 0x55d986d82800 > -5708> 2018-07-11 07:51:21.487438 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.8 10.8.128.108:0/11967 230 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:21.243067) v4 ==== 2004+0+0 (766367394 0 0) 0x55d9829e5600 con 0x55d986db0000 > -5707> 2018-07-11 07:51:21.487442 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:21.243067) v4 -- 0x55d9869bfc00 con 0 > -5706> 2018-07-11 07:51:21.487465 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:21.243067) v4 -- 0x55d9c3383200 con 0 > -5705> 2018-07-11 07:51:21.490740 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.114:0/15919 conn(0x55d986df3800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.11 seq 248 0x55d980f5a400 osd_ping(ping e501 stamp 2018-07-11 07:51:22.120886) v4 > -5704> 2018-07-11 07:51:21.490766 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.11 10.8.128.114:0/15919 248 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:22.120886) v4 ==== 2004+0+0 (1663644853 0 0) 0x55d980f5a400 con 0x55d986df3800 > -5703> 2018-07-11 07:51:21.490768 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.114:0/15919 conn(0x55d986df5000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.11 seq 248 0x55d982cdf800 osd_ping(ping e501 stamp 2018-07-11 07:51:22.120886) v4 > -5702> 2018-07-11 07:51:21.490785 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.11 10.8.128.114:0/15919 248 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:22.120886) v4 ==== 2004+0+0 (1663644853 0 0) 0x55d982cdf800 con 0x55d986df5000 > -5701> 2018-07-11 07:51:21.490780 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:22.120886) v4 -- 0x55d9869bfc00 con 0 > -5700> 2018-07-11 07:51:21.490814 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:22.120886) v4 -- 0x55d9c3383200 con 0 > -5699> 2018-07-11 07:51:21.587367 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 911 0x55d9b40fad80 rep_scrubmap(5.2e e501 from shard 2) v1 > -5698> 2018-07-11 07:51:21.587388 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 911 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12472 (1269249638 0 2822138316) 0x55d9b40fad80 con 0x55d986bf0800 > -5697> 2018-07-11 07:51:21.614486 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 905 0x55d9b40fafc0 rep_scrubmap(5.2e e501 from shard 4) v1 > -5696> 2018-07-11 07:51:21.614506 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 905 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12472 (3686815630 0 2822138316) 0x55d9b40fafc0 con 0x55d986bec000 > -5695> 2018-07-11 07:51:21.723552 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -5694> 2018-07-11 07:51:21.723589 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -5693> 2018-07-11 07:51:21.723632 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -5692> 2018-07-11 07:51:21.724087 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -5691> 2018-07-11 07:51:21.727123 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'66639,epoch:501/498,start:5:75458d9d:::benchmark_data_magna115_10867_object718856:0,end:5:7545f13c:::benchmark_data_magna115_11707_object423722:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb773b80 con 0 > -5690> 2018-07-11 07:51:21.727171 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'66639,epoch:501/498,start:5:75458d9d:::benchmark_data_magna115_10867_object718856:0,end:5:7545f13c:::benchmark_data_magna115_11707_object423722:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454c80 con 0 > -5689> 2018-07-11 07:51:21.836874 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 2996 0x55d9c74e1800 MOSDScrubReserve(11.3s5 REQUEST e501) v1 > -5688> 2018-07-11 07:51:21.836898 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 2996 ==== MOSDScrubReserve(11.3s5 REQUEST e501) v1 ==== 43+0+0 (3260170403 0 0) 0x55d9c74e1800 con 0x55d986c7e000 > -5687> 2018-07-11 07:51:21.836981 7ffab3be0700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e501) v1 -- 0x55d9bdba4600 con 0 > -5686> 2018-07-11 07:51:21.837293 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 2997 0x55d9869bfc00 MOSDScrubReserve(11.3s5 RELEASE e501) v1 > -5685> 2018-07-11 07:51:21.837324 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 2997 ==== MOSDScrubReserve(11.3s5 RELEASE e501) v1 ==== 43+0+0 (223328929 0 0) 0x55d9869bfc00 con 0x55d986c7e000 > -5684> 2018-07-11 07:51:21.947325 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 912 0x55d9b40fb200 rep_scrubmap(5.2e e501 from shard 2) v1 > -5683> 2018-07-11 07:51:21.947345 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 912 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12464 (1269249638 0 1850272349) 0x55d9b40fb200 con 0x55d986bf0800 > -5682> 2018-07-11 07:51:21.959593 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6805/15263 conn(0x55d986c41000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=31 cs=1 l=0). rx osd.5 seq 2812 0x55d986c85000 MOSDScrubReserve(6.15 REQUEST e501) v1 > -5681> 2018-07-11 07:51:21.959610 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.5 10.8.128.113:6805/15263 2812 ==== MOSDScrubReserve(6.15 REQUEST e501) v1 ==== 43+0+0 (3942742026 0 0) 0x55d986c85000 con 0x55d986c41000 > -5680> 2018-07-11 07:51:21.979911 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 906 0x55d9b40fb440 rep_scrubmap(5.2e e501 from shard 4) v1 > -5679> 2018-07-11 07:51:21.979930 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 906 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12464 (3686815630 0 1850272349) 0x55d9b40fb440 con 0x55d986bec000 > -5678> 2018-07-11 07:51:22.132395 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 31] Generated table #36800: 482081 keys, 13622058 bytes > -5677> 2018-07-11 07:51:22.132438 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295482132422, "cf_name": "default", "job": 31, "event": "table_file_creation", "file_number": 36800, "file_size": 13622058, "table_properties": {"data_size": 12263292, "index_size": 125601, "filter_size": 1232179, "raw_key_size": 13926785, "raw_average_key_size": 28, "raw_value_size": 5731408, "raw_average_value_size": 11, "num_data_blocks": 3014, "num_entries": 482081, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} > -5676> 2018-07-11 07:51:22.301801 7ffab7be8700 10 monclient: tick > -5675> 2018-07-11 07:51:22.301814 7ffab7be8700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2018-07-11 07:50:52.301813) > -5674> 2018-07-11 07:51:22.349776 7ffab6be6700 1 -- 10.8.128.107:6800/13927 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55d9ba81b600 con 0 > -5673> 2018-07-11 07:51:22.349840 7ffab6be6700 1 -- 10.8.128.107:6800/13927 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55d985053b80 con 0 > -5672> 2018-07-11 07:51:22.374557 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6805/15263 -- MOSDScrubReserve(6.15 REJECT e501) v1 -- 0x55d9829e4200 con 0 > -5671> 2018-07-11 07:51:22.374605 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -5670> 2018-07-11 07:51:22.374630 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -5669> 2018-07-11 07:51:22.374676 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -5668> 2018-07-11 07:51:22.375050 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6805/15263 conn(0x55d986c41000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=31 cs=1 l=0). rx osd.5 seq 2813 0x55d9829e4200 MOSDScrubReserve(6.15 RELEASE e501) v1 > -5667> 2018-07-11 07:51:22.375070 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.5 10.8.128.113:6805/15263 2813 ==== MOSDScrubReserve(6.15 RELEASE e501) v1 ==== 43+0+0 (605848072 0 0) 0x55d9829e4200 con 0x55d986c41000 > -5666> 2018-07-11 07:51:22.375135 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -5665> 2018-07-11 07:51:22.390470 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'66874,epoch:501/498,start:5:7545f13c:::benchmark_data_magna115_11707_object423722:0,end:5:75463d61:::benchmark_data_magna115_10867_object542977:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454500 con 0 > -5664> 2018-07-11 07:51:22.390509 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'66874,epoch:501/498,start:5:7545f13c:::benchmark_data_magna115_11707_object423722:0,end:5:75463d61:::benchmark_data_magna115_10867_object542977:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98944b680 con 0 > -5663> 2018-07-11 07:51:22.562508 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6805/15885 conn(0x55d986c7f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.7 seq 2792 0x55d9829e4c00 MOSDScrubReserve(5.4 REQUEST e501) v1 > -5662> 2018-07-11 07:51:22.562528 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.7 10.8.128.114:6805/15885 2792 ==== MOSDScrubReserve(5.4 REQUEST e501) v1 ==== 43+0+0 (1753221512 0 0) 0x55d9829e4c00 con 0x55d986c7f800 > -5661> 2018-07-11 07:51:22.562569 7ffab33df700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.114:6805/15885 -- MOSDScrubReserve(5.4 REJECT e501) v1 -- 0x55d986d32200 con 0 > -5660> 2018-07-11 07:51:22.562908 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6805/15885 conn(0x55d986c7f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.7 seq 2793 0x55d986d32200 MOSDScrubReserve(5.4 RELEASE e501) v1 > -5659> 2018-07-11 07:51:22.562923 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.7 10.8.128.114:6805/15885 2793 ==== MOSDScrubReserve(5.4 RELEASE e501) v1 ==== 43+0+0 (2812146570 0 0) 0x55d986d32200 con 0x55d986c7f800 > -5658> 2018-07-11 07:51:22.583911 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 913 0x55d9b40fb680 rep_scrubmap(5.2e e501 from shard 2) v1 > -5657> 2018-07-11 07:51:22.583933 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 913 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12474 (1269249638 0 683884596) 0x55d9b40fb680 con 0x55d986bf0800 > -5656> 2018-07-11 07:51:22.606336 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 907 0x55d9b40fb8c0 rep_scrubmap(5.2e e501 from shard 4) v1 > -5655> 2018-07-11 07:51:22.606355 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 907 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12474 (3686815630 0 683884596) 0x55d9b40fb8c0 con 0x55d986bec000 > -5654> 2018-07-11 07:51:22.678664 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -5653> 2018-07-11 07:51:22.678705 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -5652> 2018-07-11 07:51:22.678762 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -5651> 2018-07-11 07:51:22.679228 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -5650> 2018-07-11 07:51:22.698699 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'66651,epoch:501/498,start:5:75463d61:::benchmark_data_magna115_10867_object542977:0,end:5:75469e30:::benchmark_data_magna115_10867_object3595252:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455180 con 0 > -5649> 2018-07-11 07:51:22.698750 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'66651,epoch:501/498,start:5:75463d61:::benchmark_data_magna115_10867_object542977:0,end:5:75469e30:::benchmark_data_magna115_10867_object3595252:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454780 con 0 > -5648> 2018-07-11 07:51:22.837068 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 2998 0x55d980f5a400 MOSDScrubReserve(11.3s5 REQUEST e501) v1 > -5647> 2018-07-11 07:51:22.837087 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 2998 ==== MOSDScrubReserve(11.3s5 REQUEST e501) v1 ==== 43+0+0 (3260170403 0 0) 0x55d980f5a400 con 0x55d986c7e000 > -5646> 2018-07-11 07:51:22.837162 7ffab3be0700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e501) v1 -- 0x55d9869bfc00 con 0 > -5645> 2018-07-11 07:51:22.837596 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 2999 0x55d9869bfc00 MOSDScrubReserve(11.3s5 RELEASE e501) v1 > -5644> 2018-07-11 07:51:22.837613 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 2999 ==== MOSDScrubReserve(11.3s5 RELEASE e501) v1 ==== 43+0+0 (223328929 0 0) 0x55d9869bfc00 con 0x55d986c7e000 > -5643> 2018-07-11 07:51:22.862743 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12677 conn(0x55d986db4800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.1 seq 251 0x55d9829e5600 osd_ping(ping e501 stamp 2018-07-11 07:51:22.910229) v4 > -5642> 2018-07-11 07:51:22.862758 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12677 conn(0x55d986df2000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.1 seq 251 0x55d9829e4800 osd_ping(ping e501 stamp 2018-07-11 07:51:22.910229) v4 > -5641> 2018-07-11 07:51:22.862772 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.1 10.8.128.106:0/12677 251 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:22.910229) v4 ==== 2004+0+0 (1202073 0 0) 0x55d9829e5600 con 0x55d986db4800 > -5640> 2018-07-11 07:51:22.862776 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.1 10.8.128.106:0/12677 251 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:22.910229) v4 ==== 2004+0+0 (1202073 0 0) 0x55d9829e4800 con 0x55d986df2000 > -5639> 2018-07-11 07:51:22.862784 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:22.910229) v4 -- 0x55d9c3383200 con 0 > -5638> 2018-07-11 07:51:22.862810 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:22.910229) v4 -- 0x55d9869be600 con 0 > -5637> 2018-07-11 07:51:22.882179 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15287 conn(0x55d98123f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 259 0x55d982cdf800 osd_ping(ping e501 stamp 2018-07-11 07:51:24.113003) v4 > -5636> 2018-07-11 07:51:22.882204 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.4 10.8.128.113:0/15287 259 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:24.113003) v4 ==== 2004+0+0 (290984147 0 0) 0x55d982cdf800 con 0x55d98123f000 > -5635> 2018-07-11 07:51:22.882198 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15287 conn(0x55d981240800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 259 0x55d986d96e00 osd_ping(ping e501 stamp 2018-07-11 07:51:24.113003) v4 > -5634> 2018-07-11 07:51:22.882216 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:24.113003) v4 -- 0x55d9c3383200 con 0 > -5633> 2018-07-11 07:51:22.882217 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.4 10.8.128.113:0/15287 259 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:24.113003) v4 ==== 2004+0+0 (290984147 0 0) 0x55d986d96e00 con 0x55d981240800 > -5632> 2018-07-11 07:51:22.882240 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:24.113003) v4 -- 0x55d9869be600 con 0 > -5631> 2018-07-11 07:51:22.920077 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 914 0x55d9b40fbb00 rep_scrubmap(5.2e e501 from shard 2) v1 > -5630> 2018-07-11 07:51:22.920097 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 914 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12470 (1269249638 0 339537805) 0x55d9b40fbb00 con 0x55d986bf0800 > -5629> 2018-07-11 07:51:22.941506 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 908 0x55d9b40fbd40 rep_scrubmap(5.2e e501 from shard 4) v1 > -5628> 2018-07-11 07:51:22.941525 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 908 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12470 (3686815630 0 339537805) 0x55d9b40fbd40 con 0x55d986bec000 > -5627> 2018-07-11 07:51:23.002310 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -5626> 2018-07-11 07:51:23.002336 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -5625> 2018-07-11 07:51:23.002385 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -5624> 2018-07-11 07:51:23.002842 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -5623> 2018-07-11 07:51:23.040601 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75469e30:::benchmark_data_magna115_10867_object3595252:0,end:5:7546f330:::benchmark_data_magna115_10867_object894704:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454f00 con 0 > -5622> 2018-07-11 07:51:23.040651 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75469e30:::benchmark_data_magna115_10867_object3595252:0,end:5:7546f330:::benchmark_data_magna115_10867_object894704:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498280 con 0 > -5621> 2018-07-11 07:51:23.228384 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 915 0x55d99de46000 rep_scrubmap(5.2e e501 from shard 2) v1 > -5620> 2018-07-11 07:51:23.228404 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 915 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12476 (1269249638 0 3072998950) 0x55d99de46000 con 0x55d986bf0800 > -5619> 2018-07-11 07:51:23.249896 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 909 0x55d99de46240 rep_scrubmap(5.2e e501 from shard 4) v1 > -5618> 2018-07-11 07:51:23.249916 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 909 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12476 (3686815630 0 3072998950) 0x55d99de46240 con 0x55d986bec000 > -5617> 2018-07-11 07:51:23.272313 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -5616> 2018-07-11 07:51:23.272339 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -5615> 2018-07-11 07:51:23.272385 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -5614> 2018-07-11 07:51:23.272839 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -5613> 2018-07-11 07:51:23.304162 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67050,epoch:501/498,start:5:7546f330:::benchmark_data_magna115_10867_object894704:0,end:5:75473472:::benchmark_data_magna115_10867_object3612652:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455400 con 0 > -5612> 2018-07-11 07:51:23.304194 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67050,epoch:501/498,start:5:7546f330:::benchmark_data_magna115_10867_object894704:0,end:5:75473472:::benchmark_data_magna115_10867_object3612652:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498000 con 0 > -5611> 2018-07-11 07:51:23.521315 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 916 0x55d99de46480 rep_scrubmap(5.2e e501 from shard 2) v1 > -5610> 2018-07-11 07:51:23.521335 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 916 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12464 (1269249638 0 3577955631) 0x55d99de46480 con 0x55d986bf0800 > -5609> 2018-07-11 07:51:23.530625 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 910 0x55d99de466c0 rep_scrubmap(5.2e e501 from shard 4) v1 > -5608> 2018-07-11 07:51:23.530644 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 910 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12464 (3686815630 0 3577955631) 0x55d99de466c0 con 0x55d986bec000 > -5607> 2018-07-11 07:51:23.553631 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -5606> 2018-07-11 07:51:23.553669 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -5605> 2018-07-11 07:51:23.553713 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -5604> 2018-07-11 07:51:23.554187 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -5603> 2018-07-11 07:51:23.598998 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'66879,epoch:501/498,start:5:75473472:::benchmark_data_magna115_10867_object3612652:0,end:5:75477b2c:::benchmark_data_magna115_10867_object1822911:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455680 con 0 > -5602> 2018-07-11 07:51:23.599048 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'66879,epoch:501/498,start:5:75473472:::benchmark_data_magna115_10867_object3612652:0,end:5:75477b2c:::benchmark_data_magna115_10867_object1822911:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455b80 con 0 > -5601> 2018-07-11 07:51:23.794114 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 917 0x55d99de46900 rep_scrubmap(5.2e e501 from shard 2) v1 > -5600> 2018-07-11 07:51:23.794134 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 917 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12472 (1269249638 0 3344758077) 0x55d99de46900 con 0x55d986bf0800 > -5599> 2018-07-11 07:51:23.803987 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 911 0x55d99de46b40 rep_scrubmap(5.2e e501 from shard 4) v1 > -5598> 2018-07-11 07:51:23.804006 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 911 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12472 (3686815630 0 3344758077) 0x55d99de46b40 con 0x55d986bec000 > -5597> 2018-07-11 07:51:23.837268 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 3000 0x55d9bdba4600 MOSDScrubReserve(11.3s5 REQUEST e501) v1 > -5596> 2018-07-11 07:51:23.837286 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 3000 ==== MOSDScrubReserve(11.3s5 REQUEST e501) v1 ==== 43+0+0 (3260170403 0 0) 0x55d9bdba4600 con 0x55d986c7e000 > -5595> 2018-07-11 07:51:23.837333 7ffab3be0700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e501) v1 -- 0x55d9869bfc00 con 0 > -5594> 2018-07-11 07:51:23.837782 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 3001 0x55d9869bfc00 MOSDScrubReserve(11.3s5 RELEASE e501) v1 > -5593> 2018-07-11 07:51:23.837795 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 3001 ==== MOSDScrubReserve(11.3s5 RELEASE e501) v1 ==== 43+0+0 (223328929 0 0) 0x55d9869bfc00 con 0x55d986c7e000 > -5592> 2018-07-11 07:51:23.855753 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -5591> 2018-07-11 07:51:23.855776 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -5590> 2018-07-11 07:51:23.855834 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -5589> 2018-07-11 07:51:23.856323 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -5588> 2018-07-11 07:51:23.894179 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75477b2c:::benchmark_data_magna115_10867_object1822911:0,end:5:7547dabd:::benchmark_data_magna115_10867_object1505703:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455900 con 0 > -5587> 2018-07-11 07:51:23.894223 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75477b2c:::benchmark_data_magna115_10867_object1822911:0,end:5:7547dabd:::benchmark_data_magna115_10867_object1505703:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b14f0500 con 0 > -5586> 2018-07-11 07:51:23.961111 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6801/15919 conn(0x55d986c3f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=29 cs=1 l=0). rx osd.11 seq 232 0x55d9869d9600 MOSDScrubReserve(10.f REQUEST e501) v1 > -5585> 2018-07-11 07:51:23.961129 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.11 10.8.128.114:6801/15919 232 ==== MOSDScrubReserve(10.f REQUEST e501) v1 ==== 43+0+0 (2892966580 0 0) 0x55d9869d9600 con 0x55d986c3f800 > -5584> 2018-07-11 07:51:23.961198 7ffab53e3700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.114:6801/15919 -- MOSDScrubReserve(10.f REJECT e501) v1 -- 0x55d9b60de600 con 0 > -5583> 2018-07-11 07:51:23.961505 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6801/15919 conn(0x55d986c3f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=29 cs=1 l=0). rx osd.11 seq 233 0x55d9b60de600 MOSDScrubReserve(10.f RELEASE e501) v1 > -5582> 2018-07-11 07:51:23.961519 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.11 10.8.128.114:6801/15919 233 ==== MOSDScrubReserve(10.f RELEASE e501) v1 ==== 43+0+0 (1668468918 0 0) 0x55d9b60de600 con 0x55d986c3f800 > -5581> 2018-07-11 07:51:23.963130 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12677 conn(0x55d986db4800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.1 seq 252 0x55d9829e5600 osd_ping(ping e501 stamp 2018-07-11 07:51:24.010714) v4 > -5580> 2018-07-11 07:51:23.963146 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12677 conn(0x55d986df2000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.1 seq 252 0x55d9829e4800 osd_ping(ping e501 stamp 2018-07-11 07:51:24.010714) v4 > -5579> 2018-07-11 07:51:23.963155 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.1 10.8.128.106:0/12677 252 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:24.010714) v4 ==== 2004+0+0 (2458751408 0 0) 0x55d9829e5600 con 0x55d986db4800 > -5578> 2018-07-11 07:51:23.963165 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.1 10.8.128.106:0/12677 252 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:24.010714) v4 ==== 2004+0+0 (2458751408 0 0) 0x55d9829e4800 con 0x55d986df2000 > -5577> 2018-07-11 07:51:23.963168 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:24.010714) v4 -- 0x55d9c3383200 con 0 > -5576> 2018-07-11 07:51:23.963192 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:24.010714) v4 -- 0x55d9869be600 con 0 > -5575> 2018-07-11 07:51:24.110791 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 918 0x55d99de46d80 rep_scrubmap(5.2e e501 from shard 2) v1 > -5574> 2018-07-11 07:51:24.110811 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 918 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12472 (1269249638 0 1314884203) 0x55d99de46d80 con 0x55d986bf0800 > -5573> 2018-07-11 07:51:24.126480 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 912 0x55d99de46fc0 rep_scrubmap(5.2e e501 from shard 4) v1 > -5572> 2018-07-11 07:51:24.126499 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 912 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12472 (3686815630 0 1314884203) 0x55d99de46fc0 con 0x55d986bec000 > -5571> 2018-07-11 07:51:24.126647 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -5570> 2018-07-11 07:51:24.126672 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -5569> 2018-07-11 07:51:24.126720 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -5568> 2018-07-11 07:51:24.127188 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -5567> 2018-07-11 07:51:24.174263 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7547dabd:::benchmark_data_magna115_10867_object1505703:0,end:5:75483902:::benchmark_data_magna115_10867_object3188285:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498780 con 0 > -5566> 2018-07-11 07:51:24.174299 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7547dabd:::benchmark_data_magna115_10867_object1505703:0,end:5:75483902:::benchmark_data_magna115_10867_object3188285:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98696e000 con 0 > -5565> 2018-07-11 07:51:24.317334 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.114:0/15885 conn(0x55d986d81000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.7 seq 254 0x55d9829e4800 osd_ping(ping e501 stamp 2018-07-11 07:51:24.947494) v4 > -5564> 2018-07-11 07:51:24.317349 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.114:0/15885 conn(0x55d986d7f800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.7 seq 254 0x55d982cdf800 osd_ping(ping e501 stamp 2018-07-11 07:51:24.947494) v4 > -5563> 2018-07-11 07:51:24.317357 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.7 10.8.128.114:0/15885 254 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:24.947494) v4 ==== 2004+0+0 (3986918861 0 0) 0x55d9829e4800 con 0x55d986d81000 > -5562> 2018-07-11 07:51:24.317375 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.7 10.8.128.114:0/15885 254 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:24.947494) v4 ==== 2004+0+0 (3986918861 0 0) 0x55d982cdf800 con 0x55d986d7f800 > -5561> 2018-07-11 07:51:24.317383 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:24.947494) v4 -- 0x55d9869be600 con 0 > -5560> 2018-07-11 07:51:24.317415 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:24.947494) v4 -- 0x55d9c3383200 con 0 > -5559> 2018-07-11 07:51:24.400054 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 913 0x55d99de47200 rep_scrubmap(5.2e e501 from shard 4) v1 > -5558> 2018-07-11 07:51:24.400075 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 913 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12478 (3686815630 0 2912139537) 0x55d99de47200 con 0x55d986bec000 > -5557> 2018-07-11 07:51:24.403393 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 919 0x55d99de47440 rep_scrubmap(5.2e e501 from shard 2) v1 > -5556> 2018-07-11 07:51:24.403413 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 919 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12478 (1269249638 0 2912139537) 0x55d99de47440 con 0x55d986bf0800 > -5555> 2018-07-11 07:51:24.421315 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12651 conn(0x55d986db3000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.2 seq 247 0x55d9b60df800 osd_ping(ping e501 stamp 2018-07-11 07:51:24.468847) v4 > -5554> 2018-07-11 07:51:24.421342 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.2 10.8.128.106:0/12651 247 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:24.468847) v4 ==== 2004+0+0 (2256168485 0 0) 0x55d9b60df800 con 0x55d986db3000 > -5553> 2018-07-11 07:51:24.421340 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12651 conn(0x55d986db1800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.2 seq 247 0x55d986d96e00 osd_ping(ping e501 stamp 2018-07-11 07:51:24.468847) v4 > -5552> 2018-07-11 07:51:24.421356 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:24.468847) v4 -- 0x55d9b67d9e00 con 0 > -5551> 2018-07-11 07:51:24.421363 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.2 10.8.128.106:0/12651 247 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:24.468847) v4 ==== 2004+0+0 (2256168485 0 0) 0x55d986d96e00 con 0x55d986db1800 > -5550> 2018-07-11 07:51:24.421380 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:24.468847) v4 -- 0x55d9869be600 con 0 > -5549> 2018-07-11 07:51:24.445758 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.107:0/12617 conn(0x55d98123c000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 236 0x55d9829e4800 osd_ping(ping e501 stamp 2018-07-11 07:51:24.445406) v4 > -5548> 2018-07-11 07:51:24.445774 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.107:0/12617 conn(0x55d98123d800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 236 0x55d986d33e00 osd_ping(ping e501 stamp 2018-07-11 07:51:24.445406) v4 > -5547> 2018-07-11 07:51:24.445787 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.9 10.8.128.107:0/12617 236 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:24.445406) v4 ==== 2004+0+0 (179146843 0 0) 0x55d9829e4800 con 0x55d98123c000 > -5546> 2018-07-11 07:51:24.445798 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.9 10.8.128.107:0/12617 236 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:24.445406) v4 ==== 2004+0+0 (179146843 0 0) 0x55d986d33e00 con 0x55d98123d800 > -5545> 2018-07-11 07:51:24.445804 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:24.445406) v4 -- 0x55d9869be600 con 0 > -5544> 2018-07-11 07:51:24.445872 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:24.445406) v4 -- 0x55d9b67d9e00 con 0 > -5543> 2018-07-11 07:51:24.459894 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -5542> 2018-07-11 07:51:24.459917 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -5541> 2018-07-11 07:51:24.459975 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -5540> 2018-07-11 07:51:24.460447 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -5539> 2018-07-11 07:51:24.461467 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67307,epoch:501/498,start:5:75483902:::benchmark_data_magna115_10867_object3188285:0,end:5:754897ed:::benchmark_data_magna115_10867_object723474:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98696f180 con 0 > -5538> 2018-07-11 07:51:24.461502 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67307,epoch:501/498,start:5:75483902:::benchmark_data_magna115_10867_object3188285:0,end:5:754897ed:::benchmark_data_magna115_10867_object723474:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98696f900 con 0 > -5537> 2018-07-11 07:51:24.673932 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 920 0x55d99de47680 rep_scrubmap(5.2e e501 from shard 2) v1 > -5536> 2018-07-11 07:51:24.673953 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 920 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12472 (1269249638 0 2063266023) 0x55d99de47680 con 0x55d986bf0800 > -5535> 2018-07-11 07:51:24.692488 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 914 0x55d99de478c0 rep_scrubmap(5.2e e501 from shard 4) v1 > -5534> 2018-07-11 07:51:24.692507 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 914 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12472 (3686815630 0 2063266023) 0x55d99de478c0 con 0x55d986bec000 > -5533> 2018-07-11 07:51:24.717332 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -5532> 2018-07-11 07:51:24.717356 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -5531> 2018-07-11 07:51:24.717400 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -5530> 2018-07-11 07:51:24.717840 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -5529> 2018-07-11 07:51:24.766096 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67830,epoch:501/498,start:5:754897ed:::benchmark_data_magna115_10867_object723474:0,end:5:7548f86f:::benchmark_data_magna115_10867_object3227489:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498a00 con 0 > -5528> 2018-07-11 07:51:24.766133 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67830,epoch:501/498,start:5:754897ed:::benchmark_data_magna115_10867_object723474:0,end:5:7548f86f:::benchmark_data_magna115_10867_object3227489:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98696ef00 con 0 > -5527> 2018-07-11 07:51:24.816806 7ffab1bdc700 5 osd.10 501 heartbeat: osd_stat(788 GB used, 142 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist []) > -5526> 2018-07-11 07:51:24.816819 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6802/15255 -- osd_ping(ping e501 stamp 2018-07-11 07:51:24.816818) v4 -- 0x55d9c34cc600 con 0 > -5525> 2018-07-11 07:51:24.816857 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6803/15255 -- osd_ping(ping e501 stamp 2018-07-11 07:51:24.816818) v4 -- 0x55d9bcd78600 con 0 > -5524> 2018-07-11 07:51:24.816872 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6806/12677 -- osd_ping(ping e501 stamp 2018-07-11 07:51:24.816818) v4 -- 0x55d9869bf800 con 0 > -5523> 2018-07-11 07:51:24.816886 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6807/12677 -- osd_ping(ping e501 stamp 2018-07-11 07:51:24.816818) v4 -- 0x55d980f5a200 con 0 > -5522> 2018-07-11 07:51:24.816898 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6808/12651 -- osd_ping(ping e501 stamp 2018-07-11 07:51:24.816818) v4 -- 0x55d982cdee00 con 0 > -5521> 2018-07-11 07:51:24.816909 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6810/12651 -- osd_ping(ping e501 stamp 2018-07-11 07:51:24.816818) v4 -- 0x55d986d33600 con 0 > -5520> 2018-07-11 07:51:24.816921 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6809/12650 -- osd_ping(ping e501 stamp 2018-07-11 07:51:24.816818) v4 -- 0x55d9c3383e00 con 0 > -5519> 2018-07-11 07:51:24.816939 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6811/12650 -- osd_ping(ping e501 stamp 2018-07-11 07:51:24.816818) v4 -- 0x55d986c23a00 con 0 > -5518> 2018-07-11 07:51:24.816951 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6810/15287 -- osd_ping(ping e501 stamp 2018-07-11 07:51:24.816818) v4 -- 0x55d9b2783000 con 0 > -5517> 2018-07-11 07:51:24.816965 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6811/15287 -- osd_ping(ping e501 stamp 2018-07-11 07:51:24.816818) v4 -- 0x55d9b437a800 con 0 > -5516> 2018-07-11 07:51:24.816981 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6806/15263 -- osd_ping(ping e501 stamp 2018-07-11 07:51:24.816818) v4 -- 0x55d986c84400 con 0 > -5515> 2018-07-11 07:51:24.816995 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6807/15263 -- osd_ping(ping e501 stamp 2018-07-11 07:51:24.816818) v4 -- 0x55d986d33c00 con 0 > -5514> 2018-07-11 07:51:24.817009 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.115:6802/13323 -- osd_ping(ping e501 stamp 2018-07-11 07:51:24.816818) v4 -- 0x55d9b67d9200 con 0 > -5513> 2018-07-11 07:51:24.817021 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.115:6803/13323 -- osd_ping(ping e501 stamp 2018-07-11 07:51:24.816818) v4 -- 0x55d986d4f400 con 0 > -5512> 2018-07-11 07:51:24.817036 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6806/15885 -- osd_ping(ping e501 stamp 2018-07-11 07:51:24.816818) v4 -- 0x55d9869bfe00 con 0 > -5511> 2018-07-11 07:51:24.817052 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6807/15885 -- osd_ping(ping e501 stamp 2018-07-11 07:51:24.816818) v4 -- 0x55d9869bfa00 con 0 > -5510> 2018-07-11 07:51:24.817064 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.108:6802/11967 -- osd_ping(ping e501 stamp 2018-07-11 07:51:24.816818) v4 -- 0x55d9c908fe00 con 0 > -5509> 2018-07-11 07:51:24.817075 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.108:6803/11967 -- osd_ping(ping e501 stamp 2018-07-11 07:51:24.816818) v4 -- 0x55d9b9ac3200 con 0 > -5508> 2018-07-11 07:51:24.817088 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.107:6806/12617 -- osd_ping(ping e501 stamp 2018-07-11 07:51:24.816818) v4 -- 0x55d9b9ac3400 con 0 > -5507> 2018-07-11 07:51:24.817101 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.107:6807/12617 -- osd_ping(ping e501 stamp 2018-07-11 07:51:24.816818) v4 -- 0x55d9b3db3c00 con 0 > -5506> 2018-07-11 07:51:24.817113 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6802/15919 -- osd_ping(ping e501 stamp 2018-07-11 07:51:24.816818) v4 -- 0x55d982cde200 con 0 > -5505> 2018-07-11 07:51:24.817125 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6803/15919 -- osd_ping(ping e501 stamp 2018-07-11 07:51:24.816818) v4 -- 0x55d9869bec00 con 0 > -5504> 2018-07-11 07:51:24.817265 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.107:6806/12617 conn(0x55d986bd2000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.9 seq 240 0x55d982cdf800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:24.816818) v4 > -5503> 2018-07-11 07:51:24.817286 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.9 10.8.128.107:6806/12617 240 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:24.816818) v4 ==== 2004+0+0 (748963925 0 0) 0x55d982cdf800 con 0x55d986bd2000 > -5502> 2018-07-11 07:51:24.817306 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.107:6807/12617 conn(0x55d986bd3800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.9 seq 240 0x55d9b60df800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:24.816818) v4 > -5501> 2018-07-11 07:51:24.817349 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6807/12677 conn(0x55d986b59800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.1 seq 240 0x55d9829e4800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:24.816818) v4 > -5500> 2018-07-11 07:51:24.817331 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.9 10.8.128.107:6807/12617 240 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:24.816818) v4 ==== 2004+0+0 (748963925 0 0) 0x55d9b60df800 con 0x55d986bd3800 > -5499> 2018-07-11 07:51:24.817368 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.1 10.8.128.106:6807/12677 240 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:24.816818) v4 ==== 2004+0+0 (748963925 0 0) 0x55d9829e4800 con 0x55d986b59800 > -5498> 2018-07-11 07:51:24.817389 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6810/12651 conn(0x55d986b9b800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.2 seq 240 0x55d982cdf800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:24.816818) v4 > -5497> 2018-07-11 07:51:24.817414 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6808/12651 conn(0x55d986b9a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.2 seq 240 0x55d9829e4800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:24.816818) v4 > -5496> 2018-07-11 07:51:24.817421 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.2 10.8.128.106:6810/12651 240 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:24.816818) v4 ==== 2004+0+0 (748963925 0 0) 0x55d982cdf800 con 0x55d986b9b800 > -5495> 2018-07-11 07:51:24.817426 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.2 10.8.128.106:6808/12651 240 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:24.816818) v4 ==== 2004+0+0 (748963925 0 0) 0x55d9829e4800 con 0x55d986b9a000 > -5494> 2018-07-11 07:51:24.817438 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6806/12677 conn(0x55d986b58000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.1 seq 240 0x55d9b60df800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:24.816818) v4 > -5493> 2018-07-11 07:51:24.817458 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.1 10.8.128.106:6806/12677 240 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:24.816818) v4 ==== 2004+0+0 (748963925 0 0) 0x55d9b60df800 con 0x55d986b58000 > -5492> 2018-07-11 07:51:24.817468 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6802/15255 conn(0x55d986bc8000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.0 seq 240 0x55d9829e4800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:24.816818) v4 > -5491> 2018-07-11 07:51:24.817481 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.0 10.8.128.113:6802/15255 240 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:24.816818) v4 ==== 2004+0+0 (748963925 0 0) 0x55d9829e4800 con 0x55d986bc8000 > -5490> 2018-07-11 07:51:24.817473 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6803/15255 conn(0x55d986bc9800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.0 seq 240 0x55d982cdf800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:24.816818) v4 > -5489> 2018-07-11 07:51:24.817517 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.0 10.8.128.113:6803/15255 240 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:24.816818) v4 ==== 2004+0+0 (748963925 0 0) 0x55d982cdf800 con 0x55d986bc9800 > -5488> 2018-07-11 07:51:24.817522 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6806/15263 conn(0x55d986b9d000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.5 seq 240 0x55d9b60df800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:24.816818) v4 > -5487> 2018-07-11 07:51:24.817528 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6811/12650 conn(0x55d986bcc800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 240 0x55d9829e4800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:24.816818) v4 > -5486> 2018-07-11 07:51:24.817541 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.3 10.8.128.106:6811/12650 240 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:24.816818) v4 ==== 2004+0+0 (748963925 0 0) 0x55d9829e4800 con 0x55d986bcc800 > -5485> 2018-07-11 07:51:24.817541 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.5 10.8.128.113:6806/15263 240 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:24.816818) v4 ==== 2004+0+0 (748963925 0 0) 0x55d9b60df800 con 0x55d986b9d000 > -5484> 2018-07-11 07:51:24.817571 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6807/15263 conn(0x55d986b9e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.5 seq 240 0x55d9829e4800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:24.816818) v4 > -5483> 2018-07-11 07:51:24.817571 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6811/15287 conn(0x55d986bb0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 240 0x55d982cdf800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:24.816818) v4 > -5482> 2018-07-11 07:51:24.817589 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.5 10.8.128.113:6807/15263 240 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:24.816818) v4 ==== 2004+0+0 (748963925 0 0) 0x55d9829e4800 con 0x55d986b9e800 > -5481> 2018-07-11 07:51:24.817596 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.4 10.8.128.113:6811/15287 240 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:24.816818) v4 ==== 2004+0+0 (748963925 0 0) 0x55d982cdf800 con 0x55d986bb0800 > -5480> 2018-07-11 07:51:24.817601 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6809/12650 conn(0x55d986bcb000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 240 0x55d9b60df800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:24.816818) v4 > -5479> 2018-07-11 07:51:24.817617 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6810/15287 conn(0x55d986baf000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 240 0x55d9829e4800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:24.816818) v4 > -5478> 2018-07-11 07:51:24.817619 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.3 10.8.128.106:6809/12650 240 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:24.816818) v4 ==== 2004+0+0 (748963925 0 0) 0x55d9b60df800 con 0x55d986bcb000 > -5477> 2018-07-11 07:51:24.817628 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.4 10.8.128.113:6810/15287 240 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:24.816818) v4 ==== 2004+0+0 (748963925 0 0) 0x55d9829e4800 con 0x55d986baf000 > -5476> 2018-07-11 07:51:24.817645 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.115:6802/13323 conn(0x55d986bc1000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 240 0x55d982cdf800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:24.816818) v4 > -5475> 2018-07-11 07:51:24.817672 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6807/15885 conn(0x55d986bad800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.7 seq 240 0x55d9b60df800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:24.816818) v4 > -5474> 2018-07-11 07:51:24.817675 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.6 10.8.128.115:6802/13323 240 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:24.816818) v4 ==== 2004+0+0 (748963925 0 0) 0x55d982cdf800 con 0x55d986bc1000 > -5473> 2018-07-11 07:51:24.817694 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.7 10.8.128.114:6807/15885 240 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:24.816818) v4 ==== 2004+0+0 (748963925 0 0) 0x55d9b60df800 con 0x55d986bad800 > -5472> 2018-07-11 07:51:24.817730 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.108:6803/11967 conn(0x55d986bbf800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.8 seq 240 0x55d9829e4800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:24.816818) v4 > -5471> 2018-07-11 07:51:24.817736 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.108:6802/11967 conn(0x55d986bbe000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.8 seq 240 0x55d9b60df800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:24.816818) v4 > -5470> 2018-07-11 07:51:24.817732 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6806/15885 conn(0x55d986bac000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.7 seq 240 0x55d982cdf800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:24.816818) v4 > -5469> 2018-07-11 07:51:24.817753 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.8 10.8.128.108:6803/11967 240 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:24.816818) v4 ==== 2004+0+0 (748963925 0 0) 0x55d9829e4800 con 0x55d986bbf800 > -5468> 2018-07-11 07:51:24.817751 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.8 10.8.128.108:6802/11967 240 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:24.816818) v4 ==== 2004+0+0 (748963925 0 0) 0x55d9b60df800 con 0x55d986bbe000 > -5467> 2018-07-11 07:51:24.817756 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.7 10.8.128.114:6806/15885 240 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:24.816818) v4 ==== 2004+0+0 (748963925 0 0) 0x55d982cdf800 con 0x55d986bac000 > -5466> 2018-07-11 07:51:24.817799 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6802/15919 conn(0x55d986b5b000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.11 seq 240 0x55d982cdf800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:24.816818) v4 > -5465> 2018-07-11 07:51:24.817813 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.11 10.8.128.114:6802/15919 240 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:24.816818) v4 ==== 2004+0+0 (748963925 0 0) 0x55d982cdf800 con 0x55d986b5b000 > -5464> 2018-07-11 07:51:24.817820 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.115:6803/13323 conn(0x55d986bc2800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 240 0x55d9b60df800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:24.816818) v4 > -5463> 2018-07-11 07:51:24.817839 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.6 10.8.128.115:6803/13323 240 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:24.816818) v4 ==== 2004+0+0 (748963925 0 0) 0x55d9b60df800 con 0x55d986bc2800 > -5462> 2018-07-11 07:51:24.817880 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6803/15919 conn(0x55d986b5c800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.11 seq 240 0x55d9b60df800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:24.816818) v4 > -5461> 2018-07-11 07:51:24.817896 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.11 10.8.128.114:6803/15919 240 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:24.816818) v4 ==== 2004+0+0 (748963925 0 0) 0x55d9b60df800 con 0x55d986b5c800 > -5460> 2018-07-11 07:51:24.960062 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 921 0x55d99de47b00 rep_scrubmap(5.2e e501 from shard 2) v1 > -5459> 2018-07-11 07:51:24.960082 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 921 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12466 (1269249638 0 532551269) 0x55d99de47b00 con 0x55d986bf0800 > -5458> 2018-07-11 07:51:24.989460 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 915 0x55d99de47d40 rep_scrubmap(5.2e e501 from shard 4) v1 > -5457> 2018-07-11 07:51:24.989480 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 915 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12466 (3686815630 0 532551269) 0x55d99de47d40 con 0x55d986bec000 > -5456> 2018-07-11 07:51:25.025616 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -5455> 2018-07-11 07:51:25.025640 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -5454> 2018-07-11 07:51:25.025686 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -5453> 2018-07-11 07:51:25.026131 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -5452> 2018-07-11 07:51:25.054888 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67564,epoch:501/498,start:5:7548f86f:::benchmark_data_magna115_10867_object3227489:0,end:5:754947dd:::benchmark_data_magna115_10867_object3363906:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b14f0a00 con 0 > -5451> 2018-07-11 07:51:25.054939 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67564,epoch:501/498,start:5:7548f86f:::benchmark_data_magna115_10867_object3227489:0,end:5:754947dd:::benchmark_data_magna115_10867_object3363906:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98696f680 con 0 > -5450> 2018-07-11 07:51:25.275929 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 922 0x55d98bf7c000 rep_scrubmap(5.2e e501 from shard 2) v1 > -5449> 2018-07-11 07:51:25.275950 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 922 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12484 (1269249638 0 1960805976) 0x55d98bf7c000 con 0x55d986bf0800 > -5448> 2018-07-11 07:51:25.299586 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 916 0x55d98bf7c240 rep_scrubmap(5.2e e501 from shard 4) v1 > -5447> 2018-07-11 07:51:25.299606 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 916 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12484 (3686815630 0 1960805976) 0x55d98bf7c240 con 0x55d986bec000 > -5446> 2018-07-11 07:51:25.299748 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -5445> 2018-07-11 07:51:25.299774 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -5444> 2018-07-11 07:51:25.299821 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -5443> 2018-07-11 07:51:25.300286 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -5442> 2018-07-11 07:51:25.316608 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:754947dd:::benchmark_data_magna115_10867_object3363906:0,end:5:7549cae7:::benchmark_data_magna115_11707_object296364:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98696ec80 con 0 > -5441> 2018-07-11 07:51:25.316655 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:754947dd:::benchmark_data_magna115_10867_object3363906:0,end:5:7549cae7:::benchmark_data_magna115_11707_object296364:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1499400 con 0 > -5440> 2018-07-11 07:51:25.509568 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 923 0x55d98bf7c480 rep_scrubmap(5.2e e501 from shard 2) v1 > -5439> 2018-07-11 07:51:25.509590 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 923 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12480 (1269249638 0 73482109) 0x55d98bf7c480 con 0x55d986bf0800 > -5438> 2018-07-11 07:51:25.534916 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 917 0x55d98bf7c6c0 rep_scrubmap(5.2e e501 from shard 4) v1 > -5437> 2018-07-11 07:51:25.534938 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 917 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12480 (3686815630 0 73482109) 0x55d98bf7c6c0 con 0x55d986bec000 > -5436> 2018-07-11 07:51:25.596819 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -5435> 2018-07-11 07:51:25.596844 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -5434> 2018-07-11 07:51:25.596900 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -5433> 2018-07-11 07:51:25.597393 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -5432> 2018-07-11 07:51:25.598281 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7549cae7:::benchmark_data_magna115_11707_object296364:0,end:5:754a4daf:::benchmark_data_magna115_10867_object1144195:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498f00 con 0 > -5431> 2018-07-11 07:51:25.598316 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7549cae7:::benchmark_data_magna115_11707_object296364:0,end:5:754a4daf:::benchmark_data_magna115_10867_object1144195:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1499b80 con 0 > -5430> 2018-07-11 07:51:25.665766 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15255 conn(0x55d986c82800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.0 seq 249 0x55d982cdf800 osd_ping(ping e501 stamp 2018-07-11 07:51:26.896617) v4 > -5429> 2018-07-11 07:51:25.665769 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15255 conn(0x55d986d20000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.0 seq 249 0x55d986d96e00 osd_ping(ping e501 stamp 2018-07-11 07:51:26.896617) v4 > -5428> 2018-07-11 07:51:25.665790 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.0 10.8.128.113:0/15255 249 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:26.896617) v4 ==== 2004+0+0 (3415743640 0 0) 0x55d982cdf800 con 0x55d986c82800 > -5427> 2018-07-11 07:51:25.665791 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.0 10.8.128.113:0/15255 249 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:26.896617) v4 ==== 2004+0+0 (3415743640 0 0) 0x55d986d96e00 con 0x55d986d20000 > -5426> 2018-07-11 07:51:25.665801 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:26.896617) v4 -- 0x55d982cde200 con 0 > -5425> 2018-07-11 07:51:25.665825 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:26.896617) v4 -- 0x55d9b9ac3200 con 0 > -5424> 2018-07-11 07:51:25.782642 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15287 conn(0x55d981240800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 260 0x55d9829e4800 osd_ping(ping e501 stamp 2018-07-11 07:51:27.013433) v4 > -5423> 2018-07-11 07:51:25.782667 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.4 10.8.128.113:0/15287 260 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:27.013433) v4 ==== 2004+0+0 (3402279964 0 0) 0x55d9829e4800 con 0x55d981240800 > -5422> 2018-07-11 07:51:25.782665 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15287 conn(0x55d98123f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 260 0x55d9829e5600 osd_ping(ping e501 stamp 2018-07-11 07:51:27.013433) v4 > -5421> 2018-07-11 07:51:25.782678 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:27.013433) v4 -- 0x55d9b9ac3200 con 0 > -5420> 2018-07-11 07:51:25.782682 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.4 10.8.128.113:0/15287 260 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:27.013433) v4 ==== 2004+0+0 (3402279964 0 0) 0x55d9829e5600 con 0x55d98123f000 > -5419> 2018-07-11 07:51:25.782696 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:27.013433) v4 -- 0x55d982cde200 con 0 > -5418> 2018-07-11 07:51:25.816282 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 924 0x55d98bf7c900 rep_scrubmap(5.2e e501 from shard 2) v1 > -5417> 2018-07-11 07:51:25.816305 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 924 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12474 (1269249638 0 698404927) 0x55d98bf7c900 con 0x55d986bf0800 > -5416> 2018-07-11 07:51:25.823702 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 918 0x55d98bf7cb40 rep_scrubmap(5.2e e501 from shard 4) v1 > -5415> 2018-07-11 07:51:25.823722 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 918 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12474 (3686815630 0 698404927) 0x55d98bf7cb40 con 0x55d986bec000 > -5414> 2018-07-11 07:51:25.910213 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -5413> 2018-07-11 07:51:25.910239 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -5412> 2018-07-11 07:51:25.910308 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -5411> 2018-07-11 07:51:25.910770 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -5410> 2018-07-11 07:51:25.914738 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12650 conn(0x55d986d23000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 241 0x55d982cdf800 osd_ping(ping e501 stamp 2018-07-11 07:51:25.962284) v4 > -5409> 2018-07-11 07:51:25.914750 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12650 conn(0x55d986d21800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 241 0x55d9b60df800 osd_ping(ping e501 stamp 2018-07-11 07:51:25.962284) v4 > -5408> 2018-07-11 07:51:25.914759 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.3 10.8.128.106:0/12650 241 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:25.962284) v4 ==== 2004+0+0 (3603817709 0 0) 0x55d982cdf800 con 0x55d986d23000 > -5407> 2018-07-11 07:51:25.914774 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.3 10.8.128.106:0/12650 241 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:25.962284) v4 ==== 2004+0+0 (3603817709 0 0) 0x55d9b60df800 con 0x55d986d21800 > -5406> 2018-07-11 07:51:25.914780 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:25.962284) v4 -- 0x55d982cde200 con 0 > -5405> 2018-07-11 07:51:25.914805 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:25.962284) v4 -- 0x55d9869bec00 con 0 > -5404> 2018-07-11 07:51:25.933491 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67810,epoch:501/498,start:5:754a4daf:::benchmark_data_magna115_10867_object1144195:0,end:5:754ac15a:::benchmark_data_magna115_10867_object1003752:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1499180 con 0 > -5403> 2018-07-11 07:51:25.933540 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67810,epoch:501/498,start:5:754a4daf:::benchmark_data_magna115_10867_object1144195:0,end:5:754ac15a:::benchmark_data_magna115_10867_object1003752:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498c80 con 0 > -5402> 2018-07-11 07:51:26.100550 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.115:0/13323 conn(0x55d986d7e000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 249 0x55d986d33e00 osd_ping(ping e501 stamp 2018-07-11 07:51:26.446161) v4 > -5401> 2018-07-11 07:51:26.100574 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.6 10.8.128.115:0/13323 249 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:26.446161) v4 ==== 2004+0+0 (2833400582 0 0) 0x55d986d33e00 con 0x55d986d7e000 > -5400> 2018-07-11 07:51:26.100565 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.115:0/13323 conn(0x55d986d24800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 249 0x55d9829e4800 osd_ping(ping e501 stamp 2018-07-11 07:51:26.446161) v4 > -5399> 2018-07-11 07:51:26.100588 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:26.446161) v4 -- 0x55d9869bec00 con 0 > -5398> 2018-07-11 07:51:26.100587 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.6 10.8.128.115:0/13323 249 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:26.446161) v4 ==== 2004+0+0 (2833400582 0 0) 0x55d9829e4800 con 0x55d986d24800 > -5397> 2018-07-11 07:51:26.100615 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:26.446161) v4 -- 0x55d9b9ac3200 con 0 > -5396> 2018-07-11 07:51:26.121736 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12651 conn(0x55d986db3000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.2 seq 248 0x55d9b60df800 osd_ping(ping e501 stamp 2018-07-11 07:51:26.169313) v4 > -5395> 2018-07-11 07:51:26.121763 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.2 10.8.128.106:0/12651 248 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:26.169313) v4 ==== 2004+0+0 (249468539 0 0) 0x55d9b60df800 con 0x55d986db3000 > -5394> 2018-07-11 07:51:26.121780 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:26.169313) v4 -- 0x55d9869bec00 con 0 > -5393> 2018-07-11 07:51:26.121783 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12651 conn(0x55d986db1800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.2 seq 248 0x55d986d96e00 osd_ping(ping e501 stamp 2018-07-11 07:51:26.169313) v4 > -5392> 2018-07-11 07:51:26.121808 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.2 10.8.128.106:0/12651 248 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:26.169313) v4 ==== 2004+0+0 (249468539 0 0) 0x55d986d96e00 con 0x55d986db1800 > -5391> 2018-07-11 07:51:26.121825 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:26.169313) v4 -- 0x55d9b9ac3200 con 0 > -5390> 2018-07-11 07:51:26.141397 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 925 0x55d98bf7cd80 rep_scrubmap(5.2e e501 from shard 2) v1 > -5389> 2018-07-11 07:51:26.141424 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 925 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12474 (1269249638 0 3276358747) 0x55d98bf7cd80 con 0x55d986bf0800 > -5388> 2018-07-11 07:51:26.147430 7ffacbbd1700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15263 conn(0x55d985d44000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.5 seq 233 0x55d986d33e00 osd_ping(ping e501 stamp 2018-07-11 07:51:27.378322) v4 > -5387> 2018-07-11 07:51:26.147454 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 <== osd.5 10.8.128.113:0/15263 233 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:27.378322) v4 ==== 2004+0+0 (3958798934 0 0) 0x55d986d33e00 con 0x55d985d44000 > -5386> 2018-07-11 07:51:26.147442 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15263 conn(0x55d985d45800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.5 seq 233 0x55d9829e5600 osd_ping(ping e501 stamp 2018-07-11 07:51:27.378322) v4 > -5385> 2018-07-11 07:51:26.147464 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:27.378322) v4 -- 0x55d9869bec00 con 0 > -5384> 2018-07-11 07:51:26.147464 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.5 10.8.128.113:0/15263 233 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:27.378322) v4 ==== 2004+0+0 (3958798934 0 0) 0x55d9829e5600 con 0x55d985d45800 > -5383> 2018-07-11 07:51:26.147492 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:27.378322) v4 -- 0x55d982cde200 con 0 > -5382> 2018-07-11 07:51:26.149266 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 919 0x55d98bf7cfc0 rep_scrubmap(5.2e e501 from shard 4) v1 > -5381> 2018-07-11 07:51:26.149286 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 919 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12474 (3686815630 0 3276358747) 0x55d98bf7cfc0 con 0x55d986bec000 > -5380> 2018-07-11 07:51:26.283175 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15287 conn(0x55d981240800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 261 0x55d9829e4800 osd_ping(ping e501 stamp 2018-07-11 07:51:27.514000) v4 > -5379> 2018-07-11 07:51:26.283196 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.4 10.8.128.113:0/15287 261 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:27.514000) v4 ==== 2004+0+0 (734359529 0 0) 0x55d9829e4800 con 0x55d981240800 > -5378> 2018-07-11 07:51:26.283205 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:27.514000) v4 -- 0x55d9b9ac3200 con 0 > -5377> 2018-07-11 07:51:26.283214 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15287 conn(0x55d98123f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 261 0x55d982cdf800 osd_ping(ping e501 stamp 2018-07-11 07:51:27.514000) v4 > -5376> 2018-07-11 07:51:26.283236 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.4 10.8.128.113:0/15287 261 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:27.514000) v4 ==== 2004+0+0 (734359529 0 0) 0x55d982cdf800 con 0x55d98123f000 > -5375> 2018-07-11 07:51:26.283255 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:27.514000) v4 -- 0x55d982cde200 con 0 > -5374> 2018-07-11 07:51:26.517312 7ffab1bdc700 5 osd.10 501 heartbeat: osd_stat(788 GB used, 142 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,0,2]) > -5373> 2018-07-11 07:51:26.517327 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6802/15255 -- osd_ping(ping e501 stamp 2018-07-11 07:51:26.517325) v4 -- 0x55d986d4fe00 con 0 > -5372> 2018-07-11 07:51:26.517363 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6803/15255 -- osd_ping(ping e501 stamp 2018-07-11 07:51:26.517325) v4 -- 0x55d9b60dfa00 con 0 > -5371> 2018-07-11 07:51:26.517376 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6806/12677 -- osd_ping(ping e501 stamp 2018-07-11 07:51:26.517325) v4 -- 0x55d9829e4e00 con 0 > -5370> 2018-07-11 07:51:26.517390 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6807/12677 -- osd_ping(ping e501 stamp 2018-07-11 07:51:26.517325) v4 -- 0x55d9b60df400 con 0 > -5369> 2018-07-11 07:51:26.517400 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6808/12651 -- osd_ping(ping e501 stamp 2018-07-11 07:51:26.517325) v4 -- 0x55d9b60df000 con 0 > -5368> 2018-07-11 07:51:26.517424 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6810/12651 -- osd_ping(ping e501 stamp 2018-07-11 07:51:26.517325) v4 -- 0x55d980457a00 con 0 > -5367> 2018-07-11 07:51:26.517435 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6809/12650 -- osd_ping(ping e501 stamp 2018-07-11 07:51:26.517325) v4 -- 0x55d9c3383a00 con 0 > -5366> 2018-07-11 07:51:26.517446 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6811/12650 -- osd_ping(ping e501 stamp 2018-07-11 07:51:26.517325) v4 -- 0x55d9bdba4800 con 0 > -5365> 2018-07-11 07:51:26.517459 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6810/15287 -- osd_ping(ping e501 stamp 2018-07-11 07:51:26.517325) v4 -- 0x55d982d30600 con 0 > -5364> 2018-07-11 07:51:26.517471 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6811/15287 -- osd_ping(ping e501 stamp 2018-07-11 07:51:26.517325) v4 -- 0x55d9b60df600 con 0 > -5363> 2018-07-11 07:51:26.517490 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6806/15263 -- osd_ping(ping e501 stamp 2018-07-11 07:51:26.517325) v4 -- 0x55d982d31c00 con 0 > -5362> 2018-07-11 07:51:26.517503 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6807/15263 -- osd_ping(ping e501 stamp 2018-07-11 07:51:26.517325) v4 -- 0x55d9b4388200 con 0 > -5361> 2018-07-11 07:51:26.517515 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.115:6802/13323 -- osd_ping(ping e501 stamp 2018-07-11 07:51:26.517325) v4 -- 0x55d9829e5a00 con 0 > -5360> 2018-07-11 07:51:26.517528 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.115:6803/13323 -- osd_ping(ping e501 stamp 2018-07-11 07:51:26.517325) v4 -- 0x55d9b3db2600 con 0 > -5359> 2018-07-11 07:51:26.517541 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6806/15885 -- osd_ping(ping e501 stamp 2018-07-11 07:51:26.517325) v4 -- 0x55d9c9123000 con 0 > -5358> 2018-07-11 07:51:26.517553 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6807/15885 -- osd_ping(ping e501 stamp 2018-07-11 07:51:26.517325) v4 -- 0x55d986d40600 con 0 > -5357> 2018-07-11 07:51:26.517566 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.108:6802/11967 -- osd_ping(ping e501 stamp 2018-07-11 07:51:26.517325) v4 -- 0x55d9bdba5600 con 0 > -5356> 2018-07-11 07:51:26.517581 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.108:6803/11967 -- osd_ping(ping e501 stamp 2018-07-11 07:51:26.517325) v4 -- 0x55d9c4c1be00 con 0 > -5355> 2018-07-11 07:51:26.517594 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.107:6806/12617 -- osd_ping(ping e501 stamp 2018-07-11 07:51:26.517325) v4 -- 0x55d982cdea00 con 0 > -5354> 2018-07-11 07:51:26.517607 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.107:6807/12617 -- osd_ping(ping e501 stamp 2018-07-11 07:51:26.517325) v4 -- 0x55d986d97000 con 0 > -5353> 2018-07-11 07:51:26.517622 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6802/15919 -- osd_ping(ping e501 stamp 2018-07-11 07:51:26.517325) v4 -- 0x55d986d4e400 con 0 > -5352> 2018-07-11 07:51:26.517634 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6803/15919 -- osd_ping(ping e501 stamp 2018-07-11 07:51:26.517325) v4 -- 0x55d986d97e00 con 0 > -5351> 2018-07-11 07:51:26.517794 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.107:6806/12617 conn(0x55d986bd2000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.9 seq 241 0x55d9829e5600 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:26.517325) v4 > -5350> 2018-07-11 07:51:26.517816 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.9 10.8.128.107:6806/12617 241 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:26.517325) v4 ==== 2004+0+0 (2092291059 0 0) 0x55d9829e5600 con 0x55d986bd2000 > -5349> 2018-07-11 07:51:26.517832 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.107:6807/12617 conn(0x55d986bd3800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.9 seq 241 0x55d9b60df800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:26.517325) v4 > -5348> 2018-07-11 07:51:26.517851 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.9 10.8.128.107:6807/12617 241 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:26.517325) v4 ==== 2004+0+0 (2092291059 0 0) 0x55d9b60df800 con 0x55d986bd3800 > -5347> 2018-07-11 07:51:26.517902 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6803/15255 conn(0x55d986bc9800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.0 seq 241 0x55d9829e5600 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:26.517325) v4 > -5346> 2018-07-11 07:51:26.517917 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.0 10.8.128.113:6803/15255 241 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:26.517325) v4 ==== 2004+0+0 (2092291059 0 0) 0x55d9829e5600 con 0x55d986bc9800 > -5345> 2018-07-11 07:51:26.517964 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6810/12651 conn(0x55d986b9b800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.2 seq 241 0x55d9829e5600 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:26.517325) v4 > -5344> 2018-07-11 07:51:26.517958 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6802/15255 conn(0x55d986bc8000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.0 seq 241 0x55d986d96e00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:26.517325) v4 > -5343> 2018-07-11 07:51:26.517978 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.2 10.8.128.106:6810/12651 241 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:26.517325) v4 ==== 2004+0+0 (2092291059 0 0) 0x55d9829e5600 con 0x55d986b9b800 > -5342> 2018-07-11 07:51:26.517989 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.0 10.8.128.113:6802/15255 241 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:26.517325) v4 ==== 2004+0+0 (2092291059 0 0) 0x55d986d96e00 con 0x55d986bc8000 > -5341> 2018-07-11 07:51:26.517986 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6809/12650 conn(0x55d986bcb000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 241 0x55d9b60df800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:26.517325) v4 > -5340> 2018-07-11 07:51:26.518018 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.3 10.8.128.106:6809/12650 241 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:26.517325) v4 ==== 2004+0+0 (2092291059 0 0) 0x55d9b60df800 con 0x55d986bcb000 > -5339> 2018-07-11 07:51:26.518029 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6807/15263 conn(0x55d986b9e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.5 seq 241 0x55d986d96e00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:26.517325) v4 > -5338> 2018-07-11 07:51:26.518042 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.5 10.8.128.113:6807/15263 241 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:26.517325) v4 ==== 2004+0+0 (2092291059 0 0) 0x55d986d96e00 con 0x55d986b9e800 > -5337> 2018-07-11 07:51:26.518050 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6811/15287 conn(0x55d986bb0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 241 0x55d9829e5600 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:26.517325) v4 > -5336> 2018-07-11 07:51:26.518071 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.4 10.8.128.113:6811/15287 241 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:26.517325) v4 ==== 2004+0+0 (2092291059 0 0) 0x55d9829e5600 con 0x55d986bb0800 > -5335> 2018-07-11 07:51:26.518072 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6808/12651 conn(0x55d986b9a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.2 seq 241 0x55d986d96e00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:26.517325) v4 > -5334> 2018-07-11 07:51:26.518088 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.2 10.8.128.106:6808/12651 241 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:26.517325) v4 ==== 2004+0+0 (2092291059 0 0) 0x55d986d96e00 con 0x55d986b9a000 > -5333> 2018-07-11 07:51:26.518078 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6806/15263 conn(0x55d986b9d000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.5 seq 241 0x55d9b60df800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:26.517325) v4 > -5332> 2018-07-11 07:51:26.518104 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.5 10.8.128.113:6806/15263 241 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:26.517325) v4 ==== 2004+0+0 (2092291059 0 0) 0x55d9b60df800 con 0x55d986b9d000 > -5331> 2018-07-11 07:51:26.518126 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.108:6803/11967 conn(0x55d986bbf800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.8 seq 241 0x55d986d96e00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:26.517325) v4 > -5330> 2018-07-11 07:51:26.518139 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6806/12677 conn(0x55d986b58000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.1 seq 241 0x55d9b60df800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:26.517325) v4 > -5329> 2018-07-11 07:51:26.518151 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.1 10.8.128.106:6806/12677 241 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:26.517325) v4 ==== 2004+0+0 (2092291059 0 0) 0x55d9b60df800 con 0x55d986b58000 > -5328> 2018-07-11 07:51:26.518138 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.8 10.8.128.108:6803/11967 241 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:26.517325) v4 ==== 2004+0+0 (2092291059 0 0) 0x55d986d96e00 con 0x55d986bbf800 > -5327> 2018-07-11 07:51:26.518168 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.115:6802/13323 conn(0x55d986bc1000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 241 0x55d9829e5600 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:26.517325) v4 > -5326> 2018-07-11 07:51:26.518185 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.6 10.8.128.115:6802/13323 241 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:26.517325) v4 ==== 2004+0+0 (2092291059 0 0) 0x55d9829e5600 con 0x55d986bc1000 > -5325> 2018-07-11 07:51:26.518192 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6807/12677 conn(0x55d986b59800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.1 seq 241 0x55d986d96e00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:26.517325) v4 > -5324> 2018-07-11 07:51:26.518197 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.115:6803/13323 conn(0x55d986bc2800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 241 0x55d9b60df800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:26.517325) v4 > -5323> 2018-07-11 07:51:26.518205 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.1 10.8.128.106:6807/12677 241 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:26.517325) v4 ==== 2004+0+0 (2092291059 0 0) 0x55d986d96e00 con 0x55d986b59800 > -5322> 2018-07-11 07:51:26.518209 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.6 10.8.128.115:6803/13323 241 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:26.517325) v4 ==== 2004+0+0 (2092291059 0 0) 0x55d9b60df800 con 0x55d986bc2800 > -5321> 2018-07-11 07:51:26.518238 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.108:6802/11967 conn(0x55d986bbe000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.8 seq 241 0x55d9b60df800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:26.517325) v4 > -5320> 2018-07-11 07:51:26.518240 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6811/12650 conn(0x55d986bcc800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 241 0x55d986d96e00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:26.517325) v4 > -5319> 2018-07-11 07:51:26.518258 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.8 10.8.128.108:6802/11967 241 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:26.517325) v4 ==== 2004+0+0 (2092291059 0 0) 0x55d9b60df800 con 0x55d986bbe000 > -5318> 2018-07-11 07:51:26.518262 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.3 10.8.128.106:6811/12650 241 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:26.517325) v4 ==== 2004+0+0 (2092291059 0 0) 0x55d986d96e00 con 0x55d986bcc800 > -5317> 2018-07-11 07:51:26.518287 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6810/15287 conn(0x55d986baf000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 241 0x55d986d96e00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:26.517325) v4 > -5316> 2018-07-11 07:51:26.518294 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6807/15885 conn(0x55d986bad800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.7 seq 241 0x55d9b60df800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:26.517325) v4 > -5315> 2018-07-11 07:51:26.518298 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.4 10.8.128.113:6810/15287 241 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:26.517325) v4 ==== 2004+0+0 (2092291059 0 0) 0x55d986d96e00 con 0x55d986baf000 > -5314> 2018-07-11 07:51:26.518292 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6802/15919 conn(0x55d986b5b000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.11 seq 241 0x55d9829e5600 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:26.517325) v4 > -5313> 2018-07-11 07:51:26.518309 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.7 10.8.128.114:6807/15885 241 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:26.517325) v4 ==== 2004+0+0 (2092291059 0 0) 0x55d9b60df800 con 0x55d986bad800 > -5312> 2018-07-11 07:51:26.518315 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.11 10.8.128.114:6802/15919 241 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:26.517325) v4 ==== 2004+0+0 (2092291059 0 0) 0x55d9829e5600 con 0x55d986b5b000 > -5311> 2018-07-11 07:51:26.518365 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6806/15885 conn(0x55d986bac000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.7 seq 241 0x55d9829e5600 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:26.517325) v4 > -5310> 2018-07-11 07:51:26.518375 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6803/15919 conn(0x55d986b5c800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.11 seq 241 0x55d9b60df800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:26.517325) v4 > -5309> 2018-07-11 07:51:26.518380 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.7 10.8.128.114:6806/15885 241 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:26.517325) v4 ==== 2004+0+0 (2092291059 0 0) 0x55d9829e5600 con 0x55d986bac000 > -5308> 2018-07-11 07:51:26.518389 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.11 10.8.128.114:6803/15919 241 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:26.517325) v4 ==== 2004+0+0 (2092291059 0 0) 0x55d9b60df800 con 0x55d986b5c800 > -5307> 2018-07-11 07:51:26.563062 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6805/15885 conn(0x55d986c7f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.7 seq 2794 0x55d986d96e00 MOSDScrubReserve(5.4 REQUEST e501) v1 > -5306> 2018-07-11 07:51:26.563081 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.7 10.8.128.114:6805/15885 2794 ==== MOSDScrubReserve(5.4 REQUEST e501) v1 ==== 43+0+0 (1753221512 0 0) 0x55d986d96e00 con 0x55d986c7f800 > -5305> 2018-07-11 07:51:26.563141 7ffab33df700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.114:6805/15885 -- MOSDScrubReserve(5.4 REJECT e501) v1 -- 0x55d986d32200 con 0 > -5304> 2018-07-11 07:51:26.563452 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6805/15885 conn(0x55d986c7f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.7 seq 2795 0x55d986d32200 MOSDScrubReserve(5.4 RELEASE e501) v1 > -5303> 2018-07-11 07:51:26.563467 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.7 10.8.128.114:6805/15885 2795 ==== MOSDScrubReserve(5.4 RELEASE e501) v1 ==== 43+0+0 (2812146570 0 0) 0x55d986d32200 con 0x55d986c7f800 > -5302> 2018-07-11 07:51:26.787903 7ffacbbd1700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.108:0/11967 conn(0x55d986d82800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.8 seq 231 0x55d9b60df800 osd_ping(ping e501 stamp 2018-07-11 07:51:26.543518) v4 > -5301> 2018-07-11 07:51:26.787924 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 <== osd.8 10.8.128.108:0/11967 231 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:26.543518) v4 ==== 2004+0+0 (1291892185 0 0) 0x55d9b60df800 con 0x55d986d82800 > -5300> 2018-07-11 07:51:26.787912 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.108:0/11967 conn(0x55d986db0000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.8 seq 231 0x55d9829e5600 osd_ping(ping e501 stamp 2018-07-11 07:51:26.543518) v4 > -5299> 2018-07-11 07:51:26.787934 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:26.543518) v4 -- 0x55d986d97e00 con 0 > -5298> 2018-07-11 07:51:26.787935 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.8 10.8.128.108:0/11967 231 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:26.543518) v4 ==== 2004+0+0 (1291892185 0 0) 0x55d9829e5600 con 0x55d986db0000 > -5297> 2018-07-11 07:51:26.787957 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:26.543518) v4 -- 0x55d986d4e400 con 0 > -5296> 2018-07-11 07:51:26.791141 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.114:0/15919 conn(0x55d986df3800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.11 seq 249 0x55d986d33e00 osd_ping(ping e501 stamp 2018-07-11 07:51:27.421307) v4 > -5295> 2018-07-11 07:51:26.791163 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.11 10.8.128.114:0/15919 249 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:27.421307) v4 ==== 2004+0+0 (1735157368 0 0) 0x55d986d33e00 con 0x55d986df3800 > -5294> 2018-07-11 07:51:26.791174 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:27.421307) v4 -- 0x55d986d97e00 con 0 > -5293> 2018-07-11 07:51:26.791188 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.114:0/15919 conn(0x55d986df5000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.11 seq 249 0x55d982cdf800 osd_ping(ping e501 stamp 2018-07-11 07:51:27.421307) v4 > -5292> 2018-07-11 07:51:26.791209 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.11 10.8.128.114:0/15919 249 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:27.421307) v4 ==== 2004+0+0 (1735157368 0 0) 0x55d982cdf800 con 0x55d986df5000 > -5291> 2018-07-11 07:51:26.791220 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:27.421307) v4 -- 0x55d986d4e400 con 0 > -5290> 2018-07-11 07:51:26.837855 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 3002 0x55d9b60df800 MOSDScrubReserve(11.3s5 REQUEST e501) v1 > -5289> 2018-07-11 07:51:26.837872 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 3002 ==== MOSDScrubReserve(11.3s5 REQUEST e501) v1 ==== 43+0+0 (3260170403 0 0) 0x55d9b60df800 con 0x55d986c7e000 > -5288> 2018-07-11 07:51:26.837991 7ffab3be0700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e501) v1 -- 0x55d9869bfc00 con 0 > -5287> 2018-07-11 07:51:26.838283 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 3003 0x55d986d97e00 MOSDScrubReserve(11.3s5 RELEASE e501) v1 > -5286> 2018-07-11 07:51:26.838312 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 3003 ==== MOSDScrubReserve(11.3s5 RELEASE e501) v1 ==== 43+0+0 (223328929 0 0) 0x55d986d97e00 con 0x55d986c7e000 > -5285> 2018-07-11 07:51:26.960155 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6805/15263 conn(0x55d986c41000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=31 cs=1 l=0). rx osd.5 seq 2814 0x55d9829e4800 MOSDScrubReserve(6.15 REQUEST e501) v1 > -5284> 2018-07-11 07:51:26.960173 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.5 10.8.128.113:6805/15263 2814 ==== MOSDScrubReserve(6.15 REQUEST e501) v1 ==== 43+0+0 (3942742026 0 0) 0x55d9829e4800 con 0x55d986c41000 > -5283> 2018-07-11 07:51:27.015170 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12650 conn(0x55d986d23000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 242 0x55d9829e5600 osd_ping(ping e501 stamp 2018-07-11 07:51:27.062725) v4 > -5282> 2018-07-11 07:51:27.015193 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.3 10.8.128.106:0/12650 242 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:27.062725) v4 ==== 2004+0+0 (2581574663 0 0) 0x55d9829e5600 con 0x55d986d23000 > -5281> 2018-07-11 07:51:27.015209 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:27.062725) v4 -- 0x55d986d4e400 con 0 > -5280> 2018-07-11 07:51:27.015228 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12650 conn(0x55d986d21800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 242 0x55d986d33e00 osd_ping(ping e501 stamp 2018-07-11 07:51:27.062725) v4 > -5279> 2018-07-11 07:51:27.015256 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.3 10.8.128.106:0/12650 242 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:27.062725) v4 ==== 2004+0+0 (2581574663 0 0) 0x55d986d33e00 con 0x55d986d21800 > -5278> 2018-07-11 07:51:27.015268 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:27.062725) v4 -- 0x55d986d97000 con 0 > -5277> 2018-07-11 07:51:27.289403 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 31] Generated table #36801: 3106037 keys, 75716797 bytes > -5276> 2018-07-11 07:51:27.289450 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295487289423, "cf_name": "default", "job": 31, "event": "table_file_creation", "file_number": 36801, "file_size": 75716797, "table_properties": {"data_size": 67111720, "index_size": 689754, "filter_size": 7914331, "raw_key_size": 89729961, "raw_average_key_size": 28, "raw_value_size": 25038442, "raw_average_value_size": 8, "num_data_blocks": 16486, "num_entries": 3106037, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1000000", "kMergeOperands": "0"}} > -5275> 2018-07-11 07:51:27.291519 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.114:0/15919 conn(0x55d986df3800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.11 seq 250 0x55d9869bfc00 osd_ping(ping e501 stamp 2018-07-11 07:51:27.921743) v4 > -5274> 2018-07-11 07:51:27.291545 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.11 10.8.128.114:0/15919 250 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:27.921743) v4 ==== 2004+0+0 (445006450 0 0) 0x55d9869bfc00 con 0x55d986df3800 > -5273> 2018-07-11 07:51:27.291563 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:27.921743) v4 -- 0x55d986d97000 con 0 > -5272> 2018-07-11 07:51:27.291582 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.114:0/15919 conn(0x55d986df5000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.11 seq 250 0x55d982cdf800 osd_ping(ping e501 stamp 2018-07-11 07:51:27.921743) v4 > -5271> 2018-07-11 07:51:27.291608 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.11 10.8.128.114:0/15919 250 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:27.921743) v4 ==== 2004+0+0 (445006450 0 0) 0x55d982cdf800 con 0x55d986df5000 > -5270> 2018-07-11 07:51:27.291624 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:27.921743) v4 -- 0x55d986d4e400 con 0 > -5269> 2018-07-11 07:51:27.299799 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6805/15263 -- MOSDScrubReserve(6.15 REJECT e501) v1 -- 0x55d9829e4200 con 0 > -5268> 2018-07-11 07:51:27.299863 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -5267> 2018-07-11 07:51:27.299888 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -5266> 2018-07-11 07:51:27.299934 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -5265> 2018-07-11 07:51:27.300231 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6805/15263 conn(0x55d986c41000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=31 cs=1 l=0). rx osd.5 seq 2815 0x55d9829e4200 MOSDScrubReserve(6.15 RELEASE e501) v1 > -5264> 2018-07-11 07:51:27.300257 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.5 10.8.128.113:6805/15263 2815 ==== MOSDScrubReserve(6.15 RELEASE e501) v1 ==== 43+0+0 (605848072 0 0) 0x55d9829e4200 con 0x55d986c41000 > -5263> 2018-07-11 07:51:27.300410 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -5262> 2018-07-11 07:51:27.316667 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67845,epoch:501/498,start:5:754ac15a:::benchmark_data_magna115_10867_object1003752:0,end:5:754b2506:::benchmark_data_magna115_10867_object2973661:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498500 con 0 > -5261> 2018-07-11 07:51:27.316699 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67845,epoch:501/498,start:5:754ac15a:::benchmark_data_magna115_10867_object1003752:0,end:5:754b2506:::benchmark_data_magna115_10867_object2973661:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1499900 con 0 > -5260> 2018-07-11 07:51:27.350048 7ffab6be6700 1 -- 10.8.128.107:6800/13927 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55d983c49b80 con 0 > -5259> 2018-07-11 07:51:27.350122 7ffab6be6700 1 -- 10.8.128.107:6800/13927 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55d9869d7600 con 0 > -5258> 2018-07-11 07:51:27.557292 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 926 0x55d98bf7d200 rep_scrubmap(5.2e e501 from shard 2) v1 > -5257> 2018-07-11 07:51:27.557312 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 926 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12466 (1269249638 0 4158432551) 0x55d98bf7d200 con 0x55d986bf0800 > -5256> 2018-07-11 07:51:27.574123 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 920 0x55d98bf7d440 rep_scrubmap(5.2e e501 from shard 4) v1 > -5255> 2018-07-11 07:51:27.574143 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 920 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12466 (3686815630 0 4158432551) 0x55d98bf7d440 con 0x55d986bec000 > -5254> 2018-07-11 07:51:27.594349 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -5253> 2018-07-11 07:51:27.594374 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -5252> 2018-07-11 07:51:27.594434 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -5251> 2018-07-11 07:51:27.594898 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -5250> 2018-07-11 07:51:27.617135 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:754b2506:::benchmark_data_magna115_10867_object2973661:0,end:5:754b92a9:::benchmark_data_magna115_10867_object1599929:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1499680 con 0 > -5249> 2018-07-11 07:51:27.617183 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:754b2506:::benchmark_data_magna115_10867_object2973661:0,end:5:754b92a9:::benchmark_data_magna115_10867_object1599929:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b14f0000 con 0 > -5248> 2018-07-11 07:51:27.805903 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 927 0x55d98bf7d680 rep_scrubmap(5.2e e501 from shard 2) v1 > -5247> 2018-07-11 07:51:27.805925 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 927 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12468 (1269249638 0 2602561421) 0x55d98bf7d680 con 0x55d986bf0800 > -5246> 2018-07-11 07:51:27.822301 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12651 conn(0x55d986db3000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.2 seq 249 0x55d986d33e00 osd_ping(ping e501 stamp 2018-07-11 07:51:27.869812) v4 > -5245> 2018-07-11 07:51:27.822326 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.2 10.8.128.106:0/12651 249 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:27.869812) v4 ==== 2004+0+0 (2376582757 0 0) 0x55d986d33e00 con 0x55d986db3000 > -5244> 2018-07-11 07:51:27.822338 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:27.869812) v4 -- 0x55d986d97000 con 0 > -5243> 2018-07-11 07:51:27.822437 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12651 conn(0x55d986db1800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.2 seq 249 0x55d9b4388200 osd_ping(ping e501 stamp 2018-07-11 07:51:27.869812) v4 > -5242> 2018-07-11 07:51:27.822456 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.2 10.8.128.106:0/12651 249 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:27.869812) v4 ==== 2004+0+0 (2376582757 0 0) 0x55d9b4388200 con 0x55d986db1800 > -5241> 2018-07-11 07:51:27.822472 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:27.869812) v4 -- 0x55d982d30600 con 0 > -5240> 2018-07-11 07:51:27.836755 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 921 0x55d98bf7d8c0 rep_scrubmap(5.2e e501 from shard 4) v1 > -5239> 2018-07-11 07:51:27.836775 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 921 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12468 (3686815630 0 2602561421) 0x55d98bf7d8c0 con 0x55d986bec000 > -5238> 2018-07-11 07:51:27.900469 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -5237> 2018-07-11 07:51:27.900495 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -5236> 2018-07-11 07:51:27.900542 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -5235> 2018-07-11 07:51:27.901006 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -5234> 2018-07-11 07:51:27.950233 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67976,epoch:501/498,start:5:754b92a9:::benchmark_data_magna115_10867_object1599929:0,end:5:754c010b:::benchmark_data_magna115_11707_object196617:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b14f0280 con 0 > -5233> 2018-07-11 07:51:27.950279 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67976,epoch:501/498,start:5:754b92a9:::benchmark_data_magna115_10867_object1599929:0,end:5:754c010b:::benchmark_data_magna115_11707_object196617:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98696f400 con 0 > -5232> 2018-07-11 07:51:27.960322 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6805/15263 conn(0x55d986c41000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=31 cs=1 l=0). rx osd.5 seq 2816 0x55d9b4388200 MOSDScrubReserve(6.15 REQUEST e501) v1 > -5231> 2018-07-11 07:51:27.960341 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.5 10.8.128.113:6805/15263 2816 ==== MOSDScrubReserve(6.15 REQUEST e501) v1 ==== 43+0+0 (3942742026 0 0) 0x55d9b4388200 con 0x55d986c41000 > -5230> 2018-07-11 07:51:28.063701 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12677 conn(0x55d986df2000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.1 seq 253 0x55d9c4c1be00 osd_ping(ping e501 stamp 2018-07-11 07:51:28.111248) v4 > -5229> 2018-07-11 07:51:28.063723 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.1 10.8.128.106:0/12677 253 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:28.111248) v4 ==== 2004+0+0 (2768763504 0 0) 0x55d9c4c1be00 con 0x55d986df2000 > -5228> 2018-07-11 07:51:28.063722 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12677 conn(0x55d986db4800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.1 seq 253 0x55d9829e5600 osd_ping(ping e501 stamp 2018-07-11 07:51:28.111248) v4 > -5227> 2018-07-11 07:51:28.063734 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:28.111248) v4 -- 0x55d9bdba4800 con 0 > -5226> 2018-07-11 07:51:28.063745 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.1 10.8.128.106:0/12677 253 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:28.111248) v4 ==== 2004+0+0 (2768763504 0 0) 0x55d9829e5600 con 0x55d986db4800 > -5225> 2018-07-11 07:51:28.063772 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:28.111248) v4 -- 0x55d986d4e400 con 0 > -5224> 2018-07-11 07:51:28.170977 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 928 0x55d98bf7db00 rep_scrubmap(5.2e e501 from shard 2) v1 > -5223> 2018-07-11 07:51:28.170996 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 928 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12470 (1269249638 0 42828504) 0x55d98bf7db00 con 0x55d986bf0800 > -5222> 2018-07-11 07:51:28.189195 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 922 0x55d98bf7dd40 rep_scrubmap(5.2e e501 from shard 4) v1 > -5221> 2018-07-11 07:51:28.189214 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 922 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12470 (3686815630 0 42828504) 0x55d98bf7dd40 con 0x55d986bec000 > -5220> 2018-07-11 07:51:28.361577 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 31] Generated table #36802: 532207 keys, 15038478 bytes > -5219> 2018-07-11 07:51:28.361617 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295488361591, "cf_name": "default", "job": 31, "event": "table_file_creation", "file_number": 36802, "file_size": 15038478, "table_properties": {"data_size": 13538404, "index_size": 138798, "filter_size": 1360290, "raw_key_size": 15374875, "raw_average_key_size": 28, "raw_value_size": 6327356, "raw_average_value_size": 11, "num_data_blocks": 3327, "num_entries": 532207, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} > -5218> 2018-07-11 07:51:28.519454 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6805/15263 -- MOSDScrubReserve(6.15 REJECT e501) v1 -- 0x55d9829e4200 con 0 > -5217> 2018-07-11 07:51:28.519658 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -5216> 2018-07-11 07:51:28.519683 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -5215> 2018-07-11 07:51:28.519728 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -5214> 2018-07-11 07:51:28.519923 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6805/15263 conn(0x55d986c41000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=31 cs=1 l=0). rx osd.5 seq 2817 0x55d9829e4200 MOSDScrubReserve(6.15 RELEASE e501) v1 > -5213> 2018-07-11 07:51:28.519942 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.5 10.8.128.113:6805/15263 2817 ==== MOSDScrubReserve(6.15 RELEASE e501) v1 ==== 43+0+0 (605848072 0 0) 0x55d9829e4200 con 0x55d986c41000 > -5212> 2018-07-11 07:51:28.520186 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -5211> 2018-07-11 07:51:28.542270 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67092,epoch:501/498,start:5:754c010b:::benchmark_data_magna115_11707_object196617:0,end:5:754c6001:::benchmark_data_magna115_10867_object3419571:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498780 con 0 > -5210> 2018-07-11 07:51:28.542308 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67092,epoch:501/498,start:5:754c010b:::benchmark_data_magna115_11707_object196617:0,end:5:754c6001:::benchmark_data_magna115_10867_object3419571:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b14f0500 con 0 > -5209> 2018-07-11 07:51:28.583655 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15287 conn(0x55d98123f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 262 0x55d982cdf800 osd_ping(ping e501 stamp 2018-07-11 07:51:29.814592) v4 > -5208> 2018-07-11 07:51:28.583670 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15287 conn(0x55d981240800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 262 0x55d9c4c1be00 osd_ping(ping e501 stamp 2018-07-11 07:51:29.814592) v4 > -5207> 2018-07-11 07:51:28.583682 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.4 10.8.128.113:0/15287 262 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:29.814592) v4 ==== 2004+0+0 (2977255286 0 0) 0x55d982cdf800 con 0x55d98123f000 > -5206> 2018-07-11 07:51:28.583687 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.4 10.8.128.113:0/15287 262 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:29.814592) v4 ==== 2004+0+0 (2977255286 0 0) 0x55d9c4c1be00 con 0x55d981240800 > -5205> 2018-07-11 07:51:28.583696 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:29.814592) v4 -- 0x55d986d4e400 con 0 > -5204> 2018-07-11 07:51:28.583726 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:29.814592) v4 -- 0x55d9bdba4800 con 0 > -5203> 2018-07-11 07:51:28.749024 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 923 0x55d9b5c5a000 rep_scrubmap(5.2e e501 from shard 4) v1 > -5202> 2018-07-11 07:51:28.749044 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 923 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12474 (3686815630 0 787847078) 0x55d9b5c5a000 con 0x55d986bec000 > -5201> 2018-07-11 07:51:28.760130 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 929 0x55d9b5c5a240 rep_scrubmap(5.2e e501 from shard 2) v1 > -5200> 2018-07-11 07:51:28.760150 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 929 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12474 (1269249638 0 787847078) 0x55d9b5c5a240 con 0x55d986bf0800 > -5199> 2018-07-11 07:51:28.807838 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -5198> 2018-07-11 07:51:28.807863 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -5197> 2018-07-11 07:51:28.807921 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -5196> 2018-07-11 07:51:28.808414 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -5195> 2018-07-11 07:51:28.809431 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67610,epoch:501/498,start:5:754c6001:::benchmark_data_magna115_10867_object3419571:0,end:5:754cabe3:::benchmark_data_magna115_11707_object298065:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455900 con 0 > -5194> 2018-07-11 07:51:28.809466 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67610,epoch:501/498,start:5:754c6001:::benchmark_data_magna115_10867_object3419571:0,end:5:754cabe3:::benchmark_data_magna115_11707_object298065:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455680 con 0 > -5193> 2018-07-11 07:51:28.960510 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6805/15263 conn(0x55d986c41000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=31 cs=1 l=0). rx osd.5 seq 2818 0x55d9c4c1be00 MOSDScrubReserve(6.15 REQUEST e501) v1 > -5192> 2018-07-11 07:51:28.960529 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.5 10.8.128.113:6805/15263 2818 ==== MOSDScrubReserve(6.15 REQUEST e501) v1 ==== 43+0+0 (3942742026 0 0) 0x55d9c4c1be00 con 0x55d986c41000 > -5191> 2018-07-11 07:51:29.017703 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.114:0/15885 conn(0x55d986d7f800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.7 seq 255 0x55d9829e5600 osd_ping(ping e501 stamp 2018-07-11 07:51:29.647895) v4 > -5190> 2018-07-11 07:51:29.017713 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.114:0/15885 conn(0x55d986d81000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.7 seq 255 0x55d982d30600 osd_ping(ping e501 stamp 2018-07-11 07:51:29.647895) v4 > -5189> 2018-07-11 07:51:29.017730 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.7 10.8.128.114:0/15885 255 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:29.647895) v4 ==== 2004+0+0 (16709039 0 0) 0x55d982d30600 con 0x55d986d81000 > -5188> 2018-07-11 07:51:29.017728 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.7 10.8.128.114:0/15885 255 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:29.647895) v4 ==== 2004+0+0 (16709039 0 0) 0x55d9829e5600 con 0x55d986d7f800 > -5187> 2018-07-11 07:51:29.017748 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:29.647895) v4 -- 0x55d9b60df000 con 0 > -5186> 2018-07-11 07:51:29.017775 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:29.647895) v4 -- 0x55d986d4e400 con 0 > -5185> 2018-07-11 07:51:29.048313 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 930 0x55d9b5c5a480 rep_scrubmap(5.2e e501 from shard 2) v1 > -5184> 2018-07-11 07:51:29.048333 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 930 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12472 (1269249638 0 4103807136) 0x55d9b5c5a480 con 0x55d986bf0800 > -5183> 2018-07-11 07:51:29.049514 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 924 0x55d9b5c5a6c0 rep_scrubmap(5.2e e501 from shard 4) v1 > -5182> 2018-07-11 07:51:29.049545 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 924 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12472 (3686815630 0 4103807136) 0x55d9b5c5a6c0 con 0x55d986bec000 > -5181> 2018-07-11 07:51:29.125908 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6805/15263 -- MOSDScrubReserve(6.15 REJECT e501) v1 -- 0x55d9829e4200 con 0 > -5180> 2018-07-11 07:51:29.126091 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -5179> 2018-07-11 07:51:29.126113 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -5178> 2018-07-11 07:51:29.126158 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -5177> 2018-07-11 07:51:29.126328 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6805/15263 conn(0x55d986c41000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=31 cs=1 l=0). rx osd.5 seq 2819 0x55d9829e4200 MOSDScrubReserve(6.15 RELEASE e501) v1 > -5176> 2018-07-11 07:51:29.126347 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.5 10.8.128.113:6805/15263 2819 ==== MOSDScrubReserve(6.15 RELEASE e501) v1 ==== 43+0+0 (605848072 0 0) 0x55d9829e4200 con 0x55d986c41000 > -5175> 2018-07-11 07:51:29.126624 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -5174> 2018-07-11 07:51:29.142283 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67987,epoch:501/498,start:5:754cabe3:::benchmark_data_magna115_11707_object298065:0,end:5:754d2fd1:::benchmark_data_magna115_10867_object380695:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455b80 con 0 > -5173> 2018-07-11 07:51:29.142320 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67987,epoch:501/498,start:5:754cabe3:::benchmark_data_magna115_11707_object298065:0,end:5:754d2fd1:::benchmark_data_magna115_10867_object380695:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498000 con 0 > -5172> 2018-07-11 07:51:29.164229 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12677 conn(0x55d986df2000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.1 seq 254 0x55d9bdba4800 osd_ping(ping e501 stamp 2018-07-11 07:51:29.211788) v4 > -5171> 2018-07-11 07:51:29.164230 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12677 conn(0x55d986db4800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.1 seq 254 0x55d982cdf800 osd_ping(ping e501 stamp 2018-07-11 07:51:29.211788) v4 > -5170> 2018-07-11 07:51:29.164270 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.1 10.8.128.106:0/12677 254 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:29.211788) v4 ==== 2004+0+0 (1668918177 0 0) 0x55d9bdba4800 con 0x55d986df2000 > -5169> 2018-07-11 07:51:29.164273 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.1 10.8.128.106:0/12677 254 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:29.211788) v4 ==== 2004+0+0 (1668918177 0 0) 0x55d982cdf800 con 0x55d986db4800 > -5168> 2018-07-11 07:51:29.164285 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:29.211788) v4 -- 0x55d9b60df000 con 0 > -5167> 2018-07-11 07:51:29.164310 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:29.211788) v4 -- 0x55d986d4e400 con 0 > -5166> 2018-07-11 07:51:29.329965 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 931 0x55d9b5c5a900 rep_scrubmap(5.2e e501 from shard 2) v1 > -5165> 2018-07-11 07:51:29.329985 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 931 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12476 (1269249638 0 2697512628) 0x55d9b5c5a900 con 0x55d986bf0800 > -5164> 2018-07-11 07:51:29.362111 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 925 0x55d9b5c5ab40 rep_scrubmap(5.2e e501 from shard 4) v1 > -5163> 2018-07-11 07:51:29.362132 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 925 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12476 (3686815630 0 2697512628) 0x55d9b5c5ab40 con 0x55d986bec000 > -5162> 2018-07-11 07:51:29.600939 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.115:0/13323 conn(0x55d986d7e000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 250 0x55d9869bfc00 osd_ping(ping e501 stamp 2018-07-11 07:51:29.946637) v4 > -5161> 2018-07-11 07:51:29.600961 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.6 10.8.128.115:0/13323 250 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:29.946637) v4 ==== 2004+0+0 (418840195 0 0) 0x55d9869bfc00 con 0x55d986d7e000 > -5160> 2018-07-11 07:51:29.600971 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:29.946637) v4 -- 0x55d986d97000 con 0 > -5159> 2018-07-11 07:51:29.601010 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.115:0/13323 conn(0x55d986d24800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 250 0x55d982d30600 osd_ping(ping e501 stamp 2018-07-11 07:51:29.946637) v4 > -5158> 2018-07-11 07:51:29.601031 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.6 10.8.128.115:0/13323 250 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:29.946637) v4 ==== 2004+0+0 (418840195 0 0) 0x55d982d30600 con 0x55d986d24800 > -5157> 2018-07-11 07:51:29.601042 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:29.946637) v4 -- 0x55d9b60df000 con 0 > -5156> 2018-07-11 07:51:29.647842 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15263 conn(0x55d985d45800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.5 seq 234 0x55d9829e5600 osd_ping(ping e501 stamp 2018-07-11 07:51:30.878751) v4 > -5155> 2018-07-11 07:51:29.647863 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.5 10.8.128.113:0/15263 234 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:30.878751) v4 ==== 2004+0+0 (3174670659 0 0) 0x55d9829e5600 con 0x55d985d45800 > -5154> 2018-07-11 07:51:29.647873 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:30.878751) v4 -- 0x55d986d4e400 con 0 > -5153> 2018-07-11 07:51:29.647872 7ffacbbd1700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15263 conn(0x55d985d44000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.5 seq 234 0x55d986d33e00 osd_ping(ping e501 stamp 2018-07-11 07:51:30.878751) v4 > -5152> 2018-07-11 07:51:29.647894 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 <== osd.5 10.8.128.113:0/15263 234 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:30.878751) v4 ==== 2004+0+0 (3174670659 0 0) 0x55d986d33e00 con 0x55d985d44000 > -5151> 2018-07-11 07:51:29.647905 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:30.878751) v4 -- 0x55d986d97000 con 0 > -5150> 2018-07-11 07:51:29.685644 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -5149> 2018-07-11 07:51:29.685670 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -5148> 2018-07-11 07:51:29.685716 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -5147> 2018-07-11 07:51:29.686214 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -5146> 2018-07-11 07:51:29.687975 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:754d2fd1:::benchmark_data_magna115_10867_object380695:0,end:5:754daf82:::benchmark_data_magna115_10867_object140329:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455400 con 0 > -5145> 2018-07-11 07:51:29.688021 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:754d2fd1:::benchmark_data_magna115_10867_object380695:0,end:5:754daf82:::benchmark_data_magna115_10867_object140329:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498280 con 0 > -5144> 2018-07-11 07:51:29.730412 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 31] Generated table #36803: 871498 keys, 14126086 bytes > -5143> 2018-07-11 07:51:29.730453 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295489730438, "cf_name": "default", "job": 31, "event": "table_file_creation", "file_number": 36803, "file_size": 14126086, "table_properties": {"data_size": 11798209, "index_size": 120752, "filter_size": 2206138, "raw_key_size": 25176603, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 2896, "num_entries": 871498, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "871498", "kMergeOperands": "0"}} > -5142> 2018-07-11 07:51:29.730485 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 31] Compacted 1@2 + 10@3 files to L3 => 599518257 bytes > -5141> 2018-07-11 07:51:29.744224 7ffabc3f1700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 51734252169 > -5140> 2018-07-11 07:51:29.744498 7ffabc3f1700 4 rocksdb: (Original Log Time 2018/07/11-07:51:29.744190) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 11 69 469 3641 796 0] max score 2.21, MB/sec: 13.4 rd, 12.4 wr, level 3, files in(1, 10) out(12) MB in(76.6, 537.5) out(571.7), read-write-amplify(15.5) write-amplify(7.5) OK, records in: 24904415, records dropped: 2000000 > > -5139> 2018-07-11 07:51:29.744506 7ffabc3f1700 4 rocksdb: (Original Log Time 2018/07/11-07:51:29.744216) EVENT_LOG_v1 {"time_micros": 1531295489744205, "job": 31, "event": "compaction_finished", "compaction_time_micros": 48176202, "output_level": 3, "num_output_files": 12, "total_output_size": 599518257, "num_input_records": 24904415, "num_output_records": 22904415, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 11, 69, 469, 3641, 796, 0]} > -5138> 2018-07-11 07:51:29.745101 7ffabc3f1700 5 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:307] [JOB 31] Delete db.slow/036613.sst type=2 #36613 -- OK > > -5137> 2018-07-11 07:51:29.745127 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295489745110, "job": 31, "event": "table_file_deletion", "file_number": 36613} > -5136> 2018-07-11 07:51:29.745139 7ffabc3f1700 5 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:307] [JOB 31] Delete db.slow/016613.sst type=2 #16613 -- OK > > -5135> 2018-07-11 07:51:29.745145 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295489745142, "job": 31, "event": "table_file_deletion", "file_number": 16613} > -5134> 2018-07-11 07:51:29.745151 7ffabc3f1700 5 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:307] [JOB 31] Delete db.slow/016609.sst type=2 #16609 -- OK > > -5133> 2018-07-11 07:51:29.745156 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295489745154, "job": 31, "event": "table_file_deletion", "file_number": 16609} > -5132> 2018-07-11 07:51:29.745162 7ffabc3f1700 5 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:307] [JOB 31] Delete db.slow/016607.sst type=2 #16607 -- OK > > -5131> 2018-07-11 07:51:29.745167 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295489745165, "job": 31, "event": "table_file_deletion", "file_number": 16607} > -5130> 2018-07-11 07:51:29.745173 7ffabc3f1700 5 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:307] [JOB 31] Delete db.slow/016606.sst type=2 #16606 -- OK > > -5129> 2018-07-11 07:51:29.745178 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295489745176, "job": 31, "event": "table_file_deletion", "file_number": 16606} > -5128> 2018-07-11 07:51:29.745185 7ffabc3f1700 5 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:307] [JOB 31] Delete db.slow/016605.sst type=2 #16605 -- OK > > -5127> 2018-07-11 07:51:29.745190 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295489745188, "job": 31, "event": "table_file_deletion", "file_number": 16605} > -5126> 2018-07-11 07:51:29.745198 7ffabc3f1700 5 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:307] [JOB 31] Delete db.slow/016604.sst type=2 #16604 -- OK > > -5125> 2018-07-11 07:51:29.745203 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295489745201, "job": 31, "event": "table_file_deletion", "file_number": 16604} > -5124> 2018-07-11 07:51:29.745212 7ffabc3f1700 5 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:307] [JOB 31] Delete db.slow/016599.sst type=2 #16599 -- OK > > -5123> 2018-07-11 07:51:29.745217 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295489745215, "job": 31, "event": "table_file_deletion", "file_number": 16599} > -5122> 2018-07-11 07:51:29.745225 7ffabc3f1700 5 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:307] [JOB 31] Delete db.slow/016598.sst type=2 #16598 -- OK > > -5121> 2018-07-11 07:51:29.745230 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295489745228, "job": 31, "event": "table_file_deletion", "file_number": 16598} > -5120> 2018-07-11 07:51:29.745235 7ffabc3f1700 5 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:307] [JOB 31] Delete db.slow/016595.sst type=2 #16595 -- OK > > -5119> 2018-07-11 07:51:29.745241 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295489745239, "job": 31, "event": "table_file_deletion", "file_number": 16595} > -5118> 2018-07-11 07:51:29.745257 7ffabc3f1700 5 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:307] [JOB 31] Delete db.slow/015298.sst type=2 #15298 -- OK > > -5117> 2018-07-11 07:51:29.745262 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295489745260, "job": 31, "event": "table_file_deletion", "file_number": 15298} > -5116> 2018-07-11 07:51:29.745441 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 32] Compacting 1@2 + 2@3 files to L3, score 2.21 > -5115> 2018-07-11 07:51:29.745453 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 31 Base level 2, inputs: [36582(76MB)], [26977(71MB) 36708(71MB)] > > -5114> 2018-07-11 07:51:29.745466 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295489745456, "job": 32, "event": "compaction_started", "files_L2": [36582], "files_L3": [26977, 36708], "score": 2.20867, "input_data_size": 229978276} > -5113> 2018-07-11 07:51:29.746384 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.107:0/12617 conn(0x55d98123c000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 237 0x55d982d30600 osd_ping(ping e501 stamp 2018-07-11 07:51:29.745877) v4 > -5112> 2018-07-11 07:51:29.746404 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.9 10.8.128.107:0/12617 237 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:29.745877) v4 ==== 2004+0+0 (2682173727 0 0) 0x55d982d30600 con 0x55d98123c000 > -5111> 2018-07-11 07:51:29.746415 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.107:0/12617 conn(0x55d98123d800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 237 0x55d9869bfc00 osd_ping(ping e501 stamp 2018-07-11 07:51:29.745877) v4 > -5110> 2018-07-11 07:51:29.746451 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.9 10.8.128.107:0/12617 237 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:29.745877) v4 ==== 2004+0+0 (2682173727 0 0) 0x55d9869bfc00 con 0x55d98123d800 > -5109> 2018-07-11 07:51:29.746445 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:29.745877) v4 -- 0x55d9b60df000 con 0 > -5108> 2018-07-11 07:51:29.746496 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:29.745877) v4 -- 0x55d986d97000 con 0 > -5107> 2018-07-11 07:51:29.907393 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 932 0x55d9b5c5ad80 rep_scrubmap(5.2e e501 from shard 2) v1 > -5106> 2018-07-11 07:51:29.907412 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 932 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12466 (1269249638 0 1432227524) 0x55d9b5c5ad80 con 0x55d986bf0800 > -5105> 2018-07-11 07:51:29.938944 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 926 0x55d9b5c5afc0 rep_scrubmap(5.2e e501 from shard 4) v1 > -5104> 2018-07-11 07:51:29.938963 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 926 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12466 (3686815630 0 1432227524) 0x55d9b5c5afc0 con 0x55d986bec000 > -5103> 2018-07-11 07:51:29.961896 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6801/15919 conn(0x55d986c3f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=29 cs=1 l=0). rx osd.11 seq 234 0x55d986d33e00 MOSDScrubReserve(10.f REQUEST e501) v1 > -5102> 2018-07-11 07:51:29.961915 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.11 10.8.128.114:6801/15919 234 ==== MOSDScrubReserve(10.f REQUEST e501) v1 ==== 43+0+0 (2892966580 0 0) 0x55d986d33e00 con 0x55d986c3f800 > -5101> 2018-07-11 07:51:29.961948 7ffab53e3700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.114:6801/15919 -- MOSDScrubReserve(10.f REJECT e501) v1 -- 0x55d9b60de600 con 0 > -5100> 2018-07-11 07:51:29.962284 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6801/15919 conn(0x55d986c3f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=29 cs=1 l=0). rx osd.11 seq 235 0x55d9b60de600 MOSDScrubReserve(10.f RELEASE e501) v1 > -5099> 2018-07-11 07:51:29.962298 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.11 10.8.128.114:6801/15919 235 ==== MOSDScrubReserve(10.f RELEASE e501) v1 ==== 43+0+0 (1668468918 0 0) 0x55d9b60de600 con 0x55d986c3f800 > -5098> 2018-07-11 07:51:30.118116 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.114:0/15885 conn(0x55d986d7f800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.7 seq 256 0x55d982cdf800 osd_ping(ping e501 stamp 2018-07-11 07:51:30.748309) v4 > -5097> 2018-07-11 07:51:30.118151 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.7 10.8.128.114:0/15885 256 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:30.748309) v4 ==== 2004+0+0 (674966304 0 0) 0x55d982cdf800 con 0x55d986d7f800 > -5096> 2018-07-11 07:51:30.118148 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.114:0/15885 conn(0x55d986d81000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.7 seq 256 0x55d9bdba4800 osd_ping(ping e501 stamp 2018-07-11 07:51:30.748309) v4 > -5095> 2018-07-11 07:51:30.118166 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.7 10.8.128.114:0/15885 256 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:30.748309) v4 ==== 2004+0+0 (674966304 0 0) 0x55d9bdba4800 con 0x55d986d81000 > -5094> 2018-07-11 07:51:30.118168 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:30.748309) v4 -- 0x55d986d4e400 con 0 > -5093> 2018-07-11 07:51:30.118200 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:30.748309) v4 -- 0x55d9b60df000 con 0 > -5092> 2018-07-11 07:51:30.122996 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12651 conn(0x55d986db3000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.2 seq 250 0x55d9869bfc00 osd_ping(ping e501 stamp 2018-07-11 07:51:30.170341) v4 > -5091> 2018-07-11 07:51:30.123017 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.2 10.8.128.106:0/12651 250 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:30.170341) v4 ==== 2004+0+0 (1998875153 0 0) 0x55d9869bfc00 con 0x55d986db3000 > -5090> 2018-07-11 07:51:30.123027 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:30.170341) v4 -- 0x55d9bdba5600 con 0 > -5089> 2018-07-11 07:51:30.123100 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12651 conn(0x55d986db1800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.2 seq 250 0x55d982d30600 osd_ping(ping e501 stamp 2018-07-11 07:51:30.170341) v4 > -5088> 2018-07-11 07:51:30.123119 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.2 10.8.128.106:0/12651 250 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:30.170341) v4 ==== 2004+0+0 (1998875153 0 0) 0x55d982d30600 con 0x55d986db1800 > -5087> 2018-07-11 07:51:30.123129 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:30.170341) v4 -- 0x55d9b60df000 con 0 > -5086> 2018-07-11 07:51:30.314179 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -5085> 2018-07-11 07:51:30.314211 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -5084> 2018-07-11 07:51:30.314278 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -5083> 2018-07-11 07:51:30.314745 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -5082> 2018-07-11 07:51:30.334160 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'68025,epoch:501/498,start:5:754daf82:::benchmark_data_magna115_10867_object140329:0,end:5:754e0e47:::benchmark_data_magna115_11707_object409566:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454f00 con 0 > -5081> 2018-07-11 07:51:30.334213 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'68025,epoch:501/498,start:5:754daf82:::benchmark_data_magna115_10867_object140329:0,end:5:754e0e47:::benchmark_data_magna115_11707_object409566:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454780 con 0 > -5080> 2018-07-11 07:51:30.544639 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 933 0x55d9b5c5b200 rep_scrubmap(5.2e e501 from shard 2) v1 > -5079> 2018-07-11 07:51:30.544661 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 933 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12474 (1269249638 0 3498272134) 0x55d9b5c5b200 con 0x55d986bf0800 > -5078> 2018-07-11 07:51:30.568983 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 927 0x55d9b5c5b440 rep_scrubmap(5.2e e501 from shard 4) v1 > -5077> 2018-07-11 07:51:30.569004 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 927 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12474 (3686815630 0 3498272134) 0x55d9b5c5b440 con 0x55d986bec000 > -5076> 2018-07-11 07:51:30.569145 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -5075> 2018-07-11 07:51:30.569171 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -5074> 2018-07-11 07:51:30.569218 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -5073> 2018-07-11 07:51:30.569683 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -5072> 2018-07-11 07:51:30.617919 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'66624,epoch:501/498,start:5:754e0e47:::benchmark_data_magna115_11707_object409566:0,end:5:754e4f6a:::benchmark_data_magna115_10867_object1189186:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455180 con 0 > -5071> 2018-07-11 07:51:30.617974 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'66624,epoch:501/498,start:5:754e0e47:::benchmark_data_magna115_11707_object409566:0,end:5:754e4f6a:::benchmark_data_magna115_10867_object1189186:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98944b680 con 0 > -5070> 2018-07-11 07:51:30.701435 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.115:0/13323 conn(0x55d986d24800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 251 0x55d9bdba4800 osd_ping(ping e501 stamp 2018-07-11 07:51:31.047082) v4 > -5069> 2018-07-11 07:51:30.701463 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.6 10.8.128.115:0/13323 251 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:31.047082) v4 ==== 2004+0+0 (1126687286 0 0) 0x55d9bdba4800 con 0x55d986d24800 > -5068> 2018-07-11 07:51:30.701455 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.115:0/13323 conn(0x55d986d7e000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 251 0x55d986d97000 osd_ping(ping e501 stamp 2018-07-11 07:51:31.047082) v4 > -5067> 2018-07-11 07:51:30.701474 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:31.047082) v4 -- 0x55d9b60df000 con 0 > -5066> 2018-07-11 07:51:30.701474 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.6 10.8.128.115:0/13323 251 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:31.047082) v4 ==== 2004+0+0 (1126687286 0 0) 0x55d986d97000 con 0x55d986d7e000 > -5065> 2018-07-11 07:51:30.701503 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:31.047082) v4 -- 0x55d9bdba5600 con 0 > -5064> 2018-07-11 07:51:30.850164 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 934 0x55d9b5c5b680 rep_scrubmap(5.2e e501 from shard 2) v1 > -5063> 2018-07-11 07:51:30.850184 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 934 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12482 (1269249638 0 3813264274) 0x55d9b5c5b680 con 0x55d986bf0800 > -5062> 2018-07-11 07:51:30.859264 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 928 0x55d9b5c5b8c0 rep_scrubmap(5.2e e501 from shard 4) v1 > -5061> 2018-07-11 07:51:30.859285 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 928 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12482 (3686815630 0 3813264274) 0x55d9b5c5b8c0 con 0x55d986bec000 > -5060> 2018-07-11 07:51:30.860415 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -5059> 2018-07-11 07:51:30.860440 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -5058> 2018-07-11 07:51:30.860485 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -5057> 2018-07-11 07:51:30.860945 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -5056> 2018-07-11 07:51:30.897160 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:754e4f6a:::benchmark_data_magna115_10867_object1189186:0,end:5:754ec160:::benchmark_data_magna115_10867_object1249910:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454500 con 0 > -5055> 2018-07-11 07:51:30.897212 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:754e4f6a:::benchmark_data_magna115_10867_object1189186:0,end:5:754ec160:::benchmark_data_magna115_10867_object1249910:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb773b80 con 0 > -5054> 2018-07-11 07:51:31.093837 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 935 0x55d9b5c5bb00 rep_scrubmap(5.2e e501 from shard 2) v1 > -5053> 2018-07-11 07:51:31.093857 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 935 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12478 (1269249638 0 3243673949) 0x55d9b5c5bb00 con 0x55d986bf0800 > -5052> 2018-07-11 07:51:31.115339 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 929 0x55d9b5c5bd40 rep_scrubmap(5.2e e501 from shard 4) v1 > -5051> 2018-07-11 07:51:31.115360 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 929 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12478 (3686815630 0 3243673949) 0x55d9b5c5bd40 con 0x55d986bec000 > -5050> 2018-07-11 07:51:31.115525 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12650 conn(0x55d986d23000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 243 0x55d9829e5600 osd_ping(ping e501 stamp 2018-07-11 07:51:31.163177) v4 > -5049> 2018-07-11 07:51:31.115541 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12650 conn(0x55d986d21800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 243 0x55d9869bfc00 osd_ping(ping e501 stamp 2018-07-11 07:51:31.163177) v4 > -5048> 2018-07-11 07:51:31.115548 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.3 10.8.128.106:0/12650 243 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:31.163177) v4 ==== 2004+0+0 (1163136169 0 0) 0x55d9829e5600 con 0x55d986d23000 > -5047> 2018-07-11 07:51:31.115566 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.3 10.8.128.106:0/12650 243 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:31.163177) v4 ==== 2004+0+0 (1163136169 0 0) 0x55d9869bfc00 con 0x55d986d21800 > -5046> 2018-07-11 07:51:31.115576 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:31.163177) v4 -- 0x55d986d4e400 con 0 > -5045> 2018-07-11 07:51:31.115611 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:31.163177) v4 -- 0x55d9bdba5600 con 0 > -5044> 2018-07-11 07:51:31.193102 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -5043> 2018-07-11 07:51:31.193129 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -5042> 2018-07-11 07:51:31.193187 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -5041> 2018-07-11 07:51:31.193665 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -5040> 2018-07-11 07:51:31.209703 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:754ec160:::benchmark_data_magna115_10867_object1249910:0,end:5:754ef64c:::benchmark_data_magna115_10867_object2045215:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454c80 con 0 > -5039> 2018-07-11 07:51:31.209753 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:754ec160:::benchmark_data_magna115_10867_object1249910:0,end:5:754ef64c:::benchmark_data_magna115_10867_object2045215:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454280 con 0 > -5038> 2018-07-11 07:51:31.434044 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 936 0x55d9c841a000 rep_scrubmap(5.2e e501 from shard 2) v1 > -5037> 2018-07-11 07:51:31.434067 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 936 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12474 (1269249638 0 1185524465) 0x55d9c841a000 con 0x55d986bf0800 > -5036> 2018-07-11 07:51:31.445139 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 930 0x55d9c841a240 rep_scrubmap(5.2e e501 from shard 4) v1 > -5035> 2018-07-11 07:51:31.445160 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 930 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12474 (3686815630 0 1185524465) 0x55d9c841a240 con 0x55d986bec000 > -5034> 2018-07-11 07:51:31.488358 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.108:0/11967 conn(0x55d986db0000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.8 seq 232 0x55d982cdf800 osd_ping(ping e501 stamp 2018-07-11 07:51:31.243967) v4 > -5033> 2018-07-11 07:51:31.488363 7ffacbbd1700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.108:0/11967 conn(0x55d986d82800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.8 seq 232 0x55d986d97000 osd_ping(ping e501 stamp 2018-07-11 07:51:31.243967) v4 > -5032> 2018-07-11 07:51:31.488384 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 <== osd.8 10.8.128.108:0/11967 232 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:31.243967) v4 ==== 2004+0+0 (4104079820 0 0) 0x55d986d97000 con 0x55d986d82800 > -5031> 2018-07-11 07:51:31.488383 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.8 10.8.128.108:0/11967 232 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:31.243967) v4 ==== 2004+0+0 (4104079820 0 0) 0x55d982cdf800 con 0x55d986db0000 > -5030> 2018-07-11 07:51:31.488396 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:31.243967) v4 -- 0x55d9bdba5600 con 0 > -5029> 2018-07-11 07:51:31.488422 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:31.243967) v4 -- 0x55d986d4e400 con 0 > -5028> 2018-07-11 07:51:31.537077 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -5027> 2018-07-11 07:51:31.537103 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -5026> 2018-07-11 07:51:31.537160 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -5025> 2018-07-11 07:51:31.537646 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -5024> 2018-07-11 07:51:31.538719 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:754ef64c:::benchmark_data_magna115_10867_object2045215:0,end:5:754f6e21:::benchmark_data_magna115_10867_object1965717:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454000 con 0 > -5023> 2018-07-11 07:51:31.538767 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:754ef64c:::benchmark_data_magna115_10867_object2045215:0,end:5:754f6e21:::benchmark_data_magna115_10867_object1965717:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb773680 con 0 > -5022> 2018-07-11 07:51:31.566296 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15255 conn(0x55d986c82800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.0 seq 250 0x55d9829e5600 osd_ping(ping e501 stamp 2018-07-11 07:51:32.797103) v4 > -5021> 2018-07-11 07:51:31.566292 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15255 conn(0x55d986d20000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.0 seq 250 0x55d982d30600 osd_ping(ping e501 stamp 2018-07-11 07:51:32.797103) v4 > -5020> 2018-07-11 07:51:31.566321 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.0 10.8.128.113:0/15255 250 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:32.797103) v4 ==== 2004+0+0 (4156759268 0 0) 0x55d9829e5600 con 0x55d986c82800 > -5019> 2018-07-11 07:51:31.566324 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.0 10.8.128.113:0/15255 250 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:32.797103) v4 ==== 2004+0+0 (4156759268 0 0) 0x55d982d30600 con 0x55d986d20000 > -5018> 2018-07-11 07:51:31.566330 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:32.797103) v4 -- 0x55d986d4e400 con 0 > -5017> 2018-07-11 07:51:31.566352 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:32.797103) v4 -- 0x55d9b60df000 con 0 > -5016> 2018-07-11 07:51:31.744706 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 937 0x55d9c841a480 rep_scrubmap(5.2e e501 from shard 2) v1 > -5015> 2018-07-11 07:51:31.744726 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 937 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12468 (1269249638 0 3014936740) 0x55d9c841a480 con 0x55d986bf0800 > -5014> 2018-07-11 07:51:31.754201 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 931 0x55d9c841a6c0 rep_scrubmap(5.2e e501 from shard 4) v1 > -5013> 2018-07-11 07:51:31.754221 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 931 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12468 (3686815630 0 3014936740) 0x55d9c841a6c0 con 0x55d986bec000 > -5012> 2018-07-11 07:51:31.792269 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -5011> 2018-07-11 07:51:31.792295 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -5010> 2018-07-11 07:51:31.792340 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -5009> 2018-07-11 07:51:31.792801 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -5008> 2018-07-11 07:51:31.809870 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67313,epoch:501/498,start:5:754f6e21:::benchmark_data_magna115_10867_object1965717:0,end:5:754fe8da:::benchmark_data_magna115_10867_object334310:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb773180 con 0 > -5007> 2018-07-11 07:51:31.809904 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67313,epoch:501/498,start:5:754f6e21:::benchmark_data_magna115_10867_object1965717:0,end:5:754fe8da:::benchmark_data_magna115_10867_object334310:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb773400 con 0 > -5006> 2018-07-11 07:51:31.823465 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12651 conn(0x55d986db3000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.2 seq 251 0x55d9869bfc00 osd_ping(ping e501 stamp 2018-07-11 07:51:31.870985) v4 > -5005> 2018-07-11 07:51:31.823486 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.2 10.8.128.106:0/12651 251 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:31.870985) v4 ==== 2004+0+0 (605670940 0 0) 0x55d9869bfc00 con 0x55d986db3000 > -5004> 2018-07-11 07:51:31.823498 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:31.870985) v4 -- 0x55d9bdba5600 con 0 > -5003> 2018-07-11 07:51:31.823502 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12651 conn(0x55d986db1800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.2 seq 251 0x55d982d30600 osd_ping(ping e501 stamp 2018-07-11 07:51:31.870985) v4 > -5002> 2018-07-11 07:51:31.823521 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.2 10.8.128.106:0/12651 251 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:31.870985) v4 ==== 2004+0+0 (605670940 0 0) 0x55d982d30600 con 0x55d986db1800 > -5001> 2018-07-11 07:51:31.823534 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:31.870985) v4 -- 0x55d9b60df000 con 0 > -5000> 2018-07-11 07:51:31.960841 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6805/15263 conn(0x55d986c41000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=31 cs=1 l=0). rx osd.5 seq 2820 0x55d9bdba4800 MOSDScrubReserve(6.15 REQUEST e501) v1 > -4999> 2018-07-11 07:51:31.960859 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.5 10.8.128.113:6805/15263 2820 ==== MOSDScrubReserve(6.15 REQUEST e501) v1 ==== 43+0+0 (3942742026 0 0) 0x55d9bdba4800 con 0x55d986c41000 > -4998> 2018-07-11 07:51:31.991965 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.114:0/15919 conn(0x55d986df3800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.11 seq 251 0x55d986d97000 osd_ping(ping e501 stamp 2018-07-11 07:51:32.622176) v4 > -4997> 2018-07-11 07:51:31.991986 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.11 10.8.128.114:0/15919 251 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:32.622176) v4 ==== 2004+0+0 (946688496 0 0) 0x55d986d97000 con 0x55d986df3800 > -4996> 2018-07-11 07:51:31.991995 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:32.622176) v4 -- 0x55d9bdba5600 con 0 > -4995> 2018-07-11 07:51:31.992060 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.114:0/15919 conn(0x55d986df5000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.11 seq 251 0x55d982cdf800 osd_ping(ping e501 stamp 2018-07-11 07:51:32.622176) v4 > -4994> 2018-07-11 07:51:31.992083 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.11 10.8.128.114:0/15919 251 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:32.622176) v4 ==== 2004+0+0 (946688496 0 0) 0x55d982cdf800 con 0x55d986df5000 > -4993> 2018-07-11 07:51:31.992094 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:32.622176) v4 -- 0x55d986d4e400 con 0 > -4992> 2018-07-11 07:51:32.049493 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 932 0x55d9c841a900 rep_scrubmap(5.2e e501 from shard 4) v1 > -4991> 2018-07-11 07:51:32.049515 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 932 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12470 (3686815630 0 294765598) 0x55d9c841a900 con 0x55d986bec000 > -4990> 2018-07-11 07:51:32.062857 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 938 0x55d9c841ab40 rep_scrubmap(5.2e e501 from shard 2) v1 > -4989> 2018-07-11 07:51:32.062877 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 938 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12470 (1269249638 0 294765598) 0x55d9c841ab40 con 0x55d986bf0800 > -4988> 2018-07-11 07:51:32.084054 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15287 conn(0x55d981240800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 263 0x55d982d30600 osd_ping(ping e501 stamp 2018-07-11 07:51:33.315022) v4 > -4987> 2018-07-11 07:51:32.084054 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15287 conn(0x55d98123f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 263 0x55d9829e5600 osd_ping(ping e501 stamp 2018-07-11 07:51:33.315022) v4 > -4986> 2018-07-11 07:51:32.084079 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.4 10.8.128.113:0/15287 263 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:33.315022) v4 ==== 2004+0+0 (966355908 0 0) 0x55d982d30600 con 0x55d981240800 > -4985> 2018-07-11 07:51:32.084089 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:33.315022) v4 -- 0x55d9b60df400 con 0 > -4984> 2018-07-11 07:51:32.084088 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.4 10.8.128.113:0/15287 263 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:33.315022) v4 ==== 2004+0+0 (966355908 0 0) 0x55d9829e5600 con 0x55d98123f000 > -4983> 2018-07-11 07:51:32.084114 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:33.315022) v4 -- 0x55d986d4e400 con 0 > -4982> 2018-07-11 07:51:32.109585 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6805/15263 -- MOSDScrubReserve(6.15 REJECT e501) v1 -- 0x55d9829e4200 con 0 > -4981> 2018-07-11 07:51:32.109794 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -4980> 2018-07-11 07:51:32.109816 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -4979> 2018-07-11 07:51:32.109862 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -4978> 2018-07-11 07:51:32.110066 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6805/15263 conn(0x55d986c41000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=31 cs=1 l=0). rx osd.5 seq 2821 0x55d9829e4200 MOSDScrubReserve(6.15 RELEASE e501) v1 > -4977> 2018-07-11 07:51:32.110086 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.5 10.8.128.113:6805/15263 2821 ==== MOSDScrubReserve(6.15 RELEASE e501) v1 ==== 43+0+0 (605848072 0 0) 0x55d9829e4200 con 0x55d986c41000 > -4976> 2018-07-11 07:51:32.110326 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -4975> 2018-07-11 07:51:32.151800 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:754fe8da:::benchmark_data_magna115_10867_object334310:0,end:5:7550437c:::benchmark_data_magna115_10867_object2422972:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb773900 con 0 > -4974> 2018-07-11 07:51:32.151852 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:754fe8da:::benchmark_data_magna115_10867_object334310:0,end:5:7550437c:::benchmark_data_magna115_10867_object2422972:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb772f00 con 0 > -4973> 2018-07-11 07:51:32.301865 7ffab7be8700 10 monclient: tick > -4972> 2018-07-11 07:51:32.301883 7ffab7be8700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2018-07-11 07:51:02.301878) > -4971> 2018-07-11 07:51:32.324040 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12651 conn(0x55d986db3000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.2 seq 252 0x55d9869bfc00 osd_ping(ping e501 stamp 2018-07-11 07:51:32.371597) v4 > -4970> 2018-07-11 07:51:32.324061 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.2 10.8.128.106:0/12651 252 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:32.371597) v4 ==== 2004+0+0 (1315358384 0 0) 0x55d9869bfc00 con 0x55d986db3000 > -4969> 2018-07-11 07:51:32.324071 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:32.371597) v4 -- 0x55d9bdba5600 con 0 > -4968> 2018-07-11 07:51:32.324081 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12651 conn(0x55d986db1800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.2 seq 252 0x55d9b60df000 osd_ping(ping e501 stamp 2018-07-11 07:51:32.371597) v4 > -4967> 2018-07-11 07:51:32.324102 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.2 10.8.128.106:0/12651 252 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:32.371597) v4 ==== 2004+0+0 (1315358384 0 0) 0x55d9b60df000 con 0x55d986db1800 > -4966> 2018-07-11 07:51:32.324118 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:32.371597) v4 -- 0x55d9b60df400 con 0 > -4965> 2018-07-11 07:51:32.348059 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 939 0x55d9c841ad80 rep_scrubmap(5.2e e501 from shard 2) v1 > -4964> 2018-07-11 07:51:32.348078 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 939 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12466 (1269249638 0 129653446) 0x55d9c841ad80 con 0x55d986bf0800 > -4963> 2018-07-11 07:51:32.350360 7ffab6be6700 1 -- 10.8.128.107:6800/13927 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55d9869ca2c0 con 0 > -4962> 2018-07-11 07:51:32.350431 7ffab6be6700 1 -- 10.8.128.107:6800/13927 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55d986cfedc0 con 0 > -4961> 2018-07-11 07:51:32.384363 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 933 0x55d9c841afc0 rep_scrubmap(5.2e e501 from shard 4) v1 > -4960> 2018-07-11 07:51:32.384383 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 933 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12466 (3686815630 0 129653446) 0x55d9c841afc0 con 0x55d986bec000 > -4959> 2018-07-11 07:51:32.417807 7ffab1bdc700 5 osd.10 501 heartbeat: osd_stat(788 GB used, 142 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,1,1]) > -4958> 2018-07-11 07:51:32.417820 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6802/15255 -- osd_ping(ping e501 stamp 2018-07-11 07:51:32.417819) v4 -- 0x55d9c3383600 con 0 > -4957> 2018-07-11 07:51:32.417855 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6803/15255 -- osd_ping(ping e501 stamp 2018-07-11 07:51:32.417819) v4 -- 0x55d9bdba5e00 con 0 > -4956> 2018-07-11 07:51:32.417870 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6806/12677 -- osd_ping(ping e501 stamp 2018-07-11 07:51:32.417819) v4 -- 0x55d986d97200 con 0 > -4955> 2018-07-11 07:51:32.417884 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6807/12677 -- osd_ping(ping e501 stamp 2018-07-11 07:51:32.417819) v4 -- 0x55d9829e4200 con 0 > -4954> 2018-07-11 07:51:32.417907 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6808/12651 -- osd_ping(ping e501 stamp 2018-07-11 07:51:32.417819) v4 -- 0x55d9bdba4800 con 0 > -4953> 2018-07-11 07:51:32.417918 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6810/12651 -- osd_ping(ping e501 stamp 2018-07-11 07:51:32.417819) v4 -- 0x55d9c4c1be00 con 0 > -4952> 2018-07-11 07:51:32.417928 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6809/12650 -- osd_ping(ping e501 stamp 2018-07-11 07:51:32.417819) v4 -- 0x55d9b4388200 con 0 > -4951> 2018-07-11 07:51:32.417940 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6811/12650 -- osd_ping(ping e501 stamp 2018-07-11 07:51:32.417819) v4 -- 0x55d9829e4800 con 0 > -4950> 2018-07-11 07:51:32.417954 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6810/15287 -- osd_ping(ping e501 stamp 2018-07-11 07:51:32.417819) v4 -- 0x55d986c85000 con 0 > -4949> 2018-07-11 07:51:32.417970 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6811/15287 -- osd_ping(ping e501 stamp 2018-07-11 07:51:32.417819) v4 -- 0x55d986b6d400 con 0 > -4948> 2018-07-11 07:51:32.417985 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6806/15263 -- osd_ping(ping e501 stamp 2018-07-11 07:51:32.417819) v4 -- 0x55d986d33200 con 0 > -4947> 2018-07-11 07:51:32.417999 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6807/15263 -- osd_ping(ping e501 stamp 2018-07-11 07:51:32.417819) v4 -- 0x55d9b2528000 con 0 > -4946> 2018-07-11 07:51:32.418011 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.115:6802/13323 -- osd_ping(ping e501 stamp 2018-07-11 07:51:32.417819) v4 -- 0x55d9b437ac00 con 0 > -4945> 2018-07-11 07:51:32.418023 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.115:6803/13323 -- osd_ping(ping e501 stamp 2018-07-11 07:51:32.417819) v4 -- 0x55d986cd5200 con 0 > -4944> 2018-07-11 07:51:32.418036 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6806/15885 -- osd_ping(ping e501 stamp 2018-07-11 07:51:32.417819) v4 -- 0x55d9c34cc000 con 0 > -4943> 2018-07-11 07:51:32.418049 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6807/15885 -- osd_ping(ping e501 stamp 2018-07-11 07:51:32.417819) v4 -- 0x55d97f5adc00 con 0 > -4942> 2018-07-11 07:51:32.418061 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.108:6802/11967 -- osd_ping(ping e501 stamp 2018-07-11 07:51:32.417819) v4 -- 0x55d986d4e200 con 0 > -4941> 2018-07-11 07:51:32.418073 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.108:6803/11967 -- osd_ping(ping e501 stamp 2018-07-11 07:51:32.417819) v4 -- 0x55d9bf804400 con 0 > -4940> 2018-07-11 07:51:32.418085 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.107:6806/12617 -- osd_ping(ping e501 stamp 2018-07-11 07:51:32.417819) v4 -- 0x55d9ba819a00 con 0 > -4939> 2018-07-11 07:51:32.418098 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.107:6807/12617 -- osd_ping(ping e501 stamp 2018-07-11 07:51:32.417819) v4 -- 0x55d9b4388400 con 0 > -4938> 2018-07-11 07:51:32.418111 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6802/15919 -- osd_ping(ping e501 stamp 2018-07-11 07:51:32.417819) v4 -- 0x55d986cd6a00 con 0 > -4937> 2018-07-11 07:51:32.418124 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6803/15919 -- osd_ping(ping e501 stamp 2018-07-11 07:51:32.417819) v4 -- 0x55d986d33000 con 0 > -4936> 2018-07-11 07:51:32.418265 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.107:6806/12617 conn(0x55d986bd2000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.9 seq 242 0x55d982cdf800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:32.417819) v4 > -4935> 2018-07-11 07:51:32.418289 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.9 10.8.128.107:6806/12617 242 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:32.417819) v4 ==== 2004+0+0 (972020328 0 0) 0x55d982cdf800 con 0x55d986bd2000 > -4934> 2018-07-11 07:51:32.418287 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.107:6807/12617 conn(0x55d986bd3800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.9 seq 242 0x55d986d97000 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:32.417819) v4 > -4933> 2018-07-11 07:51:32.418327 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.9 10.8.128.107:6807/12617 242 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:32.417819) v4 ==== 2004+0+0 (972020328 0 0) 0x55d986d97000 con 0x55d986bd3800 > -4932> 2018-07-11 07:51:32.418436 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6810/12651 conn(0x55d986b9b800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.2 seq 242 0x55d982cdf800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:32.417819) v4 > -4931> 2018-07-11 07:51:32.418452 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.2 10.8.128.106:6810/12651 242 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:32.417819) v4 ==== 2004+0+0 (972020328 0 0) 0x55d982cdf800 con 0x55d986b9b800 > -4930> 2018-07-11 07:51:32.418449 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6809/12650 conn(0x55d986bcb000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 242 0x55d986d97000 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:32.417819) v4 > -4929> 2018-07-11 07:51:32.418473 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.3 10.8.128.106:6809/12650 242 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:32.417819) v4 ==== 2004+0+0 (972020328 0 0) 0x55d986d97000 con 0x55d986bcb000 > -4928> 2018-07-11 07:51:32.418485 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6811/15287 conn(0x55d986bb0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 242 0x55d982cdf800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:32.417819) v4 > -4927> 2018-07-11 07:51:32.418513 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.4 10.8.128.113:6811/15287 242 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:32.417819) v4 ==== 2004+0+0 (972020328 0 0) 0x55d982cdf800 con 0x55d986bb0800 > -4926> 2018-07-11 07:51:32.418498 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6802/15255 conn(0x55d986bc8000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.0 seq 242 0x55d986d40a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:32.417819) v4 > -4925> 2018-07-11 07:51:32.418515 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6806/15263 conn(0x55d986b9d000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.5 seq 242 0x55d986d97000 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:32.417819) v4 > -4924> 2018-07-11 07:51:32.418524 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.0 10.8.128.113:6802/15255 242 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:32.417819) v4 ==== 2004+0+0 (972020328 0 0) 0x55d986d40a00 con 0x55d986bc8000 > -4923> 2018-07-11 07:51:32.418537 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.5 10.8.128.113:6806/15263 242 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:32.417819) v4 ==== 2004+0+0 (972020328 0 0) 0x55d986d97000 con 0x55d986b9d000 > -4922> 2018-07-11 07:51:32.418545 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6803/15255 conn(0x55d986bc9800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.0 seq 242 0x55d982cdf800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:32.417819) v4 > -4921> 2018-07-11 07:51:32.418557 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.0 10.8.128.113:6803/15255 242 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:32.417819) v4 ==== 2004+0+0 (972020328 0 0) 0x55d982cdf800 con 0x55d986bc9800 > -4920> 2018-07-11 07:51:32.418572 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6808/12651 conn(0x55d986b9a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.2 seq 242 0x55d986d40a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:32.417819) v4 > -4919> 2018-07-11 07:51:32.418577 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.115:6803/13323 conn(0x55d986bc2800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 242 0x55d986d97000 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:32.417819) v4 > -4918> 2018-07-11 07:51:32.418587 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.6 10.8.128.115:6803/13323 242 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:32.417819) v4 ==== 2004+0+0 (972020328 0 0) 0x55d986d97000 con 0x55d986bc2800 > -4917> 2018-07-11 07:51:32.418587 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.2 10.8.128.106:6808/12651 242 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:32.417819) v4 ==== 2004+0+0 (972020328 0 0) 0x55d986d40a00 con 0x55d986b9a000 > -4916> 2018-07-11 07:51:32.418628 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6811/12650 conn(0x55d986bcc800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 242 0x55d986d40a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:32.417819) v4 > -4915> 2018-07-11 07:51:32.418627 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6806/12677 conn(0x55d986b58000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.1 seq 242 0x55d986d97000 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:32.417819) v4 > -4914> 2018-07-11 07:51:32.418641 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.3 10.8.128.106:6811/12650 242 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:32.417819) v4 ==== 2004+0+0 (972020328 0 0) 0x55d986d40a00 con 0x55d986bcc800 > -4913> 2018-07-11 07:51:32.418643 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.1 10.8.128.106:6806/12677 242 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:32.417819) v4 ==== 2004+0+0 (972020328 0 0) 0x55d986d97000 con 0x55d986b58000 > -4912> 2018-07-11 07:51:32.418669 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6807/15263 conn(0x55d986b9e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.5 seq 242 0x55d986d40a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:32.417819) v4 > -4911> 2018-07-11 07:51:32.418680 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.5 10.8.128.113:6807/15263 242 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:32.417819) v4 ==== 2004+0+0 (972020328 0 0) 0x55d986d40a00 con 0x55d986b9e800 > -4910> 2018-07-11 07:51:32.418677 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.115:6802/13323 conn(0x55d986bc1000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 242 0x55d982cdf800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:32.417819) v4 > -4909> 2018-07-11 07:51:32.418698 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.6 10.8.128.115:6802/13323 242 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:32.417819) v4 ==== 2004+0+0 (972020328 0 0) 0x55d982cdf800 con 0x55d986bc1000 > -4908> 2018-07-11 07:51:32.418708 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.108:6802/11967 conn(0x55d986bbe000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.8 seq 242 0x55d986d97000 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:32.417819) v4 > -4907> 2018-07-11 07:51:32.418713 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6807/12677 conn(0x55d986b59800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.1 seq 242 0x55d986d40a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:32.417819) v4 > -4906> 2018-07-11 07:51:32.418732 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.1 10.8.128.106:6807/12677 242 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:32.417819) v4 ==== 2004+0+0 (972020328 0 0) 0x55d986d40a00 con 0x55d986b59800 > -4905> 2018-07-11 07:51:32.418728 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.8 10.8.128.108:6802/11967 242 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:32.417819) v4 ==== 2004+0+0 (972020328 0 0) 0x55d986d97000 con 0x55d986bbe000 > -4904> 2018-07-11 07:51:32.418771 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.108:6803/11967 conn(0x55d986bbf800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.8 seq 242 0x55d986d40a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:32.417819) v4 > -4903> 2018-07-11 07:51:32.418764 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6806/15885 conn(0x55d986bac000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.7 seq 242 0x55d982cdf800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:32.417819) v4 > -4902> 2018-07-11 07:51:32.418784 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.8 10.8.128.108:6803/11967 242 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:32.417819) v4 ==== 2004+0+0 (972020328 0 0) 0x55d986d40a00 con 0x55d986bbf800 > -4901> 2018-07-11 07:51:32.418786 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.7 10.8.128.114:6806/15885 242 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:32.417819) v4 ==== 2004+0+0 (972020328 0 0) 0x55d982cdf800 con 0x55d986bac000 > -4900> 2018-07-11 07:51:32.418787 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6807/15885 conn(0x55d986bad800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.7 seq 242 0x55d986d97000 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:32.417819) v4 > -4899> 2018-07-11 07:51:32.418801 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.7 10.8.128.114:6807/15885 242 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:32.417819) v4 ==== 2004+0+0 (972020328 0 0) 0x55d986d97000 con 0x55d986bad800 > -4898> 2018-07-11 07:51:32.418821 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6810/15287 conn(0x55d986baf000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 242 0x55d986d40a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:32.417819) v4 > -4897> 2018-07-11 07:51:32.418833 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.4 10.8.128.113:6810/15287 242 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:32.417819) v4 ==== 2004+0+0 (972020328 0 0) 0x55d986d40a00 con 0x55d986baf000 > -4896> 2018-07-11 07:51:32.418828 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6802/15919 conn(0x55d986b5b000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.11 seq 242 0x55d982cdf800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:32.417819) v4 > -4895> 2018-07-11 07:51:32.418845 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.11 10.8.128.114:6802/15919 242 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:32.417819) v4 ==== 2004+0+0 (972020328 0 0) 0x55d982cdf800 con 0x55d986b5b000 > -4894> 2018-07-11 07:51:32.418844 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6803/15919 conn(0x55d986b5c800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.11 seq 242 0x55d986d97000 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:32.417819) v4 > -4893> 2018-07-11 07:51:32.418867 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.11 10.8.128.114:6803/15919 242 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:32.417819) v4 ==== 2004+0+0 (972020328 0 0) 0x55d986d97000 con 0x55d986b5c800 > -4892> 2018-07-11 07:51:32.467204 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -4891> 2018-07-11 07:51:32.467230 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -4890> 2018-07-11 07:51:32.467286 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -4889> 2018-07-11 07:51:32.467744 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -4888> 2018-07-11 07:51:32.503208 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67599,epoch:501/498,start:5:7550437c:::benchmark_data_magna115_10867_object2422972:0,end:5:7550b4df:::benchmark_data_magna115_10867_object3248924:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb772c80 con 0 > -4887> 2018-07-11 07:51:32.503242 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67599,epoch:501/498,start:5:7550437c:::benchmark_data_magna115_10867_object2422972:0,end:5:7550b4df:::benchmark_data_magna115_10867_object3248924:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb772780 con 0 > -4886> 2018-07-11 07:51:32.740799 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 940 0x55d9c841b200 rep_scrubmap(5.2e e501 from shard 2) v1 > -4885> 2018-07-11 07:51:32.740818 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 940 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12468 (1269249638 0 2081280855) 0x55d9c841b200 con 0x55d986bf0800 > -4884> 2018-07-11 07:51:32.742417 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 934 0x55d9c841b440 rep_scrubmap(5.2e e501 from shard 4) v1 > -4883> 2018-07-11 07:51:32.742436 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 934 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12468 (3686815630 0 2081280855) 0x55d9c841b440 con 0x55d986bec000 > -4882> 2018-07-11 07:51:32.742576 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -4881> 2018-07-11 07:51:32.742601 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -4880> 2018-07-11 07:51:32.742647 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -4879> 2018-07-11 07:51:32.743100 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -4878> 2018-07-11 07:51:32.768603 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67101,epoch:501/498,start:5:7550b4df:::benchmark_data_magna115_10867_object3248924:0,end:5:75514c15:::benchmark_data_magna115_10867_object98996:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb772500 con 0 > -4877> 2018-07-11 07:51:32.768646 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67101,epoch:501/498,start:5:7550b4df:::benchmark_data_magna115_10867_object3248924:0,end:5:75514c15:::benchmark_data_magna115_10867_object98996:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb772a00 con 0 > -4876> 2018-07-11 07:51:32.961069 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6805/15263 conn(0x55d986c41000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=31 cs=1 l=0). rx osd.5 seq 2822 0x55d9b60df000 MOSDScrubReserve(6.15 REQUEST e501) v1 > -4875> 2018-07-11 07:51:32.961089 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.5 10.8.128.113:6805/15263 2822 ==== MOSDScrubReserve(6.15 REQUEST e501) v1 ==== 43+0+0 (3942742026 0 0) 0x55d9b60df000 con 0x55d986c41000 > -4874> 2018-07-11 07:51:32.967149 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 941 0x55d9c841b680 rep_scrubmap(5.2e e501 from shard 2) v1 > -4873> 2018-07-11 07:51:32.967169 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 941 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12468 (1269249638 0 2461053880) 0x55d9c841b680 con 0x55d986bf0800 > -4872> 2018-07-11 07:51:32.974279 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 935 0x55d9c841b8c0 rep_scrubmap(5.2e e501 from shard 4) v1 > -4871> 2018-07-11 07:51:32.974299 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 935 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12468 (3686815630 0 2461053880) 0x55d9c841b8c0 con 0x55d986bec000 > -4870> 2018-07-11 07:51:33.018520 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.114:0/15885 conn(0x55d986d7f800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.7 seq 257 0x55d982cdf800 osd_ping(ping e501 stamp 2018-07-11 07:51:33.648729) v4 > -4869> 2018-07-11 07:51:33.018534 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.114:0/15885 conn(0x55d986d81000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.7 seq 257 0x55d986d40a00 osd_ping(ping e501 stamp 2018-07-11 07:51:33.648729) v4 > -4868> 2018-07-11 07:51:33.018546 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.7 10.8.128.114:0/15885 257 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:33.648729) v4 ==== 2004+0+0 (564043072 0 0) 0x55d982cdf800 con 0x55d986d7f800 > -4867> 2018-07-11 07:51:33.018552 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.7 10.8.128.114:0/15885 257 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:33.648729) v4 ==== 2004+0+0 (564043072 0 0) 0x55d986d40a00 con 0x55d986d81000 > -4866> 2018-07-11 07:51:33.018561 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:33.648729) v4 -- 0x55d986cd6a00 con 0 > -4865> 2018-07-11 07:51:33.018591 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:33.648729) v4 -- 0x55d986d33a00 con 0 > -4864> 2018-07-11 07:51:33.079410 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6805/15263 -- MOSDScrubReserve(6.15 REJECT e501) v1 -- 0x55d986c22e00 con 0 > -4863> 2018-07-11 07:51:33.079602 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -4862> 2018-07-11 07:51:33.079626 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -4861> 2018-07-11 07:51:33.079671 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -4860> 2018-07-11 07:51:33.079828 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6805/15263 conn(0x55d986c41000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=31 cs=1 l=0). rx osd.5 seq 2823 0x55d986c22e00 MOSDScrubReserve(6.15 RELEASE e501) v1 > -4859> 2018-07-11 07:51:33.079847 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.5 10.8.128.113:6805/15263 2823 ==== MOSDScrubReserve(6.15 RELEASE e501) v1 ==== 43+0+0 (605848072 0 0) 0x55d986c22e00 con 0x55d986c41000 > -4858> 2018-07-11 07:51:33.080127 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -4857> 2018-07-11 07:51:33.081025 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75514c15:::benchmark_data_magna115_10867_object98996:0,end:5:7551bb7f:::benchmark_data_magna115_10867_object672484:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb772000 con 0 > -4856> 2018-07-11 07:51:33.081069 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75514c15:::benchmark_data_magna115_10867_object98996:0,end:5:7551bb7f:::benchmark_data_magna115_10867_object672484:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98944b400 con 0 > -4855> 2018-07-11 07:51:33.092328 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.114:0/15919 conn(0x55d986df5000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.11 seq 252 0x55d9829e5600 osd_ping(ping e501 stamp 2018-07-11 07:51:33.722558) v4 > -4854> 2018-07-11 07:51:33.092329 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.114:0/15919 conn(0x55d986df3800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.11 seq 252 0x55d986d97000 osd_ping(ping e501 stamp 2018-07-11 07:51:33.722558) v4 > -4853> 2018-07-11 07:51:33.092354 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.11 10.8.128.114:0/15919 252 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:33.722558) v4 ==== 2004+0+0 (2996777693 0 0) 0x55d9829e5600 con 0x55d986df5000 > -4852> 2018-07-11 07:51:33.092363 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:33.722558) v4 -- 0x55d986cd6a00 con 0 > -4851> 2018-07-11 07:51:33.092361 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.11 10.8.128.114:0/15919 252 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:33.722558) v4 ==== 2004+0+0 (2996777693 0 0) 0x55d986d97000 con 0x55d986df3800 > -4850> 2018-07-11 07:51:33.092387 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:33.722558) v4 -- 0x55d986d33000 con 0 > -4849> 2018-07-11 07:51:33.285724 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 942 0x55d9c841bb00 rep_scrubmap(5.2e e501 from shard 2) v1 > -4848> 2018-07-11 07:51:33.285745 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 942 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12476 (1269249638 0 766723028) 0x55d9c841bb00 con 0x55d986bf0800 > -4847> 2018-07-11 07:51:33.330143 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 936 0x55d9c841bd40 rep_scrubmap(5.2e e501 from shard 4) v1 > -4846> 2018-07-11 07:51:33.330163 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 936 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12476 (3686815630 0 766723028) 0x55d9c841bd40 con 0x55d986bec000 > -4845> 2018-07-11 07:51:33.330314 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -4844> 2018-07-11 07:51:33.330340 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -4843> 2018-07-11 07:51:33.330386 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -4842> 2018-07-11 07:51:33.330850 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -4841> 2018-07-11 07:51:33.354268 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7551bb7f:::benchmark_data_magna115_10867_object672484:0,end:5:7552013f:::benchmark_data_magna115_10867_object3857507:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb772280 con 0 > -4840> 2018-07-11 07:51:33.354304 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7551bb7f:::benchmark_data_magna115_10867_object672484:0,end:5:7552013f:::benchmark_data_magna115_10867_object3857507:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98944bb80 con 0 > -4839> 2018-07-11 07:51:33.552918 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 943 0x55d9bf2c2000 rep_scrubmap(5.2e e501 from shard 2) v1 > -4838> 2018-07-11 07:51:33.552938 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 943 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12472 (1269249638 0 3159083856) 0x55d9bf2c2000 con 0x55d986bf0800 > -4837> 2018-07-11 07:51:33.561369 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 937 0x55d9bf2c2240 rep_scrubmap(5.2e e501 from shard 4) v1 > -4836> 2018-07-11 07:51:33.561387 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 937 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12472 (3686815630 0 3159083856) 0x55d9bf2c2240 con 0x55d986bec000 > -4835> 2018-07-11 07:51:33.564018 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6805/15885 conn(0x55d986c7f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.7 seq 2796 0x55d9c9123200 MOSDScrubReserve(5.4 REQUEST e501) v1 > -4834> 2018-07-11 07:51:33.564036 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.7 10.8.128.114:6805/15885 2796 ==== MOSDScrubReserve(5.4 REQUEST e501) v1 ==== 43+0+0 (1753221512 0 0) 0x55d9c9123200 con 0x55d986c7f800 > -4833> 2018-07-11 07:51:33.564158 7ffab33df700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.114:6805/15885 -- MOSDScrubReserve(5.4 REJECT e501) v1 -- 0x55d986d32200 con 0 > -4832> 2018-07-11 07:51:33.564451 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6805/15885 conn(0x55d986c7f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.7 seq 2797 0x55d986d32200 MOSDScrubReserve(5.4 RELEASE e501) v1 > -4831> 2018-07-11 07:51:33.564466 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.7 10.8.128.114:6805/15885 2797 ==== MOSDScrubReserve(5.4 RELEASE e501) v1 ==== 43+0+0 (2812146570 0 0) 0x55d986d32200 con 0x55d986c7f800 > -4830> 2018-07-11 07:51:33.592718 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.114:0/15919 conn(0x55d986df3800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.11 seq 253 0x55d9b2783a00 osd_ping(ping e501 stamp 2018-07-11 07:51:34.222936) v4 > -4829> 2018-07-11 07:51:33.592731 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.114:0/15919 conn(0x55d986df5000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.11 seq 253 0x55d982cdf800 osd_ping(ping e501 stamp 2018-07-11 07:51:34.222936) v4 > -4828> 2018-07-11 07:51:33.592752 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.11 10.8.128.114:0/15919 253 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:34.222936) v4 ==== 2004+0+0 (728127683 0 0) 0x55d982cdf800 con 0x55d986df5000 > -4827> 2018-07-11 07:51:33.592750 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.11 10.8.128.114:0/15919 253 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:34.222936) v4 ==== 2004+0+0 (728127683 0 0) 0x55d9b2783a00 con 0x55d986df3800 > -4826> 2018-07-11 07:51:33.592761 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:34.222936) v4 -- 0x55d986cd6a00 con 0 > -4825> 2018-07-11 07:51:33.592785 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:34.222936) v4 -- 0x55d986d33000 con 0 > -4824> 2018-07-11 07:51:33.644375 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -4823> 2018-07-11 07:51:33.644400 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -4822> 2018-07-11 07:51:33.644445 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -4821> 2018-07-11 07:51:33.644904 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -4820> 2018-07-11 07:51:33.646954 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7552013f:::benchmark_data_magna115_10867_object3857507:0,end:5:75526fa3:::benchmark_data_magna115_11707_object173741:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454a00 con 0 > -4819> 2018-07-11 07:51:33.647001 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7552013f:::benchmark_data_magna115_10867_object3857507:0,end:5:75526fa3:::benchmark_data_magna115_11707_object173741:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98944b900 con 0 > -4818> 2018-07-11 07:51:33.788817 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.108:0/11967 conn(0x55d986db0000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.8 seq 233 0x55d9829e5600 osd_ping(ping e501 stamp 2018-07-11 07:51:33.544442) v4 > -4817> 2018-07-11 07:51:33.788822 7ffacbbd1700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.108:0/11967 conn(0x55d986d82800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.8 seq 233 0x55d986d97000 osd_ping(ping e501 stamp 2018-07-11 07:51:33.544442) v4 > -4816> 2018-07-11 07:51:33.788840 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.8 10.8.128.108:0/11967 233 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:33.544442) v4 ==== 2004+0+0 (2416166882 0 0) 0x55d9829e5600 con 0x55d986db0000 > -4815> 2018-07-11 07:51:33.788850 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:33.544442) v4 -- 0x55d986cd6a00 con 0 > -4814> 2018-07-11 07:51:33.788845 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 <== osd.8 10.8.128.108:0/11967 233 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:33.544442) v4 ==== 2004+0+0 (2416166882 0 0) 0x55d986d97000 con 0x55d986d82800 > -4813> 2018-07-11 07:51:33.788873 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:33.544442) v4 -- 0x55d986d33000 con 0 > -4812> 2018-07-11 07:51:33.860875 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 944 0x55d9bf2c2480 rep_scrubmap(5.2e e501 from shard 2) v1 > -4811> 2018-07-11 07:51:33.860894 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 944 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12480 (1269249638 0 925503479) 0x55d9bf2c2480 con 0x55d986bf0800 > -4810> 2018-07-11 07:51:33.901645 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 938 0x55d9bf2c26c0 rep_scrubmap(5.2e e501 from shard 4) v1 > -4809> 2018-07-11 07:51:33.901665 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 938 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12480 (3686815630 0 925503479) 0x55d9bf2c26c0 con 0x55d986bec000 > -4808> 2018-07-11 07:51:33.962437 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -4807> 2018-07-11 07:51:33.962462 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -4806> 2018-07-11 07:51:33.962521 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -4805> 2018-07-11 07:51:33.962992 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -4804> 2018-07-11 07:51:34.010688 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75526fa3:::benchmark_data_magna115_11707_object173741:0,end:5:7552eb14:::benchmark_data_magna115_10867_object2369450:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454500 con 0 > -4803> 2018-07-11 07:51:34.010744 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75526fa3:::benchmark_data_magna115_11707_object173741:0,end:5:7552eb14:::benchmark_data_magna115_10867_object2369450:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98944b680 con 0 > -4802> 2018-07-11 07:51:34.201637 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 939 0x55d9bf2c2900 rep_scrubmap(5.2e e501 from shard 4) v1 > -4801> 2018-07-11 07:51:34.201656 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 939 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12472 (3686815630 0 3129703622) 0x55d9bf2c2900 con 0x55d986bec000 > -4800> 2018-07-11 07:51:34.207806 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 945 0x55d9bf2c2b40 rep_scrubmap(5.2e e501 from shard 2) v1 > -4799> 2018-07-11 07:51:34.207825 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 945 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12472 (1269249638 0 3129703622) 0x55d9bf2c2b40 con 0x55d986bf0800 > -4798> 2018-07-11 07:51:34.307376 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -4797> 2018-07-11 07:51:34.307403 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -4796> 2018-07-11 07:51:34.307447 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -4795> 2018-07-11 07:51:34.307912 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -4794> 2018-07-11 07:51:34.308961 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7552eb14:::benchmark_data_magna115_10867_object2369450:0,end:5:75534475:::benchmark_data_magna115_11707_object396250:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455180 con 0 > -4793> 2018-07-11 07:51:34.309008 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7552eb14:::benchmark_data_magna115_10867_object2369450:0,end:5:75534475:::benchmark_data_magna115_11707_object396250:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454780 con 0 > -4792> 2018-07-11 07:51:34.348280 7ffacbbd1700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15263 conn(0x55d985d44000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.5 seq 235 0x55d9b2783a00 osd_ping(ping e501 stamp 2018-07-11 07:51:35.579239) v4 > -4791> 2018-07-11 07:51:34.348305 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 <== osd.5 10.8.128.113:0/15263 235 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:35.579239) v4 ==== 2004+0+0 (645904450 0 0) 0x55d9b2783a00 con 0x55d985d44000 > -4790> 2018-07-11 07:51:34.348298 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15263 conn(0x55d985d45800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.5 seq 235 0x55d982cdf800 osd_ping(ping e501 stamp 2018-07-11 07:51:35.579239) v4 > -4789> 2018-07-11 07:51:34.348315 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:35.579239) v4 -- 0x55d986d33000 con 0 > -4788> 2018-07-11 07:51:34.348316 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.5 10.8.128.113:0/15263 235 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:35.579239) v4 ==== 2004+0+0 (645904450 0 0) 0x55d982cdf800 con 0x55d985d45800 > -4787> 2018-07-11 07:51:34.348338 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:35.579239) v4 -- 0x55d986cd6a00 con 0 > -4786> 2018-07-11 07:51:34.446944 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.107:0/12617 conn(0x55d98123d800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 238 0x55d986d97000 osd_ping(ping e501 stamp 2018-07-11 07:51:34.446600) v4 > -4785> 2018-07-11 07:51:34.446953 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.107:0/12617 conn(0x55d98123c000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 238 0x55d986d33a00 osd_ping(ping e501 stamp 2018-07-11 07:51:34.446600) v4 > -4784> 2018-07-11 07:51:34.446968 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.9 10.8.128.107:0/12617 238 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:34.446600) v4 ==== 2004+0+0 (3196113576 0 0) 0x55d986d97000 con 0x55d98123d800 > -4783> 2018-07-11 07:51:34.446973 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.9 10.8.128.107:0/12617 238 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:34.446600) v4 ==== 2004+0+0 (3196113576 0 0) 0x55d986d33a00 con 0x55d98123c000 > -4782> 2018-07-11 07:51:34.446981 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:34.446600) v4 -- 0x55d986d33000 con 0 > -4781> 2018-07-11 07:51:34.447006 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:34.446600) v4 -- 0x55d9ba818e00 con 0 > -4780> 2018-07-11 07:51:34.464621 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12677 conn(0x55d986db4800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.1 seq 255 0x55d9829e5600 osd_ping(ping e501 stamp 2018-07-11 07:51:34.512297) v4 > -4779> 2018-07-11 07:51:34.464628 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12677 conn(0x55d986df2000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.1 seq 255 0x55d986d40a00 osd_ping(ping e501 stamp 2018-07-11 07:51:34.512297) v4 > -4778> 2018-07-11 07:51:34.464642 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.1 10.8.128.106:0/12677 255 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:34.512297) v4 ==== 2004+0+0 (3656936549 0 0) 0x55d9829e5600 con 0x55d986db4800 > -4777> 2018-07-11 07:51:34.464645 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.1 10.8.128.106:0/12677 255 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:34.512297) v4 ==== 2004+0+0 (3656936549 0 0) 0x55d986d40a00 con 0x55d986df2000 > -4776> 2018-07-11 07:51:34.464651 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:34.512297) v4 -- 0x55d986cd6a00 con 0 > -4775> 2018-07-11 07:51:34.464674 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:34.512297) v4 -- 0x55d9ba818e00 con 0 > -4774> 2018-07-11 07:51:34.533055 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 940 0x55d9bf2c2d80 rep_scrubmap(5.2e e501 from shard 4) v1 > -4773> 2018-07-11 07:51:34.533082 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 940 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12472 (3686815630 0 2260377032) 0x55d9bf2c2d80 con 0x55d986bec000 > -4772> 2018-07-11 07:51:34.544723 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 946 0x55d9bf2c2fc0 rep_scrubmap(5.2e e501 from shard 2) v1 > -4771> 2018-07-11 07:51:34.544742 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 946 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12472 (1269249638 0 2260377032) 0x55d9bf2c2fc0 con 0x55d986bf0800 > -4770> 2018-07-11 07:51:34.718292 7ffab1bdc700 5 osd.10 501 heartbeat: osd_stat(788 GB used, 142 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,2]) > -4769> 2018-07-11 07:51:34.718306 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6802/15255 -- osd_ping(ping e501 stamp 2018-07-11 07:51:34.718304) v4 -- 0x55d986ca9a00 con 0 > -4768> 2018-07-11 07:51:34.718345 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6803/15255 -- osd_ping(ping e501 stamp 2018-07-11 07:51:34.718304) v4 -- 0x55d9869bf400 con 0 > -4767> 2018-07-11 07:51:34.718362 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6806/12677 -- osd_ping(ping e501 stamp 2018-07-11 07:51:34.718304) v4 -- 0x55d986cd4c00 con 0 > -4766> 2018-07-11 07:51:34.718376 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6807/12677 -- osd_ping(ping e501 stamp 2018-07-11 07:51:34.718304) v4 -- 0x55d9869be400 con 0 > -4765> 2018-07-11 07:51:34.718402 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6808/12651 -- osd_ping(ping e501 stamp 2018-07-11 07:51:34.718304) v4 -- 0x55d9b3db2200 con 0 > -4764> 2018-07-11 07:51:34.718414 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6810/12651 -- osd_ping(ping e501 stamp 2018-07-11 07:51:34.718304) v4 -- 0x55d9bf805e00 con 0 > -4763> 2018-07-11 07:51:34.718426 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6809/12650 -- osd_ping(ping e501 stamp 2018-07-11 07:51:34.718304) v4 -- 0x55d9b67d8600 con 0 > -4762> 2018-07-11 07:51:34.718438 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6811/12650 -- osd_ping(ping e501 stamp 2018-07-11 07:51:34.718304) v4 -- 0x55d9bcd78000 con 0 > -4761> 2018-07-11 07:51:34.718450 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6810/15287 -- osd_ping(ping e501 stamp 2018-07-11 07:51:34.718304) v4 -- 0x55d9b67d9a00 con 0 > -4760> 2018-07-11 07:51:34.718466 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6811/15287 -- osd_ping(ping e501 stamp 2018-07-11 07:51:34.718304) v4 -- 0x55d986c15000 con 0 > -4759> 2018-07-11 07:51:34.718478 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6806/15263 -- osd_ping(ping e501 stamp 2018-07-11 07:51:34.718304) v4 -- 0x55d9c3382000 con 0 > -4758> 2018-07-11 07:51:34.718491 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6807/15263 -- osd_ping(ping e501 stamp 2018-07-11 07:51:34.718304) v4 -- 0x55d980f5ac00 con 0 > -4757> 2018-07-11 07:51:34.718503 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.115:6802/13323 -- osd_ping(ping e501 stamp 2018-07-11 07:51:34.718304) v4 -- 0x55d986d96800 con 0 > -4756> 2018-07-11 07:51:34.718517 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.115:6803/13323 -- osd_ping(ping e501 stamp 2018-07-11 07:51:34.718304) v4 -- 0x55d9869bfc00 con 0 > -4755> 2018-07-11 07:51:34.718530 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6806/15885 -- osd_ping(ping e501 stamp 2018-07-11 07:51:34.718304) v4 -- 0x55d9b4388400 con 0 > -4754> 2018-07-11 07:51:34.718543 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6807/15885 -- osd_ping(ping e501 stamp 2018-07-11 07:51:34.718304) v4 -- 0x55d986d4e200 con 0 > -4753> 2018-07-11 07:51:34.718555 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.108:6802/11967 -- osd_ping(ping e501 stamp 2018-07-11 07:51:34.718304) v4 -- 0x55d97f5adc00 con 0 > -4752> 2018-07-11 07:51:34.718566 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.108:6803/11967 -- osd_ping(ping e501 stamp 2018-07-11 07:51:34.718304) v4 -- 0x55d986cd5200 con 0 > -4751> 2018-07-11 07:51:34.718580 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.107:6806/12617 -- osd_ping(ping e501 stamp 2018-07-11 07:51:34.718304) v4 -- 0x55d986d33200 con 0 > -4750> 2018-07-11 07:51:34.718592 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.107:6807/12617 -- osd_ping(ping e501 stamp 2018-07-11 07:51:34.718304) v4 -- 0x55d9b4388200 con 0 > -4749> 2018-07-11 07:51:34.718605 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6802/15919 -- osd_ping(ping e501 stamp 2018-07-11 07:51:34.718304) v4 -- 0x55d986d97200 con 0 > -4748> 2018-07-11 07:51:34.718617 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6803/15919 -- osd_ping(ping e501 stamp 2018-07-11 07:51:34.718304) v4 -- 0x55d9bdba5600 con 0 > -4747> 2018-07-11 07:51:34.718783 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.107:6806/12617 conn(0x55d986bd2000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.9 seq 243 0x55d9829e5600 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:34.718304) v4 > -4746> 2018-07-11 07:51:34.718806 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.9 10.8.128.107:6806/12617 243 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:34.718304) v4 ==== 2004+0+0 (4044449363 0 0) 0x55d9829e5600 con 0x55d986bd2000 > -4745> 2018-07-11 07:51:34.718828 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.107:6807/12617 conn(0x55d986bd3800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.9 seq 243 0x55d9b2783a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:34.718304) v4 > -4744> 2018-07-11 07:51:34.718853 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.9 10.8.128.107:6807/12617 243 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:34.718304) v4 ==== 2004+0+0 (4044449363 0 0) 0x55d9b2783a00 con 0x55d986bd3800 > -4743> 2018-07-11 07:51:34.718910 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6808/12651 conn(0x55d986b9a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.2 seq 243 0x55d986d33a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:34.718304) v4 > -4742> 2018-07-11 07:51:34.718939 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.2 10.8.128.106:6808/12651 243 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:34.718304) v4 ==== 2004+0+0 (4044449363 0 0) 0x55d986d33a00 con 0x55d986b9a000 > -4741> 2018-07-11 07:51:34.718940 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6803/15255 conn(0x55d986bc9800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.0 seq 243 0x55d9829e5600 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:34.718304) v4 > -4740> 2018-07-11 07:51:34.718957 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6809/12650 conn(0x55d986bcb000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 243 0x55d9b2783a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:34.718304) v4 > -4739> 2018-07-11 07:51:34.718964 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.0 10.8.128.113:6803/15255 243 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:34.718304) v4 ==== 2004+0+0 (4044449363 0 0) 0x55d9829e5600 con 0x55d986bc9800 > -4738> 2018-07-11 07:51:34.718973 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.3 10.8.128.106:6809/12650 243 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:34.718304) v4 ==== 2004+0+0 (4044449363 0 0) 0x55d9b2783a00 con 0x55d986bcb000 > -4737> 2018-07-11 07:51:34.718981 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6802/15255 conn(0x55d986bc8000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.0 seq 243 0x55d986d33a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:34.718304) v4 > -4736> 2018-07-11 07:51:34.718995 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.0 10.8.128.113:6802/15255 243 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:34.718304) v4 ==== 2004+0+0 (4044449363 0 0) 0x55d986d33a00 con 0x55d986bc8000 > -4735> 2018-07-11 07:51:34.719005 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6810/12651 conn(0x55d986b9b800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.2 seq 243 0x55d9829e5600 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:34.718304) v4 > -4734> 2018-07-11 07:51:34.719035 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6806/12677 conn(0x55d986b58000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.1 seq 243 0x55d9b2783a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:34.718304) v4 > -4733> 2018-07-11 07:51:34.719038 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.2 10.8.128.106:6810/12651 243 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:34.718304) v4 ==== 2004+0+0 (4044449363 0 0) 0x55d9829e5600 con 0x55d986b9b800 > -4732> 2018-07-11 07:51:34.719027 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6807/15263 conn(0x55d986b9e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.5 seq 243 0x55d986d33a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:34.718304) v4 > -4731> 2018-07-11 07:51:34.719053 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.1 10.8.128.106:6806/12677 243 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:34.718304) v4 ==== 2004+0+0 (4044449363 0 0) 0x55d9b2783a00 con 0x55d986b58000 > -4730> 2018-07-11 07:51:34.719059 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.5 10.8.128.113:6807/15263 243 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:34.718304) v4 ==== 2004+0+0 (4044449363 0 0) 0x55d986d33a00 con 0x55d986b9e800 > -4729> 2018-07-11 07:51:34.719071 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6811/15287 conn(0x55d986bb0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 243 0x55d9829e5600 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:34.718304) v4 > -4728> 2018-07-11 07:51:34.719087 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.4 10.8.128.113:6811/15287 243 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:34.718304) v4 ==== 2004+0+0 (4044449363 0 0) 0x55d9829e5600 con 0x55d986bb0800 > -4727> 2018-07-11 07:51:34.719087 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6806/15263 conn(0x55d986b9d000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.5 seq 243 0x55d9b2783a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:34.718304) v4 > -4726> 2018-07-11 07:51:34.719101 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6807/12677 conn(0x55d986b59800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.1 seq 243 0x55d986d33a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:34.718304) v4 > -4725> 2018-07-11 07:51:34.719101 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.5 10.8.128.113:6806/15263 243 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:34.718304) v4 ==== 2004+0+0 (4044449363 0 0) 0x55d9b2783a00 con 0x55d986b9d000 > -4724> 2018-07-11 07:51:34.719116 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.1 10.8.128.106:6807/12677 243 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:34.718304) v4 ==== 2004+0+0 (4044449363 0 0) 0x55d986d33a00 con 0x55d986b59800 > -4723> 2018-07-11 07:51:34.719133 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.115:6802/13323 conn(0x55d986bc1000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 243 0x55d9829e5600 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:34.718304) v4 > -4722> 2018-07-11 07:51:34.719150 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.6 10.8.128.115:6802/13323 243 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:34.718304) v4 ==== 2004+0+0 (4044449363 0 0) 0x55d9829e5600 con 0x55d986bc1000 > -4721> 2018-07-11 07:51:34.719151 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.115:6803/13323 conn(0x55d986bc2800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 243 0x55d9b2783a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:34.718304) v4 > -4720> 2018-07-11 07:51:34.719164 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.6 10.8.128.115:6803/13323 243 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:34.718304) v4 ==== 2004+0+0 (4044449363 0 0) 0x55d9b2783a00 con 0x55d986bc2800 > -4719> 2018-07-11 07:51:34.719161 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6811/12650 conn(0x55d986bcc800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 243 0x55d986d33a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:34.718304) v4 > -4718> 2018-07-11 07:51:34.719181 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.3 10.8.128.106:6811/12650 243 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:34.718304) v4 ==== 2004+0+0 (4044449363 0 0) 0x55d986d33a00 con 0x55d986bcc800 > -4717> 2018-07-11 07:51:34.719215 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.108:6803/11967 conn(0x55d986bbf800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.8 seq 243 0x55d986d33a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:34.718304) v4 > -4716> 2018-07-11 07:51:34.719214 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6807/15885 conn(0x55d986bad800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.7 seq 243 0x55d9b2783a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:34.718304) v4 > -4715> 2018-07-11 07:51:34.719198 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6802/15919 conn(0x55d986b5b000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.11 seq 243 0x55d9829e5600 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:34.718304) v4 > -4714> 2018-07-11 07:51:34.719229 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.7 10.8.128.114:6807/15885 243 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:34.718304) v4 ==== 2004+0+0 (4044449363 0 0) 0x55d9b2783a00 con 0x55d986bad800 > -4713> 2018-07-11 07:51:34.719230 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.8 10.8.128.108:6803/11967 243 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:34.718304) v4 ==== 2004+0+0 (4044449363 0 0) 0x55d986d33a00 con 0x55d986bbf800 > -4712> 2018-07-11 07:51:34.719235 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.11 10.8.128.114:6802/15919 243 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:34.718304) v4 ==== 2004+0+0 (4044449363 0 0) 0x55d9829e5600 con 0x55d986b5b000 > -4711> 2018-07-11 07:51:34.719268 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6810/15287 conn(0x55d986baf000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 243 0x55d986d33a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:34.718304) v4 > -4710> 2018-07-11 07:51:34.719280 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.4 10.8.128.113:6810/15287 243 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:34.718304) v4 ==== 2004+0+0 (4044449363 0 0) 0x55d986d33a00 con 0x55d986baf000 > -4709> 2018-07-11 07:51:34.719282 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.108:6802/11967 conn(0x55d986bbe000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.8 seq 243 0x55d9b2783a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:34.718304) v4 > -4708> 2018-07-11 07:51:34.719294 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.8 10.8.128.108:6802/11967 243 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:34.718304) v4 ==== 2004+0+0 (4044449363 0 0) 0x55d9b2783a00 con 0x55d986bbe000 > -4707> 2018-07-11 07:51:34.719300 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6806/15885 conn(0x55d986bac000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.7 seq 243 0x55d9829e5600 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:34.718304) v4 > -4706> 2018-07-11 07:51:34.719317 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.7 10.8.128.114:6806/15885 243 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:34.718304) v4 ==== 2004+0+0 (4044449363 0 0) 0x55d9829e5600 con 0x55d986bac000 > -4705> 2018-07-11 07:51:34.719322 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6803/15919 conn(0x55d986b5c800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.11 seq 243 0x55d9b2783a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:34.718304) v4 > -4704> 2018-07-11 07:51:34.719332 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.11 10.8.128.114:6803/15919 243 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:34.718304) v4 ==== 2004+0+0 (4044449363 0 0) 0x55d9b2783a00 con 0x55d986b5c800 > -4703> 2018-07-11 07:51:34.962526 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6801/15919 conn(0x55d986c3f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=29 cs=1 l=0). rx osd.11 seq 236 0x55d9b2783a00 MOSDScrubReserve(10.f REQUEST e501) v1 > -4702> 2018-07-11 07:51:34.962545 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.11 10.8.128.114:6801/15919 236 ==== MOSDScrubReserve(10.f REQUEST e501) v1 ==== 43+0+0 (2892966580 0 0) 0x55d9b2783a00 con 0x55d986c3f800 > -4701> 2018-07-11 07:51:34.962612 7ffab53e3700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.114:6801/15919 -- MOSDScrubReserve(10.f REJECT e501) v1 -- 0x55d9b60de600 con 0 > -4700> 2018-07-11 07:51:34.962927 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6801/15919 conn(0x55d986c3f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=29 cs=1 l=0). rx osd.11 seq 237 0x55d9b60de600 MOSDScrubReserve(10.f RELEASE e501) v1 > -4699> 2018-07-11 07:51:34.962942 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.11 10.8.128.114:6801/15919 237 ==== MOSDScrubReserve(10.f RELEASE e501) v1 ==== 43+0+0 (1668468918 0 0) 0x55d9b60de600 con 0x55d986c3f800 > -4698> 2018-07-11 07:51:35.066691 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15255 conn(0x55d986c82800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.0 seq 251 0x55d9829e5600 osd_ping(ping e501 stamp 2018-07-11 07:51:36.297699) v4 > -4697> 2018-07-11 07:51:35.066700 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15255 conn(0x55d986d20000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.0 seq 251 0x55d986d33a00 osd_ping(ping e501 stamp 2018-07-11 07:51:36.297699) v4 > -4696> 2018-07-11 07:51:35.066711 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.0 10.8.128.113:0/15255 251 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:36.297699) v4 ==== 2004+0+0 (506792828 0 0) 0x55d9829e5600 con 0x55d986c82800 > -4695> 2018-07-11 07:51:35.066716 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.0 10.8.128.113:0/15255 251 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:36.297699) v4 ==== 2004+0+0 (506792828 0 0) 0x55d986d33a00 con 0x55d986d20000 > -4694> 2018-07-11 07:51:35.066732 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:36.297699) v4 -- 0x55d986d97200 con 0 > -4693> 2018-07-11 07:51:35.066763 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:36.297699) v4 -- 0x55d986cd5200 con 0 > -4692> 2018-07-11 07:51:35.564433 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6805/15885 conn(0x55d986c7f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.7 seq 2798 0x55d986d40a00 MOSDScrubReserve(5.4 REQUEST e501) v1 > -4691> 2018-07-11 07:51:35.564451 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.7 10.8.128.114:6805/15885 2798 ==== MOSDScrubReserve(5.4 REQUEST e501) v1 ==== 43+0+0 (1753221512 0 0) 0x55d986d40a00 con 0x55d986c7f800 > -4690> 2018-07-11 07:51:35.564525 7ffab33df700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.114:6805/15885 -- MOSDScrubReserve(5.4 REJECT e501) v1 -- 0x55d986d32200 con 0 > -4689> 2018-07-11 07:51:35.564798 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6805/15885 conn(0x55d986c7f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.7 seq 2799 0x55d986cd5200 MOSDScrubReserve(5.4 RELEASE e501) v1 > -4688> 2018-07-11 07:51:35.564829 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.7 10.8.128.114:6805/15885 2799 ==== MOSDScrubReserve(5.4 RELEASE e501) v1 ==== 43+0+0 (2812146570 0 0) 0x55d986cd5200 con 0x55d986c7f800 > -4687> 2018-07-11 07:51:35.618526 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 32] Generated table #36804: 2757316 keys, 74844399 bytes > -4686> 2018-07-11 07:51:35.618572 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295495618544, "cf_name": "default", "job": 32, "event": "table_file_creation", "file_number": 36804, "file_size": 74844399, "table_properties": {"data_size": 67112064, "index_size": 690050, "filter_size": 7041293, "raw_key_size": 79655814, "raw_average_key_size": 28, "raw_value_size": 29755235, "raw_average_value_size": 10, "num_data_blocks": 16489, "num_entries": 2757316, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "254540", "kMergeOperands": "0"}} > -4685> 2018-07-11 07:51:35.645460 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -4684> 2018-07-11 07:51:35.645491 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -4683> 2018-07-11 07:51:35.645543 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -4682> 2018-07-11 07:51:35.646012 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -4681> 2018-07-11 07:51:35.660968 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'66891,epoch:501/498,start:5:75534475:::benchmark_data_magna115_11707_object396250:0,end:5:7553815c:::benchmark_data_magna115_11707_object152496:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454f00 con 0 > -4680> 2018-07-11 07:51:35.661020 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'66891,epoch:501/498,start:5:75534475:::benchmark_data_magna115_11707_object396250:0,end:5:7553815c:::benchmark_data_magna115_11707_object152496:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455400 con 0 > -4679> 2018-07-11 07:51:35.816120 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12650 conn(0x55d986d23000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 244 0x55d982cdf800 osd_ping(ping e501 stamp 2018-07-11 07:51:35.863660) v4 > -4678> 2018-07-11 07:51:35.816144 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.3 10.8.128.106:0/12650 244 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:35.863660) v4 ==== 2004+0+0 (978231166 0 0) 0x55d982cdf800 con 0x55d986d23000 > -4677> 2018-07-11 07:51:35.816142 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12650 conn(0x55d986d21800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 244 0x55d986d97000 osd_ping(ping e501 stamp 2018-07-11 07:51:35.863660) v4 > -4676> 2018-07-11 07:51:35.816153 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:35.863660) v4 -- 0x55d986d97200 con 0 > -4675> 2018-07-11 07:51:35.816161 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.3 10.8.128.106:0/12650 244 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:35.863660) v4 ==== 2004+0+0 (978231166 0 0) 0x55d986d97000 con 0x55d986d21800 > -4674> 2018-07-11 07:51:35.816173 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:35.863660) v4 -- 0x55d9b4388200 con 0 > -4673> 2018-07-11 07:51:35.874923 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 947 0x55d9bf2c3200 rep_scrubmap(5.2e e501 from shard 2) v1 > -4672> 2018-07-11 07:51:35.874944 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 947 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12466 (1269249638 0 2676362158) 0x55d9bf2c3200 con 0x55d986bf0800 > -4671> 2018-07-11 07:51:35.881569 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 941 0x55d9bf2c3440 rep_scrubmap(5.2e e501 from shard 4) v1 > -4670> 2018-07-11 07:51:35.881589 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 941 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12466 (3686815630 0 2676362158) 0x55d9bf2c3440 con 0x55d986bec000 > -4669> 2018-07-11 07:51:35.962768 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6801/15919 conn(0x55d986c3f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=29 cs=1 l=0). rx osd.11 seq 238 0x55d9bdba5600 MOSDScrubReserve(10.f REQUEST e501) v1 > -4668> 2018-07-11 07:51:35.962785 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.11 10.8.128.114:6801/15919 238 ==== MOSDScrubReserve(10.f REQUEST e501) v1 ==== 43+0+0 (2892966580 0 0) 0x55d9bdba5600 con 0x55d986c3f800 > -4667> 2018-07-11 07:51:35.962914 7ffab53e3700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.114:6801/15919 -- MOSDScrubReserve(10.f REJECT e501) v1 -- 0x55d9b60de600 con 0 > -4666> 2018-07-11 07:51:35.963139 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6801/15919 conn(0x55d986c3f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=29 cs=1 l=0). rx osd.11 seq 239 0x55d9b4388200 MOSDScrubReserve(10.f RELEASE e501) v1 > -4665> 2018-07-11 07:51:35.963153 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.11 10.8.128.114:6801/15919 239 ==== MOSDScrubReserve(10.f RELEASE e501) v1 ==== 43+0+0 (1668468918 0 0) 0x55d9b4388200 con 0x55d986c3f800 > -4664> 2018-07-11 07:51:36.001969 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.115:0/13323 conn(0x55d986d7e000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 252 0x55d986d97000 osd_ping(ping e501 stamp 2018-07-11 07:51:36.347735) v4 > -4663> 2018-07-11 07:51:36.001992 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.6 10.8.128.115:0/13323 252 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:36.347735) v4 ==== 2004+0+0 (4143195650 0 0) 0x55d986d97000 con 0x55d986d7e000 > -4662> 2018-07-11 07:51:36.002002 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:36.347735) v4 -- 0x55d97f5adc00 con 0 > -4661> 2018-07-11 07:51:36.002056 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.115:0/13323 conn(0x55d986d24800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 252 0x55d986d32200 osd_ping(ping e501 stamp 2018-07-11 07:51:36.347735) v4 > -4660> 2018-07-11 07:51:36.002081 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.6 10.8.128.115:0/13323 252 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:36.347735) v4 ==== 2004+0+0 (4143195650 0 0) 0x55d986d32200 con 0x55d986d24800 > -4659> 2018-07-11 07:51:36.002093 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:36.347735) v4 -- 0x55d980f5ac00 con 0 > -4658> 2018-07-11 07:51:36.147594 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.107:0/12617 conn(0x55d98123c000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 239 0x55d986d33a00 osd_ping(ping e501 stamp 2018-07-11 07:51:36.147075) v4 > -4657> 2018-07-11 07:51:36.147617 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.9 10.8.128.107:0/12617 239 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:36.147075) v4 ==== 2004+0+0 (353181829 0 0) 0x55d986d33a00 con 0x55d98123c000 > -4656> 2018-07-11 07:51:36.147634 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:36.147075) v4 -- 0x55d980f5ac00 con 0 > -4655> 2018-07-11 07:51:36.147705 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.107:0/12617 conn(0x55d98123d800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 239 0x55d9b60de600 osd_ping(ping e501 stamp 2018-07-11 07:51:36.147075) v4 > -4654> 2018-07-11 07:51:36.147726 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.9 10.8.128.107:0/12617 239 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:36.147075) v4 ==== 2004+0+0 (353181829 0 0) 0x55d9b60de600 con 0x55d98123d800 > -4653> 2018-07-11 07:51:36.147737 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:36.147075) v4 -- 0x55d97f5adc00 con 0 > -4652> 2018-07-11 07:51:36.213043 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -4651> 2018-07-11 07:51:36.213074 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -4650> 2018-07-11 07:51:36.213120 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -4649> 2018-07-11 07:51:36.213590 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -4648> 2018-07-11 07:51:36.236146 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7553815c:::benchmark_data_magna115_11707_object152496:0,end:5:7553cfb3:::benchmark_data_magna115_10867_object3146302:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498280 con 0 > -4647> 2018-07-11 07:51:36.236186 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7553815c:::benchmark_data_magna115_11707_object152496:0,end:5:7553cfb3:::benchmark_data_magna115_10867_object3146302:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498000 con 0 > -4646> 2018-07-11 07:51:36.468033 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 948 0x55d9bf2c3680 rep_scrubmap(5.2e e501 from shard 2) v1 > -4645> 2018-07-11 07:51:36.468052 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 948 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12454 (1269249638 0 2120516689) 0x55d9bf2c3680 con 0x55d986bf0800 > -4644> 2018-07-11 07:51:36.492624 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 942 0x55d9bf2c38c0 rep_scrubmap(5.2e e501 from shard 4) v1 > -4643> 2018-07-11 07:51:36.492644 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 942 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12454 (3686815630 0 2120516689) 0x55d9bf2c38c0 con 0x55d986bec000 > -4642> 2018-07-11 07:51:36.492786 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -4641> 2018-07-11 07:51:36.492812 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -4640> 2018-07-11 07:51:36.492858 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -4639> 2018-07-11 07:51:36.493152 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.114:0/15919 conn(0x55d986df3800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.11 seq 254 0x55d986d97000 osd_ping(ping e501 stamp 2018-07-11 07:51:37.123350) v4 > -4638> 2018-07-11 07:51:36.493148 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.114:0/15919 conn(0x55d986df5000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.11 seq 254 0x55d9829e5600 osd_ping(ping e501 stamp 2018-07-11 07:51:37.123350) v4 > -4637> 2018-07-11 07:51:36.493172 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.11 10.8.128.114:0/15919 254 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:37.123350) v4 ==== 2004+0+0 (1596122715 0 0) 0x55d986d97000 con 0x55d986df3800 > -4636> 2018-07-11 07:51:36.493178 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.11 10.8.128.114:0/15919 254 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:37.123350) v4 ==== 2004+0+0 (1596122715 0 0) 0x55d9829e5600 con 0x55d986df5000 > -4635> 2018-07-11 07:51:36.493184 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:37.123350) v4 -- 0x55d97f5adc00 con 0 > -4634> 2018-07-11 07:51:36.493218 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:37.123350) v4 -- 0x55d986d97200 con 0 > -4633> 2018-07-11 07:51:36.493347 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -4632> 2018-07-11 07:51:36.494160 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67911,epoch:501/498,start:5:7553cfb3:::benchmark_data_magna115_10867_object3146302:0,end:5:7554220b:::benchmark_data_magna115_11707_object143229:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455b80 con 0 > -4631> 2018-07-11 07:51:36.494194 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67911,epoch:501/498,start:5:7553cfb3:::benchmark_data_magna115_10867_object3146302:0,end:5:7554220b:::benchmark_data_magna115_11707_object143229:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455680 con 0 > -4630> 2018-07-11 07:51:36.648167 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.107:0/12617 conn(0x55d98123c000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 240 0x55d986d32200 osd_ping(ping e501 stamp 2018-07-11 07:51:36.647737) v4 > -4629> 2018-07-11 07:51:36.648188 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.9 10.8.128.107:0/12617 240 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:36.647737) v4 ==== 2004+0+0 (908800408 0 0) 0x55d986d32200 con 0x55d98123c000 > -4628> 2018-07-11 07:51:36.648201 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:36.647737) v4 -- 0x55d980f5ac00 con 0 > -4627> 2018-07-11 07:51:36.648293 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.107:0/12617 conn(0x55d98123d800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 240 0x55d9b60de600 osd_ping(ping e501 stamp 2018-07-11 07:51:36.647737) v4 > -4626> 2018-07-11 07:51:36.648315 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.9 10.8.128.107:0/12617 240 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:36.647737) v4 ==== 2004+0+0 (908800408 0 0) 0x55d9b60de600 con 0x55d98123d800 > -4625> 2018-07-11 07:51:36.648326 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:36.647737) v4 -- 0x55d97f5adc00 con 0 > -4624> 2018-07-11 07:51:36.688566 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 949 0x55d9bf2c3b00 rep_scrubmap(5.2e e501 from shard 2) v1 > -4623> 2018-07-11 07:51:36.688584 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 949 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12470 (1269249638 0 928983030) 0x55d9bf2c3b00 con 0x55d986bf0800 > -4622> 2018-07-11 07:51:36.724032 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 943 0x55d9bf2c3d40 rep_scrubmap(5.2e e501 from shard 4) v1 > -4621> 2018-07-11 07:51:36.724052 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 943 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12470 (3686815630 0 928983030) 0x55d9bf2c3d40 con 0x55d986bec000 > -4620> 2018-07-11 07:51:36.741922 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -4619> 2018-07-11 07:51:36.741945 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -4618> 2018-07-11 07:51:36.742000 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -4617> 2018-07-11 07:51:36.742447 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -4616> 2018-07-11 07:51:36.767118 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15255 conn(0x55d986c82800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.0 seq 252 0x55d982cdf800 osd_ping(ping e501 stamp 2018-07-11 07:51:37.998157) v4 > -4615> 2018-07-11 07:51:36.767139 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.0 10.8.128.113:0/15255 252 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:37.998157) v4 ==== 2004+0+0 (3211448917 0 0) 0x55d982cdf800 con 0x55d986c82800 > -4614> 2018-07-11 07:51:36.767148 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:37.998157) v4 -- 0x55d986d97200 con 0 > -4613> 2018-07-11 07:51:36.767137 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15255 conn(0x55d986d20000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.0 seq 252 0x55d986d33a00 osd_ping(ping e501 stamp 2018-07-11 07:51:37.998157) v4 > -4612> 2018-07-11 07:51:36.767157 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.0 10.8.128.113:0/15255 252 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:37.998157) v4 ==== 2004+0+0 (3211448917 0 0) 0x55d986d33a00 con 0x55d986d20000 > -4611> 2018-07-11 07:51:36.767167 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:37.998157) v4 -- 0x55d980f5ac00 con 0 > -4610> 2018-07-11 07:51:36.786393 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7554220b:::benchmark_data_magna115_11707_object143229:0,end:5:7554b9ec:::benchmark_data_magna115_10867_object3098413:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455900 con 0 > -4609> 2018-07-11 07:51:36.786431 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7554220b:::benchmark_data_magna115_11707_object143229:0,end:5:7554b9ec:::benchmark_data_magna115_10867_object3098413:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b14f0500 con 0 > -4608> 2018-07-11 07:51:37.012952 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 950 0x55d9b64f6000 rep_scrubmap(5.2e e501 from shard 2) v1 > -4607> 2018-07-11 07:51:37.012973 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 950 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12462 (1269249638 0 2399443972) 0x55d9b64f6000 con 0x55d986bf0800 > -4606> 2018-07-11 07:51:37.038231 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 944 0x55d9b64f6240 rep_scrubmap(5.2e e501 from shard 4) v1 > -4605> 2018-07-11 07:51:37.038259 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 944 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12462 (3686815630 0 2399443972) 0x55d9b64f6240 con 0x55d986bec000 > -4604> 2018-07-11 07:51:37.063102 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -4603> 2018-07-11 07:51:37.063128 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -4602> 2018-07-11 07:51:37.063183 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -4601> 2018-07-11 07:51:37.063661 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -4600> 2018-07-11 07:51:37.064731 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7554b9ec:::benchmark_data_magna115_10867_object3098413:0,end:5:75550089:::benchmark_data_magna115_10867_object1799326:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498780 con 0 > -4599> 2018-07-11 07:51:37.064777 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7554b9ec:::benchmark_data_magna115_10867_object3098413:0,end:5:75550089:::benchmark_data_magna115_10867_object1799326:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98696f400 con 0 > -4598> 2018-07-11 07:51:37.252331 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 951 0x55d9b64f6480 rep_scrubmap(5.2e e501 from shard 2) v1 > -4597> 2018-07-11 07:51:37.252352 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 951 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12476 (1269249638 0 1585652527) 0x55d9b64f6480 con 0x55d986bf0800 > -4596> 2018-07-11 07:51:37.275234 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 945 0x55d9b64f66c0 rep_scrubmap(5.2e e501 from shard 4) v1 > -4595> 2018-07-11 07:51:37.275263 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 945 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12476 (3686815630 0 1585652527) 0x55d9b64f66c0 con 0x55d986bec000 > -4594> 2018-07-11 07:51:37.312040 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -4593> 2018-07-11 07:51:37.312063 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -4592> 2018-07-11 07:51:37.312117 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -4591> 2018-07-11 07:51:37.312577 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -4590> 2018-07-11 07:51:37.350645 7ffab6be6700 1 -- 10.8.128.107:6800/13927 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55d9ba81ab00 con 0 > -4589> 2018-07-11 07:51:37.350718 7ffab6be6700 1 -- 10.8.128.107:6800/13927 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55d9869ca000 con 0 > -4588> 2018-07-11 07:51:37.353108 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67556,epoch:501/498,start:5:75550089:::benchmark_data_magna115_10867_object1799326:0,end:5:75556fba:::benchmark_data_magna115_10867_object3301208:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b14f0280 con 0 > -4587> 2018-07-11 07:51:37.353159 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67556,epoch:501/498,start:5:75550089:::benchmark_data_magna115_10867_object1799326:0,end:5:75556fba:::benchmark_data_magna115_10867_object3301208:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b14f0000 con 0 > -4586> 2018-07-11 07:51:37.365076 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12677 conn(0x55d986df2000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.1 seq 256 0x55d986d32200 osd_ping(ping e501 stamp 2018-07-11 07:51:37.412759) v4 > -4585> 2018-07-11 07:51:37.365072 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12677 conn(0x55d986db4800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.1 seq 256 0x55d9829e5600 osd_ping(ping e501 stamp 2018-07-11 07:51:37.412759) v4 > -4584> 2018-07-11 07:51:37.365104 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.1 10.8.128.106:0/12677 256 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:37.412759) v4 ==== 2004+0+0 (3932379757 0 0) 0x55d9829e5600 con 0x55d986db4800 > -4583> 2018-07-11 07:51:37.365101 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.1 10.8.128.106:0/12677 256 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:37.412759) v4 ==== 2004+0+0 (3932379757 0 0) 0x55d986d32200 con 0x55d986df2000 > -4582> 2018-07-11 07:51:37.365121 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:37.412759) v4 -- 0x55d986d97200 con 0 > -4581> 2018-07-11 07:51:37.365145 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:37.412759) v4 -- 0x55d980f5ac00 con 0 > -4580> 2018-07-11 07:51:37.384304 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15287 conn(0x55d981240800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 264 0x55d986d33a00 osd_ping(ping e501 stamp 2018-07-11 07:51:38.615419) v4 > -4579> 2018-07-11 07:51:37.384326 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.4 10.8.128.113:0/15287 264 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:38.615419) v4 ==== 2004+0+0 (4098127886 0 0) 0x55d986d33a00 con 0x55d981240800 > -4578> 2018-07-11 07:51:37.384342 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:38.615419) v4 -- 0x55d980f5ac00 con 0 > -4577> 2018-07-11 07:51:37.384348 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15287 conn(0x55d98123f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 264 0x55d982cdf800 osd_ping(ping e501 stamp 2018-07-11 07:51:38.615419) v4 > -4576> 2018-07-11 07:51:37.384369 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.4 10.8.128.113:0/15287 264 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:38.615419) v4 ==== 2004+0+0 (4098127886 0 0) 0x55d982cdf800 con 0x55d98123f000 > -4575> 2018-07-11 07:51:37.384380 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:38.615419) v4 -- 0x55d986d97200 con 0 > -4574> 2018-07-11 07:51:37.583091 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 952 0x55d9b64f6900 rep_scrubmap(5.2e e501 from shard 2) v1 > -4573> 2018-07-11 07:51:37.583111 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 952 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12474 (1269249638 0 365015885) 0x55d9b64f6900 con 0x55d986bf0800 > -4572> 2018-07-11 07:51:37.594355 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 946 0x55d9b64f6b40 rep_scrubmap(5.2e e501 from shard 4) v1 > -4571> 2018-07-11 07:51:37.594375 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 946 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12474 (3686815630 0 365015885) 0x55d9b64f6b40 con 0x55d986bec000 > -4570> 2018-07-11 07:51:37.594515 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -4569> 2018-07-11 07:51:37.594541 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -4568> 2018-07-11 07:51:37.594590 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -4567> 2018-07-11 07:51:37.595046 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -4566> 2018-07-11 07:51:37.596098 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75556fba:::benchmark_data_magna115_10867_object3301208:0,end:5:7555cf6c:::benchmark_data_magna115_11707_object105979:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1499680 con 0 > -4565> 2018-07-11 07:51:37.596144 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75556fba:::benchmark_data_magna115_10867_object3301208:0,end:5:7555cf6c:::benchmark_data_magna115_11707_object105979:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498500 con 0 > -4564> 2018-07-11 07:51:37.624498 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12651 conn(0x55d986db3000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.2 seq 253 0x55d986d97000 osd_ping(ping e501 stamp 2018-07-11 07:51:37.672186) v4 > -4563> 2018-07-11 07:51:37.624520 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.2 10.8.128.106:0/12651 253 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:37.672186) v4 ==== 2004+0+0 (1228415927 0 0) 0x55d986d97000 con 0x55d986db3000 > -4562> 2018-07-11 07:51:37.624529 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:37.672186) v4 -- 0x55d97f5adc00 con 0 > -4561> 2018-07-11 07:51:37.624537 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12651 conn(0x55d986db1800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.2 seq 253 0x55d986d33a00 osd_ping(ping e501 stamp 2018-07-11 07:51:37.672186) v4 > -4560> 2018-07-11 07:51:37.624558 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.2 10.8.128.106:0/12651 253 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:37.672186) v4 ==== 2004+0+0 (1228415927 0 0) 0x55d986d33a00 con 0x55d986db1800 > -4559> 2018-07-11 07:51:37.624574 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:37.672186) v4 -- 0x55d980f5ac00 con 0 > -4558> 2018-07-11 07:51:37.748677 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.107:0/12617 conn(0x55d98123c000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 241 0x55d986d32200 osd_ping(ping e501 stamp 2018-07-11 07:51:37.748356) v4 > -4557> 2018-07-11 07:51:37.748706 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.9 10.8.128.107:0/12617 241 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:37.748356) v4 ==== 2004+0+0 (24400194 0 0) 0x55d986d32200 con 0x55d98123c000 > -4556> 2018-07-11 07:51:37.748718 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:37.748356) v4 -- 0x55d980f5ac00 con 0 > -4555> 2018-07-11 07:51:37.748703 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.107:0/12617 conn(0x55d98123d800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 241 0x55d9b60de600 osd_ping(ping e501 stamp 2018-07-11 07:51:37.748356) v4 > -4554> 2018-07-11 07:51:37.748733 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.9 10.8.128.107:0/12617 241 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:37.748356) v4 ==== 2004+0+0 (24400194 0 0) 0x55d9b60de600 con 0x55d98123d800 > -4553> 2018-07-11 07:51:37.748751 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:37.748356) v4 -- 0x55d97f5adc00 con 0 > -4552> 2018-07-11 07:51:37.806933 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 947 0x55d9b64f6d80 rep_scrubmap(5.2e e501 from shard 4) v1 > -4551> 2018-07-11 07:51:37.806952 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 947 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12468 (3686815630 0 1846292674) 0x55d9b64f6d80 con 0x55d986bec000 > -4550> 2018-07-11 07:51:37.807144 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 953 0x55d9b64f6fc0 rep_scrubmap(5.2e e501 from shard 2) v1 > -4549> 2018-07-11 07:51:37.807164 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 953 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12468 (1269249638 0 1846292674) 0x55d9b64f6fc0 con 0x55d986bf0800 > -4548> 2018-07-11 07:51:37.821864 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -4547> 2018-07-11 07:51:37.821890 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -4546> 2018-07-11 07:51:37.821935 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -4545> 2018-07-11 07:51:37.822403 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -4544> 2018-07-11 07:51:37.839129 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 3004 0x55d986d97000 MOSDScrubReserve(11.3s5 REQUEST e501) v1 > -4543> 2018-07-11 07:51:37.839148 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 3004 ==== MOSDScrubReserve(11.3s5 REQUEST e501) v1 ==== 43+0+0 (3260170403 0 0) 0x55d986d97000 con 0x55d986c7e000 > -4542> 2018-07-11 07:51:37.839187 7ffab3be0700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e501) v1 -- 0x55d986d97e00 con 0 > -4541> 2018-07-11 07:51:37.839588 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 3005 0x55d986d97e00 MOSDScrubReserve(11.3s5 RELEASE e501) v1 > -4540> 2018-07-11 07:51:37.839607 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 3005 ==== MOSDScrubReserve(11.3s5 RELEASE e501) v1 ==== 43+0+0 (223328929 0 0) 0x55d986d97e00 con 0x55d986c7e000 > -4539> 2018-07-11 07:51:37.869990 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7555cf6c:::benchmark_data_magna115_11707_object105979:0,end:5:75564963:::benchmark_data_magna115_10867_object2055206:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1499900 con 0 > -4538> 2018-07-11 07:51:37.870027 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7555cf6c:::benchmark_data_magna115_11707_object105979:0,end:5:75564963:::benchmark_data_magna115_10867_object2055206:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498c80 con 0 > -4537> 2018-07-11 07:51:37.889162 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.108:0/11967 conn(0x55d986db0000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.8 seq 234 0x55d9829e5600 osd_ping(ping e501 stamp 2018-07-11 07:51:37.644892) v4 > -4536> 2018-07-11 07:51:37.889199 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.8 10.8.128.108:0/11967 234 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:37.644892) v4 ==== 2004+0+0 (4291914556 0 0) 0x55d9829e5600 con 0x55d986db0000 > -4535> 2018-07-11 07:51:37.889212 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:37.644892) v4 -- 0x55d986d97200 con 0 > -4534> 2018-07-11 07:51:37.889229 7ffacbbd1700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.108:0/11967 conn(0x55d986d82800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.8 seq 234 0x55d9b60de600 osd_ping(ping e501 stamp 2018-07-11 07:51:37.644892) v4 > -4533> 2018-07-11 07:51:37.889257 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 <== osd.8 10.8.128.108:0/11967 234 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:37.644892) v4 ==== 2004+0+0 (4291914556 0 0) 0x55d9b60de600 con 0x55d986d82800 > -4532> 2018-07-11 07:51:37.889268 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:37.644892) v4 -- 0x55d986d4e200 con 0 > -4531> 2018-07-11 07:51:37.961701 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6805/15263 conn(0x55d986c41000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=31 cs=1 l=0). rx osd.5 seq 2824 0x55d986d32200 MOSDScrubReserve(6.15 REQUEST e501) v1 > -4530> 2018-07-11 07:51:37.961719 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.5 10.8.128.113:6805/15263 2824 ==== MOSDScrubReserve(6.15 REQUEST e501) v1 ==== 43+0+0 (3942742026 0 0) 0x55d986d32200 con 0x55d986c41000 > -4529> 2018-07-11 07:51:38.078952 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6805/15263 -- MOSDScrubReserve(6.15 REJECT e501) v1 -- 0x55d986c22e00 con 0 > -4528> 2018-07-11 07:51:38.079427 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6805/15263 conn(0x55d986c41000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=31 cs=1 l=0). rx osd.5 seq 2825 0x55d986c22e00 MOSDScrubReserve(6.15 RELEASE e501) v1 > -4527> 2018-07-11 07:51:38.079445 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.5 10.8.128.113:6805/15263 2825 ==== MOSDScrubReserve(6.15 RELEASE e501) v1 ==== 43+0+0 (605848072 0 0) 0x55d986c22e00 con 0x55d986c41000 > -4526> 2018-07-11 07:51:38.085015 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 954 0x55d9b64f7200 rep_scrubmap(5.2e e501 from shard 2) v1 > -4525> 2018-07-11 07:51:38.085034 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 954 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12480 (1269249638 0 775905819) 0x55d9b64f7200 con 0x55d986bf0800 > -4524> 2018-07-11 07:51:38.112524 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 948 0x55d9b64f7440 rep_scrubmap(5.2e e501 from shard 4) v1 > -4523> 2018-07-11 07:51:38.112544 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 948 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12480 (3686815630 0 775905819) 0x55d9b64f7440 con 0x55d986bec000 > -4522> 2018-07-11 07:51:38.112661 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -4521> 2018-07-11 07:51:38.112684 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -4520> 2018-07-11 07:51:38.112731 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -4519> 2018-07-11 07:51:38.113195 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -4518> 2018-07-11 07:51:38.114183 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75564963:::benchmark_data_magna115_10867_object2055206:0,end:5:7556a6db:::benchmark_data_magna115_10867_object3789943:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1499180 con 0 > -4517> 2018-07-11 07:51:38.114228 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75564963:::benchmark_data_magna115_10867_object2055206:0,end:5:7556a6db:::benchmark_data_magna115_10867_object3789943:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1499b80 con 0 > -4516> 2018-07-11 07:51:38.249147 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.107:0/12617 conn(0x55d98123c000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 242 0x55d980f5ac00 osd_ping(ping e501 stamp 2018-07-11 07:51:38.248811) v4 > -4515> 2018-07-11 07:51:38.249169 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.9 10.8.128.107:0/12617 242 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:38.248811) v4 ==== 2004+0+0 (759671026 0 0) 0x55d980f5ac00 con 0x55d98123c000 > -4514> 2018-07-11 07:51:38.249181 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:38.248811) v4 -- 0x55d9b67d9a00 con 0 > -4513> 2018-07-11 07:51:38.249171 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.107:0/12617 conn(0x55d98123d800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 242 0x55d97f5adc00 osd_ping(ping e501 stamp 2018-07-11 07:51:38.248811) v4 > -4512> 2018-07-11 07:51:38.249201 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.9 10.8.128.107:0/12617 242 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:38.248811) v4 ==== 2004+0+0 (759671026 0 0) 0x55d97f5adc00 con 0x55d98123d800 > -4511> 2018-07-11 07:51:38.249222 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:38.248811) v4 -- 0x55d986d4e200 con 0 > -4510> 2018-07-11 07:51:38.315143 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 955 0x55d9b64f7680 rep_scrubmap(5.2e e501 from shard 2) v1 > -4509> 2018-07-11 07:51:38.315162 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 955 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12474 (1269249638 0 954900964) 0x55d9b64f7680 con 0x55d986bf0800 > -4508> 2018-07-11 07:51:38.318910 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.114:0/15885 conn(0x55d986d7f800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.7 seq 258 0x55d982cdf800 osd_ping(ping e501 stamp 2018-07-11 07:51:38.949137) v4 > -4507> 2018-07-11 07:51:38.318919 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.114:0/15885 conn(0x55d986d81000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.7 seq 258 0x55d980f5ac00 osd_ping(ping e501 stamp 2018-07-11 07:51:38.949137) v4 > -4506> 2018-07-11 07:51:38.318936 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.7 10.8.128.114:0/15885 258 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:38.949137) v4 ==== 2004+0+0 (3803254682 0 0) 0x55d980f5ac00 con 0x55d986d81000 > -4505> 2018-07-11 07:51:38.318932 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.7 10.8.128.114:0/15885 258 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:38.949137) v4 ==== 2004+0+0 (3803254682 0 0) 0x55d982cdf800 con 0x55d986d7f800 > -4504> 2018-07-11 07:51:38.318945 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:38.949137) v4 -- 0x55d9b67d9a00 con 0 > -4503> 2018-07-11 07:51:38.318969 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:38.949137) v4 -- 0x55d986d97200 con 0 > -4502> 2018-07-11 07:51:38.335146 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 949 0x55d9b64f78c0 rep_scrubmap(5.2e e501 from shard 4) v1 > -4501> 2018-07-11 07:51:38.335166 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 949 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12474 (3686815630 0 954900964) 0x55d9b64f78c0 con 0x55d986bec000 > -4500> 2018-07-11 07:51:38.335331 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -4499> 2018-07-11 07:51:38.335357 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -4498> 2018-07-11 07:51:38.335404 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -4497> 2018-07-11 07:51:38.335867 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -4496> 2018-07-11 07:51:38.361933 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7556a6db:::benchmark_data_magna115_10867_object3789943:0,end:5:7556e48c:::benchmark_data_magna115_10867_object1917368:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498f00 con 0 > -4495> 2018-07-11 07:51:38.361983 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7556a6db:::benchmark_data_magna115_10867_object3789943:0,end:5:7556e48c:::benchmark_data_magna115_10867_object1917368:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1499400 con 0 > -4494> 2018-07-11 07:51:38.563866 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 956 0x55d9b64f7b00 rep_scrubmap(5.2e e501 from shard 2) v1 > -4493> 2018-07-11 07:51:38.563886 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 956 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12476 (1269249638 0 36920818) 0x55d9b64f7b00 con 0x55d986bf0800 > -4492> 2018-07-11 07:51:38.594436 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 950 0x55d9b64f7d40 rep_scrubmap(5.2e e501 from shard 4) v1 > -4491> 2018-07-11 07:51:38.594456 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 950 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12476 (3686815630 0 36920818) 0x55d9b64f7d40 con 0x55d986bec000 > -4490> 2018-07-11 07:51:38.640279 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -4489> 2018-07-11 07:51:38.640306 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -4488> 2018-07-11 07:51:38.640350 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -4487> 2018-07-11 07:51:38.640814 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -4486> 2018-07-11 07:51:38.641834 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7556e48c:::benchmark_data_magna115_10867_object1917368:0,end:5:7557398a:::benchmark_data_magna115_11707_object155540:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98696ec80 con 0 > -4485> 2018-07-11 07:51:38.641881 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7556e48c:::benchmark_data_magna115_10867_object1917368:0,end:5:7557398a:::benchmark_data_magna115_11707_object155540:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b14f0a00 con 0 > -4484> 2018-07-11 07:51:38.819303 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.114:0/15885 conn(0x55d986d7f800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.7 seq 259 0x55d9829e5600 osd_ping(ping e501 stamp 2018-07-11 07:51:39.449541) v4 > -4483> 2018-07-11 07:51:38.819319 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.114:0/15885 conn(0x55d986d81000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.7 seq 259 0x55d980f5ac00 osd_ping(ping e501 stamp 2018-07-11 07:51:39.449541) v4 > -4482> 2018-07-11 07:51:38.819330 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.7 10.8.128.114:0/15885 259 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:39.449541) v4 ==== 2004+0+0 (1236146028 0 0) 0x55d9829e5600 con 0x55d986d7f800 > -4481> 2018-07-11 07:51:38.819342 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.7 10.8.128.114:0/15885 259 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:39.449541) v4 ==== 2004+0+0 (1236146028 0 0) 0x55d980f5ac00 con 0x55d986d81000 > -4480> 2018-07-11 07:51:38.819351 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:39.449541) v4 -- 0x55d986d97200 con 0 > -4479> 2018-07-11 07:51:38.819376 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:39.449541) v4 -- 0x55d9b67d9a00 con 0 > -4478> 2018-07-11 07:51:38.845891 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 951 0x55d986a86000 rep_scrubmap(5.2e e501 from shard 4) v1 > -4477> 2018-07-11 07:51:38.845910 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 951 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12476 (3686815630 0 916776537) 0x55d986a86000 con 0x55d986bec000 > -4476> 2018-07-11 07:51:38.856502 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 957 0x55d986a86240 rep_scrubmap(5.2e e501 from shard 2) v1 > -4475> 2018-07-11 07:51:38.856522 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 957 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12476 (1269249638 0 916776537) 0x55d986a86240 con 0x55d986bf0800 > -4474> 2018-07-11 07:51:38.877633 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -4473> 2018-07-11 07:51:38.877657 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -4472> 2018-07-11 07:51:38.877720 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -4471> 2018-07-11 07:51:38.878175 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -4470> 2018-07-11 07:51:38.920661 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'66880,epoch:501/498,start:5:7557398a:::benchmark_data_magna115_11707_object155540:0,end:5:7557a098:::benchmark_data_magna115_10867_object200922:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98696f680 con 0 > -4469> 2018-07-11 07:51:38.920712 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'66880,epoch:501/498,start:5:7557398a:::benchmark_data_magna115_11707_object155540:0,end:5:7557a098:::benchmark_data_magna115_10867_object200922:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98696ef00 con 0 > -4468> 2018-07-11 07:51:39.139821 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 952 0x55d986a86480 rep_scrubmap(5.2e e501 from shard 4) v1 > -4467> 2018-07-11 07:51:39.139842 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 952 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12474 (3686815630 0 3308402651) 0x55d986a86480 con 0x55d986bec000 > -4466> 2018-07-11 07:51:39.153120 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 958 0x55d986a866c0 rep_scrubmap(5.2e e501 from shard 2) v1 > -4465> 2018-07-11 07:51:39.153140 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 958 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12474 (1269249638 0 3308402651) 0x55d986a866c0 con 0x55d986bf0800 > -4464> 2018-07-11 07:51:39.190088 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -4463> 2018-07-11 07:51:39.190113 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -4462> 2018-07-11 07:51:39.190172 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -4461> 2018-07-11 07:51:39.190650 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -4460> 2018-07-11 07:51:39.191553 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7557a098:::benchmark_data_magna115_10867_object200922:0,end:5:75580d35:::benchmark_data_magna115_10867_object544532:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498a00 con 0 > -4459> 2018-07-11 07:51:39.191600 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7557a098:::benchmark_data_magna115_10867_object200922:0,end:5:75580d35:::benchmark_data_magna115_10867_object544532:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98696f900 con 0 > -4458> 2018-07-11 07:51:39.393414 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.114:0/15919 conn(0x55d986df3800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.11 seq 255 0x55d9b60de600 osd_ping(ping e501 stamp 2018-07-11 07:51:40.023734) v4 > -4457> 2018-07-11 07:51:39.393439 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.11 10.8.128.114:0/15919 255 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:40.023734) v4 ==== 2004+0+0 (2294923917 0 0) 0x55d9b60de600 con 0x55d986df3800 > -4456> 2018-07-11 07:51:39.393449 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:40.023734) v4 -- 0x55d986d4e200 con 0 > -4455> 2018-07-11 07:51:39.393615 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.114:0/15919 conn(0x55d986df5000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.11 seq 255 0x55d982cdf800 osd_ping(ping e501 stamp 2018-07-11 07:51:40.023734) v4 > -4454> 2018-07-11 07:51:39.393634 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.11 10.8.128.114:0/15919 255 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:40.023734) v4 ==== 2004+0+0 (2294923917 0 0) 0x55d982cdf800 con 0x55d986df5000 > -4453> 2018-07-11 07:51:39.393645 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:40.023734) v4 -- 0x55d986d97200 con 0 > -4452> 2018-07-11 07:51:39.394857 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 953 0x55d986a86900 rep_scrubmap(5.2e e501 from shard 4) v1 > -4451> 2018-07-11 07:51:39.394877 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 953 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12474 (3686815630 0 518132336) 0x55d986a86900 con 0x55d986bec000 > -4450> 2018-07-11 07:51:39.395307 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 959 0x55d986a86b40 rep_scrubmap(5.2e e501 from shard 2) v1 > -4449> 2018-07-11 07:51:39.395327 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 959 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12474 (1269249638 0 518132336) 0x55d986a86b40 con 0x55d986bf0800 > -4448> 2018-07-11 07:51:39.425541 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -4447> 2018-07-11 07:51:39.425565 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -4446> 2018-07-11 07:51:39.425620 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -4445> 2018-07-11 07:51:39.426067 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -4444> 2018-07-11 07:51:39.462597 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67383,epoch:501/498,start:5:75580d35:::benchmark_data_magna115_10867_object544532:0,end:5:755889fe:::benchmark_data_magna115_10867_object1805022:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98696f180 con 0 > -4443> 2018-07-11 07:51:39.462650 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67383,epoch:501/498,start:5:75580d35:::benchmark_data_magna115_10867_object544532:0,end:5:755889fe:::benchmark_data_magna115_10867_object1805022:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98696e000 con 0 > -4442> 2018-07-11 07:51:39.564787 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6805/15885 conn(0x55d986c7f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.7 seq 2800 0x55d980f5ac00 MOSDScrubReserve(5.4 REQUEST e501) v1 > -4441> 2018-07-11 07:51:39.564804 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.7 10.8.128.114:6805/15885 2800 ==== MOSDScrubReserve(5.4 REQUEST e501) v1 ==== 43+0+0 (1753221512 0 0) 0x55d980f5ac00 con 0x55d986c7f800 > -4440> 2018-07-11 07:51:39.564929 7ffab33df700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.114:6805/15885 -- MOSDScrubReserve(5.4 REJECT e501) v1 -- 0x55d986cd5200 con 0 > -4439> 2018-07-11 07:51:39.565231 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6805/15885 conn(0x55d986c7f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.7 seq 2801 0x55d986cd5200 MOSDScrubReserve(5.4 RELEASE e501) v1 > -4438> 2018-07-11 07:51:39.565256 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.7 10.8.128.114:6805/15885 2801 ==== MOSDScrubReserve(5.4 RELEASE e501) v1 ==== 43+0+0 (2812146570 0 0) 0x55d986cd5200 con 0x55d986c7f800 > -4437> 2018-07-11 07:51:39.648635 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15263 conn(0x55d985d45800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.5 seq 236 0x55d9829e5600 osd_ping(ping e501 stamp 2018-07-11 07:51:40.879719) v4 > -4436> 2018-07-11 07:51:39.648635 7ffacbbd1700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15263 conn(0x55d985d44000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.5 seq 236 0x55d97f5adc00 osd_ping(ping e501 stamp 2018-07-11 07:51:40.879719) v4 > -4435> 2018-07-11 07:51:39.648664 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.5 10.8.128.113:0/15263 236 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:40.879719) v4 ==== 2004+0+0 (4142648186 0 0) 0x55d9829e5600 con 0x55d985d45800 > -4434> 2018-07-11 07:51:39.648667 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 <== osd.5 10.8.128.113:0/15263 236 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:40.879719) v4 ==== 2004+0+0 (4142648186 0 0) 0x55d97f5adc00 con 0x55d985d44000 > -4433> 2018-07-11 07:51:39.648684 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:40.879719) v4 -- 0x55d986d97200 con 0 > -4432> 2018-07-11 07:51:39.648713 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:40.879719) v4 -- 0x55d986d4e200 con 0 > -4431> 2018-07-11 07:51:39.700537 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 954 0x55d986a86d80 rep_scrubmap(5.2e e501 from shard 4) v1 > -4430> 2018-07-11 07:51:39.700558 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 954 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12470 (3686815630 0 1991285306) 0x55d986a86d80 con 0x55d986bec000 > -4429> 2018-07-11 07:51:39.704721 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 960 0x55d986a86fc0 rep_scrubmap(5.2e e501 from shard 2) v1 > -4428> 2018-07-11 07:51:39.704742 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 960 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12470 (1269249638 0 1991285306) 0x55d986a86fc0 con 0x55d986bf0800 > -4427> 2018-07-11 07:51:39.716760 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -4426> 2018-07-11 07:51:39.716788 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -4425> 2018-07-11 07:51:39.716845 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -4424> 2018-07-11 07:51:39.717315 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -4423> 2018-07-11 07:51:39.718143 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67785,epoch:501/498,start:5:755889fe:::benchmark_data_magna115_10867_object1805022:0,end:5:7558ebe7:::benchmark_data_magna115_10867_object1904301:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498780 con 0 > -4422> 2018-07-11 07:51:39.718190 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67785,epoch:501/498,start:5:755889fe:::benchmark_data_magna115_10867_object1805022:0,end:5:7558ebe7:::benchmark_data_magna115_10867_object1904301:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455900 con 0 > -4421> 2018-07-11 07:51:39.910765 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 961 0x55d986a87200 rep_scrubmap(5.2e e501 from shard 2) v1 > -4420> 2018-07-11 07:51:39.910786 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 961 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12476 (1269249638 0 3310396879) 0x55d986a87200 con 0x55d986bf0800 > -4419> 2018-07-11 07:51:39.918845 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 955 0x55d986a87440 rep_scrubmap(5.2e e501 from shard 4) v1 > -4418> 2018-07-11 07:51:39.918865 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 955 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12476 (3686815630 0 3310396879) 0x55d986a87440 con 0x55d986bec000 > -4417> 2018-07-11 07:51:39.919688 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.114:0/15885 conn(0x55d986d7f800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.7 seq 260 0x55d982cdf800 osd_ping(ping e501 stamp 2018-07-11 07:51:40.549948) v4 > -4416> 2018-07-11 07:51:39.919697 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.114:0/15885 conn(0x55d986d81000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.7 seq 260 0x55d9b67d9a00 osd_ping(ping e501 stamp 2018-07-11 07:51:40.549948) v4 > -4415> 2018-07-11 07:51:39.919711 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.7 10.8.128.114:0/15885 260 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:40.549948) v4 ==== 2004+0+0 (4193276335 0 0) 0x55d982cdf800 con 0x55d986d7f800 > -4414> 2018-07-11 07:51:39.919716 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.7 10.8.128.114:0/15885 260 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:40.549948) v4 ==== 2004+0+0 (4193276335 0 0) 0x55d9b67d9a00 con 0x55d986d81000 > -4413> 2018-07-11 07:51:39.919728 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:40.549948) v4 -- 0x55d986d97200 con 0 > -4412> 2018-07-11 07:51:39.919759 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:40.549948) v4 -- 0x55d9bcd78000 con 0 > -4411> 2018-07-11 07:51:39.961905 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6805/15263 conn(0x55d986c41000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=31 cs=1 l=0). rx osd.5 seq 2826 0x55d986d33a00 MOSDScrubReserve(6.15 REQUEST e501) v1 > -4410> 2018-07-11 07:51:39.961923 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.5 10.8.128.113:6805/15263 2826 ==== MOSDScrubReserve(6.15 REQUEST e501) v1 ==== 43+0+0 (3942742026 0 0) 0x55d986d33a00 con 0x55d986c41000 > -4409> 2018-07-11 07:51:39.987224 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6805/15263 -- MOSDScrubReserve(6.15 REJECT e501) v1 -- 0x55d986c22e00 con 0 > -4408> 2018-07-11 07:51:39.987276 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -4407> 2018-07-11 07:51:39.987296 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -4406> 2018-07-11 07:51:39.987344 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -4405> 2018-07-11 07:51:39.987621 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6805/15263 conn(0x55d986c41000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=31 cs=1 l=0). rx osd.5 seq 2827 0x55d986c22e00 MOSDScrubReserve(6.15 RELEASE e501) v1 > -4404> 2018-07-11 07:51:39.987639 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.5 10.8.128.113:6805/15263 2827 ==== MOSDScrubReserve(6.15 RELEASE e501) v1 ==== 43+0+0 (605848072 0 0) 0x55d986c22e00 con 0x55d986c41000 > -4403> 2018-07-11 07:51:39.987798 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -4402> 2018-07-11 07:51:40.004538 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7558ebe7:::benchmark_data_magna115_10867_object1904301:0,end:5:75594344:::benchmark_data_magna115_10867_object1213770:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b14f0500 con 0 > -4401> 2018-07-11 07:51:40.004588 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7558ebe7:::benchmark_data_magna115_10867_object1904301:0,end:5:75594344:::benchmark_data_magna115_10867_object1213770:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455680 con 0 > -4400> 2018-07-11 07:51:40.245279 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 956 0x55d986a87680 rep_scrubmap(5.2e e501 from shard 4) v1 > -4399> 2018-07-11 07:51:40.245300 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 956 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12474 (3686815630 0 3555351644) 0x55d986a87680 con 0x55d986bec000 > -4398> 2018-07-11 07:51:40.248121 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 962 0x55d986a878c0 rep_scrubmap(5.2e e501 from shard 2) v1 > -4397> 2018-07-11 07:51:40.248141 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 962 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12474 (1269249638 0 3555351644) 0x55d986a878c0 con 0x55d986bf0800 > -4396> 2018-07-11 07:51:40.289238 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -4395> 2018-07-11 07:51:40.289271 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -4394> 2018-07-11 07:51:40.289329 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -4393> 2018-07-11 07:51:40.289788 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -4392> 2018-07-11 07:51:40.290631 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67776,epoch:501/498,start:5:75594344:::benchmark_data_magna115_10867_object1213770:0,end:5:7559a465:::benchmark_data_magna115_11707_object404210:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455b80 con 0 > -4391> 2018-07-11 07:51:40.290677 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67776,epoch:501/498,start:5:75594344:::benchmark_data_magna115_10867_object1213770:0,end:5:7559a465:::benchmark_data_magna115_11707_object404210:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498000 con 0 > -4390> 2018-07-11 07:51:40.481780 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 963 0x55d986a87b00 rep_scrubmap(5.2e e501 from shard 2) v1 > -4389> 2018-07-11 07:51:40.481800 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 963 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12474 (1269249638 0 1123518973) 0x55d986a87b00 con 0x55d986bf0800 > -4388> 2018-07-11 07:51:40.507215 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 957 0x55d986a87d40 rep_scrubmap(5.2e e501 from shard 4) v1 > -4387> 2018-07-11 07:51:40.507235 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 957 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12474 (3686815630 0 1123518973) 0x55d986a87d40 con 0x55d986bec000 > -4386> 2018-07-11 07:51:40.549632 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.107:0/12617 conn(0x55d98123c000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 243 0x55d9b67d9a00 osd_ping(ping e501 stamp 2018-07-11 07:51:40.549290) v4 > -4385> 2018-07-11 07:51:40.549647 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.107:0/12617 conn(0x55d98123d800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 243 0x55d9b60de600 osd_ping(ping e501 stamp 2018-07-11 07:51:40.549290) v4 > -4384> 2018-07-11 07:51:40.549656 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.9 10.8.128.107:0/12617 243 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:40.549290) v4 ==== 2004+0+0 (1534017677 0 0) 0x55d9b67d9a00 con 0x55d98123c000 > -4383> 2018-07-11 07:51:40.549676 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.9 10.8.128.107:0/12617 243 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:40.549290) v4 ==== 2004+0+0 (1534017677 0 0) 0x55d9b60de600 con 0x55d98123d800 > -4382> 2018-07-11 07:51:40.549684 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:40.549290) v4 -- 0x55d9b3db2200 con 0 > -4381> 2018-07-11 07:51:40.549713 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:40.549290) v4 -- 0x55d986d4e200 con 0 > -4380> 2018-07-11 07:51:40.564977 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6805/15885 conn(0x55d986c7f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.7 seq 2802 0x55d9bcd78000 MOSDScrubReserve(5.4 REQUEST e501) v1 > -4379> 2018-07-11 07:51:40.564996 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.7 10.8.128.114:6805/15885 2802 ==== MOSDScrubReserve(5.4 REQUEST e501) v1 ==== 43+0+0 (1753221512 0 0) 0x55d9bcd78000 con 0x55d986c7f800 > -4378> 2018-07-11 07:51:40.565031 7ffab33df700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.114:6805/15885 -- MOSDScrubReserve(5.4 REJECT e501) v1 -- 0x55d986cd5200 con 0 > -4377> 2018-07-11 07:51:40.565366 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6805/15885 conn(0x55d986c7f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.7 seq 2803 0x55d986cd5200 MOSDScrubReserve(5.4 RELEASE e501) v1 > -4376> 2018-07-11 07:51:40.565381 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.7 10.8.128.114:6805/15885 2803 ==== MOSDScrubReserve(5.4 RELEASE e501) v1 ==== 43+0+0 (2812146570 0 0) 0x55d986cd5200 con 0x55d986c7f800 > -4375> 2018-07-11 07:51:40.597902 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -4374> 2018-07-11 07:51:40.597940 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -4373> 2018-07-11 07:51:40.597983 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -4372> 2018-07-11 07:51:40.598462 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -4371> 2018-07-11 07:51:40.618734 7ffab1bdc700 5 osd.10 501 heartbeat: osd_stat(788 GB used, 142 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist []) > -4370> 2018-07-11 07:51:40.618747 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6802/15255 -- osd_ping(ping e501 stamp 2018-07-11 07:51:40.618745) v4 -- 0x55d986d40600 con 0 > -4369> 2018-07-11 07:51:40.618771 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6803/15255 -- osd_ping(ping e501 stamp 2018-07-11 07:51:40.618745) v4 -- 0x55d9b3db2600 con 0 > -4368> 2018-07-11 07:51:40.618788 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6806/12677 -- osd_ping(ping e501 stamp 2018-07-11 07:51:40.618745) v4 -- 0x55d982d31c00 con 0 > -4367> 2018-07-11 07:51:40.618803 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6807/12677 -- osd_ping(ping e501 stamp 2018-07-11 07:51:40.618745) v4 -- 0x55d9c3383a00 con 0 > -4366> 2018-07-11 07:51:40.618816 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6808/12651 -- osd_ping(ping e501 stamp 2018-07-11 07:51:40.618745) v4 -- 0x55d9829e4e00 con 0 > -4365> 2018-07-11 07:51:40.618824 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6810/12651 -- osd_ping(ping e501 stamp 2018-07-11 07:51:40.618745) v4 -- 0x55d9869bec00 con 0 > -4364> 2018-07-11 07:51:40.618834 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6809/12650 -- osd_ping(ping e501 stamp 2018-07-11 07:51:40.618745) v4 -- 0x55d9b3db3c00 con 0 > -4363> 2018-07-11 07:51:40.618851 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6811/12650 -- osd_ping(ping e501 stamp 2018-07-11 07:51:40.618745) v4 -- 0x55d9c908fe00 con 0 > -4362> 2018-07-11 07:51:40.618864 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6810/15287 -- osd_ping(ping e501 stamp 2018-07-11 07:51:40.618745) v4 -- 0x55d9869bfa00 con 0 > -4361> 2018-07-11 07:51:40.618875 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6811/15287 -- osd_ping(ping e501 stamp 2018-07-11 07:51:40.618745) v4 -- 0x55d986d4f400 con 0 > -4360> 2018-07-11 07:51:40.618888 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6806/15263 -- osd_ping(ping e501 stamp 2018-07-11 07:51:40.618745) v4 -- 0x55d986c84400 con 0 > -4359> 2018-07-11 07:51:40.618904 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6807/15263 -- osd_ping(ping e501 stamp 2018-07-11 07:51:40.618745) v4 -- 0x55d9c3383e00 con 0 > -4358> 2018-07-11 07:51:40.618916 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.115:6802/13323 -- osd_ping(ping e501 stamp 2018-07-11 07:51:40.618745) v4 -- 0x55d9869bf800 con 0 > -4357> 2018-07-11 07:51:40.618927 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.115:6803/13323 -- osd_ping(ping e501 stamp 2018-07-11 07:51:40.618745) v4 -- 0x55d9b67d9e00 con 0 > -4356> 2018-07-11 07:51:40.618942 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6806/15885 -- osd_ping(ping e501 stamp 2018-07-11 07:51:40.618745) v4 -- 0x55d982cdfc00 con 0 > -4355> 2018-07-11 07:51:40.618954 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6807/15885 -- osd_ping(ping e501 stamp 2018-07-11 07:51:40.618745) v4 -- 0x55d9b67d8400 con 0 > -4354> 2018-07-11 07:51:40.618967 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.108:6802/11967 -- osd_ping(ping e501 stamp 2018-07-11 07:51:40.618745) v4 -- 0x55d9b9ac2200 con 0 > -4353> 2018-07-11 07:51:40.618978 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.108:6803/11967 -- osd_ping(ping e501 stamp 2018-07-11 07:51:40.618745) v4 -- 0x55d986c22c00 con 0 > -4352> 2018-07-11 07:51:40.618991 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.107:6806/12617 -- osd_ping(ping e501 stamp 2018-07-11 07:51:40.618745) v4 -- 0x55d986d4f800 con 0 > -4351> 2018-07-11 07:51:40.619003 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.107:6807/12617 -- osd_ping(ping e501 stamp 2018-07-11 07:51:40.618745) v4 -- 0x55d9869be000 con 0 > -4350> 2018-07-11 07:51:40.619015 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6802/15919 -- osd_ping(ping e501 stamp 2018-07-11 07:51:40.618745) v4 -- 0x55d9c4c1b600 con 0 > -4349> 2018-07-11 07:51:40.619027 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6803/15919 -- osd_ping(ping e501 stamp 2018-07-11 07:51:40.618745) v4 -- 0x55d9b67d8a00 con 0 > -4348> 2018-07-11 07:51:40.619193 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.107:6807/12617 conn(0x55d986bd3800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.9 seq 244 0x55d97f5adc00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:40.618745) v4 > -4347> 2018-07-11 07:51:40.619214 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.9 10.8.128.107:6807/12617 244 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:40.618745) v4 ==== 2004+0+0 (691221057 0 0) 0x55d97f5adc00 con 0x55d986bd3800 > -4346> 2018-07-11 07:51:40.619205 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.107:6806/12617 conn(0x55d986bd2000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.9 seq 244 0x55d982cdf800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:40.618745) v4 > -4345> 2018-07-11 07:51:40.619234 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.9 10.8.128.107:6806/12617 244 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:40.618745) v4 ==== 2004+0+0 (691221057 0 0) 0x55d982cdf800 con 0x55d986bd2000 > -4344> 2018-07-11 07:51:40.619270 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6802/15255 conn(0x55d986bc8000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.0 seq 244 0x55d9b3db2200 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:40.618745) v4 > -4343> 2018-07-11 07:51:40.619294 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.0 10.8.128.113:6802/15255 244 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:40.618745) v4 ==== 2004+0+0 (691221057 0 0) 0x55d9b3db2200 con 0x55d986bc8000 > -4342> 2018-07-11 07:51:40.619306 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6809/12650 conn(0x55d986bcb000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 244 0x55d97f5adc00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:40.618745) v4 > -4341> 2018-07-11 07:51:40.619308 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6803/15255 conn(0x55d986bc9800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.0 seq 244 0x55d982cdf800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:40.618745) v4 > -4340> 2018-07-11 07:51:40.619321 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.3 10.8.128.106:6809/12650 244 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:40.618745) v4 ==== 2004+0+0 (691221057 0 0) 0x55d97f5adc00 con 0x55d986bcb000 > -4339> 2018-07-11 07:51:40.619322 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.0 10.8.128.113:6803/15255 244 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:40.618745) v4 ==== 2004+0+0 (691221057 0 0) 0x55d982cdf800 con 0x55d986bc9800 > -4338> 2018-07-11 07:51:40.619347 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6808/12651 conn(0x55d986b9a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.2 seq 244 0x55d9b3db2200 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:40.618745) v4 > -4337> 2018-07-11 07:51:40.619360 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.2 10.8.128.106:6808/12651 244 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:40.618745) v4 ==== 2004+0+0 (691221057 0 0) 0x55d9b3db2200 con 0x55d986b9a000 > -4336> 2018-07-11 07:51:40.619383 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6810/12651 conn(0x55d986b9b800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.2 seq 244 0x55d982cdf800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:40.618745) v4 > -4335> 2018-07-11 07:51:40.619384 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6806/15263 conn(0x55d986b9d000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.5 seq 244 0x55d97f5adc00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:40.618745) v4 > -4334> 2018-07-11 07:51:40.619397 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.2 10.8.128.106:6810/12651 244 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:40.618745) v4 ==== 2004+0+0 (691221057 0 0) 0x55d982cdf800 con 0x55d986b9b800 > -4333> 2018-07-11 07:51:40.619413 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.5 10.8.128.113:6806/15263 244 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:40.618745) v4 ==== 2004+0+0 (691221057 0 0) 0x55d97f5adc00 con 0x55d986b9d000 > -4332> 2018-07-11 07:51:40.619394 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6811/12650 conn(0x55d986bcc800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 244 0x55d9b3db2200 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:40.618745) v4 > -4331> 2018-07-11 07:51:40.619435 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.3 10.8.128.106:6811/12650 244 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:40.618745) v4 ==== 2004+0+0 (691221057 0 0) 0x55d9b3db2200 con 0x55d986bcc800 > -4330> 2018-07-11 07:51:40.619447 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6806/12677 conn(0x55d986b58000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.1 seq 244 0x55d97f5adc00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:40.618745) v4 > -4329> 2018-07-11 07:51:40.619455 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6811/15287 conn(0x55d986bb0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 244 0x55d982cdf800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:40.618745) v4 > -4328> 2018-07-11 07:51:40.619464 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.1 10.8.128.106:6806/12677 244 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:40.618745) v4 ==== 2004+0+0 (691221057 0 0) 0x55d97f5adc00 con 0x55d986b58000 > -4327> 2018-07-11 07:51:40.619467 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.4 10.8.128.113:6811/15287 244 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:40.618745) v4 ==== 2004+0+0 (691221057 0 0) 0x55d982cdf800 con 0x55d986bb0800 > -4326> 2018-07-11 07:51:40.619469 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6807/15263 conn(0x55d986b9e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.5 seq 244 0x55d9b3db2200 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:40.618745) v4 > -4325> 2018-07-11 07:51:40.619481 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.5 10.8.128.113:6807/15263 244 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:40.618745) v4 ==== 2004+0+0 (691221057 0 0) 0x55d9b3db2200 con 0x55d986b9e800 > -4324> 2018-07-11 07:51:40.619513 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6807/12677 conn(0x55d986b59800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.1 seq 244 0x55d9b3db2200 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:40.618745) v4 > -4323> 2018-07-11 07:51:40.619526 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.1 10.8.128.106:6807/12677 244 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:40.618745) v4 ==== 2004+0+0 (691221057 0 0) 0x55d9b3db2200 con 0x55d986b59800 > -4322> 2018-07-11 07:51:40.619525 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6807/15885 conn(0x55d986bad800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.7 seq 244 0x55d97f5adc00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:40.618745) v4 > -4321> 2018-07-11 07:51:40.619536 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.7 10.8.128.114:6807/15885 244 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:40.618745) v4 ==== 2004+0+0 (691221057 0 0) 0x55d97f5adc00 con 0x55d986bad800 > -4320> 2018-07-11 07:51:40.619543 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.115:6802/13323 conn(0x55d986bc1000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 244 0x55d982cdf800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:40.618745) v4 > -4319> 2018-07-11 07:51:40.619555 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6810/15287 conn(0x55d986baf000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 244 0x55d9b3db2200 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:40.618745) v4 > -4318> 2018-07-11 07:51:40.619564 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.4 10.8.128.113:6810/15287 244 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:40.618745) v4 ==== 2004+0+0 (691221057 0 0) 0x55d9b3db2200 con 0x55d986baf000 > -4317> 2018-07-11 07:51:40.619563 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.6 10.8.128.115:6802/13323 244 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:40.618745) v4 ==== 2004+0+0 (691221057 0 0) 0x55d982cdf800 con 0x55d986bc1000 > -4316> 2018-07-11 07:51:40.619584 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.108:6802/11967 conn(0x55d986bbe000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.8 seq 244 0x55d97f5adc00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:40.618745) v4 > -4315> 2018-07-11 07:51:40.619597 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.8 10.8.128.108:6802/11967 244 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:40.618745) v4 ==== 2004+0+0 (691221057 0 0) 0x55d97f5adc00 con 0x55d986bbe000 > -4314> 2018-07-11 07:51:40.619596 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.108:6803/11967 conn(0x55d986bbf800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.8 seq 244 0x55d9b3db2200 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:40.618745) v4 > -4313> 2018-07-11 07:51:40.619620 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.8 10.8.128.108:6803/11967 244 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:40.618745) v4 ==== 2004+0+0 (691221057 0 0) 0x55d9b3db2200 con 0x55d986bbf800 > -4312> 2018-07-11 07:51:40.619616 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6802/15919 conn(0x55d986b5b000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.11 seq 244 0x55d982cdf800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:40.618745) v4 > -4311> 2018-07-11 07:51:40.619633 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.11 10.8.128.114:6802/15919 244 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:40.618745) v4 ==== 2004+0+0 (691221057 0 0) 0x55d982cdf800 con 0x55d986b5b000 > -4310> 2018-07-11 07:51:40.619643 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6803/15919 conn(0x55d986b5c800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.11 seq 244 0x55d97f5adc00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:40.618745) v4 > -4309> 2018-07-11 07:51:40.619656 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.11 10.8.128.114:6803/15919 244 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:40.618745) v4 ==== 2004+0+0 (691221057 0 0) 0x55d97f5adc00 con 0x55d986b5c800 > -4308> 2018-07-11 07:51:40.619677 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6806/15885 conn(0x55d986bac000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.7 seq 244 0x55d982cdf800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:40.618745) v4 > -4307> 2018-07-11 07:51:40.619688 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.115:6803/13323 conn(0x55d986bc2800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 244 0x55d97f5adc00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:40.618745) v4 > -4306> 2018-07-11 07:51:40.619692 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.7 10.8.128.114:6806/15885 244 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:40.618745) v4 ==== 2004+0+0 (691221057 0 0) 0x55d982cdf800 con 0x55d986bac000 > -4305> 2018-07-11 07:51:40.619700 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.6 10.8.128.115:6803/13323 244 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:40.618745) v4 ==== 2004+0+0 (691221057 0 0) 0x55d97f5adc00 con 0x55d986bc2800 > -4304> 2018-07-11 07:51:40.621350 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67733,epoch:501/498,start:5:7559a465:::benchmark_data_magna115_11707_object404210:0,end:5:755a1b11:::benchmark_data_magna115_10867_object1674582:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498280 con 0 > -4303> 2018-07-11 07:51:40.621388 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67733,epoch:501/498,start:5:7559a465:::benchmark_data_magna115_11707_object404210:0,end:5:755a1b11:::benchmark_data_magna115_10867_object1674582:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455400 con 0 > -4302> 2018-07-11 07:51:40.825506 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 958 0x55d9bc1a8000 rep_scrubmap(5.2e e501 from shard 4) v1 > -4301> 2018-07-11 07:51:40.825527 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 958 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12472 (3686815630 0 779944749) 0x55d9bc1a8000 con 0x55d986bec000 > -4300> 2018-07-11 07:51:40.838264 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 964 0x55d9bc1a8240 rep_scrubmap(5.2e e501 from shard 2) v1 > -4299> 2018-07-11 07:51:40.838284 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 964 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12472 (1269249638 0 779944749) 0x55d9bc1a8240 con 0x55d986bf0800 > -4298> 2018-07-11 07:51:40.868226 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -4297> 2018-07-11 07:51:40.868258 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -4296> 2018-07-11 07:51:40.868318 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -4295> 2018-07-11 07:51:40.868782 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -4294> 2018-07-11 07:51:40.884860 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15287 conn(0x55d98123f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 265 0x55d982cdf800 osd_ping(ping e501 stamp 2018-07-11 07:51:42.115911) v4 > -4293> 2018-07-11 07:51:40.884882 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.4 10.8.128.113:0/15287 265 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:42.115911) v4 ==== 2004+0+0 (1251513018 0 0) 0x55d982cdf800 con 0x55d98123f000 > -4292> 2018-07-11 07:51:40.884905 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:42.115911) v4 -- 0x55d9c4c1b600 con 0 > -4291> 2018-07-11 07:51:40.884904 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15287 conn(0x55d981240800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 265 0x55d9b3db2200 osd_ping(ping e501 stamp 2018-07-11 07:51:42.115911) v4 > -4290> 2018-07-11 07:51:40.884925 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.4 10.8.128.113:0/15287 265 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:42.115911) v4 ==== 2004+0+0 (1251513018 0 0) 0x55d9b3db2200 con 0x55d981240800 > -4289> 2018-07-11 07:51:40.884936 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:42.115911) v4 -- 0x55d986c22c00 con 0 > -4288> 2018-07-11 07:51:40.911445 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67219,epoch:501/498,start:5:755a1b11:::benchmark_data_magna115_10867_object1674582:0,end:5:755aad59:::benchmark_data_magna115_10867_object3405968:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454f00 con 0 > -4287> 2018-07-11 07:51:40.911484 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67219,epoch:501/498,start:5:755a1b11:::benchmark_data_magna115_10867_object1674582:0,end:5:755aad59:::benchmark_data_magna115_10867_object3405968:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454780 con 0 > -4286> 2018-07-11 07:51:40.963234 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6801/15919 conn(0x55d986c3f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=29 cs=1 l=0). rx osd.11 seq 240 0x55d97f5adc00 MOSDScrubReserve(10.f REQUEST e501) v1 > -4285> 2018-07-11 07:51:40.963265 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.11 10.8.128.114:6801/15919 240 ==== MOSDScrubReserve(10.f REQUEST e501) v1 ==== 43+0+0 (2892966580 0 0) 0x55d97f5adc00 con 0x55d986c3f800 > -4284> 2018-07-11 07:51:40.963340 7ffab53e3700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.114:6801/15919 -- MOSDScrubReserve(10.f REJECT e501) v1 -- 0x55d9b4388200 con 0 > -4283> 2018-07-11 07:51:40.963649 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6801/15919 conn(0x55d986c3f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=29 cs=1 l=0). rx osd.11 seq 241 0x55d9b4388200 MOSDScrubReserve(10.f RELEASE e501) v1 > -4282> 2018-07-11 07:51:40.963663 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.11 10.8.128.114:6801/15919 241 ==== MOSDScrubReserve(10.f RELEASE e501) v1 ==== 43+0+0 (1668468918 0 0) 0x55d9b4388200 con 0x55d986c3f800 > -4281> 2018-07-11 07:51:41.090536 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 965 0x55d9bc1a8480 rep_scrubmap(5.2e e501 from shard 2) v1 > -4280> 2018-07-11 07:51:41.090581 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 965 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12472 (1269249638 0 88632702) 0x55d9bc1a8480 con 0x55d986bf0800 > -4279> 2018-07-11 07:51:41.093882 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.114:0/15919 conn(0x55d986df3800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.11 seq 256 0x55d9b60de600 osd_ping(ping e501 stamp 2018-07-11 07:51:41.724127) v4 > -4278> 2018-07-11 07:51:41.093909 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.11 10.8.128.114:0/15919 256 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:41.724127) v4 ==== 2004+0+0 (1342461633 0 0) 0x55d9b60de600 con 0x55d986df3800 > -4277> 2018-07-11 07:51:41.093921 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:41.724127) v4 -- 0x55d9869be000 con 0 > -4276> 2018-07-11 07:51:41.093908 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.114:0/15919 conn(0x55d986df5000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.11 seq 256 0x55d9c3383c00 osd_ping(ping e501 stamp 2018-07-11 07:51:41.724127) v4 > -4275> 2018-07-11 07:51:41.093932 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.11 10.8.128.114:0/15919 256 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:41.724127) v4 ==== 2004+0+0 (1342461633 0 0) 0x55d9c3383c00 con 0x55d986df5000 > -4274> 2018-07-11 07:51:41.093949 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:41.724127) v4 -- 0x55d9c4c1b600 con 0 > -4273> 2018-07-11 07:51:41.121908 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 959 0x55d9bc1a86c0 rep_scrubmap(5.2e e501 from shard 4) v1 > -4272> 2018-07-11 07:51:41.121927 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 959 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12472 (3686815630 0 88632702) 0x55d9bc1a86c0 con 0x55d986bec000 > -4271> 2018-07-11 07:51:41.302426 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.115:0/13323 conn(0x55d986d7e000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 253 0x55d9b67d8a00 osd_ping(ping e501 stamp 2018-07-11 07:51:41.648183) v4 > -4270> 2018-07-11 07:51:41.302446 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.6 10.8.128.115:0/13323 253 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:41.648183) v4 ==== 2004+0+0 (3522718348 0 0) 0x55d9b67d8a00 con 0x55d986d7e000 > -4269> 2018-07-11 07:51:41.302456 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:41.648183) v4 -- 0x55d9869be000 con 0 > -4268> 2018-07-11 07:51:41.302514 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.115:0/13323 conn(0x55d986d24800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 253 0x55d9b3db2200 osd_ping(ping e501 stamp 2018-07-11 07:51:41.648183) v4 > -4267> 2018-07-11 07:51:41.302546 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.6 10.8.128.115:0/13323 253 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:41.648183) v4 ==== 2004+0+0 (3522718348 0 0) 0x55d9b3db2200 con 0x55d986d24800 > -4266> 2018-07-11 07:51:41.302557 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:41.648183) v4 -- 0x55d986c22c00 con 0 > -4265> 2018-07-11 07:51:41.465488 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12677 conn(0x55d986db4800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.1 seq 257 0x55d982cdf800 osd_ping(ping e501 stamp 2018-07-11 07:51:41.513201) v4 > -4264> 2018-07-11 07:51:41.465496 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12677 conn(0x55d986df2000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.1 seq 257 0x55d9b67d9a00 osd_ping(ping e501 stamp 2018-07-11 07:51:41.513201) v4 > -4263> 2018-07-11 07:51:41.465514 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.1 10.8.128.106:0/12677 257 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:41.513201) v4 ==== 2004+0+0 (4211148813 0 0) 0x55d982cdf800 con 0x55d986db4800 > -4262> 2018-07-11 07:51:41.465516 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.1 10.8.128.106:0/12677 257 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:41.513201) v4 ==== 2004+0+0 (4211148813 0 0) 0x55d9b67d9a00 con 0x55d986df2000 > -4261> 2018-07-11 07:51:41.465523 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:41.513201) v4 -- 0x55d9c4c1b600 con 0 > -4260> 2018-07-11 07:51:41.465548 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:41.513201) v4 -- 0x55d986c22c00 con 0 > -4259> 2018-07-11 07:51:41.565154 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6805/15885 conn(0x55d986c7f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.7 seq 2804 0x55d9b3db2200 MOSDScrubReserve(5.4 REQUEST e501) v1 > -4258> 2018-07-11 07:51:41.565171 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.7 10.8.128.114:6805/15885 2804 ==== MOSDScrubReserve(5.4 REQUEST e501) v1 ==== 43+0+0 (1753221512 0 0) 0x55d9b3db2200 con 0x55d986c7f800 > -4257> 2018-07-11 07:51:41.565294 7ffab33df700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.114:6805/15885 -- MOSDScrubReserve(5.4 REJECT e501) v1 -- 0x55d986cd5200 con 0 > -4256> 2018-07-11 07:51:41.565529 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6805/15885 conn(0x55d986c7f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.7 seq 2805 0x55d986c22c00 MOSDScrubReserve(5.4 RELEASE e501) v1 > -4255> 2018-07-11 07:51:41.565544 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.7 10.8.128.114:6805/15885 2805 ==== MOSDScrubReserve(5.4 RELEASE e501) v1 ==== 43+0+0 (2812146570 0 0) 0x55d986c22c00 con 0x55d986c7f800 > -4254> 2018-07-11 07:51:41.716548 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12650 conn(0x55d986d21800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 245 0x55d9b60de600 osd_ping(ping e501 stamp 2018-07-11 07:51:41.764243) v4 > -4253> 2018-07-11 07:51:41.716569 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.3 10.8.128.106:0/12650 245 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:41.764243) v4 ==== 2004+0+0 (1297745320 0 0) 0x55d9b60de600 con 0x55d986d21800 > -4252> 2018-07-11 07:51:41.716578 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:41.764243) v4 -- 0x55d9869be000 con 0 > -4251> 2018-07-11 07:51:41.716572 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12650 conn(0x55d986d23000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 245 0x55d9c3383c00 osd_ping(ping e501 stamp 2018-07-11 07:51:41.764243) v4 > -4250> 2018-07-11 07:51:41.716597 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.3 10.8.128.106:0/12650 245 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:41.764243) v4 ==== 2004+0+0 (1297745320 0 0) 0x55d9c3383c00 con 0x55d986d23000 > -4249> 2018-07-11 07:51:41.716609 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:41.764243) v4 -- 0x55d9c4c1b600 con 0 > -4248> 2018-07-11 07:51:41.905466 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -4247> 2018-07-11 07:51:41.905495 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -4246> 2018-07-11 07:51:41.905552 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -4245> 2018-07-11 07:51:41.906010 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -4244> 2018-07-11 07:51:41.938281 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67502,epoch:501/498,start:5:755aad59:::benchmark_data_magna115_10867_object3405968:0,end:5:755b0c25:::benchmark_data_magna115_11707_object223120:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455180 con 0 > -4243> 2018-07-11 07:51:41.938330 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67502,epoch:501/498,start:5:755aad59:::benchmark_data_magna115_10867_object3405968:0,end:5:755b0c25:::benchmark_data_magna115_11707_object223120:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454500 con 0 > -4242> 2018-07-11 07:51:42.048127 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 32] Generated table #36805: 4509432 keys, 79228604 bytes > -4241> 2018-07-11 07:51:42.048180 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295502048152, "cf_name": "default", "job": 32, "event": "table_file_creation", "file_number": 36805, "file_size": 79228604, "table_properties": {"data_size": 67110483, "index_size": 689344, "filter_size": 11427784, "raw_key_size": 130272483, "raw_average_key_size": 28, "raw_value_size": 6056579, "raw_average_value_size": 1, "num_data_blocks": 16474, "num_entries": 4509432, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4000000", "kMergeOperands": "0"}} > -4240> 2018-07-11 07:51:42.067500 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15255 conn(0x55d986d20000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.0 seq 253 0x55d986cd5200 osd_ping(ping e501 stamp 2018-07-11 07:51:43.298627) v4 > -4239> 2018-07-11 07:51:42.067522 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.0 10.8.128.113:0/15255 253 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:43.298627) v4 ==== 2004+0+0 (4030509970 0 0) 0x55d986cd5200 con 0x55d986d20000 > -4238> 2018-07-11 07:51:42.067516 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15255 conn(0x55d986c82800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.0 seq 253 0x55d982cdf800 osd_ping(ping e501 stamp 2018-07-11 07:51:43.298627) v4 > -4237> 2018-07-11 07:51:42.067538 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:43.298627) v4 -- 0x55d9c3383e00 con 0 > -4236> 2018-07-11 07:51:42.067537 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.0 10.8.128.113:0/15255 253 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:43.298627) v4 ==== 2004+0+0 (4030509970 0 0) 0x55d982cdf800 con 0x55d986c82800 > -4235> 2018-07-11 07:51:42.067569 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:43.298627) v4 -- 0x55d9c4c1b600 con 0 > -4234> 2018-07-11 07:51:42.158637 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 966 0x55d9bc1a8900 rep_scrubmap(5.2e e501 from shard 2) v1 > -4233> 2018-07-11 07:51:42.158659 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 966 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12480 (1269249638 0 187782605) 0x55d9bc1a8900 con 0x55d986bf0800 > -4232> 2018-07-11 07:51:42.187409 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 960 0x55d9bc1a8b40 rep_scrubmap(5.2e e501 from shard 4) v1 > -4231> 2018-07-11 07:51:42.187430 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 960 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12480 (3686815630 0 187782605) 0x55d9bc1a8b40 con 0x55d986bec000 > -4230> 2018-07-11 07:51:42.301993 7ffab7be8700 10 monclient: tick > -4229> 2018-07-11 07:51:42.302007 7ffab7be8700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2018-07-11 07:51:12.302005) > -4228> 2018-07-11 07:51:42.350956 7ffab6be6700 1 -- 10.8.128.107:6800/13927 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55d9ba81a840 con 0 > -4227> 2018-07-11 07:51:42.351021 7ffab6be6700 1 -- 10.8.128.107:6800/13927 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55d9b3fdf080 con 0 > -4226> 2018-07-11 07:51:42.360447 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -4225> 2018-07-11 07:51:42.360474 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -4224> 2018-07-11 07:51:42.360534 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -4223> 2018-07-11 07:51:42.360995 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -4222> 2018-07-11 07:51:42.405043 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67816,epoch:501/498,start:5:755b0c25:::benchmark_data_magna115_11707_object223120:0,end:5:755b82d7:::benchmark_data_magna115_10867_object393323:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98944b680 con 0 > -4221> 2018-07-11 07:51:42.405083 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67816,epoch:501/498,start:5:755b0c25:::benchmark_data_magna115_11707_object223120:0,end:5:755b82d7:::benchmark_data_magna115_10867_object393323:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454a00 con 0 > -4220> 2018-07-11 07:51:42.565429 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6805/15885 conn(0x55d986c7f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.7 seq 2806 0x55d9b67d9a00 MOSDScrubReserve(5.4 REQUEST e501) v1 > -4219> 2018-07-11 07:51:42.565448 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.7 10.8.128.114:6805/15885 2806 ==== MOSDScrubReserve(5.4 REQUEST e501) v1 ==== 43+0+0 (1753221512 0 0) 0x55d9b67d9a00 con 0x55d986c7f800 > -4218> 2018-07-11 07:51:42.565572 7ffab33df700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.114:6805/15885 -- MOSDScrubReserve(5.4 REJECT e501) v1 -- 0x55d986c22c00 con 0 > -4217> 2018-07-11 07:51:42.565908 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6805/15885 conn(0x55d986c7f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.7 seq 2807 0x55d986c22c00 MOSDScrubReserve(5.4 RELEASE e501) v1 > -4216> 2018-07-11 07:51:42.565922 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.7 10.8.128.114:6805/15885 2807 ==== MOSDScrubReserve(5.4 RELEASE e501) v1 ==== 43+0+0 (2812146570 0 0) 0x55d986c22c00 con 0x55d986c7f800 > -4215> 2018-07-11 07:51:42.589595 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.108:0/11967 conn(0x55d986db0000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.8 seq 235 0x55d982cdf800 osd_ping(ping e501 stamp 2018-07-11 07:51:42.345346) v4 > -4214> 2018-07-11 07:51:42.589642 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.8 10.8.128.108:0/11967 235 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:42.345346) v4 ==== 2004+0+0 (523968779 0 0) 0x55d982cdf800 con 0x55d986db0000 > -4213> 2018-07-11 07:51:42.589659 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:42.345346) v4 -- 0x55d9c4c1b600 con 0 > -4212> 2018-07-11 07:51:42.589716 7ffacbbd1700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.108:0/11967 conn(0x55d986d82800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.8 seq 235 0x55d9b67d8a00 osd_ping(ping e501 stamp 2018-07-11 07:51:42.345346) v4 > -4211> 2018-07-11 07:51:42.589738 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 <== osd.8 10.8.128.108:0/11967 235 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:42.345346) v4 ==== 2004+0+0 (523968779 0 0) 0x55d9b67d8a00 con 0x55d986d82800 > -4210> 2018-07-11 07:51:42.589750 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:42.345346) v4 -- 0x55d9869be000 con 0 > -4209> 2018-07-11 07:51:42.597982 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 967 0x55d9bc1a8d80 rep_scrubmap(5.2e e501 from shard 2) v1 > -4208> 2018-07-11 07:51:42.598001 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 967 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12470 (1269249638 0 600833191) 0x55d9bc1a8d80 con 0x55d986bf0800 > -4207> 2018-07-11 07:51:42.650568 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 961 0x55d9bc1a8fc0 rep_scrubmap(5.2e e501 from shard 4) v1 > -4206> 2018-07-11 07:51:42.650589 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 961 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12470 (3686815630 0 600833191) 0x55d9bc1a8fc0 con 0x55d986bec000 > -4205> 2018-07-11 07:51:42.723954 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -4204> 2018-07-11 07:51:42.723979 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -4203> 2018-07-11 07:51:42.724040 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -4202> 2018-07-11 07:51:42.724544 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -4201> 2018-07-11 07:51:42.771670 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67520,epoch:501/498,start:5:755b82d7:::benchmark_data_magna115_10867_object393323:0,end:5:755bf187:::benchmark_data_magna115_10867_object2582981:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98944b900 con 0 > -4200> 2018-07-11 07:51:42.771720 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67520,epoch:501/498,start:5:755b82d7:::benchmark_data_magna115_10867_object393323:0,end:5:755bf187:::benchmark_data_magna115_10867_object2582981:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98944bb80 con 0 > -4199> 2018-07-11 07:51:42.839889 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 3006 0x55d9b60de600 MOSDScrubReserve(11.3s5 REQUEST e501) v1 > -4198> 2018-07-11 07:51:42.839906 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 3006 ==== MOSDScrubReserve(11.3s5 REQUEST e501) v1 ==== 43+0+0 (3260170403 0 0) 0x55d9b60de600 con 0x55d986c7e000 > -4197> 2018-07-11 07:51:42.840025 7ffab3be0700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e501) v1 -- 0x55d986d97e00 con 0 > -4196> 2018-07-11 07:51:42.840399 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 3007 0x55d9869be000 MOSDScrubReserve(11.3s5 RELEASE e501) v1 > -4195> 2018-07-11 07:51:42.840427 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 3007 ==== MOSDScrubReserve(11.3s5 RELEASE e501) v1 ==== 43+0+0 (223328929 0 0) 0x55d9869be000 con 0x55d986c7e000 > -4194> 2018-07-11 07:51:42.991785 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 962 0x55d9bc1a9200 rep_scrubmap(5.2e e501 from shard 4) v1 > -4193> 2018-07-11 07:51:42.991806 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 962 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12478 (3686815630 0 705503564) 0x55d9bc1a9200 con 0x55d986bec000 > -4192> 2018-07-11 07:51:42.992611 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 968 0x55d9bc1a9440 rep_scrubmap(5.2e e501 from shard 2) v1 > -4191> 2018-07-11 07:51:42.992631 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 968 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12478 (1269249638 0 705503564) 0x55d9bc1a9440 con 0x55d986bf0800 > -4190> 2018-07-11 07:51:43.024174 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -4189> 2018-07-11 07:51:43.024198 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -4188> 2018-07-11 07:51:43.024265 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -4187> 2018-07-11 07:51:43.024719 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -4186> 2018-07-11 07:51:43.063787 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:755bf187:::benchmark_data_magna115_10867_object2582981:0,end:5:755c6bc0:::benchmark_data_magna115_11707_object67783:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb772280 con 0 > -4185> 2018-07-11 07:51:43.063825 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:755bf187:::benchmark_data_magna115_10867_object2582981:0,end:5:755c6bc0:::benchmark_data_magna115_11707_object67783:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98944b400 con 0 > -4184> 2018-07-11 07:51:43.270751 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 969 0x55d9bc1a9680 rep_scrubmap(5.2e e501 from shard 2) v1 > -4183> 2018-07-11 07:51:43.270769 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 969 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12464 (1269249638 0 3194026828) 0x55d9bc1a9680 con 0x55d986bf0800 > -4182> 2018-07-11 07:51:43.274862 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 963 0x55d9bc1a98c0 rep_scrubmap(5.2e e501 from shard 4) v1 > -4181> 2018-07-11 07:51:43.274881 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 963 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12464 (3686815630 0 3194026828) 0x55d9bc1a98c0 con 0x55d986bec000 > -4180> 2018-07-11 07:51:43.275032 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -4179> 2018-07-11 07:51:43.275059 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -4178> 2018-07-11 07:51:43.275105 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -4177> 2018-07-11 07:51:43.275583 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -4176> 2018-07-11 07:51:43.313612 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:755c6bc0:::benchmark_data_magna115_11707_object67783:0,end:5:755cc8d0:::benchmark_data_magna115_11707_object404078:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb772000 con 0 > -4175> 2018-07-11 07:51:43.313649 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:755c6bc0:::benchmark_data_magna115_11707_object67783:0,end:5:755cc8d0:::benchmark_data_magna115_11707_object404078:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb772500 con 0 > -4174> 2018-07-11 07:51:43.450198 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.107:0/12617 conn(0x55d98123c000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 244 0x55d9c3383e00 osd_ping(ping e501 stamp 2018-07-11 07:51:43.449749) v4 > -4173> 2018-07-11 07:51:43.450219 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.107:0/12617 conn(0x55d98123d800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 244 0x55d9b67d8a00 osd_ping(ping e501 stamp 2018-07-11 07:51:43.449749) v4 > -4172> 2018-07-11 07:51:43.450230 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.9 10.8.128.107:0/12617 244 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:43.449749) v4 ==== 2004+0+0 (2312523265 0 0) 0x55d9c3383e00 con 0x55d98123c000 > -4171> 2018-07-11 07:51:43.450240 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.9 10.8.128.107:0/12617 244 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:43.449749) v4 ==== 2004+0+0 (2312523265 0 0) 0x55d9b67d8a00 con 0x55d98123d800 > -4170> 2018-07-11 07:51:43.450261 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:43.449749) v4 -- 0x55d9869bfa00 con 0 > -4169> 2018-07-11 07:51:43.450296 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:43.449749) v4 -- 0x55d9b9ac2200 con 0 > -4168> 2018-07-11 07:51:43.523064 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 970 0x55d9bc1a9b00 rep_scrubmap(5.2e e501 from shard 2) v1 > -4167> 2018-07-11 07:51:43.523084 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 970 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12470 (1269249638 0 19282331) 0x55d9bc1a9b00 con 0x55d986bf0800 > -4166> 2018-07-11 07:51:43.524964 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12651 conn(0x55d986db3000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.2 seq 254 0x55d986d97e00 osd_ping(ping e501 stamp 2018-07-11 07:51:43.572731) v4 > -4165> 2018-07-11 07:51:43.524987 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.2 10.8.128.106:0/12651 254 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:43.572731) v4 ==== 2004+0+0 (563745740 0 0) 0x55d986d97e00 con 0x55d986db3000 > -4164> 2018-07-11 07:51:43.524996 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:43.572731) v4 -- 0x55d9b9ac2200 con 0 > -4163> 2018-07-11 07:51:43.525066 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12651 conn(0x55d986db1800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.2 seq 254 0x55d9c3383e00 osd_ping(ping e501 stamp 2018-07-11 07:51:43.572731) v4 > -4162> 2018-07-11 07:51:43.525090 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.2 10.8.128.106:0/12651 254 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:43.572731) v4 ==== 2004+0+0 (563745740 0 0) 0x55d9c3383e00 con 0x55d986db1800 > -4161> 2018-07-11 07:51:43.525106 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:43.572731) v4 -- 0x55d9869bfa00 con 0 > -4160> 2018-07-11 07:51:43.565588 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6805/15885 conn(0x55d986c7f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.7 seq 2808 0x55d986cd5200 MOSDScrubReserve(5.4 REQUEST e501) v1 > -4159> 2018-07-11 07:51:43.565605 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.7 10.8.128.114:6805/15885 2808 ==== MOSDScrubReserve(5.4 REQUEST e501) v1 ==== 43+0+0 (1753221512 0 0) 0x55d986cd5200 con 0x55d986c7f800 > -4158> 2018-07-11 07:51:43.565726 7ffab33df700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.114:6805/15885 -- MOSDScrubReserve(5.4 REJECT e501) v1 -- 0x55d986c22c00 con 0 > -4157> 2018-07-11 07:51:43.565962 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6805/15885 conn(0x55d986c7f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.7 seq 2809 0x55d9869bfa00 MOSDScrubReserve(5.4 RELEASE e501) v1 > -4156> 2018-07-11 07:51:43.565976 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.7 10.8.128.114:6805/15885 2809 ==== MOSDScrubReserve(5.4 RELEASE e501) v1 ==== 43+0+0 (2812146570 0 0) 0x55d9869bfa00 con 0x55d986c7f800 > -4155> 2018-07-11 07:51:43.569471 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 964 0x55d9bc1a9d40 rep_scrubmap(5.2e e501 from shard 4) v1 > -4154> 2018-07-11 07:51:43.569493 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 964 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12470 (3686815630 0 19282331) 0x55d9bc1a9d40 con 0x55d986bec000 > -4153> 2018-07-11 07:51:43.569637 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -4152> 2018-07-11 07:51:43.569663 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -4151> 2018-07-11 07:51:43.569711 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -4150> 2018-07-11 07:51:43.570175 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -4149> 2018-07-11 07:51:43.571494 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:755cc8d0:::benchmark_data_magna115_11707_object404078:0,end:5:755d2ceb:::benchmark_data_magna115_10867_object1394296:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb772a00 con 0 > -4148> 2018-07-11 07:51:43.571540 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:755cc8d0:::benchmark_data_magna115_11707_object404078:0,end:5:755d2ceb:::benchmark_data_magna115_10867_object1394296:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb772c80 con 0 > -4147> 2018-07-11 07:51:43.785342 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15287 conn(0x55d981240800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 266 0x55d9c3383e00 osd_ping(ping e501 stamp 2018-07-11 07:51:45.016456) v4 > -4146> 2018-07-11 07:51:43.785368 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.4 10.8.128.113:0/15287 266 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:45.016456) v4 ==== 2004+0+0 (3712370265 0 0) 0x55d9c3383e00 con 0x55d981240800 > -4145> 2018-07-11 07:51:43.785378 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:45.016456) v4 -- 0x55d9c908fe00 con 0 > -4144> 2018-07-11 07:51:43.785393 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15287 conn(0x55d98123f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 266 0x55d9c3383c00 osd_ping(ping e501 stamp 2018-07-11 07:51:45.016456) v4 > -4143> 2018-07-11 07:51:43.785420 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.4 10.8.128.113:0/15287 266 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:45.016456) v4 ==== 2004+0+0 (3712370265 0 0) 0x55d9c3383c00 con 0x55d98123f000 > -4142> 2018-07-11 07:51:43.785441 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:45.016456) v4 -- 0x55d9c4c1b600 con 0 > -4141> 2018-07-11 07:51:43.785836 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 971 0x55d9b2710000 rep_scrubmap(5.2e e501 from shard 2) v1 > -4140> 2018-07-11 07:51:43.785856 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 971 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12464 (1269249638 0 3808660276) 0x55d9b2710000 con 0x55d986bf0800 > -4139> 2018-07-11 07:51:43.791022 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 965 0x55d9b2710240 rep_scrubmap(5.2e e501 from shard 4) v1 > -4138> 2018-07-11 07:51:43.791041 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 965 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12464 (3686815630 0 3808660276) 0x55d9b2710240 con 0x55d986bec000 > -4137> 2018-07-11 07:51:43.840147 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 3008 0x55d9b67d8a00 MOSDScrubReserve(11.3s5 REQUEST e501) v1 > -4136> 2018-07-11 07:51:43.840167 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 3008 ==== MOSDScrubReserve(11.3s5 REQUEST e501) v1 ==== 43+0+0 (3260170403 0 0) 0x55d9b67d8a00 con 0x55d986c7e000 > -4135> 2018-07-11 07:51:43.840201 7ffab3be0700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e501) v1 -- 0x55d9869be000 con 0 > -4134> 2018-07-11 07:51:43.840591 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 3009 0x55d9869be000 MOSDScrubReserve(11.3s5 RELEASE e501) v1 > -4133> 2018-07-11 07:51:43.840605 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 3009 ==== MOSDScrubReserve(11.3s5 RELEASE e501) v1 ==== 43+0+0 (223328929 0 0) 0x55d9869be000 con 0x55d986c7e000 > -4132> 2018-07-11 07:51:43.890771 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -4131> 2018-07-11 07:51:43.890795 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -4130> 2018-07-11 07:51:43.890851 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -4129> 2018-07-11 07:51:43.891324 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -4128> 2018-07-11 07:51:43.939202 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:755d2ceb:::benchmark_data_magna115_10867_object1394296:0,end:5:755d85df:::benchmark_data_magna115_10867_object164731:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb772780 con 0 > -4127> 2018-07-11 07:51:43.939260 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:755d2ceb:::benchmark_data_magna115_10867_object1394296:0,end:5:755d85df:::benchmark_data_magna115_10867_object164731:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb772f00 con 0 > -4126> 2018-07-11 07:51:44.017058 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12650 conn(0x55d986d23000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 246 0x55d982cdf800 osd_ping(ping e501 stamp 2018-07-11 07:51:44.064792) v4 > -4125> 2018-07-11 07:51:44.017060 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12650 conn(0x55d986d21800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 246 0x55d986d97e00 osd_ping(ping e501 stamp 2018-07-11 07:51:44.064792) v4 > -4124> 2018-07-11 07:51:44.017082 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.3 10.8.128.106:0/12650 246 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:44.064792) v4 ==== 2004+0+0 (3256666139 0 0) 0x55d982cdf800 con 0x55d986d23000 > -4123> 2018-07-11 07:51:44.017087 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.3 10.8.128.106:0/12650 246 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:44.064792) v4 ==== 2004+0+0 (3256666139 0 0) 0x55d986d97e00 con 0x55d986d21800 > -4122> 2018-07-11 07:51:44.017093 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:44.064792) v4 -- 0x55d9c4c1b600 con 0 > -4121> 2018-07-11 07:51:44.017114 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:44.064792) v4 -- 0x55d9b67d8400 con 0 > -4120> 2018-07-11 07:51:44.119228 7ffab1bdc700 5 osd.10 501 heartbeat: osd_stat(788 GB used, 142 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist []) > -4119> 2018-07-11 07:51:44.119241 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6802/15255 -- osd_ping(ping e501 stamp 2018-07-11 07:51:44.119239) v4 -- 0x55d986c22000 con 0 > -4118> 2018-07-11 07:51:44.119288 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6803/15255 -- osd_ping(ping e501 stamp 2018-07-11 07:51:44.119239) v4 -- 0x55d986d4f400 con 0 > -4117> 2018-07-11 07:51:44.119303 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6806/12677 -- osd_ping(ping e501 stamp 2018-07-11 07:51:44.119239) v4 -- 0x55d9c9122a00 con 0 > -4116> 2018-07-11 07:51:44.119329 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6807/12677 -- osd_ping(ping e501 stamp 2018-07-11 07:51:44.119239) v4 -- 0x55d9c74e0200 con 0 > -4115> 2018-07-11 07:51:44.119340 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6808/12651 -- osd_ping(ping e501 stamp 2018-07-11 07:51:44.119239) v4 -- 0x55d9b60dea00 con 0 > -4114> 2018-07-11 07:51:44.119351 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6810/12651 -- osd_ping(ping e501 stamp 2018-07-11 07:51:44.119239) v4 -- 0x55d9b2783e00 con 0 > -4113> 2018-07-11 07:51:44.119364 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6809/12650 -- osd_ping(ping e501 stamp 2018-07-11 07:51:44.119239) v4 -- 0x55d986c23e00 con 0 > -4112> 2018-07-11 07:51:44.119376 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6811/12650 -- osd_ping(ping e501 stamp 2018-07-11 07:51:44.119239) v4 -- 0x55d986c22600 con 0 > -4111> 2018-07-11 07:51:44.119396 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6810/15287 -- osd_ping(ping e501 stamp 2018-07-11 07:51:44.119239) v4 -- 0x55d9b9ac3c00 con 0 > -4110> 2018-07-11 07:51:44.119408 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6811/15287 -- osd_ping(ping e501 stamp 2018-07-11 07:51:44.119239) v4 -- 0x55d9869bf200 con 0 > -4109> 2018-07-11 07:51:44.119421 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6806/15263 -- osd_ping(ping e501 stamp 2018-07-11 07:51:44.119239) v4 -- 0x55d986d96600 con 0 > -4108> 2018-07-11 07:51:44.119434 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6807/15263 -- osd_ping(ping e501 stamp 2018-07-11 07:51:44.119239) v4 -- 0x55d9c908ec00 con 0 > -4107> 2018-07-11 07:51:44.119450 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.115:6802/13323 -- osd_ping(ping e501 stamp 2018-07-11 07:51:44.119239) v4 -- 0x55d9b9ac2c00 con 0 > -4106> 2018-07-11 07:51:44.119463 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.115:6803/13323 -- osd_ping(ping e501 stamp 2018-07-11 07:51:44.119239) v4 -- 0x55d9bdba4c00 con 0 > -4105> 2018-07-11 07:51:44.119476 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6806/15885 -- osd_ping(ping e501 stamp 2018-07-11 07:51:44.119239) v4 -- 0x55d9b437b200 con 0 > -4104> 2018-07-11 07:51:44.119488 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6807/15885 -- osd_ping(ping e501 stamp 2018-07-11 07:51:44.119239) v4 -- 0x55d982cdf600 con 0 > -4103> 2018-07-11 07:51:44.119502 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.108:6802/11967 -- osd_ping(ping e501 stamp 2018-07-11 07:51:44.119239) v4 -- 0x55d980f5b200 con 0 > -4102> 2018-07-11 07:51:44.119515 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.108:6803/11967 -- osd_ping(ping e501 stamp 2018-07-11 07:51:44.119239) v4 -- 0x55d9ba818800 con 0 > -4101> 2018-07-11 07:51:44.119528 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.107:6806/12617 -- osd_ping(ping e501 stamp 2018-07-11 07:51:44.119239) v4 -- 0x55d9bdba4e00 con 0 > -4100> 2018-07-11 07:51:44.119540 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.107:6807/12617 -- osd_ping(ping e501 stamp 2018-07-11 07:51:44.119239) v4 -- 0x55d986d97400 con 0 > -4099> 2018-07-11 07:51:44.119553 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6802/15919 -- osd_ping(ping e501 stamp 2018-07-11 07:51:44.119239) v4 -- 0x55d9b67d9c00 con 0 > -4098> 2018-07-11 07:51:44.119565 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6803/15919 -- osd_ping(ping e501 stamp 2018-07-11 07:51:44.119239) v4 -- 0x55d985074400 con 0 > -4097> 2018-07-11 07:51:44.119735 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.107:6807/12617 conn(0x55d986bd3800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.9 seq 245 0x55d9b9ac2200 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:44.119239) v4 > -4096> 2018-07-11 07:51:44.119762 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.9 10.8.128.107:6807/12617 245 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:44.119239) v4 ==== 2004+0+0 (2097268797 0 0) 0x55d9b9ac2200 con 0x55d986bd3800 > -4095> 2018-07-11 07:51:44.119768 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.107:6806/12617 conn(0x55d986bd2000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.9 seq 245 0x55d982cdf800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:44.119239) v4 > -4094> 2018-07-11 07:51:44.119789 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.9 10.8.128.107:6806/12617 245 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:44.119239) v4 ==== 2004+0+0 (2097268797 0 0) 0x55d982cdf800 con 0x55d986bd2000 > -4093> 2018-07-11 07:51:44.119825 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6802/15255 conn(0x55d986bc8000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.0 seq 245 0x55d9c3383e00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:44.119239) v4 > -4092> 2018-07-11 07:51:44.119829 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6803/15255 conn(0x55d986bc9800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.0 seq 245 0x55d982cdf800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:44.119239) v4 > -4091> 2018-07-11 07:51:44.119844 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.0 10.8.128.113:6802/15255 245 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:44.119239) v4 ==== 2004+0+0 (2097268797 0 0) 0x55d9c3383e00 con 0x55d986bc8000 > -4090> 2018-07-11 07:51:44.119847 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.0 10.8.128.113:6803/15255 245 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:44.119239) v4 ==== 2004+0+0 (2097268797 0 0) 0x55d982cdf800 con 0x55d986bc9800 > -4089> 2018-07-11 07:51:44.119875 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6809/12650 conn(0x55d986bcb000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 245 0x55d9b9ac2200 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:44.119239) v4 > -4088> 2018-07-11 07:51:44.119896 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.3 10.8.128.106:6809/12650 245 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:44.119239) v4 ==== 2004+0+0 (2097268797 0 0) 0x55d9b9ac2200 con 0x55d986bcb000 > -4087> 2018-07-11 07:51:44.119895 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6810/15287 conn(0x55d986baf000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 245 0x55d9c3383e00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:44.119239) v4 > -4086> 2018-07-11 07:51:44.119896 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6810/12651 conn(0x55d986b9b800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.2 seq 245 0x55d982cdf800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:44.119239) v4 > -4085> 2018-07-11 07:51:44.119907 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.4 10.8.128.113:6810/15287 245 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:44.119239) v4 ==== 2004+0+0 (2097268797 0 0) 0x55d9c3383e00 con 0x55d986baf000 > -4084> 2018-07-11 07:51:44.119907 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.2 10.8.128.106:6810/12651 245 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:44.119239) v4 ==== 2004+0+0 (2097268797 0 0) 0x55d982cdf800 con 0x55d986b9b800 > -4083> 2018-07-11 07:51:44.119939 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6806/15263 conn(0x55d986b9d000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.5 seq 245 0x55d9b9ac2200 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:44.119239) v4 > -4082> 2018-07-11 07:51:44.119954 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.5 10.8.128.113:6806/15263 245 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:44.119239) v4 ==== 2004+0+0 (2097268797 0 0) 0x55d9b9ac2200 con 0x55d986b9d000 > -4081> 2018-07-11 07:51:44.119954 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6807/15263 conn(0x55d986b9e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.5 seq 245 0x55d9c3383e00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:44.119239) v4 > -4080> 2018-07-11 07:51:44.119965 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.5 10.8.128.113:6807/15263 245 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:44.119239) v4 ==== 2004+0+0 (2097268797 0 0) 0x55d9c3383e00 con 0x55d986b9e800 > -4079> 2018-07-11 07:51:44.119995 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6808/12651 conn(0x55d986b9a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.2 seq 245 0x55d9c3383e00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:44.119239) v4 > -4078> 2018-07-11 07:51:44.120007 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.2 10.8.128.106:6808/12651 245 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:44.119239) v4 ==== 2004+0+0 (2097268797 0 0) 0x55d9c3383e00 con 0x55d986b9a000 > -4077> 2018-07-11 07:51:44.120016 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6806/12677 conn(0x55d986b58000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.1 seq 245 0x55d9b9ac2200 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:44.119239) v4 > -4076> 2018-07-11 07:51:44.120030 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.1 10.8.128.106:6806/12677 245 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:44.119239) v4 ==== 2004+0+0 (2097268797 0 0) 0x55d9b9ac2200 con 0x55d986b58000 > -4075> 2018-07-11 07:51:44.120027 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6807/12677 conn(0x55d986b59800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.1 seq 245 0x55d9c3383e00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:44.119239) v4 > -4074> 2018-07-11 07:51:44.120059 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.1 10.8.128.106:6807/12677 245 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:44.119239) v4 ==== 2004+0+0 (2097268797 0 0) 0x55d9c3383e00 con 0x55d986b59800 > -4073> 2018-07-11 07:51:44.120053 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6811/15287 conn(0x55d986bb0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 245 0x55d982cdf800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:44.119239) v4 > -4072> 2018-07-11 07:51:44.120075 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6807/15885 conn(0x55d986bad800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.7 seq 245 0x55d9b9ac2200 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:44.119239) v4 > -4071> 2018-07-11 07:51:44.120091 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6811/12650 conn(0x55d986bcc800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 245 0x55d9c3383e00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:44.119239) v4 > -4070> 2018-07-11 07:51:44.120100 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.7 10.8.128.114:6807/15885 245 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:44.119239) v4 ==== 2004+0+0 (2097268797 0 0) 0x55d9b9ac2200 con 0x55d986bad800 > -4069> 2018-07-11 07:51:44.120095 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.4 10.8.128.113:6811/15287 245 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:44.119239) v4 ==== 2004+0+0 (2097268797 0 0) 0x55d982cdf800 con 0x55d986bb0800 > -4068> 2018-07-11 07:51:44.120101 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.3 10.8.128.106:6811/12650 245 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:44.119239) v4 ==== 2004+0+0 (2097268797 0 0) 0x55d9c3383e00 con 0x55d986bcc800 > -4067> 2018-07-11 07:51:44.120137 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.115:6803/13323 conn(0x55d986bc2800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 245 0x55d9b9ac2200 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:44.119239) v4 > -4066> 2018-07-11 07:51:44.120150 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.6 10.8.128.115:6803/13323 245 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:44.119239) v4 ==== 2004+0+0 (2097268797 0 0) 0x55d9b9ac2200 con 0x55d986bc2800 > -4065> 2018-07-11 07:51:44.120152 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.115:6802/13323 conn(0x55d986bc1000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 245 0x55d982cdf800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:44.119239) v4 > -4064> 2018-07-11 07:51:44.120171 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.6 10.8.128.115:6802/13323 245 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:44.119239) v4 ==== 2004+0+0 (2097268797 0 0) 0x55d982cdf800 con 0x55d986bc1000 > -4063> 2018-07-11 07:51:44.120179 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.108:6802/11967 conn(0x55d986bbe000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.8 seq 245 0x55d9b9ac2200 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:44.119239) v4 > -4062> 2018-07-11 07:51:44.120189 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.8 10.8.128.108:6802/11967 245 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:44.119239) v4 ==== 2004+0+0 (2097268797 0 0) 0x55d9b9ac2200 con 0x55d986bbe000 > -4061> 2018-07-11 07:51:44.120190 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.108:6803/11967 conn(0x55d986bbf800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.8 seq 245 0x55d9c3383e00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:44.119239) v4 > -4060> 2018-07-11 07:51:44.120208 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.8 10.8.128.108:6803/11967 245 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:44.119239) v4 ==== 2004+0+0 (2097268797 0 0) 0x55d9c3383e00 con 0x55d986bbf800 > -4059> 2018-07-11 07:51:44.120220 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6803/15919 conn(0x55d986b5c800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.11 seq 245 0x55d9b9ac2200 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:44.119239) v4 > -4058> 2018-07-11 07:51:44.120218 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6806/15885 conn(0x55d986bac000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.7 seq 245 0x55d982cdf800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:44.119239) v4 > -4057> 2018-07-11 07:51:44.120233 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.11 10.8.128.114:6803/15919 245 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:44.119239) v4 ==== 2004+0+0 (2097268797 0 0) 0x55d9b9ac2200 con 0x55d986b5c800 > -4056> 2018-07-11 07:51:44.120235 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.7 10.8.128.114:6806/15885 245 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:44.119239) v4 ==== 2004+0+0 (2097268797 0 0) 0x55d982cdf800 con 0x55d986bac000 > -4055> 2018-07-11 07:51:44.120281 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6802/15919 conn(0x55d986b5b000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.11 seq 245 0x55d982cdf800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:44.119239) v4 > -4054> 2018-07-11 07:51:44.120296 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.11 10.8.128.114:6802/15919 245 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:44.119239) v4 ==== 2004+0+0 (2097268797 0 0) 0x55d982cdf800 con 0x55d986b5b000 > -4053> 2018-07-11 07:51:44.168173 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 972 0x55d9b2710480 rep_scrubmap(5.2e e501 from shard 2) v1 > -4052> 2018-07-11 07:51:44.168193 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 972 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12482 (1269249638 0 1648153153) 0x55d9b2710480 con 0x55d986bf0800 > -4051> 2018-07-11 07:51:44.176890 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 966 0x55d9b27106c0 rep_scrubmap(5.2e e501 from shard 4) v1 > -4050> 2018-07-11 07:51:44.176909 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 966 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12482 (3686815630 0 1648153153) 0x55d9b27106c0 con 0x55d986bec000 > -4049> 2018-07-11 07:51:44.234151 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -4048> 2018-07-11 07:51:44.234181 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -4047> 2018-07-11 07:51:44.234273 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -4046> 2018-07-11 07:51:44.235015 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -4045> 2018-07-11 07:51:44.235939 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67950,epoch:501/498,start:5:755d85df:::benchmark_data_magna115_10867_object164731:0,end:5:755dd0b6:::benchmark_data_magna115_11707_object174164:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb773900 con 0 > -4044> 2018-07-11 07:51:44.235990 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67950,epoch:501/498,start:5:755d85df:::benchmark_data_magna115_10867_object164731:0,end:5:755dd0b6:::benchmark_data_magna115_11707_object174164:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb773180 con 0 > -4043> 2018-07-11 07:51:44.423463 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 973 0x55d9b2710900 rep_scrubmap(5.2e e501 from shard 2) v1 > -4042> 2018-07-11 07:51:44.423483 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 973 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12478 (1269249638 0 3489248556) 0x55d9b2710900 con 0x55d986bf0800 > -4041> 2018-07-11 07:51:44.470234 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 967 0x55d9b2710b40 rep_scrubmap(5.2e e501 from shard 4) v1 > -4040> 2018-07-11 07:51:44.470264 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 967 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12478 (3686815630 0 3489248556) 0x55d9b2710b40 con 0x55d986bec000 > -4039> 2018-07-11 07:51:44.500913 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -4038> 2018-07-11 07:51:44.500938 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -4037> 2018-07-11 07:51:44.500983 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -4036> 2018-07-11 07:51:44.501450 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -4035> 2018-07-11 07:51:44.547958 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:755dd0b6:::benchmark_data_magna115_11707_object174164:0,end:5:755e3d88:::benchmark_data_magna115_10867_object1708785:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb773400 con 0 > -4034> 2018-07-11 07:51:44.547997 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:755dd0b6:::benchmark_data_magna115_11707_object174164:0,end:5:755e3d88:::benchmark_data_magna115_10867_object1708785:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb773680 con 0 > -4033> 2018-07-11 07:51:44.761263 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 974 0x55d9b2710d80 rep_scrubmap(5.2e e501 from shard 2) v1 > -4032> 2018-07-11 07:51:44.761282 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 974 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12474 (1269249638 0 155099142) 0x55d9b2710d80 con 0x55d986bf0800 > -4031> 2018-07-11 07:51:44.779518 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 968 0x55d9b2710fc0 rep_scrubmap(5.2e e501 from shard 4) v1 > -4030> 2018-07-11 07:51:44.779538 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 968 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12474 (3686815630 0 155099142) 0x55d9b2710fc0 con 0x55d986bec000 > -4029> 2018-07-11 07:51:44.779677 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -4028> 2018-07-11 07:51:44.779702 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -4027> 2018-07-11 07:51:44.779748 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -4026> 2018-07-11 07:51:44.780226 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -4025> 2018-07-11 07:51:44.798238 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67725,epoch:501/498,start:5:755e3d88:::benchmark_data_magna115_10867_object1708785:0,end:5:755ea945:::benchmark_data_magna115_10867_object795187:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454000 con 0 > -4024> 2018-07-11 07:51:44.798301 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67725,epoch:501/498,start:5:755e3d88:::benchmark_data_magna115_10867_object1708785:0,end:5:755ea945:::benchmark_data_magna115_10867_object795187:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454c80 con 0 > -4023> 2018-07-11 07:51:44.885759 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15287 conn(0x55d98123f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 267 0x55d982cdf800 osd_ping(ping e501 stamp 2018-07-11 07:51:46.116925) v4 > -4022> 2018-07-11 07:51:44.885780 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15287 conn(0x55d981240800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 267 0x55d986c22c00 osd_ping(ping e501 stamp 2018-07-11 07:51:46.116925) v4 > -4021> 2018-07-11 07:51:44.885787 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.4 10.8.128.113:0/15287 267 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:46.116925) v4 ==== 2004+0+0 (724667144 0 0) 0x55d982cdf800 con 0x55d98123f000 > -4020> 2018-07-11 07:51:44.885799 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.4 10.8.128.113:0/15287 267 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:46.116925) v4 ==== 2004+0+0 (724667144 0 0) 0x55d986c22c00 con 0x55d981240800 > -4019> 2018-07-11 07:51:44.885802 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:46.116925) v4 -- 0x55d9b67d9c00 con 0 > -4018> 2018-07-11 07:51:44.885842 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:46.116925) v4 -- 0x55d9ba818800 con 0 > -4017> 2018-07-11 07:51:44.963824 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6801/15919 conn(0x55d986c3f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=29 cs=1 l=0). rx osd.11 seq 242 0x55d9b9ac2200 MOSDScrubReserve(10.f REQUEST e501) v1 > -4016> 2018-07-11 07:51:44.963843 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.11 10.8.128.114:6801/15919 242 ==== MOSDScrubReserve(10.f REQUEST e501) v1 ==== 43+0+0 (2892966580 0 0) 0x55d9b9ac2200 con 0x55d986c3f800 > -4015> 2018-07-11 07:51:44.963874 7ffab53e3700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.114:6801/15919 -- MOSDScrubReserve(10.f REJECT e501) v1 -- 0x55d9b4388200 con 0 > -4014> 2018-07-11 07:51:44.964231 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6801/15919 conn(0x55d986c3f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=29 cs=1 l=0). rx osd.11 seq 243 0x55d9b4388200 MOSDScrubReserve(10.f RELEASE e501) v1 > -4013> 2018-07-11 07:51:44.964257 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.11 10.8.128.114:6801/15919 243 ==== MOSDScrubReserve(10.f RELEASE e501) v1 ==== 43+0+0 (1668468918 0 0) 0x55d9b4388200 con 0x55d986c3f800 > -4012> 2018-07-11 07:51:44.997997 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 975 0x55d9b2711200 rep_scrubmap(5.2e e501 from shard 2) v1 > -4011> 2018-07-11 07:51:44.998017 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 975 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12464 (1269249638 0 3875310732) 0x55d9b2711200 con 0x55d986bf0800 > -4010> 2018-07-11 07:51:45.027553 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 969 0x55d9b2711440 rep_scrubmap(5.2e e501 from shard 4) v1 > -4009> 2018-07-11 07:51:45.027572 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 969 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12464 (3686815630 0 3875310732) 0x55d9b2711440 con 0x55d986bec000 > -4008> 2018-07-11 07:51:45.027713 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -4007> 2018-07-11 07:51:45.027738 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -4006> 2018-07-11 07:51:45.027785 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -4005> 2018-07-11 07:51:45.028270 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -4004> 2018-07-11 07:51:45.073159 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'66595,epoch:501/498,start:5:755ea945:::benchmark_data_magna115_10867_object795187:0,end:5:755f0046:::benchmark_data_magna115_11707_object98672:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454280 con 0 > -4003> 2018-07-11 07:51:45.073209 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'66595,epoch:501/498,start:5:755ea945:::benchmark_data_magna115_10867_object795187:0,end:5:755f0046:::benchmark_data_magna115_11707_object98672:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb773b80 con 0 > -4002> 2018-07-11 07:51:45.150669 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.107:0/12617 conn(0x55d98123c000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 245 0x55d986c22c00 osd_ping(ping e501 stamp 2018-07-11 07:51:45.150330) v4 > -4001> 2018-07-11 07:51:45.150695 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.9 10.8.128.107:0/12617 245 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:45.150330) v4 ==== 2004+0+0 (741740743 0 0) 0x55d986c22c00 con 0x55d98123c000 > -4000> 2018-07-11 07:51:45.150696 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.107:0/12617 conn(0x55d98123d800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 245 0x55d986d97e00 osd_ping(ping e501 stamp 2018-07-11 07:51:45.150330) v4 > -3999> 2018-07-11 07:51:45.150717 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:45.150330) v4 -- 0x55d9ba818800 con 0 > -3998> 2018-07-11 07:51:45.150721 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.9 10.8.128.107:0/12617 245 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:45.150330) v4 ==== 2004+0+0 (741740743 0 0) 0x55d986d97e00 con 0x55d98123d800 > -3997> 2018-07-11 07:51:45.150742 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:45.150330) v4 -- 0x55d986d97400 con 0 > -3996> 2018-07-11 07:51:45.220099 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.114:0/15885 conn(0x55d986d81000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.7 seq 261 0x55d9c3383e00 osd_ping(ping e501 stamp 2018-07-11 07:51:45.850371) v4 > -3995> 2018-07-11 07:51:45.220102 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.114:0/15885 conn(0x55d986d7f800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.7 seq 261 0x55d9c3383c00 osd_ping(ping e501 stamp 2018-07-11 07:51:45.850371) v4 > -3994> 2018-07-11 07:51:45.220123 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.7 10.8.128.114:0/15885 261 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:45.850371) v4 ==== 2004+0+0 (4163539544 0 0) 0x55d9c3383e00 con 0x55d986d81000 > -3993> 2018-07-11 07:51:45.220126 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.7 10.8.128.114:0/15885 261 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:45.850371) v4 ==== 2004+0+0 (4163539544 0 0) 0x55d9c3383c00 con 0x55d986d7f800 > -3992> 2018-07-11 07:51:45.220132 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:45.850371) v4 -- 0x55d9ba818800 con 0 > -3991> 2018-07-11 07:51:45.220152 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:45.850371) v4 -- 0x55d9b67d9c00 con 0 > -3990> 2018-07-11 07:51:45.298104 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 970 0x55d9b2711680 rep_scrubmap(5.2e e501 from shard 4) v1 > -3989> 2018-07-11 07:51:45.298123 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 970 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12462 (3686815630 0 3270193042) 0x55d9b2711680 con 0x55d986bec000 > -3988> 2018-07-11 07:51:45.309980 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 976 0x55d9b27118c0 rep_scrubmap(5.2e e501 from shard 2) v1 > -3987> 2018-07-11 07:51:45.309999 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 976 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12462 (1269249638 0 3270193042) 0x55d9b27118c0 con 0x55d986bf0800 > -3986> 2018-07-11 07:51:45.310148 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -3985> 2018-07-11 07:51:45.310174 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -3984> 2018-07-11 07:51:45.310220 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -3983> 2018-07-11 07:51:45.310686 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -3982> 2018-07-11 07:51:45.345265 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67109,epoch:501/498,start:5:755f0046:::benchmark_data_magna115_11707_object98672:0,end:5:755f6ad2:::benchmark_data_magna115_10867_object2438152:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98944b680 con 0 > -3981> 2018-07-11 07:51:45.345304 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67109,epoch:501/498,start:5:755f0046:::benchmark_data_magna115_11707_object98672:0,end:5:755f6ad2:::benchmark_data_magna115_10867_object2438152:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454500 con 0 > -3980> 2018-07-11 07:51:45.548855 7ffacbbd1700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15263 conn(0x55d985d44000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.5 seq 237 0x55d985074400 osd_ping(ping e501 stamp 2018-07-11 07:51:46.780087) v4 > -3979> 2018-07-11 07:51:45.548856 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15263 conn(0x55d985d45800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.5 seq 237 0x55d982cdf800 osd_ping(ping e501 stamp 2018-07-11 07:51:46.780087) v4 > -3978> 2018-07-11 07:51:45.548881 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 <== osd.5 10.8.128.113:0/15263 237 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:46.780087) v4 ==== 2004+0+0 (3755033158 0 0) 0x55d985074400 con 0x55d985d44000 > -3977> 2018-07-11 07:51:45.548884 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.5 10.8.128.113:0/15263 237 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:46.780087) v4 ==== 2004+0+0 (3755033158 0 0) 0x55d982cdf800 con 0x55d985d45800 > -3976> 2018-07-11 07:51:45.548899 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:46.780087) v4 -- 0x55d986d97400 con 0 > -3975> 2018-07-11 07:51:45.548922 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:46.780087) v4 -- 0x55d9b67d9c00 con 0 > -3974> 2018-07-11 07:51:45.565717 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 977 0x55d9b2711b00 rep_scrubmap(5.2e e501 from shard 2) v1 > -3973> 2018-07-11 07:51:45.565738 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 977 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12462 (1269249638 0 790648664) 0x55d9b2711b00 con 0x55d986bf0800 > -3972> 2018-07-11 07:51:45.572509 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 971 0x55d9b2711d40 rep_scrubmap(5.2e e501 from shard 4) v1 > -3971> 2018-07-11 07:51:45.572529 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 971 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12462 (3686815630 0 790648664) 0x55d9b2711d40 con 0x55d986bec000 > -3970> 2018-07-11 07:51:45.651095 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.107:0/12617 conn(0x55d98123c000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 246 0x55d9c3383e00 osd_ping(ping e501 stamp 2018-07-11 07:51:45.650779) v4 > -3969> 2018-07-11 07:51:45.651120 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.9 10.8.128.107:0/12617 246 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:45.650779) v4 ==== 2004+0+0 (486489301 0 0) 0x55d9c3383e00 con 0x55d98123c000 > -3968> 2018-07-11 07:51:45.651116 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.107:0/12617 conn(0x55d98123d800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 246 0x55d986d97e00 osd_ping(ping e501 stamp 2018-07-11 07:51:45.650779) v4 > -3967> 2018-07-11 07:51:45.651131 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:45.650779) v4 -- 0x55d9ba818800 con 0 > -3966> 2018-07-11 07:51:45.651142 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.9 10.8.128.107:0/12617 246 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:45.650779) v4 ==== 2004+0+0 (486489301 0 0) 0x55d986d97e00 con 0x55d98123d800 > -3965> 2018-07-11 07:51:45.651161 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:45.650779) v4 -- 0x55d986d97400 con 0 > -3964> 2018-07-11 07:51:45.675233 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -3963> 2018-07-11 07:51:45.675275 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -3962> 2018-07-11 07:51:45.675319 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -3961> 2018-07-11 07:51:45.675778 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -3960> 2018-07-11 07:51:45.690938 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'66999,epoch:501/498,start:5:755f6ad2:::benchmark_data_magna115_10867_object2438152:0,end:5:755fcb7d:::benchmark_data_magna115_10867_object1105996:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455180 con 0 > -3959> 2018-07-11 07:51:45.690977 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'66999,epoch:501/498,start:5:755f6ad2:::benchmark_data_magna115_10867_object2438152:0,end:5:755fcb7d:::benchmark_data_magna115_10867_object1105996:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454f00 con 0 > -3958> 2018-07-11 07:51:45.913351 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 978 0x55d9bf776000 rep_scrubmap(5.2e e501 from shard 2) v1 > -3957> 2018-07-11 07:51:45.913371 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 978 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12468 (1269249638 0 3199384004) 0x55d9bf776000 con 0x55d986bf0800 > -3956> 2018-07-11 07:51:45.954718 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 972 0x55d9bf776240 rep_scrubmap(5.2e e501 from shard 4) v1 > -3955> 2018-07-11 07:51:45.954737 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 972 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12468 (3686815630 0 3199384004) 0x55d9bf776240 con 0x55d986bec000 > -3954> 2018-07-11 07:51:46.007273 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -3953> 2018-07-11 07:51:46.007298 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -3952> 2018-07-11 07:51:46.007347 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -3951> 2018-07-11 07:51:46.007809 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -3950> 2018-07-11 07:51:46.008863 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:755fcb7d:::benchmark_data_magna115_10867_object1105996:0,end:5:7560442a:::benchmark_data_magna115_10867_object3446987:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454780 con 0 > -3949> 2018-07-11 07:51:46.008909 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:755fcb7d:::benchmark_data_magna115_10867_object1105996:0,end:5:7560442a:::benchmark_data_magna115_10867_object3446987:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455400 con 0 > -3948> 2018-07-11 07:51:46.167973 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15255 conn(0x55d986d20000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.0 seq 254 0x55d9c3383e00 osd_ping(ping e501 stamp 2018-07-11 07:51:47.399059) v4 > -3947> 2018-07-11 07:51:46.167996 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.0 10.8.128.113:0/15255 254 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:47.399059) v4 ==== 2004+0+0 (1260523437 0 0) 0x55d9c3383e00 con 0x55d986d20000 > -3946> 2018-07-11 07:51:46.168006 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:47.399059) v4 -- 0x55d9ba818800 con 0 > -3945> 2018-07-11 07:51:46.168025 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15255 conn(0x55d986c82800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.0 seq 254 0x55d9c3383c00 osd_ping(ping e501 stamp 2018-07-11 07:51:47.399059) v4 > -3944> 2018-07-11 07:51:46.168046 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.0 10.8.128.113:0/15255 254 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:47.399059) v4 ==== 2004+0+0 (1260523437 0 0) 0x55d9c3383c00 con 0x55d986c82800 > -3943> 2018-07-11 07:51:46.168063 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:47.399059) v4 -- 0x55d9b67d9c00 con 0 > -3942> 2018-07-11 07:51:46.217740 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 979 0x55d9bf776480 rep_scrubmap(5.2e e501 from shard 2) v1 > -3941> 2018-07-11 07:51:46.217761 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 979 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12470 (1269249638 0 334914441) 0x55d9bf776480 con 0x55d986bf0800 > -3940> 2018-07-11 07:51:46.227469 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 973 0x55d9bf7766c0 rep_scrubmap(5.2e e501 from shard 4) v1 > -3939> 2018-07-11 07:51:46.227488 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 973 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12470 (3686815630 0 334914441) 0x55d9bf7766c0 con 0x55d986bec000 > -3938> 2018-07-11 07:51:46.320638 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -3937> 2018-07-11 07:51:46.320677 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -3936> 2018-07-11 07:51:46.320733 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -3935> 2018-07-11 07:51:46.321195 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -3934> 2018-07-11 07:51:46.340992 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67161,epoch:501/498,start:5:7560442a:::benchmark_data_magna115_10867_object3446987:0,end:5:756098ce:::benchmark_data_magna115_10867_object1064302:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498280 con 0 > -3933> 2018-07-11 07:51:46.341042 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67161,epoch:501/498,start:5:7560442a:::benchmark_data_magna115_10867_object3446987:0,end:5:756098ce:::benchmark_data_magna115_10867_object1064302:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455b80 con 0 > -3932> 2018-07-11 07:51:46.419721 7ffab1bdc700 5 osd.10 501 heartbeat: osd_stat(788 GB used, 142 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist []) > -3931> 2018-07-11 07:51:46.419733 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6802/15255 -- osd_ping(ping e501 stamp 2018-07-11 07:51:46.419732) v4 -- 0x55d9b4389a00 con 0 > -3930> 2018-07-11 07:51:46.419769 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6803/15255 -- osd_ping(ping e501 stamp 2018-07-11 07:51:46.419732) v4 -- 0x55d9b40a4c00 con 0 > -3929> 2018-07-11 07:51:46.419785 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6806/12677 -- osd_ping(ping e501 stamp 2018-07-11 07:51:46.419732) v4 -- 0x55d982cde600 con 0 > -3928> 2018-07-11 07:51:46.419800 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6807/12677 -- osd_ping(ping e501 stamp 2018-07-11 07:51:46.419732) v4 -- 0x55d986d96c00 con 0 > -3927> 2018-07-11 07:51:46.419824 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6808/12651 -- osd_ping(ping e501 stamp 2018-07-11 07:51:46.419732) v4 -- 0x55d9b9ac2a00 con 0 > -3926> 2018-07-11 07:51:46.419833 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6810/12651 -- osd_ping(ping e501 stamp 2018-07-11 07:51:46.419732) v4 -- 0x55d9c3382e00 con 0 > -3925> 2018-07-11 07:51:46.419841 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6809/12650 -- osd_ping(ping e501 stamp 2018-07-11 07:51:46.419732) v4 -- 0x55d9c3382200 con 0 > -3924> 2018-07-11 07:51:46.419851 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6811/12650 -- osd_ping(ping e501 stamp 2018-07-11 07:51:46.419732) v4 -- 0x55d9b437a200 con 0 > -3923> 2018-07-11 07:51:46.419863 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6810/15287 -- osd_ping(ping e501 stamp 2018-07-11 07:51:46.419732) v4 -- 0x55d9829e4600 con 0 > -3922> 2018-07-11 07:51:46.419873 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6811/15287 -- osd_ping(ping e501 stamp 2018-07-11 07:51:46.419732) v4 -- 0x55d9b9ac3000 con 0 > -3921> 2018-07-11 07:51:46.419890 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6806/15263 -- osd_ping(ping e501 stamp 2018-07-11 07:51:46.419732) v4 -- 0x55d9bcd72e00 con 0 > -3920> 2018-07-11 07:51:46.419904 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6807/15263 -- osd_ping(ping e501 stamp 2018-07-11 07:51:46.419732) v4 -- 0x55d9829e5600 con 0 > -3919> 2018-07-11 07:51:46.419915 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.115:6802/13323 -- osd_ping(ping e501 stamp 2018-07-11 07:51:46.419732) v4 -- 0x55d9869bec00 con 0 > -3918> 2018-07-11 07:51:46.419927 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.115:6803/13323 -- osd_ping(ping e501 stamp 2018-07-11 07:51:46.419732) v4 -- 0x55d9b3db2600 con 0 > -3917> 2018-07-11 07:51:46.419943 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6806/15885 -- osd_ping(ping e501 stamp 2018-07-11 07:51:46.419732) v4 -- 0x55d986d97200 con 0 > -3916> 2018-07-11 07:51:46.419957 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6807/15885 -- osd_ping(ping e501 stamp 2018-07-11 07:51:46.419732) v4 -- 0x55d986d33200 con 0 > -3915> 2018-07-11 07:51:46.419970 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.108:6802/11967 -- osd_ping(ping e501 stamp 2018-07-11 07:51:46.419732) v4 -- 0x55d9b4388400 con 0 > -3914> 2018-07-11 07:51:46.419984 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.108:6803/11967 -- osd_ping(ping e501 stamp 2018-07-11 07:51:46.419732) v4 -- 0x55d986d96800 con 0 > -3913> 2018-07-11 07:51:46.419997 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.107:6806/12617 -- osd_ping(ping e501 stamp 2018-07-11 07:51:46.419732) v4 -- 0x55d986c15000 con 0 > -3912> 2018-07-11 07:51:46.420010 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.107:6807/12617 -- osd_ping(ping e501 stamp 2018-07-11 07:51:46.419732) v4 -- 0x55d9bf805e00 con 0 > -3911> 2018-07-11 07:51:46.420022 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6802/15919 -- osd_ping(ping e501 stamp 2018-07-11 07:51:46.419732) v4 -- 0x55d9869bf400 con 0 > -3910> 2018-07-11 07:51:46.420035 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6803/15919 -- osd_ping(ping e501 stamp 2018-07-11 07:51:46.419732) v4 -- 0x55d986cd6a00 con 0 > -3909> 2018-07-11 07:51:46.420201 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.107:6807/12617 conn(0x55d986bd3800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.9 seq 246 0x55d985074400 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:46.419732) v4 > -3908> 2018-07-11 07:51:46.420229 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.107:6806/12617 conn(0x55d986bd2000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.9 seq 246 0x55d982cdf800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:46.419732) v4 > -3907> 2018-07-11 07:51:46.420242 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.9 10.8.128.107:6807/12617 246 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:46.419732) v4 ==== 2004+0+0 (781256902 0 0) 0x55d985074400 con 0x55d986bd3800 > -3906> 2018-07-11 07:51:46.420262 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.9 10.8.128.107:6806/12617 246 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:46.419732) v4 ==== 2004+0+0 (781256902 0 0) 0x55d982cdf800 con 0x55d986bd2000 > -3905> 2018-07-11 07:51:46.420297 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6806/12677 conn(0x55d986b58000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.1 seq 246 0x55d985074400 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:46.419732) v4 > -3904> 2018-07-11 07:51:46.420331 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.1 10.8.128.106:6806/12677 246 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:46.419732) v4 ==== 2004+0+0 (781256902 0 0) 0x55d985074400 con 0x55d986b58000 > -3903> 2018-07-11 07:51:46.420349 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6811/15287 conn(0x55d986bb0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 246 0x55d982cdf800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:46.419732) v4 > -3902> 2018-07-11 07:51:46.420367 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.4 10.8.128.113:6811/15287 246 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:46.419732) v4 ==== 2004+0+0 (781256902 0 0) 0x55d982cdf800 con 0x55d986bb0800 > -3901> 2018-07-11 07:51:46.420364 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6809/12650 conn(0x55d986bcb000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 246 0x55d985074400 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:46.419732) v4 > -3900> 2018-07-11 07:51:46.420383 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.3 10.8.128.106:6809/12650 246 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:46.419732) v4 ==== 2004+0+0 (781256902 0 0) 0x55d985074400 con 0x55d986bcb000 > -3899> 2018-07-11 07:51:46.420407 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6810/12651 conn(0x55d986b9b800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.2 seq 246 0x55d982cdf800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:46.419732) v4 > -3898> 2018-07-11 07:51:46.420393 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6808/12651 conn(0x55d986b9a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.2 seq 246 0x55d986c22c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:46.419732) v4 > -3897> 2018-07-11 07:51:46.420423 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.2 10.8.128.106:6810/12651 246 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:46.419732) v4 ==== 2004+0+0 (781256902 0 0) 0x55d982cdf800 con 0x55d986b9b800 > -3896> 2018-07-11 07:51:46.420429 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.2 10.8.128.106:6808/12651 246 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:46.419732) v4 ==== 2004+0+0 (781256902 0 0) 0x55d986c22c00 con 0x55d986b9a000 > -3895> 2018-07-11 07:51:46.420439 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6806/15263 conn(0x55d986b9d000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.5 seq 246 0x55d985074400 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:46.419732) v4 > -3894> 2018-07-11 07:51:46.420456 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6803/15255 conn(0x55d986bc9800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.0 seq 246 0x55d982cdf800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:46.419732) v4 > -3893> 2018-07-11 07:51:46.420466 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6802/15255 conn(0x55d986bc8000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.0 seq 246 0x55d986c22c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:46.419732) v4 > -3892> 2018-07-11 07:51:46.420472 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.5 10.8.128.113:6806/15263 246 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:46.419732) v4 ==== 2004+0+0 (781256902 0 0) 0x55d985074400 con 0x55d986b9d000 > -3891> 2018-07-11 07:51:46.420478 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.0 10.8.128.113:6802/15255 246 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:46.419732) v4 ==== 2004+0+0 (781256902 0 0) 0x55d986c22c00 con 0x55d986bc8000 > -3890> 2018-07-11 07:51:46.420477 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.0 10.8.128.113:6803/15255 246 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:46.419732) v4 ==== 2004+0+0 (781256902 0 0) 0x55d982cdf800 con 0x55d986bc9800 > -3889> 2018-07-11 07:51:46.420511 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6811/12650 conn(0x55d986bcc800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 246 0x55d986c22c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:46.419732) v4 > -3888> 2018-07-11 07:51:46.420524 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.3 10.8.128.106:6811/12650 246 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:46.419732) v4 ==== 2004+0+0 (781256902 0 0) 0x55d986c22c00 con 0x55d986bcc800 > -3887> 2018-07-11 07:51:46.420553 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6807/12677 conn(0x55d986b59800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.1 seq 246 0x55d986c22c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:46.419732) v4 > -3886> 2018-07-11 07:51:46.420549 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.115:6802/13323 conn(0x55d986bc1000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 246 0x55d982cdf800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:46.419732) v4 > -3885> 2018-07-11 07:51:46.420565 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.1 10.8.128.106:6807/12677 246 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:46.419732) v4 ==== 2004+0+0 (781256902 0 0) 0x55d986c22c00 con 0x55d986b59800 > -3884> 2018-07-11 07:51:46.420567 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.6 10.8.128.115:6802/13323 246 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:46.419732) v4 ==== 2004+0+0 (781256902 0 0) 0x55d982cdf800 con 0x55d986bc1000 > -3883> 2018-07-11 07:51:46.420583 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6807/15263 conn(0x55d986b9e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.5 seq 246 0x55d986c22c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:46.419732) v4 > -3882> 2018-07-11 07:51:46.420613 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.5 10.8.128.113:6807/15263 246 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:46.419732) v4 ==== 2004+0+0 (781256902 0 0) 0x55d986c22c00 con 0x55d986b9e800 > -3881> 2018-07-11 07:51:46.420625 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6802/15919 conn(0x55d986b5b000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.11 seq 246 0x55d982cdf800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:46.419732) v4 > -3880> 2018-07-11 07:51:46.420641 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.11 10.8.128.114:6802/15919 246 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:46.419732) v4 ==== 2004+0+0 (781256902 0 0) 0x55d982cdf800 con 0x55d986b5b000 > -3879> 2018-07-11 07:51:46.420648 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6810/15287 conn(0x55d986baf000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 246 0x55d986c22c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:46.419732) v4 > -3878> 2018-07-11 07:51:46.420660 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.4 10.8.128.113:6810/15287 246 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:46.419732) v4 ==== 2004+0+0 (781256902 0 0) 0x55d986c22c00 con 0x55d986baf000 > -3877> 2018-07-11 07:51:46.420642 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.108:6802/11967 conn(0x55d986bbe000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.8 seq 246 0x55d985074400 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:46.419732) v4 > -3876> 2018-07-11 07:51:46.420673 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.8 10.8.128.108:6802/11967 246 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:46.419732) v4 ==== 2004+0+0 (781256902 0 0) 0x55d985074400 con 0x55d986bbe000 > -3875> 2018-07-11 07:51:46.420692 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.108:6803/11967 conn(0x55d986bbf800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.8 seq 246 0x55d986c22c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:46.419732) v4 > -3874> 2018-07-11 07:51:46.420703 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.8 10.8.128.108:6803/11967 246 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:46.419732) v4 ==== 2004+0+0 (781256902 0 0) 0x55d986c22c00 con 0x55d986bbf800 > -3873> 2018-07-11 07:51:46.420723 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.115:6803/13323 conn(0x55d986bc2800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 246 0x55d985074400 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:46.419732) v4 > -3872> 2018-07-11 07:51:46.420733 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6806/15885 conn(0x55d986bac000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.7 seq 246 0x55d982cdf800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:46.419732) v4 > -3871> 2018-07-11 07:51:46.420738 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.6 10.8.128.115:6803/13323 246 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:46.419732) v4 ==== 2004+0+0 (781256902 0 0) 0x55d985074400 con 0x55d986bc2800 > -3870> 2018-07-11 07:51:46.420748 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.7 10.8.128.114:6806/15885 246 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:46.419732) v4 ==== 2004+0+0 (781256902 0 0) 0x55d982cdf800 con 0x55d986bac000 > -3869> 2018-07-11 07:51:46.420782 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6807/15885 conn(0x55d986bad800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.7 seq 246 0x55d985074400 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:46.419732) v4 > -3868> 2018-07-11 07:51:46.420796 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.7 10.8.128.114:6807/15885 246 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:46.419732) v4 ==== 2004+0+0 (781256902 0 0) 0x55d985074400 con 0x55d986bad800 > -3867> 2018-07-11 07:51:46.420834 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6803/15919 conn(0x55d986b5c800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.11 seq 246 0x55d985074400 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:46.419732) v4 > -3866> 2018-07-11 07:51:46.420847 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.11 10.8.128.114:6803/15919 246 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:46.419732) v4 ==== 2004+0+0 (781256902 0 0) 0x55d985074400 con 0x55d986b5c800 > -3865> 2018-07-11 07:51:46.582843 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 974 0x55d9bf776900 rep_scrubmap(5.2e e501 from shard 4) v1 > -3864> 2018-07-11 07:51:46.582863 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 974 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12464 (3686815630 0 536427325) 0x55d9bf776900 con 0x55d986bec000 > -3863> 2018-07-11 07:51:46.586696 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 980 0x55d9bf776b40 rep_scrubmap(5.2e e501 from shard 2) v1 > -3862> 2018-07-11 07:51:46.586715 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 980 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12464 (1269249638 0 536427325) 0x55d9bf776b40 con 0x55d986bf0800 > -3861> 2018-07-11 07:51:46.645075 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -3860> 2018-07-11 07:51:46.645101 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -3859> 2018-07-11 07:51:46.645156 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -3858> 2018-07-11 07:51:46.645637 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -3857> 2018-07-11 07:51:46.646660 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67508,epoch:501/498,start:5:756098ce:::benchmark_data_magna115_10867_object1064302:0,end:5:7560f800:::benchmark_data_magna115_10867_object3418944:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498000 con 0 > -3856> 2018-07-11 07:51:46.646707 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67508,epoch:501/498,start:5:756098ce:::benchmark_data_magna115_10867_object1064302:0,end:5:7560f800:::benchmark_data_magna115_10867_object3418944:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b14f0500 con 0 > -3855> 2018-07-11 07:51:46.840594 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 3010 0x55d985074400 MOSDScrubReserve(11.3s5 REQUEST e501) v1 > -3854> 2018-07-11 07:51:46.840613 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 3010 ==== MOSDScrubReserve(11.3s5 REQUEST e501) v1 ==== 43+0+0 (3260170403 0 0) 0x55d985074400 con 0x55d986c7e000 > -3853> 2018-07-11 07:51:46.840684 7ffab3be0700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e501) v1 -- 0x55d9869be000 con 0 > -3852> 2018-07-11 07:51:46.841100 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 3011 0x55d9869be000 MOSDScrubReserve(11.3s5 RELEASE e501) v1 > -3851> 2018-07-11 07:51:46.841117 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 3011 ==== MOSDScrubReserve(11.3s5 RELEASE e501) v1 ==== 43+0+0 (223328929 0 0) 0x55d9869be000 con 0x55d986c7e000 > -3850> 2018-07-11 07:51:46.859820 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 981 0x55d9bf776d80 rep_scrubmap(5.2e e501 from shard 2) v1 > -3849> 2018-07-11 07:51:46.859851 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 981 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12464 (1269249638 0 4192554902) 0x55d9bf776d80 con 0x55d986bf0800 > -3848> 2018-07-11 07:51:46.873721 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 975 0x55d9bf776fc0 rep_scrubmap(5.2e e501 from shard 4) v1 > -3847> 2018-07-11 07:51:46.873740 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 975 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12464 (3686815630 0 4192554902) 0x55d9bf776fc0 con 0x55d986bec000 > -3846> 2018-07-11 07:51:46.874298 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -3845> 2018-07-11 07:51:46.874322 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -3844> 2018-07-11 07:51:46.874365 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -3843> 2018-07-11 07:51:46.874806 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -3842> 2018-07-11 07:51:46.916272 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67262,epoch:501/498,start:5:7560f800:::benchmark_data_magna115_10867_object3418944:0,end:5:75614514:::benchmark_data_magna115_10867_object622855:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455680 con 0 > -3841> 2018-07-11 07:51:46.916310 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67262,epoch:501/498,start:5:7560f800:::benchmark_data_magna115_10867_object3418944:0,end:5:75614514:::benchmark_data_magna115_10867_object622855:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455900 con 0 > -3840> 2018-07-11 07:51:46.962729 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6805/15263 conn(0x55d986c41000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=31 cs=1 l=0). rx osd.5 seq 2828 0x55d986c22c00 MOSDScrubReserve(6.15 REQUEST e501) v1 > -3839> 2018-07-11 07:51:46.962748 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.5 10.8.128.113:6805/15263 2828 ==== MOSDScrubReserve(6.15 REQUEST e501) v1 ==== 43+0+0 (3942742026 0 0) 0x55d986c22c00 con 0x55d986c41000 > -3838> 2018-07-11 07:51:46.994334 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.114:0/15919 conn(0x55d986df3800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.11 seq 257 0x55d986d97e00 osd_ping(ping e501 stamp 2018-07-11 07:51:47.624710) v4 > -3837> 2018-07-11 07:51:46.994374 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.11 10.8.128.114:0/15919 257 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:47.624710) v4 ==== 2004+0+0 (13236709 0 0) 0x55d986d97e00 con 0x55d986df3800 > -3836> 2018-07-11 07:51:46.994388 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:47.624710) v4 -- 0x55d9bf805e00 con 0 > -3835> 2018-07-11 07:51:46.994410 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.114:0/15919 conn(0x55d986df5000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.11 seq 257 0x55d982cdf800 osd_ping(ping e501 stamp 2018-07-11 07:51:47.624710) v4 > -3834> 2018-07-11 07:51:46.994432 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.11 10.8.128.114:0/15919 257 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:47.624710) v4 ==== 2004+0+0 (13236709 0 0) 0x55d982cdf800 con 0x55d986df5000 > -3833> 2018-07-11 07:51:46.994443 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:47.624710) v4 -- 0x55d9869bf400 con 0 > -3832> 2018-07-11 07:51:47.025470 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12651 conn(0x55d986db3000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.2 seq 255 0x55d986cd6a00 osd_ping(ping e501 stamp 2018-07-11 07:51:47.073260) v4 > -3831> 2018-07-11 07:51:47.025491 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.2 10.8.128.106:0/12651 255 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:47.073260) v4 ==== 2004+0+0 (1453212103 0 0) 0x55d986cd6a00 con 0x55d986db3000 > -3830> 2018-07-11 07:51:47.025482 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12651 conn(0x55d986db1800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.2 seq 255 0x55d9c3383e00 osd_ping(ping e501 stamp 2018-07-11 07:51:47.073260) v4 > -3829> 2018-07-11 07:51:47.025500 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:47.073260) v4 -- 0x55d9bf805e00 con 0 > -3828> 2018-07-11 07:51:47.025504 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.2 10.8.128.106:0/12651 255 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:47.073260) v4 ==== 2004+0+0 (1453212103 0 0) 0x55d9c3383e00 con 0x55d986db1800 > -3827> 2018-07-11 07:51:47.025517 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:47.073260) v4 -- 0x55d9829e5600 con 0 > -3826> 2018-07-11 07:51:47.142925 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 982 0x55d9bf777200 rep_scrubmap(5.2e e501 from shard 2) v1 > -3825> 2018-07-11 07:51:47.142946 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 982 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12478 (1269249638 0 3098315432) 0x55d9bf777200 con 0x55d986bf0800 > -3824> 2018-07-11 07:51:47.148691 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 976 0x55d9bf777440 rep_scrubmap(5.2e e501 from shard 4) v1 > -3823> 2018-07-11 07:51:47.148711 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 976 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12478 (3686815630 0 3098315432) 0x55d9bf777440 con 0x55d986bec000 > -3822> 2018-07-11 07:51:47.194139 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6805/15263 -- MOSDScrubReserve(6.15 REJECT e501) v1 -- 0x55d986c22e00 con 0 > -3821> 2018-07-11 07:51:47.194352 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -3820> 2018-07-11 07:51:47.194374 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -3819> 2018-07-11 07:51:47.194419 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -3818> 2018-07-11 07:51:47.194677 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6805/15263 conn(0x55d986c41000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=31 cs=1 l=0). rx osd.5 seq 2829 0x55d986c22e00 MOSDScrubReserve(6.15 RELEASE e501) v1 > -3817> 2018-07-11 07:51:47.194695 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.5 10.8.128.113:6805/15263 2829 ==== MOSDScrubReserve(6.15 RELEASE e501) v1 ==== 43+0+0 (605848072 0 0) 0x55d986c22e00 con 0x55d986c41000 > -3816> 2018-07-11 07:51:47.194873 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -3815> 2018-07-11 07:51:47.195922 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'66892,epoch:501/498,start:5:75614514:::benchmark_data_magna115_10867_object622855:0,end:5:7561ad95:::benchmark_data_magna115_10867_object772169:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498780 con 0 > -3814> 2018-07-11 07:51:47.195966 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'66892,epoch:501/498,start:5:75614514:::benchmark_data_magna115_10867_object622855:0,end:5:7561ad95:::benchmark_data_magna115_10867_object772169:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98696e000 con 0 > -3813> 2018-07-11 07:51:47.202814 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.115:0/13323 conn(0x55d986d7e000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 254 0x55d986d97e00 osd_ping(ping e501 stamp 2018-07-11 07:51:47.548662) v4 > -3812> 2018-07-11 07:51:47.202836 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.6 10.8.128.115:0/13323 254 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:47.548662) v4 ==== 2004+0+0 (637481241 0 0) 0x55d986d97e00 con 0x55d986d7e000 > -3811> 2018-07-11 07:51:47.202846 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:47.548662) v4 -- 0x55d9bf805e00 con 0 > -3810> 2018-07-11 07:51:47.202904 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.115:0/13323 conn(0x55d986d24800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 254 0x55d9c3383e00 osd_ping(ping e501 stamp 2018-07-11 07:51:47.548662) v4 > -3809> 2018-07-11 07:51:47.202934 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.6 10.8.128.115:0/13323 254 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:47.548662) v4 ==== 2004+0+0 (637481241 0 0) 0x55d9c3383e00 con 0x55d986d24800 > -3808> 2018-07-11 07:51:47.202948 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:47.548662) v4 -- 0x55d9829e5600 con 0 > -3807> 2018-07-11 07:51:47.351220 7ffab6be6700 1 -- 10.8.128.107:6800/13927 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55d9bd9cc2c0 con 0 > -3806> 2018-07-11 07:51:47.351299 7ffab6be6700 1 -- 10.8.128.107:6800/13927 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55d9b3fde580 con 0 > -3805> 2018-07-11 07:51:47.365770 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12677 conn(0x55d986db4800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.1 seq 258 0x55d9c3383c00 osd_ping(ping e501 stamp 2018-07-11 07:51:47.413614) v4 > -3804> 2018-07-11 07:51:47.365783 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12677 conn(0x55d986df2000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.1 seq 258 0x55d986d96800 osd_ping(ping e501 stamp 2018-07-11 07:51:47.413614) v4 > -3803> 2018-07-11 07:51:47.365795 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.1 10.8.128.106:0/12677 258 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:47.413614) v4 ==== 2004+0+0 (2889478097 0 0) 0x55d9c3383c00 con 0x55d986db4800 > -3802> 2018-07-11 07:51:47.365800 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.1 10.8.128.106:0/12677 258 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:47.413614) v4 ==== 2004+0+0 (2889478097 0 0) 0x55d986d96800 con 0x55d986df2000 > -3801> 2018-07-11 07:51:47.365806 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:47.413614) v4 -- 0x55d9869bf400 con 0 > -3800> 2018-07-11 07:51:47.365843 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:47.413614) v4 -- 0x55d9829e5600 con 0 > -3799> 2018-07-11 07:51:47.399243 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 983 0x55d9bf777680 rep_scrubmap(5.2e e501 from shard 2) v1 > -3798> 2018-07-11 07:51:47.399271 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 983 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12472 (1269249638 0 3751191361) 0x55d9bf777680 con 0x55d986bf0800 > -3797> 2018-07-11 07:51:47.433761 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 977 0x55d9bf7778c0 rep_scrubmap(5.2e e501 from shard 4) v1 > -3796> 2018-07-11 07:51:47.433779 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 977 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12472 (3686815630 0 3751191361) 0x55d9bf7778c0 con 0x55d986bec000 > -3795> 2018-07-11 07:51:47.890125 7ffacbbd1700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.108:0/11967 conn(0x55d986d82800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.8 seq 236 0x55d986cd6a00 osd_ping(ping e501 stamp 2018-07-11 07:51:47.645801) v4 > -3794> 2018-07-11 07:51:47.890128 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.108:0/11967 conn(0x55d986db0000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.8 seq 236 0x55d982cdf800 osd_ping(ping e501 stamp 2018-07-11 07:51:47.645801) v4 > -3793> 2018-07-11 07:51:47.890151 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 <== osd.8 10.8.128.108:0/11967 236 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:47.645801) v4 ==== 2004+0+0 (1673693817 0 0) 0x55d986cd6a00 con 0x55d986d82800 > -3792> 2018-07-11 07:51:47.890155 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.8 10.8.128.108:0/11967 236 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:47.645801) v4 ==== 2004+0+0 (1673693817 0 0) 0x55d982cdf800 con 0x55d986db0000 > -3791> 2018-07-11 07:51:47.890166 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:47.645801) v4 -- 0x55d9bf805e00 con 0 > -3790> 2018-07-11 07:51:47.890200 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:47.645801) v4 -- 0x55d9869bf400 con 0 > -3789> 2018-07-11 07:51:47.951783 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.107:0/12617 conn(0x55d98123c000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 247 0x55d9c3383e00 osd_ping(ping e501 stamp 2018-07-11 07:51:47.951244) v4 > -3788> 2018-07-11 07:51:47.951804 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.9 10.8.128.107:0/12617 247 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:47.951244) v4 ==== 2004+0+0 (1740768112 0 0) 0x55d9c3383e00 con 0x55d98123c000 > -3787> 2018-07-11 07:51:47.951814 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:47.951244) v4 -- 0x55d9829e5600 con 0 > -3786> 2018-07-11 07:51:47.951785 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.107:0/12617 conn(0x55d98123d800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 247 0x55d986d97e00 osd_ping(ping e501 stamp 2018-07-11 07:51:47.951244) v4 > -3785> 2018-07-11 07:51:47.951842 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.9 10.8.128.107:0/12617 247 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:47.951244) v4 ==== 2004+0+0 (1740768112 0 0) 0x55d986d97e00 con 0x55d98123d800 > -3784> 2018-07-11 07:51:47.951854 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:47.951244) v4 -- 0x55d9bf805e00 con 0 > -3783> 2018-07-11 07:51:48.289145 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -3782> 2018-07-11 07:51:48.289174 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -3781> 2018-07-11 07:51:48.289220 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -3780> 2018-07-11 07:51:48.289692 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -3779> 2018-07-11 07:51:48.316485 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67127,epoch:501/498,start:5:7561ad95:::benchmark_data_magna115_10867_object772169:0,end:5:75620ad6:::benchmark_data_magna115_10867_object73195:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98696f180 con 0 > -3778> 2018-07-11 07:51:48.316535 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67127,epoch:501/498,start:5:7561ad95:::benchmark_data_magna115_10867_object772169:0,end:5:75620ad6:::benchmark_data_magna115_10867_object73195:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498a00 con 0 > -3777> 2018-07-11 07:51:48.409955 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 32] Generated table #36806: 3182184 keys, 75905673 bytes > -3776> 2018-07-11 07:51:48.409989 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295508409974, "cf_name": "default", "job": 32, "event": "table_file_creation", "file_number": 36806, "file_size": 75905673, "table_properties": {"data_size": 67110118, "index_size": 689611, "filter_size": 8104952, "raw_key_size": 91929756, "raw_average_key_size": 28, "raw_value_size": 24007109, "raw_average_value_size": 7, "num_data_blocks": 16484, "num_entries": 3182184, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1162895", "kMergeOperands": "0"}} > -3775> 2018-07-11 07:51:48.410018 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 32] Compacted 1@2 + 2@3 files to L3 => 229978676 bytes > -3774> 2018-07-11 07:51:48.426988 7ffabc3f1700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 51362925809 > -3773> 2018-07-11 07:51:48.427200 7ffabc3f1700 4 rocksdb: (Original Log Time 2018/07/11-07:51:48.426957) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 11 68 470 3641 796 0] max score 2.20, MB/sec: 12.3 rd, 12.3 wr, level 3, files in(1, 2) out(3) MB in(76.6, 142.7) out(219.3), read-write-amplify(5.7) write-amplify(2.9) OK, records in: 10448932, records dropped: 0 > > -3772> 2018-07-11 07:51:48.427210 7ffabc3f1700 4 rocksdb: (Original Log Time 2018/07/11-07:51:48.426981) EVENT_LOG_v1 {"time_micros": 1531295508426971, "job": 32, "event": "compaction_finished", "compaction_time_micros": 18664538, "output_level": 3, "num_output_files": 3, "total_output_size": 229978676, "num_input_records": 10448932, "num_output_records": 10448932, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 11, 68, 470, 3641, 796, 0]} > -3771> 2018-07-11 07:51:48.427812 7ffabc3f1700 5 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:307] [JOB 32] Delete db.slow/036708.sst type=2 #36708 -- OK > > -3770> 2018-07-11 07:51:48.427825 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295508427821, "job": 32, "event": "table_file_deletion", "file_number": 36708} > -3769> 2018-07-11 07:51:48.427850 7ffabc3f1700 5 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:307] [JOB 32] Delete db.slow/036582.sst type=2 #36582 -- OK > > -3768> 2018-07-11 07:51:48.427855 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295508427853, "job": 32, "event": "table_file_deletion", "file_number": 36582} > -3767> 2018-07-11 07:51:48.427864 7ffabc3f1700 5 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:307] [JOB 32] Delete db.slow/026977.sst type=2 #26977 -- OK > > -3766> 2018-07-11 07:51:48.427869 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295508427867, "job": 32, "event": "table_file_deletion", "file_number": 26977} > -3765> 2018-07-11 07:51:48.427961 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_picker.cc:461] [default] Expanding@1 1+1(74546956+68114896 bytes) to 4+1 (298185581+68114896bytes) > > -3764> 2018-07-11 07:51:48.428046 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 33] Compacting 4@1 + 1@2 files to L2, score 2.20 > -3763> 2018-07-11 07:51:48.428057 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 32 Base level 1, inputs: [36572(71MB) 36573(71MB) 36574(71MB) 36575(71MB)], [36290(64MB)] > > -3762> 2018-07-11 07:51:48.428071 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295508428061, "job": 33, "event": "compaction_started", "files_L1": [36572, 36573, 36574, 36575], "files_L2": [36290], "score": 2.19535, "input_data_size": 366300477} > -3761> 2018-07-11 07:51:48.468354 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15255 conn(0x55d986d20000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.0 seq 255 0x55d9c3383e00 osd_ping(ping e501 stamp 2018-07-11 07:51:49.699614) v4 > -3760> 2018-07-11 07:51:48.468376 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15255 conn(0x55d986c82800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.0 seq 255 0x55d9c3383c00 osd_ping(ping e501 stamp 2018-07-11 07:51:49.699614) v4 > -3759> 2018-07-11 07:51:48.468385 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.0 10.8.128.113:0/15255 255 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:49.699614) v4 ==== 2004+0+0 (2892618752 0 0) 0x55d9c3383e00 con 0x55d986d20000 > -3758> 2018-07-11 07:51:48.468393 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.0 10.8.128.113:0/15255 255 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:49.699614) v4 ==== 2004+0+0 (2892618752 0 0) 0x55d9c3383c00 con 0x55d986c82800 > -3757> 2018-07-11 07:51:48.468397 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:49.699614) v4 -- 0x55d9829e5600 con 0 > -3756> 2018-07-11 07:51:48.468428 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:49.699614) v4 -- 0x55d9869bf400 con 0 > -3755> 2018-07-11 07:51:48.541641 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 984 0x55d9bf777b00 rep_scrubmap(5.2e e501 from shard 2) v1 > -3754> 2018-07-11 07:51:48.541661 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 984 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12466 (1269249638 0 2666241036) 0x55d9bf777b00 con 0x55d986bf0800 > -3753> 2018-07-11 07:51:48.557456 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 978 0x55d9bf777d40 rep_scrubmap(5.2e e501 from shard 4) v1 > -3752> 2018-07-11 07:51:48.557475 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 978 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12466 (3686815630 0 2666241036) 0x55d9bf777d40 con 0x55d986bec000 > -3751> 2018-07-11 07:51:48.566282 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6805/15885 conn(0x55d986c7f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.7 seq 2810 0x55d986d96800 MOSDScrubReserve(5.4 REQUEST e501) v1 > -3750> 2018-07-11 07:51:48.566300 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.7 10.8.128.114:6805/15885 2810 ==== MOSDScrubReserve(5.4 REQUEST e501) v1 ==== 43+0+0 (1753221512 0 0) 0x55d986d96800 con 0x55d986c7f800 > -3749> 2018-07-11 07:51:48.566428 7ffab33df700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.114:6805/15885 -- MOSDScrubReserve(5.4 REJECT e501) v1 -- 0x55d9869bfa00 con 0 > -3748> 2018-07-11 07:51:48.566679 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6805/15885 conn(0x55d986c7f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.7 seq 2811 0x55d9829e5600 MOSDScrubReserve(5.4 RELEASE e501) v1 > -3747> 2018-07-11 07:51:48.566694 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.7 10.8.128.114:6805/15885 2811 ==== MOSDScrubReserve(5.4 RELEASE e501) v1 ==== 43+0+0 (2812146570 0 0) 0x55d9829e5600 con 0x55d986c7f800 > -3746> 2018-07-11 07:51:48.717545 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12650 conn(0x55d986d23000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 247 0x55d982cdf800 osd_ping(ping e501 stamp 2018-07-11 07:51:48.765247) v4 > -3745> 2018-07-11 07:51:48.717569 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.3 10.8.128.106:0/12650 247 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:48.765247) v4 ==== 2004+0+0 (398981594 0 0) 0x55d982cdf800 con 0x55d986d23000 > -3744> 2018-07-11 07:51:48.717579 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:48.765247) v4 -- 0x55d9869bf400 con 0 > -3743> 2018-07-11 07:51:48.717619 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12650 conn(0x55d986d21800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 247 0x55d986cd6a00 osd_ping(ping e501 stamp 2018-07-11 07:51:48.765247) v4 > -3742> 2018-07-11 07:51:48.717641 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.3 10.8.128.106:0/12650 247 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:48.765247) v4 ==== 2004+0+0 (398981594 0 0) 0x55d986cd6a00 con 0x55d986d21800 > -3741> 2018-07-11 07:51:48.717652 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:48.765247) v4 -- 0x55d9bf805e00 con 0 > -3740> 2018-07-11 07:51:48.720465 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.114:0/15885 conn(0x55d986d7f800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.7 seq 262 0x55d9c3383c00 osd_ping(ping e501 stamp 2018-07-11 07:51:49.350793) v4 > -3739> 2018-07-11 07:51:48.720493 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.7 10.8.128.114:0/15885 262 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:49.350793) v4 ==== 2004+0+0 (3171076132 0 0) 0x55d9c3383c00 con 0x55d986d7f800 > -3738> 2018-07-11 07:51:48.720508 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:49.350793) v4 -- 0x55d9869bf400 con 0 > -3737> 2018-07-11 07:51:48.720507 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.114:0/15885 conn(0x55d986d81000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.7 seq 262 0x55d9c3383e00 osd_ping(ping e501 stamp 2018-07-11 07:51:49.350793) v4 > -3736> 2018-07-11 07:51:48.720531 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.7 10.8.128.114:0/15885 262 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:49.350793) v4 ==== 2004+0+0 (3171076132 0 0) 0x55d9c3383e00 con 0x55d986d81000 > -3735> 2018-07-11 07:51:48.720542 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:49.350793) v4 -- 0x55d9829e4600 con 0 > -3734> 2018-07-11 07:51:48.841037 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 3012 0x55d986d97e00 MOSDScrubReserve(11.3s5 REQUEST e501) v1 > -3733> 2018-07-11 07:51:48.841054 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 3012 ==== MOSDScrubReserve(11.3s5 REQUEST e501) v1 ==== 43+0+0 (3260170403 0 0) 0x55d986d97e00 con 0x55d986c7e000 > -3732> 2018-07-11 07:51:48.841177 7ffab3be0700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e501) v1 -- 0x55d9869be000 con 0 > -3731> 2018-07-11 07:51:48.841470 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 3013 0x55d9bf805e00 MOSDScrubReserve(11.3s5 RELEASE e501) v1 > -3730> 2018-07-11 07:51:48.841485 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 3013 ==== MOSDScrubReserve(11.3s5 RELEASE e501) v1 ==== 43+0+0 (223328929 0 0) 0x55d9bf805e00 con 0x55d986c7e000 > -3729> 2018-07-11 07:51:48.964237 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6801/15919 conn(0x55d986c3f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=29 cs=1 l=0). rx osd.11 seq 244 0x55d986cd6a00 MOSDScrubReserve(10.f REQUEST e501) v1 > -3728> 2018-07-11 07:51:48.964262 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.11 10.8.128.114:6801/15919 244 ==== MOSDScrubReserve(10.f REQUEST e501) v1 ==== 43+0+0 (2892966580 0 0) 0x55d986cd6a00 con 0x55d986c3f800 > -3727> 2018-07-11 07:51:48.964384 7ffab53e3700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.114:6801/15919 -- MOSDScrubReserve(10.f REJECT e501) v1 -- 0x55d9b4388200 con 0 > -3726> 2018-07-11 07:51:48.964691 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6801/15919 conn(0x55d986c3f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=29 cs=1 l=0). rx osd.11 seq 245 0x55d9b4388200 MOSDScrubReserve(10.f RELEASE e501) v1 > -3725> 2018-07-11 07:51:48.964719 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.11 10.8.128.114:6801/15919 245 ==== MOSDScrubReserve(10.f RELEASE e501) v1 ==== 43+0+0 (1668468918 0 0) 0x55d9b4388200 con 0x55d986c3f800 > -3724> 2018-07-11 07:51:48.974462 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -3723> 2018-07-11 07:51:48.974491 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -3722> 2018-07-11 07:51:48.974535 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -3721> 2018-07-11 07:51:48.974987 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -3720> 2018-07-11 07:51:48.986355 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15287 conn(0x55d98123f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 268 0x55d982cdf800 osd_ping(ping e501 stamp 2018-07-11 07:51:50.217493) v4 > -3719> 2018-07-11 07:51:48.986367 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15287 conn(0x55d981240800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 268 0x55d9869bfa00 osd_ping(ping e501 stamp 2018-07-11 07:51:50.217493) v4 > -3718> 2018-07-11 07:51:48.986388 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.4 10.8.128.113:0/15287 268 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:50.217493) v4 ==== 2004+0+0 (70499679 0 0) 0x55d9869bfa00 con 0x55d981240800 > -3717> 2018-07-11 07:51:48.986386 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.4 10.8.128.113:0/15287 268 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:50.217493) v4 ==== 2004+0+0 (70499679 0 0) 0x55d982cdf800 con 0x55d98123f000 > -3716> 2018-07-11 07:51:48.986404 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:50.217493) v4 -- 0x55d9829e4600 con 0 > -3715> 2018-07-11 07:51:48.986435 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:50.217493) v4 -- 0x55d9869bf400 con 0 > -3714> 2018-07-11 07:51:49.033471 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75620ad6:::benchmark_data_magna115_10867_object73195:0,end:5:756278ea:::benchmark_data_magna115_10867_object2204434:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98696f900 con 0 > -3713> 2018-07-11 07:51:49.033513 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75620ad6:::benchmark_data_magna115_10867_object73195:0,end:5:756278ea:::benchmark_data_magna115_10867_object2204434:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98696f680 con 0 > -3712> 2018-07-11 07:51:49.251563 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 985 0x55d984afa000 rep_scrubmap(5.2e e501 from shard 2) v1 > -3711> 2018-07-11 07:51:49.251583 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 985 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12476 (1269249638 0 1259370029) 0x55d984afa000 con 0x55d986bf0800 > -3710> 2018-07-11 07:51:49.266109 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 979 0x55d984afa240 rep_scrubmap(5.2e e501 from shard 4) v1 > -3709> 2018-07-11 07:51:49.266128 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 979 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12476 (3686815630 0 1259370029) 0x55d984afa240 con 0x55d986bec000 > -3708> 2018-07-11 07:51:49.420148 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -3707> 2018-07-11 07:51:49.420175 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -3706> 2018-07-11 07:51:49.420235 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -3705> 2018-07-11 07:51:49.420717 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -3704> 2018-07-11 07:51:49.441970 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67766,epoch:501/498,start:5:756278ea:::benchmark_data_magna115_10867_object2204434:0,end:5:7562c8f2:::benchmark_data_magna115_10867_object2619580:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98696ef00 con 0 > -3703> 2018-07-11 07:51:49.442017 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67766,epoch:501/498,start:5:756278ea:::benchmark_data_magna115_10867_object2204434:0,end:5:7562c8f2:::benchmark_data_magna115_10867_object2619580:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98696ec80 con 0 > -3702> 2018-07-11 07:51:49.655044 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 986 0x55d984afa480 rep_scrubmap(5.2e e501 from shard 2) v1 > -3701> 2018-07-11 07:51:49.655066 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 986 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12482 (1269249638 0 2247472097) 0x55d984afa480 con 0x55d986bf0800 > -3700> 2018-07-11 07:51:49.669573 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 980 0x55d984afa6c0 rep_scrubmap(5.2e e501 from shard 4) v1 > -3699> 2018-07-11 07:51:49.669594 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 980 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12482 (3686815630 0 2247472097) 0x55d984afa6c0 con 0x55d986bec000 > -3698> 2018-07-11 07:51:49.814705 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -3697> 2018-07-11 07:51:49.814731 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -3696> 2018-07-11 07:51:49.814791 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -3695> 2018-07-11 07:51:49.815278 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -3694> 2018-07-11 07:51:49.833626 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7562c8f2:::benchmark_data_magna115_10867_object2619580:0,end:5:75633093:::benchmark_data_magna115_10867_object603563:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b14f0a00 con 0 > -3693> 2018-07-11 07:51:49.833674 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7562c8f2:::benchmark_data_magna115_10867_object2619580:0,end:5:75633093:::benchmark_data_magna115_10867_object603563:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1499400 con 0 > -3692> 2018-07-11 07:51:50.040769 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 987 0x55d984afa900 rep_scrubmap(5.2e e501 from shard 2) v1 > -3691> 2018-07-11 07:51:50.040791 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 987 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12470 (1269249638 0 248603104) 0x55d984afa900 con 0x55d986bf0800 > -3690> 2018-07-11 07:51:50.064857 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 981 0x55d984afab40 rep_scrubmap(5.2e e501 from shard 4) v1 > -3689> 2018-07-11 07:51:50.064877 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 981 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12470 (3686815630 0 248603104) 0x55d984afab40 con 0x55d986bec000 > -3688> 2018-07-11 07:51:50.206861 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -3687> 2018-07-11 07:51:50.206887 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -3686> 2018-07-11 07:51:50.206935 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -3685> 2018-07-11 07:51:50.207425 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -3684> 2018-07-11 07:51:50.258830 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75633093:::benchmark_data_magna115_10867_object603563:0,end:5:75638fad:::benchmark_data_magna115_10867_object3813978:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498f00 con 0 > -3683> 2018-07-11 07:51:50.258882 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75633093:::benchmark_data_magna115_10867_object603563:0,end:5:75638fad:::benchmark_data_magna115_10867_object3813978:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1499b80 con 0 > -3682> 2018-07-11 07:51:50.483814 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 982 0x55d984afad80 rep_scrubmap(5.2e e501 from shard 4) v1 > -3681> 2018-07-11 07:51:50.483835 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 982 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12476 (3686815630 0 19778024) 0x55d984afad80 con 0x55d986bec000 > -3680> 2018-07-11 07:51:50.490067 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 988 0x55d984afafc0 rep_scrubmap(5.2e e501 from shard 2) v1 > -3679> 2018-07-11 07:51:50.490088 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 988 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12476 (1269249638 0 19778024) 0x55d984afafc0 con 0x55d986bf0800 > -3678> 2018-07-11 07:51:50.588324 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -3677> 2018-07-11 07:51:50.588351 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -3676> 2018-07-11 07:51:50.588398 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -3675> 2018-07-11 07:51:50.588862 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -3674> 2018-07-11 07:51:50.608757 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75638fad:::benchmark_data_magna115_10867_object3813978:0,end:5:7563e61f:::benchmark_data_magna115_10867_object1583416:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1499180 con 0 > -3673> 2018-07-11 07:51:50.608806 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75638fad:::benchmark_data_magna115_10867_object3813978:0,end:5:7563e61f:::benchmark_data_magna115_10867_object1583416:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498c80 con 0 > -3672> 2018-07-11 07:51:50.686843 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15287 conn(0x55d981240800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 269 0x55d9869bfa00 osd_ping(ping e501 stamp 2018-07-11 07:51:51.918087) v4 > -3671> 2018-07-11 07:51:50.686850 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15287 conn(0x55d98123f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 269 0x55d9c3383c00 osd_ping(ping e501 stamp 2018-07-11 07:51:51.918087) v4 > -3670> 2018-07-11 07:51:50.686882 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.4 10.8.128.113:0/15287 269 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:51.918087) v4 ==== 2004+0+0 (4166390813 0 0) 0x55d9869bfa00 con 0x55d981240800 > -3669> 2018-07-11 07:51:50.686884 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.4 10.8.128.113:0/15287 269 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:51.918087) v4 ==== 2004+0+0 (4166390813 0 0) 0x55d9c3383c00 con 0x55d98123f000 > -3668> 2018-07-11 07:51:50.686898 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:51.918087) v4 -- 0x55d9829e4600 con 0 > -3667> 2018-07-11 07:51:50.686925 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:51.918087) v4 -- 0x55d9869bf400 con 0 > -3666> 2018-07-11 07:51:50.768899 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15255 conn(0x55d986c82800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.0 seq 256 0x55d982cdf800 osd_ping(ping e501 stamp 2018-07-11 07:51:52.000045) v4 > -3665> 2018-07-11 07:51:50.768898 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15255 conn(0x55d986d20000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.0 seq 256 0x55d9c3383e00 osd_ping(ping e501 stamp 2018-07-11 07:51:52.000045) v4 > -3664> 2018-07-11 07:51:50.768921 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.0 10.8.128.113:0/15255 256 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:52.000045) v4 ==== 2004+0+0 (1652698812 0 0) 0x55d982cdf800 con 0x55d986c82800 > -3663> 2018-07-11 07:51:50.768922 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.0 10.8.128.113:0/15255 256 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:52.000045) v4 ==== 2004+0+0 (1652698812 0 0) 0x55d9c3383e00 con 0x55d986d20000 > -3662> 2018-07-11 07:51:50.768930 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:52.000045) v4 -- 0x55d9869bf400 con 0 > -3661> 2018-07-11 07:51:50.768956 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:52.000045) v4 -- 0x55d9829e4600 con 0 > -3660> 2018-07-11 07:51:50.821832 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 989 0x55d984afb200 rep_scrubmap(5.2e e501 from shard 2) v1 > -3659> 2018-07-11 07:51:50.821852 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 989 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12476 (1269249638 0 442735675) 0x55d984afb200 con 0x55d986bf0800 > -3658> 2018-07-11 07:51:50.838470 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 983 0x55d984afb440 rep_scrubmap(5.2e e501 from shard 4) v1 > -3657> 2018-07-11 07:51:50.838489 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 983 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12476 (3686815630 0 442735675) 0x55d984afb440 con 0x55d986bec000 > -3656> 2018-07-11 07:51:50.849280 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15263 conn(0x55d985d45800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.5 seq 238 0x55d9c3383c00 osd_ping(ping e501 stamp 2018-07-11 07:51:52.080525) v4 > -3655> 2018-07-11 07:51:50.849281 7ffacbbd1700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15263 conn(0x55d985d44000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.5 seq 238 0x55d9869be000 osd_ping(ping e501 stamp 2018-07-11 07:51:52.080525) v4 > -3654> 2018-07-11 07:51:50.849306 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.5 10.8.128.113:0/15263 238 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:52.080525) v4 ==== 2004+0+0 (2249797917 0 0) 0x55d9c3383c00 con 0x55d985d45800 > -3653> 2018-07-11 07:51:50.849309 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 <== osd.5 10.8.128.113:0/15263 238 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:52.080525) v4 ==== 2004+0+0 (2249797917 0 0) 0x55d9869be000 con 0x55d985d44000 > -3652> 2018-07-11 07:51:50.849318 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:52.080525) v4 -- 0x55d9869bf400 con 0 > -3651> 2018-07-11 07:51:50.849343 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:52.080525) v4 -- 0x55d986d33200 con 0 > -3650> 2018-07-11 07:51:50.935969 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -3649> 2018-07-11 07:51:50.935992 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -3648> 2018-07-11 07:51:50.936048 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -3647> 2018-07-11 07:51:50.936500 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -3646> 2018-07-11 07:51:50.976138 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'68011,epoch:501/498,start:5:7563e61f:::benchmark_data_magna115_10867_object1583416:0,end:5:756445ec:::benchmark_data_magna115_10867_object3754513:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1499900 con 0 > -3645> 2018-07-11 07:51:50.976190 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'68011,epoch:501/498,start:5:7563e61f:::benchmark_data_magna115_10867_object1583416:0,end:5:756445ec:::benchmark_data_magna115_10867_object3754513:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498500 con 0 > -3644> 2018-07-11 07:51:51.120171 7ffab1bdc700 5 osd.10 501 heartbeat: osd_stat(788 GB used, 142 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist []) > -3643> 2018-07-11 07:51:51.120186 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6802/15255 -- osd_ping(ping e501 stamp 2018-07-11 07:51:51.120184) v4 -- 0x55d9ba819a00 con 0 > -3642> 2018-07-11 07:51:51.120224 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6803/15255 -- osd_ping(ping e501 stamp 2018-07-11 07:51:51.120184) v4 -- 0x55d9c34cc000 con 0 > -3641> 2018-07-11 07:51:51.120241 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6806/12677 -- osd_ping(ping e501 stamp 2018-07-11 07:51:51.120184) v4 -- 0x55d9b437ac00 con 0 > -3640> 2018-07-11 07:51:51.120271 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6807/12677 -- osd_ping(ping e501 stamp 2018-07-11 07:51:51.120184) v4 -- 0x55d986b6d400 con 0 > -3639> 2018-07-11 07:51:51.120282 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6808/12651 -- osd_ping(ping e501 stamp 2018-07-11 07:51:51.120184) v4 -- 0x55d9c4c1be00 con 0 > -3638> 2018-07-11 07:51:51.120291 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6810/12651 -- osd_ping(ping e501 stamp 2018-07-11 07:51:51.120184) v4 -- 0x55d9bdba5e00 con 0 > -3637> 2018-07-11 07:51:51.120310 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6809/12650 -- osd_ping(ping e501 stamp 2018-07-11 07:51:51.120184) v4 -- 0x55d986d4e400 con 0 > -3636> 2018-07-11 07:51:51.120323 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6811/12650 -- osd_ping(ping e501 stamp 2018-07-11 07:51:51.120184) v4 -- 0x55d982cdea00 con 0 > -3635> 2018-07-11 07:51:51.120335 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6810/15287 -- osd_ping(ping e501 stamp 2018-07-11 07:51:51.120184) v4 -- 0x55d9c9123000 con 0 > -3634> 2018-07-11 07:51:51.120348 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6811/15287 -- osd_ping(ping e501 stamp 2018-07-11 07:51:51.120184) v4 -- 0x55d9829e5a00 con 0 > -3633> 2018-07-11 07:51:51.120361 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6806/15263 -- osd_ping(ping e501 stamp 2018-07-11 07:51:51.120184) v4 -- 0x55d9b60df600 con 0 > -3632> 2018-07-11 07:51:51.120374 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6807/15263 -- osd_ping(ping e501 stamp 2018-07-11 07:51:51.120184) v4 -- 0x55d980457a00 con 0 > -3631> 2018-07-11 07:51:51.120386 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.115:6802/13323 -- osd_ping(ping e501 stamp 2018-07-11 07:51:51.120184) v4 -- 0x55d9b60dfa00 con 0 > -3630> 2018-07-11 07:51:51.120399 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.115:6803/13323 -- osd_ping(ping e501 stamp 2018-07-11 07:51:51.120184) v4 -- 0x55d982cde200 con 0 > -3629> 2018-07-11 07:51:51.120414 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6806/15885 -- osd_ping(ping e501 stamp 2018-07-11 07:51:51.120184) v4 -- 0x55d9b9ac3400 con 0 > -3628> 2018-07-11 07:51:51.120427 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6807/15885 -- osd_ping(ping e501 stamp 2018-07-11 07:51:51.120184) v4 -- 0x55d9869bfe00 con 0 > -3627> 2018-07-11 07:51:51.120439 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.108:6802/11967 -- osd_ping(ping e501 stamp 2018-07-11 07:51:51.120184) v4 -- 0x55d9b67d9200 con 0 > -3626> 2018-07-11 07:51:51.120450 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.108:6803/11967 -- osd_ping(ping e501 stamp 2018-07-11 07:51:51.120184) v4 -- 0x55d9b437a800 con 0 > -3625> 2018-07-11 07:51:51.120462 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.107:6806/12617 -- osd_ping(ping e501 stamp 2018-07-11 07:51:51.120184) v4 -- 0x55d986d33600 con 0 > -3624> 2018-07-11 07:51:51.120475 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.107:6807/12617 -- osd_ping(ping e501 stamp 2018-07-11 07:51:51.120184) v4 -- 0x55d9bcd78600 con 0 > -3623> 2018-07-11 07:51:51.120488 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6802/15919 -- osd_ping(ping e501 stamp 2018-07-11 07:51:51.120184) v4 -- 0x55d9c3383200 con 0 > -3622> 2018-07-11 07:51:51.120502 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6803/15919 -- osd_ping(ping e501 stamp 2018-07-11 07:51:51.120184) v4 -- 0x55d982d30600 con 0 > -3621> 2018-07-11 07:51:51.120682 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.107:6806/12617 conn(0x55d986bd2000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.9 seq 247 0x55d9c3383c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:51.120184) v4 > -3620> 2018-07-11 07:51:51.120704 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.9 10.8.128.107:6806/12617 247 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:51.120184) v4 ==== 2004+0+0 (1521290819 0 0) 0x55d9c3383c00 con 0x55d986bd2000 > -3619> 2018-07-11 07:51:51.120686 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.107:6807/12617 conn(0x55d986bd3800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.9 seq 247 0x55d9b4388400 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:51.120184) v4 > -3618> 2018-07-11 07:51:51.120731 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.9 10.8.128.107:6807/12617 247 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:51.120184) v4 ==== 2004+0+0 (1521290819 0 0) 0x55d9b4388400 con 0x55d986bd3800 > -3617> 2018-07-11 07:51:51.120744 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6803/15255 conn(0x55d986bc9800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.0 seq 247 0x55d9c3383c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:51.120184) v4 > -3616> 2018-07-11 07:51:51.120778 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.0 10.8.128.113:6803/15255 247 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:51.120184) v4 ==== 2004+0+0 (1521290819 0 0) 0x55d9c3383c00 con 0x55d986bc9800 > -3615> 2018-07-11 07:51:51.120788 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6809/12650 conn(0x55d986bcb000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 247 0x55d9b4388400 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:51.120184) v4 > -3614> 2018-07-11 07:51:51.120817 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6810/12651 conn(0x55d986b9b800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.2 seq 247 0x55d9c3383c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:51.120184) v4 > -3613> 2018-07-11 07:51:51.120819 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.3 10.8.128.106:6809/12650 247 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:51.120184) v4 ==== 2004+0+0 (1521290819 0 0) 0x55d9b4388400 con 0x55d986bcb000 > -3612> 2018-07-11 07:51:51.120825 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.2 10.8.128.106:6810/12651 247 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:51.120184) v4 ==== 2004+0+0 (1521290819 0 0) 0x55d9c3383c00 con 0x55d986b9b800 > -3611> 2018-07-11 07:51:51.120822 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6802/15255 conn(0x55d986bc8000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.0 seq 247 0x55d9869bfa00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:51.120184) v4 > -3610> 2018-07-11 07:51:51.120836 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.0 10.8.128.113:6802/15255 247 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:51.120184) v4 ==== 2004+0+0 (1521290819 0 0) 0x55d9869bfa00 con 0x55d986bc8000 > -3609> 2018-07-11 07:51:51.120868 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6806/15263 conn(0x55d986b9d000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.5 seq 247 0x55d9b4388400 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:51.120184) v4 > -3608> 2018-07-11 07:51:51.120886 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.5 10.8.128.113:6806/15263 247 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:51.120184) v4 ==== 2004+0+0 (1521290819 0 0) 0x55d9b4388400 con 0x55d986b9d000 > -3607> 2018-07-11 07:51:51.120884 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6810/15287 conn(0x55d986baf000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 247 0x55d9869bfa00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:51.120184) v4 > -3606> 2018-07-11 07:51:51.120897 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.4 10.8.128.113:6810/15287 247 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:51.120184) v4 ==== 2004+0+0 (1521290819 0 0) 0x55d9869bfa00 con 0x55d986baf000 > -3605> 2018-07-11 07:51:51.120928 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6808/12651 conn(0x55d986b9a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.2 seq 247 0x55d9869bfa00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:51.120184) v4 > -3604> 2018-07-11 07:51:51.120940 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.2 10.8.128.106:6808/12651 247 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:51.120184) v4 ==== 2004+0+0 (1521290819 0 0) 0x55d9869bfa00 con 0x55d986b9a000 > -3603> 2018-07-11 07:51:51.120955 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6811/15287 conn(0x55d986bb0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 247 0x55d9c3383c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:51.120184) v4 > -3602> 2018-07-11 07:51:51.120974 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.4 10.8.128.113:6811/15287 247 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:51.120184) v4 ==== 2004+0+0 (1521290819 0 0) 0x55d9c3383c00 con 0x55d986bb0800 > -3601> 2018-07-11 07:51:51.120972 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6811/12650 conn(0x55d986bcc800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 247 0x55d9869bfa00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:51.120184) v4 > -3600> 2018-07-11 07:51:51.120990 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.3 10.8.128.106:6811/12650 247 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:51.120184) v4 ==== 2004+0+0 (1521290819 0 0) 0x55d9869bfa00 con 0x55d986bcc800 > -3599> 2018-07-11 07:51:51.120994 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6806/12677 conn(0x55d986b58000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.1 seq 247 0x55d9b4388400 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:51.120184) v4 > -3598> 2018-07-11 07:51:51.121014 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.115:6802/13323 conn(0x55d986bc1000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 247 0x55d9c3383c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:51.120184) v4 > -3597> 2018-07-11 07:51:51.121016 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.1 10.8.128.106:6806/12677 247 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:51.120184) v4 ==== 2004+0+0 (1521290819 0 0) 0x55d9b4388400 con 0x55d986b58000 > -3596> 2018-07-11 07:51:51.121020 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6807/12677 conn(0x55d986b59800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.1 seq 247 0x55d9869bfa00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:51.120184) v4 > -3595> 2018-07-11 07:51:51.121027 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.6 10.8.128.115:6802/13323 247 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:51.120184) v4 ==== 2004+0+0 (1521290819 0 0) 0x55d9c3383c00 con 0x55d986bc1000 > -3594> 2018-07-11 07:51:51.121030 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.1 10.8.128.106:6807/12677 247 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:51.120184) v4 ==== 2004+0+0 (1521290819 0 0) 0x55d9869bfa00 con 0x55d986b59800 > -3593> 2018-07-11 07:51:51.121059 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6802/15919 conn(0x55d986b5b000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.11 seq 247 0x55d9c3383c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:51.120184) v4 > -3592> 2018-07-11 07:51:51.121071 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.11 10.8.128.114:6802/15919 247 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:51.120184) v4 ==== 2004+0+0 (1521290819 0 0) 0x55d9c3383c00 con 0x55d986b5b000 > -3591> 2018-07-11 07:51:51.121073 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6807/15263 conn(0x55d986b9e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.5 seq 247 0x55d9869bfa00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:51.120184) v4 > -3590> 2018-07-11 07:51:51.121086 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.5 10.8.128.113:6807/15263 247 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:51.120184) v4 ==== 2004+0+0 (1521290819 0 0) 0x55d9869bfa00 con 0x55d986b9e800 > -3589> 2018-07-11 07:51:51.121089 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6803/15919 conn(0x55d986b5c800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.11 seq 247 0x55d9b4388400 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:51.120184) v4 > -3588> 2018-07-11 07:51:51.121105 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.11 10.8.128.114:6803/15919 247 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:51.120184) v4 ==== 2004+0+0 (1521290819 0 0) 0x55d9b4388400 con 0x55d986b5c800 > -3587> 2018-07-11 07:51:51.121119 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.108:6803/11967 conn(0x55d986bbf800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.8 seq 247 0x55d9869bfa00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:51.120184) v4 > -3586> 2018-07-11 07:51:51.121132 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.8 10.8.128.108:6803/11967 247 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:51.120184) v4 ==== 2004+0+0 (1521290819 0 0) 0x55d9869bfa00 con 0x55d986bbf800 > -3585> 2018-07-11 07:51:51.121141 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.115:6803/13323 conn(0x55d986bc2800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 247 0x55d9b4388400 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:51.120184) v4 > -3584> 2018-07-11 07:51:51.121153 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.6 10.8.128.115:6803/13323 247 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:51.120184) v4 ==== 2004+0+0 (1521290819 0 0) 0x55d9b4388400 con 0x55d986bc2800 > -3583> 2018-07-11 07:51:51.121180 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6807/15885 conn(0x55d986bad800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.7 seq 247 0x55d9b4388400 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:51.120184) v4 > -3582> 2018-07-11 07:51:51.121190 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.7 10.8.128.114:6807/15885 247 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:51.120184) v4 ==== 2004+0+0 (1521290819 0 0) 0x55d9b4388400 con 0x55d986bad800 > -3581> 2018-07-11 07:51:51.121186 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6806/15885 conn(0x55d986bac000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.7 seq 247 0x55d9c3383c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:51.120184) v4 > -3580> 2018-07-11 07:51:51.121205 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.7 10.8.128.114:6806/15885 247 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:51.120184) v4 ==== 2004+0+0 (1521290819 0 0) 0x55d9c3383c00 con 0x55d986bac000 > -3579> 2018-07-11 07:51:51.121220 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.108:6802/11967 conn(0x55d986bbe000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.8 seq 247 0x55d9b4388400 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:51.120184) v4 > -3578> 2018-07-11 07:51:51.121232 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.8 10.8.128.108:6802/11967 247 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:51.120184) v4 ==== 2004+0+0 (1521290819 0 0) 0x55d9b4388400 con 0x55d986bbe000 > -3577> 2018-07-11 07:51:51.200527 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 984 0x55d984afb680 rep_scrubmap(5.2e e501 from shard 4) v1 > -3576> 2018-07-11 07:51:51.200546 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 984 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12478 (3686815630 0 2334158172) 0x55d984afb680 con 0x55d986bec000 > -3575> 2018-07-11 07:51:51.216841 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 990 0x55d984afb8c0 rep_scrubmap(5.2e e501 from shard 2) v1 > -3574> 2018-07-11 07:51:51.216860 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 990 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12478 (1269249638 0 2334158172) 0x55d984afb8c0 con 0x55d986bf0800 > -3573> 2018-07-11 07:51:51.225862 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -3572> 2018-07-11 07:51:51.225885 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -3571> 2018-07-11 07:51:51.225942 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -3570> 2018-07-11 07:51:51.226390 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -3569> 2018-07-11 07:51:51.275661 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'66937,epoch:501/498,start:5:756445ec:::benchmark_data_magna115_10867_object3754513:0,end:5:7564c00a:::benchmark_data_magna115_10867_object2443978:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1499680 con 0 > -3568> 2018-07-11 07:51:51.275713 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'66937,epoch:501/498,start:5:756445ec:::benchmark_data_magna115_10867_object3754513:0,end:5:7564c00a:::benchmark_data_magna115_10867_object2443978:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b14f0000 con 0 > -3567> 2018-07-11 07:51:51.390543 7ffacbbd1700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.108:0/11967 conn(0x55d986d82800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.8 seq 237 0x55d9b4388400 osd_ping(ping e501 stamp 2018-07-11 07:51:51.146256) v4 > -3566> 2018-07-11 07:51:51.390564 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 <== osd.8 10.8.128.108:0/11967 237 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:51.146256) v4 ==== 2004+0+0 (1941491158 0 0) 0x55d9b4388400 con 0x55d986d82800 > -3565> 2018-07-11 07:51:51.390557 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.108:0/11967 conn(0x55d986db0000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.8 seq 237 0x55d9c3383c00 osd_ping(ping e501 stamp 2018-07-11 07:51:51.146256) v4 > -3564> 2018-07-11 07:51:51.390575 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:51.146256) v4 -- 0x55d982d30600 con 0 > -3563> 2018-07-11 07:51:51.390583 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.8 10.8.128.108:0/11967 237 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:51.146256) v4 ==== 2004+0+0 (1941491158 0 0) 0x55d9c3383c00 con 0x55d986db0000 > -3562> 2018-07-11 07:51:51.390599 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:51.146256) v4 -- 0x55d9c3383200 con 0 > -3561> 2018-07-11 07:51:51.456012 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 991 0x55d984afbb00 rep_scrubmap(5.2e e501 from shard 2) v1 > -3560> 2018-07-11 07:51:51.456032 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 991 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12474 (1269249638 0 3001966480) 0x55d984afbb00 con 0x55d986bf0800 > -3559> 2018-07-11 07:51:51.523296 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 985 0x55d984afbd40 rep_scrubmap(5.2e e501 from shard 4) v1 > -3558> 2018-07-11 07:51:51.523316 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 985 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12474 (3686815630 0 3001966480) 0x55d984afbd40 con 0x55d986bec000 > -3557> 2018-07-11 07:51:51.523419 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -3556> 2018-07-11 07:51:51.523444 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -3555> 2018-07-11 07:51:51.523493 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -3554> 2018-07-11 07:51:51.523959 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -3553> 2018-07-11 07:51:51.547929 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7564c00a:::benchmark_data_magna115_10867_object2443978:0,end:5:75650d53:::benchmark_data_magna115_10867_object1406848:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b14f0280 con 0 > -3552> 2018-07-11 07:51:51.547981 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7564c00a:::benchmark_data_magna115_10867_object2443978:0,end:5:75650d53:::benchmark_data_magna115_10867_object1406848:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98696f400 con 0 > -3551> 2018-07-11 07:51:51.767411 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 992 0x55d9b0610000 rep_scrubmap(5.2e e501 from shard 2) v1 > -3550> 2018-07-11 07:51:51.767432 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 992 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12474 (1269249638 0 96562925) 0x55d9b0610000 con 0x55d986bf0800 > -3549> 2018-07-11 07:51:51.779649 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 986 0x55d9b0610240 rep_scrubmap(5.2e e501 from shard 4) v1 > -3548> 2018-07-11 07:51:51.779669 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 986 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12474 (3686815630 0 96562925) 0x55d9b0610240 con 0x55d986bec000 > -3547> 2018-07-11 07:51:51.779772 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -3546> 2018-07-11 07:51:51.779795 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -3545> 2018-07-11 07:51:51.779843 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -3544> 2018-07-11 07:51:51.780332 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -3543> 2018-07-11 07:51:51.787295 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15287 conn(0x55d981240800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 270 0x55d9869bfa00 osd_ping(ping e501 stamp 2018-07-11 07:51:53.018543) v4 > -3542> 2018-07-11 07:51:51.787319 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.4 10.8.128.113:0/15287 270 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:53.018543) v4 ==== 2004+0+0 (116028052 0 0) 0x55d9869bfa00 con 0x55d981240800 > -3541> 2018-07-11 07:51:51.787314 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15287 conn(0x55d98123f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 270 0x55d982cdf800 osd_ping(ping e501 stamp 2018-07-11 07:51:53.018543) v4 > -3540> 2018-07-11 07:51:51.787330 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:53.018543) v4 -- 0x55d9b437a800 con 0 > -3539> 2018-07-11 07:51:51.787334 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.4 10.8.128.113:0/15287 270 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:53.018543) v4 ==== 2004+0+0 (116028052 0 0) 0x55d982cdf800 con 0x55d98123f000 > -3538> 2018-07-11 07:51:51.787355 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:53.018543) v4 -- 0x55d9c3383200 con 0 > -3537> 2018-07-11 07:51:51.826043 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75650d53:::benchmark_data_magna115_10867_object1406848:0,end:5:756573d0:::benchmark_data_magna115_10867_object536890:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498780 con 0 > -3536> 2018-07-11 07:51:51.826093 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75650d53:::benchmark_data_magna115_10867_object1406848:0,end:5:756573d0:::benchmark_data_magna115_10867_object536890:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455680 con 0 > -3535> 2018-07-11 07:51:51.869343 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15255 conn(0x55d986c82800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.0 seq 257 0x55d9c3383c00 osd_ping(ping e501 stamp 2018-07-11 07:51:53.100585) v4 > -3534> 2018-07-11 07:51:51.869366 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.0 10.8.128.113:0/15255 257 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:53.100585) v4 ==== 2004+0+0 (2088528903 0 0) 0x55d9c3383c00 con 0x55d986c82800 > -3533> 2018-07-11 07:51:51.869381 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:53.100585) v4 -- 0x55d9c3383200 con 0 > -3532> 2018-07-11 07:51:51.869374 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15255 conn(0x55d986d20000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.0 seq 257 0x55d9869bfa00 osd_ping(ping e501 stamp 2018-07-11 07:51:53.100585) v4 > -3531> 2018-07-11 07:51:51.869401 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.0 10.8.128.113:0/15255 257 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:53.100585) v4 ==== 2004+0+0 (2088528903 0 0) 0x55d9869bfa00 con 0x55d986d20000 > -3530> 2018-07-11 07:51:51.869412 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:53.100585) v4 -- 0x55d9b437a800 con 0 > -3529> 2018-07-11 07:51:51.963162 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6805/15263 conn(0x55d986c41000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=31 cs=1 l=0). rx osd.5 seq 2830 0x55d9c3383e00 MOSDScrubReserve(6.15 REQUEST e501) v1 > -3528> 2018-07-11 07:51:51.963181 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.5 10.8.128.113:6805/15263 2830 ==== MOSDScrubReserve(6.15 REQUEST e501) v1 ==== 43+0+0 (3942742026 0 0) 0x55d9c3383e00 con 0x55d986c41000 > -3527> 2018-07-11 07:51:52.013263 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 993 0x55d9b0610480 rep_scrubmap(5.2e e501 from shard 2) v1 > -3526> 2018-07-11 07:51:52.013281 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 993 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12478 (1269249638 0 3490051234) 0x55d9b0610480 con 0x55d986bf0800 > -3525> 2018-07-11 07:51:52.036006 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 987 0x55d9b06106c0 rep_scrubmap(5.2e e501 from shard 4) v1 > -3524> 2018-07-11 07:51:52.036026 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 987 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12478 (3686815630 0 3490051234) 0x55d9b06106c0 con 0x55d986bec000 > -3523> 2018-07-11 07:51:52.187766 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6805/15263 -- MOSDScrubReserve(6.15 REJECT e501) v1 -- 0x55d986c22e00 con 0 > -3522> 2018-07-11 07:51:52.187967 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -3521> 2018-07-11 07:51:52.187991 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -3520> 2018-07-11 07:51:52.188041 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -3519> 2018-07-11 07:51:52.188273 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6805/15263 conn(0x55d986c41000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=31 cs=1 l=0). rx osd.5 seq 2831 0x55d986c22e00 MOSDScrubReserve(6.15 RELEASE e501) v1 > -3518> 2018-07-11 07:51:52.188291 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.5 10.8.128.113:6805/15263 2831 ==== MOSDScrubReserve(6.15 RELEASE e501) v1 ==== 43+0+0 (605848072 0 0) 0x55d986c22e00 con 0x55d986c41000 > -3517> 2018-07-11 07:51:52.188512 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -3516> 2018-07-11 07:51:52.189347 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67714,epoch:501/498,start:5:756573d0:::benchmark_data_magna115_10867_object536890:0,end:5:7565ef9a:::benchmark_data_magna115_10867_object2097881:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455900 con 0 > -3515> 2018-07-11 07:51:52.189381 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67714,epoch:501/498,start:5:756573d0:::benchmark_data_magna115_10867_object536890:0,end:5:7565ef9a:::benchmark_data_magna115_10867_object2097881:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498000 con 0 > -3514> 2018-07-11 07:51:52.220929 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.114:0/15885 conn(0x55d986d7f800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.7 seq 263 0x55d982cdf800 osd_ping(ping e501 stamp 2018-07-11 07:51:52.851214) v4 > -3513> 2018-07-11 07:51:52.220948 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.114:0/15885 conn(0x55d986d81000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.7 seq 263 0x55d9869bfa00 osd_ping(ping e501 stamp 2018-07-11 07:51:52.851214) v4 > -3512> 2018-07-11 07:51:52.220957 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.7 10.8.128.114:0/15885 263 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:52.851214) v4 ==== 2004+0+0 (3658405430 0 0) 0x55d982cdf800 con 0x55d986d7f800 > -3511> 2018-07-11 07:51:52.220965 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.7 10.8.128.114:0/15885 263 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:52.851214) v4 ==== 2004+0+0 (3658405430 0 0) 0x55d9869bfa00 con 0x55d986d81000 > -3510> 2018-07-11 07:51:52.220970 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:52.851214) v4 -- 0x55d9c3383200 con 0 > -3509> 2018-07-11 07:51:52.221001 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:52.851214) v4 -- 0x55d980457a00 con 0 > -3508> 2018-07-11 07:51:52.294795 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.114:0/15919 conn(0x55d986df3800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.11 seq 258 0x55d9869be000 osd_ping(ping e501 stamp 2018-07-11 07:51:52.925124) v4 > -3507> 2018-07-11 07:51:52.294816 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.11 10.8.128.114:0/15919 258 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:52.925124) v4 ==== 2004+0+0 (4082956824 0 0) 0x55d9869be000 con 0x55d986df3800 > -3506> 2018-07-11 07:51:52.294826 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:52.925124) v4 -- 0x55d982d30600 con 0 > -3505> 2018-07-11 07:51:52.294846 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.114:0/15919 conn(0x55d986df5000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.11 seq 258 0x55d9c3383c00 osd_ping(ping e501 stamp 2018-07-11 07:51:52.925124) v4 > -3504> 2018-07-11 07:51:52.294867 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.11 10.8.128.114:0/15919 258 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:52.925124) v4 ==== 2004+0+0 (4082956824 0 0) 0x55d9c3383c00 con 0x55d986df5000 > -3503> 2018-07-11 07:51:52.294878 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:52.925124) v4 -- 0x55d9c3383200 con 0 > -3502> 2018-07-11 07:51:52.302113 7ffab7be8700 10 monclient: tick > -3501> 2018-07-11 07:51:52.302139 7ffab7be8700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2018-07-11 07:51:22.302137) > -3500> 2018-07-11 07:51:52.351530 7ffab6be6700 1 -- 10.8.128.107:6800/13927 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55d9ba81b8c0 con 0 > -3499> 2018-07-11 07:51:52.351599 7ffab6be6700 1 -- 10.8.128.107:6800/13927 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55d9869d7340 con 0 > -3498> 2018-07-11 07:51:52.405755 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 994 0x55d9b0610900 rep_scrubmap(5.2e e501 from shard 2) v1 > -3497> 2018-07-11 07:51:52.405775 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 994 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12462 (1269249638 0 2956236238) 0x55d9b0610900 con 0x55d986bf0800 > -3496> 2018-07-11 07:51:52.428770 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 988 0x55d9b0610b40 rep_scrubmap(5.2e e501 from shard 4) v1 > -3495> 2018-07-11 07:51:52.428790 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 988 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12462 (3686815630 0 2956236238) 0x55d9b0610b40 con 0x55d986bec000 > -3494> 2018-07-11 07:51:52.463103 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -3493> 2018-07-11 07:51:52.463127 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -3492> 2018-07-11 07:51:52.463169 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -3491> 2018-07-11 07:51:52.463647 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -3490> 2018-07-11 07:51:52.518003 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7565ef9a:::benchmark_data_magna115_10867_object2097881:0,end:5:756659f5:::benchmark_data_magna115_10867_object3520720:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b14f0500 con 0 > -3489> 2018-07-11 07:51:52.518053 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7565ef9a:::benchmark_data_magna115_10867_object2097881:0,end:5:756659f5:::benchmark_data_magna115_10867_object3520720:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498280 con 0 > -3488> 2018-07-11 07:51:52.666137 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12677 conn(0x55d986db4800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.1 seq 259 0x55d9c3383c00 osd_ping(ping e501 stamp 2018-07-11 07:51:52.714045) v4 > -3487> 2018-07-11 07:51:52.666164 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.1 10.8.128.106:0/12677 259 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:52.714045) v4 ==== 2004+0+0 (948237440 0 0) 0x55d9c3383c00 con 0x55d986db4800 > -3486> 2018-07-11 07:51:52.666164 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12677 conn(0x55d986df2000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.1 seq 259 0x55d9b437a800 osd_ping(ping e501 stamp 2018-07-11 07:51:52.714045) v4 > -3485> 2018-07-11 07:51:52.666176 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:52.714045) v4 -- 0x55d9c3383200 con 0 > -3484> 2018-07-11 07:51:52.666187 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.1 10.8.128.106:0/12677 259 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:52.714045) v4 ==== 2004+0+0 (948237440 0 0) 0x55d9b437a800 con 0x55d986df2000 > -3483> 2018-07-11 07:51:52.666208 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:52.714045) v4 -- 0x55d980457a00 con 0 > -3482> 2018-07-11 07:51:52.704341 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 989 0x55d9b0610d80 rep_scrubmap(5.2e e501 from shard 4) v1 > -3481> 2018-07-11 07:51:52.704362 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 989 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12476 (3686815630 0 4194843303) 0x55d9b0610d80 con 0x55d986bec000 > -3480> 2018-07-11 07:51:52.725813 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 995 0x55d9b0610fc0 rep_scrubmap(5.2e e501 from shard 2) v1 > -3479> 2018-07-11 07:51:52.725833 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 995 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12476 (1269249638 0 4194843303) 0x55d9b0610fc0 con 0x55d986bf0800 > -3478> 2018-07-11 07:51:52.826987 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -3477> 2018-07-11 07:51:52.827014 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -3476> 2018-07-11 07:51:52.827062 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -3475> 2018-07-11 07:51:52.827530 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -3474> 2018-07-11 07:51:52.828409 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:756659f5:::benchmark_data_magna115_10867_object3520720:0,end:5:7566a38e:::benchmark_data_magna115_10867_object2462711:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455b80 con 0 > -3473> 2018-07-11 07:51:52.828445 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:756659f5:::benchmark_data_magna115_10867_object3520720:0,end:5:7566a38e:::benchmark_data_magna115_10867_object2462711:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455400 con 0 > -3472> 2018-07-11 07:51:52.841543 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 3014 0x55d9b4388400 MOSDScrubReserve(11.3s5 REQUEST e501) v1 > -3471> 2018-07-11 07:51:52.841562 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 3014 ==== MOSDScrubReserve(11.3s5 REQUEST e501) v1 ==== 43+0+0 (3260170403 0 0) 0x55d9b4388400 con 0x55d986c7e000 > -3470> 2018-07-11 07:51:52.841626 7ffab3be0700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e501) v1 -- 0x55d9bf805e00 con 0 > -3469> 2018-07-11 07:51:52.842052 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 3015 0x55d9bf805e00 MOSDScrubReserve(11.3s5 RELEASE e501) v1 > -3468> 2018-07-11 07:51:52.842069 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 3015 ==== MOSDScrubReserve(11.3s5 RELEASE e501) v1 ==== 43+0+0 (223328929 0 0) 0x55d9bf805e00 con 0x55d986c7e000 > -3467> 2018-07-11 07:51:52.925852 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12651 conn(0x55d986db1800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.2 seq 256 0x55d9869bfa00 osd_ping(ping e501 stamp 2018-07-11 07:51:52.973713) v4 > -3466> 2018-07-11 07:51:52.925864 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12651 conn(0x55d986db3000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.2 seq 256 0x55d9869be000 osd_ping(ping e501 stamp 2018-07-11 07:51:52.973713) v4 > -3465> 2018-07-11 07:51:52.925874 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.2 10.8.128.106:0/12651 256 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:52.973713) v4 ==== 2004+0+0 (3129840435 0 0) 0x55d9869bfa00 con 0x55d986db1800 > -3464> 2018-07-11 07:51:52.925884 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.2 10.8.128.106:0/12651 256 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:52.973713) v4 ==== 2004+0+0 (3129840435 0 0) 0x55d9869be000 con 0x55d986db3000 > -3463> 2018-07-11 07:51:52.925889 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:52.973713) v4 -- 0x55d980457a00 con 0 > -3462> 2018-07-11 07:51:52.925920 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:52.973713) v4 -- 0x55d9bcd78600 con 0 > -3461> 2018-07-11 07:51:53.019844 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 996 0x55d9b0611200 rep_scrubmap(5.2e e501 from shard 2) v1 > -3460> 2018-07-11 07:51:53.019863 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 996 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12482 (1269249638 0 2594622778) 0x55d9b0611200 con 0x55d986bf0800 > -3459> 2018-07-11 07:51:53.085716 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 990 0x55d9b0611440 rep_scrubmap(5.2e e501 from shard 4) v1 > -3458> 2018-07-11 07:51:53.085738 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 990 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12482 (3686815630 0 2594622778) 0x55d9b0611440 con 0x55d986bec000 > -3457> 2018-07-11 07:51:53.085882 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -3456> 2018-07-11 07:51:53.085907 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -3455> 2018-07-11 07:51:53.085952 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -3454> 2018-07-11 07:51:53.086420 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -3453> 2018-07-11 07:51:53.093196 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7566a38e:::benchmark_data_magna115_10867_object2462711:0,end:5:75670d69:::benchmark_data_magna115_10867_object3254561:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454780 con 0 > -3452> 2018-07-11 07:51:53.093253 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7566a38e:::benchmark_data_magna115_10867_object2462711:0,end:5:75670d69:::benchmark_data_magna115_10867_object3254561:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454f00 con 0 > -3451> 2018-07-11 07:51:53.103209 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.115:0/13323 conn(0x55d986d7e000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 255 0x55d982d30600 osd_ping(ping e501 stamp 2018-07-11 07:51:53.449107) v4 > -3450> 2018-07-11 07:51:53.103230 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.6 10.8.128.115:0/13323 255 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:53.449107) v4 ==== 2004+0+0 (1745523658 0 0) 0x55d982d30600 con 0x55d986d7e000 > -3449> 2018-07-11 07:51:53.103240 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:53.449107) v4 -- 0x55d9bcd78600 con 0 > -3448> 2018-07-11 07:51:53.103306 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.115:0/13323 conn(0x55d986d24800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 255 0x55d9b437a800 osd_ping(ping e501 stamp 2018-07-11 07:51:53.449107) v4 > -3447> 2018-07-11 07:51:53.103332 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.6 10.8.128.115:0/13323 255 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:53.449107) v4 ==== 2004+0+0 (1745523658 0 0) 0x55d9b437a800 con 0x55d986d24800 > -3446> 2018-07-11 07:51:53.103347 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:53.449107) v4 -- 0x55d980457a00 con 0 > -3445> 2018-07-11 07:51:53.300102 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 991 0x55d9b0611680 rep_scrubmap(5.2e e501 from shard 4) v1 > -3444> 2018-07-11 07:51:53.300121 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 991 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12476 (3686815630 0 256932338) 0x55d9b0611680 con 0x55d986bec000 > -3443> 2018-07-11 07:51:53.301703 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 997 0x55d9b06118c0 rep_scrubmap(5.2e e501 from shard 2) v1 > -3442> 2018-07-11 07:51:53.301722 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 997 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12476 (1269249638 0 256932338) 0x55d9b06118c0 con 0x55d986bf0800 > -3441> 2018-07-11 07:51:53.407972 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -3440> 2018-07-11 07:51:53.407997 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -3439> 2018-07-11 07:51:53.408056 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -3438> 2018-07-11 07:51:53.408556 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -3437> 2018-07-11 07:51:53.449381 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67720,epoch:501/498,start:5:75670d69:::benchmark_data_magna115_10867_object3254561:0,end:5:75675272:::benchmark_data_magna115_10867_object3349768:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455180 con 0 > -3436> 2018-07-11 07:51:53.449421 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67720,epoch:501/498,start:5:75670d69:::benchmark_data_magna115_10867_object3254561:0,end:5:75675272:::benchmark_data_magna115_10867_object3349768:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98944b680 con 0 > -3435> 2018-07-11 07:51:53.603630 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.115:0/13323 conn(0x55d986d7e000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 256 0x55d9869be000 osd_ping(ping e501 stamp 2018-07-11 07:51:53.949530) v4 > -3434> 2018-07-11 07:51:53.603656 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.6 10.8.128.115:0/13323 256 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:53.949530) v4 ==== 2004+0+0 (1867063319 0 0) 0x55d9869be000 con 0x55d986d7e000 > -3433> 2018-07-11 07:51:53.603668 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:53.949530) v4 -- 0x55d9bcd78600 con 0 > -3432> 2018-07-11 07:51:53.603684 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.115:0/13323 conn(0x55d986d24800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 256 0x55d9b437a800 osd_ping(ping e501 stamp 2018-07-11 07:51:53.949530) v4 > -3431> 2018-07-11 07:51:53.603707 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.6 10.8.128.115:0/13323 256 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:53.949530) v4 ==== 2004+0+0 (1867063319 0 0) 0x55d9b437a800 con 0x55d986d24800 > -3430> 2018-07-11 07:51:53.603723 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:53.949530) v4 -- 0x55d980457a00 con 0 > -3429> 2018-07-11 07:51:53.669178 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 998 0x55d9b0611b00 rep_scrubmap(5.2e e501 from shard 2) v1 > -3428> 2018-07-11 07:51:53.669198 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 998 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12484 (1269249638 0 1769601280) 0x55d9b0611b00 con 0x55d986bf0800 > -3427> 2018-07-11 07:51:53.679649 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 992 0x55d9b0611d40 rep_scrubmap(5.2e e501 from shard 4) v1 > -3426> 2018-07-11 07:51:53.679668 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 992 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12484 (3686815630 0 1769601280) 0x55d9b0611d40 con 0x55d986bec000 > -3425> 2018-07-11 07:51:53.766619 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12677 conn(0x55d986db4800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.1 seq 260 0x55d982cdf800 osd_ping(ping e501 stamp 2018-07-11 07:51:53.814485) v4 > -3424> 2018-07-11 07:51:53.766621 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12677 conn(0x55d986df2000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.1 seq 260 0x55d9869bfa00 osd_ping(ping e501 stamp 2018-07-11 07:51:53.814485) v4 > -3423> 2018-07-11 07:51:53.766648 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.1 10.8.128.106:0/12677 260 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:53.814485) v4 ==== 2004+0+0 (364441270 0 0) 0x55d982cdf800 con 0x55d986db4800 > -3422> 2018-07-11 07:51:53.766652 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.1 10.8.128.106:0/12677 260 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:53.814485) v4 ==== 2004+0+0 (364441270 0 0) 0x55d9869bfa00 con 0x55d986df2000 > -3421> 2018-07-11 07:51:53.766660 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:53.814485) v4 -- 0x55d9c3383200 con 0 > -3420> 2018-07-11 07:51:53.766698 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:53.814485) v4 -- 0x55d980457a00 con 0 > -3419> 2018-07-11 07:51:53.841999 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 3016 0x55d982d30600 MOSDScrubReserve(11.3s5 REQUEST e501) v1 > -3418> 2018-07-11 07:51:53.842018 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 3016 ==== MOSDScrubReserve(11.3s5 REQUEST e501) v1 ==== 43+0+0 (3260170403 0 0) 0x55d982d30600 con 0x55d986c7e000 > -3417> 2018-07-11 07:51:53.842093 7ffab3be0700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e501) v1 -- 0x55d9bf805e00 con 0 > -3416> 2018-07-11 07:51:53.842499 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 3017 0x55d9bcd78600 MOSDScrubReserve(11.3s5 RELEASE e501) v1 > -3415> 2018-07-11 07:51:53.842543 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 3017 ==== MOSDScrubReserve(11.3s5 RELEASE e501) v1 ==== 43+0+0 (223328929 0 0) 0x55d9bcd78600 con 0x55d986c7e000 > -3414> 2018-07-11 07:51:53.852602 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.107:0/12617 conn(0x55d98123c000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 248 0x55d9b437a800 osd_ping(ping e501 stamp 2018-07-11 07:51:53.851988) v4 > -3413> 2018-07-11 07:51:53.852608 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.107:0/12617 conn(0x55d98123d800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 248 0x55d9869be000 osd_ping(ping e501 stamp 2018-07-11 07:51:53.851988) v4 > -3412> 2018-07-11 07:51:53.852627 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.9 10.8.128.107:0/12617 248 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:53.851988) v4 ==== 2004+0+0 (4280149958 0 0) 0x55d9869be000 con 0x55d98123d800 > -3411> 2018-07-11 07:51:53.852627 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.9 10.8.128.107:0/12617 248 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:53.851988) v4 ==== 2004+0+0 (4280149958 0 0) 0x55d9b437a800 con 0x55d98123c000 > -3410> 2018-07-11 07:51:53.852637 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:53.851988) v4 -- 0x55d9b67d9200 con 0 > -3409> 2018-07-11 07:51:53.852694 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:53.851988) v4 -- 0x55d980457a00 con 0 > -3408> 2018-07-11 07:51:53.964865 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6801/15919 conn(0x55d986c3f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=29 cs=1 l=0). rx osd.11 seq 246 0x55d9bf805e00 MOSDScrubReserve(10.f REQUEST e501) v1 > -3407> 2018-07-11 07:51:53.964881 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.11 10.8.128.114:6801/15919 246 ==== MOSDScrubReserve(10.f REQUEST e501) v1 ==== 43+0+0 (2892966580 0 0) 0x55d9bf805e00 con 0x55d986c3f800 > -3406> 2018-07-11 07:51:53.964927 7ffab53e3700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.114:6801/15919 -- MOSDScrubReserve(10.f REJECT e501) v1 -- 0x55d9b4388200 con 0 > -3405> 2018-07-11 07:51:53.965303 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6801/15919 conn(0x55d986c3f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=29 cs=1 l=0). rx osd.11 seq 247 0x55d9b4388200 MOSDScrubReserve(10.f RELEASE e501) v1 > -3404> 2018-07-11 07:51:53.965318 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.11 10.8.128.114:6801/15919 247 ==== MOSDScrubReserve(10.f RELEASE e501) v1 ==== 43+0+0 (1668468918 0 0) 0x55d9b4388200 con 0x55d986c3f800 > -3403> 2018-07-11 07:51:54.087652 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15287 conn(0x55d98123f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 271 0x55d9c3383c00 osd_ping(ping e501 stamp 2018-07-11 07:51:55.318971) v4 > -3402> 2018-07-11 07:51:54.087672 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.4 10.8.128.113:0/15287 271 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:55.318971) v4 ==== 2004+0+0 (903334331 0 0) 0x55d9c3383c00 con 0x55d98123f000 > -3401> 2018-07-11 07:51:54.087681 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:55.318971) v4 -- 0x55d9c3383200 con 0 > -3400> 2018-07-11 07:51:54.087670 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15287 conn(0x55d981240800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 271 0x55d9869bfa00 osd_ping(ping e501 stamp 2018-07-11 07:51:55.318971) v4 > -3399> 2018-07-11 07:51:54.087688 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.4 10.8.128.113:0/15287 271 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:55.318971) v4 ==== 2004+0+0 (903334331 0 0) 0x55d9869bfa00 con 0x55d981240800 > -3398> 2018-07-11 07:51:54.087701 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:55.318971) v4 -- 0x55d980457a00 con 0 > -3397> 2018-07-11 07:51:54.478371 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 33] Generated table #36807: 2638240 keys, 74546968 bytes > -3396> 2018-07-11 07:51:54.478418 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295514478402, "cf_name": "default", "job": 33, "event": "table_file_creation", "file_number": 36807, "file_size": 74546968, "table_properties": {"data_size": 67112152, "index_size": 690645, "filter_size": 6743181, "raw_key_size": 76215836, "raw_average_key_size": 28, "raw_value_size": 31365756, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} > -3395> 2018-07-11 07:51:54.562859 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -3394> 2018-07-11 07:51:54.562890 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -3393> 2018-07-11 07:51:54.562948 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -3392> 2018-07-11 07:51:54.563426 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -3391> 2018-07-11 07:51:54.585125 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75675272:::benchmark_data_magna115_10867_object3349768:0,end:5:7567b4ca:::benchmark_data_magna115_10867_object3190021:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454500 con 0 > -3390> 2018-07-11 07:51:54.585180 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75675272:::benchmark_data_magna115_10867_object3349768:0,end:5:7567b4ca:::benchmark_data_magna115_10867_object3190021:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb773b80 con 0 > -3389> 2018-07-11 07:51:54.618097 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12650 conn(0x55d986d23000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 248 0x55d982cdf800 osd_ping(ping e501 stamp 2018-07-11 07:51:54.665876) v4 > -3388> 2018-07-11 07:51:54.618120 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.3 10.8.128.106:0/12650 248 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:54.665876) v4 ==== 2004+0+0 (2781991220 0 0) 0x55d982cdf800 con 0x55d986d23000 > -3387> 2018-07-11 07:51:54.618130 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:54.665876) v4 -- 0x55d9c3383200 con 0 > -3386> 2018-07-11 07:51:54.618143 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12650 conn(0x55d986d21800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 248 0x55d9869be000 osd_ping(ping e501 stamp 2018-07-11 07:51:54.665876) v4 > -3385> 2018-07-11 07:51:54.618164 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.3 10.8.128.106:0/12650 248 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:54.665876) v4 ==== 2004+0+0 (2781991220 0 0) 0x55d9869be000 con 0x55d986d21800 > -3384> 2018-07-11 07:51:54.618176 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:54.665876) v4 -- 0x55d9869bfe00 con 0 > -3383> 2018-07-11 07:51:54.620642 7ffab1bdc700 5 osd.10 501 heartbeat: osd_stat(788 GB used, 142 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist []) > -3382> 2018-07-11 07:51:54.620659 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6802/15255 -- osd_ping(ping e501 stamp 2018-07-11 07:51:54.620655) v4 -- 0x55d9bf804400 con 0 > -3381> 2018-07-11 07:51:54.620682 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6803/15255 -- osd_ping(ping e501 stamp 2018-07-11 07:51:54.620655) v4 -- 0x55d9b2528000 con 0 > -3380> 2018-07-11 07:51:54.620699 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6806/12677 -- osd_ping(ping e501 stamp 2018-07-11 07:51:54.620655) v4 -- 0x55d986c85000 con 0 > -3379> 2018-07-11 07:51:54.620714 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6807/12677 -- osd_ping(ping e501 stamp 2018-07-11 07:51:54.620655) v4 -- 0x55d9829e4800 con 0 > -3378> 2018-07-11 07:51:54.620724 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6808/12651 -- osd_ping(ping e501 stamp 2018-07-11 07:51:54.620655) v4 -- 0x55d9bdba4800 con 0 > -3377> 2018-07-11 07:51:54.620732 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6810/12651 -- osd_ping(ping e501 stamp 2018-07-11 07:51:54.620655) v4 -- 0x55d9829e4200 con 0 > -3376> 2018-07-11 07:51:54.620751 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6809/12650 -- osd_ping(ping e501 stamp 2018-07-11 07:51:54.620655) v4 -- 0x55d9c3383600 con 0 > -3375> 2018-07-11 07:51:54.620766 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6811/12650 -- osd_ping(ping e501 stamp 2018-07-11 07:51:54.620655) v4 -- 0x55d9b60df400 con 0 > -3374> 2018-07-11 07:51:54.620778 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6810/15287 -- osd_ping(ping e501 stamp 2018-07-11 07:51:54.620655) v4 -- 0x55d986d4fe00 con 0 > -3373> 2018-07-11 07:51:54.620788 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6811/15287 -- osd_ping(ping e501 stamp 2018-07-11 07:51:54.620655) v4 -- 0x55d9b9ac3200 con 0 > -3372> 2018-07-11 07:51:54.620804 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6806/15263 -- osd_ping(ping e501 stamp 2018-07-11 07:51:54.620655) v4 -- 0x55d986d33c00 con 0 > -3371> 2018-07-11 07:51:54.620817 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6807/15263 -- osd_ping(ping e501 stamp 2018-07-11 07:51:54.620655) v4 -- 0x55d9b2783000 con 0 > -3370> 2018-07-11 07:51:54.620829 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.115:6802/13323 -- osd_ping(ping e501 stamp 2018-07-11 07:51:54.620655) v4 -- 0x55d986c23a00 con 0 > -3369> 2018-07-11 07:51:54.620842 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.115:6803/13323 -- osd_ping(ping e501 stamp 2018-07-11 07:51:54.620655) v4 -- 0x55d982cdee00 con 0 > -3368> 2018-07-11 07:51:54.620858 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6806/15885 -- osd_ping(ping e501 stamp 2018-07-11 07:51:54.620655) v4 -- 0x55d980f5a200 con 0 > -3367> 2018-07-11 07:51:54.620870 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6807/15885 -- osd_ping(ping e501 stamp 2018-07-11 07:51:54.620655) v4 -- 0x55d9c34cc600 con 0 > -3366> 2018-07-11 07:51:54.620882 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.108:6802/11967 -- osd_ping(ping e501 stamp 2018-07-11 07:51:54.620655) v4 -- 0x55d9869be600 con 0 > -3365> 2018-07-11 07:51:54.620894 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.108:6803/11967 -- osd_ping(ping e501 stamp 2018-07-11 07:51:54.620655) v4 -- 0x55d9c4c1aa00 con 0 > -3364> 2018-07-11 07:51:54.620906 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.107:6806/12617 -- osd_ping(ping e501 stamp 2018-07-11 07:51:54.620655) v4 -- 0x55d9b2783400 con 0 > -3363> 2018-07-11 07:51:54.620919 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.107:6807/12617 -- osd_ping(ping e501 stamp 2018-07-11 07:51:54.620655) v4 -- 0x55d9829e5800 con 0 > -3362> 2018-07-11 07:51:54.620931 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6802/15919 -- osd_ping(ping e501 stamp 2018-07-11 07:51:54.620655) v4 -- 0x55d982d31600 con 0 > -3361> 2018-07-11 07:51:54.620945 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6803/15919 -- osd_ping(ping e501 stamp 2018-07-11 07:51:54.620655) v4 -- 0x55d980f5a800 con 0 > -3360> 2018-07-11 07:51:54.621076 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6806/12677 conn(0x55d986b58000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.1 seq 248 0x55d986d33000 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:54.620655) v4 > -3359> 2018-07-11 07:51:54.621082 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.107:6806/12617 conn(0x55d986bd2000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.9 seq 248 0x55d982cdf800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:54.620655) v4 > -3358> 2018-07-11 07:51:54.621098 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.1 10.8.128.106:6806/12677 248 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:54.620655) v4 ==== 2004+0+0 (1338603181 0 0) 0x55d986d33000 con 0x55d986b58000 > -3357> 2018-07-11 07:51:54.621100 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.9 10.8.128.107:6806/12617 248 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:54.620655) v4 ==== 2004+0+0 (1338603181 0 0) 0x55d982cdf800 con 0x55d986bd2000 > -3356> 2018-07-11 07:51:54.621148 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.107:6807/12617 conn(0x55d986bd3800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.9 seq 248 0x55d986d33000 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:54.620655) v4 > -3355> 2018-07-11 07:51:54.621162 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.9 10.8.128.107:6807/12617 248 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:54.620655) v4 ==== 2004+0+0 (1338603181 0 0) 0x55d986d33000 con 0x55d986bd3800 > -3354> 2018-07-11 07:51:54.621161 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6810/12651 conn(0x55d986b9b800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.2 seq 248 0x55d982cdf800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:54.620655) v4 > -3353> 2018-07-11 07:51:54.621164 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6807/12677 conn(0x55d986b59800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.1 seq 248 0x55d9b437a800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:54.620655) v4 > -3352> 2018-07-11 07:51:54.621174 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.2 10.8.128.106:6810/12651 248 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:54.620655) v4 ==== 2004+0+0 (1338603181 0 0) 0x55d982cdf800 con 0x55d986b9b800 > -3351> 2018-07-11 07:51:54.621199 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.1 10.8.128.106:6807/12677 248 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:54.620655) v4 ==== 2004+0+0 (1338603181 0 0) 0x55d9b437a800 con 0x55d986b59800 > -3350> 2018-07-11 07:51:54.621224 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6806/15263 conn(0x55d986b9d000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.5 seq 248 0x55d986d33000 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:54.620655) v4 > -3349> 2018-07-11 07:51:54.621238 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.5 10.8.128.113:6806/15263 248 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:54.620655) v4 ==== 2004+0+0 (1338603181 0 0) 0x55d986d33000 con 0x55d986b9d000 > -3348> 2018-07-11 07:51:54.621237 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6803/15255 conn(0x55d986bc9800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.0 seq 248 0x55d982cdf800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:54.620655) v4 > -3347> 2018-07-11 07:51:54.621262 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.0 10.8.128.113:6803/15255 248 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:54.620655) v4 ==== 2004+0+0 (1338603181 0 0) 0x55d982cdf800 con 0x55d986bc9800 > -3346> 2018-07-11 07:51:54.621274 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6811/12650 conn(0x55d986bcc800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 248 0x55d9b437a800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:54.620655) v4 > -3345> 2018-07-11 07:51:54.621290 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6809/12650 conn(0x55d986bcb000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 248 0x55d986d33000 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:54.620655) v4 > -3344> 2018-07-11 07:51:54.621295 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.3 10.8.128.106:6811/12650 248 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:54.620655) v4 ==== 2004+0+0 (1338603181 0 0) 0x55d9b437a800 con 0x55d986bcc800 > -3343> 2018-07-11 07:51:54.621305 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.3 10.8.128.106:6809/12650 248 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:54.620655) v4 ==== 2004+0+0 (1338603181 0 0) 0x55d986d33000 con 0x55d986bcb000 > -3342> 2018-07-11 07:51:54.621314 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6811/15287 conn(0x55d986bb0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 248 0x55d982cdf800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:54.620655) v4 > -3341> 2018-07-11 07:51:54.621328 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.4 10.8.128.113:6811/15287 248 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:54.620655) v4 ==== 2004+0+0 (1338603181 0 0) 0x55d982cdf800 con 0x55d986bb0800 > -3340> 2018-07-11 07:51:54.621351 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6807/15263 conn(0x55d986b9e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.5 seq 248 0x55d9b437a800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:54.620655) v4 > -3339> 2018-07-11 07:51:54.621360 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.115:6802/13323 conn(0x55d986bc1000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 248 0x55d982cdf800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:54.620655) v4 > -3338> 2018-07-11 07:51:54.621365 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.5 10.8.128.113:6807/15263 248 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:54.620655) v4 ==== 2004+0+0 (1338603181 0 0) 0x55d9b437a800 con 0x55d986b9e800 > -3337> 2018-07-11 07:51:54.621375 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.6 10.8.128.115:6802/13323 248 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:54.620655) v4 ==== 2004+0+0 (1338603181 0 0) 0x55d982cdf800 con 0x55d986bc1000 > -3336> 2018-07-11 07:51:54.621395 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6802/15255 conn(0x55d986bc8000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.0 seq 248 0x55d9b437a800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:54.620655) v4 > -3335> 2018-07-11 07:51:54.621410 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.0 10.8.128.113:6802/15255 248 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:54.620655) v4 ==== 2004+0+0 (1338603181 0 0) 0x55d9b437a800 con 0x55d986bc8000 > -3334> 2018-07-11 07:51:54.621441 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6808/12651 conn(0x55d986b9a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.2 seq 248 0x55d9b437a800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:54.620655) v4 > -3333> 2018-07-11 07:51:54.621442 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.108:6802/11967 conn(0x55d986bbe000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.8 seq 248 0x55d986d33000 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:54.620655) v4 > -3332> 2018-07-11 07:51:54.621458 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.8 10.8.128.108:6802/11967 248 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:54.620655) v4 ==== 2004+0+0 (1338603181 0 0) 0x55d986d33000 con 0x55d986bbe000 > -3331> 2018-07-11 07:51:54.621457 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.2 10.8.128.106:6808/12651 248 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:54.620655) v4 ==== 2004+0+0 (1338603181 0 0) 0x55d9b437a800 con 0x55d986b9a000 > -3330> 2018-07-11 07:51:54.621500 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6810/15287 conn(0x55d986baf000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 248 0x55d9b437a800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:54.620655) v4 > -3329> 2018-07-11 07:51:54.621512 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.4 10.8.128.113:6810/15287 248 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:54.620655) v4 ==== 2004+0+0 (1338603181 0 0) 0x55d9b437a800 con 0x55d986baf000 > -3328> 2018-07-11 07:51:54.621506 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.115:6803/13323 conn(0x55d986bc2800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 248 0x55d986d33000 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:54.620655) v4 > -3327> 2018-07-11 07:51:54.621511 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6802/15919 conn(0x55d986b5b000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.11 seq 248 0x55d982cdf800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:54.620655) v4 > -3326> 2018-07-11 07:51:54.621521 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.6 10.8.128.115:6803/13323 248 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:54.620655) v4 ==== 2004+0+0 (1338603181 0 0) 0x55d986d33000 con 0x55d986bc2800 > -3325> 2018-07-11 07:51:54.621525 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.11 10.8.128.114:6802/15919 248 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:54.620655) v4 ==== 2004+0+0 (1338603181 0 0) 0x55d982cdf800 con 0x55d986b5b000 > -3324> 2018-07-11 07:51:54.621552 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6806/15885 conn(0x55d986bac000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.7 seq 248 0x55d982cdf800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:54.620655) v4 > -3323> 2018-07-11 07:51:54.621554 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6803/15919 conn(0x55d986b5c800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.11 seq 248 0x55d986d33000 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:54.620655) v4 > -3322> 2018-07-11 07:51:54.621559 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.108:6803/11967 conn(0x55d986bbf800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.8 seq 248 0x55d9b437a800 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:54.620655) v4 > -3321> 2018-07-11 07:51:54.621562 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.7 10.8.128.114:6806/15885 248 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:54.620655) v4 ==== 2004+0+0 (1338603181 0 0) 0x55d982cdf800 con 0x55d986bac000 > -3320> 2018-07-11 07:51:54.621571 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.8 10.8.128.108:6803/11967 248 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:54.620655) v4 ==== 2004+0+0 (1338603181 0 0) 0x55d9b437a800 con 0x55d986bbf800 > -3319> 2018-07-11 07:51:54.621572 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.11 10.8.128.114:6803/15919 248 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:54.620655) v4 ==== 2004+0+0 (1338603181 0 0) 0x55d986d33000 con 0x55d986b5c800 > -3318> 2018-07-11 07:51:54.621602 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6807/15885 conn(0x55d986bad800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.7 seq 248 0x55d986d33000 osd_ping(ping_reply e501 stamp 2018-07-11 07:51:54.620655) v4 > -3317> 2018-07-11 07:51:54.621613 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.7 10.8.128.114:6807/15885 248 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:51:54.620655) v4 ==== 2004+0+0 (1338603181 0 0) 0x55d986d33000 con 0x55d986bad800 > -3316> 2018-07-11 07:51:54.769860 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15255 conn(0x55d986c82800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.0 seq 258 0x55d982cdf800 osd_ping(ping e501 stamp 2018-07-11 07:51:56.001134) v4 > -3315> 2018-07-11 07:51:54.769886 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.0 10.8.128.113:0/15255 258 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:56.001134) v4 ==== 2004+0+0 (183541711 0 0) 0x55d982cdf800 con 0x55d986c82800 > -3314> 2018-07-11 07:51:54.769900 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:56.001134) v4 -- 0x55d982d31600 con 0 > -3313> 2018-07-11 07:51:54.769894 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15255 conn(0x55d986d20000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.0 seq 258 0x55d9b437a800 osd_ping(ping e501 stamp 2018-07-11 07:51:56.001134) v4 > -3312> 2018-07-11 07:51:54.769916 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.0 10.8.128.113:0/15255 258 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:56.001134) v4 ==== 2004+0+0 (183541711 0 0) 0x55d9b437a800 con 0x55d986d20000 > -3311> 2018-07-11 07:51:54.769930 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:56.001134) v4 -- 0x55d9c4c1aa00 con 0 > -3310> 2018-07-11 07:51:54.793362 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 999 0x55d984b36000 rep_scrubmap(5.2e e501 from shard 2) v1 > -3309> 2018-07-11 07:51:54.793383 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 999 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12476 (1269249638 0 3852207718) 0x55d984b36000 con 0x55d986bf0800 > -3308> 2018-07-11 07:51:54.811656 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 993 0x55d984b36240 rep_scrubmap(5.2e e501 from shard 4) v1 > -3307> 2018-07-11 07:51:54.811676 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 993 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12476 (3686815630 0 3852207718) 0x55d984b36240 con 0x55d986bec000 > -3306> 2018-07-11 07:51:54.890994 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.108:0/11967 conn(0x55d986db0000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.8 seq 238 0x55d9869d9800 osd_ping(ping e501 stamp 2018-07-11 07:51:54.646718) v4 > -3305> 2018-07-11 07:51:54.891014 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.8 10.8.128.108:0/11967 238 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:54.646718) v4 ==== 2004+0+0 (1796696934 0 0) 0x55d9869d9800 con 0x55d986db0000 > -3304> 2018-07-11 07:51:54.891005 7ffacbbd1700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.108:0/11967 conn(0x55d986d82800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.8 seq 238 0x55d986d33000 osd_ping(ping e501 stamp 2018-07-11 07:51:54.646718) v4 > -3303> 2018-07-11 07:51:54.891023 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:54.646718) v4 -- 0x55d982d31600 con 0 > -3302> 2018-07-11 07:51:54.891023 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 <== osd.8 10.8.128.108:0/11967 238 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:54.646718) v4 ==== 2004+0+0 (1796696934 0 0) 0x55d986d33000 con 0x55d986d82800 > -3301> 2018-07-11 07:51:54.891044 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:54.646718) v4 -- 0x55d980f5a800 con 0 > -3300> 2018-07-11 07:51:54.898452 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -3299> 2018-07-11 07:51:54.898476 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -3298> 2018-07-11 07:51:54.898521 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -3297> 2018-07-11 07:51:54.898966 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -3296> 2018-07-11 07:51:54.949675 7ffacbbd1700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15263 conn(0x55d985d44000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.5 seq 239 0x55d9869be000 osd_ping(ping e501 stamp 2018-07-11 07:51:56.181003) v4 > -3295> 2018-07-11 07:51:54.949677 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15263 conn(0x55d985d45800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.5 seq 239 0x55d982cdf800 osd_ping(ping e501 stamp 2018-07-11 07:51:56.181003) v4 > -3294> 2018-07-11 07:51:54.949700 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 <== osd.5 10.8.128.113:0/15263 239 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:56.181003) v4 ==== 2004+0+0 (1333636048 0 0) 0x55d9869be000 con 0x55d985d44000 > -3293> 2018-07-11 07:51:54.949709 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:56.181003) v4 -- 0x55d980f5a800 con 0 > -3292> 2018-07-11 07:51:54.949707 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.5 10.8.128.113:0/15263 239 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:56.181003) v4 ==== 2004+0+0 (1333636048 0 0) 0x55d982cdf800 con 0x55d985d45800 > -3291> 2018-07-11 07:51:54.949735 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:56.181003) v4 -- 0x55d982d31600 con 0 > -3290> 2018-07-11 07:51:54.951896 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'66804,epoch:501/498,start:5:7567b4ca:::benchmark_data_magna115_10867_object3190021:0,end:5:756819c2:::benchmark_data_magna115_10867_object1147422:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454280 con 0 > -3289> 2018-07-11 07:51:54.951934 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'66804,epoch:501/498,start:5:7567b4ca:::benchmark_data_magna115_10867_object3190021:0,end:5:756819c2:::benchmark_data_magna115_10867_object1147422:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454c80 con 0 > -3288> 2018-07-11 07:51:54.963559 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6805/15263 conn(0x55d986c41000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=31 cs=1 l=0). rx osd.5 seq 2832 0x55d9b437a800 MOSDScrubReserve(6.15 REQUEST e501) v1 > -3287> 2018-07-11 07:51:54.963577 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.5 10.8.128.113:6805/15263 2832 ==== MOSDScrubReserve(6.15 REQUEST e501) v1 ==== 43+0+0 (3942742026 0 0) 0x55d9b437a800 con 0x55d986c41000 > -3286> 2018-07-11 07:51:55.179538 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1000 0x55d984b36480 rep_scrubmap(5.2e e501 from shard 2) v1 > -3285> 2018-07-11 07:51:55.179558 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1000 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12474 (1269249638 0 3218570133) 0x55d984b36480 con 0x55d986bf0800 > -3284> 2018-07-11 07:51:55.195198 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.114:0/15919 conn(0x55d986df5000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.11 seq 259 0x55d9869d9800 osd_ping(ping e501 stamp 2018-07-11 07:51:55.825551) v4 > -3283> 2018-07-11 07:51:55.195219 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.11 10.8.128.114:0/15919 259 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:55.825551) v4 ==== 2004+0+0 (493720874 0 0) 0x55d9869d9800 con 0x55d986df5000 > -3282> 2018-07-11 07:51:55.195229 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:55.825551) v4 -- 0x55d982d31600 con 0 > -3281> 2018-07-11 07:51:55.195232 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.114:0/15919 conn(0x55d986df3800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.11 seq 259 0x55d986d33000 osd_ping(ping e501 stamp 2018-07-11 07:51:55.825551) v4 > -3280> 2018-07-11 07:51:55.195265 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.11 10.8.128.114:0/15919 259 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:55.825551) v4 ==== 2004+0+0 (493720874 0 0) 0x55d986d33000 con 0x55d986df3800 > -3279> 2018-07-11 07:51:55.195277 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:55.825551) v4 -- 0x55d980f5a800 con 0 > -3278> 2018-07-11 07:51:55.214740 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 994 0x55d984b366c0 rep_scrubmap(5.2e e501 from shard 4) v1 > -3277> 2018-07-11 07:51:55.214760 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 994 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12474 (3686815630 0 3218570133) 0x55d984b366c0 con 0x55d986bec000 > -3276> 2018-07-11 07:51:55.269445 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6805/15263 -- MOSDScrubReserve(6.15 REJECT e501) v1 -- 0x55d986c22e00 con 0 > -3275> 2018-07-11 07:51:55.269640 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -3274> 2018-07-11 07:51:55.269663 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -3273> 2018-07-11 07:51:55.269712 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -3272> 2018-07-11 07:51:55.269924 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6805/15263 conn(0x55d986c41000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=31 cs=1 l=0). rx osd.5 seq 2833 0x55d986c22e00 MOSDScrubReserve(6.15 RELEASE e501) v1 > -3271> 2018-07-11 07:51:55.269943 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.5 10.8.128.113:6805/15263 2833 ==== MOSDScrubReserve(6.15 RELEASE e501) v1 ==== 43+0+0 (605848072 0 0) 0x55d986c22e00 con 0x55d986c41000 > -3270> 2018-07-11 07:51:55.270170 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -3269> 2018-07-11 07:51:55.271033 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67343,epoch:501/498,start:5:756819c2:::benchmark_data_magna115_10867_object1147422:0,end:5:756871db:::benchmark_data_magna115_10867_object1754976:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454000 con 0 > -3268> 2018-07-11 07:51:55.271077 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67343,epoch:501/498,start:5:756819c2:::benchmark_data_magna115_10867_object1147422:0,end:5:756871db:::benchmark_data_magna115_10867_object1754976:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb773400 con 0 > -3267> 2018-07-11 07:51:55.463326 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1001 0x55d984b36900 rep_scrubmap(5.2e e501 from shard 2) v1 > -3266> 2018-07-11 07:51:55.463347 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1001 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12470 (1269249638 0 1480050341) 0x55d984b36900 con 0x55d986bf0800 > -3265> 2018-07-11 07:51:55.492474 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 995 0x55d984b36b40 rep_scrubmap(5.2e e501 from shard 4) v1 > -3264> 2018-07-11 07:51:55.492495 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 995 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12470 (3686815630 0 1480050341) 0x55d984b36b40 con 0x55d986bec000 > -3263> 2018-07-11 07:51:55.567074 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6805/15885 conn(0x55d986c7f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.7 seq 2812 0x55d9c4c1aa00 MOSDScrubReserve(5.4 REQUEST e501) v1 > -3262> 2018-07-11 07:51:55.567093 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.7 10.8.128.114:6805/15885 2812 ==== MOSDScrubReserve(5.4 REQUEST e501) v1 ==== 43+0+0 (1753221512 0 0) 0x55d9c4c1aa00 con 0x55d986c7f800 > -3261> 2018-07-11 07:51:55.567166 7ffab33df700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.114:6805/15885 -- MOSDScrubReserve(5.4 REJECT e501) v1 -- 0x55d9829e5600 con 0 > -3260> 2018-07-11 07:51:55.567484 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6805/15885 conn(0x55d986c7f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.7 seq 2813 0x55d9829e5600 MOSDScrubReserve(5.4 RELEASE e501) v1 > -3259> 2018-07-11 07:51:55.567500 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.7 10.8.128.114:6805/15885 2813 ==== MOSDScrubReserve(5.4 RELEASE e501) v1 ==== 43+0+0 (2812146570 0 0) 0x55d9829e5600 con 0x55d986c7f800 > -3258> 2018-07-11 07:51:55.611440 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -3257> 2018-07-11 07:51:55.611467 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -3256> 2018-07-11 07:51:55.611514 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -3255> 2018-07-11 07:51:55.611977 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -3254> 2018-07-11 07:51:55.660395 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67986,epoch:501/498,start:5:756871db:::benchmark_data_magna115_10867_object1754976:0,end:5:7568d54d:::benchmark_data_magna115_10867_object673072:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb773680 con 0 > -3253> 2018-07-11 07:51:55.660433 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67986,epoch:501/498,start:5:756871db:::benchmark_data_magna115_10867_object1754976:0,end:5:7568d54d:::benchmark_data_magna115_10867_object673072:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb773180 con 0 > -3252> 2018-07-11 07:51:55.718524 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12650 conn(0x55d986d23000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 249 0x55d982cdf800 osd_ping(ping e501 stamp 2018-07-11 07:51:55.766443) v4 > -3251> 2018-07-11 07:51:55.718548 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.3 10.8.128.106:0/12650 249 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:55.766443) v4 ==== 2004+0+0 (4245467046 0 0) 0x55d982cdf800 con 0x55d986d23000 > -3250> 2018-07-11 07:51:55.718561 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:55.766443) v4 -- 0x55d982d31600 con 0 > -3249> 2018-07-11 07:51:55.718568 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12650 conn(0x55d986d21800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 249 0x55d9869be000 osd_ping(ping e501 stamp 2018-07-11 07:51:55.766443) v4 > -3248> 2018-07-11 07:51:55.718589 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.3 10.8.128.106:0/12650 249 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:55.766443) v4 ==== 2004+0+0 (4245467046 0 0) 0x55d9869be000 con 0x55d986d21800 > -3247> 2018-07-11 07:51:55.718604 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:55.766443) v4 -- 0x55d980f5a800 con 0 > -3246> 2018-07-11 07:51:55.872621 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1002 0x55d984b36d80 rep_scrubmap(5.2e e501 from shard 2) v1 > -3245> 2018-07-11 07:51:55.872641 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1002 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12476 (1269249638 0 211952654) 0x55d984b36d80 con 0x55d986bf0800 > -3244> 2018-07-11 07:51:55.889476 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 996 0x55d984b36fc0 rep_scrubmap(5.2e e501 from shard 4) v1 > -3243> 2018-07-11 07:51:55.889494 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 996 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12476 (3686815630 0 211952654) 0x55d984b36fc0 con 0x55d986bec000 > -3242> 2018-07-11 07:51:55.904112 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.115:0/13323 conn(0x55d986d7e000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 257 0x55d986d33000 osd_ping(ping e501 stamp 2018-07-11 07:51:56.249968) v4 > -3241> 2018-07-11 07:51:55.904133 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.6 10.8.128.115:0/13323 257 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:56.249968) v4 ==== 2004+0+0 (686387217 0 0) 0x55d986d33000 con 0x55d986d7e000 > -3240> 2018-07-11 07:51:55.904143 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:56.249968) v4 -- 0x55d980f5a800 con 0 > -3239> 2018-07-11 07:51:55.904236 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.115:0/13323 conn(0x55d986d24800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 257 0x55d9869bfa00 osd_ping(ping e501 stamp 2018-07-11 07:51:56.249968) v4 > -3238> 2018-07-11 07:51:55.904269 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.6 10.8.128.115:0/13323 257 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:56.249968) v4 ==== 2004+0+0 (686387217 0 0) 0x55d9869bfa00 con 0x55d986d24800 > -3237> 2018-07-11 07:51:55.904282 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:56.249968) v4 -- 0x55d986d4fe00 con 0 > -3236> 2018-07-11 07:51:55.954231 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -3235> 2018-07-11 07:51:55.954265 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -3234> 2018-07-11 07:51:55.954326 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -3233> 2018-07-11 07:51:55.954770 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -3232> 2018-07-11 07:51:55.963876 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6805/15263 conn(0x55d986c41000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=31 cs=1 l=0). rx osd.5 seq 2834 0x55d9b2783000 MOSDScrubReserve(6.15 REQUEST e501) v1 > -3231> 2018-07-11 07:51:55.963893 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.5 10.8.128.113:6805/15263 2834 ==== MOSDScrubReserve(6.15 REQUEST e501) v1 ==== 43+0+0 (3942742026 0 0) 0x55d9b2783000 con 0x55d986c41000 > -3230> 2018-07-11 07:51:55.990729 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67478,epoch:501/498,start:5:7568d54d:::benchmark_data_magna115_10867_object673072:0,end:5:75693178:::benchmark_data_magna115_10867_object1810620:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb773900 con 0 > -3229> 2018-07-11 07:51:55.990780 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67478,epoch:501/498,start:5:7568d54d:::benchmark_data_magna115_10867_object673072:0,end:5:75693178:::benchmark_data_magna115_10867_object1810620:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb772780 con 0 > -3228> 2018-07-11 07:51:56.153149 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.107:0/12617 conn(0x55d98123c000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 249 0x55d986d4fe00 osd_ping(ping e501 stamp 2018-07-11 07:51:56.152691) v4 > -3227> 2018-07-11 07:51:56.153169 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.107:0/12617 conn(0x55d98123d800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 249 0x55d9869be000 osd_ping(ping e501 stamp 2018-07-11 07:51:56.152691) v4 > -3226> 2018-07-11 07:51:56.153195 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.9 10.8.128.107:0/12617 249 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:56.152691) v4 ==== 2004+0+0 (1721494647 0 0) 0x55d986d4fe00 con 0x55d98123c000 > -3225> 2018-07-11 07:51:56.153197 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.9 10.8.128.107:0/12617 249 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:56.152691) v4 ==== 2004+0+0 (1721494647 0 0) 0x55d9869be000 con 0x55d98123d800 > -3224> 2018-07-11 07:51:56.153213 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:56.152691) v4 -- 0x55d9b60df400 con 0 > -3223> 2018-07-11 07:51:56.153235 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:56.152691) v4 -- 0x55d980f5a800 con 0 > -3222> 2018-07-11 07:51:56.208656 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1003 0x55d984b37200 rep_scrubmap(5.2e e501 from shard 2) v1 > -3221> 2018-07-11 07:51:56.208676 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1003 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12470 (1269249638 0 2647531514) 0x55d984b37200 con 0x55d986bf0800 > -3220> 2018-07-11 07:51:56.210685 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 997 0x55d984b37440 rep_scrubmap(5.2e e501 from shard 4) v1 > -3219> 2018-07-11 07:51:56.210704 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 997 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12470 (3686815630 0 2647531514) 0x55d984b37440 con 0x55d986bec000 > -3218> 2018-07-11 07:51:56.258773 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6805/15263 -- MOSDScrubReserve(6.15 REJECT e501) v1 -- 0x55d986c22e00 con 0 > -3217> 2018-07-11 07:51:56.258958 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -3216> 2018-07-11 07:51:56.258980 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -3215> 2018-07-11 07:51:56.259028 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -3214> 2018-07-11 07:51:56.259244 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6805/15263 conn(0x55d986c41000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=31 cs=1 l=0). rx osd.5 seq 2835 0x55d986c22e00 MOSDScrubReserve(6.15 RELEASE e501) v1 > -3213> 2018-07-11 07:51:56.259271 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.5 10.8.128.113:6805/15263 2835 ==== MOSDScrubReserve(6.15 RELEASE e501) v1 ==== 43+0+0 (605848072 0 0) 0x55d986c22e00 con 0x55d986c41000 > -3212> 2018-07-11 07:51:56.259501 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -3211> 2018-07-11 07:51:56.310726 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75693178:::benchmark_data_magna115_10867_object1810620:0,end:5:75699fb6:::benchmark_data_magna115_11707_object355693:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb772f00 con 0 > -3210> 2018-07-11 07:51:56.310768 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75693178:::benchmark_data_magna115_10867_object1810620:0,end:5:75699fb6:::benchmark_data_magna115_11707_object355693:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb772a00 con 0 > -3209> 2018-07-11 07:51:56.538647 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 998 0x55d984b37680 rep_scrubmap(5.2e e501 from shard 4) v1 > -3208> 2018-07-11 07:51:56.538666 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 998 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12480 (3686815630 0 2090596596) 0x55d984b37680 con 0x55d986bec000 > -3207> 2018-07-11 07:51:56.546619 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1004 0x55d984b378c0 rep_scrubmap(5.2e e501 from shard 2) v1 > -3206> 2018-07-11 07:51:56.546639 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1004 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12480 (1269249638 0 2090596596) 0x55d984b378c0 con 0x55d986bf0800 > -3205> 2018-07-11 07:51:56.546743 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -3204> 2018-07-11 07:51:56.546765 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -3203> 2018-07-11 07:51:56.546811 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -3202> 2018-07-11 07:51:56.547281 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -3201> 2018-07-11 07:51:56.562562 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75699fb6:::benchmark_data_magna115_11707_object355693:0,end:5:756a0682:::benchmark_data_magna115_10867_object1683563:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb772c80 con 0 > -3200> 2018-07-11 07:51:56.562612 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75699fb6:::benchmark_data_magna115_11707_object355693:0,end:5:756a0682:::benchmark_data_magna115_10867_object1683563:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb772500 con 0 > -3199> 2018-07-11 07:51:56.650132 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15263 conn(0x55d985d45800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.5 seq 240 0x55d9869d9800 osd_ping(ping e501 stamp 2018-07-11 07:51:57.881523) v4 > -3198> 2018-07-11 07:51:56.650153 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.5 10.8.128.113:0/15263 240 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:57.881523) v4 ==== 2004+0+0 (1605871972 0 0) 0x55d9869d9800 con 0x55d985d45800 > -3197> 2018-07-11 07:51:56.650144 7ffacbbd1700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15263 conn(0x55d985d44000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.5 seq 240 0x55d986d33000 osd_ping(ping e501 stamp 2018-07-11 07:51:57.881523) v4 > -3196> 2018-07-11 07:51:56.650169 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:57.881523) v4 -- 0x55d982d31600 con 0 > -3195> 2018-07-11 07:51:56.650175 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 <== osd.5 10.8.128.113:0/15263 240 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:57.881523) v4 ==== 2004+0+0 (1605871972 0 0) 0x55d986d33000 con 0x55d985d44000 > -3194> 2018-07-11 07:51:56.650186 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:57.881523) v4 -- 0x55d980f5a800 con 0 > -3193> 2018-07-11 07:51:56.760448 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1005 0x55d984b37b00 rep_scrubmap(5.2e e501 from shard 2) v1 > -3192> 2018-07-11 07:51:56.760468 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1005 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12464 (1269249638 0 3637799118) 0x55d984b37b00 con 0x55d986bf0800 > -3191> 2018-07-11 07:51:56.781203 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 999 0x55d984b37d40 rep_scrubmap(5.2e e501 from shard 4) v1 > -3190> 2018-07-11 07:51:56.781223 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 999 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12464 (3686815630 0 3637799118) 0x55d984b37d40 con 0x55d986bec000 > -3189> 2018-07-11 07:51:56.781330 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -3188> 2018-07-11 07:51:56.781352 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -3187> 2018-07-11 07:51:56.781399 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -3186> 2018-07-11 07:51:56.781852 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -3185> 2018-07-11 07:51:56.819006 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12650 conn(0x55d986d23000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 250 0x55d982cdf800 osd_ping(ping e501 stamp 2018-07-11 07:51:56.866897) v4 > -3184> 2018-07-11 07:51:56.819024 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.3 10.8.128.106:0/12650 250 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:56.866897) v4 ==== 2004+0+0 (3224338743 0 0) 0x55d982cdf800 con 0x55d986d23000 > -3183> 2018-07-11 07:51:56.819033 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:56.866897) v4 -- 0x55d982d31600 con 0 > -3182> 2018-07-11 07:51:56.819041 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12650 conn(0x55d986d21800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 250 0x55d9869be000 osd_ping(ping e501 stamp 2018-07-11 07:51:56.866897) v4 > -3181> 2018-07-11 07:51:56.819064 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.3 10.8.128.106:0/12650 250 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:56.866897) v4 ==== 2004+0+0 (3224338743 0 0) 0x55d9869be000 con 0x55d986d21800 > -3180> 2018-07-11 07:51:56.819080 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:56.866897) v4 -- 0x55d980f5a800 con 0 > -3179> 2018-07-11 07:51:56.827550 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'66549,epoch:501/498,start:5:756a0682:::benchmark_data_magna115_10867_object1683563:0,end:5:756a4d5f:::benchmark_data_magna115_10867_object590082:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb772000 con 0 > -3178> 2018-07-11 07:51:56.827600 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'66549,epoch:501/498,start:5:756a0682:::benchmark_data_magna115_10867_object1683563:0,end:5:756a4d5f:::benchmark_data_magna115_10867_object590082:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98944b400 con 0 > -3177> 2018-07-11 07:51:56.842370 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 3018 0x55d986d33000 MOSDScrubReserve(11.3s5 REQUEST e501) v1 > -3176> 2018-07-11 07:51:56.842389 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 3018 ==== MOSDScrubReserve(11.3s5 REQUEST e501) v1 ==== 43+0+0 (3260170403 0 0) 0x55d986d33000 con 0x55d986c7e000 > -3175> 2018-07-11 07:51:56.842464 7ffab3be0700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e501) v1 -- 0x55d9bcd78600 con 0 > -3174> 2018-07-11 07:51:56.842903 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 3019 0x55d9bcd78600 MOSDScrubReserve(11.3s5 RELEASE e501) v1 > -3173> 2018-07-11 07:51:56.842921 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 3019 ==== MOSDScrubReserve(11.3s5 RELEASE e501) v1 ==== 43+0+0 (223328929 0 0) 0x55d9bcd78600 con 0x55d986c7e000 > -3172> 2018-07-11 07:51:56.988117 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15287 conn(0x55d98123f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 272 0x55d9869d9800 osd_ping(ping e501 stamp 2018-07-11 07:51:58.219513) v4 > -3171> 2018-07-11 07:51:56.988141 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.4 10.8.128.113:0/15287 272 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:58.219513) v4 ==== 2004+0+0 (3863039981 0 0) 0x55d9869d9800 con 0x55d98123f000 > -3170> 2018-07-11 07:51:56.988140 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15287 conn(0x55d981240800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 272 0x55d986d4fe00 osd_ping(ping e501 stamp 2018-07-11 07:51:58.219513) v4 > -3169> 2018-07-11 07:51:56.988158 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.4 10.8.128.113:0/15287 272 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:58.219513) v4 ==== 2004+0+0 (3863039981 0 0) 0x55d986d4fe00 con 0x55d981240800 > -3168> 2018-07-11 07:51:56.988159 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:58.219513) v4 -- 0x55d982d31600 con 0 > -3167> 2018-07-11 07:51:56.988192 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:58.219513) v4 -- 0x55d9b60df400 con 0 > -3166> 2018-07-11 07:51:57.051447 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1006 0x55d9b1500000 rep_scrubmap(5.2e e501 from shard 2) v1 > -3165> 2018-07-11 07:51:57.051467 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1006 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12480 (1269249638 0 3662237627) 0x55d9b1500000 con 0x55d986bf0800 > -3164> 2018-07-11 07:51:57.054243 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1000 0x55d9b1500240 rep_scrubmap(5.2e e501 from shard 4) v1 > -3163> 2018-07-11 07:51:57.054271 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1000 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12480 (3686815630 0 3662237627) 0x55d9b1500240 con 0x55d986bec000 > -3162> 2018-07-11 07:51:57.150630 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15263 conn(0x55d985d45800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.5 seq 241 0x55d982cdf800 osd_ping(ping e501 stamp 2018-07-11 07:51:58.381987) v4 > -3161> 2018-07-11 07:51:57.150631 7ffacbbd1700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15263 conn(0x55d985d44000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.5 seq 241 0x55d9869be000 osd_ping(ping e501 stamp 2018-07-11 07:51:58.381987) v4 > -3160> 2018-07-11 07:51:57.150654 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.5 10.8.128.113:0/15263 241 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:58.381987) v4 ==== 2004+0+0 (3670889232 0 0) 0x55d982cdf800 con 0x55d985d45800 > -3159> 2018-07-11 07:51:57.150657 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 <== osd.5 10.8.128.113:0/15263 241 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:58.381987) v4 ==== 2004+0+0 (3670889232 0 0) 0x55d9869be000 con 0x55d985d44000 > -3158> 2018-07-11 07:51:57.150663 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:58.381987) v4 -- 0x55d982d31600 con 0 > -3157> 2018-07-11 07:51:57.150685 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:58.381987) v4 -- 0x55d9829e5800 con 0 > -3156> 2018-07-11 07:51:57.171441 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -3155> 2018-07-11 07:51:57.171469 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -3154> 2018-07-11 07:51:57.171515 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -3153> 2018-07-11 07:51:57.171981 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -3152> 2018-07-11 07:51:57.191391 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.108:0/11967 conn(0x55d986db0000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.8 seq 239 0x55d9869d9800 osd_ping(ping e501 stamp 2018-07-11 07:51:56.947168) v4 > -3151> 2018-07-11 07:51:57.191412 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.8 10.8.128.108:0/11967 239 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:56.947168) v4 ==== 2004+0+0 (552507747 0 0) 0x55d9869d9800 con 0x55d986db0000 > -3150> 2018-07-11 07:51:57.191421 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:56.947168) v4 -- 0x55d982d31600 con 0 > -3149> 2018-07-11 07:51:57.191429 7ffacbbd1700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.108:0/11967 conn(0x55d986d82800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.8 seq 239 0x55d980f5a800 osd_ping(ping e501 stamp 2018-07-11 07:51:56.947168) v4 > -3148> 2018-07-11 07:51:57.191449 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 <== osd.8 10.8.128.108:0/11967 239 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:56.947168) v4 ==== 2004+0+0 (552507747 0 0) 0x55d980f5a800 con 0x55d986d82800 > -3147> 2018-07-11 07:51:57.191464 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:56.947168) v4 -- 0x55d9829e5800 con 0 > -3146> 2018-07-11 07:51:57.195276 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:756a4d5f:::benchmark_data_magna115_10867_object590082:0,end:5:756aa107:::benchmark_data_magna115_10867_object3747070:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb772280 con 0 > -3145> 2018-07-11 07:51:57.195328 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:756a4d5f:::benchmark_data_magna115_10867_object590082:0,end:5:756aa107:::benchmark_data_magna115_10867_object3747070:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98944b900 con 0 > -3144> 2018-07-11 07:51:57.351787 7ffab6be6700 1 -- 10.8.128.107:6800/13927 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55d9ba81a840 con 0 > -3143> 2018-07-11 07:51:57.351859 7ffab6be6700 1 -- 10.8.128.107:6800/13927 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55d9869ca000 con 0 > -3142> 2018-07-11 07:51:57.413904 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1007 0x55d9b1500480 rep_scrubmap(5.2e e501 from shard 2) v1 > -3141> 2018-07-11 07:51:57.413925 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1007 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12472 (1269249638 0 593809960) 0x55d9b1500480 con 0x55d986bf0800 > -3140> 2018-07-11 07:51:57.417966 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1001 0x55d9b15006c0 rep_scrubmap(5.2e e501 from shard 4) v1 > -3139> 2018-07-11 07:51:57.417986 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1001 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12472 (3686815630 0 593809960) 0x55d9b15006c0 con 0x55d986bec000 > -3138> 2018-07-11 07:51:57.521239 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.114:0/15885 conn(0x55d986d7f800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.7 seq 264 0x55d982cdf800 osd_ping(ping e501 stamp 2018-07-11 07:51:58.151629) v4 > -3137> 2018-07-11 07:51:57.521268 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.114:0/15885 conn(0x55d986d81000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.7 seq 264 0x55d986d4fe00 osd_ping(ping e501 stamp 2018-07-11 07:51:58.151629) v4 > -3136> 2018-07-11 07:51:57.521274 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.7 10.8.128.114:0/15885 264 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:58.151629) v4 ==== 2004+0+0 (2213561888 0 0) 0x55d982cdf800 con 0x55d986d7f800 > -3135> 2018-07-11 07:51:57.521286 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.7 10.8.128.114:0/15885 264 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:58.151629) v4 ==== 2004+0+0 (2213561888 0 0) 0x55d986d4fe00 con 0x55d986d81000 > -3134> 2018-07-11 07:51:57.521289 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:58.151629) v4 -- 0x55d982d31600 con 0 > -3133> 2018-07-11 07:51:57.521318 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:58.151629) v4 -- 0x55d9b60df400 con 0 > -3132> 2018-07-11 07:51:57.567388 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6805/15885 conn(0x55d986c7f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.7 seq 2814 0x55d9869bfa00 MOSDScrubReserve(5.4 REQUEST e501) v1 > -3131> 2018-07-11 07:51:57.567407 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.7 10.8.128.114:6805/15885 2814 ==== MOSDScrubReserve(5.4 REQUEST e501) v1 ==== 43+0+0 (1753221512 0 0) 0x55d9869bfa00 con 0x55d986c7f800 > -3130> 2018-07-11 07:51:57.567481 7ffab33df700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.114:6805/15885 -- MOSDScrubReserve(5.4 REJECT e501) v1 -- 0x55d9829e5600 con 0 > -3129> 2018-07-11 07:51:57.567809 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6805/15885 conn(0x55d986c7f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.7 seq 2815 0x55d9829e5600 MOSDScrubReserve(5.4 RELEASE e501) v1 > -3128> 2018-07-11 07:51:57.567823 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.7 10.8.128.114:6805/15885 2815 ==== MOSDScrubReserve(5.4 RELEASE e501) v1 ==== 43+0+0 (2812146570 0 0) 0x55d9829e5600 con 0x55d986c7f800 > -3127> 2018-07-11 07:51:57.572168 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -3126> 2018-07-11 07:51:57.572195 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -3125> 2018-07-11 07:51:57.572263 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -3124> 2018-07-11 07:51:57.572729 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -3123> 2018-07-11 07:51:57.594370 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:756aa107:::benchmark_data_magna115_10867_object3747070:0,end:5:756af01b:::benchmark_data_magna115_10867_object2273039:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98944bb80 con 0 > -3122> 2018-07-11 07:51:57.594420 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:756aa107:::benchmark_data_magna115_10867_object3747070:0,end:5:756af01b:::benchmark_data_magna115_10867_object2273039:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454a00 con 0 > -3121> 2018-07-11 07:51:57.691835 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.108:0/11967 conn(0x55d986db0000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.8 seq 240 0x55d9869d9800 osd_ping(ping e501 stamp 2018-07-11 07:51:57.447629) v4 > -3120> 2018-07-11 07:51:57.691857 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.8 10.8.128.108:0/11967 240 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:57.447629) v4 ==== 2004+0+0 (3456250585 0 0) 0x55d9869d9800 con 0x55d986db0000 > -3119> 2018-07-11 07:51:57.691867 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:57.447629) v4 -- 0x55d982d31600 con 0 > -3118> 2018-07-11 07:51:57.691869 7ffacbbd1700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.108:0/11967 conn(0x55d986d82800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.8 seq 240 0x55d9869be000 osd_ping(ping e501 stamp 2018-07-11 07:51:57.447629) v4 > -3117> 2018-07-11 07:51:57.691886 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 <== osd.8 10.8.128.108:0/11967 240 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:57.447629) v4 ==== 2004+0+0 (3456250585 0 0) 0x55d9869be000 con 0x55d986d82800 > -3116> 2018-07-11 07:51:57.691902 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:57.447629) v4 -- 0x55d9829e5800 con 0 > -3115> 2018-07-11 07:51:57.810136 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1008 0x55d9b1500900 rep_scrubmap(5.2e e501 from shard 2) v1 > -3114> 2018-07-11 07:51:57.810158 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1008 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12476 (1269249638 0 80254729) 0x55d9b1500900 con 0x55d986bf0800 > -3113> 2018-07-11 07:51:57.818618 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1002 0x55d9b1500b40 rep_scrubmap(5.2e e501 from shard 4) v1 > -3112> 2018-07-11 07:51:57.818638 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1002 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12476 (3686815630 0 80254729) 0x55d9b1500b40 con 0x55d986bec000 > -3111> 2018-07-11 07:51:57.919552 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12650 conn(0x55d986d21800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 251 0x55d980f5a800 osd_ping(ping e501 stamp 2018-07-11 07:51:57.967453) v4 > -3110> 2018-07-11 07:51:57.919559 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12650 conn(0x55d986d23000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 251 0x55d982cdf800 osd_ping(ping e501 stamp 2018-07-11 07:51:57.967453) v4 > -3109> 2018-07-11 07:51:57.919575 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.3 10.8.128.106:0/12650 251 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:57.967453) v4 ==== 2004+0+0 (3701538981 0 0) 0x55d982cdf800 con 0x55d986d23000 > -3108> 2018-07-11 07:51:57.919574 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.3 10.8.128.106:0/12650 251 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:57.967453) v4 ==== 2004+0+0 (3701538981 0 0) 0x55d980f5a800 con 0x55d986d21800 > -3107> 2018-07-11 07:51:57.919594 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:57.967453) v4 -- 0x55d982d31600 con 0 > -3106> 2018-07-11 07:51:57.919616 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:57.967453) v4 -- 0x55d9829e5800 con 0 > -3105> 2018-07-11 07:51:57.942192 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -3104> 2018-07-11 07:51:57.942216 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -3103> 2018-07-11 07:51:57.942279 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -3102> 2018-07-11 07:51:57.942726 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -3101> 2018-07-11 07:51:57.986346 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:756af01b:::benchmark_data_magna115_10867_object2273039:0,end:5:756b37ba:::benchmark_data_magna115_10867_object1924346:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454500 con 0 > -3100> 2018-07-11 07:51:57.986386 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:756af01b:::benchmark_data_magna115_10867_object2273039:0,end:5:756b37ba:::benchmark_data_magna115_10867_object1924346:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98944b680 con 0 > -3099> 2018-07-11 07:51:58.190597 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1009 0x55d9b1500d80 rep_scrubmap(5.2e e501 from shard 2) v1 > -3098> 2018-07-11 07:51:58.190617 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1009 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12484 (1269249638 0 1943344178) 0x55d9b1500d80 con 0x55d986bf0800 > -3097> 2018-07-11 07:51:58.192306 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.108:0/11967 conn(0x55d986db0000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.8 seq 241 0x55d9869d9800 osd_ping(ping e501 stamp 2018-07-11 07:51:57.948083) v4 > -3096> 2018-07-11 07:51:58.192315 7ffacbbd1700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.108:0/11967 conn(0x55d986d82800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.8 seq 241 0x55d9869be000 osd_ping(ping e501 stamp 2018-07-11 07:51:57.948083) v4 > -3095> 2018-07-11 07:51:58.192329 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.8 10.8.128.108:0/11967 241 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:57.948083) v4 ==== 2004+0+0 (138437309 0 0) 0x55d9869d9800 con 0x55d986db0000 > -3094> 2018-07-11 07:51:58.192338 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:57.948083) v4 -- 0x55d982d31600 con 0 > -3093> 2018-07-11 07:51:58.192335 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 <== osd.8 10.8.128.108:0/11967 241 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:57.948083) v4 ==== 2004+0+0 (138437309 0 0) 0x55d9869be000 con 0x55d986d82800 > -3092> 2018-07-11 07:51:58.192362 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:57.948083) v4 -- 0x55d9829e5800 con 0 > -3091> 2018-07-11 07:51:58.206707 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1003 0x55d9b1500fc0 rep_scrubmap(5.2e e501 from shard 4) v1 > -3090> 2018-07-11 07:51:58.206740 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1003 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12484 (3686815630 0 1943344178) 0x55d9b1500fc0 con 0x55d986bec000 > -3089> 2018-07-11 07:51:58.226334 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12651 conn(0x55d986db3000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.2 seq 257 0x55d980f5a800 osd_ping(ping e501 stamp 2018-07-11 07:51:58.274228) v4 > -3088> 2018-07-11 07:51:58.226366 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.2 10.8.128.106:0/12651 257 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:58.274228) v4 ==== 2004+0+0 (2879980704 0 0) 0x55d980f5a800 con 0x55d986db3000 > -3087> 2018-07-11 07:51:58.226381 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:58.274228) v4 -- 0x55d9829e5800 con 0 > -3086> 2018-07-11 07:51:58.226586 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12651 conn(0x55d986db1800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.2 seq 257 0x55d986d4fe00 osd_ping(ping e501 stamp 2018-07-11 07:51:58.274228) v4 > -3085> 2018-07-11 07:51:58.226606 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.2 10.8.128.106:0/12651 257 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:58.274228) v4 ==== 2004+0+0 (2879980704 0 0) 0x55d986d4fe00 con 0x55d986db1800 > -3084> 2018-07-11 07:51:58.226622 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:58.274228) v4 -- 0x55d9bdba4800 con 0 > -3083> 2018-07-11 07:51:58.364852 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -3082> 2018-07-11 07:51:58.364879 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -3081> 2018-07-11 07:51:58.364937 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -3080> 2018-07-11 07:51:58.365409 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -3079> 2018-07-11 07:51:58.386192 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67932,epoch:501/498,start:5:756b37ba:::benchmark_data_magna115_10867_object1924346:0,end:5:756b892a:::benchmark_data_magna115_10867_object1018678:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455180 con 0 > -3078> 2018-07-11 07:51:58.386244 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67932,epoch:501/498,start:5:756b37ba:::benchmark_data_magna115_10867_object1924346:0,end:5:756b892a:::benchmark_data_magna115_10867_object1018678:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454f00 con 0 > -3077> 2018-07-11 07:51:58.607115 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1010 0x55d9b1501200 rep_scrubmap(5.2e e501 from shard 2) v1 > -3076> 2018-07-11 07:51:58.607135 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1010 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12478 (1269249638 0 2286713227) 0x55d9b1501200 con 0x55d986bf0800 > -3075> 2018-07-11 07:51:58.636043 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1004 0x55d9b1501440 rep_scrubmap(5.2e e501 from shard 4) v1 > -3074> 2018-07-11 07:51:58.636063 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1004 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12478 (3686815630 0 2286713227) 0x55d9b1501440 con 0x55d986bec000 > -3073> 2018-07-11 07:51:58.804558 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.115:0/13323 conn(0x55d986d7e000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 258 0x55d9869be000 osd_ping(ping e501 stamp 2018-07-11 07:51:59.150450) v4 > -3072> 2018-07-11 07:51:58.804580 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.6 10.8.128.115:0/13323 258 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:59.150450) v4 ==== 2004+0+0 (1469253664 0 0) 0x55d9869be000 con 0x55d986d7e000 > -3071> 2018-07-11 07:51:58.804591 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:59.150450) v4 -- 0x55d9829e5800 con 0 > -3070> 2018-07-11 07:51:58.804578 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.115:0/13323 conn(0x55d986d24800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 258 0x55d986d4fe00 osd_ping(ping e501 stamp 2018-07-11 07:51:59.150450) v4 > -3069> 2018-07-11 07:51:58.804602 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.6 10.8.128.115:0/13323 258 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:59.150450) v4 ==== 2004+0+0 (1469253664 0 0) 0x55d986d4fe00 con 0x55d986d24800 > -3068> 2018-07-11 07:51:58.804615 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:59.150450) v4 -- 0x55d9bdba4800 con 0 > -3067> 2018-07-11 07:51:58.937195 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -3066> 2018-07-11 07:51:58.937223 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -3065> 2018-07-11 07:51:58.937294 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -3064> 2018-07-11 07:51:58.937763 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -3063> 2018-07-11 07:51:58.953120 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67942,epoch:501/498,start:5:756b892a:::benchmark_data_magna115_10867_object1018678:0,end:5:756be1a9:::benchmark_data_magna115_10867_object1661033:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454780 con 0 > -3062> 2018-07-11 07:51:58.953172 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67942,epoch:501/498,start:5:756b892a:::benchmark_data_magna115_10867_object1018678:0,end:5:756be1a9:::benchmark_data_magna115_10867_object1661033:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455400 con 0 > -3061> 2018-07-11 07:51:58.964302 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6805/15263 conn(0x55d986c41000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=31 cs=1 l=0). rx osd.5 seq 2836 0x55d986d4fe00 MOSDScrubReserve(6.15 REQUEST e501) v1 > -3060> 2018-07-11 07:51:58.964319 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.5 10.8.128.113:6805/15263 2836 ==== MOSDScrubReserve(6.15 REQUEST e501) v1 ==== 43+0+0 (3942742026 0 0) 0x55d986d4fe00 con 0x55d986c41000 > -3059> 2018-07-11 07:51:59.146735 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1011 0x55d9b1501680 rep_scrubmap(5.2e e501 from shard 2) v1 > -3058> 2018-07-11 07:51:59.146775 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1011 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12468 (1269249638 0 2478807716) 0x55d9b1501680 con 0x55d986bf0800 > -3057> 2018-07-11 07:51:59.164927 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1005 0x55d9b15018c0 rep_scrubmap(5.2e e501 from shard 4) v1 > -3056> 2018-07-11 07:51:59.164946 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1005 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12468 (3686815630 0 2478807716) 0x55d9b15018c0 con 0x55d986bec000 > -3055> 2018-07-11 07:51:59.470432 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15255 conn(0x55d986d20000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.0 seq 259 0x55d9b60df400 osd_ping(ping e501 stamp 2018-07-11 07:52:00.701729) v4 > -3054> 2018-07-11 07:51:59.470463 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.0 10.8.128.113:0/15255 259 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:00.701729) v4 ==== 2004+0+0 (3789482413 0 0) 0x55d9b60df400 con 0x55d986d20000 > -3053> 2018-07-11 07:51:59.470466 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15255 conn(0x55d986c82800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.0 seq 259 0x55d982cdf800 osd_ping(ping e501 stamp 2018-07-11 07:52:00.701729) v4 > -3052> 2018-07-11 07:51:59.470476 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:00.701729) v4 -- 0x55d9829e4800 con 0 > -3051> 2018-07-11 07:51:59.470494 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.0 10.8.128.113:0/15255 259 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:00.701729) v4 ==== 2004+0+0 (3789482413 0 0) 0x55d982cdf800 con 0x55d986c82800 > -3050> 2018-07-11 07:51:59.470519 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:00.701729) v4 -- 0x55d982d31600 con 0 > -3049> 2018-07-11 07:51:59.666978 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12677 conn(0x55d986db4800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.1 seq 261 0x55d9869d9800 osd_ping(ping e501 stamp 2018-07-11 07:51:59.714908) v4 > -3048> 2018-07-11 07:51:59.666990 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12677 conn(0x55d986df2000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.1 seq 261 0x55d9bdba4800 osd_ping(ping e501 stamp 2018-07-11 07:51:59.714908) v4 > -3047> 2018-07-11 07:51:59.667006 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.1 10.8.128.106:0/12677 261 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:59.714908) v4 ==== 2004+0+0 (1307099649 0 0) 0x55d9869d9800 con 0x55d986db4800 > -3046> 2018-07-11 07:51:59.667006 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.1 10.8.128.106:0/12677 261 ==== osd_ping(ping e501 stamp 2018-07-11 07:51:59.714908) v4 ==== 2004+0+0 (1307099649 0 0) 0x55d9bdba4800 con 0x55d986df2000 > -3045> 2018-07-11 07:51:59.667017 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:59.714908) v4 -- 0x55d982d31600 con 0 > -3044> 2018-07-11 07:51:59.667043 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:51:59.714908) v4 -- 0x55d9829e4800 con 0 > -3043> 2018-07-11 07:52:00.140218 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6805/15263 -- MOSDScrubReserve(6.15 REJECT e501) v1 -- 0x55d986c22e00 con 0 > -3042> 2018-07-11 07:52:00.140461 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -3041> 2018-07-11 07:52:00.140488 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -3040> 2018-07-11 07:52:00.140534 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -3039> 2018-07-11 07:52:00.140845 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6805/15263 conn(0x55d986c41000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=31 cs=1 l=0). rx osd.5 seq 2837 0x55d986c22e00 MOSDScrubReserve(6.15 RELEASE e501) v1 > -3038> 2018-07-11 07:52:00.140865 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.5 10.8.128.113:6805/15263 2837 ==== MOSDScrubReserve(6.15 RELEASE e501) v1 ==== 43+0+0 (605848072 0 0) 0x55d986c22e00 con 0x55d986c41000 > -3037> 2018-07-11 07:52:00.141018 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -3036> 2018-07-11 07:52:00.180140 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'66542,epoch:501/498,start:5:756be1a9:::benchmark_data_magna115_10867_object1661033:0,end:5:756c3102:::benchmark_data_magna115_10867_object2287015:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455b80 con 0 > -3035> 2018-07-11 07:52:00.180188 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'66542,epoch:501/498,start:5:756be1a9:::benchmark_data_magna115_10867_object1661033:0,end:5:756c3102:::benchmark_data_magna115_10867_object2287015:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498280 con 0 > -3034> 2018-07-11 07:52:00.259916 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 33] Generated table #36808: 2638241 keys, 74546238 bytes > -3033> 2018-07-11 07:52:00.259959 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295520259932, "cf_name": "default", "job": 33, "event": "table_file_creation", "file_number": 36808, "file_size": 74546238, "table_properties": {"data_size": 67112128, "index_size": 689937, "filter_size": 6743183, "raw_key_size": 76215852, "raw_average_key_size": 28, "raw_value_size": 31365755, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} > -3032> 2018-07-11 07:52:00.398496 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1012 0x55d9b1501b00 rep_scrubmap(5.2e e501 from shard 2) v1 > -3031> 2018-07-11 07:52:00.398518 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1012 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12476 (1269249638 0 919452336) 0x55d9b1501b00 con 0x55d986bf0800 > -3030> 2018-07-11 07:52:00.421653 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.114:0/15885 conn(0x55d986d7f800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.7 seq 265 0x55d982cdf800 osd_ping(ping e501 stamp 2018-07-11 07:52:01.052050) v4 > -3029> 2018-07-11 07:52:00.421681 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.7 10.8.128.114:0/15885 265 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:01.052050) v4 ==== 2004+0+0 (3448585417 0 0) 0x55d982cdf800 con 0x55d986d7f800 > -3028> 2018-07-11 07:52:00.421696 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:01.052050) v4 -- 0x55d982d31600 con 0 > -3027> 2018-07-11 07:52:00.421694 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.114:0/15885 conn(0x55d986d81000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.7 seq 265 0x55d9b60df400 osd_ping(ping e501 stamp 2018-07-11 07:52:01.052050) v4 > -3026> 2018-07-11 07:52:00.421720 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.7 10.8.128.114:0/15885 265 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:01.052050) v4 ==== 2004+0+0 (3448585417 0 0) 0x55d9b60df400 con 0x55d986d81000 > -3025> 2018-07-11 07:52:00.421736 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:01.052050) v4 -- 0x55d9829e4800 con 0 > -3024> 2018-07-11 07:52:00.426188 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1006 0x55d9b1501d40 rep_scrubmap(5.2e e501 from shard 4) v1 > -3023> 2018-07-11 07:52:00.426209 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1006 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12476 (3686815630 0 919452336) 0x55d9b1501d40 con 0x55d986bec000 > -3022> 2018-07-11 07:52:00.495773 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.114:0/15919 conn(0x55d986df3800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.11 seq 260 0x55d980f5a800 osd_ping(ping e501 stamp 2018-07-11 07:52:01.126073) v4 > -3021> 2018-07-11 07:52:00.495774 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.114:0/15919 conn(0x55d986df5000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.11 seq 260 0x55d9869d9800 osd_ping(ping e501 stamp 2018-07-11 07:52:01.126073) v4 > -3020> 2018-07-11 07:52:00.495801 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.11 10.8.128.114:0/15919 260 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:01.126073) v4 ==== 2004+0+0 (4205105065 0 0) 0x55d980f5a800 con 0x55d986df3800 > -3019> 2018-07-11 07:52:00.495806 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.11 10.8.128.114:0/15919 260 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:01.126073) v4 ==== 2004+0+0 (4205105065 0 0) 0x55d9869d9800 con 0x55d986df5000 > -3018> 2018-07-11 07:52:00.495812 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:01.126073) v4 -- 0x55d9829e5800 con 0 > -3017> 2018-07-11 07:52:00.495835 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:01.126073) v4 -- 0x55d982d31600 con 0 > -3016> 2018-07-11 07:52:00.504912 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.115:0/13323 conn(0x55d986d7e000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 259 0x55d9869be000 osd_ping(ping e501 stamp 2018-07-11 07:52:00.850842) v4 > -3015> 2018-07-11 07:52:00.504934 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.6 10.8.128.115:0/13323 259 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:00.850842) v4 ==== 2004+0+0 (546491803 0 0) 0x55d9869be000 con 0x55d986d7e000 > -3014> 2018-07-11 07:52:00.504936 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.115:0/13323 conn(0x55d986d24800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 259 0x55d9b60df400 osd_ping(ping e501 stamp 2018-07-11 07:52:00.850842) v4 > -3013> 2018-07-11 07:52:00.504950 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:00.850842) v4 -- 0x55d9829e5800 con 0 > -3012> 2018-07-11 07:52:00.504964 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.6 10.8.128.115:0/13323 259 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:00.850842) v4 ==== 2004+0+0 (546491803 0 0) 0x55d9b60df400 con 0x55d986d24800 > -3011> 2018-07-11 07:52:00.504982 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:00.850842) v4 -- 0x55d9829e4800 con 0 > -3010> 2018-07-11 07:52:00.521111 7ffab1bdc700 5 osd.10 501 heartbeat: osd_stat(788 GB used, 142 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,2]) > -3009> 2018-07-11 07:52:00.521125 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6802/15255 -- osd_ping(ping e501 stamp 2018-07-11 07:52:00.521124) v4 -- 0x55d9b40a4000 con 0 > -3008> 2018-07-11 07:52:00.521162 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6803/15255 -- osd_ping(ping e501 stamp 2018-07-11 07:52:00.521124) v4 -- 0x55d9b9ac2000 con 0 > -3007> 2018-07-11 07:52:00.521176 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6806/12677 -- osd_ping(ping e501 stamp 2018-07-11 07:52:00.521124) v4 -- 0x55d980f5a000 con 0 > -3006> 2018-07-11 07:52:00.521189 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6807/12677 -- osd_ping(ping e501 stamp 2018-07-11 07:52:00.521124) v4 -- 0x55d986d4e600 con 0 > -3005> 2018-07-11 07:52:00.521212 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6808/12651 -- osd_ping(ping e501 stamp 2018-07-11 07:52:00.521124) v4 -- 0x55d9bcd72800 con 0 > -3004> 2018-07-11 07:52:00.521223 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6810/12651 -- osd_ping(ping e501 stamp 2018-07-11 07:52:00.521124) v4 -- 0x55d982cdf400 con 0 > -3003> 2018-07-11 07:52:00.521233 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6809/12650 -- osd_ping(ping e501 stamp 2018-07-11 07:52:00.521124) v4 -- 0x55d9bdba4200 con 0 > -3002> 2018-07-11 07:52:00.521258 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6811/12650 -- osd_ping(ping e501 stamp 2018-07-11 07:52:00.521124) v4 -- 0x55d982d30400 con 0 > -3001> 2018-07-11 07:52:00.521274 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6810/15287 -- osd_ping(ping e501 stamp 2018-07-11 07:52:00.521124) v4 -- 0x55d986d41600 con 0 > -3000> 2018-07-11 07:52:00.521292 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6811/15287 -- osd_ping(ping e501 stamp 2018-07-11 07:52:00.521124) v4 -- 0x55d9c3383c00 con 0 > -2999> 2018-07-11 07:52:00.521306 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6806/15263 -- osd_ping(ping e501 stamp 2018-07-11 07:52:00.521124) v4 -- 0x55d9b2783400 con 0 > -2998> 2018-07-11 07:52:00.521319 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6807/15263 -- osd_ping(ping e501 stamp 2018-07-11 07:52:00.521124) v4 -- 0x55d980f5a200 con 0 > -2997> 2018-07-11 07:52:00.521329 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.115:6802/13323 -- osd_ping(ping e501 stamp 2018-07-11 07:52:00.521124) v4 -- 0x55d986c23a00 con 0 > -2996> 2018-07-11 07:52:00.521342 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.115:6803/13323 -- osd_ping(ping e501 stamp 2018-07-11 07:52:00.521124) v4 -- 0x55d9b9ac3200 con 0 > -2995> 2018-07-11 07:52:00.521355 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6806/15885 -- osd_ping(ping e501 stamp 2018-07-11 07:52:00.521124) v4 -- 0x55d9829e4200 con 0 > -2994> 2018-07-11 07:52:00.521368 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6807/15885 -- osd_ping(ping e501 stamp 2018-07-11 07:52:00.521124) v4 -- 0x55d9b2528000 con 0 > -2993> 2018-07-11 07:52:00.521383 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.108:6802/11967 -- osd_ping(ping e501 stamp 2018-07-11 07:52:00.521124) v4 -- 0x55d9c3383200 con 0 > -2992> 2018-07-11 07:52:00.521396 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.108:6803/11967 -- osd_ping(ping e501 stamp 2018-07-11 07:52:00.521124) v4 -- 0x55d986d33600 con 0 > -2991> 2018-07-11 07:52:00.521409 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.107:6806/12617 -- osd_ping(ping e501 stamp 2018-07-11 07:52:00.521124) v4 -- 0x55d9b9ac3400 con 0 > -2990> 2018-07-11 07:52:00.521422 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.107:6807/12617 -- osd_ping(ping e501 stamp 2018-07-11 07:52:00.521124) v4 -- 0x55d9b60dfa00 con 0 > -2989> 2018-07-11 07:52:00.521437 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6802/15919 -- osd_ping(ping e501 stamp 2018-07-11 07:52:00.521124) v4 -- 0x55d9829e5a00 con 0 > -2988> 2018-07-11 07:52:00.521449 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6803/15919 -- osd_ping(ping e501 stamp 2018-07-11 07:52:00.521124) v4 -- 0x55d9bdba5e00 con 0 > -2987> 2018-07-11 07:52:00.521592 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.107:6806/12617 conn(0x55d986bd2000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.9 seq 249 0x55d982cdf800 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:00.521124) v4 > -2986> 2018-07-11 07:52:00.521615 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.9 10.8.128.107:6806/12617 249 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:00.521124) v4 ==== 2004+0+0 (365664941 0 0) 0x55d982cdf800 con 0x55d986bd2000 > -2985> 2018-07-11 07:52:00.521625 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.107:6807/12617 conn(0x55d986bd3800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.9 seq 249 0x55d980f5a800 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:00.521124) v4 > -2984> 2018-07-11 07:52:00.521641 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.9 10.8.128.107:6807/12617 249 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:00.521124) v4 ==== 2004+0+0 (365664941 0 0) 0x55d980f5a800 con 0x55d986bd3800 > -2983> 2018-07-11 07:52:00.521736 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6809/12650 conn(0x55d986bcb000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 249 0x55d980f5a800 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:00.521124) v4 > -2982> 2018-07-11 07:52:00.521744 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6810/12651 conn(0x55d986b9b800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.2 seq 249 0x55d982cdf800 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:00.521124) v4 > -2981> 2018-07-11 07:52:00.521762 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.3 10.8.128.106:6809/12650 249 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:00.521124) v4 ==== 2004+0+0 (365664941 0 0) 0x55d980f5a800 con 0x55d986bcb000 > -2980> 2018-07-11 07:52:00.521763 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.2 10.8.128.106:6810/12651 249 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:00.521124) v4 ==== 2004+0+0 (365664941 0 0) 0x55d982cdf800 con 0x55d986b9b800 > -2979> 2018-07-11 07:52:00.521772 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6808/12651 conn(0x55d986b9a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.2 seq 249 0x55d9b60df400 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:00.521124) v4 > -2978> 2018-07-11 07:52:00.521788 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.2 10.8.128.106:6808/12651 249 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:00.521124) v4 ==== 2004+0+0 (365664941 0 0) 0x55d9b60df400 con 0x55d986b9a000 > -2977> 2018-07-11 07:52:00.521808 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6806/15263 conn(0x55d986b9d000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.5 seq 249 0x55d980f5a800 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:00.521124) v4 > -2976> 2018-07-11 07:52:00.521811 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.115:6802/13323 conn(0x55d986bc1000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 249 0x55d982cdf800 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:00.521124) v4 > -2975> 2018-07-11 07:52:00.521830 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.5 10.8.128.113:6806/15263 249 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:00.521124) v4 ==== 2004+0+0 (365664941 0 0) 0x55d980f5a800 con 0x55d986b9d000 > -2974> 2018-07-11 07:52:00.521838 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6802/15255 conn(0x55d986bc8000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.0 seq 249 0x55d9b60df400 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:00.521124) v4 > -2973> 2018-07-11 07:52:00.521833 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.6 10.8.128.115:6802/13323 249 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:00.521124) v4 ==== 2004+0+0 (365664941 0 0) 0x55d982cdf800 con 0x55d986bc1000 > -2972> 2018-07-11 07:52:00.521851 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.0 10.8.128.113:6802/15255 249 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:00.521124) v4 ==== 2004+0+0 (365664941 0 0) 0x55d9b60df400 con 0x55d986bc8000 > -2971> 2018-07-11 07:52:00.521879 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6806/12677 conn(0x55d986b58000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.1 seq 249 0x55d980f5a800 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:00.521124) v4 > -2970> 2018-07-11 07:52:00.521888 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6803/15255 conn(0x55d986bc9800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.0 seq 249 0x55d982cdf800 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:00.521124) v4 > -2969> 2018-07-11 07:52:00.521896 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6811/12650 conn(0x55d986bcc800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 249 0x55d9b60df400 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:00.521124) v4 > -2968> 2018-07-11 07:52:00.521904 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.0 10.8.128.113:6803/15255 249 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:00.521124) v4 ==== 2004+0+0 (365664941 0 0) 0x55d982cdf800 con 0x55d986bc9800 > -2967> 2018-07-11 07:52:00.521893 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.1 10.8.128.106:6806/12677 249 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:00.521124) v4 ==== 2004+0+0 (365664941 0 0) 0x55d980f5a800 con 0x55d986b58000 > -2966> 2018-07-11 07:52:00.521911 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.3 10.8.128.106:6811/12650 249 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:00.521124) v4 ==== 2004+0+0 (365664941 0 0) 0x55d9b60df400 con 0x55d986bcc800 > -2965> 2018-07-11 07:52:00.521943 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6807/12677 conn(0x55d986b59800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.1 seq 249 0x55d9b60df400 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:00.521124) v4 > -2964> 2018-07-11 07:52:00.521951 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6811/15287 conn(0x55d986bb0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 249 0x55d982cdf800 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:00.521124) v4 > -2963> 2018-07-11 07:52:00.521955 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.1 10.8.128.106:6807/12677 249 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:00.521124) v4 ==== 2004+0+0 (365664941 0 0) 0x55d9b60df400 con 0x55d986b59800 > -2962> 2018-07-11 07:52:00.521963 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.4 10.8.128.113:6811/15287 249 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:00.521124) v4 ==== 2004+0+0 (365664941 0 0) 0x55d982cdf800 con 0x55d986bb0800 > -2961> 2018-07-11 07:52:00.521984 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6807/15263 conn(0x55d986b9e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.5 seq 249 0x55d9b60df400 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:00.521124) v4 > -2960> 2018-07-11 07:52:00.521990 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.108:6802/11967 conn(0x55d986bbe000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.8 seq 249 0x55d980f5a800 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:00.521124) v4 > -2959> 2018-07-11 07:52:00.522005 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.8 10.8.128.108:6802/11967 249 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:00.521124) v4 ==== 2004+0+0 (365664941 0 0) 0x55d980f5a800 con 0x55d986bbe000 > -2958> 2018-07-11 07:52:00.521996 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.5 10.8.128.113:6807/15263 249 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:00.521124) v4 ==== 2004+0+0 (365664941 0 0) 0x55d9b60df400 con 0x55d986b9e800 > -2957> 2018-07-11 07:52:00.522036 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6807/15885 conn(0x55d986bad800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.7 seq 249 0x55d980f5a800 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:00.521124) v4 > -2956> 2018-07-11 07:52:00.522048 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.7 10.8.128.114:6807/15885 249 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:00.521124) v4 ==== 2004+0+0 (365664941 0 0) 0x55d980f5a800 con 0x55d986bad800 > -2955> 2018-07-11 07:52:00.522044 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6806/15885 conn(0x55d986bac000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.7 seq 249 0x55d982cdf800 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:00.521124) v4 > -2954> 2018-07-11 07:52:00.522049 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6810/15287 conn(0x55d986baf000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 249 0x55d9b60df400 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:00.521124) v4 > -2953> 2018-07-11 07:52:00.522061 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.4 10.8.128.113:6810/15287 249 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:00.521124) v4 ==== 2004+0+0 (365664941 0 0) 0x55d9b60df400 con 0x55d986baf000 > -2952> 2018-07-11 07:52:00.522069 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.115:6803/13323 conn(0x55d986bc2800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 249 0x55d980f5a800 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:00.521124) v4 > -2951> 2018-07-11 07:52:00.522062 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.7 10.8.128.114:6806/15885 249 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:00.521124) v4 ==== 2004+0+0 (365664941 0 0) 0x55d982cdf800 con 0x55d986bac000 > -2950> 2018-07-11 07:52:00.522081 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.6 10.8.128.115:6803/13323 249 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:00.521124) v4 ==== 2004+0+0 (365664941 0 0) 0x55d980f5a800 con 0x55d986bc2800 > -2949> 2018-07-11 07:52:00.522095 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.108:6803/11967 conn(0x55d986bbf800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.8 seq 249 0x55d9b60df400 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:00.521124) v4 > -2948> 2018-07-11 07:52:00.522106 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.8 10.8.128.108:6803/11967 249 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:00.521124) v4 ==== 2004+0+0 (365664941 0 0) 0x55d9b60df400 con 0x55d986bbf800 > -2947> 2018-07-11 07:52:00.522106 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6802/15919 conn(0x55d986b5b000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.11 seq 249 0x55d982cdf800 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:00.521124) v4 > -2946> 2018-07-11 07:52:00.522116 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.11 10.8.128.114:6802/15919 249 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:00.521124) v4 ==== 2004+0+0 (365664941 0 0) 0x55d982cdf800 con 0x55d986b5b000 > -2945> 2018-07-11 07:52:00.522123 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6803/15919 conn(0x55d986b5c800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.11 seq 249 0x55d980f5a800 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:00.521124) v4 > -2944> 2018-07-11 07:52:00.522136 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.11 10.8.128.114:6803/15919 249 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:00.521124) v4 ==== 2004+0+0 (365664941 0 0) 0x55d980f5a800 con 0x55d986b5c800 > -2943> 2018-07-11 07:52:00.664226 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -2942> 2018-07-11 07:52:00.664277 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -2941> 2018-07-11 07:52:00.664327 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -2940> 2018-07-11 07:52:00.664789 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -2939> 2018-07-11 07:52:00.686762 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'66857,epoch:501/498,start:5:756c3102:::benchmark_data_magna115_10867_object2287015:0,end:5:756c9665:::benchmark_data_magna115_10867_object178838:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b14f0500 con 0 > -2938> 2018-07-11 07:52:00.686813 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'66857,epoch:501/498,start:5:756c3102:::benchmark_data_magna115_10867_object2287015:0,end:5:756c9665:::benchmark_data_magna115_10867_object178838:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498000 con 0 > -2937> 2018-07-11 07:52:00.842983 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 3020 0x55d980f5a800 MOSDScrubReserve(11.3s5 REQUEST e501) v1 > -2936> 2018-07-11 07:52:00.843001 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 3020 ==== MOSDScrubReserve(11.3s5 REQUEST e501) v1 ==== 43+0+0 (3260170403 0 0) 0x55d980f5a800 con 0x55d986c7e000 > -2935> 2018-07-11 07:52:00.843121 7ffab3be0700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e501) v1 -- 0x55d9bcd78600 con 0 > -2934> 2018-07-11 07:52:00.843432 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 3021 0x55d9bdba5e00 MOSDScrubReserve(11.3s5 RELEASE e501) v1 > -2933> 2018-07-11 07:52:00.843460 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 3021 ==== MOSDScrubReserve(11.3s5 RELEASE e501) v1 ==== 43+0+0 (223328929 0 0) 0x55d9bdba5e00 con 0x55d986c7e000 > -2932> 2018-07-11 07:52:00.896856 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1013 0x55d9c48c0000 rep_scrubmap(5.2e e501 from shard 2) v1 > -2931> 2018-07-11 07:52:00.896878 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1013 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12464 (1269249638 0 94671118) 0x55d9c48c0000 con 0x55d986bf0800 > -2930> 2018-07-11 07:52:00.904866 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1007 0x55d9c48c0240 rep_scrubmap(5.2e e501 from shard 4) v1 > -2929> 2018-07-11 07:52:00.904886 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1007 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12464 (3686815630 0 94671118) 0x55d9c48c0240 con 0x55d986bec000 > -2928> 2018-07-11 07:52:00.965779 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6801/15919 conn(0x55d986c3f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=29 cs=1 l=0). rx osd.11 seq 248 0x55d9869be000 MOSDScrubReserve(10.f REQUEST e501) v1 > -2927> 2018-07-11 07:52:00.965796 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.11 10.8.128.114:6801/15919 248 ==== MOSDScrubReserve(10.f REQUEST e501) v1 ==== 43+0+0 (2892966580 0 0) 0x55d9869be000 con 0x55d986c3f800 > -2926> 2018-07-11 07:52:00.965919 7ffab53e3700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.114:6801/15919 -- MOSDScrubReserve(10.f REJECT e501) v1 -- 0x55d9b4388200 con 0 > -2925> 2018-07-11 07:52:00.966196 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6801/15919 conn(0x55d986c3f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=29 cs=1 l=0). rx osd.11 seq 249 0x55d9b4388200 MOSDScrubReserve(10.f RELEASE e501) v1 > -2924> 2018-07-11 07:52:00.966210 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.11 10.8.128.114:6801/15919 249 ==== MOSDScrubReserve(10.f RELEASE e501) v1 ==== 43+0+0 (1668468918 0 0) 0x55d9b4388200 con 0x55d986c3f800 > -2923> 2018-07-11 07:52:00.996078 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.114:0/15919 conn(0x55d986df5000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.11 seq 261 0x55d982cdf800 osd_ping(ping e501 stamp 2018-07-11 07:52:01.626510) v4 > -2922> 2018-07-11 07:52:00.996085 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.114:0/15919 conn(0x55d986df3800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.11 seq 261 0x55d9bcd78600 osd_ping(ping e501 stamp 2018-07-11 07:52:01.626510) v4 > -2921> 2018-07-11 07:52:00.996121 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.11 10.8.128.114:0/15919 261 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:01.626510) v4 ==== 2004+0+0 (679193078 0 0) 0x55d982cdf800 con 0x55d986df5000 > -2920> 2018-07-11 07:52:00.996124 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.11 10.8.128.114:0/15919 261 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:01.626510) v4 ==== 2004+0+0 (679193078 0 0) 0x55d9bcd78600 con 0x55d986df3800 > -2919> 2018-07-11 07:52:00.996137 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:01.626510) v4 -- 0x55d9829e5a00 con 0 > -2918> 2018-07-11 07:52:00.996162 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:01.626510) v4 -- 0x55d9c3383200 con 0 > -2917> 2018-07-11 07:52:01.033423 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -2916> 2018-07-11 07:52:01.033446 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -2915> 2018-07-11 07:52:01.033504 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -2914> 2018-07-11 07:52:01.033990 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -2913> 2018-07-11 07:52:01.078564 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:756c9665:::benchmark_data_magna115_10867_object178838:0,end:5:756d06de:::benchmark_data_magna115_10867_object2822829:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455900 con 0 > -2912> 2018-07-11 07:52:01.078616 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:756c9665:::benchmark_data_magna115_10867_object178838:0,end:5:756d06de:::benchmark_data_magna115_10867_object2822829:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455680 con 0 > -2911> 2018-07-11 07:52:01.126891 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12651 conn(0x55d986db3000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.2 seq 258 0x55d9b60dfa00 osd_ping(ping e501 stamp 2018-07-11 07:52:01.174823) v4 > -2910> 2018-07-11 07:52:01.126901 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12651 conn(0x55d986db1800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.2 seq 258 0x55d9b60df400 osd_ping(ping e501 stamp 2018-07-11 07:52:01.174823) v4 > -2909> 2018-07-11 07:52:01.126912 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.2 10.8.128.106:0/12651 258 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:01.174823) v4 ==== 2004+0+0 (1679030248 0 0) 0x55d9b60dfa00 con 0x55d986db3000 > -2908> 2018-07-11 07:52:01.126922 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:01.174823) v4 -- 0x55d9c3383200 con 0 > -2907> 2018-07-11 07:52:01.126921 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.2 10.8.128.106:0/12651 258 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:01.174823) v4 ==== 2004+0+0 (1679030248 0 0) 0x55d9b60df400 con 0x55d986db1800 > -2906> 2018-07-11 07:52:01.126941 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:01.174823) v4 -- 0x55d986d33600 con 0 > -2905> 2018-07-11 07:52:01.250934 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15263 conn(0x55d985d45800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.5 seq 242 0x55d9869d9800 osd_ping(ping e501 stamp 2018-07-11 07:52:02.482387) v4 > -2904> 2018-07-11 07:52:01.250955 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.5 10.8.128.113:0/15263 242 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:02.482387) v4 ==== 2004+0+0 (2625631913 0 0) 0x55d9869d9800 con 0x55d985d45800 > -2903> 2018-07-11 07:52:01.250950 7ffacbbd1700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15263 conn(0x55d985d44000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.5 seq 242 0x55d9bcd78600 osd_ping(ping e501 stamp 2018-07-11 07:52:02.482387) v4 > -2902> 2018-07-11 07:52:01.250966 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:02.482387) v4 -- 0x55d9829e5a00 con 0 > -2901> 2018-07-11 07:52:01.250967 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 <== osd.5 10.8.128.113:0/15263 242 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:02.482387) v4 ==== 2004+0+0 (2625631913 0 0) 0x55d9bcd78600 con 0x55d985d44000 > -2900> 2018-07-11 07:52:01.250989 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:02.482387) v4 -- 0x55d9c3383200 con 0 > -2899> 2018-07-11 07:52:01.293001 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1014 0x55d9c48c0480 rep_scrubmap(5.2e e501 from shard 2) v1 > -2898> 2018-07-11 07:52:01.293020 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1014 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12476 (1269249638 0 1276163420) 0x55d9c48c0480 con 0x55d986bf0800 > -2897> 2018-07-11 07:52:01.337020 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1008 0x55d9c48c06c0 rep_scrubmap(5.2e e501 from shard 4) v1 > -2896> 2018-07-11 07:52:01.337040 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1008 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12476 (3686815630 0 1276163420) 0x55d9c48c06c0 con 0x55d986bec000 > -2895> 2018-07-11 07:52:01.337190 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -2894> 2018-07-11 07:52:01.337216 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -2893> 2018-07-11 07:52:01.337272 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -2892> 2018-07-11 07:52:01.337730 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -2891> 2018-07-11 07:52:01.338601 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:756d06de:::benchmark_data_magna115_10867_object2822829:0,end:5:756d72d2:::benchmark_data_magna115_10867_object1896560:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498780 con 0 > -2890> 2018-07-11 07:52:01.338648 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:756d06de:::benchmark_data_magna115_10867_object2822829:0,end:5:756d72d2:::benchmark_data_magna115_10867_object1896560:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98696f400 con 0 > -2889> 2018-07-11 07:52:01.566269 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1015 0x55d9c48c0900 rep_scrubmap(5.2e e501 from shard 2) v1 > -2888> 2018-07-11 07:52:01.566288 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1015 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12478 (1269249638 0 693507247) 0x55d9c48c0900 con 0x55d986bf0800 > -2887> 2018-07-11 07:52:01.567758 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1009 0x55d9c48c0b40 rep_scrubmap(5.2e e501 from shard 4) v1 > -2886> 2018-07-11 07:52:01.567777 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1009 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12478 (3686815630 0 693507247) 0x55d9c48c0b40 con 0x55d986bec000 > -2885> 2018-07-11 07:52:01.658502 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -2884> 2018-07-11 07:52:01.658526 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -2883> 2018-07-11 07:52:01.658582 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -2882> 2018-07-11 07:52:01.659031 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -2881> 2018-07-11 07:52:01.688516 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15287 conn(0x55d981240800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 273 0x55d9b60df400 osd_ping(ping e501 stamp 2018-07-11 07:52:02.919958) v4 > -2880> 2018-07-11 07:52:01.688540 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.4 10.8.128.113:0/15287 273 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:02.919958) v4 ==== 2004+0+0 (3398416423 0 0) 0x55d9b60df400 con 0x55d981240800 > -2879> 2018-07-11 07:52:01.688551 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:02.919958) v4 -- 0x55d986d33600 con 0 > -2878> 2018-07-11 07:52:01.688561 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15287 conn(0x55d98123f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 273 0x55d982cdf800 osd_ping(ping e501 stamp 2018-07-11 07:52:02.919958) v4 > -2877> 2018-07-11 07:52:01.688583 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.4 10.8.128.113:0/15287 273 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:02.919958) v4 ==== 2004+0+0 (3398416423 0 0) 0x55d982cdf800 con 0x55d98123f000 > -2876> 2018-07-11 07:52:01.688594 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:02.919958) v4 -- 0x55d9829e5a00 con 0 > -2875> 2018-07-11 07:52:01.703765 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:756d72d2:::benchmark_data_magna115_10867_object1896560:0,end:5:756ded4e:::benchmark_data_magna115_10867_object925944:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b14f0280 con 0 > -2874> 2018-07-11 07:52:01.703817 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:756d72d2:::benchmark_data_magna115_10867_object1896560:0,end:5:756ded4e:::benchmark_data_magna115_10867_object925944:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b14f0000 con 0 > -2873> 2018-07-11 07:52:01.909316 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1016 0x55d9c48c0d80 rep_scrubmap(5.2e e501 from shard 2) v1 > -2872> 2018-07-11 07:52:01.909337 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1016 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12474 (1269249638 0 1450830926) 0x55d9c48c0d80 con 0x55d986bf0800 > -2871> 2018-07-11 07:52:01.971056 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1010 0x55d9c48c0fc0 rep_scrubmap(5.2e e501 from shard 4) v1 > -2870> 2018-07-11 07:52:01.971075 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1010 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12474 (3686815630 0 1450830926) 0x55d9c48c0fc0 con 0x55d986bec000 > -2869> 2018-07-11 07:52:02.008856 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -2868> 2018-07-11 07:52:02.008882 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -2867> 2018-07-11 07:52:02.008937 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -2866> 2018-07-11 07:52:02.009411 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -2865> 2018-07-11 07:52:02.010258 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:756ded4e:::benchmark_data_magna115_10867_object925944:0,end:5:756e4868:::benchmark_data_magna115_10867_object1847287:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1499680 con 0 > -2864> 2018-07-11 07:52:02.010293 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:756ded4e:::benchmark_data_magna115_10867_object925944:0,end:5:756e4868:::benchmark_data_magna115_10867_object1847287:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1499900 con 0 > -2863> 2018-07-11 07:52:02.053707 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.107:0/12617 conn(0x55d98123c000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 250 0x55d9bdba4800 osd_ping(ping e501 stamp 2018-07-11 07:52:02.053376) v4 > -2862> 2018-07-11 07:52:02.053729 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.9 10.8.128.107:0/12617 250 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:02.053376) v4 ==== 2004+0+0 (3682028198 0 0) 0x55d9bdba4800 con 0x55d98123c000 > -2861> 2018-07-11 07:52:02.053738 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:02.053376) v4 -- 0x55d986d33600 con 0 > -2860> 2018-07-11 07:52:02.053745 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.107:0/12617 conn(0x55d98123d800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 250 0x55d9b60dfa00 osd_ping(ping e501 stamp 2018-07-11 07:52:02.053376) v4 > -2859> 2018-07-11 07:52:02.053772 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.9 10.8.128.107:0/12617 250 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:02.053376) v4 ==== 2004+0+0 (3682028198 0 0) 0x55d9b60dfa00 con 0x55d98123d800 > -2858> 2018-07-11 07:52:02.053788 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:02.053376) v4 -- 0x55d9c3383200 con 0 > -2857> 2018-07-11 07:52:02.096640 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.114:0/15919 conn(0x55d986df3800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.11 seq 262 0x55d9bcd78600 osd_ping(ping e501 stamp 2018-07-11 07:52:02.727052) v4 > -2856> 2018-07-11 07:52:02.096666 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.11 10.8.128.114:0/15919 262 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:02.727052) v4 ==== 2004+0+0 (2585776256 0 0) 0x55d9bcd78600 con 0x55d986df3800 > -2855> 2018-07-11 07:52:02.096659 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.114:0/15919 conn(0x55d986df5000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.11 seq 262 0x55d9869d9800 osd_ping(ping e501 stamp 2018-07-11 07:52:02.727052) v4 > -2854> 2018-07-11 07:52:02.096680 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:02.727052) v4 -- 0x55d9c3383200 con 0 > -2853> 2018-07-11 07:52:02.096684 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.11 10.8.128.114:0/15919 262 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:02.727052) v4 ==== 2004+0+0 (2585776256 0 0) 0x55d9869d9800 con 0x55d986df5000 > -2852> 2018-07-11 07:52:02.096704 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:02.727052) v4 -- 0x55d9829e5a00 con 0 > -2851> 2018-07-11 07:52:02.212178 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1017 0x55d9c48c1200 rep_scrubmap(5.2e e501 from shard 2) v1 > -2850> 2018-07-11 07:52:02.212199 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1017 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12474 (1269249638 0 1254539758) 0x55d9c48c1200 con 0x55d986bf0800 > -2849> 2018-07-11 07:52:02.221634 7ffab1bdc700 5 osd.10 501 heartbeat: osd_stat(788 GB used, 142 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,1]) > -2848> 2018-07-11 07:52:02.221647 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6802/15255 -- osd_ping(ping e501 stamp 2018-07-11 07:52:02.221646) v4 -- 0x55d9c34cc000 con 0 > -2847> 2018-07-11 07:52:02.221683 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6803/15255 -- osd_ping(ping e501 stamp 2018-07-11 07:52:02.221646) v4 -- 0x55d9869bf400 con 0 > -2846> 2018-07-11 07:52:02.221697 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6806/12677 -- osd_ping(ping e501 stamp 2018-07-11 07:52:02.221646) v4 -- 0x55d986c15000 con 0 > -2845> 2018-07-11 07:52:02.221711 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6807/12677 -- osd_ping(ping e501 stamp 2018-07-11 07:52:02.221646) v4 -- 0x55d986d97200 con 0 > -2844> 2018-07-11 07:52:02.221721 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6808/12651 -- osd_ping(ping e501 stamp 2018-07-11 07:52:02.221646) v4 -- 0x55d9869bec00 con 0 > -2843> 2018-07-11 07:52:02.221745 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6810/12651 -- osd_ping(ping e501 stamp 2018-07-11 07:52:02.221646) v4 -- 0x55d9b9ac3000 con 0 > -2842> 2018-07-11 07:52:02.221756 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6809/12650 -- osd_ping(ping e501 stamp 2018-07-11 07:52:02.221646) v4 -- 0x55d9c3382e00 con 0 > -2841> 2018-07-11 07:52:02.221768 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6811/12650 -- osd_ping(ping e501 stamp 2018-07-11 07:52:02.221646) v4 -- 0x55d9b40a4c00 con 0 > -2840> 2018-07-11 07:52:02.221780 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6810/15287 -- osd_ping(ping e501 stamp 2018-07-11 07:52:02.221646) v4 -- 0x55d9b67d9c00 con 0 > -2839> 2018-07-11 07:52:02.221790 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6811/15287 -- osd_ping(ping e501 stamp 2018-07-11 07:52:02.221646) v4 -- 0x55d9bdba4e00 con 0 > -2838> 2018-07-11 07:52:02.221809 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6806/15263 -- osd_ping(ping e501 stamp 2018-07-11 07:52:02.221646) v4 -- 0x55d9b437b200 con 0 > -2837> 2018-07-11 07:52:02.221825 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6807/15263 -- osd_ping(ping e501 stamp 2018-07-11 07:52:02.221646) v4 -- 0x55d9b9ac2c00 con 0 > -2836> 2018-07-11 07:52:02.221837 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.115:6802/13323 -- osd_ping(ping e501 stamp 2018-07-11 07:52:02.221646) v4 -- 0x55d9869bf200 con 0 > -2835> 2018-07-11 07:52:02.221850 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.115:6803/13323 -- osd_ping(ping e501 stamp 2018-07-11 07:52:02.221646) v4 -- 0x55d9b2783e00 con 0 > -2834> 2018-07-11 07:52:02.221863 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6806/15885 -- osd_ping(ping e501 stamp 2018-07-11 07:52:02.221646) v4 -- 0x55d986d4f400 con 0 > -2833> 2018-07-11 07:52:02.221875 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6807/15885 -- osd_ping(ping e501 stamp 2018-07-11 07:52:02.221646) v4 -- 0x55d9c4c1b600 con 0 > -2832> 2018-07-11 07:52:02.221887 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.108:6802/11967 -- osd_ping(ping e501 stamp 2018-07-11 07:52:02.221646) v4 -- 0x55d986d4f800 con 0 > -2831> 2018-07-11 07:52:02.221898 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.108:6803/11967 -- osd_ping(ping e501 stamp 2018-07-11 07:52:02.221646) v4 -- 0x55d982cdfc00 con 0 > -2830> 2018-07-11 07:52:02.221910 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.107:6806/12617 -- osd_ping(ping e501 stamp 2018-07-11 07:52:02.221646) v4 -- 0x55d9869bf800 con 0 > -2829> 2018-07-11 07:52:02.221923 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.107:6807/12617 -- osd_ping(ping e501 stamp 2018-07-11 07:52:02.221646) v4 -- 0x55d986d97200 con 0 > -2828> 2018-07-11 07:52:02.221935 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6802/15919 -- osd_ping(ping e501 stamp 2018-07-11 07:52:02.221646) v4 -- 0x55d9c34cc000 con 0 > -2827> 2018-07-11 07:52:02.221946 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6803/15919 -- osd_ping(ping e501 stamp 2018-07-11 07:52:02.221646) v4 -- 0x55d986d33600 con 0 > -2826> 2018-07-11 07:52:02.222102 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.107:6806/12617 conn(0x55d986bd2000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.9 seq 250 0x55d982cdf800 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:02.221646) v4 > -2825> 2018-07-11 07:52:02.222124 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.9 10.8.128.107:6806/12617 250 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:02.221646) v4 ==== 2004+0+0 (2683204152 0 0) 0x55d982cdf800 con 0x55d986bd2000 > -2824> 2018-07-11 07:52:02.222120 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.107:6807/12617 conn(0x55d986bd3800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.9 seq 250 0x55d9b60dfa00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:02.221646) v4 > -2823> 2018-07-11 07:52:02.222138 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.9 10.8.128.107:6807/12617 250 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:02.221646) v4 ==== 2004+0+0 (2683204152 0 0) 0x55d9b60dfa00 con 0x55d986bd3800 > -2822> 2018-07-11 07:52:02.222260 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6810/12651 conn(0x55d986b9b800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.2 seq 250 0x55d982cdf800 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:02.221646) v4 > -2821> 2018-07-11 07:52:02.222268 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6809/12650 conn(0x55d986bcb000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 250 0x55d9b60dfa00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:02.221646) v4 > -2820> 2018-07-11 07:52:02.222282 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.2 10.8.128.106:6810/12651 250 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:02.221646) v4 ==== 2004+0+0 (2683204152 0 0) 0x55d982cdf800 con 0x55d986b9b800 > -2819> 2018-07-11 07:52:02.222299 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6808/12651 conn(0x55d986b9a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.2 seq 250 0x55d9bdba4800 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:02.221646) v4 > -2818> 2018-07-11 07:52:02.222303 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.3 10.8.128.106:6809/12650 250 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:02.221646) v4 ==== 2004+0+0 (2683204152 0 0) 0x55d9b60dfa00 con 0x55d986bcb000 > -2817> 2018-07-11 07:52:02.222316 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.2 10.8.128.106:6808/12651 250 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:02.221646) v4 ==== 2004+0+0 (2683204152 0 0) 0x55d9bdba4800 con 0x55d986b9a000 > -2816> 2018-07-11 07:52:02.222350 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6811/15287 conn(0x55d986bb0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 250 0x55d982cdf800 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:02.221646) v4 > -2815> 2018-07-11 07:52:02.222354 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6806/15263 conn(0x55d986b9d000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.5 seq 250 0x55d9b60dfa00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:02.221646) v4 > -2814> 2018-07-11 07:52:02.222365 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.4 10.8.128.113:6811/15287 250 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:02.221646) v4 ==== 2004+0+0 (2683204152 0 0) 0x55d982cdf800 con 0x55d986bb0800 > -2813> 2018-07-11 07:52:02.222365 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.5 10.8.128.113:6806/15263 250 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:02.221646) v4 ==== 2004+0+0 (2683204152 0 0) 0x55d9b60dfa00 con 0x55d986b9d000 > -2812> 2018-07-11 07:52:02.222369 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6807/12677 conn(0x55d986b59800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.1 seq 250 0x55d9bdba4800 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:02.221646) v4 > -2811> 2018-07-11 07:52:02.222398 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.1 10.8.128.106:6807/12677 250 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:02.221646) v4 ==== 2004+0+0 (2683204152 0 0) 0x55d9bdba4800 con 0x55d986b59800 > -2810> 2018-07-11 07:52:02.222385 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6803/15255 conn(0x55d986bc9800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.0 seq 250 0x55d982cdf800 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:02.221646) v4 > -2809> 2018-07-11 07:52:02.222416 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.0 10.8.128.113:6803/15255 250 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:02.221646) v4 ==== 2004+0+0 (2683204152 0 0) 0x55d982cdf800 con 0x55d986bc9800 > -2808> 2018-07-11 07:52:02.222413 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6806/12677 conn(0x55d986b58000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.1 seq 250 0x55d9b60dfa00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:02.221646) v4 > -2807> 2018-07-11 07:52:02.222434 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.1 10.8.128.106:6806/12677 250 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:02.221646) v4 ==== 2004+0+0 (2683204152 0 0) 0x55d9b60dfa00 con 0x55d986b58000 > -2806> 2018-07-11 07:52:02.222435 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6807/15263 conn(0x55d986b9e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.5 seq 250 0x55d9bdba4800 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:02.221646) v4 > -2805> 2018-07-11 07:52:02.222445 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.5 10.8.128.113:6807/15263 250 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:02.221646) v4 ==== 2004+0+0 (2683204152 0 0) 0x55d9bdba4800 con 0x55d986b9e800 > -2804> 2018-07-11 07:52:02.222451 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.115:6802/13323 conn(0x55d986bc1000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 250 0x55d982cdf800 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:02.221646) v4 > -2803> 2018-07-11 07:52:02.222464 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.6 10.8.128.115:6802/13323 250 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:02.221646) v4 ==== 2004+0+0 (2683204152 0 0) 0x55d982cdf800 con 0x55d986bc1000 > -2802> 2018-07-11 07:52:02.222475 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6802/15255 conn(0x55d986bc8000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.0 seq 250 0x55d9bdba4800 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:02.221646) v4 > -2801> 2018-07-11 07:52:02.222487 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.0 10.8.128.113:6802/15255 250 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:02.221646) v4 ==== 2004+0+0 (2683204152 0 0) 0x55d9bdba4800 con 0x55d986bc8000 > -2800> 2018-07-11 07:52:02.222471 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.115:6803/13323 conn(0x55d986bc2800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 250 0x55d9b60dfa00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:02.221646) v4 > -2799> 2018-07-11 07:52:02.222504 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.6 10.8.128.115:6803/13323 250 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:02.221646) v4 ==== 2004+0+0 (2683204152 0 0) 0x55d9b60dfa00 con 0x55d986bc2800 > -2798> 2018-07-11 07:52:02.222515 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6811/12650 conn(0x55d986bcc800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 250 0x55d9bdba4800 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:02.221646) v4 > -2797> 2018-07-11 07:52:02.222526 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.3 10.8.128.106:6811/12650 250 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:02.221646) v4 ==== 2004+0+0 (2683204152 0 0) 0x55d9bdba4800 con 0x55d986bcc800 > -2796> 2018-07-11 07:52:02.222539 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6807/15885 conn(0x55d986bad800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.7 seq 250 0x55d9b60dfa00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:02.221646) v4 > -2795> 2018-07-11 07:52:02.222551 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.7 10.8.128.114:6807/15885 250 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:02.221646) v4 ==== 2004+0+0 (2683204152 0 0) 0x55d9b60dfa00 con 0x55d986bad800 > -2794> 2018-07-11 07:52:02.222557 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6810/15287 conn(0x55d986baf000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 250 0x55d9bdba4800 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:02.221646) v4 > -2793> 2018-07-11 07:52:02.222569 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.4 10.8.128.113:6810/15287 250 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:02.221646) v4 ==== 2004+0+0 (2683204152 0 0) 0x55d9bdba4800 con 0x55d986baf000 > -2792> 2018-07-11 07:52:02.222566 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6806/15885 conn(0x55d986bac000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.7 seq 250 0x55d982cdf800 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:02.221646) v4 > -2791> 2018-07-11 07:52:02.222581 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.108:6802/11967 conn(0x55d986bbe000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.8 seq 250 0x55d9b60dfa00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:02.221646) v4 > -2790> 2018-07-11 07:52:02.222590 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.8 10.8.128.108:6802/11967 250 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:02.221646) v4 ==== 2004+0+0 (2683204152 0 0) 0x55d9b60dfa00 con 0x55d986bbe000 > -2789> 2018-07-11 07:52:02.222588 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.7 10.8.128.114:6806/15885 250 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:02.221646) v4 ==== 2004+0+0 (2683204152 0 0) 0x55d982cdf800 con 0x55d986bac000 > -2788> 2018-07-11 07:52:02.222628 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6803/15919 conn(0x55d986b5c800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.11 seq 250 0x55d9b60dfa00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:02.221646) v4 > -2787> 2018-07-11 07:52:02.222633 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.108:6803/11967 conn(0x55d986bbf800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.8 seq 250 0x55d9bdba4800 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:02.221646) v4 > -2786> 2018-07-11 07:52:02.222644 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.11 10.8.128.114:6803/15919 250 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:02.221646) v4 ==== 2004+0+0 (2683204152 0 0) 0x55d9b60dfa00 con 0x55d986b5c800 > -2785> 2018-07-11 07:52:02.222646 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.8 10.8.128.108:6803/11967 250 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:02.221646) v4 ==== 2004+0+0 (2683204152 0 0) 0x55d9bdba4800 con 0x55d986bbf800 > -2784> 2018-07-11 07:52:02.222669 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6802/15919 conn(0x55d986b5b000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.11 seq 250 0x55d982cdf800 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:02.221646) v4 > -2783> 2018-07-11 07:52:02.222688 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.11 10.8.128.114:6802/15919 250 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:02.221646) v4 ==== 2004+0+0 (2683204152 0 0) 0x55d982cdf800 con 0x55d986b5b000 > -2782> 2018-07-11 07:52:02.227348 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1011 0x55d9c48c1440 rep_scrubmap(5.2e e501 from shard 4) v1 > -2781> 2018-07-11 07:52:02.227367 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1011 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12474 (3686815630 0 1254539758) 0x55d9c48c1440 con 0x55d986bec000 > -2780> 2018-07-11 07:52:02.255337 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -2779> 2018-07-11 07:52:02.255362 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -2778> 2018-07-11 07:52:02.255410 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -2777> 2018-07-11 07:52:02.255868 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -2776> 2018-07-11 07:52:02.302225 7ffab7be8700 10 monclient: tick > -2775> 2018-07-11 07:52:02.302239 7ffab7be8700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2018-07-11 07:51:32.302237) > -2774> 2018-07-11 07:52:02.304021 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:756e4868:::benchmark_data_magna115_10867_object1847287:0,end:5:756e9966:::benchmark_data_magna115_10867_object253744:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498500 con 0 > -2773> 2018-07-11 07:52:02.304063 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:756e4868:::benchmark_data_magna115_10867_object1847287:0,end:5:756e9966:::benchmark_data_magna115_10867_object253744:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498c80 con 0 > -2772> 2018-07-11 07:52:02.352065 7ffab6be6700 1 -- 10.8.128.107:6800/13927 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55d9bd9cdb80 con 0 > -2771> 2018-07-11 07:52:02.352122 7ffab6be6700 1 -- 10.8.128.107:6800/13927 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55d9ba81ab00 con 0 > -2770> 2018-07-11 07:52:02.370859 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15255 conn(0x55d986c82800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.0 seq 260 0x55d982cdf800 osd_ping(ping e501 stamp 2018-07-11 07:52:03.602310) v4 > -2769> 2018-07-11 07:52:02.370863 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15255 conn(0x55d986d20000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.0 seq 260 0x55d9b60df400 osd_ping(ping e501 stamp 2018-07-11 07:52:03.602310) v4 > -2768> 2018-07-11 07:52:02.370879 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.0 10.8.128.113:0/15255 260 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:03.602310) v4 ==== 2004+0+0 (4162034525 0 0) 0x55d982cdf800 con 0x55d986c82800 > -2767> 2018-07-11 07:52:02.370883 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.0 10.8.128.113:0/15255 260 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:03.602310) v4 ==== 2004+0+0 (4162034525 0 0) 0x55d9b60df400 con 0x55d986d20000 > -2766> 2018-07-11 07:52:02.370889 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:03.602310) v4 -- 0x55d9c34cc000 con 0 > -2765> 2018-07-11 07:52:02.370910 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:03.602310) v4 -- 0x55d982cdfc00 con 0 > -2764> 2018-07-11 07:52:02.534912 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1018 0x55d9c48c1680 rep_scrubmap(5.2e e501 from shard 2) v1 > -2763> 2018-07-11 07:52:02.534932 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1018 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12468 (1269249638 0 1573253855) 0x55d9c48c1680 con 0x55d986bf0800 > -2762> 2018-07-11 07:52:02.538767 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1012 0x55d9c48c18c0 rep_scrubmap(5.2e e501 from shard 4) v1 > -2761> 2018-07-11 07:52:02.538788 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1012 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12468 (3686815630 0 1573253855) 0x55d9c48c18c0 con 0x55d986bec000 > -2760> 2018-07-11 07:52:02.567374 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12677 conn(0x55d986db4800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.1 seq 262 0x55d9869d9800 osd_ping(ping e501 stamp 2018-07-11 07:52:02.615364) v4 > -2759> 2018-07-11 07:52:02.567390 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12677 conn(0x55d986df2000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.1 seq 262 0x55d9bdba4800 osd_ping(ping e501 stamp 2018-07-11 07:52:02.615364) v4 > -2758> 2018-07-11 07:52:02.567399 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.1 10.8.128.106:0/12677 262 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:02.615364) v4 ==== 2004+0+0 (1564155102 0 0) 0x55d9869d9800 con 0x55d986db4800 > -2757> 2018-07-11 07:52:02.567414 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.1 10.8.128.106:0/12677 262 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:02.615364) v4 ==== 2004+0+0 (1564155102 0 0) 0x55d9bdba4800 con 0x55d986df2000 > -2756> 2018-07-11 07:52:02.567418 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:02.615364) v4 -- 0x55d9c34cc000 con 0 > -2755> 2018-07-11 07:52:02.567442 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:02.615364) v4 -- 0x55d982cdfc00 con 0 > -2754> 2018-07-11 07:52:02.619955 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12650 conn(0x55d986d23000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 252 0x55d982cdf800 osd_ping(ping e501 stamp 2018-07-11 07:52:02.667869) v4 > -2753> 2018-07-11 07:52:02.619989 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.3 10.8.128.106:0/12650 252 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:02.667869) v4 ==== 2004+0+0 (3167810169 0 0) 0x55d982cdf800 con 0x55d986d23000 > -2752> 2018-07-11 07:52:02.620004 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:02.667869) v4 -- 0x55d9c34cc000 con 0 > -2751> 2018-07-11 07:52:02.620046 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12650 conn(0x55d986d21800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 252 0x55d9b60dfa00 osd_ping(ping e501 stamp 2018-07-11 07:52:02.667869) v4 > -2750> 2018-07-11 07:52:02.620068 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.3 10.8.128.106:0/12650 252 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:02.667869) v4 ==== 2004+0+0 (3167810169 0 0) 0x55d9b60dfa00 con 0x55d986d21800 > -2749> 2018-07-11 07:52:02.620079 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:02.667869) v4 -- 0x55d986d33600 con 0 > -2748> 2018-07-11 07:52:02.709420 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -2747> 2018-07-11 07:52:02.709445 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -2746> 2018-07-11 07:52:02.709501 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -2745> 2018-07-11 07:52:02.709946 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -2744> 2018-07-11 07:52:02.753926 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67149,epoch:501/498,start:5:756e9966:::benchmark_data_magna115_10867_object253744:0,end:5:756f0e0e:::benchmark_data_magna115_10867_object2848144:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1499180 con 0 > -2743> 2018-07-11 07:52:02.753979 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67149,epoch:501/498,start:5:756e9966:::benchmark_data_magna115_10867_object253744:0,end:5:756f0e0e:::benchmark_data_magna115_10867_object2848144:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1499b80 con 0 > -2742> 2018-07-11 07:52:02.892787 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.108:0/11967 conn(0x55d986db0000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.8 seq 242 0x55d9869d9800 osd_ping(ping e501 stamp 2018-07-11 07:52:02.648562) v4 > -2741> 2018-07-11 07:52:02.892809 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.8 10.8.128.108:0/11967 242 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:02.648562) v4 ==== 2004+0+0 (3611064777 0 0) 0x55d9869d9800 con 0x55d986db0000 > -2740> 2018-07-11 07:52:02.892818 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:02.648562) v4 -- 0x55d9c34cc000 con 0 > -2739> 2018-07-11 07:52:02.892809 7ffacbbd1700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.108:0/11967 conn(0x55d986d82800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.8 seq 242 0x55d9bcd78600 osd_ping(ping e501 stamp 2018-07-11 07:52:02.648562) v4 > -2738> 2018-07-11 07:52:02.892832 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 <== osd.8 10.8.128.108:0/11967 242 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:02.648562) v4 ==== 2004+0+0 (3611064777 0 0) 0x55d9bcd78600 con 0x55d986d82800 > -2737> 2018-07-11 07:52:02.892843 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:02.648562) v4 -- 0x55d986d33600 con 0 > -2736> 2018-07-11 07:52:02.941655 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1013 0x55d9c48c1b00 rep_scrubmap(5.2e e501 from shard 4) v1 > -2735> 2018-07-11 07:52:02.941676 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1013 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12482 (3686815630 0 1979932289) 0x55d9c48c1b00 con 0x55d986bec000 > -2734> 2018-07-11 07:52:02.950269 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1019 0x55d9c48c1d40 rep_scrubmap(5.2e e501 from shard 2) v1 > -2733> 2018-07-11 07:52:02.950290 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1019 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12482 (1269249638 0 1979932289) 0x55d9c48c1d40 con 0x55d986bf0800 > -2732> 2018-07-11 07:52:02.951349 7ffacbbd1700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15263 conn(0x55d985d44000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.5 seq 243 0x55d9b60dfa00 osd_ping(ping e501 stamp 2018-07-11 07:52:04.182802) v4 > -2731> 2018-07-11 07:52:02.951349 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15263 conn(0x55d985d45800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.5 seq 243 0x55d982cdf800 osd_ping(ping e501 stamp 2018-07-11 07:52:04.182802) v4 > -2730> 2018-07-11 07:52:02.951373 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 <== osd.5 10.8.128.113:0/15263 243 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:04.182802) v4 ==== 2004+0+0 (1122433266 0 0) 0x55d9b60dfa00 con 0x55d985d44000 > -2729> 2018-07-11 07:52:02.951377 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.5 10.8.128.113:0/15263 243 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:04.182802) v4 ==== 2004+0+0 (1122433266 0 0) 0x55d982cdf800 con 0x55d985d45800 > -2728> 2018-07-11 07:52:02.951384 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:04.182802) v4 -- 0x55d986d33600 con 0 > -2727> 2018-07-11 07:52:02.951408 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:04.182802) v4 -- 0x55d9c34cc000 con 0 > -2726> 2018-07-11 07:52:03.060800 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -2725> 2018-07-11 07:52:03.060826 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -2724> 2018-07-11 07:52:03.060887 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -2723> 2018-07-11 07:52:03.061382 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -2722> 2018-07-11 07:52:03.063677 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:756f0e0e:::benchmark_data_magna115_10867_object2848144:0,end:5:756f791e:::benchmark_data_magna115_10867_object3167336:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498f00 con 0 > -2721> 2018-07-11 07:52:03.063725 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:756f0e0e:::benchmark_data_magna115_10867_object2848144:0,end:5:756f791e:::benchmark_data_magna115_10867_object3167336:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b14f0a00 con 0 > -2720> 2018-07-11 07:52:03.292617 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1014 0x55d9b0e0c000 rep_scrubmap(5.2e e501 from shard 4) v1 > -2719> 2018-07-11 07:52:03.292637 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1014 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12466 (3686815630 0 1998904543) 0x55d9b0e0c000 con 0x55d986bec000 > -2718> 2018-07-11 07:52:03.295751 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1020 0x55d9b0e0c240 rep_scrubmap(5.2e e501 from shard 2) v1 > -2717> 2018-07-11 07:52:03.295770 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1020 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12466 (1269249638 0 1998904543) 0x55d9b0e0c240 con 0x55d986bf0800 > -2716> 2018-07-11 07:52:03.322068 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.114:0/15885 conn(0x55d986d7f800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.7 seq 266 0x55d9869d9800 osd_ping(ping e501 stamp 2018-07-11 07:52:03.952462) v4 > -2715> 2018-07-11 07:52:03.322091 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.7 10.8.128.114:0/15885 266 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:03.952462) v4 ==== 2004+0+0 (3577465778 0 0) 0x55d9869d9800 con 0x55d986d7f800 > -2714> 2018-07-11 07:52:03.322101 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:03.952462) v4 -- 0x55d9c34cc000 con 0 > -2713> 2018-07-11 07:52:03.322103 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.114:0/15885 conn(0x55d986d81000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.7 seq 266 0x55d9b60df400 osd_ping(ping e501 stamp 2018-07-11 07:52:03.952462) v4 > -2712> 2018-07-11 07:52:03.322121 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.7 10.8.128.114:0/15885 266 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:03.952462) v4 ==== 2004+0+0 (3577465778 0 0) 0x55d9b60df400 con 0x55d986d81000 > -2711> 2018-07-11 07:52:03.322130 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:03.952462) v4 -- 0x55d982cdfc00 con 0 > -2710> 2018-07-11 07:52:03.407331 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -2709> 2018-07-11 07:52:03.407358 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -2708> 2018-07-11 07:52:03.407407 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -2707> 2018-07-11 07:52:03.407870 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -2706> 2018-07-11 07:52:03.429311 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67998,epoch:501/498,start:5:756f791e:::benchmark_data_magna115_10867_object3167336:0,end:5:756fcc31:::benchmark_data_magna115_10867_object3446835:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1499400 con 0 > -2705> 2018-07-11 07:52:03.429351 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67998,epoch:501/498,start:5:756f791e:::benchmark_data_magna115_10867_object3167336:0,end:5:756fcc31:::benchmark_data_magna115_10867_object3446835:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98696ef00 con 0 > -2704> 2018-07-11 07:52:03.643991 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1021 0x55d9b0e0c480 rep_scrubmap(5.2e e501 from shard 2) v1 > -2703> 2018-07-11 07:52:03.644012 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1021 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12464 (1269249638 0 85191151) 0x55d9b0e0c480 con 0x55d986bf0800 > -2702> 2018-07-11 07:52:03.658497 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1015 0x55d9b0e0c6c0 rep_scrubmap(5.2e e501 from shard 4) v1 > -2701> 2018-07-11 07:52:03.658517 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1015 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12464 (3686815630 0 85191151) 0x55d9b0e0c6c0 con 0x55d986bec000 > -2700> 2018-07-11 07:52:03.768805 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -2699> 2018-07-11 07:52:03.768830 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -2698> 2018-07-11 07:52:03.768888 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -2697> 2018-07-11 07:52:03.769379 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -2696> 2018-07-11 07:52:03.770295 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:756fcc31:::benchmark_data_magna115_10867_object3446835:0,end:5:75704980:::benchmark_data_magna115_10867_object2498268:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98696ec80 con 0 > -2695> 2018-07-11 07:52:03.770330 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:756fcc31:::benchmark_data_magna115_10867_object3446835:0,end:5:75704980:::benchmark_data_magna115_10867_object2498268:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98696f680 con 0 > -2694> 2018-07-11 07:52:03.843422 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 3022 0x55d9bcd78600 MOSDScrubReserve(11.3s5 REQUEST e501) v1 > -2693> 2018-07-11 07:52:03.843441 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 3022 ==== MOSDScrubReserve(11.3s5 REQUEST e501) v1 ==== 43+0+0 (3260170403 0 0) 0x55d9bcd78600 con 0x55d986c7e000 > -2692> 2018-07-11 07:52:03.843516 7ffab3be0700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e501) v1 -- 0x55d9bdba5e00 con 0 > -2691> 2018-07-11 07:52:03.843996 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 3023 0x55d9bdba5e00 MOSDScrubReserve(11.3s5 RELEASE e501) v1 > -2690> 2018-07-11 07:52:03.844013 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 3023 ==== MOSDScrubReserve(11.3s5 RELEASE e501) v1 ==== 43+0+0 (223328929 0 0) 0x55d9bdba5e00 con 0x55d986c7e000 > -2689> 2018-07-11 07:52:03.978521 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1022 0x55d9b0e0c900 rep_scrubmap(5.2e e501 from shard 2) v1 > -2688> 2018-07-11 07:52:03.978541 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1022 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12472 (1269249638 0 177257655) 0x55d9b0e0c900 con 0x55d986bf0800 > -2687> 2018-07-11 07:52:04.035499 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1016 0x55d9b0e0cb40 rep_scrubmap(5.2e e501 from shard 4) v1 > -2686> 2018-07-11 07:52:04.035521 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1016 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12472 (3686815630 0 177257655) 0x55d9b0e0cb40 con 0x55d986bec000 > -2685> 2018-07-11 07:52:04.170174 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -2684> 2018-07-11 07:52:04.170198 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -2683> 2018-07-11 07:52:04.170263 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -2682> 2018-07-11 07:52:04.170730 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -2681> 2018-07-11 07:52:04.229707 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75704980:::benchmark_data_magna115_10867_object2498268:0,end:5:75707897:::benchmark_data_magna115_10867_object3579696:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98696f900 con 0 > -2680> 2018-07-11 07:52:04.229756 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75704980:::benchmark_data_magna115_10867_object2498268:0,end:5:75707897:::benchmark_data_magna115_10867_object3579696:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498a00 con 0 > -2679> 2018-07-11 07:52:04.439449 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1023 0x55d9b0e0cd80 rep_scrubmap(5.2e e501 from shard 2) v1 > -2678> 2018-07-11 07:52:04.439471 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1023 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12472 (1269249638 0 3218136532) 0x55d9b0e0cd80 con 0x55d986bf0800 > -2677> 2018-07-11 07:52:04.439611 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1017 0x55d9b0e0cfc0 rep_scrubmap(5.2e e501 from shard 4) v1 > -2676> 2018-07-11 07:52:04.439625 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1017 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12472 (3686815630 0 3218136532) 0x55d9b0e0cfc0 con 0x55d986bec000 > -2675> 2018-07-11 07:52:04.608679 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -2674> 2018-07-11 07:52:04.608706 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -2673> 2018-07-11 07:52:04.608767 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -2672> 2018-07-11 07:52:04.609229 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -2671> 2018-07-11 07:52:04.654516 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67334,epoch:501/498,start:5:75707897:::benchmark_data_magna115_10867_object3579696:0,end:5:7570d890:::benchmark_data_magna115_10867_object119252:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98696f180 con 0 > -2670> 2018-07-11 07:52:04.654568 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67334,epoch:501/498,start:5:75707897:::benchmark_data_magna115_10867_object3579696:0,end:5:7570d890:::benchmark_data_magna115_10867_object119252:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98696e000 con 0 > -2669> 2018-07-11 07:52:04.671399 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15255 conn(0x55d986d20000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.0 seq 261 0x55d9bdba4800 osd_ping(ping e501 stamp 2018-07-11 07:52:05.902817) v4 > -2668> 2018-07-11 07:52:04.671403 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15255 conn(0x55d986c82800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.0 seq 261 0x55d982cdf800 osd_ping(ping e501 stamp 2018-07-11 07:52:05.902817) v4 > -2667> 2018-07-11 07:52:04.671438 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.0 10.8.128.113:0/15255 261 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:05.902817) v4 ==== 2004+0+0 (2985050469 0 0) 0x55d9bdba4800 con 0x55d986d20000 > -2666> 2018-07-11 07:52:04.671443 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.0 10.8.128.113:0/15255 261 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:05.902817) v4 ==== 2004+0+0 (2985050469 0 0) 0x55d982cdf800 con 0x55d986c82800 > -2665> 2018-07-11 07:52:04.671451 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:05.902817) v4 -- 0x55d982cdfc00 con 0 > -2664> 2018-07-11 07:52:04.671475 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:05.902817) v4 -- 0x55d9c34cc000 con 0 > -2663> 2018-07-11 07:52:04.843686 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 3024 0x55d9b60dfa00 MOSDScrubReserve(11.3s5 REQUEST e501) v1 > -2662> 2018-07-11 07:52:04.843712 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 3024 ==== MOSDScrubReserve(11.3s5 REQUEST e501) v1 ==== 43+0+0 (3260170403 0 0) 0x55d9b60dfa00 con 0x55d986c7e000 > -2661> 2018-07-11 07:52:04.843796 7ffab3be0700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e501) v1 -- 0x55d9bdba5e00 con 0 > -2660> 2018-07-11 07:52:04.844150 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 3025 0x55d986d97200 MOSDScrubReserve(11.3s5 RELEASE e501) v1 > -2659> 2018-07-11 07:52:04.844164 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 3025 ==== MOSDScrubReserve(11.3s5 RELEASE e501) v1 ==== 43+0+0 (223328929 0 0) 0x55d986d97200 con 0x55d986c7e000 > -2658> 2018-07-11 07:52:04.865917 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1024 0x55d9b0e0d200 rep_scrubmap(5.2e e501 from shard 2) v1 > -2657> 2018-07-11 07:52:04.865938 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1024 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12464 (1269249638 0 1122513514) 0x55d9b0e0d200 con 0x55d986bf0800 > -2656> 2018-07-11 07:52:04.910195 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1018 0x55d9b0e0d440 rep_scrubmap(5.2e e501 from shard 4) v1 > -2655> 2018-07-11 07:52:04.910216 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1018 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12464 (3686815630 0 1122513514) 0x55d9b0e0d440 con 0x55d986bec000 > -2654> 2018-07-11 07:52:05.467834 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12677 conn(0x55d986df2000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.1 seq 263 0x55d9b60df400 osd_ping(ping e501 stamp 2018-07-11 07:52:05.515833) v4 > -2653> 2018-07-11 07:52:05.467848 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12677 conn(0x55d986db4800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.1 seq 263 0x55d9869d9800 osd_ping(ping e501 stamp 2018-07-11 07:52:05.515833) v4 > -2652> 2018-07-11 07:52:05.467859 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.1 10.8.128.106:0/12677 263 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:05.515833) v4 ==== 2004+0+0 (3765399491 0 0) 0x55d9b60df400 con 0x55d986df2000 > -2651> 2018-07-11 07:52:05.467869 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:05.515833) v4 -- 0x55d982cdfc00 con 0 > -2650> 2018-07-11 07:52:05.467868 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.1 10.8.128.106:0/12677 263 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:05.515833) v4 ==== 2004+0+0 (3765399491 0 0) 0x55d9869d9800 con 0x55d986db4800 > -2649> 2018-07-11 07:52:05.467897 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:05.515833) v4 -- 0x55d9c34cc000 con 0 > -2648> 2018-07-11 07:52:05.554206 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.107:0/12617 conn(0x55d98123c000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 251 0x55d9bdba4800 osd_ping(ping e501 stamp 2018-07-11 07:52:05.553883) v4 > -2647> 2018-07-11 07:52:05.554230 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.9 10.8.128.107:0/12617 251 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:05.553883) v4 ==== 2004+0+0 (3611829341 0 0) 0x55d9bdba4800 con 0x55d98123c000 > -2646> 2018-07-11 07:52:05.554240 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:05.553883) v4 -- 0x55d982cdfc00 con 0 > -2645> 2018-07-11 07:52:05.554229 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.107:0/12617 conn(0x55d98123d800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 251 0x55d986d33600 osd_ping(ping e501 stamp 2018-07-11 07:52:05.553883) v4 > -2644> 2018-07-11 07:52:05.554264 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.9 10.8.128.107:0/12617 251 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:05.553883) v4 ==== 2004+0+0 (3611829341 0 0) 0x55d986d33600 con 0x55d98123d800 > -2643> 2018-07-11 07:52:05.554283 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:05.553883) v4 -- 0x55d986d4f800 con 0 > -2642> 2018-07-11 07:52:05.722071 7ffab1bdc700 5 osd.10 501 heartbeat: osd_stat(789 GB used, 142 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,0,0,2]) > -2641> 2018-07-11 07:52:05.722087 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6802/15255 -- osd_ping(ping e501 stamp 2018-07-11 07:52:05.722083) v4 -- 0x55d980f5a200 con 0 > -2640> 2018-07-11 07:52:05.722125 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6803/15255 -- osd_ping(ping e501 stamp 2018-07-11 07:52:05.722083) v4 -- 0x55d986d41600 con 0 > -2639> 2018-07-11 07:52:05.722141 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6806/12677 -- osd_ping(ping e501 stamp 2018-07-11 07:52:05.722083) v4 -- 0x55d982d30400 con 0 > -2638> 2018-07-11 07:52:05.722155 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6807/12677 -- osd_ping(ping e501 stamp 2018-07-11 07:52:05.722083) v4 -- 0x55d9bcd72800 con 0 > -2637> 2018-07-11 07:52:05.722167 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6808/12651 -- osd_ping(ping e501 stamp 2018-07-11 07:52:05.722083) v4 -- 0x55d986d4e600 con 0 > -2636> 2018-07-11 07:52:05.722183 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6810/12651 -- osd_ping(ping e501 stamp 2018-07-11 07:52:05.722083) v4 -- 0x55d9b40a4000 con 0 > -2635> 2018-07-11 07:52:05.722202 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6809/12650 -- osd_ping(ping e501 stamp 2018-07-11 07:52:05.722083) v4 -- 0x55d9829e4800 con 0 > -2634> 2018-07-11 07:52:05.722215 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6811/12650 -- osd_ping(ping e501 stamp 2018-07-11 07:52:05.722083) v4 -- 0x55d9bf804400 con 0 > -2633> 2018-07-11 07:52:05.722229 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6810/15287 -- osd_ping(ping e501 stamp 2018-07-11 07:52:05.722083) v4 -- 0x55d980457a00 con 0 > -2632> 2018-07-11 07:52:05.722256 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6811/15287 -- osd_ping(ping e501 stamp 2018-07-11 07:52:05.722083) v4 -- 0x55d9c9123000 con 0 > -2631> 2018-07-11 07:52:05.722271 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6806/15263 -- osd_ping(ping e501 stamp 2018-07-11 07:52:05.722083) v4 -- 0x55d982cdea00 con 0 > -2630> 2018-07-11 07:52:05.722284 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6807/15263 -- osd_ping(ping e501 stamp 2018-07-11 07:52:05.722083) v4 -- 0x55d9c4c1be00 con 0 > -2629> 2018-07-11 07:52:05.722297 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.115:6802/13323 -- osd_ping(ping e501 stamp 2018-07-11 07:52:05.722083) v4 -- 0x55d986b6d400 con 0 > -2628> 2018-07-11 07:52:05.722311 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.115:6803/13323 -- osd_ping(ping e501 stamp 2018-07-11 07:52:05.722083) v4 -- 0x55d9ba819a00 con 0 > -2627> 2018-07-11 07:52:05.722329 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6806/15885 -- osd_ping(ping e501 stamp 2018-07-11 07:52:05.722083) v4 -- 0x55d9829e4600 con 0 > -2626> 2018-07-11 07:52:05.722342 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6807/15885 -- osd_ping(ping e501 stamp 2018-07-11 07:52:05.722083) v4 -- 0x55d9b437a200 con 0 > -2625> 2018-07-11 07:52:05.722354 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.108:6802/11967 -- osd_ping(ping e501 stamp 2018-07-11 07:52:05.722083) v4 -- 0x55d9b9ac2a00 con 0 > -2624> 2018-07-11 07:52:05.722366 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.108:6803/11967 -- osd_ping(ping e501 stamp 2018-07-11 07:52:05.722083) v4 -- 0x55d986d96c00 con 0 > -2623> 2018-07-11 07:52:05.722378 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.107:6806/12617 -- osd_ping(ping e501 stamp 2018-07-11 07:52:05.722083) v4 -- 0x55d9b4389a00 con 0 > -2622> 2018-07-11 07:52:05.722391 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.107:6807/12617 -- osd_ping(ping e501 stamp 2018-07-11 07:52:05.722083) v4 -- 0x55d9ba818800 con 0 > -2621> 2018-07-11 07:52:05.722403 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6802/15919 -- osd_ping(ping e501 stamp 2018-07-11 07:52:05.722083) v4 -- 0x55d9c908ec00 con 0 > -2620> 2018-07-11 07:52:05.722437 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6803/15919 -- osd_ping(ping e501 stamp 2018-07-11 07:52:05.722083) v4 -- 0x55d9b9ac3c00 con 0 > -2619> 2018-07-11 07:52:05.722587 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.107:6806/12617 conn(0x55d986bd2000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.9 seq 251 0x55d9869d9800 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:05.722083) v4 > -2618> 2018-07-11 07:52:05.722620 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.9 10.8.128.107:6806/12617 251 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:05.722083) v4 ==== 2004+0+0 (724951937 0 0) 0x55d9869d9800 con 0x55d986bd2000 > -2617> 2018-07-11 07:52:05.722629 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.107:6807/12617 conn(0x55d986bd3800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.9 seq 251 0x55d9bdba5e00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:05.722083) v4 > -2616> 2018-07-11 07:52:05.722652 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.9 10.8.128.107:6807/12617 251 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:05.722083) v4 ==== 2004+0+0 (724951937 0 0) 0x55d9bdba5e00 con 0x55d986bd3800 > -2615> 2018-07-11 07:52:05.722639 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6808/12651 conn(0x55d986b9a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.2 seq 251 0x55d9bdba4800 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:05.722083) v4 > -2614> 2018-07-11 07:52:05.722664 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6810/12651 conn(0x55d986b9b800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.2 seq 251 0x55d9869d9800 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:05.722083) v4 > -2613> 2018-07-11 07:52:05.722667 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.2 10.8.128.106:6808/12651 251 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:05.722083) v4 ==== 2004+0+0 (724951937 0 0) 0x55d9bdba4800 con 0x55d986b9a000 > -2612> 2018-07-11 07:52:05.722680 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.2 10.8.128.106:6810/12651 251 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:05.722083) v4 ==== 2004+0+0 (724951937 0 0) 0x55d9869d9800 con 0x55d986b9b800 > -2611> 2018-07-11 07:52:05.722710 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6806/12677 conn(0x55d986b58000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.1 seq 251 0x55d9bdba5e00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:05.722083) v4 > -2610> 2018-07-11 07:52:05.722725 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.1 10.8.128.106:6806/12677 251 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:05.722083) v4 ==== 2004+0+0 (724951937 0 0) 0x55d9bdba5e00 con 0x55d986b58000 > -2609> 2018-07-11 07:52:05.722735 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6807/12677 conn(0x55d986b59800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.1 seq 251 0x55d9bdba4800 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:05.722083) v4 > -2608> 2018-07-11 07:52:05.722736 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6803/15255 conn(0x55d986bc9800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.0 seq 251 0x55d9869d9800 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:05.722083) v4 > -2607> 2018-07-11 07:52:05.722751 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.1 10.8.128.106:6807/12677 251 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:05.722083) v4 ==== 2004+0+0 (724951937 0 0) 0x55d9bdba4800 con 0x55d986b59800 > -2606> 2018-07-11 07:52:05.722767 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.0 10.8.128.113:6803/15255 251 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:05.722083) v4 ==== 2004+0+0 (724951937 0 0) 0x55d9869d9800 con 0x55d986bc9800 > -2605> 2018-07-11 07:52:05.722772 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6809/12650 conn(0x55d986bcb000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 251 0x55d9bdba5e00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:05.722083) v4 > -2604> 2018-07-11 07:52:05.722790 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.3 10.8.128.106:6809/12650 251 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:05.722083) v4 ==== 2004+0+0 (724951937 0 0) 0x55d9bdba5e00 con 0x55d986bcb000 > -2603> 2018-07-11 07:52:05.722797 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6802/15255 conn(0x55d986bc8000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.0 seq 251 0x55d9bdba4800 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:05.722083) v4 > -2602> 2018-07-11 07:52:05.722805 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6811/15287 conn(0x55d986bb0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 251 0x55d9869d9800 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:05.722083) v4 > -2601> 2018-07-11 07:52:05.722810 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.0 10.8.128.113:6802/15255 251 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:05.722083) v4 ==== 2004+0+0 (724951937 0 0) 0x55d9bdba4800 con 0x55d986bc8000 > -2600> 2018-07-11 07:52:05.722821 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.4 10.8.128.113:6811/15287 251 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:05.722083) v4 ==== 2004+0+0 (724951937 0 0) 0x55d9869d9800 con 0x55d986bb0800 > -2599> 2018-07-11 07:52:05.722838 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6806/15263 conn(0x55d986b9d000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.5 seq 251 0x55d9bdba5e00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:05.722083) v4 > -2598> 2018-07-11 07:52:05.722849 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6811/12650 conn(0x55d986bcc800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 251 0x55d9bdba4800 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:05.722083) v4 > -2597> 2018-07-11 07:52:05.722861 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.3 10.8.128.106:6811/12650 251 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:05.722083) v4 ==== 2004+0+0 (724951937 0 0) 0x55d9bdba4800 con 0x55d986bcc800 > -2596> 2018-07-11 07:52:05.722864 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.115:6802/13323 conn(0x55d986bc1000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 251 0x55d9869d9800 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:05.722083) v4 > -2595> 2018-07-11 07:52:05.722878 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.6 10.8.128.115:6802/13323 251 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:05.722083) v4 ==== 2004+0+0 (724951937 0 0) 0x55d9869d9800 con 0x55d986bc1000 > -2594> 2018-07-11 07:52:05.722852 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.5 10.8.128.113:6806/15263 251 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:05.722083) v4 ==== 2004+0+0 (724951937 0 0) 0x55d9bdba5e00 con 0x55d986b9d000 > -2593> 2018-07-11 07:52:05.722895 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6810/15287 conn(0x55d986baf000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 251 0x55d9bdba4800 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:05.722083) v4 > -2592> 2018-07-11 07:52:05.722907 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.4 10.8.128.113:6810/15287 251 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:05.722083) v4 ==== 2004+0+0 (724951937 0 0) 0x55d9bdba4800 con 0x55d986baf000 > -2591> 2018-07-11 07:52:05.722911 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6806/15885 conn(0x55d986bac000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.7 seq 251 0x55d9869d9800 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:05.722083) v4 > -2590> 2018-07-11 07:52:05.722923 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.7 10.8.128.114:6806/15885 251 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:05.722083) v4 ==== 2004+0+0 (724951937 0 0) 0x55d9869d9800 con 0x55d986bac000 > -2589> 2018-07-11 07:52:05.722935 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6807/15263 conn(0x55d986b9e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.5 seq 251 0x55d9bdba4800 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:05.722083) v4 > -2588> 2018-07-11 07:52:05.722939 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6807/15885 conn(0x55d986bad800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.7 seq 251 0x55d9bdba5e00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:05.722083) v4 > -2587> 2018-07-11 07:52:05.722947 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.5 10.8.128.113:6807/15263 251 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:05.722083) v4 ==== 2004+0+0 (724951937 0 0) 0x55d9bdba4800 con 0x55d986b9e800 > -2586> 2018-07-11 07:52:05.722954 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6802/15919 conn(0x55d986b5b000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.11 seq 251 0x55d9869d9800 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:05.722083) v4 > -2585> 2018-07-11 07:52:05.722958 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.7 10.8.128.114:6807/15885 251 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:05.722083) v4 ==== 2004+0+0 (724951937 0 0) 0x55d9bdba5e00 con 0x55d986bad800 > -2584> 2018-07-11 07:52:05.722964 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.11 10.8.128.114:6802/15919 251 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:05.722083) v4 ==== 2004+0+0 (724951937 0 0) 0x55d9869d9800 con 0x55d986b5b000 > -2583> 2018-07-11 07:52:05.722987 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.108:6803/11967 conn(0x55d986bbf800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.8 seq 251 0x55d9bdba4800 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:05.722083) v4 > -2582> 2018-07-11 07:52:05.723000 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.8 10.8.128.108:6803/11967 251 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:05.722083) v4 ==== 2004+0+0 (724951937 0 0) 0x55d9bdba4800 con 0x55d986bbf800 > -2581> 2018-07-11 07:52:05.723009 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.115:6803/13323 conn(0x55d986bc2800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 251 0x55d9bdba5e00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:05.722083) v4 > -2580> 2018-07-11 07:52:05.723025 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.6 10.8.128.115:6803/13323 251 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:05.722083) v4 ==== 2004+0+0 (724951937 0 0) 0x55d9bdba5e00 con 0x55d986bc2800 > -2579> 2018-07-11 07:52:05.723061 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.108:6802/11967 conn(0x55d986bbe000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.8 seq 251 0x55d9bdba5e00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:05.722083) v4 > -2578> 2018-07-11 07:52:05.723074 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.8 10.8.128.108:6802/11967 251 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:05.722083) v4 ==== 2004+0+0 (724951937 0 0) 0x55d9bdba5e00 con 0x55d986bbe000 > -2577> 2018-07-11 07:52:05.723113 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6803/15919 conn(0x55d986b5c800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.11 seq 251 0x55d9bdba5e00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:05.722083) v4 > -2576> 2018-07-11 07:52:05.723127 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.11 10.8.128.114:6803/15919 251 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:05.722083) v4 ==== 2004+0+0 (724951937 0 0) 0x55d9bdba5e00 con 0x55d986b5c800 > -2575> 2018-07-11 07:52:05.797827 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -2574> 2018-07-11 07:52:05.797856 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -2573> 2018-07-11 07:52:05.797901 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -2572> 2018-07-11 07:52:05.798384 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -2571> 2018-07-11 07:52:05.827370 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12651 conn(0x55d986db3000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.2 seq 259 0x55d9bdba5e00 osd_ping(ping e501 stamp 2018-07-11 07:52:05.875335) v4 > -2570> 2018-07-11 07:52:05.827379 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12651 conn(0x55d986db1800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.2 seq 259 0x55d9bdba4800 osd_ping(ping e501 stamp 2018-07-11 07:52:05.875335) v4 > -2569> 2018-07-11 07:52:05.827390 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.2 10.8.128.106:0/12651 259 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:05.875335) v4 ==== 2004+0+0 (3172738972 0 0) 0x55d9bdba5e00 con 0x55d986db3000 > -2568> 2018-07-11 07:52:05.827397 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.2 10.8.128.106:0/12651 259 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:05.875335) v4 ==== 2004+0+0 (3172738972 0 0) 0x55d9bdba4800 con 0x55d986db1800 > -2567> 2018-07-11 07:52:05.827402 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:05.875335) v4 -- 0x55d9b9ac3c00 con 0 > -2566> 2018-07-11 07:52:05.827442 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:05.875335) v4 -- 0x55d986d96c00 con 0 > -2565> 2018-07-11 07:52:05.855325 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7570d890:::benchmark_data_magna115_10867_object119252:0,end:5:7571340f:::benchmark_data_magna115_11707_object279972:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498780 con 0 > -2564> 2018-07-11 07:52:05.855376 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7570d890:::benchmark_data_magna115_10867_object119252:0,end:5:7571340f:::benchmark_data_magna115_11707_object279972:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455680 con 0 > -2563> 2018-07-11 07:52:05.857258 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 33] Generated table #36809: 2638242 keys, 74546979 bytes > -2562> 2018-07-11 07:52:05.857299 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295525857273, "cf_name": "default", "job": 33, "event": "table_file_creation", "file_number": 36809, "file_size": 74546979, "table_properties": {"data_size": 67112152, "index_size": 690651, "filter_size": 6743186, "raw_key_size": 76215888, "raw_average_key_size": 28, "raw_value_size": 31365774, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638242, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} > -2561> 2018-07-11 07:52:05.964982 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6805/15263 conn(0x55d986c41000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=31 cs=1 l=0). rx osd.5 seq 2838 0x55d9bdba4800 MOSDScrubReserve(6.15 REQUEST e501) v1 > -2560> 2018-07-11 07:52:05.965003 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.5 10.8.128.113:6805/15263 2838 ==== MOSDScrubReserve(6.15 REQUEST e501) v1 ==== 43+0+0 (3942742026 0 0) 0x55d9bdba4800 con 0x55d986c41000 > -2559> 2018-07-11 07:52:06.063206 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1025 0x55d9b0e0d680 rep_scrubmap(5.2e e501 from shard 2) v1 > -2558> 2018-07-11 07:52:06.063228 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1025 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12468 (1269249638 0 1037618635) 0x55d9b0e0d680 con 0x55d986bf0800 > -2557> 2018-07-11 07:52:06.088986 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1019 0x55d9b0e0d8c0 rep_scrubmap(5.2e e501 from shard 4) v1 > -2556> 2018-07-11 07:52:06.089006 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1019 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12468 (3686815630 0 1037618635) 0x55d9b0e0d8c0 con 0x55d986bec000 > -2555> 2018-07-11 07:52:06.120397 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12650 conn(0x55d986d23000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 253 0x55d9869d9800 osd_ping(ping e501 stamp 2018-07-11 07:52:06.168410) v4 > -2554> 2018-07-11 07:52:06.120420 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.3 10.8.128.106:0/12650 253 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:06.168410) v4 ==== 2004+0+0 (2058592378 0 0) 0x55d9869d9800 con 0x55d986d23000 > -2553> 2018-07-11 07:52:06.120431 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:06.168410) v4 -- 0x55d9c908ec00 con 0 > -2552> 2018-07-11 07:52:06.120474 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12650 conn(0x55d986d21800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 253 0x55d986d33600 osd_ping(ping e501 stamp 2018-07-11 07:52:06.168410) v4 > -2551> 2018-07-11 07:52:06.120496 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.3 10.8.128.106:0/12650 253 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:06.168410) v4 ==== 2004+0+0 (2058592378 0 0) 0x55d986d33600 con 0x55d986d21800 > -2550> 2018-07-11 07:52:06.120508 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:06.168410) v4 -- 0x55d9b9ac3c00 con 0 > -2549> 2018-07-11 07:52:06.197067 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.114:0/15919 conn(0x55d986df3800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.11 seq 263 0x55d9bdba5e00 osd_ping(ping e501 stamp 2018-07-11 07:52:06.827485) v4 > -2548> 2018-07-11 07:52:06.197088 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.11 10.8.128.114:0/15919 263 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:06.827485) v4 ==== 2004+0+0 (2035842638 0 0) 0x55d9bdba5e00 con 0x55d986df3800 > -2547> 2018-07-11 07:52:06.197097 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:06.827485) v4 -- 0x55d9b9ac3c00 con 0 > -2546> 2018-07-11 07:52:06.197123 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.114:0/15919 conn(0x55d986df5000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.11 seq 263 0x55d982cdf800 osd_ping(ping e501 stamp 2018-07-11 07:52:06.827485) v4 > -2545> 2018-07-11 07:52:06.197144 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.11 10.8.128.114:0/15919 263 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:06.827485) v4 ==== 2004+0+0 (2035842638 0 0) 0x55d982cdf800 con 0x55d986df5000 > -2544> 2018-07-11 07:52:06.197156 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:06.827485) v4 -- 0x55d9c908ec00 con 0 > -2543> 2018-07-11 07:52:06.231675 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6805/15263 -- MOSDScrubReserve(6.15 REJECT e501) v1 -- 0x55d986c22e00 con 0 > -2542> 2018-07-11 07:52:06.232002 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -2541> 2018-07-11 07:52:06.232036 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -2540> 2018-07-11 07:52:06.232112 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -2539> 2018-07-11 07:52:06.232179 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6805/15263 conn(0x55d986c41000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=31 cs=1 l=0). rx osd.5 seq 2839 0x55d986c22e00 MOSDScrubReserve(6.15 RELEASE e501) v1 > -2538> 2018-07-11 07:52:06.232198 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.5 10.8.128.113:6805/15263 2839 ==== MOSDScrubReserve(6.15 RELEASE e501) v1 ==== 43+0+0 (605848072 0 0) 0x55d986c22e00 con 0x55d986c41000 > -2537> 2018-07-11 07:52:06.232941 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -2536> 2018-07-11 07:52:06.254813 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'66886,epoch:501/498,start:5:7571340f:::benchmark_data_magna115_11707_object279972:0,end:5:75719faa:::benchmark_data_magna115_10867_object2469716:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455900 con 0 > -2535> 2018-07-11 07:52:06.254866 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'66886,epoch:501/498,start:5:7571340f:::benchmark_data_magna115_11707_object279972:0,end:5:75719faa:::benchmark_data_magna115_10867_object2469716:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b14f0500 con 0 > -2534> 2018-07-11 07:52:06.389003 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15287 conn(0x55d98123f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 274 0x55d9869d9800 osd_ping(ping e501 stamp 2018-07-11 07:52:07.620368) v4 > -2533> 2018-07-11 07:52:06.389028 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.4 10.8.128.113:0/15287 274 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:07.620368) v4 ==== 2004+0+0 (2383630443 0 0) 0x55d9869d9800 con 0x55d98123f000 > -2532> 2018-07-11 07:52:06.389037 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:07.620368) v4 -- 0x55d9c908ec00 con 0 > -2531> 2018-07-11 07:52:06.389030 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15287 conn(0x55d981240800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 274 0x55d986d96c00 osd_ping(ping e501 stamp 2018-07-11 07:52:07.620368) v4 > -2530> 2018-07-11 07:52:06.389047 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.4 10.8.128.113:0/15287 274 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:07.620368) v4 ==== 2004+0+0 (2383630443 0 0) 0x55d986d96c00 con 0x55d981240800 > -2529> 2018-07-11 07:52:06.389064 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:07.620368) v4 -- 0x55d9c4c1be00 con 0 > -2528> 2018-07-11 07:52:06.393227 7ffacbbd1700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.108:0/11967 conn(0x55d986d82800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.8 seq 243 0x55d986d33600 osd_ping(ping e501 stamp 2018-07-11 07:52:06.149012) v4 > -2527> 2018-07-11 07:52:06.393233 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.108:0/11967 conn(0x55d986db0000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.8 seq 243 0x55d982cdf800 osd_ping(ping e501 stamp 2018-07-11 07:52:06.149012) v4 > -2526> 2018-07-11 07:52:06.393260 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 <== osd.8 10.8.128.108:0/11967 243 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:06.149012) v4 ==== 2004+0+0 (880917834 0 0) 0x55d986d33600 con 0x55d986d82800 > -2525> 2018-07-11 07:52:06.393261 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.8 10.8.128.108:0/11967 243 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:06.149012) v4 ==== 2004+0+0 (880917834 0 0) 0x55d982cdf800 con 0x55d986db0000 > -2524> 2018-07-11 07:52:06.393281 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:06.149012) v4 -- 0x55d9b9ac3c00 con 0 > -2523> 2018-07-11 07:52:06.393311 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:06.149012) v4 -- 0x55d9c908ec00 con 0 > -2522> 2018-07-11 07:52:06.405344 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.115:0/13323 conn(0x55d986d7e000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 260 0x55d9bdba5e00 osd_ping(ping e501 stamp 2018-07-11 07:52:06.751314) v4 > -2521> 2018-07-11 07:52:06.405365 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.6 10.8.128.115:0/13323 260 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:06.751314) v4 ==== 2004+0+0 (607114919 0 0) 0x55d9bdba5e00 con 0x55d986d7e000 > -2520> 2018-07-11 07:52:06.405374 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:06.751314) v4 -- 0x55d9b9ac3c00 con 0 > -2519> 2018-07-11 07:52:06.405401 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.115:0/13323 conn(0x55d986d24800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 260 0x55d9b60df400 osd_ping(ping e501 stamp 2018-07-11 07:52:06.751314) v4 > -2518> 2018-07-11 07:52:06.405429 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.6 10.8.128.115:0/13323 260 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:06.751314) v4 ==== 2004+0+0 (607114919 0 0) 0x55d9b60df400 con 0x55d986d24800 > -2517> 2018-07-11 07:52:06.405440 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:06.751314) v4 -- 0x55d9c4c1be00 con 0 > -2516> 2018-07-11 07:52:06.507159 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1026 0x55d9b0e0db00 rep_scrubmap(5.2e e501 from shard 2) v1 > -2515> 2018-07-11 07:52:06.507179 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1026 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12466 (1269249638 0 4045489029) 0x55d9b0e0db00 con 0x55d986bf0800 > -2514> 2018-07-11 07:52:06.510255 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1020 0x55d9b0e0dd40 rep_scrubmap(5.2e e501 from shard 4) v1 > -2513> 2018-07-11 07:52:06.510274 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1020 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12466 (3686815630 0 4045489029) 0x55d9b0e0dd40 con 0x55d986bec000 > -2512> 2018-07-11 07:52:06.568432 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6805/15885 conn(0x55d986c7f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.7 seq 2816 0x55d986d96c00 MOSDScrubReserve(5.4 REQUEST e501) v1 > -2511> 2018-07-11 07:52:06.568450 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.7 10.8.128.114:6805/15885 2816 ==== MOSDScrubReserve(5.4 REQUEST e501) v1 ==== 43+0+0 (1753221512 0 0) 0x55d986d96c00 con 0x55d986c7f800 > -2510> 2018-07-11 07:52:06.568521 7ffab33df700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.114:6805/15885 -- MOSDScrubReserve(5.4 REJECT e501) v1 -- 0x55d9829e5600 con 0 > -2509> 2018-07-11 07:52:06.568855 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6805/15885 conn(0x55d986c7f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.7 seq 2817 0x55d9829e5600 MOSDScrubReserve(5.4 RELEASE e501) v1 > -2508> 2018-07-11 07:52:06.568870 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.7 10.8.128.114:6805/15885 2817 ==== MOSDScrubReserve(5.4 RELEASE e501) v1 ==== 43+0+0 (2812146570 0 0) 0x55d9829e5600 con 0x55d986c7f800 > -2507> 2018-07-11 07:52:06.599140 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -2506> 2018-07-11 07:52:06.599166 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -2505> 2018-07-11 07:52:06.599228 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -2504> 2018-07-11 07:52:06.599692 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -2503> 2018-07-11 07:52:06.646715 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67200,epoch:501/498,start:5:75719faa:::benchmark_data_magna115_10867_object2469716:0,end:5:7572099a:::benchmark_data_magna115_10867_object1537064:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498000 con 0 > -2502> 2018-07-11 07:52:06.646765 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67200,epoch:501/498,start:5:75719faa:::benchmark_data_magna115_10867_object2469716:0,end:5:7572099a:::benchmark_data_magna115_10867_object1537064:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455b80 con 0 > -2501> 2018-07-11 07:52:06.697467 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.114:0/15919 conn(0x55d986df3800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.11 seq 264 0x55d986d33600 osd_ping(ping e501 stamp 2018-07-11 07:52:07.327916) v4 > -2500> 2018-07-11 07:52:06.697488 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.11 10.8.128.114:0/15919 264 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:07.327916) v4 ==== 2004+0+0 (3787653481 0 0) 0x55d986d33600 con 0x55d986df3800 > -2499> 2018-07-11 07:52:06.697498 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:07.327916) v4 -- 0x55d9b9ac3c00 con 0 > -2498> 2018-07-11 07:52:06.697520 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.114:0/15919 conn(0x55d986df5000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.11 seq 264 0x55d9869d9800 osd_ping(ping e501 stamp 2018-07-11 07:52:07.327916) v4 > -2497> 2018-07-11 07:52:06.697543 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.11 10.8.128.114:0/15919 264 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:07.327916) v4 ==== 2004+0+0 (3787653481 0 0) 0x55d9869d9800 con 0x55d986df5000 > -2496> 2018-07-11 07:52:06.697558 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:07.327916) v4 -- 0x55d9c908ec00 con 0 > -2495> 2018-07-11 07:52:06.838997 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1027 0x55d9c2152000 rep_scrubmap(5.2e e501 from shard 2) v1 > -2494> 2018-07-11 07:52:06.839016 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1027 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12468 (1269249638 0 1587094317) 0x55d9c2152000 con 0x55d986bf0800 > -2493> 2018-07-11 07:52:06.884671 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1021 0x55d9c2152240 rep_scrubmap(5.2e e501 from shard 4) v1 > -2492> 2018-07-11 07:52:06.884690 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1021 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12468 (3686815630 0 1587094317) 0x55d9c2152240 con 0x55d986bec000 > -2491> 2018-07-11 07:52:06.904287 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -2490> 2018-07-11 07:52:06.904311 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -2489> 2018-07-11 07:52:06.904362 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -2488> 2018-07-11 07:52:06.904805 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -2487> 2018-07-11 07:52:06.948071 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7572099a:::benchmark_data_magna115_10867_object1537064:0,end:5:757258ac:::benchmark_data_magna115_10867_object3274920:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498280 con 0 > -2486> 2018-07-11 07:52:06.948123 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7572099a:::benchmark_data_magna115_10867_object1537064:0,end:5:757258ac:::benchmark_data_magna115_10867_object3274920:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454780 con 0 > -2485> 2018-07-11 07:52:07.166145 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1028 0x55d9c2152480 rep_scrubmap(5.2e e501 from shard 2) v1 > -2484> 2018-07-11 07:52:07.166166 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1028 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12468 (1269249638 0 3670744074) 0x55d9c2152480 con 0x55d986bf0800 > -2483> 2018-07-11 07:52:07.175299 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1022 0x55d9c21526c0 rep_scrubmap(5.2e e501 from shard 4) v1 > -2482> 2018-07-11 07:52:07.175319 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1022 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12468 (3686815630 0 3670744074) 0x55d9c21526c0 con 0x55d986bec000 > -2481> 2018-07-11 07:52:07.255727 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -2480> 2018-07-11 07:52:07.255754 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -2479> 2018-07-11 07:52:07.255813 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -2478> 2018-07-11 07:52:07.256304 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -2477> 2018-07-11 07:52:07.271805 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:757258ac:::benchmark_data_magna115_10867_object3274920:0,end:5:7572c035:::benchmark_data_magna115_11707_object251411:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455400 con 0 > -2476> 2018-07-11 07:52:07.271856 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:757258ac:::benchmark_data_magna115_10867_object3274920:0,end:5:7572c035:::benchmark_data_magna115_11707_object251411:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454f00 con 0 > -2475> 2018-07-11 07:52:07.352338 7ffab6be6700 1 -- 10.8.128.107:6800/13927 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55d986cfe000 con 0 > -2474> 2018-07-11 07:52:07.352410 7ffab6be6700 1 -- 10.8.128.107:6800/13927 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55d986cc7b80 con 0 > -2473> 2018-07-11 07:52:07.422617 7ffab1bdc700 5 osd.10 501 heartbeat: osd_stat(789 GB used, 142 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist []) > -2472> 2018-07-11 07:52:07.422630 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6802/15255 -- osd_ping(ping e501 stamp 2018-07-11 07:52:07.422628) v4 -- 0x55d986c22600 con 0 > -2471> 2018-07-11 07:52:07.422665 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6803/15255 -- osd_ping(ping e501 stamp 2018-07-11 07:52:07.422628) v4 -- 0x55d9b60dea00 con 0 > -2470> 2018-07-11 07:52:07.422681 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6806/12677 -- osd_ping(ping e501 stamp 2018-07-11 07:52:07.422628) v4 -- 0x55d9c74e0200 con 0 > -2469> 2018-07-11 07:52:07.422695 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6807/12677 -- osd_ping(ping e501 stamp 2018-07-11 07:52:07.422628) v4 -- 0x55d986c22000 con 0 > -2468> 2018-07-11 07:52:07.422720 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6808/12651 -- osd_ping(ping e501 stamp 2018-07-11 07:52:07.422628) v4 -- 0x55d9c908fe00 con 0 > -2467> 2018-07-11 07:52:07.422731 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6810/12651 -- osd_ping(ping e501 stamp 2018-07-11 07:52:07.422628) v4 -- 0x55d9829e4e00 con 0 > -2466> 2018-07-11 07:52:07.422743 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6809/12650 -- osd_ping(ping e501 stamp 2018-07-11 07:52:07.422628) v4 -- 0x55d9c3383a00 con 0 > -2465> 2018-07-11 07:52:07.422757 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6811/12650 -- osd_ping(ping e501 stamp 2018-07-11 07:52:07.422628) v4 -- 0x55d9b67d9200 con 0 > -2464> 2018-07-11 07:52:07.422776 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6810/15287 -- osd_ping(ping e501 stamp 2018-07-11 07:52:07.422628) v4 -- 0x55d982cdee00 con 0 > -2463> 2018-07-11 07:52:07.422789 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6811/15287 -- osd_ping(ping e501 stamp 2018-07-11 07:52:07.422628) v4 -- 0x55d986d33c00 con 0 > -2462> 2018-07-11 07:52:07.422800 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6806/15263 -- osd_ping(ping e501 stamp 2018-07-11 07:52:07.422628) v4 -- 0x55d9c3383600 con 0 > -2461> 2018-07-11 07:52:07.422814 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6807/15263 -- osd_ping(ping e501 stamp 2018-07-11 07:52:07.422628) v4 -- 0x55d986c85000 con 0 > -2460> 2018-07-11 07:52:07.422826 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.115:6802/13323 -- osd_ping(ping e501 stamp 2018-07-11 07:52:07.422628) v4 -- 0x55d9869bfe00 con 0 > -2459> 2018-07-11 07:52:07.422838 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.115:6803/13323 -- osd_ping(ping e501 stamp 2018-07-11 07:52:07.422628) v4 -- 0x55d982cde200 con 0 > -2458> 2018-07-11 07:52:07.422854 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6806/15885 -- osd_ping(ping e501 stamp 2018-07-11 07:52:07.422628) v4 -- 0x55d9b60df600 con 0 > -2457> 2018-07-11 07:52:07.422867 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6807/15885 -- osd_ping(ping e501 stamp 2018-07-11 07:52:07.422628) v4 -- 0x55d986d4e400 con 0 > -2456> 2018-07-11 07:52:07.422879 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.108:6802/11967 -- osd_ping(ping e501 stamp 2018-07-11 07:52:07.422628) v4 -- 0x55d9b437ac00 con 0 > -2455> 2018-07-11 07:52:07.422891 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.108:6803/11967 -- osd_ping(ping e501 stamp 2018-07-11 07:52:07.422628) v4 -- 0x55d986d33200 con 0 > -2454> 2018-07-11 07:52:07.422904 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.107:6806/12617 -- osd_ping(ping e501 stamp 2018-07-11 07:52:07.422628) v4 -- 0x55d9b3db2600 con 0 > -2453> 2018-07-11 07:52:07.422916 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.107:6807/12617 -- osd_ping(ping e501 stamp 2018-07-11 07:52:07.422628) v4 -- 0x55d9bcd72e00 con 0 > -2452> 2018-07-11 07:52:07.422929 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6802/15919 -- osd_ping(ping e501 stamp 2018-07-11 07:52:07.422628) v4 -- 0x55d9c3382200 con 0 > -2451> 2018-07-11 07:52:07.422940 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6803/15919 -- osd_ping(ping e501 stamp 2018-07-11 07:52:07.422628) v4 -- 0x55d982cde600 con 0 > -2450> 2018-07-11 07:52:07.423123 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.107:6806/12617 conn(0x55d986bd2000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.9 seq 252 0x55d9869d9800 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:07.422628) v4 > -2449> 2018-07-11 07:52:07.423135 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.107:6807/12617 conn(0x55d986bd3800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.9 seq 252 0x55d9bdba5e00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:07.422628) v4 > -2448> 2018-07-11 07:52:07.423146 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.9 10.8.128.107:6806/12617 252 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:07.422628) v4 ==== 2004+0+0 (2802212333 0 0) 0x55d9869d9800 con 0x55d986bd2000 > -2447> 2018-07-11 07:52:07.423158 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.9 10.8.128.107:6807/12617 252 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:07.422628) v4 ==== 2004+0+0 (2802212333 0 0) 0x55d9bdba5e00 con 0x55d986bd3800 > -2446> 2018-07-11 07:52:07.423167 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6802/15255 conn(0x55d986bc8000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.0 seq 252 0x55d9c4c1be00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:07.422628) v4 > -2445> 2018-07-11 07:52:07.423185 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.0 10.8.128.113:6802/15255 252 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:07.422628) v4 ==== 2004+0+0 (2802212333 0 0) 0x55d9c4c1be00 con 0x55d986bc8000 > -2444> 2018-07-11 07:52:07.423204 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6803/15255 conn(0x55d986bc9800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.0 seq 252 0x55d9869d9800 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:07.422628) v4 > -2443> 2018-07-11 07:52:07.423219 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.0 10.8.128.113:6803/15255 252 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:07.422628) v4 ==== 2004+0+0 (2802212333 0 0) 0x55d9869d9800 con 0x55d986bc9800 > -2442> 2018-07-11 07:52:07.423223 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6808/12651 conn(0x55d986b9a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.2 seq 252 0x55d9c4c1be00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:07.422628) v4 > -2441> 2018-07-11 07:52:07.423236 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.2 10.8.128.106:6808/12651 252 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:07.422628) v4 ==== 2004+0+0 (2802212333 0 0) 0x55d9c4c1be00 con 0x55d986b9a000 > -2440> 2018-07-11 07:52:07.423272 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6810/12651 conn(0x55d986b9b800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.2 seq 252 0x55d9869d9800 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:07.422628) v4 > -2439> 2018-07-11 07:52:07.423290 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.2 10.8.128.106:6810/12651 252 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:07.422628) v4 ==== 2004+0+0 (2802212333 0 0) 0x55d9869d9800 con 0x55d986b9b800 > -2438> 2018-07-11 07:52:07.423289 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6810/15287 conn(0x55d986baf000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 252 0x55d9c4c1be00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:07.422628) v4 > -2437> 2018-07-11 07:52:07.423301 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.4 10.8.128.113:6810/15287 252 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:07.422628) v4 ==== 2004+0+0 (2802212333 0 0) 0x55d9c4c1be00 con 0x55d986baf000 > -2436> 2018-07-11 07:52:07.423340 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6811/12650 conn(0x55d986bcc800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 252 0x55d9c4c1be00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:07.422628) v4 > -2435> 2018-07-11 07:52:07.423322 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6806/12677 conn(0x55d986b58000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.1 seq 252 0x55d9bdba5e00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:07.422628) v4 > -2434> 2018-07-11 07:52:07.423353 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.3 10.8.128.106:6811/12650 252 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:07.422628) v4 ==== 2004+0+0 (2802212333 0 0) 0x55d9c4c1be00 con 0x55d986bcc800 > -2433> 2018-07-11 07:52:07.423359 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.1 10.8.128.106:6806/12677 252 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:07.422628) v4 ==== 2004+0+0 (2802212333 0 0) 0x55d9bdba5e00 con 0x55d986b58000 > -2432> 2018-07-11 07:52:07.423384 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6807/15263 conn(0x55d986b9e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.5 seq 252 0x55d9c4c1be00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:07.422628) v4 > -2431> 2018-07-11 07:52:07.423386 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6809/12650 conn(0x55d986bcb000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 252 0x55d9bdba5e00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:07.422628) v4 > -2430> 2018-07-11 07:52:07.423388 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6811/15287 conn(0x55d986bb0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 252 0x55d9869d9800 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:07.422628) v4 > -2429> 2018-07-11 07:52:07.423399 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.5 10.8.128.113:6807/15263 252 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:07.422628) v4 ==== 2004+0+0 (2802212333 0 0) 0x55d9c4c1be00 con 0x55d986b9e800 > -2428> 2018-07-11 07:52:07.423404 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.3 10.8.128.106:6809/12650 252 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:07.422628) v4 ==== 2004+0+0 (2802212333 0 0) 0x55d9bdba5e00 con 0x55d986bcb000 > -2427> 2018-07-11 07:52:07.423405 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.4 10.8.128.113:6811/15287 252 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:07.422628) v4 ==== 2004+0+0 (2802212333 0 0) 0x55d9869d9800 con 0x55d986bb0800 > -2426> 2018-07-11 07:52:07.423431 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6807/12677 conn(0x55d986b59800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.1 seq 252 0x55d9c4c1be00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:07.422628) v4 > -2425> 2018-07-11 07:52:07.423443 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.1 10.8.128.106:6807/12677 252 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:07.422628) v4 ==== 2004+0+0 (2802212333 0 0) 0x55d9c4c1be00 con 0x55d986b59800 > -2424> 2018-07-11 07:52:07.423445 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6806/15263 conn(0x55d986b9d000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.5 seq 252 0x55d9bdba5e00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:07.422628) v4 > -2423> 2018-07-11 07:52:07.423446 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6806/15885 conn(0x55d986bac000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.7 seq 252 0x55d9869d9800 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:07.422628) v4 > -2422> 2018-07-11 07:52:07.423460 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.5 10.8.128.113:6806/15263 252 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:07.422628) v4 ==== 2004+0+0 (2802212333 0 0) 0x55d9bdba5e00 con 0x55d986b9d000 > -2421> 2018-07-11 07:52:07.423465 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.7 10.8.128.114:6806/15885 252 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:07.422628) v4 ==== 2004+0+0 (2802212333 0 0) 0x55d9869d9800 con 0x55d986bac000 > -2420> 2018-07-11 07:52:07.423475 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.108:6803/11967 conn(0x55d986bbf800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.8 seq 252 0x55d9c4c1be00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:07.422628) v4 > -2419> 2018-07-11 07:52:07.423492 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6807/15885 conn(0x55d986bad800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.7 seq 252 0x55d9bdba5e00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:07.422628) v4 > -2418> 2018-07-11 07:52:07.423501 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.7 10.8.128.114:6807/15885 252 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:07.422628) v4 ==== 2004+0+0 (2802212333 0 0) 0x55d9bdba5e00 con 0x55d986bad800 > -2417> 2018-07-11 07:52:07.423501 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.8 10.8.128.108:6803/11967 252 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:07.422628) v4 ==== 2004+0+0 (2802212333 0 0) 0x55d9c4c1be00 con 0x55d986bbf800 > -2416> 2018-07-11 07:52:07.423498 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.115:6802/13323 conn(0x55d986bc1000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 252 0x55d9869d9800 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:07.422628) v4 > -2415> 2018-07-11 07:52:07.423521 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.6 10.8.128.115:6802/13323 252 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:07.422628) v4 ==== 2004+0+0 (2802212333 0 0) 0x55d9869d9800 con 0x55d986bc1000 > -2414> 2018-07-11 07:52:07.423549 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.108:6802/11967 conn(0x55d986bbe000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.8 seq 252 0x55d9bdba5e00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:07.422628) v4 > -2413> 2018-07-11 07:52:07.423562 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.8 10.8.128.108:6802/11967 252 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:07.422628) v4 ==== 2004+0+0 (2802212333 0 0) 0x55d9bdba5e00 con 0x55d986bbe000 > -2412> 2018-07-11 07:52:07.423564 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6802/15919 conn(0x55d986b5b000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.11 seq 252 0x55d9869d9800 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:07.422628) v4 > -2411> 2018-07-11 07:52:07.423577 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.11 10.8.128.114:6802/15919 252 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:07.422628) v4 ==== 2004+0+0 (2802212333 0 0) 0x55d9869d9800 con 0x55d986b5b000 > -2410> 2018-07-11 07:52:07.423595 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.115:6803/13323 conn(0x55d986bc2800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 252 0x55d9bdba5e00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:07.422628) v4 > -2409> 2018-07-11 07:52:07.423607 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.6 10.8.128.115:6803/13323 252 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:07.422628) v4 ==== 2004+0+0 (2802212333 0 0) 0x55d9bdba5e00 con 0x55d986bc2800 > -2408> 2018-07-11 07:52:07.423634 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6803/15919 conn(0x55d986b5c800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.11 seq 252 0x55d9bdba5e00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:07.422628) v4 > -2407> 2018-07-11 07:52:07.423645 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.11 10.8.128.114:6803/15919 252 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:07.422628) v4 ==== 2004+0+0 (2802212333 0 0) 0x55d9bdba5e00 con 0x55d986b5c800 > -2406> 2018-07-11 07:52:07.478411 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1029 0x55d9c2152900 rep_scrubmap(5.2e e501 from shard 2) v1 > -2405> 2018-07-11 07:52:07.478430 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1029 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12480 (1269249638 0 1739887415) 0x55d9c2152900 con 0x55d986bf0800 > -2404> 2018-07-11 07:52:07.482231 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1023 0x55d9c2152b40 rep_scrubmap(5.2e e501 from shard 4) v1 > -2403> 2018-07-11 07:52:07.482259 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1023 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12480 (3686815630 0 1739887415) 0x55d9c2152b40 con 0x55d986bec000 > -2402> 2018-07-11 07:52:07.493691 7ffacbbd1700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.108:0/11967 conn(0x55d986d82800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.8 seq 244 0x55d9bdba5e00 osd_ping(ping e501 stamp 2018-07-11 07:52:07.249484) v4 > -2401> 2018-07-11 07:52:07.493699 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.108:0/11967 conn(0x55d986db0000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.8 seq 244 0x55d9869d9800 osd_ping(ping e501 stamp 2018-07-11 07:52:07.249484) v4 > -2400> 2018-07-11 07:52:07.493712 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 <== osd.8 10.8.128.108:0/11967 244 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:07.249484) v4 ==== 2004+0+0 (3693303446 0 0) 0x55d9bdba5e00 con 0x55d986d82800 > -2399> 2018-07-11 07:52:07.493725 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:07.249484) v4 -- 0x55d982cde600 con 0 > -2398> 2018-07-11 07:52:07.493718 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.8 10.8.128.108:0/11967 244 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:07.249484) v4 ==== 2004+0+0 (3693303446 0 0) 0x55d9869d9800 con 0x55d986db0000 > -2397> 2018-07-11 07:52:07.493751 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:07.249484) v4 -- 0x55d9c3382200 con 0 > -2396> 2018-07-11 07:52:07.505738 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.115:0/13323 conn(0x55d986d7e000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 261 0x55d986d33600 osd_ping(ping e501 stamp 2018-07-11 07:52:07.851757) v4 > -2395> 2018-07-11 07:52:07.505759 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.6 10.8.128.115:0/13323 261 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:07.851757) v4 ==== 2004+0+0 (1510578958 0 0) 0x55d986d33600 con 0x55d986d7e000 > -2394> 2018-07-11 07:52:07.505768 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:07.851757) v4 -- 0x55d982cde600 con 0 > -2393> 2018-07-11 07:52:07.505835 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.115:0/13323 conn(0x55d986d24800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 261 0x55d9c4c1be00 osd_ping(ping e501 stamp 2018-07-11 07:52:07.851757) v4 > -2392> 2018-07-11 07:52:07.505860 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.6 10.8.128.115:0/13323 261 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:07.851757) v4 ==== 2004+0+0 (1510578958 0 0) 0x55d9c4c1be00 con 0x55d986d24800 > -2391> 2018-07-11 07:52:07.505871 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:07.851757) v4 -- 0x55d986d33200 con 0 > -2390> 2018-07-11 07:52:07.619753 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -2389> 2018-07-11 07:52:07.619779 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -2388> 2018-07-11 07:52:07.619837 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -2387> 2018-07-11 07:52:07.620308 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -2386> 2018-07-11 07:52:07.621364 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'66854,epoch:501/498,start:5:7572c035:::benchmark_data_magna115_11707_object251411:0,end:5:75731fb3:::benchmark_data_magna115_10867_object3393149:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455180 con 0 > -2385> 2018-07-11 07:52:07.621399 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'66854,epoch:501/498,start:5:7572c035:::benchmark_data_magna115_11707_object251411:0,end:5:75731fb3:::benchmark_data_magna115_10867_object3393149:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454500 con 0 > -2384> 2018-07-11 07:52:07.651702 7ffacbbd1700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15263 conn(0x55d985d44000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.5 seq 244 0x55d9bdba5e00 osd_ping(ping e501 stamp 2018-07-11 07:52:08.883275) v4 > -2383> 2018-07-11 07:52:07.651723 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 <== osd.5 10.8.128.113:0/15263 244 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:08.883275) v4 ==== 2004+0+0 (3771515485 0 0) 0x55d9bdba5e00 con 0x55d985d44000 > -2382> 2018-07-11 07:52:07.651718 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15263 conn(0x55d985d45800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.5 seq 244 0x55d982cdf800 osd_ping(ping e501 stamp 2018-07-11 07:52:08.883275) v4 > -2381> 2018-07-11 07:52:07.651732 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:08.883275) v4 -- 0x55d982cde600 con 0 > -2380> 2018-07-11 07:52:07.651735 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.5 10.8.128.113:0/15263 244 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:08.883275) v4 ==== 2004+0+0 (3771515485 0 0) 0x55d982cdf800 con 0x55d985d45800 > -2379> 2018-07-11 07:52:07.651754 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:08.883275) v4 -- 0x55d9c3382200 con 0 > -2378> 2018-07-11 07:52:07.844192 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 3026 0x55d986d33600 MOSDScrubReserve(11.3s5 REQUEST e501) v1 > -2377> 2018-07-11 07:52:07.844210 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 3026 ==== MOSDScrubReserve(11.3s5 REQUEST e501) v1 ==== 43+0+0 (3260170403 0 0) 0x55d986d33600 con 0x55d986c7e000 > -2376> 2018-07-11 07:52:07.844334 7ffab3be0700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e501) v1 -- 0x55d986d97200 con 0 > -2375> 2018-07-11 07:52:07.844677 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 3027 0x55d982cde600 MOSDScrubReserve(11.3s5 RELEASE e501) v1 > -2374> 2018-07-11 07:52:07.844692 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 3027 ==== MOSDScrubReserve(11.3s5 RELEASE e501) v1 ==== 43+0+0 (223328929 0 0) 0x55d982cde600 con 0x55d986c7e000 > -2373> 2018-07-11 07:52:07.850969 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1030 0x55d9c2152d80 rep_scrubmap(5.2e e501 from shard 2) v1 > -2372> 2018-07-11 07:52:07.850990 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1030 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12472 (1269249638 0 409652251) 0x55d9c2152d80 con 0x55d986bf0800 > -2371> 2018-07-11 07:52:07.856769 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1024 0x55d9c2152fc0 rep_scrubmap(5.2e e501 from shard 4) v1 > -2370> 2018-07-11 07:52:07.856788 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1024 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12472 (3686815630 0 409652251) 0x55d9c2152fc0 con 0x55d986bec000 > -2369> 2018-07-11 07:52:07.945044 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -2368> 2018-07-11 07:52:07.945083 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -2367> 2018-07-11 07:52:07.945140 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -2366> 2018-07-11 07:52:07.945622 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -2365> 2018-07-11 07:52:07.963803 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75731fb3:::benchmark_data_magna115_10867_object3393149:0,end:5:7573917a:::benchmark_data_magna115_10867_object1027455:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98944b680 con 0 > -2364> 2018-07-11 07:52:07.963837 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75731fb3:::benchmark_data_magna115_10867_object3393149:0,end:5:7573917a:::benchmark_data_magna115_10867_object1027455:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454a00 con 0 > -2363> 2018-07-11 07:52:07.965146 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6805/15263 conn(0x55d986c41000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=31 cs=1 l=0). rx osd.5 seq 2840 0x55d9b60df400 MOSDScrubReserve(6.15 REQUEST e501) v1 > -2362> 2018-07-11 07:52:07.965164 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.5 10.8.128.113:6805/15263 2840 ==== MOSDScrubReserve(6.15 REQUEST e501) v1 ==== 43+0+0 (3942742026 0 0) 0x55d9b60df400 con 0x55d986c41000 > -2361> 2018-07-11 07:52:08.006192 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.115:0/13323 conn(0x55d986d24800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 262 0x55d9c4c1be00 osd_ping(ping e501 stamp 2018-07-11 07:52:08.352160) v4 > -2360> 2018-07-11 07:52:08.006218 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.6 10.8.128.115:0/13323 262 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:08.352160) v4 ==== 2004+0+0 (2470001809 0 0) 0x55d9c4c1be00 con 0x55d986d24800 > -2359> 2018-07-11 07:52:08.006228 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:08.352160) v4 -- 0x55d986c85000 con 0 > -2358> 2018-07-11 07:52:08.006260 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.115:0/13323 conn(0x55d986d7e000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 262 0x55d9bdba5e00 osd_ping(ping e501 stamp 2018-07-11 07:52:08.352160) v4 > -2357> 2018-07-11 07:52:08.006278 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.6 10.8.128.115:0/13323 262 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:08.352160) v4 ==== 2004+0+0 (2470001809 0 0) 0x55d9bdba5e00 con 0x55d986d7e000 > -2356> 2018-07-11 07:52:08.006294 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:08.352160) v4 -- 0x55d9bcd72e00 con 0 > -2355> 2018-07-11 07:52:08.022485 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.114:0/15885 conn(0x55d986d81000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.7 seq 267 0x55d986d33200 osd_ping(ping e501 stamp 2018-07-11 07:52:08.652888) v4 > -2354> 2018-07-11 07:52:08.022510 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.7 10.8.128.114:0/15885 267 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:08.652888) v4 ==== 2004+0+0 (3767137606 0 0) 0x55d986d33200 con 0x55d986d81000 > -2353> 2018-07-11 07:52:08.022523 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:08.652888) v4 -- 0x55d986c85000 con 0 > -2352> 2018-07-11 07:52:08.022529 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.114:0/15885 conn(0x55d986d7f800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.7 seq 267 0x55d9869d9800 osd_ping(ping e501 stamp 2018-07-11 07:52:08.652888) v4 > -2351> 2018-07-11 07:52:08.022553 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.7 10.8.128.114:0/15885 267 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:08.652888) v4 ==== 2004+0+0 (3767137606 0 0) 0x55d9869d9800 con 0x55d986d7f800 > -2350> 2018-07-11 07:52:08.022572 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:08.652888) v4 -- 0x55d9c3382200 con 0 > -2349> 2018-07-11 07:52:08.171860 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15255 conn(0x55d986c82800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.0 seq 262 0x55d982cdf800 osd_ping(ping e501 stamp 2018-07-11 07:52:09.403362) v4 > -2348> 2018-07-11 07:52:08.171868 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15255 conn(0x55d986d20000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.0 seq 262 0x55d9c4c1be00 osd_ping(ping e501 stamp 2018-07-11 07:52:09.403362) v4 > -2347> 2018-07-11 07:52:08.171882 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.0 10.8.128.113:0/15255 262 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:09.403362) v4 ==== 2004+0+0 (996671055 0 0) 0x55d982cdf800 con 0x55d986c82800 > -2346> 2018-07-11 07:52:08.171892 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:09.403362) v4 -- 0x55d9c3382200 con 0 > -2345> 2018-07-11 07:52:08.171891 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.0 10.8.128.113:0/15255 262 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:09.403362) v4 ==== 2004+0+0 (996671055 0 0) 0x55d9c4c1be00 con 0x55d986d20000 > -2344> 2018-07-11 07:52:08.171917 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:09.403362) v4 -- 0x55d986c85000 con 0 > -2343> 2018-07-11 07:52:08.188619 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1031 0x55d9c2153200 rep_scrubmap(5.2e e501 from shard 2) v1 > -2342> 2018-07-11 07:52:08.188638 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1031 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12456 (1269249638 0 2568970895) 0x55d9c2153200 con 0x55d986bf0800 > -2341> 2018-07-11 07:52:08.201843 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1025 0x55d9c2153440 rep_scrubmap(5.2e e501 from shard 4) v1 > -2340> 2018-07-11 07:52:08.201863 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1025 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12456 (3686815630 0 2568970895) 0x55d9c2153440 con 0x55d986bec000 > -2339> 2018-07-11 07:52:08.378429 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6805/15263 -- MOSDScrubReserve(6.15 REJECT e501) v1 -- 0x55d986c22e00 con 0 > -2338> 2018-07-11 07:52:08.378632 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -2337> 2018-07-11 07:52:08.378656 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -2336> 2018-07-11 07:52:08.378702 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -2335> 2018-07-11 07:52:08.378870 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6805/15263 conn(0x55d986c41000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=31 cs=1 l=0). rx osd.5 seq 2841 0x55d986c22e00 MOSDScrubReserve(6.15 RELEASE e501) v1 > -2334> 2018-07-11 07:52:08.378889 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.5 10.8.128.113:6805/15263 2841 ==== MOSDScrubReserve(6.15 RELEASE e501) v1 ==== 43+0+0 (605848072 0 0) 0x55d986c22e00 con 0x55d986c41000 > -2333> 2018-07-11 07:52:08.379165 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -2332> 2018-07-11 07:52:08.397420 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7573917a:::benchmark_data_magna115_10867_object1027455:0,end:5:7573f788:::benchmark_data_magna115_10867_object3097492:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98944bb80 con 0 > -2331> 2018-07-11 07:52:08.397468 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7573917a:::benchmark_data_magna115_10867_object1027455:0,end:5:7573f788:::benchmark_data_magna115_10867_object3097492:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98944b900 con 0 > -2330> 2018-07-11 07:52:08.454658 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.107:0/12617 conn(0x55d98123c000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 252 0x55d9c4c1be00 osd_ping(ping e501 stamp 2018-07-11 07:52:08.454341) v4 > -2329> 2018-07-11 07:52:08.454674 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.107:0/12617 conn(0x55d98123d800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 252 0x55d986d97200 osd_ping(ping e501 stamp 2018-07-11 07:52:08.454341) v4 > -2328> 2018-07-11 07:52:08.454696 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.9 10.8.128.107:0/12617 252 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:08.454341) v4 ==== 2004+0+0 (1187653231 0 0) 0x55d986d97200 con 0x55d98123d800 > -2327> 2018-07-11 07:52:08.454697 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.9 10.8.128.107:0/12617 252 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:08.454341) v4 ==== 2004+0+0 (1187653231 0 0) 0x55d9c4c1be00 con 0x55d98123c000 > -2326> 2018-07-11 07:52:08.454715 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:08.454341) v4 -- 0x55d9bcd72e00 con 0 > -2325> 2018-07-11 07:52:08.454746 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:08.454341) v4 -- 0x55d986c85000 con 0 > -2324> 2018-07-11 07:52:08.624488 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1032 0x55d9c2153680 rep_scrubmap(5.2e e501 from shard 2) v1 > -2323> 2018-07-11 07:52:08.624505 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1032 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12476 (1269249638 0 248373781) 0x55d9c2153680 con 0x55d986bf0800 > -2322> 2018-07-11 07:52:08.635365 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1026 0x55d9c21538c0 rep_scrubmap(5.2e e501 from shard 4) v1 > -2321> 2018-07-11 07:52:08.635384 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1026 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12476 (3686815630 0 248373781) 0x55d9c21538c0 con 0x55d986bec000 > -2320> 2018-07-11 07:52:08.668242 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -2319> 2018-07-11 07:52:08.668273 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -2318> 2018-07-11 07:52:08.668321 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -2317> 2018-07-11 07:52:08.668811 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -2316> 2018-07-11 07:52:08.707480 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7573f788:::benchmark_data_magna115_10867_object3097492:0,end:5:757435ab:::benchmark_data_magna115_10867_object3109707:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb772280 con 0 > -2315> 2018-07-11 07:52:08.707532 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7573f788:::benchmark_data_magna115_10867_object3097492:0,end:5:757435ab:::benchmark_data_magna115_10867_object3109707:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98944b400 con 0 > -2314> 2018-07-11 07:52:08.890280 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1033 0x55d9c2153b00 rep_scrubmap(5.2e e501 from shard 2) v1 > -2313> 2018-07-11 07:52:08.890299 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1033 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12472 (1269249638 0 1170675470) 0x55d9c2153b00 con 0x55d986bf0800 > -2312> 2018-07-11 07:52:08.927352 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1027 0x55d9c2153d40 rep_scrubmap(5.2e e501 from shard 4) v1 > -2311> 2018-07-11 07:52:08.927371 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1027 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12472 (3686815630 0 1170675470) 0x55d9c2153d40 con 0x55d986bec000 > -2310> 2018-07-11 07:52:08.968229 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12677 conn(0x55d986db4800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.1 seq 264 0x55d9869d9800 osd_ping(ping e501 stamp 2018-07-11 07:52:09.016274) v4 > -2309> 2018-07-11 07:52:08.968232 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12677 conn(0x55d986df2000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.1 seq 264 0x55d9c4c1be00 osd_ping(ping e501 stamp 2018-07-11 07:52:09.016274) v4 > -2308> 2018-07-11 07:52:08.968265 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.1 10.8.128.106:0/12677 264 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:09.016274) v4 ==== 2004+0+0 (3120561005 0 0) 0x55d9869d9800 con 0x55d986db4800 > -2307> 2018-07-11 07:52:08.968269 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.1 10.8.128.106:0/12677 264 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:09.016274) v4 ==== 2004+0+0 (3120561005 0 0) 0x55d9c4c1be00 con 0x55d986df2000 > -2306> 2018-07-11 07:52:08.968276 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:09.016274) v4 -- 0x55d9c3382200 con 0 > -2305> 2018-07-11 07:52:08.968303 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:09.016274) v4 -- 0x55d986c85000 con 0 > -2304> 2018-07-11 07:52:08.972586 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -2303> 2018-07-11 07:52:08.972625 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -2302> 2018-07-11 07:52:08.972680 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -2301> 2018-07-11 07:52:08.973141 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -2300> 2018-07-11 07:52:08.989326 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:757435ab:::benchmark_data_magna115_10867_object3109707:0,end:5:75748688:::benchmark_data_magna115_10867_object2568386:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb772000 con 0 > -2299> 2018-07-11 07:52:08.989375 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:757435ab:::benchmark_data_magna115_10867_object3109707:0,end:5:75748688:::benchmark_data_magna115_10867_object2568386:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb772500 con 0 > -2298> 2018-07-11 07:52:09.021015 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12650 conn(0x55d986d23000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 254 0x55d982cdf800 osd_ping(ping e501 stamp 2018-07-11 07:52:09.068961) v4 > -2297> 2018-07-11 07:52:09.021036 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.3 10.8.128.106:0/12650 254 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:09.068961) v4 ==== 2004+0+0 (1814859367 0 0) 0x55d982cdf800 con 0x55d986d23000 > -2296> 2018-07-11 07:52:09.021047 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:09.068961) v4 -- 0x55d9c3382200 con 0 > -2295> 2018-07-11 07:52:09.021063 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12650 conn(0x55d986d21800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 254 0x55d9bdba5e00 osd_ping(ping e501 stamp 2018-07-11 07:52:09.068961) v4 > -2294> 2018-07-11 07:52:09.021084 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.3 10.8.128.106:0/12650 254 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:09.068961) v4 ==== 2004+0+0 (1814859367 0 0) 0x55d9bdba5e00 con 0x55d986d21800 > -2293> 2018-07-11 07:52:09.021095 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:09.068961) v4 -- 0x55d9bcd72e00 con 0 > -2292> 2018-07-11 07:52:09.183886 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1034 0x55d9c795c000 rep_scrubmap(5.2e e501 from shard 2) v1 > -2291> 2018-07-11 07:52:09.183907 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1034 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12474 (1269249638 0 3778002578) 0x55d9c795c000 con 0x55d986bf0800 > -2290> 2018-07-11 07:52:09.194110 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.108:0/11967 conn(0x55d986db0000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.8 seq 245 0x55d9869d9800 osd_ping(ping e501 stamp 2018-07-11 07:52:08.949927) v4 > -2289> 2018-07-11 07:52:09.194131 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.8 10.8.128.108:0/11967 245 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:08.949927) v4 ==== 2004+0+0 (588974784 0 0) 0x55d9869d9800 con 0x55d986db0000 > -2288> 2018-07-11 07:52:09.194147 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:08.949927) v4 -- 0x55d9c3382200 con 0 > -2287> 2018-07-11 07:52:09.194171 7ffacbbd1700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.108:0/11967 conn(0x55d986d82800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.8 seq 245 0x55d986d97200 osd_ping(ping e501 stamp 2018-07-11 07:52:08.949927) v4 > -2286> 2018-07-11 07:52:09.194196 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 <== osd.8 10.8.128.108:0/11967 245 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:08.949927) v4 ==== 2004+0+0 (588974784 0 0) 0x55d986d97200 con 0x55d986d82800 > -2285> 2018-07-11 07:52:09.194211 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:08.949927) v4 -- 0x55d9bcd72e00 con 0 > -2284> 2018-07-11 07:52:09.227358 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1028 0x55d9c795c240 rep_scrubmap(5.2e e501 from shard 4) v1 > -2283> 2018-07-11 07:52:09.227378 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1028 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12474 (3686815630 0 3778002578) 0x55d9c795c240 con 0x55d986bec000 > -2282> 2018-07-11 07:52:09.293579 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -2281> 2018-07-11 07:52:09.293603 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -2280> 2018-07-11 07:52:09.293661 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -2279> 2018-07-11 07:52:09.294124 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -2278> 2018-07-11 07:52:09.337555 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67719,epoch:501/498,start:5:75748688:::benchmark_data_magna115_10867_object2568386:0,end:5:75754030:::benchmark_data_magna115_10867_object2706591:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb772c80 con 0 > -2277> 2018-07-11 07:52:09.337592 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67719,epoch:501/498,start:5:75748688:::benchmark_data_magna115_10867_object2568386:0,end:5:75754030:::benchmark_data_magna115_10867_object2706591:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb772a00 con 0 > -2276> 2018-07-11 07:52:09.468616 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12677 conn(0x55d986db4800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.1 seq 265 0x55d982cdf800 osd_ping(ping e501 stamp 2018-07-11 07:52:09.516685) v4 > -2275> 2018-07-11 07:52:09.468619 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12677 conn(0x55d986df2000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.1 seq 265 0x55d986d33200 osd_ping(ping e501 stamp 2018-07-11 07:52:09.516685) v4 > -2274> 2018-07-11 07:52:09.468644 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.1 10.8.128.106:0/12677 265 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:09.516685) v4 ==== 2004+0+0 (639610122 0 0) 0x55d982cdf800 con 0x55d986db4800 > -2273> 2018-07-11 07:52:09.468647 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.1 10.8.128.106:0/12677 265 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:09.516685) v4 ==== 2004+0+0 (639610122 0 0) 0x55d986d33200 con 0x55d986df2000 > -2272> 2018-07-11 07:52:09.468653 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:09.516685) v4 -- 0x55d9c3382200 con 0 > -2271> 2018-07-11 07:52:09.468675 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:09.516685) v4 -- 0x55d986c85000 con 0 > -2270> 2018-07-11 07:52:09.528984 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1035 0x55d9c795c480 rep_scrubmap(5.2e e501 from shard 2) v1 > -2269> 2018-07-11 07:52:09.529002 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1035 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12478 (1269249638 0 3666853059) 0x55d9c795c480 con 0x55d986bf0800 > -2268> 2018-07-11 07:52:09.562119 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1029 0x55d9c795c6c0 rep_scrubmap(5.2e e501 from shard 4) v1 > -2267> 2018-07-11 07:52:09.562139 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1029 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12478 (3686815630 0 3666853059) 0x55d9c795c6c0 con 0x55d986bec000 > -2266> 2018-07-11 07:52:09.568937 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6805/15885 conn(0x55d986c7f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.7 seq 2818 0x55d9c4c1be00 MOSDScrubReserve(5.4 REQUEST e501) v1 > -2265> 2018-07-11 07:52:09.568968 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.7 10.8.128.114:6805/15885 2818 ==== MOSDScrubReserve(5.4 REQUEST e501) v1 ==== 43+0+0 (1753221512 0 0) 0x55d9c4c1be00 con 0x55d986c7f800 > -2264> 2018-07-11 07:52:09.569026 7ffab33df700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.114:6805/15885 -- MOSDScrubReserve(5.4 REJECT e501) v1 -- 0x55d9829e5600 con 0 > -2263> 2018-07-11 07:52:09.569317 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6805/15885 conn(0x55d986c7f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.7 seq 2819 0x55d9829e5600 MOSDScrubReserve(5.4 RELEASE e501) v1 > -2262> 2018-07-11 07:52:09.569332 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.7 10.8.128.114:6805/15885 2819 ==== MOSDScrubReserve(5.4 RELEASE e501) v1 ==== 43+0+0 (2812146570 0 0) 0x55d9829e5600 con 0x55d986c7f800 > -2261> 2018-07-11 07:52:09.674680 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -2260> 2018-07-11 07:52:09.674705 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -2259> 2018-07-11 07:52:09.674748 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -2258> 2018-07-11 07:52:09.675210 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -2257> 2018-07-11 07:52:09.722577 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67915,epoch:501/498,start:5:75754030:::benchmark_data_magna115_10867_object2706591:0,end:5:7575aec0:::benchmark_data_magna115_10867_object2814667:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb772f00 con 0 > -2256> 2018-07-11 07:52:09.722615 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67915,epoch:501/498,start:5:75754030:::benchmark_data_magna115_10867_object2706591:0,end:5:7575aec0:::benchmark_data_magna115_10867_object2814667:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb773900 con 0 > -2255> 2018-07-11 07:52:09.921568 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1036 0x55d9c795c900 rep_scrubmap(5.2e e501 from shard 2) v1 > -2254> 2018-07-11 07:52:09.921587 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1036 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12480 (1269249638 0 1035315882) 0x55d9c795c900 con 0x55d986bf0800 > -2253> 2018-07-11 07:52:09.951567 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1030 0x55d9c795cb40 rep_scrubmap(5.2e e501 from shard 4) v1 > -2252> 2018-07-11 07:52:09.951586 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1030 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12480 (3686815630 0 1035315882) 0x55d9c795cb40 con 0x55d986bec000 > -2251> 2018-07-11 07:52:09.969098 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12677 conn(0x55d986db4800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.1 seq 266 0x55d9869d9800 osd_ping(ping e501 stamp 2018-07-11 07:52:10.017164) v4 > -2250> 2018-07-11 07:52:09.969119 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.1 10.8.128.106:0/12677 266 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:10.017164) v4 ==== 2004+0+0 (1446183892 0 0) 0x55d9869d9800 con 0x55d986db4800 > -2249> 2018-07-11 07:52:09.969128 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:10.017164) v4 -- 0x55d9c3382200 con 0 > -2248> 2018-07-11 07:52:09.969132 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12677 conn(0x55d986df2000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.1 seq 266 0x55d986d33200 osd_ping(ping e501 stamp 2018-07-11 07:52:10.017164) v4 > -2247> 2018-07-11 07:52:09.969155 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.1 10.8.128.106:0/12677 266 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:10.017164) v4 ==== 2004+0+0 (1446183892 0 0) 0x55d986d33200 con 0x55d986df2000 > -2246> 2018-07-11 07:52:09.969166 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:10.017164) v4 -- 0x55d982cdee00 con 0 > -2245> 2018-07-11 07:52:10.167170 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -2244> 2018-07-11 07:52:10.167201 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -2243> 2018-07-11 07:52:10.167273 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -2242> 2018-07-11 07:52:10.167738 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -2241> 2018-07-11 07:52:10.189611 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7575aec0:::benchmark_data_magna115_10867_object2814667:0,end:5:75760585:::benchmark_data_magna115_10867_object1926395:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb772780 con 0 > -2240> 2018-07-11 07:52:10.189666 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7575aec0:::benchmark_data_magna115_10867_object2814667:0,end:5:75760585:::benchmark_data_magna115_10867_object1926395:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb773180 con 0 > -2239> 2018-07-11 07:52:10.391706 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1037 0x55d9c795cd80 rep_scrubmap(5.2e e501 from shard 2) v1 > -2238> 2018-07-11 07:52:10.391726 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1037 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12474 (1269249638 0 3384038035) 0x55d9c795cd80 con 0x55d986bf0800 > -2237> 2018-07-11 07:52:10.424593 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1031 0x55d9c795cfc0 rep_scrubmap(5.2e e501 from shard 4) v1 > -2236> 2018-07-11 07:52:10.424613 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1031 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12474 (3686815630 0 3384038035) 0x55d9c795cfc0 con 0x55d986bec000 > -2235> 2018-07-11 07:52:10.527805 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12651 conn(0x55d986db3000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.2 seq 260 0x55d9bdba5e00 osd_ping(ping e501 stamp 2018-07-11 07:52:10.575831) v4 > -2234> 2018-07-11 07:52:10.527805 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12651 conn(0x55d986db1800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.2 seq 260 0x55d986d33200 osd_ping(ping e501 stamp 2018-07-11 07:52:10.575831) v4 > -2233> 2018-07-11 07:52:10.527841 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.2 10.8.128.106:0/12651 260 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:10.575831) v4 ==== 2004+0+0 (379520224 0 0) 0x55d9bdba5e00 con 0x55d986db3000 > -2232> 2018-07-11 07:52:10.527844 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.2 10.8.128.106:0/12651 260 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:10.575831) v4 ==== 2004+0+0 (379520224 0 0) 0x55d986d33200 con 0x55d986db1800 > -2231> 2018-07-11 07:52:10.527852 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:10.575831) v4 -- 0x55d9bcd72e00 con 0 > -2230> 2018-07-11 07:52:10.527879 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:10.575831) v4 -- 0x55d982cdee00 con 0 > -2229> 2018-07-11 07:52:10.844699 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 3028 0x55d986d97200 MOSDScrubReserve(11.3s5 REQUEST e501) v1 > -2228> 2018-07-11 07:52:10.844717 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 3028 ==== MOSDScrubReserve(11.3s5 REQUEST e501) v1 ==== 43+0+0 (3260170403 0 0) 0x55d986d97200 con 0x55d986c7e000 > -2227> 2018-07-11 07:52:10.844842 7ffab3be0700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e501) v1 -- 0x55d982cde600 con 0 > -2226> 2018-07-11 07:52:10.845060 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 3029 0x55d9bcd72e00 MOSDScrubReserve(11.3s5 RELEASE e501) v1 > -2225> 2018-07-11 07:52:10.845074 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 3029 ==== MOSDScrubReserve(11.3s5 RELEASE e501) v1 ==== 43+0+0 (223328929 0 0) 0x55d9bcd72e00 con 0x55d986c7e000 > -2224> 2018-07-11 07:52:10.923104 7ffab1bdc700 5 osd.10 501 heartbeat: osd_stat(789 GB used, 142 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,0,1,1]) > -2223> 2018-07-11 07:52:10.923119 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6802/15255 -- osd_ping(ping e501 stamp 2018-07-11 07:52:10.923117) v4 -- 0x55d986d97400 con 0 > -2222> 2018-07-11 07:52:10.923154 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6803/15255 -- osd_ping(ping e501 stamp 2018-07-11 07:52:10.923117) v4 -- 0x55d980f5b200 con 0 > -2221> 2018-07-11 07:52:10.923168 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6806/12677 -- osd_ping(ping e501 stamp 2018-07-11 07:52:10.923117) v4 -- 0x55d982cdf600 con 0 > -2220> 2018-07-11 07:52:10.923182 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6807/12677 -- osd_ping(ping e501 stamp 2018-07-11 07:52:10.923117) v4 -- 0x55d9bdba4c00 con 0 > -2219> 2018-07-11 07:52:10.923192 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6808/12651 -- osd_ping(ping e501 stamp 2018-07-11 07:52:10.923117) v4 -- 0x55d986d96600 con 0 > -2218> 2018-07-11 07:52:10.923217 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6810/12651 -- osd_ping(ping e501 stamp 2018-07-11 07:52:10.923117) v4 -- 0x55d986c23e00 con 0 > -2217> 2018-07-11 07:52:10.923228 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6809/12650 -- osd_ping(ping e501 stamp 2018-07-11 07:52:10.923117) v4 -- 0x55d9c9122a00 con 0 > -2216> 2018-07-11 07:52:10.923240 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6811/12650 -- osd_ping(ping e501 stamp 2018-07-11 07:52:10.923117) v4 -- 0x55d9b67d8400 con 0 > -2215> 2018-07-11 07:52:10.923265 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6810/15287 -- osd_ping(ping e501 stamp 2018-07-11 07:52:10.923117) v4 -- 0x55d9b67d9e00 con 0 > -2214> 2018-07-11 07:52:10.923276 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6811/15287 -- osd_ping(ping e501 stamp 2018-07-11 07:52:10.923117) v4 -- 0x55d986c84400 con 0 > -2213> 2018-07-11 07:52:10.923291 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6806/15263 -- osd_ping(ping e501 stamp 2018-07-11 07:52:10.923117) v4 -- 0x55d9b3db3c00 con 0 > -2212> 2018-07-11 07:52:10.923303 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6807/15263 -- osd_ping(ping e501 stamp 2018-07-11 07:52:10.923117) v4 -- 0x55d982d31c00 con 0 > -2211> 2018-07-11 07:52:10.923315 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.115:6802/13323 -- osd_ping(ping e501 stamp 2018-07-11 07:52:10.923117) v4 -- 0x55d986d4e200 con 0 > -2210> 2018-07-11 07:52:10.923327 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.115:6803/13323 -- osd_ping(ping e501 stamp 2018-07-11 07:52:10.923117) v4 -- 0x55d9869bfc00 con 0 > -2209> 2018-07-11 07:52:10.923343 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6806/15885 -- osd_ping(ping e501 stamp 2018-07-11 07:52:10.923117) v4 -- 0x55d9c3382000 con 0 > -2208> 2018-07-11 07:52:10.923355 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6807/15885 -- osd_ping(ping e501 stamp 2018-07-11 07:52:10.923117) v4 -- 0x55d9b67d8600 con 0 > -2207> 2018-07-11 07:52:10.923367 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.108:6802/11967 -- osd_ping(ping e501 stamp 2018-07-11 07:52:10.923117) v4 -- 0x55d986cd4c00 con 0 > -2206> 2018-07-11 07:52:10.923377 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.108:6803/11967 -- osd_ping(ping e501 stamp 2018-07-11 07:52:10.923117) v4 -- 0x55d9b437ac00 con 0 > -2205> 2018-07-11 07:52:10.923392 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.107:6806/12617 -- osd_ping(ping e501 stamp 2018-07-11 07:52:10.923117) v4 -- 0x55d9b3db3c00 con 0 > -2204> 2018-07-11 07:52:10.923404 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.107:6807/12617 -- osd_ping(ping e501 stamp 2018-07-11 07:52:10.923117) v4 -- 0x55d9c9122a00 con 0 > -2203> 2018-07-11 07:52:10.923416 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6802/15919 -- osd_ping(ping e501 stamp 2018-07-11 07:52:10.923117) v4 -- 0x55d982cdf600 con 0 > -2202> 2018-07-11 07:52:10.923428 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6803/15919 -- osd_ping(ping e501 stamp 2018-07-11 07:52:10.923117) v4 -- 0x55d9bdba5e00 con 0 > -2201> 2018-07-11 07:52:10.923609 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.107:6806/12617 conn(0x55d986bd2000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.9 seq 253 0x55d982d31600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:10.923117) v4 > -2200> 2018-07-11 07:52:10.923634 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.9 10.8.128.107:6806/12617 253 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:10.923117) v4 ==== 2004+0+0 (4198959997 0 0) 0x55d982d31600 con 0x55d986bd2000 > -2199> 2018-07-11 07:52:10.923672 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.107:6807/12617 conn(0x55d986bd3800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.9 seq 253 0x55d9c34cc600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:10.923117) v4 > -2198> 2018-07-11 07:52:10.923690 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.9 10.8.128.107:6807/12617 253 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:10.923117) v4 ==== 2004+0+0 (4198959997 0 0) 0x55d9c34cc600 con 0x55d986bd3800 > -2197> 2018-07-11 07:52:10.923743 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6808/12651 conn(0x55d986b9a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.2 seq 253 0x55d986c85000 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:10.923117) v4 > -2196> 2018-07-11 07:52:10.923740 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6809/12650 conn(0x55d986bcb000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 253 0x55d9c34cc600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:10.923117) v4 > -2195> 2018-07-11 07:52:10.923766 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.2 10.8.128.106:6808/12651 253 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:10.923117) v4 ==== 2004+0+0 (4198959997 0 0) 0x55d986c85000 con 0x55d986b9a000 > -2194> 2018-07-11 07:52:10.923754 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6803/15255 conn(0x55d986bc9800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.0 seq 253 0x55d982d31600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:10.923117) v4 > -2193> 2018-07-11 07:52:10.923775 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.3 10.8.128.106:6809/12650 253 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:10.923117) v4 ==== 2004+0+0 (4198959997 0 0) 0x55d9c34cc600 con 0x55d986bcb000 > -2192> 2018-07-11 07:52:10.923780 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.0 10.8.128.113:6803/15255 253 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:10.923117) v4 ==== 2004+0+0 (4198959997 0 0) 0x55d982d31600 con 0x55d986bc9800 > -2191> 2018-07-11 07:52:10.923812 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6802/15255 conn(0x55d986bc8000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.0 seq 253 0x55d986c85000 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:10.923117) v4 > -2190> 2018-07-11 07:52:10.923826 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.0 10.8.128.113:6802/15255 253 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:10.923117) v4 ==== 2004+0+0 (4198959997 0 0) 0x55d986c85000 con 0x55d986bc8000 > -2189> 2018-07-11 07:52:10.923841 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6806/15263 conn(0x55d986b9d000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.5 seq 253 0x55d9c34cc600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:10.923117) v4 > -2188> 2018-07-11 07:52:10.923861 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.5 10.8.128.113:6806/15263 253 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:10.923117) v4 ==== 2004+0+0 (4198959997 0 0) 0x55d9c34cc600 con 0x55d986b9d000 > -2187> 2018-07-11 07:52:10.923861 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6810/12651 conn(0x55d986b9b800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.2 seq 253 0x55d982d31600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:10.923117) v4 > -2186> 2018-07-11 07:52:10.923861 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6811/12650 conn(0x55d986bcc800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 253 0x55d986c85000 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:10.923117) v4 > -2185> 2018-07-11 07:52:10.923874 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.2 10.8.128.106:6810/12651 253 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:10.923117) v4 ==== 2004+0+0 (4198959997 0 0) 0x55d982d31600 con 0x55d986b9b800 > -2184> 2018-07-11 07:52:10.923878 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.3 10.8.128.106:6811/12650 253 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:10.923117) v4 ==== 2004+0+0 (4198959997 0 0) 0x55d986c85000 con 0x55d986bcc800 > -2183> 2018-07-11 07:52:10.923909 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6806/12677 conn(0x55d986b58000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.1 seq 253 0x55d9c34cc600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:10.923117) v4 > -2182> 2018-07-11 07:52:10.923925 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.1 10.8.128.106:6806/12677 253 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:10.923117) v4 ==== 2004+0+0 (4198959997 0 0) 0x55d9c34cc600 con 0x55d986b58000 > -2181> 2018-07-11 07:52:10.923927 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6807/15263 conn(0x55d986b9e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.5 seq 253 0x55d986c85000 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:10.923117) v4 > -2180> 2018-07-11 07:52:10.923911 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6811/15287 conn(0x55d986bb0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 253 0x55d982d31600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:10.923117) v4 > -2179> 2018-07-11 07:52:10.923938 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.5 10.8.128.113:6807/15263 253 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:10.923117) v4 ==== 2004+0+0 (4198959997 0 0) 0x55d986c85000 con 0x55d986b9e800 > -2178> 2018-07-11 07:52:10.923946 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.4 10.8.128.113:6811/15287 253 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:10.923117) v4 ==== 2004+0+0 (4198959997 0 0) 0x55d982d31600 con 0x55d986bb0800 > -2177> 2018-07-11 07:52:10.923974 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6807/12677 conn(0x55d986b59800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.1 seq 253 0x55d986c85000 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:10.923117) v4 > -2176> 2018-07-11 07:52:10.923978 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.115:6802/13323 conn(0x55d986bc1000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 253 0x55d982d31600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:10.923117) v4 > -2175> 2018-07-11 07:52:10.923982 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.115:6803/13323 conn(0x55d986bc2800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 253 0x55d9c34cc600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:10.923117) v4 > -2174> 2018-07-11 07:52:10.923987 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.1 10.8.128.106:6807/12677 253 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:10.923117) v4 ==== 2004+0+0 (4198959997 0 0) 0x55d986c85000 con 0x55d986b59800 > -2173> 2018-07-11 07:52:10.924000 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.6 10.8.128.115:6803/13323 253 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:10.923117) v4 ==== 2004+0+0 (4198959997 0 0) 0x55d9c34cc600 con 0x55d986bc2800 > -2172> 2018-07-11 07:52:10.923988 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.6 10.8.128.115:6802/13323 253 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:10.923117) v4 ==== 2004+0+0 (4198959997 0 0) 0x55d982d31600 con 0x55d986bc1000 > -2171> 2018-07-11 07:52:10.924028 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.108:6803/11967 conn(0x55d986bbf800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.8 seq 253 0x55d986c85000 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:10.923117) v4 > -2170> 2018-07-11 07:52:10.924030 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.108:6802/11967 conn(0x55d986bbe000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.8 seq 253 0x55d9c34cc600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:10.923117) v4 > -2169> 2018-07-11 07:52:10.924044 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.8 10.8.128.108:6803/11967 253 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:10.923117) v4 ==== 2004+0+0 (4198959997 0 0) 0x55d986c85000 con 0x55d986bbf800 > -2168> 2018-07-11 07:52:10.924037 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6802/15919 conn(0x55d986b5b000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.11 seq 253 0x55d982d31600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:10.923117) v4 > -2167> 2018-07-11 07:52:10.924051 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.8 10.8.128.108:6802/11967 253 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:10.923117) v4 ==== 2004+0+0 (4198959997 0 0) 0x55d9c34cc600 con 0x55d986bbe000 > -2166> 2018-07-11 07:52:10.924068 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.11 10.8.128.114:6802/15919 253 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:10.923117) v4 ==== 2004+0+0 (4198959997 0 0) 0x55d982d31600 con 0x55d986b5b000 > -2165> 2018-07-11 07:52:10.924075 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6810/15287 conn(0x55d986baf000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 253 0x55d986c85000 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:10.923117) v4 > -2164> 2018-07-11 07:52:10.924088 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.4 10.8.128.113:6810/15287 253 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:10.923117) v4 ==== 2004+0+0 (4198959997 0 0) 0x55d986c85000 con 0x55d986baf000 > -2163> 2018-07-11 07:52:10.924083 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6803/15919 conn(0x55d986b5c800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.11 seq 253 0x55d9c34cc600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:10.923117) v4 > -2162> 2018-07-11 07:52:10.924097 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.11 10.8.128.114:6803/15919 253 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:10.923117) v4 ==== 2004+0+0 (4198959997 0 0) 0x55d9c34cc600 con 0x55d986b5c800 > -2161> 2018-07-11 07:52:10.924105 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6806/15885 conn(0x55d986bac000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.7 seq 253 0x55d982d31600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:10.923117) v4 > -2160> 2018-07-11 07:52:10.924116 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.7 10.8.128.114:6806/15885 253 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:10.923117) v4 ==== 2004+0+0 (4198959997 0 0) 0x55d982d31600 con 0x55d986bac000 > -2159> 2018-07-11 07:52:10.924139 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6807/15885 conn(0x55d986bad800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.7 seq 253 0x55d9c34cc600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:10.923117) v4 > -2158> 2018-07-11 07:52:10.924152 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.7 10.8.128.114:6807/15885 253 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:10.923117) v4 ==== 2004+0+0 (4198959997 0 0) 0x55d9c34cc600 con 0x55d986bad800 > -2157> 2018-07-11 07:52:11.089443 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15287 conn(0x55d981240800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 275 0x55d986c85000 osd_ping(ping e501 stamp 2018-07-11 07:52:12.320979) v4 > -2156> 2018-07-11 07:52:11.089464 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.4 10.8.128.113:0/15287 275 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:12.320979) v4 ==== 2004+0+0 (1855150931 0 0) 0x55d986c85000 con 0x55d981240800 > -2155> 2018-07-11 07:52:11.089479 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:12.320979) v4 -- 0x55d9b437ac00 con 0 > -2154> 2018-07-11 07:52:11.089600 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15287 conn(0x55d98123f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 275 0x55d982d31600 osd_ping(ping e501 stamp 2018-07-11 07:52:12.320979) v4 > -2153> 2018-07-11 07:52:11.089619 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.4 10.8.128.113:0/15287 275 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:12.320979) v4 ==== 2004+0+0 (1855150931 0 0) 0x55d982d31600 con 0x55d98123f000 > -2152> 2018-07-11 07:52:11.089636 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:12.320979) v4 -- 0x55d982cdf600 con 0 > -2151> 2018-07-11 07:52:11.096395 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -2150> 2018-07-11 07:52:11.096424 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -2149> 2018-07-11 07:52:11.096472 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -2148> 2018-07-11 07:52:11.096933 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -2147> 2018-07-11 07:52:11.393558 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67606,epoch:501/498,start:5:75760585:::benchmark_data_magna115_10867_object1926395:0,end:5:75765843:::benchmark_data_magna115_11707_object163684:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb773680 con 0 > -2146> 2018-07-11 07:52:11.393595 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67606,epoch:501/498,start:5:75760585:::benchmark_data_magna115_10867_object1926395:0,end:5:75765843:::benchmark_data_magna115_11707_object163684:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb773400 con 0 > -2145> 2018-07-11 07:52:11.626022 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1032 0x55d9c795d200 rep_scrubmap(5.2e e501 from shard 4) v1 > -2144> 2018-07-11 07:52:11.626042 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1032 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12476 (3686815630 0 3178971223) 0x55d9c795d200 con 0x55d986bec000 > -2143> 2018-07-11 07:52:11.628226 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1038 0x55d9c795d440 rep_scrubmap(5.2e e501 from shard 2) v1 > -2142> 2018-07-11 07:52:11.628255 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1038 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12476 (1269249638 0 3178971223) 0x55d9c795d440 con 0x55d986bf0800 > -2141> 2018-07-11 07:52:11.628371 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -2140> 2018-07-11 07:52:11.628396 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -2139> 2018-07-11 07:52:11.628441 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -2138> 2018-07-11 07:52:11.628896 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -2137> 2018-07-11 07:52:11.643481 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75765843:::benchmark_data_magna115_11707_object163684:0,end:5:7576b8c1:::benchmark_data_magna115_10867_object3506577:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454000 con 0 > -2136> 2018-07-11 07:52:11.643518 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75765843:::benchmark_data_magna115_11707_object163684:0,end:5:7576b8c1:::benchmark_data_magna115_10867_object3506577:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454280 con 0 > -2135> 2018-07-11 07:52:11.713466 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 33] Generated table #36810: 2638241 keys, 74546373 bytes > -2134> 2018-07-11 07:52:11.713498 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295531713484, "cf_name": "default", "job": 33, "event": "table_file_creation", "file_number": 36810, "file_size": 74546373, "table_properties": {"data_size": 67112143, "index_size": 690057, "filter_size": 6743183, "raw_key_size": 76215847, "raw_average_key_size": 28, "raw_value_size": 31365750, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} > -2133> 2018-07-11 07:52:11.849288 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1033 0x55d9c795d680 rep_scrubmap(5.2e e501 from shard 4) v1 > -2132> 2018-07-11 07:52:11.849309 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1033 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12476 (3686815630 0 2631146752) 0x55d9c795d680 con 0x55d986bec000 > -2131> 2018-07-11 07:52:11.854619 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1039 0x55d9c795d8c0 rep_scrubmap(5.2e e501 from shard 2) v1 > -2130> 2018-07-11 07:52:11.854638 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1039 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12476 (1269249638 0 2631146752) 0x55d9c795d8c0 con 0x55d986bf0800 > -2129> 2018-07-11 07:52:11.916454 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -2128> 2018-07-11 07:52:11.916481 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -2127> 2018-07-11 07:52:11.916524 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -2126> 2018-07-11 07:52:11.916981 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -2125> 2018-07-11 07:52:11.956329 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67926,epoch:501/498,start:5:7576b8c1:::benchmark_data_magna115_10867_object3506577:0,end:5:7577335f:::benchmark_data_magna115_11707_object186754:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454c80 con 0 > -2124> 2018-07-11 07:52:11.956362 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67926,epoch:501/498,start:5:7576b8c1:::benchmark_data_magna115_10867_object3506577:0,end:5:7577335f:::benchmark_data_magna115_11707_object186754:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb773b80 con 0 > -2123> 2018-07-11 07:52:12.106716 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.115:0/13323 conn(0x55d986d7e000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 263 0x55d9c34cc600 osd_ping(ping e501 stamp 2018-07-11 07:52:12.452781) v4 > -2122> 2018-07-11 07:52:12.106737 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.6 10.8.128.115:0/13323 263 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:12.452781) v4 ==== 2004+0+0 (552719583 0 0) 0x55d9c34cc600 con 0x55d986d7e000 > -2121> 2018-07-11 07:52:12.106747 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:12.452781) v4 -- 0x55d9bdba5e00 con 0 > -2120> 2018-07-11 07:52:12.106824 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.115:0/13323 conn(0x55d986d24800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 263 0x55d986d33200 osd_ping(ping e501 stamp 2018-07-11 07:52:12.452781) v4 > -2119> 2018-07-11 07:52:12.106849 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.6 10.8.128.115:0/13323 263 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:12.452781) v4 ==== 2004+0+0 (552719583 0 0) 0x55d986d33200 con 0x55d986d24800 > -2118> 2018-07-11 07:52:12.106861 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:12.452781) v4 -- 0x55d9b437ac00 con 0 > -2117> 2018-07-11 07:52:12.176138 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1034 0x55d9c795db00 rep_scrubmap(5.2e e501 from shard 4) v1 > -2116> 2018-07-11 07:52:12.176158 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1034 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12468 (3686815630 0 337803364) 0x55d9c795db00 con 0x55d986bec000 > -2115> 2018-07-11 07:52:12.193479 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1040 0x55d9c795dd40 rep_scrubmap(5.2e e501 from shard 2) v1 > -2114> 2018-07-11 07:52:12.193499 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1040 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12468 (1269249638 0 337803364) 0x55d9c795dd40 con 0x55d986bf0800 > -2113> 2018-07-11 07:52:12.262587 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -2112> 2018-07-11 07:52:12.262612 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -2111> 2018-07-11 07:52:12.262667 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -2110> 2018-07-11 07:52:12.263112 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -2109> 2018-07-11 07:52:12.297573 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7577335f:::benchmark_data_magna115_11707_object186754:0,end:5:7577822a:::benchmark_data_magna115_10867_object1256055:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454a00 con 0 > -2108> 2018-07-11 07:52:12.297613 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7577335f:::benchmark_data_magna115_11707_object186754:0,end:5:7577822a:::benchmark_data_magna115_10867_object1256055:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455180 con 0 > -2107> 2018-07-11 07:52:12.302348 7ffab7be8700 10 monclient: tick > -2106> 2018-07-11 07:52:12.302363 7ffab7be8700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2018-07-11 07:51:42.302361) > -2105> 2018-07-11 07:52:12.352622 7ffab6be6700 1 -- 10.8.128.107:6800/13927 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55d985053b80 con 0 > -2104> 2018-07-11 07:52:12.352695 7ffab6be6700 1 -- 10.8.128.107:6800/13927 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55d9b3fde000 con 0 > -2103> 2018-07-11 07:52:12.499804 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1041 0x55d99c796000 rep_scrubmap(5.2e e501 from shard 2) v1 > -2102> 2018-07-11 07:52:12.499823 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1041 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12478 (1269249638 0 1195787259) 0x55d99c796000 con 0x55d986bf0800 > -2101> 2018-07-11 07:52:12.512012 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1035 0x55d99c796240 rep_scrubmap(5.2e e501 from shard 4) v1 > -2100> 2018-07-11 07:52:12.512031 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1035 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12478 (3686815630 0 1195787259) 0x55d99c796240 con 0x55d986bec000 > -2099> 2018-07-11 07:52:12.555332 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.107:0/12617 conn(0x55d98123c000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 253 0x55d986c85000 osd_ping(ping e501 stamp 2018-07-11 07:52:12.554831) v4 > -2098> 2018-07-11 07:52:12.555354 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.9 10.8.128.107:0/12617 253 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:12.554831) v4 ==== 2004+0+0 (175816241 0 0) 0x55d986c85000 con 0x55d98123c000 > -2097> 2018-07-11 07:52:12.555363 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.107:0/12617 conn(0x55d98123d800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 253 0x55d982cde600 osd_ping(ping e501 stamp 2018-07-11 07:52:12.554831) v4 > -2096> 2018-07-11 07:52:12.555372 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:12.554831) v4 -- 0x55d9b437ac00 con 0 > -2095> 2018-07-11 07:52:12.555388 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.9 10.8.128.107:0/12617 253 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:12.554831) v4 ==== 2004+0+0 (175816241 0 0) 0x55d982cde600 con 0x55d98123d800 > -2094> 2018-07-11 07:52:12.555415 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:12.554831) v4 -- 0x55d9bdba5e00 con 0 > -2093> 2018-07-11 07:52:12.557894 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -2092> 2018-07-11 07:52:12.557919 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -2091> 2018-07-11 07:52:12.557965 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -2090> 2018-07-11 07:52:12.558433 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -2089> 2018-07-11 07:52:12.569237 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6805/15885 conn(0x55d986c7f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.7 seq 2820 0x55d986d33200 MOSDScrubReserve(5.4 REQUEST e501) v1 > -2088> 2018-07-11 07:52:12.569261 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.7 10.8.128.114:6805/15885 2820 ==== MOSDScrubReserve(5.4 REQUEST e501) v1 ==== 43+0+0 (1753221512 0 0) 0x55d986d33200 con 0x55d986c7f800 > -2087> 2018-07-11 07:52:12.569295 7ffab33df700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.114:6805/15885 -- MOSDScrubReserve(5.4 REJECT e501) v1 -- 0x55d9829e5600 con 0 > -2086> 2018-07-11 07:52:12.569618 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6805/15885 conn(0x55d986c7f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.7 seq 2821 0x55d9829e5600 MOSDScrubReserve(5.4 RELEASE e501) v1 > -2085> 2018-07-11 07:52:12.569633 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.7 10.8.128.114:6805/15885 2821 ==== MOSDScrubReserve(5.4 RELEASE e501) v1 ==== 43+0+0 (2812146570 0 0) 0x55d9829e5600 con 0x55d986c7f800 > -2084> 2018-07-11 07:52:12.597893 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.114:0/15919 conn(0x55d986df3800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.11 seq 265 0x55d9c34cc600 osd_ping(ping e501 stamp 2018-07-11 07:52:13.228322) v4 > -2083> 2018-07-11 07:52:12.597893 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.114:0/15919 conn(0x55d986df5000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.11 seq 265 0x55d9869d9800 osd_ping(ping e501 stamp 2018-07-11 07:52:13.228322) v4 > -2082> 2018-07-11 07:52:12.597912 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.11 10.8.128.114:0/15919 265 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:13.228322) v4 ==== 2004+0+0 (1115049817 0 0) 0x55d9c34cc600 con 0x55d986df3800 > -2081> 2018-07-11 07:52:12.597923 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:13.228322) v4 -- 0x55d9bdba5e00 con 0 > -2080> 2018-07-11 07:52:12.597922 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.11 10.8.128.114:0/15919 265 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:13.228322) v4 ==== 2004+0+0 (1115049817 0 0) 0x55d9869d9800 con 0x55d986df5000 > -2079> 2018-07-11 07:52:12.597945 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:13.228322) v4 -- 0x55d982cdf600 con 0 > -2078> 2018-07-11 07:52:12.607190 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7577822a:::benchmark_data_magna115_10867_object1256055:0,end:5:7577d608:::benchmark_data_magna115_10867_object1169566:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454500 con 0 > -2077> 2018-07-11 07:52:12.607238 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7577822a:::benchmark_data_magna115_10867_object1256055:0,end:5:7577d608:::benchmark_data_magna115_10867_object1169566:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454f00 con 0 > -2076> 2018-07-11 07:52:12.722794 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.114:0/15885 conn(0x55d986d7f800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.7 seq 268 0x55d982d31600 osd_ping(ping e501 stamp 2018-07-11 07:52:13.353270) v4 > -2075> 2018-07-11 07:52:12.722820 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.7 10.8.128.114:0/15885 268 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:13.353270) v4 ==== 2004+0+0 (2055556302 0 0) 0x55d982d31600 con 0x55d986d7f800 > -2074> 2018-07-11 07:52:12.722834 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:13.353270) v4 -- 0x55d982cdf600 con 0 > -2073> 2018-07-11 07:52:12.722825 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.114:0/15885 conn(0x55d986d81000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.7 seq 268 0x55d9b437ac00 osd_ping(ping e501 stamp 2018-07-11 07:52:13.353270) v4 > -2072> 2018-07-11 07:52:12.722863 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.7 10.8.128.114:0/15885 268 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:13.353270) v4 ==== 2004+0+0 (2055556302 0 0) 0x55d9b437ac00 con 0x55d986d81000 > -2071> 2018-07-11 07:52:12.722876 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:13.353270) v4 -- 0x55d982d31c00 con 0 > -2070> 2018-07-11 07:52:12.789967 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15287 conn(0x55d98123f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 276 0x55d9869d9800 osd_ping(ping e501 stamp 2018-07-11 07:52:14.021569) v4 > -2069> 2018-07-11 07:52:12.789961 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15287 conn(0x55d981240800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 276 0x55d986c85000 osd_ping(ping e501 stamp 2018-07-11 07:52:14.021569) v4 > -2068> 2018-07-11 07:52:12.789991 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.4 10.8.128.113:0/15287 276 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:14.021569) v4 ==== 2004+0+0 (2291899850 0 0) 0x55d9869d9800 con 0x55d98123f000 > -2067> 2018-07-11 07:52:12.789994 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.4 10.8.128.113:0/15287 276 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:14.021569) v4 ==== 2004+0+0 (2291899850 0 0) 0x55d986c85000 con 0x55d981240800 > -2066> 2018-07-11 07:52:12.790010 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:14.021569) v4 -- 0x55d982cdf600 con 0 > -2065> 2018-07-11 07:52:12.790038 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:14.021569) v4 -- 0x55d982d31c00 con 0 > -2064> 2018-07-11 07:52:12.824409 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1042 0x55d99c796480 rep_scrubmap(5.2e e501 from shard 2) v1 > -2063> 2018-07-11 07:52:12.824429 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1042 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12480 (1269249638 0 2799125529) 0x55d99c796480 con 0x55d986bf0800 > -2062> 2018-07-11 07:52:12.828158 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12651 conn(0x55d986db1800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.2 seq 261 0x55d986c85000 osd_ping(ping e501 stamp 2018-07-11 07:52:12.876277) v4 > -2061> 2018-07-11 07:52:12.828184 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.2 10.8.128.106:0/12651 261 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:12.876277) v4 ==== 2004+0+0 (505647421 0 0) 0x55d986c85000 con 0x55d986db1800 > -2060> 2018-07-11 07:52:12.828177 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12651 conn(0x55d986db3000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.2 seq 261 0x55d982cde600 osd_ping(ping e501 stamp 2018-07-11 07:52:12.876277) v4 > -2059> 2018-07-11 07:52:12.828204 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:12.876277) v4 -- 0x55d982d31c00 con 0 > -2058> 2018-07-11 07:52:12.828210 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.2 10.8.128.106:0/12651 261 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:12.876277) v4 ==== 2004+0+0 (505647421 0 0) 0x55d982cde600 con 0x55d986db3000 > -2057> 2018-07-11 07:52:12.828227 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:12.876277) v4 -- 0x55d9bdba5e00 con 0 > -2056> 2018-07-11 07:52:12.840176 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1036 0x55d99c7966c0 rep_scrubmap(5.2e e501 from shard 4) v1 > -2055> 2018-07-11 07:52:12.840197 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1036 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12480 (3686815630 0 2799125529) 0x55d99c7966c0 con 0x55d986bec000 > -2054> 2018-07-11 07:52:12.911329 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -2053> 2018-07-11 07:52:12.911354 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -2052> 2018-07-11 07:52:12.911397 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -2051> 2018-07-11 07:52:12.911862 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -2050> 2018-07-11 07:52:12.966023 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7577d608:::benchmark_data_magna115_10867_object1169566:0,end:5:75783285:::benchmark_data_magna115_10867_object3729537:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455400 con 0 > -2049> 2018-07-11 07:52:12.966061 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7577d608:::benchmark_data_magna115_10867_object1169566:0,end:5:75783285:::benchmark_data_magna115_10867_object3729537:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498280 con 0 > -2048> 2018-07-11 07:52:12.966966 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6801/15919 conn(0x55d986c3f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=29 cs=1 l=0). rx osd.11 seq 250 0x55d9c34cc600 MOSDScrubReserve(10.f REQUEST e501) v1 > -2047> 2018-07-11 07:52:12.966986 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.11 10.8.128.114:6801/15919 250 ==== MOSDScrubReserve(10.f REQUEST e501) v1 ==== 43+0+0 (2892966580 0 0) 0x55d9c34cc600 con 0x55d986c3f800 > -2046> 2018-07-11 07:52:12.967022 7ffab53e3700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.114:6801/15919 -- MOSDScrubReserve(10.f REJECT e501) v1 -- 0x55d9b4388200 con 0 > -2045> 2018-07-11 07:52:12.967348 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6801/15919 conn(0x55d986c3f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=29 cs=1 l=0). rx osd.11 seq 251 0x55d9b4388200 MOSDScrubReserve(10.f RELEASE e501) v1 > -2044> 2018-07-11 07:52:12.967363 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.11 10.8.128.114:6801/15919 251 ==== MOSDScrubReserve(10.f RELEASE e501) v1 ==== 43+0+0 (1668468918 0 0) 0x55d9b4388200 con 0x55d986c3f800 > -2043> 2018-07-11 07:52:13.121377 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12650 conn(0x55d986d23000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 255 0x55d982d31600 osd_ping(ping e501 stamp 2018-07-11 07:52:13.169508) v4 > -2042> 2018-07-11 07:52:13.121402 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.3 10.8.128.106:0/12650 255 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:13.169508) v4 ==== 2004+0+0 (629561145 0 0) 0x55d982d31600 con 0x55d986d23000 > -2041> 2018-07-11 07:52:13.121414 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:13.169508) v4 -- 0x55d982cdf600 con 0 > -2040> 2018-07-11 07:52:13.121441 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12650 conn(0x55d986d21800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 255 0x55d982cde600 osd_ping(ping e501 stamp 2018-07-11 07:52:13.169508) v4 > -2039> 2018-07-11 07:52:13.121462 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.3 10.8.128.106:0/12650 255 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:13.169508) v4 ==== 2004+0+0 (629561145 0 0) 0x55d982cde600 con 0x55d986d21800 > -2038> 2018-07-11 07:52:13.121478 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:13.169508) v4 -- 0x55d9c9122a00 con 0 > -2037> 2018-07-11 07:52:13.165952 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1037 0x55d99c796900 rep_scrubmap(5.2e e501 from shard 4) v1 > -2036> 2018-07-11 07:52:13.165971 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1037 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12476 (3686815630 0 2115330723) 0x55d99c796900 con 0x55d986bec000 > -2035> 2018-07-11 07:52:13.178483 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1043 0x55d99c796b40 rep_scrubmap(5.2e e501 from shard 2) v1 > -2034> 2018-07-11 07:52:13.178502 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1043 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12476 (1269249638 0 2115330723) 0x55d99c796b40 con 0x55d986bf0800 > -2033> 2018-07-11 07:52:13.213581 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -2032> 2018-07-11 07:52:13.213607 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -2031> 2018-07-11 07:52:13.213652 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -2030> 2018-07-11 07:52:13.214114 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -2029> 2018-07-11 07:52:13.215186 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75783285:::benchmark_data_magna115_10867_object3729537:0,end:5:757882ad:::benchmark_data_magna115_10867_object563927:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454780 con 0 > -2028> 2018-07-11 07:52:13.215233 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75783285:::benchmark_data_magna115_10867_object3729537:0,end:5:757882ad:::benchmark_data_magna115_10867_object563927:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498000 con 0 > -2027> 2018-07-11 07:52:13.441986 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1044 0x55d99c796d80 rep_scrubmap(5.2e e501 from shard 2) v1 > -2026> 2018-07-11 07:52:13.442008 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1044 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12476 (1269249638 0 223534779) 0x55d99c796d80 con 0x55d986bf0800 > -2025> 2018-07-11 07:52:13.481096 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1038 0x55d99c796fc0 rep_scrubmap(5.2e e501 from shard 4) v1 > -2024> 2018-07-11 07:52:13.481116 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1038 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12476 (3686815630 0 223534779) 0x55d99c796fc0 con 0x55d986bec000 > -2023> 2018-07-11 07:52:13.522190 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -2022> 2018-07-11 07:52:13.522214 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -2021> 2018-07-11 07:52:13.522281 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -2020> 2018-07-11 07:52:13.522744 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -2019> 2018-07-11 07:52:13.552123 7ffacbbd1700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15263 conn(0x55d985d44000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.5 seq 245 0x55d9bdba5e00 osd_ping(ping e501 stamp 2018-07-11 07:52:14.783764) v4 > -2018> 2018-07-11 07:52:13.552131 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15263 conn(0x55d985d45800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.5 seq 245 0x55d9869d9800 osd_ping(ping e501 stamp 2018-07-11 07:52:14.783764) v4 > -2017> 2018-07-11 07:52:13.552145 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 <== osd.5 10.8.128.113:0/15263 245 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:14.783764) v4 ==== 2004+0+0 (3366173745 0 0) 0x55d9bdba5e00 con 0x55d985d44000 > -2016> 2018-07-11 07:52:13.552149 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.5 10.8.128.113:0/15263 245 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:14.783764) v4 ==== 2004+0+0 (3366173745 0 0) 0x55d9869d9800 con 0x55d985d45800 > -2015> 2018-07-11 07:52:13.552155 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:14.783764) v4 -- 0x55d9c9122a00 con 0 > -2014> 2018-07-11 07:52:13.552179 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:14.783764) v4 -- 0x55d982cdf600 con 0 > -2013> 2018-07-11 07:52:13.566568 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:757882ad:::benchmark_data_magna115_10867_object563927:0,end:5:7578ba87:::benchmark_data_magna115_10867_object3054607:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455b80 con 0 > -2012> 2018-07-11 07:52:13.566619 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:757882ad:::benchmark_data_magna115_10867_object563927:0,end:5:7578ba87:::benchmark_data_magna115_10867_object3054607:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b14f0500 con 0 > -2011> 2018-07-11 07:52:13.772002 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1045 0x55d99c797200 rep_scrubmap(5.2e e501 from shard 2) v1 > -2010> 2018-07-11 07:52:13.772021 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1045 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12472 (1269249638 0 313788982) 0x55d99c797200 con 0x55d986bf0800 > -2009> 2018-07-11 07:52:13.793455 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1039 0x55d99c797440 rep_scrubmap(5.2e e501 from shard 4) v1 > -2008> 2018-07-11 07:52:13.793473 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1039 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12472 (3686815630 0 313788982) 0x55d99c797440 con 0x55d986bec000 > -2007> 2018-07-11 07:52:13.793627 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -2006> 2018-07-11 07:52:13.793653 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -2005> 2018-07-11 07:52:13.793700 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -2004> 2018-07-11 07:52:13.794162 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -2003> 2018-07-11 07:52:13.806287 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67177,epoch:501/498,start:5:7578ba87:::benchmark_data_magna115_10867_object3054607:0,end:5:75792abc:::benchmark_data_magna115_10867_object2586811:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455900 con 0 > -2002> 2018-07-11 07:52:13.806325 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67177,epoch:501/498,start:5:7578ba87:::benchmark_data_magna115_10867_object3054607:0,end:5:75792abc:::benchmark_data_magna115_10867_object2586811:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455680 con 0 > -2001> 2018-07-11 07:52:13.845005 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 3030 0x55d982cde600 MOSDScrubReserve(11.3s5 REQUEST e501) v1 > -2000> 2018-07-11 07:52:13.845027 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 3030 ==== MOSDScrubReserve(11.3s5 REQUEST e501) v1 ==== 43+0+0 (3260170403 0 0) 0x55d982cde600 con 0x55d986c7e000 > -1999> 2018-07-11 07:52:13.845097 7ffab3be0700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e501) v1 -- 0x55d9bcd72e00 con 0 > -1998> 2018-07-11 07:52:13.845460 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 3031 0x55d9bcd72e00 MOSDScrubReserve(11.3s5 RELEASE e501) v1 > -1997> 2018-07-11 07:52:13.845478 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 3031 ==== MOSDScrubReserve(11.3s5 RELEASE e501) v1 ==== 43+0+0 (223328929 0 0) 0x55d9bcd72e00 con 0x55d986c7e000 > -1996> 2018-07-11 07:52:13.890501 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15287 conn(0x55d98123f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 277 0x55d982d31600 osd_ping(ping e501 stamp 2018-07-11 07:52:15.122041) v4 > -1995> 2018-07-11 07:52:13.890524 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.4 10.8.128.113:0/15287 277 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:15.122041) v4 ==== 2004+0+0 (983213890 0 0) 0x55d982d31600 con 0x55d98123f000 > -1994> 2018-07-11 07:52:13.890521 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15287 conn(0x55d981240800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 277 0x55d986c85000 osd_ping(ping e501 stamp 2018-07-11 07:52:15.122041) v4 > -1993> 2018-07-11 07:52:13.890546 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:15.122041) v4 -- 0x55d982cdf600 con 0 > -1992> 2018-07-11 07:52:13.890550 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.4 10.8.128.113:0/15287 277 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:15.122041) v4 ==== 2004+0+0 (983213890 0 0) 0x55d986c85000 con 0x55d981240800 > -1991> 2018-07-11 07:52:13.890564 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:15.122041) v4 -- 0x55d982d31c00 con 0 > -1990> 2018-07-11 07:52:13.966019 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6805/15263 conn(0x55d986c41000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=31 cs=1 l=0). rx osd.5 seq 2842 0x55d9b437ac00 MOSDScrubReserve(6.15 REQUEST e501) v1 > -1989> 2018-07-11 07:52:13.966038 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.5 10.8.128.113:6805/15263 2842 ==== MOSDScrubReserve(6.15 REQUEST e501) v1 ==== 43+0+0 (3942742026 0 0) 0x55d9b437ac00 con 0x55d986c41000 > -1988> 2018-07-11 07:52:14.028090 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1046 0x55d99c797680 rep_scrubmap(5.2e e501 from shard 2) v1 > -1987> 2018-07-11 07:52:14.028110 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1046 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12478 (1269249638 0 2468536129) 0x55d99c797680 con 0x55d986bf0800 > -1986> 2018-07-11 07:52:14.048764 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1040 0x55d99c7978c0 rep_scrubmap(5.2e e501 from shard 4) v1 > -1985> 2018-07-11 07:52:14.048782 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1040 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12478 (3686815630 0 2468536129) 0x55d99c7978c0 con 0x55d986bec000 > -1984> 2018-07-11 07:52:14.072243 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15255 conn(0x55d986c82800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.0 seq 263 0x55d9869d9800 osd_ping(ping e501 stamp 2018-07-11 07:52:15.303923) v4 > -1983> 2018-07-11 07:52:14.072276 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.0 10.8.128.113:0/15255 263 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:15.303923) v4 ==== 2004+0+0 (105787732 0 0) 0x55d9869d9800 con 0x55d986c82800 > -1982> 2018-07-11 07:52:14.072288 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:15.303923) v4 -- 0x55d982cdf600 con 0 > -1981> 2018-07-11 07:52:14.072317 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15255 conn(0x55d986d20000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.0 seq 263 0x55d986c85000 osd_ping(ping e501 stamp 2018-07-11 07:52:15.303923) v4 > -1980> 2018-07-11 07:52:14.072338 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.0 10.8.128.113:0/15255 263 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:15.303923) v4 ==== 2004+0+0 (105787732 0 0) 0x55d986c85000 con 0x55d986d20000 > -1979> 2018-07-11 07:52:14.072354 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:15.303923) v4 -- 0x55d9b67d9e00 con 0 > -1978> 2018-07-11 07:52:14.142936 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6805/15263 -- MOSDScrubReserve(6.15 REJECT e501) v1 -- 0x55d986c22e00 con 0 > -1977> 2018-07-11 07:52:14.143140 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -1976> 2018-07-11 07:52:14.143163 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -1975> 2018-07-11 07:52:14.143209 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -1974> 2018-07-11 07:52:14.143401 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6805/15263 conn(0x55d986c41000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=31 cs=1 l=0). rx osd.5 seq 2843 0x55d986c22e00 MOSDScrubReserve(6.15 RELEASE e501) v1 > -1973> 2018-07-11 07:52:14.143419 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.5 10.8.128.113:6805/15263 2843 ==== MOSDScrubReserve(6.15 RELEASE e501) v1 ==== 43+0+0 (605848072 0 0) 0x55d986c22e00 con 0x55d986c41000 > -1972> 2018-07-11 07:52:14.143675 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -1971> 2018-07-11 07:52:14.158438 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67723,epoch:501/498,start:5:75792abc:::benchmark_data_magna115_10867_object2586811:0,end:5:75798ee2:::benchmark_data_magna115_10867_object95390:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498780 con 0 > -1970> 2018-07-11 07:52:14.158476 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67723,epoch:501/498,start:5:75792abc:::benchmark_data_magna115_10867_object2586811:0,end:5:75798ee2:::benchmark_data_magna115_10867_object95390:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98696e000 con 0 > -1969> 2018-07-11 07:52:14.255818 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.107:0/12617 conn(0x55d98123c000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 254 0x55d986c85000 osd_ping(ping e501 stamp 2018-07-11 07:52:14.255493) v4 > -1968> 2018-07-11 07:52:14.255844 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.9 10.8.128.107:0/12617 254 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:14.255493) v4 ==== 2004+0+0 (3739596140 0 0) 0x55d986c85000 con 0x55d98123c000 > -1967> 2018-07-11 07:52:14.255848 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.107:0/12617 conn(0x55d98123d800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 254 0x55d9bdba5e00 osd_ping(ping e501 stamp 2018-07-11 07:52:14.255493) v4 > -1966> 2018-07-11 07:52:14.255855 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:14.255493) v4 -- 0x55d9b67d9e00 con 0 > -1965> 2018-07-11 07:52:14.255868 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.9 10.8.128.107:0/12617 254 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:14.255493) v4 ==== 2004+0+0 (3739596140 0 0) 0x55d9bdba5e00 con 0x55d98123d800 > -1964> 2018-07-11 07:52:14.255890 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:14.255493) v4 -- 0x55d986cd4c00 con 0 > -1963> 2018-07-11 07:52:14.361283 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1041 0x55d99c797b00 rep_scrubmap(5.2e e501 from shard 4) v1 > -1962> 2018-07-11 07:52:14.361299 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1041 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12472 (3686815630 0 690701238) 0x55d99c797b00 con 0x55d986bec000 > -1961> 2018-07-11 07:52:14.383005 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1047 0x55d99c797d40 rep_scrubmap(5.2e e501 from shard 2) v1 > -1960> 2018-07-11 07:52:14.383024 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1047 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12472 (1269249638 0 690701238) 0x55d99c797d40 con 0x55d986bf0800 > -1959> 2018-07-11 07:52:14.490533 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -1958> 2018-07-11 07:52:14.490556 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -1957> 2018-07-11 07:52:14.490614 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -1956> 2018-07-11 07:52:14.491071 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -1955> 2018-07-11 07:52:14.531220 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75798ee2:::benchmark_data_magna115_10867_object95390:0,end:5:7579e353:::benchmark_data_magna115_10867_object865685:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98696f180 con 0 > -1954> 2018-07-11 07:52:14.531278 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75798ee2:::benchmark_data_magna115_10867_object95390:0,end:5:7579e353:::benchmark_data_magna115_10867_object865685:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498a00 con 0 > -1953> 2018-07-11 07:52:14.569620 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6805/15885 conn(0x55d986c7f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.7 seq 2822 0x55d986c85000 MOSDScrubReserve(5.4 REQUEST e501) v1 > -1952> 2018-07-11 07:52:14.569640 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.7 10.8.128.114:6805/15885 2822 ==== MOSDScrubReserve(5.4 REQUEST e501) v1 ==== 43+0+0 (1753221512 0 0) 0x55d986c85000 con 0x55d986c7f800 > -1951> 2018-07-11 07:52:14.569678 7ffab33df700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.114:6805/15885 -- MOSDScrubReserve(5.4 REJECT e501) v1 -- 0x55d9829e5600 con 0 > -1950> 2018-07-11 07:52:14.569982 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6805/15885 conn(0x55d986c7f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.7 seq 2823 0x55d9829e5600 MOSDScrubReserve(5.4 RELEASE e501) v1 > -1949> 2018-07-11 07:52:14.569997 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.7 10.8.128.114:6805/15885 2823 ==== MOSDScrubReserve(5.4 RELEASE e501) v1 ==== 43+0+0 (2812146570 0 0) 0x55d9829e5600 con 0x55d986c7f800 > -1948> 2018-07-11 07:52:14.774474 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1042 0x55d9af004000 rep_scrubmap(5.2e e501 from shard 4) v1 > -1947> 2018-07-11 07:52:14.774496 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1042 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12466 (3686815630 0 1309381189) 0x55d9af004000 con 0x55d986bec000 > -1946> 2018-07-11 07:52:14.780873 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1048 0x55d9af004240 rep_scrubmap(5.2e e501 from shard 2) v1 > -1945> 2018-07-11 07:52:14.780894 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1048 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12466 (1269249638 0 1309381189) 0x55d9af004240 con 0x55d986bf0800 > -1944> 2018-07-11 07:52:14.822834 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -1943> 2018-07-11 07:52:14.822859 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -1942> 2018-07-11 07:52:14.822918 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -1941> 2018-07-11 07:52:14.823411 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -1940> 2018-07-11 07:52:14.875603 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7579e353:::benchmark_data_magna115_10867_object865685:0,end:5:757a439f:::benchmark_data_magna115_10867_object644146:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98696f900 con 0 > -1939> 2018-07-11 07:52:14.875655 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7579e353:::benchmark_data_magna115_10867_object865685:0,end:5:757a439f:::benchmark_data_magna115_10867_object644146:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98696f680 con 0 > -1938> 2018-07-11 07:52:15.094516 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.108:0/11967 conn(0x55d986db0000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.8 seq 246 0x55d982d31600 osd_ping(ping e501 stamp 2018-07-11 07:52:14.850376) v4 > -1937> 2018-07-11 07:52:15.094540 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.8 10.8.128.108:0/11967 246 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:14.850376) v4 ==== 2004+0+0 (492534961 0 0) 0x55d982d31600 con 0x55d986db0000 > -1936> 2018-07-11 07:52:15.094553 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:14.850376) v4 -- 0x55d982cdf600 con 0 > -1935> 2018-07-11 07:52:15.094582 7ffacbbd1700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.108:0/11967 conn(0x55d986d82800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.8 seq 246 0x55d9c9122a00 osd_ping(ping e501 stamp 2018-07-11 07:52:14.850376) v4 > -1934> 2018-07-11 07:52:15.094602 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 <== osd.8 10.8.128.108:0/11967 246 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:14.850376) v4 ==== 2004+0+0 (492534961 0 0) 0x55d9c9122a00 con 0x55d986d82800 > -1933> 2018-07-11 07:52:15.094613 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:14.850376) v4 -- 0x55d986cd4c00 con 0 > -1932> 2018-07-11 07:52:15.095145 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1049 0x55d9af004480 rep_scrubmap(5.2e e501 from shard 2) v1 > -1931> 2018-07-11 07:52:15.095164 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1049 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12478 (1269249638 0 2903752251) 0x55d9af004480 con 0x55d986bf0800 > -1930> 2018-07-11 07:52:15.115243 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1043 0x55d9af0046c0 rep_scrubmap(5.2e e501 from shard 4) v1 > -1929> 2018-07-11 07:52:15.115275 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1043 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12478 (3686815630 0 2903752251) 0x55d9af0046c0 con 0x55d986bec000 > -1928> 2018-07-11 07:52:15.151647 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -1927> 2018-07-11 07:52:15.151671 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -1926> 2018-07-11 07:52:15.151713 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -1925> 2018-07-11 07:52:15.152161 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -1924> 2018-07-11 07:52:15.191600 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67241,epoch:501/498,start:5:757a439f:::benchmark_data_magna115_10867_object644146:0,end:5:757abdbd:::benchmark_data_magna115_10867_object2971928:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98696ec80 con 0 > -1923> 2018-07-11 07:52:15.191650 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67241,epoch:501/498,start:5:757a439f:::benchmark_data_magna115_10867_object644146:0,end:5:757abdbd:::benchmark_data_magna115_10867_object2971928:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98696ef00 con 0 > -1922> 2018-07-11 07:52:15.269502 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12677 conn(0x55d986db4800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.1 seq 267 0x55d9869d9800 osd_ping(ping e501 stamp 2018-07-11 07:52:15.317593) v4 > -1921> 2018-07-11 07:52:15.269509 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12677 conn(0x55d986df2000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.1 seq 267 0x55d982d31c00 osd_ping(ping e501 stamp 2018-07-11 07:52:15.317593) v4 > -1920> 2018-07-11 07:52:15.269527 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.1 10.8.128.106:0/12677 267 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:15.317593) v4 ==== 2004+0+0 (3739037815 0 0) 0x55d982d31c00 con 0x55d986df2000 > -1919> 2018-07-11 07:52:15.269523 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.1 10.8.128.106:0/12677 267 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:15.317593) v4 ==== 2004+0+0 (3739037815 0 0) 0x55d9869d9800 con 0x55d986db4800 > -1918> 2018-07-11 07:52:15.269536 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:15.317593) v4 -- 0x55d9b67d8400 con 0 > -1917> 2018-07-11 07:52:15.269559 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:15.317593) v4 -- 0x55d982cdf600 con 0 > -1916> 2018-07-11 07:52:15.356343 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.107:0/12617 conn(0x55d98123c000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 255 0x55d9b67d9e00 osd_ping(ping e501 stamp 2018-07-11 07:52:15.355978) v4 > -1915> 2018-07-11 07:52:15.356347 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.107:0/12617 conn(0x55d98123d800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 255 0x55d9bdba5e00 osd_ping(ping e501 stamp 2018-07-11 07:52:15.355978) v4 > -1914> 2018-07-11 07:52:15.356369 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.9 10.8.128.107:0/12617 255 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:15.355978) v4 ==== 2004+0+0 (2510818810 0 0) 0x55d9b67d9e00 con 0x55d98123c000 > -1913> 2018-07-11 07:52:15.356374 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.9 10.8.128.107:0/12617 255 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:15.355978) v4 ==== 2004+0+0 (2510818810 0 0) 0x55d9bdba5e00 con 0x55d98123d800 > -1912> 2018-07-11 07:52:15.356382 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:15.355978) v4 -- 0x55d9b67d8400 con 0 > -1911> 2018-07-11 07:52:15.356406 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:15.355978) v4 -- 0x55d986cd4c00 con 0 > -1910> 2018-07-11 07:52:15.443139 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1050 0x55d9af004900 rep_scrubmap(5.2e e501 from shard 2) v1 > -1909> 2018-07-11 07:52:15.443156 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1050 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12462 (1269249638 0 1596775858) 0x55d9af004900 con 0x55d986bf0800 > -1908> 2018-07-11 07:52:15.444887 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1044 0x55d9af004b40 rep_scrubmap(5.2e e501 from shard 4) v1 > -1907> 2018-07-11 07:52:15.444907 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1044 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12462 (3686815630 0 1596775858) 0x55d9af004b40 con 0x55d986bec000 > -1906> 2018-07-11 07:52:15.466940 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -1905> 2018-07-11 07:52:15.466962 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -1904> 2018-07-11 07:52:15.467021 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -1903> 2018-07-11 07:52:15.467487 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -1902> 2018-07-11 07:52:15.498290 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.114:0/15919 conn(0x55d986df3800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.11 seq 266 0x55d9c9122a00 osd_ping(ping e501 stamp 2018-07-11 07:52:16.128763) v4 > -1901> 2018-07-11 07:52:15.498309 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.11 10.8.128.114:0/15919 266 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:16.128763) v4 ==== 2004+0+0 (2725336170 0 0) 0x55d9c9122a00 con 0x55d986df3800 > -1900> 2018-07-11 07:52:15.498322 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:16.128763) v4 -- 0x55d986cd4c00 con 0 > -1899> 2018-07-11 07:52:15.498321 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.114:0/15919 conn(0x55d986df5000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.11 seq 266 0x55d982d31600 osd_ping(ping e501 stamp 2018-07-11 07:52:16.128763) v4 > -1898> 2018-07-11 07:52:15.498338 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.11 10.8.128.114:0/15919 266 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:16.128763) v4 ==== 2004+0+0 (2725336170 0 0) 0x55d982d31600 con 0x55d986df5000 > -1897> 2018-07-11 07:52:15.498352 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:16.128763) v4 -- 0x55d982cdf600 con 0 > -1896> 2018-07-11 07:52:15.509111 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:757abdbd:::benchmark_data_magna115_10867_object2971928:0,end:5:757b3321:::benchmark_data_magna115_11707_object328318:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1499400 con 0 > -1895> 2018-07-11 07:52:15.509160 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:757abdbd:::benchmark_data_magna115_10867_object2971928:0,end:5:757b3321:::benchmark_data_magna115_11707_object328318:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b14f0a00 con 0 > -1894> 2018-07-11 07:52:15.623546 7ffab1bdc700 5 osd.10 501 heartbeat: osd_stat(789 GB used, 142 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist []) > -1893> 2018-07-11 07:52:15.623560 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6802/15255 -- osd_ping(ping e501 stamp 2018-07-11 07:52:15.623558) v4 -- 0x55d986d4e400 con 0 > -1892> 2018-07-11 07:52:15.623596 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6803/15255 -- osd_ping(ping e501 stamp 2018-07-11 07:52:15.623558) v4 -- 0x55d982cde200 con 0 > -1891> 2018-07-11 07:52:15.623613 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6806/12677 -- osd_ping(ping e501 stamp 2018-07-11 07:52:15.623558) v4 -- 0x55d9c3383600 con 0 > -1890> 2018-07-11 07:52:15.623629 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6807/12677 -- osd_ping(ping e501 stamp 2018-07-11 07:52:15.623558) v4 -- 0x55d9c3383a00 con 0 > -1889> 2018-07-11 07:52:15.623639 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6808/12651 -- osd_ping(ping e501 stamp 2018-07-11 07:52:15.623558) v4 -- 0x55d9c74e0200 con 0 > -1888> 2018-07-11 07:52:15.623647 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6810/12651 -- osd_ping(ping e501 stamp 2018-07-11 07:52:15.623558) v4 -- 0x55d9b9ac3c00 con 0 > -1887> 2018-07-11 07:52:15.623665 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6809/12650 -- osd_ping(ping e501 stamp 2018-07-11 07:52:15.623558) v4 -- 0x55d9ba818800 con 0 > -1886> 2018-07-11 07:52:15.623678 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6811/12650 -- osd_ping(ping e501 stamp 2018-07-11 07:52:15.623558) v4 -- 0x55d9b9ac2a00 con 0 > -1885> 2018-07-11 07:52:15.623690 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6810/15287 -- osd_ping(ping e501 stamp 2018-07-11 07:52:15.623558) v4 -- 0x55d9b437a200 con 0 > -1884> 2018-07-11 07:52:15.623702 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6811/15287 -- osd_ping(ping e501 stamp 2018-07-11 07:52:15.623558) v4 -- 0x55d9ba819a00 con 0 > -1883> 2018-07-11 07:52:15.623718 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6806/15263 -- osd_ping(ping e501 stamp 2018-07-11 07:52:15.623558) v4 -- 0x55d982cdea00 con 0 > -1882> 2018-07-11 07:52:15.623732 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6807/15263 -- osd_ping(ping e501 stamp 2018-07-11 07:52:15.623558) v4 -- 0x55d9829e4800 con 0 > -1881> 2018-07-11 07:52:15.623744 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.115:6802/13323 -- osd_ping(ping e501 stamp 2018-07-11 07:52:15.623558) v4 -- 0x55d982d30400 con 0 > -1880> 2018-07-11 07:52:15.623757 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.115:6803/13323 -- osd_ping(ping e501 stamp 2018-07-11 07:52:15.623558) v4 -- 0x55d986d4f800 con 0 > -1879> 2018-07-11 07:52:15.623773 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6806/15885 -- osd_ping(ping e501 stamp 2018-07-11 07:52:15.623558) v4 -- 0x55d9c4c1b600 con 0 > -1878> 2018-07-11 07:52:15.623786 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6807/15885 -- osd_ping(ping e501 stamp 2018-07-11 07:52:15.623558) v4 -- 0x55d9b2783e00 con 0 > -1877> 2018-07-11 07:52:15.623798 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.108:6802/11967 -- osd_ping(ping e501 stamp 2018-07-11 07:52:15.623558) v4 -- 0x55d9b437b200 con 0 > -1876> 2018-07-11 07:52:15.623811 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.108:6803/11967 -- osd_ping(ping e501 stamp 2018-07-11 07:52:15.623558) v4 -- 0x55d9c3382e00 con 0 > -1875> 2018-07-11 07:52:15.623824 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.107:6806/12617 -- osd_ping(ping e501 stamp 2018-07-11 07:52:15.623558) v4 -- 0x55d986c15000 con 0 > -1874> 2018-07-11 07:52:15.623837 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.107:6807/12617 -- osd_ping(ping e501 stamp 2018-07-11 07:52:15.623558) v4 -- 0x55d9c3383200 con 0 > -1873> 2018-07-11 07:52:15.623850 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6802/15919 -- osd_ping(ping e501 stamp 2018-07-11 07:52:15.623558) v4 -- 0x55d9b2528000 con 0 > -1872> 2018-07-11 07:52:15.623863 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6803/15919 -- osd_ping(ping e501 stamp 2018-07-11 07:52:15.623558) v4 -- 0x55d9b9ac3200 con 0 > -1871> 2018-07-11 07:52:15.624024 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.107:6807/12617 conn(0x55d986bd3800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.9 seq 254 0x55d9bdba5e00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:15.623558) v4 > -1870> 2018-07-11 07:52:15.624050 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.9 10.8.128.107:6807/12617 254 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:15.623558) v4 ==== 2004+0+0 (3024425343 0 0) 0x55d9bdba5e00 con 0x55d986bd3800 > -1869> 2018-07-11 07:52:15.624058 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.107:6806/12617 conn(0x55d986bd2000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.9 seq 254 0x55d982d31600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:15.623558) v4 > -1868> 2018-07-11 07:52:15.624085 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.9 10.8.128.107:6806/12617 254 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:15.623558) v4 ==== 2004+0+0 (3024425343 0 0) 0x55d982d31600 con 0x55d986bd2000 > -1867> 2018-07-11 07:52:15.624126 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6803/15255 conn(0x55d986bc9800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.0 seq 254 0x55d982d31600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:15.623558) v4 > -1866> 2018-07-11 07:52:15.624140 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.0 10.8.128.113:6803/15255 254 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:15.623558) v4 ==== 2004+0+0 (3024425343 0 0) 0x55d982d31600 con 0x55d986bc9800 > -1865> 2018-07-11 07:52:15.624174 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6806/15263 conn(0x55d986b9d000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.5 seq 254 0x55d9bdba5e00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:15.623558) v4 > -1864> 2018-07-11 07:52:15.624195 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.5 10.8.128.113:6806/15263 254 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:15.623558) v4 ==== 2004+0+0 (3024425343 0 0) 0x55d9bdba5e00 con 0x55d986b9d000 > -1863> 2018-07-11 07:52:15.624187 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6807/12677 conn(0x55d986b59800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.1 seq 254 0x55d982d31c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:15.623558) v4 > -1862> 2018-07-11 07:52:15.624226 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.1 10.8.128.106:6807/12677 254 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:15.623558) v4 ==== 2004+0+0 (3024425343 0 0) 0x55d982d31c00 con 0x55d986b59800 > -1861> 2018-07-11 07:52:15.624234 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6810/12651 conn(0x55d986b9b800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.2 seq 254 0x55d982d31600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:15.623558) v4 > -1860> 2018-07-11 07:52:15.624255 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6806/12677 conn(0x55d986b58000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.1 seq 254 0x55d9bdba5e00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:15.623558) v4 > -1859> 2018-07-11 07:52:15.624262 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.2 10.8.128.106:6810/12651 254 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:15.623558) v4 ==== 2004+0+0 (3024425343 0 0) 0x55d982d31600 con 0x55d986b9b800 > -1858> 2018-07-11 07:52:15.624265 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6808/12651 conn(0x55d986b9a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.2 seq 254 0x55d982d31c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:15.623558) v4 > -1857> 2018-07-11 07:52:15.624270 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.1 10.8.128.106:6806/12677 254 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:15.623558) v4 ==== 2004+0+0 (3024425343 0 0) 0x55d9bdba5e00 con 0x55d986b58000 > -1856> 2018-07-11 07:52:15.624278 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.2 10.8.128.106:6808/12651 254 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:15.623558) v4 ==== 2004+0+0 (3024425343 0 0) 0x55d982d31c00 con 0x55d986b9a000 > -1855> 2018-07-11 07:52:15.624307 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6802/15255 conn(0x55d986bc8000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.0 seq 254 0x55d982d31c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:15.623558) v4 > -1854> 2018-07-11 07:52:15.624320 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.0 10.8.128.113:6802/15255 254 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:15.623558) v4 ==== 2004+0+0 (3024425343 0 0) 0x55d982d31c00 con 0x55d986bc8000 > -1853> 2018-07-11 07:52:15.624332 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6811/15287 conn(0x55d986bb0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 254 0x55d982d31600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:15.623558) v4 > -1852> 2018-07-11 07:52:15.624337 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6809/12650 conn(0x55d986bcb000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 254 0x55d9bdba5e00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:15.623558) v4 > -1851> 2018-07-11 07:52:15.624355 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6811/12650 conn(0x55d986bcc800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 254 0x55d982d31c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:15.623558) v4 > -1850> 2018-07-11 07:52:15.624359 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.3 10.8.128.106:6809/12650 254 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:15.623558) v4 ==== 2004+0+0 (3024425343 0 0) 0x55d9bdba5e00 con 0x55d986bcb000 > -1849> 2018-07-11 07:52:15.624370 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.3 10.8.128.106:6811/12650 254 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:15.623558) v4 ==== 2004+0+0 (3024425343 0 0) 0x55d982d31c00 con 0x55d986bcc800 > -1848> 2018-07-11 07:52:15.624354 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.4 10.8.128.113:6811/15287 254 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:15.623558) v4 ==== 2004+0+0 (3024425343 0 0) 0x55d982d31600 con 0x55d986bb0800 > -1847> 2018-07-11 07:52:15.624398 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6810/15287 conn(0x55d986baf000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 254 0x55d982d31c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:15.623558) v4 > -1846> 2018-07-11 07:52:15.624410 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.4 10.8.128.113:6810/15287 254 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:15.623558) v4 ==== 2004+0+0 (3024425343 0 0) 0x55d982d31c00 con 0x55d986baf000 > -1845> 2018-07-11 07:52:15.624411 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.115:6802/13323 conn(0x55d986bc1000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 254 0x55d982d31600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:15.623558) v4 > -1844> 2018-07-11 07:52:15.624429 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.6 10.8.128.115:6802/13323 254 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:15.623558) v4 ==== 2004+0+0 (3024425343 0 0) 0x55d982d31600 con 0x55d986bc1000 > -1843> 2018-07-11 07:52:15.624431 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6807/15885 conn(0x55d986bad800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.7 seq 254 0x55d9bdba5e00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:15.623558) v4 > -1842> 2018-07-11 07:52:15.624448 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.7 10.8.128.114:6807/15885 254 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:15.623558) v4 ==== 2004+0+0 (3024425343 0 0) 0x55d9bdba5e00 con 0x55d986bad800 > -1841> 2018-07-11 07:52:15.624453 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6807/15263 conn(0x55d986b9e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.5 seq 254 0x55d982d31c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:15.623558) v4 > -1840> 2018-07-11 07:52:15.624466 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.5 10.8.128.113:6807/15263 254 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:15.623558) v4 ==== 2004+0+0 (3024425343 0 0) 0x55d982d31c00 con 0x55d986b9e800 > -1839> 2018-07-11 07:52:15.624478 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6803/15919 conn(0x55d986b5c800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.11 seq 254 0x55d9bdba5e00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:15.623558) v4 > -1838> 2018-07-11 07:52:15.624493 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.11 10.8.128.114:6803/15919 254 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:15.623558) v4 ==== 2004+0+0 (3024425343 0 0) 0x55d9bdba5e00 con 0x55d986b5c800 > -1837> 2018-07-11 07:52:15.624472 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6806/15885 conn(0x55d986bac000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.7 seq 254 0x55d982d31600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:15.623558) v4 > -1836> 2018-07-11 07:52:15.624502 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.108:6803/11967 conn(0x55d986bbf800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.8 seq 254 0x55d982d31c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:15.623558) v4 > -1835> 2018-07-11 07:52:15.624509 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.7 10.8.128.114:6806/15885 254 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:15.623558) v4 ==== 2004+0+0 (3024425343 0 0) 0x55d982d31600 con 0x55d986bac000 > -1834> 2018-07-11 07:52:15.624515 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.8 10.8.128.108:6803/11967 254 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:15.623558) v4 ==== 2004+0+0 (3024425343 0 0) 0x55d982d31c00 con 0x55d986bbf800 > -1833> 2018-07-11 07:52:15.624522 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.115:6803/13323 conn(0x55d986bc2800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 254 0x55d9bdba5e00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:15.623558) v4 > -1832> 2018-07-11 07:52:15.624538 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.6 10.8.128.115:6803/13323 254 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:15.623558) v4 ==== 2004+0+0 (3024425343 0 0) 0x55d9bdba5e00 con 0x55d986bc2800 > -1831> 2018-07-11 07:52:15.624563 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6802/15919 conn(0x55d986b5b000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.11 seq 254 0x55d982d31600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:15.623558) v4 > -1830> 2018-07-11 07:52:15.624570 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.108:6802/11967 conn(0x55d986bbe000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.8 seq 254 0x55d9bdba5e00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:15.623558) v4 > -1829> 2018-07-11 07:52:15.624585 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.11 10.8.128.114:6802/15919 254 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:15.623558) v4 ==== 2004+0+0 (3024425343 0 0) 0x55d982d31600 con 0x55d986b5b000 > -1828> 2018-07-11 07:52:15.624590 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.8 10.8.128.108:6802/11967 254 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:15.623558) v4 ==== 2004+0+0 (3024425343 0 0) 0x55d9bdba5e00 con 0x55d986bbe000 > -1827> 2018-07-11 07:52:15.688829 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1051 0x55d9af004d80 rep_scrubmap(5.2e e501 from shard 2) v1 > -1826> 2018-07-11 07:52:15.688848 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1051 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12462 (1269249638 0 4096529302) 0x55d9af004d80 con 0x55d986bf0800 > -1825> 2018-07-11 07:52:15.704366 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1045 0x55d9af004fc0 rep_scrubmap(5.2e e501 from shard 4) v1 > -1824> 2018-07-11 07:52:15.704386 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1045 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12462 (3686815630 0 4096529302) 0x55d9af004fc0 con 0x55d986bec000 > -1823> 2018-07-11 07:52:15.752836 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -1822> 2018-07-11 07:52:15.752861 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -1821> 2018-07-11 07:52:15.752919 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -1820> 2018-07-11 07:52:15.753388 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -1819> 2018-07-11 07:52:15.787623 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67777,epoch:501/498,start:5:757b3321:::benchmark_data_magna115_11707_object328318:0,end:5:757b941d:::benchmark_data_magna115_10867_object1913241:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498f00 con 0 > -1818> 2018-07-11 07:52:15.787660 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67777,epoch:501/498,start:5:757b3321:::benchmark_data_magna115_11707_object328318:0,end:5:757b941d:::benchmark_data_magna115_10867_object1913241:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1499b80 con 0 > -1817> 2018-07-11 07:52:16.004026 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1046 0x55d9af005200 rep_scrubmap(5.2e e501 from shard 4) v1 > -1816> 2018-07-11 07:52:16.004046 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1046 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12472 (3686815630 0 3789441997) 0x55d9af005200 con 0x55d986bec000 > -1815> 2018-07-11 07:52:16.018411 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1052 0x55d9af005440 rep_scrubmap(5.2e e501 from shard 2) v1 > -1814> 2018-07-11 07:52:16.018431 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1052 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12472 (1269249638 0 3789441997) 0x55d9af005440 con 0x55d986bf0800 > -1813> 2018-07-11 07:52:16.191061 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15287 conn(0x55d98123f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 278 0x55d982d31600 osd_ping(ping e501 stamp 2018-07-11 07:52:17.422615) v4 > -1812> 2018-07-11 07:52:16.191091 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.4 10.8.128.113:0/15287 278 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:17.422615) v4 ==== 2004+0+0 (1042185928 0 0) 0x55d982d31600 con 0x55d98123f000 > -1811> 2018-07-11 07:52:16.191102 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:17.422615) v4 -- 0x55d9b2528000 con 0 > -1810> 2018-07-11 07:52:16.191105 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15287 conn(0x55d981240800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 278 0x55d9b67d9e00 osd_ping(ping e501 stamp 2018-07-11 07:52:17.422615) v4 > -1809> 2018-07-11 07:52:16.191132 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.4 10.8.128.113:0/15287 278 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:17.422615) v4 ==== 2004+0+0 (1042185928 0 0) 0x55d9b67d9e00 con 0x55d981240800 > -1808> 2018-07-11 07:52:16.191148 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:17.422615) v4 -- 0x55d9c3382e00 con 0 > -1807> 2018-07-11 07:52:16.223271 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.114:0/15885 conn(0x55d986d7f800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.7 seq 269 0x55d9869d9800 osd_ping(ping e501 stamp 2018-07-11 07:52:16.853696) v4 > -1806> 2018-07-11 07:52:16.223293 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.7 10.8.128.114:0/15885 269 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:16.853696) v4 ==== 2004+0+0 (3824119434 0 0) 0x55d9869d9800 con 0x55d986d7f800 > -1805> 2018-07-11 07:52:16.223293 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.114:0/15885 conn(0x55d986d81000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.7 seq 269 0x55d982d31c00 osd_ping(ping e501 stamp 2018-07-11 07:52:16.853696) v4 > -1804> 2018-07-11 07:52:16.223307 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:16.853696) v4 -- 0x55d9b2528000 con 0 > -1803> 2018-07-11 07:52:16.223312 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.7 10.8.128.114:0/15885 269 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:16.853696) v4 ==== 2004+0+0 (3824119434 0 0) 0x55d982d31c00 con 0x55d986d81000 > -1802> 2018-07-11 07:52:16.223336 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:16.853696) v4 -- 0x55d9c3382e00 con 0 > -1801> 2018-07-11 07:52:16.456842 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.107:0/12617 conn(0x55d98123c000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 256 0x55d9b67d9e00 osd_ping(ping e501 stamp 2018-07-11 07:52:16.456516) v4 > -1800> 2018-07-11 07:52:16.456850 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.107:0/12617 conn(0x55d98123d800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 256 0x55d9bdba5e00 osd_ping(ping e501 stamp 2018-07-11 07:52:16.456516) v4 > -1799> 2018-07-11 07:52:16.456889 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.9 10.8.128.107:0/12617 256 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:16.456516) v4 ==== 2004+0+0 (2050476123 0 0) 0x55d9bdba5e00 con 0x55d98123d800 > -1798> 2018-07-11 07:52:16.456864 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.9 10.8.128.107:0/12617 256 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:16.456516) v4 ==== 2004+0+0 (2050476123 0 0) 0x55d9b67d9e00 con 0x55d98123c000 > -1797> 2018-07-11 07:52:16.456903 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:16.456516) v4 -- 0x55d9b9ac3200 con 0 > -1796> 2018-07-11 07:52:16.456928 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:16.456516) v4 -- 0x55d9c3382e00 con 0 > -1795> 2018-07-11 07:52:16.807129 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.115:0/13323 conn(0x55d986d7e000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 264 0x55d9c9122a00 osd_ping(ping e501 stamp 2018-07-11 07:52:17.153259) v4 > -1794> 2018-07-11 07:52:16.807149 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.6 10.8.128.115:0/13323 264 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:17.153259) v4 ==== 2004+0+0 (1873392894 0 0) 0x55d9c9122a00 con 0x55d986d7e000 > -1793> 2018-07-11 07:52:16.807158 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:17.153259) v4 -- 0x55d9b9ac3200 con 0 > -1792> 2018-07-11 07:52:16.807232 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.115:0/13323 conn(0x55d986d24800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 264 0x55d982d31c00 osd_ping(ping e501 stamp 2018-07-11 07:52:17.153259) v4 > -1791> 2018-07-11 07:52:16.807259 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.6 10.8.128.115:0/13323 264 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:17.153259) v4 ==== 2004+0+0 (1873392894 0 0) 0x55d982d31c00 con 0x55d986d24800 > -1790> 2018-07-11 07:52:16.807270 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:17.153259) v4 -- 0x55d9c3382e00 con 0 > -1789> 2018-07-11 07:52:16.926418 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -1788> 2018-07-11 07:52:16.926446 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -1787> 2018-07-11 07:52:16.926503 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -1786> 2018-07-11 07:52:16.926970 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -1785> 2018-07-11 07:52:16.959419 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:757b941d:::benchmark_data_magna115_10867_object1913241:0,end:5:757bf90a:::benchmark_data_magna115_10867_object1801904:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1499180 con 0 > -1784> 2018-07-11 07:52:16.959458 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:757b941d:::benchmark_data_magna115_10867_object1913241:0,end:5:757bf90a:::benchmark_data_magna115_10867_object1801904:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498500 con 0 > -1783> 2018-07-11 07:52:17.036705 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 33] Generated table #36811: 2406128 keys, 68114287 bytes > -1782> 2018-07-11 07:52:17.036749 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295537036722, "cf_name": "default", "job": 33, "event": "table_file_creation", "file_number": 36811, "file_size": 68114287, "table_properties": {"data_size": 61332638, "index_size": 630459, "filter_size": 6150201, "raw_key_size": 69526219, "raw_average_key_size": 28, "raw_value_size": 28711960, "raw_average_value_size": 11, "num_data_blocks": 15071, "num_entries": 2406128, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "11", "kMergeOperands": "0"}} > -1781> 2018-07-11 07:52:17.036786 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 33] Compacted 4@1 + 1@2 files to L2 => 366300845 bytes > -1780> 2018-07-11 07:52:17.051339 7ffabc3f1700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 49121545030 > -1779> 2018-07-11 07:52:17.051576 7ffabc3f1700 4 rocksdb: (Original Log Time 2018/07/11-07:52:17.051307) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[0 7 72 470 3641 796 0] max score 2.26, MB/sec: 12.8 rd, 12.8 wr, level 2, files in(4, 1) out(5) MB in(284.4, 65.0) out(349.3), read-write-amplify(2.5) write-amplify(1.2) OK, records in: 12959092, records dropped: 0 > > -1778> 2018-07-11 07:52:17.051585 7ffabc3f1700 4 rocksdb: (Original Log Time 2018/07/11-07:52:17.051332) EVENT_LOG_v1 {"time_micros": 1531295537051321, "job": 33, "event": "compaction_finished", "compaction_time_micros": 28608701, "output_level": 2, "num_output_files": 5, "total_output_size": 366300845, "num_input_records": 12959092, "num_output_records": 12959092, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 7, 72, 470, 3641, 796, 0]} > -1777> 2018-07-11 07:52:17.052194 7ffabc3f1700 5 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:307] [JOB 33] Delete db/036575.sst type=2 #36575 -- OK > > -1776> 2018-07-11 07:52:17.052219 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295537052203, "job": 33, "event": "table_file_deletion", "file_number": 36575} > -1775> 2018-07-11 07:52:17.052229 7ffabc3f1700 5 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:307] [JOB 33] Delete db/036574.sst type=2 #36574 -- OK > > -1774> 2018-07-11 07:52:17.052235 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295537052233, "job": 33, "event": "table_file_deletion", "file_number": 36574} > -1773> 2018-07-11 07:52:17.052250 7ffabc3f1700 5 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:307] [JOB 33] Delete db/036573.sst type=2 #36573 -- OK > > -1772> 2018-07-11 07:52:17.052255 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295537052253, "job": 33, "event": "table_file_deletion", "file_number": 36573} > -1771> 2018-07-11 07:52:17.052265 7ffabc3f1700 5 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:307] [JOB 33] Delete db/036572.sst type=2 #36572 -- OK > > -1770> 2018-07-11 07:52:17.052271 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295537052268, "job": 33, "event": "table_file_deletion", "file_number": 36572} > -1769> 2018-07-11 07:52:17.052282 7ffabc3f1700 5 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:307] [JOB 33] Delete db.slow/036290.sst type=2 #36290 -- OK > > -1768> 2018-07-11 07:52:17.052289 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295537052287, "job": 33, "event": "table_file_deletion", "file_number": 36290} > -1767> 2018-07-11 07:52:17.052468 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 34] Compacting 1@2 + 8@3 files to L3, score 2.26 > -1766> 2018-07-11 07:52:17.052483 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 33 Base level 2, inputs: [36612(76MB)], [15174(71MB) 15177(577KB) 15288(70MB) 15289(71MB) 15290(71MB) 15291(71MB) 15292(71MB) 15293(1249B)] > > -1765> 2018-07-11 07:52:17.052497 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295537052486, "job": 34, "event": "compaction_started", "files_L2": [36612], "files_L3": [15174, 15177, 15288, 15289, 15290, 15291, 15292, 15293], "score": 2.26028, "input_data_size": 527530353} > -1764> 2018-07-11 07:52:17.135768 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1053 0x55d9af005680 rep_scrubmap(5.2e e501 from shard 2) v1 > -1763> 2018-07-11 07:52:17.135786 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1053 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12482 (1269249638 0 1030957041) 0x55d9af005680 con 0x55d986bf0800 > -1762> 2018-07-11 07:52:17.166133 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1047 0x55d9af0058c0 rep_scrubmap(5.2e e501 from shard 4) v1 > -1761> 2018-07-11 07:52:17.166148 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1047 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12482 (3686815630 0 1030957041) 0x55d9af0058c0 con 0x55d986bec000 > -1760> 2018-07-11 07:52:17.221949 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12650 conn(0x55d986d23000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 256 0x55d982d31600 osd_ping(ping e501 stamp 2018-07-11 07:52:17.270054) v4 > -1759> 2018-07-11 07:52:17.221970 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.3 10.8.128.106:0/12650 256 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:17.270054) v4 ==== 2004+0+0 (1359459024 0 0) 0x55d982d31600 con 0x55d986d23000 > -1758> 2018-07-11 07:52:17.221970 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12650 conn(0x55d986d21800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 256 0x55d9bdba5e00 osd_ping(ping e501 stamp 2018-07-11 07:52:17.270054) v4 > -1757> 2018-07-11 07:52:17.221987 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.3 10.8.128.106:0/12650 256 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:17.270054) v4 ==== 2004+0+0 (1359459024 0 0) 0x55d9bdba5e00 con 0x55d986d21800 > -1756> 2018-07-11 07:52:17.221987 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:17.270054) v4 -- 0x55d9b2528000 con 0 > -1755> 2018-07-11 07:52:17.222027 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:17.270054) v4 -- 0x55d9b9ac3200 con 0 > -1754> 2018-07-11 07:52:17.267268 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -1753> 2018-07-11 07:52:17.267296 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -1752> 2018-07-11 07:52:17.267341 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -1751> 2018-07-11 07:52:17.267804 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -1750> 2018-07-11 07:52:17.301016 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:757bf90a:::benchmark_data_magna115_10867_object1801904:0,end:5:757c3cf8:::benchmark_data_magna115_10867_object3043123:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498c80 con 0 > -1749> 2018-07-11 07:52:17.301065 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:757bf90a:::benchmark_data_magna115_10867_object1801904:0,end:5:757c3cf8:::benchmark_data_magna115_10867_object3043123:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1499680 con 0 > -1748> 2018-07-11 07:52:17.324014 7ffab1bdc700 5 osd.10 501 heartbeat: osd_stat(789 GB used, 142 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist []) > -1747> 2018-07-11 07:52:17.324027 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6802/15255 -- osd_ping(ping e501 stamp 2018-07-11 07:52:17.324025) v4 -- 0x55d9b2783400 con 0 > -1746> 2018-07-11 07:52:17.324061 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6803/15255 -- osd_ping(ping e501 stamp 2018-07-11 07:52:17.324025) v4 -- 0x55d9bdba4200 con 0 > -1745> 2018-07-11 07:52:17.324075 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6806/12677 -- osd_ping(ping e501 stamp 2018-07-11 07:52:17.324025) v4 -- 0x55d980f5a000 con 0 > -1744> 2018-07-11 07:52:17.324090 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6807/12677 -- osd_ping(ping e501 stamp 2018-07-11 07:52:17.324025) v4 -- 0x55d9829e5800 con 0 > -1743> 2018-07-11 07:52:17.324113 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6808/12651 -- osd_ping(ping e501 stamp 2018-07-11 07:52:17.324025) v4 -- 0x55d9869be600 con 0 > -1742> 2018-07-11 07:52:17.324126 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6810/12651 -- osd_ping(ping e501 stamp 2018-07-11 07:52:17.324025) v4 -- 0x55d982cdf800 con 0 > -1741> 2018-07-11 07:52:17.324137 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6809/12650 -- osd_ping(ping e501 stamp 2018-07-11 07:52:17.324025) v4 -- 0x55d986c84400 con 0 > -1740> 2018-07-11 07:52:17.324155 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6811/12650 -- osd_ping(ping e501 stamp 2018-07-11 07:52:17.324025) v4 -- 0x55d986c23e00 con 0 > -1739> 2018-07-11 07:52:17.324168 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6810/15287 -- osd_ping(ping e501 stamp 2018-07-11 07:52:17.324025) v4 -- 0x55d980f5b200 con 0 > -1738> 2018-07-11 07:52:17.324180 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6811/15287 -- osd_ping(ping e501 stamp 2018-07-11 07:52:17.324025) v4 -- 0x55d9c3382200 con 0 > -1737> 2018-07-11 07:52:17.324198 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6806/15263 -- osd_ping(ping e501 stamp 2018-07-11 07:52:17.324025) v4 -- 0x55d9b3db2600 con 0 > -1736> 2018-07-11 07:52:17.324212 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6807/15263 -- osd_ping(ping e501 stamp 2018-07-11 07:52:17.324025) v4 -- 0x55d9b60df600 con 0 > -1735> 2018-07-11 07:52:17.324224 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.115:6802/13323 -- osd_ping(ping e501 stamp 2018-07-11 07:52:17.324025) v4 -- 0x55d9869bfe00 con 0 > -1734> 2018-07-11 07:52:17.324240 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.115:6803/13323 -- osd_ping(ping e501 stamp 2018-07-11 07:52:17.324025) v4 -- 0x55d986d33c00 con 0 > -1733> 2018-07-11 07:52:17.324268 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6806/15885 -- osd_ping(ping e501 stamp 2018-07-11 07:52:17.324025) v4 -- 0x55d9829e4e00 con 0 > -1732> 2018-07-11 07:52:17.324282 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6807/15885 -- osd_ping(ping e501 stamp 2018-07-11 07:52:17.324025) v4 -- 0x55d9b60dea00 con 0 > -1731> 2018-07-11 07:52:17.324295 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.108:6802/11967 -- osd_ping(ping e501 stamp 2018-07-11 07:52:17.324025) v4 -- 0x55d9c908ec00 con 0 > -1730> 2018-07-11 07:52:17.324307 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.108:6803/11967 -- osd_ping(ping e501 stamp 2018-07-11 07:52:17.324025) v4 -- 0x55d9b4389a00 con 0 > -1729> 2018-07-11 07:52:17.324319 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.107:6806/12617 -- osd_ping(ping e501 stamp 2018-07-11 07:52:17.324025) v4 -- 0x55d9829e4600 con 0 > -1728> 2018-07-11 07:52:17.324354 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.107:6807/12617 -- osd_ping(ping e501 stamp 2018-07-11 07:52:17.324025) v4 -- 0x55d986b6d400 con 0 > -1727> 2018-07-11 07:52:17.324370 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6802/15919 -- osd_ping(ping e501 stamp 2018-07-11 07:52:17.324025) v4 -- 0x55d9c9123000 con 0 > -1726> 2018-07-11 07:52:17.324383 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6803/15919 -- osd_ping(ping e501 stamp 2018-07-11 07:52:17.324025) v4 -- 0x55d9b40a4000 con 0 > -1725> 2018-07-11 07:52:17.324547 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.107:6806/12617 conn(0x55d986bd2000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.9 seq 255 0x55d982d31600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:17.324025) v4 > -1724> 2018-07-11 07:52:17.324558 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6802/15255 conn(0x55d986bc8000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.0 seq 255 0x55d986d40600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:17.324025) v4 > -1723> 2018-07-11 07:52:17.324571 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.0 10.8.128.113:6802/15255 255 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:17.324025) v4 ==== 2004+0+0 (2624303893 0 0) 0x55d986d40600 con 0x55d986bc8000 > -1722> 2018-07-11 07:52:17.324570 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.9 10.8.128.107:6806/12617 255 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:17.324025) v4 ==== 2004+0+0 (2624303893 0 0) 0x55d982d31600 con 0x55d986bd2000 > -1721> 2018-07-11 07:52:17.324568 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.107:6807/12617 conn(0x55d986bd3800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.9 seq 255 0x55d9c9122a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:17.324025) v4 > -1720> 2018-07-11 07:52:17.324588 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.9 10.8.128.107:6807/12617 255 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:17.324025) v4 ==== 2004+0+0 (2624303893 0 0) 0x55d9c9122a00 con 0x55d986bd3800 > -1719> 2018-07-11 07:52:17.324642 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6803/15255 conn(0x55d986bc9800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.0 seq 255 0x55d982d31600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:17.324025) v4 > -1718> 2018-07-11 07:52:17.324649 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6808/12651 conn(0x55d986b9a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.2 seq 255 0x55d986d40600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:17.324025) v4 > -1717> 2018-07-11 07:52:17.324648 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6809/12650 conn(0x55d986bcb000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 255 0x55d9c9122a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:17.324025) v4 > -1716> 2018-07-11 07:52:17.324658 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.0 10.8.128.113:6803/15255 255 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:17.324025) v4 ==== 2004+0+0 (2624303893 0 0) 0x55d982d31600 con 0x55d986bc9800 > -1715> 2018-07-11 07:52:17.324665 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.2 10.8.128.106:6808/12651 255 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:17.324025) v4 ==== 2004+0+0 (2624303893 0 0) 0x55d986d40600 con 0x55d986b9a000 > -1714> 2018-07-11 07:52:17.324669 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.3 10.8.128.106:6809/12650 255 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:17.324025) v4 ==== 2004+0+0 (2624303893 0 0) 0x55d9c9122a00 con 0x55d986bcb000 > -1713> 2018-07-11 07:52:17.324701 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6811/15287 conn(0x55d986bb0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 255 0x55d982d31600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:17.324025) v4 > -1712> 2018-07-11 07:52:17.324704 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6806/12677 conn(0x55d986b58000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.1 seq 255 0x55d9c9122a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:17.324025) v4 > -1711> 2018-07-11 07:52:17.324713 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6807/15263 conn(0x55d986b9e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.5 seq 255 0x55d986d40600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:17.324025) v4 > -1710> 2018-07-11 07:52:17.324729 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.1 10.8.128.106:6806/12677 255 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:17.324025) v4 ==== 2004+0+0 (2624303893 0 0) 0x55d9c9122a00 con 0x55d986b58000 > -1709> 2018-07-11 07:52:17.324734 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.5 10.8.128.113:6807/15263 255 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:17.324025) v4 ==== 2004+0+0 (2624303893 0 0) 0x55d986d40600 con 0x55d986b9e800 > -1708> 2018-07-11 07:52:17.324719 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.4 10.8.128.113:6811/15287 255 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:17.324025) v4 ==== 2004+0+0 (2624303893 0 0) 0x55d982d31600 con 0x55d986bb0800 > -1707> 2018-07-11 07:52:17.324774 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6806/15263 conn(0x55d986b9d000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.5 seq 255 0x55d9c9122a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:17.324025) v4 > -1706> 2018-07-11 07:52:17.324787 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.5 10.8.128.113:6806/15263 255 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:17.324025) v4 ==== 2004+0+0 (2624303893 0 0) 0x55d9c9122a00 con 0x55d986b9d000 > -1705> 2018-07-11 07:52:17.324792 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6810/12651 conn(0x55d986b9b800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.2 seq 255 0x55d982d31600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:17.324025) v4 > -1704> 2018-07-11 07:52:17.324799 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6807/12677 conn(0x55d986b59800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.1 seq 255 0x55d986d40600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:17.324025) v4 > -1703> 2018-07-11 07:52:17.324810 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.1 10.8.128.106:6807/12677 255 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:17.324025) v4 ==== 2004+0+0 (2624303893 0 0) 0x55d986d40600 con 0x55d986b59800 > -1702> 2018-07-11 07:52:17.324809 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.2 10.8.128.106:6810/12651 255 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:17.324025) v4 ==== 2004+0+0 (2624303893 0 0) 0x55d982d31600 con 0x55d986b9b800 > -1701> 2018-07-11 07:52:17.324824 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.115:6803/13323 conn(0x55d986bc2800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 255 0x55d9c9122a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:17.324025) v4 > -1700> 2018-07-11 07:52:17.324839 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.6 10.8.128.115:6803/13323 255 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:17.324025) v4 ==== 2004+0+0 (2624303893 0 0) 0x55d9c9122a00 con 0x55d986bc2800 > -1699> 2018-07-11 07:52:17.324843 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6811/12650 conn(0x55d986bcc800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 255 0x55d986d40600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:17.324025) v4 > -1698> 2018-07-11 07:52:17.324859 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.3 10.8.128.106:6811/12650 255 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:17.324025) v4 ==== 2004+0+0 (2624303893 0 0) 0x55d986d40600 con 0x55d986bcc800 > -1697> 2018-07-11 07:52:17.324877 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.115:6802/13323 conn(0x55d986bc1000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 255 0x55d982d31600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:17.324025) v4 > -1696> 2018-07-11 07:52:17.324892 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6810/15287 conn(0x55d986baf000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 255 0x55d986d40600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:17.324025) v4 > -1695> 2018-07-11 07:52:17.324891 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.6 10.8.128.115:6802/13323 255 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:17.324025) v4 ==== 2004+0+0 (2624303893 0 0) 0x55d982d31600 con 0x55d986bc1000 > -1694> 2018-07-11 07:52:17.324906 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.4 10.8.128.113:6810/15287 255 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:17.324025) v4 ==== 2004+0+0 (2624303893 0 0) 0x55d986d40600 con 0x55d986baf000 > -1693> 2018-07-11 07:52:17.324950 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6802/15919 conn(0x55d986b5b000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.11 seq 255 0x55d982d31600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:17.324025) v4 > -1692> 2018-07-11 07:52:17.324965 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.11 10.8.128.114:6802/15919 255 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:17.324025) v4 ==== 2004+0+0 (2624303893 0 0) 0x55d982d31600 con 0x55d986b5b000 > -1691> 2018-07-11 07:52:17.324963 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.108:6802/11967 conn(0x55d986bbe000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.8 seq 255 0x55d9c9122a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:17.324025) v4 > -1690> 2018-07-11 07:52:17.324995 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.8 10.8.128.108:6802/11967 255 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:17.324025) v4 ==== 2004+0+0 (2624303893 0 0) 0x55d9c9122a00 con 0x55d986bbe000 > -1689> 2018-07-11 07:52:17.325017 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6806/15885 conn(0x55d986bac000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.7 seq 255 0x55d982d31600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:17.324025) v4 > -1688> 2018-07-11 07:52:17.325019 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6807/15885 conn(0x55d986bad800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.7 seq 255 0x55d9c9122a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:17.324025) v4 > -1687> 2018-07-11 07:52:17.325018 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.108:6803/11967 conn(0x55d986bbf800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.8 seq 255 0x55d986d40600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:17.324025) v4 > -1686> 2018-07-11 07:52:17.325035 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.7 10.8.128.114:6806/15885 255 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:17.324025) v4 ==== 2004+0+0 (2624303893 0 0) 0x55d982d31600 con 0x55d986bac000 > -1685> 2018-07-11 07:52:17.325040 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.7 10.8.128.114:6807/15885 255 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:17.324025) v4 ==== 2004+0+0 (2624303893 0 0) 0x55d9c9122a00 con 0x55d986bad800 > -1684> 2018-07-11 07:52:17.325044 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.8 10.8.128.108:6803/11967 255 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:17.324025) v4 ==== 2004+0+0 (2624303893 0 0) 0x55d986d40600 con 0x55d986bbf800 > -1683> 2018-07-11 07:52:17.325065 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6803/15919 conn(0x55d986b5c800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.11 seq 255 0x55d9c9122a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:17.324025) v4 > -1682> 2018-07-11 07:52:17.325075 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.11 10.8.128.114:6803/15919 255 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:17.324025) v4 ==== 2004+0+0 (2624303893 0 0) 0x55d9c9122a00 con 0x55d986b5c800 > -1681> 2018-07-11 07:52:17.352901 7ffab6be6700 1 -- 10.8.128.107:6800/13927 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55d9bd9cc2c0 con 0 > -1680> 2018-07-11 07:52:17.352956 7ffab6be6700 1 -- 10.8.128.107:6800/13927 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55d983c498c0 con 0 > -1679> 2018-07-11 07:52:17.523416 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1054 0x55d9af005b00 rep_scrubmap(5.2e e501 from shard 2) v1 > -1678> 2018-07-11 07:52:17.523449 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1054 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12482 (1269249638 0 1917101591) 0x55d9af005b00 con 0x55d986bf0800 > -1677> 2018-07-11 07:52:17.550232 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1048 0x55d9af005d40 rep_scrubmap(5.2e e501 from shard 4) v1 > -1676> 2018-07-11 07:52:17.550263 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1048 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12482 (3686815630 0 1917101591) 0x55d9af005d40 con 0x55d986bec000 > -1675> 2018-07-11 07:52:17.569917 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12677 conn(0x55d986db4800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.1 seq 268 0x55d982d31600 osd_ping(ping e501 stamp 2018-07-11 07:52:17.618024) v4 > -1674> 2018-07-11 07:52:17.569923 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12677 conn(0x55d986df2000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.1 seq 268 0x55d986d40600 osd_ping(ping e501 stamp 2018-07-11 07:52:17.618024) v4 > -1673> 2018-07-11 07:52:17.569947 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.1 10.8.128.106:0/12677 268 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:17.618024) v4 ==== 2004+0+0 (1885620087 0 0) 0x55d982d31600 con 0x55d986db4800 > -1672> 2018-07-11 07:52:17.569948 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.1 10.8.128.106:0/12677 268 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:17.618024) v4 ==== 2004+0+0 (1885620087 0 0) 0x55d986d40600 con 0x55d986df2000 > -1671> 2018-07-11 07:52:17.569958 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:17.618024) v4 -- 0x55d9c9123000 con 0 > -1670> 2018-07-11 07:52:17.569987 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:17.618024) v4 -- 0x55d9b4389a00 con 0 > -1669> 2018-07-11 07:52:17.572680 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15255 conn(0x55d986c82800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.0 seq 264 0x55d9869d9800 osd_ping(ping e501 stamp 2018-07-11 07:52:18.804379) v4 > -1668> 2018-07-11 07:52:17.572695 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15255 conn(0x55d986d20000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.0 seq 264 0x55d982d31c00 osd_ping(ping e501 stamp 2018-07-11 07:52:18.804379) v4 > -1667> 2018-07-11 07:52:17.572708 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.0 10.8.128.113:0/15255 264 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:18.804379) v4 ==== 2004+0+0 (3706520116 0 0) 0x55d9869d9800 con 0x55d986c82800 > -1666> 2018-07-11 07:52:17.572721 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.0 10.8.128.113:0/15255 264 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:18.804379) v4 ==== 2004+0+0 (3706520116 0 0) 0x55d982d31c00 con 0x55d986d20000 > -1665> 2018-07-11 07:52:17.572722 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:18.804379) v4 -- 0x55d9c9123000 con 0 > -1664> 2018-07-11 07:52:17.572751 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:18.804379) v4 -- 0x55d9b4389a00 con 0 > -1663> 2018-07-11 07:52:17.722413 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12650 conn(0x55d986d23000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 257 0x55d982d31600 osd_ping(ping e501 stamp 2018-07-11 07:52:17.770563) v4 > -1662> 2018-07-11 07:52:17.722443 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.3 10.8.128.106:0/12650 257 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:17.770563) v4 ==== 2004+0+0 (3093798245 0 0) 0x55d982d31600 con 0x55d986d23000 > -1661> 2018-07-11 07:52:17.722453 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:17.770563) v4 -- 0x55d9c9123000 con 0 > -1660> 2018-07-11 07:52:17.722600 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12650 conn(0x55d986d21800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 257 0x55d9c9122a00 osd_ping(ping e501 stamp 2018-07-11 07:52:17.770563) v4 > -1659> 2018-07-11 07:52:17.722621 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.3 10.8.128.106:0/12650 257 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:17.770563) v4 ==== 2004+0+0 (3093798245 0 0) 0x55d9c9122a00 con 0x55d986d21800 > -1658> 2018-07-11 07:52:17.722633 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:17.770563) v4 -- 0x55d9b40a4000 con 0 > -1657> 2018-07-11 07:52:17.989585 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -1656> 2018-07-11 07:52:17.989614 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -1655> 2018-07-11 07:52:17.989659 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -1654> 2018-07-11 07:52:17.990108 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -1653> 2018-07-11 07:52:18.034614 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:757c3cf8:::benchmark_data_magna115_10867_object3043123:0,end:5:757ca6ba:::benchmark_data_magna115_10867_object2918308:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1499900 con 0 > -1652> 2018-07-11 07:52:18.034669 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:757c3cf8:::benchmark_data_magna115_10867_object3043123:0,end:5:757ca6ba:::benchmark_data_magna115_10867_object2918308:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b14f0280 con 0 > -1651> 2018-07-11 07:52:18.128710 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12651 conn(0x55d986db1800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.2 seq 262 0x55d982d31c00 osd_ping(ping e501 stamp 2018-07-11 07:52:18.176771) v4 > -1650> 2018-07-11 07:52:18.128723 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12651 conn(0x55d986db3000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.2 seq 262 0x55d9bdba5e00 osd_ping(ping e501 stamp 2018-07-11 07:52:18.176771) v4 > -1649> 2018-07-11 07:52:18.128740 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.2 10.8.128.106:0/12651 262 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:18.176771) v4 ==== 2004+0+0 (1977182348 0 0) 0x55d9bdba5e00 con 0x55d986db3000 > -1648> 2018-07-11 07:52:18.128737 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.2 10.8.128.106:0/12651 262 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:18.176771) v4 ==== 2004+0+0 (1977182348 0 0) 0x55d982d31c00 con 0x55d986db1800 > -1647> 2018-07-11 07:52:18.128749 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:18.176771) v4 -- 0x55d9b40a4000 con 0 > -1646> 2018-07-11 07:52:18.128773 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:18.176771) v4 -- 0x55d9b4389a00 con 0 > -1645> 2018-07-11 07:52:18.243061 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1055 0x55d9c2a5e000 rep_scrubmap(5.2e e501 from shard 2) v1 > -1644> 2018-07-11 07:52:18.243081 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1055 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12472 (1269249638 0 8709826) 0x55d9c2a5e000 con 0x55d986bf0800 > -1643> 2018-07-11 07:52:18.258925 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1049 0x55d9c2a5e240 rep_scrubmap(5.2e e501 from shard 4) v1 > -1642> 2018-07-11 07:52:18.258945 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1049 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12472 (3686815630 0 8709826) 0x55d9c2a5e240 con 0x55d986bec000 > -1641> 2018-07-11 07:52:18.356800 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -1640> 2018-07-11 07:52:18.356825 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -1639> 2018-07-11 07:52:18.356886 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -1638> 2018-07-11 07:52:18.357383 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -1637> 2018-07-11 07:52:18.401384 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:757ca6ba:::benchmark_data_magna115_10867_object2918308:0,end:5:757d05d0:::benchmark_data_magna115_10867_object3535108:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b14f0000 con 0 > -1636> 2018-07-11 07:52:18.401424 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:757ca6ba:::benchmark_data_magna115_10867_object2918308:0,end:5:757d05d0:::benchmark_data_magna115_10867_object3535108:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98696f400 con 0 > -1635> 2018-07-11 07:52:18.424526 7ffab1bdc700 5 osd.10 501 heartbeat: osd_stat(789 GB used, 142 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist []) > -1634> 2018-07-11 07:52:18.424540 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6802/15255 -- osd_ping(ping e501 stamp 2018-07-11 07:52:18.424539) v4 -- 0x55d986d41600 con 0 > -1633> 2018-07-11 07:52:18.424564 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6803/15255 -- osd_ping(ping e501 stamp 2018-07-11 07:52:18.424539) v4 -- 0x55d9c34cc000 con 0 > -1632> 2018-07-11 07:52:18.424579 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6806/12677 -- osd_ping(ping e501 stamp 2018-07-11 07:52:18.424539) v4 -- 0x55d9869bf800 con 0 > -1631> 2018-07-11 07:52:18.424594 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6807/12677 -- osd_ping(ping e501 stamp 2018-07-11 07:52:18.424539) v4 -- 0x55d986d4f400 con 0 > -1630> 2018-07-11 07:52:18.424606 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6808/12651 -- osd_ping(ping e501 stamp 2018-07-11 07:52:18.424539) v4 -- 0x55d9869bf200 con 0 > -1629> 2018-07-11 07:52:18.424617 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6810/12651 -- osd_ping(ping e501 stamp 2018-07-11 07:52:18.424539) v4 -- 0x55d9bdba4e00 con 0 > -1628> 2018-07-11 07:52:18.424630 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6809/12650 -- osd_ping(ping e501 stamp 2018-07-11 07:52:18.424539) v4 -- 0x55d9b9ac3000 con 0 > -1627> 2018-07-11 07:52:18.424643 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6811/12650 -- osd_ping(ping e501 stamp 2018-07-11 07:52:18.424539) v4 -- 0x55d9869bf400 con 0 > -1626> 2018-07-11 07:52:18.424667 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6810/15287 -- osd_ping(ping e501 stamp 2018-07-11 07:52:18.424539) v4 -- 0x55d9829e5a00 con 0 > -1625> 2018-07-11 07:52:18.424680 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6811/15287 -- osd_ping(ping e501 stamp 2018-07-11 07:52:18.424539) v4 -- 0x55d9b9ac3400 con 0 > -1624> 2018-07-11 07:52:18.424694 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6806/15263 -- osd_ping(ping e501 stamp 2018-07-11 07:52:18.424539) v4 -- 0x55d9829e4200 con 0 > -1623> 2018-07-11 07:52:18.424709 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6807/15263 -- osd_ping(ping e501 stamp 2018-07-11 07:52:18.424539) v4 -- 0x55d986c23a00 con 0 > -1622> 2018-07-11 07:52:18.424722 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.115:6802/13323 -- osd_ping(ping e501 stamp 2018-07-11 07:52:18.424539) v4 -- 0x55d9c3383c00 con 0 > -1621> 2018-07-11 07:52:18.424735 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.115:6803/13323 -- osd_ping(ping e501 stamp 2018-07-11 07:52:18.424539) v4 -- 0x55d982cdf400 con 0 > -1620> 2018-07-11 07:52:18.424746 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6806/15885 -- osd_ping(ping e501 stamp 2018-07-11 07:52:18.424539) v4 -- 0x55d9b9ac2000 con 0 > -1619> 2018-07-11 07:52:18.424763 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6807/15885 -- osd_ping(ping e501 stamp 2018-07-11 07:52:18.424539) v4 -- 0x55d9b67d9e00 con 0 > -1618> 2018-07-11 07:52:18.424777 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.108:6802/11967 -- osd_ping(ping e501 stamp 2018-07-11 07:52:18.424539) v4 -- 0x55d980f5b200 con 0 > -1617> 2018-07-11 07:52:18.424788 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.108:6803/11967 -- osd_ping(ping e501 stamp 2018-07-11 07:52:18.424539) v4 -- 0x55d986c23e00 con 0 > -1616> 2018-07-11 07:52:18.424801 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.107:6806/12617 -- osd_ping(ping e501 stamp 2018-07-11 07:52:18.424539) v4 -- 0x55d9869be600 con 0 > -1615> 2018-07-11 07:52:18.424813 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.107:6807/12617 -- osd_ping(ping e501 stamp 2018-07-11 07:52:18.424539) v4 -- 0x55d9829e5800 con 0 > -1614> 2018-07-11 07:52:18.424825 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6802/15919 -- osd_ping(ping e501 stamp 2018-07-11 07:52:18.424539) v4 -- 0x55d9b2783400 con 0 > -1613> 2018-07-11 07:52:18.424837 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6803/15919 -- osd_ping(ping e501 stamp 2018-07-11 07:52:18.424539) v4 -- 0x55d9c3382e00 con 0 > -1612> 2018-07-11 07:52:18.425007 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.107:6806/12617 conn(0x55d986bd2000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.9 seq 256 0x55d982d31600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:18.424539) v4 > -1611> 2018-07-11 07:52:18.425028 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.9 10.8.128.107:6806/12617 256 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:18.424539) v4 ==== 2004+0+0 (2954031976 0 0) 0x55d982d31600 con 0x55d986bd2000 > -1610> 2018-07-11 07:52:18.425020 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.107:6807/12617 conn(0x55d986bd3800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.9 seq 256 0x55d9c9122a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:18.424539) v4 > -1609> 2018-07-11 07:52:18.425044 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.9 10.8.128.107:6807/12617 256 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:18.424539) v4 ==== 2004+0+0 (2954031976 0 0) 0x55d9c9122a00 con 0x55d986bd3800 > -1608> 2018-07-11 07:52:18.425072 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6803/15255 conn(0x55d986bc9800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.0 seq 256 0x55d982d31600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:18.424539) v4 > -1607> 2018-07-11 07:52:18.425086 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.0 10.8.128.113:6803/15255 256 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:18.424539) v4 ==== 2004+0+0 (2954031976 0 0) 0x55d982d31600 con 0x55d986bc9800 > -1606> 2018-07-11 07:52:18.425171 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6808/12651 conn(0x55d986b9a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.2 seq 256 0x55d982d31c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:18.424539) v4 > -1605> 2018-07-11 07:52:18.425190 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.2 10.8.128.106:6808/12651 256 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:18.424539) v4 ==== 2004+0+0 (2954031976 0 0) 0x55d982d31c00 con 0x55d986b9a000 > -1604> 2018-07-11 07:52:18.425180 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6806/15263 conn(0x55d986b9d000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.5 seq 256 0x55d9c9122a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:18.424539) v4 > -1603> 2018-07-11 07:52:18.425202 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.5 10.8.128.113:6806/15263 256 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:18.424539) v4 ==== 2004+0+0 (2954031976 0 0) 0x55d9c9122a00 con 0x55d986b9d000 > -1602> 2018-07-11 07:52:18.425206 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6810/12651 conn(0x55d986b9b800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.2 seq 256 0x55d982d31600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:18.424539) v4 > -1601> 2018-07-11 07:52:18.425226 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6802/15255 conn(0x55d986bc8000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.0 seq 256 0x55d982d31c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:18.424539) v4 > -1600> 2018-07-11 07:52:18.425232 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6809/12650 conn(0x55d986bcb000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 256 0x55d9c9122a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:18.424539) v4 > -1599> 2018-07-11 07:52:18.425231 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.2 10.8.128.106:6810/12651 256 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:18.424539) v4 ==== 2004+0+0 (2954031976 0 0) 0x55d982d31600 con 0x55d986b9b800 > -1598> 2018-07-11 07:52:18.425241 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.3 10.8.128.106:6809/12650 256 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:18.424539) v4 ==== 2004+0+0 (2954031976 0 0) 0x55d9c9122a00 con 0x55d986bcb000 > -1597> 2018-07-11 07:52:18.425238 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.0 10.8.128.113:6802/15255 256 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:18.424539) v4 ==== 2004+0+0 (2954031976 0 0) 0x55d982d31c00 con 0x55d986bc8000 > -1596> 2018-07-11 07:52:18.425293 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6807/12677 conn(0x55d986b59800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.1 seq 256 0x55d982d31c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:18.424539) v4 > -1595> 2018-07-11 07:52:18.425306 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.1 10.8.128.106:6807/12677 256 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:18.424539) v4 ==== 2004+0+0 (2954031976 0 0) 0x55d982d31c00 con 0x55d986b59800 > -1594> 2018-07-11 07:52:18.425311 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6806/12677 conn(0x55d986b58000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.1 seq 256 0x55d9c9122a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:18.424539) v4 > -1593> 2018-07-11 07:52:18.425327 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.1 10.8.128.106:6806/12677 256 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:18.424539) v4 ==== 2004+0+0 (2954031976 0 0) 0x55d9c9122a00 con 0x55d986b58000 > -1592> 2018-07-11 07:52:18.425335 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6807/15263 conn(0x55d986b9e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.5 seq 256 0x55d982d31c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:18.424539) v4 > -1591> 2018-07-11 07:52:18.425347 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.5 10.8.128.113:6807/15263 256 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:18.424539) v4 ==== 2004+0+0 (2954031976 0 0) 0x55d982d31c00 con 0x55d986b9e800 > -1590> 2018-07-11 07:52:18.425358 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6811/15287 conn(0x55d986bb0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 256 0x55d982d31600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:18.424539) v4 > -1589> 2018-07-11 07:52:18.425366 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6807/15885 conn(0x55d986bad800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.7 seq 256 0x55d9c9122a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:18.424539) v4 > -1588> 2018-07-11 07:52:18.425387 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.7 10.8.128.114:6807/15885 256 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:18.424539) v4 ==== 2004+0+0 (2954031976 0 0) 0x55d9c9122a00 con 0x55d986bad800 > -1587> 2018-07-11 07:52:18.425383 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.4 10.8.128.113:6811/15287 256 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:18.424539) v4 ==== 2004+0+0 (2954031976 0 0) 0x55d982d31600 con 0x55d986bb0800 > -1586> 2018-07-11 07:52:18.425395 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6810/15287 conn(0x55d986baf000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 256 0x55d982d31c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:18.424539) v4 > -1585> 2018-07-11 07:52:18.425408 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.4 10.8.128.113:6810/15287 256 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:18.424539) v4 ==== 2004+0+0 (2954031976 0 0) 0x55d982d31c00 con 0x55d986baf000 > -1584> 2018-07-11 07:52:18.425424 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.115:6803/13323 conn(0x55d986bc2800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 256 0x55d9c9122a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:18.424539) v4 > -1583> 2018-07-11 07:52:18.425437 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.6 10.8.128.115:6803/13323 256 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:18.424539) v4 ==== 2004+0+0 (2954031976 0 0) 0x55d9c9122a00 con 0x55d986bc2800 > -1582> 2018-07-11 07:52:18.425438 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6811/12650 conn(0x55d986bcc800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 256 0x55d982d31c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:18.424539) v4 > -1581> 2018-07-11 07:52:18.425448 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.3 10.8.128.106:6811/12650 256 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:18.424539) v4 ==== 2004+0+0 (2954031976 0 0) 0x55d982d31c00 con 0x55d986bcc800 > -1580> 2018-07-11 07:52:18.425442 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.115:6802/13323 conn(0x55d986bc1000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 256 0x55d982d31600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:18.424539) v4 > -1579> 2018-07-11 07:52:18.425458 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.6 10.8.128.115:6802/13323 256 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:18.424539) v4 ==== 2004+0+0 (2954031976 0 0) 0x55d982d31600 con 0x55d986bc1000 > -1578> 2018-07-11 07:52:18.425468 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.108:6802/11967 conn(0x55d986bbe000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.8 seq 256 0x55d9c9122a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:18.424539) v4 > -1577> 2018-07-11 07:52:18.425480 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.8 10.8.128.108:6802/11967 256 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:18.424539) v4 ==== 2004+0+0 (2954031976 0 0) 0x55d9c9122a00 con 0x55d986bbe000 > -1576> 2018-07-11 07:52:18.425503 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6806/15885 conn(0x55d986bac000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.7 seq 256 0x55d982d31600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:18.424539) v4 > -1575> 2018-07-11 07:52:18.425517 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.7 10.8.128.114:6806/15885 256 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:18.424539) v4 ==== 2004+0+0 (2954031976 0 0) 0x55d982d31600 con 0x55d986bac000 > -1574> 2018-07-11 07:52:18.425564 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6802/15919 conn(0x55d986b5b000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.11 seq 256 0x55d982d31600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:18.424539) v4 > -1573> 2018-07-11 07:52:18.425579 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.11 10.8.128.114:6802/15919 256 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:18.424539) v4 ==== 2004+0+0 (2954031976 0 0) 0x55d982d31600 con 0x55d986b5b000 > -1572> 2018-07-11 07:52:18.425583 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6803/15919 conn(0x55d986b5c800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.11 seq 256 0x55d9c9122a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:18.424539) v4 > -1571> 2018-07-11 07:52:18.425600 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.11 10.8.128.114:6803/15919 256 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:18.424539) v4 ==== 2004+0+0 (2954031976 0 0) 0x55d9c9122a00 con 0x55d986b5c800 > -1570> 2018-07-11 07:52:18.425593 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.108:6803/11967 conn(0x55d986bbf800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.8 seq 256 0x55d982d31c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:18.424539) v4 > -1569> 2018-07-11 07:52:18.425611 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.8 10.8.128.108:6803/11967 256 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:18.424539) v4 ==== 2004+0+0 (2954031976 0 0) 0x55d982d31c00 con 0x55d986bbf800 > -1568> 2018-07-11 07:52:18.600081 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1056 0x55d9c2a5e480 rep_scrubmap(5.2e e501 from shard 2) v1 > -1567> 2018-07-11 07:52:18.600101 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1056 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12478 (1269249638 0 161343654) 0x55d9c2a5e480 con 0x55d986bf0800 > -1566> 2018-07-11 07:52:18.656951 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1050 0x55d9c2a5e6c0 rep_scrubmap(5.2e e501 from shard 4) v1 > -1565> 2018-07-11 07:52:18.656971 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1050 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12478 (3686815630 0 161343654) 0x55d9c2a5e6c0 con 0x55d986bec000 > -1564> 2018-07-11 07:52:18.715394 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -1563> 2018-07-11 07:52:18.715420 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -1562> 2018-07-11 07:52:18.715467 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -1561> 2018-07-11 07:52:18.715930 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -1560> 2018-07-11 07:52:18.768039 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'66620,epoch:501/498,start:5:757d05d0:::benchmark_data_magna115_10867_object3535108:0,end:5:757d3450:::benchmark_data_magna115_10867_object1359053:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98696e000 con 0 > -1559> 2018-07-11 07:52:18.768091 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'66620,epoch:501/498,start:5:757d05d0:::benchmark_data_magna115_10867_object3535108:0,end:5:757d3450:::benchmark_data_magna115_10867_object1359053:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455680 con 0 > -1558> 2018-07-11 07:52:18.823198 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12650 conn(0x55d986d23000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 258 0x55d982d31600 osd_ping(ping e501 stamp 2018-07-11 07:52:18.871159) v4 > -1557> 2018-07-11 07:52:18.823202 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12650 conn(0x55d986d21800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 258 0x55d9c9122a00 osd_ping(ping e501 stamp 2018-07-11 07:52:18.871159) v4 > -1556> 2018-07-11 07:52:18.823220 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.3 10.8.128.106:0/12650 258 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:18.871159) v4 ==== 2004+0+0 (3023083250 0 0) 0x55d982d31600 con 0x55d986d23000 > -1555> 2018-07-11 07:52:18.823223 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.3 10.8.128.106:0/12650 258 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:18.871159) v4 ==== 2004+0+0 (3023083250 0 0) 0x55d9c9122a00 con 0x55d986d21800 > -1554> 2018-07-11 07:52:18.823237 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:18.871159) v4 -- 0x55d9b2783400 con 0 > -1553> 2018-07-11 07:52:18.823279 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:18.871159) v4 -- 0x55d9c3382e00 con 0 > -1552> 2018-07-11 07:52:18.852370 7ffacbbd1700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15263 conn(0x55d985d44000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.5 seq 246 0x55d9bdba5e00 osd_ping(ping e501 stamp 2018-07-11 07:52:20.084131) v4 > -1551> 2018-07-11 07:52:18.852367 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15263 conn(0x55d985d45800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.5 seq 246 0x55d9869d9800 osd_ping(ping e501 stamp 2018-07-11 07:52:20.084131) v4 > -1550> 2018-07-11 07:52:18.852395 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 <== osd.5 10.8.128.113:0/15263 246 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:20.084131) v4 ==== 2004+0+0 (2965101207 0 0) 0x55d9bdba5e00 con 0x55d985d44000 > -1549> 2018-07-11 07:52:18.852398 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.5 10.8.128.113:0/15263 246 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:20.084131) v4 ==== 2004+0+0 (2965101207 0 0) 0x55d9869d9800 con 0x55d985d45800 > -1548> 2018-07-11 07:52:18.852406 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:20.084131) v4 -- 0x55d9c3382e00 con 0 > -1547> 2018-07-11 07:52:18.852428 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:20.084131) v4 -- 0x55d9b2783400 con 0 > -1546> 2018-07-11 07:52:18.942029 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1057 0x55d9c2a5e900 rep_scrubmap(5.2e e501 from shard 2) v1 > -1545> 2018-07-11 07:52:18.942048 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1057 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12478 (1269249638 0 4126234282) 0x55d9c2a5e900 con 0x55d986bf0800 > -1544> 2018-07-11 07:52:18.967820 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6801/15919 conn(0x55d986c3f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=29 cs=1 l=0). rx osd.11 seq 252 0x55d9c9122a00 MOSDScrubReserve(10.f REQUEST e501) v1 > -1543> 2018-07-11 07:52:18.967840 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.11 10.8.128.114:6801/15919 252 ==== MOSDScrubReserve(10.f REQUEST e501) v1 ==== 43+0+0 (2892966580 0 0) 0x55d9c9122a00 con 0x55d986c3f800 > -1542> 2018-07-11 07:52:18.967872 7ffab53e3700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.114:6801/15919 -- MOSDScrubReserve(10.f REJECT e501) v1 -- 0x55d9b4388200 con 0 > -1541> 2018-07-11 07:52:18.968202 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6801/15919 conn(0x55d986c3f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=29 cs=1 l=0). rx osd.11 seq 253 0x55d9b4388200 MOSDScrubReserve(10.f RELEASE e501) v1 > -1540> 2018-07-11 07:52:18.968216 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.11 10.8.128.114:6801/15919 253 ==== MOSDScrubReserve(10.f RELEASE e501) v1 ==== 43+0+0 (1668468918 0 0) 0x55d9b4388200 con 0x55d986c3f800 > -1539> 2018-07-11 07:52:18.992367 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1051 0x55d9c2a5eb40 rep_scrubmap(5.2e e501 from shard 4) v1 > -1538> 2018-07-11 07:52:18.992387 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1051 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12478 (3686815630 0 4126234282) 0x55d9c2a5eb40 con 0x55d986bec000 > -1537> 2018-07-11 07:52:19.121297 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -1536> 2018-07-11 07:52:19.121324 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -1535> 2018-07-11 07:52:19.121382 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -1534> 2018-07-11 07:52:19.121845 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -1533> 2018-07-11 07:52:19.168254 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67645,epoch:501/498,start:5:757d3450:::benchmark_data_magna115_10867_object1359053:0,end:5:757d98b0:::benchmark_data_magna115_10867_object1367490:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455900 con 0 > -1532> 2018-07-11 07:52:19.168293 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67645,epoch:501/498,start:5:757d3450:::benchmark_data_magna115_10867_object1359053:0,end:5:757d98b0:::benchmark_data_magna115_10867_object1367490:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455b80 con 0 > -1531> 2018-07-11 07:52:19.380399 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1058 0x55d9c2a5ed80 rep_scrubmap(5.2e e501 from shard 2) v1 > -1530> 2018-07-11 07:52:19.380420 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1058 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12474 (1269249638 0 3344126670) 0x55d9c2a5ed80 con 0x55d986bf0800 > -1529> 2018-07-11 07:52:19.419611 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1052 0x55d9c2a5efc0 rep_scrubmap(5.2e e501 from shard 4) v1 > -1528> 2018-07-11 07:52:19.419631 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1052 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12474 (3686815630 0 3344126670) 0x55d9c2a5efc0 con 0x55d986bec000 > -1527> 2018-07-11 07:52:19.430911 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -1526> 2018-07-11 07:52:19.430934 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -1525> 2018-07-11 07:52:19.430988 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -1524> 2018-07-11 07:52:19.431434 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -1523> 2018-07-11 07:52:19.470264 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:757d98b0:::benchmark_data_magna115_10867_object1367490:0,end:5:757df210:::benchmark_data_magna115_10867_object3791792:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b14f0500 con 0 > -1522> 2018-07-11 07:52:19.470304 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:757d98b0:::benchmark_data_magna115_10867_object1367490:0,end:5:757df210:::benchmark_data_magna115_10867_object3791792:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498000 con 0 > -1521> 2018-07-11 07:52:19.570301 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6805/15885 conn(0x55d986c7f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.7 seq 2824 0x55d986d40600 MOSDScrubReserve(5.4 REQUEST e501) v1 > -1520> 2018-07-11 07:52:19.570319 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.7 10.8.128.114:6805/15885 2824 ==== MOSDScrubReserve(5.4 REQUEST e501) v1 ==== 43+0+0 (1753221512 0 0) 0x55d986d40600 con 0x55d986c7f800 > -1519> 2018-07-11 07:52:19.570438 7ffab33df700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.114:6805/15885 -- MOSDScrubReserve(5.4 REJECT e501) v1 -- 0x55d9829e5600 con 0 > -1518> 2018-07-11 07:52:19.570773 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6805/15885 conn(0x55d986c7f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.7 seq 2825 0x55d9829e5600 MOSDScrubReserve(5.4 RELEASE e501) v1 > -1517> 2018-07-11 07:52:19.570787 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.7 10.8.128.114:6805/15885 2825 ==== MOSDScrubReserve(5.4 RELEASE e501) v1 ==== 43+0+0 (2812146570 0 0) 0x55d9829e5600 con 0x55d986c7f800 > -1516> 2018-07-11 07:52:19.677972 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1059 0x55d9c2a5f200 rep_scrubmap(5.2e e501 from shard 2) v1 > -1515> 2018-07-11 07:52:19.677993 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1059 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12468 (1269249638 0 322864300) 0x55d9c2a5f200 con 0x55d986bf0800 > -1514> 2018-07-11 07:52:19.688426 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1053 0x55d9c2a5f440 rep_scrubmap(5.2e e501 from shard 4) v1 > -1513> 2018-07-11 07:52:19.688447 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1053 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12468 (3686815630 0 322864300) 0x55d9c2a5f440 con 0x55d986bec000 > -1512> 2018-07-11 07:52:19.691543 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15287 conn(0x55d981240800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 279 0x55d982d31c00 osd_ping(ping e501 stamp 2018-07-11 07:52:20.923214) v4 > -1511> 2018-07-11 07:52:19.691566 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.4 10.8.128.113:0/15287 279 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:20.923214) v4 ==== 2004+0+0 (2823169838 0 0) 0x55d982d31c00 con 0x55d981240800 > -1510> 2018-07-11 07:52:19.691576 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:20.923214) v4 -- 0x55d986c23a00 con 0 > -1509> 2018-07-11 07:52:19.691601 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15287 conn(0x55d98123f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 279 0x55d982d31600 osd_ping(ping e501 stamp 2018-07-11 07:52:20.923214) v4 > -1508> 2018-07-11 07:52:19.691627 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.4 10.8.128.113:0/15287 279 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:20.923214) v4 ==== 2004+0+0 (2823169838 0 0) 0x55d982d31600 con 0x55d98123f000 > -1507> 2018-07-11 07:52:19.691643 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:20.923214) v4 -- 0x55d9b2783400 con 0 > -1506> 2018-07-11 07:52:19.776764 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -1505> 2018-07-11 07:52:19.776787 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -1504> 2018-07-11 07:52:19.776844 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -1503> 2018-07-11 07:52:19.777320 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -1502> 2018-07-11 07:52:19.794963 7ffacbbd1700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.108:0/11967 conn(0x55d986d82800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.8 seq 247 0x55d9bdba5e00 osd_ping(ping e501 stamp 2018-07-11 07:52:19.550836) v4 > -1501> 2018-07-11 07:52:19.794967 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.108:0/11967 conn(0x55d986db0000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.8 seq 247 0x55d9869d9800 osd_ping(ping e501 stamp 2018-07-11 07:52:19.550836) v4 > -1500> 2018-07-11 07:52:19.794984 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 <== osd.8 10.8.128.108:0/11967 247 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:19.550836) v4 ==== 2004+0+0 (424838396 0 0) 0x55d9bdba5e00 con 0x55d986d82800 > -1499> 2018-07-11 07:52:19.794987 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.8 10.8.128.108:0/11967 247 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:19.550836) v4 ==== 2004+0+0 (424838396 0 0) 0x55d9869d9800 con 0x55d986db0000 > -1498> 2018-07-11 07:52:19.794994 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:19.550836) v4 -- 0x55d9829e5800 con 0 > -1497> 2018-07-11 07:52:19.795026 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:19.550836) v4 -- 0x55d9b2783400 con 0 > -1496> 2018-07-11 07:52:19.818481 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67612,epoch:501/498,start:5:757df210:::benchmark_data_magna115_10867_object3791792:0,end:5:757e56d3:::benchmark_data_magna115_10867_object148797:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454780 con 0 > -1495> 2018-07-11 07:52:19.818521 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67612,epoch:501/498,start:5:757df210:::benchmark_data_magna115_10867_object3791792:0,end:5:757e56d3:::benchmark_data_magna115_10867_object148797:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498280 con 0 > -1494> 2018-07-11 07:52:19.829110 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12651 conn(0x55d986db3000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.2 seq 263 0x55d9c3382e00 osd_ping(ping e501 stamp 2018-07-11 07:52:19.877291) v4 > -1493> 2018-07-11 07:52:19.829139 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.2 10.8.128.106:0/12651 263 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:19.877291) v4 ==== 2004+0+0 (58578236 0 0) 0x55d9c3382e00 con 0x55d986db3000 > -1492> 2018-07-11 07:52:19.829152 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:19.877291) v4 -- 0x55d9829e5800 con 0 > -1491> 2018-07-11 07:52:19.829163 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12651 conn(0x55d986db1800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.2 seq 263 0x55d982d31c00 osd_ping(ping e501 stamp 2018-07-11 07:52:19.877291) v4 > -1490> 2018-07-11 07:52:19.829188 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.2 10.8.128.106:0/12651 263 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:19.877291) v4 ==== 2004+0+0 (58578236 0 0) 0x55d982d31c00 con 0x55d986db1800 > -1489> 2018-07-11 07:52:19.829205 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:19.877291) v4 -- 0x55d986c23a00 con 0 > -1488> 2018-07-11 07:52:19.845875 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 3032 0x55d9bdba5e00 MOSDScrubReserve(11.3s5 REQUEST e501) v1 > -1487> 2018-07-11 07:52:19.845894 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 3032 ==== MOSDScrubReserve(11.3s5 REQUEST e501) v1 ==== 43+0+0 (3260170403 0 0) 0x55d9bdba5e00 con 0x55d986c7e000 > -1486> 2018-07-11 07:52:19.845934 7ffab3be0700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e501) v1 -- 0x55d9bcd72e00 con 0 > -1485> 2018-07-11 07:52:19.846279 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 3033 0x55d9829e5800 MOSDScrubReserve(11.3s5 RELEASE e501) v1 > -1484> 2018-07-11 07:52:19.846294 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 3033 ==== MOSDScrubReserve(11.3s5 RELEASE e501) v1 ==== 43+0+0 (223328929 0 0) 0x55d9829e5800 con 0x55d986c7e000 > -1483> 2018-07-11 07:52:19.968052 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6801/15919 conn(0x55d986c3f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=29 cs=1 l=0). rx osd.11 seq 254 0x55d9c3382e00 MOSDScrubReserve(10.f REQUEST e501) v1 > -1482> 2018-07-11 07:52:19.968071 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.11 10.8.128.114:6801/15919 254 ==== MOSDScrubReserve(10.f REQUEST e501) v1 ==== 43+0+0 (2892966580 0 0) 0x55d9c3382e00 con 0x55d986c3f800 > -1481> 2018-07-11 07:52:19.968136 7ffab53e3700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.114:6801/15919 -- MOSDScrubReserve(10.f REJECT e501) v1 -- 0x55d9b4388200 con 0 > -1480> 2018-07-11 07:52:19.968473 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6801/15919 conn(0x55d986c3f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=29 cs=1 l=0). rx osd.11 seq 255 0x55d9b4388200 MOSDScrubReserve(10.f RELEASE e501) v1 > -1479> 2018-07-11 07:52:19.968489 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.11 10.8.128.114:6801/15919 255 ==== MOSDScrubReserve(10.f RELEASE e501) v1 ==== 43+0+0 (1668468918 0 0) 0x55d9b4388200 con 0x55d986c3f800 > -1478> 2018-07-11 07:52:20.027253 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1060 0x55d9c2a5f680 rep_scrubmap(5.2e e501 from shard 2) v1 > -1477> 2018-07-11 07:52:20.027272 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1060 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12474 (1269249638 0 84270430) 0x55d9c2a5f680 con 0x55d986bf0800 > -1476> 2018-07-11 07:52:20.058508 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1054 0x55d9c2a5f8c0 rep_scrubmap(5.2e e501 from shard 4) v1 > -1475> 2018-07-11 07:52:20.058528 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1054 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12474 (3686815630 0 84270430) 0x55d9c2a5f8c0 con 0x55d986bec000 > -1474> 2018-07-11 07:52:20.116022 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -1473> 2018-07-11 07:52:20.116050 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -1472> 2018-07-11 07:52:20.116107 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -1471> 2018-07-11 07:52:20.116589 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -1470> 2018-07-11 07:52:20.160109 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67309,epoch:501/498,start:5:757e56d3:::benchmark_data_magna115_10867_object148797:0,end:5:757eb5c1:::benchmark_data_magna115_11707_object158846:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455400 con 0 > -1469> 2018-07-11 07:52:20.160160 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67309,epoch:501/498,start:5:757e56d3:::benchmark_data_magna115_10867_object148797:0,end:5:757eb5c1:::benchmark_data_magna115_11707_object158846:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454500 con 0 > -1468> 2018-07-11 07:52:20.323567 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.114:0/15885 conn(0x55d986d81000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.7 seq 270 0x55d982d31c00 osd_ping(ping e501 stamp 2018-07-11 07:52:20.954067) v4 > -1467> 2018-07-11 07:52:20.323567 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.114:0/15885 conn(0x55d986d7f800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.7 seq 270 0x55d982d31600 osd_ping(ping e501 stamp 2018-07-11 07:52:20.954067) v4 > -1466> 2018-07-11 07:52:20.323599 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.7 10.8.128.114:0/15885 270 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:20.954067) v4 ==== 2004+0+0 (2706689752 0 0) 0x55d982d31c00 con 0x55d986d81000 > -1465> 2018-07-11 07:52:20.323603 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.7 10.8.128.114:0/15885 270 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:20.954067) v4 ==== 2004+0+0 (2706689752 0 0) 0x55d982d31600 con 0x55d986d7f800 > -1464> 2018-07-11 07:52:20.323619 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:20.954067) v4 -- 0x55d986c23a00 con 0 > -1463> 2018-07-11 07:52:20.323649 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:20.954067) v4 -- 0x55d9b2783400 con 0 > -1462> 2018-07-11 07:52:20.351802 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1055 0x55d9c2a5fb00 rep_scrubmap(5.2e e501 from shard 4) v1 > -1461> 2018-07-11 07:52:20.351822 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1055 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12478 (3686815630 0 2222569222) 0x55d9c2a5fb00 con 0x55d986bec000 > -1460> 2018-07-11 07:52:20.386237 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1061 0x55d9c2a5fd40 rep_scrubmap(5.2e e501 from shard 2) v1 > -1459> 2018-07-11 07:52:20.386267 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1061 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12478 (1269249638 0 2222569222) 0x55d9c2a5fd40 con 0x55d986bf0800 > -1458> 2018-07-11 07:52:20.436375 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -1457> 2018-07-11 07:52:20.436401 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -1456> 2018-07-11 07:52:20.436443 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -1455> 2018-07-11 07:52:20.436903 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -1454> 2018-07-11 07:52:20.437988 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:757eb5c1:::benchmark_data_magna115_11707_object158846:0,end:5:757f1d6b:::benchmark_data_magna115_10867_object2200556:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454f00 con 0 > -1453> 2018-07-11 07:52:20.438035 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:757eb5c1:::benchmark_data_magna115_11707_object158846:0,end:5:757f1d6b:::benchmark_data_magna115_10867_object2200556:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455180 con 0 > -1452> 2018-07-11 07:52:20.653006 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1056 0x55d9bd688000 rep_scrubmap(5.2e e501 from shard 4) v1 > -1451> 2018-07-11 07:52:20.653026 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1056 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12478 (3686815630 0 2932721650) 0x55d9bd688000 con 0x55d986bec000 > -1450> 2018-07-11 07:52:20.662966 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1062 0x55d9bd688240 rep_scrubmap(5.2e e501 from shard 2) v1 > -1449> 2018-07-11 07:52:20.662986 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1062 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12478 (1269249638 0 2932721650) 0x55d9bd688240 con 0x55d986bf0800 > -1448> 2018-07-11 07:52:20.689434 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -1447> 2018-07-11 07:52:20.689457 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -1446> 2018-07-11 07:52:20.689512 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -1445> 2018-07-11 07:52:20.689959 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -1444> 2018-07-11 07:52:20.735511 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'66819,epoch:501/498,start:5:757f1d6b:::benchmark_data_magna115_10867_object2200556:0,end:5:757f9cbd:::benchmark_data_magna115_10867_object2534393:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454a00 con 0 > -1443> 2018-07-11 07:52:20.735563 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'66819,epoch:501/498,start:5:757f1d6b:::benchmark_data_magna115_10867_object2200556:0,end:5:757f9cbd:::benchmark_data_magna115_10867_object2534393:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454c80 con 0 > -1442> 2018-07-11 07:52:20.956724 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1063 0x55d9bd688480 rep_scrubmap(5.2e e501 from shard 2) v1 > -1441> 2018-07-11 07:52:20.956743 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1063 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12472 (1269249638 0 3598234803) 0x55d9bd688480 con 0x55d986bf0800 > -1440> 2018-07-11 07:52:20.964148 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1057 0x55d9bd6886c0 rep_scrubmap(5.2e e501 from shard 4) v1 > -1439> 2018-07-11 07:52:20.964168 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1057 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12472 (3686815630 0 3598234803) 0x55d9bd6886c0 con 0x55d986bec000 > -1438> 2018-07-11 07:52:21.046884 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -1437> 2018-07-11 07:52:21.046909 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -1436> 2018-07-11 07:52:21.046965 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -1435> 2018-07-11 07:52:21.047450 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -1434> 2018-07-11 07:52:21.070310 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12677 conn(0x55d986df2000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.1 seq 269 0x55d986c23e00 osd_ping(ping e501 stamp 2018-07-11 07:52:21.118468) v4 > -1433> 2018-07-11 07:52:21.070312 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12677 conn(0x55d986db4800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.1 seq 269 0x55d9869d9800 osd_ping(ping e501 stamp 2018-07-11 07:52:21.118468) v4 > -1432> 2018-07-11 07:52:21.070334 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.1 10.8.128.106:0/12677 269 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:21.118468) v4 ==== 2004+0+0 (3125621725 0 0) 0x55d986c23e00 con 0x55d986df2000 > -1431> 2018-07-11 07:52:21.070337 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.1 10.8.128.106:0/12677 269 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:21.118468) v4 ==== 2004+0+0 (3125621725 0 0) 0x55d9869d9800 con 0x55d986db4800 > -1430> 2018-07-11 07:52:21.070346 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:21.118468) v4 -- 0x55d986c23a00 con 0 > -1429> 2018-07-11 07:52:21.070373 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:21.118468) v4 -- 0x55d9b2783400 con 0 > -1428> 2018-07-11 07:52:21.102158 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:757f9cbd:::benchmark_data_magna115_10867_object2534393:0,end:5:757fe790:::benchmark_data_magna115_10867_object623478:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb773b80 con 0 > -1427> 2018-07-11 07:52:21.102210 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:757f9cbd:::benchmark_data_magna115_10867_object2534393:0,end:5:757fe790:::benchmark_data_magna115_10867_object623478:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454280 con 0 > -1426> 2018-07-11 07:52:21.157370 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.107:0/12617 conn(0x55d98123d800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 257 0x55d9bcd72e00 osd_ping(ping e501 stamp 2018-07-11 07:52:21.156994) v4 > -1425> 2018-07-11 07:52:21.157372 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.107:0/12617 conn(0x55d98123c000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 257 0x55d986c23e00 osd_ping(ping e501 stamp 2018-07-11 07:52:21.156994) v4 > -1424> 2018-07-11 07:52:21.157392 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.9 10.8.128.107:0/12617 257 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:21.156994) v4 ==== 2004+0+0 (1707008439 0 0) 0x55d9bcd72e00 con 0x55d98123d800 > -1423> 2018-07-11 07:52:21.157392 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.9 10.8.128.107:0/12617 257 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:21.156994) v4 ==== 2004+0+0 (1707008439 0 0) 0x55d986c23e00 con 0x55d98123c000 > -1422> 2018-07-11 07:52:21.157402 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:21.156994) v4 -- 0x55d9b67d9e00 con 0 > -1421> 2018-07-11 07:52:21.157429 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:21.156994) v4 -- 0x55d986c23a00 con 0 > -1420> 2018-07-11 07:52:21.332094 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1058 0x55d9bd688900 rep_scrubmap(5.2e e501 from shard 4) v1 > -1419> 2018-07-11 07:52:21.332113 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1058 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12484 (3686815630 0 2089002269) 0x55d9bd688900 con 0x55d986bec000 > -1418> 2018-07-11 07:52:21.334970 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1064 0x55d9bd688b40 rep_scrubmap(5.2e e501 from shard 2) v1 > -1417> 2018-07-11 07:52:21.334989 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1064 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12484 (1269249638 0 2089002269) 0x55d9bd688b40 con 0x55d986bf0800 > -1416> 2018-07-11 07:52:21.398646 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.114:0/15919 conn(0x55d986df3800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.11 seq 267 0x55d980f5b200 osd_ping(ping e501 stamp 2018-07-11 07:52:22.029184) v4 > -1415> 2018-07-11 07:52:21.398673 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.11 10.8.128.114:0/15919 267 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:22.029184) v4 ==== 2004+0+0 (507375831 0 0) 0x55d980f5b200 con 0x55d986df3800 > -1414> 2018-07-11 07:52:21.398693 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:22.029184) v4 -- 0x55d9b67d9e00 con 0 > -1413> 2018-07-11 07:52:21.398688 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.114:0/15919 conn(0x55d986df5000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.11 seq 267 0x55d982d31600 osd_ping(ping e501 stamp 2018-07-11 07:52:22.029184) v4 > -1412> 2018-07-11 07:52:21.398718 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.11 10.8.128.114:0/15919 267 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:22.029184) v4 ==== 2004+0+0 (507375831 0 0) 0x55d982d31600 con 0x55d986df5000 > -1411> 2018-07-11 07:52:21.398738 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:22.029184) v4 -- 0x55d9b2783400 con 0 > -1410> 2018-07-11 07:52:21.414225 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -1409> 2018-07-11 07:52:21.414259 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -1408> 2018-07-11 07:52:21.414306 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -1407> 2018-07-11 07:52:21.414768 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -1406> 2018-07-11 07:52:21.415833 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:757fe790:::benchmark_data_magna115_10867_object623478:0,end:5:75804e2d:::benchmark_data_magna115_10867_object2633008:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454000 con 0 > -1405> 2018-07-11 07:52:21.415880 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:757fe790:::benchmark_data_magna115_10867_object623478:0,end:5:75804e2d:::benchmark_data_magna115_10867_object2633008:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb773400 con 0 > -1404> 2018-07-11 07:52:21.570506 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6805/15885 conn(0x55d986c7f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.7 seq 2826 0x55d986c23e00 MOSDScrubReserve(5.4 REQUEST e501) v1 > -1403> 2018-07-11 07:52:21.570525 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.7 10.8.128.114:6805/15885 2826 ==== MOSDScrubReserve(5.4 REQUEST e501) v1 ==== 43+0+0 (1753221512 0 0) 0x55d986c23e00 con 0x55d986c7f800 > -1402> 2018-07-11 07:52:21.570594 7ffab33df700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.114:6805/15885 -- MOSDScrubReserve(5.4 REJECT e501) v1 -- 0x55d9829e5600 con 0 > -1401> 2018-07-11 07:52:21.570916 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6805/15885 conn(0x55d986c7f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.7 seq 2827 0x55d9829e5600 MOSDScrubReserve(5.4 RELEASE e501) v1 > -1400> 2018-07-11 07:52:21.570930 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.7 10.8.128.114:6805/15885 2827 ==== MOSDScrubReserve(5.4 RELEASE e501) v1 ==== 43+0+0 (2812146570 0 0) 0x55d9829e5600 con 0x55d986c7f800 > -1399> 2018-07-11 07:52:21.612990 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1065 0x55d9bd688d80 rep_scrubmap(5.2e e501 from shard 2) v1 > -1398> 2018-07-11 07:52:21.613013 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1065 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12462 (1269249638 0 1108989476) 0x55d9bd688d80 con 0x55d986bf0800 > -1397> 2018-07-11 07:52:21.631928 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1059 0x55d9bd688fc0 rep_scrubmap(5.2e e501 from shard 4) v1 > -1396> 2018-07-11 07:52:21.631947 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1059 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12462 (3686815630 0 1108989476) 0x55d9bd688fc0 con 0x55d986bec000 > -1395> 2018-07-11 07:52:21.657819 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.107:0/12617 conn(0x55d98123d800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 258 0x55d9bcd72e00 osd_ping(ping e501 stamp 2018-07-11 07:52:21.657488) v4 > -1394> 2018-07-11 07:52:21.657828 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.107:0/12617 conn(0x55d98123c000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 258 0x55d982d31c00 osd_ping(ping e501 stamp 2018-07-11 07:52:21.657488) v4 > -1393> 2018-07-11 07:52:21.657841 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.9 10.8.128.107:0/12617 258 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:21.657488) v4 ==== 2004+0+0 (1312813391 0 0) 0x55d9bcd72e00 con 0x55d98123d800 > -1392> 2018-07-11 07:52:21.657846 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.9 10.8.128.107:0/12617 258 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:21.657488) v4 ==== 2004+0+0 (1312813391 0 0) 0x55d982d31c00 con 0x55d98123c000 > -1391> 2018-07-11 07:52:21.657854 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:21.657488) v4 -- 0x55d9b67d9e00 con 0 > -1390> 2018-07-11 07:52:21.657879 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:21.657488) v4 -- 0x55d9829e5a00 con 0 > -1389> 2018-07-11 07:52:21.684847 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -1388> 2018-07-11 07:52:21.684872 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -1387> 2018-07-11 07:52:21.684915 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -1386> 2018-07-11 07:52:21.685382 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -1385> 2018-07-11 07:52:21.702405 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75804e2d:::benchmark_data_magna115_10867_object2633008:0,end:5:7580c019:::benchmark_data_magna115_10867_object691092:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb773680 con 0 > -1384> 2018-07-11 07:52:21.702443 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75804e2d:::benchmark_data_magna115_10867_object2633008:0,end:5:7580c019:::benchmark_data_magna115_10867_object691092:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb773180 con 0 > -1383> 2018-07-11 07:52:21.846261 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 3034 0x55d980f5b200 MOSDScrubReserve(11.3s5 REQUEST e501) v1 > -1382> 2018-07-11 07:52:21.846285 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 3034 ==== MOSDScrubReserve(11.3s5 REQUEST e501) v1 ==== 43+0+0 (3260170403 0 0) 0x55d980f5b200 con 0x55d986c7e000 > -1381> 2018-07-11 07:52:21.846416 7ffab3be0700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e501) v1 -- 0x55d9829e5800 con 0 > -1380> 2018-07-11 07:52:21.846686 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 3035 0x55d9b67d9e00 MOSDScrubReserve(11.3s5 RELEASE e501) v1 > -1379> 2018-07-11 07:52:21.846700 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 3035 ==== MOSDScrubReserve(11.3s5 RELEASE e501) v1 ==== 43+0+0 (223328929 0 0) 0x55d9b67d9e00 con 0x55d986c7e000 > -1378> 2018-07-11 07:52:21.937733 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1066 0x55d9bd689200 rep_scrubmap(5.2e e501 from shard 2) v1 > -1377> 2018-07-11 07:52:21.937754 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1066 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12470 (1269249638 0 388887045) 0x55d9bd689200 con 0x55d986bf0800 > -1376> 2018-07-11 07:52:21.945826 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1060 0x55d9bd689440 rep_scrubmap(5.2e e501 from shard 4) v1 > -1375> 2018-07-11 07:52:21.945845 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1060 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12470 (3686815630 0 388887045) 0x55d9bd689440 con 0x55d986bec000 > -1374> 2018-07-11 07:52:22.095428 7ffacbbd1700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.108:0/11967 conn(0x55d986d82800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.8 seq 248 0x55d9829e5800 osd_ping(ping e501 stamp 2018-07-11 07:52:21.851308) v4 > -1373> 2018-07-11 07:52:22.095430 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.108:0/11967 conn(0x55d986db0000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.8 seq 248 0x55d9869d9800 osd_ping(ping e501 stamp 2018-07-11 07:52:21.851308) v4 > -1372> 2018-07-11 07:52:22.095459 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 <== osd.8 10.8.128.108:0/11967 248 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:21.851308) v4 ==== 2004+0+0 (2959848342 0 0) 0x55d9829e5800 con 0x55d986d82800 > -1371> 2018-07-11 07:52:22.095462 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.8 10.8.128.108:0/11967 248 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:21.851308) v4 ==== 2004+0+0 (2959848342 0 0) 0x55d9869d9800 con 0x55d986db0000 > -1370> 2018-07-11 07:52:22.095469 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:21.851308) v4 -- 0x55d9bcd72e00 con 0 > -1369> 2018-07-11 07:52:22.095492 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:21.851308) v4 -- 0x55d9b2783400 con 0 > -1368> 2018-07-11 07:52:22.107623 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.115:0/13323 conn(0x55d986d7e000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 265 0x55d982cdf400 osd_ping(ping e501 stamp 2018-07-11 07:52:22.453702) v4 > -1367> 2018-07-11 07:52:22.107645 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.6 10.8.128.115:0/13323 265 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:22.453702) v4 ==== 2004+0+0 (3859443484 0 0) 0x55d982cdf400 con 0x55d986d7e000 > -1366> 2018-07-11 07:52:22.107654 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:22.453702) v4 -- 0x55d9bcd72e00 con 0 > -1365> 2018-07-11 07:52:22.107660 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.115:0/13323 conn(0x55d986d24800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 265 0x55d982d31c00 osd_ping(ping e501 stamp 2018-07-11 07:52:22.453702) v4 > -1364> 2018-07-11 07:52:22.107685 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.6 10.8.128.115:0/13323 265 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:22.453702) v4 ==== 2004+0+0 (3859443484 0 0) 0x55d982d31c00 con 0x55d986d24800 > -1363> 2018-07-11 07:52:22.107697 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:22.453702) v4 -- 0x55d9829e5a00 con 0 > -1362> 2018-07-11 07:52:22.302462 7ffab7be8700 10 monclient: tick > -1361> 2018-07-11 07:52:22.302475 7ffab7be8700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2018-07-11 07:51:52.302473) > -1360> 2018-07-11 07:52:22.353206 7ffab6be6700 1 -- 10.8.128.107:6800/13927 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55d9ba81b8c0 con 0 > -1359> 2018-07-11 07:52:22.353283 7ffab6be6700 1 -- 10.8.128.107:6800/13927 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55d9b3fdedc0 con 0 > -1358> 2018-07-11 07:52:22.608066 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.115:0/13323 conn(0x55d986d7e000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 266 0x55d9829e5800 osd_ping(ping e501 stamp 2018-07-11 07:52:22.954151) v4 > -1357> 2018-07-11 07:52:22.608088 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.6 10.8.128.115:0/13323 266 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:22.954151) v4 ==== 2004+0+0 (2821630134 0 0) 0x55d9829e5800 con 0x55d986d7e000 > -1356> 2018-07-11 07:52:22.608097 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:22.954151) v4 -- 0x55d9bcd72e00 con 0 > -1355> 2018-07-11 07:52:22.608098 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.115:0/13323 conn(0x55d986d24800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 266 0x55d982d31c00 osd_ping(ping e501 stamp 2018-07-11 07:52:22.954151) v4 > -1354> 2018-07-11 07:52:22.608119 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.6 10.8.128.115:0/13323 266 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:22.954151) v4 ==== 2004+0+0 (2821630134 0 0) 0x55d982d31c00 con 0x55d986d24800 > -1353> 2018-07-11 07:52:22.608130 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:22.954151) v4 -- 0x55d9829e5a00 con 0 > -1352> 2018-07-11 07:52:22.862199 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 34] Generated table #36812: 2658304 keys, 74596892 bytes > -1351> 2018-07-11 07:52:22.862241 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295542862215, "cf_name": "default", "job": 34, "event": "table_file_creation", "file_number": 36812, "file_size": 74596892, "table_properties": {"data_size": 67112830, "index_size": 689658, "filter_size": 6793412, "raw_key_size": 76795453, "raw_average_key_size": 28, "raw_value_size": 31095052, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2658304, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "42832", "kMergeOperands": "0"}} > -1350> 2018-07-11 07:52:22.873121 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15255 conn(0x55d986d20000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.0 seq 265 0x55d986c23a00 osd_ping(ping e501 stamp 2018-07-11 07:52:24.104814) v4 > -1349> 2018-07-11 07:52:22.873127 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15255 conn(0x55d986c82800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.0 seq 265 0x55d982d31600 osd_ping(ping e501 stamp 2018-07-11 07:52:24.104814) v4 > -1348> 2018-07-11 07:52:22.873148 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.0 10.8.128.113:0/15255 265 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:24.104814) v4 ==== 2004+0+0 (1197809645 0 0) 0x55d986c23a00 con 0x55d986d20000 > -1347> 2018-07-11 07:52:22.873150 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.0 10.8.128.113:0/15255 265 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:24.104814) v4 ==== 2004+0+0 (1197809645 0 0) 0x55d982d31600 con 0x55d986c82800 > -1346> 2018-07-11 07:52:22.873160 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:24.104814) v4 -- 0x55d9829e5a00 con 0 > -1345> 2018-07-11 07:52:22.873184 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:24.104814) v4 -- 0x55d9b2783400 con 0 > -1344> 2018-07-11 07:52:22.952674 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15263 conn(0x55d985d45800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.5 seq 247 0x55d9869d9800 osd_ping(ping e501 stamp 2018-07-11 07:52:24.184535) v4 > -1343> 2018-07-11 07:52:22.952675 7ffacbbd1700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15263 conn(0x55d985d44000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.5 seq 247 0x55d982cdf400 osd_ping(ping e501 stamp 2018-07-11 07:52:24.184535) v4 > -1342> 2018-07-11 07:52:22.952698 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.5 10.8.128.113:0/15263 247 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:24.184535) v4 ==== 2004+0+0 (884293653 0 0) 0x55d9869d9800 con 0x55d985d45800 > -1341> 2018-07-11 07:52:22.952704 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 <== osd.5 10.8.128.113:0/15263 247 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:24.184535) v4 ==== 2004+0+0 (884293653 0 0) 0x55d982cdf400 con 0x55d985d44000 > -1340> 2018-07-11 07:52:22.952711 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:24.184535) v4 -- 0x55d9b2783400 con 0 > -1339> 2018-07-11 07:52:22.952736 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:24.184535) v4 -- 0x55d9bcd72e00 con 0 > -1338> 2018-07-11 07:52:23.023783 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -1337> 2018-07-11 07:52:23.023811 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -1336> 2018-07-11 07:52:23.023871 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -1335> 2018-07-11 07:52:23.024365 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -1334> 2018-07-11 07:52:23.063362 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'66621,epoch:501/498,start:5:7580c019:::benchmark_data_magna115_10867_object691092:0,end:5:75810f9f:::benchmark_data_magna115_10867_object2613219:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb772780 con 0 > -1333> 2018-07-11 07:52:23.063405 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'66621,epoch:501/498,start:5:7580c019:::benchmark_data_magna115_10867_object691092:0,end:5:75810f9f:::benchmark_data_magna115_10867_object2613219:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb773900 con 0 > -1332> 2018-07-11 07:52:23.252453 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1067 0x55d9bd689680 rep_scrubmap(5.2e e501 from shard 2) v1 > -1331> 2018-07-11 07:52:23.252478 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1067 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12476 (1269249638 0 2254201827) 0x55d9bd689680 con 0x55d986bf0800 > -1330> 2018-07-11 07:52:23.261752 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1061 0x55d9bd6898c0 rep_scrubmap(5.2e e501 from shard 4) v1 > -1329> 2018-07-11 07:52:23.261772 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1061 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12476 (3686815630 0 2254201827) 0x55d9bd6898c0 con 0x55d986bec000 > -1328> 2018-07-11 07:52:23.469658 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -1327> 2018-07-11 07:52:23.469687 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -1326> 2018-07-11 07:52:23.469743 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -1325> 2018-07-11 07:52:23.470192 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -1324> 2018-07-11 07:52:23.519288 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'66830,epoch:501/498,start:5:75810f9f:::benchmark_data_magna115_10867_object2613219:0,end:5:758176e3:::benchmark_data_magna115_10867_object648460:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb772f00 con 0 > -1323> 2018-07-11 07:52:23.519331 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'66830,epoch:501/498,start:5:75810f9f:::benchmark_data_magna115_10867_object2613219:0,end:5:758176e3:::benchmark_data_magna115_10867_object648460:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb772a00 con 0 > -1322> 2018-07-11 07:52:23.570884 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6805/15885 conn(0x55d986c7f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.7 seq 2828 0x55d986c23a00 MOSDScrubReserve(5.4 REQUEST e501) v1 > -1321> 2018-07-11 07:52:23.570904 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.7 10.8.128.114:6805/15885 2828 ==== MOSDScrubReserve(5.4 REQUEST e501) v1 ==== 43+0+0 (1753221512 0 0) 0x55d986c23a00 con 0x55d986c7f800 > -1320> 2018-07-11 07:52:23.570978 7ffab33df700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.114:6805/15885 -- MOSDScrubReserve(5.4 REJECT e501) v1 -- 0x55d9829e5600 con 0 > -1319> 2018-07-11 07:52:23.571315 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6805/15885 conn(0x55d986c7f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.7 seq 2829 0x55d9829e5600 MOSDScrubReserve(5.4 RELEASE e501) v1 > -1318> 2018-07-11 07:52:23.571330 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.7 10.8.128.114:6805/15885 2829 ==== MOSDScrubReserve(5.4 RELEASE e501) v1 ==== 43+0+0 (2812146570 0 0) 0x55d9829e5600 con 0x55d986c7f800 > -1317> 2018-07-11 07:52:23.708534 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.115:0/13323 conn(0x55d986d7e000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 267 0x55d9829e5800 osd_ping(ping e501 stamp 2018-07-11 07:52:24.054631) v4 > -1316> 2018-07-11 07:52:23.708563 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.6 10.8.128.115:0/13323 267 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:24.054631) v4 ==== 2004+0+0 (1998787384 0 0) 0x55d9829e5800 con 0x55d986d7e000 > -1315> 2018-07-11 07:52:23.708580 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:24.054631) v4 -- 0x55d9bcd72e00 con 0 > -1314> 2018-07-11 07:52:23.708574 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.115:0/13323 conn(0x55d986d24800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 267 0x55d982d31c00 osd_ping(ping e501 stamp 2018-07-11 07:52:24.054631) v4 > -1313> 2018-07-11 07:52:23.708596 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.6 10.8.128.115:0/13323 267 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:24.054631) v4 ==== 2004+0+0 (1998787384 0 0) 0x55d982d31c00 con 0x55d986d24800 > -1312> 2018-07-11 07:52:23.708609 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:24.054631) v4 -- 0x55d9869bf400 con 0 > -1311> 2018-07-11 07:52:23.729380 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1068 0x55d9bd689b00 rep_scrubmap(5.2e e501 from shard 2) v1 > -1310> 2018-07-11 07:52:23.729399 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1068 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12478 (1269249638 0 435744842) 0x55d9bd689b00 con 0x55d986bf0800 > -1309> 2018-07-11 07:52:23.775288 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1062 0x55d9bd689d40 rep_scrubmap(5.2e e501 from shard 4) v1 > -1308> 2018-07-11 07:52:23.775308 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1062 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12478 (3686815630 0 435744842) 0x55d9bd689d40 con 0x55d986bec000 > -1307> 2018-07-11 07:52:23.795790 7ffacbbd1700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.108:0/11967 conn(0x55d986d82800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.8 seq 249 0x55d982cdf400 osd_ping(ping e501 stamp 2018-07-11 07:52:23.551712) v4 > -1306> 2018-07-11 07:52:23.795812 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 <== osd.8 10.8.128.108:0/11967 249 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:23.551712) v4 ==== 2004+0+0 (3087360117 0 0) 0x55d982cdf400 con 0x55d986d82800 > -1305> 2018-07-11 07:52:23.795804 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.108:0/11967 conn(0x55d986db0000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.8 seq 249 0x55d982d31600 osd_ping(ping e501 stamp 2018-07-11 07:52:23.551712) v4 > -1304> 2018-07-11 07:52:23.795821 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:23.551712) v4 -- 0x55d9bcd72e00 con 0 > -1303> 2018-07-11 07:52:23.795821 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.8 10.8.128.108:0/11967 249 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:23.551712) v4 ==== 2004+0+0 (3087360117 0 0) 0x55d982d31600 con 0x55d986db0000 > -1302> 2018-07-11 07:52:23.795848 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:23.551712) v4 -- 0x55d9b2783400 con 0 > -1301> 2018-07-11 07:52:23.818742 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -1300> 2018-07-11 07:52:23.818767 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -1299> 2018-07-11 07:52:23.818825 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -1298> 2018-07-11 07:52:23.819322 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -1297> 2018-07-11 07:52:23.861242 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:758176e3:::benchmark_data_magna115_10867_object648460:0,end:5:7581b0c3:::benchmark_data_magna115_10867_object1139733:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb772c80 con 0 > -1296> 2018-07-11 07:52:23.861291 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:758176e3:::benchmark_data_magna115_10867_object648460:0,end:5:7581b0c3:::benchmark_data_magna115_10867_object1139733:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb772500 con 0 > -1295> 2018-07-11 07:52:23.958444 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.107:0/12617 conn(0x55d98123c000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 259 0x55d982d31c00 osd_ping(ping e501 stamp 2018-07-11 07:52:23.957946) v4 > -1294> 2018-07-11 07:52:23.958466 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.9 10.8.128.107:0/12617 259 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:23.957946) v4 ==== 2004+0+0 (1441673799 0 0) 0x55d982d31c00 con 0x55d98123c000 > -1293> 2018-07-11 07:52:23.958463 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.107:0/12617 conn(0x55d98123d800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 259 0x55d9829e5800 osd_ping(ping e501 stamp 2018-07-11 07:52:23.957946) v4 > -1292> 2018-07-11 07:52:23.958487 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.9 10.8.128.107:0/12617 259 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:23.957946) v4 ==== 2004+0+0 (1441673799 0 0) 0x55d9829e5800 con 0x55d98123d800 > -1291> 2018-07-11 07:52:23.958486 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:23.957946) v4 -- 0x55d9869bf400 con 0 > -1290> 2018-07-11 07:52:23.958523 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:23.957946) v4 -- 0x55d9bcd72e00 con 0 > -1289> 2018-07-11 07:52:23.966976 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6805/15263 conn(0x55d986c41000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=31 cs=1 l=0). rx osd.5 seq 2844 0x55d9829e5a00 MOSDScrubReserve(6.15 REQUEST e501) v1 > -1288> 2018-07-11 07:52:23.966991 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.5 10.8.128.113:6805/15263 2844 ==== MOSDScrubReserve(6.15 REQUEST e501) v1 ==== 43+0+0 (3942742026 0 0) 0x55d9829e5a00 con 0x55d986c41000 > -1287> 2018-07-11 07:52:23.970601 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12677 conn(0x55d986db4800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.1 seq 270 0x55d9869d9800 osd_ping(ping e501 stamp 2018-07-11 07:52:24.018893) v4 > -1286> 2018-07-11 07:52:23.970619 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.1 10.8.128.106:0/12677 270 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:24.018893) v4 ==== 2004+0+0 (3693096775 0 0) 0x55d9869d9800 con 0x55d986db4800 > -1285> 2018-07-11 07:52:23.970628 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:24.018893) v4 -- 0x55d9b2783400 con 0 > -1284> 2018-07-11 07:52:23.970644 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12677 conn(0x55d986df2000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.1 seq 270 0x55d982d31c00 osd_ping(ping e501 stamp 2018-07-11 07:52:24.018893) v4 > -1283> 2018-07-11 07:52:23.970663 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.1 10.8.128.106:0/12677 270 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:24.018893) v4 ==== 2004+0+0 (3693096775 0 0) 0x55d982d31c00 con 0x55d986df2000 > -1282> 2018-07-11 07:52:23.970680 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:24.018893) v4 -- 0x55d9869bf200 con 0 > -1281> 2018-07-11 07:52:24.044909 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1069 0x55d9b0d26000 rep_scrubmap(5.2e e501 from shard 2) v1 > -1280> 2018-07-11 07:52:24.044929 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1069 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12462 (1269249638 0 1747920272) 0x55d9b0d26000 con 0x55d986bf0800 > -1279> 2018-07-11 07:52:24.079552 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1063 0x55d9b0d26240 rep_scrubmap(5.2e e501 from shard 4) v1 > -1278> 2018-07-11 07:52:24.079573 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1063 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12462 (3686815630 0 1747920272) 0x55d9b0d26240 con 0x55d986bec000 > -1277> 2018-07-11 07:52:24.171317 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6805/15263 -- MOSDScrubReserve(6.15 REJECT e501) v1 -- 0x55d986c22e00 con 0 > -1276> 2018-07-11 07:52:24.171531 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -1275> 2018-07-11 07:52:24.171554 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -1274> 2018-07-11 07:52:24.171601 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -1273> 2018-07-11 07:52:24.171840 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6805/15263 conn(0x55d986c41000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=31 cs=1 l=0). rx osd.5 seq 2845 0x55d986c22e00 MOSDScrubReserve(6.15 RELEASE e501) v1 > -1272> 2018-07-11 07:52:24.171860 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.5 10.8.128.113:6805/15263 2845 ==== MOSDScrubReserve(6.15 RELEASE e501) v1 ==== 43+0+0 (605848072 0 0) 0x55d986c22e00 con 0x55d986c41000 > -1271> 2018-07-11 07:52:24.172063 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -1270> 2018-07-11 07:52:24.219710 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7581b0c3:::benchmark_data_magna115_10867_object1139733:0,end:5:75820c3e:::benchmark_data_magna115_10867_object2443866:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb772000 con 0 > -1269> 2018-07-11 07:52:24.219763 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7581b0c3:::benchmark_data_magna115_10867_object1139733:0,end:5:75820c3e:::benchmark_data_magna115_10867_object2443866:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98944b400 con 0 > -1268> 2018-07-11 07:52:24.324973 7ffab1bdc700 5 osd.10 501 heartbeat: osd_stat(789 GB used, 142 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist []) > -1267> 2018-07-11 07:52:24.324987 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6802/15255 -- osd_ping(ping e501 stamp 2018-07-11 07:52:24.324985) v4 -- 0x55d9829e4800 con 0 > -1266> 2018-07-11 07:52:24.325024 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6803/15255 -- osd_ping(ping e501 stamp 2018-07-11 07:52:24.324985) v4 -- 0x55d9b437a200 con 0 > -1265> 2018-07-11 07:52:24.325040 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6806/12677 -- osd_ping(ping e501 stamp 2018-07-11 07:52:24.324985) v4 -- 0x55d9b9ac2a00 con 0 > -1264> 2018-07-11 07:52:24.325054 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6807/12677 -- osd_ping(ping e501 stamp 2018-07-11 07:52:24.324985) v4 -- 0x55d9c74e0200 con 0 > -1263> 2018-07-11 07:52:24.325068 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6808/12651 -- osd_ping(ping e501 stamp 2018-07-11 07:52:24.324985) v4 -- 0x55d9c3383a00 con 0 > -1262> 2018-07-11 07:52:24.325078 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6810/12651 -- osd_ping(ping e501 stamp 2018-07-11 07:52:24.324985) v4 -- 0x55d986d4e400 con 0 > -1261> 2018-07-11 07:52:24.325090 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6809/12650 -- osd_ping(ping e501 stamp 2018-07-11 07:52:24.324985) v4 -- 0x55d9b67d8400 con 0 > -1260> 2018-07-11 07:52:24.325102 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6811/12650 -- osd_ping(ping e501 stamp 2018-07-11 07:52:24.324985) v4 -- 0x55d986d96600 con 0 > -1259> 2018-07-11 07:52:24.325120 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6810/15287 -- osd_ping(ping e501 stamp 2018-07-11 07:52:24.324985) v4 -- 0x55d9bdba4c00 con 0 > -1258> 2018-07-11 07:52:24.325133 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6811/15287 -- osd_ping(ping e501 stamp 2018-07-11 07:52:24.324985) v4 -- 0x55d986d97400 con 0 > -1257> 2018-07-11 07:52:24.325146 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6806/15263 -- osd_ping(ping e501 stamp 2018-07-11 07:52:24.324985) v4 -- 0x55d982cdee00 con 0 > -1256> 2018-07-11 07:52:24.325161 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6807/15263 -- osd_ping(ping e501 stamp 2018-07-11 07:52:24.324985) v4 -- 0x55d9b67d9200 con 0 > -1255> 2018-07-11 07:52:24.325175 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.115:6802/13323 -- osd_ping(ping e501 stamp 2018-07-11 07:52:24.324985) v4 -- 0x55d9c908fe00 con 0 > -1254> 2018-07-11 07:52:24.325188 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.115:6803/13323 -- osd_ping(ping e501 stamp 2018-07-11 07:52:24.324985) v4 -- 0x55d986c22000 con 0 > -1253> 2018-07-11 07:52:24.325200 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6806/15885 -- osd_ping(ping e501 stamp 2018-07-11 07:52:24.324985) v4 -- 0x55d986c22600 con 0 > -1252> 2018-07-11 07:52:24.325217 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6807/15885 -- osd_ping(ping e501 stamp 2018-07-11 07:52:24.324985) v4 -- 0x55d980457a00 con 0 > -1251> 2018-07-11 07:52:24.325230 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.108:6802/11967 -- osd_ping(ping e501 stamp 2018-07-11 07:52:24.324985) v4 -- 0x55d9bf804400 con 0 > -1250> 2018-07-11 07:52:24.325241 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.108:6803/11967 -- osd_ping(ping e501 stamp 2018-07-11 07:52:24.324985) v4 -- 0x55d986d4e600 con 0 > -1249> 2018-07-11 07:52:24.325269 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.107:6806/12617 -- osd_ping(ping e501 stamp 2018-07-11 07:52:24.324985) v4 -- 0x55d9bcd72800 con 0 > -1248> 2018-07-11 07:52:24.325283 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.107:6807/12617 -- osd_ping(ping e501 stamp 2018-07-11 07:52:24.324985) v4 -- 0x55d980f5a200 con 0 > -1247> 2018-07-11 07:52:24.325296 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6802/15919 -- osd_ping(ping e501 stamp 2018-07-11 07:52:24.324985) v4 -- 0x55d982cdfc00 con 0 > -1246> 2018-07-11 07:52:24.325336 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6803/15919 -- osd_ping(ping e501 stamp 2018-07-11 07:52:24.324985) v4 -- 0x55d9b9ac2c00 con 0 > -1245> 2018-07-11 07:52:24.325466 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.107:6806/12617 conn(0x55d986bd2000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.9 seq 257 0x55d986d4e200 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:24.324985) v4 > -1244> 2018-07-11 07:52:24.325490 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.9 10.8.128.107:6806/12617 257 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:24.324985) v4 ==== 2004+0+0 (3371923037 0 0) 0x55d986d4e200 con 0x55d986bd2000 > -1243> 2018-07-11 07:52:24.325513 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.107:6807/12617 conn(0x55d986bd3800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.9 seq 257 0x55d982cdf400 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:24.324985) v4 > -1242> 2018-07-11 07:52:24.325534 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.9 10.8.128.107:6807/12617 257 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:24.324985) v4 ==== 2004+0+0 (3371923037 0 0) 0x55d982cdf400 con 0x55d986bd3800 > -1241> 2018-07-11 07:52:24.325548 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6803/15255 conn(0x55d986bc9800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.0 seq 257 0x55d986d4e200 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:24.324985) v4 > -1240> 2018-07-11 07:52:24.325564 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.0 10.8.128.113:6803/15255 257 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:24.324985) v4 ==== 2004+0+0 (3371923037 0 0) 0x55d986d4e200 con 0x55d986bc9800 > -1239> 2018-07-11 07:52:24.325577 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6809/12650 conn(0x55d986bcb000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 257 0x55d982cdf400 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:24.324985) v4 > -1238> 2018-07-11 07:52:24.325577 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6811/12650 conn(0x55d986bcc800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 257 0x55d982d31c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:24.324985) v4 > -1237> 2018-07-11 07:52:24.325592 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.3 10.8.128.106:6809/12650 257 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:24.324985) v4 ==== 2004+0+0 (3371923037 0 0) 0x55d982cdf400 con 0x55d986bcb000 > -1236> 2018-07-11 07:52:24.325593 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.3 10.8.128.106:6811/12650 257 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:24.324985) v4 ==== 2004+0+0 (3371923037 0 0) 0x55d982d31c00 con 0x55d986bcc800 > -1235> 2018-07-11 07:52:24.325601 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6810/12651 conn(0x55d986b9b800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.2 seq 257 0x55d986d4e200 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:24.324985) v4 > -1234> 2018-07-11 07:52:24.325620 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.2 10.8.128.106:6810/12651 257 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:24.324985) v4 ==== 2004+0+0 (3371923037 0 0) 0x55d986d4e200 con 0x55d986b9b800 > -1233> 2018-07-11 07:52:24.325622 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6802/15255 conn(0x55d986bc8000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.0 seq 257 0x55d982d31c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:24.324985) v4 > -1232> 2018-07-11 07:52:24.325631 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.0 10.8.128.113:6802/15255 257 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:24.324985) v4 ==== 2004+0+0 (3371923037 0 0) 0x55d982d31c00 con 0x55d986bc8000 > -1231> 2018-07-11 07:52:24.325652 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6806/12677 conn(0x55d986b58000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.1 seq 257 0x55d982cdf400 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:24.324985) v4 > -1230> 2018-07-11 07:52:24.325657 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6811/15287 conn(0x55d986bb0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 257 0x55d986d4e200 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:24.324985) v4 > -1229> 2018-07-11 07:52:24.325662 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6810/15287 conn(0x55d986baf000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 257 0x55d982d31c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:24.324985) v4 > -1228> 2018-07-11 07:52:24.325666 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.1 10.8.128.106:6806/12677 257 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:24.324985) v4 ==== 2004+0+0 (3371923037 0 0) 0x55d982cdf400 con 0x55d986b58000 > -1227> 2018-07-11 07:52:24.325667 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.4 10.8.128.113:6811/15287 257 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:24.324985) v4 ==== 2004+0+0 (3371923037 0 0) 0x55d986d4e200 con 0x55d986bb0800 > -1226> 2018-07-11 07:52:24.325677 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.4 10.8.128.113:6810/15287 257 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:24.324985) v4 ==== 2004+0+0 (3371923037 0 0) 0x55d982d31c00 con 0x55d986baf000 > -1225> 2018-07-11 07:52:24.325703 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6806/15263 conn(0x55d986b9d000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.5 seq 257 0x55d982cdf400 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:24.324985) v4 > -1224> 2018-07-11 07:52:24.325710 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6808/12651 conn(0x55d986b9a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.2 seq 257 0x55d982d31c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:24.324985) v4 > -1223> 2018-07-11 07:52:24.325722 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.2 10.8.128.106:6808/12651 257 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:24.324985) v4 ==== 2004+0+0 (3371923037 0 0) 0x55d982d31c00 con 0x55d986b9a000 > -1222> 2018-07-11 07:52:24.325716 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.5 10.8.128.113:6806/15263 257 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:24.324985) v4 ==== 2004+0+0 (3371923037 0 0) 0x55d982cdf400 con 0x55d986b9d000 > -1221> 2018-07-11 07:52:24.325754 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6807/15263 conn(0x55d986b9e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.5 seq 257 0x55d982d31c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:24.324985) v4 > -1220> 2018-07-11 07:52:24.325755 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.115:6802/13323 conn(0x55d986bc1000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 257 0x55d986d4e200 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:24.324985) v4 > -1219> 2018-07-11 07:52:24.325789 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.5 10.8.128.113:6807/15263 257 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:24.324985) v4 ==== 2004+0+0 (3371923037 0 0) 0x55d982d31c00 con 0x55d986b9e800 > -1218> 2018-07-11 07:52:24.325803 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.6 10.8.128.115:6802/13323 257 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:24.324985) v4 ==== 2004+0+0 (3371923037 0 0) 0x55d986d4e200 con 0x55d986bc1000 > -1217> 2018-07-11 07:52:24.325810 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6807/15885 conn(0x55d986bad800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.7 seq 257 0x55d982cdf400 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:24.324985) v4 > -1216> 2018-07-11 07:52:24.325824 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.7 10.8.128.114:6807/15885 257 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:24.324985) v4 ==== 2004+0+0 (3371923037 0 0) 0x55d982cdf400 con 0x55d986bad800 > -1215> 2018-07-11 07:52:24.325830 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6807/12677 conn(0x55d986b59800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.1 seq 257 0x55d982d31c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:24.324985) v4 > -1214> 2018-07-11 07:52:24.325843 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.1 10.8.128.106:6807/12677 257 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:24.324985) v4 ==== 2004+0+0 (3371923037 0 0) 0x55d982d31c00 con 0x55d986b59800 > -1213> 2018-07-11 07:52:24.325858 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.115:6803/13323 conn(0x55d986bc2800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 257 0x55d982cdf400 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:24.324985) v4 > -1212> 2018-07-11 07:52:24.325879 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.108:6803/11967 conn(0x55d986bbf800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.8 seq 257 0x55d982d31c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:24.324985) v4 > -1211> 2018-07-11 07:52:24.325884 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.6 10.8.128.115:6803/13323 257 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:24.324985) v4 ==== 2004+0+0 (3371923037 0 0) 0x55d982cdf400 con 0x55d986bc2800 > -1210> 2018-07-11 07:52:24.325891 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.8 10.8.128.108:6803/11967 257 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:24.324985) v4 ==== 2004+0+0 (3371923037 0 0) 0x55d982d31c00 con 0x55d986bbf800 > -1209> 2018-07-11 07:52:24.325889 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6802/15919 conn(0x55d986b5b000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.11 seq 257 0x55d986d4e200 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:24.324985) v4 > -1208> 2018-07-11 07:52:24.325916 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.11 10.8.128.114:6802/15919 257 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:24.324985) v4 ==== 2004+0+0 (3371923037 0 0) 0x55d986d4e200 con 0x55d986b5b000 > -1207> 2018-07-11 07:52:24.325915 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.108:6802/11967 conn(0x55d986bbe000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.8 seq 257 0x55d982cdf400 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:24.324985) v4 > -1206> 2018-07-11 07:52:24.325933 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.8 10.8.128.108:6802/11967 257 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:24.324985) v4 ==== 2004+0+0 (3371923037 0 0) 0x55d982cdf400 con 0x55d986bbe000 > -1205> 2018-07-11 07:52:24.325948 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6806/15885 conn(0x55d986bac000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.7 seq 257 0x55d986d4e200 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:24.324985) v4 > -1204> 2018-07-11 07:52:24.325959 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.7 10.8.128.114:6806/15885 257 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:24.324985) v4 ==== 2004+0+0 (3371923037 0 0) 0x55d986d4e200 con 0x55d986bac000 > -1203> 2018-07-11 07:52:24.325966 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6803/15919 conn(0x55d986b5c800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.11 seq 257 0x55d982cdf400 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:24.324985) v4 > -1202> 2018-07-11 07:52:24.325978 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.11 10.8.128.114:6803/15919 257 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:24.324985) v4 ==== 2004+0+0 (3371923037 0 0) 0x55d982cdf400 con 0x55d986b5c800 > -1201> 2018-07-11 07:52:24.392008 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15287 conn(0x55d981240800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 280 0x55d982d31c00 osd_ping(ping e501 stamp 2018-07-11 07:52:25.623795) v4 > -1200> 2018-07-11 07:52:24.392034 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.4 10.8.128.113:0/15287 280 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:25.623795) v4 ==== 2004+0+0 (2581191656 0 0) 0x55d982d31c00 con 0x55d981240800 > -1199> 2018-07-11 07:52:24.392048 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:25.623795) v4 -- 0x55d986d4e600 con 0 > -1198> 2018-07-11 07:52:24.392057 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15287 conn(0x55d98123f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 280 0x55d986d4e200 osd_ping(ping e501 stamp 2018-07-11 07:52:25.623795) v4 > -1197> 2018-07-11 07:52:24.392079 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.4 10.8.128.113:0/15287 280 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:25.623795) v4 ==== 2004+0+0 (2581191656 0 0) 0x55d986d4e200 con 0x55d98123f000 > -1196> 2018-07-11 07:52:24.392090 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:25.623795) v4 -- 0x55d9869d9800 con 0 > -1195> 2018-07-11 07:52:24.432039 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1070 0x55d9b0d26480 rep_scrubmap(5.2e e501 from shard 2) v1 > -1194> 2018-07-11 07:52:24.432058 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1070 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12466 (1269249638 0 1978270026) 0x55d9b0d26480 con 0x55d986bf0800 > -1193> 2018-07-11 07:52:24.457881 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1064 0x55d9b0d266c0 rep_scrubmap(5.2e e501 from shard 4) v1 > -1192> 2018-07-11 07:52:24.457902 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1064 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12466 (3686815630 0 1978270026) 0x55d9b0d266c0 con 0x55d986bec000 > -1191> 2018-07-11 07:52:24.481673 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -1190> 2018-07-11 07:52:24.481700 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -1189> 2018-07-11 07:52:24.481756 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -1188> 2018-07-11 07:52:24.482224 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -1187> 2018-07-11 07:52:24.483043 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67822,epoch:501/498,start:5:75820c3e:::benchmark_data_magna115_10867_object2443866:0,end:5:75826f8f:::benchmark_data_magna115_10867_object1109316:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb772280 con 0 > -1186> 2018-07-11 07:52:24.483090 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67822,epoch:501/498,start:5:75820c3e:::benchmark_data_magna115_10867_object2443866:0,end:5:75826f8f:::benchmark_data_magna115_10867_object1109316:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98944b900 con 0 > -1185> 2018-07-11 07:52:24.694714 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1065 0x55d9b0d26900 rep_scrubmap(5.2e e501 from shard 4) v1 > -1184> 2018-07-11 07:52:24.694735 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1065 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12458 (3686815630 0 3376724486) 0x55d9b0d26900 con 0x55d986bec000 > -1183> 2018-07-11 07:52:24.705737 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1071 0x55d9b0d26b40 rep_scrubmap(5.2e e501 from shard 2) v1 > -1182> 2018-07-11 07:52:24.705758 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1071 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12458 (1269249638 0 3376724486) 0x55d9b0d26b40 con 0x55d986bf0800 > -1181> 2018-07-11 07:52:24.723570 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12650 conn(0x55d986d23000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 259 0x55d982d31600 osd_ping(ping e501 stamp 2018-07-11 07:52:24.771785) v4 > -1180> 2018-07-11 07:52:24.723577 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12650 conn(0x55d986d21800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 259 0x55d982cdf400 osd_ping(ping e501 stamp 2018-07-11 07:52:24.771785) v4 > -1179> 2018-07-11 07:52:24.723592 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.3 10.8.128.106:0/12650 259 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:24.771785) v4 ==== 2004+0+0 (1383382943 0 0) 0x55d982d31600 con 0x55d986d23000 > -1178> 2018-07-11 07:52:24.723595 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.3 10.8.128.106:0/12650 259 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:24.771785) v4 ==== 2004+0+0 (1383382943 0 0) 0x55d982cdf400 con 0x55d986d21800 > -1177> 2018-07-11 07:52:24.723603 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:24.771785) v4 -- 0x55d9869d9800 con 0 > -1176> 2018-07-11 07:52:24.723629 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:24.771785) v4 -- 0x55d9b9ac2c00 con 0 > -1175> 2018-07-11 07:52:24.830033 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -1174> 2018-07-11 07:52:24.830059 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -1173> 2018-07-11 07:52:24.830118 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -1172> 2018-07-11 07:52:24.830597 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -1171> 2018-07-11 07:52:24.846683 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 3036 0x55d9829e5800 MOSDScrubReserve(11.3s5 REQUEST e501) v1 > -1170> 2018-07-11 07:52:24.846700 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 3036 ==== MOSDScrubReserve(11.3s5 REQUEST e501) v1 ==== 43+0+0 (3260170403 0 0) 0x55d9829e5800 con 0x55d986c7e000 > -1169> 2018-07-11 07:52:24.846820 7ffab3be0700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e501) v1 -- 0x55d9b67d9e00 con 0 > -1168> 2018-07-11 07:52:24.847140 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 3037 0x55d9b9ac2c00 MOSDScrubReserve(11.3s5 RELEASE e501) v1 > -1167> 2018-07-11 07:52:24.847165 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 3037 ==== MOSDScrubReserve(11.3s5 RELEASE e501) v1 ==== 43+0+0 (223328929 0 0) 0x55d9b9ac2c00 con 0x55d986c7e000 > -1166> 2018-07-11 07:52:24.853372 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75826f8f:::benchmark_data_magna115_10867_object1109316:0,end:5:7582cb2c:::benchmark_data_magna115_10867_object187780:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98944bb80 con 0 > -1165> 2018-07-11 07:52:24.853421 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75826f8f:::benchmark_data_magna115_10867_object1109316:0,end:5:7582cb2c:::benchmark_data_magna115_10867_object187780:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98944b680 con 0 > -1164> 2018-07-11 07:52:25.059132 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.107:0/12617 conn(0x55d98123c000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 260 0x55d982d31c00 osd_ping(ping e501 stamp 2018-07-11 07:52:25.058616) v4 > -1163> 2018-07-11 07:52:25.059145 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.107:0/12617 conn(0x55d98123d800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 260 0x55d982cdf400 osd_ping(ping e501 stamp 2018-07-11 07:52:25.058616) v4 > -1162> 2018-07-11 07:52:25.059158 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.9 10.8.128.107:0/12617 260 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:25.058616) v4 ==== 2004+0+0 (4210794168 0 0) 0x55d982d31c00 con 0x55d98123c000 > -1161> 2018-07-11 07:52:25.059174 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.9 10.8.128.107:0/12617 260 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:25.058616) v4 ==== 2004+0+0 (4210794168 0 0) 0x55d982cdf400 con 0x55d98123d800 > -1160> 2018-07-11 07:52:25.059183 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:25.058616) v4 -- 0x55d986d4e600 con 0 > -1159> 2018-07-11 07:52:25.059220 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:25.058616) v4 -- 0x55d980f5a200 con 0 > -1158> 2018-07-11 07:52:25.076586 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1066 0x55d9b0d26d80 rep_scrubmap(5.2e e501 from shard 4) v1 > -1157> 2018-07-11 07:52:25.076606 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1066 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12470 (3686815630 0 1355904682) 0x55d9b0d26d80 con 0x55d986bec000 > -1156> 2018-07-11 07:52:25.092199 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1072 0x55d9b0d26fc0 rep_scrubmap(5.2e e501 from shard 2) v1 > -1155> 2018-07-11 07:52:25.092219 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1072 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12470 (1269249638 0 1355904682) 0x55d9b0d26fc0 con 0x55d986bf0800 > -1154> 2018-07-11 07:52:25.220850 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -1153> 2018-07-11 07:52:25.220877 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -1152> 2018-07-11 07:52:25.220934 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -1151> 2018-07-11 07:52:25.221404 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -1150> 2018-07-11 07:52:25.236575 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7582cb2c:::benchmark_data_magna115_10867_object187780:0,end:5:75834015:::benchmark_data_magna115_10867_object2248491:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454a00 con 0 > -1149> 2018-07-11 07:52:25.236626 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7582cb2c:::benchmark_data_magna115_10867_object187780:0,end:5:75834015:::benchmark_data_magna115_10867_object2248491:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455180 con 0 > -1148> 2018-07-11 07:52:25.438724 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1073 0x55d9b0d27200 rep_scrubmap(5.2e e501 from shard 2) v1 > -1147> 2018-07-11 07:52:25.438744 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1073 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12478 (1269249638 0 2811696664) 0x55d9b0d27200 con 0x55d986bf0800 > -1146> 2018-07-11 07:52:25.445261 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1067 0x55d9b0d27440 rep_scrubmap(5.2e e501 from shard 4) v1 > -1145> 2018-07-11 07:52:25.445280 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1067 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12478 (3686815630 0 2811696664) 0x55d9b0d27440 con 0x55d986bec000 > -1144> 2018-07-11 07:52:25.499020 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.114:0/15919 conn(0x55d986df3800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.11 seq 268 0x55d9b67d9e00 osd_ping(ping e501 stamp 2018-07-11 07:52:26.129595) v4 > -1143> 2018-07-11 07:52:25.499040 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.11 10.8.128.114:0/15919 268 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:26.129595) v4 ==== 2004+0+0 (2832407974 0 0) 0x55d9b67d9e00 con 0x55d986df3800 > -1142> 2018-07-11 07:52:25.499050 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:26.129595) v4 -- 0x55d980f5a200 con 0 > -1141> 2018-07-11 07:52:25.499201 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.114:0/15919 conn(0x55d986df5000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.11 seq 268 0x55d986d4e200 osd_ping(ping e501 stamp 2018-07-11 07:52:26.129595) v4 > -1140> 2018-07-11 07:52:25.499219 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.11 10.8.128.114:0/15919 268 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:26.129595) v4 ==== 2004+0+0 (2832407974 0 0) 0x55d986d4e200 con 0x55d986df5000 > -1139> 2018-07-11 07:52:25.499231 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:26.129595) v4 -- 0x55d9869d9800 con 0 > -1138> 2018-07-11 07:52:25.507283 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -1137> 2018-07-11 07:52:25.507306 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -1136> 2018-07-11 07:52:25.507365 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -1135> 2018-07-11 07:52:25.507828 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -1134> 2018-07-11 07:52:25.546693 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75834015:::benchmark_data_magna115_10867_object2248491:0,end:5:75838e15:::benchmark_data_magna115_10867_object2273514:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454f00 con 0 > -1133> 2018-07-11 07:52:25.546731 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75834015:::benchmark_data_magna115_10867_object2248491:0,end:5:75838e15:::benchmark_data_magna115_10867_object2273514:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454500 con 0 > -1132> 2018-07-11 07:52:25.623930 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.114:0/15885 conn(0x55d986d7f800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.7 seq 271 0x55d982d31600 osd_ping(ping e501 stamp 2018-07-11 07:52:26.254444) v4 > -1131> 2018-07-11 07:52:25.623955 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.7 10.8.128.114:0/15885 271 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:26.254444) v4 ==== 2004+0+0 (2719324215 0 0) 0x55d982d31600 con 0x55d986d7f800 > -1130> 2018-07-11 07:52:25.623973 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:26.254444) v4 -- 0x55d9869d9800 con 0 > -1129> 2018-07-11 07:52:25.623972 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.114:0/15885 conn(0x55d986d81000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.7 seq 271 0x55d9869bf400 osd_ping(ping e501 stamp 2018-07-11 07:52:26.254444) v4 > -1128> 2018-07-11 07:52:25.623994 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.7 10.8.128.114:0/15885 271 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:26.254444) v4 ==== 2004+0+0 (2719324215 0 0) 0x55d9869bf400 con 0x55d986d81000 > -1127> 2018-07-11 07:52:25.624005 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:26.254444) v4 -- 0x55d986d4e600 con 0 > -1126> 2018-07-11 07:52:25.729551 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12651 conn(0x55d986db3000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.2 seq 264 0x55d982cdf400 osd_ping(ping e501 stamp 2018-07-11 07:52:25.777780) v4 > -1125> 2018-07-11 07:52:25.729561 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12651 conn(0x55d986db1800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.2 seq 264 0x55d982d31c00 osd_ping(ping e501 stamp 2018-07-11 07:52:25.777780) v4 > -1124> 2018-07-11 07:52:25.729573 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.2 10.8.128.106:0/12651 264 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:25.777780) v4 ==== 2004+0+0 (352377841 0 0) 0x55d982cdf400 con 0x55d986db3000 > -1123> 2018-07-11 07:52:25.729580 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.2 10.8.128.106:0/12651 264 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:25.777780) v4 ==== 2004+0+0 (352377841 0 0) 0x55d982d31c00 con 0x55d986db1800 > -1122> 2018-07-11 07:52:25.729583 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:25.777780) v4 -- 0x55d980f5a200 con 0 > -1121> 2018-07-11 07:52:25.729626 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:25.777780) v4 -- 0x55d986d4e600 con 0 > -1120> 2018-07-11 07:52:25.766215 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1074 0x55d9b0d27680 rep_scrubmap(5.2e e501 from shard 2) v1 > -1119> 2018-07-11 07:52:25.766235 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1074 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12470 (1269249638 0 193535164) 0x55d9b0d27680 con 0x55d986bf0800 > -1118> 2018-07-11 07:52:25.795975 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1068 0x55d9b0d278c0 rep_scrubmap(5.2e e501 from shard 4) v1 > -1117> 2018-07-11 07:52:25.795995 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1068 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12470 (3686815630 0 193535164) 0x55d9b0d278c0 con 0x55d986bec000 > -1116> 2018-07-11 07:52:25.846911 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 3038 0x55d9b67d9e00 MOSDScrubReserve(11.3s5 REQUEST e501) v1 > -1115> 2018-07-11 07:52:25.846929 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 3038 ==== MOSDScrubReserve(11.3s5 REQUEST e501) v1 ==== 43+0+0 (3260170403 0 0) 0x55d9b67d9e00 con 0x55d986c7e000 > -1114> 2018-07-11 07:52:25.847054 7ffab3be0700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e501) v1 -- 0x55d9b9ac2c00 con 0 > -1113> 2018-07-11 07:52:25.847393 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 3039 0x55d980f5a200 MOSDScrubReserve(11.3s5 RELEASE e501) v1 > -1112> 2018-07-11 07:52:25.847407 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 3039 ==== MOSDScrubReserve(11.3s5 RELEASE e501) v1 ==== 43+0+0 (223328929 0 0) 0x55d980f5a200 con 0x55d986c7e000 > -1111> 2018-07-11 07:52:25.907578 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -1110> 2018-07-11 07:52:25.907606 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -1109> 2018-07-11 07:52:25.907664 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -1108> 2018-07-11 07:52:25.908133 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -1107> 2018-07-11 07:52:25.928554 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75838e15:::benchmark_data_magna115_10867_object2273514:0,end:5:7583e76d:::benchmark_data_magna115_10867_object1048868:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455400 con 0 > -1106> 2018-07-11 07:52:25.928603 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75838e15:::benchmark_data_magna115_10867_object2273514:0,end:5:7583e76d:::benchmark_data_magna115_10867_object1048868:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454780 con 0 > -1105> 2018-07-11 07:52:25.967302 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6805/15263 conn(0x55d986c41000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=31 cs=1 l=0). rx osd.5 seq 2846 0x55d982d31c00 MOSDScrubReserve(6.15 REQUEST e501) v1 > -1104> 2018-07-11 07:52:25.967319 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.5 10.8.128.113:6805/15263 2846 ==== MOSDScrubReserve(6.15 REQUEST e501) v1 ==== 43+0+0 (3942742026 0 0) 0x55d982d31c00 con 0x55d986c41000 > -1103> 2018-07-11 07:52:26.008943 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.115:0/13323 conn(0x55d986d7e000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 268 0x55d982cdf400 osd_ping(ping e501 stamp 2018-07-11 07:52:26.355084) v4 > -1102> 2018-07-11 07:52:26.008965 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.6 10.8.128.115:0/13323 268 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:26.355084) v4 ==== 2004+0+0 (4007534155 0 0) 0x55d982cdf400 con 0x55d986d7e000 > -1101> 2018-07-11 07:52:26.008982 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:26.355084) v4 -- 0x55d9bf804400 con 0 > -1100> 2018-07-11 07:52:26.009009 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.115:0/13323 conn(0x55d986d24800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 268 0x55d9869bf400 osd_ping(ping e501 stamp 2018-07-11 07:52:26.355084) v4 > -1099> 2018-07-11 07:52:26.009035 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.6 10.8.128.115:0/13323 268 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:26.355084) v4 ==== 2004+0+0 (4007534155 0 0) 0x55d9869bf400 con 0x55d986d24800 > -1098> 2018-07-11 07:52:26.009047 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:26.355084) v4 -- 0x55d9b67d9200 con 0 > -1097> 2018-07-11 07:52:26.109810 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1075 0x55d9b0d27b00 rep_scrubmap(5.2e e501 from shard 2) v1 > -1096> 2018-07-11 07:52:26.109831 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1075 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12472 (1269249638 0 2661814863) 0x55d9b0d27b00 con 0x55d986bf0800 > -1095> 2018-07-11 07:52:26.136528 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1069 0x55d9b0d27d40 rep_scrubmap(5.2e e501 from shard 4) v1 > -1094> 2018-07-11 07:52:26.136548 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1069 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12472 (3686815630 0 2661814863) 0x55d9b0d27d40 con 0x55d986bec000 > -1093> 2018-07-11 07:52:26.159766 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.107:0/12617 conn(0x55d98123c000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 261 0x55d986d4e600 osd_ping(ping e501 stamp 2018-07-11 07:52:26.159293) v4 > -1092> 2018-07-11 07:52:26.159804 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.9 10.8.128.107:0/12617 261 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:26.159293) v4 ==== 2004+0+0 (829395421 0 0) 0x55d986d4e600 con 0x55d98123c000 > -1091> 2018-07-11 07:52:26.159815 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:26.159293) v4 -- 0x55d9b67d9200 con 0 > -1090> 2018-07-11 07:52:26.159900 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.107:0/12617 conn(0x55d98123d800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 261 0x55d9b9ac2c00 osd_ping(ping e501 stamp 2018-07-11 07:52:26.159293) v4 > -1089> 2018-07-11 07:52:26.159922 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.9 10.8.128.107:0/12617 261 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:26.159293) v4 ==== 2004+0+0 (829395421 0 0) 0x55d9b9ac2c00 con 0x55d98123d800 > -1088> 2018-07-11 07:52:26.159935 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:26.159293) v4 -- 0x55d9bf804400 con 0 > -1087> 2018-07-11 07:52:26.206080 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6805/15263 -- MOSDScrubReserve(6.15 REJECT e501) v1 -- 0x55d986c22e00 con 0 > -1086> 2018-07-11 07:52:26.206275 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -1085> 2018-07-11 07:52:26.206297 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -1084> 2018-07-11 07:52:26.206343 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -1083> 2018-07-11 07:52:26.206536 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6805/15263 conn(0x55d986c41000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=31 cs=1 l=0). rx osd.5 seq 2847 0x55d986c22e00 MOSDScrubReserve(6.15 RELEASE e501) v1 > -1082> 2018-07-11 07:52:26.206555 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.5 10.8.128.113:6805/15263 2847 ==== MOSDScrubReserve(6.15 RELEASE e501) v1 ==== 43+0+0 (605848072 0 0) 0x55d986c22e00 con 0x55d986c41000 > -1081> 2018-07-11 07:52:26.206796 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -1080> 2018-07-11 07:52:26.238653 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'66605,epoch:501/498,start:5:7583e76d:::benchmark_data_magna115_10867_object1048868:0,end:5:75843357:::benchmark_data_magna115_10867_object2313570:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498280 con 0 > -1079> 2018-07-11 07:52:26.238690 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'66605,epoch:501/498,start:5:7583e76d:::benchmark_data_magna115_10867_object1048868:0,end:5:75843357:::benchmark_data_magna115_10867_object2313570:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498000 con 0 > -1078> 2018-07-11 07:52:26.468073 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1076 0x55d983050000 rep_scrubmap(5.2e e501 from shard 2) v1 > -1077> 2018-07-11 07:52:26.468093 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1076 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12468 (1269249638 0 3193462505) 0x55d983050000 con 0x55d986bf0800 > -1076> 2018-07-11 07:52:26.496370 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1070 0x55d983050240 rep_scrubmap(5.2e e501 from shard 4) v1 > -1075> 2018-07-11 07:52:26.496389 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1070 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12468 (3686815630 0 3193462505) 0x55d983050240 con 0x55d986bec000 > -1074> 2018-07-11 07:52:26.540031 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -1073> 2018-07-11 07:52:26.540057 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -1072> 2018-07-11 07:52:26.540115 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -1071> 2018-07-11 07:52:26.540600 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -1070> 2018-07-11 07:52:26.562292 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75843357:::benchmark_data_magna115_10867_object2313570:0,end:5:7584c7e3:::benchmark_data_magna115_10867_object3852935:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b14f0500 con 0 > -1069> 2018-07-11 07:52:26.562341 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75843357:::benchmark_data_magna115_10867_object2313570:0,end:5:7584c7e3:::benchmark_data_magna115_10867_object3852935:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455900 con 0 > -1068> 2018-07-11 07:52:26.571348 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6805/15885 conn(0x55d986c7f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.7 seq 2830 0x55d9869bf400 MOSDScrubReserve(5.4 REQUEST e501) v1 > -1067> 2018-07-11 07:52:26.571365 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.7 10.8.128.114:6805/15885 2830 ==== MOSDScrubReserve(5.4 REQUEST e501) v1 ==== 43+0+0 (1753221512 0 0) 0x55d9869bf400 con 0x55d986c7f800 > -1066> 2018-07-11 07:52:26.571485 7ffab33df700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.114:6805/15885 -- MOSDScrubReserve(5.4 REJECT e501) v1 -- 0x55d9829e5600 con 0 > -1065> 2018-07-11 07:52:26.571710 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6805/15885 conn(0x55d986c7f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.7 seq 2831 0x55d9b67d9200 MOSDScrubReserve(5.4 RELEASE e501) v1 > -1064> 2018-07-11 07:52:26.571725 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.7 10.8.128.114:6805/15885 2831 ==== MOSDScrubReserve(5.4 RELEASE e501) v1 ==== 43+0+0 (2812146570 0 0) 0x55d9b67d9200 con 0x55d986c7f800 > -1063> 2018-07-11 07:52:26.766342 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1071 0x55d983050480 rep_scrubmap(5.2e e501 from shard 4) v1 > -1062> 2018-07-11 07:52:26.766362 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1071 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12468 (3686815630 0 1006979641) 0x55d983050480 con 0x55d986bec000 > -1061> 2018-07-11 07:52:26.781283 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1077 0x55d9830506c0 rep_scrubmap(5.2e e501 from shard 2) v1 > -1060> 2018-07-11 07:52:26.781302 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1077 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12468 (1269249638 0 1006979641) 0x55d9830506c0 con 0x55d986bf0800 > -1059> 2018-07-11 07:52:26.847107 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 3040 0x55d982cdf400 MOSDScrubReserve(11.3s5 REQUEST e501) v1 > -1058> 2018-07-11 07:52:26.847126 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 3040 ==== MOSDScrubReserve(11.3s5 REQUEST e501) v1 ==== 43+0+0 (3260170403 0 0) 0x55d982cdf400 con 0x55d986c7e000 > -1057> 2018-07-11 07:52:26.847200 7ffab3be0700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e501) v1 -- 0x55d980f5a200 con 0 > -1056> 2018-07-11 07:52:26.847602 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 3041 0x55d980f5a200 MOSDScrubReserve(11.3s5 RELEASE e501) v1 > -1055> 2018-07-11 07:52:26.847616 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 3041 ==== MOSDScrubReserve(11.3s5 RELEASE e501) v1 ==== 43+0+0 (223328929 0 0) 0x55d980f5a200 con 0x55d986c7e000 > -1054> 2018-07-11 07:52:26.936786 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -1053> 2018-07-11 07:52:26.936813 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -1052> 2018-07-11 07:52:26.936857 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -1051> 2018-07-11 07:52:26.937351 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -1050> 2018-07-11 07:52:26.954025 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7584c7e3:::benchmark_data_magna115_10867_object3852935:0,end:5:75851922:::benchmark_data_magna115_10867_object1519105:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455b80 con 0 > -1049> 2018-07-11 07:52:26.954075 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7584c7e3:::benchmark_data_magna115_10867_object3852935:0,end:5:75851922:::benchmark_data_magna115_10867_object1519105:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455680 con 0 > -1048> 2018-07-11 07:52:26.967519 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6805/15263 conn(0x55d986c41000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=31 cs=1 l=0). rx osd.5 seq 2848 0x55d9829e5600 MOSDScrubReserve(6.15 REQUEST e501) v1 > -1047> 2018-07-11 07:52:26.967536 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.5 10.8.128.113:6805/15263 2848 ==== MOSDScrubReserve(6.15 REQUEST e501) v1 ==== 43+0+0 (3942742026 0 0) 0x55d9829e5600 con 0x55d986c41000 > -1046> 2018-07-11 07:52:26.973608 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15255 conn(0x55d986c82800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.0 seq 266 0x55d986d4e200 osd_ping(ping e501 stamp 2018-07-11 07:52:28.205385) v4 > -1045> 2018-07-11 07:52:26.973610 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15255 conn(0x55d986d20000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.0 seq 266 0x55d986d4e600 osd_ping(ping e501 stamp 2018-07-11 07:52:28.205385) v4 > -1044> 2018-07-11 07:52:26.973632 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.0 10.8.128.113:0/15255 266 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:28.205385) v4 ==== 2004+0+0 (2401875384 0 0) 0x55d986d4e200 con 0x55d986c82800 > -1043> 2018-07-11 07:52:26.973633 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.0 10.8.128.113:0/15255 266 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:28.205385) v4 ==== 2004+0+0 (2401875384 0 0) 0x55d986d4e600 con 0x55d986d20000 > -1042> 2018-07-11 07:52:26.973643 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:28.205385) v4 -- 0x55d9869d9800 con 0 > -1041> 2018-07-11 07:52:26.973668 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:28.205385) v4 -- 0x55d986d96600 con 0 > -1040> 2018-07-11 07:52:27.164666 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1078 0x55d983050900 rep_scrubmap(5.2e e501 from shard 2) v1 > -1039> 2018-07-11 07:52:27.164696 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1078 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12468 (1269249638 0 2427214691) 0x55d983050900 con 0x55d986bf0800 > -1038> 2018-07-11 07:52:27.188536 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1072 0x55d983050b40 rep_scrubmap(5.2e e501 from shard 4) v1 > -1037> 2018-07-11 07:52:27.188554 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1072 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12468 (3686815630 0 2427214691) 0x55d983050b40 con 0x55d986bec000 > -1036> 2018-07-11 07:52:27.353513 7ffab6be6700 1 -- 10.8.128.107:6800/13927 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55d983c49600 con 0 > -1035> 2018-07-11 07:52:27.353582 7ffab6be6700 1 -- 10.8.128.107:6800/13927 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55d986cff8c0 con 0 > -1034> 2018-07-11 07:52:27.624039 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12650 conn(0x55d986d23000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 260 0x55d986d4e200 osd_ping(ping e501 stamp 2018-07-11 07:52:27.672255) v4 > -1033> 2018-07-11 07:52:27.624076 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.3 10.8.128.106:0/12650 260 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:27.672255) v4 ==== 2004+0+0 (3393737928 0 0) 0x55d986d4e200 con 0x55d986d23000 > -1032> 2018-07-11 07:52:27.624088 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:27.672255) v4 -- 0x55d9869d9800 con 0 > -1031> 2018-07-11 07:52:27.624137 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12650 conn(0x55d986d21800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 260 0x55d9b9ac2c00 osd_ping(ping e501 stamp 2018-07-11 07:52:27.672255) v4 > -1030> 2018-07-11 07:52:27.624159 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.3 10.8.128.106:0/12650 260 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:27.672255) v4 ==== 2004+0+0 (3393737928 0 0) 0x55d9b9ac2c00 con 0x55d986d21800 > -1029> 2018-07-11 07:52:27.624171 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:27.672255) v4 -- 0x55d980457a00 con 0 > -1028> 2018-07-11 07:52:27.653043 7ffacbbd1700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15263 conn(0x55d985d44000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.5 seq 248 0x55d9bf804400 osd_ping(ping e501 stamp 2018-07-11 07:52:28.884906) v4 > -1027> 2018-07-11 07:52:27.653049 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15263 conn(0x55d985d45800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.5 seq 248 0x55d982d31600 osd_ping(ping e501 stamp 2018-07-11 07:52:28.884906) v4 > -1026> 2018-07-11 07:52:27.653064 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 <== osd.5 10.8.128.113:0/15263 248 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:28.884906) v4 ==== 2004+0+0 (963220814 0 0) 0x55d9bf804400 con 0x55d985d44000 > -1025> 2018-07-11 07:52:27.653067 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.5 10.8.128.113:0/15263 248 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:28.884906) v4 ==== 2004+0+0 (963220814 0 0) 0x55d982d31600 con 0x55d985d45800 > -1024> 2018-07-11 07:52:27.653073 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:28.884906) v4 -- 0x55d980457a00 con 0 > -1023> 2018-07-11 07:52:27.653104 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:28.884906) v4 -- 0x55d9869d9800 con 0 > -1022> 2018-07-11 07:52:27.847384 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 3042 0x55d9b9ac2c00 MOSDScrubReserve(11.3s5 REQUEST e501) v1 > -1021> 2018-07-11 07:52:27.847401 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 3042 ==== MOSDScrubReserve(11.3s5 REQUEST e501) v1 ==== 43+0+0 (3260170403 0 0) 0x55d9b9ac2c00 con 0x55d986c7e000 > -1020> 2018-07-11 07:52:27.847527 7ffab3be0700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e501) v1 -- 0x55d980f5a200 con 0 > -1019> 2018-07-11 07:52:27.847930 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 3043 0x55d980f5a200 MOSDScrubReserve(11.3s5 RELEASE e501) v1 > -1018> 2018-07-11 07:52:27.847944 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 3043 ==== MOSDScrubReserve(11.3s5 RELEASE e501) v1 ==== 43+0+0 (223328929 0 0) 0x55d980f5a200 con 0x55d986c7e000 > -1017> 2018-07-11 07:52:27.896280 7ffacbbd1700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.108:0/11967 conn(0x55d986d82800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.8 seq 250 0x55d9bf804400 osd_ping(ping e501 stamp 2018-07-11 07:52:27.652145) v4 > -1016> 2018-07-11 07:52:27.896286 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.108:0/11967 conn(0x55d986db0000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.8 seq 250 0x55d986d4e200 osd_ping(ping e501 stamp 2018-07-11 07:52:27.652145) v4 > -1015> 2018-07-11 07:52:27.896302 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 <== osd.8 10.8.128.108:0/11967 250 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:27.652145) v4 ==== 2004+0+0 (2836439439 0 0) 0x55d9bf804400 con 0x55d986d82800 > -1014> 2018-07-11 07:52:27.896307 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.8 10.8.128.108:0/11967 250 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:27.652145) v4 ==== 2004+0+0 (2836439439 0 0) 0x55d986d4e200 con 0x55d986db0000 > -1013> 2018-07-11 07:52:27.896312 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:27.652145) v4 -- 0x55d986c22000 con 0 > -1012> 2018-07-11 07:52:27.896334 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:27.652145) v4 -- 0x55d9869d9800 con 0 > -1011> 2018-07-11 07:52:27.924304 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.114:0/15885 conn(0x55d986d7f800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.7 seq 272 0x55d982d31600 osd_ping(ping e501 stamp 2018-07-11 07:52:28.554862) v4 > -1010> 2018-07-11 07:52:27.924330 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.7 10.8.128.114:0/15885 272 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:28.554862) v4 ==== 2004+0+0 (4097807665 0 0) 0x55d982d31600 con 0x55d986d7f800 > -1009> 2018-07-11 07:52:27.924347 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:28.554862) v4 -- 0x55d9869d9800 con 0 > -1008> 2018-07-11 07:52:27.924341 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.114:0/15885 conn(0x55d986d81000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.7 seq 272 0x55d9bdba4c00 osd_ping(ping e501 stamp 2018-07-11 07:52:28.554862) v4 > -1007> 2018-07-11 07:52:27.924370 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.7 10.8.128.114:0/15885 272 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:28.554862) v4 ==== 2004+0+0 (4097807665 0 0) 0x55d9bdba4c00 con 0x55d986d81000 > -1006> 2018-07-11 07:52:27.924382 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:28.554862) v4 -- 0x55d986d96600 con 0 > -1005> 2018-07-11 07:52:28.153596 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15263 conn(0x55d985d45800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.5 seq 249 0x55d986d4e200 osd_ping(ping e501 stamp 2018-07-11 07:52:29.385454) v4 > -1004> 2018-07-11 07:52:28.153617 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.5 10.8.128.113:0/15263 249 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:29.385454) v4 ==== 2004+0+0 (2809643815 0 0) 0x55d986d4e200 con 0x55d985d45800 > -1003> 2018-07-11 07:52:28.153634 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:29.385454) v4 -- 0x55d9869d9800 con 0 > -1002> 2018-07-11 07:52:28.153679 7ffacbbd1700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15263 conn(0x55d985d44000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.5 seq 249 0x55d980457a00 osd_ping(ping e501 stamp 2018-07-11 07:52:29.385454) v4 > -1001> 2018-07-11 07:52:28.153700 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 <== osd.5 10.8.128.113:0/15263 249 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:29.385454) v4 ==== 2004+0+0 (2809643815 0 0) 0x55d980457a00 con 0x55d985d44000 > -1000> 2018-07-11 07:52:28.153711 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:29.385454) v4 -- 0x55d986c22000 con 0 > -999> 2018-07-11 07:52:28.267153 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 34] Generated table #36813: 2638237 keys, 74546406 bytes > -998> 2018-07-11 07:52:28.267197 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295548267171, "cf_name": "default", "job": 34, "event": "table_file_creation", "file_number": 36813, "file_size": 74546406, "table_properties": {"data_size": 67112112, "index_size": 690129, "filter_size": 6743175, "raw_key_size": 76215749, "raw_average_key_size": 28, "raw_value_size": 31365720, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638237, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} > -997> 2018-07-11 07:52:28.399492 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.114:0/15919 conn(0x55d986df3800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.11 seq 269 0x55d9bf804400 osd_ping(ping e501 stamp 2018-07-11 07:52:29.030014) v4 > -996> 2018-07-11 07:52:28.399512 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.11 10.8.128.114:0/15919 269 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:29.030014) v4 ==== 2004+0+0 (4155166323 0 0) 0x55d9bf804400 con 0x55d986df3800 > -995> 2018-07-11 07:52:28.399527 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:29.030014) v4 -- 0x55d986c22000 con 0 > -994> 2018-07-11 07:52:28.399536 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.114:0/15919 conn(0x55d986df5000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.11 seq 269 0x55d982d31600 osd_ping(ping e501 stamp 2018-07-11 07:52:29.030014) v4 > -993> 2018-07-11 07:52:28.399562 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.11 10.8.128.114:0/15919 269 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:29.030014) v4 ==== 2004+0+0 (4155166323 0 0) 0x55d982d31600 con 0x55d986df5000 > -992> 2018-07-11 07:52:28.399578 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:29.030014) v4 -- 0x55d9869d9800 con 0 > -991> 2018-07-11 07:52:28.425510 7ffab1bdc700 5 osd.10 501 heartbeat: osd_stat(789 GB used, 142 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,0,0,0,3]) > -990> 2018-07-11 07:52:28.425526 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6802/15255 -- osd_ping(ping e501 stamp 2018-07-11 07:52:28.425524) v4 -- 0x55d9b67d9c00 con 0 > -989> 2018-07-11 07:52:28.425565 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6803/15255 -- osd_ping(ping e501 stamp 2018-07-11 07:52:28.425524) v4 -- 0x55d9b40a4c00 con 0 > -988> 2018-07-11 07:52:28.425581 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6806/12677 -- osd_ping(ping e501 stamp 2018-07-11 07:52:28.425524) v4 -- 0x55d9869bec00 con 0 > -987> 2018-07-11 07:52:28.425596 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6807/12677 -- osd_ping(ping e501 stamp 2018-07-11 07:52:28.425524) v4 -- 0x55d9829e5200 con 0 > -986> 2018-07-11 07:52:28.425620 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6808/12651 -- osd_ping(ping e501 stamp 2018-07-11 07:52:28.425524) v4 -- 0x55d986d41000 con 0 > -985> 2018-07-11 07:52:28.425631 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6810/12651 -- osd_ping(ping e501 stamp 2018-07-11 07:52:28.425524) v4 -- 0x55d9869bf600 con 0 > -984> 2018-07-11 07:52:28.425643 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6809/12650 -- osd_ping(ping e501 stamp 2018-07-11 07:52:28.425524) v4 -- 0x55d986d32c00 con 0 > -983> 2018-07-11 07:52:28.425664 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6811/12650 -- osd_ping(ping e501 stamp 2018-07-11 07:52:28.425524) v4 -- 0x55d9bf804200 con 0 > -982> 2018-07-11 07:52:28.425677 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6810/15287 -- osd_ping(ping e501 stamp 2018-07-11 07:52:28.425524) v4 -- 0x55d9c34cca00 con 0 > -981> 2018-07-11 07:52:28.425688 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6811/15287 -- osd_ping(ping e501 stamp 2018-07-11 07:52:28.425524) v4 -- 0x55d980f5ba00 con 0 > -980> 2018-07-11 07:52:28.425703 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6806/15263 -- osd_ping(ping e501 stamp 2018-07-11 07:52:28.425524) v4 -- 0x55d9b67d9400 con 0 > -979> 2018-07-11 07:52:28.425719 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6807/15263 -- osd_ping(ping e501 stamp 2018-07-11 07:52:28.425524) v4 -- 0x55d986c60800 con 0 > -978> 2018-07-11 07:52:28.425731 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.115:6802/13323 -- osd_ping(ping e501 stamp 2018-07-11 07:52:28.425524) v4 -- 0x55d9c908fa00 con 0 > -977> 2018-07-11 07:52:28.425744 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.115:6803/13323 -- osd_ping(ping e501 stamp 2018-07-11 07:52:28.425524) v4 -- 0x55d9c4c1bc00 con 0 > -976> 2018-07-11 07:52:28.425757 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6806/15885 -- osd_ping(ping e501 stamp 2018-07-11 07:52:28.425524) v4 -- 0x55d986d32800 con 0 > -975> 2018-07-11 07:52:28.425768 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6807/15885 -- osd_ping(ping e501 stamp 2018-07-11 07:52:28.425524) v4 -- 0x55d9b4388a00 con 0 > -974> 2018-07-11 07:52:28.425780 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.108:6802/11967 -- osd_ping(ping e501 stamp 2018-07-11 07:52:28.425524) v4 -- 0x55d9bcd78a00 con 0 > -973> 2018-07-11 07:52:28.425792 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.108:6803/11967 -- osd_ping(ping e501 stamp 2018-07-11 07:52:28.425524) v4 -- 0x55d986c84200 con 0 > -972> 2018-07-11 07:52:28.425807 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.107:6806/12617 -- osd_ping(ping e501 stamp 2018-07-11 07:52:28.425524) v4 -- 0x55d9b60de400 con 0 > -971> 2018-07-11 07:52:28.425821 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.107:6807/12617 -- osd_ping(ping e501 stamp 2018-07-11 07:52:28.425524) v4 -- 0x55d986cd5000 con 0 > -970> 2018-07-11 07:52:28.425834 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6802/15919 -- osd_ping(ping e501 stamp 2018-07-11 07:52:28.425524) v4 -- 0x55d980f5b000 con 0 > -969> 2018-07-11 07:52:28.425846 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6803/15919 -- osd_ping(ping e501 stamp 2018-07-11 07:52:28.425524) v4 -- 0x55d9c9122e00 con 0 > -968> 2018-07-11 07:52:28.426000 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.107:6806/12617 conn(0x55d986bd2000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.9 seq 258 0x55d982d31600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:28.425524) v4 > -967> 2018-07-11 07:52:28.426023 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.9 10.8.128.107:6806/12617 258 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:28.425524) v4 ==== 2004+0+0 (2238430661 0 0) 0x55d982d31600 con 0x55d986bd2000 > -966> 2018-07-11 07:52:28.426031 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.107:6807/12617 conn(0x55d986bd3800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.9 seq 258 0x55d980457a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:28.425524) v4 > -965> 2018-07-11 07:52:28.426050 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.9 10.8.128.107:6807/12617 258 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:28.425524) v4 ==== 2004+0+0 (2238430661 0 0) 0x55d980457a00 con 0x55d986bd3800 > -964> 2018-07-11 07:52:28.426099 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6803/15255 conn(0x55d986bc9800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.0 seq 258 0x55d982d31600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:28.425524) v4 > -963> 2018-07-11 07:52:28.426129 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.0 10.8.128.113:6803/15255 258 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:28.425524) v4 ==== 2004+0+0 (2238430661 0 0) 0x55d982d31600 con 0x55d986bc9800 > -962> 2018-07-11 07:52:28.426146 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6808/12651 conn(0x55d986b9a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.2 seq 258 0x55d986d4e600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:28.425524) v4 > -961> 2018-07-11 07:52:28.426155 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6806/15263 conn(0x55d986b9d000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.5 seq 258 0x55d980457a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:28.425524) v4 > -960> 2018-07-11 07:52:28.426163 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.2 10.8.128.106:6808/12651 258 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:28.425524) v4 ==== 2004+0+0 (2238430661 0 0) 0x55d986d4e600 con 0x55d986b9a000 > -959> 2018-07-11 07:52:28.426185 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.5 10.8.128.113:6806/15263 258 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:28.425524) v4 ==== 2004+0+0 (2238430661 0 0) 0x55d980457a00 con 0x55d986b9d000 > -958> 2018-07-11 07:52:28.426203 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6802/15255 conn(0x55d986bc8000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.0 seq 258 0x55d986d4e600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:28.425524) v4 > -957> 2018-07-11 07:52:28.426216 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.0 10.8.128.113:6802/15255 258 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:28.425524) v4 ==== 2004+0+0 (2238430661 0 0) 0x55d986d4e600 con 0x55d986bc8000 > -956> 2018-07-11 07:52:28.426192 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6810/12651 conn(0x55d986b9b800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.2 seq 258 0x55d982d31600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:28.425524) v4 > -955> 2018-07-11 07:52:28.426222 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6809/12650 conn(0x55d986bcb000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 258 0x55d980457a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:28.425524) v4 > -954> 2018-07-11 07:52:28.426238 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.2 10.8.128.106:6810/12651 258 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:28.425524) v4 ==== 2004+0+0 (2238430661 0 0) 0x55d982d31600 con 0x55d986b9b800 > -953> 2018-07-11 07:52:28.426266 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6807/15263 conn(0x55d986b9e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.5 seq 258 0x55d986d4e600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:28.425524) v4 > -952> 2018-07-11 07:52:28.426279 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.5 10.8.128.113:6807/15263 258 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:28.425524) v4 ==== 2004+0+0 (2238430661 0 0) 0x55d986d4e600 con 0x55d986b9e800 > -951> 2018-07-11 07:52:28.426243 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.3 10.8.128.106:6809/12650 258 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:28.425524) v4 ==== 2004+0+0 (2238430661 0 0) 0x55d980457a00 con 0x55d986bcb000 > -950> 2018-07-11 07:52:28.426318 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6811/15287 conn(0x55d986bb0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 258 0x55d982d31600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:28.425524) v4 > -949> 2018-07-11 07:52:28.426332 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6806/12677 conn(0x55d986b58000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.1 seq 258 0x55d980457a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:28.425524) v4 > -948> 2018-07-11 07:52:28.426325 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6810/15287 conn(0x55d986baf000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 258 0x55d986d4e600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:28.425524) v4 > -947> 2018-07-11 07:52:28.426333 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.4 10.8.128.113:6811/15287 258 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:28.425524) v4 ==== 2004+0+0 (2238430661 0 0) 0x55d982d31600 con 0x55d986bb0800 > -946> 2018-07-11 07:52:28.426349 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.1 10.8.128.106:6806/12677 258 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:28.425524) v4 ==== 2004+0+0 (2238430661 0 0) 0x55d980457a00 con 0x55d986b58000 > -945> 2018-07-11 07:52:28.426349 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.4 10.8.128.113:6810/15287 258 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:28.425524) v4 ==== 2004+0+0 (2238430661 0 0) 0x55d986d4e600 con 0x55d986baf000 > -944> 2018-07-11 07:52:28.426386 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6807/15885 conn(0x55d986bad800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.7 seq 258 0x55d980457a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:28.425524) v4 > -943> 2018-07-11 07:52:28.426391 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6807/12677 conn(0x55d986b59800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.1 seq 258 0x55d986d4e600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:28.425524) v4 > -942> 2018-07-11 07:52:28.426398 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.7 10.8.128.114:6807/15885 258 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:28.425524) v4 ==== 2004+0+0 (2238430661 0 0) 0x55d980457a00 con 0x55d986bad800 > -941> 2018-07-11 07:52:28.426401 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.1 10.8.128.106:6807/12677 258 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:28.425524) v4 ==== 2004+0+0 (2238430661 0 0) 0x55d986d4e600 con 0x55d986b59800 > -940> 2018-07-11 07:52:28.426400 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.115:6802/13323 conn(0x55d986bc1000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 258 0x55d982d31600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:28.425524) v4 > -939> 2018-07-11 07:52:28.426418 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.6 10.8.128.115:6802/13323 258 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:28.425524) v4 ==== 2004+0+0 (2238430661 0 0) 0x55d982d31600 con 0x55d986bc1000 > -938> 2018-07-11 07:52:28.426426 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6803/15919 conn(0x55d986b5c800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.11 seq 258 0x55d980457a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:28.425524) v4 > -937> 2018-07-11 07:52:28.426433 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6811/12650 conn(0x55d986bcc800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 258 0x55d986d4e600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:28.425524) v4 > -936> 2018-07-11 07:52:28.426438 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.11 10.8.128.114:6803/15919 258 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:28.425524) v4 ==== 2004+0+0 (2238430661 0 0) 0x55d980457a00 con 0x55d986b5c800 > -935> 2018-07-11 07:52:28.426448 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.3 10.8.128.106:6811/12650 258 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:28.425524) v4 ==== 2004+0+0 (2238430661 0 0) 0x55d986d4e600 con 0x55d986bcc800 > -934> 2018-07-11 07:52:28.426469 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6806/15885 conn(0x55d986bac000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.7 seq 258 0x55d982d31600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:28.425524) v4 > -933> 2018-07-11 07:52:28.426483 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.7 10.8.128.114:6806/15885 258 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:28.425524) v4 ==== 2004+0+0 (2238430661 0 0) 0x55d982d31600 con 0x55d986bac000 > -932> 2018-07-11 07:52:28.426489 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.115:6803/13323 conn(0x55d986bc2800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 258 0x55d980457a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:28.425524) v4 > -931> 2018-07-11 07:52:28.426496 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.108:6803/11967 conn(0x55d986bbf800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.8 seq 258 0x55d986d4e600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:28.425524) v4 > -930> 2018-07-11 07:52:28.426502 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.6 10.8.128.115:6803/13323 258 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:28.425524) v4 ==== 2004+0+0 (2238430661 0 0) 0x55d980457a00 con 0x55d986bc2800 > -929> 2018-07-11 07:52:28.426508 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.8 10.8.128.108:6803/11967 258 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:28.425524) v4 ==== 2004+0+0 (2238430661 0 0) 0x55d986d4e600 con 0x55d986bbf800 > -928> 2018-07-11 07:52:28.426531 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6802/15919 conn(0x55d986b5b000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.11 seq 258 0x55d982d31600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:28.425524) v4 > -927> 2018-07-11 07:52:28.426536 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.108:6802/11967 conn(0x55d986bbe000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.8 seq 258 0x55d980457a00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:28.425524) v4 > -926> 2018-07-11 07:52:28.426546 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.8 10.8.128.108:6802/11967 258 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:28.425524) v4 ==== 2004+0+0 (2238430661 0 0) 0x55d980457a00 con 0x55d986bbe000 > -925> 2018-07-11 07:52:28.426545 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.11 10.8.128.114:6802/15919 258 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:28.425524) v4 ==== 2004+0+0 (2238430661 0 0) 0x55d982d31600 con 0x55d986b5b000 > -924> 2018-07-11 07:52:28.614023 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6805/15263 -- MOSDScrubReserve(6.15 REJECT e501) v1 -- 0x55d986c22e00 con 0 > -923> 2018-07-11 07:52:28.614243 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -922> 2018-07-11 07:52:28.614278 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -921> 2018-07-11 07:52:28.614325 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -920> 2018-07-11 07:52:28.614505 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6805/15263 conn(0x55d986c41000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=31 cs=1 l=0). rx osd.5 seq 2849 0x55d986c22e00 MOSDScrubReserve(6.15 RELEASE e501) v1 > -919> 2018-07-11 07:52:28.614524 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.5 10.8.128.113:6805/15263 2849 ==== MOSDScrubReserve(6.15 RELEASE e501) v1 ==== 43+0+0 (605848072 0 0) 0x55d986c22e00 con 0x55d986c41000 > -918> 2018-07-11 07:52:28.614783 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -917> 2018-07-11 07:52:28.630885 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 34] Generated table #36814: 19370 keys, 548228 bytes > -916> 2018-07-11 07:52:28.630924 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295548630910, "cf_name": "default", "job": 34, "event": "table_file_creation", "file_number": 36814, "file_size": 548228, "table_properties": {"data_size": 492756, "index_size": 4977, "filter_size": 49516, "raw_key_size": 559577, "raw_average_key_size": 28, "raw_value_size": 230287, "raw_average_value_size": 11, "num_data_blocks": 122, "num_entries": 19370, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} > -915> 2018-07-11 07:52:28.637405 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75851922:::benchmark_data_magna115_10867_object1519105:0,end:5:75856c42:::benchmark_data_magna115_11707_object156095:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98696e000 con 0 > -914> 2018-07-11 07:52:28.637443 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75851922:::benchmark_data_magna115_10867_object1519105:0,end:5:75856c42:::benchmark_data_magna115_11707_object156095:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98696f400 con 0 > -913> 2018-07-11 07:52:28.671028 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12677 conn(0x55d986db4800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.1 seq 271 0x55d982d31600 osd_ping(ping e501 stamp 2018-07-11 07:52:28.719313) v4 > -912> 2018-07-11 07:52:28.671054 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.1 10.8.128.106:0/12677 271 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:28.719313) v4 ==== 2004+0+0 (300616111 0 0) 0x55d982d31600 con 0x55d986db4800 > -911> 2018-07-11 07:52:28.671054 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12677 conn(0x55d986df2000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.1 seq 271 0x55d9bdba4c00 osd_ping(ping e501 stamp 2018-07-11 07:52:28.719313) v4 > -910> 2018-07-11 07:52:28.671069 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:28.719313) v4 -- 0x55d980f5b000 con 0 > -909> 2018-07-11 07:52:28.671078 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.1 10.8.128.106:0/12677 271 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:28.719313) v4 ==== 2004+0+0 (300616111 0 0) 0x55d9bdba4c00 con 0x55d986df2000 > -908> 2018-07-11 07:52:28.671099 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:28.719313) v4 -- 0x55d986c84200 con 0 > -907> 2018-07-11 07:52:28.835900 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1079 0x55d983050d80 rep_scrubmap(5.2e e501 from shard 2) v1 > -906> 2018-07-11 07:52:28.835920 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1079 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12474 (1269249638 0 92703710) 0x55d983050d80 con 0x55d986bf0800 > -905> 2018-07-11 07:52:28.844017 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1073 0x55d983050fc0 rep_scrubmap(5.2e e501 from shard 4) v1 > -904> 2018-07-11 07:52:28.844036 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1073 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12474 (3686815630 0 92703710) 0x55d983050fc0 con 0x55d986bec000 > -903> 2018-07-11 07:52:28.881687 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -902> 2018-07-11 07:52:28.881711 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -901> 2018-07-11 07:52:28.881771 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -900> 2018-07-11 07:52:28.882227 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -899> 2018-07-11 07:52:28.899872 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.114:0/15919 conn(0x55d986df3800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.11 seq 270 0x55d980457a00 osd_ping(ping e501 stamp 2018-07-11 07:52:29.530439) v4 > -898> 2018-07-11 07:52:28.899893 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.11 10.8.128.114:0/15919 270 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:29.530439) v4 ==== 2004+0+0 (4121344081 0 0) 0x55d980457a00 con 0x55d986df3800 > -897> 2018-07-11 07:52:28.899902 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:29.530439) v4 -- 0x55d9c9122e00 con 0 > -896> 2018-07-11 07:52:28.899899 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.114:0/15919 conn(0x55d986df5000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.11 seq 270 0x55d986d4e200 osd_ping(ping e501 stamp 2018-07-11 07:52:29.530439) v4 > -895> 2018-07-11 07:52:28.899925 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.11 10.8.128.114:0/15919 270 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:29.530439) v4 ==== 2004+0+0 (4121344081 0 0) 0x55d986d4e200 con 0x55d986df5000 > -894> 2018-07-11 07:52:28.899941 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:29.530439) v4 -- 0x55d980f5b000 con 0 > -893> 2018-07-11 07:52:28.927559 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75856c42:::benchmark_data_magna115_11707_object156095:0,end:5:7585d239:::benchmark_data_magna115_11707_object285494:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b14f0000 con 0 > -892> 2018-07-11 07:52:28.927610 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75856c42:::benchmark_data_magna115_11707_object156095:0,end:5:7585d239:::benchmark_data_magna115_11707_object285494:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b14f0280 con 0 > -891> 2018-07-11 07:52:29.092465 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15287 conn(0x55d98123f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 281 0x55d982d31600 osd_ping(ping e501 stamp 2018-07-11 07:52:30.324297) v4 > -890> 2018-07-11 07:52:29.092486 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.4 10.8.128.113:0/15287 281 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:30.324297) v4 ==== 2004+0+0 (2192020503 0 0) 0x55d982d31600 con 0x55d98123f000 > -889> 2018-07-11 07:52:29.092502 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:30.324297) v4 -- 0x55d980f5b000 con 0 > -888> 2018-07-11 07:52:29.092517 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15287 conn(0x55d981240800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 281 0x55d9bdba4c00 osd_ping(ping e501 stamp 2018-07-11 07:52:30.324297) v4 > -887> 2018-07-11 07:52:29.092543 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.4 10.8.128.113:0/15287 281 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:30.324297) v4 ==== 2004+0+0 (2192020503 0 0) 0x55d9bdba4c00 con 0x55d981240800 > -886> 2018-07-11 07:52:29.092559 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:30.324297) v4 -- 0x55d986c84200 con 0 > -885> 2018-07-11 07:52:29.144876 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1080 0x55d983051200 rep_scrubmap(5.2e e501 from shard 2) v1 > -884> 2018-07-11 07:52:29.144896 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1080 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12474 (1269249638 0 2232166371) 0x55d983051200 con 0x55d986bf0800 > -883> 2018-07-11 07:52:29.162276 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1074 0x55d983051440 rep_scrubmap(5.2e e501 from shard 4) v1 > -882> 2018-07-11 07:52:29.162295 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1074 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12474 (3686815630 0 2232166371) 0x55d983051440 con 0x55d986bec000 > -881> 2018-07-11 07:52:29.254146 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15263 conn(0x55d985d45800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.5 seq 250 0x55d986d4e200 osd_ping(ping e501 stamp 2018-07-11 07:52:30.486029) v4 > -880> 2018-07-11 07:52:29.254144 7ffacbbd1700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15263 conn(0x55d985d44000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.5 seq 250 0x55d9869bfc00 osd_ping(ping e501 stamp 2018-07-11 07:52:30.486029) v4 > -879> 2018-07-11 07:52:29.254171 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.5 10.8.128.113:0/15263 250 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:30.486029) v4 ==== 2004+0+0 (773153675 0 0) 0x55d986d4e200 con 0x55d985d45800 > -878> 2018-07-11 07:52:29.254176 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 <== osd.5 10.8.128.113:0/15263 250 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:30.486029) v4 ==== 2004+0+0 (773153675 0 0) 0x55d9869bfc00 con 0x55d985d44000 > -877> 2018-07-11 07:52:29.254188 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:30.486029) v4 -- 0x55d980f5b000 con 0 > -876> 2018-07-11 07:52:29.254218 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:30.486029) v4 -- 0x55d9c9122e00 con 0 > -875> 2018-07-11 07:52:29.293907 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -874> 2018-07-11 07:52:29.293933 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -873> 2018-07-11 07:52:29.293979 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -872> 2018-07-11 07:52:29.294448 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -871> 2018-07-11 07:52:29.314578 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7585d239:::benchmark_data_magna115_11707_object285494:0,end:5:758657ba:::benchmark_data_magna115_10867_object851934:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1499900 con 0 > -870> 2018-07-11 07:52:29.314628 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7585d239:::benchmark_data_magna115_11707_object285494:0,end:5:758657ba:::benchmark_data_magna115_10867_object851934:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1499680 con 0 > -869> 2018-07-11 07:52:29.503579 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1081 0x55d983051680 rep_scrubmap(5.2e e501 from shard 2) v1 > -868> 2018-07-11 07:52:29.503599 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1081 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12474 (1269249638 0 3294557354) 0x55d983051680 con 0x55d986bf0800 > -867> 2018-07-11 07:52:29.509391 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.115:0/13323 conn(0x55d986d7e000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 269 0x55d980457a00 osd_ping(ping e501 stamp 2018-07-11 07:52:29.855548) v4 > -866> 2018-07-11 07:52:29.509412 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.6 10.8.128.115:0/13323 269 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:29.855548) v4 ==== 2004+0+0 (2167273203 0 0) 0x55d980457a00 con 0x55d986d7e000 > -865> 2018-07-11 07:52:29.509429 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:29.855548) v4 -- 0x55d9c9122e00 con 0 > -864> 2018-07-11 07:52:29.509455 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.115:0/13323 conn(0x55d986d24800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 269 0x55d9bdba4c00 osd_ping(ping e501 stamp 2018-07-11 07:52:29.855548) v4 > -863> 2018-07-11 07:52:29.509476 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.6 10.8.128.115:0/13323 269 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:29.855548) v4 ==== 2004+0+0 (2167273203 0 0) 0x55d9bdba4c00 con 0x55d986d24800 > -862> 2018-07-11 07:52:29.509487 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:29.855548) v4 -- 0x55d986c84200 con 0 > -861> 2018-07-11 07:52:29.533465 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1075 0x55d9830518c0 rep_scrubmap(5.2e e501 from shard 4) v1 > -860> 2018-07-11 07:52:29.533483 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1075 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12474 (3686815630 0 3294557354) 0x55d9830518c0 con 0x55d986bec000 > -859> 2018-07-11 07:52:29.692444 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -858> 2018-07-11 07:52:29.692467 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -857> 2018-07-11 07:52:29.692523 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -856> 2018-07-11 07:52:29.692980 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -855> 2018-07-11 07:52:29.737584 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:758657ba:::benchmark_data_magna115_10867_object851934:0,end:5:7586a32b:::benchmark_data_magna115_10867_object2234409:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498c80 con 0 > -854> 2018-07-11 07:52:29.737635 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:758657ba:::benchmark_data_magna115_10867_object851934:0,end:5:7586a32b:::benchmark_data_magna115_10867_object2234409:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498500 con 0 > -853> 2018-07-11 07:52:29.754457 7ffacbbd1700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15263 conn(0x55d985d44000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.5 seq 251 0x55d9869bfc00 osd_ping(ping e501 stamp 2018-07-11 07:52:30.986472) v4 > -852> 2018-07-11 07:52:29.754478 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 <== osd.5 10.8.128.113:0/15263 251 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:30.986472) v4 ==== 2004+0+0 (3518208590 0 0) 0x55d9869bfc00 con 0x55d985d44000 > -851> 2018-07-11 07:52:29.754489 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:30.986472) v4 -- 0x55d9c9122e00 con 0 > -850> 2018-07-11 07:52:29.754533 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15263 conn(0x55d985d45800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.5 seq 251 0x55d982d31600 osd_ping(ping e501 stamp 2018-07-11 07:52:30.986472) v4 > -849> 2018-07-11 07:52:29.754553 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.5 10.8.128.113:0/15263 251 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:30.986472) v4 ==== 2004+0+0 (3518208590 0 0) 0x55d982d31600 con 0x55d985d45800 > -848> 2018-07-11 07:52:29.754569 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:30.986472) v4 -- 0x55d980f5b000 con 0 > -847> 2018-07-11 07:52:29.971021 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1082 0x55d983051b00 rep_scrubmap(5.2e e501 from shard 2) v1 > -846> 2018-07-11 07:52:29.971040 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1082 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12476 (1269249638 0 883569618) 0x55d983051b00 con 0x55d986bf0800 > -845> 2018-07-11 07:52:29.976904 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1076 0x55d983051d40 rep_scrubmap(5.2e e501 from shard 4) v1 > -844> 2018-07-11 07:52:29.976924 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1076 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12476 (3686815630 0 883569618) 0x55d983051d40 con 0x55d986bec000 > -843> 2018-07-11 07:52:30.084694 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -842> 2018-07-11 07:52:30.084717 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -841> 2018-07-11 07:52:30.084774 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -840> 2018-07-11 07:52:30.085228 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -839> 2018-07-11 07:52:30.129578 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7586a32b:::benchmark_data_magna115_10867_object2234409:0,end:5:7586fe7f:::benchmark_data_magna115_11707_object136642:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1499180 con 0 > -838> 2018-07-11 07:52:30.129625 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7586a32b:::benchmark_data_magna115_10867_object2234409:0,end:5:7586fe7f:::benchmark_data_magna115_11707_object136642:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498f00 con 0 > -837> 2018-07-11 07:52:30.224719 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.114:0/15885 conn(0x55d986d7f800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.7 seq 273 0x55d986d4e200 osd_ping(ping e501 stamp 2018-07-11 07:52:30.855262) v4 > -836> 2018-07-11 07:52:30.224728 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.114:0/15885 conn(0x55d986d81000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.7 seq 273 0x55d9bdba4c00 osd_ping(ping e501 stamp 2018-07-11 07:52:30.855262) v4 > -835> 2018-07-11 07:52:30.224747 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.7 10.8.128.114:0/15885 273 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:30.855262) v4 ==== 2004+0+0 (2920946660 0 0) 0x55d9bdba4c00 con 0x55d986d81000 > -834> 2018-07-11 07:52:30.224745 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.7 10.8.128.114:0/15885 273 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:30.855262) v4 ==== 2004+0+0 (2920946660 0 0) 0x55d986d4e200 con 0x55d986d7f800 > -833> 2018-07-11 07:52:30.224767 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:30.855262) v4 -- 0x55d986c84200 con 0 > -832> 2018-07-11 07:52:30.224791 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:30.855262) v4 -- 0x55d980f5b000 con 0 > -831> 2018-07-11 07:52:30.341130 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1083 0x55d9b36e8000 rep_scrubmap(5.2e e501 from shard 2) v1 > -830> 2018-07-11 07:52:30.341151 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1083 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12478 (1269249638 0 2955335627) 0x55d9b36e8000 con 0x55d986bf0800 > -829> 2018-07-11 07:52:30.356093 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1077 0x55d9b36e8240 rep_scrubmap(5.2e e501 from shard 4) v1 > -828> 2018-07-11 07:52:30.356114 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1077 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12478 (3686815630 0 2955335627) 0x55d9b36e8240 con 0x55d986bec000 > -827> 2018-07-11 07:52:30.474062 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15255 conn(0x55d986d20000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.0 seq 267 0x55d986d4e600 osd_ping(ping e501 stamp 2018-07-11 07:52:31.705917) v4 > -826> 2018-07-11 07:52:30.474064 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15255 conn(0x55d986c82800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.0 seq 267 0x55d982d31600 osd_ping(ping e501 stamp 2018-07-11 07:52:31.705917) v4 > -825> 2018-07-11 07:52:30.474087 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.0 10.8.128.113:0/15255 267 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:31.705917) v4 ==== 2004+0+0 (2044783808 0 0) 0x55d986d4e600 con 0x55d986d20000 > -824> 2018-07-11 07:52:30.474091 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.0 10.8.128.113:0/15255 267 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:31.705917) v4 ==== 2004+0+0 (2044783808 0 0) 0x55d982d31600 con 0x55d986c82800 > -823> 2018-07-11 07:52:30.474099 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:31.705917) v4 -- 0x55d986c84200 con 0 > -822> 2018-07-11 07:52:30.474121 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:31.705917) v4 -- 0x55d980f5b000 con 0 > -821> 2018-07-11 07:52:30.516947 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -820> 2018-07-11 07:52:30.516973 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -819> 2018-07-11 07:52:30.517018 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -818> 2018-07-11 07:52:30.517483 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -817> 2018-07-11 07:52:30.562988 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7586fe7f:::benchmark_data_magna115_11707_object136642:0,end:5:75874b0d:::benchmark_data_magna115_11707_object42080:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1499b80 con 0 > -816> 2018-07-11 07:52:30.563040 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7586fe7f:::benchmark_data_magna115_11707_object136642:0,end:5:75874b0d:::benchmark_data_magna115_11707_object42080:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1499400 con 0 > -815> 2018-07-11 07:52:30.784053 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1084 0x55d9b36e8480 rep_scrubmap(5.2e e501 from shard 2) v1 > -814> 2018-07-11 07:52:30.784073 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1084 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12480 (1269249638 0 1429643441) 0x55d9b36e8480 con 0x55d986bf0800 > -813> 2018-07-11 07:52:30.795127 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1078 0x55d9b36e86c0 rep_scrubmap(5.2e e501 from shard 4) v1 > -812> 2018-07-11 07:52:30.795145 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1078 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12480 (3686815630 0 1429643441) 0x55d9b36e86c0 con 0x55d986bec000 > -811> 2018-07-11 07:52:30.841093 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -810> 2018-07-11 07:52:30.841122 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -809> 2018-07-11 07:52:30.841204 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -808> 2018-07-11 07:52:30.842013 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -807> 2018-07-11 07:52:30.847736 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 3044 0x55d980457a00 MOSDScrubReserve(11.3s5 REQUEST e501) v1 > -806> 2018-07-11 07:52:30.847756 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 3044 ==== MOSDScrubReserve(11.3s5 REQUEST e501) v1 ==== 43+0+0 (3260170403 0 0) 0x55d980457a00 con 0x55d986c7e000 > -805> 2018-07-11 07:52:30.847829 7ffab3be0700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e501) v1 -- 0x55d980f5a200 con 0 > -804> 2018-07-11 07:52:30.848275 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 3045 0x55d980f5a200 MOSDScrubReserve(11.3s5 RELEASE e501) v1 > -803> 2018-07-11 07:52:30.848293 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 3045 ==== MOSDScrubReserve(11.3s5 RELEASE e501) v1 ==== 43+0+0 (223328929 0 0) 0x55d980f5a200 con 0x55d986c7e000 > -802> 2018-07-11 07:52:30.860528 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.107:0/12617 conn(0x55d98123c000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 262 0x55d986d4e600 osd_ping(ping e501 stamp 2018-07-11 07:52:30.860008) v4 > -801> 2018-07-11 07:52:30.860554 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.9 10.8.128.107:0/12617 262 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:30.860008) v4 ==== 2004+0+0 (1445236388 0 0) 0x55d986d4e600 con 0x55d98123c000 > -800> 2018-07-11 07:52:30.860557 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.107:0/12617 conn(0x55d98123d800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 262 0x55d9869bfc00 osd_ping(ping e501 stamp 2018-07-11 07:52:30.860008) v4 > -799> 2018-07-11 07:52:30.860572 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:30.860008) v4 -- 0x55d986c84200 con 0 > -798> 2018-07-11 07:52:30.860579 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.9 10.8.128.107:0/12617 262 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:30.860008) v4 ==== 2004+0+0 (1445236388 0 0) 0x55d9869bfc00 con 0x55d98123d800 > -797> 2018-07-11 07:52:30.860607 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:30.860008) v4 -- 0x55d986cd5000 con 0 > -796> 2018-07-11 07:52:30.862879 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75874b0d:::benchmark_data_magna115_11707_object42080:0,end:5:758796b7:::benchmark_data_magna115_10867_object2443456:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b14f0a00 con 0 > -795> 2018-07-11 07:52:30.862925 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75874b0d:::benchmark_data_magna115_11707_object42080:0,end:5:758796b7:::benchmark_data_magna115_10867_object2443456:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98696ef00 con 0 > -794> 2018-07-11 07:52:31.030007 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12651 conn(0x55d986db3000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.2 seq 265 0x55d9c9122e00 osd_ping(ping e501 stamp 2018-07-11 07:52:31.078262) v4 > -793> 2018-07-11 07:52:31.030028 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.2 10.8.128.106:0/12651 265 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:31.078262) v4 ==== 2004+0+0 (2662994242 0 0) 0x55d9c9122e00 con 0x55d986db3000 > -792> 2018-07-11 07:52:31.030015 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12651 conn(0x55d986db1800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.2 seq 265 0x55d9bdba4c00 osd_ping(ping e501 stamp 2018-07-11 07:52:31.078262) v4 > -791> 2018-07-11 07:52:31.030038 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:31.078262) v4 -- 0x55d986cd5000 con 0 > -790> 2018-07-11 07:52:31.030038 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.2 10.8.128.106:0/12651 265 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:31.078262) v4 ==== 2004+0+0 (2662994242 0 0) 0x55d9bdba4c00 con 0x55d986db1800 > -789> 2018-07-11 07:52:31.030057 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:31.078262) v4 -- 0x55d986c84200 con 0 > -788> 2018-07-11 07:52:31.067455 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1085 0x55d9b36e8900 rep_scrubmap(5.2e e501 from shard 2) v1 > -787> 2018-07-11 07:52:31.067474 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1085 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12472 (1269249638 0 3660850328) 0x55d9b36e8900 con 0x55d986bf0800 > -786> 2018-07-11 07:52:31.115056 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1079 0x55d9b36e8b40 rep_scrubmap(5.2e e501 from shard 4) v1 > -785> 2018-07-11 07:52:31.115076 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1079 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12472 (3686815630 0 3660850328) 0x55d9b36e8b40 con 0x55d986bec000 > -784> 2018-07-11 07:52:31.157275 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -783> 2018-07-11 07:52:31.157299 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -782> 2018-07-11 07:52:31.157356 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -781> 2018-07-11 07:52:31.157817 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -780> 2018-07-11 07:52:31.190955 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:758796b7:::benchmark_data_magna115_10867_object2443456:0,end:5:7587fba4:::benchmark_data_magna115_11707_object419006:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98696ec80 con 0 > -779> 2018-07-11 07:52:31.191006 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:758796b7:::benchmark_data_magna115_10867_object2443456:0,end:5:7587fba4:::benchmark_data_magna115_11707_object419006:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98696f680 con 0 > -778> 2018-07-11 07:52:31.396651 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.108:0/11967 conn(0x55d986db0000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.8 seq 251 0x55d986d4e200 osd_ping(ping e501 stamp 2018-07-11 07:52:31.152590) v4 > -777> 2018-07-11 07:52:31.396683 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.8 10.8.128.108:0/11967 251 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:31.152590) v4 ==== 2004+0+0 (2714877935 0 0) 0x55d986d4e200 con 0x55d986db0000 > -776> 2018-07-11 07:52:31.396682 7ffacbbd1700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.108:0/11967 conn(0x55d986d82800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.8 seq 251 0x55d9869bfc00 osd_ping(ping e501 stamp 2018-07-11 07:52:31.152590) v4 > -775> 2018-07-11 07:52:31.396700 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 <== osd.8 10.8.128.108:0/11967 251 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:31.152590) v4 ==== 2004+0+0 (2714877935 0 0) 0x55d9869bfc00 con 0x55d986d82800 > -774> 2018-07-11 07:52:31.396702 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:31.152590) v4 -- 0x55d980f5b000 con 0 > -773> 2018-07-11 07:52:31.396733 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:31.152590) v4 -- 0x55d986cd5000 con 0 > -772> 2018-07-11 07:52:31.407866 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1080 0x55d9b36e8d80 rep_scrubmap(5.2e e501 from shard 4) v1 > -771> 2018-07-11 07:52:31.407886 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1080 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12464 (3686815630 0 4025952156) 0x55d9b36e8d80 con 0x55d986bec000 > -770> 2018-07-11 07:52:31.430652 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1086 0x55d9b36e8fc0 rep_scrubmap(5.2e e501 from shard 2) v1 > -769> 2018-07-11 07:52:31.430672 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1086 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12464 (1269249638 0 4025952156) 0x55d9b36e8fc0 con 0x55d986bf0800 > -768> 2018-07-11 07:52:31.462357 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -767> 2018-07-11 07:52:31.462381 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -766> 2018-07-11 07:52:31.462440 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -765> 2018-07-11 07:52:31.462900 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -764> 2018-07-11 07:52:31.513178 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'66792,epoch:501/498,start:5:7587fba4:::benchmark_data_magna115_11707_object419006:0,end:5:758874c2:::benchmark_data_magna115_10867_object1091127:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98696f900 con 0 > -763> 2018-07-11 07:52:31.513229 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'66792,epoch:501/498,start:5:7587fba4:::benchmark_data_magna115_11707_object419006:0,end:5:758874c2:::benchmark_data_magna115_10867_object1091127:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98696f180 con 0 > -762> 2018-07-11 07:52:31.699432 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1087 0x55d9b36e9200 rep_scrubmap(5.2e e501 from shard 2) v1 > -761> 2018-07-11 07:52:31.699453 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1087 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12476 (1269249638 0 3290644257) 0x55d9b36e9200 con 0x55d986bf0800 > -760> 2018-07-11 07:52:31.709885 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1081 0x55d9b36e9440 rep_scrubmap(5.2e e501 from shard 4) v1 > -759> 2018-07-11 07:52:31.709905 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1081 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12476 (3686815630 0 3290644257) 0x55d9b36e9440 con 0x55d986bec000 > -758> 2018-07-11 07:52:31.724475 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12650 conn(0x55d986d21800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 261 0x55d9c9122e00 osd_ping(ping e501 stamp 2018-07-11 07:52:31.772806) v4 > -757> 2018-07-11 07:52:31.724497 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.3 10.8.128.106:0/12650 261 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:31.772806) v4 ==== 2004+0+0 (4085912028 0 0) 0x55d9c9122e00 con 0x55d986d21800 > -756> 2018-07-11 07:52:31.724506 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:31.772806) v4 -- 0x55d986cd5000 con 0 > -755> 2018-07-11 07:52:31.724500 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12650 conn(0x55d986d23000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 261 0x55d982d31600 osd_ping(ping e501 stamp 2018-07-11 07:52:31.772806) v4 > -754> 2018-07-11 07:52:31.724518 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.3 10.8.128.106:0/12650 261 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:31.772806) v4 ==== 2004+0+0 (4085912028 0 0) 0x55d982d31600 con 0x55d986d23000 > -753> 2018-07-11 07:52:31.724531 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:31.772806) v4 -- 0x55d980f5b000 con 0 > -752> 2018-07-11 07:52:31.785928 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -751> 2018-07-11 07:52:31.785951 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -750> 2018-07-11 07:52:31.786005 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -749> 2018-07-11 07:52:31.786452 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -748> 2018-07-11 07:52:31.829102 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67291,epoch:501/498,start:5:758874c2:::benchmark_data_magna115_10867_object1091127:0,end:5:7588d76d:::benchmark_data_magna115_10867_object3502447:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498a00 con 0 > -747> 2018-07-11 07:52:31.829154 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67291,epoch:501/498,start:5:758874c2:::benchmark_data_magna115_10867_object1091127:0,end:5:7588d76d:::benchmark_data_magna115_10867_object3502447:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498780 con 0 > -746> 2018-07-11 07:52:31.969356 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6801/15919 conn(0x55d986c3f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=29 cs=1 l=0). rx osd.11 seq 256 0x55d9869bfc00 MOSDScrubReserve(10.f REQUEST e501) v1 > -745> 2018-07-11 07:52:31.969375 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.11 10.8.128.114:6801/15919 256 ==== MOSDScrubReserve(10.f REQUEST e501) v1 ==== 43+0+0 (2892966580 0 0) 0x55d9869bfc00 con 0x55d986c3f800 > -744> 2018-07-11 07:52:31.969408 7ffab53e3700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.114:6801/15919 -- MOSDScrubReserve(10.f REJECT e501) v1 -- 0x55d9b4388200 con 0 > -743> 2018-07-11 07:52:31.969758 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6801/15919 conn(0x55d986c3f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=29 cs=1 l=0). rx osd.11 seq 257 0x55d9b4388200 MOSDScrubReserve(10.f RELEASE e501) v1 > -742> 2018-07-11 07:52:31.969772 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.11 10.8.128.114:6801/15919 257 ==== MOSDScrubReserve(10.f RELEASE e501) v1 ==== 43+0+0 (1668468918 0 0) 0x55d9b4388200 con 0x55d986c3f800 > -741> 2018-07-11 07:52:32.050028 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1088 0x55d9b36e9680 rep_scrubmap(5.2e e501 from shard 2) v1 > -740> 2018-07-11 07:52:32.050047 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1088 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12456 (1269249638 0 3475025999) 0x55d9b36e9680 con 0x55d986bf0800 > -739> 2018-07-11 07:52:32.068027 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1082 0x55d9b36e98c0 rep_scrubmap(5.2e e501 from shard 4) v1 > -738> 2018-07-11 07:52:32.068047 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1082 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12456 (3686815630 0 3475025999) 0x55d9b36e98c0 con 0x55d986bec000 > -737> 2018-07-11 07:52:32.130738 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -736> 2018-07-11 07:52:32.130765 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -735> 2018-07-11 07:52:32.130823 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -734> 2018-07-11 07:52:32.131316 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -733> 2018-07-11 07:52:32.146738 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7588d76d:::benchmark_data_magna115_10867_object3502447:0,end:5:7589447b:::benchmark_data_magna115_10867_object3391491:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98696e000 con 0 > -732> 2018-07-11 07:52:32.146790 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7588d76d:::benchmark_data_magna115_10867_object3502447:0,end:5:7589447b:::benchmark_data_magna115_10867_object3391491:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455680 con 0 > -731> 2018-07-11 07:52:32.302567 7ffab7be8700 10 monclient: tick > -730> 2018-07-11 07:52:32.302582 7ffab7be8700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2018-07-11 07:52:02.302580) > -729> 2018-07-11 07:52:32.341209 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1089 0x55d9b36e9b00 rep_scrubmap(5.2e e501 from shard 2) v1 > -728> 2018-07-11 07:52:32.341228 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1089 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12476 (1269249638 0 1331614478) 0x55d9b36e9b00 con 0x55d986bf0800 > -727> 2018-07-11 07:52:32.353808 7ffab6be6700 1 -- 10.8.128.107:6800/13927 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55d9b3fdf080 con 0 > -726> 2018-07-11 07:52:32.353877 7ffab6be6700 1 -- 10.8.128.107:6800/13927 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55d986cfeb00 con 0 > -725> 2018-07-11 07:52:32.368665 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1083 0x55d9b36e9d40 rep_scrubmap(5.2e e501 from shard 4) v1 > -724> 2018-07-11 07:52:32.368684 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1083 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12476 (3686815630 0 1331614478) 0x55d9b36e9d40 con 0x55d986bec000 > -723> 2018-07-11 07:52:32.453009 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -722> 2018-07-11 07:52:32.453038 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -721> 2018-07-11 07:52:32.453120 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -720> 2018-07-11 07:52:32.453914 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -719> 2018-07-11 07:52:32.455093 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7589447b:::benchmark_data_magna115_10867_object3391491:0,end:5:7589b898:::benchmark_data_magna115_10867_object2226920:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455b80 con 0 > -718> 2018-07-11 07:52:32.455146 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7589447b:::benchmark_data_magna115_10867_object3391491:0,end:5:7589b898:::benchmark_data_magna115_10867_object2226920:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455900 con 0 > -717> 2018-07-11 07:52:32.526006 7ffab1bdc700 5 osd.10 501 heartbeat: osd_stat(789 GB used, 142 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist []) > -716> 2018-07-11 07:52:32.526018 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6802/15255 -- osd_ping(ping e501 stamp 2018-07-11 07:52:32.526017) v4 -- 0x55d985055600 con 0 > -715> 2018-07-11 07:52:32.526055 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6803/15255 -- osd_ping(ping e501 stamp 2018-07-11 07:52:32.526017) v4 -- 0x55d9b67d9000 con 0 > -714> 2018-07-11 07:52:32.526069 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6806/12677 -- osd_ping(ping e501 stamp 2018-07-11 07:52:32.526017) v4 -- 0x55d9b9ac2800 con 0 > -713> 2018-07-11 07:52:32.526083 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6807/12677 -- osd_ping(ping e501 stamp 2018-07-11 07:52:32.526017) v4 -- 0x55d986c22a00 con 0 > -712> 2018-07-11 07:52:32.526108 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6808/12651 -- osd_ping(ping e501 stamp 2018-07-11 07:52:32.526017) v4 -- 0x55d986d97a00 con 0 > -711> 2018-07-11 07:52:32.526120 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6810/12651 -- osd_ping(ping e501 stamp 2018-07-11 07:52:32.526017) v4 -- 0x55d9869be800 con 0 > -710> 2018-07-11 07:52:32.526132 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6809/12650 -- osd_ping(ping e501 stamp 2018-07-11 07:52:32.526017) v4 -- 0x55d9829e5c00 con 0 > -709> 2018-07-11 07:52:32.526144 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6811/12650 -- osd_ping(ping e501 stamp 2018-07-11 07:52:32.526017) v4 -- 0x55d986d32600 con 0 > -708> 2018-07-11 07:52:32.526162 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6810/15287 -- osd_ping(ping e501 stamp 2018-07-11 07:52:32.526017) v4 -- 0x55d986c23600 con 0 > -707> 2018-07-11 07:52:32.526174 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6811/15287 -- osd_ping(ping e501 stamp 2018-07-11 07:52:32.526017) v4 -- 0x55d9b2782200 con 0 > -706> 2018-07-11 07:52:32.526187 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6806/15263 -- osd_ping(ping e501 stamp 2018-07-11 07:52:32.526017) v4 -- 0x55d986d97c00 con 0 > -705> 2018-07-11 07:52:32.526203 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6807/15263 -- osd_ping(ping e501 stamp 2018-07-11 07:52:32.526017) v4 -- 0x55d9c34cc200 con 0 > -704> 2018-07-11 07:52:32.526215 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.115:6802/13323 -- osd_ping(ping e501 stamp 2018-07-11 07:52:32.526017) v4 -- 0x55d9b3db3000 con 0 > -703> 2018-07-11 07:52:32.526229 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.115:6803/13323 -- osd_ping(ping e501 stamp 2018-07-11 07:52:32.526017) v4 -- 0x55d9bf804400 con 0 > -702> 2018-07-11 07:52:32.526242 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6806/15885 -- osd_ping(ping e501 stamp 2018-07-11 07:52:32.526017) v4 -- 0x55d9b4388a00 con 0 > -701> 2018-07-11 07:52:32.526270 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6807/15885 -- osd_ping(ping e501 stamp 2018-07-11 07:52:32.526017) v4 -- 0x55d9c4c1bc00 con 0 > -700> 2018-07-11 07:52:32.526284 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.108:6802/11967 -- osd_ping(ping e501 stamp 2018-07-11 07:52:32.526017) v4 -- 0x55d9b67d9400 con 0 > -699> 2018-07-11 07:52:32.526296 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.108:6803/11967 -- osd_ping(ping e501 stamp 2018-07-11 07:52:32.526017) v4 -- 0x55d986d32c00 con 0 > -698> 2018-07-11 07:52:32.526309 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.107:6806/12617 -- osd_ping(ping e501 stamp 2018-07-11 07:52:32.526017) v4 -- 0x55d9869bec00 con 0 > -697> 2018-07-11 07:52:32.526322 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.107:6807/12617 -- osd_ping(ping e501 stamp 2018-07-11 07:52:32.526017) v4 -- 0x55d986c22000 con 0 > -696> 2018-07-11 07:52:32.526335 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6802/15919 -- osd_ping(ping e501 stamp 2018-07-11 07:52:32.526017) v4 -- 0x55d982cdee00 con 0 > -695> 2018-07-11 07:52:32.526376 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6803/15919 -- osd_ping(ping e501 stamp 2018-07-11 07:52:32.526017) v4 -- 0x55d9b67d8400 con 0 > -694> 2018-07-11 07:52:32.526483 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6802/15255 conn(0x55d986bc8000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.0 seq 259 0x55d9bdba4c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:32.526017) v4 > -693> 2018-07-11 07:52:32.526499 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.0 10.8.128.113:6802/15255 259 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:32.526017) v4 ==== 2004+0+0 (4283430883 0 0) 0x55d9bdba4c00 con 0x55d986bc8000 > -692> 2018-07-11 07:52:32.526500 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.107:6807/12617 conn(0x55d986bd3800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.9 seq 259 0x55d9c9122e00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:32.526017) v4 > -691> 2018-07-11 07:52:32.526526 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.9 10.8.128.107:6807/12617 259 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:32.526017) v4 ==== 2004+0+0 (4283430883 0 0) 0x55d9c9122e00 con 0x55d986bd3800 > -690> 2018-07-11 07:52:32.526551 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.107:6806/12617 conn(0x55d986bd2000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.9 seq 259 0x55d982d31600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:32.526017) v4 > -689> 2018-07-11 07:52:32.526567 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6808/12651 conn(0x55d986b9a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.2 seq 259 0x55d9bdba4c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:32.526017) v4 > -688> 2018-07-11 07:52:32.526573 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.9 10.8.128.107:6806/12617 259 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:32.526017) v4 ==== 2004+0+0 (4283430883 0 0) 0x55d982d31600 con 0x55d986bd2000 > -687> 2018-07-11 07:52:32.526584 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.2 10.8.128.106:6808/12651 259 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:32.526017) v4 ==== 2004+0+0 (4283430883 0 0) 0x55d9bdba4c00 con 0x55d986b9a000 > -686> 2018-07-11 07:52:32.526620 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6803/15255 conn(0x55d986bc9800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.0 seq 259 0x55d982d31600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:32.526017) v4 > -685> 2018-07-11 07:52:32.526630 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6806/15263 conn(0x55d986b9d000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.5 seq 259 0x55d9c9122e00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:32.526017) v4 > -684> 2018-07-11 07:52:32.526638 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.0 10.8.128.113:6803/15255 259 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:32.526017) v4 ==== 2004+0+0 (4283430883 0 0) 0x55d982d31600 con 0x55d986bc9800 > -683> 2018-07-11 07:52:32.526652 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.5 10.8.128.113:6806/15263 259 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:32.526017) v4 ==== 2004+0+0 (4283430883 0 0) 0x55d9c9122e00 con 0x55d986b9d000 > -682> 2018-07-11 07:52:32.526688 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6811/12650 conn(0x55d986bcc800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 259 0x55d9bdba4c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:32.526017) v4 > -681> 2018-07-11 07:52:32.526704 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.3 10.8.128.106:6811/12650 259 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:32.526017) v4 ==== 2004+0+0 (4283430883 0 0) 0x55d9bdba4c00 con 0x55d986bcc800 > -680> 2018-07-11 07:52:32.526709 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6810/12651 conn(0x55d986b9b800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.2 seq 259 0x55d982d31600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:32.526017) v4 > -679> 2018-07-11 07:52:32.526725 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.2 10.8.128.106:6810/12651 259 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:32.526017) v4 ==== 2004+0+0 (4283430883 0 0) 0x55d982d31600 con 0x55d986b9b800 > -678> 2018-07-11 07:52:32.526729 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6807/12677 conn(0x55d986b59800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.1 seq 259 0x55d9bdba4c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:32.526017) v4 > -677> 2018-07-11 07:52:32.526726 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6806/12677 conn(0x55d986b58000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.1 seq 259 0x55d9c9122e00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:32.526017) v4 > -676> 2018-07-11 07:52:32.526747 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.1 10.8.128.106:6806/12677 259 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:32.526017) v4 ==== 2004+0+0 (4283430883 0 0) 0x55d9c9122e00 con 0x55d986b58000 > -675> 2018-07-11 07:52:32.526754 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.1 10.8.128.106:6807/12677 259 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:32.526017) v4 ==== 2004+0+0 (4283430883 0 0) 0x55d9bdba4c00 con 0x55d986b59800 > -674> 2018-07-11 07:52:32.526773 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6811/15287 conn(0x55d986bb0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 259 0x55d982d31600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:32.526017) v4 > -673> 2018-07-11 07:52:32.526782 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6809/12650 conn(0x55d986bcb000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 259 0x55d9c9122e00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:32.526017) v4 > -672> 2018-07-11 07:52:32.526777 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6807/15263 conn(0x55d986b9e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.5 seq 259 0x55d9bdba4c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:32.526017) v4 > -671> 2018-07-11 07:52:32.526798 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.3 10.8.128.106:6809/12650 259 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:32.526017) v4 ==== 2004+0+0 (4283430883 0 0) 0x55d9c9122e00 con 0x55d986bcb000 > -670> 2018-07-11 07:52:32.526789 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.4 10.8.128.113:6811/15287 259 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:32.526017) v4 ==== 2004+0+0 (4283430883 0 0) 0x55d982d31600 con 0x55d986bb0800 > -669> 2018-07-11 07:52:32.526805 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.5 10.8.128.113:6807/15263 259 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:32.526017) v4 ==== 2004+0+0 (4283430883 0 0) 0x55d9bdba4c00 con 0x55d986b9e800 > -668> 2018-07-11 07:52:32.526832 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.115:6803/13323 conn(0x55d986bc2800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 259 0x55d9c9122e00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:32.526017) v4 > -667> 2018-07-11 07:52:32.526838 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6810/15287 conn(0x55d986baf000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 259 0x55d9bdba4c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:32.526017) v4 > -666> 2018-07-11 07:52:32.526839 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.115:6802/13323 conn(0x55d986bc1000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 259 0x55d982d31600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:32.526017) v4 > -665> 2018-07-11 07:52:32.526847 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.4 10.8.128.113:6810/15287 259 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:32.526017) v4 ==== 2004+0+0 (4283430883 0 0) 0x55d9bdba4c00 con 0x55d986baf000 > -664> 2018-07-11 07:52:32.526844 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.6 10.8.128.115:6803/13323 259 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:32.526017) v4 ==== 2004+0+0 (4283430883 0 0) 0x55d9c9122e00 con 0x55d986bc2800 > -663> 2018-07-11 07:52:32.526855 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.6 10.8.128.115:6802/13323 259 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:32.526017) v4 ==== 2004+0+0 (4283430883 0 0) 0x55d982d31600 con 0x55d986bc1000 > -662> 2018-07-11 07:52:32.526915 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6806/15885 conn(0x55d986bac000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.7 seq 259 0x55d982d31600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:32.526017) v4 > -661> 2018-07-11 07:52:32.526924 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.108:6803/11967 conn(0x55d986bbf800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.8 seq 259 0x55d9bdba4c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:32.526017) v4 > -660> 2018-07-11 07:52:32.526925 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.108:6802/11967 conn(0x55d986bbe000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.8 seq 259 0x55d9c9122e00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:32.526017) v4 > -659> 2018-07-11 07:52:32.526930 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.7 10.8.128.114:6806/15885 259 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:32.526017) v4 ==== 2004+0+0 (4283430883 0 0) 0x55d982d31600 con 0x55d986bac000 > -658> 2018-07-11 07:52:32.526942 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.8 10.8.128.108:6803/11967 259 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:32.526017) v4 ==== 2004+0+0 (4283430883 0 0) 0x55d9bdba4c00 con 0x55d986bbf800 > -657> 2018-07-11 07:52:32.526945 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.8 10.8.128.108:6802/11967 259 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:32.526017) v4 ==== 2004+0+0 (4283430883 0 0) 0x55d9c9122e00 con 0x55d986bbe000 > -656> 2018-07-11 07:52:32.526974 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6802/15919 conn(0x55d986b5b000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.11 seq 259 0x55d982d31600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:32.526017) v4 > -655> 2018-07-11 07:52:32.526987 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.11 10.8.128.114:6802/15919 259 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:32.526017) v4 ==== 2004+0+0 (4283430883 0 0) 0x55d982d31600 con 0x55d986b5b000 > -654> 2018-07-11 07:52:32.526999 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6803/15919 conn(0x55d986b5c800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.11 seq 259 0x55d9c9122e00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:32.526017) v4 > -653> 2018-07-11 07:52:32.527014 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.11 10.8.128.114:6803/15919 259 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:32.526017) v4 ==== 2004+0+0 (4283430883 0 0) 0x55d9c9122e00 con 0x55d986b5c800 > -652> 2018-07-11 07:52:32.527094 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6807/15885 conn(0x55d986bad800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.7 seq 259 0x55d9c9122e00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:32.526017) v4 > -651> 2018-07-11 07:52:32.527111 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.7 10.8.128.114:6807/15885 259 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:32.526017) v4 ==== 2004+0+0 (4283430883 0 0) 0x55d9c9122e00 con 0x55d986bad800 > -650> 2018-07-11 07:52:32.572093 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6805/15885 conn(0x55d986c7f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.7 seq 2832 0x55d9bdba4c00 MOSDScrubReserve(5.4 REQUEST e501) v1 > -649> 2018-07-11 07:52:32.572112 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.7 10.8.128.114:6805/15885 2832 ==== MOSDScrubReserve(5.4 REQUEST e501) v1 ==== 43+0+0 (1753221512 0 0) 0x55d9bdba4c00 con 0x55d986c7f800 > -648> 2018-07-11 07:52:32.572144 7ffab33df700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.114:6805/15885 -- MOSDScrubReserve(5.4 REJECT e501) v1 -- 0x55d9b67d9200 con 0 > -647> 2018-07-11 07:52:32.572502 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6805/15885 conn(0x55d986c7f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.7 seq 2833 0x55d9b67d9200 MOSDScrubReserve(5.4 RELEASE e501) v1 > -646> 2018-07-11 07:52:32.572530 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.7 10.8.128.114:6805/15885 2833 ==== MOSDScrubReserve(5.4 RELEASE e501) v1 ==== 43+0+0 (2812146570 0 0) 0x55d9b67d9200 con 0x55d986c7f800 > -645> 2018-07-11 07:52:32.697740 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1090 0x55d9897e0000 rep_scrubmap(5.2e e501 from shard 2) v1 > -644> 2018-07-11 07:52:32.697760 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1090 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12478 (1269249638 0 139107108) 0x55d9897e0000 con 0x55d986bf0800 > -643> 2018-07-11 07:52:32.702078 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1084 0x55d9897e0240 rep_scrubmap(5.2e e501 from shard 4) v1 > -642> 2018-07-11 07:52:32.702099 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1084 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12478 (3686815630 0 139107108) 0x55d9897e0240 con 0x55d986bec000 > -641> 2018-07-11 07:52:32.702244 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -640> 2018-07-11 07:52:32.702280 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -639> 2018-07-11 07:52:32.702326 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -638> 2018-07-11 07:52:32.702791 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -637> 2018-07-11 07:52:32.713776 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67620,epoch:501/498,start:5:7589b898:::benchmark_data_magna115_10867_object2226920:0,end:5:758a4396:::benchmark_data_magna115_10867_object656374:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b14f0500 con 0 > -636> 2018-07-11 07:52:32.713814 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67620,epoch:501/498,start:5:7589b898:::benchmark_data_magna115_10867_object2226920:0,end:5:758a4396:::benchmark_data_magna115_10867_object656374:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498000 con 0 > -635> 2018-07-11 07:52:32.906934 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1091 0x55d9897e0480 rep_scrubmap(5.2e e501 from shard 2) v1 > -634> 2018-07-11 07:52:32.906953 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1091 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12470 (1269249638 0 3857039038) 0x55d9897e0480 con 0x55d986bf0800 > -633> 2018-07-11 07:52:32.956989 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1085 0x55d9897e06c0 rep_scrubmap(5.2e e501 from shard 4) v1 > -632> 2018-07-11 07:52:32.957008 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1085 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12470 (3686815630 0 3857039038) 0x55d9897e06c0 con 0x55d986bec000 > -631> 2018-07-11 07:52:32.957161 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -630> 2018-07-11 07:52:32.957187 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -629> 2018-07-11 07:52:32.957232 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -628> 2018-07-11 07:52:32.957714 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -627> 2018-07-11 07:52:32.958590 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:758a4396:::benchmark_data_magna115_10867_object656374:0,end:5:758aa4db:::benchmark_data_magna115_10867_object2750792:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498280 con 0 > -626> 2018-07-11 07:52:32.958637 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:758a4396:::benchmark_data_magna115_10867_object656374:0,end:5:758aa4db:::benchmark_data_magna115_10867_object2750792:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454780 con 0 > -625> 2018-07-11 07:52:33.000288 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.114:0/15919 conn(0x55d986df5000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.11 seq 271 0x55d982d31600 osd_ping(ping e501 stamp 2018-07-11 07:52:33.630880) v4 > -624> 2018-07-11 07:52:33.000311 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.11 10.8.128.114:0/15919 271 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:33.630880) v4 ==== 2004+0+0 (2798175010 0 0) 0x55d982d31600 con 0x55d986df5000 > -623> 2018-07-11 07:52:33.000302 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.114:0/15919 conn(0x55d986df3800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.11 seq 271 0x55d9c9122e00 osd_ping(ping e501 stamp 2018-07-11 07:52:33.630880) v4 > -622> 2018-07-11 07:52:33.000320 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:33.630880) v4 -- 0x55d982cdee00 con 0 > -621> 2018-07-11 07:52:33.000324 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.11 10.8.128.114:0/15919 271 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:33.630880) v4 ==== 2004+0+0 (2798175010 0 0) 0x55d9c9122e00 con 0x55d986df3800 > -620> 2018-07-11 07:52:33.000342 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:33.630880) v4 -- 0x55d9b67d8400 con 0 > -619> 2018-07-11 07:52:33.097017 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.108:0/11967 conn(0x55d986db0000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.8 seq 252 0x55d986d4e200 osd_ping(ping e501 stamp 2018-07-11 07:52:32.852990) v4 > -618> 2018-07-11 07:52:33.097020 7ffacbbd1700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.108:0/11967 conn(0x55d986d82800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.8 seq 252 0x55d986cd5000 osd_ping(ping e501 stamp 2018-07-11 07:52:32.852990) v4 > -617> 2018-07-11 07:52:33.097039 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.8 10.8.128.108:0/11967 252 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:32.852990) v4 ==== 2004+0+0 (962038549 0 0) 0x55d986d4e200 con 0x55d986db0000 > -616> 2018-07-11 07:52:33.097041 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 <== osd.8 10.8.128.108:0/11967 252 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:32.852990) v4 ==== 2004+0+0 (962038549 0 0) 0x55d986cd5000 con 0x55d986d82800 > -615> 2018-07-11 07:52:33.097049 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:32.852990) v4 -- 0x55d982cdee00 con 0 > -614> 2018-07-11 07:52:33.097070 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:32.852990) v4 -- 0x55d9b67d8400 con 0 > -613> 2018-07-11 07:52:33.182942 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1086 0x55d9897e0900 rep_scrubmap(5.2e e501 from shard 4) v1 > -612> 2018-07-11 07:52:33.182962 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1086 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12482 (3686815630 0 1410598402) 0x55d9897e0900 con 0x55d986bec000 > -611> 2018-07-11 07:52:33.200846 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1092 0x55d9897e0b40 rep_scrubmap(5.2e e501 from shard 2) v1 > -610> 2018-07-11 07:52:33.200866 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1092 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12482 (1269249638 0 1410598402) 0x55d9897e0b40 con 0x55d986bf0800 > -609> 2018-07-11 07:52:33.235584 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -608> 2018-07-11 07:52:33.235609 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -607> 2018-07-11 07:52:33.235666 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -606> 2018-07-11 07:52:33.236124 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -605> 2018-07-11 07:52:33.289043 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67154,epoch:501/498,start:5:758aa4db:::benchmark_data_magna115_10867_object2750792:0,end:5:758af2dd:::benchmark_data_magna115_10867_object3112643:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455400 con 0 > -604> 2018-07-11 07:52:33.289095 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67154,epoch:501/498,start:5:758aa4db:::benchmark_data_magna115_10867_object2750792:0,end:5:758af2dd:::benchmark_data_magna115_10867_object3112643:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454500 con 0 > -603> 2018-07-11 07:52:33.330462 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12651 conn(0x55d986db3000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.2 seq 266 0x55d9c9122e00 osd_ping(ping e501 stamp 2018-07-11 07:52:33.378775) v4 > -602> 2018-07-11 07:52:33.330491 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.2 10.8.128.106:0/12651 266 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:33.378775) v4 ==== 2004+0+0 (1768749569 0 0) 0x55d9c9122e00 con 0x55d986db3000 > -601> 2018-07-11 07:52:33.330496 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12651 conn(0x55d986db1800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.2 seq 266 0x55d986d32c00 osd_ping(ping e501 stamp 2018-07-11 07:52:33.378775) v4 > -600> 2018-07-11 07:52:33.330505 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:33.378775) v4 -- 0x55d9b67d8400 con 0 > -599> 2018-07-11 07:52:33.330514 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.2 10.8.128.106:0/12651 266 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:33.378775) v4 ==== 2004+0+0 (1768749569 0 0) 0x55d986d32c00 con 0x55d986db1800 > -598> 2018-07-11 07:52:33.330533 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:33.378775) v4 -- 0x55d9c34cc200 con 0 > -597> 2018-07-11 07:52:33.371353 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12677 conn(0x55d986db4800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.1 seq 272 0x55d982d31600 osd_ping(ping e501 stamp 2018-07-11 07:52:33.419705) v4 > -596> 2018-07-11 07:52:33.371378 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.1 10.8.128.106:0/12677 272 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:33.419705) v4 ==== 2004+0+0 (4072937556 0 0) 0x55d982d31600 con 0x55d986db4800 > -595> 2018-07-11 07:52:33.371373 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12677 conn(0x55d986df2000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.1 seq 272 0x55d986d4e600 osd_ping(ping e501 stamp 2018-07-11 07:52:33.419705) v4 > -594> 2018-07-11 07:52:33.371388 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:33.419705) v4 -- 0x55d982cdee00 con 0 > -593> 2018-07-11 07:52:33.371392 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.1 10.8.128.106:0/12677 272 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:33.419705) v4 ==== 2004+0+0 (4072937556 0 0) 0x55d986d4e600 con 0x55d986df2000 > -592> 2018-07-11 07:52:33.371413 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:33.419705) v4 -- 0x55d9c34cc200 con 0 > -591> 2018-07-11 07:52:33.502831 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1093 0x55d9897e0d80 rep_scrubmap(5.2e e501 from shard 2) v1 > -590> 2018-07-11 07:52:33.502852 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1093 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12468 (1269249638 0 298273641) 0x55d9897e0d80 con 0x55d986bf0800 > -589> 2018-07-11 07:52:33.532272 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1087 0x55d9897e0fc0 rep_scrubmap(5.2e e501 from shard 4) v1 > -588> 2018-07-11 07:52:33.532292 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1087 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12468 (3686815630 0 298273641) 0x55d9897e0fc0 con 0x55d986bec000 > -587> 2018-07-11 07:52:33.565445 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -586> 2018-07-11 07:52:33.565468 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -585> 2018-07-11 07:52:33.565523 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -584> 2018-07-11 07:52:33.565965 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -583> 2018-07-11 07:52:33.605211 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:758af2dd:::benchmark_data_magna115_10867_object3112643:0,end:5:758b4cba:::benchmark_data_magna115_10867_object1842629:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454f00 con 0 > -582> 2018-07-11 07:52:33.605272 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:758af2dd:::benchmark_data_magna115_10867_object3112643:0,end:5:758b4cba:::benchmark_data_magna115_10867_object1842629:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455180 con 0 > -581> 2018-07-11 07:52:33.609833 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.115:0/13323 conn(0x55d986d7e000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 270 0x55d986cd5000 osd_ping(ping e501 stamp 2018-07-11 07:52:33.955998) v4 > -580> 2018-07-11 07:52:33.609854 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.6 10.8.128.115:0/13323 270 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:33.955998) v4 ==== 2004+0+0 (1630362826 0 0) 0x55d986cd5000 con 0x55d986d7e000 > -579> 2018-07-11 07:52:33.609863 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:33.955998) v4 -- 0x55d9b67d8400 con 0 > -578> 2018-07-11 07:52:33.609874 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.115:0/13323 conn(0x55d986d24800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 270 0x55d986d4e600 osd_ping(ping e501 stamp 2018-07-11 07:52:33.955998) v4 > -577> 2018-07-11 07:52:33.609900 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.6 10.8.128.115:0/13323 270 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:33.955998) v4 ==== 2004+0+0 (1630362826 0 0) 0x55d986d4e600 con 0x55d986d24800 > -576> 2018-07-11 07:52:33.609916 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:33.955998) v4 -- 0x55d9c34cc200 con 0 > -575> 2018-07-11 07:52:33.626510 7ffab1bdc700 5 osd.10 501 heartbeat: osd_stat(789 GB used, 142 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist []) > -574> 2018-07-11 07:52:33.626524 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6802/15255 -- osd_ping(ping e501 stamp 2018-07-11 07:52:33.626522) v4 -- 0x55d9b9ac2a00 con 0 > -573> 2018-07-11 07:52:33.626556 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6803/15255 -- osd_ping(ping e501 stamp 2018-07-11 07:52:33.626522) v4 -- 0x55d9bcd72e00 con 0 > -572> 2018-07-11 07:52:33.626581 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6806/12677 -- osd_ping(ping e501 stamp 2018-07-11 07:52:33.626522) v4 -- 0x55d9829e4200 con 0 > -571> 2018-07-11 07:52:33.626601 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6807/12677 -- osd_ping(ping e501 stamp 2018-07-11 07:52:33.626522) v4 -- 0x55d9b9ac3000 con 0 > -570> 2018-07-11 07:52:33.626619 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6808/12651 -- osd_ping(ping e501 stamp 2018-07-11 07:52:33.626522) v4 -- 0x55d9869bf800 con 0 > -569> 2018-07-11 07:52:33.626635 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6810/12651 -- osd_ping(ping e501 stamp 2018-07-11 07:52:33.626522) v4 -- 0x55d9b40a4000 con 0 > -568> 2018-07-11 07:52:33.626661 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6809/12650 -- osd_ping(ping e501 stamp 2018-07-11 07:52:33.626522) v4 -- 0x55d986b6d400 con 0 > -567> 2018-07-11 07:52:33.626681 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6811/12650 -- osd_ping(ping e501 stamp 2018-07-11 07:52:33.626522) v4 -- 0x55d9c908ec00 con 0 > -566> 2018-07-11 07:52:33.626699 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6810/15287 -- osd_ping(ping e501 stamp 2018-07-11 07:52:33.626522) v4 -- 0x55d9b60dea00 con 0 > -565> 2018-07-11 07:52:33.626716 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6811/15287 -- osd_ping(ping e501 stamp 2018-07-11 07:52:33.626522) v4 -- 0x55d986d33c00 con 0 > -564> 2018-07-11 07:52:33.626733 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6806/15263 -- osd_ping(ping e501 stamp 2018-07-11 07:52:33.626522) v4 -- 0x55d9b3db2600 con 0 > -563> 2018-07-11 07:52:33.626755 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6807/15263 -- osd_ping(ping e501 stamp 2018-07-11 07:52:33.626522) v4 -- 0x55d986c84400 con 0 > -562> 2018-07-11 07:52:33.626772 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.115:6802/13323 -- osd_ping(ping e501 stamp 2018-07-11 07:52:33.626522) v4 -- 0x55d980f5a000 con 0 > -561> 2018-07-11 07:52:33.626815 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.115:6803/13323 -- osd_ping(ping e501 stamp 2018-07-11 07:52:33.626522) v4 -- 0x55d9b9ac3200 con 0 > -560> 2018-07-11 07:52:33.626837 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6806/15885 -- osd_ping(ping e501 stamp 2018-07-11 07:52:33.626522) v4 -- 0x55d9c3383200 con 0 > -559> 2018-07-11 07:52:33.626856 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6807/15885 -- osd_ping(ping e501 stamp 2018-07-11 07:52:33.626522) v4 -- 0x55d9b437b200 con 0 > -558> 2018-07-11 07:52:33.626873 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.108:6802/11967 -- osd_ping(ping e501 stamp 2018-07-11 07:52:33.626522) v4 -- 0x55d9b2783e00 con 0 > -557> 2018-07-11 07:52:33.626906 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.108:6803/11967 -- osd_ping(ping e501 stamp 2018-07-11 07:52:33.626522) v4 -- 0x55d986d4f800 con 0 > -556> 2018-07-11 07:52:33.626926 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.107:6806/12617 -- osd_ping(ping e501 stamp 2018-07-11 07:52:33.626522) v4 -- 0x55d982cdea00 con 0 > -555> 2018-07-11 07:52:33.626945 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.107:6807/12617 -- osd_ping(ping e501 stamp 2018-07-11 07:52:33.626522) v4 -- 0x55d9ba818800 con 0 > -554> 2018-07-11 07:52:33.626962 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6802/15919 -- osd_ping(ping e501 stamp 2018-07-11 07:52:33.626522) v4 -- 0x55d9c3383600 con 0 > -553> 2018-07-11 07:52:33.626999 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6803/15919 -- osd_ping(ping e501 stamp 2018-07-11 07:52:33.626522) v4 -- 0x55d986cd4c00 con 0 > -552> 2018-07-11 07:52:33.627015 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6802/15255 conn(0x55d986bc8000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.0 seq 260 0x55d986d4e600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:33.626522) v4 > -551> 2018-07-11 07:52:33.627036 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.0 10.8.128.113:6802/15255 260 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:33.626522) v4 ==== 2004+0+0 (1177914654 0 0) 0x55d986d4e600 con 0x55d986bc8000 > -550> 2018-07-11 07:52:33.627094 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6803/15255 conn(0x55d986bc9800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.0 seq 260 0x55d982d31600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:33.626522) v4 > -549> 2018-07-11 07:52:33.627106 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6810/15287 conn(0x55d986baf000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 260 0x55d986d4e600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:33.626522) v4 > -548> 2018-07-11 07:52:33.627104 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6806/15263 conn(0x55d986b9d000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.5 seq 260 0x55d9c9122e00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:33.626522) v4 > -547> 2018-07-11 07:52:33.627118 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.4 10.8.128.113:6810/15287 260 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:33.626522) v4 ==== 2004+0+0 (1177914654 0 0) 0x55d986d4e600 con 0x55d986baf000 > -546> 2018-07-11 07:52:33.627117 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.0 10.8.128.113:6803/15255 260 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:33.626522) v4 ==== 2004+0+0 (1177914654 0 0) 0x55d982d31600 con 0x55d986bc9800 > -545> 2018-07-11 07:52:33.627131 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.5 10.8.128.113:6806/15263 260 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:33.626522) v4 ==== 2004+0+0 (1177914654 0 0) 0x55d9c9122e00 con 0x55d986b9d000 > -544> 2018-07-11 07:52:33.627170 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6808/12651 conn(0x55d986b9a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.2 seq 260 0x55d986d4e600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:33.626522) v4 > -543> 2018-07-11 07:52:33.627172 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.107:6806/12617 conn(0x55d986bd2000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.9 seq 260 0x55d982d31600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:33.626522) v4 > -542> 2018-07-11 07:52:33.627186 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.2 10.8.128.106:6808/12651 260 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:33.626522) v4 ==== 2004+0+0 (1177914654 0 0) 0x55d986d4e600 con 0x55d986b9a000 > -541> 2018-07-11 07:52:33.627186 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.9 10.8.128.107:6806/12617 260 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:33.626522) v4 ==== 2004+0+0 (1177914654 0 0) 0x55d982d31600 con 0x55d986bd2000 > -540> 2018-07-11 07:52:33.627196 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6809/12650 conn(0x55d986bcb000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 260 0x55d9c9122e00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:33.626522) v4 > -539> 2018-07-11 07:52:33.627213 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.3 10.8.128.106:6809/12650 260 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:33.626522) v4 ==== 2004+0+0 (1177914654 0 0) 0x55d9c9122e00 con 0x55d986bcb000 > -538> 2018-07-11 07:52:33.627225 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6807/12677 conn(0x55d986b59800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.1 seq 260 0x55d986d4e600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:33.626522) v4 > -537> 2018-07-11 07:52:33.627233 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6810/12651 conn(0x55d986b9b800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.2 seq 260 0x55d982d31600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:33.626522) v4 > -536> 2018-07-11 07:52:33.627239 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.1 10.8.128.106:6807/12677 260 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:33.626522) v4 ==== 2004+0+0 (1177914654 0 0) 0x55d986d4e600 con 0x55d986b59800 > -535> 2018-07-11 07:52:33.627256 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.107:6807/12617 conn(0x55d986bd3800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.9 seq 260 0x55d9c9122e00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:33.626522) v4 > -534> 2018-07-11 07:52:33.627260 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.2 10.8.128.106:6810/12651 260 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:33.626522) v4 ==== 2004+0+0 (1177914654 0 0) 0x55d982d31600 con 0x55d986b9b800 > -533> 2018-07-11 07:52:33.627270 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.9 10.8.128.107:6807/12617 260 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:33.626522) v4 ==== 2004+0+0 (1177914654 0 0) 0x55d9c9122e00 con 0x55d986bd3800 > -532> 2018-07-11 07:52:33.627286 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6807/15263 conn(0x55d986b9e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.5 seq 260 0x55d986d4e600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:33.626522) v4 > -531> 2018-07-11 07:52:33.627297 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6811/15287 conn(0x55d986bb0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 260 0x55d982d31600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:33.626522) v4 > -530> 2018-07-11 07:52:33.627300 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.5 10.8.128.113:6807/15263 260 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:33.626522) v4 ==== 2004+0+0 (1177914654 0 0) 0x55d986d4e600 con 0x55d986b9e800 > -529> 2018-07-11 07:52:33.627311 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.4 10.8.128.113:6811/15287 260 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:33.626522) v4 ==== 2004+0+0 (1177914654 0 0) 0x55d982d31600 con 0x55d986bb0800 > -528> 2018-07-11 07:52:33.627310 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.115:6803/13323 conn(0x55d986bc2800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 260 0x55d9c9122e00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:33.626522) v4 > -527> 2018-07-11 07:52:33.627320 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.6 10.8.128.115:6803/13323 260 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:33.626522) v4 ==== 2004+0+0 (1177914654 0 0) 0x55d9c9122e00 con 0x55d986bc2800 > -526> 2018-07-11 07:52:33.627351 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.115:6802/13323 conn(0x55d986bc1000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 260 0x55d982d31600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:33.626522) v4 > -525> 2018-07-11 07:52:33.627350 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6811/12650 conn(0x55d986bcc800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 260 0x55d986d4e600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:33.626522) v4 > -524> 2018-07-11 07:52:33.627353 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6806/12677 conn(0x55d986b58000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.1 seq 260 0x55d9c9122e00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:33.626522) v4 > -523> 2018-07-11 07:52:33.627366 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.3 10.8.128.106:6811/12650 260 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:33.626522) v4 ==== 2004+0+0 (1177914654 0 0) 0x55d986d4e600 con 0x55d986bcc800 > -522> 2018-07-11 07:52:33.627367 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.6 10.8.128.115:6802/13323 260 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:33.626522) v4 ==== 2004+0+0 (1177914654 0 0) 0x55d982d31600 con 0x55d986bc1000 > -521> 2018-07-11 07:52:33.627372 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.1 10.8.128.106:6806/12677 260 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:33.626522) v4 ==== 2004+0+0 (1177914654 0 0) 0x55d9c9122e00 con 0x55d986b58000 > -520> 2018-07-11 07:52:33.627401 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.108:6803/11967 conn(0x55d986bbf800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.8 seq 260 0x55d986d4e600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:33.626522) v4 > -519> 2018-07-11 07:52:33.627413 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.8 10.8.128.108:6803/11967 260 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:33.626522) v4 ==== 2004+0+0 (1177914654 0 0) 0x55d986d4e600 con 0x55d986bbf800 > -518> 2018-07-11 07:52:33.627421 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.108:6802/11967 conn(0x55d986bbe000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.8 seq 260 0x55d9c9122e00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:33.626522) v4 > -517> 2018-07-11 07:52:33.627441 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.8 10.8.128.108:6802/11967 260 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:33.626522) v4 ==== 2004+0+0 (1177914654 0 0) 0x55d9c9122e00 con 0x55d986bbe000 > -516> 2018-07-11 07:52:33.627442 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6806/15885 conn(0x55d986bac000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.7 seq 260 0x55d982d31600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:33.626522) v4 > -515> 2018-07-11 07:52:33.627463 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.7 10.8.128.114:6806/15885 260 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:33.626522) v4 ==== 2004+0+0 (1177914654 0 0) 0x55d982d31600 con 0x55d986bac000 > -514> 2018-07-11 07:52:33.627499 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6803/15919 conn(0x55d986b5c800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.11 seq 260 0x55d9c9122e00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:33.626522) v4 > -513> 2018-07-11 07:52:33.627512 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6802/15919 conn(0x55d986b5b000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.11 seq 260 0x55d982d31600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:33.626522) v4 > -512> 2018-07-11 07:52:33.627520 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.11 10.8.128.114:6803/15919 260 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:33.626522) v4 ==== 2004+0+0 (1177914654 0 0) 0x55d9c9122e00 con 0x55d986b5c800 > -511> 2018-07-11 07:52:33.627528 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.11 10.8.128.114:6802/15919 260 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:33.626522) v4 ==== 2004+0+0 (1177914654 0 0) 0x55d982d31600 con 0x55d986b5b000 > -510> 2018-07-11 07:52:33.627552 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6807/15885 conn(0x55d986bad800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.7 seq 260 0x55d9c9122e00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:33.626522) v4 > -509> 2018-07-11 07:52:33.627568 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.7 10.8.128.114:6807/15885 260 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:33.626522) v4 ==== 2004+0+0 (1177914654 0 0) 0x55d9c9122e00 con 0x55d986bad800 > -508> 2018-07-11 07:52:33.792890 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15287 conn(0x55d981240800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 282 0x55d986d4e600 osd_ping(ping e501 stamp 2018-07-11 07:52:35.024900) v4 > -507> 2018-07-11 07:52:33.792910 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.4 10.8.128.113:0/15287 282 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:35.024900) v4 ==== 2004+0+0 (2514649647 0 0) 0x55d986d4e600 con 0x55d981240800 > -506> 2018-07-11 07:52:33.792925 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:35.024900) v4 -- 0x55d986d4f800 con 0 > -505> 2018-07-11 07:52:33.792927 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15287 conn(0x55d98123f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 282 0x55d982d31600 osd_ping(ping e501 stamp 2018-07-11 07:52:35.024900) v4 > -504> 2018-07-11 07:52:33.792948 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.4 10.8.128.113:0/15287 282 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:35.024900) v4 ==== 2004+0+0 (2514649647 0 0) 0x55d982d31600 con 0x55d98123f000 > -503> 2018-07-11 07:52:33.792959 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:35.024900) v4 -- 0x55d9c3383600 con 0 > -502> 2018-07-11 07:52:33.843797 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1094 0x55d9897e1200 rep_scrubmap(5.2e e501 from shard 2) v1 > -501> 2018-07-11 07:52:33.843817 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1094 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12470 (1269249638 0 2727315725) 0x55d9897e1200 con 0x55d986bf0800 > -500> 2018-07-11 07:52:33.846814 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1088 0x55d9897e1440 rep_scrubmap(5.2e e501 from shard 4) v1 > -499> 2018-07-11 07:52:33.846832 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1088 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12470 (3686815630 0 2727315725) 0x55d9897e1440 con 0x55d986bec000 > -498> 2018-07-11 07:52:33.848334 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 3046 0x55d9c9122e00 MOSDScrubReserve(11.3s5 REQUEST e501) v1 > -497> 2018-07-11 07:52:33.848351 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 3046 ==== MOSDScrubReserve(11.3s5 REQUEST e501) v1 ==== 43+0+0 (3260170403 0 0) 0x55d9c9122e00 con 0x55d986c7e000 > -496> 2018-07-11 07:52:33.848475 7ffab3be0700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e501) v1 -- 0x55d980f5a200 con 0 > -495> 2018-07-11 07:52:33.848820 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 3047 0x55d986cd4c00 MOSDScrubReserve(11.3s5 RELEASE e501) v1 > -494> 2018-07-11 07:52:33.848858 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 3047 ==== MOSDScrubReserve(11.3s5 RELEASE e501) v1 ==== 43+0+0 (223328929 0 0) 0x55d986cd4c00 con 0x55d986c7e000 > -493> 2018-07-11 07:52:33.908596 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -492> 2018-07-11 07:52:33.908622 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -491> 2018-07-11 07:52:33.908679 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -490> 2018-07-11 07:52:33.909142 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -489> 2018-07-11 07:52:33.931067 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:758b4cba:::benchmark_data_magna115_10867_object1842629:0,end:5:758b9530:::benchmark_data_magna115_10867_object1115258:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454a00 con 0 > -488> 2018-07-11 07:52:33.931114 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:758b4cba:::benchmark_data_magna115_10867_object1842629:0,end:5:758b9530:::benchmark_data_magna115_10867_object1115258:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98944b680 con 0 > -487> 2018-07-11 07:52:33.968289 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6805/15263 conn(0x55d986c41000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=31 cs=1 l=0). rx osd.5 seq 2850 0x55d986d4e600 MOSDScrubReserve(6.15 REQUEST e501) v1 > -486> 2018-07-11 07:52:33.968309 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.5 10.8.128.113:6805/15263 2850 ==== MOSDScrubReserve(6.15 REQUEST e501) v1 ==== 43+0+0 (3942742026 0 0) 0x55d986d4e600 con 0x55d986c41000 > -485> 2018-07-11 07:52:34.130297 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1089 0x55d9897e1680 rep_scrubmap(5.2e e501 from shard 4) v1 > -484> 2018-07-11 07:52:34.130329 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1089 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12484 (3686815630 0 133854266) 0x55d9897e1680 con 0x55d986bec000 > -483> 2018-07-11 07:52:34.142264 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1095 0x55d9897e18c0 rep_scrubmap(5.2e e501 from shard 2) v1 > -482> 2018-07-11 07:52:34.142283 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1095 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12484 (1269249638 0 133854266) 0x55d9897e18c0 con 0x55d986bf0800 > -481> 2018-07-11 07:52:34.275829 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6805/15263 -- MOSDScrubReserve(6.15 REJECT e501) v1 -- 0x55d986c22e00 con 0 > -480> 2018-07-11 07:52:34.276039 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -479> 2018-07-11 07:52:34.276062 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -478> 2018-07-11 07:52:34.276109 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -477> 2018-07-11 07:52:34.276229 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6805/15263 conn(0x55d986c41000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=31 cs=1 l=0). rx osd.5 seq 2851 0x55d986c22e00 MOSDScrubReserve(6.15 RELEASE e501) v1 > -476> 2018-07-11 07:52:34.276257 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.5 10.8.128.113:6805/15263 2851 ==== MOSDScrubReserve(6.15 RELEASE e501) v1 ==== 43+0+0 (605848072 0 0) 0x55d986c22e00 con 0x55d986c41000 > -475> 2018-07-11 07:52:34.276593 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -474> 2018-07-11 07:52:34.322616 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:758b9530:::benchmark_data_magna115_10867_object1115258:0,end:5:758c0061:::benchmark_data_magna115_11707_object60697:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98944bb80 con 0 > -473> 2018-07-11 07:52:34.322667 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:758b9530:::benchmark_data_magna115_10867_object1115258:0,end:5:758c0061:::benchmark_data_magna115_11707_object60697:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98944b900 con 0 > -472> 2018-07-11 07:52:34.361034 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.107:0/12617 conn(0x55d98123c000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 263 0x55d986d4f800 osd_ping(ping e501 stamp 2018-07-11 07:52:34.360705) v4 > -471> 2018-07-11 07:52:34.361057 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.9 10.8.128.107:0/12617 263 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:34.360705) v4 ==== 2004+0+0 (913904965 0 0) 0x55d986d4f800 con 0x55d98123c000 > -470> 2018-07-11 07:52:34.361056 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.107:0/12617 conn(0x55d98123d800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 263 0x55d986cd5000 osd_ping(ping e501 stamp 2018-07-11 07:52:34.360705) v4 > -469> 2018-07-11 07:52:34.361068 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:34.360705) v4 -- 0x55d986c84400 con 0 > -468> 2018-07-11 07:52:34.361076 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.9 10.8.128.107:0/12617 263 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:34.360705) v4 ==== 2004+0+0 (913904965 0 0) 0x55d986cd5000 con 0x55d98123d800 > -467> 2018-07-11 07:52:34.361110 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:34.360705) v4 -- 0x55d9ba818800 con 0 > -466> 2018-07-11 07:52:34.540539 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1096 0x55d9897e1b00 rep_scrubmap(5.2e e501 from shard 2) v1 > -465> 2018-07-11 07:52:34.540559 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1096 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12474 (1269249638 0 1616718838) 0x55d9897e1b00 con 0x55d986bf0800 > -464> 2018-07-11 07:52:34.558519 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1090 0x55d9897e1d40 rep_scrubmap(5.2e e501 from shard 4) v1 > -463> 2018-07-11 07:52:34.558541 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1090 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12474 (3686815630 0 1616718838) 0x55d9897e1d40 con 0x55d986bec000 > -462> 2018-07-11 07:52:34.558687 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -461> 2018-07-11 07:52:34.558713 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -460> 2018-07-11 07:52:34.558759 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -459> 2018-07-11 07:52:34.559217 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -458> 2018-07-11 07:52:34.591338 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:758c0061:::benchmark_data_magna115_11707_object60697:0,end:5:758c6c67:::benchmark_data_magna115_10867_object696573:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb772280 con 0 > -457> 2018-07-11 07:52:34.591380 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:758c0061:::benchmark_data_magna115_11707_object60697:0,end:5:758c6c67:::benchmark_data_magna115_10867_object696573:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98944b400 con 0 > -456> 2018-07-11 07:52:34.727131 7ffab1bdc700 5 osd.10 501 heartbeat: osd_stat(789 GB used, 142 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist []) > -455> 2018-07-11 07:52:34.727148 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6802/15255 -- osd_ping(ping e501 stamp 2018-07-11 07:52:34.727143) v4 -- 0x55d9b67d8600 con 0 > -454> 2018-07-11 07:52:34.727179 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6803/15255 -- osd_ping(ping e501 stamp 2018-07-11 07:52:34.727143) v4 -- 0x55d9bf805800 con 0 > -453> 2018-07-11 07:52:34.727213 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6806/12677 -- osd_ping(ping e501 stamp 2018-07-11 07:52:34.727143) v4 -- 0x55d9bf804600 con 0 > -452> 2018-07-11 07:52:34.727233 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6807/12677 -- osd_ping(ping e501 stamp 2018-07-11 07:52:34.727143) v4 -- 0x55d9c908f600 con 0 > -451> 2018-07-11 07:52:34.727262 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6808/12651 -- osd_ping(ping e501 stamp 2018-07-11 07:52:34.727143) v4 -- 0x55d9b3db3600 con 0 > -450> 2018-07-11 07:52:34.727282 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6810/12651 -- osd_ping(ping e501 stamp 2018-07-11 07:52:34.727143) v4 -- 0x55d9bf805000 con 0 > -449> 2018-07-11 07:52:34.727307 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6809/12650 -- osd_ping(ping e501 stamp 2018-07-11 07:52:34.727143) v4 -- 0x55d9c3382800 con 0 > -448> 2018-07-11 07:52:34.727325 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6811/12650 -- osd_ping(ping e501 stamp 2018-07-11 07:52:34.727143) v4 -- 0x55d9b67d8c00 con 0 > -447> 2018-07-11 07:52:34.727346 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6810/15287 -- osd_ping(ping e501 stamp 2018-07-11 07:52:34.727143) v4 -- 0x55d982d31e00 con 0 > -446> 2018-07-11 07:52:34.727364 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6811/15287 -- osd_ping(ping e501 stamp 2018-07-11 07:52:34.727143) v4 -- 0x55d986d41e00 con 0 > -445> 2018-07-11 07:52:34.727403 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6806/15263 -- osd_ping(ping e501 stamp 2018-07-11 07:52:34.727143) v4 -- 0x55d9b60df200 con 0 > -444> 2018-07-11 07:52:34.727425 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6807/15263 -- osd_ping(ping e501 stamp 2018-07-11 07:52:34.727143) v4 -- 0x55d9bfbde400 con 0 > -443> 2018-07-11 07:52:34.727444 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.115:6802/13323 -- osd_ping(ping e501 stamp 2018-07-11 07:52:34.727143) v4 -- 0x55d9829e4400 con 0 > -442> 2018-07-11 07:52:34.727462 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.115:6803/13323 -- osd_ping(ping e501 stamp 2018-07-11 07:52:34.727143) v4 -- 0x55d9b67d9600 con 0 > -441> 2018-07-11 07:52:34.727504 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6806/15885 -- osd_ping(ping e501 stamp 2018-07-11 07:52:34.727143) v4 -- 0x55d9c3382a00 con 0 > -440> 2018-07-11 07:52:34.727527 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6807/15885 -- osd_ping(ping e501 stamp 2018-07-11 07:52:34.727143) v4 -- 0x55d986c22400 con 0 > -439> 2018-07-11 07:52:34.727545 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.108:6802/11967 -- osd_ping(ping e501 stamp 2018-07-11 07:52:34.727143) v4 -- 0x55d986d96400 con 0 > -438> 2018-07-11 07:52:34.727536 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6802/15255 conn(0x55d986bc8000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.0 seq 261 0x55d9b60df600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:34.727143) v4 > -437> 2018-07-11 07:52:34.727564 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.0 10.8.128.113:6802/15255 261 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:34.727143) v4 ==== 2004+0+0 (2692470257 0 0) 0x55d9b60df600 con 0x55d986bc8000 > -436> 2018-07-11 07:52:34.727569 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.108:6803/11967 -- osd_ping(ping e501 stamp 2018-07-11 07:52:34.727143) v4 -- 0x55d986d40000 con 0 > -435> 2018-07-11 07:52:34.727593 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.107:6806/12617 -- osd_ping(ping e501 stamp 2018-07-11 07:52:34.727143) v4 -- 0x55d986cd5400 con 0 > -434> 2018-07-11 07:52:34.727611 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.107:6807/12617 -- osd_ping(ping e501 stamp 2018-07-11 07:52:34.727143) v4 -- 0x55d9bdba5200 con 0 > -433> 2018-07-11 07:52:34.727648 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6802/15919 -- osd_ping(ping e501 stamp 2018-07-11 07:52:34.727143) v4 -- 0x55d9bf805200 con 0 > -432> 2018-07-11 07:52:34.727667 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6803/15919 -- osd_ping(ping e501 stamp 2018-07-11 07:52:34.727143) v4 -- 0x55d9c4c1ae00 con 0 > -431> 2018-07-11 07:52:34.727676 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6803/15255 conn(0x55d986bc9800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.0 seq 261 0x55d986d4e200 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:34.727143) v4 > -430> 2018-07-11 07:52:34.727697 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.0 10.8.128.113:6803/15255 261 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:34.727143) v4 ==== 2004+0+0 (2692470257 0 0) 0x55d986d4e200 con 0x55d986bc9800 > -429> 2018-07-11 07:52:34.727757 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6810/12651 conn(0x55d986b9b800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.2 seq 261 0x55d986d4e200 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:34.727143) v4 > -428> 2018-07-11 07:52:34.727774 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.2 10.8.128.106:6810/12651 261 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:34.727143) v4 ==== 2004+0+0 (2692470257 0 0) 0x55d986d4e200 con 0x55d986b9b800 > -427> 2018-07-11 07:52:34.727775 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6806/12677 conn(0x55d986b58000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.1 seq 261 0x55d986cd5000 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:34.727143) v4 > -426> 2018-07-11 07:52:34.727780 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6808/12651 conn(0x55d986b9a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.2 seq 261 0x55d986d32c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:34.727143) v4 > -425> 2018-07-11 07:52:34.727802 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.1 10.8.128.106:6806/12677 261 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:34.727143) v4 ==== 2004+0+0 (2692470257 0 0) 0x55d986cd5000 con 0x55d986b58000 > -424> 2018-07-11 07:52:34.727808 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.2 10.8.128.106:6808/12651 261 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:34.727143) v4 ==== 2004+0+0 (2692470257 0 0) 0x55d986d32c00 con 0x55d986b9a000 > -423> 2018-07-11 07:52:34.727828 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6811/15287 conn(0x55d986bb0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 261 0x55d982d31600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:34.727143) v4 > -422> 2018-07-11 07:52:34.727842 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.4 10.8.128.113:6811/15287 261 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:34.727143) v4 ==== 2004+0+0 (2692470257 0 0) 0x55d982d31600 con 0x55d986bb0800 > -421> 2018-07-11 07:52:34.727848 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6807/12677 conn(0x55d986b59800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.1 seq 261 0x55d986d32c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:34.727143) v4 > -420> 2018-07-11 07:52:34.727853 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6809/12650 conn(0x55d986bcb000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 261 0x55d986cd5000 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:34.727143) v4 > -419> 2018-07-11 07:52:34.727861 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.1 10.8.128.106:6807/12677 261 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:34.727143) v4 ==== 2004+0+0 (2692470257 0 0) 0x55d986d32c00 con 0x55d986b59800 > -418> 2018-07-11 07:52:34.727869 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.3 10.8.128.106:6809/12650 261 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:34.727143) v4 ==== 2004+0+0 (2692470257 0 0) 0x55d986cd5000 con 0x55d986bcb000 > -417> 2018-07-11 07:52:34.727884 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.115:6802/13323 conn(0x55d986bc1000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 261 0x55d982d31600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:34.727143) v4 > -416> 2018-07-11 07:52:34.727894 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6811/12650 conn(0x55d986bcc800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 261 0x55d986d32c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:34.727143) v4 > -415> 2018-07-11 07:52:34.727897 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.6 10.8.128.115:6802/13323 261 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:34.727143) v4 ==== 2004+0+0 (2692470257 0 0) 0x55d982d31600 con 0x55d986bc1000 > -414> 2018-07-11 07:52:34.727904 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6806/15263 conn(0x55d986b9d000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.5 seq 261 0x55d986cd5000 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:34.727143) v4 > -413> 2018-07-11 07:52:34.727908 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.3 10.8.128.106:6811/12650 261 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:34.727143) v4 ==== 2004+0+0 (2692470257 0 0) 0x55d986d32c00 con 0x55d986bcc800 > -412> 2018-07-11 07:52:34.727918 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.5 10.8.128.113:6806/15263 261 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:34.727143) v4 ==== 2004+0+0 (2692470257 0 0) 0x55d986cd5000 con 0x55d986b9d000 > -411> 2018-07-11 07:52:34.727928 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.107:6806/12617 conn(0x55d986bd2000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.9 seq 261 0x55d982d31600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:34.727143) v4 > -410> 2018-07-11 07:52:34.727939 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.9 10.8.128.107:6806/12617 261 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:34.727143) v4 ==== 2004+0+0 (2692470257 0 0) 0x55d982d31600 con 0x55d986bd2000 > -409> 2018-07-11 07:52:34.727949 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6807/15263 conn(0x55d986b9e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.5 seq 261 0x55d986d32c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:34.727143) v4 > -408> 2018-07-11 07:52:34.727961 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.5 10.8.128.113:6807/15263 261 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:34.727143) v4 ==== 2004+0+0 (2692470257 0 0) 0x55d986d32c00 con 0x55d986b9e800 > -407> 2018-07-11 07:52:34.727985 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.107:6807/12617 conn(0x55d986bd3800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.9 seq 261 0x55d980f5a200 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:34.727143) v4 > -406> 2018-07-11 07:52:34.728000 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.9 10.8.128.107:6807/12617 261 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:34.727143) v4 ==== 2004+0+0 (2692470257 0 0) 0x55d980f5a200 con 0x55d986bd3800 > -405> 2018-07-11 07:52:34.728008 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6810/15287 conn(0x55d986baf000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 261 0x55d986d32c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:34.727143) v4 > -404> 2018-07-11 07:52:34.728022 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.4 10.8.128.113:6810/15287 261 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:34.727143) v4 ==== 2004+0+0 (2692470257 0 0) 0x55d986d32c00 con 0x55d986baf000 > -403> 2018-07-11 07:52:34.728031 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.115:6803/13323 conn(0x55d986bc2800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 261 0x55d980f5a200 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:34.727143) v4 > -402> 2018-07-11 07:52:34.728043 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.6 10.8.128.115:6803/13323 261 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:34.727143) v4 ==== 2004+0+0 (2692470257 0 0) 0x55d980f5a200 con 0x55d986bc2800 > -401> 2018-07-11 07:52:34.728054 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.108:6803/11967 conn(0x55d986bbf800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.8 seq 261 0x55d986d32c00 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:34.727143) v4 > -400> 2018-07-11 07:52:34.728066 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.8 10.8.128.108:6803/11967 261 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:34.727143) v4 ==== 2004+0+0 (2692470257 0 0) 0x55d986d32c00 con 0x55d986bbf800 > -399> 2018-07-11 07:52:34.728081 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.108:6802/11967 conn(0x55d986bbe000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.8 seq 261 0x55d980f5a200 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:34.727143) v4 > -398> 2018-07-11 07:52:34.728089 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6806/15885 conn(0x55d986bac000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.7 seq 261 0x55d982d31600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:34.727143) v4 > -397> 2018-07-11 07:52:34.728094 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.8 10.8.128.108:6802/11967 261 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:34.727143) v4 ==== 2004+0+0 (2692470257 0 0) 0x55d980f5a200 con 0x55d986bbe000 > -396> 2018-07-11 07:52:34.728107 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.7 10.8.128.114:6806/15885 261 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:34.727143) v4 ==== 2004+0+0 (2692470257 0 0) 0x55d982d31600 con 0x55d986bac000 > -395> 2018-07-11 07:52:34.728137 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6807/15885 conn(0x55d986bad800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.7 seq 261 0x55d980f5a200 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:34.727143) v4 > -394> 2018-07-11 07:52:34.728151 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.7 10.8.128.114:6807/15885 261 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:34.727143) v4 ==== 2004+0+0 (2692470257 0 0) 0x55d980f5a200 con 0x55d986bad800 > -393> 2018-07-11 07:52:34.728171 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6802/15919 conn(0x55d986b5b000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.11 seq 261 0x55d982d31600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:34.727143) v4 > -392> 2018-07-11 07:52:34.728192 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.11 10.8.128.114:6802/15919 261 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:34.727143) v4 ==== 2004+0+0 (2692470257 0 0) 0x55d982d31600 con 0x55d986b5b000 > -391> 2018-07-11 07:52:34.728227 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6803/15919 conn(0x55d986b5c800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.11 seq 261 0x55d980f5a200 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:34.727143) v4 > -390> 2018-07-11 07:52:34.728242 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.11 10.8.128.114:6803/15919 261 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:34.727143) v4 ==== 2004+0+0 (2692470257 0 0) 0x55d980f5a200 con 0x55d986b5c800 > -389> 2018-07-11 07:52:34.781058 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1091 0x55d9b314e000 rep_scrubmap(5.2e e501 from shard 4) v1 > -388> 2018-07-11 07:52:34.781078 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1091 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12460 (3686815630 0 3395207776) 0x55d9b314e000 con 0x55d986bec000 > -387> 2018-07-11 07:52:34.815520 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1097 0x55d9b314e240 rep_scrubmap(5.2e e501 from shard 2) v1 > -386> 2018-07-11 07:52:34.815540 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1097 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12460 (1269249638 0 3395207776) 0x55d9b314e240 con 0x55d986bf0800 > -385> 2018-07-11 07:52:34.828776 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -384> 2018-07-11 07:52:34.828800 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -383> 2018-07-11 07:52:34.828858 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -382> 2018-07-11 07:52:34.829322 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -381> 2018-07-11 07:52:34.872958 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'66690,epoch:501/498,start:5:758c6c67:::benchmark_data_magna115_10867_object696573:0,end:5:758ce7ee:::benchmark_data_magna115_10867_object2055687:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb772000 con 0 > -380> 2018-07-11 07:52:34.873007 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'66690,epoch:501/498,start:5:758c6c67:::benchmark_data_magna115_10867_object696573:0,end:5:758ce7ee:::benchmark_data_magna115_10867_object2055687:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb772500 con 0 > -379> 2018-07-11 07:52:34.925073 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.114:0/15885 conn(0x55d986d81000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.7 seq 274 0x55d986d32c00 osd_ping(ping e501 stamp 2018-07-11 07:52:35.555695) v4 > -378> 2018-07-11 07:52:34.925095 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.7 10.8.128.114:0/15885 274 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:35.555695) v4 ==== 2004+0+0 (1423291767 0 0) 0x55d986d32c00 con 0x55d986d81000 > -377> 2018-07-11 07:52:34.925094 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.114:0/15885 conn(0x55d986d7f800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.7 seq 274 0x55d982d31600 osd_ping(ping e501 stamp 2018-07-11 07:52:35.555695) v4 > -376> 2018-07-11 07:52:34.925119 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:35.555695) v4 -- 0x55d986d40000 con 0 > -375> 2018-07-11 07:52:34.925123 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.7 10.8.128.114:0/15885 274 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:35.555695) v4 ==== 2004+0+0 (1423291767 0 0) 0x55d982d31600 con 0x55d986d7f800 > -374> 2018-07-11 07:52:34.925151 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:35.555695) v4 -- 0x55d9bf805200 con 0 > -373> 2018-07-11 07:52:34.968431 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6805/15263 conn(0x55d986c41000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=31 cs=1 l=0). rx osd.5 seq 2852 0x55d9b60df600 MOSDScrubReserve(6.15 REQUEST e501) v1 > -372> 2018-07-11 07:52:34.968450 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.5 10.8.128.113:6805/15263 2852 ==== MOSDScrubReserve(6.15 REQUEST e501) v1 ==== 43+0+0 (3942742026 0 0) 0x55d9b60df600 con 0x55d986c41000 > -371> 2018-07-11 07:52:35.031034 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12651 conn(0x55d986db3000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.2 seq 267 0x55d980f5a200 osd_ping(ping e501 stamp 2018-07-11 07:52:35.079310) v4 > -370> 2018-07-11 07:52:35.031036 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12651 conn(0x55d986db1800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.2 seq 267 0x55d986d32c00 osd_ping(ping e501 stamp 2018-07-11 07:52:35.079310) v4 > -369> 2018-07-11 07:52:35.031069 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.2 10.8.128.106:0/12651 267 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:35.079310) v4 ==== 2004+0+0 (3000702957 0 0) 0x55d980f5a200 con 0x55d986db3000 > -368> 2018-07-11 07:52:35.031073 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.2 10.8.128.106:0/12651 267 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:35.079310) v4 ==== 2004+0+0 (3000702957 0 0) 0x55d986d32c00 con 0x55d986db1800 > -367> 2018-07-11 07:52:35.031079 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:35.079310) v4 -- 0x55d9c4c1ae00 con 0 > -366> 2018-07-11 07:52:35.031106 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:35.079310) v4 -- 0x55d9bfbde400 con 0 > -365> 2018-07-11 07:52:35.110218 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1098 0x55d9b314e480 rep_scrubmap(5.2e e501 from shard 2) v1 > -364> 2018-07-11 07:52:35.110239 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1098 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12464 (1269249638 0 751163074) 0x55d9b314e480 con 0x55d986bf0800 > -363> 2018-07-11 07:52:35.114664 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1092 0x55d9b314e6c0 rep_scrubmap(5.2e e501 from shard 4) v1 > -362> 2018-07-11 07:52:35.114683 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1092 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12464 (3686815630 0 751163074) 0x55d9b314e6c0 con 0x55d986bec000 > -361> 2018-07-11 07:52:35.174474 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15255 conn(0x55d986d20000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.0 seq 268 0x55d986d40000 osd_ping(ping e501 stamp 2018-07-11 07:52:36.406515) v4 > -360> 2018-07-11 07:52:35.174476 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15255 conn(0x55d986c82800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.0 seq 268 0x55d986d4e200 osd_ping(ping e501 stamp 2018-07-11 07:52:36.406515) v4 > -359> 2018-07-11 07:52:35.174496 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.0 10.8.128.113:0/15255 268 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:36.406515) v4 ==== 2004+0+0 (2592555795 0 0) 0x55d986d4e200 con 0x55d986c82800 > -358> 2018-07-11 07:52:35.174496 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.0 10.8.128.113:0/15255 268 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:36.406515) v4 ==== 2004+0+0 (2592555795 0 0) 0x55d986d40000 con 0x55d986d20000 > -357> 2018-07-11 07:52:35.174507 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:36.406515) v4 -- 0x55d9bf805200 con 0 > -356> 2018-07-11 07:52:35.174538 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:36.406515) v4 -- 0x55d9bfbde400 con 0 > -355> 2018-07-11 07:52:35.397465 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.108:0/11967 conn(0x55d986db0000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.8 seq 253 0x55d982d31600 osd_ping(ping e501 stamp 2018-07-11 07:52:35.153460) v4 > -354> 2018-07-11 07:52:35.397486 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.8 10.8.128.108:0/11967 253 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:35.153460) v4 ==== 2004+0+0 (2888130002 0 0) 0x55d982d31600 con 0x55d986db0000 > -353> 2018-07-11 07:52:35.397495 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:35.153460) v4 -- 0x55d9bf805200 con 0 > -352> 2018-07-11 07:52:35.397525 7ffacbbd1700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.108:0/11967 conn(0x55d986d82800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.8 seq 253 0x55d986cd5000 osd_ping(ping e501 stamp 2018-07-11 07:52:35.153460) v4 > -351> 2018-07-11 07:52:35.397549 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 <== osd.8 10.8.128.108:0/11967 253 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:35.153460) v4 ==== 2004+0+0 (2888130002 0 0) 0x55d986cd5000 con 0x55d986d82800 > -350> 2018-07-11 07:52:35.397565 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:35.153460) v4 -- 0x55d9c4c1ae00 con 0 > -349> 2018-07-11 07:52:35.531564 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12651 conn(0x55d986db3000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.2 seq 268 0x55d980f5a200 osd_ping(ping e501 stamp 2018-07-11 07:52:35.579848) v4 > -348> 2018-07-11 07:52:35.531597 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.2 10.8.128.106:0/12651 268 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:35.579848) v4 ==== 2004+0+0 (2390527721 0 0) 0x55d980f5a200 con 0x55d986db3000 > -347> 2018-07-11 07:52:35.531606 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:35.579848) v4 -- 0x55d9c4c1ae00 con 0 > -346> 2018-07-11 07:52:35.531700 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12651 conn(0x55d986db1800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.2 seq 268 0x55d986d32c00 osd_ping(ping e501 stamp 2018-07-11 07:52:35.579848) v4 > -345> 2018-07-11 07:52:35.531720 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.2 10.8.128.106:0/12651 268 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:35.579848) v4 ==== 2004+0+0 (2390527721 0 0) 0x55d986d32c00 con 0x55d986db1800 > -344> 2018-07-11 07:52:35.531731 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:35.579848) v4 -- 0x55d9bfbde400 con 0 > -343> 2018-07-11 07:52:35.654972 7ffacbbd1700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15263 conn(0x55d985d44000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.5 seq 252 0x55d986cd5000 osd_ping(ping e501 stamp 2018-07-11 07:52:36.886957) v4 > -342> 2018-07-11 07:52:35.654979 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15263 conn(0x55d985d45800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.5 seq 252 0x55d986d4e200 osd_ping(ping e501 stamp 2018-07-11 07:52:36.886957) v4 > -341> 2018-07-11 07:52:35.654992 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 <== osd.5 10.8.128.113:0/15263 252 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:36.886957) v4 ==== 2004+0+0 (2383712439 0 0) 0x55d986cd5000 con 0x55d985d44000 > -340> 2018-07-11 07:52:35.654995 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.5 10.8.128.113:0/15263 252 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:36.886957) v4 ==== 2004+0+0 (2383712439 0 0) 0x55d986d4e200 con 0x55d985d45800 > -339> 2018-07-11 07:52:35.655004 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:36.886957) v4 -- 0x55d9c4c1ae00 con 0 > -338> 2018-07-11 07:52:35.655034 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:36.886957) v4 -- 0x55d9bf805200 con 0 > -337> 2018-07-11 07:52:35.671804 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12677 conn(0x55d986db4800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.1 seq 273 0x55d982d31600 osd_ping(ping e501 stamp 2018-07-11 07:52:35.720142) v4 > -336> 2018-07-11 07:52:35.671825 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.1 10.8.128.106:0/12677 273 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:35.720142) v4 ==== 2004+0+0 (2575881897 0 0) 0x55d982d31600 con 0x55d986db4800 > -335> 2018-07-11 07:52:35.671822 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12677 conn(0x55d986df2000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.1 seq 273 0x55d986d40000 osd_ping(ping e501 stamp 2018-07-11 07:52:35.720142) v4 > -334> 2018-07-11 07:52:35.671850 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:35.720142) v4 -- 0x55d9bf805200 con 0 > -333> 2018-07-11 07:52:35.671855 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.1 10.8.128.106:0/12677 273 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:35.720142) v4 ==== 2004+0+0 (2575881897 0 0) 0x55d986d40000 con 0x55d986df2000 > -332> 2018-07-11 07:52:35.671881 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:35.720142) v4 -- 0x55d9bfbde400 con 0 > -331> 2018-07-11 07:52:35.674900 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15255 conn(0x55d986c82800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.0 seq 269 0x55d986d4e200 osd_ping(ping e501 stamp 2018-07-11 07:52:36.906890) v4 > -330> 2018-07-11 07:52:35.674920 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.0 10.8.128.113:0/15255 269 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:36.906890) v4 ==== 2004+0+0 (1986791492 0 0) 0x55d986d4e200 con 0x55d986c82800 > -329> 2018-07-11 07:52:35.674929 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:36.906890) v4 -- 0x55d9bf805200 con 0 > -328> 2018-07-11 07:52:35.674919 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15255 conn(0x55d986d20000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.0 seq 269 0x55d986d32c00 osd_ping(ping e501 stamp 2018-07-11 07:52:36.906890) v4 > -327> 2018-07-11 07:52:35.674950 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.0 10.8.128.113:0/15255 269 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:36.906890) v4 ==== 2004+0+0 (1986791492 0 0) 0x55d986d32c00 con 0x55d986d20000 > -326> 2018-07-11 07:52:35.674966 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:36.906890) v4 -- 0x55d9bfbde400 con 0 > -325> 2018-07-11 07:52:35.824934 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12650 conn(0x55d986d23000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 262 0x55d982d31600 osd_ping(ping e501 stamp 2018-07-11 07:52:35.873257) v4 > -324> 2018-07-11 07:52:35.824967 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.3 10.8.128.106:0/12650 262 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:35.873257) v4 ==== 2004+0+0 (1215669741 0 0) 0x55d982d31600 con 0x55d986d23000 > -323> 2018-07-11 07:52:35.824976 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:35.873257) v4 -- 0x55d9bf805200 con 0 > -322> 2018-07-11 07:52:35.825028 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12650 conn(0x55d986d21800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 262 0x55d980f5a200 osd_ping(ping e501 stamp 2018-07-11 07:52:35.873257) v4 > -321> 2018-07-11 07:52:35.825050 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.3 10.8.128.106:0/12650 262 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:35.873257) v4 ==== 2004+0+0 (1215669741 0 0) 0x55d980f5a200 con 0x55d986d21800 > -320> 2018-07-11 07:52:35.825060 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:35.873257) v4 -- 0x55d9c4c1ae00 con 0 > -319> 2018-07-11 07:52:36.025545 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.114:0/15885 conn(0x55d986d81000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.7 seq 275 0x55d986d40000 osd_ping(ping e501 stamp 2018-07-11 07:52:36.656092) v4 > -318> 2018-07-11 07:52:36.025558 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.114:0/15885 conn(0x55d986d7f800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.7 seq 275 0x55d986d4e200 osd_ping(ping e501 stamp 2018-07-11 07:52:36.656092) v4 > -317> 2018-07-11 07:52:36.025566 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.7 10.8.128.114:0/15885 275 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:36.656092) v4 ==== 2004+0+0 (3184789035 0 0) 0x55d986d40000 con 0x55d986d81000 > -316> 2018-07-11 07:52:36.025581 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.7 10.8.128.114:0/15885 275 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:36.656092) v4 ==== 2004+0+0 (3184789035 0 0) 0x55d986d4e200 con 0x55d986d7f800 > -315> 2018-07-11 07:52:36.025588 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:36.656092) v4 -- 0x55d9bfbde400 con 0 > -314> 2018-07-11 07:52:36.025620 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:36.656092) v4 -- 0x55d9bf805200 con 0 > -313> 2018-07-11 07:52:36.237080 7ffabc3f1700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 34] Generated table #36815: 3672198 keys, 77132968 bytes > -312> 2018-07-11 07:52:36.237128 7ffabc3f1700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295556237100, "cf_name": "default", "job": 34, "event": "table_file_creation", "file_number": 36815, "file_size": 77132968, "table_properties": {"data_size": 67110610, "index_size": 689631, "filter_size": 9331734, "raw_key_size": 106085701, "raw_average_key_size": 28, "raw_value_size": 17380103, "raw_average_value_size": 4, "num_data_blocks": 16481, "num_entries": 3672198, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "2210319", "kMergeOperands": "0"}} > -311> 2018-07-11 07:52:36.325468 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12650 conn(0x55d986d21800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 263 0x55d986cd5000 osd_ping(ping e501 stamp 2018-07-11 07:52:36.373814) v4 > -310> 2018-07-11 07:52:36.325493 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.3 10.8.128.106:0/12650 263 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:36.373814) v4 ==== 2004+0+0 (3983078768 0 0) 0x55d986cd5000 con 0x55d986d21800 > -309> 2018-07-11 07:52:36.325504 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:36.373814) v4 -- 0x55d9c4c1ae00 con 0 > -308> 2018-07-11 07:52:36.325505 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12650 conn(0x55d986d23000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 263 0x55d982d31600 osd_ping(ping e501 stamp 2018-07-11 07:52:36.373814) v4 > -307> 2018-07-11 07:52:36.325527 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.3 10.8.128.106:0/12650 263 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:36.373814) v4 ==== 2004+0+0 (3983078768 0 0) 0x55d982d31600 con 0x55d986d23000 > -306> 2018-07-11 07:52:36.325538 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:36.373814) v4 -- 0x55d9bf805200 con 0 > -305> 2018-07-11 07:52:36.378715 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6805/15263 -- MOSDScrubReserve(6.15 REJECT e501) v1 -- 0x55d986c22e00 con 0 > -304> 2018-07-11 07:52:36.378952 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -303> 2018-07-11 07:52:36.378978 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -302> 2018-07-11 07:52:36.379026 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -301> 2018-07-11 07:52:36.379241 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6805/15263 conn(0x55d986c41000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=31 cs=1 l=0). rx osd.5 seq 2853 0x55d986c22e00 MOSDScrubReserve(6.15 RELEASE e501) v1 > -300> 2018-07-11 07:52:36.379276 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.5 10.8.128.113:6805/15263 2853 ==== MOSDScrubReserve(6.15 RELEASE e501) v1 ==== 43+0+0 (605848072 0 0) 0x55d986c22e00 con 0x55d986c41000 > -299> 2018-07-11 07:52:36.379507 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -298> 2018-07-11 07:52:36.398001 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67339,epoch:501/498,start:5:758ce7ee:::benchmark_data_magna115_10867_object2055687:0,end:5:758d500f:::benchmark_data_magna115_10867_object2501491:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb772c80 con 0 > -297> 2018-07-11 07:52:36.398049 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67339,epoch:501/498,start:5:758ce7ee:::benchmark_data_magna115_10867_object2055687:0,end:5:758d500f:::benchmark_data_magna115_10867_object2501491:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb772f00 con 0 > -296> 2018-07-11 07:52:36.600418 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1099 0x55d9b314e900 rep_scrubmap(5.2e e501 from shard 2) v1 > -295> 2018-07-11 07:52:36.600440 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1099 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12464 (1269249638 0 123434470) 0x55d9b314e900 con 0x55d986bf0800 > -294> 2018-07-11 07:52:36.616192 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1093 0x55d9b314eb40 rep_scrubmap(5.2e e501 from shard 4) v1 > -293> 2018-07-11 07:52:36.616214 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1093 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12464 (3686815630 0 123434470) 0x55d9b314eb40 con 0x55d986bec000 > -292> 2018-07-11 07:52:36.693452 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15287 conn(0x55d981240800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 283 0x55d986d40000 osd_ping(ping e501 stamp 2018-07-11 07:52:37.925429) v4 > -291> 2018-07-11 07:52:36.693460 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15287 conn(0x55d98123f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 283 0x55d986d4e200 osd_ping(ping e501 stamp 2018-07-11 07:52:37.925429) v4 > -290> 2018-07-11 07:52:36.693474 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.4 10.8.128.113:0/15287 283 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:37.925429) v4 ==== 2004+0+0 (1454539513 0 0) 0x55d986d40000 con 0x55d981240800 > -289> 2018-07-11 07:52:36.693477 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.4 10.8.128.113:0/15287 283 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:37.925429) v4 ==== 2004+0+0 (1454539513 0 0) 0x55d986d4e200 con 0x55d98123f000 > -288> 2018-07-11 07:52:36.693484 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:37.925429) v4 -- 0x55d9bfbde400 con 0 > -287> 2018-07-11 07:52:36.693506 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:37.925429) v4 -- 0x55d9bf805200 con 0 > -286> 2018-07-11 07:52:36.737041 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -285> 2018-07-11 07:52:36.737067 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -284> 2018-07-11 07:52:36.737123 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -283> 2018-07-11 07:52:36.737603 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -282> 2018-07-11 07:52:36.756681 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:758d500f:::benchmark_data_magna115_10867_object2501491:0,end:5:758da4da:::benchmark_data_magna115_11707_object208628:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb772a00 con 0 > -281> 2018-07-11 07:52:36.756728 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:758d500f:::benchmark_data_magna115_10867_object2501491:0,end:5:758da4da:::benchmark_data_magna115_11707_object208628:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb772780 con 0 > -280> 2018-07-11 07:52:36.848837 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 3048 0x55d980f5a200 MOSDScrubReserve(11.3s5 REQUEST e501) v1 > -279> 2018-07-11 07:52:36.848855 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 3048 ==== MOSDScrubReserve(11.3s5 REQUEST e501) v1 ==== 43+0+0 (3260170403 0 0) 0x55d980f5a200 con 0x55d986c7e000 > -278> 2018-07-11 07:52:36.848975 7ffab3be0700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e501) v1 -- 0x55d986cd4c00 con 0 > -277> 2018-07-11 07:52:36.849362 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 3049 0x55d986cd4c00 MOSDScrubReserve(11.3s5 RELEASE e501) v1 > -276> 2018-07-11 07:52:36.849380 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 3049 ==== MOSDScrubReserve(11.3s5 RELEASE e501) v1 ==== 43+0+0 (223328929 0 0) 0x55d986cd4c00 con 0x55d986c7e000 > -275> 2018-07-11 07:52:36.970941 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1100 0x55d9b314ed80 rep_scrubmap(5.2e e501 from shard 2) v1 > -274> 2018-07-11 07:52:36.970963 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1100 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12466 (1269249638 0 1871360384) 0x55d9b314ed80 con 0x55d986bf0800 > -273> 2018-07-11 07:52:36.975063 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1094 0x55d9b314efc0 rep_scrubmap(5.2e e501 from shard 4) v1 > -272> 2018-07-11 07:52:36.975082 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1094 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12466 (3686815630 0 1871360384) 0x55d9b314efc0 con 0x55d986bec000 > -271> 2018-07-11 07:52:37.098034 7ffacbbd1700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.108:0/11967 conn(0x55d986d82800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.8 seq 254 0x55d986cd5000 osd_ping(ping e501 stamp 2018-07-11 07:52:36.853907) v4 > -270> 2018-07-11 07:52:37.098057 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 <== osd.8 10.8.128.108:0/11967 254 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:36.853907) v4 ==== 2004+0+0 (123566741 0 0) 0x55d986cd5000 con 0x55d986d82800 > -269> 2018-07-11 07:52:37.098052 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.108:0/11967 conn(0x55d986db0000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.8 seq 254 0x55d982d31600 osd_ping(ping e501 stamp 2018-07-11 07:52:36.853907) v4 > -268> 2018-07-11 07:52:37.098079 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:36.853907) v4 -- 0x55d9bdba5200 con 0 > -267> 2018-07-11 07:52:37.098081 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.8 10.8.128.108:0/11967 254 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:36.853907) v4 ==== 2004+0+0 (123566741 0 0) 0x55d982d31600 con 0x55d986db0000 > -266> 2018-07-11 07:52:37.098114 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:36.853907) v4 -- 0x55d9bf805200 con 0 > -265> 2018-07-11 07:52:37.100887 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.114:0/15919 conn(0x55d986df3800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.11 seq 272 0x55d9c4c1ae00 osd_ping(ping e501 stamp 2018-07-11 07:52:37.731440) v4 > -264> 2018-07-11 07:52:37.100887 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.114:0/15919 conn(0x55d986df5000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.11 seq 272 0x55d986d4e200 osd_ping(ping e501 stamp 2018-07-11 07:52:37.731440) v4 > -263> 2018-07-11 07:52:37.100914 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.11 10.8.128.114:0/15919 272 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:37.731440) v4 ==== 2004+0+0 (3789998297 0 0) 0x55d986d4e200 con 0x55d986df5000 > -262> 2018-07-11 07:52:37.100912 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.11 10.8.128.114:0/15919 272 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:37.731440) v4 ==== 2004+0+0 (3789998297 0 0) 0x55d9c4c1ae00 con 0x55d986df3800 > -261> 2018-07-11 07:52:37.100931 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:37.731440) v4 -- 0x55d9bf805200 con 0 > -260> 2018-07-11 07:52:37.100955 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:37.731440) v4 -- 0x55d9bdba5200 con 0 > -259> 2018-07-11 07:52:37.110269 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.115:0/13323 conn(0x55d986d7e000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 271 0x55d986cd5000 osd_ping(ping e501 stamp 2018-07-11 07:52:37.456461) v4 > -258> 2018-07-11 07:52:37.110291 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.6 10.8.128.115:0/13323 271 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:37.456461) v4 ==== 2004+0+0 (4003054625 0 0) 0x55d986cd5000 con 0x55d986d7e000 > -257> 2018-07-11 07:52:37.110289 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.115:0/13323 conn(0x55d986d24800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 271 0x55d986d40000 osd_ping(ping e501 stamp 2018-07-11 07:52:37.456461) v4 > -256> 2018-07-11 07:52:37.110307 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:37.456461) v4 -- 0x55d9bdba5200 con 0 > -255> 2018-07-11 07:52:37.110320 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.6 10.8.128.115:0/13323 271 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:37.456461) v4 ==== 2004+0+0 (4003054625 0 0) 0x55d986d40000 con 0x55d986d24800 > -254> 2018-07-11 07:52:37.110336 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:37.456461) v4 -- 0x55d9bfbde400 con 0 > -253> 2018-07-11 07:52:37.256452 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -252> 2018-07-11 07:52:37.256480 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -251> 2018-07-11 07:52:37.256536 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -250> 2018-07-11 07:52:37.256983 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -249> 2018-07-11 07:52:37.298202 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:758da4da:::benchmark_data_magna115_11707_object208628:0,end:5:758e24f2:::benchmark_data_magna115_10867_object871330:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb773900 con 0 > -248> 2018-07-11 07:52:37.298267 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:758da4da:::benchmark_data_magna115_11707_object208628:0,end:5:758e24f2:::benchmark_data_magna115_10867_object871330:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb773180 con 0 > -247> 2018-07-11 07:52:37.354107 7ffab6be6700 1 -- 10.8.128.107:6800/13927 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55d9ba81b8c0 con 0 > -246> 2018-07-11 07:52:37.354186 7ffab6be6700 1 -- 10.8.128.107:6800/13927 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55d983c498c0 con 0 > -245> 2018-07-11 07:52:37.485597 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1101 0x55d9b314f200 rep_scrubmap(5.2e e501 from shard 2) v1 > -244> 2018-07-11 07:52:37.485617 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1101 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12470 (1269249638 0 3873110229) 0x55d9b314f200 con 0x55d986bf0800 > -243> 2018-07-11 07:52:37.526211 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1095 0x55d9b314f440 rep_scrubmap(5.2e e501 from shard 4) v1 > -242> 2018-07-11 07:52:37.526230 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1095 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12470 (3686815630 0 3873110229) 0x55d9b314f440 con 0x55d986bec000 > -241> 2018-07-11 07:52:37.572819 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6805/15885 conn(0x55d986c7f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.7 seq 2834 0x55d986d32c00 MOSDScrubReserve(5.4 REQUEST e501) v1 > -240> 2018-07-11 07:52:37.572837 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.7 10.8.128.114:6805/15885 2834 ==== MOSDScrubReserve(5.4 REQUEST e501) v1 ==== 43+0+0 (1753221512 0 0) 0x55d986d32c00 con 0x55d986c7f800 > -239> 2018-07-11 07:52:37.572894 7ffab33df700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.114:6805/15885 -- MOSDScrubReserve(5.4 REJECT e501) v1 -- 0x55d9b67d9200 con 0 > -238> 2018-07-11 07:52:37.573279 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6805/15885 conn(0x55d986c7f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=0). rx osd.7 seq 2835 0x55d9b67d9200 MOSDScrubReserve(5.4 RELEASE e501) v1 > -237> 2018-07-11 07:52:37.573294 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.7 10.8.128.114:6805/15885 2835 ==== MOSDScrubReserve(5.4 RELEASE e501) v1 ==== 43+0+0 (2812146570 0 0) 0x55d9b67d9200 con 0x55d986c7f800 > -236> 2018-07-11 07:52:37.593935 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -235> 2018-07-11 07:52:37.593961 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -234> 2018-07-11 07:52:37.594017 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -233> 2018-07-11 07:52:37.594486 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -232> 2018-07-11 07:52:37.615092 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:758e24f2:::benchmark_data_magna115_10867_object871330:0,end:5:758e9a90:::benchmark_data_magna115_10867_object1334927:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb773680 con 0 > -231> 2018-07-11 07:52:37.615142 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:758e24f2:::benchmark_data_magna115_10867_object871330:0,end:5:758e9a90:::benchmark_data_magna115_10867_object1334927:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454000 con 0 > -230> 2018-07-11 07:52:37.827559 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1102 0x55d9b314f680 rep_scrubmap(5.2e e501 from shard 2) v1 > -229> 2018-07-11 07:52:37.827579 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1102 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12474 (1269249638 0 1954246777) 0x55d9b314f680 con 0x55d986bf0800 > -228> 2018-07-11 07:52:37.849081 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 3050 0x55d9c4c1ae00 MOSDScrubReserve(11.3s5 REQUEST e501) v1 > -227> 2018-07-11 07:52:37.849099 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 3050 ==== MOSDScrubReserve(11.3s5 REQUEST e501) v1 ==== 43+0+0 (3260170403 0 0) 0x55d9c4c1ae00 con 0x55d986c7e000 > -226> 2018-07-11 07:52:37.849158 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1096 0x55d9b314f8c0 rep_scrubmap(5.2e e501 from shard 4) v1 > -225> 2018-07-11 07:52:37.849172 7ffab3be0700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e501) v1 -- 0x55d986cd4c00 con 0 > -224> 2018-07-11 07:52:37.849179 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1096 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12474 (3686815630 0 1954246777) 0x55d9b314f8c0 con 0x55d986bec000 > -223> 2018-07-11 07:52:37.849548 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 3051 0x55d986cd4c00 MOSDScrubReserve(11.3s5 RELEASE e501) v1 > -222> 2018-07-11 07:52:37.849565 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 3051 ==== MOSDScrubReserve(11.3s5 RELEASE e501) v1 ==== 43+0+0 (223328929 0 0) 0x55d986cd4c00 con 0x55d986c7e000 > -221> 2018-07-11 07:52:37.922682 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -220> 2018-07-11 07:52:37.922706 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -219> 2018-07-11 07:52:37.922763 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -218> 2018-07-11 07:52:37.923211 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -217> 2018-07-11 07:52:37.970178 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6801/15919 conn(0x55d986c3f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=29 cs=1 l=0). rx osd.11 seq 258 0x55d986cd5000 MOSDScrubReserve(10.f REQUEST e501) v1 > -216> 2018-07-11 07:52:37.970198 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.11 10.8.128.114:6801/15919 258 ==== MOSDScrubReserve(10.f REQUEST e501) v1 ==== 43+0+0 (2892966580 0 0) 0x55d986cd5000 con 0x55d986c3f800 > -215> 2018-07-11 07:52:37.970229 7ffab53e3700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.114:6801/15919 -- MOSDScrubReserve(10.f REJECT e501) v1 -- 0x55d9b4388200 con 0 > -214> 2018-07-11 07:52:37.970506 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6801/15919 conn(0x55d986c3f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=29 cs=1 l=0). rx osd.11 seq 259 0x55d986d96400 MOSDScrubReserve(10.f RELEASE e501) v1 > -213> 2018-07-11 07:52:37.970520 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.11 10.8.128.114:6801/15919 259 ==== MOSDScrubReserve(10.f RELEASE e501) v1 ==== 43+0+0 (1668468918 0 0) 0x55d986d96400 con 0x55d986c3f800 > -212> 2018-07-11 07:52:37.973370 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'66963,epoch:501/498,start:5:758e9a90:::benchmark_data_magna115_10867_object1334927:0,end:5:758ef33a:::benchmark_data_magna115_10867_object2843543:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb773400 con 0 > -211> 2018-07-11 07:52:37.973408 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'66963,epoch:501/498,start:5:758e9a90:::benchmark_data_magna115_10867_object1334927:0,end:5:758ef33a:::benchmark_data_magna115_10867_object2843543:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454280 con 0 > -210> 2018-07-11 07:52:38.168095 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1103 0x55d9b314fb00 rep_scrubmap(5.2e e501 from shard 2) v1 > -209> 2018-07-11 07:52:38.168117 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1103 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12472 (1269249638 0 20143920) 0x55d9b314fb00 con 0x55d986bf0800 > -208> 2018-07-11 07:52:38.181925 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1097 0x55d9b314fd40 rep_scrubmap(5.2e e501 from shard 4) v1 > -207> 2018-07-11 07:52:38.181944 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1097 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12472 (3686815630 0 20143920) 0x55d9b314fd40 con 0x55d986bec000 > -206> 2018-07-11 07:52:38.324789 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -205> 2018-07-11 07:52:38.324817 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -204> 2018-07-11 07:52:38.324874 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -203> 2018-07-11 07:52:38.325367 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -202> 2018-07-11 07:52:38.342363 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:758ef33a:::benchmark_data_magna115_10867_object2843543:0,end:5:758f3a06:::benchmark_data_magna115_10867_object2596026:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9bb773b80 con 0 > -201> 2018-07-11 07:52:38.342412 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:758ef33a:::benchmark_data_magna115_10867_object2843543:0,end:5:758f3a06:::benchmark_data_magna115_10867_object2596026:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454c80 con 0 > -200> 2018-07-11 07:52:38.393817 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15287 conn(0x55d981240800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 284 0x55d986d40000 osd_ping(ping e501 stamp 2018-07-11 07:52:39.625859) v4 > -199> 2018-07-11 07:52:38.393844 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.4 10.8.128.113:0/15287 284 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:39.625859) v4 ==== 2004+0+0 (2934216484 0 0) 0x55d986d40000 con 0x55d981240800 > -198> 2018-07-11 07:52:38.393858 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:39.625859) v4 -- 0x55d986d40400 con 0 > -197> 2018-07-11 07:52:38.393852 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15287 conn(0x55d98123f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 284 0x55d982d31600 osd_ping(ping e501 stamp 2018-07-11 07:52:39.625859) v4 > -196> 2018-07-11 07:52:38.393873 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.4 10.8.128.113:0/15287 284 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:39.625859) v4 ==== 2004+0+0 (2934216484 0 0) 0x55d982d31600 con 0x55d98123f000 > -195> 2018-07-11 07:52:38.393886 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:39.625859) v4 -- 0x55d9bf805200 con 0 > -194> 2018-07-11 07:52:38.561566 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1104 0x55d99664e000 rep_scrubmap(5.2e e501 from shard 2) v1 > -193> 2018-07-11 07:52:38.561588 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1104 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12480 (1269249638 0 3265768225) 0x55d99664e000 con 0x55d986bf0800 > -192> 2018-07-11 07:52:38.602905 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1098 0x55d99664e240 rep_scrubmap(5.2e e501 from shard 4) v1 > -191> 2018-07-11 07:52:38.602926 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1098 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12480 (3686815630 0 3265768225) 0x55d99664e240 con 0x55d986bec000 > -190> 2018-07-11 07:52:38.709534 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -189> 2018-07-11 07:52:38.709560 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -188> 2018-07-11 07:52:38.709617 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -187> 2018-07-11 07:52:38.710082 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -186> 2018-07-11 07:52:38.732376 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:758f3a06:::benchmark_data_magna115_10867_object2596026:0,end:5:758f9daa:::benchmark_data_magna115_11707_object301193:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454a00 con 0 > -185> 2018-07-11 07:52:38.732416 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:758f3a06:::benchmark_data_magna115_10867_object2596026:0,end:5:758f9daa:::benchmark_data_magna115_11707_object301193:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455180 con 0 > -184> 2018-07-11 07:52:38.827822 7ffab1bdc700 5 osd.10 501 heartbeat: osd_stat(789 GB used, 142 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist []) > -183> 2018-07-11 07:52:38.827837 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6802/15255 -- osd_ping(ping e501 stamp 2018-07-11 07:52:38.827835) v4 -- 0x55d9bf804800 con 0 > -182> 2018-07-11 07:52:38.827871 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6803/15255 -- osd_ping(ping e501 stamp 2018-07-11 07:52:38.827835) v4 -- 0x55d9829e5000 con 0 > -181> 2018-07-11 07:52:38.827906 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6806/12677 -- osd_ping(ping e501 stamp 2018-07-11 07:52:38.827835) v4 -- 0x55d986b6dc00 con 0 > -180> 2018-07-11 07:52:38.827926 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6807/12677 -- osd_ping(ping e501 stamp 2018-07-11 07:52:38.827835) v4 -- 0x55d9829e5e00 con 0 > -179> 2018-07-11 07:52:38.827943 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6808/12651 -- osd_ping(ping e501 stamp 2018-07-11 07:52:38.827835) v4 -- 0x55d986d96200 con 0 > -178> 2018-07-11 07:52:38.827960 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6810/12651 -- osd_ping(ping e501 stamp 2018-07-11 07:52:38.827835) v4 -- 0x55d986c61600 con 0 > -177> 2018-07-11 07:52:38.827985 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6809/12650 -- osd_ping(ping e501 stamp 2018-07-11 07:52:38.827835) v4 -- 0x55d9829e4a00 con 0 > -176> 2018-07-11 07:52:38.828004 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.106:6811/12650 -- osd_ping(ping e501 stamp 2018-07-11 07:52:38.827835) v4 -- 0x55d9b40a5800 con 0 > -175> 2018-07-11 07:52:38.828025 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6810/15287 -- osd_ping(ping e501 stamp 2018-07-11 07:52:38.827835) v4 -- 0x55d9b3db3200 con 0 > -174> 2018-07-11 07:52:38.828042 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6811/15287 -- osd_ping(ping e501 stamp 2018-07-11 07:52:38.827835) v4 -- 0x55d9c908ee00 con 0 > -173> 2018-07-11 07:52:38.828060 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6806/15263 -- osd_ping(ping e501 stamp 2018-07-11 07:52:38.827835) v4 -- 0x55d9bfbde200 con 0 > -172> 2018-07-11 07:52:38.828080 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.113:6807/15263 -- osd_ping(ping e501 stamp 2018-07-11 07:52:38.827835) v4 -- 0x55d982d31e00 con 0 > -171> 2018-07-11 07:52:38.828097 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.115:6802/13323 -- osd_ping(ping e501 stamp 2018-07-11 07:52:38.827835) v4 -- 0x55d9869be400 con 0 > -170> 2018-07-11 07:52:38.828119 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.115:6803/13323 -- osd_ping(ping e501 stamp 2018-07-11 07:52:38.827835) v4 -- 0x55d986ca9a00 con 0 > -169> 2018-07-11 07:52:38.828137 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6806/15885 -- osd_ping(ping e501 stamp 2018-07-11 07:52:38.827835) v4 -- 0x55d9ba818e00 con 0 > -168> 2018-07-11 07:52:38.828155 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6807/15885 -- osd_ping(ping e501 stamp 2018-07-11 07:52:38.827835) v4 -- 0x55d985054000 con 0 > -167> 2018-07-11 07:52:38.828173 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.108:6802/11967 -- osd_ping(ping e501 stamp 2018-07-11 07:52:38.827835) v4 -- 0x55d986c84600 con 0 > -166> 2018-07-11 07:52:38.828189 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.108:6803/11967 -- osd_ping(ping e501 stamp 2018-07-11 07:52:38.827835) v4 -- 0x55d9b4389800 con 0 > -165> 2018-07-11 07:52:38.828235 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.107:6806/12617 -- osd_ping(ping e501 stamp 2018-07-11 07:52:38.827835) v4 -- 0x55d9b2782e00 con 0 > -164> 2018-07-11 07:52:38.828270 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.107:6807/12617 -- osd_ping(ping e501 stamp 2018-07-11 07:52:38.827835) v4 -- 0x55d986c22800 con 0 > -163> 2018-07-11 07:52:38.828289 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6802/15919 -- osd_ping(ping e501 stamp 2018-07-11 07:52:38.827835) v4 -- 0x55d985074000 con 0 > -162> 2018-07-11 07:52:38.828306 7ffab1bdc700 1 -- 10.8.128.107:0/13927 --> 10.8.128.114:6803/15919 -- osd_ping(ping e501 stamp 2018-07-11 07:52:38.827835) v4 -- 0x55d9bf804a00 con 0 > -161> 2018-07-11 07:52:38.828421 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6809/12650 conn(0x55d986bcb000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 262 0x55d9bdba5200 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:38.827835) v4 > -160> 2018-07-11 07:52:38.828446 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.3 10.8.128.106:6809/12650 262 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:38.827835) v4 ==== 2004+0+0 (4290667359 0 0) 0x55d9bdba5200 con 0x55d986bcb000 > -159> 2018-07-11 07:52:38.828460 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6802/15255 conn(0x55d986bc8000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.0 seq 262 0x55d9bfbde400 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:38.827835) v4 > -158> 2018-07-11 07:52:38.828483 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.0 10.8.128.113:6802/15255 262 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:38.827835) v4 ==== 2004+0+0 (4290667359 0 0) 0x55d9bfbde400 con 0x55d986bc8000 > -157> 2018-07-11 07:52:38.828481 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6803/15255 conn(0x55d986bc9800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=14 cs=1 l=1). rx osd.0 seq 262 0x55d982d31600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:38.827835) v4 > -156> 2018-07-11 07:52:38.828493 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.107:6807/12617 conn(0x55d986bd3800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.9 seq 262 0x55d9bdba5200 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:38.827835) v4 > -155> 2018-07-11 07:52:38.828502 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.0 10.8.128.113:6803/15255 262 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:38.827835) v4 ==== 2004+0+0 (4290667359 0 0) 0x55d982d31600 con 0x55d986bc9800 > -154> 2018-07-11 07:52:38.828506 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.9 10.8.128.107:6807/12617 262 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:38.827835) v4 ==== 2004+0+0 (4290667359 0 0) 0x55d9bdba5200 con 0x55d986bd3800 > -153> 2018-07-11 07:52:38.828541 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6810/12651 conn(0x55d986b9b800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.2 seq 262 0x55d982d31600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:38.827835) v4 > -152> 2018-07-11 07:52:38.828545 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6807/12677 conn(0x55d986b59800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.1 seq 262 0x55d9bfbde400 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:38.827835) v4 > -151> 2018-07-11 07:52:38.828554 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.2 10.8.128.106:6810/12651 262 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:38.827835) v4 ==== 2004+0+0 (4290667359 0 0) 0x55d982d31600 con 0x55d986b9b800 > -150> 2018-07-11 07:52:38.828556 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.1 10.8.128.106:6807/12677 262 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:38.827835) v4 ==== 2004+0+0 (4290667359 0 0) 0x55d9bfbde400 con 0x55d986b59800 > -149> 2018-07-11 07:52:38.828561 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6806/15263 conn(0x55d986b9d000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.5 seq 262 0x55d9bdba5200 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:38.827835) v4 > -148> 2018-07-11 07:52:38.828575 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.5 10.8.128.113:6806/15263 262 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:38.827835) v4 ==== 2004+0+0 (4290667359 0 0) 0x55d9bdba5200 con 0x55d986b9d000 > -147> 2018-07-11 07:52:38.828591 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6811/15287 conn(0x55d986bb0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 262 0x55d982d31600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:38.827835) v4 > -146> 2018-07-11 07:52:38.828603 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.4 10.8.128.113:6811/15287 262 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:38.827835) v4 ==== 2004+0+0 (4290667359 0 0) 0x55d982d31600 con 0x55d986bb0800 > -145> 2018-07-11 07:52:38.828609 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6808/12651 conn(0x55d986b9a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.2 seq 262 0x55d9bfbde400 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:38.827835) v4 > -144> 2018-07-11 07:52:38.828623 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.2 10.8.128.106:6808/12651 262 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:38.827835) v4 ==== 2004+0+0 (4290667359 0 0) 0x55d9bfbde400 con 0x55d986b9a000 > -143> 2018-07-11 07:52:38.828617 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6806/12677 conn(0x55d986b58000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.1 seq 262 0x55d9bdba5200 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:38.827835) v4 > -142> 2018-07-11 07:52:38.828633 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.1 10.8.128.106:6806/12677 262 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:38.827835) v4 ==== 2004+0+0 (4290667359 0 0) 0x55d9bdba5200 con 0x55d986b58000 > -141> 2018-07-11 07:52:38.828638 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.107:6806/12617 conn(0x55d986bd2000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.9 seq 262 0x55d982d31600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:38.827835) v4 > -140> 2018-07-11 07:52:38.828651 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.9 10.8.128.107:6806/12617 262 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:38.827835) v4 ==== 2004+0+0 (4290667359 0 0) 0x55d982d31600 con 0x55d986bd2000 > -139> 2018-07-11 07:52:38.828654 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6807/15263 conn(0x55d986b9e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.5 seq 262 0x55d9bfbde400 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:38.827835) v4 > -138> 2018-07-11 07:52:38.828666 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.5 10.8.128.113:6807/15263 262 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:38.827835) v4 ==== 2004+0+0 (4290667359 0 0) 0x55d9bfbde400 con 0x55d986b9e800 > -137> 2018-07-11 07:52:38.828670 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6807/15885 conn(0x55d986bad800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.7 seq 262 0x55d9bdba5200 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:38.827835) v4 > -136> 2018-07-11 07:52:38.828683 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.7 10.8.128.114:6807/15885 262 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:38.827835) v4 ==== 2004+0+0 (4290667359 0 0) 0x55d9bdba5200 con 0x55d986bad800 > -135> 2018-07-11 07:52:38.828684 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.115:6802/13323 conn(0x55d986bc1000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 262 0x55d982d31600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:38.827835) v4 > -134> 2018-07-11 07:52:38.828693 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.6 10.8.128.115:6802/13323 262 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:38.827835) v4 ==== 2004+0+0 (4290667359 0 0) 0x55d982d31600 con 0x55d986bc1000 > -133> 2018-07-11 07:52:38.828695 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.113:6810/15287 conn(0x55d986baf000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.4 seq 262 0x55d9bfbde400 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:38.827835) v4 > -132> 2018-07-11 07:52:38.828706 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.4 10.8.128.113:6810/15287 262 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:38.827835) v4 ==== 2004+0+0 (4290667359 0 0) 0x55d9bfbde400 con 0x55d986baf000 > -131> 2018-07-11 07:52:38.828715 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.108:6802/11967 conn(0x55d986bbe000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.8 seq 262 0x55d9bdba5200 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:38.827835) v4 > -130> 2018-07-11 07:52:38.828727 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.8 10.8.128.108:6802/11967 262 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:38.827835) v4 ==== 2004+0+0 (4290667359 0 0) 0x55d9bdba5200 con 0x55d986bbe000 > -129> 2018-07-11 07:52:38.828735 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.106:6811/12650 conn(0x55d986bcc800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.3 seq 262 0x55d9bfbde400 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:38.827835) v4 > -128> 2018-07-11 07:52:38.828747 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.3 10.8.128.106:6811/12650 262 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:38.827835) v4 ==== 2004+0+0 (4290667359 0 0) 0x55d9bfbde400 con 0x55d986bcc800 > -127> 2018-07-11 07:52:38.828748 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6806/15885 conn(0x55d986bac000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.7 seq 262 0x55d982d31600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:38.827835) v4 > -126> 2018-07-11 07:52:38.828756 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.115:6803/13323 conn(0x55d986bc2800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.6 seq 262 0x55d9bdba5200 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:38.827835) v4 > -125> 2018-07-11 07:52:38.828762 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.7 10.8.128.114:6806/15885 262 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:38.827835) v4 ==== 2004+0+0 (4290667359 0 0) 0x55d982d31600 con 0x55d986bac000 > -124> 2018-07-11 07:52:38.828768 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.6 10.8.128.115:6803/13323 262 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:38.827835) v4 ==== 2004+0+0 (4290667359 0 0) 0x55d9bdba5200 con 0x55d986bc2800 > -123> 2018-07-11 07:52:38.828778 7ffacb3d0700 5 -- 10.8.128.107:0/13927 >> 10.8.128.108:6803/11967 conn(0x55d986bbf800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=12 cs=1 l=1). rx osd.8 seq 262 0x55d9bfbde400 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:38.827835) v4 > -122> 2018-07-11 07:52:38.828807 7ffacb3d0700 1 -- 10.8.128.107:0/13927 <== osd.8 10.8.128.108:6803/11967 262 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:38.827835) v4 ==== 2004+0+0 (4290667359 0 0) 0x55d9bfbde400 con 0x55d986bbf800 > -121> 2018-07-11 07:52:38.828806 7ffacbbd1700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6803/15919 conn(0x55d986b5c800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.11 seq 262 0x55d9bdba5200 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:38.827835) v4 > -120> 2018-07-11 07:52:38.828817 7ffacbbd1700 1 -- 10.8.128.107:0/13927 <== osd.11 10.8.128.114:6803/15919 262 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:38.827835) v4 ==== 2004+0+0 (4290667359 0 0) 0x55d9bdba5200 con 0x55d986b5c800 > -119> 2018-07-11 07:52:38.828824 7ffacabcf700 5 -- 10.8.128.107:0/13927 >> 10.8.128.114:6802/15919 conn(0x55d986b5b000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.11 seq 262 0x55d982d31600 osd_ping(ping_reply e501 stamp 2018-07-11 07:52:38.827835) v4 > -118> 2018-07-11 07:52:38.828838 7ffacabcf700 1 -- 10.8.128.107:0/13927 <== osd.11 10.8.128.114:6802/15919 262 ==== osd_ping(ping_reply e501 stamp 2018-07-11 07:52:38.827835) v4 ==== 2004+0+0 (4290667359 0 0) 0x55d982d31600 con 0x55d986b5b000 > -117> 2018-07-11 07:52:38.849238 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 3052 0x55d9bdba5200 MOSDScrubReserve(11.3s5 REQUEST e501) v1 > -116> 2018-07-11 07:52:38.849268 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 3052 ==== MOSDScrubReserve(11.3s5 REQUEST e501) v1 ==== 43+0+0 (3260170403 0 0) 0x55d9bdba5200 con 0x55d986c7e000 > -115> 2018-07-11 07:52:38.849331 7ffab3be0700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e501) v1 -- 0x55d986cd4c00 con 0 > -114> 2018-07-11 07:52:38.849729 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.115:6801/13323 conn(0x55d986c7e000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=0). rx osd.6 seq 3053 0x55d986cd4c00 MOSDScrubReserve(11.3s5 RELEASE e501) v1 > -113> 2018-07-11 07:52:38.849747 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.6 10.8.128.115:6801/13323 3053 ==== MOSDScrubReserve(11.3s5 RELEASE e501) v1 ==== 43+0+0 (223328929 0 0) 0x55d986cd4c00 con 0x55d986c7e000 > -112> 2018-07-11 07:52:38.925902 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.114:0/15885 conn(0x55d986d7f800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.7 seq 276 0x55d982d31600 osd_ping(ping e501 stamp 2018-07-11 07:52:39.556498) v4 > -111> 2018-07-11 07:52:38.925927 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.7 10.8.128.114:0/15885 276 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:39.556498) v4 ==== 2004+0+0 (3155650411 0 0) 0x55d982d31600 con 0x55d986d7f800 > -110> 2018-07-11 07:52:38.925931 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.114:0/15885 conn(0x55d986d81000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.7 seq 276 0x55d9bfbde400 osd_ping(ping e501 stamp 2018-07-11 07:52:39.556498) v4 > -109> 2018-07-11 07:52:38.925943 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:39.556498) v4 -- 0x55d985074000 con 0 > -108> 2018-07-11 07:52:38.925954 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.7 10.8.128.114:0/15885 276 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:39.556498) v4 ==== 2004+0+0 (3155650411 0 0) 0x55d9bfbde400 con 0x55d986d81000 > -107> 2018-07-11 07:52:38.925972 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:39.556498) v4 -- 0x55d9b4389800 con 0 > -106> 2018-07-11 07:52:38.940856 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1105 0x55d99664e480 rep_scrubmap(5.2e e501 from shard 2) v1 > -105> 2018-07-11 07:52:38.940875 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1105 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12474 (1269249638 0 703608172) 0x55d99664e480 con 0x55d986bf0800 > -104> 2018-07-11 07:52:38.968907 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6805/15263 conn(0x55d986c41000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=31 cs=1 l=0). rx osd.5 seq 2854 0x55d9bfbde400 MOSDScrubReserve(6.15 REQUEST e501) v1 > -103> 2018-07-11 07:52:38.968926 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.5 10.8.128.113:6805/15263 2854 ==== MOSDScrubReserve(6.15 REQUEST e501) v1 ==== 43+0+0 (3942742026 0 0) 0x55d9bfbde400 con 0x55d986c41000 > -102> 2018-07-11 07:52:38.976200 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1099 0x55d99664e6c0 rep_scrubmap(5.2e e501 from shard 4) v1 > -101> 2018-07-11 07:52:38.976219 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1099 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12474 (3686815630 0 703608172) 0x55d99664e6c0 con 0x55d986bec000 > -100> 2018-07-11 07:52:39.037465 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6805/15263 -- MOSDScrubReserve(6.15 REJECT e501) v1 -- 0x55d986c22e00 con 0 > -99> 2018-07-11 07:52:39.037521 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -98> 2018-07-11 07:52:39.037544 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -97> 2018-07-11 07:52:39.037591 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -96> 2018-07-11 07:52:39.037867 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6805/15263 conn(0x55d986c41000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=31 cs=1 l=0). rx osd.5 seq 2855 0x55d986c22e00 MOSDScrubReserve(6.15 RELEASE e501) v1 > -95> 2018-07-11 07:52:39.037886 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.5 10.8.128.113:6805/15263 2855 ==== MOSDScrubReserve(6.15 RELEASE e501) v1 ==== 43+0+0 (605848072 0 0) 0x55d986c22e00 con 0x55d986c41000 > -94> 2018-07-11 07:52:39.038046 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -93> 2018-07-11 07:52:39.082093 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'66825,epoch:501/498,start:5:758f9daa:::benchmark_data_magna115_11707_object301193:0,end:5:75901b83:::benchmark_data_magna115_10867_object2037709:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454f00 con 0 > -92> 2018-07-11 07:52:39.082145 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'66825,epoch:501/498,start:5:758f9daa:::benchmark_data_magna115_11707_object301193:0,end:5:75901b83:::benchmark_data_magna115_10867_object2037709:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455400 con 0 > -91> 2018-07-11 07:52:39.155372 7ffacbbd1700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.113:0/15263 conn(0x55d985d44000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.5 seq 253 0x55d9b4388200 osd_ping(ping e501 stamp 2018-07-11 07:52:40.387468) v4 > -90> 2018-07-11 07:52:39.155394 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 <== osd.5 10.8.128.113:0/15263 253 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:40.387468) v4 ==== 2004+0+0 (4090684812 0 0) 0x55d9b4388200 con 0x55d985d44000 > -89> 2018-07-11 07:52:39.155382 7ffacabcf700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.113:0/15263 conn(0x55d985d45800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.5 seq 253 0x55d9b40a4c00 osd_ping(ping e501 stamp 2018-07-11 07:52:40.387468) v4 > -88> 2018-07-11 07:52:39.155404 7ffacbbd1700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:40.387468) v4 -- 0x55d986c22800 con 0 > -87> 2018-07-11 07:52:39.155415 7ffacabcf700 1 -- 10.8.128.107:6803/13927 <== osd.5 10.8.128.113:0/15263 253 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:40.387468) v4 ==== 2004+0+0 (4090684812 0 0) 0x55d9b40a4c00 con 0x55d985d45800 > -86> 2018-07-11 07:52:39.155438 7ffacabcf700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:40.387468) v4 -- 0x55d985074000 con 0 > -85> 2018-07-11 07:52:39.323765 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1100 0x55d99664e900 rep_scrubmap(5.2e e501 from shard 4) v1 > -84> 2018-07-11 07:52:39.323786 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1100 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12470 (3686815630 0 3672318136) 0x55d99664e900 con 0x55d986bec000 > -83> 2018-07-11 07:52:39.331948 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1106 0x55d99664eb40 rep_scrubmap(5.2e e501 from shard 2) v1 > -82> 2018-07-11 07:52:39.331968 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1106 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12470 (1269249638 0 3672318136) 0x55d99664eb40 con 0x55d986bf0800 > -81> 2018-07-11 07:52:39.332096 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -80> 2018-07-11 07:52:39.332122 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -79> 2018-07-11 07:52:39.332168 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -78> 2018-07-11 07:52:39.332638 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -77> 2018-07-11 07:52:39.333704 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75901b83:::benchmark_data_magna115_10867_object2037709:0,end:5:759085f0:::benchmark_data_magna115_10867_object1102751:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454500 con 0 > -76> 2018-07-11 07:52:39.333749 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:75901b83:::benchmark_data_magna115_10867_object2037709:0,end:5:759085f0:::benchmark_data_magna115_10867_object1102751:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b454780 con 0 > -75> 2018-07-11 07:52:39.426311 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.114:0/15885 conn(0x55d986d7f800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.7 seq 277 0x55d982d31600 osd_ping(ping e501 stamp 2018-07-11 07:52:40.056915) v4 > -74> 2018-07-11 07:52:39.426320 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.114:0/15885 conn(0x55d986d81000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.7 seq 277 0x55d9b4389800 osd_ping(ping e501 stamp 2018-07-11 07:52:40.056915) v4 > -73> 2018-07-11 07:52:39.426348 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.7 10.8.128.114:0/15885 277 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:40.056915) v4 ==== 2004+0+0 (3860075324 0 0) 0x55d982d31600 con 0x55d986d7f800 > -72> 2018-07-11 07:52:39.426357 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.7 10.8.128.114:0/15885 277 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:40.056915) v4 ==== 2004+0+0 (3860075324 0 0) 0x55d9b4389800 con 0x55d986d81000 > -71> 2018-07-11 07:52:39.426366 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:40.056915) v4 -- 0x55d985074000 con 0 > -70> 2018-07-11 07:52:39.426398 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:40.056915) v4 -- 0x55d982d31e00 con 0 > -69> 2018-07-11 07:52:39.546857 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1107 0x55d99664ed80 rep_scrubmap(5.2e e501 from shard 2) v1 > -68> 2018-07-11 07:52:39.546877 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1107 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12470 (1269249638 0 220703025) 0x55d99664ed80 con 0x55d986bf0800 > -67> 2018-07-11 07:52:39.556861 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1101 0x55d99664efc0 rep_scrubmap(5.2e e501 from shard 4) v1 > -66> 2018-07-11 07:52:39.556880 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1101 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12470 (3686815630 0 220703025) 0x55d99664efc0 con 0x55d986bec000 > -65> 2018-07-11 07:52:39.587872 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -64> 2018-07-11 07:52:39.587897 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -63> 2018-07-11 07:52:39.587952 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -62> 2018-07-11 07:52:39.588401 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -61> 2018-07-11 07:52:39.632626 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:166'67168,epoch:501/498,start:5:759085f0:::benchmark_data_magna115_10867_object1102751:0,end:5:7590ea6d:::benchmark_data_magna115_10867_object2853227:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498280 con 0 > -60> 2018-07-11 07:52:39.632678 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:166'67168,epoch:501/498,start:5:759085f0:::benchmark_data_magna115_10867_object1102751:0,end:5:7590ea6d:::benchmark_data_magna115_10867_object2853227:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b14f0500 con 0 > -59> 2018-07-11 07:52:39.772167 7ffacb3d0700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.106:0/12677 conn(0x55d986df2000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.1 seq 274 0x55d9b4389800 osd_ping(ping e501 stamp 2018-07-11 07:52:39.820548) v4 > -58> 2018-07-11 07:52:39.772164 7ffacabcf700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.106:0/12677 conn(0x55d986db4800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx osd.1 seq 274 0x55d9b40a4c00 osd_ping(ping e501 stamp 2018-07-11 07:52:39.820548) v4 > -57> 2018-07-11 07:52:39.772191 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 <== osd.1 10.8.128.106:0/12677 274 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:39.820548) v4 ==== 2004+0+0 (3299437081 0 0) 0x55d9b4389800 con 0x55d986df2000 > -56> 2018-07-11 07:52:39.772194 7ffacabcf700 1 -- 10.8.128.107:6802/13927 <== osd.1 10.8.128.106:0/12677 274 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:39.820548) v4 ==== 2004+0+0 (3299437081 0 0) 0x55d9b40a4c00 con 0x55d986db4800 > -55> 2018-07-11 07:52:39.772203 7ffacb3d0700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:39.820548) v4 -- 0x55d982d31e00 con 0 > -54> 2018-07-11 07:52:39.772226 7ffacabcf700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:39.820548) v4 -- 0x55d985074000 con 0 > -53> 2018-07-11 07:52:39.863663 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1102 0x55d99664f200 rep_scrubmap(5.2e e501 from shard 4) v1 > -52> 2018-07-11 07:52:39.863681 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1102 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12466 (3686815630 0 3685216837) 0x55d99664f200 con 0x55d986bec000 > -51> 2018-07-11 07:52:39.873100 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1108 0x55d99664f440 rep_scrubmap(5.2e e501 from shard 2) v1 > -50> 2018-07-11 07:52:39.873120 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1108 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12466 (1269249638 0 3685216837) 0x55d99664f440 con 0x55d986bf0800 > -49> 2018-07-11 07:52:39.934742 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -48> 2018-07-11 07:52:39.934767 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -47> 2018-07-11 07:52:39.934814 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -46> 2018-07-11 07:52:39.935283 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -45> 2018-07-11 07:52:39.969062 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7590ea6d:::benchmark_data_magna115_10867_object2853227:0,end:5:759154d3:::benchmark_data_magna115_10867_object396289:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498000 con 0 > -44> 2018-07-11 07:52:39.969114 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7590ea6d:::benchmark_data_magna115_10867_object2853227:0,end:5:759154d3:::benchmark_data_magna115_10867_object396289:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455900 con 0 > -43> 2018-07-11 07:52:39.970412 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6801/15919 conn(0x55d986c3f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=29 cs=1 l=0). rx osd.11 seq 260 0x55d9bf804a00 MOSDScrubReserve(10.f REQUEST e501) v1 > -42> 2018-07-11 07:52:39.970431 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.11 10.8.128.114:6801/15919 260 ==== MOSDScrubReserve(10.f REQUEST e501) v1 ==== 43+0+0 (2892966580 0 0) 0x55d9bf804a00 con 0x55d986c3f800 > -41> 2018-07-11 07:52:39.970467 7ffab53e3700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.114:6801/15919 -- MOSDScrubReserve(10.f REJECT e501) v1 -- 0x55d986d96400 con 0 > -40> 2018-07-11 07:52:39.970794 7ffacbbd1700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.114:6801/15919 conn(0x55d986c3f800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=29 cs=1 l=0). rx osd.11 seq 261 0x55d986d96400 MOSDScrubReserve(10.f RELEASE e501) v1 > -39> 2018-07-11 07:52:39.970809 7ffacbbd1700 1 -- 10.8.128.107:6801/13927 <== osd.11 10.8.128.114:6801/15919 261 ==== MOSDScrubReserve(10.f RELEASE e501) v1 ==== 43+0+0 (1668468918 0 0) 0x55d986d96400 con 0x55d986c3f800 > -38> 2018-07-11 07:52:40.141814 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1109 0x55d99664f680 rep_scrubmap(5.2e e501 from shard 2) v1 > -37> 2018-07-11 07:52:40.141834 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1109 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12484 (1269249638 0 707970655) 0x55d99664f680 con 0x55d986bf0800 > -36> 2018-07-11 07:52:40.179327 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1103 0x55d99664f8c0 rep_scrubmap(5.2e e501 from shard 4) v1 > -35> 2018-07-11 07:52:40.179348 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1103 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12484 (3686815630 0 707970655) 0x55d99664f8c0 con 0x55d986bec000 > -34> 2018-07-11 07:52:40.230619 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -33> 2018-07-11 07:52:40.230641 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -32> 2018-07-11 07:52:40.230696 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -31> 2018-07-11 07:52:40.231139 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -30> 2018-07-11 07:52:40.261539 7ffacb3d0700 5 -- 10.8.128.107:6802/13927 >> 10.8.128.107:0/12617 conn(0x55d98123c000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 264 0x55d986d40000 osd_ping(ping e501 stamp 2018-07-11 07:52:40.261183) v4 > -29> 2018-07-11 07:52:40.261562 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 <== osd.9 10.8.128.107:0/12617 264 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:40.261183) v4 ==== 2004+0+0 (2210996576 0 0) 0x55d986d40000 con 0x55d98123c000 > -28> 2018-07-11 07:52:40.261560 7ffacbbd1700 5 -- 10.8.128.107:6803/13927 >> 10.8.128.107:0/12617 conn(0x55d98123d800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.9 seq 264 0x55d9b4388200 osd_ping(ping e501 stamp 2018-07-11 07:52:40.261183) v4 > -27> 2018-07-11 07:52:40.261579 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 <== osd.9 10.8.128.107:0/12617 264 ==== osd_ping(ping e501 stamp 2018-07-11 07:52:40.261183) v4 ==== 2004+0+0 (2210996576 0 0) 0x55d9b4388200 con 0x55d98123d800 > -26> 2018-07-11 07:52:40.261574 7ffacb3d0700 1 -- 10.8.128.107:6802/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:40.261183) v4 -- 0x55d982d31e00 con 0 > -25> 2018-07-11 07:52:40.261624 7ffacbbd1700 1 -- 10.8.128.107:6803/13927 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e501 stamp 2018-07-11 07:52:40.261183) v4 -- 0x55d986c84600 con 0 > -24> 2018-07-11 07:52:40.283626 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:759154d3:::benchmark_data_magna115_10867_object396289:0,end:5:7591c51a:::benchmark_data_magna115_10867_object2925540:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455b80 con 0 > -23> 2018-07-11 07:52:40.283664 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:759154d3:::benchmark_data_magna115_10867_object396289:0,end:5:7591c51a:::benchmark_data_magna115_10867_object2925540:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98b455680 con 0 > -22> 2018-07-11 07:52:40.508285 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.113:6809/15287 conn(0x55d986bec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=0). rx osd.4 seq 1104 0x55d99664fb00 rep_scrubmap(5.2e e501 from shard 4) v1 > -21> 2018-07-11 07:52:40.508304 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.4 10.8.128.113:6809/15287 1104 ==== rep_scrubmap(5.2e e501 from shard 4) v1 ==== 39+0+12470 (3686815630 0 3984372486) 0x55d99664fb00 con 0x55d986bec000 > -20> 2018-07-11 07:52:40.519800 7ffacb3d0700 5 -- 10.8.128.107:6801/13927 >> 10.8.128.106:6802/12651 conn(0x55d986bf0800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=34 cs=1 l=0). rx osd.2 seq 1110 0x55d99664fd40 rep_scrubmap(5.2e e501 from shard 2) v1 > -19> 2018-07-11 07:52:40.519820 7ffacb3d0700 1 -- 10.8.128.107:6801/13927 <== osd.2 10.8.128.106:6802/12651 1110 ==== rep_scrubmap(5.2e e501 from shard 2) v1 ==== 39+0+12470 (1269249638 0 3984372486) 0x55d99664fd40 con 0x55d986bf0800 > -18> 2018-07-11 07:52:40.542961 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 2 has 25 items > -17> 2018-07-11 07:52:40.542986 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps replica 4 has 25 items > -16> 2018-07-11 07:52:40.543034 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub_compare_maps osd.10 has 25 items > -15> 2018-07-11 07:52:40.543498 7ffab4be2700 2 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=498 crt=166'68041 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] > -14> 2018-07-11 07:52:40.577766 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.106:6802/12651 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7591c51a:::benchmark_data_magna115_10867_object2925540:0,end:5:7592283d:::benchmark_data_magna115_10867_object208658:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d98696e000 con 0 > -13> 2018-07-11 07:52:40.577804 7ffab4be2700 1 -- 10.8.128.107:6801/13927 --> 10.8.128.113:6809/15287 -- replica scrub(pg: 5.2e,from:0'0,to:0'0,epoch:501/498,start:5:7591c51a:::benchmark_data_magna115_10867_object2925540:0,end:5:7592283d:::benchmark_data_magna115_10867_object208658:0,chunky:1,deep:1,seed:4294967295,version:7) v7 -- 0x55d9b1498780 con 0 > -12> 2018-07-11 07:52:40.598802 7ffabc3f1700 -1 bluefs _allocate failed to allocate 0x4800000 on bdev 2, free 0x0 > -11> 2018-07-11 07:52:40.598814 7ffabc3f1700 0 stupidalloc 0x0x55d97f6aa070 dump free bin 0: 0 extents > -10> 2018-07-11 07:52:40.598816 7ffabc3f1700 0 stupidalloc 0x0x55d97f6aa070 dump free bin 1: 0 extents > -9> 2018-07-11 07:52:40.598830 7ffabc3f1700 0 stupidalloc 0x0x55d97f6aa070 dump free bin 2: 0 extents > -8> 2018-07-11 07:52:40.598832 7ffabc3f1700 0 stupidalloc 0x0x55d97f6aa070 dump free bin 3: 0 extents > -7> 2018-07-11 07:52:40.598833 7ffabc3f1700 0 stupidalloc 0x0x55d97f6aa070 dump free bin 4: 0 extents > -6> 2018-07-11 07:52:40.598834 7ffabc3f1700 0 stupidalloc 0x0x55d97f6aa070 dump free bin 5: 0 extents > -5> 2018-07-11 07:52:40.598835 7ffabc3f1700 0 stupidalloc 0x0x55d97f6aa070 dump free bin 6: 0 extents > -4> 2018-07-11 07:52:40.598836 7ffabc3f1700 0 stupidalloc 0x0x55d97f6aa070 dump free bin 7: 0 extents > -3> 2018-07-11 07:52:40.598837 7ffabc3f1700 0 stupidalloc 0x0x55d97f6aa070 dump free bin 8: 0 extents > -2> 2018-07-11 07:52:40.598838 7ffabc3f1700 0 stupidalloc 0x0x55d97f6aa070 dump free bin 9: 0 extents > -1> 2018-07-11 07:52:40.598839 7ffabc3f1700 -1 bluefs _flush_range allocated: 0x0 offset: 0x0 length: 0x4717d2d > 0> 2018-07-11 07:52:40.604936 7ffabc3f1700 -1 /builddir/build/BUILD/ceph-12.2.5/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, uint64_t)' thread 7ffabc3f1700 time 2018-07-11 07:52:40.598851 >/builddir/build/BUILD/ceph-12.2.5/src/os/bluestore/BlueFS.cc: 1525: FAILED assert(0 == "bluefs enospc") > > ceph version 12.2.5-27.el7cp (6b7d1a8cf2e2376fd0a12424947c8c97aa1c0890) luminous (stable) > 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x110) [0x55d97539a710] > 2: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1b1b) [0x55d97531913b] > 3: (BlueFS::_flush(BlueFS::FileWriter*, bool)+0x188) [0x55d9753196a8] > 4: (BlueRocksWritableFile::Flush()+0x3c) [0x55d97533523c] > 5: (rocksdb::WritableFileWriter::Flush()+0xaf) [0x55d97570455f] > 6: (rocksdb::WritableFileWriter::Sync(bool)+0x33) [0x55d975704773] > 7: (rocksdb::CompactionJob::FinishCompactionOutputFile(rocksdb::Status const&, rocksdb::CompactionJob::SubcompactionState*, rocksdb::RangeDelAggregator*, CompactionIterationStats*, rocksdb::Slice const*)+0xa1b) [0x55d9757ca7ab] > 8: (rocksdb::CompactionJob::ProcessKeyValueCompaction(rocksdb::CompactionJob::SubcompactionState*)+0xea2) [0x55d9757cd222] > 9: (rocksdb::CompactionJob::Run()+0x1de) [0x55d9757cdcde] > 10: (rocksdb::DBImpl::BackgroundCompaction(bool*, rocksdb::JobContext*, rocksdb::LogBuffer*, void*)+0x9c7) [0x55d975759687] > 11: (rocksdb::DBImpl::BackgroundCallCompaction(void*)+0x283) [0x55d97575cc73] > 12: (rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long)+0x271) [0x55d9757c3e31] > 13: (rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*)+0x3d) [0x55d9757c401d] > 14: (()+0xb5070) [0x7ffacde3f070] > 15: (()+0x7dd5) [0x7fface4b2dd5] > 16: (clone()+0x6d) [0x7ffacd5a3b3d] > NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. > >--- logging levels --- > 0/ 5 none > 0/ 1 lockdep > 0/ 1 context > 1/ 1 crush > 1/ 5 mds > 1/ 5 mds_balancer > 1/ 5 mds_locker > 1/ 5 mds_log > 1/ 5 mds_log_expire > 1/ 5 mds_migrator > 0/ 1 buffer > 0/ 1 timer > 0/ 1 filer > 0/ 1 striper > 0/ 1 objecter > 0/ 5 rados > 0/ 5 rbd > 0/ 5 rbd_mirror > 0/ 5 rbd_replay > 0/ 5 journaler > 0/ 5 objectcacher > 0/ 5 client > 1/ 5 osd > 0/ 5 optracker > 0/ 5 objclass > 1/ 3 filestore > 1/ 3 journal > 0/ 5 ms > 1/ 5 mon > 0/10 monc > 1/ 5 paxos > 0/ 5 tp > 1/ 5 auth > 1/ 5 crypto > 1/ 1 finisher > 1/ 1 reserver > 1/ 5 heartbeatmap > 1/ 5 perfcounter > 1/ 5 rgw > 1/10 civetweb > 1/ 5 javaclient > 1/ 5 asok > 1/ 1 throttle > 0/ 0 refs > 1/ 5 xio > 1/ 5 compressor > 1/ 5 bluestore > 1/ 5 bluefs > 1/ 3 bdev > 1/ 5 kstore > 4/ 5 rocksdb > 4/ 5 leveldb > 4/ 5 memdb > 1/ 5 kinetic > 1/ 5 fuse > 1/ 5 mgr > 1/ 5 mgrc > 1/ 5 dpdk > 1/ 5 eventtrace > -2/-2 (syslog threshold) > -1/-1 (stderr threshold) > max_recent 10000 > max_new 1000 > log_file /var/log/ceph/ceph-osd.10.log >--- end dump of recent events --- >2018-07-11 07:52:40.653690 7ffabc3f1700 -1 *** Caught signal (Aborted) ** > in thread 7ffabc3f1700 thread_name:rocksdb:bg0 > > ceph version 12.2.5-27.el7cp (6b7d1a8cf2e2376fd0a12424947c8c97aa1c0890) luminous (stable) > 1: (()+0xa42d81) [0x55d97535bd81] > 2: (()+0xf680) [0x7fface4ba680] > 3: (gsignal()+0x37) [0x7ffacd4db207] > 4: (abort()+0x148) [0x7ffacd4dc8f8] > 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x284) [0x55d97539a884] > 6: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1b1b) [0x55d97531913b] > 7: (BlueFS::_flush(BlueFS::FileWriter*, bool)+0x188) [0x55d9753196a8] > 8: (BlueRocksWritableFile::Flush()+0x3c) [0x55d97533523c] > 9: (rocksdb::WritableFileWriter::Flush()+0xaf) [0x55d97570455f] > 10: (rocksdb::WritableFileWriter::Sync(bool)+0x33) [0x55d975704773] > 11: (rocksdb::CompactionJob::FinishCompactionOutputFile(rocksdb::Status const&, rocksdb::CompactionJob::SubcompactionState*, rocksdb::RangeDelAggregator*, CompactionIterationStats*, rocksdb::Slice const*)+0xa1b) [0x55d9757ca7ab] > 12: (rocksdb::CompactionJob::ProcessKeyValueCompaction(rocksdb::CompactionJob::SubcompactionState*)+0xea2) [0x55d9757cd222] > 13: (rocksdb::CompactionJob::Run()+0x1de) [0x55d9757cdcde] > 14: (rocksdb::DBImpl::BackgroundCompaction(bool*, rocksdb::JobContext*, rocksdb::LogBuffer*, void*)+0x9c7) [0x55d975759687] > 15: (rocksdb::DBImpl::BackgroundCallCompaction(void*)+0x283) [0x55d97575cc73] > 16: (rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long)+0x271) [0x55d9757c3e31] > 17: (rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*)+0x3d) [0x55d9757c401d] > 18: (()+0xb5070) [0x7ffacde3f070] > 19: (()+0x7dd5) [0x7fface4b2dd5] > 20: (clone()+0x6d) [0x7ffacd5a3b3d] > NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. > >--- begin dump of recent events --- > 0> 2018-07-11 07:52:40.653690 7ffabc3f1700 -1 *** Caught signal (Aborted) ** > in thread 7ffabc3f1700 thread_name:rocksdb:bg0 > > ceph version 12.2.5-27.el7cp (6b7d1a8cf2e2376fd0a12424947c8c97aa1c0890) luminous (stable) > 1: (()+0xa42d81) [0x55d97535bd81] > 2: (()+0xf680) [0x7fface4ba680] > 3: (gsignal()+0x37) [0x7ffacd4db207] > 4: (abort()+0x148) [0x7ffacd4dc8f8] > 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x284) [0x55d97539a884] > 6: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1b1b) [0x55d97531913b] > 7: (BlueFS::_flush(BlueFS::FileWriter*, bool)+0x188) [0x55d9753196a8] > 8: (BlueRocksWritableFile::Flush()+0x3c) [0x55d97533523c] > 9: (rocksdb::WritableFileWriter::Flush()+0xaf) [0x55d97570455f] > 10: (rocksdb::WritableFileWriter::Sync(bool)+0x33) [0x55d975704773] > 11: (rocksdb::CompactionJob::FinishCompactionOutputFile(rocksdb::Status const&, rocksdb::CompactionJob::SubcompactionState*, rocksdb::RangeDelAggregator*, CompactionIterationStats*, rocksdb::Slice const*)+0xa1b) [0x55d9757ca7ab] > 12: (rocksdb::CompactionJob::ProcessKeyValueCompaction(rocksdb::CompactionJob::SubcompactionState*)+0xea2) [0x55d9757cd222] > 13: (rocksdb::CompactionJob::Run()+0x1de) [0x55d9757cdcde] > 14: (rocksdb::DBImpl::BackgroundCompaction(bool*, rocksdb::JobContext*, rocksdb::LogBuffer*, void*)+0x9c7) [0x55d975759687] > 15: (rocksdb::DBImpl::BackgroundCallCompaction(void*)+0x283) [0x55d97575cc73] > 16: (rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long)+0x271) [0x55d9757c3e31] > 17: (rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*)+0x3d) [0x55d9757c401d] > 18: (()+0xb5070) [0x7ffacde3f070] > 19: (()+0x7dd5) [0x7fface4b2dd5] > 20: (clone()+0x6d) [0x7ffacd5a3b3d] > NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. > >--- logging levels --- > 0/ 5 none > 0/ 1 lockdep > 0/ 1 context > 1/ 1 crush > 1/ 5 mds > 1/ 5 mds_balancer > 1/ 5 mds_locker > 1/ 5 mds_log > 1/ 5 mds_log_expire > 1/ 5 mds_migrator > 0/ 1 buffer > 0/ 1 timer > 0/ 1 filer > 0/ 1 striper > 0/ 1 objecter > 0/ 5 rados > 0/ 5 rbd > 0/ 5 rbd_mirror > 0/ 5 rbd_replay > 0/ 5 journaler > 0/ 5 objectcacher > 0/ 5 client > 1/ 5 osd > 0/ 5 optracker > 0/ 5 objclass > 1/ 3 filestore > 1/ 3 journal > 0/ 5 ms > 1/ 5 mon > 0/10 monc > 1/ 5 paxos > 0/ 5 tp > 1/ 5 auth > 1/ 5 crypto > 1/ 1 finisher > 1/ 1 reserver > 1/ 5 heartbeatmap > 1/ 5 perfcounter > 1/ 5 rgw > 1/10 civetweb > 1/ 5 javaclient > 1/ 5 asok > 1/ 1 throttle > 0/ 0 refs > 1/ 5 xio > 1/ 5 compressor > 1/ 5 bluestore > 1/ 5 bluefs > 1/ 3 bdev > 1/ 5 kstore > 4/ 5 rocksdb > 4/ 5 leveldb > 4/ 5 memdb > 1/ 5 kinetic > 1/ 5 fuse > 1/ 5 mgr > 1/ 5 mgrc > 1/ 5 dpdk > 1/ 5 eventtrace > -2/-2 (syslog threshold) > -1/-1 (stderr threshold) > max_recent 10000 > max_new 1000 > log_file /var/log/ceph/ceph-osd.10.log >--- end dump of recent events --- >2018-07-11 07:53:14.879809 7f493724cd80 0 set uid:gid to 167:167 (ceph:ceph) >2018-07-11 07:53:14.879845 7f493724cd80 0 ceph version 12.2.5-27.el7cp (6b7d1a8cf2e2376fd0a12424947c8c97aa1c0890) luminous (stable), process (unknown), pid 14173 >2018-07-11 07:53:14.886827 7f493724cd80 0 pidfile_write: ignore empty --pid-file >2018-07-11 07:53:14.924179 7f493724cd80 0 load: jerasure load: lrc load: isa >2018-07-11 07:53:14.924288 7f493724cd80 1 bdev create path /var/lib/ceph/osd/ceph-10/block type kernel >2018-07-11 07:53:14.924300 7f493724cd80 1 bdev(0x55f64c8ace00 /var/lib/ceph/osd/ceph-10/block) open path /var/lib/ceph/osd/ceph-10/block >2018-07-11 07:53:14.924813 7f493724cd80 1 bdev(0x55f64c8ace00 /var/lib/ceph/osd/ceph-10/block) open size 1000098959360 (0xe8da8b1000, 931 GB) block_size 4096 (4096 B) rotational >2018-07-11 07:53:14.925055 7f493724cd80 1 bluestore(/var/lib/ceph/osd/ceph-10) _set_cache_sizes max 0.5 < ratio 0.99 >2018-07-11 07:53:14.925089 7f493724cd80 1 bluestore(/var/lib/ceph/osd/ceph-10) _set_cache_sizes cache_size 1073741824 meta 0.5 kv 0.5 data 0 >2018-07-11 07:53:14.925097 7f493724cd80 1 bdev(0x55f64c8ace00 /var/lib/ceph/osd/ceph-10/block) close >2018-07-11 07:53:15.180818 7f493724cd80 1 bluestore(/var/lib/ceph/osd/ceph-10) _mount path /var/lib/ceph/osd/ceph-10 >2018-07-11 07:53:15.181094 7f493724cd80 1 bdev create path /var/lib/ceph/osd/ceph-10/block type kernel >2018-07-11 07:53:15.181102 7f493724cd80 1 bdev(0x55f64c8ada00 /var/lib/ceph/osd/ceph-10/block) open path /var/lib/ceph/osd/ceph-10/block >2018-07-11 07:53:15.181321 7f493724cd80 1 bdev(0x55f64c8ada00 /var/lib/ceph/osd/ceph-10/block) open size 1000098959360 (0xe8da8b1000, 931 GB) block_size 4096 (4096 B) rotational >2018-07-11 07:53:15.181539 7f493724cd80 1 bluestore(/var/lib/ceph/osd/ceph-10) _set_cache_sizes max 0.5 < ratio 0.99 >2018-07-11 07:53:15.181551 7f493724cd80 1 bluestore(/var/lib/ceph/osd/ceph-10) _set_cache_sizes cache_size 1073741824 meta 0.5 kv 0.5 data 0 >2018-07-11 07:53:15.181633 7f493724cd80 1 bdev create path /var/lib/ceph/osd/ceph-10/block.db type kernel >2018-07-11 07:53:15.181640 7f493724cd80 1 bdev(0x55f64c8adc00 /var/lib/ceph/osd/ceph-10/block.db) open path /var/lib/ceph/osd/ceph-10/block.db >2018-07-11 07:53:15.181867 7f493724cd80 1 bdev(0x55f64c8adc00 /var/lib/ceph/osd/ceph-10/block.db) open size 1073741824 (0x40000000, 1024 MB) block_size 4096 (4096 B) rotational >2018-07-11 07:53:15.181877 7f493724cd80 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-10/block.db size 1024 MB >2018-07-11 07:53:15.182134 7f493724cd80 1 bdev create path /var/lib/ceph/osd/ceph-10/block type kernel >2018-07-11 07:53:15.182141 7f493724cd80 1 bdev(0x55f64c8ad800 /var/lib/ceph/osd/ceph-10/block) open path /var/lib/ceph/osd/ceph-10/block >2018-07-11 07:53:15.182326 7f493724cd80 1 bdev(0x55f64c8ad800 /var/lib/ceph/osd/ceph-10/block) open size 1000098959360 (0xe8da8b1000, 931 GB) block_size 4096 (4096 B) rotational >2018-07-11 07:53:15.182335 7f493724cd80 1 bluefs add_block_device bdev 2 path /var/lib/ceph/osd/ceph-10/block size 931 GB >2018-07-11 07:53:15.182374 7f493724cd80 1 bdev create path /var/lib/ceph/osd/ceph-10/block.wal type kernel >2018-07-11 07:53:15.182380 7f493724cd80 1 bdev(0x55f64c8ac200 /var/lib/ceph/osd/ceph-10/block.wal) open path /var/lib/ceph/osd/ceph-10/block.wal >2018-07-11 07:53:15.182547 7f493724cd80 1 bdev(0x55f64c8ac200 /var/lib/ceph/osd/ceph-10/block.wal) open size 603979776 (0x24000000, 576 MB) block_size 4096 (4096 B) rotational >2018-07-11 07:53:15.182555 7f493724cd80 1 bluefs add_block_device bdev 0 path /var/lib/ceph/osd/ceph-10/block.wal size 576 MB >2018-07-11 07:53:15.182784 7f493724cd80 1 bluefs mount >2018-07-11 07:53:15.305225 7f493724cd80 0 set rocksdb option compaction_readahead_size = 2097152 >2018-07-11 07:53:15.305242 7f493724cd80 0 set rocksdb option compression = kNoCompression >2018-07-11 07:53:15.305254 7f493724cd80 0 set rocksdb option max_write_buffer_number = 4 >2018-07-11 07:53:15.305259 7f493724cd80 0 set rocksdb option min_write_buffer_number_to_merge = 1 >2018-07-11 07:53:15.305263 7f493724cd80 0 set rocksdb option recycle_log_file_num = 4 >2018-07-11 07:53:15.305266 7f493724cd80 0 set rocksdb option writable_file_max_buffer_size = 0 >2018-07-11 07:53:15.305270 7f493724cd80 0 set rocksdb option write_buffer_size = 268435456 >2018-07-11 07:53:15.305297 7f493724cd80 0 set rocksdb option compaction_readahead_size = 2097152 >2018-07-11 07:53:15.305301 7f493724cd80 0 set rocksdb option compression = kNoCompression >2018-07-11 07:53:15.305305 7f493724cd80 0 set rocksdb option max_write_buffer_number = 4 >2018-07-11 07:53:15.305308 7f493724cd80 0 set rocksdb option min_write_buffer_number_to_merge = 1 >2018-07-11 07:53:15.305312 7f493724cd80 0 set rocksdb option recycle_log_file_num = 4 >2018-07-11 07:53:15.305315 7f493724cd80 0 set rocksdb option writable_file_max_buffer_size = 0 >2018-07-11 07:53:15.305319 7f493724cd80 0 set rocksdb option write_buffer_size = 268435456 >2018-07-11 07:53:15.305475 7f493724cd80 4 rocksdb: RocksDB version: 5.4.0 > >2018-07-11 07:53:15.305486 7f493724cd80 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ >2018-07-11 07:53:15.305488 7f493724cd80 4 rocksdb: Compile date Jun 28 2018 >2018-07-11 07:53:15.305491 7f493724cd80 4 rocksdb: DB SUMMARY > >2018-07-11 07:53:15.305521 7f493724cd80 4 rocksdb: CURRENT file: CURRENT > >2018-07-11 07:53:15.305531 7f493724cd80 4 rocksdb: IDENTITY file: IDENTITY > >2018-07-11 07:53:15.305539 7f493724cd80 4 rocksdb: MANIFEST file: MANIFEST-036595 size: 379986 Bytes > >2018-07-11 07:53:15.305544 7f493724cd80 4 rocksdb: SST files in db dir, Total Num: 7, files: 036405.sst 036503.sst 036564.sst 036570.sst 036571.sst 036576.sst 036577.sst > >2018-07-11 07:53:15.308800 7f493724cd80 4 rocksdb: SST files in db.slow dir, Total Num: 4983, files: 000549.sst 000550.sst 000565.sst 000566.sst 000605.sst 000617.sst 000620.sst 000622.sst 000634.sst > >2018-07-11 07:53:15.308883 7f493724cd80 4 rocksdb: Write Ahead Log file in db.wal: 036596.log size: 523413 ; > >2018-07-11 07:53:15.308889 7f493724cd80 4 rocksdb: Options.error_if_exists: 0 >2018-07-11 07:53:15.308891 7f493724cd80 4 rocksdb: Options.create_if_missing: 0 >2018-07-11 07:53:15.308892 7f493724cd80 4 rocksdb: Options.paranoid_checks: 1 >2018-07-11 07:53:15.308892 7f493724cd80 4 rocksdb: Options.env: 0x55f64c8a5060 >2018-07-11 07:53:15.308894 7f493724cd80 4 rocksdb: Options.info_log: 0x55f64cadae00 >2018-07-11 07:53:15.308894 7f493724cd80 4 rocksdb: Options.max_open_files: -1 >2018-07-11 07:53:15.308895 7f493724cd80 4 rocksdb: Options.max_file_opening_threads: 16 >2018-07-11 07:53:15.308896 7f493724cd80 4 rocksdb: Options.use_fsync: 0 >2018-07-11 07:53:15.308897 7f493724cd80 4 rocksdb: Options.max_log_file_size: 0 >2018-07-11 07:53:15.308898 7f493724cd80 4 rocksdb: Options.max_manifest_file_size: 18446744073709551615 >2018-07-11 07:53:15.308899 7f493724cd80 4 rocksdb: Options.log_file_time_to_roll: 0 >2018-07-11 07:53:15.308899 7f493724cd80 4 rocksdb: Options.keep_log_file_num: 1000 >2018-07-11 07:53:15.308900 7f493724cd80 4 rocksdb: Options.recycle_log_file_num: 4 >2018-07-11 07:53:15.308901 7f493724cd80 4 rocksdb: Options.allow_fallocate: 1 >2018-07-11 07:53:15.308902 7f493724cd80 4 rocksdb: Options.allow_mmap_reads: 0 >2018-07-11 07:53:15.308902 7f493724cd80 4 rocksdb: Options.allow_mmap_writes: 0 >2018-07-11 07:53:15.308903 7f493724cd80 4 rocksdb: Options.use_direct_reads: 0 >2018-07-11 07:53:15.308904 7f493724cd80 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 >2018-07-11 07:53:15.308905 7f493724cd80 4 rocksdb: Options.create_missing_column_families: 0 >2018-07-11 07:53:15.308905 7f493724cd80 4 rocksdb: Options.db_log_dir: >2018-07-11 07:53:15.308906 7f493724cd80 4 rocksdb: Options.wal_dir: db.wal >2018-07-11 07:53:15.308907 7f493724cd80 4 rocksdb: Options.table_cache_numshardbits: 6 >2018-07-11 07:53:15.308908 7f493724cd80 4 rocksdb: Options.max_subcompactions: 1 >2018-07-11 07:53:15.308909 7f493724cd80 4 rocksdb: Options.max_background_flushes: 1 >2018-07-11 07:53:15.308913 7f493724cd80 4 rocksdb: Options.WAL_ttl_seconds: 0 >2018-07-11 07:53:15.308914 7f493724cd80 4 rocksdb: Options.WAL_size_limit_MB: 0 >2018-07-11 07:53:15.308915 7f493724cd80 4 rocksdb: Options.manifest_preallocation_size: 4194304 >2018-07-11 07:53:15.308916 7f493724cd80 4 rocksdb: Options.is_fd_close_on_exec: 1 >2018-07-11 07:53:15.308917 7f493724cd80 4 rocksdb: Options.advise_random_on_open: 1 >2018-07-11 07:53:15.308917 7f493724cd80 4 rocksdb: Options.db_write_buffer_size: 0 >2018-07-11 07:53:15.308918 7f493724cd80 4 rocksdb: Options.access_hint_on_compaction_start: 1 >2018-07-11 07:53:15.308919 7f493724cd80 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 1 >2018-07-11 07:53:15.308919 7f493724cd80 4 rocksdb: Options.compaction_readahead_size: 2097152 >2018-07-11 07:53:15.308920 7f493724cd80 4 rocksdb: Options.random_access_max_buffer_size: 1048576 >2018-07-11 07:53:15.308921 7f493724cd80 4 rocksdb: Options.writable_file_max_buffer_size: 0 >2018-07-11 07:53:15.308922 7f493724cd80 4 rocksdb: Options.use_adaptive_mutex: 0 >2018-07-11 07:53:15.308922 7f493724cd80 4 rocksdb: Options.rate_limiter: (nil) >2018-07-11 07:53:15.308923 7f493724cd80 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 >2018-07-11 07:53:15.308924 7f493724cd80 4 rocksdb: Options.bytes_per_sync: 0 >2018-07-11 07:53:15.308925 7f493724cd80 4 rocksdb: Options.wal_bytes_per_sync: 0 >2018-07-11 07:53:15.308926 7f493724cd80 4 rocksdb: Options.wal_recovery_mode: 0 >2018-07-11 07:53:15.308926 7f493724cd80 4 rocksdb: Options.enable_thread_tracking: 0 >2018-07-11 07:53:15.308927 7f493724cd80 4 rocksdb: Options.allow_concurrent_memtable_write: 1 >2018-07-11 07:53:15.308928 7f493724cd80 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 >2018-07-11 07:53:15.308928 7f493724cd80 4 rocksdb: Options.write_thread_max_yield_usec: 100 >2018-07-11 07:53:15.308929 7f493724cd80 4 rocksdb: Options.write_thread_slow_yield_usec: 3 >2018-07-11 07:53:15.308930 7f493724cd80 4 rocksdb: Options.row_cache: None >2018-07-11 07:53:15.308931 7f493724cd80 4 rocksdb: Options.wal_filter: None >2018-07-11 07:53:15.308931 7f493724cd80 4 rocksdb: Options.avoid_flush_during_recovery: 0 >2018-07-11 07:53:15.308936 7f493724cd80 4 rocksdb: Options.base_background_compactions: 1 >2018-07-11 07:53:15.308937 7f493724cd80 4 rocksdb: Options.max_background_compactions: 1 >2018-07-11 07:53:15.308937 7f493724cd80 4 rocksdb: Options.avoid_flush_during_shutdown: 0 >2018-07-11 07:53:15.308938 7f493724cd80 4 rocksdb: Options.delayed_write_rate : 16777216 >2018-07-11 07:53:15.308939 7f493724cd80 4 rocksdb: Options.max_total_wal_size: 0 >2018-07-11 07:53:15.308940 7f493724cd80 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 >2018-07-11 07:53:15.308941 7f493724cd80 4 rocksdb: Options.stats_dump_period_sec: 600 >2018-07-11 07:53:15.308941 7f493724cd80 4 rocksdb: Compression algorithms supported: >2018-07-11 07:53:15.308944 7f493724cd80 4 rocksdb: Snappy supported: 0 >2018-07-11 07:53:15.308945 7f493724cd80 4 rocksdb: Zlib supported: 0 >2018-07-11 07:53:15.308946 7f493724cd80 4 rocksdb: Bzip supported: 0 >2018-07-11 07:53:15.308947 7f493724cd80 4 rocksdb: LZ4 supported: 0 >2018-07-11 07:53:15.308947 7f493724cd80 4 rocksdb: ZSTD supported: 0 >2018-07-11 07:53:15.308948 7f493724cd80 4 rocksdb: Fast CRC32 supported: 1 >2018-07-11 07:53:15.313379 7f493724cd80 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/version_set.cc:2609] Recovering from manifest file: MANIFEST-036595 > >2018-07-11 07:53:15.313429 7f493724cd80 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:407] --------------- Options for column family [default]: > >2018-07-11 07:53:15.313433 7f493724cd80 4 rocksdb: Options.comparator: leveldb.BytewiseComparator >2018-07-11 07:53:15.313443 7f493724cd80 4 rocksdb: Options.merge_operator: .T:int64_array.b:bitwise_xor >2018-07-11 07:53:15.313444 7f493724cd80 4 rocksdb: Options.compaction_filter: None >2018-07-11 07:53:15.313445 7f493724cd80 4 rocksdb: Options.compaction_filter_factory: None >2018-07-11 07:53:15.313446 7f493724cd80 4 rocksdb: Options.memtable_factory: SkipListFactory >2018-07-11 07:53:15.313447 7f493724cd80 4 rocksdb: Options.table_factory: BlockBasedTable >2018-07-11 07:53:15.313467 7f493724cd80 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55f64c544248) > cache_index_and_filter_blocks: 1 > cache_index_and_filter_blocks_with_high_priority: 1 > pin_l0_filter_and_index_blocks_in_cache: 1 > index_type: 0 > hash_index_allow_collision: 1 > checksum: 1 > no_block_cache: 0 > block_cache: 0x55f64c9a82b8 > block_cache_name: LRUCache > block_cache_options: > capacity : 536870912 > num_shard_bits : 4 > strict_capacity_limit : 0 > high_pri_pool_ratio: 0.000 > block_cache_compressed: (nil) > persistent_cache: (nil) > block_size: 4096 > block_size_deviation: 10 > block_restart_interval: 16 > index_block_restart_interval: 1 > filter_policy: rocksdb.BuiltinBloomFilter > whole_key_filtering: 1 > format_version: 2 > >2018-07-11 07:53:15.313474 7f493724cd80 4 rocksdb: Options.write_buffer_size: 268435456 >2018-07-11 07:53:15.313476 7f493724cd80 4 rocksdb: Options.max_write_buffer_number: 4 >2018-07-11 07:53:15.313481 7f493724cd80 4 rocksdb: Options.compression: NoCompression >2018-07-11 07:53:15.313483 7f493724cd80 4 rocksdb: Options.bottommost_compression: Disabled >2018-07-11 07:53:15.313484 7f493724cd80 4 rocksdb: Options.prefix_extractor: nullptr >2018-07-11 07:53:15.313485 7f493724cd80 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr >2018-07-11 07:53:15.313486 7f493724cd80 4 rocksdb: Options.num_levels: 7 >2018-07-11 07:53:15.313487 7f493724cd80 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 >2018-07-11 07:53:15.313487 7f493724cd80 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 >2018-07-11 07:53:15.313488 7f493724cd80 4 rocksdb: Options.compression_opts.window_bits: -14 >2018-07-11 07:53:15.313489 7f493724cd80 4 rocksdb: Options.compression_opts.level: -1 >2018-07-11 07:53:15.313490 7f493724cd80 4 rocksdb: Options.compression_opts.strategy: 0 >2018-07-11 07:53:15.313491 7f493724cd80 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 >2018-07-11 07:53:15.313491 7f493724cd80 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 >2018-07-11 07:53:15.313492 7f493724cd80 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 >2018-07-11 07:53:15.313493 7f493724cd80 4 rocksdb: Options.level0_stop_writes_trigger: 36 >2018-07-11 07:53:15.313494 7f493724cd80 4 rocksdb: Options.target_file_size_base: 67108864 >2018-07-11 07:53:15.313495 7f493724cd80 4 rocksdb: Options.target_file_size_multiplier: 1 >2018-07-11 07:53:15.313495 7f493724cd80 4 rocksdb: Options.max_bytes_for_level_base: 268435456 >2018-07-11 07:53:15.313496 7f493724cd80 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0 >2018-07-11 07:53:15.313497 7f493724cd80 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 >2018-07-11 07:53:15.313506 7f493724cd80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 >2018-07-11 07:53:15.313508 7f493724cd80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 >2018-07-11 07:53:15.313509 7f493724cd80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 >2018-07-11 07:53:15.313509 7f493724cd80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 >2018-07-11 07:53:15.313510 7f493724cd80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 >2018-07-11 07:53:15.313511 7f493724cd80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 >2018-07-11 07:53:15.313512 7f493724cd80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 >2018-07-11 07:53:15.313513 7f493724cd80 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 >2018-07-11 07:53:15.313513 7f493724cd80 4 rocksdb: Options.max_compaction_bytes: 1677721600 >2018-07-11 07:53:15.313514 7f493724cd80 4 rocksdb: Options.arena_block_size: 33554432 >2018-07-11 07:53:15.313515 7f493724cd80 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 >2018-07-11 07:53:15.313516 7f493724cd80 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 >2018-07-11 07:53:15.313517 7f493724cd80 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 >2018-07-11 07:53:15.313518 7f493724cd80 4 rocksdb: Options.disable_auto_compactions: 0 >2018-07-11 07:53:15.313524 7f493724cd80 4 rocksdb: Options.compaction_style: kCompactionStyleLevel >2018-07-11 07:53:15.313525 7f493724cd80 4 rocksdb: Options.compaction_pri: kByCompensatedSize >2018-07-11 07:53:15.313526 7f493724cd80 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 >2018-07-11 07:53:15.313527 7f493724cd80 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 >2018-07-11 07:53:15.313528 7f493724cd80 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 >2018-07-11 07:53:15.313528 7f493724cd80 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 >2018-07-11 07:53:15.313529 7f493724cd80 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 >2018-07-11 07:53:15.313530 7f493724cd80 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 >2018-07-11 07:53:15.313531 7f493724cd80 4 rocksdb: Options.table_properties_collectors: >2018-07-11 07:53:15.313532 7f493724cd80 4 rocksdb: Options.inplace_update_support: 0 >2018-07-11 07:53:15.313533 7f493724cd80 4 rocksdb: Options.inplace_update_num_locks: 10000 >2018-07-11 07:53:15.313533 7f493724cd80 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 >2018-07-11 07:53:15.313535 7f493724cd80 4 rocksdb: Options.memtable_huge_page_size: 0 >2018-07-11 07:53:15.313536 7f493724cd80 4 rocksdb: Options.bloom_locality: 0 >2018-07-11 07:53:15.313536 7f493724cd80 4 rocksdb: Options.max_successive_merges: 0 >2018-07-11 07:53:15.313537 7f493724cd80 4 rocksdb: Options.optimize_filters_for_hits: 0 >2018-07-11 07:53:15.313538 7f493724cd80 4 rocksdb: Options.paranoid_file_checks: 0 >2018-07-11 07:53:15.313538 7f493724cd80 4 rocksdb: Options.force_consistency_checks: 0 >2018-07-11 07:53:15.313539 7f493724cd80 4 rocksdb: Options.report_bg_io_stats: 0 >2018-07-11 07:54:23.698801 7f493724cd80 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/version_set.cc:2859] Recovered from manifest file:db/MANIFEST-036595 succeeded,manifest_file_number is 36595, next_file_number is 36813, last_sequence is 12881316064, log_number is 0,prev_log_number is 0,max_column_family is 0 > >2018-07-11 07:54:23.698819 7f493724cd80 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/version_set.cc:2867] Column family [default] (ID 0), log number is 36493 > >2018-07-11 07:54:23.706151 7f493724cd80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295663706141, "job": 1, "event": "recovery_started", "log_files": [36596]} >2018-07-11 07:54:23.706159 7f493724cd80 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_open.cc:482] Recovering log #36596 mode 0 >2018-07-11 07:54:23.740639 7f493724cd80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295663740622, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 36813, "file_size": 224004, "table_properties": {"data_size": 220942, "index_size": 720, "filter_size": 1367, "raw_key_size": 11298, "raw_average_key_size": 31, "raw_value_size": 211572, "raw_average_value_size": 590, "num_data_blocks": 24, "num_entries": 358, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "4", "kMergeOperands": "7"}} >2018-07-11 07:54:23.740868 7f493724cd80 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/version_set.cc:2395] Creating manifest 36814 > >2018-07-11 07:54:23.773012 7f493724cd80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295663773008, "job": 1, "event": "recovery_finished"} >2018-07-11 07:54:23.773058 7f493724cd80 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 49121545030 >2018-07-11 07:54:23.775331 7f493724cd80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295663775327, "job": 2, "event": "table_file_deletion", "file_number": 36815} >2018-07-11 07:54:23.775346 7f493724cd80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295663775343, "job": 2, "event": "table_file_deletion", "file_number": 36814} >2018-07-11 07:54:23.775359 7f493724cd80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295663775357, "job": 2, "event": "table_file_deletion", "file_number": 36812} >2018-07-11 07:54:23.790577 7f493724cd80 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_open.cc:1063] DB pointer 0x55f64cc44000 >2018-07-11 07:54:23.790678 7f493724cd80 1 bluestore(/var/lib/ceph/osd/ceph-10) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152 >2018-07-11 07:54:23.791138 7f4922851700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS ------- >2018-07-11 07:54:23.791147 7f4922851700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] >** DB Stats ** >Uptime(secs): 68.5 total, 68.5 interval >Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s >Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s >Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent >Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s >Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s >Interval stall: 00:00:0.000 H:M:S, 0.0 percent > >** Compaction Stats [default] ** >Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop >---------------------------------------------------------------------------------------------------------------------------------------------------------- > L0 1/0 218.75 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 10.9 0 1 0.020 0 0 > L1 7/0 240.38 MB 1.1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 > L2 72/0 4.28 GB 2.3 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 > L3 470/0 26.18 GB 1.1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 > L4 3641/0 250.00 GB 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 > L5 796/0 55.24 GB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 > Sum 4987/0 335.94 GB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 10.9 0 1 0.020 0 0 > Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 10.9 0 1 0.020 0 0 >Uptime(secs): 68.5 total, 68.5 interval >Flush(GB): cumulative 0.000, interval 0.000 >AddFile(GB): cumulative 0.000, interval 0.000 >AddFile(Total Files): cumulative 0, interval 0 >AddFile(L0 Files): cumulative 0, interval 0 >AddFile(Keys): cumulative 0, interval 0 >Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds >Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds >Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count > >** File Read Latency Histogram By Level [default] ** > >** Compaction Stats [default] ** >Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop >---------------------------------------------------------------------------------------------------------------------------------------------------------- > L0 1/0 218.75 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 10.9 0 1 0.020 0 0 > L1 7/0 240.38 MB 1.1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 > L2 72/0 4.28 GB 2.3 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 > L3 470/0 26.18 GB 1.1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 > L4 3641/0 250.00 GB 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 > L5 796/0 55.24 GB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 > Sum 4987/0 335.94 GB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 10.9 0 1 0.020 0 0 > Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 >Uptime(secs): 68.5 total, 68.5 interval >Flush(GB): cumulative 0.000, interval 0.000 >AddFile(GB): cumulative 0.000, interval 0.000 >AddFile(Total Files): cumulative 0, interval 0 >AddFile(L0 Files): cumulative 0, interval 0 >AddFile(Keys): cumulative 0, interval 0 >Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds >Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds >Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count > >** File Read Latency Histogram By Level [default] ** > >2018-07-11 07:54:23.791358 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 3] Compacting 1@2 + 8@3 files to L3, score 2.26 >2018-07-11 07:54:23.791382 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 2 Base level 2, inputs: [36612(76MB)], [15174(71MB) 15177(577KB) 15288(70MB) 15289(71MB) 15290(71MB) 15291(71MB) 15292(71MB) 15293(1249B)] > >2018-07-11 07:54:23.791405 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295663791388, "job": 3, "event": "compaction_started", "files_L2": [36612], "files_L3": [15174, 15177, 15288, 15289, 15290, 15291, 15292, 15293], "score": 2.26028, "input_data_size": 527530353} >2018-07-11 07:54:23.832019 7f493724cd80 1 freelist init >2018-07-11 07:54:24.293207 7f493724cd80 1 bluestore(/var/lib/ceph/osd/ceph-10) _open_alloc opening allocation metadata >2018-07-11 07:54:24.633563 7f493724cd80 1 bluestore(/var/lib/ceph/osd/ceph-10) _open_alloc loaded 495 G in 77587 extents >2018-07-11 07:54:25.039342 7f493724cd80 0 _get_class not permitted to load kvs >2018-07-11 07:54:25.039355 7f493724cd80 0 _get_class not permitted to load lua >2018-07-11 07:54:25.042459 7f493724cd80 0 <cls> /builddir/build/BUILD/ceph-12.2.5/src/cls/cephfs/cls_cephfs.cc:197: loading cephfs >2018-07-11 07:54:25.044829 7f493724cd80 0 <cls> /builddir/build/BUILD/ceph-12.2.5/src/cls/hello/cls_hello.cc:296: loading cls_hello >2018-07-11 07:54:25.045818 7f493724cd80 0 _get_class not permitted to load sdk >2018-07-11 07:54:25.046457 7f493724cd80 0 osd.10 501 crush map has features 288514119978713088, adjusting msgr requires for clients >2018-07-11 07:54:25.046470 7f493724cd80 0 osd.10 501 crush map has features 288514394856620032 was 8705, adjusting msgr requires for mons >2018-07-11 07:54:25.046477 7f493724cd80 0 osd.10 501 crush map has features 1009090335235915776, adjusting msgr requires for osds >2018-07-11 07:54:27.216376 7f493724cd80 0 osd.10 501 load_pgs >2018-07-11 07:54:28.284177 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 3] Generated table #36818: 2658304 keys, 74596892 bytes >2018-07-11 07:54:28.284217 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295668284202, "cf_name": "default", "job": 3, "event": "table_file_creation", "file_number": 36818, "file_size": 74596892, "table_properties": {"data_size": 67112830, "index_size": 689658, "filter_size": 6793412, "raw_key_size": 76795453, "raw_average_key_size": 28, "raw_value_size": 31095052, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2658304, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "42832", "kMergeOperands": "0"}} >2018-07-11 07:54:31.817953 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 3] Generated table #36819: 2638237 keys, 74546406 bytes >2018-07-11 07:54:31.818000 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295671817981, "cf_name": "default", "job": 3, "event": "table_file_creation", "file_number": 36819, "file_size": 74546406, "table_properties": {"data_size": 67112112, "index_size": 690129, "filter_size": 6743175, "raw_key_size": 76215749, "raw_average_key_size": 28, "raw_value_size": 31365720, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638237, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:54:32.024201 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 3] Generated table #36820: 19370 keys, 548228 bytes >2018-07-11 07:54:32.024232 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295672024217, "cf_name": "default", "job": 3, "event": "table_file_creation", "file_number": 36820, "file_size": 548228, "table_properties": {"data_size": 492756, "index_size": 4977, "filter_size": 49516, "raw_key_size": 559577, "raw_average_key_size": 28, "raw_value_size": 230287, "raw_average_value_size": 11, "num_data_blocks": 122, "num_entries": 19370, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:54:36.908481 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 3] Generated table #36821: 3672198 keys, 77132968 bytes >2018-07-11 07:54:36.908529 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295676908514, "cf_name": "default", "job": 3, "event": "table_file_creation", "file_number": 36821, "file_size": 77132968, "table_properties": {"data_size": 67110610, "index_size": 689631, "filter_size": 9331734, "raw_key_size": 106085701, "raw_average_key_size": 28, "raw_value_size": 17380103, "raw_average_value_size": 4, "num_data_blocks": 16481, "num_entries": 3672198, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "2210319", "kMergeOperands": "0"}} >2018-07-11 07:54:40.353948 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 3] Generated table #36822: 2638240 keys, 74546477 bytes >2018-07-11 07:54:40.354024 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295680353981, "cf_name": "default", "job": 3, "event": "table_file_creation", "file_number": 36822, "file_size": 74546477, "table_properties": {"data_size": 67112110, "index_size": 690196, "filter_size": 6743181, "raw_key_size": 76215837, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:54:40.425691 7f493724cd80 0 osd.10 501 load_pgs opened 71 pgs >2018-07-11 07:54:40.425716 7f493724cd80 0 osd.10 501 using weightedpriority op queue with priority op cut off at 64. >2018-07-11 07:54:40.426656 7f493724cd80 -1 osd.10 501 log_to_monitors {default=true} >2018-07-11 07:54:40.432492 7f491c044700 1 osd.10 pg_epoch: 501 pg[5.35( v 166'67898 (166'66348,166'67898] local-lis/les=498/499 n=67898 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/470) [0,11,10] r=2 lpr=499 crt=166'67898 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:40.432493 7f491c845700 1 osd.10 pg_epoch: 501 pg[5.37( v 166'68122 (166'66622,166'68122] local-lis/les=498/500 n=68122 ec=71/71 lis/c 498/498 les/c/f 500/500/0 498/498/487) [4,10,8] r=1 lpr=500 crt=166'68122 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:40.432634 7f491c044700 1 osd.10 pg_epoch: 501 pg[5.c( v 166'68190 (166'66593,166'68190] local-lis/les=498/500 n=68190 ec=71/71 lis/c 498/498 les/c/f 500/500/0 498/498/470) [1,10,4] r=1 lpr=500 crt=166'68190 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:40.432651 7f491c845700 1 osd.10 pg_epoch: 501 pg[5.33( v 166'68409 (166'66894,166'68409] local-lis/les=498/500 n=68409 ec=71/71 lis/c 498/498 les/c/f 500/500/0 498/498/465) [2,10,8] r=1 lpr=500 crt=166'68409 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:40.432738 7f491c044700 1 osd.10 pg_epoch: 501 pg[16.e( empty local-lis/les=498/500 n=0 ec=442/442 lis/c 498/498 les/c/f 500/500/0 498/498/465) [2,4,10] r=2 lpr=500 crt=0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:40.432794 7f491c845700 1 osd.10 pg_epoch: 501 pg[6.14( v 461'122884 (430'121291,461'122884] local-lis/les=498/499 n=25983 ec=76/76 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,4,3] r=0 lpr=499 crt=461'122884 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 07:54:40.432836 7f491c044700 1 osd.10 pg_epoch: 501 pg[5.24( v 166'68298 (166'66798,166'68298] local-lis/les=498/499 n=68298 ec=71/71 lis/c 498/498 les/c/f 499/500/0 498/498/498) [10,11,1] r=0 lpr=500 crt=166'68298 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 07:54:40.432968 7f491c845700 1 osd.10 pg_epoch: 501 pg[5.14( v 166'68424 (166'66924,166'68424] local-lis/les=498/500 n=68424 ec=71/71 lis/c 498/498 les/c/f 500/500/0 498/498/468) [7,10,2] r=1 lpr=500 crt=166'68424 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:40.432981 7f491c044700 1 osd.10 pg_epoch: 501 pg[13.ds1( v 377'10591 (355'1551,377'10591] local-lis/les=498/499 n=2744 ec=345/345 lis/c 498/498 les/c/f 499/500/0 498/498/470) [0,10,1,8,11,6]p0(0) r=1 lpr=500 crt=377'10591 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:40.433069 7f491c845700 1 osd.10 pg_epoch: 501 pg[6.17( v 430'108120 (430'106620,430'108120] local-lis/les=498/499 n=26407 ec=76/76 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,5] r=0 lpr=499 crt=430'108120 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 07:54:40.433099 7f491c044700 1 osd.10 pg_epoch: 501 pg[13.cs2( v 377'10679 (371'2117,377'10679] local-lis/les=498/500 n=2785 ec=345/345 lis/c 498/498 les/c/f 500/500/0 498/498/470) [1,11,10,4,8,6]p1(0) r=2 lpr=500 crt=377'10679 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:40.433200 7f491c845700 1 osd.10 pg_epoch: 501 pg[6.21( v 461'128646 (430'127048,461'128646] local-lis/les=498/499 n=25982 ec=76/76 lis/c 498/498 les/c/f 499/499/0 498/498/470) [0,2,10] r=2 lpr=499 crt=461'128646 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:40.433203 7f491c044700 1 osd.10 pg_epoch: 501 pg[16.10( empty local-lis/les=498/500 n=0 ec=442/442 lis/c 498/498 les/c/f 500/500/0 498/498/465) [2,10,7] r=1 lpr=500 crt=0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:40.433324 7f491c044700 1 osd.10 pg_epoch: 501 pg[10.a( v 386'2032 (203'500,386'2032] local-lis/les=498/499 n=516 ec=193/193 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,5] r=0 lpr=499 crt=386'2032 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 07:54:40.433336 7f491c845700 1 osd.10 pg_epoch: 501 pg[10.9( v 390'615 (0'0,390'615] local-lis/les=498/499 n=478 ec=193/193 lis/c 498/498 les/c/f 499/499/0 498/498/470) [0,2,10] r=2 lpr=499 crt=390'615 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:40.433436 7f491c845700 1 osd.10 pg_epoch: 501 pg[9.a( v 494'572 (0'0,494'572] local-lis/les=498/499 n=232 ec=190/190 lis/c 498/498 les/c/f 499/499/0 498/498/470) [0,10,11] r=1 lpr=499 crt=494'572 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:40.433463 7f491c044700 1 osd.10 pg_epoch: 501 pg[5.23( v 166'68497 (166'66949,166'68497] local-lis/les=498/500 n=68497 ec=71/71 lis/c 498/498 les/c/f 500/500/0 498/498/472) [8,3,10] r=2 lpr=500 crt=166'68497 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:40.433531 7f491c845700 1 osd.10 pg_epoch: 501 pg[9.7( v 486'503 (0'0,486'503] local-lis/les=498/500 n=248 ec=190/190 lis/c 498/498 les/c/f 500/500/0 498/498/468) [7,10,2] r=1 lpr=500 crt=486'503 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:40.433568 7f491c044700 1 osd.10 pg_epoch: 501 pg[10.f( v 383'568 (0'0,383'568] local-lis/les=498/500 n=494 ec=193/193 lis/c 498/498 les/c/f 500/500/0 498/498/478) [11,10,2] r=1 lpr=500 crt=383'568 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:40.433636 7f491c845700 1 osd.10 pg_epoch: 501 pg[5.1c( v 383'68804 (166'67302,383'68804] local-lis/les=498/499 n=68803 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,8,2] r=0 lpr=499 crt=383'68804 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 07:54:40.433673 7f491c044700 1 osd.10 pg_epoch: 501 pg[6.3( v 430'104329 (430'102829,430'104329] local-lis/les=498/499 n=25881 ec=76/76 lis/c 498/498 les/c/f 499/499/0 498/498/470) [0,10,11] r=1 lpr=499 crt=430'104329 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:40.433769 7f491c845700 1 osd.10 pg_epoch: 501 pg[6.1f( v 430'114415 (430'112915,430'114415] local-lis/les=498/499 n=26413 ec=76/76 lis/c 498/498 les/c/f 499/499/0 498/498/484) [3,10,0] r=1 lpr=499 crt=430'114415 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:40.433775 7f491c044700 1 osd.10 pg_epoch: 501 pg[6.32( v 430'104474 (430'102974,430'104474] local-lis/les=498/500 n=25649 ec=76/76 lis/c 498/498 les/c/f 500/500/0 498/498/487) [4,2,10] r=2 lpr=500 crt=430'104474 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:40.433860 7f491c845700 1 osd.10 pg_epoch: 501 pg[9.10( v 494'394 (0'0,494'394] local-lis/les=498/500 n=251 ec=190/190 lis/c 498/498 les/c/f 500/500/0 498/498/487) [4,11,10] r=2 lpr=500 crt=494'394 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:40.433870 7f491c044700 1 osd.10 pg_epoch: 501 pg[6.0( v 430'122181 (430'120681,430'122181] local-lis/les=498/499 n=25944 ec=76/76 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,7,2] r=0 lpr=499 crt=430'122181 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 07:54:40.433954 7f491c845700 1 osd.10 pg_epoch: 501 pg[10.13( v 383'839 (0'0,383'839] local-lis/les=498/500 n=523 ec=193/193 lis/c 498/498 les/c/f 500/500/0 498/498/470) [1,10,7] r=1 lpr=500 crt=383'839 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:40.433986 7f491c044700 1 osd.10 pg_epoch: 501 pg[5.10( v 166'68154 (166'66598,166'68154] local-lis/les=498/499 n=68154 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,8,4] r=0 lpr=499 crt=166'68154 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 07:54:40.434066 7f491c845700 1 osd.10 pg_epoch: 501 pg[5.9( v 166'68275 (166'66775,166'68275] local-lis/les=498/499 n=68275 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,8,7] r=0 lpr=499 crt=166'68275 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 07:54:40.434145 7f491c044700 1 osd.10 pg_epoch: 501 pg[11.4s2( v 383'60130 (352'56449,383'60130] local-lis/les=498/499 n=60130 ec=267/267 lis/c 498/498 les/c/f 499/499/0 498/498/470) [0,6,10,8,3,11]p0(0) r=2 lpr=499 crt=383'60130 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:40.434200 7f491c845700 1 osd.10 pg_epoch: 501 pg[6.a( v 430'109231 (430'107731,430'109231] local-lis/les=498/500 n=26267 ec=76/76 lis/c 498/498 les/c/f 500/500/0 498/498/467) [5,10,6] r=1 lpr=500 crt=430'109231 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:40.434276 7f491c044700 1 osd.10 pg_epoch: 501 pg[11.6s0( v 383'59732 (352'56233,383'59732] local-lis/les=498/499 n=59732 ec=267/267 lis/c 498/498 les/c/f 499/500/0 498/498/498) [10,1,11,6,5,8]p10(0) r=0 lpr=500 crt=383'59732 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 07:54:40.434311 7f491c845700 1 osd.10 pg_epoch: 501 pg[12.0( v 383'7619 (383'6093,383'7619] local-lis/les=498/500 n=7619 ec=331/331 lis/c 498/498 les/c/f 500/500/0 498/498/487) [4,10,3] r=1 lpr=500 crt=383'7619 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:40.434408 7f491c845700 1 osd.10 pg_epoch: 501 pg[5.4( v 166'68216 (166'66716,166'68216] local-lis/les=498/500 n=68216 ec=71/71 lis/c 498/498 les/c/f 500/500/0 498/498/468) [7,2,10] r=2 lpr=500 crt=166'68216 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:40.434423 7f491c044700 1 osd.10 pg_epoch: 501 pg[6.13( v 430'99276 (430'97776,430'99276] local-lis/les=498/499 n=26101 ec=76/76 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,7,0] r=0 lpr=499 crt=430'99276 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 07:54:40.434512 7f491c845700 1 osd.10 pg_epoch: 501 pg[6.7( v 430'109641 (430'108141,430'109641] local-lis/les=498/499 n=25825 ec=76/76 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,6,3] r=0 lpr=499 crt=430'109641 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 07:54:40.434550 7f491c044700 1 osd.10 pg_epoch: 501 pg[9.1c( v 494'483 (0'0,494'483] local-lis/les=498/500 n=238 ec=190/190 lis/c 498/498 les/c/f 500/500/0 498/498/470) [1,10,7] r=1 lpr=500 crt=494'483 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:40.434630 7f491c845700 1 osd.10 pg_epoch: 501 pg[5.d( v 166'68209 (166'66709,166'68209] local-lis/les=498/499 n=68209 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,7,0] r=0 lpr=499 crt=166'68209 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 07:54:40.434646 7f491c044700 1 osd.10 pg_epoch: 501 pg[9.1b( v 463'472 (0'0,463'472] local-lis/les=498/499 n=256 ec=190/190 lis/c 498/498 les/c/f 499/500/0 498/498/498) [10,1,5] r=0 lpr=500 crt=463'472 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 07:54:40.434746 7f491c845700 1 osd.10 pg_epoch: 501 pg[6.e( v 430'116090 (430'114517,430'116090] local-lis/les=498/500 n=25967 ec=76/76 lis/c 498/498 les/c/f 500/500/0 498/498/487) [4,1,10] r=2 lpr=500 crt=430'116090 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:40.434801 7f491c044700 1 osd.10 pg_epoch: 501 pg[6.d( v 461'110631 (430'109050,461'110631] local-lis/les=498/499 n=25732 ec=76/76 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,7,2] r=0 lpr=499 crt=461'110631 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 07:54:40.434871 7f491c845700 1 osd.10 pg_epoch: 501 pg[11.8s3( v 383'59629 (352'55946,383'59629] local-lis/les=498/500 n=59629 ec=267/267 lis/c 498/498 les/c/f 500/500/0 498/498/468) [7,1,6,10,5,8]p7(0) r=3 lpr=500 crt=383'59629 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:40.434929 7f491c044700 1 osd.10 pg_epoch: 501 pg[9.d( v 494'450 (0'0,494'450] local-lis/les=498/499 n=228 ec=190/190 lis/c 498/498 les/c/f 499/499/0 498/498/470) [0,6,10] r=2 lpr=499 crt=494'450 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:40.434981 7f491c845700 1 osd.10 pg_epoch: 501 pg[13.8s5( v 377'9578 (367'1954,377'9578] local-lis/les=498/500 n=2631 ec=345/345 lis/c 498/498 les/c/f 500/500/0 498/498/465) [2,11,5,8,6,10]p2(0) r=5 lpr=500 crt=377'9578 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:40.435025 7f491c044700 1 osd.10 pg_epoch: 501 pg[6.2( v 430'110237 (430'108737,430'110237] local-lis/les=498/500 n=25907 ec=76/76 lis/c 498/498 les/c/f 500/500/0 498/498/468) [7,10,2] r=1 lpr=500 crt=430'110237 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:40.435107 7f491c845700 1 osd.10 pg_epoch: 501 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=499 crt=166'68041 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 07:54:40.435143 7f491c044700 1 osd.10 pg_epoch: 501 pg[11.as3( v 383'60057 (367'56547,383'60057] local-lis/les=498/499 n=60057 ec=267/267 lis/c 498/498 les/c/f 499/499/0 498/498/484) [3,5,8,10,11,6]p3(0) r=3 lpr=499 crt=383'60057 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:40.435255 7f491c044700 1 osd.10 pg_epoch: 501 pg[13.bs4( v 377'9782 (355'1491,377'9782] local-lis/les=498/500 n=2694 ec=345/345 lis/c 498/498 les/c/f 500/500/0 498/498/478) [11,8,1,4,10,6]p11(0) r=4 lpr=500 crt=377'9782 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:40.435278 7f491c845700 1 osd.10 pg_epoch: 501 pg[11.1s0( v 383'60378 (352'56674,383'60378] local-lis/les=498/499 n=60378 ec=267/267 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,5,2,8,6,11]p10(0) r=0 lpr=499 crt=383'60378 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 07:54:40.435361 7f491c044700 1 osd.10 pg_epoch: 501 pg[6.26( v 461'123787 (430'122194,461'123787] local-lis/les=498/500 n=26121 ec=76/76 lis/c 498/498 les/c/f 500/500/0 498/498/468) [7,10,8] r=1 lpr=500 crt=461'123787 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:40.435423 7f491c845700 1 osd.10 pg_epoch: 501 pg[13.4s3( v 377'9519 (355'1503,377'9519] local-lis/les=498/500 n=2672 ec=345/345 lis/c 498/498 les/c/f 500/500/0 498/498/467) [5,1,8,10,11,6]p5(0) r=3 lpr=500 crt=377'9519 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:40.435463 7f491c044700 1 osd.10 pg_epoch: 501 pg[13.9s1( v 377'10260 (352'219,377'10260] local-lis/les=498/500 n=2727 ec=345/345 lis/c 498/498 les/c/f 500/500/0 498/498/467) [5,10,3,6,8,11]p5(0) r=1 lpr=500 crt=377'10254 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:40.435525 7f491c845700 1 osd.10 pg_epoch: 501 pg[5.2f( v 166'67864 (166'66267,166'67864] local-lis/les=498/500 n=67864 ec=71/71 lis/c 498/498 les/c/f 500/500/0 498/498/472) [8,10,4] r=1 lpr=500 crt=166'67864 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:40.435584 7f491c044700 1 osd.10 pg_epoch: 501 pg[11.ds3( v 383'59865 (367'56373,383'59865] local-lis/les=498/500 n=59865 ec=267/267 lis/c 498/498 les/c/f 500/500/0 498/498/465) [2,4,11,10,8,6]p2(0) r=3 lpr=500 crt=383'59865 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:40.435628 7f491c845700 1 osd.10 pg_epoch: 501 pg[6.2c( v 457'120217 (430'118627,457'120217] local-lis/les=498/500 n=26249 ec=76/76 lis/c 498/498 les/c/f 500/500/0 498/498/487) [4,6,10] r=2 lpr=500 crt=457'120217 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:40.435693 7f491c044700 1 osd.10 pg_epoch: 501 pg[16.d( empty local-lis/les=498/499 n=0 ec=442/442 lis/c 498/498 les/c/f 499/499/0 498/498/470) [0,3,10] r=2 lpr=499 crt=0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:40.479806 7f491c845700 1 osd.10 pg_epoch: 501 pg[10.1c( v 386'649 (0'0,386'649] lb MIN (bitwise) local-lis/les=419/420 n=500 ec=193/193 lis/c 419/419 les/c/f 420/420/0 487/487/478) [11,4,8] r=-1 lpr=494 pi=[419,487)/2 crt=386'649 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:40.479837 7f491c044700 1 osd.10 pg_epoch: 501 pg[13.fs4( v 377'8989 (0'0,377'8989] local-lis/les=498/500 n=2712 ec=345/345 lis/c 498/498 les/c/f 500/500/0 498/498/467) [5,7,3,6,10,8]p5(0) r=4 lpr=500 crt=377'8979 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:40.479915 7f491c845700 1 osd.10 pg_epoch: 501 pg[10.7( v 386'575 (0'0,386'575] local-lis/les=498/499 n=507 ec=193/193 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,6,3] r=0 lpr=499 crt=386'575 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 07:54:40.479946 7f491c044700 1 osd.10 pg_epoch: 501 pg[16.1a( empty local-lis/les=498/500 n=0 ec=442/442 lis/c 498/498 les/c/f 500/500/0 498/498/478) [11,10,3] r=1 lpr=500 crt=0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:40.480043 7f491c044700 1 osd.10 pg_epoch: 501 pg[9.1a( v 494'341 (0'0,494'341] local-lis/les=498/500 n=242 ec=190/190 lis/c 498/498 les/c/f 500/500/0 498/498/468) [7,2,10] r=2 lpr=500 crt=494'341 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:40.480139 7f491c044700 1 osd.10 pg_epoch: 501 pg[6.15( v 463'106940 (430'105357,463'106940] local-lis/les=498/500 n=25757 ec=76/76 lis/c 498/498 les/c/f 500/500/0 498/498/467) [5,10,11] r=1 lpr=500 crt=463'106940 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:40.480226 7f491c044700 1 osd.10 pg_epoch: 501 pg[16.3( empty local-lis/les=498/500 n=0 ec=442/442 lis/c 498/498 les/c/f 500/500/0 498/498/467) [5,2,10] r=2 lpr=500 crt=0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:40.480330 7f491c044700 1 osd.10 pg_epoch: 501 pg[13.3s1( v 377'10405 (371'2081,377'10405] local-lis/les=498/500 n=2736 ec=345/345 lis/c 498/498 les/c/f 500/500/0 498/498/470) [1,10,8,6,7,5]p1(0) r=1 lpr=500 crt=377'10405 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:40.480453 7f491c044700 1 osd.10 pg_epoch: 501 pg[11.fs3( v 383'63252 (352'59530,383'63252] local-lis/les=498/499 n=60320 ec=267/267 lis/c 498/498 les/c/f 499/499/0 498/498/470) [0,6,2,10,7,8]p0(0) r=3 lpr=499 crt=383'63252 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:40.480564 7f491c044700 1 osd.10 pg_epoch: 501 pg[9.1e( v 494'752 (0'0,494'752] local-lis/les=498/500 n=270 ec=190/190 lis/c 498/498 les/c/f 500/500/0 498/498/465) [2,4,10] r=2 lpr=500 crt=494'752 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:40.480659 7f491c044700 1 osd.10 pg_epoch: 501 pg[13.2s4( v 377'9209 (0'0,377'9209] local-lis/les=498/500 n=2712 ec=345/345 lis/c 498/498 les/c/f 500/500/0 498/498/467) [5,8,6,7,10,3]p5(0) r=4 lpr=500 crt=377'9201 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:40.480755 7f493724cd80 0 osd.10 501 done with init, starting boot process >2018-07-11 07:54:40.480798 7f493724cd80 1 osd.10 501 start_boot >2018-07-11 07:54:41.148593 7f4932031700 0 -- 10.8.128.107:6801/14173 >> 10.8.128.106:6803/12650 conn(0x55f668a26000 :6801 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg: challenging authorizer >2018-07-11 07:54:41.175180 7f493102f700 0 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg: challenging authorizer >2018-07-11 07:54:41.179946 7f4931830700 0 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6801/15255 conn(0x55f64e3c2000 :6801 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg: challenging authorizer >2018-07-11 07:54:41.184109 7f4923853700 1 osd.10 504 state: booting -> active >2018-07-11 07:54:41.184358 7f491c845700 1 osd.10 pg_epoch: 502 pg[16.e( empty local-lis/les=498/500 n=0 ec=442/442 lis/c 498/498 les/c/f 500/500/0 502/502/465) [2,4] r=-1 lpr=502 pi=[498,502)/1 crt=0'0 unknown NOTIFY] start_peering_interval up [2,4,10] -> [2,4], acting [2,4,10] -> [2,4], acting_primary 2 -> 2, up_primary 2 -> 2, role 2 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.184369 7f491c044700 1 osd.10 pg_epoch: 502 pg[5.37( v 166'68122 (166'66622,166'68122] local-lis/les=498/500 n=68122 ec=71/71 lis/c 498/498 les/c/f 500/500/0 502/502/487) [4,8] r=-1 lpr=502 pi=[498,502)/1 crt=166'68122 lcod 0'0 unknown NOTIFY] start_peering_interval up [4,10,8] -> [4,8], acting [4,10,8] -> [4,8], acting_primary 4 -> 4, up_primary 4 -> 4, role 1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.184505 7f491c845700 1 osd.10 pg_epoch: 504 pg[16.e( empty local-lis/les=498/500 n=0 ec=442/442 lis/c 498/498 les/c/f 500/500/0 504/504/465) [2,4,10] r=2 lpr=504 pi=[498,504)/1 crt=0'0 unknown NOTIFY] start_peering_interval up [2,4] -> [2,4,10], acting [2,4] -> [2,4,10], acting_primary 2 -> 2, up_primary 2 -> 2, role -1 -> 2, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.184504 7f491c044700 1 osd.10 pg_epoch: 504 pg[5.37( v 166'68122 (166'66622,166'68122] local-lis/les=498/500 n=68122 ec=71/71 lis/c 498/498 les/c/f 500/500/0 504/504/487) [4,10,8] r=1 lpr=504 pi=[498,504)/1 crt=166'68122 lcod 0'0 unknown NOTIFY] start_peering_interval up [4,8] -> [4,10,8], acting [4,8] -> [4,10,8], acting_primary 4 -> 4, up_primary 4 -> 4, role -1 -> 1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.184601 7f491c845700 1 osd.10 pg_epoch: 504 pg[16.e( empty local-lis/les=498/500 n=0 ec=442/442 lis/c 498/498 les/c/f 500/500/0 504/504/465) [2,4,10] r=2 lpr=504 pi=[498,504)/1 crt=0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:41.184668 7f491c044700 1 osd.10 pg_epoch: 504 pg[5.37( v 166'68122 (166'66622,166'68122] local-lis/les=498/500 n=68122 ec=71/71 lis/c 498/498 les/c/f 500/500/0 504/504/487) [4,10,8] r=1 lpr=504 pi=[498,504)/1 crt=166'68122 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:41.184862 7f491c845700 1 osd.10 pg_epoch: 502 pg[5.24( v 166'68298 (166'66798,166'68298] local-lis/les=498/499 n=68298 ec=71/71 lis/c 498/498 les/c/f 499/500/0 498/498/498) [10,11,1] r=0 lpr=500 crt=166'68298 lcod 0'0 mlcod 0'0 peering] state<Started/Primary/Peering>: Peering, affected_by_map, going to Reset >2018-07-11 07:54:41.184897 7f491c044700 1 osd.10 pg_epoch: 502 pg[5.33( v 166'68409 (166'66894,166'68409] local-lis/les=498/500 n=68409 ec=71/71 lis/c 498/498 les/c/f 500/500/0 502/502/465) [2,8] r=-1 lpr=502 pi=[498,502)/1 crt=166'68409 lcod 0'0 unknown NOTIFY] start_peering_interval up [2,10,8] -> [2,8], acting [2,10,8] -> [2,8], acting_primary 2 -> 2, up_primary 2 -> 2, role 1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.185003 7f491c044700 1 osd.10 pg_epoch: 504 pg[5.33( v 166'68409 (166'66894,166'68409] local-lis/les=498/500 n=68409 ec=71/71 lis/c 498/498 les/c/f 500/500/0 504/504/465) [2,10,8] r=1 lpr=504 pi=[498,504)/1 crt=166'68409 lcod 0'0 unknown NOTIFY] start_peering_interval up [2,8] -> [2,10,8], acting [2,8] -> [2,10,8], acting_primary 2 -> 2, up_primary 2 -> 2, role -1 -> 1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.185067 7f491c845700 1 osd.10 pg_epoch: 502 pg[5.24( v 166'68298 (166'66798,166'68298] local-lis/les=498/499 n=68298 ec=71/71 lis/c 498/498 les/c/f 499/500/0 502/502/502) [11,1] r=-1 lpr=502 pi=[498,502)/1 crt=166'68298 lcod 0'0 unknown] start_peering_interval up [10,11,1] -> [11,1], acting [10,11,1] -> [11,1], acting_primary 10 -> 11, up_primary 10 -> 11, role 0 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.185137 7f491c845700 1 osd.10 pg_epoch: 504 pg[5.24( v 166'68298 (166'66798,166'68298] local-lis/les=498/499 n=68298 ec=71/71 lis/c 498/498 les/c/f 499/500/0 504/504/504) [10,11,1] r=0 lpr=504 pi=[498,504)/1 crt=166'68298 lcod 0'0 mlcod 0'0 unknown NOTIFY] start_peering_interval up [11,1] -> [10,11,1], acting [11,1] -> [10,11,1], acting_primary 11 -> 10, up_primary 11 -> 10, role -1 -> 0, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.185157 7f491c044700 1 osd.10 pg_epoch: 504 pg[5.33( v 166'68409 (166'66894,166'68409] local-lis/les=498/500 n=68409 ec=71/71 lis/c 498/498 les/c/f 500/500/0 504/504/465) [2,10,8] r=1 lpr=504 pi=[498,504)/1 crt=166'68409 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:41.185300 7f491c845700 1 osd.10 pg_epoch: 504 pg[5.24( v 166'68298 (166'66798,166'68298] local-lis/les=498/499 n=68298 ec=71/71 lis/c 498/498 les/c/f 499/500/0 504/504/504) [10,11,1] r=0 lpr=504 pi=[498,504)/1 crt=166'68298 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 07:54:41.185410 7f491c044700 1 osd.10 pg_epoch: 502 pg[5.35( v 166'67898 (166'66348,166'67898] local-lis/les=498/499 n=67898 ec=71/71 lis/c 498/498 les/c/f 499/499/0 502/502/470) [0,11] r=-1 lpr=502 pi=[498,502)/1 crt=166'67898 lcod 0'0 unknown NOTIFY] start_peering_interval up [0,11,10] -> [0,11], acting [0,11,10] -> [0,11], acting_primary 0 -> 0, up_primary 0 -> 0, role 2 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.185497 7f491c044700 1 osd.10 pg_epoch: 504 pg[5.35( v 166'67898 (166'66348,166'67898] local-lis/les=498/499 n=67898 ec=71/71 lis/c 498/498 les/c/f 499/499/0 504/504/470) [0,11,10] r=2 lpr=504 pi=[498,504)/1 crt=166'67898 lcod 0'0 unknown NOTIFY] start_peering_interval up [0,11] -> [0,11,10], acting [0,11] -> [0,11,10], acting_primary 0 -> 0, up_primary 0 -> 0, role -1 -> 2, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.185529 7f491c845700 1 osd.10 pg_epoch: 502 pg[13.ds1( v 377'10591 (355'1551,377'10591] local-lis/les=498/499 n=2744 ec=345/345 lis/c 498/498 les/c/f 499/500/0 502/502/470) [0,2147483647,1,8,11,6]p0(0) r=-1 lpr=502 pi=[498,502)/1 crt=377'10591 lcod 0'0 unknown NOTIFY] start_peering_interval up [0,10,1,8,11,6] -> [0,2147483647,1,8,11,6], acting [0,10,1,8,11,6] -> [0,2147483647,1,8,11,6], acting_primary 0(0) -> 0, up_primary 0(0) -> 0, role 1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.185570 7f491c044700 1 osd.10 pg_epoch: 504 pg[5.35( v 166'67898 (166'66348,166'67898] local-lis/les=498/499 n=67898 ec=71/71 lis/c 498/498 les/c/f 499/499/0 504/504/470) [0,11,10] r=2 lpr=504 pi=[498,504)/1 crt=166'67898 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:41.185639 7f491c845700 1 osd.10 pg_epoch: 504 pg[13.ds1( v 377'10591 (355'1551,377'10591] local-lis/les=498/499 n=2744 ec=345/345 lis/c 498/498 les/c/f 499/500/0 504/504/470) [0,10,1,8,11,6]p0(0) r=1 lpr=504 pi=[498,504)/1 crt=377'10591 lcod 0'0 unknown NOTIFY] start_peering_interval up [0,2147483647,1,8,11,6] -> [0,10,1,8,11,6], acting [0,2147483647,1,8,11,6] -> [0,10,1,8,11,6], acting_primary 0(0) -> 0, up_primary 0(0) -> 0, role -1 -> 1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.185709 7f491c044700 1 osd.10 pg_epoch: 502 pg[5.c( v 166'68190 (166'66593,166'68190] local-lis/les=498/500 n=68190 ec=71/71 lis/c 498/498 les/c/f 500/500/0 502/502/470) [1,4] r=-1 lpr=502 pi=[498,502)/1 crt=166'68190 lcod 0'0 unknown NOTIFY] start_peering_interval up [1,10,4] -> [1,4], acting [1,10,4] -> [1,4], acting_primary 1 -> 1, up_primary 1 -> 1, role 1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.185720 7f491c845700 1 osd.10 pg_epoch: 504 pg[13.ds1( v 377'10591 (355'1551,377'10591] local-lis/les=498/499 n=2744 ec=345/345 lis/c 498/498 les/c/f 499/500/0 504/504/470) [0,10,1,8,11,6]p0(0) r=1 lpr=504 pi=[498,504)/1 crt=377'10591 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:41.185768 7f491c044700 1 osd.10 pg_epoch: 504 pg[5.c( v 166'68190 (166'66593,166'68190] local-lis/les=498/500 n=68190 ec=71/71 lis/c 498/498 les/c/f 500/500/0 504/504/470) [1,10,4] r=1 lpr=504 pi=[498,504)/1 crt=166'68190 lcod 0'0 unknown NOTIFY] start_peering_interval up [1,4] -> [1,10,4], acting [1,4] -> [1,10,4], acting_primary 1 -> 1, up_primary 1 -> 1, role -1 -> 1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.185840 7f491c044700 1 osd.10 pg_epoch: 504 pg[5.c( v 166'68190 (166'66593,166'68190] local-lis/les=498/500 n=68190 ec=71/71 lis/c 498/498 les/c/f 500/500/0 504/504/470) [1,10,4] r=1 lpr=504 pi=[498,504)/1 crt=166'68190 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:41.185883 7f491c845700 1 osd.10 pg_epoch: 502 pg[13.cs2( v 377'10679 (371'2117,377'10679] local-lis/les=498/500 n=2785 ec=345/345 lis/c 498/498 les/c/f 500/500/0 502/502/470) [1,11,2147483647,4,8,6]p1(0) r=-1 lpr=502 pi=[498,502)/1 crt=377'10679 lcod 0'0 unknown NOTIFY] start_peering_interval up [1,11,10,4,8,6] -> [1,11,2147483647,4,8,6], acting [1,11,10,4,8,6] -> [1,11,2147483647,4,8,6], acting_primary 1(0) -> 1, up_primary 1(0) -> 1, role 2 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.185956 7f491c845700 1 osd.10 pg_epoch: 504 pg[13.cs2( v 377'10679 (371'2117,377'10679] local-lis/les=498/500 n=2785 ec=345/345 lis/c 498/498 les/c/f 500/500/0 504/504/470) [1,11,10,4,8,6]p1(0) r=2 lpr=504 pi=[498,504)/1 crt=377'10679 lcod 0'0 unknown NOTIFY] start_peering_interval up [1,11,2147483647,4,8,6] -> [1,11,10,4,8,6], acting [1,11,2147483647,4,8,6] -> [1,11,10,4,8,6], acting_primary 1(0) -> 1, up_primary 1(0) -> 1, role -1 -> 2, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.186025 7f491c044700 1 osd.10 pg_epoch: 502 pg[6.17( v 430'108120 (430'106620,430'108120] local-lis/les=498/499 n=26407 ec=76/76 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,5] r=0 lpr=499 crt=430'108120 lcod 0'0 mlcod 0'0 peering] state<Started/Primary/Peering>: Peering, affected_by_map, going to Reset >2018-07-11 07:54:41.186040 7f491c845700 1 osd.10 pg_epoch: 504 pg[13.cs2( v 377'10679 (371'2117,377'10679] local-lis/les=498/500 n=2785 ec=345/345 lis/c 498/498 les/c/f 500/500/0 504/504/470) [1,11,10,4,8,6]p1(0) r=2 lpr=504 pi=[498,504)/1 crt=377'10679 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:41.186158 7f491c044700 1 osd.10 pg_epoch: 502 pg[6.17( v 430'108120 (430'106620,430'108120] local-lis/les=498/499 n=26407 ec=76/76 lis/c 498/498 les/c/f 499/499/0 502/502/502) [2,5] r=-1 lpr=502 pi=[498,502)/1 crt=430'108120 lcod 0'0 unknown] start_peering_interval up [10,2,5] -> [2,5], acting [10,2,5] -> [2,5], acting_primary 10 -> 2, up_primary 10 -> 2, role 0 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.186182 7f491c845700 1 osd.10 pg_epoch: 502 pg[16.10( empty local-lis/les=498/500 n=0 ec=442/442 lis/c 498/498 les/c/f 500/500/0 502/502/465) [2,7] r=-1 lpr=502 pi=[498,502)/1 crt=0'0 unknown NOTIFY] start_peering_interval up [2,10,7] -> [2,7], acting [2,10,7] -> [2,7], acting_primary 2 -> 2, up_primary 2 -> 2, role 1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.186259 7f491c845700 1 osd.10 pg_epoch: 504 pg[16.10( empty local-lis/les=498/500 n=0 ec=442/442 lis/c 498/498 les/c/f 500/500/0 504/504/465) [2,10,7] r=1 lpr=504 pi=[498,504)/1 crt=0'0 unknown NOTIFY] start_peering_interval up [2,7] -> [2,10,7], acting [2,7] -> [2,10,7], acting_primary 2 -> 2, up_primary 2 -> 2, role -1 -> 1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.186266 7f491c044700 1 osd.10 pg_epoch: 504 pg[6.17( v 430'108120 (430'106620,430'108120] local-lis/les=498/499 n=26407 ec=76/76 lis/c 498/498 les/c/f 499/499/0 504/504/504) [10,2,5] r=0 lpr=504 pi=[498,504)/1 crt=430'108120 lcod 0'0 mlcod 0'0 unknown NOTIFY] start_peering_interval up [2,5] -> [10,2,5], acting [2,5] -> [10,2,5], acting_primary 2 -> 10, up_primary 2 -> 10, role -1 -> 0, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.186336 7f491c845700 1 osd.10 pg_epoch: 504 pg[16.10( empty local-lis/les=498/500 n=0 ec=442/442 lis/c 498/498 les/c/f 500/500/0 504/504/465) [2,10,7] r=1 lpr=504 pi=[498,504)/1 crt=0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:41.186379 7f491c044700 1 osd.10 pg_epoch: 504 pg[6.17( v 430'108120 (430'106620,430'108120] local-lis/les=498/499 n=26407 ec=76/76 lis/c 498/498 les/c/f 499/499/0 504/504/504) [10,2,5] r=0 lpr=504 pi=[498,504)/1 crt=430'108120 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 07:54:41.186413 7f491c845700 1 osd.10 pg_epoch: 502 pg[10.a( v 386'2032 (203'500,386'2032] local-lis/les=498/499 n=516 ec=193/193 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,5] r=0 lpr=499 crt=386'2032 lcod 0'0 mlcod 0'0 peering] state<Started/Primary/Peering>: Peering, affected_by_map, going to Reset >2018-07-11 07:54:41.186515 7f491c845700 1 osd.10 pg_epoch: 502 pg[10.a( v 386'2032 (203'500,386'2032] local-lis/les=498/499 n=516 ec=193/193 lis/c 498/498 les/c/f 499/499/0 502/502/502) [2,5] r=-1 lpr=502 pi=[498,502)/1 crt=386'2032 lcod 0'0 unknown] start_peering_interval up [10,2,5] -> [2,5], acting [10,2,5] -> [2,5], acting_primary 10 -> 2, up_primary 10 -> 2, role 0 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.186574 7f491c845700 1 osd.10 pg_epoch: 504 pg[10.a( v 386'2032 (203'500,386'2032] local-lis/les=498/499 n=516 ec=193/193 lis/c 498/498 les/c/f 499/499/0 504/504/504) [10,2,5] r=0 lpr=504 pi=[498,504)/1 crt=386'2032 lcod 0'0 mlcod 0'0 unknown NOTIFY] start_peering_interval up [2,5] -> [10,2,5], acting [2,5] -> [10,2,5], acting_primary 2 -> 10, up_primary 2 -> 10, role -1 -> 0, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.186616 7f491c044700 1 osd.10 pg_epoch: 502 pg[6.21( v 461'128646 (430'127048,461'128646] local-lis/les=498/499 n=25982 ec=76/76 lis/c 498/498 les/c/f 499/499/0 502/502/470) [0,2] r=-1 lpr=502 pi=[498,502)/1 crt=461'128646 lcod 0'0 unknown NOTIFY] start_peering_interval up [0,2,10] -> [0,2], acting [0,2,10] -> [0,2], acting_primary 0 -> 0, up_primary 0 -> 0, role 2 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.186641 7f491c845700 1 osd.10 pg_epoch: 504 pg[10.a( v 386'2032 (203'500,386'2032] local-lis/les=498/499 n=516 ec=193/193 lis/c 498/498 les/c/f 499/499/0 504/504/504) [10,2,5] r=0 lpr=504 pi=[498,504)/1 crt=386'2032 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 07:54:41.186690 7f491c044700 1 osd.10 pg_epoch: 504 pg[6.21( v 461'128646 (430'127048,461'128646] local-lis/les=498/499 n=25982 ec=76/76 lis/c 498/498 les/c/f 499/499/0 504/504/470) [0,2,10] r=2 lpr=504 pi=[498,504)/1 crt=461'128646 lcod 0'0 unknown NOTIFY] start_peering_interval up [0,2] -> [0,2,10], acting [0,2] -> [0,2,10], acting_primary 0 -> 0, up_primary 0 -> 0, role -1 -> 2, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.186786 7f491c044700 1 osd.10 pg_epoch: 504 pg[6.21( v 461'128646 (430'127048,461'128646] local-lis/les=498/499 n=25982 ec=76/76 lis/c 498/498 les/c/f 499/499/0 504/504/470) [0,2,10] r=2 lpr=504 pi=[498,504)/1 crt=461'128646 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:41.186822 7f491c845700 1 osd.10 pg_epoch: 502 pg[5.23( v 166'68497 (166'66949,166'68497] local-lis/les=498/500 n=68497 ec=71/71 lis/c 498/498 les/c/f 500/500/0 502/502/472) [8,3] r=-1 lpr=502 pi=[498,502)/1 crt=166'68497 lcod 0'0 unknown NOTIFY] start_peering_interval up [8,3,10] -> [8,3], acting [8,3,10] -> [8,3], acting_primary 8 -> 8, up_primary 8 -> 8, role 2 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.186880 7f491c845700 1 osd.10 pg_epoch: 504 pg[5.23( v 166'68497 (166'66949,166'68497] local-lis/les=498/500 n=68497 ec=71/71 lis/c 498/498 les/c/f 500/500/0 504/504/472) [8,3,10] r=2 lpr=504 pi=[498,504)/1 crt=166'68497 lcod 0'0 unknown NOTIFY] start_peering_interval up [8,3] -> [8,3,10], acting [8,3] -> [8,3,10], acting_primary 8 -> 8, up_primary 8 -> 8, role -1 -> 2, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.186949 7f491c845700 1 osd.10 pg_epoch: 504 pg[5.23( v 166'68497 (166'66949,166'68497] local-lis/les=498/500 n=68497 ec=71/71 lis/c 498/498 les/c/f 500/500/0 504/504/472) [8,3,10] r=2 lpr=504 pi=[498,504)/1 crt=166'68497 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:41.186988 7f491c044700 1 osd.10 pg_epoch: 502 pg[10.9( v 390'615 (0'0,390'615] local-lis/les=498/499 n=478 ec=193/193 lis/c 498/498 les/c/f 499/499/0 502/502/470) [0,2] r=-1 lpr=502 pi=[498,502)/1 crt=390'615 lcod 0'0 unknown NOTIFY] start_peering_interval up [0,2,10] -> [0,2], acting [0,2,10] -> [0,2], acting_primary 0 -> 0, up_primary 0 -> 0, role 2 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.187070 7f491c044700 1 osd.10 pg_epoch: 504 pg[10.9( v 390'615 (0'0,390'615] local-lis/les=498/499 n=478 ec=193/193 lis/c 498/498 les/c/f 499/499/0 504/504/470) [0,2,10] r=2 lpr=504 pi=[498,504)/1 crt=390'615 lcod 0'0 unknown NOTIFY] start_peering_interval up [0,2] -> [0,2,10], acting [0,2] -> [0,2,10], acting_primary 0 -> 0, up_primary 0 -> 0, role -1 -> 2, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.187088 7f491c845700 1 osd.10 pg_epoch: 502 pg[10.f( v 383'568 (0'0,383'568] local-lis/les=498/500 n=494 ec=193/193 lis/c 498/498 les/c/f 500/500/0 502/502/478) [11,2] r=-1 lpr=502 pi=[498,502)/1 crt=383'568 lcod 0'0 unknown NOTIFY] start_peering_interval up [11,10,2] -> [11,2], acting [11,10,2] -> [11,2], acting_primary 11 -> 11, up_primary 11 -> 11, role 1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.187146 7f491c044700 1 osd.10 pg_epoch: 504 pg[10.9( v 390'615 (0'0,390'615] local-lis/les=498/499 n=478 ec=193/193 lis/c 498/498 les/c/f 499/499/0 504/504/470) [0,2,10] r=2 lpr=504 pi=[498,504)/1 crt=390'615 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:41.187145 7f491c845700 1 osd.10 pg_epoch: 504 pg[10.f( v 383'568 (0'0,383'568] local-lis/les=498/500 n=494 ec=193/193 lis/c 498/498 les/c/f 500/500/0 504/504/478) [11,10,2] r=1 lpr=504 pi=[498,504)/1 crt=383'568 lcod 0'0 unknown NOTIFY] start_peering_interval up [11,2] -> [11,10,2], acting [11,2] -> [11,10,2], acting_primary 11 -> 11, up_primary 11 -> 11, role -1 -> 1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.187208 7f491c845700 1 osd.10 pg_epoch: 504 pg[10.f( v 383'568 (0'0,383'568] local-lis/les=498/500 n=494 ec=193/193 lis/c 498/498 les/c/f 500/500/0 504/504/478) [11,10,2] r=1 lpr=504 pi=[498,504)/1 crt=383'568 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:41.187291 7f491c044700 1 osd.10 pg_epoch: 502 pg[9.a( v 494'572 (0'0,494'572] local-lis/les=498/499 n=232 ec=190/190 lis/c 498/498 les/c/f 499/499/0 502/502/470) [0,11] r=-1 lpr=502 pi=[498,502)/1 crt=494'572 lcod 0'0 unknown NOTIFY] start_peering_interval up [0,10,11] -> [0,11], acting [0,10,11] -> [0,11], acting_primary 0 -> 0, up_primary 0 -> 0, role 1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.187350 7f491c845700 1 osd.10 pg_epoch: 502 pg[6.3( v 430'104329 (430'102829,430'104329] local-lis/les=498/499 n=25881 ec=76/76 lis/c 498/498 les/c/f 499/499/0 502/502/470) [0,11] r=-1 lpr=502 pi=[498,502)/1 crt=430'104329 lcod 0'0 unknown NOTIFY] start_peering_interval up [0,10,11] -> [0,11], acting [0,10,11] -> [0,11], acting_primary 0 -> 0, up_primary 0 -> 0, role 1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.187353 7f491c044700 1 osd.10 pg_epoch: 504 pg[9.a( v 494'572 (0'0,494'572] local-lis/les=498/499 n=232 ec=190/190 lis/c 498/498 les/c/f 499/499/0 504/504/470) [0,10,11] r=1 lpr=504 pi=[498,504)/1 crt=494'572 lcod 0'0 unknown NOTIFY] start_peering_interval up [0,11] -> [0,10,11], acting [0,11] -> [0,10,11], acting_primary 0 -> 0, up_primary 0 -> 0, role -1 -> 1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.187419 7f491c044700 1 osd.10 pg_epoch: 504 pg[9.a( v 494'572 (0'0,494'572] local-lis/les=498/499 n=232 ec=190/190 lis/c 498/498 les/c/f 499/499/0 504/504/470) [0,10,11] r=1 lpr=504 pi=[498,504)/1 crt=494'572 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:41.187413 7f491c845700 1 osd.10 pg_epoch: 504 pg[6.3( v 430'104329 (430'102829,430'104329] local-lis/les=498/499 n=25881 ec=76/76 lis/c 498/498 les/c/f 499/499/0 504/504/470) [0,10,11] r=1 lpr=504 pi=[498,504)/1 crt=430'104329 lcod 0'0 unknown NOTIFY] start_peering_interval up [0,11] -> [0,10,11], acting [0,11] -> [0,10,11], acting_primary 0 -> 0, up_primary 0 -> 0, role -1 -> 1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.187482 7f491c845700 1 osd.10 pg_epoch: 504 pg[6.3( v 430'104329 (430'102829,430'104329] local-lis/les=498/499 n=25881 ec=76/76 lis/c 498/498 les/c/f 499/499/0 504/504/470) [0,10,11] r=1 lpr=504 pi=[498,504)/1 crt=430'104329 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:41.187556 7f491c044700 1 osd.10 pg_epoch: 502 pg[9.7( v 486'503 (0'0,486'503] local-lis/les=498/500 n=248 ec=190/190 lis/c 498/498 les/c/f 500/500/0 502/502/468) [7,2] r=-1 lpr=502 pi=[498,502)/1 crt=486'503 lcod 0'0 unknown NOTIFY] start_peering_interval up [7,10,2] -> [7,2], acting [7,10,2] -> [7,2], acting_primary 7 -> 7, up_primary 7 -> 7, role 1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.187614 7f491c044700 1 osd.10 pg_epoch: 504 pg[9.7( v 486'503 (0'0,486'503] local-lis/les=498/500 n=248 ec=190/190 lis/c 498/498 les/c/f 500/500/0 504/504/468) [7,10,2] r=1 lpr=504 pi=[498,504)/1 crt=486'503 lcod 0'0 unknown NOTIFY] start_peering_interval up [7,2] -> [7,10,2], acting [7,2] -> [7,10,2], acting_primary 7 -> 7, up_primary 7 -> 7, role -1 -> 1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.187625 7f491c845700 1 osd.10 pg_epoch: 502 pg[6.32( v 430'104474 (430'102974,430'104474] local-lis/les=498/500 n=25649 ec=76/76 lis/c 498/498 les/c/f 500/500/0 502/502/487) [4,2] r=-1 lpr=502 pi=[498,502)/1 crt=430'104474 lcod 0'0 unknown NOTIFY] start_peering_interval up [4,2,10] -> [4,2], acting [4,2,10] -> [4,2], acting_primary 4 -> 4, up_primary 4 -> 4, role 2 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.187680 7f491c044700 1 osd.10 pg_epoch: 504 pg[9.7( v 486'503 (0'0,486'503] local-lis/les=498/500 n=248 ec=190/190 lis/c 498/498 les/c/f 500/500/0 504/504/468) [7,10,2] r=1 lpr=504 pi=[498,504)/1 crt=486'503 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:41.187683 7f491c845700 1 osd.10 pg_epoch: 504 pg[6.32( v 430'104474 (430'102974,430'104474] local-lis/les=498/500 n=25649 ec=76/76 lis/c 498/498 les/c/f 500/500/0 504/504/487) [4,2,10] r=2 lpr=504 pi=[498,504)/1 crt=430'104474 lcod 0'0 unknown NOTIFY] start_peering_interval up [4,2] -> [4,2,10], acting [4,2] -> [4,2,10], acting_primary 4 -> 4, up_primary 4 -> 4, role -1 -> 2, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.187748 7f491c845700 1 osd.10 pg_epoch: 504 pg[6.32( v 430'104474 (430'102974,430'104474] local-lis/les=498/500 n=25649 ec=76/76 lis/c 498/498 les/c/f 500/500/0 504/504/487) [4,2,10] r=2 lpr=504 pi=[498,504)/1 crt=430'104474 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:41.187755 7f491c044700 1 osd.10 pg_epoch: 502 pg[5.1c( v 383'68804 (166'67302,383'68804] local-lis/les=498/499 n=68803 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,8,2] r=0 lpr=499 crt=383'68804 lcod 0'0 mlcod 0'0 peering] state<Started/Primary/Peering>: Peering, affected_by_map, going to Reset >2018-07-11 07:54:41.187828 7f491c845700 1 osd.10 pg_epoch: 502 pg[6.0( v 430'122181 (430'120681,430'122181] local-lis/les=498/499 n=25944 ec=76/76 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,7,2] r=0 lpr=499 crt=430'122181 lcod 0'0 mlcod 0'0 peering] state<Started/Primary/Peering>: Peering, affected_by_map, going to Reset >2018-07-11 07:54:41.187861 7f491c044700 1 osd.10 pg_epoch: 502 pg[5.1c( v 383'68804 (166'67302,383'68804] local-lis/les=498/499 n=68803 ec=71/71 lis/c 498/498 les/c/f 499/499/0 502/502/502) [8,2] r=-1 lpr=502 pi=[498,502)/1 crt=383'68804 lcod 0'0 unknown] start_peering_interval up [10,8,2] -> [8,2], acting [10,8,2] -> [8,2], acting_primary 10 -> 8, up_primary 10 -> 8, role 0 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.187921 7f491c845700 1 osd.10 pg_epoch: 502 pg[6.0( v 430'122181 (430'120681,430'122181] local-lis/les=498/499 n=25944 ec=76/76 lis/c 498/498 les/c/f 499/499/0 502/502/502) [7,2] r=-1 lpr=502 pi=[498,502)/1 crt=430'122181 lcod 0'0 unknown] start_peering_interval up [10,7,2] -> [7,2], acting [10,7,2] -> [7,2], acting_primary 10 -> 7, up_primary 10 -> 7, role 0 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.187921 7f491c044700 1 osd.10 pg_epoch: 504 pg[5.1c( v 383'68804 (166'67302,383'68804] local-lis/les=498/499 n=68803 ec=71/71 lis/c 498/498 les/c/f 499/499/0 504/504/504) [10,8,2] r=0 lpr=504 pi=[498,504)/1 crt=383'68804 lcod 0'0 mlcod 0'0 unknown NOTIFY] start_peering_interval up [8,2] -> [10,8,2], acting [8,2] -> [10,8,2], acting_primary 8 -> 10, up_primary 8 -> 10, role -1 -> 0, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.187980 7f491c845700 1 osd.10 pg_epoch: 504 pg[6.0( v 430'122181 (430'120681,430'122181] local-lis/les=498/499 n=25944 ec=76/76 lis/c 498/498 les/c/f 499/499/0 504/504/504) [10,7,2] r=0 lpr=504 pi=[498,504)/1 crt=430'122181 lcod 0'0 mlcod 0'0 unknown NOTIFY] start_peering_interval up [7,2] -> [10,7,2], acting [7,2] -> [10,7,2], acting_primary 7 -> 10, up_primary 7 -> 10, role -1 -> 0, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.188001 7f491c044700 1 osd.10 pg_epoch: 504 pg[5.1c( v 383'68804 (166'67302,383'68804] local-lis/les=498/499 n=68803 ec=71/71 lis/c 498/498 les/c/f 499/499/0 504/504/504) [10,8,2] r=0 lpr=504 pi=[498,504)/1 crt=383'68804 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 07:54:41.188052 7f491c845700 1 osd.10 pg_epoch: 504 pg[6.0( v 430'122181 (430'120681,430'122181] local-lis/les=498/499 n=25944 ec=76/76 lis/c 498/498 les/c/f 499/499/0 504/504/504) [10,7,2] r=0 lpr=504 pi=[498,504)/1 crt=430'122181 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 07:54:41.188159 7f491c845700 1 osd.10 pg_epoch: 502 pg[5.10( v 166'68154 (166'66598,166'68154] local-lis/les=498/499 n=68154 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,8,4] r=0 lpr=499 crt=166'68154 lcod 0'0 mlcod 0'0 peering] state<Started/Primary/Peering>: Peering, affected_by_map, going to Reset >2018-07-11 07:54:41.188169 7f491c044700 1 osd.10 pg_epoch: 502 pg[6.1f( v 430'114415 (430'112915,430'114415] local-lis/les=498/499 n=26413 ec=76/76 lis/c 498/498 les/c/f 499/499/0 502/502/484) [3,0] r=-1 lpr=502 pi=[498,502)/1 crt=430'114415 lcod 0'0 unknown NOTIFY] start_peering_interval up [3,10,0] -> [3,0], acting [3,10,0] -> [3,0], acting_primary 3 -> 3, up_primary 3 -> 3, role 1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.188223 7f491c044700 1 osd.10 pg_epoch: 504 pg[6.1f( v 430'114415 (430'112915,430'114415] local-lis/les=498/499 n=26413 ec=76/76 lis/c 498/498 les/c/f 499/499/0 504/504/484) [3,10,0] r=1 lpr=504 pi=[498,504)/1 crt=430'114415 lcod 0'0 unknown NOTIFY] start_peering_interval up [3,0] -> [3,10,0], acting [3,0] -> [3,10,0], acting_primary 3 -> 3, up_primary 3 -> 3, role -1 -> 1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.188273 7f491c845700 1 osd.10 pg_epoch: 502 pg[5.10( v 166'68154 (166'66598,166'68154] local-lis/les=498/499 n=68154 ec=71/71 lis/c 498/498 les/c/f 499/499/0 502/502/502) [8,4] r=-1 lpr=502 pi=[498,502)/1 crt=166'68154 lcod 0'0 unknown] start_peering_interval up [10,8,4] -> [8,4], acting [10,8,4] -> [8,4], acting_primary 10 -> 8, up_primary 10 -> 8, role 0 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.188304 7f491c044700 1 osd.10 pg_epoch: 504 pg[6.1f( v 430'114415 (430'112915,430'114415] local-lis/les=498/499 n=26413 ec=76/76 lis/c 498/498 les/c/f 499/499/0 504/504/484) [3,10,0] r=1 lpr=504 pi=[498,504)/1 crt=430'114415 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:41.188346 7f491c845700 1 osd.10 pg_epoch: 504 pg[5.10( v 166'68154 (166'66598,166'68154] local-lis/les=498/499 n=68154 ec=71/71 lis/c 498/498 les/c/f 499/499/0 504/504/504) [10,8,4] r=0 lpr=504 pi=[498,504)/1 crt=166'68154 lcod 0'0 mlcod 0'0 unknown NOTIFY] start_peering_interval up [8,4] -> [10,8,4], acting [8,4] -> [10,8,4], acting_primary 8 -> 10, up_primary 8 -> 10, role -1 -> 0, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.188417 7f491c845700 1 osd.10 pg_epoch: 504 pg[5.10( v 166'68154 (166'66598,166'68154] local-lis/les=498/499 n=68154 ec=71/71 lis/c 498/498 les/c/f 499/499/0 504/504/504) [10,8,4] r=0 lpr=504 pi=[498,504)/1 crt=166'68154 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 07:54:41.188438 7f491c044700 1 osd.10 pg_epoch: 502 pg[9.10( v 494'394 (0'0,494'394] local-lis/les=498/500 n=251 ec=190/190 lis/c 498/498 les/c/f 500/500/0 502/502/487) [4,11] r=-1 lpr=502 pi=[498,502)/1 crt=494'394 lcod 0'0 unknown NOTIFY] start_peering_interval up [4,11,10] -> [4,11], acting [4,11,10] -> [4,11], acting_primary 4 -> 4, up_primary 4 -> 4, role 2 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.188492 7f491c044700 1 osd.10 pg_epoch: 504 pg[9.10( v 494'394 (0'0,494'394] local-lis/les=498/500 n=251 ec=190/190 lis/c 498/498 les/c/f 500/500/0 504/504/487) [4,11,10] r=2 lpr=504 pi=[498,504)/1 crt=494'394 lcod 0'0 unknown NOTIFY] start_peering_interval up [4,11] -> [4,11,10], acting [4,11] -> [4,11,10], acting_primary 4 -> 4, up_primary 4 -> 4, role -1 -> 2, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.188597 7f491c044700 1 osd.10 pg_epoch: 504 pg[9.10( v 494'394 (0'0,494'394] local-lis/les=498/500 n=251 ec=190/190 lis/c 498/498 les/c/f 500/500/0 504/504/487) [4,11,10] r=2 lpr=504 pi=[498,504)/1 crt=494'394 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:41.188610 7f491c845700 1 osd.10 pg_epoch: 502 pg[11.4s2( v 383'60130 (352'56449,383'60130] local-lis/les=498/499 n=60130 ec=267/267 lis/c 498/498 les/c/f 499/499/0 502/502/470) [0,6,2147483647,8,3,11]p0(0) r=-1 lpr=502 pi=[498,502)/1 crt=383'60130 lcod 0'0 unknown NOTIFY] start_peering_interval up [0,6,10,8,3,11] -> [0,6,2147483647,8,3,11], acting [0,6,10,8,3,11] -> [0,6,2147483647,8,3,11], acting_primary 0(0) -> 0, up_primary 0(0) -> 0, role 2 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.188699 7f491c845700 1 osd.10 pg_epoch: 504 pg[11.4s2( v 383'60130 (352'56449,383'60130] local-lis/les=498/499 n=60130 ec=267/267 lis/c 498/498 les/c/f 499/499/0 504/504/470) [0,6,10,8,3,11]p0(0) r=2 lpr=504 pi=[498,504)/1 crt=383'60130 lcod 0'0 unknown NOTIFY] start_peering_interval up [0,6,2147483647,8,3,11] -> [0,6,10,8,3,11], acting [0,6,2147483647,8,3,11] -> [0,6,10,8,3,11], acting_primary 0(0) -> 0, up_primary 0(0) -> 0, role -1 -> 2, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.188762 7f491c044700 1 osd.10 pg_epoch: 502 pg[10.13( v 383'839 (0'0,383'839] local-lis/les=498/500 n=523 ec=193/193 lis/c 498/498 les/c/f 500/500/0 502/502/470) [1,7] r=-1 lpr=502 pi=[498,502)/1 crt=383'839 lcod 0'0 unknown NOTIFY] start_peering_interval up [1,10,7] -> [1,7], acting [1,10,7] -> [1,7], acting_primary 1 -> 1, up_primary 1 -> 1, role 1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.188774 7f491c845700 1 osd.10 pg_epoch: 504 pg[11.4s2( v 383'60130 (352'56449,383'60130] local-lis/les=498/499 n=60130 ec=267/267 lis/c 498/498 les/c/f 499/499/0 504/504/470) [0,6,10,8,3,11]p0(0) r=2 lpr=504 pi=[498,504)/1 crt=383'60130 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:41.188827 7f491c044700 1 osd.10 pg_epoch: 504 pg[10.13( v 383'839 (0'0,383'839] local-lis/les=498/500 n=523 ec=193/193 lis/c 498/498 les/c/f 500/500/0 504/504/470) [1,10,7] r=1 lpr=504 pi=[498,504)/1 crt=383'839 lcod 0'0 unknown NOTIFY] start_peering_interval up [1,7] -> [1,10,7], acting [1,7] -> [1,10,7], acting_primary 1 -> 1, up_primary 1 -> 1, role -1 -> 1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.188898 7f491c044700 1 osd.10 pg_epoch: 504 pg[10.13( v 383'839 (0'0,383'839] local-lis/les=498/500 n=523 ec=193/193 lis/c 498/498 les/c/f 500/500/0 504/504/470) [1,10,7] r=1 lpr=504 pi=[498,504)/1 crt=383'839 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:41.188893 7f491c845700 1 osd.10 pg_epoch: 502 pg[13.1s1( v 377'9368 (355'1456,377'9368] local-lis/les=498/500 n=2661 ec=345/345 lis/c 498/498 les/c/f 500/501/0 502/502/472) [8,2147483647,0,2,6,7]p8(0) r=-1 lpr=502 pi=[498,502)/1 crt=377'9368 lcod 0'0 unknown NOTIFY] start_peering_interval up [8,10,0,2,6,7] -> [8,2147483647,0,2,6,7], acting [8,10,0,2,6,7] -> [8,2147483647,0,2,6,7], acting_primary 8(0) -> 8, up_primary 8(0) -> 8, role 1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.188979 7f491c044700 1 osd.10 pg_epoch: 502 pg[5.9( v 166'68275 (166'66775,166'68275] local-lis/les=498/499 n=68275 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,8,7] r=0 lpr=499 crt=166'68275 lcod 0'0 mlcod 0'0 peering] state<Started/Primary/Peering>: Peering, affected_by_map, going to Reset >2018-07-11 07:54:41.188980 7f491c845700 1 osd.10 pg_epoch: 504 pg[13.1s1( v 377'9368 (355'1456,377'9368] local-lis/les=498/500 n=2661 ec=345/345 lis/c 498/498 les/c/f 500/501/0 504/504/472) [8,10,0,2,6,7]p8(0) r=1 lpr=504 pi=[498,504)/1 crt=377'9368 lcod 0'0 unknown NOTIFY] start_peering_interval up [8,2147483647,0,2,6,7] -> [8,10,0,2,6,7], acting [8,2147483647,0,2,6,7] -> [8,10,0,2,6,7], acting_primary 8(0) -> 8, up_primary 8(0) -> 8, role -1 -> 1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.189084 7f491c044700 1 osd.10 pg_epoch: 502 pg[5.9( v 166'68275 (166'66775,166'68275] local-lis/les=498/499 n=68275 ec=71/71 lis/c 498/498 les/c/f 499/499/0 502/502/502) [8,7] r=-1 lpr=502 pi=[498,502)/1 crt=166'68275 lcod 0'0 unknown] start_peering_interval up [10,8,7] -> [8,7], acting [10,8,7] -> [8,7], acting_primary 10 -> 8, up_primary 10 -> 8, role 0 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.189082 7f491c845700 1 osd.10 pg_epoch: 504 pg[13.1s1( v 377'9368 (355'1456,377'9368] local-lis/les=498/500 n=2661 ec=345/345 lis/c 498/498 les/c/f 500/501/0 504/504/472) [8,10,0,2,6,7]p8(0) r=1 lpr=504 pi=[498,504)/1 crt=377'9368 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:41.189145 7f491c044700 1 osd.10 pg_epoch: 504 pg[5.9( v 166'68275 (166'66775,166'68275] local-lis/les=498/499 n=68275 ec=71/71 lis/c 498/498 les/c/f 499/499/0 504/504/504) [10,8,7] r=0 lpr=504 pi=[498,504)/1 crt=166'68275 lcod 0'0 mlcod 0'0 unknown NOTIFY] start_peering_interval up [8,7] -> [10,8,7], acting [8,7] -> [10,8,7], acting_primary 8 -> 10, up_primary 8 -> 10, role -1 -> 0, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.189214 7f491c044700 1 osd.10 pg_epoch: 504 pg[5.9( v 166'68275 (166'66775,166'68275] local-lis/les=498/499 n=68275 ec=71/71 lis/c 498/498 les/c/f 499/499/0 504/504/504) [10,8,7] r=0 lpr=504 pi=[498,504)/1 crt=166'68275 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 07:54:41.189228 7f491c845700 1 osd.10 pg_epoch: 502 pg[11.3s5( v 383'60232 (352'56579,383'60232] local-lis/les=498/500 n=60232 ec=267/267 lis/c 498/498 les/c/f 500/501/0 502/502/481) [6,8,0,11,2,2147483647]p6(0) r=-1 lpr=502 pi=[498,502)/1 crt=383'60232 lcod 0'0 unknown NOTIFY] start_peering_interval up [6,8,0,11,2,10] -> [6,8,0,11,2,2147483647], acting [6,8,0,11,2,10] -> [6,8,0,11,2,2147483647], acting_primary 6(0) -> 6, up_primary 6(0) -> 6, role 5 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.189342 7f491c845700 1 osd.10 pg_epoch: 504 pg[11.3s5( v 383'60232 (352'56579,383'60232] local-lis/les=498/500 n=60232 ec=267/267 lis/c 498/498 les/c/f 500/501/0 504/504/481) [6,8,0,11,2,10]p6(0) r=5 lpr=504 pi=[498,504)/1 crt=383'60232 lcod 0'0 unknown NOTIFY] start_peering_interval up [6,8,0,11,2,2147483647] -> [6,8,0,11,2,10], acting [6,8,0,11,2,2147483647] -> [6,8,0,11,2,10], acting_primary 6(0) -> 6, up_primary 6(0) -> 6, role -1 -> 5, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.189399 7f491c044700 1 osd.10 pg_epoch: 502 pg[6.a( v 430'109231 (430'107731,430'109231] local-lis/les=498/500 n=26267 ec=76/76 lis/c 498/498 les/c/f 500/500/0 502/502/467) [5,6] r=-1 lpr=502 pi=[498,502)/1 crt=430'109231 lcod 0'0 unknown NOTIFY] start_peering_interval up [5,10,6] -> [5,6], acting [5,10,6] -> [5,6], acting_primary 5 -> 5, up_primary 5 -> 5, role 1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.189428 7f491c845700 1 osd.10 pg_epoch: 504 pg[11.3s5( v 383'60232 (352'56579,383'60232] local-lis/les=498/500 n=60232 ec=267/267 lis/c 498/498 les/c/f 500/501/0 504/504/481) [6,8,0,11,2,10]p6(0) r=5 lpr=504 pi=[498,504)/1 crt=383'60232 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:41.189457 7f491c044700 1 osd.10 pg_epoch: 504 pg[6.a( v 430'109231 (430'107731,430'109231] local-lis/les=498/500 n=26267 ec=76/76 lis/c 498/498 les/c/f 500/500/0 504/504/467) [5,10,6] r=1 lpr=504 pi=[498,504)/1 crt=430'109231 lcod 0'0 unknown NOTIFY] start_peering_interval up [5,6] -> [5,10,6], acting [5,6] -> [5,10,6], acting_primary 5 -> 5, up_primary 5 -> 5, role -1 -> 1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.189527 7f491c044700 1 osd.10 pg_epoch: 504 pg[6.a( v 430'109231 (430'107731,430'109231] local-lis/les=498/500 n=26267 ec=76/76 lis/c 498/498 les/c/f 500/500/0 504/504/467) [5,10,6] r=1 lpr=504 pi=[498,504)/1 crt=430'109231 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:41.189529 7f491c845700 1 osd.10 pg_epoch: 502 pg[11.6s0( v 383'59732 (352'56233,383'59732] local-lis/les=498/499 n=59732 ec=267/267 lis/c 498/498 les/c/f 499/500/0 498/498/498) [10,1,11,6,5,8]p10(0) r=0 lpr=500 crt=383'59732 lcod 0'0 mlcod 0'0 peering] state<Started/Primary/Peering>: Peering, affected_by_map, going to Reset >2018-07-11 07:54:41.189639 7f491c845700 1 osd.10 pg_epoch: 502 pg[11.6s0( v 383'59732 (352'56233,383'59732] local-lis/les=498/499 n=59732 ec=267/267 lis/c 498/498 les/c/f 499/500/0 502/502/502) [2147483647,1,11,6,5,8]p1(1) r=-1 lpr=502 pi=[498,502)/1 crt=383'59732 lcod 0'0 unknown] start_peering_interval up [10,1,11,6,5,8] -> [2147483647,1,11,6,5,8], acting [10,1,11,6,5,8] -> [2147483647,1,11,6,5,8], acting_primary 10(0) -> 1, up_primary 10(0) -> 1, role 0 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.189662 7f491c044700 1 osd.10 pg_epoch: 502 pg[12.0( v 383'7619 (383'6093,383'7619] local-lis/les=498/500 n=7619 ec=331/331 lis/c 498/498 les/c/f 500/500/0 502/502/487) [4,3] r=-1 lpr=502 pi=[498,502)/1 crt=383'7619 lcod 0'0 unknown NOTIFY] start_peering_interval up [4,10,3] -> [4,3], acting [4,10,3] -> [4,3], acting_primary 4 -> 4, up_primary 4 -> 4, role 1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.189733 7f491c044700 1 osd.10 pg_epoch: 504 pg[12.0( v 383'7619 (383'6093,383'7619] local-lis/les=498/500 n=7619 ec=331/331 lis/c 498/498 les/c/f 500/500/0 504/504/487) [4,10,3] r=1 lpr=504 pi=[498,504)/1 crt=383'7619 lcod 0'0 unknown NOTIFY] start_peering_interval up [4,3] -> [4,10,3], acting [4,3] -> [4,10,3], acting_primary 4 -> 4, up_primary 4 -> 4, role -1 -> 1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.189738 7f491c845700 1 osd.10 pg_epoch: 504 pg[11.6s0( v 383'59732 (352'56233,383'59732] local-lis/les=498/499 n=59732 ec=267/267 lis/c 498/498 les/c/f 499/500/0 504/504/504) [10,1,11,6,5,8]p10(0) r=0 lpr=504 pi=[498,504)/1 crt=383'59732 lcod 0'0 mlcod 0'0 unknown NOTIFY] start_peering_interval up [2147483647,1,11,6,5,8] -> [10,1,11,6,5,8], acting [2147483647,1,11,6,5,8] -> [10,1,11,6,5,8], acting_primary 1(1) -> 10, up_primary 1(1) -> 10, role -1 -> 0, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.189803 7f491c044700 1 osd.10 pg_epoch: 504 pg[12.0( v 383'7619 (383'6093,383'7619] local-lis/les=498/500 n=7619 ec=331/331 lis/c 498/498 les/c/f 500/500/0 504/504/487) [4,10,3] r=1 lpr=504 pi=[498,504)/1 crt=383'7619 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:41.189832 7f491c845700 1 osd.10 pg_epoch: 504 pg[11.6s0( v 383'59732 (352'56233,383'59732] local-lis/les=498/499 n=59732 ec=267/267 lis/c 498/498 les/c/f 499/500/0 504/504/504) [10,1,11,6,5,8]p10(0) r=0 lpr=504 pi=[498,504)/1 crt=383'59732 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 07:54:41.189942 7f491c044700 1 osd.10 pg_epoch: 502 pg[5.4( v 166'68216 (166'66716,166'68216] local-lis/les=498/500 n=68216 ec=71/71 lis/c 498/498 les/c/f 500/500/0 502/502/468) [7,2] r=-1 lpr=502 pi=[498,502)/1 crt=166'68216 lcod 0'0 unknown NOTIFY] start_peering_interval up [7,2,10] -> [7,2], acting [7,2,10] -> [7,2], acting_primary 7 -> 7, up_primary 7 -> 7, role 2 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.189965 7f491c845700 1 osd.10 pg_epoch: 502 pg[6.13( v 430'99276 (430'97776,430'99276] local-lis/les=498/499 n=26101 ec=76/76 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,7,0] r=0 lpr=499 crt=430'99276 lcod 0'0 mlcod 0'0 peering] state<Started/Primary/Peering>: Peering, affected_by_map, going to Reset >2018-07-11 07:54:41.189998 7f491c044700 1 osd.10 pg_epoch: 504 pg[5.4( v 166'68216 (166'66716,166'68216] local-lis/les=498/500 n=68216 ec=71/71 lis/c 498/498 les/c/f 500/500/0 504/504/468) [7,2,10] r=2 lpr=504 pi=[498,504)/1 crt=166'68216 lcod 0'0 unknown NOTIFY] start_peering_interval up [7,2] -> [7,2,10], acting [7,2] -> [7,2,10], acting_primary 7 -> 7, up_primary 7 -> 7, role -1 -> 2, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.190068 7f491c044700 1 osd.10 pg_epoch: 504 pg[5.4( v 166'68216 (166'66716,166'68216] local-lis/les=498/500 n=68216 ec=71/71 lis/c 498/498 les/c/f 500/500/0 504/504/468) [7,2,10] r=2 lpr=504 pi=[498,504)/1 crt=166'68216 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:41.190069 7f491c845700 1 osd.10 pg_epoch: 502 pg[6.13( v 430'99276 (430'97776,430'99276] local-lis/les=498/499 n=26101 ec=76/76 lis/c 498/498 les/c/f 499/499/0 502/502/502) [7,0] r=-1 lpr=502 pi=[498,502)/1 crt=430'99276 lcod 0'0 unknown] start_peering_interval up [10,7,0] -> [7,0], acting [10,7,0] -> [7,0], acting_primary 10 -> 7, up_primary 10 -> 7, role 0 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.190127 7f491c845700 1 osd.10 pg_epoch: 504 pg[6.13( v 430'99276 (430'97776,430'99276] local-lis/les=498/499 n=26101 ec=76/76 lis/c 498/498 les/c/f 499/499/0 504/504/504) [10,7,0] r=0 lpr=504 pi=[498,504)/1 crt=430'99276 lcod 0'0 mlcod 0'0 unknown NOTIFY] start_peering_interval up [7,0] -> [10,7,0], acting [7,0] -> [10,7,0], acting_primary 7 -> 10, up_primary 7 -> 10, role -1 -> 0, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.190144 7f491c044700 1 osd.10 pg_epoch: 502 pg[6.7( v 430'109641 (430'108141,430'109641] local-lis/les=498/499 n=25825 ec=76/76 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,6,3] r=0 lpr=499 crt=430'109641 lcod 0'0 mlcod 0'0 peering] state<Started/Primary/Peering>: Peering, affected_by_map, going to Reset >2018-07-11 07:54:41.190200 7f491c845700 1 osd.10 pg_epoch: 504 pg[6.13( v 430'99276 (430'97776,430'99276] local-lis/les=498/499 n=26101 ec=76/76 lis/c 498/498 les/c/f 499/499/0 504/504/504) [10,7,0] r=0 lpr=504 pi=[498,504)/1 crt=430'99276 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 07:54:41.190262 7f491c044700 1 osd.10 pg_epoch: 502 pg[6.7( v 430'109641 (430'108141,430'109641] local-lis/les=498/499 n=25825 ec=76/76 lis/c 498/498 les/c/f 499/499/0 502/502/502) [6,3] r=-1 lpr=502 pi=[498,502)/1 crt=430'109641 lcod 0'0 unknown] start_peering_interval up [10,6,3] -> [6,3], acting [10,6,3] -> [6,3], acting_primary 10 -> 6, up_primary 10 -> 6, role 0 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.190322 7f491c044700 1 osd.10 pg_epoch: 504 pg[6.7( v 430'109641 (430'108141,430'109641] local-lis/les=498/499 n=25825 ec=76/76 lis/c 498/498 les/c/f 499/499/0 504/504/504) [10,6,3] r=0 lpr=504 pi=[498,504)/1 crt=430'109641 lcod 0'0 mlcod 0'0 unknown NOTIFY] start_peering_interval up [6,3] -> [10,6,3], acting [6,3] -> [10,6,3], acting_primary 6 -> 10, up_primary 6 -> 10, role -1 -> 0, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.190393 7f491c044700 1 osd.10 pg_epoch: 504 pg[6.7( v 430'109641 (430'108141,430'109641] local-lis/les=498/499 n=25825 ec=76/76 lis/c 498/498 les/c/f 499/499/0 504/504/504) [10,6,3] r=0 lpr=504 pi=[498,504)/1 crt=430'109641 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 07:54:41.190393 7f491c845700 1 osd.10 pg_epoch: 502 pg[9.1c( v 494'483 (0'0,494'483] local-lis/les=498/500 n=238 ec=190/190 lis/c 498/498 les/c/f 500/500/0 502/502/470) [1,7] r=-1 lpr=502 pi=[498,502)/1 crt=494'483 lcod 0'0 unknown NOTIFY] start_peering_interval up [1,10,7] -> [1,7], acting [1,10,7] -> [1,7], acting_primary 1 -> 1, up_primary 1 -> 1, role 1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.190453 7f491c845700 1 osd.10 pg_epoch: 504 pg[9.1c( v 494'483 (0'0,494'483] local-lis/les=498/500 n=238 ec=190/190 lis/c 498/498 les/c/f 500/500/0 504/504/470) [1,10,7] r=1 lpr=504 pi=[498,504)/1 crt=494'483 lcod 0'0 unknown NOTIFY] start_peering_interval up [1,7] -> [1,10,7], acting [1,7] -> [1,10,7], acting_primary 1 -> 1, up_primary 1 -> 1, role -1 -> 1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.190501 7f491c044700 1 osd.10 pg_epoch: 502 pg[5.d( v 166'68209 (166'66709,166'68209] local-lis/les=498/499 n=68209 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,7,0] r=0 lpr=499 crt=166'68209 lcod 0'0 mlcod 0'0 peering] state<Started/Primary/Peering>: Peering, affected_by_map, going to Reset >2018-07-11 07:54:41.190521 7f491c845700 1 osd.10 pg_epoch: 504 pg[9.1c( v 494'483 (0'0,494'483] local-lis/les=498/500 n=238 ec=190/190 lis/c 498/498 les/c/f 500/500/0 504/504/470) [1,10,7] r=1 lpr=504 pi=[498,504)/1 crt=494'483 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:41.190592 7f491c044700 1 osd.10 pg_epoch: 502 pg[5.d( v 166'68209 (166'66709,166'68209] local-lis/les=498/499 n=68209 ec=71/71 lis/c 498/498 les/c/f 499/499/0 502/502/502) [7,0] r=-1 lpr=502 pi=[498,502)/1 crt=166'68209 lcod 0'0 unknown] start_peering_interval up [10,7,0] -> [7,0], acting [10,7,0] -> [7,0], acting_primary 10 -> 7, up_primary 10 -> 7, role 0 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.190604 7f491c845700 1 osd.10 pg_epoch: 502 pg[9.1b( v 463'472 (0'0,463'472] local-lis/les=498/499 n=256 ec=190/190 lis/c 498/498 les/c/f 499/500/0 498/498/498) [10,1,5] r=0 lpr=500 crt=463'472 lcod 0'0 mlcod 0'0 peering] state<Started/Primary/Peering>: Peering, affected_by_map, going to Reset >2018-07-11 07:54:41.190650 7f491c044700 1 osd.10 pg_epoch: 504 pg[5.d( v 166'68209 (166'66709,166'68209] local-lis/les=498/499 n=68209 ec=71/71 lis/c 498/498 les/c/f 499/499/0 504/504/504) [10,7,0] r=0 lpr=504 pi=[498,504)/1 crt=166'68209 lcod 0'0 mlcod 0'0 unknown NOTIFY] start_peering_interval up [7,0] -> [10,7,0], acting [7,0] -> [10,7,0], acting_primary 7 -> 10, up_primary 7 -> 10, role -1 -> 0, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.190698 7f491c845700 1 osd.10 pg_epoch: 502 pg[9.1b( v 463'472 (0'0,463'472] local-lis/les=498/499 n=256 ec=190/190 lis/c 498/498 les/c/f 499/500/0 502/502/502) [1,5] r=-1 lpr=502 pi=[498,502)/1 crt=463'472 lcod 0'0 unknown] start_peering_interval up [10,1,5] -> [1,5], acting [10,1,5] -> [1,5], acting_primary 10 -> 1, up_primary 10 -> 1, role 0 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.190715 7f491c044700 1 osd.10 pg_epoch: 504 pg[5.d( v 166'68209 (166'66709,166'68209] local-lis/les=498/499 n=68209 ec=71/71 lis/c 498/498 les/c/f 499/499/0 504/504/504) [10,7,0] r=0 lpr=504 pi=[498,504)/1 crt=166'68209 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 07:54:41.190755 7f491c845700 1 osd.10 pg_epoch: 504 pg[9.1b( v 463'472 (0'0,463'472] local-lis/les=498/499 n=256 ec=190/190 lis/c 498/498 les/c/f 499/500/0 504/504/504) [10,1,5] r=0 lpr=504 pi=[498,504)/1 crt=463'472 lcod 0'0 mlcod 0'0 unknown NOTIFY] start_peering_interval up [1,5] -> [10,1,5], acting [1,5] -> [10,1,5], acting_primary 1 -> 10, up_primary 1 -> 10, role -1 -> 0, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.190824 7f491c845700 1 osd.10 pg_epoch: 504 pg[9.1b( v 463'472 (0'0,463'472] local-lis/les=498/499 n=256 ec=190/190 lis/c 498/498 les/c/f 499/500/0 504/504/504) [10,1,5] r=0 lpr=504 pi=[498,504)/1 crt=463'472 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 07:54:41.190873 7f491c044700 1 osd.10 pg_epoch: 502 pg[6.e( v 430'116090 (430'114517,430'116090] local-lis/les=498/500 n=25967 ec=76/76 lis/c 498/498 les/c/f 500/500/0 502/502/487) [4,1] r=-1 lpr=502 pi=[498,502)/1 crt=430'116090 lcod 0'0 unknown NOTIFY] start_peering_interval up [4,1,10] -> [4,1], acting [4,1,10] -> [4,1], acting_primary 4 -> 4, up_primary 4 -> 4, role 2 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.190935 7f491c845700 1 osd.10 pg_epoch: 502 pg[6.14( v 461'122884 (430'121291,461'122884] local-lis/les=498/499 n=25983 ec=76/76 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,4,3] r=0 lpr=499 crt=461'122884 lcod 0'0 mlcod 0'0 peering] state<Started/Primary/Peering>: Peering, affected_by_map, going to Reset >2018-07-11 07:54:41.190935 7f491c044700 1 osd.10 pg_epoch: 504 pg[6.e( v 430'116090 (430'114517,430'116090] local-lis/les=498/500 n=25967 ec=76/76 lis/c 498/498 les/c/f 500/500/0 504/504/487) [4,1,10] r=2 lpr=504 pi=[498,504)/1 crt=430'116090 lcod 0'0 unknown NOTIFY] start_peering_interval up [4,1] -> [4,1,10], acting [4,1] -> [4,1,10], acting_primary 4 -> 4, up_primary 4 -> 4, role -1 -> 2, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.191012 7f491c044700 1 osd.10 pg_epoch: 504 pg[6.e( v 430'116090 (430'114517,430'116090] local-lis/les=498/500 n=25967 ec=76/76 lis/c 498/498 les/c/f 500/500/0 504/504/487) [4,1,10] r=2 lpr=504 pi=[498,504)/1 crt=430'116090 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:41.191029 7f491c845700 1 osd.10 pg_epoch: 502 pg[6.14( v 461'122884 (430'121291,461'122884] local-lis/les=498/499 n=25983 ec=76/76 lis/c 498/498 les/c/f 499/499/0 502/502/502) [4,3] r=-1 lpr=502 pi=[498,502)/1 crt=461'122884 lcod 0'0 unknown] start_peering_interval up [10,4,3] -> [4,3], acting [10,4,3] -> [4,3], acting_primary 10 -> 4, up_primary 10 -> 4, role 0 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.191091 7f491c845700 1 osd.10 pg_epoch: 504 pg[6.14( v 461'122884 (430'121291,461'122884] local-lis/les=498/499 n=25983 ec=76/76 lis/c 498/498 les/c/f 499/499/0 504/504/504) [10,4,3] r=0 lpr=504 pi=[498,504)/1 crt=461'122884 lcod 0'0 mlcod 0'0 unknown NOTIFY] start_peering_interval up [4,3] -> [10,4,3], acting [4,3] -> [10,4,3], acting_primary 4 -> 10, up_primary 4 -> 10, role -1 -> 0, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.191163 7f491c845700 1 osd.10 pg_epoch: 504 pg[6.14( v 461'122884 (430'121291,461'122884] local-lis/les=498/499 n=25983 ec=76/76 lis/c 498/498 les/c/f 499/499/0 504/504/504) [10,4,3] r=0 lpr=504 pi=[498,504)/1 crt=461'122884 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 07:54:41.191180 7f491c044700 1 osd.10 pg_epoch: 502 pg[11.8s3( v 383'59629 (352'55946,383'59629] local-lis/les=498/500 n=59629 ec=267/267 lis/c 498/498 les/c/f 500/500/0 502/502/468) [7,1,6,2147483647,5,8]p7(0) r=-1 lpr=502 pi=[498,502)/1 crt=383'59629 lcod 0'0 unknown NOTIFY] start_peering_interval up [7,1,6,10,5,8] -> [7,1,6,2147483647,5,8], acting [7,1,6,10,5,8] -> [7,1,6,2147483647,5,8], acting_primary 7(0) -> 7, up_primary 7(0) -> 7, role 3 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.191264 7f491c044700 1 osd.10 pg_epoch: 504 pg[11.8s3( v 383'59629 (352'55946,383'59629] local-lis/les=498/500 n=59629 ec=267/267 lis/c 498/498 les/c/f 500/500/0 504/504/468) [7,1,6,10,5,8]p7(0) r=3 lpr=504 pi=[498,504)/1 crt=383'59629 lcod 0'0 unknown NOTIFY] start_peering_interval up [7,1,6,2147483647,5,8] -> [7,1,6,10,5,8], acting [7,1,6,2147483647,5,8] -> [7,1,6,10,5,8], acting_primary 7(0) -> 7, up_primary 7(0) -> 7, role -1 -> 3, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.191333 7f491c845700 1 osd.10 pg_epoch: 502 pg[5.14( v 166'68424 (166'66924,166'68424] local-lis/les=498/500 n=68424 ec=71/71 lis/c 498/498 les/c/f 500/500/0 502/502/468) [7,2] r=-1 lpr=502 pi=[498,502)/1 crt=166'68424 lcod 0'0 unknown NOTIFY] start_peering_interval up [7,10,2] -> [7,2], acting [7,10,2] -> [7,2], acting_primary 7 -> 7, up_primary 7 -> 7, role 1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.191341 7f491c044700 1 osd.10 pg_epoch: 504 pg[11.8s3( v 383'59629 (352'55946,383'59629] local-lis/les=498/500 n=59629 ec=267/267 lis/c 498/498 les/c/f 500/500/0 504/504/468) [7,1,6,10,5,8]p7(0) r=3 lpr=504 pi=[498,504)/1 crt=383'59629 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:41.191389 7f491c845700 1 osd.10 pg_epoch: 504 pg[5.14( v 166'68424 (166'66924,166'68424] local-lis/les=498/500 n=68424 ec=71/71 lis/c 498/498 les/c/f 500/500/0 504/504/468) [7,10,2] r=1 lpr=504 pi=[498,504)/1 crt=166'68424 lcod 0'0 unknown NOTIFY] start_peering_interval up [7,2] -> [7,10,2], acting [7,2] -> [7,10,2], acting_primary 7 -> 7, up_primary 7 -> 7, role -1 -> 1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.191464 7f491c845700 1 osd.10 pg_epoch: 504 pg[5.14( v 166'68424 (166'66924,166'68424] local-lis/les=498/500 n=68424 ec=71/71 lis/c 498/498 les/c/f 500/500/0 504/504/468) [7,10,2] r=1 lpr=504 pi=[498,504)/1 crt=166'68424 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:41.191498 7f491c044700 1 osd.10 pg_epoch: 502 pg[13.8s5( v 377'9578 (367'1954,377'9578] local-lis/les=498/500 n=2631 ec=345/345 lis/c 498/498 les/c/f 500/500/0 502/502/465) [2,11,5,8,6,2147483647]p2(0) r=-1 lpr=502 pi=[498,502)/1 crt=377'9578 lcod 0'0 unknown NOTIFY] start_peering_interval up [2,11,5,8,6,10] -> [2,11,5,8,6,2147483647], acting [2,11,5,8,6,10] -> [2,11,5,8,6,2147483647], acting_primary 2(0) -> 2, up_primary 2(0) -> 2, role 5 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.191577 7f491c044700 1 osd.10 pg_epoch: 504 pg[13.8s5( v 377'9578 (367'1954,377'9578] local-lis/les=498/500 n=2631 ec=345/345 lis/c 498/498 les/c/f 500/500/0 504/504/465) [2,11,5,8,6,10]p2(0) r=5 lpr=504 pi=[498,504)/1 crt=377'9578 lcod 0'0 unknown NOTIFY] start_peering_interval up [2,11,5,8,6,2147483647] -> [2,11,5,8,6,10], acting [2,11,5,8,6,2147483647] -> [2,11,5,8,6,10], acting_primary 2(0) -> 2, up_primary 2(0) -> 2, role -1 -> 5, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.191651 7f491c044700 1 osd.10 pg_epoch: 504 pg[13.8s5( v 377'9578 (367'1954,377'9578] local-lis/les=498/500 n=2631 ec=345/345 lis/c 498/498 les/c/f 500/500/0 504/504/465) [2,11,5,8,6,10]p2(0) r=5 lpr=504 pi=[498,504)/1 crt=377'9578 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:41.191732 7f491c044700 1 osd.10 pg_epoch: 502 pg[6.d( v 461'110631 (430'109050,461'110631] local-lis/les=498/499 n=25732 ec=76/76 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,7,2] r=0 lpr=499 crt=461'110631 lcod 0'0 mlcod 0'0 peering] state<Started/Primary/Peering>: Peering, affected_by_map, going to Reset >2018-07-11 07:54:41.191840 7f491c044700 1 osd.10 pg_epoch: 502 pg[6.d( v 461'110631 (430'109050,461'110631] local-lis/les=498/499 n=25732 ec=76/76 lis/c 498/498 les/c/f 499/499/0 502/502/502) [7,2] r=-1 lpr=502 pi=[498,502)/1 crt=461'110631 lcod 0'0 unknown] start_peering_interval up [10,7,2] -> [7,2], acting [10,7,2] -> [7,2], acting_primary 10 -> 7, up_primary 10 -> 7, role 0 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.191903 7f491c044700 1 osd.10 pg_epoch: 504 pg[6.d( v 461'110631 (430'109050,461'110631] local-lis/les=498/499 n=25732 ec=76/76 lis/c 498/498 les/c/f 499/499/0 504/504/504) [10,7,2] r=0 lpr=504 pi=[498,504)/1 crt=461'110631 lcod 0'0 mlcod 0'0 unknown NOTIFY] start_peering_interval up [7,2] -> [10,7,2], acting [7,2] -> [10,7,2], acting_primary 7 -> 10, up_primary 7 -> 10, role -1 -> 0, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.191954 7f491c845700 1 osd.10 pg_epoch: 502 pg[6.2( v 430'110237 (430'108737,430'110237] local-lis/les=498/500 n=25907 ec=76/76 lis/c 498/498 les/c/f 500/500/0 502/502/468) [7,2] r=-1 lpr=502 pi=[498,502)/1 crt=430'110237 lcod 0'0 unknown NOTIFY] start_peering_interval up [7,10,2] -> [7,2], acting [7,10,2] -> [7,2], acting_primary 7 -> 7, up_primary 7 -> 7, role 1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.192009 7f491c044700 1 osd.10 pg_epoch: 504 pg[6.d( v 461'110631 (430'109050,461'110631] local-lis/les=498/499 n=25732 ec=76/76 lis/c 498/498 les/c/f 499/499/0 504/504/504) [10,7,2] r=0 lpr=504 pi=[498,504)/1 crt=461'110631 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 07:54:41.192016 7f491c845700 1 osd.10 pg_epoch: 504 pg[6.2( v 430'110237 (430'108737,430'110237] local-lis/les=498/500 n=25907 ec=76/76 lis/c 498/498 les/c/f 500/500/0 504/504/468) [7,10,2] r=1 lpr=504 pi=[498,504)/1 crt=430'110237 lcod 0'0 unknown NOTIFY] start_peering_interval up [7,2] -> [7,10,2], acting [7,2] -> [7,10,2], acting_primary 7 -> 7, up_primary 7 -> 7, role -1 -> 1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.192143 7f491c845700 1 osd.10 pg_epoch: 504 pg[6.2( v 430'110237 (430'108737,430'110237] local-lis/les=498/500 n=25907 ec=76/76 lis/c 498/498 les/c/f 500/500/0 504/504/468) [7,10,2] r=1 lpr=504 pi=[498,504)/1 crt=430'110237 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:41.192260 7f491c044700 1 osd.10 pg_epoch: 502 pg[9.d( v 494'450 (0'0,494'450] local-lis/les=498/499 n=228 ec=190/190 lis/c 498/498 les/c/f 499/499/0 502/502/470) [0,6] r=-1 lpr=502 pi=[498,502)/1 crt=494'450 lcod 0'0 unknown NOTIFY] start_peering_interval up [0,6,10] -> [0,6], acting [0,6,10] -> [0,6], acting_primary 0 -> 0, up_primary 0 -> 0, role 2 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.192343 7f491c845700 1 osd.10 pg_epoch: 502 pg[11.as3( v 383'60057 (367'56547,383'60057] local-lis/les=498/499 n=60057 ec=267/267 lis/c 498/498 les/c/f 499/499/0 502/502/484) [3,5,8,2147483647,11,6]p3(0) r=-1 lpr=502 pi=[498,502)/1 crt=383'60057 lcod 0'0 unknown NOTIFY] start_peering_interval up [3,5,8,10,11,6] -> [3,5,8,2147483647,11,6], acting [3,5,8,10,11,6] -> [3,5,8,2147483647,11,6], acting_primary 3(0) -> 3, up_primary 3(0) -> 3, role 3 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.192366 7f491c044700 1 osd.10 pg_epoch: 504 pg[9.d( v 494'450 (0'0,494'450] local-lis/les=498/499 n=228 ec=190/190 lis/c 498/498 les/c/f 499/499/0 504/504/470) [0,6,10] r=2 lpr=504 pi=[498,504)/1 crt=494'450 lcod 0'0 unknown NOTIFY] start_peering_interval up [0,6] -> [0,6,10], acting [0,6] -> [0,6,10], acting_primary 0 -> 0, up_primary 0 -> 0, role -1 -> 2, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.192429 7f491c845700 1 osd.10 pg_epoch: 504 pg[11.as3( v 383'60057 (367'56547,383'60057] local-lis/les=498/499 n=60057 ec=267/267 lis/c 498/498 les/c/f 499/499/0 504/504/484) [3,5,8,10,11,6]p3(0) r=3 lpr=504 pi=[498,504)/1 crt=383'60057 lcod 0'0 unknown NOTIFY] start_peering_interval up [3,5,8,2147483647,11,6] -> [3,5,8,10,11,6], acting [3,5,8,2147483647,11,6] -> [3,5,8,10,11,6], acting_primary 3(0) -> 3, up_primary 3(0) -> 3, role -1 -> 3, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.192481 7f491c044700 1 osd.10 pg_epoch: 504 pg[9.d( v 494'450 (0'0,494'450] local-lis/les=498/499 n=228 ec=190/190 lis/c 498/498 les/c/f 499/499/0 504/504/470) [0,6,10] r=2 lpr=504 pi=[498,504)/1 crt=494'450 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:41.192517 7f491c845700 1 osd.10 pg_epoch: 504 pg[11.as3( v 383'60057 (367'56547,383'60057] local-lis/les=498/499 n=60057 ec=267/267 lis/c 498/498 les/c/f 499/499/0 504/504/484) [3,5,8,10,11,6]p3(0) r=3 lpr=504 pi=[498,504)/1 crt=383'60057 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:41.192678 7f491c845700 1 osd.10 pg_epoch: 502 pg[13.bs4( v 377'9782 (355'1491,377'9782] local-lis/les=498/500 n=2694 ec=345/345 lis/c 498/498 les/c/f 500/500/0 502/502/478) [11,8,1,4,2147483647,6]p11(0) r=-1 lpr=502 pi=[498,502)/1 crt=377'9782 lcod 0'0 unknown NOTIFY] start_peering_interval up [11,8,1,4,10,6] -> [11,8,1,4,2147483647,6], acting [11,8,1,4,10,6] -> [11,8,1,4,2147483647,6], acting_primary 11(0) -> 11, up_primary 11(0) -> 11, role 4 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.192790 7f491c845700 1 osd.10 pg_epoch: 504 pg[13.bs4( v 377'9782 (355'1491,377'9782] local-lis/les=498/500 n=2694 ec=345/345 lis/c 498/498 les/c/f 500/500/0 504/504/478) [11,8,1,4,10,6]p11(0) r=4 lpr=504 pi=[498,504)/1 crt=377'9782 lcod 0'0 unknown NOTIFY] start_peering_interval up [11,8,1,4,2147483647,6] -> [11,8,1,4,10,6], acting [11,8,1,4,2147483647,6] -> [11,8,1,4,10,6], acting_primary 11(0) -> 11, up_primary 11(0) -> 11, role -1 -> 4, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.192923 7f491c845700 1 osd.10 pg_epoch: 504 pg[13.bs4( v 377'9782 (355'1491,377'9782] local-lis/les=498/500 n=2694 ec=345/345 lis/c 498/498 les/c/f 500/500/0 504/504/478) [11,8,1,4,10,6]p11(0) r=4 lpr=504 pi=[498,504)/1 crt=377'9782 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:41.193075 7f491c044700 1 osd.10 pg_epoch: 502 pg[13.5s2( v 377'9560 (0'0,377'9560] local-lis/les=500/501 n=2691 ec=345/345 lis/c 500/500 les/c/f 501/501/0 502/502/487) [4,3,2147483647,11,2147483647,8]/[4,3,2147483647,11,8,8]p4(0) r=-1 lpr=502 pi=[500,502)/1 crt=377'9552 lcod 0'0 remapped NOTIFY] start_peering_interval up [4,3,10,11,2147483647,8] -> [4,3,2147483647,11,2147483647,8], acting [4,3,10,11,8,8] -> [4,3,2147483647,11,8,8], acting_primary 4(0) -> 4, up_primary 4(0) -> 4, role 2 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.193155 7f491c845700 1 osd.10 pg_epoch: 502 pg[6.26( v 461'123787 (430'122194,461'123787] local-lis/les=498/500 n=26121 ec=76/76 lis/c 498/498 les/c/f 500/500/0 502/502/468) [7,8] r=-1 lpr=502 pi=[498,502)/1 crt=461'123787 lcod 0'0 unknown NOTIFY] start_peering_interval up [7,10,8] -> [7,8], acting [7,10,8] -> [7,8], acting_primary 7 -> 7, up_primary 7 -> 7, role 1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.193239 7f491c845700 1 osd.10 pg_epoch: 504 pg[6.26( v 461'123787 (430'122194,461'123787] local-lis/les=498/500 n=26121 ec=76/76 lis/c 498/498 les/c/f 500/500/0 504/504/468) [7,10,8] r=1 lpr=504 pi=[498,504)/1 crt=461'123787 lcod 0'0 unknown NOTIFY] start_peering_interval up [7,8] -> [7,10,8], acting [7,8] -> [7,10,8], acting_primary 7 -> 7, up_primary 7 -> 7, role -1 -> 1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.193323 7f491c845700 1 osd.10 pg_epoch: 504 pg[6.26( v 461'123787 (430'122194,461'123787] local-lis/les=498/500 n=26121 ec=76/76 lis/c 498/498 les/c/f 500/500/0 504/504/468) [7,10,8] r=1 lpr=504 pi=[498,504)/1 crt=461'123787 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:41.193359 7f491c044700 1 osd.10 pg_epoch: 504 pg[13.5s2( v 377'9560 (0'0,377'9560] local-lis/les=500/501 n=2691 ec=345/345 lis/c 500/500 les/c/f 501/501/0 504/504/487) [4,3,10,11,2147483647,8]/[4,3,2147483647,11,8,8]p4(0) r=-1 lpr=504 pi=[500,504)/1 crt=377'9552 lcod 0'0 remapped NOTIFY] start_peering_interval up [4,3,2147483647,11,2147483647,8] -> [4,3,10,11,2147483647,8], acting [4,3,2147483647,11,8,8] -> [4,3,2147483647,11,8,8], acting_primary 4(0) -> 4, up_primary 4(0) -> 4, role -1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.193503 7f491c845700 1 osd.10 pg_epoch: 502 pg[13.9s1( v 377'10260 (352'219,377'10260] local-lis/les=498/500 n=2727 ec=345/345 lis/c 498/498 les/c/f 500/500/0 502/502/467) [5,2147483647,3,6,8,11]p5(0) r=-1 lpr=502 pi=[498,502)/1 crt=377'10254 lcod 0'0 unknown NOTIFY] start_peering_interval up [5,10,3,6,8,11] -> [5,2147483647,3,6,8,11], acting [5,10,3,6,8,11] -> [5,2147483647,3,6,8,11], acting_primary 5(0) -> 5, up_primary 5(0) -> 5, role 1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.193509 7f491c044700 1 osd.10 pg_epoch: 504 pg[13.5s2( v 377'9560 (0'0,377'9560] local-lis/les=500/501 n=2691 ec=345/345 lis/c 500/500 les/c/f 501/501/0 504/504/487) [4,3,10,11,2147483647,8]/[4,3,2147483647,11,8,8]p4(0) r=-1 lpr=504 pi=[500,504)/1 crt=377'9552 lcod 0'0 remapped NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:41.193576 7f491c845700 1 osd.10 pg_epoch: 504 pg[13.9s1( v 377'10260 (352'219,377'10260] local-lis/les=498/500 n=2727 ec=345/345 lis/c 498/498 les/c/f 500/500/0 504/504/467) [5,10,3,6,8,11]p5(0) r=1 lpr=504 pi=[498,504)/1 crt=377'10254 lcod 0'0 unknown NOTIFY] start_peering_interval up [5,2147483647,3,6,8,11] -> [5,10,3,6,8,11], acting [5,2147483647,3,6,8,11] -> [5,10,3,6,8,11], acting_primary 5(0) -> 5, up_primary 5(0) -> 5, role -1 -> 1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.193656 7f491c845700 1 osd.10 pg_epoch: 504 pg[13.9s1( v 377'10260 (352'219,377'10260] local-lis/les=498/500 n=2727 ec=345/345 lis/c 498/498 les/c/f 500/500/0 504/504/467) [5,10,3,6,8,11]p5(0) r=1 lpr=504 pi=[498,504)/1 crt=377'10254 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:41.193677 7f491c044700 1 osd.10 pg_epoch: 502 pg[13.4s3( v 377'9519 (355'1503,377'9519] local-lis/les=498/500 n=2672 ec=345/345 lis/c 498/498 les/c/f 500/500/0 502/502/467) [5,1,8,2147483647,11,6]p5(0) r=-1 lpr=502 pi=[498,502)/1 crt=377'9519 lcod 0'0 unknown NOTIFY] start_peering_interval up [5,1,8,10,11,6] -> [5,1,8,2147483647,11,6], acting [5,1,8,10,11,6] -> [5,1,8,2147483647,11,6], acting_primary 5(0) -> 5, up_primary 5(0) -> 5, role 3 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.193753 7f491c044700 1 osd.10 pg_epoch: 504 pg[13.4s3( v 377'9519 (355'1503,377'9519] local-lis/les=498/500 n=2672 ec=345/345 lis/c 498/498 les/c/f 500/500/0 504/504/467) [5,1,8,10,11,6]p5(0) r=3 lpr=504 pi=[498,504)/1 crt=377'9519 lcod 0'0 unknown NOTIFY] start_peering_interval up [5,1,8,2147483647,11,6] -> [5,1,8,10,11,6], acting [5,1,8,2147483647,11,6] -> [5,1,8,10,11,6], acting_primary 5(0) -> 5, up_primary 5(0) -> 5, role -1 -> 3, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.193826 7f491c044700 1 osd.10 pg_epoch: 504 pg[13.4s3( v 377'9519 (355'1503,377'9519] local-lis/les=498/500 n=2672 ec=345/345 lis/c 498/498 les/c/f 500/500/0 504/504/467) [5,1,8,10,11,6]p5(0) r=3 lpr=504 pi=[498,504)/1 crt=377'9519 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:41.193827 7f491c845700 1 osd.10 pg_epoch: 502 pg[11.ds3( v 383'59865 (367'56373,383'59865] local-lis/les=498/500 n=59865 ec=267/267 lis/c 498/498 les/c/f 500/500/0 502/502/465) [2,4,11,2147483647,8,6]p2(0) r=-1 lpr=502 pi=[498,502)/1 crt=383'59865 lcod 0'0 unknown NOTIFY] start_peering_interval up [2,4,11,10,8,6] -> [2,4,11,2147483647,8,6], acting [2,4,11,10,8,6] -> [2,4,11,2147483647,8,6], acting_primary 2(0) -> 2, up_primary 2(0) -> 2, role 3 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.193914 7f491c845700 1 osd.10 pg_epoch: 504 pg[11.ds3( v 383'59865 (367'56373,383'59865] local-lis/les=498/500 n=59865 ec=267/267 lis/c 498/498 les/c/f 500/500/0 504/504/465) [2,4,11,10,8,6]p2(0) r=3 lpr=504 pi=[498,504)/1 crt=383'59865 lcod 0'0 unknown NOTIFY] start_peering_interval up [2,4,11,2147483647,8,6] -> [2,4,11,10,8,6], acting [2,4,11,2147483647,8,6] -> [2,4,11,10,8,6], acting_primary 2(0) -> 2, up_primary 2(0) -> 2, role -1 -> 3, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.193963 7f491c044700 1 osd.10 pg_epoch: 502 pg[5.2f( v 166'67864 (166'66267,166'67864] local-lis/les=498/500 n=67864 ec=71/71 lis/c 498/498 les/c/f 500/500/0 502/502/472) [8,4] r=-1 lpr=502 pi=[498,502)/1 crt=166'67864 lcod 0'0 unknown NOTIFY] start_peering_interval up [8,10,4] -> [8,4], acting [8,10,4] -> [8,4], acting_primary 8 -> 8, up_primary 8 -> 8, role 1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.193987 7f491c845700 1 osd.10 pg_epoch: 504 pg[11.ds3( v 383'59865 (367'56373,383'59865] local-lis/les=498/500 n=59865 ec=267/267 lis/c 498/498 les/c/f 500/500/0 504/504/465) [2,4,11,10,8,6]p2(0) r=3 lpr=504 pi=[498,504)/1 crt=383'59865 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:41.194019 7f491c044700 1 osd.10 pg_epoch: 504 pg[5.2f( v 166'67864 (166'66267,166'67864] local-lis/les=498/500 n=67864 ec=71/71 lis/c 498/498 les/c/f 500/500/0 504/504/472) [8,10,4] r=1 lpr=504 pi=[498,504)/1 crt=166'67864 lcod 0'0 unknown NOTIFY] start_peering_interval up [8,4] -> [8,10,4], acting [8,4] -> [8,10,4], acting_primary 8 -> 8, up_primary 8 -> 8, role -1 -> 1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.194087 7f491c044700 1 osd.10 pg_epoch: 504 pg[5.2f( v 166'67864 (166'66267,166'67864] local-lis/les=498/500 n=67864 ec=71/71 lis/c 498/498 les/c/f 500/500/0 504/504/472) [8,10,4] r=1 lpr=504 pi=[498,504)/1 crt=166'67864 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:41.194122 7f491c845700 1 osd.10 pg_epoch: 502 pg[16.d( empty local-lis/les=498/499 n=0 ec=442/442 lis/c 498/498 les/c/f 499/499/0 502/502/470) [0,3] r=-1 lpr=502 pi=[498,502)/1 crt=0'0 unknown NOTIFY] start_peering_interval up [0,3,10] -> [0,3], acting [0,3,10] -> [0,3], acting_primary 0 -> 0, up_primary 0 -> 0, role 2 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.194181 7f491c845700 1 osd.10 pg_epoch: 504 pg[16.d( empty local-lis/les=498/499 n=0 ec=442/442 lis/c 498/498 les/c/f 499/499/0 504/504/470) [0,3,10] r=2 lpr=504 pi=[498,504)/1 crt=0'0 unknown NOTIFY] start_peering_interval up [0,3] -> [0,3,10], acting [0,3] -> [0,3,10], acting_primary 0 -> 0, up_primary 0 -> 0, role -1 -> 2, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.194220 7f491c044700 1 osd.10 pg_epoch: 502 pg[6.2c( v 457'120217 (430'118627,457'120217] local-lis/les=498/500 n=26249 ec=76/76 lis/c 498/498 les/c/f 500/500/0 502/502/487) [4,6] r=-1 lpr=502 pi=[498,502)/1 crt=457'120217 lcod 0'0 unknown NOTIFY] start_peering_interval up [4,6,10] -> [4,6], acting [4,6,10] -> [4,6], acting_primary 4 -> 4, up_primary 4 -> 4, role 2 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.194254 7f491c845700 1 osd.10 pg_epoch: 504 pg[16.d( empty local-lis/les=498/499 n=0 ec=442/442 lis/c 498/498 les/c/f 499/499/0 504/504/470) [0,3,10] r=2 lpr=504 pi=[498,504)/1 crt=0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:41.194287 7f491c044700 1 osd.10 pg_epoch: 504 pg[6.2c( v 457'120217 (430'118627,457'120217] local-lis/les=498/500 n=26249 ec=76/76 lis/c 498/498 les/c/f 500/500/0 504/504/487) [4,6,10] r=2 lpr=504 pi=[498,504)/1 crt=457'120217 lcod 0'0 unknown NOTIFY] start_peering_interval up [4,6] -> [4,6,10], acting [4,6] -> [4,6,10], acting_primary 4 -> 4, up_primary 4 -> 4, role -1 -> 2, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.194359 7f491c044700 1 osd.10 pg_epoch: 504 pg[6.2c( v 457'120217 (430'118627,457'120217] local-lis/les=498/500 n=26249 ec=76/76 lis/c 498/498 les/c/f 500/500/0 504/504/487) [4,6,10] r=2 lpr=504 pi=[498,504)/1 crt=457'120217 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:41.194421 7f491c845700 1 osd.10 pg_epoch: 502 pg[13.fs4( v 377'8989 (0'0,377'8989] local-lis/les=498/500 n=2712 ec=345/345 lis/c 498/498 les/c/f 500/500/0 502/502/467) [5,7,3,6,2147483647,8]p5(0) r=-1 lpr=502 pi=[498,502)/1 crt=377'8979 lcod 0'0 unknown NOTIFY] start_peering_interval up [5,7,3,6,10,8] -> [5,7,3,6,2147483647,8], acting [5,7,3,6,10,8] -> [5,7,3,6,2147483647,8], acting_primary 5(0) -> 5, up_primary 5(0) -> 5, role 4 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.194480 7f491c044700 1 osd.10 pg_epoch: 502 pg[10.7( v 386'575 (0'0,386'575] local-lis/les=498/499 n=507 ec=193/193 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,6,3] r=0 lpr=499 crt=386'575 lcod 0'0 mlcod 0'0 peering] state<Started/Primary/Peering>: Peering, affected_by_map, going to Reset >2018-07-11 07:54:41.194511 7f491c845700 1 osd.10 pg_epoch: 504 pg[13.fs4( v 377'8989 (0'0,377'8989] local-lis/les=498/500 n=2712 ec=345/345 lis/c 498/498 les/c/f 500/500/0 504/504/467) [5,7,3,6,10,8]p5(0) r=4 lpr=504 pi=[498,504)/1 crt=377'8979 lcod 0'0 unknown NOTIFY] start_peering_interval up [5,7,3,6,2147483647,8] -> [5,7,3,6,10,8], acting [5,7,3,6,2147483647,8] -> [5,7,3,6,10,8], acting_primary 5(0) -> 5, up_primary 5(0) -> 5, role -1 -> 4, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.194574 7f491c044700 1 osd.10 pg_epoch: 502 pg[10.7( v 386'575 (0'0,386'575] local-lis/les=498/499 n=507 ec=193/193 lis/c 498/498 les/c/f 499/499/0 502/502/502) [6,3] r=-1 lpr=502 pi=[498,502)/1 crt=386'575 lcod 0'0 unknown] start_peering_interval up [10,6,3] -> [6,3], acting [10,6,3] -> [6,3], acting_primary 10 -> 6, up_primary 10 -> 6, role 0 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.194584 7f491c845700 1 osd.10 pg_epoch: 504 pg[13.fs4( v 377'8989 (0'0,377'8989] local-lis/les=498/500 n=2712 ec=345/345 lis/c 498/498 les/c/f 500/500/0 504/504/467) [5,7,3,6,10,8]p5(0) r=4 lpr=504 pi=[498,504)/1 crt=377'8979 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:41.194630 7f491c044700 1 osd.10 pg_epoch: 504 pg[10.7( v 386'575 (0'0,386'575] local-lis/les=498/499 n=507 ec=193/193 lis/c 498/498 les/c/f 499/499/0 504/504/504) [10,6,3] r=0 lpr=504 pi=[498,504)/1 crt=386'575 lcod 0'0 mlcod 0'0 unknown NOTIFY] start_peering_interval up [6,3] -> [10,6,3], acting [6,3] -> [10,6,3], acting_primary 6 -> 10, up_primary 6 -> 10, role -1 -> 0, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.194699 7f491c044700 1 osd.10 pg_epoch: 504 pg[10.7( v 386'575 (0'0,386'575] local-lis/les=498/499 n=507 ec=193/193 lis/c 498/498 les/c/f 499/499/0 504/504/504) [10,6,3] r=0 lpr=504 pi=[498,504)/1 crt=386'575 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 07:54:41.194719 7f491c845700 1 osd.10 pg_epoch: 502 pg[16.1a( empty local-lis/les=498/500 n=0 ec=442/442 lis/c 498/498 les/c/f 500/500/0 502/502/478) [11,3] r=-1 lpr=502 pi=[498,502)/1 crt=0'0 unknown NOTIFY] start_peering_interval up [11,10,3] -> [11,3], acting [11,10,3] -> [11,3], acting_primary 11 -> 11, up_primary 11 -> 11, role 1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.194775 7f491c845700 1 osd.10 pg_epoch: 504 pg[16.1a( empty local-lis/les=498/500 n=0 ec=442/442 lis/c 498/498 les/c/f 500/500/0 504/504/478) [11,10,3] r=1 lpr=504 pi=[498,504)/1 crt=0'0 unknown NOTIFY] start_peering_interval up [11,3] -> [11,10,3], acting [11,3] -> [11,10,3], acting_primary 11 -> 11, up_primary 11 -> 11, role -1 -> 1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.194838 7f491c845700 1 osd.10 pg_epoch: 504 pg[16.1a( empty local-lis/les=498/500 n=0 ec=442/442 lis/c 498/498 les/c/f 500/500/0 504/504/478) [11,10,3] r=1 lpr=504 pi=[498,504)/1 crt=0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:41.194977 7f491c845700 1 osd.10 pg_epoch: 502 pg[9.1a( v 494'341 (0'0,494'341] local-lis/les=498/500 n=242 ec=190/190 lis/c 498/498 les/c/f 500/500/0 502/502/468) [7,2] r=-1 lpr=502 pi=[498,502)/1 crt=494'341 lcod 0'0 unknown NOTIFY] start_peering_interval up [7,2,10] -> [7,2], acting [7,2,10] -> [7,2], acting_primary 7 -> 7, up_primary 7 -> 7, role 2 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.195034 7f491c845700 1 osd.10 pg_epoch: 504 pg[9.1a( v 494'341 (0'0,494'341] local-lis/les=498/500 n=242 ec=190/190 lis/c 498/498 les/c/f 500/500/0 504/504/468) [7,2,10] r=2 lpr=504 pi=[498,504)/1 crt=494'341 lcod 0'0 unknown NOTIFY] start_peering_interval up [7,2] -> [7,2,10], acting [7,2] -> [7,2,10], acting_primary 7 -> 7, up_primary 7 -> 7, role -1 -> 2, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.195100 7f491c845700 1 osd.10 pg_epoch: 504 pg[9.1a( v 494'341 (0'0,494'341] local-lis/les=498/500 n=242 ec=190/190 lis/c 498/498 les/c/f 500/500/0 504/504/468) [7,2,10] r=2 lpr=504 pi=[498,504)/1 crt=494'341 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:41.195233 7f491c845700 1 osd.10 pg_epoch: 502 pg[6.15( v 463'106940 (430'105357,463'106940] local-lis/les=498/500 n=25757 ec=76/76 lis/c 498/498 les/c/f 500/500/0 502/502/467) [5,11] r=-1 lpr=502 pi=[498,502)/1 crt=463'106940 lcod 0'0 unknown NOTIFY] start_peering_interval up [5,10,11] -> [5,11], acting [5,10,11] -> [5,11], acting_primary 5 -> 5, up_primary 5 -> 5, role 1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.195299 7f491c845700 1 osd.10 pg_epoch: 504 pg[6.15( v 463'106940 (430'105357,463'106940] local-lis/les=498/500 n=25757 ec=76/76 lis/c 498/498 les/c/f 500/500/0 504/504/467) [5,10,11] r=1 lpr=504 pi=[498,504)/1 crt=463'106940 lcod 0'0 unknown NOTIFY] start_peering_interval up [5,11] -> [5,10,11], acting [5,11] -> [5,10,11], acting_primary 5 -> 5, up_primary 5 -> 5, role -1 -> 1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.195366 7f491c845700 1 osd.10 pg_epoch: 504 pg[6.15( v 463'106940 (430'105357,463'106940] local-lis/les=498/500 n=25757 ec=76/76 lis/c 498/498 les/c/f 500/500/0 504/504/467) [5,10,11] r=1 lpr=504 pi=[498,504)/1 crt=463'106940 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:41.195514 7f491c845700 1 osd.10 pg_epoch: 502 pg[16.3( empty local-lis/les=498/500 n=0 ec=442/442 lis/c 498/498 les/c/f 500/500/0 502/502/467) [5,2] r=-1 lpr=502 pi=[498,502)/1 crt=0'0 unknown NOTIFY] start_peering_interval up [5,2,10] -> [5,2], acting [5,2,10] -> [5,2], acting_primary 5 -> 5, up_primary 5 -> 5, role 2 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.195576 7f491c845700 1 osd.10 pg_epoch: 504 pg[16.3( empty local-lis/les=498/500 n=0 ec=442/442 lis/c 498/498 les/c/f 500/500/0 504/504/467) [5,2,10] r=2 lpr=504 pi=[498,504)/1 crt=0'0 unknown NOTIFY] start_peering_interval up [5,2] -> [5,2,10], acting [5,2] -> [5,2,10], acting_primary 5 -> 5, up_primary 5 -> 5, role -1 -> 2, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.195638 7f491c845700 1 osd.10 pg_epoch: 504 pg[16.3( empty local-lis/les=498/500 n=0 ec=442/442 lis/c 498/498 les/c/f 500/500/0 504/504/467) [5,2,10] r=2 lpr=504 pi=[498,504)/1 crt=0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:41.195783 7f491c845700 1 osd.10 pg_epoch: 502 pg[13.3s1( v 377'10405 (371'2081,377'10405] local-lis/les=498/500 n=2736 ec=345/345 lis/c 498/498 les/c/f 500/500/0 502/502/470) [1,2147483647,8,6,7,5]p1(0) r=-1 lpr=502 pi=[498,502)/1 crt=377'10405 lcod 0'0 unknown NOTIFY] start_peering_interval up [1,10,8,6,7,5] -> [1,2147483647,8,6,7,5], acting [1,10,8,6,7,5] -> [1,2147483647,8,6,7,5], acting_primary 1(0) -> 1, up_primary 1(0) -> 1, role 1 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.195851 7f491c845700 1 osd.10 pg_epoch: 504 pg[13.3s1( v 377'10405 (371'2081,377'10405] local-lis/les=498/500 n=2736 ec=345/345 lis/c 498/498 les/c/f 500/500/0 504/504/470) [1,10,8,6,7,5]p1(0) r=1 lpr=504 pi=[498,504)/1 crt=377'10405 lcod 0'0 unknown NOTIFY] start_peering_interval up [1,2147483647,8,6,7,5] -> [1,10,8,6,7,5], acting [1,2147483647,8,6,7,5] -> [1,10,8,6,7,5], acting_primary 1(0) -> 1, up_primary 1(0) -> 1, role -1 -> 1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.195936 7f491c845700 1 osd.10 pg_epoch: 504 pg[13.3s1( v 377'10405 (371'2081,377'10405] local-lis/les=498/500 n=2736 ec=345/345 lis/c 498/498 les/c/f 500/500/0 504/504/470) [1,10,8,6,7,5]p1(0) r=1 lpr=504 pi=[498,504)/1 crt=377'10405 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:41.196158 7f491c845700 1 osd.10 pg_epoch: 502 pg[11.fs3( v 383'63252 (352'59530,383'63252] local-lis/les=498/499 n=60320 ec=267/267 lis/c 498/498 les/c/f 499/499/0 502/502/470) [0,6,2,2147483647,7,8]p0(0) r=-1 lpr=502 pi=[498,502)/1 crt=383'63252 lcod 0'0 unknown NOTIFY] start_peering_interval up [0,6,2,10,7,8] -> [0,6,2,2147483647,7,8], acting [0,6,2,10,7,8] -> [0,6,2,2147483647,7,8], acting_primary 0(0) -> 0, up_primary 0(0) -> 0, role 3 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.196242 7f491c845700 1 osd.10 pg_epoch: 504 pg[11.fs3( v 383'63252 (352'59530,383'63252] local-lis/les=498/499 n=60320 ec=267/267 lis/c 498/498 les/c/f 499/499/0 504/504/470) [0,6,2,10,7,8]p0(0) r=3 lpr=504 pi=[498,504)/1 crt=383'63252 lcod 0'0 unknown NOTIFY] start_peering_interval up [0,6,2,2147483647,7,8] -> [0,6,2,10,7,8], acting [0,6,2,2147483647,7,8] -> [0,6,2,10,7,8], acting_primary 0(0) -> 0, up_primary 0(0) -> 0, role -1 -> 3, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.196326 7f491c845700 1 osd.10 pg_epoch: 504 pg[11.fs3( v 383'63252 (352'59530,383'63252] local-lis/les=498/499 n=60320 ec=267/267 lis/c 498/498 les/c/f 499/499/0 504/504/470) [0,6,2,10,7,8]p0(0) r=3 lpr=504 pi=[498,504)/1 crt=383'63252 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:41.196441 7f491c845700 1 osd.10 pg_epoch: 502 pg[11.5s0( v 383'60461 (352'56750,383'60461] local-lis/les=498/499 n=60461 ec=267/267 lis/c 498/498 les/c/f 499/500/0 502/502/502) [2147483647,5,8,6,1,7]p5(1) r=-1 lpr=502 pi=[498,502)/1 crt=383'60461 lcod 0'0 unknown] start_peering_interval up [10,5,8,6,1,7] -> [2147483647,5,8,6,1,7], acting [10,5,8,6,1,7] -> [2147483647,5,8,6,1,7], acting_primary 10(0) -> 5, up_primary 10(0) -> 5, role 0 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.196527 7f491c845700 1 osd.10 pg_epoch: 504 pg[11.5s0( v 383'60461 (352'56750,383'60461] local-lis/les=498/499 n=60461 ec=267/267 lis/c 498/498 les/c/f 499/500/0 504/504/504) [10,5,8,6,1,7]p10(0) r=0 lpr=504 pi=[498,504)/1 crt=383'60461 lcod 0'0 mlcod 0'0 unknown NOTIFY] start_peering_interval up [2147483647,5,8,6,1,7] -> [10,5,8,6,1,7], acting [2147483647,5,8,6,1,7] -> [10,5,8,6,1,7], acting_primary 5(1) -> 10, up_primary 5(1) -> 10, role -1 -> 0, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.196602 7f491c845700 1 osd.10 pg_epoch: 504 pg[11.5s0( v 383'60461 (352'56750,383'60461] local-lis/les=498/499 n=60461 ec=267/267 lis/c 498/498 les/c/f 499/500/0 504/504/504) [10,5,8,6,1,7]p10(0) r=0 lpr=504 pi=[498,504)/1 crt=383'60461 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 07:54:41.196752 7f491c845700 1 osd.10 pg_epoch: 502 pg[11.2s2( v 383'59971 (377'58405,383'59971] local-lis/les=498/500 n=59971 ec=267/267 lis/c 498/498 les/c/f 500/501/0 502/502/465) [2,6,2147483647,0,7,8]p2(0) r=-1 lpr=502 pi=[498,502)/1 crt=383'59970 lcod 0'0 unknown NOTIFY] start_peering_interval up [2,6,10,0,7,8] -> [2,6,2147483647,0,7,8], acting [2,6,10,0,7,8] -> [2,6,2147483647,0,7,8], acting_primary 2(0) -> 2, up_primary 2(0) -> 2, role 2 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.196834 7f491c845700 1 osd.10 pg_epoch: 504 pg[11.2s2( v 383'59971 (377'58405,383'59971] local-lis/les=498/500 n=59971 ec=267/267 lis/c 498/498 les/c/f 500/501/0 504/504/465) [2,6,10,0,7,8]p2(0) r=2 lpr=504 pi=[498,504)/1 crt=383'59970 lcod 0'0 unknown NOTIFY] start_peering_interval up [2,6,2147483647,0,7,8] -> [2,6,10,0,7,8], acting [2,6,2147483647,0,7,8] -> [2,6,10,0,7,8], acting_primary 2(0) -> 2, up_primary 2(0) -> 2, role -1 -> 2, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.196909 7f491c845700 1 osd.10 pg_epoch: 504 pg[11.2s2( v 383'59971 (377'58405,383'59971] local-lis/les=498/500 n=59971 ec=267/267 lis/c 498/498 les/c/f 500/501/0 504/504/465) [2,6,10,0,7,8]p2(0) r=2 lpr=504 pi=[498,504)/1 crt=383'59970 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:41.197126 7f491c845700 1 osd.10 pg_epoch: 502 pg[9.1e( v 494'752 (0'0,494'752] local-lis/les=498/500 n=270 ec=190/190 lis/c 498/498 les/c/f 500/500/0 502/502/465) [2,4] r=-1 lpr=502 pi=[498,502)/1 crt=494'752 lcod 0'0 unknown NOTIFY] start_peering_interval up [2,4,10] -> [2,4], acting [2,4,10] -> [2,4], acting_primary 2 -> 2, up_primary 2 -> 2, role 2 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.197229 7f491c845700 1 osd.10 pg_epoch: 504 pg[9.1e( v 494'752 (0'0,494'752] local-lis/les=498/500 n=270 ec=190/190 lis/c 498/498 les/c/f 500/500/0 504/504/465) [2,4,10] r=2 lpr=504 pi=[498,504)/1 crt=494'752 lcod 0'0 unknown NOTIFY] start_peering_interval up [2,4] -> [2,4,10], acting [2,4] -> [2,4,10], acting_primary 2 -> 2, up_primary 2 -> 2, role -1 -> 2, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.197348 7f491c845700 1 osd.10 pg_epoch: 504 pg[9.1e( v 494'752 (0'0,494'752] local-lis/les=498/500 n=270 ec=190/190 lis/c 498/498 les/c/f 500/500/0 504/504/465) [2,4,10] r=2 lpr=504 pi=[498,504)/1 crt=494'752 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:41.197537 7f491c845700 1 osd.10 pg_epoch: 502 pg[13.2s4( v 377'9209 (0'0,377'9209] local-lis/les=498/500 n=2712 ec=345/345 lis/c 498/498 les/c/f 500/500/0 502/502/467) [5,8,6,7,2147483647,3]p5(0) r=-1 lpr=502 pi=[498,502)/1 crt=377'9201 lcod 0'0 unknown NOTIFY] start_peering_interval up [5,8,6,7,10,3] -> [5,8,6,7,2147483647,3], acting [5,8,6,7,10,3] -> [5,8,6,7,2147483647,3], acting_primary 5(0) -> 5, up_primary 5(0) -> 5, role 4 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.197621 7f491c845700 1 osd.10 pg_epoch: 504 pg[13.2s4( v 377'9209 (0'0,377'9209] local-lis/les=498/500 n=2712 ec=345/345 lis/c 498/498 les/c/f 500/500/0 504/504/467) [5,8,6,7,10,3]p5(0) r=4 lpr=504 pi=[498,504)/1 crt=377'9201 lcod 0'0 unknown NOTIFY] start_peering_interval up [5,8,6,7,2147483647,3] -> [5,8,6,7,10,3], acting [5,8,6,7,2147483647,3] -> [5,8,6,7,10,3], acting_primary 5(0) -> 5, up_primary 5(0) -> 5, role -1 -> 4, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.197698 7f491c845700 1 osd.10 pg_epoch: 504 pg[13.2s4( v 377'9209 (0'0,377'9209] local-lis/les=498/500 n=2712 ec=345/345 lis/c 498/498 les/c/f 500/500/0 504/504/467) [5,8,6,7,10,3]p5(0) r=4 lpr=504 pi=[498,504)/1 crt=377'9201 lcod 0'0 unknown NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:54:41.197785 7f491c845700 1 osd.10 pg_epoch: 502 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,2,4] r=0 lpr=499 crt=166'68041 lcod 0'0 mlcod 0'0 peering] state<Started/Primary/Peering>: Peering, affected_by_map, going to Reset >2018-07-11 07:54:41.197887 7f491c845700 1 osd.10 pg_epoch: 502 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 502/502/502) [2,4] r=-1 lpr=502 pi=[498,502)/1 crt=166'68041 lcod 0'0 unknown] start_peering_interval up [10,2,4] -> [2,4], acting [10,2,4] -> [2,4], acting_primary 10 -> 2, up_primary 10 -> 2, role 0 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.197955 7f491c845700 1 osd.10 pg_epoch: 504 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 504/504/504) [10,2,4] r=0 lpr=504 pi=[498,504)/1 crt=166'68041 lcod 0'0 mlcod 0'0 unknown NOTIFY] start_peering_interval up [2,4] -> [10,2,4], acting [2,4] -> [10,2,4], acting_primary 2 -> 10, up_primary 2 -> 10, role -1 -> 0, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.198035 7f491c845700 1 osd.10 pg_epoch: 504 pg[5.2e( v 166'68041 (166'66541,166'68041] local-lis/les=498/499 n=68041 ec=71/71 lis/c 498/498 les/c/f 499/499/0 504/504/504) [10,2,4] r=0 lpr=504 pi=[498,504)/1 crt=166'68041 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 07:54:41.198153 7f491c845700 1 osd.10 pg_epoch: 502 pg[11.1s0( v 383'60378 (352'56674,383'60378] local-lis/les=498/499 n=60378 ec=267/267 lis/c 498/498 les/c/f 499/499/0 498/498/498) [10,5,2,8,6,11]p10(0) r=0 lpr=499 crt=383'60378 lcod 0'0 mlcod 0'0 peering] state<Started/Primary/Peering>: Peering, affected_by_map, going to Reset >2018-07-11 07:54:41.198285 7f491c845700 1 osd.10 pg_epoch: 502 pg[11.1s0( v 383'60378 (352'56674,383'60378] local-lis/les=498/499 n=60378 ec=267/267 lis/c 498/498 les/c/f 499/499/0 502/502/502) [2147483647,5,2,8,6,11]p5(1) r=-1 lpr=502 pi=[498,502)/1 crt=383'60378 lcod 0'0 unknown] start_peering_interval up [10,5,2,8,6,11] -> [2147483647,5,2,8,6,11], acting [10,5,2,8,6,11] -> [2147483647,5,2,8,6,11], acting_primary 10(0) -> 5, up_primary 10(0) -> 5, role 0 -> -1, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.198371 7f491c845700 1 osd.10 pg_epoch: 504 pg[11.1s0( v 383'60378 (352'56674,383'60378] local-lis/les=498/499 n=60378 ec=267/267 lis/c 498/498 les/c/f 499/499/0 504/504/504) [10,5,2,8,6,11]p10(0) r=0 lpr=504 pi=[498,504)/1 crt=383'60378 lcod 0'0 mlcod 0'0 unknown NOTIFY] start_peering_interval up [2147483647,5,2,8,6,11] -> [10,5,2,8,6,11], acting [2147483647,5,2,8,6,11] -> [10,5,2,8,6,11], acting_primary 5(1) -> 10, up_primary 5(1) -> 10, role -1 -> 0, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:54:41.198445 7f491c845700 1 osd.10 pg_epoch: 504 pg[11.1s0( v 383'60378 (352'56674,383'60378] local-lis/les=498/499 n=60378 ec=267/267 lis/c 498/498 les/c/f 499/499/0 504/504/504) [10,5,2,8,6,11]p10(0) r=0 lpr=504 pi=[498,504)/1 crt=383'60378 lcod 0'0 mlcod 0'0 unknown] state<Start>: transitioning to Primary >2018-07-11 07:54:42.881796 7f4931830700 0 -- 10.8.128.107:6801/14173 >> 10.8.128.107:6805/12617 conn(0x55f656af0800 :6801 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg: challenging authorizer >2018-07-11 07:54:42.882029 7f4931830700 0 -- 10.8.128.107:6801/14173 >> 10.8.128.107:6805/12617 conn(0x55f656af0800 :6801 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg accept connect_seq 0 vs existing csq=0 existing_state=STATE_CONNECTING >2018-07-11 07:54:42.882235 7f4931830700 0 -- 10.8.128.107:6801/14173 >> 10.8.128.107:6805/12617 conn(0x55f663f6d800 :-1 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg: challenging authorizer >2018-07-11 07:54:43.856856 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 3] Generated table #36823: 3106037 keys, 75716540 bytes >2018-07-11 07:54:43.856907 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295683856879, "cf_name": "default", "job": 3, "event": "table_file_creation", "file_number": 36823, "file_size": 75716540, "table_properties": {"data_size": 67111704, "index_size": 689513, "filter_size": 7914331, "raw_key_size": 89729965, "raw_average_key_size": 28, "raw_value_size": 25038446, "raw_average_value_size": 8, "num_data_blocks": 16486, "num_entries": 3106037, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1000000", "kMergeOperands": "0"}} >2018-07-11 07:54:46.788360 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 3] Generated table #36824: 2638239 keys, 74546312 bytes >2018-07-11 07:54:46.788403 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295686788382, "cf_name": "default", "job": 3, "event": "table_file_creation", "file_number": 36824, "file_size": 74546312, "table_properties": {"data_size": 67112147, "index_size": 689996, "filter_size": 6743179, "raw_key_size": 76215805, "raw_average_key_size": 28, "raw_value_size": 31365742, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638239, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:54:47.980523 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 3] Generated table #36825: 918894 keys, 25964675 bytes >2018-07-11 07:54:47.980561 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295687980542, "cf_name": "default", "job": 3, "event": "table_file_creation", "file_number": 36825, "file_size": 25964675, "table_properties": {"data_size": 23375031, "index_size": 240016, "filter_size": 2348640, "raw_key_size": 26545824, "raw_average_key_size": 28, "raw_value_size": 10924626, "raw_average_value_size": 11, "num_data_blocks": 5744, "num_entries": 918894, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:54:49.007067 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 3] Generated table #36826: 914333 keys, 14820357 bytes >2018-07-11 07:54:49.007111 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295689007095, "cf_name": "default", "job": 3, "event": "table_file_creation", "file_number": 36826, "file_size": 14820357, "table_properties": {"data_size": 12378104, "index_size": 126700, "filter_size": 2314566, "raw_key_size": 26414058, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 3038, "num_entries": 914333, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "914333", "kMergeOperands": "0"}} >2018-07-11 07:54:49.007151 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 3] Compacted 1@2 + 8@3 files to L3 => 492418855 bytes >2018-07-11 07:54:49.018573 7f4922851700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 48408856159 >2018-07-11 07:54:49.018841 7f4922851700 4 rocksdb: (Original Log Time 2018/07/11-07:54:49.018535) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[1 7 71 471 3641 796 0] max score 2.20, MB/sec: 20.9 rd, 19.5 wr, level 3, files in(1, 8) out(9) MB in(76.6, 426.5) out(469.6), read-write-amplify(12.7) write-amplify(6.1) OK, records in: 20783214, records dropped: 1579362 > >2018-07-11 07:54:49.018851 7f4922851700 4 rocksdb: (Original Log Time 2018/07/11-07:54:49.018561) EVENT_LOG_v1 {"time_micros": 1531295689018549, "job": 3, "event": "compaction_finished", "compaction_time_micros": 25215717, "output_level": 3, "num_output_files": 9, "total_output_size": 492418855, "num_input_records": 20783214, "num_output_records": 19203852, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 7, 71, 471, 3641, 796, 0]} >2018-07-11 07:54:49.019374 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295689019366, "job": 3, "event": "table_file_deletion", "file_number": 36612} >2018-07-11 07:54:49.019399 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295689019395, "job": 3, "event": "table_file_deletion", "file_number": 15293} >2018-07-11 07:54:49.019423 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295689019419, "job": 3, "event": "table_file_deletion", "file_number": 15292} >2018-07-11 07:54:49.019447 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295689019443, "job": 3, "event": "table_file_deletion", "file_number": 15291} >2018-07-11 07:54:49.019469 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295689019465, "job": 3, "event": "table_file_deletion", "file_number": 15290} >2018-07-11 07:54:49.019497 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295689019493, "job": 3, "event": "table_file_deletion", "file_number": 15289} >2018-07-11 07:54:49.019524 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295689019520, "job": 3, "event": "table_file_deletion", "file_number": 15288} >2018-07-11 07:54:49.019550 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295689019547, "job": 3, "event": "table_file_deletion", "file_number": 15177} >2018-07-11 07:54:49.019575 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295689019571, "job": 3, "event": "table_file_deletion", "file_number": 15174} >2018-07-11 07:54:49.035147 7f4922851700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 48057247012 >2018-07-11 07:54:49.035267 7f4922851700 4 rocksdb: (Original Log Time 2018/07/11-07:54:49.019726) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #36581 to level-3 80348254 bytes > >2018-07-11 07:54:49.035277 7f4922851700 4 rocksdb: (Original Log Time 2018/07/11-07:54:49.035172) EVENT_LOG_v1 {"time_micros": 1531295689035161, "job": 4, "event": "trivial_move", "destination_level": 3, "files": 1, "total_files_size": 80348254} >2018-07-11 07:54:49.035282 7f4922851700 4 rocksdb: (Original Log Time 2018/07/11-07:54:49.035180) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-3 80348254 bytes OK: base level 1 max bytes base 268435456 files[1 7 70 472 3641 796 0] max score 2.14 > >2018-07-11 07:54:49.035853 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 5] Compacting 1@2 + 5@3 files to L3, score 2.14 >2018-07-11 07:54:49.035869 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 4 Base level 2, inputs: [36622(76MB)], [28757(71MB) 28902(71MB) 28951(71MB) 29035(71MB) 29039(71MB)] > >2018-07-11 07:54:49.035890 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295689035875, "job": 5, "event": "compaction_started", "files_L2": [36622], "files_L3": [28757, 28902, 28951, 29035, 29039], "score": 2.14132, "input_data_size": 453077752} >2018-07-11 07:54:53.886560 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 5] Generated table #36827: 3367943 keys, 76373702 bytes >2018-07-11 07:54:53.886603 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295693886578, "cf_name": "default", "job": 5, "event": "table_file_creation", "file_number": 36827, "file_size": 76373702, "table_properties": {"data_size": 67112438, "index_size": 690249, "filter_size": 8570023, "raw_key_size": 97296149, "raw_average_key_size": 28, "raw_value_size": 21496963, "raw_average_value_size": 6, "num_data_blocks": 16484, "num_entries": 3367943, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1559788", "kMergeOperands": "0"}} >2018-07-11 07:54:56.880608 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 5] Generated table #36828: 2638240 keys, 74546032 bytes >2018-07-11 07:54:56.880659 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295696880643, "cf_name": "default", "job": 5, "event": "table_file_creation", "file_number": 36828, "file_size": 74546032, "table_properties": {"data_size": 67112109, "index_size": 689752, "filter_size": 6743181, "raw_key_size": 76215813, "raw_average_key_size": 28, "raw_value_size": 31365733, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:55:00.014370 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 5] Generated table #36829: 2638240 keys, 74546356 bytes >2018-07-11 07:55:00.014423 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295700014406, "cf_name": "default", "job": 5, "event": "table_file_creation", "file_number": 36829, "file_size": 74546356, "table_properties": {"data_size": 67112143, "index_size": 690042, "filter_size": 6743181, "raw_key_size": 76215825, "raw_average_key_size": 28, "raw_value_size": 31365745, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:55:03.260882 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 5] Generated table #36830: 2787372 keys, 74916785 bytes >2018-07-11 07:55:03.260927 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295703260910, "cf_name": "default", "job": 5, "event": "table_file_creation", "file_number": 36830, "file_size": 74916785, "table_properties": {"data_size": 67109070, "index_size": 690189, "filter_size": 7116534, "raw_key_size": 80524068, "raw_average_key_size": 28, "raw_value_size": 29345686, "raw_average_value_size": 10, "num_data_blocks": 16488, "num_entries": 2787372, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "319043", "kMergeOperands": "0"}} >2018-07-11 07:55:06.702814 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 5] Generated table #36831: 2956747 keys, 75342151 bytes >2018-07-11 07:55:06.702846 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295706702831, "cf_name": "default", "job": 5, "event": "table_file_creation", "file_number": 36831, "file_size": 75342151, "table_properties": {"data_size": 67110742, "index_size": 689845, "filter_size": 7540572, "raw_key_size": 85417147, "raw_average_key_size": 28, "raw_value_size": 27056616, "raw_average_value_size": 9, "num_data_blocks": 16487, "num_entries": 2956747, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "680957", "kMergeOperands": "0"}} >2018-07-11 07:55:07.146809 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 5] Generated table #36832: 362451 keys, 10241903 bytes >2018-07-11 07:55:07.146861 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295707146844, "cf_name": "default", "job": 5, "event": "table_file_creation", "file_number": 36832, "file_size": 10241903, "table_properties": {"data_size": 9220143, "index_size": 94367, "filter_size": 926407, "raw_key_size": 10470813, "raw_average_key_size": 28, "raw_value_size": 4309146, "raw_average_value_size": 11, "num_data_blocks": 2266, "num_entries": 362451, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:55:08.542902 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 5] Generated table #36833: 1397377 keys, 22649754 bytes >2018-07-11 07:55:08.542935 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295708542920, "cf_name": "default", "job": 5, "event": "table_file_creation", "file_number": 36833, "file_size": 22649754, "table_properties": {"data_size": 18917490, "index_size": 193917, "filter_size": 3537360, "raw_key_size": 40368662, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 4643, "num_entries": 1397377, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1397377", "kMergeOperands": "0"}} >2018-07-11 07:55:08.542969 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 5] Compacted 1@2 + 5@3 files to L3 => 408616683 bytes >2018-07-11 07:55:08.559876 7f4922851700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 47220366267 >2018-07-11 07:55:08.560114 7f4922851700 4 rocksdb: (Original Log Time 2018/07/11-07:55:08.559842) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[1 7 69 474 3641 796 0] max score 2.08, MB/sec: 23.2 rd, 20.9 wr, level 3, files in(1, 5) out(7) MB in(76.6, 355.5) out(389.7), read-write-amplify(10.7) write-amplify(5.1) OK, records in: 18148370, records dropped: 2000000 > >2018-07-11 07:55:08.560124 7f4922851700 4 rocksdb: (Original Log Time 2018/07/11-07:55:08.559868) EVENT_LOG_v1 {"time_micros": 1531295708559856, "job": 5, "event": "compaction_finished", "compaction_time_micros": 19507059, "output_level": 3, "num_output_files": 7, "total_output_size": 408616683, "num_input_records": 18148370, "num_output_records": 16148370, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 7, 69, 474, 3641, 796, 0]} >2018-07-11 07:55:08.560715 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295708560711, "job": 5, "event": "table_file_deletion", "file_number": 36622} >2018-07-11 07:55:08.560730 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295708560728, "job": 5, "event": "table_file_deletion", "file_number": 29039} >2018-07-11 07:55:08.560742 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295708560740, "job": 5, "event": "table_file_deletion", "file_number": 29035} >2018-07-11 07:55:08.560754 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295708560752, "job": 5, "event": "table_file_deletion", "file_number": 28951} >2018-07-11 07:55:08.560766 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295708560764, "job": 5, "event": "table_file_deletion", "file_number": 28902} >2018-07-11 07:55:08.560777 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295708560775, "job": 5, "event": "table_file_deletion", "file_number": 28757} >2018-07-11 07:55:08.560938 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 6] Compacting 1@2 + 8@3 files to L3, score 2.08 >2018-07-11 07:55:08.560952 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 5 Base level 2, inputs: [36619(76MB)], [33953(1329B) 33954(71MB) 33955(71MB) 33958(71MB) 33959(71MB) 33960(71MB) 33961(71MB) 33963(31MB)] > >2018-07-11 07:55:08.560966 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295708560955, "job": 6, "event": "compaction_started", "files_L2": [36619], "files_L3": [33953, 33954, 33955, 33958, 33959, 33960, 33961, 33963], "score": 2.08184, "input_data_size": 560234519} >2018-07-11 07:55:12.152344 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 6] Generated table #36834: 2938738 keys, 75296435 bytes >2018-07-11 07:55:12.152394 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295712152378, "cf_name": "default", "job": 6, "event": "table_file_creation", "file_number": 36834, "file_size": 75296435, "table_properties": {"data_size": 67110148, "index_size": 689809, "filter_size": 7495486, "raw_key_size": 84896898, "raw_average_key_size": 28, "raw_value_size": 27299561, "raw_average_value_size": 9, "num_data_blocks": 16487, "num_entries": 2938738, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "642514", "kMergeOperands": "0"}} >2018-07-11 07:55:12.761763 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 6] Generated table #36835: 342028 keys, 9664711 bytes >2018-07-11 07:55:12.761808 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295712761780, "cf_name": "default", "job": 6, "event": "table_file_creation", "file_number": 36835, "file_size": 9664711, "table_properties": {"data_size": 8700578, "index_size": 88942, "filter_size": 874205, "raw_key_size": 9880804, "raw_average_key_size": 28, "raw_value_size": 4066328, "raw_average_value_size": 11, "num_data_blocks": 2138, "num_entries": 342028, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:55:15.790041 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 6] Generated table #36836: 2638240 keys, 74545798 bytes >2018-07-11 07:55:15.790094 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295715790077, "cf_name": "default", "job": 6, "event": "table_file_creation", "file_number": 36836, "file_size": 74545798, "table_properties": {"data_size": 67112145, "index_size": 689482, "filter_size": 6743181, "raw_key_size": 76215823, "raw_average_key_size": 28, "raw_value_size": 31365743, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:55:20.172910 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 6] Generated table #36837: 2638240 keys, 74546033 bytes >2018-07-11 07:55:20.172959 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295720172943, "cf_name": "default", "job": 6, "event": "table_file_creation", "file_number": 36837, "file_size": 74546033, "table_properties": {"data_size": 67112141, "index_size": 689721, "filter_size": 6743181, "raw_key_size": 76215827, "raw_average_key_size": 28, "raw_value_size": 31365747, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:55:23.666685 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 6] Generated table #36838: 3106038 keys, 75716969 bytes >2018-07-11 07:55:23.666737 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295723666721, "cf_name": "default", "job": 6, "event": "table_file_creation", "file_number": 36838, "file_size": 75716969, "table_properties": {"data_size": 67111736, "index_size": 689909, "filter_size": 7914332, "raw_key_size": 89729987, "raw_average_key_size": 28, "raw_value_size": 25038451, "raw_average_value_size": 8, "num_data_blocks": 16486, "num_entries": 3106038, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1000000", "kMergeOperands": "0"}} >2018-07-11 07:55:26.777426 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 6] Generated table #36839: 2638240 keys, 74546086 bytes >2018-07-11 07:55:26.777469 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295726777450, "cf_name": "default", "job": 6, "event": "table_file_creation", "file_number": 36839, "file_size": 74546086, "table_properties": {"data_size": 67112143, "index_size": 689772, "filter_size": 6743181, "raw_key_size": 76215823, "raw_average_key_size": 28, "raw_value_size": 31365743, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:55:29.459389 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 6] Generated table #36840: 2324524 keys, 65681793 bytes >2018-07-11 07:55:29.459436 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295729459421, "cf_name": "default", "job": 6, "event": "table_file_creation", "file_number": 36840, "file_size": 65681793, "table_properties": {"data_size": 59131734, "index_size": 607733, "filter_size": 5941337, "raw_key_size": 67152921, "raw_average_key_size": 28, "raw_value_size": 27636013, "raw_average_value_size": 11, "num_data_blocks": 14529, "num_entries": 2324524, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:55:30.762316 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 6] Generated table #36841: 1314651 keys, 21308888 bytes >2018-07-11 07:55:30.762350 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295730762335, "cf_name": "default", "job": 6, "event": "table_file_creation", "file_number": 36841, "file_size": 21308888, "table_properties": {"data_size": 17797546, "index_size": 182411, "filter_size": 3327944, "raw_key_size": 37978798, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 4368, "num_entries": 1314651, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1314651", "kMergeOperands": "0"}} >2018-07-11 07:55:30.762387 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 6] Compacted 1@2 + 8@3 files to L3 => 471306713 bytes >2018-07-11 07:55:30.776154 7f4922851700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 45895770018 >2018-07-11 07:55:30.776600 7f4922851700 4 rocksdb: (Original Log Time 2018/07/11-07:55:30.776120) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[1 7 68 474 3641 796 0] max score 2.02, MB/sec: 25.2 rd, 21.2 wr, level 3, files in(1, 8) out(8) MB in(76.6, 457.7) out(449.5), read-write-amplify(12.8) write-amplify(5.9) OK, records in: 21940699, records dropped: 4000000 > >2018-07-11 07:55:30.776611 7f4922851700 4 rocksdb: (Original Log Time 2018/07/11-07:55:30.776146) EVENT_LOG_v1 {"time_micros": 1531295730776135, "job": 6, "event": "compaction_finished", "compaction_time_micros": 22201403, "output_level": 3, "num_output_files": 8, "total_output_size": 471306713, "num_input_records": 21940699, "num_output_records": 17940699, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 7, 68, 474, 3641, 796, 0]} >2018-07-11 07:55:30.777126 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295730777120, "job": 6, "event": "table_file_deletion", "file_number": 36619} >2018-07-11 07:55:30.777157 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295730777153, "job": 6, "event": "table_file_deletion", "file_number": 33963} >2018-07-11 07:55:30.777181 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295730777177, "job": 6, "event": "table_file_deletion", "file_number": 33961} >2018-07-11 07:55:30.777204 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295730777201, "job": 6, "event": "table_file_deletion", "file_number": 33960} >2018-07-11 07:55:30.777228 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295730777224, "job": 6, "event": "table_file_deletion", "file_number": 33959} >2018-07-11 07:55:30.777257 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295730777253, "job": 6, "event": "table_file_deletion", "file_number": 33958} >2018-07-11 07:55:30.777275 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295730777271, "job": 6, "event": "table_file_deletion", "file_number": 33955} >2018-07-11 07:55:30.777290 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295730777287, "job": 6, "event": "table_file_deletion", "file_number": 33954} >2018-07-11 07:55:30.777308 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295730777303, "job": 6, "event": "table_file_deletion", "file_number": 33953} >2018-07-11 07:55:30.777500 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 7] Compacting 1@2 + 2@3 files to L3, score 2.02 >2018-07-11 07:55:30.777513 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 6 Base level 2, inputs: [36626(76MB)], [25015(71MB) 25151(71MB)] > >2018-07-11 07:55:30.777533 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295730777518, "job": 7, "event": "compaction_started", "files_L2": [36626], "files_L3": [25015, 25151], "score": 2.02236, "input_data_size": 229439698} >2018-07-11 07:55:34.786895 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 7] Generated table #36842: 3417477 keys, 76495296 bytes >2018-07-11 07:55:34.786947 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295734786930, "cf_name": "default", "job": 7, "event": "table_file_creation", "file_number": 36842, "file_size": 76495296, "table_properties": {"data_size": 67110063, "index_size": 690212, "filter_size": 8694029, "raw_key_size": 98727130, "raw_average_key_size": 28, "raw_value_size": 20824661, "raw_average_value_size": 6, "num_data_blocks": 16483, "num_entries": 3417477, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1665870", "kMergeOperands": "0"}} >2018-07-11 07:55:38.352197 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 7] Generated table #36843: 3106038 keys, 75716780 bytes >2018-07-11 07:55:38.352238 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295738352222, "cf_name": "default", "job": 7, "event": "table_file_creation", "file_number": 36843, "file_size": 75716780, "table_properties": {"data_size": 67111749, "index_size": 689706, "filter_size": 7914333, "raw_key_size": 89729993, "raw_average_key_size": 28, "raw_value_size": 25038457, "raw_average_value_size": 8, "num_data_blocks": 16486, "num_entries": 3106038, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1000000", "kMergeOperands": "0"}} >2018-07-11 07:55:42.369970 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 7] Generated table #36844: 3710129 keys, 77228949 bytes >2018-07-11 07:55:42.370006 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295742369990, "cf_name": "default", "job": 7, "event": "table_file_creation", "file_number": 36844, "file_size": 77228949, "table_properties": {"data_size": 67111873, "index_size": 689383, "filter_size": 9426700, "raw_key_size": 107181498, "raw_average_key_size": 28, "raw_value_size": 16868398, "raw_average_value_size": 4, "num_data_blocks": 16481, "num_entries": 3710129, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "2291292", "kMergeOperands": "0"}} >2018-07-11 07:55:42.401861 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 7] Generated table #36845: 3 keys, 1075 bytes >2018-07-11 07:55:42.401897 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295742401872, "cf_name": "default", "job": 7, "event": "table_file_creation", "file_number": 36845, "file_size": 1075, "table_properties": {"data_size": 68, "index_size": 27, "filter_size": 18, "raw_key_size": 86, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 1, "num_entries": 3, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "3", "kMergeOperands": "0"}} >2018-07-11 07:55:42.401919 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 7] Compacted 1@2 + 2@3 files to L3 => 229442100 bytes >2018-07-11 07:55:42.417611 7f4922851700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 45504633257 >2018-07-11 07:55:42.418012 7f4922851700 4 rocksdb: (Original Log Time 2018/07/11-07:55:42.417577) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[1 7 67 476 3641 796 0] max score 1.96, MB/sec: 19.7 rd, 19.7 wr, level 3, files in(1, 2) out(4) MB in(76.6, 142.2) out(218.8), read-write-amplify(5.7) write-amplify(2.9) OK, records in: 10233647, records dropped: 0 > >2018-07-11 07:55:42.418022 7f4922851700 4 rocksdb: (Original Log Time 2018/07/11-07:55:42.417603) EVENT_LOG_v1 {"time_micros": 1531295742417590, "job": 7, "event": "compaction_finished", "compaction_time_micros": 11624370, "output_level": 3, "num_output_files": 4, "total_output_size": 229442100, "num_input_records": 10233647, "num_output_records": 10233647, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 7, 67, 476, 3641, 796, 0]} >2018-07-11 07:55:42.418575 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295742418570, "job": 7, "event": "table_file_deletion", "file_number": 36626} >2018-07-11 07:55:42.418603 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295742418601, "job": 7, "event": "table_file_deletion", "file_number": 25151} >2018-07-11 07:55:42.418616 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295742418614, "job": 7, "event": "table_file_deletion", "file_number": 25015} >2018-07-11 07:55:42.418789 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 8] Compacting 1@2 + 8@3 files to L3, score 1.96 >2018-07-11 07:55:42.418803 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 7 Base level 2, inputs: [36579(76MB)], [24734(11MB) 24736(39MB) 24739(41MB) 24858(71MB) 24859(49MB) 24860(71MB) 24861(1064B) 24862(51MB)] > >2018-07-11 07:55:42.418816 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295742418806, "job": 8, "event": "compaction_started", "files_L2": [36579], "files_L3": [24734, 24736, 24739, 24858, 24859, 24860, 24861, 24862], "score": 1.96288, "input_data_size": 432350876} >2018-07-11 07:55:43.361717 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 8] Generated table #36846: 546001 keys, 13909879 bytes >2018-07-11 07:55:43.361760 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295743361745, "cf_name": "default", "job": 8, "event": "table_file_creation", "file_number": 36846, "file_size": 13909879, "table_properties": {"data_size": 12389474, "index_size": 126960, "filter_size": 1392457, "raw_key_size": 15773362, "raw_average_key_size": 28, "raw_value_size": 4992924, "raw_average_value_size": 9, "num_data_blocks": 3044, "num_entries": 546001, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "126035", "kMergeOperands": "0"}} >2018-07-11 07:55:45.187743 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 8] Generated table #36847: 1482080 keys, 41877652 bytes >2018-07-11 07:55:45.187773 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295745187759, "cf_name": "default", "job": 8, "event": "table_file_creation", "file_number": 36847, "file_size": 41877652, "table_properties": {"data_size": 37701485, "index_size": 387068, "filter_size": 3788111, "raw_key_size": 42815653, "raw_average_key_size": 28, "raw_value_size": 17620293, "raw_average_value_size": 11, "num_data_blocks": 9264, "num_entries": 1482080, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:55:48.771881 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 8] Generated table #36848: 2979343 keys, 75399498 bytes >2018-07-11 07:55:48.771933 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295748771917, "cf_name": "default", "job": 8, "event": "table_file_creation", "file_number": 36848, "file_size": 75399498, "table_properties": {"data_size": 67111494, "index_size": 689865, "filter_size": 7597147, "raw_key_size": 86069937, "raw_average_key_size": 28, "raw_value_size": 26751716, "raw_average_value_size": 8, "num_data_blocks": 16487, "num_entries": 2979343, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "729200", "kMergeOperands": "0"}} >2018-07-11 07:55:52.330546 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 8] Generated table #36849: 2942590 keys, 75306285 bytes >2018-07-11 07:55:52.330599 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295752330581, "cf_name": "default", "job": 8, "event": "table_file_creation", "file_number": 36849, "file_size": 75306285, "table_properties": {"data_size": 67110221, "index_size": 689942, "filter_size": 7505130, "raw_key_size": 85008133, "raw_average_key_size": 28, "raw_value_size": 27247521, "raw_average_value_size": 9, "num_data_blocks": 16487, "num_entries": 2942590, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "650741", "kMergeOperands": "0"}} >2018-07-11 07:55:53.493203 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 8] Generated table #36850: 848176 keys, 23966377 bytes >2018-07-11 07:55:53.493256 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295753493233, "cf_name": "default", "job": 8, "event": "table_file_creation", "file_number": 36850, "file_size": 23966377, "table_properties": {"data_size": 21576080, "index_size": 221422, "filter_size": 2167887, "raw_key_size": 24502867, "raw_average_key_size": 28, "raw_value_size": 10083875, "raw_average_value_size": 11, "num_data_blocks": 5302, "num_entries": 848176, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:55:56.793957 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 8] Generated table #36851: 2638241 keys, 74545882 bytes >2018-07-11 07:55:56.794003 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295756793988, "cf_name": "default", "job": 8, "event": "table_file_creation", "file_number": 36851, "file_size": 74545882, "table_properties": {"data_size": 67112101, "index_size": 689608, "filter_size": 6743183, "raw_key_size": 76215859, "raw_average_key_size": 28, "raw_value_size": 31365762, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:55:56.918304 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 8] Generated table #36852: 3 keys, 1124 bytes >2018-07-11 07:55:56.918332 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295756918319, "cf_name": "default", "job": 8, "event": "table_file_creation", "file_number": 36852, "file_size": 1124, "table_properties": {"data_size": 116, "index_size": 27, "filter_size": 18, "raw_key_size": 87, "raw_average_key_size": 29, "raw_value_size": 36, "raw_average_value_size": 12, "num_data_blocks": 1, "num_entries": 3, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:56:00.280516 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 8] Generated table #36853: 2907057 keys, 70094238 bytes >2018-07-11 07:56:00.280561 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295760280535, "cf_name": "default", "job": 8, "event": "table_file_creation", "file_number": 36853, "file_size": 70094238, "table_properties": {"data_size": 62050014, "index_size": 637489, "filter_size": 7405744, "raw_key_size": 83981644, "raw_average_key_size": 28, "raw_value_size": 22672785, "raw_average_value_size": 7, "num_data_blocks": 15242, "num_entries": 2907057, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1000000", "kMergeOperands": "0"}} >2018-07-11 07:56:02.451460 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 8] Generated table #36854: 1831130 keys, 29680187 bytes >2018-07-11 07:56:02.451495 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295762451475, "cf_name": "default", "job": 8, "event": "table_file_creation", "file_number": 36854, "file_size": 29680187, "table_properties": {"data_size": 24789571, "index_size": 254257, "filter_size": 4635372, "raw_key_size": 52899304, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 6084, "num_entries": 1831130, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1831130", "kMergeOperands": "0"}} >2018-07-11 07:56:02.451543 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 8] Compacted 1@2 + 8@3 files to L3 => 404781122 bytes >2018-07-11 07:56:02.467321 7f4922851700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 44804462470 >2018-07-11 07:56:02.467757 7f4922851700 4 rocksdb: (Original Log Time 2018/07/11-07:56:02.467285) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[1 7 66 477 3641 796 0] max score 1.90, MB/sec: 21.6 rd, 20.2 wr, level 3, files in(1, 8) out(9) MB in(76.6, 335.7) out(386.0), read-write-amplify(10.4) write-amplify(5.0) OK, records in: 17414739, records dropped: 1240118 > >2018-07-11 07:56:02.467766 7f4922851700 4 rocksdb: (Original Log Time 2018/07/11-07:56:02.467313) EVENT_LOG_v1 {"time_micros": 1531295762467299, "job": 8, "event": "compaction_finished", "compaction_time_micros": 20032705, "output_level": 3, "num_output_files": 9, "total_output_size": 404781122, "num_input_records": 17414739, "num_output_records": 16174621, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 7, 66, 477, 3641, 796, 0]} >2018-07-11 07:56:02.468332 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295762468327, "job": 8, "event": "table_file_deletion", "file_number": 36579} >2018-07-11 07:56:02.468349 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295762468347, "job": 8, "event": "table_file_deletion", "file_number": 24862} >2018-07-11 07:56:02.468365 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295762468361, "job": 8, "event": "table_file_deletion", "file_number": 24861} >2018-07-11 07:56:02.468376 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295762468373, "job": 8, "event": "table_file_deletion", "file_number": 24860} >2018-07-11 07:56:02.468390 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295762468388, "job": 8, "event": "table_file_deletion", "file_number": 24859} >2018-07-11 07:56:02.468404 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295762468401, "job": 8, "event": "table_file_deletion", "file_number": 24858} >2018-07-11 07:56:02.468420 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295762468417, "job": 8, "event": "table_file_deletion", "file_number": 24739} >2018-07-11 07:56:02.468433 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295762468430, "job": 8, "event": "table_file_deletion", "file_number": 24736} >2018-07-11 07:56:02.468450 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295762468448, "job": 8, "event": "table_file_deletion", "file_number": 24734} >2018-07-11 07:56:02.468622 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 9] Compacting 1@2 + 2@3 files to L3, score 1.90 >2018-07-11 07:56:02.468634 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 8 Base level 2, inputs: [36627(76MB)], [25203(71MB) 25244(71MB)] > >2018-07-11 07:56:02.468647 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295762468637, "job": 9, "event": "compaction_started", "files_L2": [36627], "files_L3": [25203, 25244], "score": 1.9034, "input_data_size": 229440091} >2018-07-11 07:56:07.427971 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 9] Generated table #36855: 3437542 keys, 76546166 bytes >2018-07-11 07:56:07.428021 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295767427992, "cf_name": "default", "job": 9, "event": "table_file_creation", "file_number": 36855, "file_size": 76546166, "table_properties": {"data_size": 67110734, "index_size": 690177, "filter_size": 8744263, "raw_key_size": 99306779, "raw_average_key_size": 28, "raw_value_size": 20553947, "raw_average_value_size": 5, "num_data_blocks": 16483, "num_entries": 3437542, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1708705", "kMergeOperands": "0"}} >2018-07-11 07:56:11.076846 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 9] Generated table #36856: 2796105 keys, 63966221 bytes >2018-07-11 07:56:11.076881 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295771076866, "cf_name": "default", "job": 9, "event": "table_file_creation", "file_number": 36856, "file_size": 63966221, "table_properties": {"data_size": 56270793, "index_size": 578375, "filter_size": 7116062, "raw_key_size": 80776354, "raw_average_key_size": 28, "raw_value_size": 18399779, "raw_average_value_size": 6, "num_data_blocks": 13821, "num_entries": 2796105, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1248460", "kMergeOperands": "0"}} >2018-07-11 07:56:11.076911 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 9] Compacted 1@2 + 2@3 files to L3 => 140512387 bytes >2018-07-11 07:56:11.092121 7f4922851700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 43433069323 >2018-07-11 07:56:11.092235 7f4922851700 4 rocksdb: (Original Log Time 2018/07/11-07:56:11.092090) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[1 7 65 477 3641 796 0] max score 1.84, MB/sec: 26.7 rd, 16.3 wr, level 3, files in(1, 2) out(2) MB in(76.6, 142.2) out(134.0), read-write-amplify(4.6) write-amplify(1.7) OK, records in: 10233647, records dropped: 4000000 > >2018-07-11 07:56:11.092252 7f4922851700 4 rocksdb: (Original Log Time 2018/07/11-07:56:11.092114) EVENT_LOG_v1 {"time_micros": 1531295771092103, "job": 9, "event": "compaction_finished", "compaction_time_micros": 8608250, "output_level": 3, "num_output_files": 2, "total_output_size": 140512387, "num_input_records": 10233647, "num_output_records": 6233647, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 7, 65, 477, 3641, 796, 0]} >2018-07-11 07:56:11.092825 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295771092820, "job": 9, "event": "table_file_deletion", "file_number": 36627} >2018-07-11 07:56:11.092841 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295771092839, "job": 9, "event": "table_file_deletion", "file_number": 25244} >2018-07-11 07:56:11.092855 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295771092853, "job": 9, "event": "table_file_deletion", "file_number": 25203} >2018-07-11 07:56:11.093032 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 10] Compacting 1@2 + 1@3 files to L3, score 1.84 >2018-07-11 07:56:11.093043 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 9 Base level 2, inputs: [36635(76MB)], [33947(1064B)] > >2018-07-11 07:56:11.093057 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295771093047, "job": 10, "event": "compaction_started", "files_L2": [36635], "files_L3": [33947], "score": 1.84392, "input_data_size": 80349294} >2018-07-11 07:56:14.567449 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 10] Generated table #36857: 3008545 keys, 48764197 bytes >2018-07-11 07:56:14.567498 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295774567482, "cf_name": "default", "job": 10, "event": "table_file_creation", "file_number": 36857, "file_size": 48764197, "table_properties": {"data_size": 40729211, "index_size": 418089, "filter_size": 7615908, "raw_key_size": 86913533, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 9996, "num_entries": 3008545, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "3008545", "kMergeOperands": "0"}} >2018-07-11 07:56:16.671590 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 10] Generated table #36858: 1948621 keys, 31584509 bytes >2018-07-11 07:56:16.671633 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295776671605, "cf_name": "default", "job": 10, "event": "table_file_creation", "file_number": 36858, "file_size": 31584509, "table_properties": {"data_size": 26380152, "index_size": 270582, "filter_size": 4932788, "raw_key_size": 56293491, "raw_average_key_size": 28, "raw_value_size": 12, "raw_average_value_size": 0, "num_data_blocks": 6474, "num_entries": 1948621, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1948620", "kMergeOperands": "0"}} >2018-07-11 07:56:16.671662 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 10] Compacted 1@2 + 1@3 files to L3 => 80348706 bytes >2018-07-11 07:56:16.683856 7f4922851700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 42994844619 >2018-07-11 07:56:16.684016 7f4922851700 4 rocksdb: (Original Log Time 2018/07/11-07:56:16.683824) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[1 7 64 478 3641 796 0] max score 1.78, MB/sec: 14.4 rd, 14.4 wr, level 3, files in(1, 1) out(2) MB in(76.6, 0.0) out(76.6), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 4957166, records dropped: 0 > >2018-07-11 07:56:16.684026 7f4922851700 4 rocksdb: (Original Log Time 2018/07/11-07:56:16.683849) EVENT_LOG_v1 {"time_micros": 1531295776683838, "job": 10, "event": "compaction_finished", "compaction_time_micros": 5578592, "output_level": 3, "num_output_files": 2, "total_output_size": 80348706, "num_input_records": 4957166, "num_output_records": 4957166, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 7, 64, 478, 3641, 796, 0]} >2018-07-11 07:56:16.684635 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295776684631, "job": 10, "event": "table_file_deletion", "file_number": 36635} >2018-07-11 07:56:16.684664 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295776684661, "job": 10, "event": "table_file_deletion", "file_number": 33947} >2018-07-11 07:56:16.684829 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 11] Compacting 1@2 + 14@3 files to L3, score 1.78 >2018-07-11 07:56:16.684844 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 10 Base level 2, inputs: [36588(76MB)], [30066(71MB) 30067(1064B) 30073(39MB) 30074(31MB) 2254(71MB) 2274(71MB) 2289(71MB) 2323(71MB) 2340(71MB) 2358(71MB) 2376(71MB) 2406(71MB) 20426(1169B) 20427(71MB)] > >2018-07-11 07:56:16.684859 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295776684848, "job": 11, "event": "compaction_started", "files_L2": [36588], "files_L3": [30066, 30067, 30073, 30074, 2254, 2274, 2289, 2323, 2340, 2358, 2376, 2406, 20426, 20427], "score": 1.78445, "input_data_size": 900356772} >2018-07-11 07:56:18.418142 7f491c044700 1 osd.10 pg_epoch: 508 pg[13.5s2( v 377'9560 (0'0,377'9560] local-lis/les=500/501 n=2691 ec=345/345 lis/c 502/500 les/c/f 503/501/0 504/508/487) [4,3,10,11,2147483647,8]/[4,3,10,11,8,8]p4(0) r=2 lpr=508 pi=[500,508)/1 crt=377'9552 lcod 0'0 remapped NOTIFY] start_peering_interval up [4,3,10,11,2147483647,8] -> [4,3,10,11,2147483647,8], acting [4,3,2147483647,11,8,8] -> [4,3,10,11,8,8], acting_primary 4(0) -> 4, up_primary 4(0) -> 4, role -1 -> 2, features acting 4611087853745930235 upacting 4611087853745930235 >2018-07-11 07:56:18.418239 7f491c044700 1 osd.10 pg_epoch: 508 pg[13.5s2( v 377'9560 (0'0,377'9560] local-lis/les=500/501 n=2691 ec=345/345 lis/c 502/500 les/c/f 503/501/0 504/508/487) [4,3,10,11,2147483647,8]/[4,3,10,11,8,8]p4(0) r=2 lpr=508 pi=[500,508)/1 crt=377'9552 lcod 0'0 remapped NOTIFY] state<Start>: transitioning to Stray >2018-07-11 07:56:20.185713 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 11] Generated table #36859: 2711178 keys, 74727227 bytes >2018-07-11 07:56:20.185758 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295780185732, "cf_name": "default", "job": 11, "event": "table_file_creation", "file_number": 36859, "file_size": 74727227, "table_properties": {"data_size": 67110551, "index_size": 689904, "filter_size": 6925780, "raw_key_size": 78322933, "raw_average_key_size": 28, "raw_value_size": 30377700, "raw_average_value_size": 11, "num_data_blocks": 16489, "num_entries": 2711178, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "156045", "kMergeOperands": "0"}} >2018-07-11 07:56:23.202887 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 11] Generated table #36860: 2565667 keys, 60447923 bytes >2018-07-11 07:56:23.202931 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295783202904, "cf_name": "default", "job": 11, "event": "table_file_creation", "file_number": 36860, "file_size": 60447923, "table_properties": {"data_size": 53365658, "index_size": 548104, "filter_size": 6533170, "raw_key_size": 74119274, "raw_average_key_size": 28, "raw_value_size": 18614045, "raw_average_value_size": 7, "num_data_blocks": 13108, "num_entries": 2565667, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1000000", "kMergeOperands": "0"}} >2018-07-11 07:56:26.737135 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 11] Generated table #36861: 3106037 keys, 75716900 bytes >2018-07-11 07:56:26.737182 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295786737155, "cf_name": "default", "job": 11, "event": "table_file_creation", "file_number": 36861, "file_size": 75716900, "table_properties": {"data_size": 67111702, "index_size": 689875, "filter_size": 7914331, "raw_key_size": 89729968, "raw_average_key_size": 28, "raw_value_size": 25038449, "raw_average_value_size": 8, "num_data_blocks": 16486, "num_entries": 3106037, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1000000", "kMergeOperands": "0"}} >2018-07-11 07:56:29.902603 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 11] Generated table #36862: 2638240 keys, 74546007 bytes >2018-07-11 07:56:29.902639 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295789902623, "cf_name": "default", "job": 11, "event": "table_file_creation", "file_number": 36862, "file_size": 74546007, "table_properties": {"data_size": 67112111, "index_size": 689725, "filter_size": 6743181, "raw_key_size": 76215822, "raw_average_key_size": 28, "raw_value_size": 31365742, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:56:33.139632 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 11] Generated table #36863: 2638240 keys, 74546641 bytes >2018-07-11 07:56:33.139666 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295793139651, "cf_name": "default", "job": 11, "event": "table_file_creation", "file_number": 36863, "file_size": 74546641, "table_properties": {"data_size": 67112133, "index_size": 690337, "filter_size": 6743181, "raw_key_size": 76215830, "raw_average_key_size": 28, "raw_value_size": 31365750, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:56:35.260899 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 11] Generated table #36864: 1687892 keys, 47693299 bytes >2018-07-11 07:56:35.260946 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295795260930, "cf_name": "default", "job": 11, "event": "table_file_creation", "file_number": 36864, "file_size": 47693299, "table_properties": {"data_size": 42936937, "index_size": 441229, "filter_size": 4314145, "raw_key_size": 48761326, "raw_average_key_size": 28, "raw_value_size": 20067162, "raw_average_value_size": 11, "num_data_blocks": 10550, "num_entries": 1687892, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:56:38.438882 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 11] Generated table #36865: 2638241 keys, 74545920 bytes >2018-07-11 07:56:38.438930 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295798438914, "cf_name": "default", "job": 11, "event": "table_file_creation", "file_number": 36865, "file_size": 74545920, "table_properties": {"data_size": 67112158, "index_size": 689589, "filter_size": 6743183, "raw_key_size": 76215848, "raw_average_key_size": 28, "raw_value_size": 31365751, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:56:41.635779 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 11] Generated table #36866: 2638241 keys, 74545866 bytes >2018-07-11 07:56:41.635823 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295801635796, "cf_name": "default", "job": 11, "event": "table_file_creation", "file_number": 36866, "file_size": 74545866, "table_properties": {"data_size": 67112098, "index_size": 689595, "filter_size": 6743183, "raw_key_size": 76215859, "raw_average_key_size": 28, "raw_value_size": 31365762, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:56:44.802531 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 11] Generated table #36867: 2638241 keys, 74545901 bytes >2018-07-11 07:56:44.802577 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295804802562, "cf_name": "default", "job": 11, "event": "table_file_creation", "file_number": 36867, "file_size": 74545901, "table_properties": {"data_size": 67112141, "index_size": 689587, "filter_size": 6743183, "raw_key_size": 76215848, "raw_average_key_size": 28, "raw_value_size": 31365751, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:56:48.006658 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 11] Generated table #36868: 2638241 keys, 74545813 bytes >2018-07-11 07:56:48.006707 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295808006691, "cf_name": "default", "job": 11, "event": "table_file_creation", "file_number": 36868, "file_size": 74545813, "table_properties": {"data_size": 67112099, "index_size": 689541, "filter_size": 6743183, "raw_key_size": 76215847, "raw_average_key_size": 28, "raw_value_size": 31365750, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:56:51.117008 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 11] Generated table #36869: 2638241 keys, 74545959 bytes >2018-07-11 07:56:51.117059 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295811117043, "cf_name": "default", "job": 11, "event": "table_file_creation", "file_number": 36869, "file_size": 74545959, "table_properties": {"data_size": 67112125, "index_size": 689661, "filter_size": 6743183, "raw_key_size": 76215852, "raw_average_key_size": 28, "raw_value_size": 31365755, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:56:51.167454 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 11] Generated table #36870: 1 keys, 1047 bytes >2018-07-11 07:56:51.167481 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295811167468, "cf_name": "default", "job": 11, "event": "table_file_creation", "file_number": 36870, "file_size": 1047, "table_properties": {"data_size": 40, "index_size": 27, "filter_size": 18, "raw_key_size": 24, "raw_average_key_size": 24, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 1, "num_entries": 1, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1", "kMergeOperands": "0"}} >2018-07-11 07:56:55.391881 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 11] Generated table #36871: 3416134 keys, 76491382 bytes >2018-07-11 07:56:55.391929 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295815391914, "cf_name": "default", "job": 11, "event": "table_file_creation", "file_number": 36871, "file_size": 76491382, "table_properties": {"data_size": 67110066, "index_size": 689658, "filter_size": 8690666, "raw_key_size": 98688311, "raw_average_key_size": 28, "raw_value_size": 20842787, "raw_average_value_size": 6, "num_data_blocks": 16483, "num_entries": 3416134, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1663002", "kMergeOperands": "0"}} >2018-07-11 07:56:56.961429 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 11] Generated table #36872: 1349227 keys, 28472773 bytes >2018-07-11 07:56:56.961473 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295816961446, "cf_name": "default", "job": 11, "event": "table_file_creation", "file_number": 36872, "file_size": 28472773, "table_properties": {"data_size": 24788477, "index_size": 254399, "filter_size": 3428907, "raw_key_size": 38977655, "raw_average_key_size": 28, "raw_value_size": 6516496, "raw_average_value_size": 4, "num_data_blocks": 6088, "num_entries": 1349227, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "801110", "kMergeOperands": "0"}} >2018-07-11 07:56:56.961726 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 11] Compacted 1@2 + 14@3 files to L3 => 885372658 bytes >2018-07-11 07:56:56.974741 7f4922851700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 42378511421 >2018-07-11 07:56:56.974964 7f4922851700 4 rocksdb: (Original Log Time 2018/07/11-07:56:56.974706) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[1 7 63 478 3641 796 0] max score 1.72, MB/sec: 22.4 rd, 22.0 wr, level 3, files in(1, 14) out(14) MB in(76.6, 782.0) out(844.4), read-write-amplify(22.2) write-amplify(11.0) OK, records in: 33977815, records dropped: 673994 > >2018-07-11 07:56:56.974974 7f4922851700 4 rocksdb: (Original Log Time 2018/07/11-07:56:56.974732) EVENT_LOG_v1 {"time_micros": 1531295816974720, "job": 11, "event": "compaction_finished", "compaction_time_micros": 40276845, "output_level": 3, "num_output_files": 14, "total_output_size": 885372658, "num_input_records": 33977815, "num_output_records": 33303821, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 7, 63, 478, 3641, 796, 0]} >2018-07-11 07:56:56.975537 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295816975520, "job": 11, "event": "table_file_deletion", "file_number": 36588} >2018-07-11 07:56:56.975554 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295816975552, "job": 11, "event": "table_file_deletion", "file_number": 30074} >2018-07-11 07:56:56.975565 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295816975563, "job": 11, "event": "table_file_deletion", "file_number": 30073} >2018-07-11 07:56:56.975575 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295816975573, "job": 11, "event": "table_file_deletion", "file_number": 30067} >2018-07-11 07:56:56.975599 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295816975597, "job": 11, "event": "table_file_deletion", "file_number": 30066} >2018-07-11 07:56:56.975612 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295816975610, "job": 11, "event": "table_file_deletion", "file_number": 20427} >2018-07-11 07:56:56.975623 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295816975621, "job": 11, "event": "table_file_deletion", "file_number": 20426} >2018-07-11 07:56:56.975638 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295816975635, "job": 11, "event": "table_file_deletion", "file_number": 2406} >2018-07-11 07:56:56.975649 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295816975647, "job": 11, "event": "table_file_deletion", "file_number": 2376} >2018-07-11 07:56:56.975660 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295816975658, "job": 11, "event": "table_file_deletion", "file_number": 2358} >2018-07-11 07:56:56.975672 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295816975669, "job": 11, "event": "table_file_deletion", "file_number": 2340} >2018-07-11 07:56:56.975685 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295816975683, "job": 11, "event": "table_file_deletion", "file_number": 2323} >2018-07-11 07:56:56.975696 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295816975694, "job": 11, "event": "table_file_deletion", "file_number": 2289} >2018-07-11 07:56:56.975710 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295816975708, "job": 11, "event": "table_file_deletion", "file_number": 2274} >2018-07-11 07:56:56.975726 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295816975723, "job": 11, "event": "table_file_deletion", "file_number": 2254} >2018-07-11 07:56:56.975904 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 12] Compacting 1@2 + 13@3 files to L3, score 1.72 >2018-07-11 07:56:56.975920 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 11 Base level 2, inputs: [36540(76MB)], [15281(71MB) 15317(71MB) 15365(71MB) 29568(39MB) 29570(70MB) 29571(71MB) 29572(71MB) 29573(1064B) 29575(71MB) 29576(71MB) 29577(71MB) 29580(30MB) 29581(71MB)] > >2018-07-11 07:56:56.975937 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295816975926, "job": 12, "event": "compaction_started", "files_L2": [36540], "files_L3": [15281, 15317, 15365, 29568, 29570, 29571, 29572, 29573, 29575, 29576, 29577, 29580, 29581], "score": 1.72497, "input_data_size": 899196072} >2018-07-11 07:57:00.695071 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 12] Generated table #36873: 2843284 keys, 75058666 bytes >2018-07-11 07:57:00.695116 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295820695101, "cf_name": "default", "job": 12, "event": "table_file_creation", "file_number": 36873, "file_size": 75058666, "table_properties": {"data_size": 67110965, "index_size": 690196, "filter_size": 7256513, "raw_key_size": 82139325, "raw_average_key_size": 28, "raw_value_size": 28591396, "raw_average_value_size": 10, "num_data_blocks": 16488, "num_entries": 2843284, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "438400", "kMergeOperands": "0"}} >2018-07-11 07:57:03.997449 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 12] Generated table #36874: 2638240 keys, 74546210 bytes >2018-07-11 07:57:03.997502 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295823997483, "cf_name": "default", "job": 12, "event": "table_file_creation", "file_number": 36874, "file_size": 74546210, "table_properties": {"data_size": 67112142, "index_size": 689897, "filter_size": 6743181, "raw_key_size": 76215824, "raw_average_key_size": 28, "raw_value_size": 31365744, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:57:07.359564 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 12] Generated table #36875: 2638240 keys, 74546041 bytes >2018-07-11 07:57:07.359615 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295827359599, "cf_name": "default", "job": 12, "event": "table_file_creation", "file_number": 36875, "file_size": 74546041, "table_properties": {"data_size": 67112112, "index_size": 689758, "filter_size": 6743181, "raw_key_size": 76215827, "raw_average_key_size": 28, "raw_value_size": 31365747, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:57:09.534680 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 12] Generated table #36876: 1715921 keys, 48485804 bytes >2018-07-11 07:57:09.534713 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295829534698, "cf_name": "default", "job": 12, "event": "table_file_creation", "file_number": 36876, "file_size": 48485804, "table_properties": {"data_size": 43649997, "index_size": 449034, "filter_size": 4385785, "raw_key_size": 49571052, "raw_average_key_size": 28, "raw_value_size": 20400395, "raw_average_value_size": 11, "num_data_blocks": 10725, "num_entries": 1715921, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:57:12.980148 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 12] Generated table #36877: 2638240 keys, 74546284 bytes >2018-07-11 07:57:12.980199 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295832980183, "cf_name": "default", "job": 12, "event": "table_file_creation", "file_number": 36877, "file_size": 74546284, "table_properties": {"data_size": 67112145, "index_size": 689967, "filter_size": 6743182, "raw_key_size": 76215831, "raw_average_key_size": 28, "raw_value_size": 31365751, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:57:15.139203 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 12] Generated table #36878: 1482240 keys, 41882464 bytes >2018-07-11 07:57:15.139255 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295835139234, "cf_name": "default", "job": 12, "event": "table_file_creation", "file_number": 36878, "file_size": 41882464, "table_properties": {"data_size": 37705582, "index_size": 387374, "filter_size": 3788520, "raw_key_size": 42820269, "raw_average_key_size": 28, "raw_value_size": 17622189, "raw_average_value_size": 11, "num_data_blocks": 9265, "num_entries": 1482240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:57:18.436104 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 12] Generated table #36879: 2638241 keys, 74546041 bytes >2018-07-11 07:57:18.436155 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295838436139, "cf_name": "default", "job": 12, "event": "table_file_creation", "file_number": 36879, "file_size": 74546041, "table_properties": {"data_size": 67112147, "index_size": 689721, "filter_size": 6743183, "raw_key_size": 76215846, "raw_average_key_size": 28, "raw_value_size": 31365749, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:57:20.941043 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 12] Generated table #36880: 2156004 keys, 48873264 bytes >2018-07-11 07:57:20.941090 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295840941074, "cf_name": "default", "job": 12, "event": "table_file_creation", "file_number": 36880, "file_size": 48873264, "table_properties": {"data_size": 42944617, "index_size": 441555, "filter_size": 5486101, "raw_key_size": 62284568, "raw_average_key_size": 28, "raw_value_size": 13743610, "raw_average_value_size": 6, "num_data_blocks": 10548, "num_entries": 2156004, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1000000", "kMergeOperands": "0"}} >2018-07-11 07:57:24.216117 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 12] Generated table #36881: 2638241 keys, 74546001 bytes >2018-07-11 07:57:24.216149 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295844216134, "cf_name": "default", "job": 12, "event": "table_file_creation", "file_number": 36881, "file_size": 74546001, "table_properties": {"data_size": 67112114, "index_size": 689714, "filter_size": 6743183, "raw_key_size": 76215838, "raw_average_key_size": 28, "raw_value_size": 31365741, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:57:27.564073 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 12] Generated table #36882: 2638240 keys, 74545911 bytes >2018-07-11 07:57:27.564110 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295847564095, "cf_name": "default", "job": 12, "event": "table_file_creation", "file_number": 36882, "file_size": 74545911, "table_properties": {"data_size": 67112138, "index_size": 689602, "filter_size": 6743181, "raw_key_size": 76215820, "raw_average_key_size": 28, "raw_value_size": 31365740, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:57:30.736012 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 12] Generated table #36883: 2638241 keys, 74545819 bytes >2018-07-11 07:57:30.736045 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295850736031, "cf_name": "default", "job": 12, "event": "table_file_creation", "file_number": 36883, "file_size": 74545819, "table_properties": {"data_size": 67112157, "index_size": 689489, "filter_size": 6743183, "raw_key_size": 76215850, "raw_average_key_size": 28, "raw_value_size": 31365753, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:57:33.301020 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 12] Generated table #36884: 2149603 keys, 48691933 bytes >2018-07-11 07:57:33.301066 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295853301038, "cf_name": "default", "job": 12, "event": "table_file_creation", "file_number": 36884, "file_size": 48691933, "table_properties": {"data_size": 42781682, "index_size": 439522, "filter_size": 5469738, "raw_key_size": 62099645, "raw_average_key_size": 28, "raw_value_size": 13667504, "raw_average_value_size": 6, "num_data_blocks": 10508, "num_entries": 2149603, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1000000", "kMergeOperands": "0"}} >2018-07-11 07:57:36.502368 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 12] Generated table #36885: 2638519 keys, 74546637 bytes >2018-07-11 07:57:36.502414 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295856502399, "cf_name": "default", "job": 12, "event": "table_file_creation", "file_number": 36885, "file_size": 74546637, "table_properties": {"data_size": 67112172, "index_size": 689595, "filter_size": 6743879, "raw_key_size": 76223878, "raw_average_key_size": 28, "raw_value_size": 31361986, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638519, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "596", "kMergeOperands": "0"}} >2018-07-11 07:57:37.628017 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 12] Generated table #36886: 999087 keys, 16194110 bytes >2018-07-11 07:57:37.628053 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295857628035, "cf_name": "default", "job": 12, "event": "table_file_creation", "file_number": 36886, "file_size": 16194110, "table_properties": {"data_size": 13525510, "index_size": 138495, "filter_size": 2529118, "raw_key_size": 28862531, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 3320, "num_entries": 999087, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "999087", "kMergeOperands": "0"}} >2018-07-11 07:57:39.137473 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 12] Generated table #36887: 1457826 keys, 23629548 bytes >2018-07-11 07:57:39.137521 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295859137506, "cf_name": "default", "job": 12, "event": "table_file_creation", "file_number": 36887, "file_size": 23629548, "table_properties": {"data_size": 19735848, "index_size": 202329, "filter_size": 3690384, "raw_key_size": 42114961, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 4844, "num_entries": 1457826, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1457826", "kMergeOperands": "0"}} >2018-07-11 07:57:39.137562 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 12] Compacted 1@2 + 13@3 files to L3 => 899184733 bytes >2018-07-11 07:57:39.148980 7f4922851700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 41903297879 >2018-07-11 07:57:39.149455 7f4922851700 4 rocksdb: (Original Log Time 2018/07/11-07:57:39.148947) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[1 7 62 480 3641 796 0] max score 1.67, MB/sec: 21.3 rd, 21.3 wr, level 3, files in(1, 13) out(15) MB in(76.6, 781.0) out(857.5), read-write-amplify(22.4) write-amplify(11.2) OK, records in: 33910801, records dropped: 634 > >2018-07-11 07:57:39.149465 7f4922851700 4 rocksdb: (Original Log Time 2018/07/11-07:57:39.148972) EVENT_LOG_v1 {"time_micros": 1531295859148961, "job": 12, "event": "compaction_finished", "compaction_time_micros": 42161601, "output_level": 3, "num_output_files": 15, "total_output_size": 899184733, "num_input_records": 33910801, "num_output_records": 33910167, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 7, 62, 480, 3641, 796, 0]} >2018-07-11 07:57:39.150023 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295859150007, "job": 12, "event": "table_file_deletion", "file_number": 36540} >2018-07-11 07:57:39.150041 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295859150039, "job": 12, "event": "table_file_deletion", "file_number": 29581} >2018-07-11 07:57:39.150053 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295859150051, "job": 12, "event": "table_file_deletion", "file_number": 29580} >2018-07-11 07:57:39.150064 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295859150062, "job": 12, "event": "table_file_deletion", "file_number": 29577} >2018-07-11 07:57:39.150075 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295859150073, "job": 12, "event": "table_file_deletion", "file_number": 29576} >2018-07-11 07:57:39.150088 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295859150086, "job": 12, "event": "table_file_deletion", "file_number": 29575} >2018-07-11 07:57:39.150101 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295859150099, "job": 12, "event": "table_file_deletion", "file_number": 29573} >2018-07-11 07:57:39.150115 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295859150113, "job": 12, "event": "table_file_deletion", "file_number": 29572} >2018-07-11 07:57:39.150128 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295859150126, "job": 12, "event": "table_file_deletion", "file_number": 29571} >2018-07-11 07:57:39.150140 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295859150138, "job": 12, "event": "table_file_deletion", "file_number": 29570} >2018-07-11 07:57:39.150155 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295859150151, "job": 12, "event": "table_file_deletion", "file_number": 29568} >2018-07-11 07:57:39.150173 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295859150171, "job": 12, "event": "table_file_deletion", "file_number": 15365} >2018-07-11 07:57:39.150189 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295859150186, "job": 12, "event": "table_file_deletion", "file_number": 15317} >2018-07-11 07:57:39.150203 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295859150201, "job": 12, "event": "table_file_deletion", "file_number": 15281} >2018-07-11 07:57:39.150387 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 13] Compacting 1@2 + 5@3 files to L3, score 1.67 >2018-07-11 07:57:39.150400 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 12 Base level 2, inputs: [36518(76MB)], [29584(39MB) 29585(71MB) 29588(31MB) 29590(32MB) 18163(15MB)] > >2018-07-11 07:57:39.150415 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295859150404, "job": 13, "event": "compaction_started", "files_L2": [36518], "files_L3": [29584, 29585, 29588, 29590, 18163], "score": 1.66607, "input_data_size": 278786751} >2018-07-11 07:57:43.375287 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 13] Generated table #36888: 3306228 keys, 76216949 bytes >2018-07-11 07:57:43.375335 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295863375319, "cf_name": "default", "job": 13, "event": "table_file_creation", "file_number": 36888, "file_size": 76216949, "table_properties": {"data_size": 67110357, "index_size": 690087, "filter_size": 8415513, "raw_key_size": 95513265, "raw_average_key_size": 28, "raw_value_size": 22329545, "raw_average_value_size": 6, "num_data_blocks": 16484, "num_entries": 3306228, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1428042", "kMergeOperands": "0"}} >2018-07-11 07:57:44.335127 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 13] Generated table #36889: 760060 keys, 21476670 bytes >2018-07-11 07:57:44.335159 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295864335144, "cf_name": "default", "job": 13, "event": "table_file_creation", "file_number": 36889, "file_size": 21476670, "table_properties": {"data_size": 19334581, "index_size": 198437, "filter_size": 1942666, "raw_key_size": 21957288, "raw_average_key_size": 28, "raw_value_size": 9036268, "raw_average_value_size": 11, "num_data_blocks": 4751, "num_entries": 760060, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:57:47.576549 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 13] Generated table #36890: 2638241 keys, 74546045 bytes >2018-07-11 07:57:47.576600 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295867576571, "cf_name": "default", "job": 13, "event": "table_file_creation", "file_number": 36890, "file_size": 74546045, "table_properties": {"data_size": 67112127, "index_size": 689745, "filter_size": 6743183, "raw_key_size": 76215856, "raw_average_key_size": 28, "raw_value_size": 31365759, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:57:48.597970 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 13] Generated table #36891: 1000000 keys, 16208895 bytes >2018-07-11 07:57:48.598014 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295868597987, "cf_name": "default", "job": 13, "event": "table_file_creation", "file_number": 36891, "file_size": 16208895, "table_properties": {"data_size": 13537837, "index_size": 138642, "filter_size": 2531429, "raw_key_size": 28888890, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 3323, "num_entries": 1000000, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1000000", "kMergeOperands": "0"}} >2018-07-11 07:57:51.451570 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 13] Generated table #36892: 2482410 keys, 58095967 bytes >2018-07-11 07:57:51.451607 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295871451592, "cf_name": "default", "job": 13, "event": "table_file_creation", "file_number": 36892, "file_size": 58095967, "table_properties": {"data_size": 51247792, "index_size": 526812, "filter_size": 6320372, "raw_key_size": 71714077, "raw_average_key_size": 28, "raw_value_size": 17624217, "raw_average_value_size": 7, "num_data_blocks": 12588, "num_entries": 2482410, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1000000", "kMergeOperands": "0"}} >2018-07-11 07:57:53.172753 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 13] Generated table #36893: 1557589 keys, 32242059 bytes >2018-07-11 07:57:53.172800 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295873172784, "cf_name": "default", "job": 13, "event": "table_file_creation", "file_number": 36893, "file_size": 32242059, "table_properties": {"data_size": 27996630, "index_size": 287284, "filter_size": 3957155, "raw_key_size": 44997005, "raw_average_key_size": 28, "raw_value_size": 6903575, "raw_average_value_size": 4, "num_data_blocks": 6875, "num_entries": 1557589, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "976914", "kMergeOperands": "0"}} >2018-07-11 07:57:53.172835 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 13] Compacted 1@2 + 5@3 files to L3 => 278786585 bytes >2018-07-11 07:57:53.190403 7f4922851700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 41411290251 >2018-07-11 07:57:53.190632 7f4922851700 4 rocksdb: (Original Log Time 2018/07/11-07:57:53.190367) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[1 7 61 481 3641 796 0] max score 1.61, MB/sec: 19.9 rd, 19.9 wr, level 3, files in(1, 5) out(6) MB in(76.0, 189.9) out(265.9), read-write-amplify(7.0) write-amplify(3.5) OK, records in: 11744528, records dropped: 0 > >2018-07-11 07:57:53.190642 7f4922851700 4 rocksdb: (Original Log Time 2018/07/11-07:57:53.190395) EVENT_LOG_v1 {"time_micros": 1531295873190381, "job": 13, "event": "compaction_finished", "compaction_time_micros": 14022403, "output_level": 3, "num_output_files": 6, "total_output_size": 278786585, "num_input_records": 11744528, "num_output_records": 11744528, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 7, 61, 481, 3641, 796, 0]} >2018-07-11 07:57:53.191259 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295873191253, "job": 13, "event": "table_file_deletion", "file_number": 36518} >2018-07-11 07:57:53.191292 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295873191288, "job": 13, "event": "table_file_deletion", "file_number": 29590} >2018-07-11 07:57:53.191318 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295873191314, "job": 13, "event": "table_file_deletion", "file_number": 29588} >2018-07-11 07:57:53.191345 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295873191341, "job": 13, "event": "table_file_deletion", "file_number": 29585} >2018-07-11 07:57:53.191367 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295873191363, "job": 13, "event": "table_file_deletion", "file_number": 29584} >2018-07-11 07:57:53.191395 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295873191392, "job": 13, "event": "table_file_deletion", "file_number": 18163} >2018-07-11 07:57:53.191601 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 14] Compacting 1@2 + 24@3 files to L3, score 1.61 >2018-07-11 07:57:53.191627 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 13 Base level 2, inputs: [36605(75MB)], [8205(71MB) 8206(71MB) 8208(54MB) 8740(1113B) 8489(71MB) 8494(15MB) 7243(71MB) 7246(71MB) 7249(71MB) 7201(71MB) 7208(71MB) 7212(71MB) 7234(71MB) 8611(71MB) 8612(71MB) 8620(39MB) 8624(62MB) 8625(71MB) 8629(71MB) 8633(1223B) 7689(71MB) 7722(71MB) 7727(71MB) 7757(71MB)] > >2018-07-11 07:57:53.191654 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295873191636, "job": 14, "event": "compaction_started", "files_L2": [36605], "files_L3": [8205, 8206, 8208, 8740, 8489, 8494, 7243, 7246, 7249, 7201, 7208, 7212, 7234, 8611, 8612, 8620, 8624, 8625, 8629, 8633, 7689, 7722, 7727, 7757], "score": 1.61187, "input_data_size": 1602709684} >2018-07-11 07:57:57.231144 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 14] Generated table #36894: 3106038 keys, 75717406 bytes >2018-07-11 07:57:57.231193 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295877231177, "cf_name": "default", "job": 14, "event": "table_file_creation", "file_number": 36894, "file_size": 75717406, "table_properties": {"data_size": 67111679, "index_size": 690402, "filter_size": 7914333, "raw_key_size": 89729986, "raw_average_key_size": 28, "raw_value_size": 25038450, "raw_average_value_size": 8, "num_data_blocks": 16486, "num_entries": 3106038, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1000000", "kMergeOperands": "0"}} >2018-07-11 07:58:00.332679 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 14] Generated table #36895: 2638240 keys, 74546028 bytes >2018-07-11 07:58:00.332716 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295880332699, "cf_name": "default", "job": 14, "event": "table_file_creation", "file_number": 36895, "file_size": 74546028, "table_properties": {"data_size": 67112164, "index_size": 689693, "filter_size": 6743181, "raw_key_size": 76215829, "raw_average_key_size": 28, "raw_value_size": 31365749, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:58:03.521533 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 14] Generated table #36896: 2572920 keys, 72700430 bytes >2018-07-11 07:58:03.521578 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295883521551, "cf_name": "default", "job": 14, "event": "table_file_creation", "file_number": 36896, "file_size": 72700430, "table_properties": {"data_size": 65450504, "index_size": 672718, "filter_size": 6576219, "raw_key_size": 74328798, "raw_average_key_size": 28, "raw_value_size": 30589158, "raw_average_value_size": 11, "num_data_blocks": 16081, "num_entries": 2572920, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:58:06.619339 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 14] Generated table #36897: 2638241 keys, 74545943 bytes >2018-07-11 07:58:06.619383 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295886619356, "cf_name": "default", "job": 14, "event": "table_file_creation", "file_number": 36897, "file_size": 74545943, "table_properties": {"data_size": 67112159, "index_size": 689611, "filter_size": 6743183, "raw_key_size": 76215852, "raw_average_key_size": 28, "raw_value_size": 31365755, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:58:10.279317 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 14] Generated table #36898: 3106037 keys, 75716767 bytes >2018-07-11 07:58:10.279363 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295890279347, "cf_name": "default", "job": 14, "event": "table_file_creation", "file_number": 36898, "file_size": 75716767, "table_properties": {"data_size": 67111720, "index_size": 689725, "filter_size": 7914330, "raw_key_size": 89729963, "raw_average_key_size": 28, "raw_value_size": 25038444, "raw_average_value_size": 8, "num_data_blocks": 16486, "num_entries": 3106037, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1000000", "kMergeOperands": "0"}} >2018-07-11 07:58:13.533659 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 14] Generated table #36899: 2638239 keys, 74546048 bytes >2018-07-11 07:58:13.533704 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295893533689, "cf_name": "default", "job": 14, "event": "table_file_creation", "file_number": 36899, "file_size": 74546048, "table_properties": {"data_size": 67112080, "index_size": 689799, "filter_size": 6743179, "raw_key_size": 76215796, "raw_average_key_size": 28, "raw_value_size": 31365733, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638239, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:58:16.746094 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 14] Generated table #36900: 2638240 keys, 74546708 bytes >2018-07-11 07:58:16.746140 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295896746124, "cf_name": "default", "job": 14, "event": "table_file_creation", "file_number": 36900, "file_size": 74546708, "table_properties": {"data_size": 67112150, "index_size": 690387, "filter_size": 6743181, "raw_key_size": 76215823, "raw_average_key_size": 28, "raw_value_size": 31365743, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:58:20.087362 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 14] Generated table #36901: 2638240 keys, 74546088 bytes >2018-07-11 07:58:20.087418 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295900087401, "cf_name": "default", "job": 14, "event": "table_file_creation", "file_number": 36901, "file_size": 74546088, "table_properties": {"data_size": 67112105, "index_size": 689812, "filter_size": 6743181, "raw_key_size": 76215817, "raw_average_key_size": 28, "raw_value_size": 31365737, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:58:21.981693 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 14] Generated table #36902: 1482241 keys, 41882566 bytes >2018-07-11 07:58:21.981737 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295901981710, "cf_name": "default", "job": 14, "event": "table_file_creation", "file_number": 36902, "file_size": 41882566, "table_properties": {"data_size": 37705636, "index_size": 387419, "filter_size": 3788523, "raw_key_size": 42820303, "raw_average_key_size": 28, "raw_value_size": 17622206, "raw_average_value_size": 11, "num_data_blocks": 9265, "num_entries": 1482241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:58:25.409938 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 14] Generated table #36903: 2638240 keys, 74546284 bytes >2018-07-11 07:58:25.409987 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295905409972, "cf_name": "default", "job": 14, "event": "table_file_creation", "file_number": 36903, "file_size": 74546284, "table_properties": {"data_size": 67112150, "index_size": 689963, "filter_size": 6743181, "raw_key_size": 76215829, "raw_average_key_size": 28, "raw_value_size": 31365749, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:58:28.597045 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 14] Generated table #36904: 2638240 keys, 74546093 bytes >2018-07-11 07:58:28.597096 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295908597079, "cf_name": "default", "job": 14, "event": "table_file_creation", "file_number": 36904, "file_size": 74546093, "table_properties": {"data_size": 67112125, "index_size": 689797, "filter_size": 6743181, "raw_key_size": 76215833, "raw_average_key_size": 28, "raw_value_size": 31365753, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:58:31.831104 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 14] Generated table #36905: 2638240 keys, 74545794 bytes >2018-07-11 07:58:31.831151 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295911831123, "cf_name": "default", "job": 14, "event": "table_file_creation", "file_number": 36905, "file_size": 74545794, "table_properties": {"data_size": 67112157, "index_size": 689466, "filter_size": 6743181, "raw_key_size": 76215823, "raw_average_key_size": 28, "raw_value_size": 31365743, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:58:34.654941 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 14] Generated table #36906: 2638241 keys, 74546392 bytes >2018-07-11 07:58:34.654986 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295914654971, "cf_name": "default", "job": 14, "event": "table_file_creation", "file_number": 36906, "file_size": 74546392, "table_properties": {"data_size": 67112123, "index_size": 690096, "filter_size": 6743183, "raw_key_size": 76215855, "raw_average_key_size": 28, "raw_value_size": 31365758, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:58:37.213944 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 14] Generated table #36907: 2266856 keys, 64052484 bytes >2018-07-11 07:58:37.213981 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295917213966, "cf_name": "default", "job": 14, "event": "table_file_creation", "file_number": 36907, "file_size": 64052484, "table_properties": {"data_size": 57664768, "index_size": 592790, "filter_size": 5793937, "raw_key_size": 65486947, "raw_average_key_size": 28, "raw_value_size": 26950395, "raw_average_value_size": 11, "num_data_blocks": 14168, "num_entries": 2266856, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:58:39.880792 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 14] Generated table #36908: 2471390 keys, 57784241 bytes >2018-07-11 07:58:39.880840 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295919880824, "cf_name": "default", "job": 14, "event": "table_file_creation", "file_number": 36908, "file_size": 57784241, "table_properties": {"data_size": 50967428, "index_size": 523617, "filter_size": 6292205, "raw_key_size": 71395705, "raw_average_key_size": 28, "raw_value_size": 17493185, "raw_average_value_size": 7, "num_data_blocks": 12519, "num_entries": 2471390, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1000000", "kMergeOperands": "0"}} >2018-07-11 07:58:42.688878 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 14] Generated table #36909: 2638239 keys, 74546420 bytes >2018-07-11 07:58:42.688911 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295922688897, "cf_name": "default", "job": 14, "event": "table_file_creation", "file_number": 36909, "file_size": 74546420, "table_properties": {"data_size": 67112138, "index_size": 690113, "filter_size": 6743179, "raw_key_size": 76215797, "raw_average_key_size": 28, "raw_value_size": 31365734, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638239, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:58:45.709794 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 14] Generated table #36910: 2638239 keys, 74546942 bytes >2018-07-11 07:58:45.709840 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295925709813, "cf_name": "default", "job": 14, "event": "table_file_creation", "file_number": 36910, "file_size": 74546942, "table_properties": {"data_size": 67112130, "index_size": 690643, "filter_size": 6743179, "raw_key_size": 76215796, "raw_average_key_size": 28, "raw_value_size": 31365733, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638239, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:58:48.581747 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 14] Generated table #36911: 2638239 keys, 74546630 bytes >2018-07-11 07:58:48.581802 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295928581785, "cf_name": "default", "job": 14, "event": "table_file_creation", "file_number": 36911, "file_size": 74546630, "table_properties": {"data_size": 67112111, "index_size": 690350, "filter_size": 6743179, "raw_key_size": 76215793, "raw_average_key_size": 28, "raw_value_size": 31365730, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638239, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:58:51.336274 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 14] Generated table #36912: 2638241 keys, 74546348 bytes >2018-07-11 07:58:51.336322 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295931336295, "cf_name": "default", "job": 14, "event": "table_file_creation", "file_number": 36912, "file_size": 74546348, "table_properties": {"data_size": 67112135, "index_size": 690039, "filter_size": 6743184, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:58:53.036206 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 14] Generated table #36913: 1482240 keys, 41882556 bytes >2018-07-11 07:58:53.036256 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295933036235, "cf_name": "default", "job": 14, "event": "table_file_creation", "file_number": 36913, "file_size": 41882556, "table_properties": {"data_size": 37705577, "index_size": 387471, "filter_size": 3788520, "raw_key_size": 42820275, "raw_average_key_size": 28, "raw_value_size": 17622195, "raw_average_value_size": 11, "num_data_blocks": 9265, "num_entries": 1482240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:58:55.947984 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 14] Generated table #36914: 2638240 keys, 74546130 bytes >2018-07-11 07:58:55.948030 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295935948003, "cf_name": "default", "job": 14, "event": "table_file_creation", "file_number": 36914, "file_size": 74546130, "table_properties": {"data_size": 67112115, "index_size": 689844, "filter_size": 6743181, "raw_key_size": 76215830, "raw_average_key_size": 28, "raw_value_size": 31365750, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:58:59.259814 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 14] Generated table #36915: 3106039 keys, 75716793 bytes >2018-07-11 07:58:59.259864 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295939259847, "cf_name": "default", "job": 14, "event": "table_file_creation", "file_number": 36915, "file_size": 75716793, "table_properties": {"data_size": 67111759, "index_size": 689707, "filter_size": 7914335, "raw_key_size": 89730027, "raw_average_key_size": 28, "raw_value_size": 25038474, "raw_average_value_size": 8, "num_data_blocks": 16486, "num_entries": 3106039, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1000000", "kMergeOperands": "0"}} >2018-07-11 07:59:01.451214 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 14] Generated table #36916: 2052959 keys, 53614846 bytes >2018-07-11 07:59:01.451253 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295941451233, "cf_name": "default", "job": 14, "event": "table_file_creation", "file_number": 36916, "file_size": 53614846, "table_properties": {"data_size": 47883047, "index_size": 492513, "filter_size": 5238296, "raw_key_size": 59307696, "raw_average_key_size": 28, "raw_value_size": 20071026, "raw_average_value_size": 9, "num_data_blocks": 11764, "num_entries": 2052959, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "364742", "kMergeOperands": "0"}} >2018-07-11 07:59:01.451301 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 14] Compacted 1@2 + 24@3 files to L3 => 1602715937 bytes >2018-07-11 07:59:01.464121 7f4922851700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 40898087686 >2018-07-11 07:59:01.464383 7f4922851700 4 rocksdb: (Original Log Time 2018/07/11-07:59:01.464085) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[1 7 60 480 3641 796 0] max score 1.56, MB/sec: 23.5 rd, 23.5 wr, level 3, files in(1, 24) out(23) MB in(76.0, 1452.5) out(1528.5), read-write-amplify(40.2) write-amplify(20.1) OK, records in: 58582079, records dropped: 0 > >2018-07-11 07:59:01.464393 7f4922851700 4 rocksdb: (Original Log Time 2018/07/11-07:59:01.464113) EVENT_LOG_v1 {"time_micros": 1531295941464099, "job": 14, "event": "compaction_finished", "compaction_time_micros": 68259616, "output_level": 3, "num_output_files": 23, "total_output_size": 1602715937, "num_input_records": 58582079, "num_output_records": 58582079, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 7, 60, 480, 3641, 796, 0]} >2018-07-11 07:59:01.465003 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295941464999, "job": 14, "event": "table_file_deletion", "file_number": 36605} >2018-07-11 07:59:01.465019 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295941465016, "job": 14, "event": "table_file_deletion", "file_number": 8740} >2018-07-11 07:59:01.465031 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295941465029, "job": 14, "event": "table_file_deletion", "file_number": 8633} >2018-07-11 07:59:01.465042 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295941465040, "job": 14, "event": "table_file_deletion", "file_number": 8629} >2018-07-11 07:59:01.465053 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295941465051, "job": 14, "event": "table_file_deletion", "file_number": 8625} >2018-07-11 07:59:01.465064 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295941465062, "job": 14, "event": "table_file_deletion", "file_number": 8624} >2018-07-11 07:59:01.465078 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295941465075, "job": 14, "event": "table_file_deletion", "file_number": 8620} >2018-07-11 07:59:01.465089 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295941465087, "job": 14, "event": "table_file_deletion", "file_number": 8612} >2018-07-11 07:59:01.465103 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295941465101, "job": 14, "event": "table_file_deletion", "file_number": 8611} >2018-07-11 07:59:01.465119 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295941465115, "job": 14, "event": "table_file_deletion", "file_number": 8494} >2018-07-11 07:59:01.465133 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295941465131, "job": 14, "event": "table_file_deletion", "file_number": 8489} >2018-07-11 07:59:01.465148 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295941465145, "job": 14, "event": "table_file_deletion", "file_number": 8208} >2018-07-11 07:59:01.465161 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295941465159, "job": 14, "event": "table_file_deletion", "file_number": 8206} >2018-07-11 07:59:01.465176 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295941465172, "job": 14, "event": "table_file_deletion", "file_number": 8205} >2018-07-11 07:59:01.465191 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295941465189, "job": 14, "event": "table_file_deletion", "file_number": 7757} >2018-07-11 07:59:01.465207 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295941465205, "job": 14, "event": "table_file_deletion", "file_number": 7727} >2018-07-11 07:59:01.465220 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295941465218, "job": 14, "event": "table_file_deletion", "file_number": 7722} >2018-07-11 07:59:01.465234 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295941465231, "job": 14, "event": "table_file_deletion", "file_number": 7689} >2018-07-11 07:59:01.465254 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295941465251, "job": 14, "event": "table_file_deletion", "file_number": 7249} >2018-07-11 07:59:01.465268 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295941465265, "job": 14, "event": "table_file_deletion", "file_number": 7246} >2018-07-11 07:59:01.465280 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295941465278, "job": 14, "event": "table_file_deletion", "file_number": 7243} >2018-07-11 07:59:01.465293 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295941465291, "job": 14, "event": "table_file_deletion", "file_number": 7234} >2018-07-11 07:59:01.465306 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295941465304, "job": 14, "event": "table_file_deletion", "file_number": 7212} >2018-07-11 07:59:01.465320 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295941465317, "job": 14, "event": "table_file_deletion", "file_number": 7208} >2018-07-11 07:59:01.465332 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295941465330, "job": 14, "event": "table_file_deletion", "file_number": 7201} >2018-07-11 07:59:01.465530 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 15] Compacting 1@2 + 4@3 files to L3, score 1.56 >2018-07-11 07:59:01.465542 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 14 Base level 2, inputs: [36550(75MB)], [21178(71MB) 21179(70MB) 21439(71MB) 36767(71MB)] > >2018-07-11 07:59:01.465556 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295941465546, "job": 15, "event": "compaction_started", "files_L2": [36550], "files_L3": [21178, 21179, 21439, 36767], "score": 1.55806, "input_data_size": 377708232} >2018-07-11 07:59:04.783569 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 15] Generated table #36917: 2798441 keys, 74944330 bytes >2018-07-11 07:59:04.783622 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295944783606, "cf_name": "default", "job": 15, "event": "table_file_creation", "file_number": 36917, "file_size": 74944330, "table_properties": {"data_size": 67109423, "index_size": 689668, "filter_size": 7144247, "raw_key_size": 80843853, "raw_average_key_size": 28, "raw_value_size": 29196323, "raw_average_value_size": 10, "num_data_blocks": 16488, "num_entries": 2798441, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "342675", "kMergeOperands": "0"}} >2018-07-11 07:59:08.323740 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 15] Generated table #36918: 2870388 keys, 75127334 bytes >2018-07-11 07:59:08.323787 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295948323771, "cf_name": "default", "job": 15, "event": "table_file_creation", "file_number": 36918, "file_size": 75127334, "table_properties": {"data_size": 67111874, "index_size": 690095, "filter_size": 7324373, "raw_key_size": 82922346, "raw_average_key_size": 28, "raw_value_size": 28225676, "raw_average_value_size": 9, "num_data_blocks": 16488, "num_entries": 2870388, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "496267", "kMergeOperands": "0"}} >2018-07-11 07:59:10.650426 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 15] Generated table #36919: 2159342 keys, 61014324 bytes >2018-07-11 07:59:10.650465 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295950650449, "cf_name": "default", "job": 15, "event": "table_file_creation", "file_number": 36919, "file_size": 61014324, "table_properties": {"data_size": 54929747, "index_size": 564448, "filter_size": 5519140, "raw_key_size": 62380993, "raw_average_key_size": 28, "raw_value_size": 25672179, "raw_average_value_size": 11, "num_data_blocks": 13496, "num_entries": 2159342, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:59:14.473043 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 15] Generated table #36920: 3573835 keys, 76887282 bytes >2018-07-11 07:59:14.473095 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295954473078, "cf_name": "default", "job": 15, "event": "table_file_creation", "file_number": 36920, "file_size": 76887282, "table_properties": {"data_size": 67111266, "index_size": 689542, "filter_size": 9085482, "raw_key_size": 103244125, "raw_average_key_size": 28, "raw_value_size": 18711150, "raw_average_value_size": 5, "num_data_blocks": 16482, "num_entries": 3573835, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "2000000", "kMergeOperands": "0"}} >2018-07-11 07:59:17.322507 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 15] Generated table #36921: 2809475 keys, 67337283 bytes >2018-07-11 07:59:17.322544 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295957322529, "cf_name": "default", "job": 15, "event": "table_file_creation", "file_number": 36921, "file_size": 67337283, "table_properties": {"data_size": 59567711, "index_size": 612255, "filter_size": 7156326, "raw_key_size": 81162608, "raw_average_key_size": 28, "raw_value_size": 21512643, "raw_average_value_size": 7, "num_data_blocks": 14632, "num_entries": 2809475, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1000000", "kMergeOperands": "0"}} >2018-07-11 07:59:17.322578 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 15] Compacted 1@2 + 4@3 files to L3 => 355310553 bytes >2018-07-11 07:59:17.338837 7f4922851700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 40124771313 >2018-07-11 07:59:17.339033 7f4922851700 4 rocksdb: (Original Log Time 2018/07/11-07:59:17.338804) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[1 7 59 481 3641 796 0] max score 1.51, MB/sec: 23.8 rd, 22.4 wr, level 3, files in(1, 4) out(5) MB in(75.8, 284.4) out(338.9), read-write-amplify(9.2) write-amplify(4.5) OK, records in: 15218947, records dropped: 1007466 > >2018-07-11 07:59:17.339042 7f4922851700 4 rocksdb: (Original Log Time 2018/07/11-07:59:17.338829) EVENT_LOG_v1 {"time_micros": 1531295957338818, "job": 15, "event": "compaction_finished", "compaction_time_micros": 15857006, "output_level": 3, "num_output_files": 5, "total_output_size": 355310553, "num_input_records": 15218947, "num_output_records": 14211481, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 7, 59, 481, 3641, 796, 0]} >2018-07-11 07:59:17.339677 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295957339661, "job": 15, "event": "table_file_deletion", "file_number": 36767} >2018-07-11 07:59:17.339692 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295957339690, "job": 15, "event": "table_file_deletion", "file_number": 36550} >2018-07-11 07:59:17.339705 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295957339703, "job": 15, "event": "table_file_deletion", "file_number": 21439} >2018-07-11 07:59:17.339718 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295957339716, "job": 15, "event": "table_file_deletion", "file_number": 21179} >2018-07-11 07:59:17.339730 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295957339728, "job": 15, "event": "table_file_deletion", "file_number": 21178} >2018-07-11 07:59:17.339912 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 16] Compacting 1@2 + 3@3 files to L3, score 1.51 >2018-07-11 07:59:17.339924 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 15 Base level 2, inputs: [36641(67MB)], [36732(72MB) 36733(3438KB) 31428(71MB)] > >2018-07-11 07:59:17.339937 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295957339927, "job": 16, "event": "compaction_started", "files_L2": [36641], "files_L3": [36732, 36733, 31428], "score": 1.50526, "input_data_size": 224812483} >2018-07-11 07:59:22.244951 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 16] Generated table #36922: 3650080 keys, 77076702 bytes >2018-07-11 07:59:22.244992 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295962244976, "cf_name": "default", "job": 16, "event": "table_file_creation", "file_number": 36922, "file_size": 77076702, "table_properties": {"data_size": 67109845, "index_size": 689504, "filter_size": 9276360, "raw_key_size": 105446729, "raw_average_key_size": 28, "raw_value_size": 17678486, "raw_average_value_size": 4, "num_data_blocks": 16481, "num_entries": 3650080, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "2163104", "kMergeOperands": "0"}} >2018-07-11 07:59:25.832352 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 16] Generated table #36923: 3546088 keys, 76817137 bytes >2018-07-11 07:59:25.832405 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295965832388, "cf_name": "default", "job": 16, "event": "table_file_creation", "file_number": 36923, "file_size": 76817137, "table_properties": {"data_size": 67110379, "index_size": 689754, "filter_size": 9016012, "raw_key_size": 102442568, "raw_average_key_size": 28, "raw_value_size": 19085529, "raw_average_value_size": 5, "num_data_blocks": 16482, "num_entries": 3546088, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1940764", "kMergeOperands": "0"}} >2018-07-11 07:59:27.628465 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 16] Generated table #36924: 1719455 keys, 40314118 bytes >2018-07-11 07:59:27.628498 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295967628483, "cf_name": "default", "job": 16, "event": "table_file_creation", "file_number": 36924, "file_size": 40314118, "table_properties": {"data_size": 35569870, "index_size": 365266, "filter_size": 4377992, "raw_key_size": 49673135, "raw_average_key_size": 28, "raw_value_size": 12280221, "raw_average_value_size": 7, "num_data_blocks": 8737, "num_entries": 1719455, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "686539", "kMergeOperands": "0"}} >2018-07-11 07:59:27.628531 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 16] Compacted 1@2 + 3@3 files to L3 => 194207957 bytes >2018-07-11 07:59:27.646968 7f4922851700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 39302088055 >2018-07-11 07:59:27.647180 7f4922851700 4 rocksdb: (Original Log Time 2018/07/11-07:59:27.646935) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[1 7 58 481 3641 796 0] max score 1.45, MB/sec: 21.9 rd, 18.9 wr, level 3, files in(1, 3) out(3) MB in(67.6, 146.8) out(185.2), read-write-amplify(5.9) write-amplify(2.7) OK, records in: 10292205, records dropped: 1376582 > >2018-07-11 07:59:27.647190 7f4922851700 4 rocksdb: (Original Log Time 2018/07/11-07:59:27.646959) EVENT_LOG_v1 {"time_micros": 1531295967646948, "job": 16, "event": "compaction_finished", "compaction_time_micros": 10288580, "output_level": 3, "num_output_files": 3, "total_output_size": 194207957, "num_input_records": 10292205, "num_output_records": 8915623, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 7, 58, 481, 3641, 796, 0]} >2018-07-11 07:59:27.647834 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295967647828, "job": 16, "event": "table_file_deletion", "file_number": 36733} >2018-07-11 07:59:27.647868 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295967647864, "job": 16, "event": "table_file_deletion", "file_number": 36732} >2018-07-11 07:59:27.647895 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295967647891, "job": 16, "event": "table_file_deletion", "file_number": 36641} >2018-07-11 07:59:27.647923 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295967647919, "job": 16, "event": "table_file_deletion", "file_number": 31428} >2018-07-11 07:59:27.648112 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 17] Compacting 1@2 + 21@3 files to L3, score 1.45 >2018-07-11 07:59:27.648136 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 16 Base level 2, inputs: [36749(58MB)], [7525(71MB) 7530(71MB) 7535(71MB) 7537(12MB) 7538(71MB) 7539(71MB) 7540(71MB) 7541(71MB) 7545(71MB) 7548(68MB) 755(71MB) 770(71MB) 807(71MB) 824(71MB) 858(71MB) 888(71MB) 907(71MB) 939(71MB) 956(71MB) 976(71MB) 990(71MB)] > >2018-07-11 07:59:27.648162 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295967648145, "job": 17, "event": "compaction_started", "files_L2": [36749], "files_L3": [7525, 7530, 7535, 7537, 7538, 7539, 7540, 7541, 7545, 7548, 755, 770, 807, 824, 858, 888, 907, 939, 956, 976, 990], "score": 1.45277, "input_data_size": 1562568821} >2018-07-11 07:59:31.414406 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 17] Generated table #36925: 2638238 keys, 74546399 bytes >2018-07-11 07:59:31.414449 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295971414427, "cf_name": "default", "job": 17, "event": "table_file_creation", "file_number": 36925, "file_size": 74546399, "table_properties": {"data_size": 67112133, "index_size": 690099, "filter_size": 6743177, "raw_key_size": 76215750, "raw_average_key_size": 28, "raw_value_size": 31365704, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638238, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:59:34.397470 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 17] Generated table #36926: 2638241 keys, 74546080 bytes >2018-07-11 07:59:34.397505 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295974397489, "cf_name": "default", "job": 17, "event": "table_file_creation", "file_number": 36926, "file_size": 74546080, "table_properties": {"data_size": 67112157, "index_size": 689749, "filter_size": 6743184, "raw_key_size": 76215855, "raw_average_key_size": 28, "raw_value_size": 31365758, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:59:38.135252 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 17] Generated table #36927: 2638240 keys, 74545999 bytes >2018-07-11 07:59:38.135295 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295978135275, "cf_name": "default", "job": 17, "event": "table_file_creation", "file_number": 36927, "file_size": 74545999, "table_properties": {"data_size": 67112136, "index_size": 689692, "filter_size": 6743181, "raw_key_size": 76215820, "raw_average_key_size": 28, "raw_value_size": 31365740, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:59:40.999834 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 17] Generated table #36928: 2638240 keys, 74546493 bytes >2018-07-11 07:59:40.999880 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295980999865, "cf_name": "default", "job": 17, "event": "table_file_creation", "file_number": 36928, "file_size": 74546493, "table_properties": {"data_size": 67112130, "index_size": 690192, "filter_size": 6743181, "raw_key_size": 76215824, "raw_average_key_size": 28, "raw_value_size": 31365744, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:59:42.641439 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 17] Generated table #36929: 1441805 keys, 40739901 bytes >2018-07-11 07:59:42.641482 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295982641455, "cf_name": "default", "job": 17, "event": "table_file_creation", "file_number": 36929, "file_size": 40739901, "table_properties": {"data_size": 36676911, "index_size": 376836, "filter_size": 3685166, "raw_key_size": 41652149, "raw_average_key_size": 28, "raw_value_size": 17141464, "raw_average_value_size": 11, "num_data_blocks": 9012, "num_entries": 1441805, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:59:45.456088 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 17] Generated table #36930: 2638241 keys, 74545834 bytes >2018-07-11 07:59:45.456134 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295985456106, "cf_name": "default", "job": 17, "event": "table_file_creation", "file_number": 36930, "file_size": 74545834, "table_properties": {"data_size": 67112096, "index_size": 689565, "filter_size": 6743183, "raw_key_size": 76215857, "raw_average_key_size": 28, "raw_value_size": 31365760, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:59:48.271941 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 17] Generated table #36931: 2638241 keys, 74545900 bytes >2018-07-11 07:59:48.271987 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295988271959, "cf_name": "default", "job": 17, "event": "table_file_creation", "file_number": 36931, "file_size": 74545900, "table_properties": {"data_size": 67112126, "index_size": 689601, "filter_size": 6743183, "raw_key_size": 76215860, "raw_average_key_size": 28, "raw_value_size": 31365763, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:59:51.221819 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 17] Generated table #36932: 2638241 keys, 74545832 bytes >2018-07-11 07:59:51.221865 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295991221850, "cf_name": "default", "job": 17, "event": "table_file_creation", "file_number": 36932, "file_size": 74545832, "table_properties": {"data_size": 67112124, "index_size": 689535, "filter_size": 6743183, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:59:54.545948 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 17] Generated table #36933: 2638241 keys, 74545734 bytes >2018-07-11 07:59:54.545997 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295994545981, "cf_name": "default", "job": 17, "event": "table_file_creation", "file_number": 36933, "file_size": 74545734, "table_properties": {"data_size": 67112115, "index_size": 689446, "filter_size": 6743183, "raw_key_size": 76215846, "raw_average_key_size": 28, "raw_value_size": 31365749, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 07:59:57.725678 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 17] Generated table #36934: 3106038 keys, 75717144 bytes >2018-07-11 07:59:57.725716 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531295997725700, "cf_name": "default", "job": 17, "event": "table_file_creation", "file_number": 36934, "file_size": 75717144, "table_properties": {"data_size": 67111707, "index_size": 690112, "filter_size": 7914333, "raw_key_size": 89729985, "raw_average_key_size": 28, "raw_value_size": 25038449, "raw_average_value_size": 8, "num_data_blocks": 16486, "num_entries": 3106038, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1000000", "kMergeOperands": "0"}} >2018-07-11 08:00:00.852725 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 17] Generated table #36935: 2638240 keys, 74546036 bytes >2018-07-11 08:00:00.852772 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296000852744, "cf_name": "default", "job": 17, "event": "table_file_creation", "file_number": 36935, "file_size": 74546036, "table_properties": {"data_size": 67112143, "index_size": 689722, "filter_size": 6743181, "raw_key_size": 76215828, "raw_average_key_size": 28, "raw_value_size": 31365748, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:00:03.830439 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 17] Generated table #36936: 2638240 keys, 74546098 bytes >2018-07-11 08:00:03.830491 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296003830474, "cf_name": "default", "job": 17, "event": "table_file_creation", "file_number": 36936, "file_size": 74546098, "table_properties": {"data_size": 67112078, "index_size": 689849, "filter_size": 6743181, "raw_key_size": 76215821, "raw_average_key_size": 28, "raw_value_size": 31365741, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:00:04.600593 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 17] Generated table #36937: 532205 keys, 15038335 bytes >2018-07-11 08:00:04.600638 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296004600610, "cf_name": "default", "job": 17, "event": "table_file_creation", "file_number": 36937, "file_size": 15038335, "table_properties": {"data_size": 13538350, "index_size": 138714, "filter_size": 1360285, "raw_key_size": 15374810, "raw_average_key_size": 28, "raw_value_size": 6327325, "raw_average_value_size": 11, "num_data_blocks": 3327, "num_entries": 532205, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:00:07.654917 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 17] Generated table #36938: 2638241 keys, 74546006 bytes >2018-07-11 08:00:07.654950 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296007654935, "cf_name": "default", "job": 17, "event": "table_file_creation", "file_number": 36938, "file_size": 74546006, "table_properties": {"data_size": 67112169, "index_size": 689664, "filter_size": 6743183, "raw_key_size": 76215859, "raw_average_key_size": 28, "raw_value_size": 31365762, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:00:10.738308 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 17] Generated table #36939: 2638241 keys, 74545941 bytes >2018-07-11 08:00:10.738356 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296010738340, "cf_name": "default", "job": 17, "event": "table_file_creation", "file_number": 36939, "file_size": 74545941, "table_properties": {"data_size": 67112157, "index_size": 689611, "filter_size": 6743183, "raw_key_size": 76215847, "raw_average_key_size": 28, "raw_value_size": 31365750, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:00:13.781931 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 17] Generated table #36940: 2638241 keys, 74545935 bytes >2018-07-11 08:00:13.781986 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296013781970, "cf_name": "default", "job": 17, "event": "table_file_creation", "file_number": 36940, "file_size": 74545935, "table_properties": {"data_size": 67112148, "index_size": 689614, "filter_size": 6743183, "raw_key_size": 76215848, "raw_average_key_size": 28, "raw_value_size": 31365751, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:00:16.821007 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 17] Generated table #36941: 2638241 keys, 74545888 bytes >2018-07-11 08:00:16.821050 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296016821030, "cf_name": "default", "job": 17, "event": "table_file_creation", "file_number": 36941, "file_size": 74545888, "table_properties": {"data_size": 67112168, "index_size": 689547, "filter_size": 6743183, "raw_key_size": 76215859, "raw_average_key_size": 28, "raw_value_size": 31365762, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:00:19.910571 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 17] Generated table #36942: 2638241 keys, 74545843 bytes >2018-07-11 08:00:19.910621 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296019910605, "cf_name": "default", "job": 17, "event": "table_file_creation", "file_number": 36942, "file_size": 74545843, "table_properties": {"data_size": 67112146, "index_size": 689524, "filter_size": 6743183, "raw_key_size": 76215856, "raw_average_key_size": 28, "raw_value_size": 31365759, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:00:23.084088 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 17] Generated table #36943: 2638241 keys, 74545936 bytes >2018-07-11 08:00:23.084140 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296023084121, "cf_name": "default", "job": 17, "event": "table_file_creation", "file_number": 36943, "file_size": 74545936, "table_properties": {"data_size": 67112101, "index_size": 689662, "filter_size": 6743183, "raw_key_size": 76215851, "raw_average_key_size": 28, "raw_value_size": 31365754, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:00:23.096993 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 17] Generated table #36944: 1 keys, 1064 bytes >2018-07-11 08:00:23.097019 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296023097006, "cf_name": "default", "job": 17, "event": "table_file_creation", "file_number": 36944, "file_size": 1064, "table_properties": {"data_size": 57, "index_size": 27, "filter_size": 18, "raw_key_size": 29, "raw_average_key_size": 29, "raw_value_size": 12, "raw_average_value_size": 12, "num_data_blocks": 1, "num_entries": 1, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:00:26.259952 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 17] Generated table #36945: 2638242 keys, 74546697 bytes >2018-07-11 08:00:26.259998 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296026259971, "cf_name": "default", "job": 17, "event": "table_file_creation", "file_number": 36945, "file_size": 74546697, "table_properties": {"data_size": 67112183, "index_size": 690338, "filter_size": 6743186, "raw_key_size": 76215892, "raw_average_key_size": 28, "raw_value_size": 31365778, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638242, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:00:28.672593 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 17] Generated table #36946: 1638239 keys, 46290705 bytes >2018-07-11 08:00:28.672640 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296028672625, "cf_name": "default", "job": 17, "event": "table_file_creation", "file_number": 36946, "file_size": 46290705, "table_properties": {"data_size": 41673842, "index_size": 428641, "filter_size": 4187234, "raw_key_size": 47326904, "raw_average_key_size": 28, "raw_value_size": 19476841, "raw_average_value_size": 11, "num_data_blocks": 10239, "num_entries": 1638239, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:00:28.672685 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 17] Compacted 1@2 + 21@3 files to L3 => 1445069800 bytes >2018-07-11 08:00:28.687558 7f4922851700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 37604509879 >2018-07-11 08:00:28.687800 7f4922851700 4 rocksdb: (Original Log Time 2018/07/11-08:00:28.687524) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[1 7 57 482 3641 796 0] max score 1.41, MB/sec: 25.6 rd, 23.7 wr, level 3, files in(1, 21) out(22) MB in(58.5, 1431.7) out(1378.1), read-write-amplify(49.1) write-amplify(23.6) OK, records in: 56853697, records dropped: 5285318 > >2018-07-11 08:00:28.687810 7f4922851700 4 rocksdb: (Original Log Time 2018/07/11-08:00:28.687549) EVENT_LOG_v1 {"time_micros": 1531296028687537, "job": 17, "event": "compaction_finished", "compaction_time_micros": 61024493, "output_level": 3, "num_output_files": 22, "total_output_size": 1445069800, "num_input_records": 56853697, "num_output_records": 51568379, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 7, 57, 482, 3641, 796, 0]} >2018-07-11 08:00:28.688449 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296028688445, "job": 17, "event": "table_file_deletion", "file_number": 36749} >2018-07-11 08:00:28.688467 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296028688465, "job": 17, "event": "table_file_deletion", "file_number": 7548} >2018-07-11 08:00:28.688482 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296028688480, "job": 17, "event": "table_file_deletion", "file_number": 7545} >2018-07-11 08:00:28.688496 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296028688494, "job": 17, "event": "table_file_deletion", "file_number": 7541} >2018-07-11 08:00:28.688509 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296028688507, "job": 17, "event": "table_file_deletion", "file_number": 7540} >2018-07-11 08:00:28.688522 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296028688520, "job": 17, "event": "table_file_deletion", "file_number": 7539} >2018-07-11 08:00:28.688535 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296028688533, "job": 17, "event": "table_file_deletion", "file_number": 7538} >2018-07-11 08:00:28.688548 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296028688546, "job": 17, "event": "table_file_deletion", "file_number": 7537} >2018-07-11 08:00:28.688561 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296028688559, "job": 17, "event": "table_file_deletion", "file_number": 7535} >2018-07-11 08:00:28.688575 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296028688573, "job": 17, "event": "table_file_deletion", "file_number": 7530} >2018-07-11 08:00:28.688590 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296028688586, "job": 17, "event": "table_file_deletion", "file_number": 7525} >2018-07-11 08:00:28.688606 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296028688604, "job": 17, "event": "table_file_deletion", "file_number": 990} >2018-07-11 08:00:28.688619 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296028688617, "job": 17, "event": "table_file_deletion", "file_number": 976} >2018-07-11 08:00:28.688634 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296028688632, "job": 17, "event": "table_file_deletion", "file_number": 956} >2018-07-11 08:00:28.688649 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296028688645, "job": 17, "event": "table_file_deletion", "file_number": 939} >2018-07-11 08:00:28.688663 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296028688661, "job": 17, "event": "table_file_deletion", "file_number": 907} >2018-07-11 08:00:28.688677 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296028688674, "job": 17, "event": "table_file_deletion", "file_number": 888} >2018-07-11 08:00:28.688691 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296028688689, "job": 17, "event": "table_file_deletion", "file_number": 858} >2018-07-11 08:00:28.688704 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296028688702, "job": 17, "event": "table_file_deletion", "file_number": 824} >2018-07-11 08:00:28.688717 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296028688715, "job": 17, "event": "table_file_deletion", "file_number": 807} >2018-07-11 08:00:28.688733 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296028688731, "job": 17, "event": "table_file_deletion", "file_number": 770} >2018-07-11 08:00:28.688746 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296028688744, "job": 17, "event": "table_file_deletion", "file_number": 755} >2018-07-11 08:00:28.688928 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 18] Compacting 1@2 + 6@3 files to L3, score 1.41 >2018-07-11 08:00:28.688941 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 17 Base level 2, inputs: [36614(48MB)], [16614(71MB) 16617(71MB) 16618(31MB) 16619(824KB) 18774(71MB) 18780(1088B)] > >2018-07-11 08:00:28.688956 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296028688946, "job": 18, "event": "compaction_started", "files_L2": [36614], "files_L3": [16614, 16617, 16618, 16619, 18774, 18780], "score": 1.4087, "input_data_size": 307774193} >2018-07-11 08:00:32.043180 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 18] Generated table #36947: 2698275 keys, 74694568 bytes >2018-07-11 08:00:32.043214 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296032043199, "cf_name": "default", "job": 18, "event": "table_file_creation", "file_number": 36947, "file_size": 74694568, "table_properties": {"data_size": 67110171, "index_size": 689928, "filter_size": 6893477, "raw_key_size": 77950177, "raw_average_key_size": 28, "raw_value_size": 30551749, "raw_average_value_size": 11, "num_data_blocks": 16489, "num_entries": 2698275, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "128502", "kMergeOperands": "0"}} >2018-07-11 08:00:32.304485 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 18] Generated table #36948: 68467 keys, 1935158 bytes >2018-07-11 08:00:32.304520 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296032304506, "cf_name": "default", "job": 18, "event": "table_file_creation", "file_number": 36948, "file_size": 1935158, "table_properties": {"data_size": 1741693, "index_size": 17483, "filter_size": 175001, "raw_key_size": 1977937, "raw_average_key_size": 28, "raw_value_size": 813998, "raw_average_value_size": 11, "num_data_blocks": 428, "num_entries": 68467, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:00:35.454232 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 18] Generated table #36949: 2638240 keys, 74545861 bytes >2018-07-11 08:00:35.454283 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296035454268, "cf_name": "default", "job": 18, "event": "table_file_creation", "file_number": 36949, "file_size": 74545861, "table_properties": {"data_size": 67112157, "index_size": 689533, "filter_size": 6743181, "raw_key_size": 76215837, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:00:38.672904 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 18] Generated table #36950: 2638239 keys, 74546489 bytes >2018-07-11 08:00:38.672954 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296038672938, "cf_name": "default", "job": 18, "event": "table_file_creation", "file_number": 36950, "file_size": 74546489, "table_properties": {"data_size": 67112138, "index_size": 690182, "filter_size": 6743179, "raw_key_size": 76215796, "raw_average_key_size": 28, "raw_value_size": 31365733, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638239, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:00:40.455460 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 18] Generated table #36951: 1300872 keys, 30023757 bytes >2018-07-11 08:00:40.455504 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296040455489, "cf_name": "default", "job": 18, "event": "table_file_creation", "file_number": 36951, "file_size": 30023757, "table_properties": {"data_size": 26440078, "index_size": 271441, "filter_size": 3311248, "raw_key_size": 37580755, "raw_average_key_size": 28, "raw_value_size": 8820511, "raw_average_value_size": 6, "num_data_blocks": 6494, "num_entries": 1300872, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "558960", "kMergeOperands": "0"}} >2018-07-11 08:00:42.429619 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 18] Generated table #36952: 2000002 keys, 32417367 bytes >2018-07-11 08:00:42.429660 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296042429640, "cf_name": "default", "job": 18, "event": "table_file_creation", "file_number": 36952, "file_size": 32417367, "table_properties": {"data_size": 27075767, "index_size": 277756, "filter_size": 5062857, "raw_key_size": 57777838, "raw_average_key_size": 28, "raw_value_size": 24, "raw_average_value_size": 0, "num_data_blocks": 6645, "num_entries": 2000002, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "2000000", "kMergeOperands": "0"}} >2018-07-11 08:00:42.429699 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 18] Compacted 1@2 + 6@3 files to L3 => 288163200 bytes >2018-07-11 08:00:42.445675 7f4922851700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 37015030220 >2018-07-11 08:00:42.445822 7f4922851700 4 rocksdb: (Original Log Time 2018/07/11-08:00:42.445642) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[1 7 56 482 3641 796 0] max score 1.37, MB/sec: 22.4 rd, 21.0 wr, level 3, files in(1, 6) out(6) MB in(48.4, 245.2) out(274.8), read-write-amplify(11.8) write-amplify(5.7) OK, records in: 12226175, records dropped: 882080 > >2018-07-11 08:00:42.445832 7f4922851700 4 rocksdb: (Original Log Time 2018/07/11-08:00:42.445667) EVENT_LOG_v1 {"time_micros": 1531296042445656, "job": 18, "event": "compaction_finished", "compaction_time_micros": 13740724, "output_level": 3, "num_output_files": 6, "total_output_size": 288163200, "num_input_records": 12226175, "num_output_records": 11344095, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 7, 56, 482, 3641, 796, 0]} >2018-07-11 08:00:42.446483 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296042446477, "job": 18, "event": "table_file_deletion", "file_number": 36614} >2018-07-11 08:00:42.446512 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296042446507, "job": 18, "event": "table_file_deletion", "file_number": 18780} >2018-07-11 08:00:42.446554 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296042446550, "job": 18, "event": "table_file_deletion", "file_number": 18774} >2018-07-11 08:00:42.446582 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296042446578, "job": 18, "event": "table_file_deletion", "file_number": 16619} >2018-07-11 08:00:42.446604 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296042446600, "job": 18, "event": "table_file_deletion", "file_number": 16618} >2018-07-11 08:00:42.446626 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296042446622, "job": 18, "event": "table_file_deletion", "file_number": 16617} >2018-07-11 08:00:42.446650 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296042446646, "job": 18, "event": "table_file_deletion", "file_number": 16614} >2018-07-11 08:00:42.446836 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 19] Compacting 1@2 + 17@3 files to L3, score 1.37 >2018-07-11 08:00:42.446858 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 18 Base level 2, inputs: [36608(47MB)], [36664(50MB) 9728(71MB) 9732(71MB) 9736(1113B) 9993(1087B) 10763(70MB) 10764(71MB) 10768(71MB) 10769(71MB) 10770(71MB) 10771(71MB) 10772(1060B) 10773(329KB) 14851(1088B) 12903(1064B) 12978(70MB) 12979(71MB)] > >2018-07-11 08:00:42.446883 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296042446867, "job": 19, "event": "compaction_started", "files_L2": [36608], "files_L3": [36664, 9728, 9732, 9736, 9993, 10763, 10764, 10768, 10769, 10770, 10771, 10772, 10773, 14851, 12903, 12978, 12979], "score": 1.37116, "input_data_size": 847182995} >2018-07-11 08:00:45.871102 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 19] Generated table #36953: 2649065 keys, 74574084 bytes >2018-07-11 08:00:45.871141 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296045871122, "cf_name": "default", "job": 19, "event": "table_file_creation", "file_number": 36953, "file_size": 74574084, "table_properties": {"data_size": 67112511, "index_size": 690298, "filter_size": 6770283, "raw_key_size": 76528559, "raw_average_key_size": 28, "raw_value_size": 31219626, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2649065, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "23116", "kMergeOperands": "0"}} >2018-07-11 08:00:48.956772 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 19] Generated table #36954: 2638240 keys, 74546284 bytes >2018-07-11 08:00:48.956818 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296048956791, "cf_name": "default", "job": 19, "event": "table_file_creation", "file_number": 36954, "file_size": 74546284, "table_properties": {"data_size": 67112164, "index_size": 689949, "filter_size": 6743181, "raw_key_size": 76215831, "raw_average_key_size": 28, "raw_value_size": 31365751, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:00:52.106036 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 19] Generated table #36955: 2858790 keys, 68730500 bytes >2018-07-11 08:00:52.106084 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296052106068, "cf_name": "default", "job": 19, "event": "table_file_creation", "file_number": 36955, "file_size": 68730500, "table_properties": {"data_size": 60822172, "index_size": 624963, "filter_size": 7282374, "raw_key_size": 82587268, "raw_average_key_size": 28, "raw_value_size": 22098948, "raw_average_value_size": 7, "num_data_blocks": 14940, "num_entries": 2858790, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1000000", "kMergeOperands": "0"}} >2018-07-11 08:00:55.322593 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 19] Generated table #36956: 2626329 keys, 74209490 bytes >2018-07-11 08:00:55.322642 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296055322626, "cf_name": "default", "job": 19, "event": "table_file_creation", "file_number": 36956, "file_size": 74209490, "table_properties": {"data_size": 66809129, "index_size": 686643, "filter_size": 6712728, "raw_key_size": 75871733, "raw_average_key_size": 28, "raw_value_size": 31224140, "raw_average_value_size": 11, "num_data_blocks": 16415, "num_entries": 2626329, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:00:58.529023 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 19] Generated table #36957: 2638240 keys, 74545929 bytes >2018-07-11 08:00:58.529073 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296058529057, "cf_name": "default", "job": 19, "event": "table_file_creation", "file_number": 36957, "file_size": 74545929, "table_properties": {"data_size": 67112152, "index_size": 689606, "filter_size": 6743181, "raw_key_size": 76215820, "raw_average_key_size": 28, "raw_value_size": 31365740, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:01:01.727284 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 19] Generated table #36958: 2638241 keys, 74545965 bytes >2018-07-11 08:01:01.727339 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296061727322, "cf_name": "default", "job": 19, "event": "table_file_creation", "file_number": 36958, "file_size": 74545965, "table_properties": {"data_size": 67112136, "index_size": 689656, "filter_size": 6743183, "raw_key_size": 76215844, "raw_average_key_size": 28, "raw_value_size": 31365747, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:01:04.986297 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 19] Generated table #36959: 2638241 keys, 74545983 bytes >2018-07-11 08:01:04.986332 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296064986317, "cf_name": "default", "job": 19, "event": "table_file_creation", "file_number": 36959, "file_size": 74545983, "table_properties": {"data_size": 67112100, "index_size": 689710, "filter_size": 6743183, "raw_key_size": 76215850, "raw_average_key_size": 28, "raw_value_size": 31365753, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:01:08.267312 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 19] Generated table #36960: 2638241 keys, 74545948 bytes >2018-07-11 08:01:08.267355 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296068267329, "cf_name": "default", "job": 19, "event": "table_file_creation", "file_number": 36960, "file_size": 74545948, "table_properties": {"data_size": 67112132, "index_size": 689643, "filter_size": 6743183, "raw_key_size": 76215843, "raw_average_key_size": 28, "raw_value_size": 31365746, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:01:11.450543 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 19] Generated table #36961: 2638241 keys, 74545952 bytes >2018-07-11 08:01:11.450594 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296071450577, "cf_name": "default", "job": 19, "event": "table_file_creation", "file_number": 36961, "file_size": 74545952, "table_properties": {"data_size": 67112103, "index_size": 689676, "filter_size": 6743183, "raw_key_size": 76215849, "raw_average_key_size": 28, "raw_value_size": 31365752, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:01:12.661143 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 19] Generated table #36962: 1011916 keys, 16545621 bytes >2018-07-11 08:01:12.661190 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296072661174, "cf_name": "default", "job": 19, "event": "table_file_creation", "file_number": 36962, "file_size": 16545621, "table_properties": {"data_size": 13841004, "index_size": 141742, "filter_size": 2561886, "raw_key_size": 29233128, "raw_average_key_size": 28, "raw_value_size": 141666, "raw_average_value_size": 0, "num_data_blocks": 3397, "num_entries": 1011916, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1000000", "kMergeOperands": "0"}} >2018-07-11 08:01:16.026093 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 19] Generated table #36963: 2638240 keys, 74545752 bytes >2018-07-11 08:01:16.026142 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296076026125, "cf_name": "default", "job": 19, "event": "table_file_creation", "file_number": 36963, "file_size": 74545752, "table_properties": {"data_size": 67112129, "index_size": 689452, "filter_size": 6743181, "raw_key_size": 76215833, "raw_average_key_size": 28, "raw_value_size": 31365753, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:01:18.826117 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 19] Generated table #36964: 2626330 keys, 74209757 bytes >2018-07-11 08:01:18.826166 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296078826150, "cf_name": "default", "job": 19, "event": "table_file_creation", "file_number": 36964, "file_size": 74209757, "table_properties": {"data_size": 66809112, "index_size": 686925, "filter_size": 6712730, "raw_key_size": 75871755, "raw_average_key_size": 28, "raw_value_size": 31224145, "raw_average_value_size": 11, "num_data_blocks": 16415, "num_entries": 2626330, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:01:19.824026 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 19] Generated table #36965: 1000000 keys, 16208895 bytes >2018-07-11 08:01:19.824059 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296079824044, "cf_name": "default", "job": 19, "event": "table_file_creation", "file_number": 36965, "file_size": 16208895, "table_properties": {"data_size": 13537837, "index_size": 138642, "filter_size": 2531429, "raw_key_size": 28888890, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 3323, "num_entries": 1000000, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1000000", "kMergeOperands": "0"}} >2018-07-11 08:01:19.824094 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 19] Compacted 1@2 + 17@3 files to L3 => 846300160 bytes >2018-07-11 08:01:19.844957 7f4922851700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 36624991626 >2018-07-11 08:01:19.845157 7f4922851700 4 rocksdb: (Original Log Time 2018/07/11-08:01:19.844924) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[1 7 55 478 3641 796 0] max score 1.33, MB/sec: 22.7 rd, 22.6 wr, level 3, files in(1, 17) out(13) MB in(47.0, 760.9) out(807.1), read-write-amplify(34.3) write-amplify(17.2) OK, records in: 31279546, records dropped: 39432 > >2018-07-11 08:01:19.845166 7f4922851700 4 rocksdb: (Original Log Time 2018/07/11-08:01:19.844949) EVENT_LOG_v1 {"time_micros": 1531296079844937, "job": 19, "event": "compaction_finished", "compaction_time_micros": 37377188, "output_level": 3, "num_output_files": 13, "total_output_size": 846300160, "num_input_records": 31279546, "num_output_records": 31240114, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 7, 55, 478, 3641, 796, 0]} >2018-07-11 08:01:19.845799 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296079845782, "job": 19, "event": "table_file_deletion", "file_number": 36664} >2018-07-11 08:01:19.845815 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296079845813, "job": 19, "event": "table_file_deletion", "file_number": 36608} >2018-07-11 08:01:19.845829 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296079845827, "job": 19, "event": "table_file_deletion", "file_number": 14851} >2018-07-11 08:01:19.845845 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296079845843, "job": 19, "event": "table_file_deletion", "file_number": 12979} >2018-07-11 08:01:19.845861 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296079845858, "job": 19, "event": "table_file_deletion", "file_number": 12978} >2018-07-11 08:01:19.845877 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296079845875, "job": 19, "event": "table_file_deletion", "file_number": 12903} >2018-07-11 08:01:19.845892 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296079845890, "job": 19, "event": "table_file_deletion", "file_number": 10773} >2018-07-11 08:01:19.845905 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296079845903, "job": 19, "event": "table_file_deletion", "file_number": 10772} >2018-07-11 08:01:19.845919 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296079845916, "job": 19, "event": "table_file_deletion", "file_number": 10771} >2018-07-11 08:01:19.845932 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296079845930, "job": 19, "event": "table_file_deletion", "file_number": 10770} >2018-07-11 08:01:19.845944 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296079845942, "job": 19, "event": "table_file_deletion", "file_number": 10769} >2018-07-11 08:01:19.845956 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296079845954, "job": 19, "event": "table_file_deletion", "file_number": 10768} >2018-07-11 08:01:19.845969 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296079845967, "job": 19, "event": "table_file_deletion", "file_number": 10764} >2018-07-11 08:01:19.845982 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296079845980, "job": 19, "event": "table_file_deletion", "file_number": 10763} >2018-07-11 08:01:19.846000 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296079845998, "job": 19, "event": "table_file_deletion", "file_number": 9993} >2018-07-11 08:01:19.846016 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296079846014, "job": 19, "event": "table_file_deletion", "file_number": 9736} >2018-07-11 08:01:19.846029 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296079846027, "job": 19, "event": "table_file_deletion", "file_number": 9732} >2018-07-11 08:01:19.846042 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296079846040, "job": 19, "event": "table_file_deletion", "file_number": 9728} >2018-07-11 08:01:19.846228 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 20] Compacting 1@2 + 5@3 files to L3, score 1.33 >2018-07-11 08:01:19.846240 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 19 Base level 2, inputs: [36602(71MB)], [36684(54MB) 7383(39MB) 7389(21MB) 7390(71MB) 7395(41MB)] > >2018-07-11 08:01:19.846260 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296079846244, "job": 20, "event": "compaction_started", "files_L2": [36602], "files_L3": [36684, 7383, 7389, 7390, 7395], "score": 1.33465, "input_data_size": 313947314} >2018-07-11 08:01:23.821347 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 20] Generated table #36966: 3482080 keys, 74294899 bytes >2018-07-11 08:01:23.821394 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296083821368, "cf_name": "default", "job": 20, "event": "table_file_creation", "file_number": 36966, "file_size": 74294899, "table_properties": {"data_size": 64777149, "index_size": 665808, "filter_size": 8850951, "raw_key_size": 100593421, "raw_average_key_size": 28, "raw_value_size": 17620281, "raw_average_value_size": 5, "num_data_blocks": 15908, "num_entries": 3482080, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "2000000", "kMergeOperands": "0"}} >2018-07-11 08:01:27.047315 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 20] Generated table #36967: 3106036 keys, 75717199 bytes >2018-07-11 08:01:27.047359 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296087047333, "cf_name": "default", "job": 20, "event": "table_file_creation", "file_number": 36967, "file_size": 75717199, "table_properties": {"data_size": 67111664, "index_size": 690214, "filter_size": 7914329, "raw_key_size": 89729935, "raw_average_key_size": 28, "raw_value_size": 25038433, "raw_average_value_size": 8, "num_data_blocks": 16486, "num_entries": 3106036, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1000000", "kMergeOperands": "0"}} >2018-07-11 08:01:27.395184 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 20] Generated table #36968: 157661 keys, 4455336 bytes >2018-07-11 08:01:27.395227 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296087395201, "cf_name": "default", "job": 20, "event": "table_file_creation", "file_number": 36968, "file_size": 4455336, "table_properties": {"data_size": 4010631, "index_size": 40744, "filter_size": 402976, "raw_key_size": 4554655, "raw_average_key_size": 28, "raw_value_size": 1874418, "raw_average_value_size": 11, "num_data_blocks": 986, "num_entries": 157661, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:01:30.291807 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 20] Generated table #36969: 2638240 keys, 74545912 bytes >2018-07-11 08:01:30.291853 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296090291826, "cf_name": "default", "job": 20, "event": "table_file_creation", "file_number": 36969, "file_size": 74545912, "table_properties": {"data_size": 67112139, "index_size": 689602, "filter_size": 6743181, "raw_key_size": 76215825, "raw_average_key_size": 28, "raw_value_size": 31365745, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:01:32.746624 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 20] Generated table #36970: 2147917 keys, 73396269 bytes >2018-07-11 08:01:32.746668 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296092746641, "cf_name": "default", "job": 20, "event": "table_file_creation", "file_number": 36970, "file_size": 73396269, "table_properties": {"data_size": 67112819, "index_size": 764668, "filter_size": 5517792, "raw_key_size": 62528698, "raw_average_key_size": 29, "raw_value_size": 37840121, "raw_average_value_size": 17, "num_data_blocks": 16609, "num_entries": 2147917, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:01:33.079422 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 20] Generated table #36971: 20182 keys, 11538881 bytes >2018-07-11 08:01:33.079461 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296093079447, "cf_name": "default", "job": 20, "event": "table_file_creation", "file_number": 36971, "file_size": 11538881, "table_properties": {"data_size": 11281627, "index_size": 180649, "filter_size": 75620, "raw_key_size": 1008671, "raw_average_key_size": 49, "raw_value_size": 10857770, "raw_average_value_size": 537, "num_data_blocks": 2883, "num_entries": 20182, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:01:33.079544 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 20] Compacted 1@2 + 5@3 files to L3 => 313948496 bytes >2018-07-11 08:01:33.094904 7f4922851700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 36025758068 >2018-07-11 08:01:33.095073 7f4922851700 4 rocksdb: (Original Log Time 2018/07/11-08:01:33.094871) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[1 7 54 479 3641 796 0] max score 1.30, MB/sec: 23.7 rd, 23.7 wr, level 3, files in(1, 5) out(6) MB in(71.3, 228.1) out(299.4), read-write-amplify(8.4) write-amplify(4.2) OK, records in: 11552116, records dropped: 0 > >2018-07-11 08:01:33.095083 7f4922851700 4 rocksdb: (Original Log Time 2018/07/11-08:01:33.094896) EVENT_LOG_v1 {"time_micros": 1531296093094885, "job": 20, "event": "compaction_finished", "compaction_time_micros": 13233255, "output_level": 3, "num_output_files": 6, "total_output_size": 313948496, "num_input_records": 11552116, "num_output_records": 11552116, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 7, 54, 479, 3641, 796, 0]} >2018-07-11 08:01:33.095711 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296093095707, "job": 20, "event": "table_file_deletion", "file_number": 36684} >2018-07-11 08:01:33.095737 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296093095735, "job": 20, "event": "table_file_deletion", "file_number": 36602} >2018-07-11 08:01:33.095752 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296093095750, "job": 20, "event": "table_file_deletion", "file_number": 7395} >2018-07-11 08:01:33.095763 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296093095761, "job": 20, "event": "table_file_deletion", "file_number": 7390} >2018-07-11 08:01:33.095774 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296093095772, "job": 20, "event": "table_file_deletion", "file_number": 7389} >2018-07-11 08:01:33.095785 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296093095783, "job": 20, "event": "table_file_deletion", "file_number": 7383} >2018-07-11 08:01:33.095958 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 21] Compacting 1@2 + 8@3 files to L3, score 1.30 >2018-07-11 08:01:33.095972 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 20 Base level 2, inputs: [36520(73MB)], [21257(71MB) 21258(18MB) 21259(71MB) 21260(71MB) 21261(71MB) 21263(52MB) 19213(71MB) 19081(71MB)] > >2018-07-11 08:01:33.095985 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296093095975, "job": 21, "event": "compaction_started", "files_L2": [36520], "files_L3": [21257, 21258, 21259, 21260, 21261, 21263, 19213, 19081], "score": 1.29841, "input_data_size": 599005256} >2018-07-11 08:01:36.301647 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 21] Generated table #36972: 2638239 keys, 74546285 bytes >2018-07-11 08:01:36.301697 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296096301681, "cf_name": "default", "job": 21, "event": "table_file_creation", "file_number": 36972, "file_size": 74546285, "table_properties": {"data_size": 67112141, "index_size": 689975, "filter_size": 6743179, "raw_key_size": 76215804, "raw_average_key_size": 28, "raw_value_size": 31365741, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638239, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:01:39.077606 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 21] Generated table #36973: 2442820 keys, 56977384 bytes >2018-07-11 08:01:39.077652 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296099077637, "cf_name": "default", "job": 21, "event": "table_file_creation", "file_number": 36973, "file_size": 56977384, "table_properties": {"data_size": 50240605, "index_size": 516608, "filter_size": 6219180, "raw_key_size": 70570366, "raw_average_key_size": 28, "raw_value_size": 17153536, "raw_average_value_size": 7, "num_data_blocks": 12340, "num_entries": 2442820, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1000000", "kMergeOperands": "0"}} >2018-07-11 08:01:42.134275 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 21] Generated table #36974: 2638241 keys, 74546022 bytes >2018-07-11 08:01:42.134325 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296102134308, "cf_name": "default", "job": 21, "event": "table_file_creation", "file_number": 36974, "file_size": 74546022, "table_properties": {"data_size": 67112169, "index_size": 689680, "filter_size": 6743183, "raw_key_size": 76215862, "raw_average_key_size": 28, "raw_value_size": 31365765, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:01:45.276071 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 21] Generated table #36975: 2638241 keys, 74545903 bytes >2018-07-11 08:01:45.276121 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296105276105, "cf_name": "default", "job": 21, "event": "table_file_creation", "file_number": 36975, "file_size": 74545903, "table_properties": {"data_size": 67112144, "index_size": 689586, "filter_size": 6743183, "raw_key_size": 76215847, "raw_average_key_size": 28, "raw_value_size": 31365750, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:01:48.603520 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 21] Generated table #36976: 2638241 keys, 74545981 bytes >2018-07-11 08:01:48.603574 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296108603557, "cf_name": "default", "job": 21, "event": "table_file_creation", "file_number": 36976, "file_size": 74545981, "table_properties": {"data_size": 67112098, "index_size": 689710, "filter_size": 6743183, "raw_key_size": 76215847, "raw_average_key_size": 28, "raw_value_size": 31365750, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:01:52.112350 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 21] Generated table #36977: 3106036 keys, 75716908 bytes >2018-07-11 08:01:52.112407 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296112112387, "cf_name": "default", "job": 21, "event": "table_file_creation", "file_number": 36977, "file_size": 75716908, "table_properties": {"data_size": 67111691, "index_size": 689896, "filter_size": 7914329, "raw_key_size": 89729935, "raw_average_key_size": 28, "raw_value_size": 25038433, "raw_average_value_size": 8, "num_data_blocks": 16486, "num_entries": 3106036, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1000000", "kMergeOperands": "0"}} >2018-07-11 08:01:55.151140 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 21] Generated table #36978: 2638240 keys, 74546104 bytes >2018-07-11 08:01:55.151188 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296115151172, "cf_name": "default", "job": 21, "event": "table_file_creation", "file_number": 36978, "file_size": 74546104, "table_properties": {"data_size": 67112115, "index_size": 689818, "filter_size": 6743181, "raw_key_size": 76215822, "raw_average_key_size": 28, "raw_value_size": 31365742, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:01:55.898731 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 21] Generated table #36979: 528836 keys, 14943255 bytes >2018-07-11 08:01:55.898783 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296115898767, "cf_name": "default", "job": 21, "event": "table_file_creation", "file_number": 36979, "file_size": 14943255, "table_properties": {"data_size": 13452669, "index_size": 137922, "filter_size": 1351678, "raw_key_size": 15277486, "raw_average_key_size": 28, "raw_value_size": 6287274, "raw_average_value_size": 11, "num_data_blocks": 3306, "num_entries": 528836, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:01:58.891376 7f491a841700 0 log_channel(cluster) log [DBG] : 9.1b scrub starts >2018-07-11 08:01:58.949748 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 21] Generated table #36980: 2638241 keys, 74545870 bytes >2018-07-11 08:01:58.949785 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296118949766, "cf_name": "default", "job": 21, "event": "table_file_creation", "file_number": 36980, "file_size": 74545870, "table_properties": {"data_size": 67112103, "index_size": 689594, "filter_size": 6743183, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:01:59.069046 7f491a841700 0 log_channel(cluster) log [DBG] : 9.1b scrub ok >2018-07-11 08:01:59.293633 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 21] Generated table #36981: 252507 keys, 4093189 bytes >2018-07-11 08:01:59.293663 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296119293649, "cf_name": "default", "job": 21, "event": "table_file_creation", "file_number": 36981, "file_size": 4093189, "table_properties": {"data_size": 3418390, "index_size": 34607, "filter_size": 639207, "raw_key_size": 7294635, "raw_average_key_size": 28, "raw_value_size": 0, "raw_average_value_size": 0, "num_data_blocks": 839, "num_entries": 252507, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "252507", "kMergeOperands": "0"}} >2018-07-11 08:01:59.293690 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 21] Compacted 1@2 + 8@3 files to L3 => 599006901 bytes >2018-07-11 08:01:59.311012 7f4922851700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 35377750018 >2018-07-11 08:01:59.311155 7f4922851700 4 rocksdb: (Original Log Time 2018/07/11-08:01:59.310980) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[1 7 53 481 3641 796 0] max score 1.26, MB/sec: 22.9 rd, 22.9 wr, level 3, files in(1, 8) out(10) MB in(73.6, 497.6) out(571.3), read-write-amplify(15.5) write-amplify(7.8) OK, records in: 22159642, records dropped: 0 > >2018-07-11 08:01:59.311164 7f4922851700 4 rocksdb: (Original Log Time 2018/07/11-08:01:59.311005) EVENT_LOG_v1 {"time_micros": 1531296119310993, "job": 21, "event": "compaction_finished", "compaction_time_micros": 26197687, "output_level": 3, "num_output_files": 10, "total_output_size": 599006901, "num_input_records": 22159642, "num_output_records": 22159642, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 7, 53, 481, 3641, 796, 0]} >2018-07-11 08:01:59.311779 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296119311772, "job": 21, "event": "table_file_deletion", "file_number": 36520} >2018-07-11 08:01:59.311795 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296119311793, "job": 21, "event": "table_file_deletion", "file_number": 21263} >2018-07-11 08:01:59.311806 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296119311804, "job": 21, "event": "table_file_deletion", "file_number": 21261} >2018-07-11 08:01:59.311817 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296119311815, "job": 21, "event": "table_file_deletion", "file_number": 21260} >2018-07-11 08:01:59.311827 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296119311825, "job": 21, "event": "table_file_deletion", "file_number": 21259} >2018-07-11 08:01:59.311838 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296119311836, "job": 21, "event": "table_file_deletion", "file_number": 21258} >2018-07-11 08:01:59.311852 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296119311849, "job": 21, "event": "table_file_deletion", "file_number": 21257} >2018-07-11 08:01:59.311866 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296119311864, "job": 21, "event": "table_file_deletion", "file_number": 19213} >2018-07-11 08:01:59.311881 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296119311878, "job": 21, "event": "table_file_deletion", "file_number": 19081} >2018-07-11 08:01:59.312062 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 22] Compacting 1@2 + 6@3 files to L3, score 1.26 >2018-07-11 08:01:59.312074 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 21 Base level 2, inputs: [36603(70MB)], [6751(71MB) 6754(71MB) 6758(71MB) 6762(21MB) 6768(71MB) 6769(49MB)] > >2018-07-11 08:01:59.312089 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296119312078, "job": 22, "event": "compaction_started", "files_L2": [36603], "files_L3": [6751, 6754, 6758, 6762, 6768, 6769], "score": 1.26481, "input_data_size": 446595015} >2018-07-11 08:02:02.006451 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 22] Generated table #36982: 1552673 keys, 71999321 bytes >2018-07-11 08:02:02.006483 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296122006468, "cf_name": "default", "job": 22, "event": "table_file_creation", "file_number": 36982, "file_size": 71999321, "table_properties": {"data_size": 67112270, "index_size": 855840, "filter_size": 4030222, "raw_key_size": 45969433, "raw_average_key_size": 29, "raw_value_size": 45689328, "raw_average_value_size": 29, "num_data_blocks": 16755, "num_entries": 1552673, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:02:05.142800 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 22] Generated table #36983: 2638240 keys, 74546534 bytes >2018-07-11 08:02:05.142849 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296125142833, "cf_name": "default", "job": 22, "event": "table_file_creation", "file_number": 36983, "file_size": 74546534, "table_properties": {"data_size": 67112112, "index_size": 690251, "filter_size": 6743181, "raw_key_size": 76215831, "raw_average_key_size": 28, "raw_value_size": 31365751, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:02:07.393286 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 22] Generated table #36984: 1789114 keys, 50553952 bytes >2018-07-11 08:02:07.393332 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296127393316, "cf_name": "default", "job": 22, "event": "table_file_creation", "file_number": 36984, "file_size": 50553952, "table_properties": {"data_size": 45511884, "index_size": 468219, "filter_size": 4572861, "raw_key_size": 51685524, "raw_average_key_size": 28, "raw_value_size": 21270586, "raw_average_value_size": 11, "num_data_blocks": 11182, "num_entries": 1789114, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:02:10.667380 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 22] Generated table #36985: 2638241 keys, 74546019 bytes >2018-07-11 08:02:10.667411 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296130667397, "cf_name": "default", "job": 22, "event": "table_file_creation", "file_number": 36985, "file_size": 74546019, "table_properties": {"data_size": 67112150, "index_size": 689695, "filter_size": 6743184, "raw_key_size": 76215853, "raw_average_key_size": 28, "raw_value_size": 31365756, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:02:14.337517 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 22] Generated table #36986: 3106039 keys, 75716836 bytes >2018-07-11 08:02:14.337567 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296134337551, "cf_name": "default", "job": 22, "event": "table_file_creation", "file_number": 36986, "file_size": 75716836, "table_properties": {"data_size": 67111770, "index_size": 689738, "filter_size": 7914336, "raw_key_size": 89730019, "raw_average_key_size": 28, "raw_value_size": 25038466, "raw_average_value_size": 8, "num_data_blocks": 16486, "num_entries": 3106039, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1000000", "kMergeOperands": "0"}} >2018-07-11 08:02:17.528853 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 22] Generated table #36987: 2638240 keys, 74546276 bytes >2018-07-11 08:02:17.528891 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296137528872, "cf_name": "default", "job": 22, "event": "table_file_creation", "file_number": 36987, "file_size": 74546276, "table_properties": {"data_size": 67112173, "index_size": 689932, "filter_size": 6743181, "raw_key_size": 76215832, "raw_average_key_size": 28, "raw_value_size": 31365752, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:02:18.857405 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 22] Generated table #36988: 1127525 keys, 24687772 bytes >2018-07-11 08:02:18.857456 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296138857439, "cf_name": "default", "job": 22, "event": "table_file_creation", "file_number": 36988, "file_size": 24687772, "table_properties": {"data_size": 21597838, "index_size": 221648, "filter_size": 2867296, "raw_key_size": 32572937, "raw_average_key_size": 28, "raw_value_size": 6327457, "raw_average_value_size": 5, "num_data_blocks": 5305, "num_entries": 1127525, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "595309", "kMergeOperands": "0"}} >2018-07-11 08:02:18.857487 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 22] Compacted 1@2 + 6@3 files to L3 => 446596710 bytes >2018-07-11 08:02:18.877393 7f4922851700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 34727363913 >2018-07-11 08:02:18.877532 7f4922851700 4 rocksdb: (Original Log Time 2018/07/11-08:02:18.877360) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[1 7 52 482 3641 796 0] max score 1.23, MB/sec: 22.8 rd, 22.8 wr, level 3, files in(1, 6) out(7) MB in(70.4, 355.5) out(425.9), read-write-amplify(12.1) write-amplify(6.0) OK, records in: 15490072, records dropped: 0 > >2018-07-11 08:02:18.877541 7f4922851700 4 rocksdb: (Original Log Time 2018/07/11-08:02:18.877384) EVENT_LOG_v1 {"time_micros": 1531296138877373, "job": 22, "event": "compaction_finished", "compaction_time_micros": 19545382, "output_level": 3, "num_output_files": 7, "total_output_size": 446596710, "num_input_records": 15490072, "num_output_records": 15490072, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 7, 52, 482, 3641, 796, 0]} >2018-07-11 08:02:18.878157 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296138878152, "job": 22, "event": "table_file_deletion", "file_number": 36603} >2018-07-11 08:02:18.878173 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296138878171, "job": 22, "event": "table_file_deletion", "file_number": 6769} >2018-07-11 08:02:18.878185 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296138878183, "job": 22, "event": "table_file_deletion", "file_number": 6768} >2018-07-11 08:02:18.878196 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296138878194, "job": 22, "event": "table_file_deletion", "file_number": 6762} >2018-07-11 08:02:18.878210 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296138878205, "job": 22, "event": "table_file_deletion", "file_number": 6758} >2018-07-11 08:02:18.878222 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296138878220, "job": 22, "event": "table_file_deletion", "file_number": 6754} >2018-07-11 08:02:18.878237 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296138878234, "job": 22, "event": "table_file_deletion", "file_number": 6751} >2018-07-11 08:02:18.878428 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 23] Compacting 1@2 + 2@3 files to L3, score 1.23 >2018-07-11 08:02:18.878440 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 22 Base level 2, inputs: [36620(73MB)], [28420(71MB) 36751(72MB)] > >2018-07-11 08:02:18.878455 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296138878443, "job": 23, "event": "compaction_started", "files_L2": [36620], "files_L3": [28420, 36751], "score": 1.23198, "input_data_size": 227312558} >2018-07-11 08:02:23.233005 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 23] Generated table #36989: 3426601 keys, 76518198 bytes >2018-07-11 08:02:23.233050 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296143233035, "cf_name": "default", "job": 23, "event": "table_file_creation", "file_number": 36989, "file_size": 76518198, "table_properties": {"data_size": 67110371, "index_size": 689963, "filter_size": 8716872, "raw_key_size": 98990715, "raw_average_key_size": 28, "raw_value_size": 20701559, "raw_average_value_size": 6, "num_data_blocks": 16483, "num_entries": 3426601, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1685349", "kMergeOperands": "0"}} >2018-07-11 08:02:26.860153 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 23] Generated table #36990: 2832456 keys, 75031455 bytes >2018-07-11 08:02:26.860198 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296146860183, "cf_name": "default", "job": 23, "event": "table_file_creation", "file_number": 36990, "file_size": 75031455, "table_properties": {"data_size": 67110607, "index_size": 690451, "filter_size": 7229405, "raw_key_size": 81826481, "raw_average_key_size": 28, "raw_value_size": 28737484, "raw_average_value_size": 10, "num_data_blocks": 16488, "num_entries": 2832456, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "415283", "kMergeOperands": "0"}} >2018-07-11 08:02:30.356088 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 23] Generated table #36991: 2988814 keys, 72749754 bytes >2018-07-11 08:02:30.356133 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296150356118, "cf_name": "default", "job": 23, "event": "table_file_creation", "file_number": 36991, "file_size": 72749754, "table_properties": {"data_size": 64471132, "index_size": 662221, "filter_size": 7615410, "raw_key_size": 86343516, "raw_average_key_size": 28, "raw_value_size": 23985875, "raw_average_value_size": 8, "num_data_blocks": 15837, "num_entries": 2988814, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "971311", "kMergeOperands": "0"}} >2018-07-11 08:02:30.356164 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 23] Compacted 1@2 + 2@3 files to L3 => 224299407 bytes >2018-07-11 08:02:30.377057 7f4922851700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 33982783386 >2018-07-11 08:02:30.377178 7f4922851700 4 rocksdb: (Original Log Time 2018/07/11-08:02:30.377028) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[1 7 51 483 3641 796 0] max score 1.20, MB/sec: 19.8 rd, 19.5 wr, level 3, files in(1, 2) out(3) MB in(73.5, 143.3) out(213.9), read-write-amplify(5.9) write-amplify(2.9) OK, records in: 9383399, records dropped: 135528 > >2018-07-11 08:02:30.377187 7f4922851700 4 rocksdb: (Original Log Time 2018/07/11-08:02:30.377050) EVENT_LOG_v1 {"time_micros": 1531296150377040, "job": 23, "event": "compaction_finished", "compaction_time_micros": 11477695, "output_level": 3, "num_output_files": 3, "total_output_size": 224299407, "num_input_records": 9383399, "num_output_records": 9247871, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 7, 51, 483, 3641, 796, 0]} >2018-07-11 08:02:30.377825 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296150377809, "job": 23, "event": "table_file_deletion", "file_number": 36751} >2018-07-11 08:02:30.377840 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296150377838, "job": 23, "event": "table_file_deletion", "file_number": 36620} >2018-07-11 08:02:30.377854 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296150377852, "job": 23, "event": "table_file_deletion", "file_number": 28420} >2018-07-11 08:02:30.378034 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 24] Compacting 1@2 + 6@3 files to L3, score 1.20 >2018-07-11 08:02:30.378046 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 23 Base level 2, inputs: [36521(48MB)], [21170(59MB) 21172(71MB) 21174(35KB) 21175(71MB) 21177(11MB) 36917(71MB)] > >2018-07-11 08:02:30.378060 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296150378050, "job": 24, "event": "compaction_started", "files_L2": [36521], "files_L3": [21170, 21172, 21174, 21175, 21177, 36917], "score": 1.19918, "input_data_size": 349472249} >2018-07-11 08:02:34.235973 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 24] Generated table #36992: 2987913 keys, 75421285 bytes >2018-07-11 08:02:34.236004 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296154235989, "cf_name": "default", "job": 24, "event": "table_file_creation", "file_number": 36992, "file_size": 75421285, "table_properties": {"data_size": 67111815, "index_size": 689876, "filter_size": 7618602, "raw_key_size": 86317506, "raw_average_key_size": 28, "raw_value_size": 26636111, "raw_average_value_size": 8, "num_data_blocks": 16487, "num_entries": 2987913, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "747493", "kMergeOperands": "0"}} >2018-07-11 08:02:34.928073 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 24] Generated table #36993: 396316 keys, 11198615 bytes >2018-07-11 08:02:34.928103 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296154928088, "cf_name": "default", "job": 24, "event": "table_file_creation", "file_number": 36993, "file_size": 11198615, "table_properties": {"data_size": 10081567, "index_size": 103102, "filter_size": 1012960, "raw_key_size": 11449131, "raw_average_key_size": 28, "raw_value_size": 4711759, "raw_average_value_size": 11, "num_data_blocks": 2477, "num_entries": 396316, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:02:38.185252 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 24] Generated table #36994: 2638241 keys, 74546029 bytes >2018-07-11 08:02:38.185285 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296158185270, "cf_name": "default", "job": 24, "event": "table_file_creation", "file_number": 36994, "file_size": 74546029, "table_properties": {"data_size": 67112128, "index_size": 689728, "filter_size": 6743183, "raw_key_size": 76215851, "raw_average_key_size": 28, "raw_value_size": 31365754, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:02:41.833916 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 24] Generated table #36995: 3106037 keys, 75716798 bytes >2018-07-11 08:02:41.833960 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296161833945, "cf_name": "default", "job": 24, "event": "table_file_creation", "file_number": 36995, "file_size": 75716798, "table_properties": {"data_size": 67111687, "index_size": 689789, "filter_size": 7914330, "raw_key_size": 89729957, "raw_average_key_size": 28, "raw_value_size": 25038438, "raw_average_value_size": 8, "num_data_blocks": 16486, "num_entries": 3106037, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1000000", "kMergeOperands": "0"}} >2018-07-11 08:02:41.921286 7f4924054700 1 bluestore(/var/lib/ceph/osd/ceph-10) _balance_bluefs_freespace gifting 0xd3d9f00000~80000000 to bluefs >2018-07-11 08:02:41.935109 7f4924054700 1 bluefs add_block_extent bdev 2 0xd3d9f00000~80000000 >2018-07-11 08:02:42.528274 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 24] Generated table #36996: 533438 keys, 15073231 bytes >2018-07-11 08:02:42.528318 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296162528303, "cf_name": "default", "job": 24, "event": "table_file_creation", "file_number": 36996, "file_size": 15073231, "table_properties": {"data_size": 13569707, "index_size": 139104, "filter_size": 1363434, "raw_key_size": 15410433, "raw_average_key_size": 28, "raw_value_size": 6341987, "raw_average_value_size": 11, "num_data_blocks": 3334, "num_entries": 533438, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:02:46.479384 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 24] Generated table #36997: 3106036 keys, 75716808 bytes >2018-07-11 08:02:46.479429 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296166479413, "cf_name": "default", "job": 24, "event": "table_file_creation", "file_number": 36997, "file_size": 75716808, "table_properties": {"data_size": 67111724, "index_size": 689763, "filter_size": 7914329, "raw_key_size": 89729936, "raw_average_key_size": 28, "raw_value_size": 25038434, "raw_average_value_size": 8, "num_data_blocks": 16486, "num_entries": 3106036, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1000000", "kMergeOperands": "0"}} >2018-07-11 08:02:47.406398 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 24] Generated table #36998: 771497 keys, 21799994 bytes >2018-07-11 08:02:47.406428 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296167406414, "cf_name": "default", "job": 24, "event": "table_file_creation", "file_number": 36998, "file_size": 21799994, "table_properties": {"data_size": 19625495, "index_size": 201615, "filter_size": 1971898, "raw_key_size": 22287688, "raw_average_key_size": 28, "raw_value_size": 9172239, "raw_average_value_size": 11, "num_data_blocks": 4822, "num_entries": 771497, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:02:47.406458 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 24] Compacted 1@2 + 6@3 files to L3 => 349472760 bytes >2018-07-11 08:02:47.426873 7f4922851700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 33493078760 >2018-07-11 08:02:47.427102 7f4922851700 4 rocksdb: (Original Log Time 2018/07/11-08:02:47.426839) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[1 7 50 484 3641 796 0] max score 1.17, MB/sec: 20.5 rd, 20.5 wr, level 3, files in(1, 6) out(7) MB in(48.4, 284.8) out(333.3), read-write-amplify(13.8) write-amplify(6.9) OK, records in: 13539478, records dropped: 0 > >2018-07-11 08:02:47.427111 7f4922851700 4 rocksdb: (Original Log Time 2018/07/11-08:02:47.426865) EVENT_LOG_v1 {"time_micros": 1531296167426853, "job": 24, "event": "compaction_finished", "compaction_time_micros": 17028382, "output_level": 3, "num_output_files": 7, "total_output_size": 349472760, "num_input_records": 13539478, "num_output_records": 13539478, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 7, 50, 484, 3641, 796, 0]} >2018-07-11 08:02:47.427760 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296167427743, "job": 24, "event": "table_file_deletion", "file_number": 36917} >2018-07-11 08:02:47.427776 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296167427774, "job": 24, "event": "table_file_deletion", "file_number": 36521} >2018-07-11 08:02:47.427793 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296167427787, "job": 24, "event": "table_file_deletion", "file_number": 21177} >2018-07-11 08:02:47.427805 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296167427803, "job": 24, "event": "table_file_deletion", "file_number": 21175} >2018-07-11 08:02:47.427816 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296167427814, "job": 24, "event": "table_file_deletion", "file_number": 21174} >2018-07-11 08:02:47.427831 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296167427829, "job": 24, "event": "table_file_deletion", "file_number": 21172} >2018-07-11 08:02:47.427845 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296167427843, "job": 24, "event": "table_file_deletion", "file_number": 21170} >2018-07-11 08:02:47.428031 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 25] Compacting 1@2 + 9@3 files to L3, score 1.17 >2018-07-11 08:02:47.428045 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 24 Base level 2, inputs: [36519(63MB)], [36893(30MB) 18164(71MB) 18165(71MB) 18169(55MB) 18214(71MB) 18197(71MB) 18199(1088B) 18069(71MB) 29919(20MB)] > >2018-07-11 08:02:47.428059 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296167428048, "job": 25, "event": "compaction_started", "files_L2": [36519], "files_L3": [36893, 18164, 18165, 18169, 18214, 18197, 18199, 18069, 29919], "score": 1.16842, "input_data_size": 550762669} >2018-07-11 08:02:51.362925 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 25] Generated table #36999: 3106037 keys, 75717329 bytes >2018-07-11 08:02:51.362977 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296171362961, "cf_name": "default", "job": 25, "event": "table_file_creation", "file_number": 36999, "file_size": 75717329, "table_properties": {"data_size": 67111662, "index_size": 690344, "filter_size": 7914331, "raw_key_size": 89729957, "raw_average_key_size": 28, "raw_value_size": 25038438, "raw_average_value_size": 8, "num_data_blocks": 16486, "num_entries": 3106037, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1000000", "kMergeOperands": "0"}} >2018-07-11 08:02:54.644924 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 25] Generated table #37000: 2638240 keys, 74546084 bytes >2018-07-11 08:02:54.644956 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296174644941, "cf_name": "default", "job": 25, "event": "table_file_creation", "file_number": 37000, "file_size": 74546084, "table_properties": {"data_size": 67112159, "index_size": 689754, "filter_size": 6743181, "raw_key_size": 76215826, "raw_average_key_size": 28, "raw_value_size": 31365746, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:02:57.995639 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 25] Generated table #37001: 2638241 keys, 74546412 bytes >2018-07-11 08:02:57.995683 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296177995657, "cf_name": "default", "job": 25, "event": "table_file_creation", "file_number": 37001, "file_size": 74546412, "table_properties": {"data_size": 67112110, "index_size": 690129, "filter_size": 6743183, "raw_key_size": 76215850, "raw_average_key_size": 28, "raw_value_size": 31365753, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:02:59.862242 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 25] Generated table #37002: 1532046 keys, 31242201 bytes >2018-07-11 08:02:59.862277 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296179862263, "cf_name": "default", "job": 25, "event": "table_file_creation", "file_number": 37002, "file_size": 31242201, "table_properties": {"data_size": 27072145, "index_size": 277765, "filter_size": 3891301, "raw_key_size": 44259106, "raw_average_key_size": 28, "raw_value_size": 6325434, "raw_average_value_size": 4, "num_data_blocks": 6648, "num_entries": 1532046, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1000000", "kMergeOperands": "0"}} >2018-07-11 08:03:03.383010 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 25] Generated table #37003: 2638240 keys, 74546033 bytes >2018-07-11 08:03:03.383056 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296183383036, "cf_name": "default", "job": 25, "event": "table_file_creation", "file_number": 37003, "file_size": 74546033, "table_properties": {"data_size": 67112135, "index_size": 689727, "filter_size": 6743181, "raw_key_size": 76215820, "raw_average_key_size": 28, "raw_value_size": 31365740, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:03:06.450613 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 25] Generated table #37004: 2638240 keys, 74545876 bytes >2018-07-11 08:03:06.450658 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296186450631, "cf_name": "default", "job": 25, "event": "table_file_creation", "file_number": 37004, "file_size": 74545876, "table_properties": {"data_size": 67112093, "index_size": 689612, "filter_size": 6743181, "raw_key_size": 76215819, "raw_average_key_size": 28, "raw_value_size": 31365739, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:03:09.534847 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 25] Generated table #37005: 2638240 keys, 74546869 bytes >2018-07-11 08:03:09.534892 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296189534877, "cf_name": "default", "job": 25, "event": "table_file_creation", "file_number": 37005, "file_size": 74546869, "table_properties": {"data_size": 67112113, "index_size": 690585, "filter_size": 6743181, "raw_key_size": 76215820, "raw_average_key_size": 28, "raw_value_size": 31365740, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:03:12.873886 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 25] Generated table #37006: 2941668 keys, 71072242 bytes >2018-07-11 08:03:12.873930 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296192873903, "cf_name": "default", "job": 25, "event": "table_file_creation", "file_number": 37006, "file_size": 71072242, "table_properties": {"data_size": 62930453, "index_size": 646595, "filter_size": 7494203, "raw_key_size": 84981519, "raw_average_key_size": 28, "raw_value_size": 23084273, "raw_average_value_size": 7, "num_data_blocks": 15458, "num_entries": 2941668, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1000000", "kMergeOperands": "0"}} >2018-07-11 08:03:12.873982 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 25] Compacted 1@2 + 9@3 files to L3 => 550763046 bytes >2018-07-11 08:03:12.893183 7f4922851700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 32825220735 >2018-07-11 08:03:12.893445 7f4922851700 4 rocksdb: (Original Log Time 2018/07/11-08:03:12.893149) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[1 7 49 483 3641 796 0] max score 1.14, MB/sec: 21.6 rd, 21.6 wr, level 3, files in(1, 9) out(8) MB in(63.5, 461.8) out(525.2), read-write-amplify(16.5) write-amplify(8.3) OK, records in: 20770952, records dropped: 0 > >2018-07-11 08:03:12.893455 7f4922851700 4 rocksdb: (Original Log Time 2018/07/11-08:03:12.893174) EVENT_LOG_v1 {"time_micros": 1531296192893163, "job": 25, "event": "compaction_finished", "compaction_time_micros": 25445905, "output_level": 3, "num_output_files": 8, "total_output_size": 550763046, "num_input_records": 20770952, "num_output_records": 20770952, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 7, 49, 483, 3641, 796, 0]} >2018-07-11 08:03:12.894178 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296192894173, "job": 25, "event": "table_file_deletion", "file_number": 36893} >2018-07-11 08:03:12.894198 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296192894196, "job": 25, "event": "table_file_deletion", "file_number": 36519} >2018-07-11 08:03:12.894217 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296192894215, "job": 25, "event": "table_file_deletion", "file_number": 29919} >2018-07-11 08:03:12.894235 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296192894233, "job": 25, "event": "table_file_deletion", "file_number": 18214} >2018-07-11 08:03:12.894262 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296192894259, "job": 25, "event": "table_file_deletion", "file_number": 18199} >2018-07-11 08:03:12.894279 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296192894277, "job": 25, "event": "table_file_deletion", "file_number": 18197} >2018-07-11 08:03:12.894292 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296192894290, "job": 25, "event": "table_file_deletion", "file_number": 18169} >2018-07-11 08:03:12.894303 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296192894301, "job": 25, "event": "table_file_deletion", "file_number": 18165} >2018-07-11 08:03:12.894315 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296192894312, "job": 25, "event": "table_file_deletion", "file_number": 18164} >2018-07-11 08:03:12.894329 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296192894327, "job": 25, "event": "table_file_deletion", "file_number": 18069} >2018-07-11 08:03:12.894520 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 26] Compacting 1@3 + 14@4 files to L4, score 1.14 >2018-07-11 08:03:12.894536 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 25 Base level 3, inputs: [36629(76MB)], [27421(71MB) 27422(71MB) 27425(71MB) 27426(71MB) 27428(71MB) 26206(71MB) 26207(71MB) 26208(71MB) 26210(71MB) 26211(71MB) 26212(71MB) 26213(71MB) 26215(71MB) 26216(71MB)] > >2018-07-11 08:03:12.894553 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296192894542, "job": 26, "event": "compaction_started", "files_L3": [36629], "files_L4": [27421, 27422, 27425, 27426, 27428, 26206, 26207, 26208, 26210, 26211, 26212, 26213, 26215, 26216], "score": 1.13923, "input_data_size": 1123995901} >2018-07-11 08:03:16.935344 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 26] Generated table #37007: 2638241 keys, 74546546 bytes >2018-07-11 08:03:16.935395 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296196935379, "cf_name": "default", "job": 26, "event": "table_file_creation", "file_number": 37007, "file_size": 74546546, "table_properties": {"data_size": 67112117, "index_size": 690256, "filter_size": 6743183, "raw_key_size": 76215841, "raw_average_key_size": 28, "raw_value_size": 31365744, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:03:20.381932 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 26] Generated table #37008: 2638241 keys, 74545820 bytes >2018-07-11 08:03:20.381984 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296200381964, "cf_name": "default", "job": 26, "event": "table_file_creation", "file_number": 37008, "file_size": 74545820, "table_properties": {"data_size": 67112098, "index_size": 689549, "filter_size": 6743183, "raw_key_size": 76215851, "raw_average_key_size": 28, "raw_value_size": 31365754, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:03:23.635670 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 26] Generated table #37009: 2638241 keys, 74546347 bytes >2018-07-11 08:03:23.635705 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296203635690, "cf_name": "default", "job": 26, "event": "table_file_creation", "file_number": 37009, "file_size": 74546347, "table_properties": {"data_size": 67112177, "index_size": 689997, "filter_size": 6743183, "raw_key_size": 76215860, "raw_average_key_size": 28, "raw_value_size": 31365763, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:03:27.308409 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 26] Generated table #37010: 3106038 keys, 75717318 bytes >2018-07-11 08:03:27.308461 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296207308445, "cf_name": "default", "job": 26, "event": "table_file_creation", "file_number": 37010, "file_size": 75717318, "table_properties": {"data_size": 67111718, "index_size": 690275, "filter_size": 7914333, "raw_key_size": 89729995, "raw_average_key_size": 28, "raw_value_size": 25038459, "raw_average_value_size": 8, "num_data_blocks": 16486, "num_entries": 3106038, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "1000000", "kMergeOperands": "0"}} >2018-07-11 08:03:31.332566 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 26] Generated table #37011: 2830928 keys, 75027107 bytes >2018-07-11 08:03:31.332617 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296211332601, "cf_name": "default", "job": 26, "event": "table_file_creation", "file_number": 37011, "file_size": 75027107, "table_properties": {"data_size": 67110482, "index_size": 690054, "filter_size": 7225579, "raw_key_size": 81782357, "raw_average_key_size": 28, "raw_value_size": 28758071, "raw_average_value_size": 10, "num_data_blocks": 16488, "num_entries": 2830928, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "412025", "kMergeOperands": "0"}} >2018-07-11 08:03:35.493093 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 26] Generated table #37012: 2913190 keys, 75231599 bytes >2018-07-11 08:03:35.493144 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296215493129, "cf_name": "default", "job": 26, "event": "table_file_creation", "file_number": 37012, "file_size": 75231599, "table_properties": {"data_size": 67109247, "index_size": 689835, "filter_size": 7431525, "raw_key_size": 84158836, "raw_average_key_size": 28, "raw_value_size": 27644226, "raw_average_value_size": 9, "num_data_blocks": 16487, "num_entries": 2913190, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "587975", "kMergeOperands": "0"}} >2018-07-11 08:03:38.849751 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 26] Generated table #37013: 2638241 keys, 74545853 bytes >2018-07-11 08:03:38.849803 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296218849787, "cf_name": "default", "job": 26, "event": "table_file_creation", "file_number": 37013, "file_size": 74545853, "table_properties": {"data_size": 67112108, "index_size": 689572, "filter_size": 6743183, "raw_key_size": 76215855, "raw_average_key_size": 28, "raw_value_size": 31365758, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:03:42.167733 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 26] Generated table #37014: 2638241 keys, 74546550 bytes >2018-07-11 08:03:42.167785 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296222167769, "cf_name": "default", "job": 26, "event": "table_file_creation", "file_number": 37014, "file_size": 74546550, "table_properties": {"data_size": 67112153, "index_size": 690224, "filter_size": 6743183, "raw_key_size": 76215855, "raw_average_key_size": 28, "raw_value_size": 31365758, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:03:45.476065 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 26] Generated table #37015: 2638241 keys, 74546280 bytes >2018-07-11 08:03:45.476117 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296225476089, "cf_name": "default", "job": 26, "event": "table_file_creation", "file_number": 37015, "file_size": 74546280, "table_properties": {"data_size": 67112152, "index_size": 689955, "filter_size": 6743183, "raw_key_size": 76215848, "raw_average_key_size": 28, "raw_value_size": 31365751, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:03:48.926885 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 26] Generated table #37016: 2638241 keys, 74546371 bytes >2018-07-11 08:03:48.926932 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296228926917, "cf_name": "default", "job": 26, "event": "table_file_creation", "file_number": 37016, "file_size": 74546371, "table_properties": {"data_size": 67112164, "index_size": 690034, "filter_size": 6743183, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:03:52.162621 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 26] Generated table #37017: 2638241 keys, 74545868 bytes >2018-07-11 08:03:52.162671 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296232162654, "cf_name": "default", "job": 26, "event": "table_file_creation", "file_number": 37017, "file_size": 74545868, "table_properties": {"data_size": 67112128, "index_size": 689567, "filter_size": 6743183, "raw_key_size": 76215864, "raw_average_key_size": 28, "raw_value_size": 31365767, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:03:55.348519 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 26] Generated table #37018: 2638241 keys, 74546781 bytes >2018-07-11 08:03:55.348569 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296235348553, "cf_name": "default", "job": 26, "event": "table_file_creation", "file_number": 37018, "file_size": 74546781, "table_properties": {"data_size": 67112144, "index_size": 690464, "filter_size": 6743183, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:03:59.423679 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 26] Generated table #37019: 2638241 keys, 74546299 bytes >2018-07-11 08:03:59.423722 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296239423694, "cf_name": "default", "job": 26, "event": "table_file_creation", "file_number": 37019, "file_size": 74546299, "table_properties": {"data_size": 67112165, "index_size": 689961, "filter_size": 6743183, "raw_key_size": 76215847, "raw_average_key_size": 28, "raw_value_size": 31365750, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:04:00.495685 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 26] Generated table #37020: 745642 keys, 21069283 bytes >2018-07-11 08:04:00.495729 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296240495714, "cf_name": "default", "job": 26, "event": "table_file_creation", "file_number": 37020, "file_size": 21069283, "table_properties": {"data_size": 18967836, "index_size": 194641, "filter_size": 1905820, "raw_key_size": 21540786, "raw_average_key_size": 28, "raw_value_size": 8864872, "raw_average_value_size": 11, "num_data_blocks": 4661, "num_entries": 745642, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:04:00.495769 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 26] Compacted 1@3 + 14@4 files to L4 => 992508022 bytes >2018-07-11 08:04:00.517074 7f4922851700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 31908658051 >2018-07-11 08:04:00.517369 7f4922851700 4 rocksdb: (Original Log Time 2018/07/11-08:04:00.517039) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[1 7 49 482 3641 796 0] max score 1.14, MB/sec: 23.6 rd, 20.9 wr, level 4, files in(1, 14) out(14) MB in(76.6, 995.3) out(946.5), read-write-amplify(26.3) write-amplify(12.4) OK, records in: 41892538, records dropped: 5914330 > >2018-07-11 08:04:00.517378 7f4922851700 4 rocksdb: (Original Log Time 2018/07/11-08:04:00.517065) EVENT_LOG_v1 {"time_micros": 1531296240517053, "job": 26, "event": "compaction_finished", "compaction_time_micros": 47601193, "output_level": 4, "num_output_files": 14, "total_output_size": 992508022, "num_input_records": 41892538, "num_output_records": 35978208, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 7, 49, 482, 3641, 796, 0]} >2018-07-11 08:04:00.518012 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296240517995, "job": 26, "event": "table_file_deletion", "file_number": 36629} >2018-07-11 08:04:00.518030 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296240518027, "job": 26, "event": "table_file_deletion", "file_number": 27428} >2018-07-11 08:04:00.518041 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296240518039, "job": 26, "event": "table_file_deletion", "file_number": 27426} >2018-07-11 08:04:00.518053 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296240518050, "job": 26, "event": "table_file_deletion", "file_number": 27425} >2018-07-11 08:04:00.518063 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296240518061, "job": 26, "event": "table_file_deletion", "file_number": 27422} >2018-07-11 08:04:00.518079 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296240518076, "job": 26, "event": "table_file_deletion", "file_number": 27421} >2018-07-11 08:04:00.518095 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296240518093, "job": 26, "event": "table_file_deletion", "file_number": 26216} >2018-07-11 08:04:00.518109 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296240518106, "job": 26, "event": "table_file_deletion", "file_number": 26215} >2018-07-11 08:04:00.518122 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296240518120, "job": 26, "event": "table_file_deletion", "file_number": 26213} >2018-07-11 08:04:00.518135 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296240518133, "job": 26, "event": "table_file_deletion", "file_number": 26212} >2018-07-11 08:04:00.518148 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296240518145, "job": 26, "event": "table_file_deletion", "file_number": 26211} >2018-07-11 08:04:00.518160 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296240518158, "job": 26, "event": "table_file_deletion", "file_number": 26210} >2018-07-11 08:04:00.518175 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296240518171, "job": 26, "event": "table_file_deletion", "file_number": 26208} >2018-07-11 08:04:00.518187 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296240518185, "job": 26, "event": "table_file_deletion", "file_number": 26207} >2018-07-11 08:04:00.518200 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296240518197, "job": 26, "event": "table_file_deletion", "file_number": 26206} >2018-07-11 08:04:00.533703 7f4922851700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 31503511159 >2018-07-11 08:04:00.533802 7f4922851700 4 rocksdb: (Original Log Time 2018/07/11-08:04:00.518381) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1509] [default] Moving #36636 to level-3 39044325 bytes > >2018-07-11 08:04:00.533811 7f4922851700 4 rocksdb: (Original Log Time 2018/07/11-08:04:00.533724) EVENT_LOG_v1 {"time_micros": 1531296240533718, "job": 27, "event": "trivial_move", "destination_level": 3, "files": 1, "total_files_size": 39044325} >2018-07-11 08:04:00.533814 7f4922851700 4 rocksdb: (Original Log Time 2018/07/11-08:04:00.533730) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_compaction_flush.cc:1537] [default] Moved #1 files to level-3 39044325 bytes OK: base level 1 max bytes base 268435456 files[1 7 48 483 3641 796 0] max score 1.14 > >2018-07-11 08:04:00.534436 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 28] Compacting 1@3 + 13@4 files to L4, score 1.14 >2018-07-11 08:04:00.534451 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 27 Base level 3, inputs: [36632(76MB)], [28118(71MB) 28119(71MB) 28120(71MB) 28122(71MB) 26354(71MB) 26355(71MB) 26356(71MB) 26357(71MB) 26358(71MB) 26360(71MB) 26361(71MB) 26362(71MB) 26363(71MB)] > >2018-07-11 08:04:00.534467 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296240534455, "job": 28, "event": "compaction_started", "files_L3": [36632], "files_L4": [28118, 28119, 28120, 28122, 26354, 26355, 26356, 26357, 26358, 26360, 26361, 26362, 26363], "score": 1.13618, "input_data_size": 1049449294} >2018-07-11 08:04:04.474428 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 28] Generated table #37021: 3049846 keys, 75574653 bytes >2018-07-11 08:04:04.474467 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296244474451, "cf_name": "default", "job": 28, "event": "table_file_creation", "file_number": 37021, "file_size": 75574653, "table_properties": {"data_size": 67109868, "index_size": 690143, "filter_size": 7773650, "raw_key_size": 88106676, "raw_average_key_size": 28, "raw_value_size": 25796572, "raw_average_value_size": 8, "num_data_blocks": 16486, "num_entries": 3049846, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "880041", "kMergeOperands": "0"}} >2018-07-11 08:04:07.778405 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 28] Generated table #37022: 2638241 keys, 74546271 bytes >2018-07-11 08:04:07.778443 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296247778427, "cf_name": "default", "job": 28, "event": "table_file_creation", "file_number": 37022, "file_size": 74546271, "table_properties": {"data_size": 67112164, "index_size": 689934, "filter_size": 6743183, "raw_key_size": 76215856, "raw_average_key_size": 28, "raw_value_size": 31365759, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:04:12.397181 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 28] Generated table #37023: 2947811 keys, 75319123 bytes >2018-07-11 08:04:12.397213 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296252397198, "cf_name": "default", "job": 28, "event": "table_file_creation", "file_number": 37023, "file_size": 75319123, "table_properties": {"data_size": 67110420, "index_size": 689511, "filter_size": 7518200, "raw_key_size": 85158963, "raw_average_key_size": 28, "raw_value_size": 27177151, "raw_average_value_size": 9, "num_data_blocks": 16487, "num_entries": 2947811, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "661881", "kMergeOperands": "0"}} >2018-07-11 08:04:15.696907 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 28] Generated table #37024: 2638241 keys, 74546581 bytes >2018-07-11 08:04:15.696954 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296255696938, "cf_name": "default", "job": 28, "event": "table_file_creation", "file_number": 37024, "file_size": 74546581, "table_properties": {"data_size": 67112121, "index_size": 690287, "filter_size": 6743183, "raw_key_size": 76215859, "raw_average_key_size": 28, "raw_value_size": 31365762, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:04:19.026292 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 28] Generated table #37025: 2638241 keys, 74546767 bytes >2018-07-11 08:04:19.026339 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296259026323, "cf_name": "default", "job": 28, "event": "table_file_creation", "file_number": 37025, "file_size": 74546767, "table_properties": {"data_size": 67112150, "index_size": 690444, "filter_size": 6743183, "raw_key_size": 76215848, "raw_average_key_size": 28, "raw_value_size": 31365751, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:04:22.376263 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 28] Generated table #37026: 2638241 keys, 74546120 bytes >2018-07-11 08:04:22.376311 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296262376295, "cf_name": "default", "job": 28, "event": "table_file_creation", "file_number": 37026, "file_size": 74546120, "table_properties": {"data_size": 67112169, "index_size": 689778, "filter_size": 6743183, "raw_key_size": 76215859, "raw_average_key_size": 28, "raw_value_size": 31365762, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:04:26.053836 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 28] Generated table #37027: 2638241 keys, 74546152 bytes >2018-07-11 08:04:26.053873 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296266053857, "cf_name": "default", "job": 28, "event": "table_file_creation", "file_number": 37027, "file_size": 74546152, "table_properties": {"data_size": 67112123, "index_size": 689856, "filter_size": 6743183, "raw_key_size": 76215859, "raw_average_key_size": 28, "raw_value_size": 31365762, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:04:30.491478 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 28] Generated table #37028: 2638241 keys, 74545909 bytes >2018-07-11 08:04:30.491514 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296270491498, "cf_name": "default", "job": 28, "event": "table_file_creation", "file_number": 37028, "file_size": 74545909, "table_properties": {"data_size": 67112145, "index_size": 689591, "filter_size": 6743183, "raw_key_size": 76215846, "raw_average_key_size": 28, "raw_value_size": 31365749, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:04:34.976278 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 28] Generated table #37029: 2638240 keys, 74547029 bytes >2018-07-11 08:04:34.976325 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296274976297, "cf_name": "default", "job": 28, "event": "table_file_creation", "file_number": 37029, "file_size": 74547029, "table_properties": {"data_size": 67112089, "index_size": 690769, "filter_size": 6743181, "raw_key_size": 76215830, "raw_average_key_size": 28, "raw_value_size": 31365750, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:04:40.482366 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 28] Generated table #37030: 2638241 keys, 74546353 bytes >2018-07-11 08:04:40.482415 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296280482399, "cf_name": "default", "job": 28, "event": "table_file_creation", "file_number": 37030, "file_size": 74546353, "table_properties": {"data_size": 67112158, "index_size": 690022, "filter_size": 6743183, "raw_key_size": 76215855, "raw_average_key_size": 28, "raw_value_size": 31365758, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:04:44.804902 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 28] Generated table #37031: 2638241 keys, 74546467 bytes >2018-07-11 08:04:44.804954 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296284804937, "cf_name": "default", "job": 28, "event": "table_file_creation", "file_number": 37031, "file_size": 74546467, "table_properties": {"data_size": 67112104, "index_size": 690190, "filter_size": 6743183, "raw_key_size": 76215852, "raw_average_key_size": 28, "raw_value_size": 31365755, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:04:50.523016 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 28] Generated table #37032: 2638241 keys, 74546192 bytes >2018-07-11 08:04:50.523060 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296290523033, "cf_name": "default", "job": 28, "event": "table_file_creation", "file_number": 37032, "file_size": 74546192, "table_properties": {"data_size": 67112163, "index_size": 689855, "filter_size": 6743184, "raw_key_size": 76215867, "raw_average_key_size": 28, "raw_value_size": 31365770, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:04:50.701113 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 28] Generated table #37033: 43745 keys, 1236800 bytes >2018-07-11 08:04:50.701150 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296290701126, "cf_name": "default", "job": 28, "event": "table_file_creation", "file_number": 37033, "file_size": 1236800, "table_properties": {"data_size": 1112818, "index_size": 11188, "filter_size": 111814, "raw_key_size": 1263741, "raw_average_key_size": 28, "raw_value_size": 520076, "raw_average_value_size": 11, "num_data_blocks": 274, "num_entries": 43745, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:04:50.701179 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 28] Compacted 1@3 + 13@4 files to L4 => 897594417 bytes >2018-07-11 08:04:50.724433 7f4922851700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 30558006446 >2018-07-11 08:04:50.724920 7f4922851700 4 rocksdb: (Original Log Time 2018/07/11-08:04:50.724392) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[1 7 48 482 3641 796 0] max score 1.13, MB/sec: 20.9 rd, 17.9 wr, level 4, files in(1, 13) out(13) MB in(76.6, 924.2) out(856.0), read-write-amplify(24.2) write-amplify(11.2) OK, records in: 39254297, records dropped: 6830486 > >2018-07-11 08:04:50.724930 7f4922851700 4 rocksdb: (Original Log Time 2018/07/11-08:04:50.724426) EVENT_LOG_v1 {"time_micros": 1531296290724406, "job": 28, "event": "compaction_finished", "compaction_time_micros": 50166692, "output_level": 4, "num_output_files": 13, "total_output_size": 897594417, "num_input_records": 39254297, "num_output_records": 32423811, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 7, 48, 482, 3641, 796, 0]} >2018-07-11 08:04:50.725530 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296290725513, "job": 28, "event": "table_file_deletion", "file_number": 36632} >2018-07-11 08:04:50.725545 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296290725543, "job": 28, "event": "table_file_deletion", "file_number": 28122} >2018-07-11 08:04:50.725556 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296290725554, "job": 28, "event": "table_file_deletion", "file_number": 28120} >2018-07-11 08:04:50.725566 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296290725564, "job": 28, "event": "table_file_deletion", "file_number": 28119} >2018-07-11 08:04:50.725575 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296290725573, "job": 28, "event": "table_file_deletion", "file_number": 28118} >2018-07-11 08:04:50.725600 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296290725598, "job": 28, "event": "table_file_deletion", "file_number": 26363} >2018-07-11 08:04:50.725612 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296290725610, "job": 28, "event": "table_file_deletion", "file_number": 26362} >2018-07-11 08:04:50.725622 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296290725620, "job": 28, "event": "table_file_deletion", "file_number": 26361} >2018-07-11 08:04:50.725633 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296290725631, "job": 28, "event": "table_file_deletion", "file_number": 26360} >2018-07-11 08:04:50.725643 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296290725641, "job": 28, "event": "table_file_deletion", "file_number": 26358} >2018-07-11 08:04:50.725653 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296290725651, "job": 28, "event": "table_file_deletion", "file_number": 26357} >2018-07-11 08:04:50.725663 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296290725661, "job": 28, "event": "table_file_deletion", "file_number": 26356} >2018-07-11 08:04:50.725675 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296290725673, "job": 28, "event": "table_file_deletion", "file_number": 26355} >2018-07-11 08:04:50.725690 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296290725687, "job": 28, "event": "table_file_deletion", "file_number": 26354} >2018-07-11 08:04:50.726113 7f4922851700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS ------- >2018-07-11 08:04:50.726119 7f4922851700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] >** DB Stats ** >Uptime(secs): 695.4 total, 626.9 interval >Cumulative writes: 624 writes, 930 keys, 624 commit groups, 1.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s >Cumulative WAL: 624 writes, 136 syncs, 4.55 writes per sync, written: 0.00 GB, 0.00 MB/s >Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent >Interval writes: 624 writes, 930 keys, 624 commit groups, 1.0 writes per commit group, ingest: 0.52 MB, 0.00 MB/s >Interval WAL: 624 writes, 136 syncs, 4.55 writes per sync, written: 0.00 MB, 0.00 MB/s >Interval stall: 00:00:0.000 H:M:S, 0.0 percent > >** Compaction Stats [default] ** >Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop >---------------------------------------------------------------------------------------------------------------------------------------------------------- > L0 1/0 218.75 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 10.9 0 1 0.020 0 0 > L1 7/0 240.38 MB 1.1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 > L2 48/0 2.67 GB 1.1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 > L3 482/0 27.19 GB 1.1 11.2 1.5 9.7 10.7 1.0 0.1 7.1 21.6 20.8 529 22 24.032 460M 22M > L4 3641/0 249.88 GB 1.0 2.0 0.1 1.9 1.8 -0.1 0.0 11.8 21.2 18.4 98 2 48.884 81M 12M > L5 796/0 55.24 GB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 > Sum 4975/0 335.21 GB 0.0 13.2 1.7 11.5 12.5 0.9 0.1 59806.5 21.6 20.4 626 25 25.060 541M 34M > Int 0/0 0.00 KB 0.0 13.2 1.7 11.5 12.5 0.9 0.1 13396727908.0 21.6 20.4 626 24 26.103 541M 34M >Uptime(secs): 695.4 total, 695.4 interval >Flush(GB): cumulative 0.000, interval 0.000 >AddFile(GB): cumulative 0.000, interval 0.000 >AddFile(Total Files): cumulative 0, interval 0 >AddFile(L0 Files): cumulative 0, interval 0 >AddFile(Keys): cumulative 0, interval 0 >Cumulative compaction: 12.48 GB write, 18.37 MB/s write, 13.20 GB read, 19.44 MB/s read, 626.5 seconds >Interval compaction: 12.48 GB write, 18.37 MB/s write, 13.20 GB read, 19.44 MB/s read, 626.5 seconds >Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count > >** File Read Latency Histogram By Level [default] ** > >** Compaction Stats [default] ** >Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop >---------------------------------------------------------------------------------------------------------------------------------------------------------- > L0 1/0 218.75 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 10.9 0 1 0.020 0 0 > L1 7/0 240.38 MB 1.1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 > L2 48/0 2.67 GB 1.1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 > L3 482/0 27.19 GB 1.1 11.2 1.5 9.7 10.7 1.0 0.1 7.1 21.6 20.8 529 22 24.032 460M 22M > L4 3641/0 249.88 GB 1.0 2.0 0.1 1.9 1.8 -0.1 0.0 11.8 21.2 18.4 98 2 48.884 81M 12M > L5 796/0 55.24 GB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 > Sum 4975/0 335.21 GB 0.0 13.2 1.7 11.5 12.5 0.9 0.1 59806.5 21.6 20.4 626 25 25.060 541M 34M > Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 >Uptime(secs): 695.4 total, 695.4 interval >Flush(GB): cumulative 0.000, interval 0.000 >AddFile(GB): cumulative 0.000, interval 0.000 >AddFile(Total Files): cumulative 0, interval 0 >AddFile(L0 Files): cumulative 0, interval 0 >AddFile(Keys): cumulative 0, interval 0 >Cumulative compaction: 12.48 GB write, 18.37 MB/s write, 13.20 GB read, 19.44 MB/s read, 626.5 seconds >Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds >Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count > >** File Read Latency Histogram By Level [default] ** > >2018-07-11 08:04:50.726156 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_picker.cc:461] [default] Expanding@3 1+11(159662894+820009132 bytes) to 2+11 (193327614+820009132bytes) > >2018-07-11 08:04:50.726236 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 29] Compacting 2@3 + 11@4 files to L4, score 1.13 >2018-07-11 08:04:50.726272 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 28 Base level 3, inputs: [36785(32MB) 36581(76MB)], [27361(71MB) 27363(71MB) 27364(71MB) 27365(71MB) 27368(71MB) 27369(71MB) 26980(71MB) 26981(71MB) 26983(71MB) 26984(71MB) 26985(71MB)] > >2018-07-11 08:04:50.726287 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296290726276, "job": 29, "event": "compaction_started", "files_L3": [36785, 36581], "files_L4": [27361, 27363, 27364, 27365, 27368, 27369, 26980, 26981, 26983, 26984, 26985], "score": 1.13023, "input_data_size": 934022106} >2018-07-11 08:04:56.443258 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 29] Generated table #37034: 2638241 keys, 74546335 bytes >2018-07-11 08:04:56.443303 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296296443276, "cf_name": "default", "job": 29, "event": "table_file_creation", "file_number": 37034, "file_size": 74546335, "table_properties": {"data_size": 67112182, "index_size": 689980, "filter_size": 6743183, "raw_key_size": 76215861, "raw_average_key_size": 28, "raw_value_size": 31365764, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:05:00.904459 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 29] Generated table #37035: 2638240 keys, 74545896 bytes >2018-07-11 08:05:00.904505 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296300904489, "cf_name": "default", "job": 29, "event": "table_file_creation", "file_number": 37035, "file_size": 74545896, "table_properties": {"data_size": 67112133, "index_size": 689592, "filter_size": 6743181, "raw_key_size": 76215820, "raw_average_key_size": 28, "raw_value_size": 31365740, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:05:05.535774 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 29] Generated table #37036: 2638241 keys, 74546667 bytes >2018-07-11 08:05:05.535821 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296305535806, "cf_name": "default", "job": 29, "event": "table_file_creation", "file_number": 37036, "file_size": 74546667, "table_properties": {"data_size": 67112118, "index_size": 690376, "filter_size": 6743183, "raw_key_size": 76215855, "raw_average_key_size": 28, "raw_value_size": 31365758, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:05:12.766163 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 29] Generated table #37037: 2638241 keys, 74546650 bytes >2018-07-11 08:05:12.766211 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296312766195, "cf_name": "default", "job": 29, "event": "table_file_creation", "file_number": 37037, "file_size": 74546650, "table_properties": {"data_size": 67112152, "index_size": 690325, "filter_size": 6743183, "raw_key_size": 76215857, "raw_average_key_size": 28, "raw_value_size": 31365760, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:05:17.434897 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 29] Generated table #37038: 2638242 keys, 74546174 bytes >2018-07-11 08:05:17.434944 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296317434928, "cf_name": "default", "job": 29, "event": "table_file_creation", "file_number": 37038, "file_size": 74546174, "table_properties": {"data_size": 67112168, "index_size": 689830, "filter_size": 6743186, "raw_key_size": 76215884, "raw_average_key_size": 28, "raw_value_size": 31365770, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638242, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:05:23.426921 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 29] Generated table #37039: 2638241 keys, 74545999 bytes >2018-07-11 08:05:23.426958 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296323426942, "cf_name": "default", "job": 29, "event": "table_file_creation", "file_number": 37039, "file_size": 74545999, "table_properties": {"data_size": 67112153, "index_size": 689673, "filter_size": 6743183, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:05:28.010463 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 29] Generated table #37040: 2638240 keys, 74546953 bytes >2018-07-11 08:05:28.010499 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296328010483, "cf_name": "default", "job": 29, "event": "table_file_creation", "file_number": 37040, "file_size": 74546953, "table_properties": {"data_size": 67112080, "index_size": 690702, "filter_size": 6743181, "raw_key_size": 76215827, "raw_average_key_size": 28, "raw_value_size": 31365747, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:05:33.839222 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 29] Generated table #37041: 2638241 keys, 74546309 bytes >2018-07-11 08:05:33.839258 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296333839239, "cf_name": "default", "job": 29, "event": "table_file_creation", "file_number": 37041, "file_size": 74546309, "table_properties": {"data_size": 67112174, "index_size": 689962, "filter_size": 6743183, "raw_key_size": 76215851, "raw_average_key_size": 28, "raw_value_size": 31365754, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:05:38.534965 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 29] Generated table #37042: 2638241 keys, 74546437 bytes >2018-07-11 08:05:38.535013 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296338534997, "cf_name": "default", "job": 29, "event": "table_file_creation", "file_number": 37042, "file_size": 74546437, "table_properties": {"data_size": 67112133, "index_size": 690131, "filter_size": 6743183, "raw_key_size": 76215851, "raw_average_key_size": 28, "raw_value_size": 31365754, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:05:40.898069 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 29] Generated table #37043: 1067172 keys, 27570702 bytes >2018-07-11 08:05:40.898113 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296340898097, "cf_name": "default", "job": 29, "event": "table_file_creation", "file_number": 37043, "file_size": 27570702, "table_properties": {"data_size": 24595035, "index_size": 252308, "filter_size": 2722369, "raw_key_size": 30829425, "raw_average_key_size": 28, "raw_value_size": 10138026, "raw_average_value_size": 9, "num_data_blocks": 6042, "num_entries": 1067172, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "214442", "kMergeOperands": "0"}} >2018-07-11 08:05:40.898147 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 29] Compacted 2@3 + 11@4 files to L4 => 698488122 bytes >2018-07-11 08:05:40.907769 7f4922851700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 29184525102 >2018-07-11 08:05:40.908006 7f4922851700 4 rocksdb: (Original Log Time 2018/07/11-08:05:40.907736) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[1 7 48 480 3640 796 0] max score 1.12, MB/sec: 18.6 rd, 13.9 wr, level 4, files in(2, 11) out(10) MB in(108.7, 782.0) out(666.1), read-write-amplify(14.3) write-amplify(6.1) OK, records in: 35405602, records dropped: 10594262 > >2018-07-11 08:05:40.908015 7f4922851700 4 rocksdb: (Original Log Time 2018/07/11-08:05:40.907761) EVENT_LOG_v1 {"time_micros": 1531296340907750, "job": 29, "event": "compaction_finished", "compaction_time_micros": 50171842, "output_level": 4, "num_output_files": 10, "total_output_size": 698488122, "num_input_records": 35405602, "num_output_records": 24811340, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 7, 48, 480, 3640, 796, 0]} >2018-07-11 08:05:40.908678 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296340908662, "job": 29, "event": "table_file_deletion", "file_number": 36785} >2018-07-11 08:05:40.908698 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296340908695, "job": 29, "event": "table_file_deletion", "file_number": 36581} >2018-07-11 08:05:40.908712 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296340908709, "job": 29, "event": "table_file_deletion", "file_number": 27369} >2018-07-11 08:05:40.908725 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296340908721, "job": 29, "event": "table_file_deletion", "file_number": 27368} >2018-07-11 08:05:40.908746 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296340908743, "job": 29, "event": "table_file_deletion", "file_number": 27365} >2018-07-11 08:05:40.908760 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296340908757, "job": 29, "event": "table_file_deletion", "file_number": 27364} >2018-07-11 08:05:40.908772 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296340908770, "job": 29, "event": "table_file_deletion", "file_number": 27363} >2018-07-11 08:05:40.908785 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296340908783, "job": 29, "event": "table_file_deletion", "file_number": 27361} >2018-07-11 08:05:40.908800 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296340908797, "job": 29, "event": "table_file_deletion", "file_number": 26985} >2018-07-11 08:05:40.908813 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296340908811, "job": 29, "event": "table_file_deletion", "file_number": 26984} >2018-07-11 08:05:40.908826 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296340908824, "job": 29, "event": "table_file_deletion", "file_number": 26983} >2018-07-11 08:05:40.908839 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296340908837, "job": 29, "event": "table_file_deletion", "file_number": 26981} >2018-07-11 08:05:40.908853 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296340908850, "job": 29, "event": "table_file_deletion", "file_number": 26980} >2018-07-11 08:05:40.909047 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 30] Compacting 1@3 + 18@4 files to L4, score 1.12 >2018-07-11 08:05:40.909064 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 29 Base level 3, inputs: [36637(76MB)], [25818(71MB) 26341(71MB) 26342(71MB) 26343(71MB) 26347(71MB) 26348(71MB) 26349(71MB) 26350(71MB) 26351(71MB) 26352(71MB) 26292(71MB) 29948(71MB) 29949(71MB) 29950(71MB) 26002(71MB) 26003(71MB) 26004(71MB) 33964(71MB)] > >2018-07-11 08:05:40.909082 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296340909070, "job": 30, "event": "compaction_started", "files_L3": [36637], "files_L4": [25818, 26341, 26342, 26343, 26347, 26348, 26349, 26350, 26351, 26352, 26292, 29948, 29949, 29950, 26002, 26003, 26004, 33964], "score": 1.12303, "input_data_size": 1422044065} >2018-07-11 08:05:47.980094 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 30] Generated table #37044: 2638241 keys, 74546713 bytes >2018-07-11 08:05:47.980142 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296347980126, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 37044, "file_size": 74546713, "table_properties": {"data_size": 67112134, "index_size": 690406, "filter_size": 6743183, "raw_key_size": 76215836, "raw_average_key_size": 28, "raw_value_size": 31365739, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:05:52.477389 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 30] Generated table #37045: 2638241 keys, 74546342 bytes >2018-07-11 08:05:52.477424 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296352477408, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 37045, "file_size": 74546342, "table_properties": {"data_size": 67112125, "index_size": 690044, "filter_size": 6743183, "raw_key_size": 76215857, "raw_average_key_size": 28, "raw_value_size": 31365760, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:05:57.177065 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 30] Generated table #37046: 2638241 keys, 74546279 bytes >2018-07-11 08:05:57.177099 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296357177084, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 37046, "file_size": 74546279, "table_properties": {"data_size": 67112151, "index_size": 689955, "filter_size": 6743183, "raw_key_size": 76215853, "raw_average_key_size": 28, "raw_value_size": 31365756, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:06:01.616068 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 30] Generated table #37047: 2638240 keys, 74546053 bytes >2018-07-11 08:06:01.616105 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296361616089, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 37047, "file_size": 74546053, "table_properties": {"data_size": 67112078, "index_size": 689804, "filter_size": 6743181, "raw_key_size": 76215827, "raw_average_key_size": 28, "raw_value_size": 31365747, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:06:05.826344 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 30] Generated table #37048: 2638241 keys, 74547069 bytes >2018-07-11 08:06:05.826394 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296365826378, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 37048, "file_size": 74547069, "table_properties": {"data_size": 67112177, "index_size": 690719, "filter_size": 6743183, "raw_key_size": 76215860, "raw_average_key_size": 28, "raw_value_size": 31365763, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:06:10.368183 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 30] Generated table #37049: 2638241 keys, 74546099 bytes >2018-07-11 08:06:10.368238 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296370368221, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 37049, "file_size": 74546099, "table_properties": {"data_size": 67112109, "index_size": 689817, "filter_size": 6743183, "raw_key_size": 76215847, "raw_average_key_size": 28, "raw_value_size": 31365750, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:06:14.769019 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 30] Generated table #37050: 2638241 keys, 74546380 bytes >2018-07-11 08:06:14.769066 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296374769051, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 37050, "file_size": 74546380, "table_properties": {"data_size": 67112167, "index_size": 690040, "filter_size": 6743183, "raw_key_size": 76215859, "raw_average_key_size": 28, "raw_value_size": 31365762, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:06:19.396372 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 30] Generated table #37051: 2638241 keys, 74545767 bytes >2018-07-11 08:06:19.396412 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296379396392, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 37051, "file_size": 74545767, "table_properties": {"data_size": 67112127, "index_size": 689467, "filter_size": 6743183, "raw_key_size": 76215846, "raw_average_key_size": 28, "raw_value_size": 31365749, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:06:23.806568 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 30] Generated table #37052: 2638241 keys, 74546882 bytes >2018-07-11 08:06:23.806617 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296383806601, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 37052, "file_size": 74546882, "table_properties": {"data_size": 67112148, "index_size": 690561, "filter_size": 6743183, "raw_key_size": 76215859, "raw_average_key_size": 28, "raw_value_size": 31365762, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:06:28.573619 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 30] Generated table #37053: 2638242 keys, 74546411 bytes >2018-07-11 08:06:28.573664 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296388573637, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 37053, "file_size": 74546411, "table_properties": {"data_size": 67112161, "index_size": 690074, "filter_size": 6743186, "raw_key_size": 76215884, "raw_average_key_size": 28, "raw_value_size": 31365770, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638242, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:06:34.459789 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 30] Generated table #37054: 2638241 keys, 74546128 bytes >2018-07-11 08:06:34.459835 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296394459808, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 37054, "file_size": 74546128, "table_properties": {"data_size": 67112142, "index_size": 689813, "filter_size": 6743183, "raw_key_size": 76215851, "raw_average_key_size": 28, "raw_value_size": 31365754, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:06:39.108409 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 30] Generated table #37055: 2638241 keys, 74546305 bytes >2018-07-11 08:06:39.108454 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296399108427, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 37055, "file_size": 74546305, "table_properties": {"data_size": 67112095, "index_size": 690037, "filter_size": 6743183, "raw_key_size": 76215855, "raw_average_key_size": 28, "raw_value_size": 31365758, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:06:45.106383 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 30] Generated table #37056: 2638241 keys, 74546613 bytes >2018-07-11 08:06:45.106432 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296405106415, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 37056, "file_size": 74546613, "table_properties": {"data_size": 67112163, "index_size": 690277, "filter_size": 6743183, "raw_key_size": 76215850, "raw_average_key_size": 28, "raw_value_size": 31365753, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:06:49.765513 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 30] Generated table #37057: 2638241 keys, 74546276 bytes >2018-07-11 08:06:49.765549 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296409765534, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 37057, "file_size": 74546276, "table_properties": {"data_size": 67112146, "index_size": 689957, "filter_size": 6743183, "raw_key_size": 76215856, "raw_average_key_size": 28, "raw_value_size": 31365759, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:06:55.134533 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 30] Generated table #37058: 2638241 keys, 74545740 bytes >2018-07-11 08:06:55.134586 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296415134570, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 37058, "file_size": 74545740, "table_properties": {"data_size": 67112137, "index_size": 689430, "filter_size": 6743183, "raw_key_size": 76215851, "raw_average_key_size": 28, "raw_value_size": 31365754, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:06:59.647016 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 30] Generated table #37059: 2638239 keys, 74546655 bytes >2018-07-11 08:06:59.647068 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296419647051, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 37059, "file_size": 74546655, "table_properties": {"data_size": 67112091, "index_size": 690395, "filter_size": 6743179, "raw_key_size": 76215793, "raw_average_key_size": 28, "raw_value_size": 31365730, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638239, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:07:00.543675 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 30] Generated table #37060: 503196 keys, 14218849 bytes >2018-07-11 08:07:00.543718 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296420543691, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 37060, "file_size": 14218849, "table_properties": {"data_size": 12800421, "index_size": 131302, "filter_size": 1286140, "raw_key_size": 14536772, "raw_average_key_size": 28, "raw_value_size": 5982440, "raw_average_value_size": 11, "num_data_blocks": 3145, "num_entries": 503196, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:07:00.543758 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 30] Compacted 1@3 + 18@4 files to L4 => 1206960561 bytes >2018-07-11 08:07:00.555950 7f4922851700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 28150165975 >2018-07-11 08:07:00.556167 7f4922851700 4 rocksdb: (Original Log Time 2018/07/11-08:07:00.555916) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[1 7 48 479 3639 796 0] max score 1.12, MB/sec: 17.9 rd, 15.2 wr, level 4, files in(1, 18) out(17) MB in(76.5, 1279.7) out(1151.0), read-write-amplify(32.8) write-amplify(15.0) OK, records in: 52389470, records dropped: 9674420 > >2018-07-11 08:07:00.556176 7f4922851700 4 rocksdb: (Original Log Time 2018/07/11-08:07:00.555942) EVENT_LOG_v1 {"time_micros": 1531296420555930, "job": 30, "event": "compaction_finished", "compaction_time_micros": 79634650, "output_level": 4, "num_output_files": 17, "total_output_size": 1206960561, "num_input_records": 52389470, "num_output_records": 42715050, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 7, 48, 479, 3639, 796, 0]} >2018-07-11 08:07:00.556827 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296420556810, "job": 30, "event": "table_file_deletion", "file_number": 36637} >2018-07-11 08:07:00.556846 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296420556843, "job": 30, "event": "table_file_deletion", "file_number": 33964} >2018-07-11 08:07:00.556859 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296420556857, "job": 30, "event": "table_file_deletion", "file_number": 29950} >2018-07-11 08:07:00.556871 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296420556869, "job": 30, "event": "table_file_deletion", "file_number": 29949} >2018-07-11 08:07:00.556882 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296420556880, "job": 30, "event": "table_file_deletion", "file_number": 29948} >2018-07-11 08:07:00.556902 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296420556898, "job": 30, "event": "table_file_deletion", "file_number": 26352} >2018-07-11 08:07:00.556916 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296420556913, "job": 30, "event": "table_file_deletion", "file_number": 26351} >2018-07-11 08:07:00.556929 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296420556927, "job": 30, "event": "table_file_deletion", "file_number": 26350} >2018-07-11 08:07:00.556941 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296420556938, "job": 30, "event": "table_file_deletion", "file_number": 26349} >2018-07-11 08:07:00.556952 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296420556949, "job": 30, "event": "table_file_deletion", "file_number": 26348} >2018-07-11 08:07:00.556966 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296420556964, "job": 30, "event": "table_file_deletion", "file_number": 26347} >2018-07-11 08:07:00.556979 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296420556977, "job": 30, "event": "table_file_deletion", "file_number": 26343} >2018-07-11 08:07:00.556994 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296420556991, "job": 30, "event": "table_file_deletion", "file_number": 26342} >2018-07-11 08:07:00.557008 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296420557006, "job": 30, "event": "table_file_deletion", "file_number": 26341} >2018-07-11 08:07:00.557022 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296420557019, "job": 30, "event": "table_file_deletion", "file_number": 26292} >2018-07-11 08:07:00.557036 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296420557033, "job": 30, "event": "table_file_deletion", "file_number": 26004} >2018-07-11 08:07:00.557049 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296420557047, "job": 30, "event": "table_file_deletion", "file_number": 26003} >2018-07-11 08:07:00.557062 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296420557060, "job": 30, "event": "table_file_deletion", "file_number": 26002} >2018-07-11 08:07:00.557076 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296420557074, "job": 30, "event": "table_file_deletion", "file_number": 25818} >2018-07-11 08:07:00.557188 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_picker.cc:461] [default] Expanding@3 1+15(153984504+1043647686 bytes) to 2+15 (198992290+1043647686bytes) > >2018-07-11 08:07:00.557284 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 31] Compacting 2@3 + 15@4 files to L4, score 1.12 >2018-07-11 08:07:00.557300 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 30 Base level 3, inputs: [36833(21MB) 36786(76MB)], [31594(71MB) 31595(71MB) 32734(71MB) 32735(71MB) 32736(71MB) 32737(71MB) 32738(71MB) 31740(71MB) 31741(71MB) 31743(1064B) 31459(71MB) 31460(71MB) 31462(71MB) 29382(71MB) 30309(71MB)] > >2018-07-11 08:07:00.557317 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296420557306, "job": 31, "event": "compaction_started", "files_L3": [36833, 36786], "files_L4": [31594, 31595, 32734, 32735, 32736, 32737, 32738, 31740, 31741, 31743, 31459, 31460, 31462, 29382, 30309], "score": 1.11719, "input_data_size": 1146387448} >2018-07-11 08:07:07.356422 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 31] Generated table #37061: 2939958 keys, 75300056 bytes >2018-07-11 08:07:07.356461 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296427356445, "cf_name": "default", "job": 31, "event": "table_file_creation", "file_number": 37061, "file_size": 75300056, "table_properties": {"data_size": 67110142, "index_size": 690382, "filter_size": 7498540, "raw_key_size": 84932103, "raw_average_key_size": 28, "raw_value_size": 27283102, "raw_average_value_size": 9, "num_data_blocks": 16487, "num_entries": 2939958, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "645117", "kMergeOperands": "0"}} >2018-07-11 08:07:11.839273 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 31] Generated table #37062: 2638241 keys, 74546451 bytes >2018-07-11 08:07:11.839306 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296431839291, "cf_name": "default", "job": 31, "event": "table_file_creation", "file_number": 37062, "file_size": 74546451, "table_properties": {"data_size": 67112168, "index_size": 690110, "filter_size": 6743183, "raw_key_size": 76215859, "raw_average_key_size": 28, "raw_value_size": 31365762, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:07:17.768800 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 31] Generated table #37063: 2638241 keys, 74546343 bytes >2018-07-11 08:07:17.768849 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296437768833, "cf_name": "default", "job": 31, "event": "table_file_creation", "file_number": 37063, "file_size": 74546343, "table_properties": {"data_size": 67112098, "index_size": 690072, "filter_size": 6743183, "raw_key_size": 76215848, "raw_average_key_size": 28, "raw_value_size": 31365751, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:07:22.223528 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 31] Generated table #37064: 2638241 keys, 74546026 bytes >2018-07-11 08:07:22.223578 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296442223563, "cf_name": "default", "job": 31, "event": "table_file_creation", "file_number": 37064, "file_size": 74546026, "table_properties": {"data_size": 67112160, "index_size": 689693, "filter_size": 6743183, "raw_key_size": 76215857, "raw_average_key_size": 28, "raw_value_size": 31365760, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:07:27.295557 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 31] Generated table #37065: 2638241 keys, 74546407 bytes >2018-07-11 08:07:27.295593 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296447295578, "cf_name": "default", "job": 31, "event": "table_file_creation", "file_number": 37065, "file_size": 74546407, "table_properties": {"data_size": 67112105, "index_size": 690129, "filter_size": 6743183, "raw_key_size": 76215852, "raw_average_key_size": 28, "raw_value_size": 31365755, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:07:33.475139 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 31] Generated table #37066: 2638241 keys, 74546796 bytes >2018-07-11 08:07:33.475172 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296453475156, "cf_name": "default", "job": 31, "event": "table_file_creation", "file_number": 37066, "file_size": 74546796, "table_properties": {"data_size": 67112181, "index_size": 690442, "filter_size": 6743183, "raw_key_size": 76215841, "raw_average_key_size": 28, "raw_value_size": 31365744, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} >2018-07-11 08:07:36.936574 7f4922851700 -1 bluefs _allocate failed to allocate 0x4800000 on bdev 2, free 0x4400000 >2018-07-11 08:07:36.936588 7f4922851700 0 stupidalloc 0x0x55f64c9a8000 dump free bin 0: 0 extents >2018-07-11 08:07:36.936605 7f4922851700 0 stupidalloc 0x0x55f64c9a8000 dump free bin 1: 0 extents >2018-07-11 08:07:36.936606 7f4922851700 0 stupidalloc 0x0x55f64c9a8000 dump free bin 2: 0 extents >2018-07-11 08:07:36.936607 7f4922851700 0 stupidalloc 0x0x55f64c9a8000 dump free bin 3: 0 extents >2018-07-11 08:07:36.936608 7f4922851700 0 stupidalloc 0x0x55f64c9a8000 dump free bin 4: 0 extents >2018-07-11 08:07:36.936609 7f4922851700 0 stupidalloc 0x0x55f64c9a8000 dump free bin 5: 0 extents >2018-07-11 08:07:36.936610 7f4922851700 0 stupidalloc 0x0x55f64c9a8000 dump free bin 6: 0 extents >2018-07-11 08:07:36.936611 7f4922851700 0 stupidalloc 0x0x55f64c9a8000 dump free bin 7: 0 extents >2018-07-11 08:07:36.936612 7f4922851700 0 stupidalloc 0x0x55f64c9a8000 dump free bin 8: 0 extents >2018-07-11 08:07:36.936613 7f4922851700 0 stupidalloc 0x0x55f64c9a8000 dump free bin 9: 20 extents >2018-07-11 08:07:36.936614 7f4922851700 0 stupidalloc 0x0x55f64c9a8000 dump 0x26fcc00000~100000 >2018-07-11 08:07:36.936616 7f4922851700 0 stupidalloc 0x0x55f64c9a8000 dump 0x26ff300000~100000 >2018-07-11 08:07:36.936617 7f4922851700 0 stupidalloc 0x0x55f64c9a8000 dump 0x342e700000~a00000 >2018-07-11 08:07:36.936617 7f4922851700 0 stupidalloc 0x0x55f64c9a8000 dump 0x374f000000~400000 >2018-07-11 08:07:36.936618 7f4922851700 0 stupidalloc 0x0x55f64c9a8000 dump 0x3771a00000~100000 >2018-07-11 08:07:36.936619 7f4922851700 0 stupidalloc 0x0x55f64c9a8000 dump 0x785b300000~100000 >2018-07-11 08:07:36.936620 7f4922851700 0 stupidalloc 0x0x55f64c9a8000 dump 0x789ea00000~600000 >2018-07-11 08:07:36.936621 7f4922851700 0 stupidalloc 0x0x55f64c9a8000 dump 0x8d78600000~100000 >2018-07-11 08:07:36.936623 7f4922851700 0 stupidalloc 0x0x55f64c9a8000 dump 0x950fe00000~700000 >2018-07-11 08:07:36.936624 7f4922851700 0 stupidalloc 0x0x55f64c9a8000 dump 0x951a300000~600000 >2018-07-11 08:07:36.936625 7f4922851700 0 stupidalloc 0x0x55f64c9a8000 dump 0x9552800000~100000 >2018-07-11 08:07:36.936626 7f4922851700 0 stupidalloc 0x0x55f64c9a8000 dump 0x9791300000~900000 >2018-07-11 08:07:36.936627 7f4922851700 0 stupidalloc 0x0x55f64c9a8000 dump 0x9ab9900000~100000 >2018-07-11 08:07:36.936628 7f4922851700 0 stupidalloc 0x0x55f64c9a8000 dump 0x9b13b00000~500000 >2018-07-11 08:07:36.936628 7f4922851700 0 stupidalloc 0x0x55f64c9a8000 dump 0x9b94300000~200000 >2018-07-11 08:07:36.936629 7f4922851700 0 stupidalloc 0x0x55f64c9a8000 dump 0x9bde300000~100000 >2018-07-11 08:07:36.936630 7f4922851700 0 stupidalloc 0x0x55f64c9a8000 dump 0x9bf4c00000~100000 >2018-07-11 08:07:36.936631 7f4922851700 0 stupidalloc 0x0x55f64c9a8000 dump 0xa02b800000~800000 >2018-07-11 08:07:36.936633 7f4922851700 0 stupidalloc 0x0x55f64c9a8000 dump 0xa047000000~100000 >2018-07-11 08:07:36.936634 7f4922851700 0 stupidalloc 0x0x55f64c9a8000 dump 0xa19e000000~100000 >2018-07-11 08:07:36.936635 7f4922851700 -1 bluefs _flush_range allocated: 0x0 offset: 0x0 length: 0x4717c35 >2018-07-11 08:07:36.942503 7f4922851700 -1 /builddir/build/BUILD/ceph-12.2.5/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, uint64_t)' thread 7f4922851700 time 2018-07-11 08:07:36.936648 >/builddir/build/BUILD/ceph-12.2.5/src/os/bluestore/BlueFS.cc: 1525: FAILED assert(0 == "bluefs enospc") > > ceph version 12.2.5-27.el7cp (6b7d1a8cf2e2376fd0a12424947c8c97aa1c0890) luminous (stable) > 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x110) [0x55f642fae710] > 2: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1b1b) [0x55f642f2d13b] > 3: (BlueFS::_flush(BlueFS::FileWriter*, bool)+0x188) [0x55f642f2d6a8] > 4: (BlueRocksWritableFile::Flush()+0x3c) [0x55f642f4923c] > 5: (rocksdb::WritableFileWriter::Flush()+0xaf) [0x55f64331855f] > 6: (rocksdb::WritableFileWriter::Sync(bool)+0x33) [0x55f643318773] > 7: (rocksdb::CompactionJob::FinishCompactionOutputFile(rocksdb::Status const&, rocksdb::CompactionJob::SubcompactionState*, rocksdb::RangeDelAggregator*, CompactionIterationStats*, rocksdb::Slice const*)+0xa1b) [0x55f6433de7ab] > 8: (rocksdb::CompactionJob::ProcessKeyValueCompaction(rocksdb::CompactionJob::SubcompactionState*)+0xea2) [0x55f6433e1222] > 9: (rocksdb::CompactionJob::Run()+0x1de) [0x55f6433e1cde] > 10: (rocksdb::DBImpl::BackgroundCompaction(bool*, rocksdb::JobContext*, rocksdb::LogBuffer*, void*)+0x9c7) [0x55f64336d687] > 11: (rocksdb::DBImpl::BackgroundCallCompaction(void*)+0x283) [0x55f643370c73] > 12: (rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long)+0x271) [0x55f6433d7e31] > 13: (rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*)+0x3d) [0x55f6433d801d] > 14: (()+0xb5070) [0x7f493429f070] > 15: (()+0x7dd5) [0x7f4934912dd5] > 16: (clone()+0x6d) [0x7f4933a03b3d] > NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. > >--- begin dump of recent events --- >-10000> 2018-07-11 08:04:16.564558 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.1 10.8.128.106:0/12677 174 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:16.620407) v4 ==== 2004+0+0 (3929460852 0 0) 0x55f64cc81800 con 0x55f663ed6000 > -9999> 2018-07-11 08:04:16.564570 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:16.620407) v4 -- 0x55f664256a00 con 0 > -9998> 2018-07-11 08:04:16.652785 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.114:6805/15885 conn(0x55f66315a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.7 seq 2493 0x55f64e20f800 MOSDScrubReserve(5.4 REQUEST e509) v1 > -9997> 2018-07-11 08:04:16.652810 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.7 10.8.128.114:6805/15885 2493 ==== MOSDScrubReserve(5.4 REQUEST e509) v1 ==== 43+0+0 (3003630469 0 0) 0x55f64e20f800 con 0x55f66315a000 > -9996> 2018-07-11 08:04:16.652901 7f491983f700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.114:6805/15885 -- MOSDScrubReserve(5.4 GRANT e509) v1 -- 0x55f663c0bc00 con 0 > -9995> 2018-07-11 08:04:16.653212 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.114:6805/15885 conn(0x55f66315a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.7 seq 2494 0x55f663c0bc00 MOSDScrubReserve(5.4 RELEASE e509) v1 > -9994> 2018-07-11 08:04:16.653232 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.7 10.8.128.114:6805/15885 2494 ==== MOSDScrubReserve(5.4 RELEASE e509) v1 ==== 43+0+0 (2082093447 0 0) 0x55f663c0bc00 con 0x55f66315a000 > -9993> 2018-07-11 08:04:16.666656 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15255 conn(0x55f663ec2800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.0 seq 170 0x55f650f78a00 osd_ping(ping e509 stamp 2018-07-11 08:04:17.910032) v4 > -9992> 2018-07-11 08:04:16.666660 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15255 conn(0x55f663ec1000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.0 seq 170 0x55f65299b600 osd_ping(ping e509 stamp 2018-07-11 08:04:17.910032) v4 > -9991> 2018-07-11 08:04:16.666684 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.0 10.8.128.113:0/15255 170 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:17.910032) v4 ==== 2004+0+0 (2729805695 0 0) 0x55f650f78a00 con 0x55f663ec2800 > -9990> 2018-07-11 08:04:16.666687 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.0 10.8.128.113:0/15255 170 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:17.910032) v4 ==== 2004+0+0 (2729805695 0 0) 0x55f65299b600 con 0x55f663ec1000 > -9989> 2018-07-11 08:04:16.666694 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:17.910032) v4 -- 0x55f653d4ac00 con 0 > -9988> 2018-07-11 08:04:16.666717 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:17.910032) v4 -- 0x55f66b1cce00 con 0 > -9987> 2018-07-11 08:04:16.789239 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.108:0/11967 conn(0x55f663f6f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=1). rx osd.8 seq 170 0x55f663fb9c00 osd_ping(ping e509 stamp 2018-07-11 08:04:16.548772) v4 > -9986> 2018-07-11 08:04:16.789273 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.8 10.8.128.108:0/11967 170 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:16.548772) v4 ==== 2004+0+0 (785860640 0 0) 0x55f663fb9c00 con 0x55f663f6f000 > -9985> 2018-07-11 08:04:16.789285 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:16.548772) v4 -- 0x55f653d4ac00 con 0 > -9984> 2018-07-11 08:04:16.789303 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.108:0/11967 conn(0x55f663f70800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=1). rx osd.8 seq 170 0x55f64cc81800 osd_ping(ping e509 stamp 2018-07-11 08:04:16.548772) v4 > -9983> 2018-07-11 08:04:16.789324 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.8 10.8.128.108:0/11967 170 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:16.548772) v4 ==== 2004+0+0 (785860640 0 0) 0x55f64cc81800 con 0x55f663f70800 > -9982> 2018-07-11 08:04:16.789336 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:16.548772) v4 -- 0x55f666c46c00 con 0 > -9981> 2018-07-11 08:04:16.973189 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.107:0/12617 conn(0x55f64e3c5000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.9 seq 177 0x55f664256a00 osd_ping(ping e509 stamp 2018-07-11 08:04:16.972642) v4 > -9980> 2018-07-11 08:04:16.973208 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.9 10.8.128.107:0/12617 177 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:16.972642) v4 ==== 2004+0+0 (395155429 0 0) 0x55f664256a00 con 0x55f64e3c5000 > -9979> 2018-07-11 08:04:16.973220 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:16.972642) v4 -- 0x55f666c46c00 con 0 > -9978> 2018-07-11 08:04:16.973331 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.107:0/12617 conn(0x55f64e3c3800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.9 seq 177 0x55f650f78a00 osd_ping(ping e509 stamp 2018-07-11 08:04:16.972642) v4 > -9977> 2018-07-11 08:04:16.973352 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.9 10.8.128.107:0/12617 177 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:16.972642) v4 ==== 2004+0+0 (395155429 0 0) 0x55f650f78a00 con 0x55f64e3c3800 > -9976> 2018-07-11 08:04:16.973367 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:16.972642) v4 -- 0x55f653d4ac00 con 0 > -9975> 2018-07-11 08:04:17.044216 7f4932031700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.114:6801/15919 conn(0x55f66315d000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=30 cs=1 l=0). rx osd.11 seq 99 0x55f653d3cc00 MOSDScrubReserve(10.f REQUEST e509) v1 > -9974> 2018-07-11 08:04:17.044235 7f4932031700 1 -- 10.8.128.107:6801/14173 <== osd.11 10.8.128.114:6801/15919 99 ==== MOSDScrubReserve(10.f REQUEST e509) v1 ==== 43+0+0 (2011750585 0 0) 0x55f653d3cc00 con 0x55f66315d000 > -9973> 2018-07-11 08:04:17.044290 7f491b843700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.114:6801/15919 -- MOSDScrubReserve(10.f GRANT e509) v1 -- 0x55f67d1afe00 con 0 > -9972> 2018-07-11 08:04:17.044695 7f4932031700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.114:6801/15919 conn(0x55f66315d000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=30 cs=1 l=0). rx osd.11 seq 100 0x55f67d1afe00 MOSDScrubReserve(10.f RELEASE e509) v1 > -9971> 2018-07-11 08:04:17.044710 7f4932031700 1 -- 10.8.128.107:6801/14173 <== osd.11 10.8.128.114:6801/15919 100 ==== MOSDScrubReserve(10.f RELEASE e509) v1 ==== 43+0+0 (3103070907 0 0) 0x55f67d1afe00 con 0x55f66315d000 > -9970> 2018-07-11 08:04:17.345277 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0e800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 179 0x55f65299b600 osd_ping(ping e509 stamp 2018-07-11 08:04:18.588635) v4 > -9969> 2018-07-11 08:04:17.345276 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0d000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 179 0x55f64cc81800 osd_ping(ping e509 stamp 2018-07-11 08:04:18.588635) v4 > -9968> 2018-07-11 08:04:17.345307 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.5 10.8.128.113:0/15263 179 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:18.588635) v4 ==== 2004+0+0 (4033097265 0 0) 0x55f65299b600 con 0x55f663f0e800 > -9967> 2018-07-11 08:04:17.345310 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.5 10.8.128.113:0/15263 179 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:18.588635) v4 ==== 2004+0+0 (4033097265 0 0) 0x55f64cc81800 con 0x55f663f0d000 > -9966> 2018-07-11 08:04:17.345328 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:18.588635) v4 -- 0x55f64cc80a00 con 0 > -9965> 2018-07-11 08:04:17.345360 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:18.588635) v4 -- 0x55f666c46c00 con 0 > -9964> 2018-07-11 08:04:17.521051 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15885 conn(0x55f663217800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 176 0x55f663fb9c00 osd_ping(ping e509 stamp 2018-07-11 08:04:18.156126) v4 > -9963> 2018-07-11 08:04:17.521074 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.7 10.8.128.114:0/15885 176 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:18.156126) v4 ==== 2004+0+0 (3742302621 0 0) 0x55f663fb9c00 con 0x55f663217800 > -9962> 2018-07-11 08:04:17.521082 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15885 conn(0x55f663216000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 176 0x55f66b1cce00 osd_ping(ping e509 stamp 2018-07-11 08:04:18.156126) v4 > -9961> 2018-07-11 08:04:17.521103 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.7 10.8.128.114:0/15885 176 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:18.156126) v4 ==== 2004+0+0 (3742302621 0 0) 0x55f66b1cce00 con 0x55f663216000 > -9960> 2018-07-11 08:04:17.521103 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:18.156126) v4 -- 0x55f653d4ac00 con 0 > -9959> 2018-07-11 08:04:17.521143 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:18.156126) v4 -- 0x55f64cc80a00 con 0 > -9958> 2018-07-11 08:04:17.767055 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15255 conn(0x55f663ec1000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.0 seq 171 0x55f65299b600 osd_ping(ping e509 stamp 2018-07-11 08:04:19.010489) v4 > -9957> 2018-07-11 08:04:17.767083 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.0 10.8.128.113:0/15255 171 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:19.010489) v4 ==== 2004+0+0 (1587781496 0 0) 0x55f65299b600 con 0x55f663ec1000 > -9956> 2018-07-11 08:04:17.767098 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:19.010489) v4 -- 0x55f64cc80a00 con 0 > -9955> 2018-07-11 08:04:17.767090 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15255 conn(0x55f663ec2800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.0 seq 171 0x55f650f78a00 osd_ping(ping e509 stamp 2018-07-11 08:04:19.010489) v4 > -9954> 2018-07-11 08:04:17.767114 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.0 10.8.128.113:0/15255 171 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:19.010489) v4 ==== 2004+0+0 (1587781496 0 0) 0x55f650f78a00 con 0x55f663ec2800 > -9953> 2018-07-11 08:04:17.767133 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:19.010489) v4 -- 0x55f653d4ac00 con 0 > -9952> 2018-07-11 08:04:18.292949 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0a000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 171 0x55f66b1cce00 osd_ping(ping e509 stamp 2018-07-11 08:04:18.927994) v4 > -9951> 2018-07-11 08:04:18.292976 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0b800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 171 0x55f663fb9c00 osd_ping(ping e509 stamp 2018-07-11 08:04:18.927994) v4 > -9950> 2018-07-11 08:04:18.292984 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.11 10.8.128.114:0/15919 171 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:18.927994) v4 ==== 2004+0+0 (337030713 0 0) 0x55f66b1cce00 con 0x55f663f0a000 > -9949> 2018-07-11 08:04:18.292992 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.11 10.8.128.114:0/15919 171 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:18.927994) v4 ==== 2004+0+0 (337030713 0 0) 0x55f663fb9c00 con 0x55f663f0b800 > -9948> 2018-07-11 08:04:18.292999 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:18.927994) v4 -- 0x55f64cc80a00 con 0 > -9947> 2018-07-11 08:04:18.293033 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:18.927994) v4 -- 0x55f653d4ac00 con 0 > -9946> 2018-07-11 08:04:18.401160 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.115:0/13323 conn(0x55f663ebe000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.6 seq 168 0x55f650f78a00 osd_ping(ping e509 stamp 2018-07-11 08:04:18.753123) v4 > -9945> 2018-07-11 08:04:18.401162 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.115:0/13323 conn(0x55f663ebf800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.6 seq 168 0x55f664256a00 osd_ping(ping e509 stamp 2018-07-11 08:04:18.753123) v4 > -9944> 2018-07-11 08:04:18.401185 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.6 10.8.128.115:0/13323 168 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:18.753123) v4 ==== 2004+0+0 (3473779907 0 0) 0x55f650f78a00 con 0x55f663ebe000 > -9943> 2018-07-11 08:04:18.401186 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.6 10.8.128.115:0/13323 168 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:18.753123) v4 ==== 2004+0+0 (3473779907 0 0) 0x55f664256a00 con 0x55f663ebf800 > -9942> 2018-07-11 08:04:18.401197 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:18.753123) v4 -- 0x55f653d4ac00 con 0 > -9941> 2018-07-11 08:04:18.401224 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:18.753123) v4 -- 0x55f666c46c00 con 0 > -9940> 2018-07-11 08:04:18.489713 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.108:0/11967 conn(0x55f663f70800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=1). rx osd.8 seq 171 0x55f64cc81800 osd_ping(ping e509 stamp 2018-07-11 08:04:18.249230) v4 > -9939> 2018-07-11 08:04:18.489734 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.8 10.8.128.108:0/11967 171 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:18.249230) v4 ==== 2004+0+0 (3596299905 0 0) 0x55f64cc81800 con 0x55f663f70800 > -9938> 2018-07-11 08:04:18.489746 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:18.249230) v4 -- 0x55f666c46c00 con 0 > -9937> 2018-07-11 08:04:18.489750 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.108:0/11967 conn(0x55f663f6f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=1). rx osd.8 seq 171 0x55f663fb9c00 osd_ping(ping e509 stamp 2018-07-11 08:04:18.249230) v4 > -9936> 2018-07-11 08:04:18.489770 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.8 10.8.128.108:0/11967 171 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:18.249230) v4 ==== 2004+0+0 (3596299905 0 0) 0x55f663fb9c00 con 0x55f663f6f000 > -9935> 2018-07-11 08:04:18.489782 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:18.249230) v4 -- 0x55f653d4ac00 con 0 > -9934> 2018-07-11 08:04:18.653143 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.114:6805/15885 conn(0x55f66315a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.7 seq 2495 0x55f664256a00 MOSDScrubReserve(5.4 REQUEST e509) v1 > -9933> 2018-07-11 08:04:18.653162 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.7 10.8.128.114:6805/15885 2495 ==== MOSDScrubReserve(5.4 REQUEST e509) v1 ==== 43+0+0 (3003630469 0 0) 0x55f664256a00 con 0x55f66315a000 > -9932> 2018-07-11 08:04:18.653241 7f491983f700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.114:6805/15885 -- MOSDScrubReserve(5.4 GRANT e509) v1 -- 0x55f663c0bc00 con 0 > -9931> 2018-07-11 08:04:18.653493 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.114:6805/15885 conn(0x55f66315a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.7 seq 2496 0x55f666c46c00 MOSDScrubReserve(5.4 RELEASE e509) v1 > -9930> 2018-07-11 08:04:18.653523 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.7 10.8.128.114:6805/15885 2496 ==== MOSDScrubReserve(5.4 RELEASE e509) v1 ==== 43+0+0 (2082093447 0 0) 0x55f666c46c00 con 0x55f66315a000 > -9929> 2018-07-11 08:04:18.725691 7f491803c700 5 osd.10 509 heartbeat: osd_stat(788 GB used, 142 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist []) > -9928> 2018-07-11 08:04:18.725705 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6802/15255 -- osd_ping(ping e509 stamp 2018-07-11 08:04:18.725703) v4 -- 0x55f663193800 con 0 > -9927> 2018-07-11 08:04:18.725742 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6803/15255 -- osd_ping(ping e509 stamp 2018-07-11 08:04:18.725703) v4 -- 0x55f6548fba00 con 0 > -9926> 2018-07-11 08:04:18.725758 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6806/12677 -- osd_ping(ping e509 stamp 2018-07-11 08:04:18.725703) v4 -- 0x55f6548fa200 con 0 > -9925> 2018-07-11 08:04:18.725773 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6807/12677 -- osd_ping(ping e509 stamp 2018-07-11 08:04:18.725703) v4 -- 0x55f663f5e400 con 0 > -9924> 2018-07-11 08:04:18.725787 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6808/12651 -- osd_ping(ping e509 stamp 2018-07-11 08:04:18.725703) v4 -- 0x55f664255200 con 0 > -9923> 2018-07-11 08:04:18.725797 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6810/12651 -- osd_ping(ping e509 stamp 2018-07-11 08:04:18.725703) v4 -- 0x55f668a30000 con 0 > -9922> 2018-07-11 08:04:18.725810 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6809/12650 -- osd_ping(ping e509 stamp 2018-07-11 08:04:18.725703) v4 -- 0x55f650f78c00 con 0 > -9921> 2018-07-11 08:04:18.725830 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6811/12650 -- osd_ping(ping e509 stamp 2018-07-11 08:04:18.725703) v4 -- 0x55f663c0aa00 con 0 > -9920> 2018-07-11 08:04:18.725843 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6810/15287 -- osd_ping(ping e509 stamp 2018-07-11 08:04:18.725703) v4 -- 0x55f663c0b400 con 0 > -9919> 2018-07-11 08:04:18.725855 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6811/15287 -- osd_ping(ping e509 stamp 2018-07-11 08:04:18.725703) v4 -- 0x55f664257000 con 0 > -9918> 2018-07-11 08:04:18.725869 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6806/15263 -- osd_ping(ping e509 stamp 2018-07-11 08:04:18.725703) v4 -- 0x55f667d09000 con 0 > -9917> 2018-07-11 08:04:18.725882 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6807/15263 -- osd_ping(ping e509 stamp 2018-07-11 08:04:18.725703) v4 -- 0x55f666c46e00 con 0 > -9916> 2018-07-11 08:04:18.725895 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.115:6802/13323 -- osd_ping(ping e509 stamp 2018-07-11 08:04:18.725703) v4 -- 0x55f67d1ae400 con 0 > -9915> 2018-07-11 08:04:18.725907 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.115:6803/13323 -- osd_ping(ping e509 stamp 2018-07-11 08:04:18.725703) v4 -- 0x55f64e25dc00 con 0 > -9914> 2018-07-11 08:04:18.725924 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6806/15885 -- osd_ping(ping e509 stamp 2018-07-11 08:04:18.725703) v4 -- 0x55f65e71d200 con 0 > -9913> 2018-07-11 08:04:18.725937 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6807/15885 -- osd_ping(ping e509 stamp 2018-07-11 08:04:18.725703) v4 -- 0x55f667d09c00 con 0 > -9912> 2018-07-11 08:04:18.725951 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.108:6802/11967 -- osd_ping(ping e509 stamp 2018-07-11 08:04:18.725703) v4 -- 0x55f664256600 con 0 > -9911> 2018-07-11 08:04:18.725963 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.108:6803/11967 -- osd_ping(ping e509 stamp 2018-07-11 08:04:18.725703) v4 -- 0x55f64e25c600 con 0 > -9910> 2018-07-11 08:04:18.725976 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.107:6806/12617 -- osd_ping(ping e509 stamp 2018-07-11 08:04:18.725703) v4 -- 0x55f67d1af400 con 0 > -9909> 2018-07-11 08:04:18.725989 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.107:6807/12617 -- osd_ping(ping e509 stamp 2018-07-11 08:04:18.725703) v4 -- 0x55f66dceb000 con 0 > -9908> 2018-07-11 08:04:18.726001 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6802/15919 -- osd_ping(ping e509 stamp 2018-07-11 08:04:18.725703) v4 -- 0x55f660949800 con 0 > -9907> 2018-07-11 08:04:18.726013 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6803/15919 -- osd_ping(ping e509 stamp 2018-07-11 08:04:18.725703) v4 -- 0x55f65dad4a00 con 0 > -9906> 2018-07-11 08:04:18.726152 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6803/15255 conn(0x55f656ad5800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 176 0x55f663fb9c00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:18.725703) v4 > -9905> 2018-07-11 08:04:18.726173 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.0 10.8.128.113:6803/15255 176 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:18.725703) v4 ==== 2004+0+0 (1795057935 0 0) 0x55f663fb9c00 con 0x55f656ad5800 > -9904> 2018-07-11 08:04:18.726206 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6806/12677 conn(0x55f64e3d8000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.1 seq 176 0x55f65299b600 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:18.725703) v4 > -9903> 2018-07-11 08:04:18.726226 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.1 10.8.128.106:6806/12677 176 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:18.725703) v4 ==== 2004+0+0 (1795057935 0 0) 0x55f65299b600 con 0x55f64e3d8000 > -9902> 2018-07-11 08:04:18.726226 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.107:6806/12617 conn(0x55f656ade000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.9 seq 176 0x55f663fb9c00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:18.725703) v4 > -9901> 2018-07-11 08:04:18.726270 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6802/15255 conn(0x55f656ad4000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 176 0x55f64cc81800 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:18.725703) v4 > -9900> 2018-07-11 08:04:18.726275 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.9 10.8.128.107:6806/12617 176 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:18.725703) v4 ==== 2004+0+0 (1795057935 0 0) 0x55f663fb9c00 con 0x55f656ade000 > -9899> 2018-07-11 08:04:18.726289 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.0 10.8.128.113:6802/15255 176 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:18.725703) v4 ==== 2004+0+0 (1795057935 0 0) 0x55f64cc81800 con 0x55f656ad4000 > -9898> 2018-07-11 08:04:18.726303 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6809/12650 conn(0x55f656ad7000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.3 seq 176 0x55f65299b600 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:18.725703) v4 > -9897> 2018-07-11 08:04:18.726326 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.3 10.8.128.106:6809/12650 176 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:18.725703) v4 ==== 2004+0+0 (1795057935 0 0) 0x55f65299b600 con 0x55f656ad7000 > -9896> 2018-07-11 08:04:18.726331 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6807/12677 conn(0x55f64e3d9800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.1 seq 176 0x55f64cc81800 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:18.725703) v4 > -9895> 2018-07-11 08:04:18.726339 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6810/12651 conn(0x55f64e3eb800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.2 seq 176 0x55f663fb9c00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:18.725703) v4 > -9894> 2018-07-11 08:04:18.726343 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.1 10.8.128.106:6807/12677 176 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:18.725703) v4 ==== 2004+0+0 (1795057935 0 0) 0x55f64cc81800 con 0x55f64e3d9800 > -9893> 2018-07-11 08:04:18.726355 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.2 10.8.128.106:6810/12651 176 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:18.725703) v4 ==== 2004+0+0 (1795057935 0 0) 0x55f663fb9c00 con 0x55f64e3eb800 > -9892> 2018-07-11 08:04:18.726364 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6806/15263 conn(0x55f64e3ed000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.5 seq 176 0x55f65299b600 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:18.725703) v4 > -9891> 2018-07-11 08:04:18.726377 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.5 10.8.128.113:6806/15263 176 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:18.725703) v4 ==== 2004+0+0 (1795057935 0 0) 0x55f65299b600 con 0x55f64e3ed000 > -9890> 2018-07-11 08:04:18.726389 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6811/12650 conn(0x55f656ad8800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.3 seq 176 0x55f64cc81800 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:18.725703) v4 > -9889> 2018-07-11 08:04:18.726395 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6811/15287 conn(0x55f64e402800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.4 seq 176 0x55f663fb9c00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:18.725703) v4 > -9888> 2018-07-11 08:04:18.726402 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.3 10.8.128.106:6811/12650 176 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:18.725703) v4 ==== 2004+0+0 (1795057935 0 0) 0x55f64cc81800 con 0x55f656ad8800 > -9887> 2018-07-11 08:04:18.726398 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.107:6807/12617 conn(0x55f656adf800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.9 seq 176 0x55f65299b600 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:18.725703) v4 > -9886> 2018-07-11 08:04:18.726410 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.4 10.8.128.113:6811/15287 176 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:18.725703) v4 ==== 2004+0+0 (1795057935 0 0) 0x55f663fb9c00 con 0x55f64e402800 > -9885> 2018-07-11 08:04:18.726414 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.9 10.8.128.107:6807/12617 176 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:18.725703) v4 ==== 2004+0+0 (1795057935 0 0) 0x55f65299b600 con 0x55f656adf800 > -9884> 2018-07-11 08:04:18.726433 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6810/15287 conn(0x55f64e401000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.4 seq 176 0x55f64cc81800 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:18.725703) v4 > -9883> 2018-07-11 08:04:18.726445 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.4 10.8.128.113:6810/15287 176 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:18.725703) v4 ==== 2004+0+0 (1795057935 0 0) 0x55f64cc81800 con 0x55f64e401000 > -9882> 2018-07-11 08:04:18.726451 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.115:6803/13323 conn(0x55f656ace800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.6 seq 176 0x55f65299b600 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:18.725703) v4 > -9881> 2018-07-11 08:04:18.726465 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.6 10.8.128.115:6803/13323 176 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:18.725703) v4 ==== 2004+0+0 (1795057935 0 0) 0x55f65299b600 con 0x55f656ace800 > -9880> 2018-07-11 08:04:18.726475 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6807/15263 conn(0x55f64e3ee800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.5 seq 176 0x55f64cc81800 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:18.725703) v4 > -9879> 2018-07-11 08:04:18.726487 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.5 10.8.128.113:6807/15263 176 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:18.725703) v4 ==== 2004+0+0 (1795057935 0 0) 0x55f64cc81800 con 0x55f64e3ee800 > -9878> 2018-07-11 08:04:18.726480 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6802/15919 conn(0x55f64e3db000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.11 seq 176 0x55f663fb9c00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:18.725703) v4 > -9877> 2018-07-11 08:04:18.726516 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6808/12651 conn(0x55f64e3ea000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.2 seq 176 0x55f64cc81800 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:18.725703) v4 > -9876> 2018-07-11 08:04:18.726529 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.2 10.8.128.106:6808/12651 176 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:18.725703) v4 ==== 2004+0+0 (1795057935 0 0) 0x55f64cc81800 con 0x55f64e3ea000 > -9875> 2018-07-11 08:04:18.726506 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.11 10.8.128.114:6802/15919 176 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:18.725703) v4 ==== 2004+0+0 (1795057935 0 0) 0x55f663fb9c00 con 0x55f64e3db000 > -9874> 2018-07-11 08:04:18.726548 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.108:6802/11967 conn(0x55f656aca000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.8 seq 176 0x55f65299b600 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:18.725703) v4 > -9873> 2018-07-11 08:04:18.726562 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.8 10.8.128.108:6802/11967 176 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:18.725703) v4 ==== 2004+0+0 (1795057935 0 0) 0x55f65299b600 con 0x55f656aca000 > -9872> 2018-07-11 08:04:18.726586 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.115:6802/13323 conn(0x55f656acd000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.6 seq 176 0x55f663fb9c00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:18.725703) v4 > -9871> 2018-07-11 08:04:18.726591 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6803/15919 conn(0x55f64e3dc800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.11 seq 176 0x55f65299b600 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:18.725703) v4 > -9870> 2018-07-11 08:04:18.726608 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.11 10.8.128.114:6803/15919 176 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:18.725703) v4 ==== 2004+0+0 (1795057935 0 0) 0x55f65299b600 con 0x55f64e3dc800 > -9869> 2018-07-11 08:04:18.726605 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.6 10.8.128.115:6802/13323 176 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:18.725703) v4 ==== 2004+0+0 (1795057935 0 0) 0x55f663fb9c00 con 0x55f656acd000 > -9868> 2018-07-11 08:04:18.726643 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6807/15885 conn(0x55f64e3ff800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.7 seq 176 0x55f65299b600 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:18.725703) v4 > -9867> 2018-07-11 08:04:18.726652 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6806/15885 conn(0x55f64e3fe000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.7 seq 176 0x55f663fb9c00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:18.725703) v4 > -9866> 2018-07-11 08:04:18.726658 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.7 10.8.128.114:6807/15885 176 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:18.725703) v4 ==== 2004+0+0 (1795057935 0 0) 0x55f65299b600 con 0x55f64e3ff800 > -9865> 2018-07-11 08:04:18.726673 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.7 10.8.128.114:6806/15885 176 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:18.725703) v4 ==== 2004+0+0 (1795057935 0 0) 0x55f663fb9c00 con 0x55f64e3fe000 > -9864> 2018-07-11 08:04:18.726675 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.108:6803/11967 conn(0x55f656acb800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.8 seq 176 0x55f64cc81800 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:18.725703) v4 > -9863> 2018-07-11 08:04:18.726694 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.8 10.8.128.108:6803/11967 176 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:18.725703) v4 ==== 2004+0+0 (1795057935 0 0) 0x55f64cc81800 con 0x55f656acb800 > -9862> 2018-07-11 08:04:18.920624 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12650 conn(0x55f663eda800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.3 seq 176 0x55f64cc81800 osd_ping(ping e509 stamp 2018-07-11 08:04:18.976665) v4 > -9861> 2018-07-11 08:04:18.920629 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12650 conn(0x55f663ed9000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.3 seq 176 0x55f663fb9c00 osd_ping(ping e509 stamp 2018-07-11 08:04:18.976665) v4 > -9860> 2018-07-11 08:04:18.920650 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.3 10.8.128.106:0/12650 176 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:18.976665) v4 ==== 2004+0+0 (3991696318 0 0) 0x55f663fb9c00 con 0x55f663ed9000 > -9859> 2018-07-11 08:04:18.920645 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.3 10.8.128.106:0/12650 176 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:18.976665) v4 ==== 2004+0+0 (3991696318 0 0) 0x55f64cc81800 con 0x55f663eda800 > -9858> 2018-07-11 08:04:18.920664 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:18.976665) v4 -- 0x55f660949800 con 0 > -9857> 2018-07-11 08:04:18.920690 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:18.976665) v4 -- 0x55f64e25c600 con 0 > -9856> 2018-07-11 08:04:19.026292 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 28] Generated table #37025: 2638241 keys, 74546767 bytes > -9855> 2018-07-11 08:04:19.026339 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296259026323, "cf_name": "default", "job": 28, "event": "table_file_creation", "file_number": 37025, "file_size": 74546767, "table_properties": {"data_size": 67112150, "index_size": 690444, "filter_size": 6743183, "raw_key_size": 76215848, "raw_average_key_size": 28, "raw_value_size": 31365751, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} > -9854> 2018-07-11 08:04:19.294738 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15287 conn(0x55f66dce1000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.4 seq 187 0x55f65299b600 osd_ping(ping e509 stamp 2018-07-11 08:04:20.538018) v4 > -9853> 2018-07-11 08:04:19.294766 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.4 10.8.128.113:0/15287 187 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:20.538018) v4 ==== 2004+0+0 (3476241046 0 0) 0x55f65299b600 con 0x55f66dce1000 > -9852> 2018-07-11 08:04:19.294791 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:20.538018) v4 -- 0x55f65dad4a00 con 0 > -9851> 2018-07-11 08:04:19.294789 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15287 conn(0x55f66dce2800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.4 seq 187 0x55f663c0bc00 osd_ping(ping e509 stamp 2018-07-11 08:04:20.538018) v4 > -9850> 2018-07-11 08:04:19.294806 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.4 10.8.128.113:0/15287 187 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:20.538018) v4 ==== 2004+0+0 (3476241046 0 0) 0x55f663c0bc00 con 0x55f66dce2800 > -9849> 2018-07-11 08:04:19.294821 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:20.538018) v4 -- 0x55f64e25c600 con 0 > -9848> 2018-07-11 08:04:19.464983 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed7800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 175 0x55f66b1cce00 osd_ping(ping e509 stamp 2018-07-11 08:04:19.520948) v4 > -9847> 2018-07-11 08:04:19.465021 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.1 10.8.128.106:0/12677 175 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:19.520948) v4 ==== 2004+0+0 (42213725 0 0) 0x55f66b1cce00 con 0x55f663ed7800 > -9846> 2018-07-11 08:04:19.465031 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:19.520948) v4 -- 0x55f65dad4a00 con 0 > -9845> 2018-07-11 08:04:19.465061 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed6000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 175 0x55f64cc81800 osd_ping(ping e509 stamp 2018-07-11 08:04:19.520948) v4 > -9844> 2018-07-11 08:04:19.465082 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.1 10.8.128.106:0/12677 175 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:19.520948) v4 ==== 2004+0+0 (42213725 0 0) 0x55f64cc81800 con 0x55f663ed6000 > -9843> 2018-07-11 08:04:19.465094 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:19.520948) v4 -- 0x55f64e25c600 con 0 > -9842> 2018-07-11 08:04:19.499581 7f492d867700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.106:6802/12651 -- MOSDScrubReserve(5.2e REQUEST e509) v1 -- 0x55f64e20fc00 con 0 > -9841> 2018-07-11 08:04:19.499613 7f492d867700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6809/15287 -- MOSDScrubReserve(5.2e REQUEST e509) v1 -- 0x55f667d0d800 con 0 > -9840> 2018-07-11 08:04:19.499980 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.106:6802/12651 conn(0x55f656aec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=38 cs=1 l=0). rx osd.2 seq 76 0x55f64e20fc00 MOSDScrubReserve(5.2e REJECT e509) v1 > -9839> 2018-07-11 08:04:19.499999 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.2 10.8.128.106:6802/12651 76 ==== MOSDScrubReserve(5.2e REJECT e509) v1 ==== 43+0+0 (2335265730 0 0) 0x55f64e20fc00 con 0x55f656aec000 > -9838> 2018-07-11 08:04:19.500053 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.106:6802/12651 -- MOSDScrubReserve(5.2e RELEASE e509) v1 -- 0x55f64e20e800 con 0 > -9837> 2018-07-11 08:04:19.500070 7f4932031700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6809/15287 conn(0x55f656aef000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=27 cs=1 l=0). rx osd.4 seq 61 0x55f667d0d800 MOSDScrubReserve(5.2e REJECT e509) v1 > -9836> 2018-07-11 08:04:19.500088 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6809/15287 -- MOSDScrubReserve(5.2e RELEASE e509) v1 -- 0x55f64e20f200 con 0 > -9835> 2018-07-11 08:04:19.500086 7f4932031700 1 -- 10.8.128.107:6801/14173 <== osd.4 10.8.128.113:6809/15287 61 ==== MOSDScrubReserve(5.2e REJECT e509) v1 ==== 43+0+0 (458633770 0 0) 0x55f667d0d800 con 0x55f656aef000 > -9834> 2018-07-11 08:04:19.542732 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12651 conn(0x55f663219000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25 cs=1 l=1). rx osd.2 seq 189 0x55f64e20e800 osd_ping(ping e509 stamp 2018-07-11 08:04:19.598661) v4 > -9833> 2018-07-11 08:04:19.542754 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.2 10.8.128.106:0/12651 189 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:19.598661) v4 ==== 2004+0+0 (1635966551 0 0) 0x55f64e20e800 con 0x55f663219000 > -9832> 2018-07-11 08:04:19.542763 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:19.598661) v4 -- 0x55f64cc81800 con 0 > -9831> 2018-07-11 08:04:19.542770 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12651 conn(0x55f66321a800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25 cs=1 l=1). rx osd.2 seq 189 0x55f64e20f200 osd_ping(ping e509 stamp 2018-07-11 08:04:19.598661) v4 > -9830> 2018-07-11 08:04:19.542793 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.2 10.8.128.106:0/12651 189 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:19.598661) v4 ==== 2004+0+0 (1635966551 0 0) 0x55f64e20f200 con 0x55f66321a800 > -9829> 2018-07-11 08:04:19.542805 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:19.598661) v4 -- 0x55f65299b600 con 0 > -9828> 2018-07-11 08:04:19.645700 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0e800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 180 0x55f66b1cce00 osd_ping(ping e509 stamp 2018-07-11 08:04:20.889188) v4 > -9827> 2018-07-11 08:04:19.645707 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0d000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 180 0x55f663c0bc00 osd_ping(ping e509 stamp 2018-07-11 08:04:20.889188) v4 > -9826> 2018-07-11 08:04:19.645726 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.5 10.8.128.113:0/15263 180 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:20.889188) v4 ==== 2004+0+0 (1908359414 0 0) 0x55f663c0bc00 con 0x55f663f0d000 > -9825> 2018-07-11 08:04:19.645725 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.5 10.8.128.113:0/15263 180 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:20.889188) v4 ==== 2004+0+0 (1908359414 0 0) 0x55f66b1cce00 con 0x55f663f0e800 > -9824> 2018-07-11 08:04:19.645735 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:20.889188) v4 -- 0x55f64cc81800 con 0 > -9823> 2018-07-11 08:04:19.645763 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:20.889188) v4 -- 0x55f65299b600 con 0 > -9822> 2018-07-11 08:04:19.821487 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15885 conn(0x55f663216000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 177 0x55f64e20f200 osd_ping(ping e509 stamp 2018-07-11 08:04:20.456578) v4 > -9821> 2018-07-11 08:04:19.821506 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15885 conn(0x55f663217800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 177 0x55f650f78a00 osd_ping(ping e509 stamp 2018-07-11 08:04:20.456578) v4 > -9820> 2018-07-11 08:04:19.821512 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.7 10.8.128.114:0/15885 177 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:20.456578) v4 ==== 2004+0+0 (3869210228 0 0) 0x55f64e20f200 con 0x55f663216000 > -9819> 2018-07-11 08:04:19.821532 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.7 10.8.128.114:0/15885 177 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:20.456578) v4 ==== 2004+0+0 (3869210228 0 0) 0x55f650f78a00 con 0x55f663217800 > -9818> 2018-07-11 08:04:19.821540 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:20.456578) v4 -- 0x55f65299b600 con 0 > -9817> 2018-07-11 08:04:19.821571 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:20.456578) v4 -- 0x55f660949800 con 0 > -9816> 2018-07-11 08:04:19.954417 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 383 0x55f663fb9c00 MOSDScrubReserve(11.3s5 REQUEST e509) v1 > -9815> 2018-07-11 08:04:19.954438 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 383 ==== MOSDScrubReserve(11.3s5 REQUEST e509) v1 ==== 43+0+0 (433425070 0 0) 0x55f663fb9c00 con 0x55f668a29000 > -9814> 2018-07-11 08:04:19.954487 7f491a040700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 GRANT e509) v1 -- 0x55f650f79200 con 0 > -9813> 2018-07-11 08:04:19.954824 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 384 0x55f660949800 MOSDScrubReserve(11.3s5 RELEASE e509) v1 > -9812> 2018-07-11 08:04:19.954866 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 384 ==== MOSDScrubReserve(11.3s5 RELEASE e509) v1 ==== 43+0+0 (3603460268 0 0) 0x55f660949800 con 0x55f668a29000 > -9811> 2018-07-11 08:04:20.044657 7f4932031700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.114:6801/15919 conn(0x55f66315d000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=30 cs=1 l=0). rx osd.11 seq 101 0x55f66b1cce00 MOSDScrubReserve(10.f REQUEST e509) v1 > -9810> 2018-07-11 08:04:20.044675 7f4932031700 1 -- 10.8.128.107:6801/14173 <== osd.11 10.8.128.114:6801/15919 101 ==== MOSDScrubReserve(10.f REQUEST e509) v1 ==== 43+0+0 (2011750585 0 0) 0x55f66b1cce00 con 0x55f66315d000 > -9809> 2018-07-11 08:04:20.044747 7f491b843700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.114:6801/15919 -- MOSDScrubReserve(10.f GRANT e509) v1 -- 0x55f67d1afe00 con 0 > -9808> 2018-07-11 08:04:20.045056 7f4932031700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.114:6801/15919 conn(0x55f66315d000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=30 cs=1 l=0). rx osd.11 seq 102 0x55f67d1afe00 MOSDScrubReserve(10.f RELEASE e509) v1 > -9807> 2018-07-11 08:04:20.045071 7f4932031700 1 -- 10.8.128.107:6801/14173 <== osd.11 10.8.128.114:6801/15919 102 ==== MOSDScrubReserve(10.f RELEASE e509) v1 ==== 43+0+0 (3103070907 0 0) 0x55f67d1afe00 con 0x55f66315d000 > -9806> 2018-07-11 08:04:20.432760 7f491e048700 10 monclient: tick > -9805> 2018-07-11 08:04:20.432778 7f491e048700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2018-07-11 08:03:50.432776) > -9804> 2018-07-11 08:04:20.523485 7f491d046700 1 -- 10.8.128.107:6800/14173 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55f653d5a000 con 0 > -9803> 2018-07-11 08:04:20.523563 7f491d046700 1 -- 10.8.128.107:6800/14173 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55f6631cf600 con 0 > -9802> 2018-07-11 08:04:20.565475 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed7800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 176 0x55f64e20f200 osd_ping(ping e509 stamp 2018-07-11 08:04:20.621516) v4 > -9801> 2018-07-11 08:04:20.565514 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.1 10.8.128.106:0/12677 176 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:20.621516) v4 ==== 2004+0+0 (4204736839 0 0) 0x55f64e20f200 con 0x55f663ed7800 > -9800> 2018-07-11 08:04:20.565527 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:20.621516) v4 -- 0x55f65dad4a00 con 0 > -9799> 2018-07-11 08:04:20.565558 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed6000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 176 0x55f663c0bc00 osd_ping(ping e509 stamp 2018-07-11 08:04:20.621516) v4 > -9798> 2018-07-11 08:04:20.565586 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.1 10.8.128.106:0/12677 176 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:20.621516) v4 ==== 2004+0+0 (4204736839 0 0) 0x55f663c0bc00 con 0x55f663ed6000 > -9797> 2018-07-11 08:04:20.565603 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:20.621516) v4 -- 0x55f64cc81800 con 0 > -9796> 2018-07-11 08:04:21.221023 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12650 conn(0x55f663eda800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.3 seq 177 0x55f64e20e800 osd_ping(ping e509 stamp 2018-07-11 08:04:21.277098) v4 > -9795> 2018-07-11 08:04:21.221029 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12650 conn(0x55f663ed9000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.3 seq 177 0x55f650f78a00 osd_ping(ping e509 stamp 2018-07-11 08:04:21.277098) v4 > -9794> 2018-07-11 08:04:21.221063 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.3 10.8.128.106:0/12650 177 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:21.277098) v4 ==== 2004+0+0 (3257732880 0 0) 0x55f64e20e800 con 0x55f663eda800 > -9793> 2018-07-11 08:04:21.221075 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:21.277098) v4 -- 0x55f64cc81800 con 0 > -9792> 2018-07-11 08:04:21.221069 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.3 10.8.128.106:0/12650 177 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:21.277098) v4 ==== 2004+0+0 (3257732880 0 0) 0x55f650f78a00 con 0x55f663ed9000 > -9791> 2018-07-11 08:04:21.221102 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:21.277098) v4 -- 0x55f67d1af400 con 0 > -9790> 2018-07-11 08:04:21.499883 7f492d867700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.106:6802/12651 -- MOSDScrubReserve(5.2e REQUEST e509) v1 -- 0x55f667d0c400 con 0 > -9789> 2018-07-11 08:04:21.499941 7f492d867700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6809/15287 -- MOSDScrubReserve(5.2e REQUEST e509) v1 -- 0x55f65d024c00 con 0 > -9788> 2018-07-11 08:04:21.500370 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.106:6802/12651 conn(0x55f656aec000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=38 cs=1 l=0). rx osd.2 seq 77 0x55f667d0c400 MOSDScrubReserve(5.2e REJECT e509) v1 > -9787> 2018-07-11 08:04:21.500391 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.2 10.8.128.106:6802/12651 77 ==== MOSDScrubReserve(5.2e REJECT e509) v1 ==== 43+0+0 (2335265730 0 0) 0x55f667d0c400 con 0x55f656aec000 > -9786> 2018-07-11 08:04:21.500440 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.106:6802/12651 -- MOSDScrubReserve(5.2e RELEASE e509) v1 -- 0x55f667d0d800 con 0 > -9785> 2018-07-11 08:04:21.500470 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6809/15287 -- MOSDScrubReserve(5.2e RELEASE e509) v1 -- 0x55f64e20fc00 con 0 > -9784> 2018-07-11 08:04:21.500567 7f4932031700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6809/15287 conn(0x55f656aef000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=27 cs=1 l=0). rx osd.4 seq 62 0x55f65299b600 MOSDScrubReserve(5.2e REJECT e509) v1 > -9783> 2018-07-11 08:04:21.500584 7f4932031700 1 -- 10.8.128.107:6801/14173 <== osd.4 10.8.128.113:6809/15287 62 ==== MOSDScrubReserve(5.2e REJECT e509) v1 ==== 43+0+0 (458633770 0 0) 0x55f65299b600 con 0x55f656aef000 > -9782> 2018-07-11 08:04:21.521936 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15885 conn(0x55f663217800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 178 0x55f650f79200 osd_ping(ping e509 stamp 2018-07-11 08:04:22.157024) v4 > -9781> 2018-07-11 08:04:21.521941 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15885 conn(0x55f663216000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 178 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:04:22.157024) v4 > -9780> 2018-07-11 08:04:21.521959 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.7 10.8.128.114:0/15885 178 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:22.157024) v4 ==== 2004+0+0 (574404275 0 0) 0x55f650f79200 con 0x55f663217800 > -9779> 2018-07-11 08:04:21.521962 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.7 10.8.128.114:0/15885 178 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:22.157024) v4 ==== 2004+0+0 (574404275 0 0) 0x55f64e20fc00 con 0x55f663216000 > -9778> 2018-07-11 08:04:21.521969 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:22.157024) v4 -- 0x55f67d1af400 con 0 > -9777> 2018-07-11 08:04:21.522004 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:22.157024) v4 -- 0x55f65d024c00 con 0 > -9776> 2018-07-11 08:04:21.626125 7f491803c700 5 osd.10 509 heartbeat: osd_stat(788 GB used, 142 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist []) > -9775> 2018-07-11 08:04:21.626139 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6802/15255 -- osd_ping(ping e509 stamp 2018-07-11 08:04:21.626137) v4 -- 0x55f64cc80200 con 0 > -9774> 2018-07-11 08:04:21.626175 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6803/15255 -- osd_ping(ping e509 stamp 2018-07-11 08:04:21.626137) v4 -- 0x55f663fb8000 con 0 > -9773> 2018-07-11 08:04:21.626191 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6806/12677 -- osd_ping(ping e509 stamp 2018-07-11 08:04:21.626137) v4 -- 0x55f650f79600 con 0 > -9772> 2018-07-11 08:04:21.626206 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6807/12677 -- osd_ping(ping e509 stamp 2018-07-11 08:04:21.626137) v4 -- 0x55f663fb9800 con 0 > -9771> 2018-07-11 08:04:21.626219 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6808/12651 -- osd_ping(ping e509 stamp 2018-07-11 08:04:21.626137) v4 -- 0x55f64e25ca00 con 0 > -9770> 2018-07-11 08:04:21.626230 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6810/12651 -- osd_ping(ping e509 stamp 2018-07-11 08:04:21.626137) v4 -- 0x55f6631cd000 con 0 > -9769> 2018-07-11 08:04:21.626242 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6809/12650 -- osd_ping(ping e509 stamp 2018-07-11 08:04:21.626137) v4 -- 0x55f65299b800 con 0 > -9768> 2018-07-11 08:04:21.626269 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6811/12650 -- osd_ping(ping e509 stamp 2018-07-11 08:04:21.626137) v4 -- 0x55f64cf24000 con 0 > -9767> 2018-07-11 08:04:21.626290 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6810/15287 -- osd_ping(ping e509 stamp 2018-07-11 08:04:21.626137) v4 -- 0x55f666c47000 con 0 > -9766> 2018-07-11 08:04:21.626303 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6811/15287 -- osd_ping(ping e509 stamp 2018-07-11 08:04:21.626137) v4 -- 0x55f662489e00 con 0 > -9765> 2018-07-11 08:04:21.626316 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6806/15263 -- osd_ping(ping e509 stamp 2018-07-11 08:04:21.626137) v4 -- 0x55f67d1afc00 con 0 > -9764> 2018-07-11 08:04:21.626333 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6807/15263 -- osd_ping(ping e509 stamp 2018-07-11 08:04:21.626137) v4 -- 0x55f64e20f000 con 0 > -9763> 2018-07-11 08:04:21.626347 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.115:6802/13323 -- osd_ping(ping e509 stamp 2018-07-11 08:04:21.626137) v4 -- 0x55f681f90000 con 0 > -9762> 2018-07-11 08:04:21.626360 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.115:6803/13323 -- osd_ping(ping e509 stamp 2018-07-11 08:04:21.626137) v4 -- 0x55f664256200 con 0 > -9761> 2018-07-11 08:04:21.626373 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6806/15885 -- osd_ping(ping e509 stamp 2018-07-11 08:04:21.626137) v4 -- 0x55f64cf25a00 con 0 > -9760> 2018-07-11 08:04:21.626386 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6807/15885 -- osd_ping(ping e509 stamp 2018-07-11 08:04:21.626137) v4 -- 0x55f664256000 con 0 > -9759> 2018-07-11 08:04:21.626399 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.108:6802/11967 -- osd_ping(ping e509 stamp 2018-07-11 08:04:21.626137) v4 -- 0x55f65e71ce00 con 0 > -9758> 2018-07-11 08:04:21.626411 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.108:6803/11967 -- osd_ping(ping e509 stamp 2018-07-11 08:04:21.626137) v4 -- 0x55f65dad4e00 con 0 > -9757> 2018-07-11 08:04:21.626424 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.107:6806/12617 -- osd_ping(ping e509 stamp 2018-07-11 08:04:21.626137) v4 -- 0x55f6548fb400 con 0 > -9756> 2018-07-11 08:04:21.626437 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.107:6807/12617 -- osd_ping(ping e509 stamp 2018-07-11 08:04:21.626137) v4 -- 0x55f64cc81c00 con 0 > -9755> 2018-07-11 08:04:21.626449 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6802/15919 -- osd_ping(ping e509 stamp 2018-07-11 08:04:21.626137) v4 -- 0x55f64e25cc00 con 0 > -9754> 2018-07-11 08:04:21.626461 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6803/15919 -- osd_ping(ping e509 stamp 2018-07-11 08:04:21.626137) v4 -- 0x55f667d09600 con 0 > -9753> 2018-07-11 08:04:21.626529 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6802/15255 conn(0x55f656ad4000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 177 0x55f667d0d800 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:21.626137) v4 > -9752> 2018-07-11 08:04:21.626551 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.0 10.8.128.113:6802/15255 177 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:21.626137) v4 ==== 2004+0+0 (3898459380 0 0) 0x55f667d0d800 con 0x55f656ad4000 > -9751> 2018-07-11 08:04:21.626588 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.107:6806/12617 conn(0x55f656ade000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.9 seq 177 0x55f650f79200 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:21.626137) v4 > -9750> 2018-07-11 08:04:21.626641 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.9 10.8.128.107:6806/12617 177 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:21.626137) v4 ==== 2004+0+0 (3898459380 0 0) 0x55f650f79200 con 0x55f656ade000 > -9749> 2018-07-11 08:04:21.626645 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6811/12650 conn(0x55f656ad8800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.3 seq 177 0x55f667d0d800 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:21.626137) v4 > -9748> 2018-07-11 08:04:21.626670 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6803/15255 conn(0x55f656ad5800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 177 0x55f650f79200 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:21.626137) v4 > -9747> 2018-07-11 08:04:21.626682 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.0 10.8.128.113:6803/15255 177 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:21.626137) v4 ==== 2004+0+0 (3898459380 0 0) 0x55f650f79200 con 0x55f656ad5800 > -9746> 2018-07-11 08:04:21.626661 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.3 10.8.128.106:6811/12650 177 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:21.626137) v4 ==== 2004+0+0 (3898459380 0 0) 0x55f667d0d800 con 0x55f656ad8800 > -9745> 2018-07-11 08:04:21.626678 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.107:6807/12617 conn(0x55f656adf800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.9 seq 177 0x55f64e20f200 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:21.626137) v4 > -9744> 2018-07-11 08:04:21.626698 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.9 10.8.128.107:6807/12617 177 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:21.626137) v4 ==== 2004+0+0 (3898459380 0 0) 0x55f64e20f200 con 0x55f656adf800 > -9743> 2018-07-11 08:04:21.626742 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6811/15287 conn(0x55f64e402800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.4 seq 177 0x55f650f79200 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:21.626137) v4 > -9742> 2018-07-11 08:04:21.626756 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.4 10.8.128.113:6811/15287 177 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:21.626137) v4 ==== 2004+0+0 (3898459380 0 0) 0x55f650f79200 con 0x55f64e402800 > -9741> 2018-07-11 08:04:21.626753 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6810/15287 conn(0x55f64e401000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.4 seq 177 0x55f667d0d800 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:21.626137) v4 > -9740> 2018-07-11 08:04:21.626781 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.4 10.8.128.113:6810/15287 177 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:21.626137) v4 ==== 2004+0+0 (3898459380 0 0) 0x55f667d0d800 con 0x55f64e401000 > -9739> 2018-07-11 08:04:21.626786 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6810/12651 conn(0x55f64e3eb800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.2 seq 177 0x55f650f79200 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:21.626137) v4 > -9738> 2018-07-11 08:04:21.626784 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6809/12650 conn(0x55f656ad7000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.3 seq 177 0x55f64e20f200 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:21.626137) v4 > -9737> 2018-07-11 08:04:21.626799 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.2 10.8.128.106:6810/12651 177 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:21.626137) v4 ==== 2004+0+0 (3898459380 0 0) 0x55f650f79200 con 0x55f64e3eb800 > -9736> 2018-07-11 08:04:21.626801 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.3 10.8.128.106:6809/12650 177 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:21.626137) v4 ==== 2004+0+0 (3898459380 0 0) 0x55f64e20f200 con 0x55f656ad7000 > -9735> 2018-07-11 08:04:21.626812 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6808/12651 conn(0x55f64e3ea000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.2 seq 177 0x55f667d0d800 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:21.626137) v4 > -9734> 2018-07-11 08:04:21.626825 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.2 10.8.128.106:6808/12651 177 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:21.626137) v4 ==== 2004+0+0 (3898459380 0 0) 0x55f667d0d800 con 0x55f64e3ea000 > -9733> 2018-07-11 08:04:21.626834 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6806/12677 conn(0x55f64e3d8000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.1 seq 177 0x55f64e20f200 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:21.626137) v4 > -9732> 2018-07-11 08:04:21.626850 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.1 10.8.128.106:6806/12677 177 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:21.626137) v4 ==== 2004+0+0 (3898459380 0 0) 0x55f64e20f200 con 0x55f64e3d8000 > -9731> 2018-07-11 08:04:21.626860 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6807/12677 conn(0x55f64e3d9800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.1 seq 177 0x55f667d0d800 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:21.626137) v4 > -9730> 2018-07-11 08:04:21.626871 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6806/15263 conn(0x55f64e3ed000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.5 seq 177 0x55f64e20f200 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:21.626137) v4 > -9729> 2018-07-11 08:04:21.626873 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.1 10.8.128.106:6807/12677 177 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:21.626137) v4 ==== 2004+0+0 (3898459380 0 0) 0x55f667d0d800 con 0x55f64e3d9800 > -9728> 2018-07-11 08:04:21.626882 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.5 10.8.128.113:6806/15263 177 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:21.626137) v4 ==== 2004+0+0 (3898459380 0 0) 0x55f64e20f200 con 0x55f64e3ed000 > -9727> 2018-07-11 08:04:21.626909 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6807/15263 conn(0x55f64e3ee800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.5 seq 177 0x55f667d0d800 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:21.626137) v4 > -9726> 2018-07-11 08:04:21.626916 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6807/15885 conn(0x55f64e3ff800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.7 seq 177 0x55f64e20f200 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:21.626137) v4 > -9725> 2018-07-11 08:04:21.626922 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.5 10.8.128.113:6807/15263 177 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:21.626137) v4 ==== 2004+0+0 (3898459380 0 0) 0x55f667d0d800 con 0x55f64e3ee800 > -9724> 2018-07-11 08:04:21.626929 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.7 10.8.128.114:6807/15885 177 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:21.626137) v4 ==== 2004+0+0 (3898459380 0 0) 0x55f64e20f200 con 0x55f64e3ff800 > -9723> 2018-07-11 08:04:21.626958 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6803/15919 conn(0x55f64e3dc800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.11 seq 177 0x55f64e20f200 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:21.626137) v4 > -9722> 2018-07-11 08:04:21.626955 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6802/15919 conn(0x55f64e3db000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.11 seq 177 0x55f650f79200 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:21.626137) v4 > -9721> 2018-07-11 08:04:21.626970 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.11 10.8.128.114:6803/15919 177 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:21.626137) v4 ==== 2004+0+0 (3898459380 0 0) 0x55f64e20f200 con 0x55f64e3dc800 > -9720> 2018-07-11 08:04:21.626983 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.108:6803/11967 conn(0x55f656acb800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.8 seq 177 0x55f667d0d800 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:21.626137) v4 > -9719> 2018-07-11 08:04:21.626985 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.11 10.8.128.114:6802/15919 177 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:21.626137) v4 ==== 2004+0+0 (3898459380 0 0) 0x55f650f79200 con 0x55f64e3db000 > -9718> 2018-07-11 08:04:21.627003 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.8 10.8.128.108:6803/11967 177 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:21.626137) v4 ==== 2004+0+0 (3898459380 0 0) 0x55f667d0d800 con 0x55f656acb800 > -9717> 2018-07-11 08:04:21.627019 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.115:6803/13323 conn(0x55f656ace800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.6 seq 177 0x55f64e20f200 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:21.626137) v4 > -9716> 2018-07-11 08:04:21.627033 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.6 10.8.128.115:6803/13323 177 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:21.626137) v4 ==== 2004+0+0 (3898459380 0 0) 0x55f64e20f200 con 0x55f656ace800 > -9715> 2018-07-11 08:04:21.627054 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6806/15885 conn(0x55f64e3fe000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.7 seq 177 0x55f650f79200 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:21.626137) v4 > -9714> 2018-07-11 08:04:21.627066 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.108:6802/11967 conn(0x55f656aca000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.8 seq 177 0x55f64e20f200 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:21.626137) v4 > -9713> 2018-07-11 08:04:21.627075 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.8 10.8.128.108:6802/11967 177 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:21.626137) v4 ==== 2004+0+0 (3898459380 0 0) 0x55f64e20f200 con 0x55f656aca000 > -9712> 2018-07-11 08:04:21.627074 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.7 10.8.128.114:6806/15885 177 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:21.626137) v4 ==== 2004+0+0 (3898459380 0 0) 0x55f650f79200 con 0x55f64e3fe000 > -9711> 2018-07-11 08:04:21.627130 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.115:6802/13323 conn(0x55f656acd000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.6 seq 177 0x55f650f79200 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:21.626137) v4 > -9710> 2018-07-11 08:04:21.627151 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.6 10.8.128.115:6802/13323 177 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:21.626137) v4 ==== 2004+0+0 (3898459380 0 0) 0x55f650f79200 con 0x55f656acd000 > -9709> 2018-07-11 08:04:21.653383 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.114:6805/15885 conn(0x55f66315a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.7 seq 2497 0x55f667d0d800 MOSDScrubReserve(5.4 REQUEST e509) v1 > -9708> 2018-07-11 08:04:21.653398 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.7 10.8.128.114:6805/15885 2497 ==== MOSDScrubReserve(5.4 REQUEST e509) v1 ==== 43+0+0 (3003630469 0 0) 0x55f667d0d800 con 0x55f66315a000 > -9707> 2018-07-11 08:04:21.653471 7f491983f700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.114:6805/15885 -- MOSDScrubReserve(5.4 GRANT e509) v1 -- 0x55f666c46c00 con 0 > -9706> 2018-07-11 08:04:21.653845 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.114:6805/15885 conn(0x55f66315a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.7 seq 2498 0x55f666c46c00 MOSDScrubReserve(5.4 RELEASE e509) v1 > -9705> 2018-07-11 08:04:21.653859 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.7 10.8.128.114:6805/15885 2498 ==== MOSDScrubReserve(5.4 RELEASE e509) v1 ==== 43+0+0 (2082093447 0 0) 0x55f666c46c00 con 0x55f66315a000 > -9704> 2018-07-11 08:04:21.793319 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0b800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 172 0x55f650f79200 osd_ping(ping e509 stamp 2018-07-11 08:04:22.428413) v4 > -9703> 2018-07-11 08:04:21.793344 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.11 10.8.128.114:0/15919 172 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:22.428413) v4 ==== 2004+0+0 (1723492652 0 0) 0x55f650f79200 con 0x55f663f0b800 > -9702> 2018-07-11 08:04:21.793356 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:22.428413) v4 -- 0x55f64e25cc00 con 0 > -9701> 2018-07-11 08:04:21.793408 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0a000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 172 0x55f64e20f200 osd_ping(ping e509 stamp 2018-07-11 08:04:22.428413) v4 > -9700> 2018-07-11 08:04:21.793428 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.11 10.8.128.114:0/15919 172 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:22.428413) v4 ==== 2004+0+0 (1723492652 0 0) 0x55f64e20f200 con 0x55f663f0a000 > -9699> 2018-07-11 08:04:21.793449 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:22.428413) v4 -- 0x55f667d09600 con 0 > -9698> 2018-07-11 08:04:21.867496 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15255 conn(0x55f663ec2800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.0 seq 172 0x55f650f78a00 osd_ping(ping e509 stamp 2018-07-11 08:04:23.110995) v4 > -9697> 2018-07-11 08:04:21.867516 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.0 10.8.128.113:0/15255 172 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:23.110995) v4 ==== 2004+0+0 (3300219297 0 0) 0x55f650f78a00 con 0x55f663ec2800 > -9696> 2018-07-11 08:04:21.867524 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15255 conn(0x55f663ec1000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.0 seq 172 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:04:23.110995) v4 > -9695> 2018-07-11 08:04:21.867533 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:23.110995) v4 -- 0x55f64e25cc00 con 0 > -9694> 2018-07-11 08:04:21.867553 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.0 10.8.128.113:0/15255 172 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:23.110995) v4 ==== 2004+0+0 (3300219297 0 0) 0x55f64e20fc00 con 0x55f663ec1000 > -9693> 2018-07-11 08:04:21.867580 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:23.110995) v4 -- 0x55f667d09600 con 0 > -9692> 2018-07-11 08:04:21.901524 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.115:0/13323 conn(0x55f663ebf800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.6 seq 169 0x55f663c0bc00 osd_ping(ping e509 stamp 2018-07-11 08:04:22.253603) v4 > -9691> 2018-07-11 08:04:21.901549 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.6 10.8.128.115:0/13323 169 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:22.253603) v4 ==== 2004+0+0 (3308178236 0 0) 0x55f663c0bc00 con 0x55f663ebf800 > -9690> 2018-07-11 08:04:21.901566 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:22.253603) v4 -- 0x55f64e20f000 con 0 > -9689> 2018-07-11 08:04:21.901602 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.115:0/13323 conn(0x55f663ebe000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.6 seq 169 0x55f650f79200 osd_ping(ping e509 stamp 2018-07-11 08:04:22.253603) v4 > -9688> 2018-07-11 08:04:21.901622 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.6 10.8.128.115:0/13323 169 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:22.253603) v4 ==== 2004+0+0 (3308178236 0 0) 0x55f650f79200 con 0x55f663ebe000 > -9687> 2018-07-11 08:04:21.901633 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:22.253603) v4 -- 0x55f64e25cc00 con 0 > -9686> 2018-07-11 08:04:22.044948 7f4932031700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.114:6801/15919 conn(0x55f66315d000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=30 cs=1 l=0). rx osd.11 seq 103 0x55f64e20f200 MOSDScrubReserve(10.f REQUEST e509) v1 > -9685> 2018-07-11 08:04:22.044974 7f4932031700 1 -- 10.8.128.107:6801/14173 <== osd.11 10.8.128.114:6801/15919 103 ==== MOSDScrubReserve(10.f REQUEST e509) v1 ==== 43+0+0 (2011750585 0 0) 0x55f64e20f200 con 0x55f66315d000 > -9684> 2018-07-11 08:04:22.045098 7f491b843700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.114:6801/15919 -- MOSDScrubReserve(10.f GRANT e509) v1 -- 0x55f67d1afe00 con 0 > -9683> 2018-07-11 08:04:22.045318 7f4932031700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.114:6801/15919 conn(0x55f66315d000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=30 cs=1 l=0). rx osd.11 seq 104 0x55f667d09600 MOSDScrubReserve(10.f RELEASE e509) v1 > -9682> 2018-07-11 08:04:22.045346 7f4932031700 1 -- 10.8.128.107:6801/14173 <== osd.11 10.8.128.114:6801/15919 104 ==== MOSDScrubReserve(10.f RELEASE e509) v1 ==== 43+0+0 (3103070907 0 0) 0x55f667d09600 con 0x55f66315d000 > -9681> 2018-07-11 08:04:22.265930 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed6000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 177 0x55f65dad4e00 osd_ping(ping e509 stamp 2018-07-11 08:04:22.321977) v4 > -9680> 2018-07-11 08:04:22.265958 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.1 10.8.128.106:0/12677 177 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:22.321977) v4 ==== 2004+0+0 (2112664261 0 0) 0x55f65dad4e00 con 0x55f663ed6000 > -9679> 2018-07-11 08:04:22.265976 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:22.321977) v4 -- 0x55f64e20f000 con 0 > -9678> 2018-07-11 08:04:22.265968 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed7800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 177 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:04:22.321977) v4 > -9677> 2018-07-11 08:04:22.266006 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.1 10.8.128.106:0/12677 177 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:22.321977) v4 ==== 2004+0+0 (2112664261 0 0) 0x55f64e20fc00 con 0x55f663ed7800 > -9676> 2018-07-11 08:04:22.266024 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:22.321977) v4 -- 0x55f64cc81c00 con 0 > -9675> 2018-07-11 08:04:22.376263 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 28] Generated table #37026: 2638241 keys, 74546120 bytes > -9674> 2018-07-11 08:04:22.376311 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296262376295, "cf_name": "default", "job": 28, "event": "table_file_creation", "file_number": 37026, "file_size": 74546120, "table_properties": {"data_size": 67112169, "index_size": 689778, "filter_size": 6743183, "raw_key_size": 76215859, "raw_average_key_size": 28, "raw_value_size": 31365762, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} > -9673> 2018-07-11 08:04:22.873694 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.107:0/12617 conn(0x55f64e3c3800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.9 seq 178 0x55f650f78a00 osd_ping(ping e509 stamp 2018-07-11 08:04:22.873330) v4 > -9672> 2018-07-11 08:04:22.873721 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.9 10.8.128.107:0/12617 178 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:22.873330) v4 ==== 2004+0+0 (1645823341 0 0) 0x55f650f78a00 con 0x55f64e3c3800 > -9671> 2018-07-11 08:04:22.873734 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:22.873330) v4 -- 0x55f64e25cc00 con 0 > -9670> 2018-07-11 08:04:22.873739 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.107:0/12617 conn(0x55f64e3c5000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.9 seq 178 0x55f663c0bc00 osd_ping(ping e509 stamp 2018-07-11 08:04:22.873330) v4 > -9669> 2018-07-11 08:04:22.873757 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.9 10.8.128.107:0/12617 178 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:22.873330) v4 ==== 2004+0+0 (1645823341 0 0) 0x55f663c0bc00 con 0x55f64e3c5000 > -9668> 2018-07-11 08:04:22.873792 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:22.873330) v4 -- 0x55f64e20f000 con 0 > -9667> 2018-07-11 08:04:23.146117 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0d000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 181 0x55f65dad4e00 osd_ping(ping e509 stamp 2018-07-11 08:04:24.389600) v4 > -9666> 2018-07-11 08:04:23.146177 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.5 10.8.128.113:0/15263 181 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:24.389600) v4 ==== 2004+0+0 (3632159437 0 0) 0x55f65dad4e00 con 0x55f663f0d000 > -9665> 2018-07-11 08:04:23.146195 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:24.389600) v4 -- 0x55f64e20f000 con 0 > -9664> 2018-07-11 08:04:23.146192 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0e800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 181 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:04:24.389600) v4 > -9663> 2018-07-11 08:04:23.146210 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.5 10.8.128.113:0/15263 181 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:24.389600) v4 ==== 2004+0+0 (3632159437 0 0) 0x55f67d1afe00 con 0x55f663f0e800 > -9662> 2018-07-11 08:04:23.146224 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:24.389600) v4 -- 0x55f64cc81c00 con 0 > -9661> 2018-07-11 08:04:23.190128 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.108:0/11967 conn(0x55f663f6f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=1). rx osd.8 seq 172 0x55f650f79200 osd_ping(ping e509 stamp 2018-07-11 08:04:22.949707) v4 > -9660> 2018-07-11 08:04:23.190149 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.8 10.8.128.108:0/11967 172 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:22.949707) v4 ==== 2004+0+0 (2971101472 0 0) 0x55f650f79200 con 0x55f663f6f000 > -9659> 2018-07-11 08:04:23.190159 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:22.949707) v4 -- 0x55f64e25cc00 con 0 > -9658> 2018-07-11 08:04:23.190163 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.108:0/11967 conn(0x55f663f70800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=1). rx osd.8 seq 172 0x55f663c0bc00 osd_ping(ping e509 stamp 2018-07-11 08:04:22.949707) v4 > -9657> 2018-07-11 08:04:23.190186 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.8 10.8.128.108:0/11967 172 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:22.949707) v4 ==== 2004+0+0 (2971101472 0 0) 0x55f663c0bc00 con 0x55f663f70800 > -9656> 2018-07-11 08:04:23.190203 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:22.949707) v4 -- 0x55f64e20f000 con 0 > -9655> 2018-07-11 08:04:23.395106 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15287 conn(0x55f66dce1000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.4 seq 188 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:04:24.638626) v4 > -9654> 2018-07-11 08:04:23.395115 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15287 conn(0x55f66dce2800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.4 seq 188 0x55f65dad4e00 osd_ping(ping e509 stamp 2018-07-11 08:04:24.638626) v4 > -9653> 2018-07-11 08:04:23.395128 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.4 10.8.128.113:0/15287 188 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:24.638626) v4 ==== 2004+0+0 (977564642 0 0) 0x55f64e20fc00 con 0x55f66dce1000 > -9652> 2018-07-11 08:04:23.395132 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.4 10.8.128.113:0/15287 188 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:24.638626) v4 ==== 2004+0+0 (977564642 0 0) 0x55f65dad4e00 con 0x55f66dce2800 > -9651> 2018-07-11 08:04:23.395140 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:24.638626) v4 -- 0x55f64cc81c00 con 0 > -9650> 2018-07-11 08:04:23.395165 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:24.638626) v4 -- 0x55f64e20f000 con 0 > -9649> 2018-07-11 08:04:23.643264 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12651 conn(0x55f663219000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25 cs=1 l=1). rx osd.2 seq 190 0x55f663c0bc00 osd_ping(ping e509 stamp 2018-07-11 08:04:23.699193) v4 > -9648> 2018-07-11 08:04:23.643300 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.2 10.8.128.106:0/12651 190 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:23.699193) v4 ==== 2004+0+0 (2663836270 0 0) 0x55f663c0bc00 con 0x55f663219000 > -9647> 2018-07-11 08:04:23.643322 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:23.699193) v4 -- 0x55f64e20f000 con 0 > -9646> 2018-07-11 08:04:23.643316 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12651 conn(0x55f66321a800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25 cs=1 l=1). rx osd.2 seq 190 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:04:23.699193) v4 > -9645> 2018-07-11 08:04:23.643340 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.2 10.8.128.106:0/12651 190 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:23.699193) v4 ==== 2004+0+0 (2663836270 0 0) 0x55f67d1afe00 con 0x55f66321a800 > -9644> 2018-07-11 08:04:23.643353 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:23.699193) v4 -- 0x55f64cc81c00 con 0 > -9643> 2018-07-11 08:04:24.024225 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 160 0x55f65dad4e00 MOSDScrubReserve(6.15 REQUEST e509) v1 > -9642> 2018-07-11 08:04:24.024266 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 160 ==== MOSDScrubReserve(6.15 REQUEST e509) v1 ==== 43+0+0 (814133767 0 0) 0x55f65dad4e00 con 0x55f66315e800 > -9641> 2018-07-11 08:04:24.024340 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- MOSDScrubReserve(6.15 GRANT e509) v1 -- 0x55f65299b600 con 0 > -9640> 2018-07-11 08:04:24.025033 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 161 0x55f663138780 replica scrub(pg: 6.15,from:0'0,to:430'105930,epoch:509/504,start:6:a8000000::::head,end:6:a800b3e1:::benchmark_data_magna108.ceph.redhat.com_2478_object14545:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -9639> 2018-07-11 08:04:24.025065 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 161 ==== replica scrub(pg: 6.15,from:0'0,to:430'105930,epoch:509/504,start:6:a8000000::::head,end:6:a800b3e1:::benchmark_data_magna108.ceph.redhat.com_2478_object14545:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 190+0+0 (1649757797 0 0) 0x55f663138780 con 0x55f66315e800 > -9638> 2018-07-11 08:04:24.693756 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0a000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 173 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:04:25.328886) v4 > -9637> 2018-07-11 08:04:24.693756 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0b800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 173 0x55f650f78a00 osd_ping(ping e509 stamp 2018-07-11 08:04:25.328886) v4 > -9636> 2018-07-11 08:04:24.693795 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.11 10.8.128.114:0/15919 173 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:25.328886) v4 ==== 2004+0+0 (435020088 0 0) 0x55f64e20fc00 con 0x55f663f0a000 > -9635> 2018-07-11 08:04:24.693798 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.11 10.8.128.114:0/15919 173 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:25.328886) v4 ==== 2004+0+0 (435020088 0 0) 0x55f650f78a00 con 0x55f663f0b800 > -9634> 2018-07-11 08:04:24.693809 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:25.328886) v4 -- 0x55f65e71ce00 con 0 > -9633> 2018-07-11 08:04:24.693836 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:25.328886) v4 -- 0x55f6548fb400 con 0 > -9632> 2018-07-11 08:04:24.886035 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f6539e0b40 con 0 > -9631> 2018-07-11 08:04:24.887598 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 162 0x55f663138a00 replica scrub(pg: 6.15,from:0'0,to:430'106762,epoch:509/504,start:6:a800b3e1:::benchmark_data_magna108.ceph.redhat.com_2478_object14545:0,end:6:a80146d0:::benchmark_data_magna108_35617_object912976:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -9630> 2018-07-11 08:04:24.887630 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 162 ==== replica scrub(pg: 6.15,from:0'0,to:430'106762,epoch:509/504,start:6:a800b3e1:::benchmark_data_magna108.ceph.redhat.com_2478_object14545:0,end:6:a80146d0:::benchmark_data_magna108_35617_object912976:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 232+0+0 (83824670 0 0) 0x55f663138a00 con 0x55f66315e800 > -9629> 2018-07-11 08:04:24.890602 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.108:0/11967 conn(0x55f663f6f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=1). rx osd.8 seq 173 0x55f650f79200 osd_ping(ping e509 stamp 2018-07-11 08:04:24.650155) v4 > -9628> 2018-07-11 08:04:24.890623 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.8 10.8.128.108:0/11967 173 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:24.650155) v4 ==== 2004+0+0 (3509159479 0 0) 0x55f650f79200 con 0x55f663f6f000 > -9627> 2018-07-11 08:04:24.890633 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:24.650155) v4 -- 0x55f6548fb400 con 0 > -9626> 2018-07-11 08:04:24.890638 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.108:0/11967 conn(0x55f663f70800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=1). rx osd.8 seq 173 0x55f663c0bc00 osd_ping(ping e509 stamp 2018-07-11 08:04:24.650155) v4 > -9625> 2018-07-11 08:04:24.890660 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.8 10.8.128.108:0/11967 173 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:24.650155) v4 ==== 2004+0+0 (3509159479 0 0) 0x55f663c0bc00 con 0x55f663f70800 > -9624> 2018-07-11 08:04:24.890671 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:24.650155) v4 -- 0x55f666c47000 con 0 > -9623> 2018-07-11 08:04:24.955045 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 385 0x55f650f78a00 MOSDScrubReserve(11.3s5 REQUEST e509) v1 > -9622> 2018-07-11 08:04:24.955064 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 385 ==== MOSDScrubReserve(11.3s5 REQUEST e509) v1 ==== 43+0+0 (433425070 0 0) 0x55f650f78a00 con 0x55f668a29000 > -9621> 2018-07-11 08:04:24.955104 7f491a040700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e509) v1 -- 0x55f660949800 con 0 > -9620> 2018-07-11 08:04:24.955513 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 386 0x55f660949800 MOSDScrubReserve(11.3s5 RELEASE e509) v1 > -9619> 2018-07-11 08:04:24.955528 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 386 ==== MOSDScrubReserve(11.3s5 RELEASE e509) v1 ==== 43+0+0 (3603460268 0 0) 0x55f660949800 con 0x55f668a29000 > -9618> 2018-07-11 08:04:25.022254 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15885 conn(0x55f663216000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 179 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:04:25.657422) v4 > -9617> 2018-07-11 08:04:25.022263 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15885 conn(0x55f663217800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 179 0x55f650f79200 osd_ping(ping e509 stamp 2018-07-11 08:04:25.657422) v4 > -9616> 2018-07-11 08:04:25.022282 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.7 10.8.128.114:0/15885 179 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:25.657422) v4 ==== 2004+0+0 (199905591 0 0) 0x55f67d1afe00 con 0x55f663216000 > -9615> 2018-07-11 08:04:25.022285 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.7 10.8.128.114:0/15885 179 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:25.657422) v4 ==== 2004+0+0 (199905591 0 0) 0x55f650f79200 con 0x55f663217800 > -9614> 2018-07-11 08:04:25.022298 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:25.657422) v4 -- 0x55f664256000 con 0 > -9613> 2018-07-11 08:04:25.022325 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:25.657422) v4 -- 0x55f662489e00 con 0 > -9612> 2018-07-11 08:04:25.166472 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed7800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 178 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:04:25.222492) v4 > -9611> 2018-07-11 08:04:25.166522 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.1 10.8.128.106:0/12677 178 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:25.222492) v4 ==== 2004+0+0 (3603454860 0 0) 0x55f64e20fc00 con 0x55f663ed7800 > -9610> 2018-07-11 08:04:25.166539 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:25.222492) v4 -- 0x55f664256200 con 0 > -9609> 2018-07-11 08:04:25.166545 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed6000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 178 0x55f64e20f000 osd_ping(ping e509 stamp 2018-07-11 08:04:25.222492) v4 > -9608> 2018-07-11 08:04:25.166571 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.1 10.8.128.106:0/12677 178 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:25.222492) v4 ==== 2004+0+0 (3603454860 0 0) 0x55f64e20f000 con 0x55f663ed6000 > -9607> 2018-07-11 08:04:25.166592 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:25.222492) v4 -- 0x55f64cf24000 con 0 > -9606> 2018-07-11 08:04:25.194085 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0a000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 174 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:04:25.829265) v4 > -9605> 2018-07-11 08:04:25.194112 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.11 10.8.128.114:0/15919 174 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:25.829265) v4 ==== 2004+0+0 (1511411531 0 0) 0x55f67d1afe00 con 0x55f663f0a000 > -9604> 2018-07-11 08:04:25.194110 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0b800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 174 0x55f64cf25a00 osd_ping(ping e509 stamp 2018-07-11 08:04:25.829265) v4 > -9603> 2018-07-11 08:04:25.194130 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:25.829265) v4 -- 0x55f67d1afc00 con 0 > -9602> 2018-07-11 08:04:25.194133 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.11 10.8.128.114:0/15919 174 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:25.829265) v4 ==== 2004+0+0 (1511411531 0 0) 0x55f64cf25a00 con 0x55f663f0b800 > -9601> 2018-07-11 08:04:25.194161 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:25.829265) v4 -- 0x55f6631cd000 con 0 > -9600> 2018-07-11 08:04:25.321409 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12650 conn(0x55f663ed9000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.3 seq 178 0x55f650f79200 osd_ping(ping e509 stamp 2018-07-11 08:04:25.377530) v4 > -9599> 2018-07-11 08:04:25.321413 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12650 conn(0x55f663eda800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.3 seq 178 0x55f663c0bc00 osd_ping(ping e509 stamp 2018-07-11 08:04:25.377530) v4 > -9598> 2018-07-11 08:04:25.321441 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.3 10.8.128.106:0/12650 178 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:25.377530) v4 ==== 2004+0+0 (3673568524 0 0) 0x55f650f79200 con 0x55f663ed9000 > -9597> 2018-07-11 08:04:25.321446 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.3 10.8.128.106:0/12650 178 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:25.377530) v4 ==== 2004+0+0 (3673568524 0 0) 0x55f663c0bc00 con 0x55f663eda800 > -9596> 2018-07-11 08:04:25.321458 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:25.377530) v4 -- 0x55f6631cd000 con 0 > -9595> 2018-07-11 08:04:25.321491 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:25.377530) v4 -- 0x55f64cf24000 con 0 > -9594> 2018-07-11 08:04:25.402002 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.115:0/13323 conn(0x55f663ebe000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.6 seq 170 0x55f64cf25a00 osd_ping(ping e509 stamp 2018-07-11 08:04:25.754059) v4 > -9593> 2018-07-11 08:04:25.402004 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.115:0/13323 conn(0x55f663ebf800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.6 seq 170 0x55f64e20f000 osd_ping(ping e509 stamp 2018-07-11 08:04:25.754059) v4 > -9592> 2018-07-11 08:04:25.402023 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.6 10.8.128.115:0/13323 170 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:25.754059) v4 ==== 2004+0+0 (2542931342 0 0) 0x55f64cf25a00 con 0x55f663ebe000 > -9591> 2018-07-11 08:04:25.402024 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.6 10.8.128.115:0/13323 170 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:25.754059) v4 ==== 2004+0+0 (2542931342 0 0) 0x55f64e20f000 con 0x55f663ebf800 > -9590> 2018-07-11 08:04:25.402049 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:25.754059) v4 -- 0x55f6631cd000 con 0 > -9589> 2018-07-11 08:04:25.402079 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:25.754059) v4 -- 0x55f64cf24000 con 0 > -9588> 2018-07-11 08:04:25.523854 7f491d046700 1 -- 10.8.128.107:6800/14173 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55f662a88000 con 0 > -9587> 2018-07-11 08:04:25.523927 7f491d046700 1 -- 10.8.128.107:6800/14173 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55f64cc77b80 con 0 > -9586> 2018-07-11 08:04:25.653990 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.114:6805/15885 conn(0x55f66315a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.7 seq 2499 0x55f663c0bc00 MOSDScrubReserve(5.4 REQUEST e509) v1 > -9585> 2018-07-11 08:04:25.654016 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.7 10.8.128.114:6805/15885 2499 ==== MOSDScrubReserve(5.4 REQUEST e509) v1 ==== 43+0+0 (3003630469 0 0) 0x55f663c0bc00 con 0x55f66315a000 > -9584> 2018-07-11 08:04:25.654092 7f491983f700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.114:6805/15885 -- MOSDScrubReserve(5.4 REJECT e509) v1 -- 0x55f663fb8800 con 0 > -9583> 2018-07-11 08:04:25.654378 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.114:6805/15885 conn(0x55f66315a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.7 seq 2500 0x55f663fb8800 MOSDScrubReserve(5.4 RELEASE e509) v1 > -9582> 2018-07-11 08:04:25.654399 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.7 10.8.128.114:6805/15885 2500 ==== MOSDScrubReserve(5.4 RELEASE e509) v1 ==== 43+0+0 (2082093447 0 0) 0x55f663fb8800 con 0x55f66315a000 > -9581> 2018-07-11 08:04:25.726606 7f491803c700 5 osd.10 509 heartbeat: osd_stat(788 GB used, 142 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,0,0,1]) > -9580> 2018-07-11 08:04:25.726622 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6802/15255 -- osd_ping(ping e509 stamp 2018-07-11 08:04:25.726620) v4 -- 0x55f64cc80e00 con 0 > -9579> 2018-07-11 08:04:25.726661 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6803/15255 -- osd_ping(ping e509 stamp 2018-07-11 08:04:25.726620) v4 -- 0x55f65299a000 con 0 > -9578> 2018-07-11 08:04:25.726679 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6806/12677 -- osd_ping(ping e509 stamp 2018-07-11 08:04:25.726620) v4 -- 0x55f662489800 con 0 > -9577> 2018-07-11 08:04:25.726694 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6807/12677 -- osd_ping(ping e509 stamp 2018-07-11 08:04:25.726620) v4 -- 0x55f65d024800 con 0 > -9576> 2018-07-11 08:04:25.726707 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6808/12651 -- osd_ping(ping e509 stamp 2018-07-11 08:04:25.726620) v4 -- 0x55f662488200 con 0 > -9575> 2018-07-11 08:04:25.726719 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6810/12651 -- osd_ping(ping e509 stamp 2018-07-11 08:04:25.726620) v4 -- 0x55f6780b2600 con 0 > -9574> 2018-07-11 08:04:25.726731 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6809/12650 -- osd_ping(ping e509 stamp 2018-07-11 08:04:25.726620) v4 -- 0x55f6780b2a00 con 0 > -9573> 2018-07-11 08:04:25.726744 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6811/12650 -- osd_ping(ping e509 stamp 2018-07-11 08:04:25.726620) v4 -- 0x55f681f91e00 con 0 > -9572> 2018-07-11 08:04:25.726763 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6810/15287 -- osd_ping(ping e509 stamp 2018-07-11 08:04:25.726620) v4 -- 0x55f663fb9600 con 0 > -9571> 2018-07-11 08:04:25.726775 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6811/15287 -- osd_ping(ping e509 stamp 2018-07-11 08:04:25.726620) v4 -- 0x55f664257800 con 0 > -9570> 2018-07-11 08:04:25.726789 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6806/15263 -- osd_ping(ping e509 stamp 2018-07-11 08:04:25.726620) v4 -- 0x55f66dc70400 con 0 > -9569> 2018-07-11 08:04:25.726802 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6807/15263 -- osd_ping(ping e509 stamp 2018-07-11 08:04:25.726620) v4 -- 0x55f65e71d800 con 0 > -9568> 2018-07-11 08:04:25.726815 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.115:6802/13323 -- osd_ping(ping e509 stamp 2018-07-11 08:04:25.726620) v4 -- 0x55f656b55c00 con 0 > -9567> 2018-07-11 08:04:25.726828 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.115:6803/13323 -- osd_ping(ping e509 stamp 2018-07-11 08:04:25.726620) v4 -- 0x55f64cf24e00 con 0 > -9566> 2018-07-11 08:04:25.726843 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6806/15885 -- osd_ping(ping e509 stamp 2018-07-11 08:04:25.726620) v4 -- 0x55f64cc81400 con 0 > -9565> 2018-07-11 08:04:25.726856 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6807/15885 -- osd_ping(ping e509 stamp 2018-07-11 08:04:25.726620) v4 -- 0x55f64e20ea00 con 0 > -9564> 2018-07-11 08:04:25.726870 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.108:6802/11967 -- osd_ping(ping e509 stamp 2018-07-11 08:04:25.726620) v4 -- 0x55f66dc70600 con 0 > -9563> 2018-07-11 08:04:25.726882 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.108:6803/11967 -- osd_ping(ping e509 stamp 2018-07-11 08:04:25.726620) v4 -- 0x55f662488e00 con 0 > -9562> 2018-07-11 08:04:25.726894 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.107:6806/12617 -- osd_ping(ping e509 stamp 2018-07-11 08:04:25.726620) v4 -- 0x55f664256c00 con 0 > -9561> 2018-07-11 08:04:25.726907 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.107:6807/12617 -- osd_ping(ping e509 stamp 2018-07-11 08:04:25.726620) v4 -- 0x55f667d08a00 con 0 > -9560> 2018-07-11 08:04:25.726919 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6802/15919 -- osd_ping(ping e509 stamp 2018-07-11 08:04:25.726620) v4 -- 0x55f650f79400 con 0 > -9559> 2018-07-11 08:04:25.726931 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6803/15919 -- osd_ping(ping e509 stamp 2018-07-11 08:04:25.726620) v4 -- 0x55f650f79000 con 0 > -9558> 2018-07-11 08:04:25.727094 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6806/12677 conn(0x55f64e3d8000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.1 seq 178 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:25.726620) v4 > -9557> 2018-07-11 08:04:25.727106 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.107:6806/12617 conn(0x55f656ade000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.9 seq 178 0x55f64cf25a00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:25.726620) v4 > -9556> 2018-07-11 08:04:25.727096 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6808/12651 conn(0x55f64e3ea000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.2 seq 178 0x55f64e20f000 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:25.726620) v4 > -9555> 2018-07-11 08:04:25.727117 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.1 10.8.128.106:6806/12677 178 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:25.726620) v4 ==== 2004+0+0 (2278408662 0 0) 0x55f64e20fc00 con 0x55f64e3d8000 > -9554> 2018-07-11 08:04:25.727121 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.9 10.8.128.107:6806/12617 178 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:25.726620) v4 ==== 2004+0+0 (2278408662 0 0) 0x55f64cf25a00 con 0x55f656ade000 > -9553> 2018-07-11 08:04:25.727123 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.2 10.8.128.106:6808/12651 178 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:25.726620) v4 ==== 2004+0+0 (2278408662 0 0) 0x55f64e20f000 con 0x55f64e3ea000 > -9552> 2018-07-11 08:04:25.727146 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6809/12650 conn(0x55f656ad7000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.3 seq 178 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:25.726620) v4 > -9551> 2018-07-11 08:04:25.727158 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.3 10.8.128.106:6809/12650 178 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:25.726620) v4 ==== 2004+0+0 (2278408662 0 0) 0x55f64e20fc00 con 0x55f656ad7000 > -9550> 2018-07-11 08:04:25.727181 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6807/12677 conn(0x55f64e3d9800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.1 seq 178 0x55f64e20f000 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:25.726620) v4 > -9549> 2018-07-11 08:04:25.727191 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6803/15255 conn(0x55f656ad5800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 178 0x55f64cf25a00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:25.726620) v4 > -9548> 2018-07-11 08:04:25.727199 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.1 10.8.128.106:6807/12677 178 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:25.726620) v4 ==== 2004+0+0 (2278408662 0 0) 0x55f64e20f000 con 0x55f64e3d9800 > -9547> 2018-07-11 08:04:25.727200 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.107:6807/12617 conn(0x55f656adf800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.9 seq 178 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:25.726620) v4 > -9546> 2018-07-11 08:04:25.727207 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.0 10.8.128.113:6803/15255 178 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:25.726620) v4 ==== 2004+0+0 (2278408662 0 0) 0x55f64cf25a00 con 0x55f656ad5800 > -9545> 2018-07-11 08:04:25.727213 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.9 10.8.128.107:6807/12617 178 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:25.726620) v4 ==== 2004+0+0 (2278408662 0 0) 0x55f64e20fc00 con 0x55f656adf800 > -9544> 2018-07-11 08:04:25.727231 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6811/12650 conn(0x55f656ad8800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.3 seq 178 0x55f64e20f000 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:25.726620) v4 > -9543> 2018-07-11 08:04:25.727243 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.3 10.8.128.106:6811/12650 178 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:25.726620) v4 ==== 2004+0+0 (2278408662 0 0) 0x55f64e20f000 con 0x55f656ad8800 > -9542> 2018-07-11 08:04:25.727240 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6810/12651 conn(0x55f64e3eb800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.2 seq 178 0x55f64cf25a00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:25.726620) v4 > -9541> 2018-07-11 08:04:25.727270 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.2 10.8.128.106:6810/12651 178 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:25.726620) v4 ==== 2004+0+0 (2278408662 0 0) 0x55f64cf25a00 con 0x55f64e3eb800 > -9540> 2018-07-11 08:04:25.727301 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6802/15255 conn(0x55f656ad4000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 178 0x55f64e20f000 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:25.726620) v4 > -9539> 2018-07-11 08:04:25.727304 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6811/15287 conn(0x55f64e402800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.4 seq 178 0x55f64cf25a00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:25.726620) v4 > -9538> 2018-07-11 08:04:25.727317 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.0 10.8.128.113:6802/15255 178 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:25.726620) v4 ==== 2004+0+0 (2278408662 0 0) 0x55f64e20f000 con 0x55f656ad4000 > -9537> 2018-07-11 08:04:25.727322 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.4 10.8.128.113:6811/15287 178 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:25.726620) v4 ==== 2004+0+0 (2278408662 0 0) 0x55f64cf25a00 con 0x55f64e402800 > -9536> 2018-07-11 08:04:25.727348 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6810/15287 conn(0x55f64e401000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.4 seq 178 0x55f64e20f000 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:25.726620) v4 > -9535> 2018-07-11 08:04:25.727354 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.115:6802/13323 conn(0x55f656acd000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.6 seq 178 0x55f64cf25a00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:25.726620) v4 > -9534> 2018-07-11 08:04:25.727361 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.4 10.8.128.113:6810/15287 178 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:25.726620) v4 ==== 2004+0+0 (2278408662 0 0) 0x55f64e20f000 con 0x55f64e401000 > -9533> 2018-07-11 08:04:25.727367 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.6 10.8.128.115:6802/13323 178 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:25.726620) v4 ==== 2004+0+0 (2278408662 0 0) 0x55f64cf25a00 con 0x55f656acd000 > -9532> 2018-07-11 08:04:25.727336 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.115:6803/13323 conn(0x55f656ace800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.6 seq 178 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:25.726620) v4 > -9531> 2018-07-11 08:04:25.727378 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.6 10.8.128.115:6803/13323 178 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:25.726620) v4 ==== 2004+0+0 (2278408662 0 0) 0x55f64e20fc00 con 0x55f656ace800 > -9530> 2018-07-11 08:04:25.727402 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6807/15263 conn(0x55f64e3ee800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.5 seq 178 0x55f64e20f000 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:25.726620) v4 > -9529> 2018-07-11 08:04:25.727399 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6802/15919 conn(0x55f64e3db000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.11 seq 178 0x55f64cf25a00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:25.726620) v4 > -9528> 2018-07-11 08:04:25.727413 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.5 10.8.128.113:6807/15263 178 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:25.726620) v4 ==== 2004+0+0 (2278408662 0 0) 0x55f64e20f000 con 0x55f64e3ee800 > -9527> 2018-07-11 08:04:25.727419 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.11 10.8.128.114:6802/15919 178 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:25.726620) v4 ==== 2004+0+0 (2278408662 0 0) 0x55f64cf25a00 con 0x55f64e3db000 > -9526> 2018-07-11 08:04:25.727439 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6806/15263 conn(0x55f64e3ed000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.5 seq 178 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:25.726620) v4 > -9525> 2018-07-11 08:04:25.727454 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6806/15885 conn(0x55f64e3fe000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.7 seq 178 0x55f64cf25a00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:25.726620) v4 > -9524> 2018-07-11 08:04:25.727459 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.5 10.8.128.113:6806/15263 178 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:25.726620) v4 ==== 2004+0+0 (2278408662 0 0) 0x55f64e20fc00 con 0x55f64e3ed000 > -9523> 2018-07-11 08:04:25.727466 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.7 10.8.128.114:6806/15885 178 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:25.726620) v4 ==== 2004+0+0 (2278408662 0 0) 0x55f64cf25a00 con 0x55f64e3fe000 > -9522> 2018-07-11 08:04:25.727511 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.108:6803/11967 conn(0x55f656acb800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.8 seq 178 0x55f64e20f000 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:25.726620) v4 > -9521> 2018-07-11 08:04:25.727511 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6807/15885 conn(0x55f64e3ff800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.7 seq 178 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:25.726620) v4 > -9520> 2018-07-11 08:04:25.727525 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.8 10.8.128.108:6803/11967 178 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:25.726620) v4 ==== 2004+0+0 (2278408662 0 0) 0x55f64e20f000 con 0x55f656acb800 > -9519> 2018-07-11 08:04:25.727527 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.7 10.8.128.114:6807/15885 178 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:25.726620) v4 ==== 2004+0+0 (2278408662 0 0) 0x55f64e20fc00 con 0x55f64e3ff800 > -9518> 2018-07-11 08:04:25.727580 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.108:6802/11967 conn(0x55f656aca000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.8 seq 178 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:25.726620) v4 > -9517> 2018-07-11 08:04:25.727599 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.8 10.8.128.108:6802/11967 178 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:25.726620) v4 ==== 2004+0+0 (2278408662 0 0) 0x55f64e20fc00 con 0x55f656aca000 > -9516> 2018-07-11 08:04:25.727650 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6803/15919 conn(0x55f64e3dc800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.11 seq 178 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:25.726620) v4 > -9515> 2018-07-11 08:04:25.727669 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.11 10.8.128.114:6803/15919 178 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:25.726620) v4 ==== 2004+0+0 (2278408662 0 0) 0x55f64e20fc00 con 0x55f64e3dc800 > -9514> 2018-07-11 08:04:26.053836 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 28] Generated table #37027: 2638241 keys, 74546152 bytes > -9513> 2018-07-11 08:04:26.053873 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296266053857, "cf_name": "default", "job": 28, "event": "table_file_creation", "file_number": 37027, "file_size": 74546152, "table_properties": {"data_size": 67112123, "index_size": 689856, "filter_size": 6743183, "raw_key_size": 76215859, "raw_average_key_size": 28, "raw_value_size": 31365762, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} > -9512> 2018-07-11 08:04:26.122743 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15885 conn(0x55f663217800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 180 0x55f64cf25a00 osd_ping(ping e509 stamp 2018-07-11 08:04:26.757885) v4 > -9511> 2018-07-11 08:04:26.122765 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.7 10.8.128.114:0/15885 180 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:26.757885) v4 ==== 2004+0+0 (145455369 0 0) 0x55f64cf25a00 con 0x55f663217800 > -9510> 2018-07-11 08:04:26.122778 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:26.757885) v4 -- 0x55f6631cd000 con 0 > -9509> 2018-07-11 08:04:26.122786 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15885 conn(0x55f663216000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 180 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:04:26.757885) v4 > -9508> 2018-07-11 08:04:26.122808 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.7 10.8.128.114:0/15885 180 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:26.757885) v4 ==== 2004+0+0 (145455369 0 0) 0x55f64e20fc00 con 0x55f663216000 > -9507> 2018-07-11 08:04:26.122821 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:26.757885) v4 -- 0x55f66dc70400 con 0 > -9506> 2018-07-11 08:04:26.421802 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12650 conn(0x55f663eda800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.3 seq 179 0x55f64e20f000 osd_ping(ping e509 stamp 2018-07-11 08:04:26.477936) v4 > -9505> 2018-07-11 08:04:26.421832 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.3 10.8.128.106:0/12650 179 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:26.477936) v4 ==== 2004+0+0 (3884735371 0 0) 0x55f64e20f000 con 0x55f663eda800 > -9504> 2018-07-11 08:04:26.421842 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:26.477936) v4 -- 0x55f64cc80e00 con 0 > -9503> 2018-07-11 08:04:26.421851 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12650 conn(0x55f663ed9000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.3 seq 179 0x55f650f79200 osd_ping(ping e509 stamp 2018-07-11 08:04:26.477936) v4 > -9502> 2018-07-11 08:04:26.421873 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.3 10.8.128.106:0/12650 179 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:26.477936) v4 ==== 2004+0+0 (3884735371 0 0) 0x55f650f79200 con 0x55f663ed9000 > -9501> 2018-07-11 08:04:26.421885 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:26.477936) v4 -- 0x55f6631cd000 con 0 > -9500> 2018-07-11 08:04:26.501234 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f6679fc6c0 con 0 > -9499> 2018-07-11 08:04:26.502412 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.115:0/13323 conn(0x55f663ebf800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.6 seq 171 0x55f64e20f000 osd_ping(ping e509 stamp 2018-07-11 08:04:26.854534) v4 > -9498> 2018-07-11 08:04:26.502439 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.6 10.8.128.115:0/13323 171 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:26.854534) v4 ==== 2004+0+0 (3962244924 0 0) 0x55f64e20f000 con 0x55f663ebf800 > -9497> 2018-07-11 08:04:26.502440 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.115:0/13323 conn(0x55f663ebe000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.6 seq 171 0x55f64cf25a00 osd_ping(ping e509 stamp 2018-07-11 08:04:26.854534) v4 > -9496> 2018-07-11 08:04:26.502457 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:26.854534) v4 -- 0x55f64cc80e00 con 0 > -9495> 2018-07-11 08:04:26.502458 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.6 10.8.128.115:0/13323 171 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:26.854534) v4 ==== 2004+0+0 (3962244924 0 0) 0x55f64cf25a00 con 0x55f663ebe000 > -9494> 2018-07-11 08:04:26.502493 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:26.854534) v4 -- 0x55f6631cd000 con 0 > -9493> 2018-07-11 08:04:26.503088 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 163 0x55f666c90a00 replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:509/504,start:6:a80146d0:::benchmark_data_magna108_35617_object912976:0,end:6:a801e873:::benchmark_data_magna108.ceph.redhat.com_2478_object163693:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -9492> 2018-07-11 08:04:26.503119 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 163 ==== replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:509/504,start:6:a80146d0:::benchmark_data_magna108_35617_object912976:0,end:6:a801e873:::benchmark_data_magna108.ceph.redhat.com_2478_object163693:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (1590913157 0 0) 0x55f666c90a00 con 0x55f66315e800 > -9491> 2018-07-11 08:04:26.922260 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12650 conn(0x55f663eda800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.3 seq 180 0x55f64e20f000 osd_ping(ping e509 stamp 2018-07-11 08:04:26.978360) v4 > -9490> 2018-07-11 08:04:26.922293 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.3 10.8.128.106:0/12650 180 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:26.978360) v4 ==== 2004+0+0 (916748792 0 0) 0x55f64e20f000 con 0x55f663eda800 > -9489> 2018-07-11 08:04:26.922284 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12650 conn(0x55f663ed9000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.3 seq 180 0x55f650f79200 osd_ping(ping e509 stamp 2018-07-11 08:04:26.978360) v4 > -9488> 2018-07-11 08:04:26.922309 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:26.978360) v4 -- 0x55f64cc80e00 con 0 > -9487> 2018-07-11 08:04:26.922312 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.3 10.8.128.106:0/12650 180 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:26.978360) v4 ==== 2004+0+0 (916748792 0 0) 0x55f650f79200 con 0x55f663ed9000 > -9486> 2018-07-11 08:04:26.922344 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:26.978360) v4 -- 0x55f6631cd000 con 0 > -9485> 2018-07-11 08:04:26.974363 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.107:0/12617 conn(0x55f64e3c3800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.9 seq 179 0x55f64cf25a00 osd_ping(ping e509 stamp 2018-07-11 08:04:26.973838) v4 > -9484> 2018-07-11 08:04:26.974384 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.9 10.8.128.107:0/12617 179 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:26.973838) v4 ==== 2004+0+0 (438155791 0 0) 0x55f64cf25a00 con 0x55f64e3c3800 > -9483> 2018-07-11 08:04:26.974377 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.107:0/12617 conn(0x55f64e3c5000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.9 seq 179 0x55f64cf24000 osd_ping(ping e509 stamp 2018-07-11 08:04:26.973838) v4 > -9482> 2018-07-11 08:04:26.974393 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:26.973838) v4 -- 0x55f6631cd000 con 0 > -9481> 2018-07-11 08:04:26.974405 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.9 10.8.128.107:0/12617 179 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:26.973838) v4 ==== 2004+0+0 (438155791 0 0) 0x55f64cf24000 con 0x55f64e3c5000 > -9480> 2018-07-11 08:04:26.974452 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:26.973838) v4 -- 0x55f64cc80e00 con 0 > -9479> 2018-07-11 08:04:27.167931 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15255 conn(0x55f663ec1000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.0 seq 173 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:04:28.411418) v4 > -9478> 2018-07-11 08:04:27.167934 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15255 conn(0x55f663ec2800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.0 seq 173 0x55f650f79200 osd_ping(ping e509 stamp 2018-07-11 08:04:28.411418) v4 > -9477> 2018-07-11 08:04:27.167957 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.0 10.8.128.113:0/15255 173 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:28.411418) v4 ==== 2004+0+0 (987847064 0 0) 0x55f67d1afe00 con 0x55f663ec1000 > -9476> 2018-07-11 08:04:27.167961 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.0 10.8.128.113:0/15255 173 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:28.411418) v4 ==== 2004+0+0 (987847064 0 0) 0x55f650f79200 con 0x55f663ec2800 > -9475> 2018-07-11 08:04:27.167968 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:28.411418) v4 -- 0x55f6780b2a00 con 0 > -9474> 2018-07-11 08:04:27.167989 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:28.411418) v4 -- 0x55f663fb8000 con 0 > -9473> 2018-07-11 08:04:27.246483 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0e800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 182 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:04:28.490144) v4 > -9472> 2018-07-11 08:04:27.246514 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.5 10.8.128.113:0/15263 182 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:28.490144) v4 ==== 2004+0+0 (107756293 0 0) 0x55f64e20fc00 con 0x55f663f0e800 > -9471> 2018-07-11 08:04:27.246532 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:28.490144) v4 -- 0x55f662489800 con 0 > -9470> 2018-07-11 08:04:27.246583 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0d000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 182 0x55f64e20f000 osd_ping(ping e509 stamp 2018-07-11 08:04:28.490144) v4 > -9469> 2018-07-11 08:04:27.246602 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.5 10.8.128.113:0/15263 182 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:28.490144) v4 ==== 2004+0+0 (107756293 0 0) 0x55f64e20f000 con 0x55f663f0d000 > -9468> 2018-07-11 08:04:27.246618 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:28.490144) v4 -- 0x55f64e25ca00 con 0 > -9467> 2018-07-11 08:04:27.271156 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f65d4d6000 con 0 > -9466> 2018-07-11 08:04:27.272911 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 164 0x55f663138000 replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:509/504,start:6:a801e873:::benchmark_data_magna108.ceph.redhat.com_2478_object163693:0,end:6:a802d253:::benchmark_data_magna108_35617_object564223:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -9465> 2018-07-11 08:04:27.272940 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 164 ==== replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:509/504,start:6:a801e873:::benchmark_data_magna108.ceph.redhat.com_2478_object163693:0,end:6:a802d253:::benchmark_data_magna108_35617_object564223:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (3723872566 0 0) 0x55f663138000 con 0x55f66315e800 > -9464> 2018-07-11 08:04:27.466884 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed7800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 179 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:04:27.523025) v4 > -9463> 2018-07-11 08:04:27.466897 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed6000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 179 0x55f64cf24000 osd_ping(ping e509 stamp 2018-07-11 08:04:27.523025) v4 > -9462> 2018-07-11 08:04:27.466909 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.1 10.8.128.106:0/12677 179 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:27.523025) v4 ==== 2004+0+0 (2562649313 0 0) 0x55f67d1afe00 con 0x55f663ed7800 > -9461> 2018-07-11 08:04:27.466923 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.1 10.8.128.106:0/12677 179 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:27.523025) v4 ==== 2004+0+0 (2562649313 0 0) 0x55f64cf24000 con 0x55f663ed6000 > -9460> 2018-07-11 08:04:27.466926 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:27.523025) v4 -- 0x55f662489800 con 0 > -9459> 2018-07-11 08:04:27.466966 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:27.523025) v4 -- 0x55f663fb9800 con 0 > -9458> 2018-07-11 08:04:27.654343 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.114:6805/15885 conn(0x55f66315a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.7 seq 2501 0x55f64e25ca00 MOSDScrubReserve(5.4 REQUEST e509) v1 > -9457> 2018-07-11 08:04:27.654367 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.7 10.8.128.114:6805/15885 2501 ==== MOSDScrubReserve(5.4 REQUEST e509) v1 ==== 43+0+0 (3003630469 0 0) 0x55f64e25ca00 con 0x55f66315a000 > -9456> 2018-07-11 08:04:27.654449 7f491983f700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.114:6805/15885 -- MOSDScrubReserve(5.4 REJECT e509) v1 -- 0x55f663fb8800 con 0 > -9455> 2018-07-11 08:04:27.654783 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.114:6805/15885 conn(0x55f66315a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.7 seq 2502 0x55f663fb8800 MOSDScrubReserve(5.4 RELEASE e509) v1 > -9454> 2018-07-11 08:04:27.654802 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.7 10.8.128.114:6805/15885 2502 ==== MOSDScrubReserve(5.4 RELEASE e509) v1 ==== 43+0+0 (2082093447 0 0) 0x55f663fb8800 con 0x55f66315a000 > -9453> 2018-07-11 08:04:27.823301 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15885 conn(0x55f663216000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 181 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:04:28.458488) v4 > -9452> 2018-07-11 08:04:27.823329 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.7 10.8.128.114:0/15885 181 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:28.458488) v4 ==== 2004+0+0 (67928059 0 0) 0x55f64e20fc00 con 0x55f663216000 > -9451> 2018-07-11 08:04:27.823349 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:28.458488) v4 -- 0x55f67d1afc00 con 0 > -9450> 2018-07-11 08:04:27.823381 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15885 conn(0x55f663217800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 181 0x55f64cf25a00 osd_ping(ping e509 stamp 2018-07-11 08:04:28.458488) v4 > -9449> 2018-07-11 08:04:27.823407 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.7 10.8.128.114:0/15885 181 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:28.458488) v4 ==== 2004+0+0 (67928059 0 0) 0x55f64cf25a00 con 0x55f663217800 > -9448> 2018-07-11 08:04:27.823424 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:28.458488) v4 -- 0x55f67d1af400 con 0 > -9447> 2018-07-11 08:04:27.955534 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 387 0x55f650f79200 MOSDScrubReserve(11.3s5 REQUEST e509) v1 > -9446> 2018-07-11 08:04:27.955555 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 387 ==== MOSDScrubReserve(11.3s5 REQUEST e509) v1 ==== 43+0+0 (433425070 0 0) 0x55f650f79200 con 0x55f668a29000 > -9445> 2018-07-11 08:04:27.955598 7f491a040700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e509) v1 -- 0x55f660949800 con 0 > -9444> 2018-07-11 08:04:27.955953 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 388 0x55f65e71d200 MOSDScrubReserve(11.3s5 RELEASE e509) v1 > -9443> 2018-07-11 08:04:27.955982 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 388 ==== MOSDScrubReserve(11.3s5 RELEASE e509) v1 ==== 43+0+0 (3603460268 0 0) 0x55f65e71d200 con 0x55f668a29000 > -9442> 2018-07-11 08:04:28.011288 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f675f546c0 con 0 > -9441> 2018-07-11 08:04:28.012844 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 165 0x55f663139680 replica scrub(pg: 6.15,from:0'0,to:430'106058,epoch:509/504,start:6:a802d253:::benchmark_data_magna108_35617_object564223:0,end:6:a8038cf9:::benchmark_data_magna108_35617_object558661:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -9440> 2018-07-11 08:04:28.012875 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 165 ==== replica scrub(pg: 6.15,from:0'0,to:430'106058,epoch:509/504,start:6:a802d253:::benchmark_data_magna108_35617_object564223:0,end:6:a8038cf9:::benchmark_data_magna108_35617_object558661:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (258456477 0 0) 0x55f663139680 con 0x55f66315e800 > -9439> 2018-07-11 08:04:28.074885 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.107:0/12617 conn(0x55f64e3c3800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.9 seq 180 0x55f64cf25a00 osd_ping(ping e509 stamp 2018-07-11 08:04:28.074514) v4 > -9438> 2018-07-11 08:04:28.074906 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.9 10.8.128.107:0/12617 180 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:28.074514) v4 ==== 2004+0+0 (2626850531 0 0) 0x55f64cf25a00 con 0x55f64e3c3800 > -9437> 2018-07-11 08:04:28.074890 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.107:0/12617 conn(0x55f64e3c5000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.9 seq 180 0x55f64cf24000 osd_ping(ping e509 stamp 2018-07-11 08:04:28.074514) v4 > -9436> 2018-07-11 08:04:28.074922 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:28.074514) v4 -- 0x55f664257000 con 0 > -9435> 2018-07-11 08:04:28.074928 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.9 10.8.128.107:0/12617 180 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:28.074514) v4 ==== 2004+0+0 (2626850531 0 0) 0x55f64cf24000 con 0x55f64e3c5000 > -9434> 2018-07-11 08:04:28.074947 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:28.074514) v4 -- 0x55f64cc81800 con 0 > -9433> 2018-07-11 08:04:28.095491 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15287 conn(0x55f66dce2800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.4 seq 189 0x55f64cc80200 osd_ping(ping e509 stamp 2018-07-11 08:04:29.339063) v4 > -9432> 2018-07-11 08:04:28.095513 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.4 10.8.128.113:0/15287 189 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:29.339063) v4 ==== 2004+0+0 (3335101885 0 0) 0x55f64cc80200 con 0x55f66dce2800 > -9431> 2018-07-11 08:04:28.095530 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:29.339063) v4 -- 0x55f64e25c600 con 0 > -9430> 2018-07-11 08:04:28.095532 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15287 conn(0x55f66dce1000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.4 seq 189 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:04:29.339063) v4 > -9429> 2018-07-11 08:04:28.095554 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.4 10.8.128.113:0/15287 189 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:29.339063) v4 ==== 2004+0+0 (3335101885 0 0) 0x55f67d1afe00 con 0x55f66dce1000 > -9428> 2018-07-11 08:04:28.095566 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:29.339063) v4 -- 0x55f65299b800 con 0 > -9427> 2018-07-11 08:04:28.343743 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12651 conn(0x55f663219000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25 cs=1 l=1). rx osd.2 seq 191 0x55f64cf24000 osd_ping(ping e509 stamp 2018-07-11 08:04:28.399762) v4 > -9426> 2018-07-11 08:04:28.343766 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12651 conn(0x55f66321a800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25 cs=1 l=1). rx osd.2 seq 191 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:04:28.399762) v4 > -9425> 2018-07-11 08:04:28.343783 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.2 10.8.128.106:0/12651 191 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:28.399762) v4 ==== 2004+0+0 (3977990218 0 0) 0x55f64e20fc00 con 0x55f66321a800 > -9424> 2018-07-11 08:04:28.343781 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.2 10.8.128.106:0/12651 191 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:28.399762) v4 ==== 2004+0+0 (3977990218 0 0) 0x55f64cf24000 con 0x55f663219000 > -9423> 2018-07-11 08:04:28.343797 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:28.399762) v4 -- 0x55f650f79600 con 0 > -9422> 2018-07-11 08:04:28.343831 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:28.399762) v4 -- 0x55f666c46e00 con 0 > -9421> 2018-07-11 08:04:28.391044 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.108:0/11967 conn(0x55f663f6f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=1). rx osd.8 seq 174 0x55f660949800 osd_ping(ping e509 stamp 2018-07-11 08:04:28.150619) v4 > -9420> 2018-07-11 08:04:28.391065 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.8 10.8.128.108:0/11967 174 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:28.150619) v4 ==== 2004+0+0 (1516464077 0 0) 0x55f660949800 con 0x55f663f6f000 > -9419> 2018-07-11 08:04:28.391075 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:28.150619) v4 -- 0x55f668a30000 con 0 > -9418> 2018-07-11 08:04:28.391082 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.108:0/11967 conn(0x55f663f70800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=1). rx osd.8 seq 174 0x55f64cc80200 osd_ping(ping e509 stamp 2018-07-11 08:04:28.150619) v4 > -9417> 2018-07-11 08:04:28.391103 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.8 10.8.128.108:0/11967 174 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:28.150619) v4 ==== 2004+0+0 (1516464077 0 0) 0x55f64cc80200 con 0x55f663f70800 > -9416> 2018-07-11 08:04:28.391115 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:28.150619) v4 -- 0x55f663c0b400 con 0 > -9415> 2018-07-11 08:04:28.567396 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed7800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 180 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:04:28.623430) v4 > -9414> 2018-07-11 08:04:28.567418 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.1 10.8.128.106:0/12677 180 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:28.623430) v4 ==== 2004+0+0 (696439070 0 0) 0x55f67d1afe00 con 0x55f663ed7800 > -9413> 2018-07-11 08:04:28.567429 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:28.623430) v4 -- 0x55f650f79600 con 0 > -9412> 2018-07-11 08:04:28.567433 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed6000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 180 0x55f64cf24000 osd_ping(ping e509 stamp 2018-07-11 08:04:28.623430) v4 > -9411> 2018-07-11 08:04:28.567454 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.1 10.8.128.106:0/12677 180 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:28.623430) v4 ==== 2004+0+0 (696439070 0 0) 0x55f64cf24000 con 0x55f663ed6000 > -9410> 2018-07-11 08:04:28.567466 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:28.623430) v4 -- 0x55f663c0b400 con 0 > -9409> 2018-07-11 08:04:28.654545 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.114:6805/15885 conn(0x55f66315a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.7 seq 2503 0x55f64cc80200 MOSDScrubReserve(5.4 REQUEST e509) v1 > -9408> 2018-07-11 08:04:28.654569 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.7 10.8.128.114:6805/15885 2503 ==== MOSDScrubReserve(5.4 REQUEST e509) v1 ==== 43+0+0 (3003630469 0 0) 0x55f64cc80200 con 0x55f66315a000 > -9407> 2018-07-11 08:04:28.654657 7f491983f700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.114:6805/15885 -- MOSDScrubReserve(5.4 REJECT e509) v1 -- 0x55f663fb8400 con 0 > -9406> 2018-07-11 08:04:28.654968 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.114:6805/15885 conn(0x55f66315a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.7 seq 2504 0x55f663c0b400 MOSDScrubReserve(5.4 RELEASE e509) v1 > -9405> 2018-07-11 08:04:28.655013 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.7 10.8.128.114:6805/15885 2504 ==== MOSDScrubReserve(5.4 RELEASE e509) v1 ==== 43+0+0 (2082093447 0 0) 0x55f663c0b400 con 0x55f66315a000 > -9404> 2018-07-11 08:04:28.936973 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f66b0a26c0 con 0 > -9403> 2018-07-11 08:04:28.938662 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 166 0x55f666c90f00 replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:509/504,start:6:a8038cf9:::benchmark_data_magna108_35617_object558661:0,end:6:a804c105:::benchmark_data_magna108_35617_object15284:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -9402> 2018-07-11 08:04:28.938695 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 166 ==== replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:509/504,start:6:a8038cf9:::benchmark_data_magna108_35617_object558661:0,end:6:a804c105:::benchmark_data_magna108_35617_object15284:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 217+0+0 (506209812 0 0) 0x55f666c90f00 con 0x55f66315e800 > -9401> 2018-07-11 08:04:28.955844 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 389 0x55f64cf25a00 MOSDScrubReserve(11.3s5 REQUEST e509) v1 > -9400> 2018-07-11 08:04:28.955863 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 389 ==== MOSDScrubReserve(11.3s5 REQUEST e509) v1 ==== 43+0+0 (433425070 0 0) 0x55f64cf25a00 con 0x55f668a29000 > -9399> 2018-07-11 08:04:28.955902 7f491a040700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e509) v1 -- 0x55f65e71d200 con 0 > -9398> 2018-07-11 08:04:28.956266 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 390 0x55f6548fba00 MOSDScrubReserve(11.3s5 RELEASE e509) v1 > -9397> 2018-07-11 08:04:28.956297 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 390 ==== MOSDScrubReserve(11.3s5 RELEASE e509) v1 ==== 43+0+0 (3603460268 0 0) 0x55f6548fba00 con 0x55f668a29000 > -9396> 2018-07-11 08:04:29.956132 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 391 0x55f660949800 MOSDScrubReserve(11.3s5 REQUEST e509) v1 > -9395> 2018-07-11 08:04:29.956164 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 391 ==== MOSDScrubReserve(11.3s5 REQUEST e509) v1 ==== 43+0+0 (433425070 0 0) 0x55f660949800 con 0x55f668a29000 > -9394> 2018-07-11 08:04:29.956227 7f491a040700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e509) v1 -- 0x55f6548fba00 con 0 > -9393> 2018-07-11 08:04:29.956571 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 392 0x55f666c47600 MOSDScrubReserve(11.3s5 RELEASE e509) v1 > -9392> 2018-07-11 08:04:29.956586 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 392 ==== MOSDScrubReserve(11.3s5 RELEASE e509) v1 ==== 43+0+0 (3603460268 0 0) 0x55f666c47600 con 0x55f668a29000 > -9391> 2018-07-11 08:04:30.146908 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0e800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 183 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:04:31.390554) v4 > -9390> 2018-07-11 08:04:30.146939 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.5 10.8.128.113:0/15263 183 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:31.390554) v4 ==== 2004+0+0 (3260896616 0 0) 0x55f64e20fc00 con 0x55f663f0e800 > -9389> 2018-07-11 08:04:30.146945 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0d000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 183 0x55f64cf24000 osd_ping(ping e509 stamp 2018-07-11 08:04:31.390554) v4 > -9388> 2018-07-11 08:04:30.146960 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:31.390554) v4 -- 0x55f65d024c00 con 0 > -9387> 2018-07-11 08:04:30.146972 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.5 10.8.128.113:0/15263 183 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:31.390554) v4 ==== 2004+0+0 (3260896616 0 0) 0x55f64cf24000 con 0x55f663f0d000 > -9386> 2018-07-11 08:04:30.146989 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:31.390554) v4 -- 0x55f663193800 con 0 > -9385> 2018-07-11 08:04:30.375384 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.107:0/12617 conn(0x55f64e3c5000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.9 seq 181 0x55f664255200 osd_ping(ping e509 stamp 2018-07-11 08:04:30.374999) v4 > -9384> 2018-07-11 08:04:30.375402 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.9 10.8.128.107:0/12617 181 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:30.374999) v4 ==== 2004+0+0 (3992509554 0 0) 0x55f664255200 con 0x55f64e3c5000 > -9383> 2018-07-11 08:04:30.375406 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.107:0/12617 conn(0x55f64e3c3800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.9 seq 181 0x55f65e71d200 osd_ping(ping e509 stamp 2018-07-11 08:04:30.374999) v4 > -9382> 2018-07-11 08:04:30.375419 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:30.374999) v4 -- 0x55f663193800 con 0 > -9381> 2018-07-11 08:04:30.375428 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.9 10.8.128.107:0/12617 181 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:30.374999) v4 ==== 2004+0+0 (3992509554 0 0) 0x55f65e71d200 con 0x55f64e3c3800 > -9380> 2018-07-11 08:04:30.375451 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:30.374999) v4 -- 0x55f67d1ae600 con 0 > -9379> 2018-07-11 08:04:30.427049 7f491803c700 5 osd.10 509 heartbeat: osd_stat(788 GB used, 142 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,0,0,0,1]) > -9378> 2018-07-11 08:04:30.427062 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6802/15255 -- osd_ping(ping e509 stamp 2018-07-11 08:04:30.427061) v4 -- 0x55f64e20e000 con 0 > -9377> 2018-07-11 08:04:30.427086 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6803/15255 -- osd_ping(ping e509 stamp 2018-07-11 08:04:30.427061) v4 -- 0x55f64f768200 con 0 > -9376> 2018-07-11 08:04:30.427104 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6806/12677 -- osd_ping(ping e509 stamp 2018-07-11 08:04:30.427061) v4 -- 0x55f64f768c00 con 0 > -9375> 2018-07-11 08:04:30.427124 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6807/12677 -- osd_ping(ping e509 stamp 2018-07-11 08:04:30.427061) v4 -- 0x55f666c46c00 con 0 > -9374> 2018-07-11 08:04:30.427137 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6808/12651 -- osd_ping(ping e509 stamp 2018-07-11 08:04:30.427061) v4 -- 0x55f667d0d800 con 0 > -9373> 2018-07-11 08:04:30.427150 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6810/12651 -- osd_ping(ping e509 stamp 2018-07-11 08:04:30.427061) v4 -- 0x55f664257400 con 0 > -9372> 2018-07-11 08:04:30.427168 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6809/12650 -- osd_ping(ping e509 stamp 2018-07-11 08:04:30.427061) v4 -- 0x55f664256a00 con 0 > -9371> 2018-07-11 08:04:30.427183 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6811/12650 -- osd_ping(ping e509 stamp 2018-07-11 08:04:30.427061) v4 -- 0x55f64e20e200 con 0 > -9370> 2018-07-11 08:04:30.427196 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6810/15287 -- osd_ping(ping e509 stamp 2018-07-11 08:04:30.427061) v4 -- 0x55f64e20f200 con 0 > -9369> 2018-07-11 08:04:30.427208 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6811/15287 -- osd_ping(ping e509 stamp 2018-07-11 08:04:30.427061) v4 -- 0x55f64e20f800 con 0 > -9368> 2018-07-11 08:04:30.427224 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6806/15263 -- osd_ping(ping e509 stamp 2018-07-11 08:04:30.427061) v4 -- 0x55f64e20e400 con 0 > -9367> 2018-07-11 08:04:30.427239 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6807/15263 -- osd_ping(ping e509 stamp 2018-07-11 08:04:30.427061) v4 -- 0x55f664254200 con 0 > -9366> 2018-07-11 08:04:30.427264 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.115:6802/13323 -- osd_ping(ping e509 stamp 2018-07-11 08:04:30.427061) v4 -- 0x55f664254c00 con 0 > -9365> 2018-07-11 08:04:30.427277 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.115:6803/13323 -- osd_ping(ping e509 stamp 2018-07-11 08:04:30.427061) v4 -- 0x55f667d09600 con 0 > -9364> 2018-07-11 08:04:30.427291 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6806/15885 -- osd_ping(ping e509 stamp 2018-07-11 08:04:30.427061) v4 -- 0x55f667d08400 con 0 > -9363> 2018-07-11 08:04:30.427303 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6807/15885 -- osd_ping(ping e509 stamp 2018-07-11 08:04:30.427061) v4 -- 0x55f662488a00 con 0 > -9362> 2018-07-11 08:04:30.427316 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.108:6802/11967 -- osd_ping(ping e509 stamp 2018-07-11 08:04:30.427061) v4 -- 0x55f66dceac00 con 0 > -9361> 2018-07-11 08:04:30.427326 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.108:6803/11967 -- osd_ping(ping e509 stamp 2018-07-11 08:04:30.427061) v4 -- 0x55f66dcea400 con 0 > -9360> 2018-07-11 08:04:30.427338 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.107:6806/12617 -- osd_ping(ping e509 stamp 2018-07-11 08:04:30.427061) v4 -- 0x55f653d3c800 con 0 > -9359> 2018-07-11 08:04:30.427351 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.107:6807/12617 -- osd_ping(ping e509 stamp 2018-07-11 08:04:30.427061) v4 -- 0x55f653d3c600 con 0 > -9358> 2018-07-11 08:04:30.427364 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6802/15919 -- osd_ping(ping e509 stamp 2018-07-11 08:04:30.427061) v4 -- 0x55f653d3cc00 con 0 > -9357> 2018-07-11 08:04:30.427375 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6803/15919 -- osd_ping(ping e509 stamp 2018-07-11 08:04:30.427061) v4 -- 0x55f653d3d000 con 0 > -9356> 2018-07-11 08:04:30.427398 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6802/15255 conn(0x55f656ad4000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 179 0x55f664255200 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:30.427061) v4 > -9355> 2018-07-11 08:04:30.427419 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.0 10.8.128.113:6802/15255 179 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:30.427061) v4 ==== 2004+0+0 (648487911 0 0) 0x55f664255200 con 0x55f656ad4000 > -9354> 2018-07-11 08:04:30.427502 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.107:6806/12617 conn(0x55f656ade000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.9 seq 179 0x55f65e71d200 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:30.427061) v4 > -9353> 2018-07-11 08:04:30.427524 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.9 10.8.128.107:6806/12617 179 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:30.427061) v4 ==== 2004+0+0 (648487911 0 0) 0x55f65e71d200 con 0x55f656ade000 > -9352> 2018-07-11 08:04:30.427534 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6806/12677 conn(0x55f64e3d8000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.1 seq 179 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:30.427061) v4 > -9351> 2018-07-11 08:04:30.427552 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.1 10.8.128.106:6806/12677 179 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:30.427061) v4 ==== 2004+0+0 (648487911 0 0) 0x55f64e20fc00 con 0x55f64e3d8000 > -9350> 2018-07-11 08:04:30.427569 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6803/15255 conn(0x55f656ad5800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 179 0x55f65e71d200 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:30.427061) v4 > -9349> 2018-07-11 08:04:30.427585 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6808/12651 conn(0x55f64e3ea000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.2 seq 179 0x55f664255200 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:30.427061) v4 > -9348> 2018-07-11 08:04:30.427595 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6809/12650 conn(0x55f656ad7000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.3 seq 179 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:30.427061) v4 > -9347> 2018-07-11 08:04:30.427598 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.0 10.8.128.113:6803/15255 179 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:30.427061) v4 ==== 2004+0+0 (648487911 0 0) 0x55f65e71d200 con 0x55f656ad5800 > -9346> 2018-07-11 08:04:30.427610 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.2 10.8.128.106:6808/12651 179 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:30.427061) v4 ==== 2004+0+0 (648487911 0 0) 0x55f664255200 con 0x55f64e3ea000 > -9345> 2018-07-11 08:04:30.427609 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.3 10.8.128.106:6809/12650 179 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:30.427061) v4 ==== 2004+0+0 (648487911 0 0) 0x55f64e20fc00 con 0x55f656ad7000 > -9344> 2018-07-11 08:04:30.427641 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6810/12651 conn(0x55f64e3eb800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.2 seq 179 0x55f65e71d200 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:30.427061) v4 > -9343> 2018-07-11 08:04:30.427654 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6807/12677 conn(0x55f64e3d9800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.1 seq 179 0x55f664255200 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:30.427061) v4 > -9342> 2018-07-11 08:04:30.427668 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.1 10.8.128.106:6807/12677 179 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:30.427061) v4 ==== 2004+0+0 (648487911 0 0) 0x55f664255200 con 0x55f64e3d9800 > -9341> 2018-07-11 08:04:30.427659 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.2 10.8.128.106:6810/12651 179 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:30.427061) v4 ==== 2004+0+0 (648487911 0 0) 0x55f65e71d200 con 0x55f64e3eb800 > -9340> 2018-07-11 08:04:30.427675 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.107:6807/12617 conn(0x55f656adf800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.9 seq 179 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:30.427061) v4 > -9339> 2018-07-11 08:04:30.427690 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.9 10.8.128.107:6807/12617 179 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:30.427061) v4 ==== 2004+0+0 (648487911 0 0) 0x55f64e20fc00 con 0x55f656adf800 > -9338> 2018-07-11 08:04:30.427718 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6810/15287 conn(0x55f64e401000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.4 seq 179 0x55f664255200 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:30.427061) v4 > -9337> 2018-07-11 08:04:30.427719 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6806/15263 conn(0x55f64e3ed000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.5 seq 179 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:30.427061) v4 > -9336> 2018-07-11 08:04:30.427715 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6811/15287 conn(0x55f64e402800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.4 seq 179 0x55f65e71d200 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:30.427061) v4 > -9335> 2018-07-11 08:04:30.427737 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.4 10.8.128.113:6810/15287 179 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:30.427061) v4 ==== 2004+0+0 (648487911 0 0) 0x55f664255200 con 0x55f64e401000 > -9334> 2018-07-11 08:04:30.427740 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.5 10.8.128.113:6806/15263 179 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:30.427061) v4 ==== 2004+0+0 (648487911 0 0) 0x55f64e20fc00 con 0x55f64e3ed000 > -9333> 2018-07-11 08:04:30.427745 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.4 10.8.128.113:6811/15287 179 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:30.427061) v4 ==== 2004+0+0 (648487911 0 0) 0x55f65e71d200 con 0x55f64e402800 > -9332> 2018-07-11 08:04:30.427767 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6811/12650 conn(0x55f656ad8800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.3 seq 179 0x55f664255200 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:30.427061) v4 > -9331> 2018-07-11 08:04:30.427779 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.3 10.8.128.106:6811/12650 179 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:30.427061) v4 ==== 2004+0+0 (648487911 0 0) 0x55f664255200 con 0x55f656ad8800 > -9330> 2018-07-11 08:04:30.427783 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.115:6803/13323 conn(0x55f656ace800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.6 seq 179 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:30.427061) v4 > -9329> 2018-07-11 08:04:30.427790 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.115:6802/13323 conn(0x55f656acd000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.6 seq 179 0x55f65e71d200 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:30.427061) v4 > -9328> 2018-07-11 08:04:30.427800 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.6 10.8.128.115:6803/13323 179 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:30.427061) v4 ==== 2004+0+0 (648487911 0 0) 0x55f64e20fc00 con 0x55f656ace800 > -9327> 2018-07-11 08:04:30.427806 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.6 10.8.128.115:6802/13323 179 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:30.427061) v4 ==== 2004+0+0 (648487911 0 0) 0x55f65e71d200 con 0x55f656acd000 > -9326> 2018-07-11 08:04:30.427812 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6807/15263 conn(0x55f64e3ee800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.5 seq 179 0x55f664255200 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:30.427061) v4 > -9325> 2018-07-11 08:04:30.427836 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.5 10.8.128.113:6807/15263 179 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:30.427061) v4 ==== 2004+0+0 (648487911 0 0) 0x55f664255200 con 0x55f64e3ee800 > -9324> 2018-07-11 08:04:30.427845 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6802/15919 conn(0x55f64e3db000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.11 seq 179 0x55f65e71d200 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:30.427061) v4 > -9323> 2018-07-11 08:04:30.427858 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.11 10.8.128.114:6802/15919 179 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:30.427061) v4 ==== 2004+0+0 (648487911 0 0) 0x55f65e71d200 con 0x55f64e3db000 > -9322> 2018-07-11 08:04:30.427885 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6807/15885 conn(0x55f64e3ff800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.7 seq 179 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:30.427061) v4 > -9321> 2018-07-11 08:04:30.427896 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6806/15885 conn(0x55f64e3fe000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.7 seq 179 0x55f65e71d200 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:30.427061) v4 > -9320> 2018-07-11 08:04:30.427901 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.108:6803/11967 conn(0x55f656acb800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.8 seq 179 0x55f664255200 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:30.427061) v4 > -9319> 2018-07-11 08:04:30.427905 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.7 10.8.128.114:6807/15885 179 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:30.427061) v4 ==== 2004+0+0 (648487911 0 0) 0x55f64e20fc00 con 0x55f64e3ff800 > -9318> 2018-07-11 08:04:30.427907 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.7 10.8.128.114:6806/15885 179 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:30.427061) v4 ==== 2004+0+0 (648487911 0 0) 0x55f65e71d200 con 0x55f64e3fe000 > -9317> 2018-07-11 08:04:30.427920 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.8 10.8.128.108:6803/11967 179 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:30.427061) v4 ==== 2004+0+0 (648487911 0 0) 0x55f664255200 con 0x55f656acb800 > -9316> 2018-07-11 08:04:30.427953 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6803/15919 conn(0x55f64e3dc800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.11 seq 179 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:30.427061) v4 > -9315> 2018-07-11 08:04:30.427968 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.11 10.8.128.114:6803/15919 179 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:30.427061) v4 ==== 2004+0+0 (648487911 0 0) 0x55f64e20fc00 con 0x55f64e3dc800 > -9314> 2018-07-11 08:04:30.428009 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.108:6802/11967 conn(0x55f656aca000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.8 seq 179 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:30.427061) v4 > -9313> 2018-07-11 08:04:30.428022 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.8 10.8.128.108:6802/11967 179 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:30.427061) v4 ==== 2004+0+0 (648487911 0 0) 0x55f64e20fc00 con 0x55f656aca000 > -9312> 2018-07-11 08:04:30.432876 7f491e048700 10 monclient: tick > -9311> 2018-07-11 08:04:30.432889 7f491e048700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2018-07-11 08:04:00.432888) > -9310> 2018-07-11 08:04:30.491478 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 28] Generated table #37028: 2638241 keys, 74545909 bytes > -9309> 2018-07-11 08:04:30.491514 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296270491498, "cf_name": "default", "job": 28, "event": "table_file_creation", "file_number": 37028, "file_size": 74545909, "table_properties": {"data_size": 67112145, "index_size": 689591, "filter_size": 6743183, "raw_key_size": 76215846, "raw_average_key_size": 28, "raw_value_size": 31365749, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} > -9308> 2018-07-11 08:04:30.494543 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0b800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 175 0x55f65e71d200 osd_ping(ping e509 stamp 2018-07-11 08:04:31.129712) v4 > -9307> 2018-07-11 08:04:30.494563 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.11 10.8.128.114:0/15919 175 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:31.129712) v4 ==== 2004+0+0 (300658293 0 0) 0x55f65e71d200 con 0x55f663f0b800 > -9306> 2018-07-11 08:04:30.494554 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0a000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 175 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:04:31.129712) v4 > -9305> 2018-07-11 08:04:30.494573 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:31.129712) v4 -- 0x55f664257400 con 0 > -9304> 2018-07-11 08:04:30.494574 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.11 10.8.128.114:0/15919 175 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:31.129712) v4 ==== 2004+0+0 (300658293 0 0) 0x55f64e20fc00 con 0x55f663f0a000 > -9303> 2018-07-11 08:04:30.494599 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:31.129712) v4 -- 0x55f64e20e400 con 0 > -9302> 2018-07-11 08:04:30.524206 7f491d046700 1 -- 10.8.128.107:6800/14173 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55f663151b80 con 0 > -9301> 2018-07-11 08:04:30.524275 7f491d046700 1 -- 10.8.128.107:6800/14173 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55f6631cf8c0 con 0 > -9300> 2018-07-11 08:04:30.534173 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f675fc86c0 con 0 > -9299> 2018-07-11 08:04:30.535755 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 167 0x55f663139900 replica scrub(pg: 6.15,from:0'0,to:430'106848,epoch:509/504,start:6:a804c105:::benchmark_data_magna108_35617_object15284:0,end:6:a8056d39:::benchmark_data_magna108.ceph.redhat.com_2478_object151069:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -9298> 2018-07-11 08:04:30.535786 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 167 ==== replica scrub(pg: 6.15,from:0'0,to:430'106848,epoch:509/504,start:6:a804c105:::benchmark_data_magna108_35617_object15284:0,end:6:a8056d39:::benchmark_data_magna108.ceph.redhat.com_2478_object151069:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 232+0+0 (983376562 0 0) 0x55f663139900 con 0x55f66315e800 > -9297> 2018-07-11 08:04:30.644349 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12651 conn(0x55f663219000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25 cs=1 l=1). rx osd.2 seq 192 0x55f64cf24000 osd_ping(ping e509 stamp 2018-07-11 08:04:30.700329) v4 > -9296> 2018-07-11 08:04:30.644372 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.2 10.8.128.106:0/12651 192 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:30.700329) v4 ==== 2004+0+0 (82711821 0 0) 0x55f64cf24000 con 0x55f663219000 > -9295> 2018-07-11 08:04:30.644381 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:30.700329) v4 -- 0x55f667d0d800 con 0 > -9294> 2018-07-11 08:04:30.644414 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12651 conn(0x55f66321a800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25 cs=1 l=1). rx osd.2 seq 192 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:04:30.700329) v4 > -9293> 2018-07-11 08:04:30.644440 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.2 10.8.128.106:0/12651 192 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:30.700329) v4 ==== 2004+0+0 (82711821 0 0) 0x55f67d1afe00 con 0x55f66321a800 > -9292> 2018-07-11 08:04:30.644456 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:30.700329) v4 -- 0x55f64e20e400 con 0 > -9291> 2018-07-11 08:04:30.654867 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.114:6805/15885 conn(0x55f66315a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.7 seq 2505 0x55f664255200 MOSDScrubReserve(5.4 REQUEST e509) v1 > -9290> 2018-07-11 08:04:30.654886 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.7 10.8.128.114:6805/15885 2505 ==== MOSDScrubReserve(5.4 REQUEST e509) v1 ==== 43+0+0 (3003630469 0 0) 0x55f664255200 con 0x55f66315a000 > -9289> 2018-07-11 08:04:30.654921 7f491983f700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.114:6805/15885 -- MOSDScrubReserve(5.4 REJECT e509) v1 -- 0x55f663c0b400 con 0 > -9288> 2018-07-11 08:04:30.655281 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.114:6805/15885 conn(0x55f66315a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.7 seq 2506 0x55f663c0b400 MOSDScrubReserve(5.4 RELEASE e509) v1 > -9287> 2018-07-11 08:04:30.655301 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.7 10.8.128.114:6805/15885 2506 ==== MOSDScrubReserve(5.4 RELEASE e509) v1 ==== 43+0+0 (2082093447 0 0) 0x55f663c0b400 con 0x55f66315a000 > -9286> 2018-07-11 08:04:31.144943 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12651 conn(0x55f663219000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25 cs=1 l=1). rx osd.2 seq 193 0x55f64cf24000 osd_ping(ping e509 stamp 2018-07-11 08:04:31.200915) v4 > -9285> 2018-07-11 08:04:31.144973 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.2 10.8.128.106:0/12651 193 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:31.200915) v4 ==== 2004+0+0 (2867108774 0 0) 0x55f64cf24000 con 0x55f663219000 > -9284> 2018-07-11 08:04:31.144987 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:31.200915) v4 -- 0x55f663193800 con 0 > -9283> 2018-07-11 08:04:31.145034 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12651 conn(0x55f66321a800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25 cs=1 l=1). rx osd.2 seq 193 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:04:31.200915) v4 > -9282> 2018-07-11 08:04:31.145053 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.2 10.8.128.106:0/12651 193 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:31.200915) v4 ==== 2004+0+0 (2867108774 0 0) 0x55f64e20fc00 con 0x55f66321a800 > -9281> 2018-07-11 08:04:31.145065 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:31.200915) v4 -- 0x55f664256a00 con 0 > -9280> 2018-07-11 08:04:31.202831 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.115:0/13323 conn(0x55f663ebf800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.6 seq 172 0x55f666c46c00 osd_ping(ping e509 stamp 2018-07-11 08:04:31.554980) v4 > -9279> 2018-07-11 08:04:31.202885 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.6 10.8.128.115:0/13323 172 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:31.554980) v4 ==== 2004+0+0 (2392208859 0 0) 0x55f666c46c00 con 0x55f663ebf800 > -9278> 2018-07-11 08:04:31.202909 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:31.554980) v4 -- 0x55f666c47a00 con 0 > -9277> 2018-07-11 08:04:31.202924 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.115:0/13323 conn(0x55f663ebe000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.6 seq 172 0x55f6548fba00 osd_ping(ping e509 stamp 2018-07-11 08:04:31.554980) v4 > -9276> 2018-07-11 08:04:31.202949 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.6 10.8.128.115:0/13323 172 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:31.554980) v4 ==== 2004+0+0 (2392208859 0 0) 0x55f6548fba00 con 0x55f663ebe000 > -9275> 2018-07-11 08:04:31.202962 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:31.554980) v4 -- 0x55f64f768200 con 0 > -9274> 2018-07-11 08:04:31.332073 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f669444900 con 0 > -9273> 2018-07-11 08:04:31.333701 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 168 0x55f663ac1180 replica scrub(pg: 6.15,from:0'0,to:430'106074,epoch:509/504,start:6:a8056d39:::benchmark_data_magna108.ceph.redhat.com_2478_object151069:0,end:6:a8065b98:::benchmark_data_magna108_35617_object96212:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -9272> 2018-07-11 08:04:31.333733 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 168 ==== replica scrub(pg: 6.15,from:0'0,to:430'106074,epoch:509/504,start:6:a8056d39:::benchmark_data_magna108.ceph.redhat.com_2478_object151069:0,end:6:a8065b98:::benchmark_data_magna108_35617_object96212:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 232+0+0 (3715419172 0 0) 0x55f663ac1180 con 0x55f66315e800 > -9271> 2018-07-11 08:04:31.475870 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.107:0/12617 conn(0x55f64e3c3800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.9 seq 182 0x55f65e71d200 osd_ping(ping e509 stamp 2018-07-11 08:04:31.475519) v4 > -9270> 2018-07-11 08:04:31.475876 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.107:0/12617 conn(0x55f64e3c5000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.9 seq 182 0x55f64cf24000 osd_ping(ping e509 stamp 2018-07-11 08:04:31.475519) v4 > -9269> 2018-07-11 08:04:31.475902 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.9 10.8.128.107:0/12617 182 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:31.475519) v4 ==== 2004+0+0 (4059799200 0 0) 0x55f64cf24000 con 0x55f64e3c5000 > -9268> 2018-07-11 08:04:31.475915 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:31.475519) v4 -- 0x55f67d1af200 con 0 > -9267> 2018-07-11 08:04:31.475898 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.9 10.8.128.107:0/12617 182 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:31.475519) v4 ==== 2004+0+0 (4059799200 0 0) 0x55f65e71d200 con 0x55f64e3c3800 > -9266> 2018-07-11 08:04:31.475947 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:31.475519) v4 -- 0x55f64f768200 con 0 > -9265> 2018-07-11 08:04:31.622585 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12650 conn(0x55f663ed9000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.3 seq 181 0x55f6548fba00 osd_ping(ping e509 stamp 2018-07-11 08:04:31.678784) v4 > -9264> 2018-07-11 08:04:31.622598 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12650 conn(0x55f663eda800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.3 seq 181 0x55f666c47a00 osd_ping(ping e509 stamp 2018-07-11 08:04:31.678784) v4 > -9263> 2018-07-11 08:04:31.622611 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.3 10.8.128.106:0/12650 181 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:31.678784) v4 ==== 2004+0+0 (3936174273 0 0) 0x55f6548fba00 con 0x55f663ed9000 > -9262> 2018-07-11 08:04:31.622618 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.3 10.8.128.106:0/12650 181 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:31.678784) v4 ==== 2004+0+0 (3936174273 0 0) 0x55f666c47a00 con 0x55f663eda800 > -9261> 2018-07-11 08:04:31.622627 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:31.678784) v4 -- 0x55f64f768200 con 0 > -9260> 2018-07-11 08:04:31.622659 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:31.678784) v4 -- 0x55f67d1af200 con 0 > -9259> 2018-07-11 08:04:31.655001 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.114:6805/15885 conn(0x55f66315a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.7 seq 2507 0x55f64cf24000 MOSDScrubReserve(5.4 REQUEST e509) v1 > -9258> 2018-07-11 08:04:31.655022 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.7 10.8.128.114:6805/15885 2507 ==== MOSDScrubReserve(5.4 REQUEST e509) v1 ==== 43+0+0 (3003630469 0 0) 0x55f64cf24000 con 0x55f66315a000 > -9257> 2018-07-11 08:04:31.655116 7f491983f700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.114:6805/15885 -- MOSDScrubReserve(5.4 REJECT e509) v1 -- 0x55f66b1cd600 con 0 > -9256> 2018-07-11 08:04:31.655416 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.114:6805/15885 conn(0x55f66315a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.7 seq 2508 0x55f66b1cd600 MOSDScrubReserve(5.4 RELEASE e509) v1 > -9255> 2018-07-11 08:04:31.655434 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.7 10.8.128.114:6805/15885 2508 ==== MOSDScrubReserve(5.4 RELEASE e509) v1 ==== 43+0+0 (2082093447 0 0) 0x55f66b1cd600 con 0x55f66315a000 > -9254> 2018-07-11 08:04:31.992846 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f663c20480 con 0 > -9253> 2018-07-11 08:04:31.994399 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 169 0x55f663ac0c80 replica scrub(pg: 6.15,from:0'0,to:430'105584,epoch:509/504,start:6:a8065b98:::benchmark_data_magna108_35617_object96212:0,end:6:a807e185:::benchmark_data_magna108_35617_object31982:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -9252> 2018-07-11 08:04:31.994425 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 169 ==== replica scrub(pg: 6.15,from:0'0,to:430'105584,epoch:509/504,start:6:a8065b98:::benchmark_data_magna108_35617_object96212:0,end:6:a807e185:::benchmark_data_magna108_35617_object31982:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 216+0+0 (2115376763 0 0) 0x55f663ac0c80 con 0x55f66315e800 > -9251> 2018-07-11 08:04:32.194917 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0a000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 176 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:04:32.830137) v4 > -9250> 2018-07-11 08:04:32.194941 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.11 10.8.128.114:0/15919 176 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:32.830137) v4 ==== 2004+0+0 (2766517143 0 0) 0x55f67d1afe00 con 0x55f663f0a000 > -9249> 2018-07-11 08:04:32.194934 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0b800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 176 0x55f65e71d200 osd_ping(ping e509 stamp 2018-07-11 08:04:32.830137) v4 > -9248> 2018-07-11 08:04:32.194954 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.11 10.8.128.114:0/15919 176 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:32.830137) v4 ==== 2004+0+0 (2766517143 0 0) 0x55f65e71d200 con 0x55f663f0b800 > -9247> 2018-07-11 08:04:32.194955 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:32.830137) v4 -- 0x55f64f768c00 con 0 > -9246> 2018-07-11 08:04:32.194987 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:32.830137) v4 -- 0x55f67d1ae600 con 0 > -9245> 2018-07-11 08:04:32.195983 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15287 conn(0x55f66dce2800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.4 seq 190 0x55f666c47a00 osd_ping(ping e509 stamp 2018-07-11 08:04:33.439632) v4 > -9244> 2018-07-11 08:04:32.196010 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.4 10.8.128.113:0/15287 190 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:33.439632) v4 ==== 2004+0+0 (233481811 0 0) 0x55f666c47a00 con 0x55f66dce2800 > -9243> 2018-07-11 08:04:32.196023 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:33.439632) v4 -- 0x55f65299ae00 con 0 > -9242> 2018-07-11 08:04:32.196096 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15287 conn(0x55f66dce1000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.4 seq 190 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:04:33.439632) v4 > -9241> 2018-07-11 08:04:32.196117 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.4 10.8.128.113:0/15287 190 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:33.439632) v4 ==== 2004+0+0 (233481811 0 0) 0x55f64e20fc00 con 0x55f66dce1000 > -9240> 2018-07-11 08:04:32.196129 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:33.439632) v4 -- 0x55f65d024c00 con 0 > -9239> 2018-07-11 08:04:32.523851 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15885 conn(0x55f663217800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 182 0x55f6548fba00 osd_ping(ping e509 stamp 2018-07-11 08:04:33.159053) v4 > -9238> 2018-07-11 08:04:32.523868 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15885 conn(0x55f663216000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 182 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:04:33.159053) v4 > -9237> 2018-07-11 08:04:32.523880 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.7 10.8.128.114:0/15885 182 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:33.159053) v4 ==== 2004+0+0 (3600430542 0 0) 0x55f6548fba00 con 0x55f663217800 > -9236> 2018-07-11 08:04:32.523892 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.7 10.8.128.114:0/15885 182 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:33.159053) v4 ==== 2004+0+0 (3600430542 0 0) 0x55f67d1afe00 con 0x55f663216000 > -9235> 2018-07-11 08:04:32.523899 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:33.159053) v4 -- 0x55f6780b2200 con 0 > -9234> 2018-07-11 08:04:32.523931 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:33.159053) v4 -- 0x55f65dad4a00 con 0 > -9233> 2018-07-11 08:04:32.695384 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0a000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 177 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:04:33.330593) v4 > -9232> 2018-07-11 08:04:32.695382 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0b800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 177 0x55f65e71d200 osd_ping(ping e509 stamp 2018-07-11 08:04:33.330593) v4 > -9231> 2018-07-11 08:04:32.695411 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.11 10.8.128.114:0/15919 177 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:33.330593) v4 ==== 2004+0+0 (137318867 0 0) 0x55f64e20fc00 con 0x55f663f0a000 > -9230> 2018-07-11 08:04:32.695415 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.11 10.8.128.114:0/15919 177 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:33.330593) v4 ==== 2004+0+0 (137318867 0 0) 0x55f65e71d200 con 0x55f663f0b800 > -9229> 2018-07-11 08:04:32.695425 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:33.330593) v4 -- 0x55f66dceb000 con 0 > -9228> 2018-07-11 08:04:32.695450 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:33.330593) v4 -- 0x55f65299be00 con 0 > -9227> 2018-07-11 08:04:32.725639 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f666c59440 con 0 > -9226> 2018-07-11 08:04:32.727180 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 170 0x55f66805fb80 replica scrub(pg: 6.15,from:0'0,to:430'106311,epoch:509/504,start:6:a807e185:::benchmark_data_magna108_35617_object31982:0,end:6:a808c512:::benchmark_data_magna108_35617_object1061269:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -9225> 2018-07-11 08:04:32.727210 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 170 ==== replica scrub(pg: 6.15,from:0'0,to:430'106311,epoch:509/504,start:6:a807e185:::benchmark_data_magna108_35617_object31982:0,end:6:a808c512:::benchmark_data_magna108_35617_object1061269:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (1956267261 0 0) 0x55f66805fb80 con 0x55f66315e800 > -9224> 2018-07-11 08:04:33.068271 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15255 conn(0x55f663ec2800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.0 seq 174 0x55f6548fba00 osd_ping(ping e509 stamp 2018-07-11 08:04:34.311935) v4 > -9223> 2018-07-11 08:04:33.068298 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.0 10.8.128.113:0/15255 174 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:34.311935) v4 ==== 2004+0+0 (179137319 0 0) 0x55f6548fba00 con 0x55f663ec2800 > -9222> 2018-07-11 08:04:33.068284 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15255 conn(0x55f663ec1000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.0 seq 174 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:04:34.311935) v4 > -9221> 2018-07-11 08:04:33.068309 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:34.311935) v4 -- 0x55f65299be00 con 0 > -9220> 2018-07-11 08:04:33.068309 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.0 10.8.128.113:0/15255 174 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:34.311935) v4 ==== 2004+0+0 (179137319 0 0) 0x55f67d1afe00 con 0x55f663ec1000 > -9219> 2018-07-11 08:04:33.068337 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:34.311935) v4 -- 0x55f66dceb000 con 0 > -9218> 2018-07-11 08:04:33.091475 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.108:0/11967 conn(0x55f663f6f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=1). rx osd.8 seq 175 0x55f65e71d200 osd_ping(ping e509 stamp 2018-07-11 08:04:32.851066) v4 > -9217> 2018-07-11 08:04:33.091496 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.8 10.8.128.108:0/11967 175 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:32.851066) v4 ==== 2004+0+0 (3872392914 0 0) 0x55f65e71d200 con 0x55f663f6f000 > -9216> 2018-07-11 08:04:33.091508 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:32.851066) v4 -- 0x55f6548faa00 con 0 > -9215> 2018-07-11 08:04:33.091527 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.108:0/11967 conn(0x55f663f70800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=1). rx osd.8 seq 175 0x55f67d1aea00 osd_ping(ping e509 stamp 2018-07-11 08:04:32.851066) v4 > -9214> 2018-07-11 08:04:33.091548 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.8 10.8.128.108:0/11967 175 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:32.851066) v4 ==== 2004+0+0 (3872392914 0 0) 0x55f67d1aea00 con 0x55f663f70800 > -9213> 2018-07-11 08:04:33.091565 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:32.851066) v4 -- 0x55f6780b2000 con 0 > -9212> 2018-07-11 08:04:33.508035 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f65374c240 con 0 > -9211> 2018-07-11 08:04:33.509691 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 171 0x55f663168500 replica scrub(pg: 6.15,from:0'0,to:461'106882,epoch:509/504,start:6:a808c512:::benchmark_data_magna108_35617_object1061269:0,end:6:a809bf4e:::benchmark_data_magna108.ceph.redhat.com_2478_object337791:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -9210> 2018-07-11 08:04:33.509724 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 171 ==== replica scrub(pg: 6.15,from:0'0,to:461'106882,epoch:509/504,start:6:a808c512:::benchmark_data_magna108_35617_object1061269:0,end:6:a809bf4e:::benchmark_data_magna108.ceph.redhat.com_2478_object337791:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 234+0+0 (3563795320 0 0) 0x55f663168500 con 0x55f66315e800 > -9209> 2018-07-11 08:04:33.624289 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15885 conn(0x55f663217800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 183 0x55f6548fba00 osd_ping(ping e509 stamp 2018-07-11 08:04:34.259513) v4 > -9208> 2018-07-11 08:04:33.624293 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15885 conn(0x55f663216000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 183 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:04:34.259513) v4 > -9207> 2018-07-11 08:04:33.624321 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.7 10.8.128.114:0/15885 183 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:34.259513) v4 ==== 2004+0+0 (505902890 0 0) 0x55f6548fba00 con 0x55f663217800 > -9206> 2018-07-11 08:04:33.624322 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.7 10.8.128.114:0/15885 183 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:34.259513) v4 ==== 2004+0+0 (505902890 0 0) 0x55f64e20fc00 con 0x55f663216000 > -9205> 2018-07-11 08:04:33.624332 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:34.259513) v4 -- 0x55f6548faa00 con 0 > -9204> 2018-07-11 08:04:33.624358 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:34.259513) v4 -- 0x55f66dceb000 con 0 > -9203> 2018-07-11 08:04:33.655225 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.114:6805/15885 conn(0x55f66315a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.7 seq 2509 0x55f65299ae00 MOSDScrubReserve(5.4 REQUEST e509) v1 > -9202> 2018-07-11 08:04:33.655272 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.7 10.8.128.114:6805/15885 2509 ==== MOSDScrubReserve(5.4 REQUEST e509) v1 ==== 43+0+0 (3003630469 0 0) 0x55f65299ae00 con 0x55f66315a000 > -9201> 2018-07-11 08:04:33.655343 7f491983f700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.114:6805/15885 -- MOSDScrubReserve(5.4 REJECT e509) v1 -- 0x55f66b1cd600 con 0 > -9200> 2018-07-11 08:04:33.655672 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.114:6805/15885 conn(0x55f66315a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.7 seq 2510 0x55f66b1cd600 MOSDScrubReserve(5.4 RELEASE e509) v1 > -9199> 2018-07-11 08:04:33.655691 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.7 10.8.128.114:6805/15885 2510 ==== MOSDScrubReserve(5.4 RELEASE e509) v1 ==== 43+0+0 (2082093447 0 0) 0x55f66b1cd600 con 0x55f66315a000 > -9198> 2018-07-11 08:04:33.956701 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 393 0x55f65e71d200 MOSDScrubReserve(11.3s5 REQUEST e509) v1 > -9197> 2018-07-11 08:04:33.956725 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 393 ==== MOSDScrubReserve(11.3s5 REQUEST e509) v1 ==== 43+0+0 (433425070 0 0) 0x55f65e71d200 con 0x55f668a29000 > -9196> 2018-07-11 08:04:33.956794 7f491a040700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e509) v1 -- 0x55f666c47600 con 0 > -9195> 2018-07-11 08:04:33.957187 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 394 0x55f666c47600 MOSDScrubReserve(11.3s5 RELEASE e509) v1 > -9194> 2018-07-11 08:04:33.957203 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 394 ==== MOSDScrubReserve(11.3s5 RELEASE e509) v1 ==== 43+0+0 (3603460268 0 0) 0x55f666c47600 con 0x55f668a29000 > -9193> 2018-07-11 08:04:34.467904 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed7800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 181 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:04:34.524003) v4 > -9192> 2018-07-11 08:04:34.467933 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.1 10.8.128.106:0/12677 181 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:34.524003) v4 ==== 2004+0+0 (1099068134 0 0) 0x55f67d1afe00 con 0x55f663ed7800 > -9191> 2018-07-11 08:04:34.467944 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:34.524003) v4 -- 0x55f664256600 con 0 > -9190> 2018-07-11 08:04:34.468012 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed6000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 181 0x55f6780b2000 osd_ping(ping e509 stamp 2018-07-11 08:04:34.524003) v4 > -9189> 2018-07-11 08:04:34.468038 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.1 10.8.128.106:0/12677 181 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:34.524003) v4 ==== 2004+0+0 (1099068134 0 0) 0x55f6780b2000 con 0x55f663ed6000 > -9188> 2018-07-11 08:04:34.468056 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:34.524003) v4 -- 0x55f65d024e00 con 0 > -9187> 2018-07-11 08:04:34.655530 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.114:6805/15885 conn(0x55f66315a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.7 seq 2511 0x55f64f768000 MOSDScrubReserve(5.4 REQUEST e509) v1 > -9186> 2018-07-11 08:04:34.655549 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.7 10.8.128.114:6805/15885 2511 ==== MOSDScrubReserve(5.4 REQUEST e509) v1 ==== 43+0+0 (3003630469 0 0) 0x55f64f768000 con 0x55f66315a000 > -9185> 2018-07-11 08:04:34.655622 7f491983f700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.114:6805/15885 -- MOSDScrubReserve(5.4 REJECT e509) v1 -- 0x55f6548fbe00 con 0 > -9184> 2018-07-11 08:04:34.655955 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.114:6805/15885 conn(0x55f66315a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.7 seq 2512 0x55f6548fbe00 MOSDScrubReserve(5.4 RELEASE e509) v1 > -9183> 2018-07-11 08:04:34.655973 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.7 10.8.128.114:6805/15885 2512 ==== MOSDScrubReserve(5.4 RELEASE e509) v1 ==== 43+0+0 (2082093447 0 0) 0x55f6548fbe00 con 0x55f66315a000 > -9182> 2018-07-11 08:04:34.703243 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.115:0/13323 conn(0x55f663ebf800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.6 seq 173 0x55f6780b2000 osd_ping(ping e509 stamp 2018-07-11 08:04:35.055430) v4 > -9181> 2018-07-11 08:04:34.703278 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.6 10.8.128.115:0/13323 173 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:35.055430) v4 ==== 2004+0+0 (3010219797 0 0) 0x55f6780b2000 con 0x55f663ebf800 > -9180> 2018-07-11 08:04:34.703287 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.055430) v4 -- 0x55f65d025000 con 0 > -9179> 2018-07-11 08:04:34.703288 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.115:0/13323 conn(0x55f663ebe000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.6 seq 173 0x55f6548fba00 osd_ping(ping e509 stamp 2018-07-11 08:04:35.055430) v4 > -9178> 2018-07-11 08:04:34.703309 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.6 10.8.128.115:0/13323 173 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:35.055430) v4 ==== 2004+0+0 (3010219797 0 0) 0x55f6548fba00 con 0x55f663ebe000 > -9177> 2018-07-11 08:04:34.703326 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.055430) v4 -- 0x55f663f5e200 con 0 > -9176> 2018-07-11 08:04:34.847256 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0e800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 184 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:04:36.091001) v4 > -9175> 2018-07-11 08:04:34.847283 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.5 10.8.128.113:0/15263 184 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:36.091001) v4 ==== 2004+0+0 (3682611272 0 0) 0x55f64e20fc00 con 0x55f663f0e800 > -9174> 2018-07-11 08:04:34.847298 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:36.091001) v4 -- 0x55f664256600 con 0 > -9173> 2018-07-11 08:04:34.847290 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0d000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 184 0x55f664257c00 osd_ping(ping e509 stamp 2018-07-11 08:04:36.091001) v4 > -9172> 2018-07-11 08:04:34.847311 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.5 10.8.128.113:0/15263 184 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:36.091001) v4 ==== 2004+0+0 (3682611272 0 0) 0x55f664257c00 con 0x55f663f0d000 > -9171> 2018-07-11 08:04:34.847329 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:36.091001) v4 -- 0x55f65d025000 con 0 > -9170> 2018-07-11 08:04:34.956902 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 395 0x55f65299a600 MOSDScrubReserve(11.3s5 REQUEST e509) v1 > -9169> 2018-07-11 08:04:34.956921 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 395 ==== MOSDScrubReserve(11.3s5 REQUEST e509) v1 ==== 43+0+0 (433425070 0 0) 0x55f65299a600 con 0x55f668a29000 > -9168> 2018-07-11 08:04:34.956998 7f491a040700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e509) v1 -- 0x55f666c47600 con 0 > -9167> 2018-07-11 08:04:34.957336 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 396 0x55f666c47600 MOSDScrubReserve(11.3s5 RELEASE e509) v1 > -9166> 2018-07-11 08:04:34.957354 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 396 ==== MOSDScrubReserve(11.3s5 RELEASE e509) v1 ==== 43+0+0 (3603460268 0 0) 0x55f666c47600 con 0x55f668a29000 > -9165> 2018-07-11 08:04:34.976278 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 28] Generated table #37029: 2638240 keys, 74547029 bytes > -9164> 2018-07-11 08:04:34.976325 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296274976297, "cf_name": "default", "job": 28, "event": "table_file_creation", "file_number": 37029, "file_size": 74547029, "table_properties": {"data_size": 67112089, "index_size": 690769, "filter_size": 6743181, "raw_key_size": 76215830, "raw_average_key_size": 28, "raw_value_size": 31365750, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} > -9163> 2018-07-11 08:04:35.096419 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15287 conn(0x55f66dce1000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.4 seq 191 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:04:36.340131) v4 > -9162> 2018-07-11 08:04:35.096446 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.4 10.8.128.113:0/15287 191 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:36.340131) v4 ==== 2004+0+0 (3366010878 0 0) 0x55f67d1afe00 con 0x55f66dce1000 > -9161> 2018-07-11 08:04:35.096444 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15287 conn(0x55f66dce2800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.4 seq 191 0x55f6780b2000 osd_ping(ping e509 stamp 2018-07-11 08:04:36.340131) v4 > -9160> 2018-07-11 08:04:35.096464 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.4 10.8.128.113:0/15287 191 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:36.340131) v4 ==== 2004+0+0 (3366010878 0 0) 0x55f6780b2000 con 0x55f66dce2800 > -9159> 2018-07-11 08:04:35.096466 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:36.340131) v4 -- 0x55f667d09c00 con 0 > -9158> 2018-07-11 08:04:35.096495 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:36.340131) v4 -- 0x55f64cc80400 con 0 > -9157> 2018-07-11 08:04:35.127516 7f491803c700 5 osd.10 509 heartbeat: osd_stat(788 GB used, 142 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,0,0,0,1]) > -9156> 2018-07-11 08:04:35.127531 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6802/15255 -- osd_ping(ping e509 stamp 2018-07-11 08:04:35.127530) v4 -- 0x55f653d3c200 con 0 > -9155> 2018-07-11 08:04:35.127556 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6803/15255 -- osd_ping(ping e509 stamp 2018-07-11 08:04:35.127530) v4 -- 0x55f653d3dc00 con 0 > -9154> 2018-07-11 08:04:35.127573 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6806/12677 -- osd_ping(ping e509 stamp 2018-07-11 08:04:35.127530) v4 -- 0x55f663140a00 con 0 > -9153> 2018-07-11 08:04:35.127588 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6807/12677 -- osd_ping(ping e509 stamp 2018-07-11 08:04:35.127530) v4 -- 0x55f660948e00 con 0 > -9152> 2018-07-11 08:04:35.127601 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6808/12651 -- osd_ping(ping e509 stamp 2018-07-11 08:04:35.127530) v4 -- 0x55f6780b2800 con 0 > -9151> 2018-07-11 08:04:35.127614 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6810/12651 -- osd_ping(ping e509 stamp 2018-07-11 08:04:35.127530) v4 -- 0x55f6780b3200 con 0 > -9150> 2018-07-11 08:04:35.127627 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6809/12650 -- osd_ping(ping e509 stamp 2018-07-11 08:04:35.127530) v4 -- 0x55f64cf25400 con 0 > -9149> 2018-07-11 08:04:35.127645 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6811/12650 -- osd_ping(ping e509 stamp 2018-07-11 08:04:35.127530) v4 -- 0x55f64cf24c00 con 0 > -9148> 2018-07-11 08:04:35.127657 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6810/15287 -- osd_ping(ping e509 stamp 2018-07-11 08:04:35.127530) v4 -- 0x55f64cf25e00 con 0 > -9147> 2018-07-11 08:04:35.127670 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6811/15287 -- osd_ping(ping e509 stamp 2018-07-11 08:04:35.127530) v4 -- 0x55f64cf24800 con 0 > -9146> 2018-07-11 08:04:35.127682 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6806/15263 -- osd_ping(ping e509 stamp 2018-07-11 08:04:35.127530) v4 -- 0x55f65dad4e00 con 0 > -9145> 2018-07-11 08:04:35.127695 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6807/15263 -- osd_ping(ping e509 stamp 2018-07-11 08:04:35.127530) v4 -- 0x55f65dad4000 con 0 > -9144> 2018-07-11 08:04:35.127707 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.115:6802/13323 -- osd_ping(ping e509 stamp 2018-07-11 08:04:35.127530) v4 -- 0x55f663fb8200 con 0 > -9143> 2018-07-11 08:04:35.127720 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.115:6803/13323 -- osd_ping(ping e509 stamp 2018-07-11 08:04:35.127530) v4 -- 0x55f66b1cd600 con 0 > -9142> 2018-07-11 08:04:35.127736 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6806/15885 -- osd_ping(ping e509 stamp 2018-07-11 08:04:35.127530) v4 -- 0x55f65299ae00 con 0 > -9141> 2018-07-11 08:04:35.127749 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6807/15885 -- osd_ping(ping e509 stamp 2018-07-11 08:04:35.127530) v4 -- 0x55f64cf24000 con 0 > -9140> 2018-07-11 08:04:35.127761 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.108:6802/11967 -- osd_ping(ping e509 stamp 2018-07-11 08:04:35.127530) v4 -- 0x55f66b1cc000 con 0 > -9139> 2018-07-11 08:04:35.127773 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.108:6803/11967 -- osd_ping(ping e509 stamp 2018-07-11 08:04:35.127530) v4 -- 0x55f66b1cd000 con 0 > -9138> 2018-07-11 08:04:35.127786 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.107:6806/12617 -- osd_ping(ping e509 stamp 2018-07-11 08:04:35.127530) v4 -- 0x55f66b1cd200 con 0 > -9137> 2018-07-11 08:04:35.127799 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.107:6807/12617 -- osd_ping(ping e509 stamp 2018-07-11 08:04:35.127530) v4 -- 0x55f66b1cda00 con 0 > -9136> 2018-07-11 08:04:35.127810 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6802/15919 -- osd_ping(ping e509 stamp 2018-07-11 08:04:35.127530) v4 -- 0x55f66b1cde00 con 0 > -9135> 2018-07-11 08:04:35.127823 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6803/15919 -- osd_ping(ping e509 stamp 2018-07-11 08:04:35.127530) v4 -- 0x55f66b1cc800 con 0 > -9134> 2018-07-11 08:04:35.127946 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6802/15255 conn(0x55f656ad4000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 180 0x55f664257c00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.127530) v4 > -9133> 2018-07-11 08:04:35.127951 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6803/15255 conn(0x55f656ad5800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 180 0x55f6780b3600 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.127530) v4 > -9132> 2018-07-11 08:04:35.127975 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.0 10.8.128.113:6803/15255 180 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.127530) v4 ==== 2004+0+0 (877076964 0 0) 0x55f6780b3600 con 0x55f656ad5800 > -9131> 2018-07-11 08:04:35.127974 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.0 10.8.128.113:6802/15255 180 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.127530) v4 ==== 2004+0+0 (877076964 0 0) 0x55f664257c00 con 0x55f656ad4000 > -9130> 2018-07-11 08:04:35.128038 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.107:6806/12617 conn(0x55f656ade000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.9 seq 180 0x55f6780b3600 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.127530) v4 > -9129> 2018-07-11 08:04:35.128057 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.9 10.8.128.107:6806/12617 180 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.127530) v4 ==== 2004+0+0 (877076964 0 0) 0x55f6780b3600 con 0x55f656ade000 > -9128> 2018-07-11 08:04:35.128051 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6809/12650 conn(0x55f656ad7000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.3 seq 180 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.127530) v4 > -9127> 2018-07-11 08:04:35.128073 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.3 10.8.128.106:6809/12650 180 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.127530) v4 ==== 2004+0+0 (877076964 0 0) 0x55f64e20fc00 con 0x55f656ad7000 > -9126> 2018-07-11 08:04:35.128126 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6807/12677 conn(0x55f64e3d9800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.1 seq 180 0x55f664257c00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.127530) v4 > -9125> 2018-07-11 08:04:35.128128 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.107:6807/12617 conn(0x55f656adf800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.9 seq 180 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.127530) v4 > -9124> 2018-07-11 08:04:35.128150 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.9 10.8.128.107:6807/12617 180 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.127530) v4 ==== 2004+0+0 (877076964 0 0) 0x55f64e20fc00 con 0x55f656adf800 > -9123> 2018-07-11 08:04:35.128143 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6810/12651 conn(0x55f64e3eb800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.2 seq 180 0x55f6780b3600 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.127530) v4 > -9122> 2018-07-11 08:04:35.128148 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.1 10.8.128.106:6807/12677 180 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.127530) v4 ==== 2004+0+0 (877076964 0 0) 0x55f664257c00 con 0x55f64e3d9800 > -9121> 2018-07-11 08:04:35.128165 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.2 10.8.128.106:6810/12651 180 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.127530) v4 ==== 2004+0+0 (877076964 0 0) 0x55f6780b3600 con 0x55f64e3eb800 > -9120> 2018-07-11 08:04:35.128190 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6806/12677 conn(0x55f64e3d8000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.1 seq 180 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.127530) v4 > -9119> 2018-07-11 08:04:35.128205 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.1 10.8.128.106:6806/12677 180 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.127530) v4 ==== 2004+0+0 (877076964 0 0) 0x55f64e20fc00 con 0x55f64e3d8000 > -9118> 2018-07-11 08:04:35.128208 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6811/12650 conn(0x55f656ad8800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.3 seq 180 0x55f664257c00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.127530) v4 > -9117> 2018-07-11 08:04:35.128218 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6811/15287 conn(0x55f64e402800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.4 seq 180 0x55f6780b3600 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.127530) v4 > -9116> 2018-07-11 08:04:35.128226 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.3 10.8.128.106:6811/12650 180 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.127530) v4 ==== 2004+0+0 (877076964 0 0) 0x55f664257c00 con 0x55f656ad8800 > -9115> 2018-07-11 08:04:35.128238 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.4 10.8.128.113:6811/15287 180 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.127530) v4 ==== 2004+0+0 (877076964 0 0) 0x55f6780b3600 con 0x55f64e402800 > -9114> 2018-07-11 08:04:35.128268 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6806/15263 conn(0x55f64e3ed000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.5 seq 180 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.127530) v4 > -9113> 2018-07-11 08:04:35.128283 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6810/15287 conn(0x55f64e401000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.4 seq 180 0x55f664257c00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.127530) v4 > -9112> 2018-07-11 08:04:35.128289 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.5 10.8.128.113:6806/15263 180 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.127530) v4 ==== 2004+0+0 (877076964 0 0) 0x55f64e20fc00 con 0x55f64e3ed000 > -9111> 2018-07-11 08:04:35.128297 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.4 10.8.128.113:6810/15287 180 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.127530) v4 ==== 2004+0+0 (877076964 0 0) 0x55f664257c00 con 0x55f64e401000 > -9110> 2018-07-11 08:04:35.128326 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6802/15919 conn(0x55f64e3db000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.11 seq 180 0x55f6780b3600 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.127530) v4 > -9109> 2018-07-11 08:04:35.128337 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.115:6803/13323 conn(0x55f656ace800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.6 seq 180 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.127530) v4 > -9108> 2018-07-11 08:04:35.128345 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.11 10.8.128.114:6802/15919 180 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.127530) v4 ==== 2004+0+0 (877076964 0 0) 0x55f6780b3600 con 0x55f64e3db000 > -9107> 2018-07-11 08:04:35.128350 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.6 10.8.128.115:6803/13323 180 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.127530) v4 ==== 2004+0+0 (877076964 0 0) 0x55f64e20fc00 con 0x55f656ace800 > -9106> 2018-07-11 08:04:35.128340 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6808/12651 conn(0x55f64e3ea000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.2 seq 180 0x55f664257c00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.127530) v4 > -9105> 2018-07-11 08:04:35.128364 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.2 10.8.128.106:6808/12651 180 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.127530) v4 ==== 2004+0+0 (877076964 0 0) 0x55f664257c00 con 0x55f64e3ea000 > -9104> 2018-07-11 08:04:35.128398 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6807/15263 conn(0x55f64e3ee800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.5 seq 180 0x55f664257c00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.127530) v4 > -9103> 2018-07-11 08:04:35.128399 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.115:6802/13323 conn(0x55f656acd000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.6 seq 180 0x55f6780b3600 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.127530) v4 > -9102> 2018-07-11 08:04:35.128411 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.5 10.8.128.113:6807/15263 180 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.127530) v4 ==== 2004+0+0 (877076964 0 0) 0x55f664257c00 con 0x55f64e3ee800 > -9101> 2018-07-11 08:04:35.128405 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.108:6802/11967 conn(0x55f656aca000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.8 seq 180 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.127530) v4 > -9100> 2018-07-11 08:04:35.128415 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.6 10.8.128.115:6802/13323 180 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.127530) v4 ==== 2004+0+0 (877076964 0 0) 0x55f6780b3600 con 0x55f656acd000 > -9099> 2018-07-11 08:04:35.128426 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.8 10.8.128.108:6802/11967 180 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.127530) v4 ==== 2004+0+0 (877076964 0 0) 0x55f64e20fc00 con 0x55f656aca000 > -9098> 2018-07-11 08:04:35.128455 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6806/15885 conn(0x55f64e3fe000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.7 seq 180 0x55f6780b3600 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.127530) v4 > -9097> 2018-07-11 08:04:35.128471 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.7 10.8.128.114:6806/15885 180 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.127530) v4 ==== 2004+0+0 (877076964 0 0) 0x55f6780b3600 con 0x55f64e3fe000 > -9096> 2018-07-11 08:04:35.128470 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6803/15919 conn(0x55f64e3dc800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.11 seq 180 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.127530) v4 > -9095> 2018-07-11 08:04:35.128487 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.11 10.8.128.114:6803/15919 180 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.127530) v4 ==== 2004+0+0 (877076964 0 0) 0x55f64e20fc00 con 0x55f64e3dc800 > -9094> 2018-07-11 08:04:35.128492 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.108:6803/11967 conn(0x55f656acb800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.8 seq 180 0x55f664257c00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.127530) v4 > -9093> 2018-07-11 08:04:35.128512 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.8 10.8.128.108:6803/11967 180 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.127530) v4 ==== 2004+0+0 (877076964 0 0) 0x55f664257c00 con 0x55f656acb800 > -9092> 2018-07-11 08:04:35.128532 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6807/15885 conn(0x55f64e3ff800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.7 seq 180 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.127530) v4 > -9091> 2018-07-11 08:04:35.128547 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.7 10.8.128.114:6807/15885 180 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.127530) v4 ==== 2004+0+0 (877076964 0 0) 0x55f64e20fc00 con 0x55f64e3ff800 > -9090> 2018-07-11 08:04:35.203763 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.115:0/13323 conn(0x55f663ebe000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.6 seq 174 0x55f6780b3600 osd_ping(ping e509 stamp 2018-07-11 08:04:35.555883) v4 > -9089> 2018-07-11 08:04:35.203789 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.6 10.8.128.115:0/13323 174 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:35.555883) v4 ==== 2004+0+0 (3796789666 0 0) 0x55f6780b3600 con 0x55f663ebe000 > -9088> 2018-07-11 08:04:35.203805 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.555883) v4 -- 0x55f64cf24800 con 0 > -9087> 2018-07-11 08:04:35.203841 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.115:0/13323 conn(0x55f663ebf800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.6 seq 174 0x55f664257c00 osd_ping(ping e509 stamp 2018-07-11 08:04:35.555883) v4 > -9086> 2018-07-11 08:04:35.203859 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.6 10.8.128.115:0/13323 174 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:35.555883) v4 ==== 2004+0+0 (3796789666 0 0) 0x55f664257c00 con 0x55f663ebf800 > -9085> 2018-07-11 08:04:35.203875 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.555883) v4 -- 0x55f64cf24c00 con 0 > -9084> 2018-07-11 08:04:35.323268 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f654682fc0 con 0 > -9083> 2018-07-11 08:04:35.324919 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 172 0x55f663168780 replica scrub(pg: 6.15,from:0'0,to:430'106343,epoch:509/504,start:6:a809bf4e:::benchmark_data_magna108.ceph.redhat.com_2478_object337791:0,end:6:a80ac43d:::benchmark_data_magna108_35617_object916886:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -9082> 2018-07-11 08:04:35.324950 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 172 ==== replica scrub(pg: 6.15,from:0'0,to:430'106343,epoch:509/504,start:6:a809bf4e:::benchmark_data_magna108.ceph.redhat.com_2478_object337791:0,end:6:a80ac43d:::benchmark_data_magna108_35617_object916886:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (2133100863 0 0) 0x55f663168780 con 0x55f66315e800 > -9081> 2018-07-11 08:04:35.347665 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0e800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 185 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:04:36.591419) v4 > -9080> 2018-07-11 08:04:35.347689 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.5 10.8.128.113:0/15263 185 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:36.591419) v4 ==== 2004+0+0 (4035004383 0 0) 0x55f64e20fc00 con 0x55f663f0e800 > -9079> 2018-07-11 08:04:35.347706 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:36.591419) v4 -- 0x55f64cf24000 con 0 > -9078> 2018-07-11 08:04:35.347715 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0d000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 185 0x55f6780b2000 osd_ping(ping e509 stamp 2018-07-11 08:04:36.591419) v4 > -9077> 2018-07-11 08:04:35.347739 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.5 10.8.128.113:0/15263 185 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:36.591419) v4 ==== 2004+0+0 (4035004383 0 0) 0x55f6780b2000 con 0x55f663f0d000 > -9076> 2018-07-11 08:04:35.347752 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:36.591419) v4 -- 0x55f64cf24c00 con 0 > -9075> 2018-07-11 08:04:35.391881 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.108:0/11967 conn(0x55f663f6f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=1). rx osd.8 seq 176 0x55f6548fba00 osd_ping(ping e509 stamp 2018-07-11 08:04:35.151516) v4 > -9074> 2018-07-11 08:04:35.391884 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.108:0/11967 conn(0x55f663f70800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=1). rx osd.8 seq 176 0x55f664257c00 osd_ping(ping e509 stamp 2018-07-11 08:04:35.151516) v4 > -9073> 2018-07-11 08:04:35.391904 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.8 10.8.128.108:0/11967 176 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:35.151516) v4 ==== 2004+0+0 (4138862120 0 0) 0x55f6548fba00 con 0x55f663f6f000 > -9072> 2018-07-11 08:04:35.391906 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.8 10.8.128.108:0/11967 176 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:35.151516) v4 ==== 2004+0+0 (4138862120 0 0) 0x55f664257c00 con 0x55f663f70800 > -9071> 2018-07-11 08:04:35.391914 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.151516) v4 -- 0x55f64cf24800 con 0 > -9070> 2018-07-11 08:04:35.391948 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.151516) v4 -- 0x55f64cf24c00 con 0 > -9069> 2018-07-11 08:04:35.524535 7f491d046700 1 -- 10.8.128.107:6800/14173 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55f64c9a3340 con 0 > -9068> 2018-07-11 08:04:35.524609 7f491d046700 1 -- 10.8.128.107:6800/14173 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55f666f24840 con 0 > -9067> 2018-07-11 08:04:35.576564 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.107:0/12617 conn(0x55f64e3c5000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.9 seq 183 0x55f6780b2000 osd_ping(ping e509 stamp 2018-07-11 08:04:35.576034) v4 > -9066> 2018-07-11 08:04:35.576583 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.9 10.8.128.107:0/12617 183 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:35.576034) v4 ==== 2004+0+0 (3412852736 0 0) 0x55f6780b2000 con 0x55f64e3c5000 > -9065> 2018-07-11 08:04:35.576596 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.576034) v4 -- 0x55f64cf24c00 con 0 > -9064> 2018-07-11 08:04:35.576640 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.107:0/12617 conn(0x55f64e3c3800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.9 seq 183 0x55f6780b3600 osd_ping(ping e509 stamp 2018-07-11 08:04:35.576034) v4 > -9063> 2018-07-11 08:04:35.576663 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.9 10.8.128.107:0/12617 183 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:35.576034) v4 ==== 2004+0+0 (3412852736 0 0) 0x55f6780b3600 con 0x55f64e3c3800 > -9062> 2018-07-11 08:04:35.576681 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.576034) v4 -- 0x55f6780b3200 con 0 > -9061> 2018-07-11 08:04:35.627940 7f491803c700 5 osd.10 509 heartbeat: osd_stat(788 GB used, 142 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,0,1]) > -9060> 2018-07-11 08:04:35.627954 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6802/15255 -- osd_ping(ping e509 stamp 2018-07-11 08:04:35.627952) v4 -- 0x55f662489000 con 0 > -9059> 2018-07-11 08:04:35.627980 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6803/15255 -- osd_ping(ping e509 stamp 2018-07-11 08:04:35.627952) v4 -- 0x55f650f78800 con 0 > -9058> 2018-07-11 08:04:35.627995 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6806/12677 -- osd_ping(ping e509 stamp 2018-07-11 08:04:35.627952) v4 -- 0x55f650f79a00 con 0 > -9057> 2018-07-11 08:04:35.628011 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6807/12677 -- osd_ping(ping e509 stamp 2018-07-11 08:04:35.627952) v4 -- 0x55f663fb8800 con 0 > -9056> 2018-07-11 08:04:35.628025 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6808/12651 -- osd_ping(ping e509 stamp 2018-07-11 08:04:35.627952) v4 -- 0x55f663fb9000 con 0 > -9055> 2018-07-11 08:04:35.628037 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6810/12651 -- osd_ping(ping e509 stamp 2018-07-11 08:04:35.627952) v4 -- 0x55f663fb8600 con 0 > -9054> 2018-07-11 08:04:35.628055 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6809/12650 -- osd_ping(ping e509 stamp 2018-07-11 08:04:35.627952) v4 -- 0x55f65299a400 con 0 > -9053> 2018-07-11 08:04:35.628069 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6811/12650 -- osd_ping(ping e509 stamp 2018-07-11 08:04:35.627952) v4 -- 0x55f65299bc00 con 0 > -9052> 2018-07-11 08:04:35.628082 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6810/15287 -- osd_ping(ping e509 stamp 2018-07-11 08:04:35.627952) v4 -- 0x55f664257e00 con 0 > -9051> 2018-07-11 08:04:35.628095 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6811/15287 -- osd_ping(ping e509 stamp 2018-07-11 08:04:35.627952) v4 -- 0x55f664256400 con 0 > -9050> 2018-07-11 08:04:35.628111 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6806/15263 -- osd_ping(ping e509 stamp 2018-07-11 08:04:35.627952) v4 -- 0x55f67d1af800 con 0 > -9049> 2018-07-11 08:04:35.628124 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6807/15263 -- osd_ping(ping e509 stamp 2018-07-11 08:04:35.627952) v4 -- 0x55f67d1ae800 con 0 > -9048> 2018-07-11 08:04:35.628137 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.115:6802/13323 -- osd_ping(ping e509 stamp 2018-07-11 08:04:35.627952) v4 -- 0x55f64cf25200 con 0 > -9047> 2018-07-11 08:04:35.628150 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.115:6803/13323 -- osd_ping(ping e509 stamp 2018-07-11 08:04:35.627952) v4 -- 0x55f66dc70c00 con 0 > -9046> 2018-07-11 08:04:35.628164 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6806/15885 -- osd_ping(ping e509 stamp 2018-07-11 08:04:35.627952) v4 -- 0x55f64cc80200 con 0 > -9045> 2018-07-11 08:04:35.628175 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6807/15885 -- osd_ping(ping e509 stamp 2018-07-11 08:04:35.627952) v4 -- 0x55f64e25ca00 con 0 > -9044> 2018-07-11 08:04:35.628187 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.108:6802/11967 -- osd_ping(ping e509 stamp 2018-07-11 08:04:35.627952) v4 -- 0x55f64f768800 con 0 > -9043> 2018-07-11 08:04:35.628198 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.108:6803/11967 -- osd_ping(ping e509 stamp 2018-07-11 08:04:35.627952) v4 -- 0x55f64f769400 con 0 > -9042> 2018-07-11 08:04:35.628210 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.107:6806/12617 -- osd_ping(ping e509 stamp 2018-07-11 08:04:35.627952) v4 -- 0x55f667fd8c00 con 0 > -9041> 2018-07-11 08:04:35.628223 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.107:6807/12617 -- osd_ping(ping e509 stamp 2018-07-11 08:04:35.627952) v4 -- 0x55f6631ccc00 con 0 > -9040> 2018-07-11 08:04:35.628235 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6802/15919 -- osd_ping(ping e509 stamp 2018-07-11 08:04:35.627952) v4 -- 0x55f6631cc000 con 0 > -9039> 2018-07-11 08:04:35.628258 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6803/15919 -- osd_ping(ping e509 stamp 2018-07-11 08:04:35.627952) v4 -- 0x55f6631cc400 con 0 > -9038> 2018-07-11 08:04:35.628345 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6802/15255 conn(0x55f656ad4000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 181 0x55f6780b2000 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.627952) v4 > -9037> 2018-07-11 08:04:35.628368 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.0 10.8.128.113:6802/15255 181 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.627952) v4 ==== 2004+0+0 (240778790 0 0) 0x55f6780b2000 con 0x55f656ad4000 > -9036> 2018-07-11 08:04:35.628384 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.107:6806/12617 conn(0x55f656ade000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.9 seq 181 0x55f6780b3600 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.627952) v4 > -9035> 2018-07-11 08:04:35.628402 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.9 10.8.128.107:6806/12617 181 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.627952) v4 ==== 2004+0+0 (240778790 0 0) 0x55f6780b3600 con 0x55f656ade000 > -9034> 2018-07-11 08:04:35.628446 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6803/15255 conn(0x55f656ad5800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 181 0x55f6780b3600 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.627952) v4 > -9033> 2018-07-11 08:04:35.628461 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.0 10.8.128.113:6803/15255 181 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.627952) v4 ==== 2004+0+0 (240778790 0 0) 0x55f6780b3600 con 0x55f656ad5800 > -9032> 2018-07-11 08:04:35.628444 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6810/15287 conn(0x55f64e401000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.4 seq 181 0x55f6780b2000 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.627952) v4 > -9031> 2018-07-11 08:04:35.628478 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.4 10.8.128.113:6810/15287 181 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.627952) v4 ==== 2004+0+0 (240778790 0 0) 0x55f6780b2000 con 0x55f64e401000 > -9030> 2018-07-11 08:04:35.628474 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6809/12650 conn(0x55f656ad7000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.3 seq 181 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.627952) v4 > -9029> 2018-07-11 08:04:35.628498 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6811/15287 conn(0x55f64e402800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.4 seq 181 0x55f6780b3600 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.627952) v4 > -9028> 2018-07-11 08:04:35.628501 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.3 10.8.128.106:6809/12650 181 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.627952) v4 ==== 2004+0+0 (240778790 0 0) 0x55f64e20fc00 con 0x55f656ad7000 > -9027> 2018-07-11 08:04:35.628513 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.4 10.8.128.113:6811/15287 181 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.627952) v4 ==== 2004+0+0 (240778790 0 0) 0x55f6780b3600 con 0x55f64e402800 > -9026> 2018-07-11 08:04:35.628532 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6807/12677 conn(0x55f64e3d9800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.1 seq 181 0x55f6780b2000 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.627952) v4 > -9025> 2018-07-11 08:04:35.628558 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6806/12677 conn(0x55f64e3d8000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.1 seq 181 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.627952) v4 > -9024> 2018-07-11 08:04:35.628579 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6810/12651 conn(0x55f64e3eb800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.2 seq 181 0x55f6780b3600 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.627952) v4 > -9023> 2018-07-11 08:04:35.628621 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.1 10.8.128.106:6806/12677 181 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.627952) v4 ==== 2004+0+0 (240778790 0 0) 0x55f64e20fc00 con 0x55f64e3d8000 > -9022> 2018-07-11 08:04:35.628625 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.2 10.8.128.106:6810/12651 181 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.627952) v4 ==== 2004+0+0 (240778790 0 0) 0x55f6780b3600 con 0x55f64e3eb800 > -9021> 2018-07-11 08:04:35.628617 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.1 10.8.128.106:6807/12677 181 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.627952) v4 ==== 2004+0+0 (240778790 0 0) 0x55f6780b2000 con 0x55f64e3d9800 > -9020> 2018-07-11 08:04:35.628666 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.107:6807/12617 conn(0x55f656adf800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.9 seq 181 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.627952) v4 > -9019> 2018-07-11 08:04:35.628668 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6806/15885 conn(0x55f64e3fe000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.7 seq 181 0x55f6780b3600 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.627952) v4 > -9018> 2018-07-11 08:04:35.628680 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.9 10.8.128.107:6807/12617 181 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.627952) v4 ==== 2004+0+0 (240778790 0 0) 0x55f64e20fc00 con 0x55f656adf800 > -9017> 2018-07-11 08:04:35.628681 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.7 10.8.128.114:6806/15885 181 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.627952) v4 ==== 2004+0+0 (240778790 0 0) 0x55f6780b3600 con 0x55f64e3fe000 > -9016> 2018-07-11 08:04:35.628685 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6808/12651 conn(0x55f64e3ea000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.2 seq 181 0x55f6780b2000 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.627952) v4 > -9015> 2018-07-11 08:04:35.628704 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.2 10.8.128.106:6808/12651 181 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.627952) v4 ==== 2004+0+0 (240778790 0 0) 0x55f6780b2000 con 0x55f64e3ea000 > -9014> 2018-07-11 08:04:35.628714 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6806/15263 conn(0x55f64e3ed000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.5 seq 181 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.627952) v4 > -9013> 2018-07-11 08:04:35.628719 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.115:6802/13323 conn(0x55f656acd000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.6 seq 181 0x55f6780b3600 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.627952) v4 > -9012> 2018-07-11 08:04:35.628726 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.5 10.8.128.113:6806/15263 181 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.627952) v4 ==== 2004+0+0 (240778790 0 0) 0x55f64e20fc00 con 0x55f64e3ed000 > -9011> 2018-07-11 08:04:35.628729 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.6 10.8.128.115:6802/13323 181 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.627952) v4 ==== 2004+0+0 (240778790 0 0) 0x55f6780b3600 con 0x55f656acd000 > -9010> 2018-07-11 08:04:35.628747 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6807/15263 conn(0x55f64e3ee800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.5 seq 181 0x55f6780b2000 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.627952) v4 > -9009> 2018-07-11 08:04:35.628761 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.115:6803/13323 conn(0x55f656ace800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.6 seq 181 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.627952) v4 > -9008> 2018-07-11 08:04:35.628764 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.5 10.8.128.113:6807/15263 181 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.627952) v4 ==== 2004+0+0 (240778790 0 0) 0x55f6780b2000 con 0x55f64e3ee800 > -9007> 2018-07-11 08:04:35.628766 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6802/15919 conn(0x55f64e3db000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.11 seq 181 0x55f6780b3600 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.627952) v4 > -9006> 2018-07-11 08:04:35.628774 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.6 10.8.128.115:6803/13323 181 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.627952) v4 ==== 2004+0+0 (240778790 0 0) 0x55f64e20fc00 con 0x55f656ace800 > -9005> 2018-07-11 08:04:35.628776 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.11 10.8.128.114:6802/15919 181 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.627952) v4 ==== 2004+0+0 (240778790 0 0) 0x55f6780b3600 con 0x55f64e3db000 > -9004> 2018-07-11 08:04:35.628806 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6811/12650 conn(0x55f656ad8800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.3 seq 181 0x55f6780b2000 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.627952) v4 > -9003> 2018-07-11 08:04:35.628814 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.108:6802/11967 conn(0x55f656aca000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.8 seq 181 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.627952) v4 > -9002> 2018-07-11 08:04:35.628824 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.8 10.8.128.108:6802/11967 181 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.627952) v4 ==== 2004+0+0 (240778790 0 0) 0x55f64e20fc00 con 0x55f656aca000 > -9001> 2018-07-11 08:04:35.628824 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.3 10.8.128.106:6811/12650 181 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.627952) v4 ==== 2004+0+0 (240778790 0 0) 0x55f6780b2000 con 0x55f656ad8800 > -9000> 2018-07-11 08:04:35.628872 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.108:6803/11967 conn(0x55f656acb800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.8 seq 181 0x55f6780b2000 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.627952) v4 > -8999> 2018-07-11 08:04:35.628873 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6807/15885 conn(0x55f64e3ff800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.7 seq 181 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.627952) v4 > -8998> 2018-07-11 08:04:35.628893 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.8 10.8.128.108:6803/11967 181 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.627952) v4 ==== 2004+0+0 (240778790 0 0) 0x55f6780b2000 con 0x55f656acb800 > -8997> 2018-07-11 08:04:35.628899 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.7 10.8.128.114:6807/15885 181 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.627952) v4 ==== 2004+0+0 (240778790 0 0) 0x55f64e20fc00 con 0x55f64e3ff800 > -8996> 2018-07-11 08:04:35.628942 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6803/15919 conn(0x55f64e3dc800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.11 seq 181 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.627952) v4 > -8995> 2018-07-11 08:04:35.628955 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.11 10.8.128.114:6803/15919 181 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:35.627952) v4 ==== 2004+0+0 (240778790 0 0) 0x55f64e20fc00 con 0x55f64e3dc800 > -8994> 2018-07-11 08:04:35.957149 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 397 0x55f6780b3600 MOSDScrubReserve(11.3s5 REQUEST e509) v1 > -8993> 2018-07-11 08:04:35.957169 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 397 ==== MOSDScrubReserve(11.3s5 REQUEST e509) v1 ==== 43+0+0 (433425070 0 0) 0x55f6780b3600 con 0x55f668a29000 > -8992> 2018-07-11 08:04:35.957208 7f491a040700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e509) v1 -- 0x55f666c47600 con 0 > -8991> 2018-07-11 08:04:35.957605 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 398 0x55f666c47600 MOSDScrubReserve(11.3s5 RELEASE e509) v1 > -8990> 2018-07-11 08:04:35.957620 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 398 ==== MOSDScrubReserve(11.3s5 RELEASE e509) v1 ==== 43+0+0 (3603460268 0 0) 0x55f666c47600 con 0x55f668a29000 > -8989> 2018-07-11 08:04:36.195781 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0b800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 178 0x55f6548fba00 osd_ping(ping e509 stamp 2018-07-11 08:04:36.830983) v4 > -8988> 2018-07-11 08:04:36.195786 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0a000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 178 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:04:36.830983) v4 > -8987> 2018-07-11 08:04:36.195802 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.11 10.8.128.114:0/15919 178 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:36.830983) v4 ==== 2004+0+0 (3452952664 0 0) 0x55f6548fba00 con 0x55f663f0b800 > -8986> 2018-07-11 08:04:36.195808 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.11 10.8.128.114:0/15919 178 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:36.830983) v4 ==== 2004+0+0 (3452952664 0 0) 0x55f64e20fc00 con 0x55f663f0a000 > -8985> 2018-07-11 08:04:36.195812 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:36.830983) v4 -- 0x55f653d3dc00 con 0 > -8984> 2018-07-11 08:04:36.195842 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:36.830983) v4 -- 0x55f65299a400 con 0 > -8983> 2018-07-11 08:04:36.401377 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f6662a6240 con 0 > -8982> 2018-07-11 08:04:36.403026 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 173 0x55f667d82f00 replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:509/504,start:6:a80ac43d:::benchmark_data_magna108_35617_object916886:0,end:6:a80b7a80:::benchmark_data_magna108_35617_object230234:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -8981> 2018-07-11 08:04:36.403058 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 173 ==== replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:509/504,start:6:a80ac43d:::benchmark_data_magna108_35617_object916886:0,end:6:a80b7a80:::benchmark_data_magna108_35617_object230234:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (1485898291 0 0) 0x55f667d82f00 con 0x55f66315e800 > -8980> 2018-07-11 08:04:36.445394 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12651 conn(0x55f663219000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25 cs=1 l=1). rx osd.2 seq 194 0x55f6780b2000 osd_ping(ping e509 stamp 2018-07-11 08:04:36.501466) v4 > -8979> 2018-07-11 08:04:36.445422 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.2 10.8.128.106:0/12651 194 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:36.501466) v4 ==== 2004+0+0 (3086466510 0 0) 0x55f6780b2000 con 0x55f663219000 > -8978> 2018-07-11 08:04:36.445445 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:36.501466) v4 -- 0x55f662489000 con 0 > -8977> 2018-07-11 08:04:36.445463 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12651 conn(0x55f66321a800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25 cs=1 l=1). rx osd.2 seq 194 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:04:36.501466) v4 > -8976> 2018-07-11 08:04:36.445482 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.2 10.8.128.106:0/12651 194 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:36.501466) v4 ==== 2004+0+0 (3086466510 0 0) 0x55f67d1afe00 con 0x55f66321a800 > -8975> 2018-07-11 08:04:36.445494 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:36.501466) v4 -- 0x55f650f79a00 con 0 > -8974> 2018-07-11 08:04:36.568677 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15255 conn(0x55f663ec1000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.0 seq 175 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:04:37.812363) v4 > -8973> 2018-07-11 08:04:36.568707 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.0 10.8.128.113:0/15255 175 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:37.812363) v4 ==== 2004+0+0 (1085905718 0 0) 0x55f64e20fc00 con 0x55f663ec1000 > -8972> 2018-07-11 08:04:36.568724 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:37.812363) v4 -- 0x55f64cf24000 con 0 > -8971> 2018-07-11 08:04:36.568745 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15255 conn(0x55f663ec2800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.0 seq 175 0x55f6780b3200 osd_ping(ping e509 stamp 2018-07-11 08:04:37.812363) v4 > -8970> 2018-07-11 08:04:36.568767 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.0 10.8.128.113:0/15255 175 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:37.812363) v4 ==== 2004+0+0 (1085905718 0 0) 0x55f6780b3200 con 0x55f663ec2800 > -8969> 2018-07-11 08:04:36.568780 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:37.812363) v4 -- 0x55f669deb600 con 0 > -8968> 2018-07-11 08:04:36.922959 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12650 conn(0x55f663eda800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.3 seq 182 0x55f664257c00 osd_ping(ping e509 stamp 2018-07-11 08:04:36.979198) v4 > -8967> 2018-07-11 08:04:36.922991 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.3 10.8.128.106:0/12650 182 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:36.979198) v4 ==== 2004+0+0 (21303042 0 0) 0x55f664257c00 con 0x55f663eda800 > -8966> 2018-07-11 08:04:36.923006 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:36.979198) v4 -- 0x55f64cf24c00 con 0 > -8965> 2018-07-11 08:04:36.923056 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12650 conn(0x55f663ed9000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.3 seq 182 0x55f6548fba00 osd_ping(ping e509 stamp 2018-07-11 08:04:36.979198) v4 > -8964> 2018-07-11 08:04:36.923082 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.3 10.8.128.106:0/12650 182 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:36.979198) v4 ==== 2004+0+0 (21303042 0 0) 0x55f6548fba00 con 0x55f663ed9000 > -8963> 2018-07-11 08:04:36.923094 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:36.979198) v4 -- 0x55f667d09200 con 0 > -8962> 2018-07-11 08:04:36.957360 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 399 0x55f6780b3200 MOSDScrubReserve(11.3s5 REQUEST e509) v1 > -8961> 2018-07-11 08:04:36.957380 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 399 ==== MOSDScrubReserve(11.3s5 REQUEST e509) v1 ==== 43+0+0 (433425070 0 0) 0x55f6780b3200 con 0x55f668a29000 > -8960> 2018-07-11 08:04:36.957452 7f491a040700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e509) v1 -- 0x55f666c47600 con 0 > -8959> 2018-07-11 08:04:36.957740 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 400 0x55f667d09200 MOSDScrubReserve(11.3s5 RELEASE e509) v1 > -8958> 2018-07-11 08:04:36.957755 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 400 ==== MOSDScrubReserve(11.3s5 RELEASE e509) v1 ==== 43+0+0 (3603460268 0 0) 0x55f667d09200 con 0x55f668a29000 > -8957> 2018-07-11 08:04:37.092331 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.108:0/11967 conn(0x55f663f6f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=1). rx osd.8 seq 177 0x55f6548fba00 osd_ping(ping e509 stamp 2018-07-11 08:04:36.851955) v4 > -8956> 2018-07-11 08:04:37.092354 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.8 10.8.128.108:0/11967 177 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:36.851955) v4 ==== 2004+0+0 (2691614726 0 0) 0x55f6548fba00 con 0x55f663f6f000 > -8955> 2018-07-11 08:04:37.092364 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:36.851955) v4 -- 0x55f65dad4200 con 0 > -8954> 2018-07-11 08:04:37.092377 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.108:0/11967 conn(0x55f663f70800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=1). rx osd.8 seq 177 0x55f6780b2000 osd_ping(ping e509 stamp 2018-07-11 08:04:36.851955) v4 > -8953> 2018-07-11 08:04:37.092399 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.8 10.8.128.108:0/11967 177 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:36.851955) v4 ==== 2004+0+0 (2691614726 0 0) 0x55f6780b2000 con 0x55f663f70800 > -8952> 2018-07-11 08:04:37.092412 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:36.851955) v4 -- 0x55f6780b2800 con 0 > -8951> 2018-07-11 08:04:37.229513 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f666e5fd40 con 0 > -8950> 2018-07-11 08:04:37.231190 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 174 0x55f667d82c80 replica scrub(pg: 6.15,from:0'0,to:430'106360,epoch:509/504,start:6:a80b7a80:::benchmark_data_magna108_35617_object230234:0,end:6:a80c8403:::benchmark_data_magna108_35617_object513272:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -8949> 2018-07-11 08:04:37.231221 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 174 ==== replica scrub(pg: 6.15,from:0'0,to:430'106360,epoch:509/504,start:6:a80b7a80:::benchmark_data_magna108_35617_object230234:0,end:6:a80c8403:::benchmark_data_magna108_35617_object513272:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (2900551922 0 0) 0x55f667d82c80 con 0x55f66315e800 > -8948> 2018-07-11 08:04:37.724752 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15885 conn(0x55f663217800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 184 0x55f666c47600 osd_ping(ping e509 stamp 2018-07-11 08:04:38.359951) v4 > -8947> 2018-07-11 08:04:37.724767 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15885 conn(0x55f663216000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 184 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:04:38.359951) v4 > -8946> 2018-07-11 08:04:37.724791 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.7 10.8.128.114:0/15885 184 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:38.359951) v4 ==== 2004+0+0 (1500809650 0 0) 0x55f666c47600 con 0x55f663217800 > -8945> 2018-07-11 08:04:37.724802 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:38.359951) v4 -- 0x55f65dad4200 con 0 > -8944> 2018-07-11 08:04:37.724799 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.7 10.8.128.114:0/15885 184 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:38.359951) v4 ==== 2004+0+0 (1500809650 0 0) 0x55f67d1afe00 con 0x55f663216000 > -8943> 2018-07-11 08:04:37.724832 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:38.359951) v4 -- 0x55f64cf24000 con 0 > -8942> 2018-07-11 08:04:37.964145 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f6546a2900 con 0 > -8941> 2018-07-11 08:04:37.965981 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 175 0x55f663168c80 replica scrub(pg: 6.15,from:0'0,to:430'105940,epoch:509/504,start:6:a80c8403:::benchmark_data_magna108_35617_object513272:0,end:6:a80d6954:::benchmark_data_magna108_35617_object1064599:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -8940> 2018-07-11 08:04:37.966012 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 175 ==== replica scrub(pg: 6.15,from:0'0,to:430'105940,epoch:509/504,start:6:a80c8403:::benchmark_data_magna108_35617_object513272:0,end:6:a80d6954:::benchmark_data_magna108_35617_object1064599:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 219+0+0 (1174566281 0 0) 0x55f663168c80 con 0x55f66315e800 > -8939> 2018-07-11 08:04:38.248055 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0e800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 186 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:04:39.491866) v4 > -8938> 2018-07-11 08:04:38.248082 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0d000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 186 0x55f664257c00 osd_ping(ping e509 stamp 2018-07-11 08:04:39.491866) v4 > -8937> 2018-07-11 08:04:38.248091 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.5 10.8.128.113:0/15263 186 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:39.491866) v4 ==== 2004+0+0 (2404037374 0 0) 0x55f64e20fc00 con 0x55f663f0e800 > -8936> 2018-07-11 08:04:38.248106 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:39.491866) v4 -- 0x55f64cf24000 con 0 > -8935> 2018-07-11 08:04:38.248102 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.5 10.8.128.113:0/15263 186 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:39.491866) v4 ==== 2004+0+0 (2404037374 0 0) 0x55f664257c00 con 0x55f663f0d000 > -8934> 2018-07-11 08:04:38.248131 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:39.491866) v4 -- 0x55f660948e00 con 0 > -8933> 2018-07-11 08:04:38.528396 7f491803c700 5 osd.10 509 heartbeat: osd_stat(788 GB used, 142 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,0,0,1]) > -8932> 2018-07-11 08:04:38.528416 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6802/15255 -- osd_ping(ping e509 stamp 2018-07-11 08:04:38.528414) v4 -- 0x55f663193c00 con 0 > -8931> 2018-07-11 08:04:38.528466 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6803/15255 -- osd_ping(ping e509 stamp 2018-07-11 08:04:38.528414) v4 -- 0x55f65e71de00 con 0 > -8930> 2018-07-11 08:04:38.528482 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6806/12677 -- osd_ping(ping e509 stamp 2018-07-11 08:04:38.528414) v4 -- 0x55f65e71d600 con 0 > -8929> 2018-07-11 08:04:38.528498 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6807/12677 -- osd_ping(ping e509 stamp 2018-07-11 08:04:38.528414) v4 -- 0x55f65e71c600 con 0 > -8928> 2018-07-11 08:04:38.528510 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6808/12651 -- osd_ping(ping e509 stamp 2018-07-11 08:04:38.528414) v4 -- 0x55f669debe00 con 0 > -8927> 2018-07-11 08:04:38.528522 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6810/12651 -- osd_ping(ping e509 stamp 2018-07-11 08:04:38.528414) v4 -- 0x55f669dea800 con 0 > -8926> 2018-07-11 08:04:38.528533 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6809/12650 -- osd_ping(ping e509 stamp 2018-07-11 08:04:38.528414) v4 -- 0x55f653d3d400 con 0 > -8925> 2018-07-11 08:04:38.528545 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6811/12650 -- osd_ping(ping e509 stamp 2018-07-11 08:04:38.528414) v4 -- 0x55f6631f4c00 con 0 > -8924> 2018-07-11 08:04:38.528566 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6810/15287 -- osd_ping(ping e509 stamp 2018-07-11 08:04:38.528414) v4 -- 0x55f6631f4a00 con 0 > -8923> 2018-07-11 08:04:38.528578 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6811/15287 -- osd_ping(ping e509 stamp 2018-07-11 08:04:38.528414) v4 -- 0x55f6631f4200 con 0 > -8922> 2018-07-11 08:04:38.528591 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6806/15263 -- osd_ping(ping e509 stamp 2018-07-11 08:04:38.528414) v4 -- 0x55f6631f5800 con 0 > -8921> 2018-07-11 08:04:38.528607 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6807/15263 -- osd_ping(ping e509 stamp 2018-07-11 08:04:38.528414) v4 -- 0x55f65d024200 con 0 > -8920> 2018-07-11 08:04:38.528620 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.115:6802/13323 -- osd_ping(ping e509 stamp 2018-07-11 08:04:38.528414) v4 -- 0x55f664255200 con 0 > -8919> 2018-07-11 08:04:38.528633 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.115:6803/13323 -- osd_ping(ping e509 stamp 2018-07-11 08:04:38.528414) v4 -- 0x55f664254600 con 0 > -8918> 2018-07-11 08:04:38.528647 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6806/15885 -- osd_ping(ping e509 stamp 2018-07-11 08:04:38.528414) v4 -- 0x55f65dad4c00 con 0 > -8917> 2018-07-11 08:04:38.528658 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6807/15885 -- osd_ping(ping e509 stamp 2018-07-11 08:04:38.528414) v4 -- 0x55f65dad4800 con 0 > -8916> 2018-07-11 08:04:38.528670 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.108:6802/11967 -- osd_ping(ping e509 stamp 2018-07-11 08:04:38.528414) v4 -- 0x55f65dad5e00 con 0 > -8915> 2018-07-11 08:04:38.528680 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.108:6803/11967 -- osd_ping(ping e509 stamp 2018-07-11 08:04:38.528414) v4 -- 0x55f668a30a00 con 0 > -8914> 2018-07-11 08:04:38.528693 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.107:6806/12617 -- osd_ping(ping e509 stamp 2018-07-11 08:04:38.528414) v4 -- 0x55f663c0bc00 con 0 > -8913> 2018-07-11 08:04:38.528705 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.107:6807/12617 -- osd_ping(ping e509 stamp 2018-07-11 08:04:38.528414) v4 -- 0x55f663c0a200 con 0 > -8912> 2018-07-11 08:04:38.528717 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6802/15919 -- osd_ping(ping e509 stamp 2018-07-11 08:04:38.528414) v4 -- 0x55f663c0ba00 con 0 > -8911> 2018-07-11 08:04:38.528729 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6803/15919 -- osd_ping(ping e509 stamp 2018-07-11 08:04:38.528414) v4 -- 0x55f663c0b400 con 0 > -8910> 2018-07-11 08:04:38.528852 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6802/15255 conn(0x55f656ad4000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 182 0x55f664257c00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:38.528414) v4 > -8909> 2018-07-11 08:04:38.528873 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.0 10.8.128.113:6802/15255 182 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:38.528414) v4 ==== 2004+0+0 (3178586853 0 0) 0x55f664257c00 con 0x55f656ad4000 > -8908> 2018-07-11 08:04:38.528963 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6803/15255 conn(0x55f656ad5800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 182 0x55f666c47600 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:38.528414) v4 > -8907> 2018-07-11 08:04:38.528978 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.0 10.8.128.113:6803/15255 182 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:38.528414) v4 ==== 2004+0+0 (3178586853 0 0) 0x55f666c47600 con 0x55f656ad5800 > -8906> 2018-07-11 08:04:38.528966 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6806/12677 conn(0x55f64e3d8000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.1 seq 182 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:38.528414) v4 > -8905> 2018-07-11 08:04:38.528977 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6811/12650 conn(0x55f656ad8800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.3 seq 182 0x55f664257c00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:38.528414) v4 > -8904> 2018-07-11 08:04:38.528988 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.1 10.8.128.106:6806/12677 182 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:38.528414) v4 ==== 2004+0+0 (3178586853 0 0) 0x55f67d1afe00 con 0x55f64e3d8000 > -8903> 2018-07-11 08:04:38.528994 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.3 10.8.128.106:6811/12650 182 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:38.528414) v4 ==== 2004+0+0 (3178586853 0 0) 0x55f664257c00 con 0x55f656ad8800 > -8902> 2018-07-11 08:04:38.529018 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6810/12651 conn(0x55f64e3eb800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.2 seq 182 0x55f666c47600 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:38.528414) v4 > -8901> 2018-07-11 08:04:38.529078 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.2 10.8.128.106:6810/12651 182 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:38.528414) v4 ==== 2004+0+0 (3178586853 0 0) 0x55f666c47600 con 0x55f64e3eb800 > -8900> 2018-07-11 08:04:38.529082 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.107:6807/12617 conn(0x55f656adf800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.9 seq 182 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:38.528414) v4 > -8899> 2018-07-11 08:04:38.529102 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6807/12677 conn(0x55f64e3d9800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.1 seq 182 0x55f664257c00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:38.528414) v4 > -8898> 2018-07-11 08:04:38.529123 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.9 10.8.128.107:6807/12617 182 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:38.528414) v4 ==== 2004+0+0 (3178586853 0 0) 0x55f67d1afe00 con 0x55f656adf800 > -8897> 2018-07-11 08:04:38.529132 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.1 10.8.128.106:6807/12677 182 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:38.528414) v4 ==== 2004+0+0 (3178586853 0 0) 0x55f664257c00 con 0x55f64e3d9800 > -8896> 2018-07-11 08:04:38.529136 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6811/15287 conn(0x55f64e402800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.4 seq 182 0x55f666c47600 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:38.528414) v4 > -8895> 2018-07-11 08:04:38.529151 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.4 10.8.128.113:6811/15287 182 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:38.528414) v4 ==== 2004+0+0 (3178586853 0 0) 0x55f666c47600 con 0x55f64e402800 > -8894> 2018-07-11 08:04:38.529170 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6807/15263 conn(0x55f64e3ee800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.5 seq 182 0x55f664257c00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:38.528414) v4 > -8893> 2018-07-11 08:04:38.529183 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.5 10.8.128.113:6807/15263 182 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:38.528414) v4 ==== 2004+0+0 (3178586853 0 0) 0x55f664257c00 con 0x55f64e3ee800 > -8892> 2018-07-11 08:04:38.529175 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.107:6806/12617 conn(0x55f656ade000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.9 seq 182 0x55f666c47600 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:38.528414) v4 > -8891> 2018-07-11 08:04:38.529202 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.9 10.8.128.107:6806/12617 182 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:38.528414) v4 ==== 2004+0+0 (3178586853 0 0) 0x55f666c47600 con 0x55f656ade000 > -8890> 2018-07-11 08:04:38.529202 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6809/12650 conn(0x55f656ad7000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.3 seq 182 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:38.528414) v4 > -8889> 2018-07-11 08:04:38.529213 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6808/12651 conn(0x55f64e3ea000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.2 seq 182 0x55f664257c00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:38.528414) v4 > -8888> 2018-07-11 08:04:38.529224 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.2 10.8.128.106:6808/12651 182 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:38.528414) v4 ==== 2004+0+0 (3178586853 0 0) 0x55f664257c00 con 0x55f64e3ea000 > -8887> 2018-07-11 08:04:38.529224 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.3 10.8.128.106:6809/12650 182 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:38.528414) v4 ==== 2004+0+0 (3178586853 0 0) 0x55f67d1afe00 con 0x55f656ad7000 > -8886> 2018-07-11 08:04:38.529240 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.115:6802/13323 conn(0x55f656acd000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.6 seq 182 0x55f666c47600 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:38.528414) v4 > -8885> 2018-07-11 08:04:38.529268 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6810/15287 conn(0x55f64e401000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.4 seq 182 0x55f664257c00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:38.528414) v4 > -8884> 2018-07-11 08:04:38.529272 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.6 10.8.128.115:6802/13323 182 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:38.528414) v4 ==== 2004+0+0 (3178586853 0 0) 0x55f666c47600 con 0x55f656acd000 > -8883> 2018-07-11 08:04:38.529284 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.4 10.8.128.113:6810/15287 182 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:38.528414) v4 ==== 2004+0+0 (3178586853 0 0) 0x55f664257c00 con 0x55f64e401000 > -8882> 2018-07-11 08:04:38.529303 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6806/15263 conn(0x55f64e3ed000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.5 seq 182 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:38.528414) v4 > -8881> 2018-07-11 08:04:38.529317 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6802/15919 conn(0x55f64e3db000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.11 seq 182 0x55f666c47600 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:38.528414) v4 > -8880> 2018-07-11 08:04:38.529331 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.11 10.8.128.114:6802/15919 182 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:38.528414) v4 ==== 2004+0+0 (3178586853 0 0) 0x55f666c47600 con 0x55f64e3db000 > -8879> 2018-07-11 08:04:38.529323 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.5 10.8.128.113:6806/15263 182 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:38.528414) v4 ==== 2004+0+0 (3178586853 0 0) 0x55f67d1afe00 con 0x55f64e3ed000 > -8878> 2018-07-11 08:04:38.529360 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6806/15885 conn(0x55f64e3fe000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.7 seq 182 0x55f666c47600 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:38.528414) v4 > -8877> 2018-07-11 08:04:38.529364 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.108:6803/11967 conn(0x55f656acb800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.8 seq 182 0x55f664257c00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:38.528414) v4 > -8876> 2018-07-11 08:04:38.529376 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.7 10.8.128.114:6806/15885 182 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:38.528414) v4 ==== 2004+0+0 (3178586853 0 0) 0x55f666c47600 con 0x55f64e3fe000 > -8875> 2018-07-11 08:04:38.529379 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.8 10.8.128.108:6803/11967 182 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:38.528414) v4 ==== 2004+0+0 (3178586853 0 0) 0x55f664257c00 con 0x55f656acb800 > -8874> 2018-07-11 08:04:38.529387 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.115:6803/13323 conn(0x55f656ace800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.6 seq 182 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:38.528414) v4 > -8873> 2018-07-11 08:04:38.529413 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.6 10.8.128.115:6803/13323 182 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:38.528414) v4 ==== 2004+0+0 (3178586853 0 0) 0x55f67d1afe00 con 0x55f656ace800 > -8872> 2018-07-11 08:04:38.529464 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6803/15919 conn(0x55f64e3dc800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.11 seq 182 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:38.528414) v4 > -8871> 2018-07-11 08:04:38.529483 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.11 10.8.128.114:6803/15919 182 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:38.528414) v4 ==== 2004+0+0 (3178586853 0 0) 0x55f67d1afe00 con 0x55f64e3dc800 > -8870> 2018-07-11 08:04:38.529526 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.108:6802/11967 conn(0x55f656aca000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.8 seq 182 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:38.528414) v4 > -8869> 2018-07-11 08:04:38.529545 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.8 10.8.128.108:6802/11967 182 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:38.528414) v4 ==== 2004+0+0 (3178586853 0 0) 0x55f67d1afe00 con 0x55f656aca000 > -8868> 2018-07-11 08:04:38.529588 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6807/15885 conn(0x55f64e3ff800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.7 seq 182 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:38.528414) v4 > -8867> 2018-07-11 08:04:38.529607 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.7 10.8.128.114:6807/15885 182 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:38.528414) v4 ==== 2004+0+0 (3178586853 0 0) 0x55f67d1afe00 con 0x55f64e3ff800 > -8866> 2018-07-11 08:04:38.656040 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.114:6805/15885 conn(0x55f66315a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.7 seq 2513 0x55f664257c00 MOSDScrubReserve(5.4 REQUEST e509) v1 > -8865> 2018-07-11 08:04:38.656058 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.7 10.8.128.114:6805/15885 2513 ==== MOSDScrubReserve(5.4 REQUEST e509) v1 ==== 43+0+0 (3003630469 0 0) 0x55f664257c00 con 0x55f66315a000 > -8864> 2018-07-11 08:04:38.656126 7f491983f700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.114:6805/15885 -- MOSDScrubReserve(5.4 REJECT e509) v1 -- 0x55f6548fbe00 con 0 > -8863> 2018-07-11 08:04:38.656462 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.114:6805/15885 conn(0x55f66315a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.7 seq 2514 0x55f6548fbe00 MOSDScrubReserve(5.4 RELEASE e509) v1 > -8862> 2018-07-11 08:04:38.656477 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.7 10.8.128.114:6805/15885 2514 ==== MOSDScrubReserve(5.4 RELEASE e509) v1 ==== 43+0+0 (2082093447 0 0) 0x55f6548fbe00 con 0x55f66315a000 > -8861> 2018-07-11 08:04:38.690302 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f66a2b8240 con 0 > -8860> 2018-07-11 08:04:38.691831 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 176 0x55f66ca7ec80 replica scrub(pg: 6.15,from:0'0,to:430'106732,epoch:509/504,start:6:a80d6954:::benchmark_data_magna108_35617_object1064599:0,end:6:a80e3a46:::benchmark_data_magna108.ceph.redhat.com_2478_object249933:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -8859> 2018-07-11 08:04:38.691861 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 176 ==== replica scrub(pg: 6.15,from:0'0,to:430'106732,epoch:509/504,start:6:a80d6954:::benchmark_data_magna108_35617_object1064599:0,end:6:a80e3a46:::benchmark_data_magna108.ceph.redhat.com_2478_object249933:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 234+0+0 (217326469 0 0) 0x55f66ca7ec80 con 0x55f66315e800 > -8858> 2018-07-11 08:04:38.748452 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0d000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 187 0x55f6780b2800 osd_ping(ping e509 stamp 2018-07-11 08:04:39.992249) v4 > -8857> 2018-07-11 08:04:38.748465 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0e800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 187 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:04:39.992249) v4 > -8856> 2018-07-11 08:04:38.748474 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.5 10.8.128.113:0/15263 187 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:39.992249) v4 ==== 2004+0+0 (3311763637 0 0) 0x55f6780b2800 con 0x55f663f0d000 > -8855> 2018-07-11 08:04:38.748482 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.5 10.8.128.113:0/15263 187 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:39.992249) v4 ==== 2004+0+0 (3311763637 0 0) 0x55f67d1afe00 con 0x55f663f0e800 > -8854> 2018-07-11 08:04:38.748489 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:39.992249) v4 -- 0x55f653d3c200 con 0 > -8853> 2018-07-11 08:04:38.748516 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:39.992249) v4 -- 0x55f65e71d600 con 0 > -8852> 2018-07-11 08:04:39.096295 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0a000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 179 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:04:39.731549) v4 > -8851> 2018-07-11 08:04:39.096331 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.11 10.8.128.114:0/15919 179 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:39.731549) v4 ==== 2004+0+0 (980035411 0 0) 0x55f64e20fc00 con 0x55f663f0a000 > -8850> 2018-07-11 08:04:39.096317 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0b800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 179 0x55f666c47600 osd_ping(ping e509 stamp 2018-07-11 08:04:39.731549) v4 > -8849> 2018-07-11 08:04:39.096342 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:39.731549) v4 -- 0x55f65e71d600 con 0 > -8848> 2018-07-11 08:04:39.096347 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.11 10.8.128.114:0/15919 179 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:39.731549) v4 ==== 2004+0+0 (980035411 0 0) 0x55f666c47600 con 0x55f663f0b800 > -8847> 2018-07-11 08:04:39.096367 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:39.731549) v4 -- 0x55f65e71de00 con 0 > -8846> 2018-07-11 08:04:39.541362 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f64f2b46c0 con 0 > -8845> 2018-07-11 08:04:39.543053 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 177 0x55f6631d6000 replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:509/504,start:6:a80e3a46:::benchmark_data_magna108.ceph.redhat.com_2478_object249933:0,end:6:a80f79b5:::benchmark_data_magna108_35617_object603713:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -8844> 2018-07-11 08:04:39.543084 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 177 ==== replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:509/504,start:6:a80e3a46:::benchmark_data_magna108.ceph.redhat.com_2478_object249933:0,end:6:a80f79b5:::benchmark_data_magna108_35617_object603713:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (2180560712 0 0) 0x55f6631d6000 con 0x55f66315e800 > -8843> 2018-07-11 08:04:39.596761 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0a000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 180 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:04:40.231985) v4 > -8842> 2018-07-11 08:04:39.596785 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.11 10.8.128.114:0/15919 180 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:40.231985) v4 ==== 2004+0+0 (294077089 0 0) 0x55f67d1afe00 con 0x55f663f0a000 > -8841> 2018-07-11 08:04:39.596774 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0b800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 180 0x55f6548fba00 osd_ping(ping e509 stamp 2018-07-11 08:04:40.231985) v4 > -8840> 2018-07-11 08:04:39.596798 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:40.231985) v4 -- 0x55f65e71d600 con 0 > -8839> 2018-07-11 08:04:39.596798 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.11 10.8.128.114:0/15919 180 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:40.231985) v4 ==== 2004+0+0 (294077089 0 0) 0x55f6548fba00 con 0x55f663f0b800 > -8838> 2018-07-11 08:04:39.596825 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:40.231985) v4 -- 0x55f65e71de00 con 0 > -8837> 2018-07-11 08:04:39.945968 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12651 conn(0x55f663219000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25 cs=1 l=1). rx osd.2 seq 195 0x55f660948e00 osd_ping(ping e509 stamp 2018-07-11 08:04:40.002025) v4 > -8836> 2018-07-11 08:04:39.946000 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.2 10.8.128.106:0/12651 195 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:40.002025) v4 ==== 2004+0+0 (2816668670 0 0) 0x55f660948e00 con 0x55f663219000 > -8835> 2018-07-11 08:04:39.946012 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:40.002025) v4 -- 0x55f64cc80400 con 0 > -8834> 2018-07-11 08:04:39.946028 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12651 conn(0x55f66321a800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25 cs=1 l=1). rx osd.2 seq 195 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:04:40.002025) v4 > -8833> 2018-07-11 08:04:39.946050 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.2 10.8.128.106:0/12651 195 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:40.002025) v4 ==== 2004+0+0 (2816668670 0 0) 0x55f64e20fc00 con 0x55f66321a800 > -8832> 2018-07-11 08:04:39.946062 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:40.002025) v4 -- 0x55f64cf24000 con 0 > -8831> 2018-07-11 08:04:39.957998 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 401 0x55f666c47600 MOSDScrubReserve(11.3s5 REQUEST e509) v1 > -8830> 2018-07-11 08:04:39.958015 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 401 ==== MOSDScrubReserve(11.3s5 REQUEST e509) v1 ==== 43+0+0 (433425070 0 0) 0x55f666c47600 con 0x55f668a29000 > -8829> 2018-07-11 08:04:39.958142 7f491a040700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e509) v1 -- 0x55f667d09200 con 0 > -8828> 2018-07-11 08:04:39.958327 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 402 0x55f65dad4200 MOSDScrubReserve(11.3s5 RELEASE e509) v1 > -8827> 2018-07-11 08:04:39.958342 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 402 ==== MOSDScrubReserve(11.3s5 RELEASE e509) v1 ==== 43+0+0 (3603460268 0 0) 0x55f65dad4200 con 0x55f668a29000 > -8826> 2018-07-11 08:04:39.992750 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.108:0/11967 conn(0x55f663f6f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=1). rx osd.8 seq 178 0x55f6548fba00 osd_ping(ping e509 stamp 2018-07-11 08:04:39.752410) v4 > -8825> 2018-07-11 08:04:39.992772 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.8 10.8.128.108:0/11967 178 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:39.752410) v4 ==== 2004+0+0 (2724906184 0 0) 0x55f6548fba00 con 0x55f663f6f000 > -8824> 2018-07-11 08:04:39.992781 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:39.752410) v4 -- 0x55f669debc00 con 0 > -8823> 2018-07-11 08:04:39.992843 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.108:0/11967 conn(0x55f663f70800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=1). rx osd.8 seq 178 0x55f6780b2800 osd_ping(ping e509 stamp 2018-07-11 08:04:39.752410) v4 > -8822> 2018-07-11 08:04:39.992866 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.8 10.8.128.108:0/11967 178 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:39.752410) v4 ==== 2004+0+0 (2724906184 0 0) 0x55f6780b2800 con 0x55f663f70800 > -8821> 2018-07-11 08:04:39.992882 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:39.752410) v4 -- 0x55f64cc80400 con 0 > -8820> 2018-07-11 08:04:40.025176 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15885 conn(0x55f663217800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 185 0x55f667d09200 osd_ping(ping e509 stamp 2018-07-11 08:04:40.660394) v4 > -8819> 2018-07-11 08:04:40.025197 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.7 10.8.128.114:0/15885 185 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:40.660394) v4 ==== 2004+0+0 (4039839855 0 0) 0x55f667d09200 con 0x55f663217800 > -8818> 2018-07-11 08:04:40.025202 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15885 conn(0x55f663216000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 185 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:04:40.660394) v4 > -8817> 2018-07-11 08:04:40.025213 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:40.660394) v4 -- 0x55f669debc00 con 0 > -8816> 2018-07-11 08:04:40.025224 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.7 10.8.128.114:0/15885 185 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:40.660394) v4 ==== 2004+0+0 (4039839855 0 0) 0x55f67d1afe00 con 0x55f663216000 > -8815> 2018-07-11 08:04:40.025243 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:40.660394) v4 -- 0x55f64cf24000 con 0 > -8814> 2018-07-11 08:04:40.069096 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15255 conn(0x55f663ec2800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.0 seq 176 0x55f6548fba00 osd_ping(ping e509 stamp 2018-07-11 08:04:41.312876) v4 > -8813> 2018-07-11 08:04:40.069116 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.0 10.8.128.113:0/15255 176 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:41.312876) v4 ==== 2004+0+0 (4163969510 0 0) 0x55f6548fba00 con 0x55f663ec2800 > -8812> 2018-07-11 08:04:40.069113 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15255 conn(0x55f663ec1000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.0 seq 176 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:04:41.312876) v4 > -8811> 2018-07-11 08:04:40.069126 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:41.312876) v4 -- 0x55f669debc00 con 0 > -8810> 2018-07-11 08:04:40.069138 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.0 10.8.128.113:0/15255 176 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:41.312876) v4 ==== 2004+0+0 (4163969510 0 0) 0x55f64e20fc00 con 0x55f663ec1000 > -8809> 2018-07-11 08:04:40.069153 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:41.312876) v4 -- 0x55f64cf24000 con 0 > -8808> 2018-07-11 08:04:40.368281 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed7800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 182 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:04:40.424529) v4 > -8807> 2018-07-11 08:04:40.368302 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.1 10.8.128.106:0/12677 182 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:40.424529) v4 ==== 2004+0+0 (1393329454 0 0) 0x55f67d1afe00 con 0x55f663ed7800 > -8806> 2018-07-11 08:04:40.368301 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed6000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 182 0x55f660948e00 osd_ping(ping e509 stamp 2018-07-11 08:04:40.424529) v4 > -8805> 2018-07-11 08:04:40.368314 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:40.424529) v4 -- 0x55f66b1cd600 con 0 > -8804> 2018-07-11 08:04:40.368325 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.1 10.8.128.106:0/12677 182 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:40.424529) v4 ==== 2004+0+0 (1393329454 0 0) 0x55f660948e00 con 0x55f663ed6000 > -8803> 2018-07-11 08:04:40.368340 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:40.424529) v4 -- 0x55f65d025800 con 0 > -8802> 2018-07-11 08:04:40.432984 7f491e048700 10 monclient: tick > -8801> 2018-07-11 08:04:40.432997 7f491e048700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2018-07-11 08:04:10.432996) > -8800> 2018-07-11 08:04:40.482366 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 28] Generated table #37030: 2638241 keys, 74546353 bytes > -8799> 2018-07-11 08:04:40.482415 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296280482399, "cf_name": "default", "job": 28, "event": "table_file_creation", "file_number": 37030, "file_size": 74546353, "table_properties": {"data_size": 67112158, "index_size": 690022, "filter_size": 6743183, "raw_key_size": 76215855, "raw_average_key_size": 28, "raw_value_size": 31365758, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} > -8798> 2018-07-11 08:04:40.524883 7f491d046700 1 -- 10.8.128.107:6800/14173 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55f6631f1600 con 0 > -8797> 2018-07-11 08:04:40.524970 7f491d046700 1 -- 10.8.128.107:6800/14173 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55f66dc58840 con 0 > -8796> 2018-07-11 08:04:40.877066 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.107:0/12617 conn(0x55f64e3c5000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.9 seq 184 0x55f660948e00 osd_ping(ping e509 stamp 2018-07-11 08:04:40.876699) v4 > -8795> 2018-07-11 08:04:40.877066 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.107:0/12617 conn(0x55f64e3c3800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.9 seq 184 0x55f667d09200 osd_ping(ping e509 stamp 2018-07-11 08:04:40.876699) v4 > -8794> 2018-07-11 08:04:40.877096 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.9 10.8.128.107:0/12617 184 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:40.876699) v4 ==== 2004+0+0 (2133434086 0 0) 0x55f660948e00 con 0x55f64e3c5000 > -8793> 2018-07-11 08:04:40.877100 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.9 10.8.128.107:0/12617 184 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:40.876699) v4 ==== 2004+0+0 (2133434086 0 0) 0x55f667d09200 con 0x55f64e3c3800 > -8792> 2018-07-11 08:04:40.877109 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:40.876699) v4 -- 0x55f64c7d2e00 con 0 > -8791> 2018-07-11 08:04:40.877138 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:40.876699) v4 -- 0x55f653d4a200 con 0 > -8790> 2018-07-11 08:04:40.996797 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15287 conn(0x55f66dce2800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.4 seq 192 0x55f6780b2800 osd_ping(ping e509 stamp 2018-07-11 08:04:42.240599) v4 > -8789> 2018-07-11 08:04:40.996820 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.4 10.8.128.113:0/15287 192 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:42.240599) v4 ==== 2004+0+0 (3652684180 0 0) 0x55f6780b2800 con 0x55f66dce2800 > -8788> 2018-07-11 08:04:40.996829 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15287 conn(0x55f66dce1000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.4 seq 192 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:04:42.240599) v4 > -8787> 2018-07-11 08:04:40.996838 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:42.240599) v4 -- 0x55f669deac00 con 0 > -8786> 2018-07-11 08:04:40.996853 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.4 10.8.128.113:0/15287 192 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:42.240599) v4 ==== 2004+0+0 (3652684180 0 0) 0x55f64e20fc00 con 0x55f66dce1000 > -8785> 2018-07-11 08:04:40.996873 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:42.240599) v4 -- 0x55f65dad4e00 con 0 > -8784> 2018-07-11 08:04:41.104097 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.115:0/13323 conn(0x55f663ebf800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.6 seq 175 0x55f660948e00 osd_ping(ping e509 stamp 2018-07-11 08:04:41.456339) v4 > -8783> 2018-07-11 08:04:41.104119 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.6 10.8.128.115:0/13323 175 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:41.456339) v4 ==== 2004+0+0 (3261296512 0 0) 0x55f660948e00 con 0x55f663ebf800 > -8782> 2018-07-11 08:04:41.104129 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:41.456339) v4 -- 0x55f653d3d800 con 0 > -8781> 2018-07-11 08:04:41.104188 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.115:0/13323 conn(0x55f663ebe000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.6 seq 175 0x55f6548fba00 osd_ping(ping e509 stamp 2018-07-11 08:04:41.456339) v4 > -8780> 2018-07-11 08:04:41.104212 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.6 10.8.128.115:0/13323 175 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:41.456339) v4 ==== 2004+0+0 (3261296512 0 0) 0x55f6548fba00 con 0x55f663ebe000 > -8779> 2018-07-11 08:04:41.104230 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:41.456339) v4 -- 0x55f653d4bc00 con 0 > -8778> 2018-07-11 08:04:41.111855 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f66b5f5680 con 0 > -8777> 2018-07-11 08:04:41.113460 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 178 0x55f667d83b80 replica scrub(pg: 6.15,from:0'0,to:430'106116,epoch:509/504,start:6:a80f79b5:::benchmark_data_magna108_35617_object603713:0,end:6:a810519e:::benchmark_data_magna108_35617_object774363:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -8776> 2018-07-11 08:04:41.113490 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 178 ==== replica scrub(pg: 6.15,from:0'0,to:430'106116,epoch:509/504,start:6:a80f79b5:::benchmark_data_magna108_35617_object603713:0,end:6:a810519e:::benchmark_data_magna108_35617_object774363:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (364426712 0 0) 0x55f667d83b80 con 0x55f66315e800 > -8775> 2018-07-11 08:04:41.169489 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15255 conn(0x55f663ec1000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.0 seq 177 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:04:42.413250) v4 > -8774> 2018-07-11 08:04:41.169513 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.0 10.8.128.113:0/15255 177 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:42.413250) v4 ==== 2004+0+0 (3086089803 0 0) 0x55f67d1afe00 con 0x55f663ec1000 > -8773> 2018-07-11 08:04:41.169530 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:42.413250) v4 -- 0x55f65dad4e00 con 0 > -8772> 2018-07-11 08:04:41.169534 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15255 conn(0x55f663ec2800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.0 seq 177 0x55f667d09200 osd_ping(ping e509 stamp 2018-07-11 08:04:42.413250) v4 > -8771> 2018-07-11 08:04:41.169554 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.0 10.8.128.113:0/15255 177 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:42.413250) v4 ==== 2004+0+0 (3086089803 0 0) 0x55f667d09200 con 0x55f663ec2800 > -8770> 2018-07-11 08:04:41.169565 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:42.413250) v4 -- 0x55f66dceb800 con 0 > -8769> 2018-07-11 08:04:41.623493 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12650 conn(0x55f663eda800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.3 seq 183 0x55f6780b2800 osd_ping(ping e509 stamp 2018-07-11 08:04:41.679734) v4 > -8768> 2018-07-11 08:04:41.623523 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12650 conn(0x55f663ed9000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.3 seq 183 0x55f6548fba00 osd_ping(ping e509 stamp 2018-07-11 08:04:41.679734) v4 > -8767> 2018-07-11 08:04:41.623533 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.3 10.8.128.106:0/12650 183 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:41.679734) v4 ==== 2004+0+0 (1202702335 0 0) 0x55f6780b2800 con 0x55f663eda800 > -8766> 2018-07-11 08:04:41.623541 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.3 10.8.128.106:0/12650 183 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:41.679734) v4 ==== 2004+0+0 (1202702335 0 0) 0x55f6548fba00 con 0x55f663ed9000 > -8765> 2018-07-11 08:04:41.623553 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:41.679734) v4 -- 0x55f65dad5800 con 0 > -8764> 2018-07-11 08:04:41.623587 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:41.679734) v4 -- 0x55f663fb9a00 con 0 > -8763> 2018-07-11 08:04:41.656343 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.114:6805/15885 conn(0x55f66315a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.7 seq 2515 0x55f653d3d800 MOSDScrubReserve(5.4 REQUEST e509) v1 > -8762> 2018-07-11 08:04:41.656366 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.7 10.8.128.114:6805/15885 2515 ==== MOSDScrubReserve(5.4 REQUEST e509) v1 ==== 43+0+0 (3003630469 0 0) 0x55f653d3d800 con 0x55f66315a000 > -8761> 2018-07-11 08:04:41.656451 7f491983f700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.114:6805/15885 -- MOSDScrubReserve(5.4 REJECT e509) v1 -- 0x55f6548fbe00 con 0 > -8760> 2018-07-11 08:04:41.656789 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.114:6805/15885 conn(0x55f66315a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.7 seq 2516 0x55f6548fbe00 MOSDScrubReserve(5.4 RELEASE e509) v1 > -8759> 2018-07-11 08:04:41.656807 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.7 10.8.128.114:6805/15885 2516 ==== MOSDScrubReserve(5.4 RELEASE e509) v1 ==== 43+0+0 (2082093447 0 0) 0x55f6548fbe00 con 0x55f66315a000 > -8758> 2018-07-11 08:04:42.095917 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f66c94afc0 con 0 > -8757> 2018-07-11 08:04:42.097678 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 179 0x55f6631d6780 replica scrub(pg: 6.15,from:0'0,to:430'106703,epoch:509/504,start:6:a810519e:::benchmark_data_magna108_35617_object774363:0,end:6:a8112d76:::benchmark_data_magna108.ceph.redhat.com_2478_object147019:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -8756> 2018-07-11 08:04:42.097711 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 179 ==== replica scrub(pg: 6.15,from:0'0,to:430'106703,epoch:509/504,start:6:a810519e:::benchmark_data_magna108_35617_object774363:0,end:6:a8112d76:::benchmark_data_magna108.ceph.redhat.com_2478_object147019:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (1638683144 0 0) 0x55f6631d6780 con 0x55f66315e800 > -8755> 2018-07-11 08:04:42.628872 7f491803c700 5 osd.10 509 heartbeat: osd_stat(788 GB used, 142 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,0,0,1]) > -8754> 2018-07-11 08:04:42.628892 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6802/15255 -- osd_ping(ping e509 stamp 2018-07-11 08:04:42.628889) v4 -- 0x55f663c0b200 con 0 > -8753> 2018-07-11 08:04:42.628933 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6803/15255 -- osd_ping(ping e509 stamp 2018-07-11 08:04:42.628889) v4 -- 0x55f66b1cd800 con 0 > -8752> 2018-07-11 08:04:42.628954 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6806/12677 -- osd_ping(ping e509 stamp 2018-07-11 08:04:42.628889) v4 -- 0x55f66b1cce00 con 0 > -8751> 2018-07-11 08:04:42.628989 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6807/12677 -- osd_ping(ping e509 stamp 2018-07-11 08:04:42.628889) v4 -- 0x55f66b1cc600 con 0 > -8750> 2018-07-11 08:04:42.629007 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6808/12651 -- osd_ping(ping e509 stamp 2018-07-11 08:04:42.628889) v4 -- 0x55f673952000 con 0 > -8749> 2018-07-11 08:04:42.629025 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6810/12651 -- osd_ping(ping e509 stamp 2018-07-11 08:04:42.628889) v4 -- 0x55f673952200 con 0 > -8748> 2018-07-11 08:04:42.629049 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6809/12650 -- osd_ping(ping e509 stamp 2018-07-11 08:04:42.628889) v4 -- 0x55f673952400 con 0 > -8747> 2018-07-11 08:04:42.629069 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6811/12650 -- osd_ping(ping e509 stamp 2018-07-11 08:04:42.628889) v4 -- 0x55f673952600 con 0 > -8746> 2018-07-11 08:04:42.629087 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6810/15287 -- osd_ping(ping e509 stamp 2018-07-11 08:04:42.628889) v4 -- 0x55f673952800 con 0 > -8745> 2018-07-11 08:04:42.629108 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6811/15287 -- osd_ping(ping e509 stamp 2018-07-11 08:04:42.628889) v4 -- 0x55f673952a00 con 0 > -8744> 2018-07-11 08:04:42.629126 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6806/15263 -- osd_ping(ping e509 stamp 2018-07-11 08:04:42.628889) v4 -- 0x55f673952c00 con 0 > -8743> 2018-07-11 08:04:42.629144 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6807/15263 -- osd_ping(ping e509 stamp 2018-07-11 08:04:42.628889) v4 -- 0x55f673952e00 con 0 > -8742> 2018-07-11 08:04:42.629163 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.115:6802/13323 -- osd_ping(ping e509 stamp 2018-07-11 08:04:42.628889) v4 -- 0x55f673953000 con 0 > -8741> 2018-07-11 08:04:42.629181 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.115:6803/13323 -- osd_ping(ping e509 stamp 2018-07-11 08:04:42.628889) v4 -- 0x55f673953200 con 0 > -8740> 2018-07-11 08:04:42.629197 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6806/15885 -- osd_ping(ping e509 stamp 2018-07-11 08:04:42.628889) v4 -- 0x55f673953400 con 0 > -8739> 2018-07-11 08:04:42.629216 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6807/15885 -- osd_ping(ping e509 stamp 2018-07-11 08:04:42.628889) v4 -- 0x55f673953600 con 0 > -8738> 2018-07-11 08:04:42.629232 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.108:6802/11967 -- osd_ping(ping e509 stamp 2018-07-11 08:04:42.628889) v4 -- 0x55f673953800 con 0 > -8737> 2018-07-11 08:04:42.629262 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.108:6803/11967 -- osd_ping(ping e509 stamp 2018-07-11 08:04:42.628889) v4 -- 0x55f673953a00 con 0 > -8736> 2018-07-11 08:04:42.629283 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.107:6806/12617 -- osd_ping(ping e509 stamp 2018-07-11 08:04:42.628889) v4 -- 0x55f673953c00 con 0 > -8735> 2018-07-11 08:04:42.629301 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.107:6807/12617 -- osd_ping(ping e509 stamp 2018-07-11 08:04:42.628889) v4 -- 0x55f673953e00 con 0 > -8734> 2018-07-11 08:04:42.629321 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6802/15919 -- osd_ping(ping e509 stamp 2018-07-11 08:04:42.628889) v4 -- 0x55f66cb82000 con 0 > -8733> 2018-07-11 08:04:42.629338 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6803/15919 -- osd_ping(ping e509 stamp 2018-07-11 08:04:42.628889) v4 -- 0x55f66cb82200 con 0 > -8732> 2018-07-11 08:04:42.629386 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6803/15255 conn(0x55f656ad5800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 183 0x55f667d09200 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:42.628889) v4 > -8731> 2018-07-11 08:04:42.629392 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6802/15255 conn(0x55f656ad4000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 183 0x55f64cc80c00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:42.628889) v4 > -8730> 2018-07-11 08:04:42.629409 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.0 10.8.128.113:6803/15255 183 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:42.628889) v4 ==== 2004+0+0 (4178289131 0 0) 0x55f667d09200 con 0x55f656ad5800 > -8729> 2018-07-11 08:04:42.629419 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.0 10.8.128.113:6802/15255 183 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:42.628889) v4 ==== 2004+0+0 (4178289131 0 0) 0x55f64cc80c00 con 0x55f656ad4000 > -8728> 2018-07-11 08:04:42.629519 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6810/12651 conn(0x55f64e3eb800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.2 seq 183 0x55f6548fba00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:42.628889) v4 > -8727> 2018-07-11 08:04:42.629516 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6810/15287 conn(0x55f64e401000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.4 seq 183 0x55f64cc80c00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:42.628889) v4 > -8726> 2018-07-11 08:04:42.629552 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.2 10.8.128.106:6810/12651 183 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:42.628889) v4 ==== 2004+0+0 (4178289131 0 0) 0x55f6548fba00 con 0x55f64e3eb800 > -8725> 2018-07-11 08:04:42.629588 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6811/15287 conn(0x55f64e402800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.4 seq 183 0x55f6548fba00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:42.628889) v4 > -8724> 2018-07-11 08:04:42.629558 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.4 10.8.128.113:6810/15287 183 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:42.628889) v4 ==== 2004+0+0 (4178289131 0 0) 0x55f64cc80c00 con 0x55f64e401000 > -8723> 2018-07-11 08:04:42.629554 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6809/12650 conn(0x55f656ad7000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.3 seq 183 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:42.628889) v4 > -8722> 2018-07-11 08:04:42.629616 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.4 10.8.128.113:6811/15287 183 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:42.628889) v4 ==== 2004+0+0 (4178289131 0 0) 0x55f6548fba00 con 0x55f64e402800 > -8721> 2018-07-11 08:04:42.629624 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.3 10.8.128.106:6809/12650 183 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:42.628889) v4 ==== 2004+0+0 (4178289131 0 0) 0x55f64e20fc00 con 0x55f656ad7000 > -8720> 2018-07-11 08:04:42.629642 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6807/12677 conn(0x55f64e3d9800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.1 seq 183 0x55f64cc80c00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:42.628889) v4 > -8719> 2018-07-11 08:04:42.629652 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6806/15885 conn(0x55f64e3fe000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.7 seq 183 0x55f6548fba00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:42.628889) v4 > -8718> 2018-07-11 08:04:42.629662 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.7 10.8.128.114:6806/15885 183 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:42.628889) v4 ==== 2004+0+0 (4178289131 0 0) 0x55f6548fba00 con 0x55f64e3fe000 > -8717> 2018-07-11 08:04:42.629671 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6806/12677 conn(0x55f64e3d8000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.1 seq 183 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:42.628889) v4 > -8716> 2018-07-11 08:04:42.629662 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.1 10.8.128.106:6807/12677 183 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:42.628889) v4 ==== 2004+0+0 (4178289131 0 0) 0x55f64cc80c00 con 0x55f64e3d9800 > -8715> 2018-07-11 08:04:42.629690 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.1 10.8.128.106:6806/12677 183 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:42.628889) v4 ==== 2004+0+0 (4178289131 0 0) 0x55f64e20fc00 con 0x55f64e3d8000 > -8714> 2018-07-11 08:04:42.629687 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.107:6806/12617 conn(0x55f656ade000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.9 seq 183 0x55f6548fba00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:42.628889) v4 > -8713> 2018-07-11 08:04:42.629714 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.9 10.8.128.107:6806/12617 183 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:42.628889) v4 ==== 2004+0+0 (4178289131 0 0) 0x55f6548fba00 con 0x55f656ade000 > -8712> 2018-07-11 08:04:42.629717 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.107:6807/12617 conn(0x55f656adf800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.9 seq 183 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:42.628889) v4 > -8711> 2018-07-11 08:04:42.629726 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.9 10.8.128.107:6807/12617 183 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:42.628889) v4 ==== 2004+0+0 (4178289131 0 0) 0x55f64e20fc00 con 0x55f656adf800 > -8710> 2018-07-11 08:04:42.629726 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6811/12650 conn(0x55f656ad8800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.3 seq 183 0x55f64cc80c00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:42.628889) v4 > -8709> 2018-07-11 08:04:42.629744 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6806/15263 conn(0x55f64e3ed000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.5 seq 183 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:42.628889) v4 > -8708> 2018-07-11 08:04:42.629771 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.3 10.8.128.106:6811/12650 183 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:42.628889) v4 ==== 2004+0+0 (4178289131 0 0) 0x55f64cc80c00 con 0x55f656ad8800 > -8707> 2018-07-11 08:04:42.629752 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.115:6802/13323 conn(0x55f656acd000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.6 seq 183 0x55f6548fba00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:42.628889) v4 > -8706> 2018-07-11 08:04:42.629782 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.5 10.8.128.113:6806/15263 183 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:42.628889) v4 ==== 2004+0+0 (4178289131 0 0) 0x55f64e20fc00 con 0x55f64e3ed000 > -8705> 2018-07-11 08:04:42.629785 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.6 10.8.128.115:6802/13323 183 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:42.628889) v4 ==== 2004+0+0 (4178289131 0 0) 0x55f6548fba00 con 0x55f656acd000 > -8704> 2018-07-11 08:04:42.629812 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6808/12651 conn(0x55f64e3ea000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.2 seq 183 0x55f64cc80c00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:42.628889) v4 > -8703> 2018-07-11 08:04:42.629820 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.115:6803/13323 conn(0x55f656ace800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.6 seq 183 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:42.628889) v4 > -8702> 2018-07-11 08:04:42.629827 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6802/15919 conn(0x55f64e3db000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.11 seq 183 0x55f6548fba00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:42.628889) v4 > -8701> 2018-07-11 08:04:42.629835 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.6 10.8.128.115:6803/13323 183 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:42.628889) v4 ==== 2004+0+0 (4178289131 0 0) 0x55f64e20fc00 con 0x55f656ace800 > -8700> 2018-07-11 08:04:42.629825 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.2 10.8.128.106:6808/12651 183 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:42.628889) v4 ==== 2004+0+0 (4178289131 0 0) 0x55f64cc80c00 con 0x55f64e3ea000 > -8699> 2018-07-11 08:04:42.629842 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.11 10.8.128.114:6802/15919 183 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:42.628889) v4 ==== 2004+0+0 (4178289131 0 0) 0x55f6548fba00 con 0x55f64e3db000 > -8698> 2018-07-11 08:04:42.629864 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6807/15885 conn(0x55f64e3ff800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.7 seq 183 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:42.628889) v4 > -8697> 2018-07-11 08:04:42.629879 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.7 10.8.128.114:6807/15885 183 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:42.628889) v4 ==== 2004+0+0 (4178289131 0 0) 0x55f64e20fc00 con 0x55f64e3ff800 > -8696> 2018-07-11 08:04:42.629877 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6807/15263 conn(0x55f64e3ee800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.5 seq 183 0x55f64cc80c00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:42.628889) v4 > -8695> 2018-07-11 08:04:42.629896 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.5 10.8.128.113:6807/15263 183 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:42.628889) v4 ==== 2004+0+0 (4178289131 0 0) 0x55f64cc80c00 con 0x55f64e3ee800 > -8694> 2018-07-11 08:04:42.629915 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.108:6802/11967 conn(0x55f656aca000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.8 seq 183 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:42.628889) v4 > -8693> 2018-07-11 08:04:42.629932 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.8 10.8.128.108:6802/11967 183 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:42.628889) v4 ==== 2004+0+0 (4178289131 0 0) 0x55f64e20fc00 con 0x55f656aca000 > -8692> 2018-07-11 08:04:42.629947 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.108:6803/11967 conn(0x55f656acb800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.8 seq 183 0x55f64cc80c00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:42.628889) v4 > -8691> 2018-07-11 08:04:42.629967 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.8 10.8.128.108:6803/11967 183 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:42.628889) v4 ==== 2004+0+0 (4178289131 0 0) 0x55f64cc80c00 con 0x55f656acb800 > -8690> 2018-07-11 08:04:42.629969 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6803/15919 conn(0x55f64e3dc800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.11 seq 183 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:42.628889) v4 > -8689> 2018-07-11 08:04:42.629989 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.11 10.8.128.114:6803/15919 183 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:42.628889) v4 ==== 2004+0+0 (4178289131 0 0) 0x55f64e20fc00 con 0x55f64e3dc800 > -8688> 2018-07-11 08:04:42.656494 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.114:6805/15885 conn(0x55f66315a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.7 seq 2517 0x55f64cc80c00 MOSDScrubReserve(5.4 REQUEST e509) v1 > -8687> 2018-07-11 08:04:42.656509 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.7 10.8.128.114:6805/15885 2517 ==== MOSDScrubReserve(5.4 REQUEST e509) v1 ==== 43+0+0 (3003630469 0 0) 0x55f64cc80c00 con 0x55f66315a000 > -8686> 2018-07-11 08:04:42.656588 7f491983f700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.114:6805/15885 -- MOSDScrubReserve(5.4 REJECT e509) v1 -- 0x55f6548fbe00 con 0 > -8685> 2018-07-11 08:04:42.656917 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.114:6805/15885 conn(0x55f66315a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.7 seq 2518 0x55f6548fbe00 MOSDScrubReserve(5.4 RELEASE e509) v1 > -8684> 2018-07-11 08:04:42.656931 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.7 10.8.128.114:6805/15885 2518 ==== MOSDScrubReserve(5.4 RELEASE e509) v1 ==== 43+0+0 (2082093447 0 0) 0x55f6548fbe00 con 0x55f66315a000 > -8683> 2018-07-11 08:04:42.796912 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f65f619440 con 0 > -8682> 2018-07-11 08:04:42.798435 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 180 0x55f6631d6500 replica scrub(pg: 6.15,from:0'0,to:461'106883,epoch:509/504,start:6:a8112d76:::benchmark_data_magna108.ceph.redhat.com_2478_object147019:0,end:6:a811fbc3:::benchmark_data_magna108_35617_object588840:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -8681> 2018-07-11 08:04:42.798465 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 180 ==== replica scrub(pg: 6.15,from:0'0,to:461'106883,epoch:509/504,start:6:a8112d76:::benchmark_data_magna108.ceph.redhat.com_2478_object147019:0,end:6:a811fbc3:::benchmark_data_magna108_35617_object588840:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (353417170 0 0) 0x55f6631d6500 con 0x55f66315e800 > -8680> 2018-07-11 08:04:42.893179 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.108:0/11967 conn(0x55f663f6f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=1). rx osd.8 seq 179 0x55f6548fba00 osd_ping(ping e509 stamp 2018-07-11 08:04:42.652858) v4 > -8679> 2018-07-11 08:04:42.893204 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.8 10.8.128.108:0/11967 179 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:42.652858) v4 ==== 2004+0+0 (1074531142 0 0) 0x55f6548fba00 con 0x55f663f6f000 > -8678> 2018-07-11 08:04:42.893198 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.108:0/11967 conn(0x55f663f70800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=1). rx osd.8 seq 179 0x55f6780b2800 osd_ping(ping e509 stamp 2018-07-11 08:04:42.652858) v4 > -8677> 2018-07-11 08:04:42.893218 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:42.652858) v4 -- 0x55f673953000 con 0 > -8676> 2018-07-11 08:04:42.893217 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.8 10.8.128.108:0/11967 179 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:42.652858) v4 ==== 2004+0+0 (1074531142 0 0) 0x55f6780b2800 con 0x55f663f70800 > -8675> 2018-07-11 08:04:42.893253 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:42.652858) v4 -- 0x55f673952000 con 0 > -8674> 2018-07-11 08:04:42.925669 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15885 conn(0x55f663216000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 186 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:04:43.560898) v4 > -8673> 2018-07-11 08:04:42.925671 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15885 conn(0x55f663217800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 186 0x55f667d09200 osd_ping(ping e509 stamp 2018-07-11 08:04:43.560898) v4 > -8672> 2018-07-11 08:04:42.925697 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.7 10.8.128.114:0/15885 186 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:43.560898) v4 ==== 2004+0+0 (1184921148 0 0) 0x55f667d09200 con 0x55f663217800 > -8671> 2018-07-11 08:04:42.925694 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.7 10.8.128.114:0/15885 186 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:43.560898) v4 ==== 2004+0+0 (1184921148 0 0) 0x55f64e20fc00 con 0x55f663216000 > -8670> 2018-07-11 08:04:42.925715 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:43.560898) v4 -- 0x55f673953000 con 0 > -8669> 2018-07-11 08:04:42.925738 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:43.560898) v4 -- 0x55f673952c00 con 0 > -8668> 2018-07-11 08:04:43.446465 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12651 conn(0x55f663219000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25 cs=1 l=1). rx osd.2 seq 196 0x55f673952600 osd_ping(ping e509 stamp 2018-07-11 08:04:43.502684) v4 > -8667> 2018-07-11 08:04:43.446506 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.2 10.8.128.106:0/12651 196 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:43.502684) v4 ==== 2004+0+0 (1454679009 0 0) 0x55f673952600 con 0x55f663219000 > -8666> 2018-07-11 08:04:43.446518 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:43.502684) v4 -- 0x55f673952000 con 0 > -8665> 2018-07-11 08:04:43.446530 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12651 conn(0x55f66321a800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25 cs=1 l=1). rx osd.2 seq 196 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:04:43.502684) v4 > -8664> 2018-07-11 08:04:43.446559 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.2 10.8.128.106:0/12651 196 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:43.502684) v4 ==== 2004+0+0 (1454679009 0 0) 0x55f67d1afe00 con 0x55f66321a800 > -8663> 2018-07-11 08:04:43.446581 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:43.502684) v4 -- 0x55f673952400 con 0 > -8662> 2018-07-11 08:04:43.448797 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0d000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 188 0x55f6780b2800 osd_ping(ping e509 stamp 2018-07-11 08:04:44.692657) v4 > -8661> 2018-07-11 08:04:43.448818 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.5 10.8.128.113:0/15263 188 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:44.692657) v4 ==== 2004+0+0 (2446263281 0 0) 0x55f6780b2800 con 0x55f663f0d000 > -8660> 2018-07-11 08:04:43.448831 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:44.692657) v4 -- 0x55f66b1cc600 con 0 > -8659> 2018-07-11 08:04:43.448821 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0e800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 188 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:04:44.692657) v4 > -8658> 2018-07-11 08:04:43.448853 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.5 10.8.128.113:0/15263 188 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:44.692657) v4 ==== 2004+0+0 (2446263281 0 0) 0x55f64e20fc00 con 0x55f663f0e800 > -8657> 2018-07-11 08:04:43.448870 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:44.692657) v4 -- 0x55f66b1cce00 con 0 > -8656> 2018-07-11 08:04:43.656737 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.114:6805/15885 conn(0x55f66315a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.7 seq 2519 0x55f673952600 MOSDScrubReserve(5.4 REQUEST e509) v1 > -8655> 2018-07-11 08:04:43.656768 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.7 10.8.128.114:6805/15885 2519 ==== MOSDScrubReserve(5.4 REQUEST e509) v1 ==== 43+0+0 (3003630469 0 0) 0x55f673952600 con 0x55f66315a000 > -8654> 2018-07-11 08:04:43.656840 7f491983f700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.114:6805/15885 -- MOSDScrubReserve(5.4 REJECT e509) v1 -- 0x55f6548fbe00 con 0 > -8653> 2018-07-11 08:04:43.657176 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.114:6805/15885 conn(0x55f66315a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.7 seq 2520 0x55f6548fbe00 MOSDScrubReserve(5.4 RELEASE e509) v1 > -8652> 2018-07-11 08:04:43.657211 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.7 10.8.128.114:6805/15885 2520 ==== MOSDScrubReserve(5.4 RELEASE e509) v1 ==== 43+0+0 (2082093447 0 0) 0x55f6548fbe00 con 0x55f66315a000 > -8651> 2018-07-11 08:04:43.701793 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f66e918b40 con 0 > -8650> 2018-07-11 08:04:43.724443 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 181 0x55f6631d6280 replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:509/504,start:6:a811fbc3:::benchmark_data_magna108_35617_object588840:0,end:6:a81302ca:::benchmark_data_magna108_35617_object816665:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -8649> 2018-07-11 08:04:43.724477 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 181 ==== replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:509/504,start:6:a811fbc3:::benchmark_data_magna108_35617_object588840:0,end:6:a81302ca:::benchmark_data_magna108_35617_object816665:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (946083760 0 0) 0x55f6631d6280 con 0x55f66315e800 > -8648> 2018-07-11 08:04:43.949308 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0e800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 189 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:04:45.193075) v4 > -8647> 2018-07-11 08:04:43.949338 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.5 10.8.128.113:0/15263 189 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:45.193075) v4 ==== 2004+0+0 (1571367509 0 0) 0x55f67d1afe00 con 0x55f663f0e800 > -8646> 2018-07-11 08:04:43.949361 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:45.193075) v4 -- 0x55f65dad4e00 con 0 > -8645> 2018-07-11 08:04:43.949367 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0d000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 189 0x55f6780b2800 osd_ping(ping e509 stamp 2018-07-11 08:04:45.193075) v4 > -8644> 2018-07-11 08:04:43.949389 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.5 10.8.128.113:0/15263 189 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:45.193075) v4 ==== 2004+0+0 (1571367509 0 0) 0x55f6780b2800 con 0x55f663f0d000 > -8643> 2018-07-11 08:04:43.949401 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:45.193075) v4 -- 0x55f64c7d3200 con 0 > -8642> 2018-07-11 08:04:44.297060 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0a000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 181 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:04:44.932378) v4 > -8641> 2018-07-11 08:04:44.297084 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.11 10.8.128.114:0/15919 181 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:44.932378) v4 ==== 2004+0+0 (4221677301 0 0) 0x55f64e20fc00 con 0x55f663f0a000 > -8640> 2018-07-11 08:04:44.297101 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:44.932378) v4 -- 0x55f64cf25400 con 0 > -8639> 2018-07-11 08:04:44.297165 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0b800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 181 0x55f6548fba00 osd_ping(ping e509 stamp 2018-07-11 08:04:44.932378) v4 > -8638> 2018-07-11 08:04:44.297191 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.11 10.8.128.114:0/15919 181 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:44.932378) v4 ==== 2004+0+0 (4221677301 0 0) 0x55f6548fba00 con 0x55f663f0b800 > -8637> 2018-07-11 08:04:44.297203 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:44.932378) v4 -- 0x55f673952a00 con 0 > -8636> 2018-07-11 08:04:44.377718 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.107:0/12617 conn(0x55f64e3c5000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.9 seq 185 0x55f64cc81000 osd_ping(ping e509 stamp 2018-07-11 08:04:44.377172) v4 > -8635> 2018-07-11 08:04:44.377735 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.9 10.8.128.107:0/12617 185 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:44.377172) v4 ==== 2004+0+0 (2425788689 0 0) 0x55f64cc81000 con 0x55f64e3c5000 > -8634> 2018-07-11 08:04:44.377744 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:44.377172) v4 -- 0x55f666c46600 con 0 > -8633> 2018-07-11 08:04:44.377828 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.107:0/12617 conn(0x55f64e3c3800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.9 seq 185 0x55f667d09200 osd_ping(ping e509 stamp 2018-07-11 08:04:44.377172) v4 > -8632> 2018-07-11 08:04:44.377849 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.9 10.8.128.107:0/12617 185 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:44.377172) v4 ==== 2004+0+0 (2425788689 0 0) 0x55f667d09200 con 0x55f64e3c3800 > -8631> 2018-07-11 08:04:44.377860 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:44.377172) v4 -- 0x55f673952200 con 0 > -8630> 2018-07-11 08:04:44.497308 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15287 conn(0x55f66dce2800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.4 seq 193 0x55f6780b2800 osd_ping(ping e509 stamp 2018-07-11 08:04:45.741106) v4 > -8629> 2018-07-11 08:04:44.497329 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.4 10.8.128.113:0/15287 193 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:45.741106) v4 ==== 2004+0+0 (3780553230 0 0) 0x55f6780b2800 con 0x55f66dce2800 > -8628> 2018-07-11 08:04:44.497318 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15287 conn(0x55f66dce1000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.4 seq 193 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:04:45.741106) v4 > -8627> 2018-07-11 08:04:44.497340 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:45.741106) v4 -- 0x55f666c46600 con 0 > -8626> 2018-07-11 08:04:44.497342 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.4 10.8.128.113:0/15287 193 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:45.741106) v4 ==== 2004+0+0 (3780553230 0 0) 0x55f67d1afe00 con 0x55f66dce1000 > -8625> 2018-07-11 08:04:44.497365 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:45.741106) v4 -- 0x55f64cf25400 con 0 > -8624> 2018-07-11 08:04:44.804902 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 28] Generated table #37031: 2638241 keys, 74546467 bytes > -8623> 2018-07-11 08:04:44.804954 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296284804937, "cf_name": "default", "job": 28, "event": "table_file_creation", "file_number": 37031, "file_size": 74546467, "table_properties": {"data_size": 67112104, "index_size": 690190, "filter_size": 6743183, "raw_key_size": 76215852, "raw_average_key_size": 28, "raw_value_size": 31365755, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} > -8622> 2018-07-11 08:04:44.958696 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 403 0x55f6548fba00 MOSDScrubReserve(11.3s5 REQUEST e509) v1 > -8621> 2018-07-11 08:04:44.958719 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 403 ==== MOSDScrubReserve(11.3s5 REQUEST e509) v1 ==== 43+0+0 (433425070 0 0) 0x55f6548fba00 con 0x55f668a29000 > -8620> 2018-07-11 08:04:44.958803 7f491a040700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e509) v1 -- 0x55f65dad4200 con 0 > -8619> 2018-07-11 08:04:44.959142 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 404 0x55f65dad4200 MOSDScrubReserve(11.3s5 RELEASE e509) v1 > -8618> 2018-07-11 08:04:44.959156 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 404 ==== MOSDScrubReserve(11.3s5 RELEASE e509) v1 ==== 43+0+0 (3603460268 0 0) 0x55f65dad4200 con 0x55f668a29000 > -8617> 2018-07-11 08:04:45.068757 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed7800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 183 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:04:45.125004) v4 > -8616> 2018-07-11 08:04:45.068789 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.1 10.8.128.106:0/12677 183 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:45.125004) v4 ==== 2004+0+0 (2928870991 0 0) 0x55f64e20fc00 con 0x55f663ed7800 > -8615> 2018-07-11 08:04:45.068808 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:45.125004) v4 -- 0x55f663140a00 con 0 > -8614> 2018-07-11 08:04:45.068814 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed6000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 183 0x55f64cc81000 osd_ping(ping e509 stamp 2018-07-11 08:04:45.125004) v4 > -8613> 2018-07-11 08:04:45.068836 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.1 10.8.128.106:0/12677 183 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:45.125004) v4 ==== 2004+0+0 (2928870991 0 0) 0x55f64cc81000 con 0x55f663ed6000 > -8612> 2018-07-11 08:04:45.068847 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:45.125004) v4 -- 0x55f669deb400 con 0 > -8611> 2018-07-11 08:04:45.204500 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.115:0/13323 conn(0x55f663ebe000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.6 seq 176 0x55f667d09200 osd_ping(ping e509 stamp 2018-07-11 08:04:45.556755) v4 > -8610> 2018-07-11 08:04:45.204506 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.115:0/13323 conn(0x55f663ebf800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.6 seq 176 0x55f6780b2800 osd_ping(ping e509 stamp 2018-07-11 08:04:45.556755) v4 > -8609> 2018-07-11 08:04:45.204530 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.6 10.8.128.115:0/13323 176 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:45.556755) v4 ==== 2004+0+0 (3020976689 0 0) 0x55f667d09200 con 0x55f663ebe000 > -8608> 2018-07-11 08:04:45.204535 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.6 10.8.128.115:0/13323 176 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:45.556755) v4 ==== 2004+0+0 (3020976689 0 0) 0x55f6780b2800 con 0x55f663ebf800 > -8607> 2018-07-11 08:04:45.204546 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:45.556755) v4 -- 0x55f64f769e00 con 0 > -8606> 2018-07-11 08:04:45.204576 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:45.556755) v4 -- 0x55f64cf25600 con 0 > -8605> 2018-07-11 08:04:45.269978 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15255 conn(0x55f663ec2800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.0 seq 178 0x55f66b1cd800 osd_ping(ping e509 stamp 2018-07-11 08:04:46.513779) v4 > -8604> 2018-07-11 08:04:45.270000 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.0 10.8.128.113:0/15255 178 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:46.513779) v4 ==== 2004+0+0 (748861400 0 0) 0x55f66b1cd800 con 0x55f663ec2800 > -8603> 2018-07-11 08:04:45.269997 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15255 conn(0x55f663ec1000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.0 seq 178 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:04:46.513779) v4 > -8602> 2018-07-11 08:04:45.270015 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.0 10.8.128.113:0/15255 178 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:46.513779) v4 ==== 2004+0+0 (748861400 0 0) 0x55f67d1afe00 con 0x55f663ec1000 > -8601> 2018-07-11 08:04:45.270017 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:46.513779) v4 -- 0x55f668a30400 con 0 > -8600> 2018-07-11 08:04:45.270045 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:46.513779) v4 -- 0x55f667d09c00 con 0 > -8599> 2018-07-11 08:04:45.395647 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f675fc98c0 con 0 > -8598> 2018-07-11 08:04:45.397301 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 182 0x55f676582000 replica scrub(pg: 6.15,from:0'0,to:430'106590,epoch:509/504,start:6:a81302ca:::benchmark_data_magna108_35617_object816665:0,end:6:a813f48d:::benchmark_data_magna108_35617_object796577:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -8597> 2018-07-11 08:04:45.397327 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 182 ==== replica scrub(pg: 6.15,from:0'0,to:430'106590,epoch:509/504,start:6:a81302ca:::benchmark_data_magna108_35617_object816665:0,end:6:a813f48d:::benchmark_data_magna108_35617_object796577:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (3954363924 0 0) 0x55f676582000 con 0x55f66315e800 > -8596> 2018-07-11 08:04:45.525236 7f491d046700 1 -- 10.8.128.107:6800/14173 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55f662a89080 con 0 > -8595> 2018-07-11 08:04:45.525320 7f491d046700 1 -- 10.8.128.107:6800/14173 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55f666f258c0 con 0 > -8594> 2018-07-11 08:04:45.657086 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.114:6805/15885 conn(0x55f66315a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.7 seq 2521 0x55f64cc81000 MOSDScrubReserve(5.4 REQUEST e509) v1 > -8593> 2018-07-11 08:04:45.657104 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.7 10.8.128.114:6805/15885 2521 ==== MOSDScrubReserve(5.4 REQUEST e509) v1 ==== 43+0+0 (3003630469 0 0) 0x55f64cc81000 con 0x55f66315a000 > -8592> 2018-07-11 08:04:45.657180 7f491983f700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.114:6805/15885 -- MOSDScrubReserve(5.4 REJECT e509) v1 -- 0x55f6548fbe00 con 0 > -8591> 2018-07-11 08:04:45.657471 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.114:6805/15885 conn(0x55f66315a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.7 seq 2522 0x55f6548fbe00 MOSDScrubReserve(5.4 RELEASE e509) v1 > -8590> 2018-07-11 08:04:45.657485 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.7 10.8.128.114:6805/15885 2522 ==== MOSDScrubReserve(5.4 RELEASE e509) v1 ==== 43+0+0 (2082093447 0 0) 0x55f6548fbe00 con 0x55f66315a000 > -8589> 2018-07-11 08:04:45.958920 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 405 0x55f667d09200 MOSDScrubReserve(11.3s5 REQUEST e509) v1 > -8588> 2018-07-11 08:04:45.958940 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 405 ==== MOSDScrubReserve(11.3s5 REQUEST e509) v1 ==== 43+0+0 (433425070 0 0) 0x55f667d09200 con 0x55f668a29000 > -8587> 2018-07-11 08:04:45.958996 7f491a040700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e509) v1 -- 0x55f65dad4200 con 0 > -8586> 2018-07-11 08:04:45.959389 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 406 0x55f65dad4200 MOSDScrubReserve(11.3s5 RELEASE e509) v1 > -8585> 2018-07-11 08:04:45.959404 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 406 ==== MOSDScrubReserve(11.3s5 RELEASE e509) v1 ==== 43+0+0 (3603460268 0 0) 0x55f65dad4200 con 0x55f668a29000 > -8584> 2018-07-11 08:04:45.997649 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0a000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 182 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:04:46.632955) v4 > -8583> 2018-07-11 08:04:45.997677 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.11 10.8.128.114:0/15919 182 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:46.632955) v4 ==== 2004+0+0 (2821311588 0 0) 0x55f64e20fc00 con 0x55f663f0a000 > -8582> 2018-07-11 08:04:45.997692 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:46.632955) v4 -- 0x55f64e25dc00 con 0 > -8581> 2018-07-11 08:04:45.997748 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0b800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 182 0x55f66b1cd800 osd_ping(ping e509 stamp 2018-07-11 08:04:46.632955) v4 > -8580> 2018-07-11 08:04:45.997770 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.11 10.8.128.114:0/15919 182 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:46.632955) v4 ==== 2004+0+0 (2821311588 0 0) 0x55f66b1cd800 con 0x55f663f0b800 > -8579> 2018-07-11 08:04:45.997781 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:46.632955) v4 -- 0x55f6631f4400 con 0 > -8578> 2018-07-11 08:04:46.078244 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.107:0/12617 conn(0x55f64e3c5000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.9 seq 186 0x55f64cf25600 osd_ping(ping e509 stamp 2018-07-11 08:04:46.077858) v4 > -8577> 2018-07-11 08:04:46.078273 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.9 10.8.128.107:0/12617 186 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:46.077858) v4 ==== 2004+0+0 (2411743624 0 0) 0x55f64cf25600 con 0x55f64e3c5000 > -8576> 2018-07-11 08:04:46.078234 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.107:0/12617 conn(0x55f64e3c3800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.9 seq 186 0x55f65e71da00 osd_ping(ping e509 stamp 2018-07-11 08:04:46.077858) v4 > -8575> 2018-07-11 08:04:46.078296 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:46.077858) v4 -- 0x55f660948200 con 0 > -8574> 2018-07-11 08:04:46.078301 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.9 10.8.128.107:0/12617 186 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:46.077858) v4 ==== 2004+0+0 (2411743624 0 0) 0x55f65e71da00 con 0x55f64e3c3800 > -8573> 2018-07-11 08:04:46.078328 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:46.077858) v4 -- 0x55f6631f4400 con 0 > -8572> 2018-07-11 08:04:46.575229 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f66d611440 con 0 > -8571> 2018-07-11 08:04:46.577271 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 183 0x55f676582280 replica scrub(pg: 6.15,from:0'0,to:430'105874,epoch:509/504,start:6:a813f48d:::benchmark_data_magna108_35617_object796577:0,end:6:a814f775:::benchmark_data_magna108.ceph.redhat.com_2478_object423780:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -8570> 2018-07-11 08:04:46.577303 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 183 ==== replica scrub(pg: 6.15,from:0'0,to:430'105874,epoch:509/504,start:6:a813f48d:::benchmark_data_magna108_35617_object796577:0,end:6:a814f775:::benchmark_data_magna108.ceph.redhat.com_2478_object423780:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (4076319844 0 0) 0x55f676582280 con 0x55f66315e800 > -8569> 2018-07-11 08:04:46.657244 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.114:6805/15885 conn(0x55f66315a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.7 seq 2523 0x55f663f5f400 MOSDScrubReserve(5.4 REQUEST e509) v1 > -8568> 2018-07-11 08:04:46.657279 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.7 10.8.128.114:6805/15885 2523 ==== MOSDScrubReserve(5.4 REQUEST e509) v1 ==== 43+0+0 (3003630469 0 0) 0x55f663f5f400 con 0x55f66315a000 > -8567> 2018-07-11 08:04:46.657358 7f491983f700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.114:6805/15885 -- MOSDScrubReserve(5.4 REJECT e509) v1 -- 0x55f6548fbe00 con 0 > -8566> 2018-07-11 08:04:46.657713 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.114:6805/15885 conn(0x55f66315a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.7 seq 2524 0x55f6548fbe00 MOSDScrubReserve(5.4 RELEASE e509) v1 > -8565> 2018-07-11 08:04:46.657732 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.7 10.8.128.114:6805/15885 2524 ==== MOSDScrubReserve(5.4 RELEASE e509) v1 ==== 43+0+0 (2082093447 0 0) 0x55f6548fbe00 con 0x55f66315a000 > -8564> 2018-07-11 08:04:46.729493 7f491803c700 5 osd.10 509 heartbeat: osd_stat(789 GB used, 142 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,1]) > -8563> 2018-07-11 08:04:46.729509 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6802/15255 -- osd_ping(ping e509 stamp 2018-07-11 08:04:46.729506) v4 -- 0x55f66cb82400 con 0 > -8562> 2018-07-11 08:04:46.729544 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6803/15255 -- osd_ping(ping e509 stamp 2018-07-11 08:04:46.729506) v4 -- 0x55f66cb82600 con 0 > -8561> 2018-07-11 08:04:46.729579 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6806/12677 -- osd_ping(ping e509 stamp 2018-07-11 08:04:46.729506) v4 -- 0x55f66cb82800 con 0 > -8560> 2018-07-11 08:04:46.729600 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6807/12677 -- osd_ping(ping e509 stamp 2018-07-11 08:04:46.729506) v4 -- 0x55f66cb82a00 con 0 > -8559> 2018-07-11 08:04:46.729616 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6808/12651 -- osd_ping(ping e509 stamp 2018-07-11 08:04:46.729506) v4 -- 0x55f66cb82c00 con 0 > -8558> 2018-07-11 08:04:46.729633 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6810/12651 -- osd_ping(ping e509 stamp 2018-07-11 08:04:46.729506) v4 -- 0x55f66cb82e00 con 0 > -8557> 2018-07-11 08:04:46.729660 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6809/12650 -- osd_ping(ping e509 stamp 2018-07-11 08:04:46.729506) v4 -- 0x55f66cb83000 con 0 > -8556> 2018-07-11 08:04:46.729678 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6811/12650 -- osd_ping(ping e509 stamp 2018-07-11 08:04:46.729506) v4 -- 0x55f66cb83200 con 0 > -8555> 2018-07-11 08:04:46.729700 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6810/15287 -- osd_ping(ping e509 stamp 2018-07-11 08:04:46.729506) v4 -- 0x55f66cb83400 con 0 > -8554> 2018-07-11 08:04:46.729717 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6811/15287 -- osd_ping(ping e509 stamp 2018-07-11 08:04:46.729506) v4 -- 0x55f66cb83600 con 0 > -8553> 2018-07-11 08:04:46.729736 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6806/15263 -- osd_ping(ping e509 stamp 2018-07-11 08:04:46.729506) v4 -- 0x55f66cb83800 con 0 > -8552> 2018-07-11 08:04:46.729754 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6807/15263 -- osd_ping(ping e509 stamp 2018-07-11 08:04:46.729506) v4 -- 0x55f66cb83a00 con 0 > -8551> 2018-07-11 08:04:46.729772 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.115:6802/13323 -- osd_ping(ping e509 stamp 2018-07-11 08:04:46.729506) v4 -- 0x55f66cb83c00 con 0 > -8550> 2018-07-11 08:04:46.729790 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.115:6803/13323 -- osd_ping(ping e509 stamp 2018-07-11 08:04:46.729506) v4 -- 0x55f66cb83e00 con 0 > -8549> 2018-07-11 08:04:46.729812 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6806/15885 -- osd_ping(ping e509 stamp 2018-07-11 08:04:46.729506) v4 -- 0x55f66b1cd400 con 0 > -8548> 2018-07-11 08:04:46.729829 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6807/15885 -- osd_ping(ping e509 stamp 2018-07-11 08:04:46.729506) v4 -- 0x55f66b1cc400 con 0 > -8547> 2018-07-11 08:04:46.729848 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.108:6802/11967 -- osd_ping(ping e509 stamp 2018-07-11 08:04:46.729506) v4 -- 0x55f6780b3000 con 0 > -8546> 2018-07-11 08:04:46.729864 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.108:6803/11967 -- osd_ping(ping e509 stamp 2018-07-11 08:04:46.729506) v4 -- 0x55f6780b3400 con 0 > -8545> 2018-07-11 08:04:46.729881 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.107:6806/12617 -- osd_ping(ping e509 stamp 2018-07-11 08:04:46.729506) v4 -- 0x55f6780b3c00 con 0 > -8544> 2018-07-11 08:04:46.729899 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.107:6807/12617 -- osd_ping(ping e509 stamp 2018-07-11 08:04:46.729506) v4 -- 0x55f653d3c000 con 0 > -8543> 2018-07-11 08:04:46.729916 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6802/15919 -- osd_ping(ping e509 stamp 2018-07-11 08:04:46.729506) v4 -- 0x55f653d3da00 con 0 > -8542> 2018-07-11 08:04:46.729932 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6803/15919 -- osd_ping(ping e509 stamp 2018-07-11 08:04:46.729506) v4 -- 0x55f653d3c400 con 0 > -8541> 2018-07-11 08:04:46.729930 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6802/15255 conn(0x55f656ad4000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 184 0x55f660948200 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:46.729506) v4 > -8540> 2018-07-11 08:04:46.729951 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.0 10.8.128.113:6802/15255 184 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:46.729506) v4 ==== 2004+0+0 (2143429140 0 0) 0x55f660948200 con 0x55f656ad4000 > -8539> 2018-07-11 08:04:46.730020 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6807/12677 conn(0x55f64e3d9800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.1 seq 184 0x55f660948200 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:46.729506) v4 > -8538> 2018-07-11 08:04:46.730035 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.1 10.8.128.106:6807/12677 184 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:46.729506) v4 ==== 2004+0+0 (2143429140 0 0) 0x55f660948200 con 0x55f64e3d9800 > -8537> 2018-07-11 08:04:46.730044 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6803/15255 conn(0x55f656ad5800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 184 0x55f65e71da00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:46.729506) v4 > -8536> 2018-07-11 08:04:46.730065 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.0 10.8.128.113:6803/15255 184 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:46.729506) v4 ==== 2004+0+0 (2143429140 0 0) 0x55f65e71da00 con 0x55f656ad5800 > -8535> 2018-07-11 08:04:46.730086 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6806/12677 conn(0x55f64e3d8000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.1 seq 184 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:46.729506) v4 > -8534> 2018-07-11 08:04:46.730100 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.107:6806/12617 conn(0x55f656ade000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.9 seq 184 0x55f65e71da00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:46.729506) v4 > -8533> 2018-07-11 08:04:46.730106 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.1 10.8.128.106:6806/12677 184 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:46.729506) v4 ==== 2004+0+0 (2143429140 0 0) 0x55f67d1afe00 con 0x55f64e3d8000 > -8532> 2018-07-11 08:04:46.730112 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.9 10.8.128.107:6806/12617 184 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:46.729506) v4 ==== 2004+0+0 (2143429140 0 0) 0x55f65e71da00 con 0x55f656ade000 > -8531> 2018-07-11 08:04:46.730076 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6808/12651 conn(0x55f64e3ea000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.2 seq 184 0x55f660948200 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:46.729506) v4 > -8530> 2018-07-11 08:04:46.730136 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.2 10.8.128.106:6808/12651 184 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:46.729506) v4 ==== 2004+0+0 (2143429140 0 0) 0x55f660948200 con 0x55f64e3ea000 > -8529> 2018-07-11 08:04:46.730162 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6810/15287 conn(0x55f64e401000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.4 seq 184 0x55f660948200 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:46.729506) v4 > -8528> 2018-07-11 08:04:46.730173 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.4 10.8.128.113:6810/15287 184 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:46.729506) v4 ==== 2004+0+0 (2143429140 0 0) 0x55f660948200 con 0x55f64e401000 > -8527> 2018-07-11 08:04:46.730160 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6810/12651 conn(0x55f64e3eb800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.2 seq 184 0x55f65e71da00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:46.729506) v4 > -8526> 2018-07-11 08:04:46.730175 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6809/12650 conn(0x55f656ad7000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.3 seq 184 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:46.729506) v4 > -8525> 2018-07-11 08:04:46.730198 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.2 10.8.128.106:6810/12651 184 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:46.729506) v4 ==== 2004+0+0 (2143429140 0 0) 0x55f65e71da00 con 0x55f64e3eb800 > -8524> 2018-07-11 08:04:46.730203 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.3 10.8.128.106:6809/12650 184 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:46.729506) v4 ==== 2004+0+0 (2143429140 0 0) 0x55f67d1afe00 con 0x55f656ad7000 > -8523> 2018-07-11 08:04:46.730202 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6811/12650 conn(0x55f656ad8800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.3 seq 184 0x55f660948200 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:46.729506) v4 > -8522> 2018-07-11 08:04:46.730232 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6806/15263 conn(0x55f64e3ed000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.5 seq 184 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:46.729506) v4 > -8521> 2018-07-11 08:04:46.730233 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6811/15287 conn(0x55f64e402800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.4 seq 184 0x55f65e71da00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:46.729506) v4 > -8520> 2018-07-11 08:04:46.730233 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.3 10.8.128.106:6811/12650 184 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:46.729506) v4 ==== 2004+0+0 (2143429140 0 0) 0x55f660948200 con 0x55f656ad8800 > -8519> 2018-07-11 08:04:46.730244 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.4 10.8.128.113:6811/15287 184 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:46.729506) v4 ==== 2004+0+0 (2143429140 0 0) 0x55f65e71da00 con 0x55f64e402800 > -8518> 2018-07-11 08:04:46.730256 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.5 10.8.128.113:6806/15263 184 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:46.729506) v4 ==== 2004+0+0 (2143429140 0 0) 0x55f67d1afe00 con 0x55f64e3ed000 > -8517> 2018-07-11 08:04:46.730280 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.107:6807/12617 conn(0x55f656adf800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.9 seq 184 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:46.729506) v4 > -8516> 2018-07-11 08:04:46.730280 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6807/15263 conn(0x55f64e3ee800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.5 seq 184 0x55f660948200 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:46.729506) v4 > -8515> 2018-07-11 08:04:46.730292 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.9 10.8.128.107:6807/12617 184 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:46.729506) v4 ==== 2004+0+0 (2143429140 0 0) 0x55f67d1afe00 con 0x55f656adf800 > -8514> 2018-07-11 08:04:46.730292 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.5 10.8.128.113:6807/15263 184 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:46.729506) v4 ==== 2004+0+0 (2143429140 0 0) 0x55f660948200 con 0x55f64e3ee800 > -8513> 2018-07-11 08:04:46.730299 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6806/15885 conn(0x55f64e3fe000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.7 seq 184 0x55f65e71da00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:46.729506) v4 > -8512> 2018-07-11 08:04:46.730328 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.115:6803/13323 conn(0x55f656ace800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.6 seq 184 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:46.729506) v4 > -8511> 2018-07-11 08:04:46.730330 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.7 10.8.128.114:6806/15885 184 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:46.729506) v4 ==== 2004+0+0 (2143429140 0 0) 0x55f65e71da00 con 0x55f64e3fe000 > -8510> 2018-07-11 08:04:46.730331 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.108:6803/11967 conn(0x55f656acb800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.8 seq 184 0x55f660948200 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:46.729506) v4 > -8509> 2018-07-11 08:04:46.730345 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.6 10.8.128.115:6803/13323 184 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:46.729506) v4 ==== 2004+0+0 (2143429140 0 0) 0x55f67d1afe00 con 0x55f656ace800 > -8508> 2018-07-11 08:04:46.730350 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.8 10.8.128.108:6803/11967 184 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:46.729506) v4 ==== 2004+0+0 (2143429140 0 0) 0x55f660948200 con 0x55f656acb800 > -8507> 2018-07-11 08:04:46.730368 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.115:6802/13323 conn(0x55f656acd000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.6 seq 184 0x55f65e71da00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:46.729506) v4 > -8506> 2018-07-11 08:04:46.730376 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.108:6802/11967 conn(0x55f656aca000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.8 seq 184 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:46.729506) v4 > -8505> 2018-07-11 08:04:46.730385 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.6 10.8.128.115:6802/13323 184 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:46.729506) v4 ==== 2004+0+0 (2143429140 0 0) 0x55f65e71da00 con 0x55f656acd000 > -8504> 2018-07-11 08:04:46.730386 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.8 10.8.128.108:6802/11967 184 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:46.729506) v4 ==== 2004+0+0 (2143429140 0 0) 0x55f67d1afe00 con 0x55f656aca000 > -8503> 2018-07-11 08:04:46.730418 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6802/15919 conn(0x55f64e3db000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.11 seq 184 0x55f65e71da00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:46.729506) v4 > -8502> 2018-07-11 08:04:46.730426 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6807/15885 conn(0x55f64e3ff800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.7 seq 184 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:46.729506) v4 > -8501> 2018-07-11 08:04:46.730430 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.11 10.8.128.114:6802/15919 184 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:46.729506) v4 ==== 2004+0+0 (2143429140 0 0) 0x55f65e71da00 con 0x55f64e3db000 > -8500> 2018-07-11 08:04:46.730441 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.7 10.8.128.114:6807/15885 184 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:46.729506) v4 ==== 2004+0+0 (2143429140 0 0) 0x55f67d1afe00 con 0x55f64e3ff800 > -8499> 2018-07-11 08:04:46.730476 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6803/15919 conn(0x55f64e3dc800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.11 seq 184 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:46.729506) v4 > -8498> 2018-07-11 08:04:46.730492 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.11 10.8.128.114:6803/15919 184 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:46.729506) v4 ==== 2004+0+0 (2143429140 0 0) 0x55f67d1afe00 con 0x55f64e3dc800 > -8497> 2018-07-11 08:04:46.769323 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed7800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 184 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:04:46.825538) v4 > -8496> 2018-07-11 08:04:46.769344 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.1 10.8.128.106:0/12677 184 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:46.825538) v4 ==== 2004+0+0 (2849219603 0 0) 0x55f67d1afe00 con 0x55f663ed7800 > -8495> 2018-07-11 08:04:46.769354 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:46.825538) v4 -- 0x55f64e25dc00 con 0 > -8494> 2018-07-11 08:04:46.769367 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed6000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 184 0x55f660948200 osd_ping(ping e509 stamp 2018-07-11 08:04:46.825538) v4 > -8493> 2018-07-11 08:04:46.769388 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.1 10.8.128.106:0/12677 184 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:46.825538) v4 ==== 2004+0+0 (2849219603 0 0) 0x55f660948200 con 0x55f663ed6000 > -8492> 2018-07-11 08:04:46.769399 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:46.825538) v4 -- 0x55f66cb82a00 con 0 > -8491> 2018-07-11 08:04:46.904945 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.115:0/13323 conn(0x55f663ebf800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.6 seq 177 0x55f64cf25600 osd_ping(ping e509 stamp 2018-07-11 08:04:47.257207) v4 > -8490> 2018-07-11 08:04:46.904970 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.6 10.8.128.115:0/13323 177 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:47.257207) v4 ==== 2004+0+0 (3303330529 0 0) 0x55f64cf25600 con 0x55f663ebf800 > -8489> 2018-07-11 08:04:46.904985 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:47.257207) v4 -- 0x55f66cb82400 con 0 > -8488> 2018-07-11 08:04:46.905127 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.115:0/13323 conn(0x55f663ebe000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.6 seq 177 0x55f65e71da00 osd_ping(ping e509 stamp 2018-07-11 08:04:47.257207) v4 > -8487> 2018-07-11 08:04:46.905145 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.6 10.8.128.115:0/13323 177 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:47.257207) v4 ==== 2004+0+0 (3303330529 0 0) 0x55f65e71da00 con 0x55f663ebe000 > -8486> 2018-07-11 08:04:46.905157 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:47.257207) v4 -- 0x55f6631f4400 con 0 > -8485> 2018-07-11 08:04:46.923842 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12650 conn(0x55f663eda800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.3 seq 184 0x55f660948200 osd_ping(ping e509 stamp 2018-07-11 08:04:46.980151) v4 > -8484> 2018-07-11 08:04:46.923839 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12650 conn(0x55f663ed9000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.3 seq 184 0x55f66b1cd800 osd_ping(ping e509 stamp 2018-07-11 08:04:46.980151) v4 > -8483> 2018-07-11 08:04:46.923871 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.3 10.8.128.106:0/12650 184 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:46.980151) v4 ==== 2004+0+0 (3825562987 0 0) 0x55f66b1cd800 con 0x55f663ed9000 > -8482> 2018-07-11 08:04:46.923868 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.3 10.8.128.106:0/12650 184 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:46.980151) v4 ==== 2004+0+0 (3825562987 0 0) 0x55f660948200 con 0x55f663eda800 > -8481> 2018-07-11 08:04:46.923888 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:46.980151) v4 -- 0x55f64cc81e00 con 0 > -8480> 2018-07-11 08:04:46.923912 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:46.980151) v4 -- 0x55f6548fac00 con 0 > -8479> 2018-07-11 08:04:46.959154 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 407 0x55f65e71da00 MOSDScrubReserve(11.3s5 REQUEST e509) v1 > -8478> 2018-07-11 08:04:46.959172 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 407 ==== MOSDScrubReserve(11.3s5 REQUEST e509) v1 ==== 43+0+0 (433425070 0 0) 0x55f65e71da00 con 0x55f668a29000 > -8477> 2018-07-11 08:04:46.959259 7f491a040700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e509) v1 -- 0x55f65dad4200 con 0 > -8476> 2018-07-11 08:04:46.959582 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 408 0x55f64cc81e00 MOSDScrubReserve(11.3s5 RELEASE e509) v1 > -8475> 2018-07-11 08:04:46.959597 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 408 ==== MOSDScrubReserve(11.3s5 RELEASE e509) v1 ==== 43+0+0 (3603460268 0 0) 0x55f64cc81e00 con 0x55f668a29000 > -8474> 2018-07-11 08:04:47.269714 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed7800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 185 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:04:47.326052) v4 > -8473> 2018-07-11 08:04:47.269726 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed6000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 185 0x55f64cf25600 osd_ping(ping e509 stamp 2018-07-11 08:04:47.326052) v4 > -8472> 2018-07-11 08:04:47.269737 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.1 10.8.128.106:0/12677 185 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:47.326052) v4 ==== 2004+0+0 (1449163732 0 0) 0x55f64e20fc00 con 0x55f663ed7800 > -8471> 2018-07-11 08:04:47.269745 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.1 10.8.128.106:0/12677 185 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:47.326052) v4 ==== 2004+0+0 (1449163732 0 0) 0x55f64cf25600 con 0x55f663ed6000 > -8470> 2018-07-11 08:04:47.269749 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:47.326052) v4 -- 0x55f64e25dc00 con 0 > -8469> 2018-07-11 08:04:47.269773 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:47.326052) v4 -- 0x55f6548fac00 con 0 > -8468> 2018-07-11 08:04:47.285130 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f66ca10fc0 con 0 > -8467> 2018-07-11 08:04:47.286617 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 184 0x55f676582500 replica scrub(pg: 6.15,from:0'0,to:430'106560,epoch:509/504,start:6:a814f775:::benchmark_data_magna108.ceph.redhat.com_2478_object423780:0,end:6:a8164b54:::benchmark_data_magna108_35617_object393656:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -8466> 2018-07-11 08:04:47.286647 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 184 ==== replica scrub(pg: 6.15,from:0'0,to:430'106560,epoch:509/504,start:6:a814f775:::benchmark_data_magna108.ceph.redhat.com_2478_object423780:0,end:6:a8164b54:::benchmark_data_magna108_35617_object393656:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (1255841135 0 0) 0x55f676582500 con 0x55f66315e800 > -8465> 2018-07-11 08:04:47.449664 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0d000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 190 0x55f660948200 osd_ping(ping e509 stamp 2018-07-11 08:04:48.693621) v4 > -8464> 2018-07-11 08:04:47.449674 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0e800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 190 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:04:48.693621) v4 > -8463> 2018-07-11 08:04:47.449692 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.5 10.8.128.113:0/15263 190 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:48.693621) v4 ==== 2004+0+0 (792515365 0 0) 0x55f67d1afe00 con 0x55f663f0e800 > -8462> 2018-07-11 08:04:47.449690 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.5 10.8.128.113:0/15263 190 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:48.693621) v4 ==== 2004+0+0 (792515365 0 0) 0x55f660948200 con 0x55f663f0d000 > -8461> 2018-07-11 08:04:47.449701 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:48.693621) v4 -- 0x55f64e25dc00 con 0 > -8460> 2018-07-11 08:04:47.449726 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:48.693621) v4 -- 0x55f6548fac00 con 0 > -8459> 2018-07-11 08:04:47.547068 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12651 conn(0x55f663219000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25 cs=1 l=1). rx osd.2 seq 197 0x55f64cf25600 osd_ping(ping e509 stamp 2018-07-11 08:04:47.603220) v4 > -8458> 2018-07-11 08:04:47.547094 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.2 10.8.128.106:0/12651 197 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:47.603220) v4 ==== 2004+0+0 (1825896867 0 0) 0x55f64cf25600 con 0x55f663219000 > -8457> 2018-07-11 08:04:47.547105 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:47.603220) v4 -- 0x55f6548fac00 con 0 > -8456> 2018-07-11 08:04:47.547099 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12651 conn(0x55f66321a800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25 cs=1 l=1). rx osd.2 seq 197 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:04:47.603220) v4 > -8455> 2018-07-11 08:04:47.547124 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.2 10.8.128.106:0/12651 197 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:47.603220) v4 ==== 2004+0+0 (1825896867 0 0) 0x55f64e20fc00 con 0x55f66321a800 > -8454> 2018-07-11 08:04:47.547138 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:47.603220) v4 -- 0x55f64e25dc00 con 0 > -8453> 2018-07-11 08:04:47.593679 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.108:0/11967 conn(0x55f663f6f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=1). rx osd.8 seq 180 0x55f66b1cd800 osd_ping(ping e509 stamp 2018-07-11 08:04:47.353320) v4 > -8452> 2018-07-11 08:04:47.593701 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.8 10.8.128.108:0/11967 180 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:47.353320) v4 ==== 2004+0+0 (2460226306 0 0) 0x55f66b1cd800 con 0x55f663f6f000 > -8451> 2018-07-11 08:04:47.593710 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:47.353320) v4 -- 0x55f662489200 con 0 > -8450> 2018-07-11 08:04:47.593718 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.108:0/11967 conn(0x55f663f70800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=1). rx osd.8 seq 180 0x55f660948200 osd_ping(ping e509 stamp 2018-07-11 08:04:47.353320) v4 > -8449> 2018-07-11 08:04:47.593738 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.8 10.8.128.108:0/11967 180 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:47.353320) v4 ==== 2004+0+0 (2460226306 0 0) 0x55f660948200 con 0x55f663f70800 > -8448> 2018-07-11 08:04:47.593750 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:47.353320) v4 -- 0x55f64cc81200 con 0 > -8447> 2018-07-11 08:04:47.626191 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15885 conn(0x55f663217800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 187 0x55f65dad4200 osd_ping(ping e509 stamp 2018-07-11 08:04:48.261487) v4 > -8446> 2018-07-11 08:04:47.626212 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.7 10.8.128.114:0/15885 187 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:48.261487) v4 ==== 2004+0+0 (1840854661 0 0) 0x55f65dad4200 con 0x55f663217800 > -8445> 2018-07-11 08:04:47.626221 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:48.261487) v4 -- 0x55f662488600 con 0 > -8444> 2018-07-11 08:04:47.626222 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15885 conn(0x55f663216000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 187 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:04:48.261487) v4 > -8443> 2018-07-11 08:04:47.626243 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.7 10.8.128.114:0/15885 187 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:48.261487) v4 ==== 2004+0+0 (1840854661 0 0) 0x55f67d1afe00 con 0x55f663216000 > -8442> 2018-07-11 08:04:47.626265 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:48.261487) v4 -- 0x55f667d09000 con 0 > -8441> 2018-07-11 08:04:47.657340 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.114:6805/15885 conn(0x55f66315a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.7 seq 2525 0x55f64cf25600 MOSDScrubReserve(5.4 REQUEST e509) v1 > -8440> 2018-07-11 08:04:47.657365 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.7 10.8.128.114:6805/15885 2525 ==== MOSDScrubReserve(5.4 REQUEST e509) v1 ==== 43+0+0 (3003630469 0 0) 0x55f64cf25600 con 0x55f66315a000 > -8439> 2018-07-11 08:04:47.657431 7f491983f700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.114:6805/15885 -- MOSDScrubReserve(5.4 REJECT e509) v1 -- 0x55f6548fbe00 con 0 > -8438> 2018-07-11 08:04:47.657761 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.114:6805/15885 conn(0x55f66315a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.7 seq 2526 0x55f6548fbe00 MOSDScrubReserve(5.4 RELEASE e509) v1 > -8437> 2018-07-11 08:04:47.657776 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.7 10.8.128.114:6805/15885 2526 ==== MOSDScrubReserve(5.4 RELEASE e509) v1 ==== 43+0+0 (2082093447 0 0) 0x55f6548fbe00 con 0x55f66315a000 > -8436> 2018-07-11 08:04:47.830089 7f491803c700 5 osd.10 509 heartbeat: osd_stat(789 GB used, 142 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,0,0,1]) > -8435> 2018-07-11 08:04:47.830104 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6802/15255 -- osd_ping(ping e509 stamp 2018-07-11 08:04:47.830102) v4 -- 0x55f662488800 con 0 > -8434> 2018-07-11 08:04:47.830136 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6803/15255 -- osd_ping(ping e509 stamp 2018-07-11 08:04:47.830102) v4 -- 0x55f662488000 con 0 > -8433> 2018-07-11 08:04:47.830170 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6806/12677 -- osd_ping(ping e509 stamp 2018-07-11 08:04:47.830102) v4 -- 0x55f66dc70000 con 0 > -8432> 2018-07-11 08:04:47.830190 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6807/12677 -- osd_ping(ping e509 stamp 2018-07-11 08:04:47.830102) v4 -- 0x55f65299aa00 con 0 > -8431> 2018-07-11 08:04:47.830207 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6808/12651 -- osd_ping(ping e509 stamp 2018-07-11 08:04:47.830102) v4 -- 0x55f65299b200 con 0 > -8430> 2018-07-11 08:04:47.830224 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6810/12651 -- osd_ping(ping e509 stamp 2018-07-11 08:04:47.830102) v4 -- 0x55f65d025400 con 0 > -8429> 2018-07-11 08:04:47.830259 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6809/12650 -- osd_ping(ping e509 stamp 2018-07-11 08:04:47.830102) v4 -- 0x55f667d08000 con 0 > -8428> 2018-07-11 08:04:47.830285 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6811/12650 -- osd_ping(ping e509 stamp 2018-07-11 08:04:47.830102) v4 -- 0x55f64f768e00 con 0 > -8427> 2018-07-11 08:04:47.830305 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6810/15287 -- osd_ping(ping e509 stamp 2018-07-11 08:04:47.830102) v4 -- 0x55f666c46200 con 0 > -8426> 2018-07-11 08:04:47.830322 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6811/15287 -- osd_ping(ping e509 stamp 2018-07-11 08:04:47.830102) v4 -- 0x55f666c47400 con 0 > -8425> 2018-07-11 08:04:47.830341 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6806/15263 -- osd_ping(ping e509 stamp 2018-07-11 08:04:47.830102) v4 -- 0x55f64e25d800 con 0 > -8424> 2018-07-11 08:04:47.830359 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6807/15263 -- osd_ping(ping e509 stamp 2018-07-11 08:04:47.830102) v4 -- 0x55f67d1af000 con 0 > -8423> 2018-07-11 08:04:47.830378 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.115:6802/13323 -- osd_ping(ping e509 stamp 2018-07-11 08:04:47.830102) v4 -- 0x55f67d1afa00 con 0 > -8422> 2018-07-11 08:04:47.830396 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.115:6803/13323 -- osd_ping(ping e509 stamp 2018-07-11 08:04:47.830102) v4 -- 0x55f668a30c00 con 0 > -8421> 2018-07-11 08:04:47.830416 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6806/15885 -- osd_ping(ping e509 stamp 2018-07-11 08:04:47.830102) v4 -- 0x55f663f5f800 con 0 > -8420> 2018-07-11 08:04:47.830433 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6807/15885 -- osd_ping(ping e509 stamp 2018-07-11 08:04:47.830102) v4 -- 0x55f64d711c00 con 0 > -8419> 2018-07-11 08:04:47.830453 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.108:6802/11967 -- osd_ping(ping e509 stamp 2018-07-11 08:04:47.830102) v4 -- 0x55f653d4b200 con 0 > -8418> 2018-07-11 08:04:47.830470 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.108:6803/11967 -- osd_ping(ping e509 stamp 2018-07-11 08:04:47.830102) v4 -- 0x55f64c7d2a00 con 0 > -8417> 2018-07-11 08:04:47.830489 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.107:6806/12617 -- osd_ping(ping e509 stamp 2018-07-11 08:04:47.830102) v4 -- 0x55f6716b2000 con 0 > -8416> 2018-07-11 08:04:47.830507 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.107:6807/12617 -- osd_ping(ping e509 stamp 2018-07-11 08:04:47.830102) v4 -- 0x55f6716b2200 con 0 > -8415> 2018-07-11 08:04:47.830525 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6802/15919 -- osd_ping(ping e509 stamp 2018-07-11 08:04:47.830102) v4 -- 0x55f6716b2400 con 0 > -8414> 2018-07-11 08:04:47.830541 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6803/15919 -- osd_ping(ping e509 stamp 2018-07-11 08:04:47.830102) v4 -- 0x55f6716b2600 con 0 > -8413> 2018-07-11 08:04:47.830570 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6802/15255 conn(0x55f656ad4000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 185 0x55f65c0f2000 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:47.830102) v4 > -8412> 2018-07-11 08:04:47.830578 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6803/15255 conn(0x55f656ad5800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 185 0x55f66b1cd800 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:47.830102) v4 > -8411> 2018-07-11 08:04:47.830591 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.0 10.8.128.113:6802/15255 185 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:47.830102) v4 ==== 2004+0+0 (176029388 0 0) 0x55f65c0f2000 con 0x55f656ad4000 > -8410> 2018-07-11 08:04:47.830598 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.0 10.8.128.113:6803/15255 185 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:47.830102) v4 ==== 2004+0+0 (176029388 0 0) 0x55f66b1cd800 con 0x55f656ad5800 > -8409> 2018-07-11 08:04:47.830614 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6809/12650 conn(0x55f656ad7000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.3 seq 185 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:47.830102) v4 > -8408> 2018-07-11 08:04:47.830635 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.3 10.8.128.106:6809/12650 185 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:47.830102) v4 ==== 2004+0+0 (176029388 0 0) 0x55f64e20fc00 con 0x55f656ad7000 > -8407> 2018-07-11 08:04:47.830641 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6807/12677 conn(0x55f64e3d9800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.1 seq 185 0x55f65c0f2000 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:47.830102) v4 > -8406> 2018-07-11 08:04:47.830655 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.1 10.8.128.106:6807/12677 185 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:47.830102) v4 ==== 2004+0+0 (176029388 0 0) 0x55f65c0f2000 con 0x55f64e3d9800 > -8405> 2018-07-11 08:04:47.830689 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6806/12677 conn(0x55f64e3d8000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.1 seq 185 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:47.830102) v4 > -8404> 2018-07-11 08:04:47.830703 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.1 10.8.128.106:6806/12677 185 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:47.830102) v4 ==== 2004+0+0 (176029388 0 0) 0x55f64e20fc00 con 0x55f64e3d8000 > -8403> 2018-07-11 08:04:47.830715 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6810/12651 conn(0x55f64e3eb800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.2 seq 185 0x55f65dad4200 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:47.830102) v4 > -8402> 2018-07-11 08:04:47.830727 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6810/15287 conn(0x55f64e401000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.4 seq 185 0x55f65c0f2000 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:47.830102) v4 > -8401> 2018-07-11 08:04:47.830735 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.2 10.8.128.106:6810/12651 185 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:47.830102) v4 ==== 2004+0+0 (176029388 0 0) 0x55f65dad4200 con 0x55f64e3eb800 > -8400> 2018-07-11 08:04:47.830744 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.4 10.8.128.113:6810/15287 185 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:47.830102) v4 ==== 2004+0+0 (176029388 0 0) 0x55f65c0f2000 con 0x55f64e401000 > -8399> 2018-07-11 08:04:47.830775 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6808/12651 conn(0x55f64e3ea000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.2 seq 185 0x55f65c0f2000 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:47.830102) v4 > -8398> 2018-07-11 08:04:47.830782 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6811/15287 conn(0x55f64e402800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.4 seq 185 0x55f65dad4200 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:47.830102) v4 > -8397> 2018-07-11 08:04:47.830787 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.2 10.8.128.106:6808/12651 185 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:47.830102) v4 ==== 2004+0+0 (176029388 0 0) 0x55f65c0f2000 con 0x55f64e3ea000 > -8396> 2018-07-11 08:04:47.830799 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.4 10.8.128.113:6811/15287 185 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:47.830102) v4 ==== 2004+0+0 (176029388 0 0) 0x55f65dad4200 con 0x55f64e402800 > -8395> 2018-07-11 08:04:47.830828 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6811/12650 conn(0x55f656ad8800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.3 seq 185 0x55f65c0f2000 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:47.830102) v4 > -8394> 2018-07-11 08:04:47.830830 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.107:6806/12617 conn(0x55f656ade000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.9 seq 185 0x55f65dad4200 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:47.830102) v4 > -8393> 2018-07-11 08:04:47.830843 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.3 10.8.128.106:6811/12650 185 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:47.830102) v4 ==== 2004+0+0 (176029388 0 0) 0x55f65c0f2000 con 0x55f656ad8800 > -8392> 2018-07-11 08:04:47.830847 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.9 10.8.128.107:6806/12617 185 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:47.830102) v4 ==== 2004+0+0 (176029388 0 0) 0x55f65dad4200 con 0x55f656ade000 > -8391> 2018-07-11 08:04:47.830839 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.115:6803/13323 conn(0x55f656ace800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.6 seq 185 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:47.830102) v4 > -8390> 2018-07-11 08:04:47.830869 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.6 10.8.128.115:6803/13323 185 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:47.830102) v4 ==== 2004+0+0 (176029388 0 0) 0x55f64e20fc00 con 0x55f656ace800 > -8389> 2018-07-11 08:04:47.830874 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6807/15263 conn(0x55f64e3ee800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.5 seq 185 0x55f65c0f2000 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:47.830102) v4 > -8388> 2018-07-11 08:04:47.830884 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.5 10.8.128.113:6807/15263 185 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:47.830102) v4 ==== 2004+0+0 (176029388 0 0) 0x55f65c0f2000 con 0x55f64e3ee800 > -8387> 2018-07-11 08:04:47.830908 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6806/15263 conn(0x55f64e3ed000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.5 seq 185 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:47.830102) v4 > -8386> 2018-07-11 08:04:47.830926 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.5 10.8.128.113:6806/15263 185 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:47.830102) v4 ==== 2004+0+0 (176029388 0 0) 0x55f64e20fc00 con 0x55f64e3ed000 > -8385> 2018-07-11 08:04:47.830974 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.115:6802/13323 conn(0x55f656acd000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.6 seq 185 0x55f65dad4200 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:47.830102) v4 > -8384> 2018-07-11 08:04:47.830985 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.107:6807/12617 conn(0x55f656adf800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.9 seq 185 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:47.830102) v4 > -8383> 2018-07-11 08:04:47.830994 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.108:6803/11967 conn(0x55f656acb800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.8 seq 185 0x55f65c0f2000 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:47.830102) v4 > -8382> 2018-07-11 08:04:47.830989 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.6 10.8.128.115:6802/13323 185 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:47.830102) v4 ==== 2004+0+0 (176029388 0 0) 0x55f65dad4200 con 0x55f656acd000 > -8381> 2018-07-11 08:04:47.831006 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.8 10.8.128.108:6803/11967 185 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:47.830102) v4 ==== 2004+0+0 (176029388 0 0) 0x55f65c0f2000 con 0x55f656acb800 > -8380> 2018-07-11 08:04:47.831007 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.9 10.8.128.107:6807/12617 185 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:47.830102) v4 ==== 2004+0+0 (176029388 0 0) 0x55f64e20fc00 con 0x55f656adf800 > -8379> 2018-07-11 08:04:47.831032 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6806/15885 conn(0x55f64e3fe000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.7 seq 185 0x55f65dad4200 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:47.830102) v4 > -8378> 2018-07-11 08:04:47.831047 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.7 10.8.128.114:6806/15885 185 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:47.830102) v4 ==== 2004+0+0 (176029388 0 0) 0x55f65dad4200 con 0x55f64e3fe000 > -8377> 2018-07-11 08:04:47.831055 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6807/15885 conn(0x55f64e3ff800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.7 seq 185 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:47.830102) v4 > -8376> 2018-07-11 08:04:47.831073 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.7 10.8.128.114:6807/15885 185 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:47.830102) v4 ==== 2004+0+0 (176029388 0 0) 0x55f64e20fc00 con 0x55f64e3ff800 > -8375> 2018-07-11 08:04:47.831079 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6802/15919 conn(0x55f64e3db000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.11 seq 185 0x55f65dad4200 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:47.830102) v4 > -8374> 2018-07-11 08:04:47.831092 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.11 10.8.128.114:6802/15919 185 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:47.830102) v4 ==== 2004+0+0 (176029388 0 0) 0x55f65dad4200 con 0x55f64e3db000 > -8373> 2018-07-11 08:04:47.831117 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6803/15919 conn(0x55f64e3dc800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.11 seq 185 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:47.830102) v4 > -8372> 2018-07-11 08:04:47.831135 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.11 10.8.128.114:6803/15919 185 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:47.830102) v4 ==== 2004+0+0 (176029388 0 0) 0x55f64e20fc00 con 0x55f64e3dc800 > -8371> 2018-07-11 08:04:47.831174 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.108:6802/11967 conn(0x55f656aca000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.8 seq 185 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:47.830102) v4 > -8370> 2018-07-11 08:04:47.831191 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.8 10.8.128.108:6802/11967 185 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:47.830102) v4 ==== 2004+0+0 (176029388 0 0) 0x55f64e20fc00 con 0x55f656aca000 > -8369> 2018-07-11 08:04:47.950653 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f650486b40 con 0 > -8368> 2018-07-11 08:04:47.952204 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 185 0x55f676582780 replica scrub(pg: 6.15,from:0'0,to:430'106295,epoch:509/504,start:6:a8164b54:::benchmark_data_magna108_35617_object393656:0,end:6:a81704c4:::benchmark_data_magna108_35617_object501383:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -8367> 2018-07-11 08:04:47.952235 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 185 ==== replica scrub(pg: 6.15,from:0'0,to:430'106295,epoch:509/504,start:6:a8164b54:::benchmark_data_magna108_35617_object393656:0,end:6:a81704c4:::benchmark_data_magna108_35617_object501383:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (2093116328 0 0) 0x55f676582780 con 0x55f66315e800 > -8366> 2018-07-11 08:04:48.170518 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15255 conn(0x55f663ec2800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.0 seq 179 0x55f65dad4200 osd_ping(ping e509 stamp 2018-07-11 08:04:49.414255) v4 > -8365> 2018-07-11 08:04:48.170539 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.0 10.8.128.113:0/15255 179 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:49.414255) v4 ==== 2004+0+0 (299700771 0 0) 0x55f65dad4200 con 0x55f663ec2800 > -8364> 2018-07-11 08:04:48.170546 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15255 conn(0x55f663ec1000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.0 seq 179 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:04:49.414255) v4 > -8363> 2018-07-11 08:04:48.170555 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:49.414255) v4 -- 0x55f666c47400 con 0 > -8362> 2018-07-11 08:04:48.170572 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.0 10.8.128.113:0/15255 179 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:49.414255) v4 ==== 2004+0+0 (299700771 0 0) 0x55f64e20fc00 con 0x55f663ec1000 > -8361> 2018-07-11 08:04:48.170599 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:49.414255) v4 -- 0x55f668a30c00 con 0 > -8360> 2018-07-11 08:04:48.370288 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed7800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 186 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:04:48.426445) v4 > -8359> 2018-07-11 08:04:48.370290 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed6000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 186 0x55f65c0f2000 osd_ping(ping e509 stamp 2018-07-11 08:04:48.426445) v4 > -8358> 2018-07-11 08:04:48.370310 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.1 10.8.128.106:0/12677 186 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:48.426445) v4 ==== 2004+0+0 (1565413375 0 0) 0x55f67d1afe00 con 0x55f663ed7800 > -8357> 2018-07-11 08:04:48.370312 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.1 10.8.128.106:0/12677 186 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:48.426445) v4 ==== 2004+0+0 (1565413375 0 0) 0x55f65c0f2000 con 0x55f663ed6000 > -8356> 2018-07-11 08:04:48.370321 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:48.426445) v4 -- 0x55f64e25d800 con 0 > -8355> 2018-07-11 08:04:48.370350 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:48.426445) v4 -- 0x55f662488800 con 0 > -8354> 2018-07-11 08:04:48.502356 7f492d867700 10 monclient: _send_mon_message to mon.magna102 at 10.8.128.102:6789/0 > -8353> 2018-07-11 08:04:48.502369 7f492d867700 1 -- 10.8.128.107:6800/14173 --> 10.8.128.102:6789/0 -- osd_beacon(pgs [5.24,10.a,6.0,5.10,11.6,6.13,9.1b,6.14,6.17,5.1c,5.9,6.7,5.d,6.d,11.5,5.2e,11.1,10.7] lec 508 v509) v1 -- 0x55f65d024a00 con 0 > -8352> 2018-07-11 08:04:48.766423 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f6506578c0 con 0 > -8351> 2018-07-11 08:04:48.767947 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 186 0x55f676582a00 replica scrub(pg: 6.15,from:0'0,to:430'105616,epoch:509/504,start:6:a81704c4:::benchmark_data_magna108_35617_object501383:0,end:6:a817f1e9:::benchmark_data_magna108_35617_object816578:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -8350> 2018-07-11 08:04:48.767978 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 186 ==== replica scrub(pg: 6.15,from:0'0,to:430'105616,epoch:509/504,start:6:a81704c4:::benchmark_data_magna108_35617_object501383:0,end:6:a817f1e9:::benchmark_data_magna108_35617_object816578:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (1489105051 0 0) 0x55f676582a00 con 0x55f66315e800 > -8349> 2018-07-11 08:04:48.870643 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed6000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 187 0x55f65c0f2000 osd_ping(ping e509 stamp 2018-07-11 08:04:48.926984) v4 > -8348> 2018-07-11 08:04:48.870664 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.1 10.8.128.106:0/12677 187 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:48.926984) v4 ==== 2004+0+0 (203295086 0 0) 0x55f65c0f2000 con 0x55f663ed6000 > -8347> 2018-07-11 08:04:48.870654 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed7800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 187 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:04:48.926984) v4 > -8346> 2018-07-11 08:04:48.870674 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:48.926984) v4 -- 0x55f650f78000 con 0 > -8345> 2018-07-11 08:04:48.870687 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.1 10.8.128.106:0/12677 187 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:48.926984) v4 ==== 2004+0+0 (203295086 0 0) 0x55f64e20fc00 con 0x55f663ed7800 > -8344> 2018-07-11 08:04:48.870704 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:48.926984) v4 -- 0x55f64e25d800 con 0 > -8343> 2018-07-11 08:04:49.150110 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0d000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 191 0x55f65d024a00 osd_ping(ping e509 stamp 2018-07-11 08:04:50.394077) v4 > -8342> 2018-07-11 08:04:49.150104 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0e800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 191 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:04:50.394077) v4 > -8341> 2018-07-11 08:04:49.150142 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.5 10.8.128.113:0/15263 191 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:50.394077) v4 ==== 2004+0+0 (4047140447 0 0) 0x55f67d1afe00 con 0x55f663f0e800 > -8340> 2018-07-11 08:04:49.150140 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.5 10.8.128.113:0/15263 191 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:50.394077) v4 ==== 2004+0+0 (4047140447 0 0) 0x55f65d024a00 con 0x55f663f0d000 > -8339> 2018-07-11 08:04:49.150158 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:50.394077) v4 -- 0x55f667d08000 con 0 > -8338> 2018-07-11 08:04:49.150188 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:50.394077) v4 -- 0x55f662488800 con 0 > -8337> 2018-07-11 08:04:49.847542 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12651 conn(0x55f663219000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25 cs=1 l=1). rx osd.2 seq 198 0x55f65c0f2000 osd_ping(ping e509 stamp 2018-07-11 08:04:49.903829) v4 > -8336> 2018-07-11 08:04:49.847557 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12651 conn(0x55f66321a800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25 cs=1 l=1). rx osd.2 seq 198 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:04:49.903829) v4 > -8335> 2018-07-11 08:04:49.847581 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.2 10.8.128.106:0/12651 198 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:49.903829) v4 ==== 2004+0+0 (212225110 0 0) 0x55f65c0f2000 con 0x55f663219000 > -8334> 2018-07-11 08:04:49.847584 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.2 10.8.128.106:0/12651 198 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:49.903829) v4 ==== 2004+0+0 (212225110 0 0) 0x55f64e20fc00 con 0x55f66321a800 > -8333> 2018-07-11 08:04:49.847591 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:49.903829) v4 -- 0x55f65c0f2200 con 0 > -8332> 2018-07-11 08:04:49.847618 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:49.903829) v4 -- 0x55f66dc70000 con 0 > -8331> 2018-07-11 08:04:49.926600 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15885 conn(0x55f663217800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 188 0x55f66b1cd800 osd_ping(ping e509 stamp 2018-07-11 08:04:50.561906) v4 > -8330> 2018-07-11 08:04:49.926621 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.7 10.8.128.114:0/15885 188 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:50.561906) v4 ==== 2004+0+0 (2964419178 0 0) 0x55f66b1cd800 con 0x55f663217800 > -8329> 2018-07-11 08:04:49.926631 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:50.561906) v4 -- 0x55f65d025400 con 0 > -8328> 2018-07-11 08:04:49.926632 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15885 conn(0x55f663216000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 188 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:04:50.561906) v4 > -8327> 2018-07-11 08:04:49.926653 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.7 10.8.128.114:0/15885 188 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:50.561906) v4 ==== 2004+0+0 (2964419178 0 0) 0x55f67d1afe00 con 0x55f663216000 > -8326> 2018-07-11 08:04:49.926664 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:50.561906) v4 -- 0x55f667d09000 con 0 > -8325> 2018-07-11 08:04:50.397760 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15287 conn(0x55f66dce2800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.4 seq 194 0x55f65d024a00 osd_ping(ping e509 stamp 2018-07-11 08:04:51.641666) v4 > -8324> 2018-07-11 08:04:50.397781 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.4 10.8.128.113:0/15287 194 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:51.641666) v4 ==== 2004+0+0 (4282828338 0 0) 0x55f65d024a00 con 0x55f66dce2800 > -8323> 2018-07-11 08:04:50.397791 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:51.641666) v4 -- 0x55f65c0f2400 con 0 > -8322> 2018-07-11 08:04:50.397811 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15287 conn(0x55f66dce1000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.4 seq 194 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:04:51.641666) v4 > -8321> 2018-07-11 08:04:50.397833 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.4 10.8.128.113:0/15287 194 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:51.641666) v4 ==== 2004+0+0 (4282828338 0 0) 0x55f64e20fc00 con 0x55f66dce1000 > -8320> 2018-07-11 08:04:50.397846 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:51.641666) v4 -- 0x55f667d09000 con 0 > -8319> 2018-07-11 08:04:50.433149 7f491e048700 10 monclient: tick > -8318> 2018-07-11 08:04:50.433162 7f491e048700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2018-07-11 08:04:20.433161) > -8317> 2018-07-11 08:04:50.523016 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 28] Generated table #37032: 2638241 keys, 74546192 bytes > -8316> 2018-07-11 08:04:50.523060 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296290523033, "cf_name": "default", "job": 28, "event": "table_file_creation", "file_number": 37032, "file_size": 74546192, "table_properties": {"data_size": 67112163, "index_size": 689855, "filter_size": 6743184, "raw_key_size": 76215867, "raw_average_key_size": 28, "raw_value_size": 31365770, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} > -8315> 2018-07-11 08:04:50.525584 7f491d046700 1 -- 10.8.128.107:6800/14173 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55f653d5a000 con 0 > -8314> 2018-07-11 08:04:50.525660 7f491d046700 1 -- 10.8.128.107:6800/14173 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55f6631f18c0 con 0 > -8313> 2018-07-11 08:04:50.683848 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f653ecb8c0 con 0 > -8312> 2018-07-11 08:04:50.685753 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 187 0x55f676582c80 replica scrub(pg: 6.15,from:0'0,to:430'105529,epoch:509/504,start:6:a817f1e9:::benchmark_data_magna108_35617_object816578:0,end:6:a8188941:::benchmark_data_magna108_35617_object273789:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -8311> 2018-07-11 08:04:50.685778 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 187 ==== replica scrub(pg: 6.15,from:0'0,to:430'105529,epoch:509/504,start:6:a817f1e9:::benchmark_data_magna108_35617_object816578:0,end:6:a8188941:::benchmark_data_magna108_35617_object273789:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (1453343262 0 0) 0x55f676582c80 con 0x55f66315e800 > -8310> 2018-07-11 08:04:50.701113 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 28] Generated table #37033: 43745 keys, 1236800 bytes > -8309> 2018-07-11 08:04:50.701150 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296290701126, "cf_name": "default", "job": 28, "event": "table_file_creation", "file_number": 37033, "file_size": 1236800, "table_properties": {"data_size": 1112818, "index_size": 11188, "filter_size": 111814, "raw_key_size": 1263741, "raw_average_key_size": 28, "raw_value_size": 520076, "raw_average_value_size": 11, "num_data_blocks": 274, "num_entries": 43745, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} > -8308> 2018-07-11 08:04:50.701179 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 28] Compacted 1@3 + 13@4 files to L4 => 897594417 bytes > -8307> 2018-07-11 08:04:50.724433 7f4922851700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 30558006446 > -8306> 2018-07-11 08:04:50.724920 7f4922851700 4 rocksdb: (Original Log Time 2018/07/11-08:04:50.724392) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[1 7 48 482 3641 796 0] max score 1.13, MB/sec: 20.9 rd, 17.9 wr, level 4, files in(1, 13) out(13) MB in(76.6, 924.2) out(856.0), read-write-amplify(24.2) write-amplify(11.2) OK, records in: 39254297, records dropped: 6830486 > > -8305> 2018-07-11 08:04:50.724930 7f4922851700 4 rocksdb: (Original Log Time 2018/07/11-08:04:50.724426) EVENT_LOG_v1 {"time_micros": 1531296290724406, "job": 28, "event": "compaction_finished", "compaction_time_micros": 50166692, "output_level": 4, "num_output_files": 13, "total_output_size": 897594417, "num_input_records": 39254297, "num_output_records": 32423811, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 7, 48, 482, 3641, 796, 0]} > -8304> 2018-07-11 08:04:50.725503 7f4922851700 5 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:307] [JOB 28] Delete db.slow/036632.sst type=2 #36632 -- OK > > -8303> 2018-07-11 08:04:50.725530 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296290725513, "job": 28, "event": "table_file_deletion", "file_number": 36632} > -8302> 2018-07-11 08:04:50.725540 7f4922851700 5 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:307] [JOB 28] Delete db.slow/028122.sst type=2 #28122 -- OK > > -8301> 2018-07-11 08:04:50.725545 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296290725543, "job": 28, "event": "table_file_deletion", "file_number": 28122} > -8300> 2018-07-11 08:04:50.725551 7f4922851700 5 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:307] [JOB 28] Delete db.slow/028120.sst type=2 #28120 -- OK > > -8299> 2018-07-11 08:04:50.725556 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296290725554, "job": 28, "event": "table_file_deletion", "file_number": 28120} > -8298> 2018-07-11 08:04:50.725561 7f4922851700 5 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:307] [JOB 28] Delete db.slow/028119.sst type=2 #28119 -- OK > > -8297> 2018-07-11 08:04:50.725566 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296290725564, "job": 28, "event": "table_file_deletion", "file_number": 28119} > -8296> 2018-07-11 08:04:50.725571 7f4922851700 5 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:307] [JOB 28] Delete db.slow/028118.sst type=2 #28118 -- OK > > -8295> 2018-07-11 08:04:50.725575 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296290725573, "job": 28, "event": "table_file_deletion", "file_number": 28118} > -8294> 2018-07-11 08:04:50.725595 7f4922851700 5 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:307] [JOB 28] Delete db.slow/026363.sst type=2 #26363 -- OK > > -8293> 2018-07-11 08:04:50.725600 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296290725598, "job": 28, "event": "table_file_deletion", "file_number": 26363} > -8292> 2018-07-11 08:04:50.725607 7f4922851700 5 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:307] [JOB 28] Delete db.slow/026362.sst type=2 #26362 -- OK > > -8291> 2018-07-11 08:04:50.725612 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296290725610, "job": 28, "event": "table_file_deletion", "file_number": 26362} > -8290> 2018-07-11 08:04:50.725618 7f4922851700 5 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:307] [JOB 28] Delete db.slow/026361.sst type=2 #26361 -- OK > > -8289> 2018-07-11 08:04:50.725622 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296290725620, "job": 28, "event": "table_file_deletion", "file_number": 26361} > -8288> 2018-07-11 08:04:50.725628 7f4922851700 5 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:307] [JOB 28] Delete db.slow/026360.sst type=2 #26360 -- OK > > -8287> 2018-07-11 08:04:50.725633 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296290725631, "job": 28, "event": "table_file_deletion", "file_number": 26360} > -8286> 2018-07-11 08:04:50.725638 7f4922851700 5 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:307] [JOB 28] Delete db.slow/026358.sst type=2 #26358 -- OK > > -8285> 2018-07-11 08:04:50.725643 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296290725641, "job": 28, "event": "table_file_deletion", "file_number": 26358} > -8284> 2018-07-11 08:04:50.725648 7f4922851700 5 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:307] [JOB 28] Delete db.slow/026357.sst type=2 #26357 -- OK > > -8283> 2018-07-11 08:04:50.725653 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296290725651, "job": 28, "event": "table_file_deletion", "file_number": 26357} > -8282> 2018-07-11 08:04:50.725658 7f4922851700 5 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:307] [JOB 28] Delete db.slow/026356.sst type=2 #26356 -- OK > > -8281> 2018-07-11 08:04:50.725663 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296290725661, "job": 28, "event": "table_file_deletion", "file_number": 26356} > -8280> 2018-07-11 08:04:50.725670 7f4922851700 5 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:307] [JOB 28] Delete db.slow/026355.sst type=2 #26355 -- OK > > -8279> 2018-07-11 08:04:50.725675 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296290725673, "job": 28, "event": "table_file_deletion", "file_number": 26355} > -8278> 2018-07-11 08:04:50.725682 7f4922851700 5 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:307] [JOB 28] Delete db.slow/026354.sst type=2 #26354 -- OK > > -8277> 2018-07-11 08:04:50.725690 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296290725687, "job": 28, "event": "table_file_deletion", "file_number": 26354} > -8276> 2018-07-11 08:04:50.726113 7f4922851700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:447] ------- DUMPING STATS ------- > -8275> 2018-07-11 08:04:50.726119 7f4922851700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl.cc:448] >** DB Stats ** >Uptime(secs): 695.4 total, 626.9 interval >Cumulative writes: 624 writes, 930 keys, 624 commit groups, 1.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s >Cumulative WAL: 624 writes, 136 syncs, 4.55 writes per sync, written: 0.00 GB, 0.00 MB/s >Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent >Interval writes: 624 writes, 930 keys, 624 commit groups, 1.0 writes per commit group, ingest: 0.52 MB, 0.00 MB/s >Interval WAL: 624 writes, 136 syncs, 4.55 writes per sync, written: 0.00 MB, 0.00 MB/s >Interval stall: 00:00:0.000 H:M:S, 0.0 percent > >** Compaction Stats [default] ** >Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop >---------------------------------------------------------------------------------------------------------------------------------------------------------- > L0 1/0 218.75 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 10.9 0 1 0.020 0 0 > L1 7/0 240.38 MB 1.1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 > L2 48/0 2.67 GB 1.1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 > L3 482/0 27.19 GB 1.1 11.2 1.5 9.7 10.7 1.0 0.1 7.1 21.6 20.8 529 22 24.032 460M 22M > L4 3641/0 249.88 GB 1.0 2.0 0.1 1.9 1.8 -0.1 0.0 11.8 21.2 18.4 98 2 48.884 81M 12M > L5 796/0 55.24 GB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 > Sum 4975/0 335.21 GB 0.0 13.2 1.7 11.5 12.5 0.9 0.1 59806.5 21.6 20.4 626 25 25.060 541M 34M > Int 0/0 0.00 KB 0.0 13.2 1.7 11.5 12.5 0.9 0.1 13396727908.0 21.6 20.4 626 24 26.103 541M 34M >Uptime(secs): 695.4 total, 695.4 interval >Flush(GB): cumulative 0.000, interval 0.000 >AddFile(GB): cumulative 0.000, interval 0.000 >AddFile(Total Files): cumulative 0, interval 0 >AddFile(L0 Files): cumulative 0, interval 0 >AddFile(Keys): cumulative 0, interval 0 >Cumulative compaction: 12.48 GB write, 18.37 MB/s write, 13.20 GB read, 19.44 MB/s read, 626.5 seconds >Interval compaction: 12.48 GB write, 18.37 MB/s write, 13.20 GB read, 19.44 MB/s read, 626.5 seconds >Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count > >** File Read Latency Histogram By Level [default] ** > >** Compaction Stats [default] ** >Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop >---------------------------------------------------------------------------------------------------------------------------------------------------------- > L0 1/0 218.75 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 10.9 0 1 0.020 0 0 > L1 7/0 240.38 MB 1.1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 > L2 48/0 2.67 GB 1.1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 > L3 482/0 27.19 GB 1.1 11.2 1.5 9.7 10.7 1.0 0.1 7.1 21.6 20.8 529 22 24.032 460M 22M > L4 3641/0 249.88 GB 1.0 2.0 0.1 1.9 1.8 -0.1 0.0 11.8 21.2 18.4 98 2 48.884 81M 12M > L5 796/0 55.24 GB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 > Sum 4975/0 335.21 GB 0.0 13.2 1.7 11.5 12.5 0.9 0.1 59806.5 21.6 20.4 626 25 25.060 541M 34M > Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 >Uptime(secs): 695.4 total, 695.4 interval >Flush(GB): cumulative 0.000, interval 0.000 >AddFile(GB): cumulative 0.000, interval 0.000 >AddFile(Total Files): cumulative 0, interval 0 >AddFile(L0 Files): cumulative 0, interval 0 >AddFile(Keys): cumulative 0, interval 0 >Cumulative compaction: 12.48 GB write, 18.37 MB/s write, 13.20 GB read, 19.44 MB/s read, 626.5 seconds >Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds >Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count > >** File Read Latency Histogram By Level [default] ** > > -8274> 2018-07-11 08:04:50.726156 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_picker.cc:461] [default] Expanding@3 1+11(159662894+820009132 bytes) to 2+11 (193327614+820009132bytes) > > -8273> 2018-07-11 08:04:50.726236 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 29] Compacting 2@3 + 11@4 files to L4, score 1.13 > -8272> 2018-07-11 08:04:50.726272 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 28 Base level 3, inputs: [36785(32MB) 36581(76MB)], [27361(71MB) 27363(71MB) 27364(71MB) 27365(71MB) 27368(71MB) 27369(71MB) 26980(71MB) 26981(71MB) 26983(71MB) 26984(71MB) 26985(71MB)] > > -8271> 2018-07-11 08:04:50.726287 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296290726276, "job": 29, "event": "compaction_started", "files_L3": [36785, 36581], "files_L4": [27361, 27363, 27364, 27365, 27368, 27369, 26980, 26981, 26983, 26984, 26985], "score": 1.13023, "input_data_size": 934022106} > -8270> 2018-07-11 08:04:50.959845 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 409 0x55f65dad4200 MOSDScrubReserve(11.3s5 REQUEST e509) v1 > -8269> 2018-07-11 08:04:50.959867 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 409 ==== MOSDScrubReserve(11.3s5 REQUEST e509) v1 ==== 43+0+0 (433425070 0 0) 0x55f65dad4200 con 0x55f668a29000 > -8268> 2018-07-11 08:04:50.959903 7f491a040700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e509) v1 -- 0x55f64cc81e00 con 0 > -8267> 2018-07-11 08:04:50.960300 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 410 0x55f65d025400 MOSDScrubReserve(11.3s5 RELEASE e509) v1 > -8266> 2018-07-11 08:04:50.960315 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 410 ==== MOSDScrubReserve(11.3s5 RELEASE e509) v1 ==== 43+0+0 (3603460268 0 0) 0x55f65d025400 con 0x55f668a29000 > -8265> 2018-07-11 08:04:51.298211 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0a000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 183 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:04:51.933511) v4 > -8264> 2018-07-11 08:04:51.298240 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.11 10.8.128.114:0/15919 183 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:51.933511) v4 ==== 2004+0+0 (3838898606 0 0) 0x55f67d1afe00 con 0x55f663f0a000 > -8263> 2018-07-11 08:04:51.298268 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:51.933511) v4 -- 0x55f650f78c00 con 0 > -8262> 2018-07-11 08:04:51.298232 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0b800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 183 0x55f66b1cd800 osd_ping(ping e509 stamp 2018-07-11 08:04:51.933511) v4 > -8261> 2018-07-11 08:04:51.298291 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.11 10.8.128.114:0/15919 183 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:51.933511) v4 ==== 2004+0+0 (3838898606 0 0) 0x55f66b1cd800 con 0x55f663f0b800 > -8260> 2018-07-11 08:04:51.298306 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:51.933511) v4 -- 0x55f667d0c800 con 0 > -8259> 2018-07-11 08:04:51.899933 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f65ff0e6c0 con 0 > -8258> 2018-07-11 08:04:51.901603 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 188 0x55f676582f00 replica scrub(pg: 6.15,from:0'0,to:461'106884,epoch:509/504,start:6:a8188941:::benchmark_data_magna108_35617_object273789:0,end:6:a81995d8:::benchmark_data_magna108_35617_object794430:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -8257> 2018-07-11 08:04:51.901636 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 188 ==== replica scrub(pg: 6.15,from:0'0,to:461'106884,epoch:509/504,start:6:a8188941:::benchmark_data_magna108_35617_object273789:0,end:6:a81995d8:::benchmark_data_magna108_35617_object794430:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (1973486001 0 0) 0x55f676582f00 con 0x55f66315e800 > -8256> 2018-07-11 08:04:51.978756 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.107:0/12617 conn(0x55f64e3c3800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.9 seq 187 0x55f64cc81e00 osd_ping(ping e509 stamp 2018-07-11 08:04:51.978377) v4 > -8255> 2018-07-11 08:04:51.978756 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.107:0/12617 conn(0x55f64e3c5000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.9 seq 187 0x55f65d024a00 osd_ping(ping e509 stamp 2018-07-11 08:04:51.978377) v4 > -8254> 2018-07-11 08:04:51.978780 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.9 10.8.128.107:0/12617 187 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:51.978377) v4 ==== 2004+0+0 (735312612 0 0) 0x55f64cc81e00 con 0x55f64e3c3800 > -8253> 2018-07-11 08:04:51.978799 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.9 10.8.128.107:0/12617 187 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:51.978377) v4 ==== 2004+0+0 (735312612 0 0) 0x55f65d024a00 con 0x55f64e3c5000 > -8252> 2018-07-11 08:04:51.978829 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:51.978377) v4 -- 0x55f667d0c800 con 0 > -8251> 2018-07-11 08:04:51.978869 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:51.978377) v4 -- 0x55f65c0f2600 con 0 > -8250> 2018-07-11 08:04:52.224260 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12650 conn(0x55f663eda800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.3 seq 185 0x55f65c0f2400 osd_ping(ping e509 stamp 2018-07-11 08:04:52.280594) v4 > -8249> 2018-07-11 08:04:52.224283 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.3 10.8.128.106:0/12650 185 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:52.280594) v4 ==== 2004+0+0 (1875214979 0 0) 0x55f65c0f2400 con 0x55f663eda800 > -8248> 2018-07-11 08:04:52.224279 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12650 conn(0x55f663ed9000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.3 seq 185 0x55f66b1cd800 osd_ping(ping e509 stamp 2018-07-11 08:04:52.280594) v4 > -8247> 2018-07-11 08:04:52.224293 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:52.280594) v4 -- 0x55f65c0f2600 con 0 > -8246> 2018-07-11 08:04:52.224297 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.3 10.8.128.106:0/12650 185 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:52.280594) v4 ==== 2004+0+0 (1875214979 0 0) 0x55f66b1cd800 con 0x55f663ed9000 > -8245> 2018-07-11 08:04:52.224311 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:52.280594) v4 -- 0x55f667d0c800 con 0 > -8244> 2018-07-11 08:04:52.270922 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15255 conn(0x55f663ec1000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.0 seq 180 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:04:53.514907) v4 > -8243> 2018-07-11 08:04:52.270944 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.0 10.8.128.113:0/15255 180 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:53.514907) v4 ==== 2004+0+0 (1398046779 0 0) 0x55f64e20fc00 con 0x55f663ec1000 > -8242> 2018-07-11 08:04:52.270953 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:53.514907) v4 -- 0x55f6548fa200 con 0 > -8241> 2018-07-11 08:04:52.270965 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15255 conn(0x55f663ec2800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.0 seq 180 0x55f64cc81e00 osd_ping(ping e509 stamp 2018-07-11 08:04:53.514907) v4 > -8240> 2018-07-11 08:04:52.270986 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.0 10.8.128.113:0/15255 180 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:53.514907) v4 ==== 2004+0+0 (1398046779 0 0) 0x55f64cc81e00 con 0x55f663ec2800 > -8239> 2018-07-11 08:04:52.270998 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:53.514907) v4 -- 0x55f64f768a00 con 0 > -8238> 2018-07-11 08:04:52.294071 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.108:0/11967 conn(0x55f663f6f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=1). rx osd.8 seq 181 0x55f66b1cd800 osd_ping(ping e509 stamp 2018-07-11 08:04:52.053783) v4 > -8237> 2018-07-11 08:04:52.294104 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.8 10.8.128.108:0/11967 181 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:52.053783) v4 ==== 2004+0+0 (370808564 0 0) 0x55f66b1cd800 con 0x55f663f6f000 > -8236> 2018-07-11 08:04:52.294116 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:52.053783) v4 -- 0x55f64e25d400 con 0 > -8235> 2018-07-11 08:04:52.294165 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.108:0/11967 conn(0x55f663f70800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=1). rx osd.8 seq 181 0x55f65d024a00 osd_ping(ping e509 stamp 2018-07-11 08:04:52.053783) v4 > -8234> 2018-07-11 08:04:52.294187 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.8 10.8.128.108:0/11967 181 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:52.053783) v4 ==== 2004+0+0 (370808564 0 0) 0x55f65d024a00 con 0x55f663f70800 > -8233> 2018-07-11 08:04:52.294198 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:52.053783) v4 -- 0x55f65c0f2800 con 0 > -8232> 2018-07-11 08:04:52.530681 7f491803c700 5 osd.10 509 heartbeat: osd_stat(789 GB used, 142 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,0,0,1]) > -8231> 2018-07-11 08:04:52.530696 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6802/15255 -- osd_ping(ping e509 stamp 2018-07-11 08:04:52.530694) v4 -- 0x55f6716b2800 con 0 > -8230> 2018-07-11 08:04:52.530735 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6803/15255 -- osd_ping(ping e509 stamp 2018-07-11 08:04:52.530694) v4 -- 0x55f6716b2a00 con 0 > -8229> 2018-07-11 08:04:52.530751 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6806/12677 -- osd_ping(ping e509 stamp 2018-07-11 08:04:52.530694) v4 -- 0x55f6716b2c00 con 0 > -8228> 2018-07-11 08:04:52.530765 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6807/12677 -- osd_ping(ping e509 stamp 2018-07-11 08:04:52.530694) v4 -- 0x55f6716b2e00 con 0 > -8227> 2018-07-11 08:04:52.530777 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6808/12651 -- osd_ping(ping e509 stamp 2018-07-11 08:04:52.530694) v4 -- 0x55f6716b3000 con 0 > -8226> 2018-07-11 08:04:52.530789 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6810/12651 -- osd_ping(ping e509 stamp 2018-07-11 08:04:52.530694) v4 -- 0x55f6716b3200 con 0 > -8225> 2018-07-11 08:04:52.530801 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6809/12650 -- osd_ping(ping e509 stamp 2018-07-11 08:04:52.530694) v4 -- 0x55f6716b3400 con 0 > -8224> 2018-07-11 08:04:52.530813 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6811/12650 -- osd_ping(ping e509 stamp 2018-07-11 08:04:52.530694) v4 -- 0x55f6716b3600 con 0 > -8223> 2018-07-11 08:04:52.530832 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6810/15287 -- osd_ping(ping e509 stamp 2018-07-11 08:04:52.530694) v4 -- 0x55f6716b3800 con 0 > -8222> 2018-07-11 08:04:52.530845 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6811/15287 -- osd_ping(ping e509 stamp 2018-07-11 08:04:52.530694) v4 -- 0x55f6716b3a00 con 0 > -8221> 2018-07-11 08:04:52.530858 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6806/15263 -- osd_ping(ping e509 stamp 2018-07-11 08:04:52.530694) v4 -- 0x55f6716b3c00 con 0 > -8220> 2018-07-11 08:04:52.530873 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6807/15263 -- osd_ping(ping e509 stamp 2018-07-11 08:04:52.530694) v4 -- 0x55f6716b3e00 con 0 > -8219> 2018-07-11 08:04:52.530887 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.115:6802/13323 -- osd_ping(ping e509 stamp 2018-07-11 08:04:52.530694) v4 -- 0x55f681f90c00 con 0 > -8218> 2018-07-11 08:04:52.530900 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.115:6803/13323 -- osd_ping(ping e509 stamp 2018-07-11 08:04:52.530694) v4 -- 0x55f681f90e00 con 0 > -8217> 2018-07-11 08:04:52.530913 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6806/15885 -- osd_ping(ping e509 stamp 2018-07-11 08:04:52.530694) v4 -- 0x55f681f91600 con 0 > -8216> 2018-07-11 08:04:52.530925 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6807/15885 -- osd_ping(ping e509 stamp 2018-07-11 08:04:52.530694) v4 -- 0x55f681f91000 con 0 > -8215> 2018-07-11 08:04:52.530936 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.108:6802/11967 -- osd_ping(ping e509 stamp 2018-07-11 08:04:52.530694) v4 -- 0x55f66d000000 con 0 > -8214> 2018-07-11 08:04:52.530947 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.108:6803/11967 -- osd_ping(ping e509 stamp 2018-07-11 08:04:52.530694) v4 -- 0x55f66d000200 con 0 > -8213> 2018-07-11 08:04:52.530960 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.107:6806/12617 -- osd_ping(ping e509 stamp 2018-07-11 08:04:52.530694) v4 -- 0x55f66d000400 con 0 > -8212> 2018-07-11 08:04:52.530972 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.107:6807/12617 -- osd_ping(ping e509 stamp 2018-07-11 08:04:52.530694) v4 -- 0x55f66d000600 con 0 > -8211> 2018-07-11 08:04:52.530983 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6802/15919 -- osd_ping(ping e509 stamp 2018-07-11 08:04:52.530694) v4 -- 0x55f66d000800 con 0 > -8210> 2018-07-11 08:04:52.530995 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6803/15919 -- osd_ping(ping e509 stamp 2018-07-11 08:04:52.530694) v4 -- 0x55f66d000a00 con 0 > -8209> 2018-07-11 08:04:52.531145 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.107:6806/12617 conn(0x55f656ade000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.9 seq 186 0x55f66b1cd800 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:52.530694) v4 > -8208> 2018-07-11 08:04:52.531180 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.9 10.8.128.107:6806/12617 186 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:52.530694) v4 ==== 2004+0+0 (2619092397 0 0) 0x55f66b1cd800 con 0x55f656ade000 > -8207> 2018-07-11 08:04:52.531172 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6802/15255 conn(0x55f656ad4000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 186 0x55f65c0f2400 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:52.530694) v4 > -8206> 2018-07-11 08:04:52.531193 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.0 10.8.128.113:6802/15255 186 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:52.530694) v4 ==== 2004+0+0 (2619092397 0 0) 0x55f65c0f2400 con 0x55f656ad4000 > -8205> 2018-07-11 08:04:52.531222 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6803/15255 conn(0x55f656ad5800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 186 0x55f66b1cd800 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:52.530694) v4 > -8204> 2018-07-11 08:04:52.531229 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.107:6807/12617 conn(0x55f656adf800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.9 seq 186 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:52.530694) v4 > -8203> 2018-07-11 08:04:52.531257 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.9 10.8.128.107:6807/12617 186 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:52.530694) v4 ==== 2004+0+0 (2619092397 0 0) 0x55f64e20fc00 con 0x55f656adf800 > -8202> 2018-07-11 08:04:52.531236 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.0 10.8.128.113:6803/15255 186 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:52.530694) v4 ==== 2004+0+0 (2619092397 0 0) 0x55f66b1cd800 con 0x55f656ad5800 > -8201> 2018-07-11 08:04:52.531270 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6807/12677 conn(0x55f64e3d9800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.1 seq 186 0x55f65c0f2400 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:52.530694) v4 > -8200> 2018-07-11 08:04:52.531287 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.1 10.8.128.106:6807/12677 186 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:52.530694) v4 ==== 2004+0+0 (2619092397 0 0) 0x55f65c0f2400 con 0x55f64e3d9800 > -8199> 2018-07-11 08:04:52.531319 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6811/15287 conn(0x55f64e402800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.4 seq 186 0x55f66b1cd800 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:52.530694) v4 > -8198> 2018-07-11 08:04:52.531328 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6808/12651 conn(0x55f64e3ea000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.2 seq 186 0x55f65c0f2400 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:52.530694) v4 > -8197> 2018-07-11 08:04:52.531345 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.2 10.8.128.106:6808/12651 186 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:52.530694) v4 ==== 2004+0+0 (2619092397 0 0) 0x55f65c0f2400 con 0x55f64e3ea000 > -8196> 2018-07-11 08:04:52.531334 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.4 10.8.128.113:6811/15287 186 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:52.530694) v4 ==== 2004+0+0 (2619092397 0 0) 0x55f66b1cd800 con 0x55f64e402800 > -8195> 2018-07-11 08:04:52.531354 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6806/12677 conn(0x55f64e3d8000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.1 seq 186 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:52.530694) v4 > -8194> 2018-07-11 08:04:52.531376 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.1 10.8.128.106:6806/12677 186 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:52.530694) v4 ==== 2004+0+0 (2619092397 0 0) 0x55f64e20fc00 con 0x55f64e3d8000 > -8193> 2018-07-11 08:04:52.531390 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6811/12650 conn(0x55f656ad8800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.3 seq 186 0x55f65c0f2400 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:52.530694) v4 > -8192> 2018-07-11 08:04:52.531417 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6809/12650 conn(0x55f656ad7000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.3 seq 186 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:52.530694) v4 > -8191> 2018-07-11 08:04:52.531430 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.3 10.8.128.106:6809/12650 186 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:52.530694) v4 ==== 2004+0+0 (2619092397 0 0) 0x55f64e20fc00 con 0x55f656ad7000 > -8190> 2018-07-11 08:04:52.531453 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6810/12651 conn(0x55f64e3eb800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.2 seq 186 0x55f66b1cd800 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:52.530694) v4 > -8189> 2018-07-11 08:04:52.531461 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.115:6803/13323 conn(0x55f656ace800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.6 seq 186 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:52.530694) v4 > -8188> 2018-07-11 08:04:52.531455 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.3 10.8.128.106:6811/12650 186 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:52.530694) v4 ==== 2004+0+0 (2619092397 0 0) 0x55f65c0f2400 con 0x55f656ad8800 > -8187> 2018-07-11 08:04:52.531471 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.6 10.8.128.115:6803/13323 186 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:52.530694) v4 ==== 2004+0+0 (2619092397 0 0) 0x55f64e20fc00 con 0x55f656ace800 > -8186> 2018-07-11 08:04:52.531471 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.2 10.8.128.106:6810/12651 186 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:52.530694) v4 ==== 2004+0+0 (2619092397 0 0) 0x55f66b1cd800 con 0x55f64e3eb800 > -8185> 2018-07-11 08:04:52.531507 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6806/15263 conn(0x55f64e3ed000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.5 seq 186 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:52.530694) v4 > -8184> 2018-07-11 08:04:52.531515 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.115:6802/13323 conn(0x55f656acd000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.6 seq 186 0x55f66b1cd800 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:52.530694) v4 > -8183> 2018-07-11 08:04:52.531520 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.5 10.8.128.113:6806/15263 186 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:52.530694) v4 ==== 2004+0+0 (2619092397 0 0) 0x55f64e20fc00 con 0x55f64e3ed000 > -8182> 2018-07-11 08:04:52.531536 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.6 10.8.128.115:6802/13323 186 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:52.530694) v4 ==== 2004+0+0 (2619092397 0 0) 0x55f66b1cd800 con 0x55f656acd000 > -8181> 2018-07-11 08:04:52.531513 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6807/15263 conn(0x55f64e3ee800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.5 seq 186 0x55f65c0f2400 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:52.530694) v4 > -8180> 2018-07-11 08:04:52.531546 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.5 10.8.128.113:6807/15263 186 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:52.530694) v4 ==== 2004+0+0 (2619092397 0 0) 0x55f65c0f2400 con 0x55f64e3ee800 > -8179> 2018-07-11 08:04:52.531573 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6806/15885 conn(0x55f64e3fe000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.7 seq 186 0x55f66b1cd800 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:52.530694) v4 > -8178> 2018-07-11 08:04:52.531588 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.7 10.8.128.114:6806/15885 186 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:52.530694) v4 ==== 2004+0+0 (2619092397 0 0) 0x55f66b1cd800 con 0x55f64e3fe000 > -8177> 2018-07-11 08:04:52.531588 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.108:6803/11967 conn(0x55f656acb800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.8 seq 186 0x55f65c0f2400 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:52.530694) v4 > -8176> 2018-07-11 08:04:52.531593 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.108:6802/11967 conn(0x55f656aca000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.8 seq 186 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:52.530694) v4 > -8175> 2018-07-11 08:04:52.531613 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.8 10.8.128.108:6802/11967 186 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:52.530694) v4 ==== 2004+0+0 (2619092397 0 0) 0x55f64e20fc00 con 0x55f656aca000 > -8174> 2018-07-11 08:04:52.531610 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.8 10.8.128.108:6803/11967 186 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:52.530694) v4 ==== 2004+0+0 (2619092397 0 0) 0x55f65c0f2400 con 0x55f656acb800 > -8173> 2018-07-11 08:04:52.531623 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6802/15919 conn(0x55f64e3db000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.11 seq 186 0x55f66b1cd800 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:52.530694) v4 > -8172> 2018-07-11 08:04:52.531635 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.11 10.8.128.114:6802/15919 186 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:52.530694) v4 ==== 2004+0+0 (2619092397 0 0) 0x55f66b1cd800 con 0x55f64e3db000 > -8171> 2018-07-11 08:04:52.531650 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6807/15885 conn(0x55f64e3ff800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.7 seq 186 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:52.530694) v4 > -8170> 2018-07-11 08:04:52.531663 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.7 10.8.128.114:6807/15885 186 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:52.530694) v4 ==== 2004+0+0 (2619092397 0 0) 0x55f64e20fc00 con 0x55f64e3ff800 > -8169> 2018-07-11 08:04:52.531658 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6810/15287 conn(0x55f64e401000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.4 seq 186 0x55f65c0f2400 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:52.530694) v4 > -8168> 2018-07-11 08:04:52.531673 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.4 10.8.128.113:6810/15287 186 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:52.530694) v4 ==== 2004+0+0 (2619092397 0 0) 0x55f65c0f2400 con 0x55f64e401000 > -8167> 2018-07-11 08:04:52.531692 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6803/15919 conn(0x55f64e3dc800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.11 seq 186 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:52.530694) v4 > -8166> 2018-07-11 08:04:52.531704 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.11 10.8.128.114:6803/15919 186 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:52.530694) v4 ==== 2004+0+0 (2619092397 0 0) 0x55f64e20fc00 con 0x55f64e3dc800 > -8165> 2018-07-11 08:04:52.805407 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.115:0/13323 conn(0x55f663ebf800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.6 seq 178 0x55f65c0f2400 osd_ping(ping e509 stamp 2018-07-11 08:04:53.157691) v4 > -8164> 2018-07-11 08:04:52.805428 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.6 10.8.128.115:0/13323 178 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:53.157691) v4 ==== 2004+0+0 (213201477 0 0) 0x55f65c0f2400 con 0x55f663ebf800 > -8163> 2018-07-11 08:04:52.805419 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.115:0/13323 conn(0x55f663ebe000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.6 seq 178 0x55f66b1cd800 osd_ping(ping e509 stamp 2018-07-11 08:04:53.157691) v4 > -8162> 2018-07-11 08:04:52.805437 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:53.157691) v4 -- 0x55f6716b3000 con 0 > -8161> 2018-07-11 08:04:52.805449 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.6 10.8.128.115:0/13323 178 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:53.157691) v4 ==== 2004+0+0 (213201477 0 0) 0x55f66b1cd800 con 0x55f663ebe000 > -8160> 2018-07-11 08:04:52.805466 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:53.157691) v4 -- 0x55f6716b2a00 con 0 > -8159> 2018-07-11 08:04:52.827054 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15885 conn(0x55f663217800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 189 0x55f64cc81e00 osd_ping(ping e509 stamp 2018-07-11 08:04:53.462359) v4 > -8158> 2018-07-11 08:04:52.827075 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.7 10.8.128.114:0/15885 189 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:53.462359) v4 ==== 2004+0+0 (387268231 0 0) 0x55f64cc81e00 con 0x55f663217800 > -8157> 2018-07-11 08:04:52.827075 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15885 conn(0x55f663216000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 189 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:04:53.462359) v4 > -8156> 2018-07-11 08:04:52.827091 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:53.462359) v4 -- 0x55f6716b2a00 con 0 > -8155> 2018-07-11 08:04:52.827093 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.7 10.8.128.114:0/15885 189 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:53.462359) v4 ==== 2004+0+0 (387268231 0 0) 0x55f64e20fc00 con 0x55f663216000 > -8154> 2018-07-11 08:04:52.827125 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:53.462359) v4 -- 0x55f6716b3400 con 0 > -8153> 2018-07-11 08:04:53.031170 7f491803c700 5 osd.10 509 heartbeat: osd_stat(789 GB used, 142 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,0,0,0,1]) > -8152> 2018-07-11 08:04:53.031183 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6802/15255 -- osd_ping(ping e509 stamp 2018-07-11 08:04:53.031182) v4 -- 0x55f66d000c00 con 0 > -8151> 2018-07-11 08:04:53.031218 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6803/15255 -- osd_ping(ping e509 stamp 2018-07-11 08:04:53.031182) v4 -- 0x55f66d000e00 con 0 > -8150> 2018-07-11 08:04:53.031231 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6806/12677 -- osd_ping(ping e509 stamp 2018-07-11 08:04:53.031182) v4 -- 0x55f66d001000 con 0 > -8149> 2018-07-11 08:04:53.031243 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6807/12677 -- osd_ping(ping e509 stamp 2018-07-11 08:04:53.031182) v4 -- 0x55f66d001200 con 0 > -8148> 2018-07-11 08:04:53.031282 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6808/12651 -- osd_ping(ping e509 stamp 2018-07-11 08:04:53.031182) v4 -- 0x55f66d001400 con 0 > -8147> 2018-07-11 08:04:53.031293 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6810/12651 -- osd_ping(ping e509 stamp 2018-07-11 08:04:53.031182) v4 -- 0x55f66d001600 con 0 > -8146> 2018-07-11 08:04:53.031305 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6809/12650 -- osd_ping(ping e509 stamp 2018-07-11 08:04:53.031182) v4 -- 0x55f66d001800 con 0 > -8145> 2018-07-11 08:04:53.031317 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6811/12650 -- osd_ping(ping e509 stamp 2018-07-11 08:04:53.031182) v4 -- 0x55f66d001a00 con 0 > -8144> 2018-07-11 08:04:53.031336 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6810/15287 -- osd_ping(ping e509 stamp 2018-07-11 08:04:53.031182) v4 -- 0x55f66d001c00 con 0 > -8143> 2018-07-11 08:04:53.031347 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6811/15287 -- osd_ping(ping e509 stamp 2018-07-11 08:04:53.031182) v4 -- 0x55f66d001e00 con 0 > -8142> 2018-07-11 08:04:53.031360 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6806/15263 -- osd_ping(ping e509 stamp 2018-07-11 08:04:53.031182) v4 -- 0x55f66e944000 con 0 > -8141> 2018-07-11 08:04:53.031373 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6807/15263 -- osd_ping(ping e509 stamp 2018-07-11 08:04:53.031182) v4 -- 0x55f66e944200 con 0 > -8140> 2018-07-11 08:04:53.031390 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.115:6802/13323 -- osd_ping(ping e509 stamp 2018-07-11 08:04:53.031182) v4 -- 0x55f66e944400 con 0 > -8139> 2018-07-11 08:04:53.031403 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.115:6803/13323 -- osd_ping(ping e509 stamp 2018-07-11 08:04:53.031182) v4 -- 0x55f66e944600 con 0 > -8138> 2018-07-11 08:04:53.031416 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6806/15885 -- osd_ping(ping e509 stamp 2018-07-11 08:04:53.031182) v4 -- 0x55f66e944800 con 0 > -8137> 2018-07-11 08:04:53.031428 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6807/15885 -- osd_ping(ping e509 stamp 2018-07-11 08:04:53.031182) v4 -- 0x55f66e944a00 con 0 > -8136> 2018-07-11 08:04:53.031441 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.108:6802/11967 -- osd_ping(ping e509 stamp 2018-07-11 08:04:53.031182) v4 -- 0x55f66e944c00 con 0 > -8135> 2018-07-11 08:04:53.031452 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.108:6803/11967 -- osd_ping(ping e509 stamp 2018-07-11 08:04:53.031182) v4 -- 0x55f66e944e00 con 0 > -8134> 2018-07-11 08:04:53.031464 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.107:6806/12617 -- osd_ping(ping e509 stamp 2018-07-11 08:04:53.031182) v4 -- 0x55f66e945000 con 0 > -8133> 2018-07-11 08:04:53.031476 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.107:6807/12617 -- osd_ping(ping e509 stamp 2018-07-11 08:04:53.031182) v4 -- 0x55f66e945200 con 0 > -8132> 2018-07-11 08:04:53.031488 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6802/15919 -- osd_ping(ping e509 stamp 2018-07-11 08:04:53.031182) v4 -- 0x55f66e945400 con 0 > -8131> 2018-07-11 08:04:53.031498 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6803/15919 -- osd_ping(ping e509 stamp 2018-07-11 08:04:53.031182) v4 -- 0x55f66e945600 con 0 > -8130> 2018-07-11 08:04:53.031619 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6802/15255 conn(0x55f656ad4000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 187 0x55f65d024a00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:53.031182) v4 > -8129> 2018-07-11 08:04:53.031622 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6803/15255 conn(0x55f656ad5800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 187 0x55f66b1cd800 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:53.031182) v4 > -8128> 2018-07-11 08:04:53.031645 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.0 10.8.128.113:6802/15255 187 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:53.031182) v4 ==== 2004+0+0 (73063345 0 0) 0x55f65d024a00 con 0x55f656ad4000 > -8127> 2018-07-11 08:04:53.031649 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.0 10.8.128.113:6803/15255 187 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:53.031182) v4 ==== 2004+0+0 (73063345 0 0) 0x55f66b1cd800 con 0x55f656ad5800 > -8126> 2018-07-11 08:04:53.031682 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.107:6807/12617 conn(0x55f656adf800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.9 seq 187 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:53.031182) v4 > -8125> 2018-07-11 08:04:53.031703 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.9 10.8.128.107:6807/12617 187 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:53.031182) v4 ==== 2004+0+0 (73063345 0 0) 0x55f67d1afe00 con 0x55f656adf800 > -8124> 2018-07-11 08:04:53.031742 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6807/12677 conn(0x55f64e3d9800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.1 seq 187 0x55f65d024a00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:53.031182) v4 > -8123> 2018-07-11 08:04:53.031765 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.1 10.8.128.106:6807/12677 187 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:53.031182) v4 ==== 2004+0+0 (73063345 0 0) 0x55f65d024a00 con 0x55f64e3d9800 > -8122> 2018-07-11 08:04:53.031800 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6806/12677 conn(0x55f64e3d8000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.1 seq 187 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:53.031182) v4 > -8121> 2018-07-11 08:04:53.031816 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.1 10.8.128.106:6806/12677 187 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:53.031182) v4 ==== 2004+0+0 (73063345 0 0) 0x55f67d1afe00 con 0x55f64e3d8000 > -8120> 2018-07-11 08:04:53.031810 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6810/12651 conn(0x55f64e3eb800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.2 seq 187 0x55f66b1cd800 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:53.031182) v4 > -8119> 2018-07-11 08:04:53.031840 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6811/12650 conn(0x55f656ad8800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.3 seq 187 0x55f65d024a00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:53.031182) v4 > -8118> 2018-07-11 08:04:53.031857 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.3 10.8.128.106:6811/12650 187 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:53.031182) v4 ==== 2004+0+0 (73063345 0 0) 0x55f65d024a00 con 0x55f656ad8800 > -8117> 2018-07-11 08:04:53.031860 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6806/15263 conn(0x55f64e3ed000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.5 seq 187 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:53.031182) v4 > -8116> 2018-07-11 08:04:53.031881 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.2 10.8.128.106:6810/12651 187 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:53.031182) v4 ==== 2004+0+0 (73063345 0 0) 0x55f66b1cd800 con 0x55f64e3eb800 > -8115> 2018-07-11 08:04:53.031888 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.5 10.8.128.113:6806/15263 187 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:53.031182) v4 ==== 2004+0+0 (73063345 0 0) 0x55f67d1afe00 con 0x55f64e3ed000 > -8114> 2018-07-11 08:04:53.031910 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6810/15287 conn(0x55f64e401000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.4 seq 187 0x55f65d024a00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:53.031182) v4 > -8113> 2018-07-11 08:04:53.031920 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6809/12650 conn(0x55f656ad7000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.3 seq 187 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:53.031182) v4 > -8112> 2018-07-11 08:04:53.031916 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.107:6806/12617 conn(0x55f656ade000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.9 seq 187 0x55f66b1cd800 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:53.031182) v4 > -8111> 2018-07-11 08:04:53.031929 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.4 10.8.128.113:6810/15287 187 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:53.031182) v4 ==== 2004+0+0 (73063345 0 0) 0x55f65d024a00 con 0x55f64e401000 > -8110> 2018-07-11 08:04:53.031934 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.3 10.8.128.106:6809/12650 187 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:53.031182) v4 ==== 2004+0+0 (73063345 0 0) 0x55f67d1afe00 con 0x55f656ad7000 > -8109> 2018-07-11 08:04:53.031938 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.9 10.8.128.107:6806/12617 187 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:53.031182) v4 ==== 2004+0+0 (73063345 0 0) 0x55f66b1cd800 con 0x55f656ade000 > -8108> 2018-07-11 08:04:53.031966 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6811/15287 conn(0x55f64e402800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.4 seq 187 0x55f66b1cd800 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:53.031182) v4 > -8107> 2018-07-11 08:04:53.031961 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6808/12651 conn(0x55f64e3ea000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.2 seq 187 0x55f65d024a00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:53.031182) v4 > -8106> 2018-07-11 08:04:53.031979 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.4 10.8.128.113:6811/15287 187 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:53.031182) v4 ==== 2004+0+0 (73063345 0 0) 0x55f66b1cd800 con 0x55f64e402800 > -8105> 2018-07-11 08:04:53.031964 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6807/15885 conn(0x55f64e3ff800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.7 seq 187 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:53.031182) v4 > -8104> 2018-07-11 08:04:53.031985 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.2 10.8.128.106:6808/12651 187 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:53.031182) v4 ==== 2004+0+0 (73063345 0 0) 0x55f65d024a00 con 0x55f64e3ea000 > -8103> 2018-07-11 08:04:53.031988 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.7 10.8.128.114:6807/15885 187 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:53.031182) v4 ==== 2004+0+0 (73063345 0 0) 0x55f67d1afe00 con 0x55f64e3ff800 > -8102> 2018-07-11 08:04:53.032009 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.115:6802/13323 conn(0x55f656acd000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.6 seq 187 0x55f66b1cd800 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:53.031182) v4 > -8101> 2018-07-11 08:04:53.032023 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.6 10.8.128.115:6802/13323 187 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:53.031182) v4 ==== 2004+0+0 (73063345 0 0) 0x55f66b1cd800 con 0x55f656acd000 > -8100> 2018-07-11 08:04:53.032021 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6807/15263 conn(0x55f64e3ee800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.5 seq 187 0x55f65d024a00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:53.031182) v4 > -8099> 2018-07-11 08:04:53.032025 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.115:6803/13323 conn(0x55f656ace800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.6 seq 187 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:53.031182) v4 > -8098> 2018-07-11 08:04:53.032033 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.5 10.8.128.113:6807/15263 187 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:53.031182) v4 ==== 2004+0+0 (73063345 0 0) 0x55f65d024a00 con 0x55f64e3ee800 > -8097> 2018-07-11 08:04:53.032038 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.6 10.8.128.115:6803/13323 187 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:53.031182) v4 ==== 2004+0+0 (73063345 0 0) 0x55f67d1afe00 con 0x55f656ace800 > -8096> 2018-07-11 08:04:53.032053 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6806/15885 conn(0x55f64e3fe000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.7 seq 187 0x55f66b1cd800 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:53.031182) v4 > -8095> 2018-07-11 08:04:53.032065 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.7 10.8.128.114:6806/15885 187 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:53.031182) v4 ==== 2004+0+0 (73063345 0 0) 0x55f66b1cd800 con 0x55f64e3fe000 > -8094> 2018-07-11 08:04:53.032074 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6803/15919 conn(0x55f64e3dc800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.11 seq 187 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:53.031182) v4 > -8093> 2018-07-11 08:04:53.032087 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.11 10.8.128.114:6803/15919 187 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:53.031182) v4 ==== 2004+0+0 (73063345 0 0) 0x55f67d1afe00 con 0x55f64e3dc800 > -8092> 2018-07-11 08:04:53.032096 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6802/15919 conn(0x55f64e3db000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.11 seq 187 0x55f66b1cd800 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:53.031182) v4 > -8091> 2018-07-11 08:04:53.032108 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.11 10.8.128.114:6802/15919 187 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:53.031182) v4 ==== 2004+0+0 (73063345 0 0) 0x55f66b1cd800 con 0x55f64e3db000 > -8090> 2018-07-11 08:04:53.032117 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.108:6802/11967 conn(0x55f656aca000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.8 seq 187 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:53.031182) v4 > -8089> 2018-07-11 08:04:53.032129 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.8 10.8.128.108:6802/11967 187 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:53.031182) v4 ==== 2004+0+0 (73063345 0 0) 0x55f67d1afe00 con 0x55f656aca000 > -8088> 2018-07-11 08:04:53.032190 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.108:6803/11967 conn(0x55f656acb800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.8 seq 187 0x55f65d024a00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:53.031182) v4 > -8087> 2018-07-11 08:04:53.032205 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.8 10.8.128.108:6803/11967 187 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:53.031182) v4 ==== 2004+0+0 (73063345 0 0) 0x55f65d024a00 con 0x55f656acb800 > -8086> 2018-07-11 08:04:53.250486 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0e800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 192 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:04:54.494489) v4 > -8085> 2018-07-11 08:04:53.250519 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.5 10.8.128.113:0/15263 192 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:54.494489) v4 ==== 2004+0+0 (3833438902 0 0) 0x55f67d1afe00 con 0x55f663f0e800 > -8084> 2018-07-11 08:04:53.250515 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0d000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 192 0x55f65d024a00 osd_ping(ping e509 stamp 2018-07-11 08:04:54.494489) v4 > -8083> 2018-07-11 08:04:53.250530 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:54.494489) v4 -- 0x55f66e945600 con 0 > -8082> 2018-07-11 08:04:53.250540 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.5 10.8.128.113:0/15263 192 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:54.494489) v4 ==== 2004+0+0 (3833438902 0 0) 0x55f65d024a00 con 0x55f663f0d000 > -8081> 2018-07-11 08:04:53.250563 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:54.494489) v4 -- 0x55f66e944e00 con 0 > -8080> 2018-07-11 08:04:53.288007 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f65bffe240 con 0 > -8079> 2018-07-11 08:04:53.289689 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 189 0x55f676583180 replica scrub(pg: 6.15,from:0'0,to:430'106866,epoch:509/504,start:6:a81995d8:::benchmark_data_magna108_35617_object794430:0,end:6:a81aad1f:::benchmark_data_magna108_35617_object201548:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -8078> 2018-07-11 08:04:53.289714 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 189 ==== replica scrub(pg: 6.15,from:0'0,to:430'106866,epoch:509/504,start:6:a81995d8:::benchmark_data_magna108_35617_object794430:0,end:6:a81aad1f:::benchmark_data_magna108_35617_object201548:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (3112704556 0 0) 0x55f676583180 con 0x55f66315e800 > -8077> 2018-07-11 08:04:53.324612 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12650 conn(0x55f663eda800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.3 seq 186 0x55f65c0f2400 osd_ping(ping e509 stamp 2018-07-11 08:04:53.381043) v4 > -8076> 2018-07-11 08:04:53.324634 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.3 10.8.128.106:0/12650 186 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:53.381043) v4 ==== 2004+0+0 (1643791233 0 0) 0x55f65c0f2400 con 0x55f663eda800 > -8075> 2018-07-11 08:04:53.324632 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12650 conn(0x55f663ed9000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.3 seq 186 0x55f66b1cd800 osd_ping(ping e509 stamp 2018-07-11 08:04:53.381043) v4 > -8074> 2018-07-11 08:04:53.324650 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:53.381043) v4 -- 0x55f66e944e00 con 0 > -8073> 2018-07-11 08:04:53.324666 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.3 10.8.128.106:0/12650 186 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:53.381043) v4 ==== 2004+0+0 (1643791233 0 0) 0x55f66b1cd800 con 0x55f663ed9000 > -8072> 2018-07-11 08:04:53.324693 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:53.381043) v4 -- 0x55f66e945400 con 0 > -8071> 2018-07-11 08:04:53.571083 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed6000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 188 0x55f65d024a00 osd_ping(ping e509 stamp 2018-07-11 08:04:53.627427) v4 > -8070> 2018-07-11 08:04:53.571104 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.1 10.8.128.106:0/12677 188 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:53.627427) v4 ==== 2004+0+0 (1186124651 0 0) 0x55f65d024a00 con 0x55f663ed6000 > -8069> 2018-07-11 08:04:53.571114 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:53.627427) v4 -- 0x55f66e944200 con 0 > -8068> 2018-07-11 08:04:53.571129 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed7800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 188 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:04:53.627427) v4 > -8067> 2018-07-11 08:04:53.571150 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.1 10.8.128.106:0/12677 188 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:53.627427) v4 ==== 2004+0+0 (1186124651 0 0) 0x55f64e20fc00 con 0x55f663ed7800 > -8066> 2018-07-11 08:04:53.571166 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:53.627427) v4 -- 0x55f66e945200 con 0 > -8065> 2018-07-11 08:04:53.898230 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15287 conn(0x55f66dce2800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.4 seq 195 0x55f65c0f2400 osd_ping(ping e509 stamp 2018-07-11 08:04:55.142133) v4 > -8064> 2018-07-11 08:04:53.898265 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.4 10.8.128.113:0/15287 195 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:55.142133) v4 ==== 2004+0+0 (3490780573 0 0) 0x55f65c0f2400 con 0x55f66dce2800 > -8063> 2018-07-11 08:04:53.898275 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:55.142133) v4 -- 0x55f66d001c00 con 0 > -8062> 2018-07-11 08:04:53.898311 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15287 conn(0x55f66dce1000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.4 seq 195 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:04:55.142133) v4 > -8061> 2018-07-11 08:04:53.898333 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.4 10.8.128.113:0/15287 195 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:55.142133) v4 ==== 2004+0+0 (3490780573 0 0) 0x55f67d1afe00 con 0x55f66dce1000 > -8060> 2018-07-11 08:04:53.898343 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:55.142133) v4 -- 0x55f66e944c00 con 0 > -8059> 2018-07-11 08:04:53.971442 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15255 conn(0x55f663ec1000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.0 seq 181 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:04:55.215374) v4 > -8058> 2018-07-11 08:04:53.971463 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.0 10.8.128.113:0/15255 181 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:55.215374) v4 ==== 2004+0+0 (3960278713 0 0) 0x55f64e20fc00 con 0x55f663ec1000 > -8057> 2018-07-11 08:04:53.971475 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:55.215374) v4 -- 0x55f66e944a00 con 0 > -8056> 2018-07-11 08:04:53.971485 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15255 conn(0x55f663ec2800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.0 seq 181 0x55f64cc81e00 osd_ping(ping e509 stamp 2018-07-11 08:04:55.215374) v4 > -8055> 2018-07-11 08:04:53.971506 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.0 10.8.128.113:0/15255 181 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:55.215374) v4 ==== 2004+0+0 (3960278713 0 0) 0x55f64cc81e00 con 0x55f663ec2800 > -8054> 2018-07-11 08:04:53.971518 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:55.215374) v4 -- 0x55f66e945000 con 0 > -8053> 2018-07-11 08:04:54.055879 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f67bdf5d40 con 0 > -8052> 2018-07-11 08:04:54.057531 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 190 0x55f676583400 replica scrub(pg: 6.15,from:0'0,to:430'106463,epoch:509/504,start:6:a81aad1f:::benchmark_data_magna108_35617_object201548:0,end:6:a81ba62e:::benchmark_data_magna108.ceph.redhat.com_2478_object138007:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -8051> 2018-07-11 08:04:54.057564 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 190 ==== replica scrub(pg: 6.15,from:0'0,to:430'106463,epoch:509/504,start:6:a81aad1f:::benchmark_data_magna108_35617_object201548:0,end:6:a81ba62e:::benchmark_data_magna108.ceph.redhat.com_2478_object138007:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (4002111628 0 0) 0x55f676583400 con 0x55f66315e800 > -8050> 2018-07-11 08:04:54.198779 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0a000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 184 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:04:54.834053) v4 > -8049> 2018-07-11 08:04:54.198800 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.11 10.8.128.114:0/15919 184 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:54.834053) v4 ==== 2004+0+0 (2969610371 0 0) 0x55f67d1afe00 con 0x55f663f0a000 > -8048> 2018-07-11 08:04:54.198817 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:54.834053) v4 -- 0x55f66e944a00 con 0 > -8047> 2018-07-11 08:04:54.198809 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0b800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 184 0x55f66b1cd800 osd_ping(ping e509 stamp 2018-07-11 08:04:54.834053) v4 > -8046> 2018-07-11 08:04:54.198832 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.11 10.8.128.114:0/15919 184 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:54.834053) v4 ==== 2004+0+0 (2969610371 0 0) 0x55f66b1cd800 con 0x55f663f0b800 > -8045> 2018-07-11 08:04:54.198846 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:54.834053) v4 -- 0x55f66e945000 con 0 > -8044> 2018-07-11 08:04:54.279293 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.107:0/12617 conn(0x55f64e3c5000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.9 seq 188 0x55f65d024a00 osd_ping(ping e509 stamp 2018-07-11 08:04:54.278921) v4 > -8043> 2018-07-11 08:04:54.279314 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.107:0/12617 conn(0x55f64e3c3800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.9 seq 188 0x55f64cc81e00 osd_ping(ping e509 stamp 2018-07-11 08:04:54.278921) v4 > -8042> 2018-07-11 08:04:54.279335 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.9 10.8.128.107:0/12617 188 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:54.278921) v4 ==== 2004+0+0 (3204599243 0 0) 0x55f64cc81e00 con 0x55f64e3c3800 > -8041> 2018-07-11 08:04:54.279313 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.9 10.8.128.107:0/12617 188 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:54.278921) v4 ==== 2004+0+0 (3204599243 0 0) 0x55f65d024a00 con 0x55f64e3c5000 > -8040> 2018-07-11 08:04:54.279352 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:54.278921) v4 -- 0x55f66e945000 con 0 > -8039> 2018-07-11 08:04:54.279381 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:54.278921) v4 -- 0x55f66d001c00 con 0 > -8038> 2018-07-11 08:04:54.731673 7f491803c700 5 osd.10 509 heartbeat: osd_stat(789 GB used, 142 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,0,0,1]) > -8037> 2018-07-11 08:04:54.731692 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6802/15255 -- osd_ping(ping e509 stamp 2018-07-11 08:04:54.731690) v4 -- 0x55f66e945800 con 0 > -8036> 2018-07-11 08:04:54.731738 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6803/15255 -- osd_ping(ping e509 stamp 2018-07-11 08:04:54.731690) v4 -- 0x55f66e945a00 con 0 > -8035> 2018-07-11 08:04:54.731754 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6806/12677 -- osd_ping(ping e509 stamp 2018-07-11 08:04:54.731690) v4 -- 0x55f66e945c00 con 0 > -8034> 2018-07-11 08:04:54.731769 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6807/12677 -- osd_ping(ping e509 stamp 2018-07-11 08:04:54.731690) v4 -- 0x55f66e945e00 con 0 > -8033> 2018-07-11 08:04:54.731783 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6808/12651 -- osd_ping(ping e509 stamp 2018-07-11 08:04:54.731690) v4 -- 0x55f6743f6000 con 0 > -8032> 2018-07-11 08:04:54.731794 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6810/12651 -- osd_ping(ping e509 stamp 2018-07-11 08:04:54.731690) v4 -- 0x55f6743f6200 con 0 > -8031> 2018-07-11 08:04:54.731806 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6809/12650 -- osd_ping(ping e509 stamp 2018-07-11 08:04:54.731690) v4 -- 0x55f6743f6400 con 0 > -8030> 2018-07-11 08:04:54.731818 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6811/12650 -- osd_ping(ping e509 stamp 2018-07-11 08:04:54.731690) v4 -- 0x55f6743f6600 con 0 > -8029> 2018-07-11 08:04:54.731839 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6810/15287 -- osd_ping(ping e509 stamp 2018-07-11 08:04:54.731690) v4 -- 0x55f6743f6800 con 0 > -8028> 2018-07-11 08:04:54.731850 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6811/15287 -- osd_ping(ping e509 stamp 2018-07-11 08:04:54.731690) v4 -- 0x55f6743f6a00 con 0 > -8027> 2018-07-11 08:04:54.731863 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6806/15263 -- osd_ping(ping e509 stamp 2018-07-11 08:04:54.731690) v4 -- 0x55f6743f6c00 con 0 > -8026> 2018-07-11 08:04:54.731878 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6807/15263 -- osd_ping(ping e509 stamp 2018-07-11 08:04:54.731690) v4 -- 0x55f6743f6e00 con 0 > -8025> 2018-07-11 08:04:54.731891 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.115:6802/13323 -- osd_ping(ping e509 stamp 2018-07-11 08:04:54.731690) v4 -- 0x55f6743f7000 con 0 > -8024> 2018-07-11 08:04:54.731904 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.115:6803/13323 -- osd_ping(ping e509 stamp 2018-07-11 08:04:54.731690) v4 -- 0x55f6743f7200 con 0 > -8023> 2018-07-11 08:04:54.731918 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6806/15885 -- osd_ping(ping e509 stamp 2018-07-11 08:04:54.731690) v4 -- 0x55f6743f7400 con 0 > -8022> 2018-07-11 08:04:54.731931 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6807/15885 -- osd_ping(ping e509 stamp 2018-07-11 08:04:54.731690) v4 -- 0x55f6743f7600 con 0 > -8021> 2018-07-11 08:04:54.731943 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.108:6802/11967 -- osd_ping(ping e509 stamp 2018-07-11 08:04:54.731690) v4 -- 0x55f6743f7800 con 0 > -8020> 2018-07-11 08:04:54.731955 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.108:6803/11967 -- osd_ping(ping e509 stamp 2018-07-11 08:04:54.731690) v4 -- 0x55f6743f7a00 con 0 > -8019> 2018-07-11 08:04:54.731968 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.107:6806/12617 -- osd_ping(ping e509 stamp 2018-07-11 08:04:54.731690) v4 -- 0x55f6743f7c00 con 0 > -8018> 2018-07-11 08:04:54.731980 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.107:6807/12617 -- osd_ping(ping e509 stamp 2018-07-11 08:04:54.731690) v4 -- 0x55f6743f7e00 con 0 > -8017> 2018-07-11 08:04:54.731991 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6802/15919 -- osd_ping(ping e509 stamp 2018-07-11 08:04:54.731690) v4 -- 0x55f653732000 con 0 > -8016> 2018-07-11 08:04:54.732003 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6803/15919 -- osd_ping(ping e509 stamp 2018-07-11 08:04:54.731690) v4 -- 0x55f653732200 con 0 > -8015> 2018-07-11 08:04:54.732156 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6803/15255 conn(0x55f656ad5800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 188 0x55f64cc81e00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:54.731690) v4 > -8014> 2018-07-11 08:04:54.732167 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6802/15255 conn(0x55f656ad4000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 188 0x55f65d024a00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:54.731690) v4 > -8013> 2018-07-11 08:04:54.732177 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.0 10.8.128.113:6803/15255 188 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:54.731690) v4 ==== 2004+0+0 (1833674152 0 0) 0x55f64cc81e00 con 0x55f656ad5800 > -8012> 2018-07-11 08:04:54.732194 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.0 10.8.128.113:6802/15255 188 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:54.731690) v4 ==== 2004+0+0 (1833674152 0 0) 0x55f65d024a00 con 0x55f656ad4000 > -8011> 2018-07-11 08:04:54.732219 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.107:6806/12617 conn(0x55f656ade000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.9 seq 188 0x55f64cc81e00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:54.731690) v4 > -8010> 2018-07-11 08:04:54.732231 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.9 10.8.128.107:6806/12617 188 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:54.731690) v4 ==== 2004+0+0 (1833674152 0 0) 0x55f64cc81e00 con 0x55f656ade000 > -8009> 2018-07-11 08:04:54.732306 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.107:6807/12617 conn(0x55f656adf800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.9 seq 188 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:54.731690) v4 > -8008> 2018-07-11 08:04:54.732302 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6807/12677 conn(0x55f64e3d9800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.1 seq 188 0x55f65d024a00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:54.731690) v4 > -8007> 2018-07-11 08:04:54.732331 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.9 10.8.128.107:6807/12617 188 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:54.731690) v4 ==== 2004+0+0 (1833674152 0 0) 0x55f67d1afe00 con 0x55f656adf800 > -8006> 2018-07-11 08:04:54.732355 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6811/15287 conn(0x55f64e402800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.4 seq 188 0x55f64cc81e00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:54.731690) v4 > -8005> 2018-07-11 08:04:54.732336 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.1 10.8.128.106:6807/12677 188 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:54.731690) v4 ==== 2004+0+0 (1833674152 0 0) 0x55f65d024a00 con 0x55f64e3d9800 > -8004> 2018-07-11 08:04:54.732375 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.4 10.8.128.113:6811/15287 188 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:54.731690) v4 ==== 2004+0+0 (1833674152 0 0) 0x55f64cc81e00 con 0x55f64e402800 > -8003> 2018-07-11 08:04:54.732376 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6806/15263 conn(0x55f64e3ed000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.5 seq 188 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:54.731690) v4 > -8002> 2018-07-11 08:04:54.732394 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.5 10.8.128.113:6806/15263 188 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:54.731690) v4 ==== 2004+0+0 (1833674152 0 0) 0x55f67d1afe00 con 0x55f64e3ed000 > -8001> 2018-07-11 08:04:54.732416 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6810/12651 conn(0x55f64e3eb800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.2 seq 188 0x55f64cc81e00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:54.731690) v4 > -8000> 2018-07-11 08:04:54.732429 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.2 10.8.128.106:6810/12651 188 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:54.731690) v4 ==== 2004+0+0 (1833674152 0 0) 0x55f64cc81e00 con 0x55f64e3eb800 > -7999> 2018-07-11 08:04:54.732428 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6811/12650 conn(0x55f656ad8800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.3 seq 188 0x55f65d024a00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:54.731690) v4 > -7998> 2018-07-11 08:04:54.732438 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6806/12677 conn(0x55f64e3d8000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.1 seq 188 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:54.731690) v4 > -7997> 2018-07-11 08:04:54.732448 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.3 10.8.128.106:6811/12650 188 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:54.731690) v4 ==== 2004+0+0 (1833674152 0 0) 0x55f65d024a00 con 0x55f656ad8800 > -7996> 2018-07-11 08:04:54.732452 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.1 10.8.128.106:6806/12677 188 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:54.731690) v4 ==== 2004+0+0 (1833674152 0 0) 0x55f67d1afe00 con 0x55f64e3d8000 > -7995> 2018-07-11 08:04:54.732461 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.115:6802/13323 conn(0x55f656acd000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.6 seq 188 0x55f64cc81e00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:54.731690) v4 > -7994> 2018-07-11 08:04:54.732473 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.6 10.8.128.115:6802/13323 188 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:54.731690) v4 ==== 2004+0+0 (1833674152 0 0) 0x55f64cc81e00 con 0x55f656acd000 > -7993> 2018-07-11 08:04:54.732497 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6809/12650 conn(0x55f656ad7000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.3 seq 188 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:54.731690) v4 > -7992> 2018-07-11 08:04:54.732507 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6806/15885 conn(0x55f64e3fe000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.7 seq 188 0x55f64cc81e00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:54.731690) v4 > -7991> 2018-07-11 08:04:54.732497 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6810/15287 conn(0x55f64e401000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.4 seq 188 0x55f65d024a00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:54.731690) v4 > -7990> 2018-07-11 08:04:54.732516 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.7 10.8.128.114:6806/15885 188 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:54.731690) v4 ==== 2004+0+0 (1833674152 0 0) 0x55f64cc81e00 con 0x55f64e3fe000 > -7989> 2018-07-11 08:04:54.732517 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.3 10.8.128.106:6809/12650 188 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:54.731690) v4 ==== 2004+0+0 (1833674152 0 0) 0x55f67d1afe00 con 0x55f656ad7000 > -7988> 2018-07-11 08:04:54.732518 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.4 10.8.128.113:6810/15287 188 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:54.731690) v4 ==== 2004+0+0 (1833674152 0 0) 0x55f65d024a00 con 0x55f64e401000 > -7987> 2018-07-11 08:04:54.732549 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6802/15919 conn(0x55f64e3db000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.11 seq 188 0x55f64cc81e00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:54.731690) v4 > -7986> 2018-07-11 08:04:54.732561 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.11 10.8.128.114:6802/15919 188 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:54.731690) v4 ==== 2004+0+0 (1833674152 0 0) 0x55f64cc81e00 con 0x55f64e3db000 > -7985> 2018-07-11 08:04:54.732553 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6808/12651 conn(0x55f64e3ea000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.2 seq 188 0x55f65d024a00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:54.731690) v4 > -7984> 2018-07-11 08:04:54.732572 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.2 10.8.128.106:6808/12651 188 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:54.731690) v4 ==== 2004+0+0 (1833674152 0 0) 0x55f65d024a00 con 0x55f64e3ea000 > -7983> 2018-07-11 08:04:54.732572 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.115:6803/13323 conn(0x55f656ace800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.6 seq 188 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:54.731690) v4 > -7982> 2018-07-11 08:04:54.732594 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.6 10.8.128.115:6803/13323 188 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:54.731690) v4 ==== 2004+0+0 (1833674152 0 0) 0x55f67d1afe00 con 0x55f656ace800 > -7981> 2018-07-11 08:04:54.732630 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6807/15263 conn(0x55f64e3ee800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.5 seq 188 0x55f65d024a00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:54.731690) v4 > -7980> 2018-07-11 08:04:54.732637 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6807/15885 conn(0x55f64e3ff800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.7 seq 188 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:54.731690) v4 > -7979> 2018-07-11 08:04:54.732652 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.5 10.8.128.113:6807/15263 188 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:54.731690) v4 ==== 2004+0+0 (1833674152 0 0) 0x55f65d024a00 con 0x55f64e3ee800 > -7978> 2018-07-11 08:04:54.732656 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.7 10.8.128.114:6807/15885 188 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:54.731690) v4 ==== 2004+0+0 (1833674152 0 0) 0x55f67d1afe00 con 0x55f64e3ff800 > -7977> 2018-07-11 08:04:54.732707 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.108:6802/11967 conn(0x55f656aca000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.8 seq 188 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:54.731690) v4 > -7976> 2018-07-11 08:04:54.732707 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.108:6803/11967 conn(0x55f656acb800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.8 seq 188 0x55f65d024a00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:54.731690) v4 > -7975> 2018-07-11 08:04:54.732728 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.8 10.8.128.108:6803/11967 188 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:54.731690) v4 ==== 2004+0+0 (1833674152 0 0) 0x55f65d024a00 con 0x55f656acb800 > -7974> 2018-07-11 08:04:54.732728 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.8 10.8.128.108:6802/11967 188 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:54.731690) v4 ==== 2004+0+0 (1833674152 0 0) 0x55f67d1afe00 con 0x55f656aca000 > -7973> 2018-07-11 08:04:54.732782 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6803/15919 conn(0x55f64e3dc800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.11 seq 188 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:54.731690) v4 > -7972> 2018-07-11 08:04:54.732797 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.11 10.8.128.114:6803/15919 188 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:54.731690) v4 ==== 2004+0+0 (1833674152 0 0) 0x55f67d1afe00 con 0x55f64e3dc800 > -7971> 2018-07-11 08:04:54.826221 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f67bf0e240 con 0 > -7970> 2018-07-11 08:04:54.828022 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 191 0x55f676583680 replica scrub(pg: 6.15,from:0'0,to:430'106297,epoch:509/504,start:6:a81ba62e:::benchmark_data_magna108.ceph.redhat.com_2478_object138007:0,end:6:a81cba6e:::benchmark_data_magna108_35617_object642836:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -7969> 2018-07-11 08:04:54.828048 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 191 ==== replica scrub(pg: 6.15,from:0'0,to:430'106297,epoch:509/504,start:6:a81ba62e:::benchmark_data_magna108.ceph.redhat.com_2478_object138007:0,end:6:a81cba6e:::benchmark_data_magna108_35617_object642836:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (3528467167 0 0) 0x55f676583680 con 0x55f66315e800 > -7968> 2018-07-11 08:04:54.950922 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0d000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 193 0x55f65d024a00 osd_ping(ping e509 stamp 2018-07-11 08:04:56.194940) v4 > -7967> 2018-07-11 08:04:54.950922 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0e800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 193 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:04:56.194940) v4 > -7966> 2018-07-11 08:04:54.950954 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.5 10.8.128.113:0/15263 193 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:56.194940) v4 ==== 2004+0+0 (3755837171 0 0) 0x55f67d1afe00 con 0x55f663f0e800 > -7965> 2018-07-11 08:04:54.950952 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.5 10.8.128.113:0/15263 193 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:56.194940) v4 ==== 2004+0+0 (3755837171 0 0) 0x55f65d024a00 con 0x55f663f0d000 > -7964> 2018-07-11 08:04:54.950973 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:56.194940) v4 -- 0x55f66e945c00 con 0 > -7963> 2018-07-11 08:04:54.951003 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:56.194940) v4 -- 0x55f66e945800 con 0 > -7962> 2018-07-11 08:04:54.960417 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 411 0x55f64cc81e00 MOSDScrubReserve(11.3s5 REQUEST e509) v1 > -7961> 2018-07-11 08:04:54.960437 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 411 ==== MOSDScrubReserve(11.3s5 REQUEST e509) v1 ==== 43+0+0 (433425070 0 0) 0x55f64cc81e00 con 0x55f668a29000 > -7960> 2018-07-11 08:04:54.960502 7f491a040700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e509) v1 -- 0x55f65d025400 con 0 > -7959> 2018-07-11 08:04:54.960838 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 412 0x55f6743f6a00 MOSDScrubReserve(11.3s5 RELEASE e509) v1 > -7958> 2018-07-11 08:04:54.960879 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 412 ==== MOSDScrubReserve(11.3s5 RELEASE e509) v1 ==== 43+0+0 (3603460268 0 0) 0x55f6743f6a00 con 0x55f668a29000 > -7957> 2018-07-11 08:04:55.025104 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12650 conn(0x55f663ed9000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.3 seq 187 0x55f66b1cd800 osd_ping(ping e509 stamp 2018-07-11 08:04:55.081491) v4 > -7956> 2018-07-11 08:04:55.025110 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12650 conn(0x55f663eda800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.3 seq 187 0x55f66e945e00 osd_ping(ping e509 stamp 2018-07-11 08:04:55.081491) v4 > -7955> 2018-07-11 08:04:55.025130 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.3 10.8.128.106:0/12650 187 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:55.081491) v4 ==== 2004+0+0 (1900962272 0 0) 0x55f66b1cd800 con 0x55f663ed9000 > -7954> 2018-07-11 08:04:55.025132 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.3 10.8.128.106:0/12650 187 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:55.081491) v4 ==== 2004+0+0 (1900962272 0 0) 0x55f66e945e00 con 0x55f663eda800 > -7953> 2018-07-11 08:04:55.025152 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:55.081491) v4 -- 0x55f6743f6200 con 0 > -7952> 2018-07-11 08:04:55.025185 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:55.081491) v4 -- 0x55f66e945800 con 0 > -7951> 2018-07-11 08:04:55.071846 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15255 conn(0x55f663ec1000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.0 seq 182 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:04:56.315887) v4 > -7950> 2018-07-11 08:04:55.071854 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15255 conn(0x55f663ec2800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.0 seq 182 0x55f65d025400 osd_ping(ping e509 stamp 2018-07-11 08:04:56.315887) v4 > -7949> 2018-07-11 08:04:55.071867 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.0 10.8.128.113:0/15255 182 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:56.315887) v4 ==== 2004+0+0 (544295400 0 0) 0x55f64e20fc00 con 0x55f663ec1000 > -7948> 2018-07-11 08:04:55.071877 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:56.315887) v4 -- 0x55f66e944a00 con 0 > -7947> 2018-07-11 08:04:55.071873 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.0 10.8.128.113:0/15255 182 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:56.315887) v4 ==== 2004+0+0 (544295400 0 0) 0x55f65d025400 con 0x55f663ec2800 > -7946> 2018-07-11 08:04:55.071904 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:56.315887) v4 -- 0x55f66e945a00 con 0 > -7945> 2018-07-11 08:04:55.148132 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12651 conn(0x55f66321a800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25 cs=1 l=1). rx osd.2 seq 199 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:04:55.204370) v4 > -7944> 2018-07-11 08:04:55.148157 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.2 10.8.128.106:0/12651 199 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:55.204370) v4 ==== 2004+0+0 (2782466127 0 0) 0x55f67d1afe00 con 0x55f66321a800 > -7943> 2018-07-11 08:04:55.148146 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12651 conn(0x55f663219000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25 cs=1 l=1). rx osd.2 seq 199 0x55f65d024a00 osd_ping(ping e509 stamp 2018-07-11 08:04:55.204370) v4 > -7942> 2018-07-11 08:04:55.148168 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:55.204370) v4 -- 0x55f66e944600 con 0 > -7941> 2018-07-11 08:04:55.148175 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.2 10.8.128.106:0/12651 199 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:55.204370) v4 ==== 2004+0+0 (2782466127 0 0) 0x55f65d024a00 con 0x55f663219000 > -7940> 2018-07-11 08:04:55.148192 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:55.204370) v4 -- 0x55f66d001c00 con 0 > -7939> 2018-07-11 08:04:55.271552 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed7800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 189 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:04:55.327859) v4 > -7938> 2018-07-11 08:04:55.271574 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.1 10.8.128.106:0/12677 189 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:55.327859) v4 ==== 2004+0+0 (3031592022 0 0) 0x55f64e20fc00 con 0x55f663ed7800 > -7937> 2018-07-11 08:04:55.271565 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed6000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 189 0x55f66e945e00 osd_ping(ping e509 stamp 2018-07-11 08:04:55.327859) v4 > -7936> 2018-07-11 08:04:55.271583 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:55.327859) v4 -- 0x55f66e944000 con 0 > -7935> 2018-07-11 08:04:55.271594 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.1 10.8.128.106:0/12677 189 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:55.327859) v4 ==== 2004+0+0 (3031592022 0 0) 0x55f66e945e00 con 0x55f663ed6000 > -7934> 2018-07-11 08:04:55.271612 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:55.327859) v4 -- 0x55f66d001a00 con 0 > -7933> 2018-07-11 08:04:55.525971 7f491d046700 1 -- 10.8.128.107:6800/14173 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55f663502580 con 0 > -7932> 2018-07-11 08:04:55.526055 7f491d046700 1 -- 10.8.128.107:6800/14173 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55f663150000 con 0 > -7931> 2018-07-11 08:04:55.658392 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.114:6805/15885 conn(0x55f66315a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.7 seq 2527 0x55f65d024a00 MOSDScrubReserve(5.4 REQUEST e509) v1 > -7930> 2018-07-11 08:04:55.658413 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.7 10.8.128.114:6805/15885 2527 ==== MOSDScrubReserve(5.4 REQUEST e509) v1 ==== 43+0+0 (3003630469 0 0) 0x55f65d024a00 con 0x55f66315a000 > -7929> 2018-07-11 08:04:55.658463 7f491983f700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.114:6805/15885 -- MOSDScrubReserve(5.4 REJECT e509) v1 -- 0x55f6548fbe00 con 0 > -7928> 2018-07-11 08:04:55.658810 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.114:6805/15885 conn(0x55f66315a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.7 seq 2528 0x55f66d001400 MOSDScrubReserve(5.4 RELEASE e509) v1 > -7927> 2018-07-11 08:04:55.658825 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.7 10.8.128.114:6805/15885 2528 ==== MOSDScrubReserve(5.4 RELEASE e509) v1 ==== 43+0+0 (2082093447 0 0) 0x55f66d001400 con 0x55f66315a000 > -7926> 2018-07-11 08:04:55.705831 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.115:0/13323 conn(0x55f663ebf800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.6 seq 179 0x55f66e945e00 osd_ping(ping e509 stamp 2018-07-11 08:04:56.058156) v4 > -7925> 2018-07-11 08:04:55.705840 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.115:0/13323 conn(0x55f663ebe000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.6 seq 179 0x55f66b1cd800 osd_ping(ping e509 stamp 2018-07-11 08:04:56.058156) v4 > -7924> 2018-07-11 08:04:55.705856 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.6 10.8.128.115:0/13323 179 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:56.058156) v4 ==== 2004+0+0 (860239902 0 0) 0x55f66e945e00 con 0x55f663ebf800 > -7923> 2018-07-11 08:04:55.705868 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:56.058156) v4 -- 0x55f66d001200 con 0 > -7922> 2018-07-11 08:04:55.705865 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.6 10.8.128.115:0/13323 179 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:56.058156) v4 ==== 2004+0+0 (860239902 0 0) 0x55f66b1cd800 con 0x55f663ebe000 > -7921> 2018-07-11 08:04:55.705893 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:56.058156) v4 -- 0x55f66e945a00 con 0 > -7920> 2018-07-11 08:04:55.832193 7f491803c700 5 osd.10 509 heartbeat: osd_stat(789 GB used, 142 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,0,0,1]) > -7919> 2018-07-11 08:04:55.832207 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6802/15255 -- osd_ping(ping e509 stamp 2018-07-11 08:04:55.832205) v4 -- 0x55f653732400 con 0 > -7918> 2018-07-11 08:04:55.832242 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6803/15255 -- osd_ping(ping e509 stamp 2018-07-11 08:04:55.832205) v4 -- 0x55f653732600 con 0 > -7917> 2018-07-11 08:04:55.832270 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6806/12677 -- osd_ping(ping e509 stamp 2018-07-11 08:04:55.832205) v4 -- 0x55f653732800 con 0 > -7916> 2018-07-11 08:04:55.832298 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6807/12677 -- osd_ping(ping e509 stamp 2018-07-11 08:04:55.832205) v4 -- 0x55f653732a00 con 0 > -7915> 2018-07-11 08:04:55.832310 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6808/12651 -- osd_ping(ping e509 stamp 2018-07-11 08:04:55.832205) v4 -- 0x55f653732c00 con 0 > -7914> 2018-07-11 08:04:55.832322 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6810/12651 -- osd_ping(ping e509 stamp 2018-07-11 08:04:55.832205) v4 -- 0x55f653732e00 con 0 > -7913> 2018-07-11 08:04:55.832334 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6809/12650 -- osd_ping(ping e509 stamp 2018-07-11 08:04:55.832205) v4 -- 0x55f653733000 con 0 > -7912> 2018-07-11 08:04:55.832347 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6811/12650 -- osd_ping(ping e509 stamp 2018-07-11 08:04:55.832205) v4 -- 0x55f653733200 con 0 > -7911> 2018-07-11 08:04:55.832357 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6810/15287 -- osd_ping(ping e509 stamp 2018-07-11 08:04:55.832205) v4 -- 0x55f653733400 con 0 > -7910> 2018-07-11 08:04:55.832367 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6811/15287 -- osd_ping(ping e509 stamp 2018-07-11 08:04:55.832205) v4 -- 0x55f653733600 con 0 > -7909> 2018-07-11 08:04:55.832388 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6806/15263 -- osd_ping(ping e509 stamp 2018-07-11 08:04:55.832205) v4 -- 0x55f653733800 con 0 > -7908> 2018-07-11 08:04:55.832404 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6807/15263 -- osd_ping(ping e509 stamp 2018-07-11 08:04:55.832205) v4 -- 0x55f653733a00 con 0 > -7907> 2018-07-11 08:04:55.832419 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.115:6802/13323 -- osd_ping(ping e509 stamp 2018-07-11 08:04:55.832205) v4 -- 0x55f653733c00 con 0 > -7906> 2018-07-11 08:04:55.832435 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.115:6803/13323 -- osd_ping(ping e509 stamp 2018-07-11 08:04:55.832205) v4 -- 0x55f653733e00 con 0 > -7905> 2018-07-11 08:04:55.832450 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6806/15885 -- osd_ping(ping e509 stamp 2018-07-11 08:04:55.832205) v4 -- 0x55f669612000 con 0 > -7904> 2018-07-11 08:04:55.832462 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6807/15885 -- osd_ping(ping e509 stamp 2018-07-11 08:04:55.832205) v4 -- 0x55f669612200 con 0 > -7903> 2018-07-11 08:04:55.832474 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.108:6802/11967 -- osd_ping(ping e509 stamp 2018-07-11 08:04:55.832205) v4 -- 0x55f669612400 con 0 > -7902> 2018-07-11 08:04:55.832486 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.108:6803/11967 -- osd_ping(ping e509 stamp 2018-07-11 08:04:55.832205) v4 -- 0x55f669612600 con 0 > -7901> 2018-07-11 08:04:55.832499 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.107:6806/12617 -- osd_ping(ping e509 stamp 2018-07-11 08:04:55.832205) v4 -- 0x55f669612800 con 0 > -7900> 2018-07-11 08:04:55.832511 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.107:6807/12617 -- osd_ping(ping e509 stamp 2018-07-11 08:04:55.832205) v4 -- 0x55f669612a00 con 0 > -7899> 2018-07-11 08:04:55.832524 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6802/15919 -- osd_ping(ping e509 stamp 2018-07-11 08:04:55.832205) v4 -- 0x55f669612c00 con 0 > -7898> 2018-07-11 08:04:55.832535 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6803/15919 -- osd_ping(ping e509 stamp 2018-07-11 08:04:55.832205) v4 -- 0x55f669612e00 con 0 > -7897> 2018-07-11 08:04:55.832613 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6803/15255 conn(0x55f656ad5800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 189 0x55f65d025400 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:55.832205) v4 > -7896> 2018-07-11 08:04:55.832636 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.0 10.8.128.113:6803/15255 189 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:55.832205) v4 ==== 2004+0+0 (2300908382 0 0) 0x55f65d025400 con 0x55f656ad5800 > -7895> 2018-07-11 08:04:55.832701 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6802/15255 conn(0x55f656ad4000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 189 0x55f6548fbe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:55.832205) v4 > -7894> 2018-07-11 08:04:55.832711 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.107:6806/12617 conn(0x55f656ade000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.9 seq 189 0x55f66b1cd800 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:55.832205) v4 > -7893> 2018-07-11 08:04:55.832721 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.0 10.8.128.113:6802/15255 189 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:55.832205) v4 ==== 2004+0+0 (2300908382 0 0) 0x55f6548fbe00 con 0x55f656ad4000 > -7892> 2018-07-11 08:04:55.832725 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.9 10.8.128.107:6806/12617 189 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:55.832205) v4 ==== 2004+0+0 (2300908382 0 0) 0x55f66b1cd800 con 0x55f656ade000 > -7891> 2018-07-11 08:04:55.832748 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6809/12650 conn(0x55f656ad7000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.3 seq 189 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:55.832205) v4 > -7890> 2018-07-11 08:04:55.832769 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.3 10.8.128.106:6809/12650 189 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:55.832205) v4 ==== 2004+0+0 (2300908382 0 0) 0x55f67d1afe00 con 0x55f656ad7000 > -7889> 2018-07-11 08:04:55.832799 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.107:6807/12617 conn(0x55f656adf800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.9 seq 189 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:55.832205) v4 > -7888> 2018-07-11 08:04:55.832814 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6811/15287 conn(0x55f64e402800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.4 seq 189 0x55f66b1cd800 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:55.832205) v4 > -7887> 2018-07-11 08:04:55.832830 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.9 10.8.128.107:6807/12617 189 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:55.832205) v4 ==== 2004+0+0 (2300908382 0 0) 0x55f67d1afe00 con 0x55f656adf800 > -7886> 2018-07-11 08:04:55.832831 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.4 10.8.128.113:6811/15287 189 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:55.832205) v4 ==== 2004+0+0 (2300908382 0 0) 0x55f66b1cd800 con 0x55f64e402800 > -7885> 2018-07-11 08:04:55.832852 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6807/12677 conn(0x55f64e3d9800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.1 seq 189 0x55f6548fbe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:55.832205) v4 > -7884> 2018-07-11 08:04:55.832869 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.1 10.8.128.106:6807/12677 189 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:55.832205) v4 ==== 2004+0+0 (2300908382 0 0) 0x55f6548fbe00 con 0x55f64e3d9800 > -7883> 2018-07-11 08:04:55.832893 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6810/12651 conn(0x55f64e3eb800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.2 seq 189 0x55f66b1cd800 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:55.832205) v4 > -7882> 2018-07-11 08:04:55.832917 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6811/12650 conn(0x55f656ad8800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.3 seq 189 0x55f6548fbe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:55.832205) v4 > -7881> 2018-07-11 08:04:55.832909 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.2 10.8.128.106:6810/12651 189 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:55.832205) v4 ==== 2004+0+0 (2300908382 0 0) 0x55f66b1cd800 con 0x55f64e3eb800 > -7880> 2018-07-11 08:04:55.832931 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.3 10.8.128.106:6811/12650 189 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:55.832205) v4 ==== 2004+0+0 (2300908382 0 0) 0x55f6548fbe00 con 0x55f656ad8800 > -7879> 2018-07-11 08:04:55.832965 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.115:6802/13323 conn(0x55f656acd000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.6 seq 189 0x55f66b1cd800 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:55.832205) v4 > -7878> 2018-07-11 08:04:55.832978 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.6 10.8.128.115:6802/13323 189 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:55.832205) v4 ==== 2004+0+0 (2300908382 0 0) 0x55f66b1cd800 con 0x55f656acd000 > -7877> 2018-07-11 08:04:55.832978 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6810/15287 conn(0x55f64e401000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.4 seq 189 0x55f6548fbe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:55.832205) v4 > -7876> 2018-07-11 08:04:55.832991 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.4 10.8.128.113:6810/15287 189 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:55.832205) v4 ==== 2004+0+0 (2300908382 0 0) 0x55f6548fbe00 con 0x55f64e401000 > -7875> 2018-07-11 08:04:55.832987 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6806/12677 conn(0x55f64e3d8000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.1 seq 189 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:55.832205) v4 > -7874> 2018-07-11 08:04:55.833006 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.1 10.8.128.106:6806/12677 189 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:55.832205) v4 ==== 2004+0+0 (2300908382 0 0) 0x55f67d1afe00 con 0x55f64e3d8000 > -7873> 2018-07-11 08:04:55.833021 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6807/15263 conn(0x55f64e3ee800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.5 seq 189 0x55f6548fbe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:55.832205) v4 > -7872> 2018-07-11 08:04:55.833046 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.115:6803/13323 conn(0x55f656ace800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.6 seq 189 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:55.832205) v4 > -7871> 2018-07-11 08:04:55.833053 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.5 10.8.128.113:6807/15263 189 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:55.832205) v4 ==== 2004+0+0 (2300908382 0 0) 0x55f6548fbe00 con 0x55f64e3ee800 > -7870> 2018-07-11 08:04:55.833059 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.6 10.8.128.115:6803/13323 189 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:55.832205) v4 ==== 2004+0+0 (2300908382 0 0) 0x55f67d1afe00 con 0x55f656ace800 > -7869> 2018-07-11 08:04:55.833067 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6802/15919 conn(0x55f64e3db000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.11 seq 189 0x55f66b1cd800 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:55.832205) v4 > -7868> 2018-07-11 08:04:55.833083 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.11 10.8.128.114:6802/15919 189 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:55.832205) v4 ==== 2004+0+0 (2300908382 0 0) 0x55f66b1cd800 con 0x55f64e3db000 > -7867> 2018-07-11 08:04:55.833089 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6808/12651 conn(0x55f64e3ea000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.2 seq 189 0x55f6548fbe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:55.832205) v4 > -7866> 2018-07-11 08:04:55.833090 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6807/15885 conn(0x55f64e3ff800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.7 seq 189 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:55.832205) v4 > -7865> 2018-07-11 08:04:55.833104 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.2 10.8.128.106:6808/12651 189 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:55.832205) v4 ==== 2004+0+0 (2300908382 0 0) 0x55f6548fbe00 con 0x55f64e3ea000 > -7864> 2018-07-11 08:04:55.833111 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.7 10.8.128.114:6807/15885 189 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:55.832205) v4 ==== 2004+0+0 (2300908382 0 0) 0x55f67d1afe00 con 0x55f64e3ff800 > -7863> 2018-07-11 08:04:55.833137 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.108:6803/11967 conn(0x55f656acb800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.8 seq 189 0x55f6548fbe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:55.832205) v4 > -7862> 2018-07-11 08:04:55.833141 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6803/15919 conn(0x55f64e3dc800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.11 seq 189 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:55.832205) v4 > -7861> 2018-07-11 08:04:55.833134 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6806/15885 conn(0x55f64e3fe000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.7 seq 189 0x55f66b1cd800 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:55.832205) v4 > -7860> 2018-07-11 08:04:55.833151 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.11 10.8.128.114:6803/15919 189 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:55.832205) v4 ==== 2004+0+0 (2300908382 0 0) 0x55f67d1afe00 con 0x55f64e3dc800 > -7859> 2018-07-11 08:04:55.833152 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.8 10.8.128.108:6803/11967 189 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:55.832205) v4 ==== 2004+0+0 (2300908382 0 0) 0x55f6548fbe00 con 0x55f656acb800 > -7858> 2018-07-11 08:04:55.833156 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.7 10.8.128.114:6806/15885 189 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:55.832205) v4 ==== 2004+0+0 (2300908382 0 0) 0x55f66b1cd800 con 0x55f64e3fe000 > -7857> 2018-07-11 08:04:55.833184 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6806/15263 conn(0x55f64e3ed000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.5 seq 189 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:55.832205) v4 > -7856> 2018-07-11 08:04:55.833195 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.5 10.8.128.113:6806/15263 189 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:55.832205) v4 ==== 2004+0+0 (2300908382 0 0) 0x55f67d1afe00 con 0x55f64e3ed000 > -7855> 2018-07-11 08:04:55.833224 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.108:6802/11967 conn(0x55f656aca000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.8 seq 189 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:55.832205) v4 > -7854> 2018-07-11 08:04:55.833236 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.8 10.8.128.108:6802/11967 189 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:55.832205) v4 ==== 2004+0+0 (2300908382 0 0) 0x55f67d1afe00 con 0x55f656aca000 > -7853> 2018-07-11 08:04:55.899159 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0a000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 185 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:04:56.534488) v4 > -7852> 2018-07-11 08:04:55.899160 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0b800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 185 0x55f66b1cd800 osd_ping(ping e509 stamp 2018-07-11 08:04:56.534488) v4 > -7851> 2018-07-11 08:04:55.899186 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.11 10.8.128.114:0/15919 185 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:56.534488) v4 ==== 2004+0+0 (2246165656 0 0) 0x55f67d1afe00 con 0x55f663f0a000 > -7850> 2018-07-11 08:04:55.899198 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:56.534488) v4 -- 0x55f669612200 con 0 > -7849> 2018-07-11 08:04:55.899186 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.11 10.8.128.114:0/15919 185 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:56.534488) v4 ==== 2004+0+0 (2246165656 0 0) 0x55f66b1cd800 con 0x55f663f0b800 > -7848> 2018-07-11 08:04:55.899222 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:56.534488) v4 -- 0x55f653733600 con 0 > -7847> 2018-07-11 08:04:56.125550 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12650 conn(0x55f663ed9000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.3 seq 188 0x55f65d025400 osd_ping(ping e509 stamp 2018-07-11 08:04:56.181939) v4 > -7846> 2018-07-11 08:04:56.125571 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.3 10.8.128.106:0/12650 188 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:56.181939) v4 ==== 2004+0+0 (3303092999 0 0) 0x55f65d025400 con 0x55f663ed9000 > -7845> 2018-07-11 08:04:56.125567 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12650 conn(0x55f663eda800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.3 seq 188 0x55f6548fbe00 osd_ping(ping e509 stamp 2018-07-11 08:04:56.181939) v4 > -7844> 2018-07-11 08:04:56.125581 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:56.181939) v4 -- 0x55f653733600 con 0 > -7843> 2018-07-11 08:04:56.125593 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.3 10.8.128.106:0/12650 188 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:56.181939) v4 ==== 2004+0+0 (3303092999 0 0) 0x55f6548fbe00 con 0x55f663eda800 > -7842> 2018-07-11 08:04:56.125611 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:56.181939) v4 -- 0x55f653732c00 con 0 > -7841> 2018-07-11 08:04:56.198747 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15287 conn(0x55f66dce2800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.4 seq 196 0x55f66e945e00 osd_ping(ping e509 stamp 2018-07-11 08:04:57.442714) v4 > -7840> 2018-07-11 08:04:56.198768 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.4 10.8.128.113:0/15287 196 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:57.442714) v4 ==== 2004+0+0 (804606018 0 0) 0x55f66e945e00 con 0x55f66dce2800 > -7839> 2018-07-11 08:04:56.198777 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:57.442714) v4 -- 0x55f653732c00 con 0 > -7838> 2018-07-11 08:04:56.198916 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15287 conn(0x55f66dce1000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.4 seq 196 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:04:57.442714) v4 > -7837> 2018-07-11 08:04:56.198938 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.4 10.8.128.113:0/15287 196 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:57.442714) v4 ==== 2004+0+0 (804606018 0 0) 0x55f64e20fc00 con 0x55f66dce1000 > -7836> 2018-07-11 08:04:56.198949 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:57.442714) v4 -- 0x55f669612200 con 0 > -7835> 2018-07-11 08:04:56.394558 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.108:0/11967 conn(0x55f663f6f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=1). rx osd.8 seq 182 0x55f66b1cd800 osd_ping(ping e509 stamp 2018-07-11 08:04:56.154226) v4 > -7834> 2018-07-11 08:04:56.394562 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.108:0/11967 conn(0x55f663f70800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=1). rx osd.8 seq 182 0x55f6548fbe00 osd_ping(ping e509 stamp 2018-07-11 08:04:56.154226) v4 > -7833> 2018-07-11 08:04:56.394584 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.8 10.8.128.108:0/11967 182 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:56.154226) v4 ==== 2004+0+0 (3561944414 0 0) 0x55f66b1cd800 con 0x55f663f6f000 > -7832> 2018-07-11 08:04:56.394588 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.8 10.8.128.108:0/11967 182 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:56.154226) v4 ==== 2004+0+0 (3561944414 0 0) 0x55f6548fbe00 con 0x55f663f70800 > -7831> 2018-07-11 08:04:56.394594 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:56.154226) v4 -- 0x55f653733600 con 0 > -7830> 2018-07-11 08:04:56.394615 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:56.154226) v4 -- 0x55f653732c00 con 0 > -7829> 2018-07-11 08:04:56.443258 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 29] Generated table #37034: 2638241 keys, 74546335 bytes > -7828> 2018-07-11 08:04:56.443303 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296296443276, "cf_name": "default", "job": 29, "event": "table_file_creation", "file_number": 37034, "file_size": 74546335, "table_properties": {"data_size": 67112182, "index_size": 689980, "filter_size": 6743183, "raw_key_size": 76215861, "raw_average_key_size": 28, "raw_value_size": 31365764, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} > -7827> 2018-07-11 08:04:56.538959 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f65b22cfc0 con 0 > -7826> 2018-07-11 08:04:56.540651 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 192 0x55f676583900 replica scrub(pg: 6.15,from:0'0,to:430'106219,epoch:509/504,start:6:a81cba6e:::benchmark_data_magna108_35617_object642836:0,end:6:a81dca79:::benchmark_data_magna108_35617_object518342:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -7825> 2018-07-11 08:04:56.540683 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 192 ==== replica scrub(pg: 6.15,from:0'0,to:430'106219,epoch:509/504,start:6:a81cba6e:::benchmark_data_magna108_35617_object642836:0,end:6:a81dca79:::benchmark_data_magna108_35617_object518342:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (229481115 0 0) 0x55f676583900 con 0x55f66315e800 > -7824> 2018-07-11 08:04:57.511521 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f65b5bbb00 con 0 > -7823> 2018-07-11 08:04:57.513092 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 193 0x55f676583b80 replica scrub(pg: 6.15,from:0'0,to:430'106366,epoch:509/504,start:6:a81dca79:::benchmark_data_magna108_35617_object518342:0,end:6:a81ec982:::benchmark_data_magna108_35617_object657483:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -7822> 2018-07-11 08:04:57.513122 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 193 ==== replica scrub(pg: 6.15,from:0'0,to:430'106366,epoch:509/504,start:6:a81dca79:::benchmark_data_magna108_35617_object518342:0,end:6:a81ec982:::benchmark_data_magna108_35617_object657483:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (896014852 0 0) 0x55f676583b80 con 0x55f66315e800 > -7821> 2018-07-11 08:04:57.571997 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed6000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 190 0x55f66e945e00 osd_ping(ping e509 stamp 2018-07-11 08:04:57.628398) v4 > -7820> 2018-07-11 08:04:57.572024 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.1 10.8.128.106:0/12677 190 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:57.628398) v4 ==== 2004+0+0 (1999660021 0 0) 0x55f66e945e00 con 0x55f663ed6000 > -7819> 2018-07-11 08:04:57.572031 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed7800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 190 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:04:57.628398) v4 > -7818> 2018-07-11 08:04:57.572041 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:57.628398) v4 -- 0x55f653732c00 con 0 > -7817> 2018-07-11 08:04:57.572055 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.1 10.8.128.106:0/12677 190 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:57.628398) v4 ==== 2004+0+0 (1999660021 0 0) 0x55f67d1afe00 con 0x55f663ed7800 > -7816> 2018-07-11 08:04:57.572076 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:57.628398) v4 -- 0x55f653733e00 con 0 > -7815> 2018-07-11 08:04:58.127622 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15885 conn(0x55f663217800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 190 0x55f65d025400 osd_ping(ping e509 stamp 2018-07-11 08:04:58.762951) v4 > -7814> 2018-07-11 08:04:58.127651 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.7 10.8.128.114:0/15885 190 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:58.762951) v4 ==== 2004+0+0 (3821523933 0 0) 0x55f65d025400 con 0x55f663217800 > -7813> 2018-07-11 08:04:58.127651 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15885 conn(0x55f663216000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 190 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:04:58.762951) v4 > -7812> 2018-07-11 08:04:58.127670 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.7 10.8.128.114:0/15885 190 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:58.762951) v4 ==== 2004+0+0 (3821523933 0 0) 0x55f64e20fc00 con 0x55f663216000 > -7811> 2018-07-11 08:04:58.127669 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:58.762951) v4 -- 0x55f653732e00 con 0 > -7810> 2018-07-11 08:04:58.127713 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:58.762951) v4 -- 0x55f653733800 con 0 > -7809> 2018-07-11 08:04:58.279525 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f65696cb40 con 0 > -7808> 2018-07-11 08:04:58.280972 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 194 0x55f65a3f6000 replica scrub(pg: 6.15,from:0'0,to:430'106720,epoch:509/504,start:6:a81ec982:::benchmark_data_magna108_35617_object657483:0,end:6:a81f8afc:::benchmark_data_magna108.ceph.redhat.com_2478_object241465:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -7807> 2018-07-11 08:04:58.281003 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 194 ==== replica scrub(pg: 6.15,from:0'0,to:430'106720,epoch:509/504,start:6:a81ec982:::benchmark_data_magna108_35617_object657483:0,end:6:a81f8afc:::benchmark_data_magna108.ceph.redhat.com_2478_object241465:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (637549461 0 0) 0x55f65a3f6000 con 0x55f66315e800 > -7806> 2018-07-11 08:04:58.627987 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15885 conn(0x55f663217800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 191 0x55f66b1cd800 osd_ping(ping e509 stamp 2018-07-11 08:04:59.263377) v4 > -7805> 2018-07-11 08:04:58.627982 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15885 conn(0x55f663216000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 191 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:04:59.263377) v4 > -7804> 2018-07-11 08:04:58.628023 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.7 10.8.128.114:0/15885 191 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:59.263377) v4 ==== 2004+0+0 (989893556 0 0) 0x55f66b1cd800 con 0x55f663217800 > -7803> 2018-07-11 08:04:58.628026 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.7 10.8.128.114:0/15885 191 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:59.263377) v4 ==== 2004+0+0 (989893556 0 0) 0x55f67d1afe00 con 0x55f663216000 > -7802> 2018-07-11 08:04:58.628043 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:59.263377) v4 -- 0x55f653732e00 con 0 > -7801> 2018-07-11 08:04:58.628073 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:59.263377) v4 -- 0x55f653733800 con 0 > -7800> 2018-07-11 08:04:58.648572 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12651 conn(0x55f663219000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25 cs=1 l=1). rx osd.2 seq 200 0x55f6548fbe00 osd_ping(ping e509 stamp 2018-07-11 08:04:58.704895) v4 > -7799> 2018-07-11 08:04:58.648598 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.2 10.8.128.106:0/12651 200 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:58.704895) v4 ==== 2004+0+0 (93657809 0 0) 0x55f6548fbe00 con 0x55f663219000 > -7798> 2018-07-11 08:04:58.648608 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:58.704895) v4 -- 0x55f653732a00 con 0 > -7797> 2018-07-11 08:04:58.648650 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12651 conn(0x55f66321a800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25 cs=1 l=1). rx osd.2 seq 200 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:04:58.704895) v4 > -7796> 2018-07-11 08:04:58.648670 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.2 10.8.128.106:0/12651 200 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:58.704895) v4 ==== 2004+0+0 (93657809 0 0) 0x55f64e20fc00 con 0x55f66321a800 > -7795> 2018-07-11 08:04:58.648681 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:58.704895) v4 -- 0x55f653733800 con 0 > -7794> 2018-07-11 08:04:58.658749 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.114:6805/15885 conn(0x55f66315a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.7 seq 2529 0x55f653732c00 MOSDScrubReserve(5.4 REQUEST e509) v1 > -7793> 2018-07-11 08:04:58.658769 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.7 10.8.128.114:6805/15885 2529 ==== MOSDScrubReserve(5.4 REQUEST e509) v1 ==== 43+0+0 (3003630469 0 0) 0x55f653732c00 con 0x55f66315a000 > -7792> 2018-07-11 08:04:58.658853 7f491983f700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.114:6805/15885 -- MOSDScrubReserve(5.4 REJECT e509) v1 -- 0x55f66d001400 con 0 > -7791> 2018-07-11 08:04:58.659190 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.114:6805/15885 conn(0x55f66315a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.7 seq 2530 0x55f66d001400 MOSDScrubReserve(5.4 RELEASE e509) v1 > -7790> 2018-07-11 08:04:58.659210 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.7 10.8.128.114:6805/15885 2530 ==== MOSDScrubReserve(5.4 RELEASE e509) v1 ==== 43+0+0 (2082093447 0 0) 0x55f66d001400 con 0x55f66315a000 > -7789> 2018-07-11 08:04:58.732679 7f491803c700 5 osd.10 509 heartbeat: osd_stat(789 GB used, 142 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,0,1]) > -7788> 2018-07-11 08:04:58.732694 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6802/15255 -- osd_ping(ping e509 stamp 2018-07-11 08:04:58.732692) v4 -- 0x55f669613000 con 0 > -7787> 2018-07-11 08:04:58.732730 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6803/15255 -- osd_ping(ping e509 stamp 2018-07-11 08:04:58.732692) v4 -- 0x55f669613200 con 0 > -7786> 2018-07-11 08:04:58.732748 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6806/12677 -- osd_ping(ping e509 stamp 2018-07-11 08:04:58.732692) v4 -- 0x55f669613400 con 0 > -7785> 2018-07-11 08:04:58.732765 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6807/12677 -- osd_ping(ping e509 stamp 2018-07-11 08:04:58.732692) v4 -- 0x55f669613600 con 0 > -7784> 2018-07-11 08:04:58.732776 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6808/12651 -- osd_ping(ping e509 stamp 2018-07-11 08:04:58.732692) v4 -- 0x55f669613800 con 0 > -7783> 2018-07-11 08:04:58.732784 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6810/12651 -- osd_ping(ping e509 stamp 2018-07-11 08:04:58.732692) v4 -- 0x55f669613a00 con 0 > -7782> 2018-07-11 08:04:58.732797 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6809/12650 -- osd_ping(ping e509 stamp 2018-07-11 08:04:58.732692) v4 -- 0x55f669613c00 con 0 > -7781> 2018-07-11 08:04:58.732809 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6811/12650 -- osd_ping(ping e509 stamp 2018-07-11 08:04:58.732692) v4 -- 0x55f669613e00 con 0 > -7780> 2018-07-11 08:04:58.732831 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6810/15287 -- osd_ping(ping e509 stamp 2018-07-11 08:04:58.732692) v4 -- 0x55f663c0a000 con 0 > -7779> 2018-07-11 08:04:58.732848 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6811/15287 -- osd_ping(ping e509 stamp 2018-07-11 08:04:58.732692) v4 -- 0x55f663c0a400 con 0 > -7778> 2018-07-11 08:04:58.732861 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6806/15263 -- osd_ping(ping e509 stamp 2018-07-11 08:04:58.732692) v4 -- 0x55f65dad5c00 con 0 > -7777> 2018-07-11 08:04:58.732874 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6807/15263 -- osd_ping(ping e509 stamp 2018-07-11 08:04:58.732692) v4 -- 0x55f667d0da00 con 0 > -7776> 2018-07-11 08:04:58.732887 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.115:6802/13323 -- osd_ping(ping e509 stamp 2018-07-11 08:04:58.732692) v4 -- 0x55f66dceae00 con 0 > -7775> 2018-07-11 08:04:58.732900 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.115:6803/13323 -- osd_ping(ping e509 stamp 2018-07-11 08:04:58.732692) v4 -- 0x55f65299a800 con 0 > -7774> 2018-07-11 08:04:58.732913 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6806/15885 -- osd_ping(ping e509 stamp 2018-07-11 08:04:58.732692) v4 -- 0x55f66dc70200 con 0 > -7773> 2018-07-11 08:04:58.732929 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6807/15885 -- osd_ping(ping e509 stamp 2018-07-11 08:04:58.732692) v4 -- 0x55f66dc71000 con 0 > -7772> 2018-07-11 08:04:58.732943 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.108:6802/11967 -- osd_ping(ping e509 stamp 2018-07-11 08:04:58.732692) v4 -- 0x55f664257200 con 0 > -7771> 2018-07-11 08:04:58.732955 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.108:6803/11967 -- osd_ping(ping e509 stamp 2018-07-11 08:04:58.732692) v4 -- 0x55f64cf24a00 con 0 > -7770> 2018-07-11 08:04:58.732968 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.107:6806/12617 -- osd_ping(ping e509 stamp 2018-07-11 08:04:58.732692) v4 -- 0x55f64e25ce00 con 0 > -7769> 2018-07-11 08:04:58.732981 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.107:6807/12617 -- osd_ping(ping e509 stamp 2018-07-11 08:04:58.732692) v4 -- 0x55f64e25de00 con 0 > -7768> 2018-07-11 08:04:58.732994 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6802/15919 -- osd_ping(ping e509 stamp 2018-07-11 08:04:58.732692) v4 -- 0x55f662489a00 con 0 > -7767> 2018-07-11 08:04:58.733006 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6803/15919 -- osd_ping(ping e509 stamp 2018-07-11 08:04:58.732692) v4 -- 0x55f653d3d600 con 0 > -7766> 2018-07-11 08:04:58.733166 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6803/15255 conn(0x55f656ad5800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 190 0x55f66b1cd800 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:58.732692) v4 > -7765> 2018-07-11 08:04:58.733165 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6802/15255 conn(0x55f656ad4000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 190 0x55f6548fbe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:58.732692) v4 > -7764> 2018-07-11 08:04:58.733190 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.0 10.8.128.113:6803/15255 190 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:58.732692) v4 ==== 2004+0+0 (1594151809 0 0) 0x55f66b1cd800 con 0x55f656ad5800 > -7763> 2018-07-11 08:04:58.733193 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.0 10.8.128.113:6802/15255 190 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:58.732692) v4 ==== 2004+0+0 (1594151809 0 0) 0x55f6548fbe00 con 0x55f656ad4000 > -7762> 2018-07-11 08:04:58.733258 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.107:6806/12617 conn(0x55f656ade000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.9 seq 190 0x55f66b1cd800 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:58.732692) v4 > -7761> 2018-07-11 08:04:58.733265 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6807/12677 conn(0x55f64e3d9800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.1 seq 190 0x55f6548fbe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:58.732692) v4 > -7760> 2018-07-11 08:04:58.733273 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.9 10.8.128.107:6806/12617 190 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:58.732692) v4 ==== 2004+0+0 (1594151809 0 0) 0x55f66b1cd800 con 0x55f656ade000 > -7759> 2018-07-11 08:04:58.733280 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.1 10.8.128.106:6807/12677 190 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:58.732692) v4 ==== 2004+0+0 (1594151809 0 0) 0x55f6548fbe00 con 0x55f64e3d9800 > -7758> 2018-07-11 08:04:58.733277 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.107:6807/12617 conn(0x55f656adf800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.9 seq 190 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:58.732692) v4 > -7757> 2018-07-11 08:04:58.733314 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.9 10.8.128.107:6807/12617 190 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:58.732692) v4 ==== 2004+0+0 (1594151809 0 0) 0x55f67d1afe00 con 0x55f656adf800 > -7756> 2018-07-11 08:04:58.733333 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6810/15287 conn(0x55f64e401000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.4 seq 190 0x55f6548fbe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:58.732692) v4 > -7755> 2018-07-11 08:04:58.733341 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6811/15287 conn(0x55f64e402800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.4 seq 190 0x55f66b1cd800 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:58.732692) v4 > -7754> 2018-07-11 08:04:58.733347 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.4 10.8.128.113:6810/15287 190 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:58.732692) v4 ==== 2004+0+0 (1594151809 0 0) 0x55f6548fbe00 con 0x55f64e401000 > -7753> 2018-07-11 08:04:58.733353 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.4 10.8.128.113:6811/15287 190 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:58.732692) v4 ==== 2004+0+0 (1594151809 0 0) 0x55f66b1cd800 con 0x55f64e402800 > -7752> 2018-07-11 08:04:58.733357 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6806/12677 conn(0x55f64e3d8000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.1 seq 190 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:58.732692) v4 > -7751> 2018-07-11 08:04:58.733376 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6808/12651 conn(0x55f64e3ea000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.2 seq 190 0x55f6548fbe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:58.732692) v4 > -7750> 2018-07-11 08:04:58.733388 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.2 10.8.128.106:6808/12651 190 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:58.732692) v4 ==== 2004+0+0 (1594151809 0 0) 0x55f6548fbe00 con 0x55f64e3ea000 > -7749> 2018-07-11 08:04:58.733372 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.1 10.8.128.106:6806/12677 190 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:58.732692) v4 ==== 2004+0+0 (1594151809 0 0) 0x55f67d1afe00 con 0x55f64e3d8000 > -7748> 2018-07-11 08:04:58.733419 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6811/12650 conn(0x55f656ad8800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.3 seq 190 0x55f6548fbe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:58.732692) v4 > -7747> 2018-07-11 08:04:58.733422 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6806/15263 conn(0x55f64e3ed000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.5 seq 190 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:58.732692) v4 > -7746> 2018-07-11 08:04:58.733432 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.3 10.8.128.106:6811/12650 190 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:58.732692) v4 ==== 2004+0+0 (1594151809 0 0) 0x55f6548fbe00 con 0x55f656ad8800 > -7745> 2018-07-11 08:04:58.733442 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.5 10.8.128.113:6806/15263 190 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:58.732692) v4 ==== 2004+0+0 (1594151809 0 0) 0x55f67d1afe00 con 0x55f64e3ed000 > -7744> 2018-07-11 08:04:58.733447 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6810/12651 conn(0x55f64e3eb800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.2 seq 190 0x55f66b1cd800 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:58.732692) v4 > -7743> 2018-07-11 08:04:58.733466 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.2 10.8.128.106:6810/12651 190 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:58.732692) v4 ==== 2004+0+0 (1594151809 0 0) 0x55f66b1cd800 con 0x55f64e3eb800 > -7742> 2018-07-11 08:04:58.733472 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6807/15263 conn(0x55f64e3ee800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.5 seq 190 0x55f6548fbe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:58.732692) v4 > -7741> 2018-07-11 08:04:58.733475 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6809/12650 conn(0x55f656ad7000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.3 seq 190 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:58.732692) v4 > -7740> 2018-07-11 08:04:58.733485 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.5 10.8.128.113:6807/15263 190 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:58.732692) v4 ==== 2004+0+0 (1594151809 0 0) 0x55f6548fbe00 con 0x55f64e3ee800 > -7739> 2018-07-11 08:04:58.733491 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.3 10.8.128.106:6809/12650 190 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:58.732692) v4 ==== 2004+0+0 (1594151809 0 0) 0x55f67d1afe00 con 0x55f656ad7000 > -7738> 2018-07-11 08:04:58.733498 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6806/15885 conn(0x55f64e3fe000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.7 seq 190 0x55f66b1cd800 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:58.732692) v4 > -7737> 2018-07-11 08:04:58.733510 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.7 10.8.128.114:6806/15885 190 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:58.732692) v4 ==== 2004+0+0 (1594151809 0 0) 0x55f66b1cd800 con 0x55f64e3fe000 > -7736> 2018-07-11 08:04:58.733534 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.108:6803/11967 conn(0x55f656acb800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.8 seq 190 0x55f6548fbe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:58.732692) v4 > -7735> 2018-07-11 08:04:58.733543 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.115:6802/13323 conn(0x55f656acd000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.6 seq 190 0x55f66b1cd800 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:58.732692) v4 > -7734> 2018-07-11 08:04:58.733543 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.115:6803/13323 conn(0x55f656ace800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.6 seq 190 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:58.732692) v4 > -7733> 2018-07-11 08:04:58.733547 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.8 10.8.128.108:6803/11967 190 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:58.732692) v4 ==== 2004+0+0 (1594151809 0 0) 0x55f6548fbe00 con 0x55f656acb800 > -7732> 2018-07-11 08:04:58.733556 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.6 10.8.128.115:6803/13323 190 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:58.732692) v4 ==== 2004+0+0 (1594151809 0 0) 0x55f67d1afe00 con 0x55f656ace800 > -7731> 2018-07-11 08:04:58.733557 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.6 10.8.128.115:6802/13323 190 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:58.732692) v4 ==== 2004+0+0 (1594151809 0 0) 0x55f66b1cd800 con 0x55f656acd000 > -7730> 2018-07-11 08:04:58.733592 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6807/15885 conn(0x55f64e3ff800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.7 seq 190 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:58.732692) v4 > -7729> 2018-07-11 08:04:58.733599 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6802/15919 conn(0x55f64e3db000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.11 seq 190 0x55f66b1cd800 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:58.732692) v4 > -7728> 2018-07-11 08:04:58.733605 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.7 10.8.128.114:6807/15885 190 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:58.732692) v4 ==== 2004+0+0 (1594151809 0 0) 0x55f67d1afe00 con 0x55f64e3ff800 > -7727> 2018-07-11 08:04:58.733615 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.11 10.8.128.114:6802/15919 190 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:58.732692) v4 ==== 2004+0+0 (1594151809 0 0) 0x55f66b1cd800 con 0x55f64e3db000 > -7726> 2018-07-11 08:04:58.733639 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6803/15919 conn(0x55f64e3dc800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.11 seq 190 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:58.732692) v4 > -7725> 2018-07-11 08:04:58.733652 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.11 10.8.128.114:6803/15919 190 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:58.732692) v4 ==== 2004+0+0 (1594151809 0 0) 0x55f67d1afe00 con 0x55f64e3dc800 > -7724> 2018-07-11 08:04:58.733684 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.108:6802/11967 conn(0x55f656aca000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.8 seq 190 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:58.732692) v4 > -7723> 2018-07-11 08:04:58.733697 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.8 10.8.128.108:6802/11967 190 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:58.732692) v4 ==== 2004+0+0 (1594151809 0 0) 0x55f67d1afe00 con 0x55f656aca000 > -7722> 2018-07-11 08:04:59.025240 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f65b188900 con 0 > -7721> 2018-07-11 08:04:59.025826 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12650 conn(0x55f663eda800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.3 seq 189 0x55f653732400 osd_ping(ping e509 stamp 2018-07-11 08:04:59.082320) v4 > -7720> 2018-07-11 08:04:59.025849 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.3 10.8.128.106:0/12650 189 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:59.082320) v4 ==== 2004+0+0 (1979467110 0 0) 0x55f653732400 con 0x55f663eda800 > -7719> 2018-07-11 08:04:59.025866 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:59.082320) v4 -- 0x55f66d001200 con 0 > -7718> 2018-07-11 08:04:59.025984 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12650 conn(0x55f663ed9000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.3 seq 189 0x55f66b1cd800 osd_ping(ping e509 stamp 2018-07-11 08:04:59.082320) v4 > -7717> 2018-07-11 08:04:59.026004 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.3 10.8.128.106:0/12650 189 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:59.082320) v4 ==== 2004+0+0 (1979467110 0 0) 0x55f66b1cd800 con 0x55f663ed9000 > -7716> 2018-07-11 08:04:59.026016 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:59.082320) v4 -- 0x55f653732e00 con 0 > -7715> 2018-07-11 08:04:59.027093 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 195 0x55f65a3f6280 replica scrub(pg: 6.15,from:0'0,to:430'106449,epoch:509/504,start:6:a81f8afc:::benchmark_data_magna108.ceph.redhat.com_2478_object241465:0,end:6:a820c40b:::benchmark_data_magna108.ceph.redhat.com_2478_object395510:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -7714> 2018-07-11 08:04:59.027126 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 195 ==== replica scrub(pg: 6.15,from:0'0,to:430'106449,epoch:509/504,start:6:a81f8afc:::benchmark_data_magna108.ceph.redhat.com_2478_object241465:0,end:6:a820c40b:::benchmark_data_magna108.ceph.redhat.com_2478_object395510:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 248+0+0 (4170277274 0 0) 0x55f65a3f6280 con 0x55f66315e800 > -7713> 2018-07-11 08:04:59.128398 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15885 conn(0x55f663217800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 192 0x55f65d025400 osd_ping(ping e509 stamp 2018-07-11 08:04:59.763799) v4 > -7712> 2018-07-11 08:04:59.128420 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.7 10.8.128.114:0/15885 192 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:59.763799) v4 ==== 2004+0+0 (4248050183 0 0) 0x55f65d025400 con 0x55f663217800 > -7711> 2018-07-11 08:04:59.128433 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:59.763799) v4 -- 0x55f653732e00 con 0 > -7710> 2018-07-11 08:04:59.128442 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15885 conn(0x55f663216000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 192 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:04:59.763799) v4 > -7709> 2018-07-11 08:04:59.128465 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.7 10.8.128.114:0/15885 192 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:59.763799) v4 ==== 2004+0+0 (4248050183 0 0) 0x55f67d1afe00 con 0x55f663216000 > -7708> 2018-07-11 08:04:59.128476 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:59.763799) v4 -- 0x55f669613400 con 0 > -7707> 2018-07-11 08:04:59.651291 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0d000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 194 0x55f653732400 osd_ping(ping e509 stamp 2018-07-11 08:05:00.895461) v4 > -7706> 2018-07-11 08:04:59.651301 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0e800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 194 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:05:00.895461) v4 > -7705> 2018-07-11 08:04:59.651330 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.5 10.8.128.113:0/15263 194 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:00.895461) v4 ==== 2004+0+0 (3056781023 0 0) 0x55f653732400 con 0x55f663f0d000 > -7704> 2018-07-11 08:04:59.651342 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:00.895461) v4 -- 0x55f66d001200 con 0 > -7703> 2018-07-11 08:04:59.651335 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.5 10.8.128.113:0/15263 194 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:00.895461) v4 ==== 2004+0+0 (3056781023 0 0) 0x55f64e20fc00 con 0x55f663f0e800 > -7702> 2018-07-11 08:04:59.651367 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:00.895461) v4 -- 0x55f669613400 con 0 > -7701> 2018-07-11 08:04:59.827934 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f653f90b40 con 0 > -7700> 2018-07-11 08:04:59.829524 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 196 0x55f65a3f6500 replica scrub(pg: 6.15,from:0'0,to:461'106885,epoch:509/504,start:6:a820c40b:::benchmark_data_magna108.ceph.redhat.com_2478_object395510:0,end:6:a821f7c7:::benchmark_data_magna108_35617_object1011002:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -7699> 2018-07-11 08:04:59.829555 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 196 ==== replica scrub(pg: 6.15,from:0'0,to:461'106885,epoch:509/504,start:6:a820c40b:::benchmark_data_magna108.ceph.redhat.com_2478_object395510:0,end:6:a821f7c7:::benchmark_data_magna108_35617_object1011002:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 234+0+0 (3358935089 0 0) 0x55f65a3f6500 con 0x55f66315e800 > -7698> 2018-07-11 08:04:59.833150 7f491803c700 5 osd.10 509 heartbeat: osd_stat(789 GB used, 142 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,1]) > -7697> 2018-07-11 08:04:59.833164 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6802/15255 -- osd_ping(ping e509 stamp 2018-07-11 08:04:59.833162) v4 -- 0x55f64f769c00 con 0 > -7696> 2018-07-11 08:04:59.833187 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6803/15255 -- osd_ping(ping e509 stamp 2018-07-11 08:04:59.833162) v4 -- 0x55f6780b3a00 con 0 > -7695> 2018-07-11 08:04:59.833207 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6806/12677 -- osd_ping(ping e509 stamp 2018-07-11 08:04:59.833162) v4 -- 0x55f6780b2c00 con 0 > -7694> 2018-07-11 08:04:59.833223 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6807/12677 -- osd_ping(ping e509 stamp 2018-07-11 08:04:59.833162) v4 -- 0x55f65e71c800 con 0 > -7693> 2018-07-11 08:04:59.833236 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6808/12651 -- osd_ping(ping e509 stamp 2018-07-11 08:04:59.833162) v4 -- 0x55f6631cc200 con 0 > -7692> 2018-07-11 08:04:59.833260 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6810/12651 -- osd_ping(ping e509 stamp 2018-07-11 08:04:59.833162) v4 -- 0x55f653d4b600 con 0 > -7691> 2018-07-11 08:04:59.833281 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6809/12650 -- osd_ping(ping e509 stamp 2018-07-11 08:04:59.833162) v4 -- 0x55f65d024400 con 0 > -7690> 2018-07-11 08:04:59.833294 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6811/12650 -- osd_ping(ping e509 stamp 2018-07-11 08:04:59.833162) v4 -- 0x55f64e20fe00 con 0 > -7689> 2018-07-11 08:04:59.833308 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6810/15287 -- osd_ping(ping e509 stamp 2018-07-11 08:04:59.833162) v4 -- 0x55f6631f5a00 con 0 > -7688> 2018-07-11 08:04:59.833320 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6811/15287 -- osd_ping(ping e509 stamp 2018-07-11 08:04:59.833162) v4 -- 0x55f663f5e800 con 0 > -7687> 2018-07-11 08:04:59.833336 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6806/15263 -- osd_ping(ping e509 stamp 2018-07-11 08:04:59.833162) v4 -- 0x55f664255000 con 0 > -7686> 2018-07-11 08:04:59.833350 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6807/15263 -- osd_ping(ping e509 stamp 2018-07-11 08:04:59.833162) v4 -- 0x55f64e12e000 con 0 > -7685> 2018-07-11 08:04:59.833363 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.115:6802/13323 -- osd_ping(ping e509 stamp 2018-07-11 08:04:59.833162) v4 -- 0x55f64e12e200 con 0 > -7684> 2018-07-11 08:04:59.833375 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.115:6803/13323 -- osd_ping(ping e509 stamp 2018-07-11 08:04:59.833162) v4 -- 0x55f64e12e400 con 0 > -7683> 2018-07-11 08:04:59.833388 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6806/15885 -- osd_ping(ping e509 stamp 2018-07-11 08:04:59.833162) v4 -- 0x55f64e12e600 con 0 > -7682> 2018-07-11 08:04:59.833400 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6807/15885 -- osd_ping(ping e509 stamp 2018-07-11 08:04:59.833162) v4 -- 0x55f64e12e800 con 0 > -7681> 2018-07-11 08:04:59.833413 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.108:6802/11967 -- osd_ping(ping e509 stamp 2018-07-11 08:04:59.833162) v4 -- 0x55f64e12ea00 con 0 > -7680> 2018-07-11 08:04:59.833425 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.108:6803/11967 -- osd_ping(ping e509 stamp 2018-07-11 08:04:59.833162) v4 -- 0x55f64e12ec00 con 0 > -7679> 2018-07-11 08:04:59.833437 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.107:6806/12617 -- osd_ping(ping e509 stamp 2018-07-11 08:04:59.833162) v4 -- 0x55f64e12ee00 con 0 > -7678> 2018-07-11 08:04:59.833450 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.107:6807/12617 -- osd_ping(ping e509 stamp 2018-07-11 08:04:59.833162) v4 -- 0x55f64e12f000 con 0 > -7677> 2018-07-11 08:04:59.833462 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6802/15919 -- osd_ping(ping e509 stamp 2018-07-11 08:04:59.833162) v4 -- 0x55f64e12f200 con 0 > -7676> 2018-07-11 08:04:59.833474 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6803/15919 -- osd_ping(ping e509 stamp 2018-07-11 08:04:59.833162) v4 -- 0x55f64e12f400 con 0 > -7675> 2018-07-11 08:04:59.833540 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6802/15255 conn(0x55f656ad4000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 191 0x55f653732400 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:59.833162) v4 > -7674> 2018-07-11 08:04:59.833565 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.0 10.8.128.113:6802/15255 191 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:59.833162) v4 ==== 2004+0+0 (2867164397 0 0) 0x55f653732400 con 0x55f656ad4000 > -7673> 2018-07-11 08:04:59.833643 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6807/12677 conn(0x55f64e3d9800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.1 seq 191 0x55f653732400 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:59.833162) v4 > -7672> 2018-07-11 08:04:59.833663 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.1 10.8.128.106:6807/12677 191 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:59.833162) v4 ==== 2004+0+0 (2867164397 0 0) 0x55f653732400 con 0x55f64e3d9800 > -7671> 2018-07-11 08:04:59.833665 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.107:6806/12617 conn(0x55f656ade000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.9 seq 191 0x55f65d025400 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:59.833162) v4 > -7670> 2018-07-11 08:04:59.833729 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6808/12651 conn(0x55f64e3ea000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.2 seq 191 0x55f653732400 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:59.833162) v4 > -7669> 2018-07-11 08:04:59.833737 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.9 10.8.128.107:6806/12617 191 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:59.833162) v4 ==== 2004+0+0 (2867164397 0 0) 0x55f65d025400 con 0x55f656ade000 > -7668> 2018-07-11 08:04:59.833750 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.2 10.8.128.106:6808/12651 191 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:59.833162) v4 ==== 2004+0+0 (2867164397 0 0) 0x55f653732400 con 0x55f64e3ea000 > -7667> 2018-07-11 08:04:59.833770 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.107:6807/12617 conn(0x55f656adf800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.9 seq 191 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:59.833162) v4 > -7666> 2018-07-11 08:04:59.833791 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.9 10.8.128.107:6807/12617 191 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:59.833162) v4 ==== 2004+0+0 (2867164397 0 0) 0x55f67d1afe00 con 0x55f656adf800 > -7665> 2018-07-11 08:04:59.833805 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6803/15255 conn(0x55f656ad5800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 191 0x55f65d025400 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:59.833162) v4 > -7664> 2018-07-11 08:04:59.833825 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.0 10.8.128.113:6803/15255 191 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:59.833162) v4 ==== 2004+0+0 (2867164397 0 0) 0x55f65d025400 con 0x55f656ad5800 > -7663> 2018-07-11 08:04:59.833800 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6811/12650 conn(0x55f656ad8800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.3 seq 191 0x55f653732400 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:59.833162) v4 > -7662> 2018-07-11 08:04:59.833853 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6806/12677 conn(0x55f64e3d8000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.1 seq 191 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:59.833162) v4 > -7661> 2018-07-11 08:04:59.833876 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.1 10.8.128.106:6806/12677 191 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:59.833162) v4 ==== 2004+0+0 (2867164397 0 0) 0x55f67d1afe00 con 0x55f64e3d8000 > -7660> 2018-07-11 08:04:59.833871 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.3 10.8.128.106:6811/12650 191 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:59.833162) v4 ==== 2004+0+0 (2867164397 0 0) 0x55f653732400 con 0x55f656ad8800 > -7659> 2018-07-11 08:04:59.833892 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6810/12651 conn(0x55f64e3eb800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.2 seq 191 0x55f65d025400 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:59.833162) v4 > -7658> 2018-07-11 08:04:59.833901 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6809/12650 conn(0x55f656ad7000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.3 seq 191 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:59.833162) v4 > -7657> 2018-07-11 08:04:59.833911 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.2 10.8.128.106:6810/12651 191 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:59.833162) v4 ==== 2004+0+0 (2867164397 0 0) 0x55f65d025400 con 0x55f64e3eb800 > -7656> 2018-07-11 08:04:59.833911 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.3 10.8.128.106:6809/12650 191 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:59.833162) v4 ==== 2004+0+0 (2867164397 0 0) 0x55f67d1afe00 con 0x55f656ad7000 > -7655> 2018-07-11 08:04:59.833951 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6810/15287 conn(0x55f64e401000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.4 seq 191 0x55f653732400 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:59.833162) v4 > -7654> 2018-07-11 08:04:59.833974 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6811/15287 conn(0x55f64e402800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.4 seq 191 0x55f65d025400 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:59.833162) v4 > -7653> 2018-07-11 08:04:59.833988 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6807/15885 conn(0x55f64e3ff800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.7 seq 191 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:59.833162) v4 > -7652> 2018-07-11 08:04:59.833993 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.4 10.8.128.113:6811/15287 191 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:59.833162) v4 ==== 2004+0+0 (2867164397 0 0) 0x55f65d025400 con 0x55f64e402800 > -7651> 2018-07-11 08:04:59.833993 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.4 10.8.128.113:6810/15287 191 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:59.833162) v4 ==== 2004+0+0 (2867164397 0 0) 0x55f653732400 con 0x55f64e401000 > -7650> 2018-07-11 08:04:59.834002 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.7 10.8.128.114:6807/15885 191 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:59.833162) v4 ==== 2004+0+0 (2867164397 0 0) 0x55f67d1afe00 con 0x55f64e3ff800 > -7649> 2018-07-11 08:04:59.834034 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6806/15263 conn(0x55f64e3ed000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.5 seq 191 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:59.833162) v4 > -7648> 2018-07-11 08:04:59.834038 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6806/15885 conn(0x55f64e3fe000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.7 seq 191 0x55f65d025400 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:59.833162) v4 > -7647> 2018-07-11 08:04:59.834046 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.5 10.8.128.113:6806/15263 191 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:59.833162) v4 ==== 2004+0+0 (2867164397 0 0) 0x55f67d1afe00 con 0x55f64e3ed000 > -7646> 2018-07-11 08:04:59.834054 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6807/15263 conn(0x55f64e3ee800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.5 seq 191 0x55f653732400 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:59.833162) v4 > -7645> 2018-07-11 08:04:59.834060 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.7 10.8.128.114:6806/15885 191 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:59.833162) v4 ==== 2004+0+0 (2867164397 0 0) 0x55f65d025400 con 0x55f64e3fe000 > -7644> 2018-07-11 08:04:59.834078 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.5 10.8.128.113:6807/15263 191 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:59.833162) v4 ==== 2004+0+0 (2867164397 0 0) 0x55f653732400 con 0x55f64e3ee800 > -7643> 2018-07-11 08:04:59.834090 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.115:6803/13323 conn(0x55f656ace800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.6 seq 191 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:59.833162) v4 > -7642> 2018-07-11 08:04:59.834104 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.6 10.8.128.115:6803/13323 191 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:59.833162) v4 ==== 2004+0+0 (2867164397 0 0) 0x55f67d1afe00 con 0x55f656ace800 > -7641> 2018-07-11 08:04:59.834113 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6802/15919 conn(0x55f64e3db000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.11 seq 191 0x55f65d025400 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:59.833162) v4 > -7640> 2018-07-11 08:04:59.834126 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.108:6803/11967 conn(0x55f656acb800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.8 seq 191 0x55f653732400 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:59.833162) v4 > -7639> 2018-07-11 08:04:59.834138 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6803/15919 conn(0x55f64e3dc800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.11 seq 191 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:59.833162) v4 > -7638> 2018-07-11 08:04:59.834132 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.11 10.8.128.114:6802/15919 191 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:59.833162) v4 ==== 2004+0+0 (2867164397 0 0) 0x55f65d025400 con 0x55f64e3db000 > -7637> 2018-07-11 08:04:59.834147 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.8 10.8.128.108:6803/11967 191 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:59.833162) v4 ==== 2004+0+0 (2867164397 0 0) 0x55f653732400 con 0x55f656acb800 > -7636> 2018-07-11 08:04:59.834153 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.11 10.8.128.114:6803/15919 191 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:59.833162) v4 ==== 2004+0+0 (2867164397 0 0) 0x55f67d1afe00 con 0x55f64e3dc800 > -7635> 2018-07-11 08:04:59.834201 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.108:6802/11967 conn(0x55f656aca000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.8 seq 191 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:59.833162) v4 > -7634> 2018-07-11 08:04:59.834205 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.115:6802/13323 conn(0x55f656acd000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.6 seq 191 0x55f65d025400 osd_ping(ping_reply e509 stamp 2018-07-11 08:04:59.833162) v4 > -7633> 2018-07-11 08:04:59.834214 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.8 10.8.128.108:6802/11967 191 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:59.833162) v4 ==== 2004+0+0 (2867164397 0 0) 0x55f67d1afe00 con 0x55f656aca000 > -7632> 2018-07-11 08:04:59.834227 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.6 10.8.128.115:6802/13323 191 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:04:59.833162) v4 ==== 2004+0+0 (2867164397 0 0) 0x55f65d025400 con 0x55f656acd000 > -7631> 2018-07-11 08:04:59.872447 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed7800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 191 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:04:59.928816) v4 > -7630> 2018-07-11 08:04:59.872468 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.1 10.8.128.106:0/12677 191 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:59.928816) v4 ==== 2004+0+0 (4048465734 0 0) 0x55f67d1afe00 con 0x55f663ed7800 > -7629> 2018-07-11 08:04:59.872479 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:59.928816) v4 -- 0x55f669613400 con 0 > -7628> 2018-07-11 08:04:59.872473 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed6000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 191 0x55f653732400 osd_ping(ping e509 stamp 2018-07-11 08:04:59.928816) v4 > -7627> 2018-07-11 08:04:59.872497 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.1 10.8.128.106:0/12677 191 ==== osd_ping(ping e509 stamp 2018-07-11 08:04:59.928816) v4 ==== 2004+0+0 (4048465734 0 0) 0x55f653732400 con 0x55f663ed6000 > -7626> 2018-07-11 08:04:59.872515 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:04:59.928816) v4 -- 0x55f6631cc200 con 0 > -7625> 2018-07-11 08:05:00.180013 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.107:0/12617 conn(0x55f64e3c5000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.9 seq 189 0x55f6548fbe00 osd_ping(ping e509 stamp 2018-07-11 08:05:00.179460) v4 > -7624> 2018-07-11 08:05:00.180045 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.9 10.8.128.107:0/12617 189 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:00.179460) v4 ==== 2004+0+0 (2256756642 0 0) 0x55f6548fbe00 con 0x55f64e3c5000 > -7623> 2018-07-11 08:05:00.180088 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:00.179460) v4 -- 0x55f65e71c800 con 0 > -7622> 2018-07-11 08:05:00.180098 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.107:0/12617 conn(0x55f64e3c3800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.9 seq 189 0x55f65d025400 osd_ping(ping e509 stamp 2018-07-11 08:05:00.179460) v4 > -7621> 2018-07-11 08:05:00.180123 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.9 10.8.128.107:0/12617 189 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:00.179460) v4 ==== 2004+0+0 (2256756642 0 0) 0x55f65d025400 con 0x55f64e3c3800 > -7620> 2018-07-11 08:05:00.180139 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:00.179460) v4 -- 0x55f653732e00 con 0 > -7619> 2018-07-11 08:05:00.372258 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15255 conn(0x55f663ec1000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.0 seq 183 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:05:01.616309) v4 > -7618> 2018-07-11 08:05:00.372279 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.0 10.8.128.113:0/15255 183 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:01.616309) v4 ==== 2004+0+0 (2676715239 0 0) 0x55f64e20fc00 con 0x55f663ec1000 > -7617> 2018-07-11 08:05:00.372296 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:01.616309) v4 -- 0x55f653733800 con 0 > -7616> 2018-07-11 08:05:00.372297 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15255 conn(0x55f663ec2800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.0 seq 183 0x55f66b1cd800 osd_ping(ping e509 stamp 2018-07-11 08:05:01.616309) v4 > -7615> 2018-07-11 08:05:00.372317 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.0 10.8.128.113:0/15255 183 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:01.616309) v4 ==== 2004+0+0 (2676715239 0 0) 0x55f66b1cd800 con 0x55f663ec2800 > -7614> 2018-07-11 08:05:00.372333 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:01.616309) v4 -- 0x55f653732e00 con 0 > -7613> 2018-07-11 08:05:00.433258 7f491e048700 10 monclient: tick > -7612> 2018-07-11 08:05:00.433272 7f491e048700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2018-07-11 08:04:30.433270) > -7611> 2018-07-11 08:05:00.494966 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.108:0/11967 conn(0x55f663f6f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=1). rx osd.8 seq 183 0x55f65d025400 osd_ping(ping e509 stamp 2018-07-11 08:05:00.254693) v4 > -7610> 2018-07-11 08:05:00.494986 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.8 10.8.128.108:0/11967 183 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:00.254693) v4 ==== 2004+0+0 (1954009956 0 0) 0x55f65d025400 con 0x55f663f6f000 > -7609> 2018-07-11 08:05:00.494996 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:00.254693) v4 -- 0x55f653732e00 con 0 > -7608> 2018-07-11 08:05:00.495020 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.108:0/11967 conn(0x55f663f70800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=1). rx osd.8 seq 183 0x55f6548fbe00 osd_ping(ping e509 stamp 2018-07-11 08:05:00.254693) v4 > -7607> 2018-07-11 08:05:00.495041 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.8 10.8.128.108:0/11967 183 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:00.254693) v4 ==== 2004+0+0 (1954009956 0 0) 0x55f6548fbe00 con 0x55f663f70800 > -7606> 2018-07-11 08:05:00.495052 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:00.254693) v4 -- 0x55f65e71c800 con 0 > -7605> 2018-07-11 08:05:00.526324 7f491d046700 1 -- 10.8.128.107:6800/14173 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55f656aac000 con 0 > -7604> 2018-07-11 08:05:00.526396 7f491d046700 1 -- 10.8.128.107:6800/14173 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55f662a89600 con 0 > -7603> 2018-07-11 08:05:00.751721 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0d000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 195 0x55f653732400 osd_ping(ping e509 stamp 2018-07-11 08:05:01.995875) v4 > -7602> 2018-07-11 08:05:00.751741 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.5 10.8.128.113:0/15263 195 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:01.995875) v4 ==== 2004+0+0 (672031910 0 0) 0x55f653732400 con 0x55f663f0d000 > -7601> 2018-07-11 08:05:00.751751 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:01.995875) v4 -- 0x55f65e71c800 con 0 > -7600> 2018-07-11 08:05:00.751737 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0e800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 195 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:05:01.995875) v4 > -7599> 2018-07-11 08:05:00.751764 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.5 10.8.128.113:0/15263 195 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:01.995875) v4 ==== 2004+0+0 (672031910 0 0) 0x55f67d1afe00 con 0x55f663f0e800 > -7598> 2018-07-11 08:05:00.751781 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:01.995875) v4 -- 0x55f653733800 con 0 > -7597> 2018-07-11 08:05:00.828878 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15885 conn(0x55f663217800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 193 0x55f66b1cd800 osd_ping(ping e509 stamp 2018-07-11 08:05:01.464254) v4 > -7596> 2018-07-11 08:05:00.828890 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15885 conn(0x55f663216000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 193 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:05:01.464254) v4 > -7595> 2018-07-11 08:05:00.828900 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.7 10.8.128.114:0/15885 193 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:01.464254) v4 ==== 2004+0+0 (4217009420 0 0) 0x55f66b1cd800 con 0x55f663217800 > -7594> 2018-07-11 08:05:00.828907 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.7 10.8.128.114:0/15885 193 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:01.464254) v4 ==== 2004+0+0 (4217009420 0 0) 0x55f64e20fc00 con 0x55f663216000 > -7593> 2018-07-11 08:05:00.828917 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:01.464254) v4 -- 0x55f653732e00 con 0 > -7592> 2018-07-11 08:05:00.828948 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:01.464254) v4 -- 0x55f653733800 con 0 > -7591> 2018-07-11 08:05:00.904459 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 29] Generated table #37035: 2638240 keys, 74545896 bytes > -7590> 2018-07-11 08:05:00.904505 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296300904489, "cf_name": "default", "job": 29, "event": "table_file_creation", "file_number": 37035, "file_size": 74545896, "table_properties": {"data_size": 67112133, "index_size": 689592, "filter_size": 6743181, "raw_key_size": 76215820, "raw_average_key_size": 28, "raw_value_size": 31365740, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} > -7589> 2018-07-11 08:05:00.933616 7f491803c700 5 osd.10 509 heartbeat: osd_stat(789 GB used, 142 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,0,0,0,1]) > -7588> 2018-07-11 08:05:00.933631 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6802/15255 -- osd_ping(ping e509 stamp 2018-07-11 08:05:00.933629) v4 -- 0x55f64e12f600 con 0 > -7587> 2018-07-11 08:05:00.933673 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6803/15255 -- osd_ping(ping e509 stamp 2018-07-11 08:05:00.933629) v4 -- 0x55f64e12f800 con 0 > -7586> 2018-07-11 08:05:00.933688 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6806/12677 -- osd_ping(ping e509 stamp 2018-07-11 08:05:00.933629) v4 -- 0x55f64e12fa00 con 0 > -7585> 2018-07-11 08:05:00.933704 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6807/12677 -- osd_ping(ping e509 stamp 2018-07-11 08:05:00.933629) v4 -- 0x55f64e12fc00 con 0 > -7584> 2018-07-11 08:05:00.933717 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6808/12651 -- osd_ping(ping e509 stamp 2018-07-11 08:05:00.933629) v4 -- 0x55f64e12fe00 con 0 > -7583> 2018-07-11 08:05:00.933727 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6810/12651 -- osd_ping(ping e509 stamp 2018-07-11 08:05:00.933629) v4 -- 0x55f660ea4000 con 0 > -7582> 2018-07-11 08:05:00.933737 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6809/12650 -- osd_ping(ping e509 stamp 2018-07-11 08:05:00.933629) v4 -- 0x55f660ea4200 con 0 > -7581> 2018-07-11 08:05:00.933747 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6811/12650 -- osd_ping(ping e509 stamp 2018-07-11 08:05:00.933629) v4 -- 0x55f660ea4400 con 0 > -7580> 2018-07-11 08:05:00.933766 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6810/15287 -- osd_ping(ping e509 stamp 2018-07-11 08:05:00.933629) v4 -- 0x55f660ea4600 con 0 > -7579> 2018-07-11 08:05:00.933778 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6811/15287 -- osd_ping(ping e509 stamp 2018-07-11 08:05:00.933629) v4 -- 0x55f660ea4800 con 0 > -7578> 2018-07-11 08:05:00.933791 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6806/15263 -- osd_ping(ping e509 stamp 2018-07-11 08:05:00.933629) v4 -- 0x55f660ea4a00 con 0 > -7577> 2018-07-11 08:05:00.933806 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6807/15263 -- osd_ping(ping e509 stamp 2018-07-11 08:05:00.933629) v4 -- 0x55f660ea4c00 con 0 > -7576> 2018-07-11 08:05:00.933820 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.115:6802/13323 -- osd_ping(ping e509 stamp 2018-07-11 08:05:00.933629) v4 -- 0x55f660ea4e00 con 0 > -7575> 2018-07-11 08:05:00.933833 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.115:6803/13323 -- osd_ping(ping e509 stamp 2018-07-11 08:05:00.933629) v4 -- 0x55f660ea5000 con 0 > -7574> 2018-07-11 08:05:00.933845 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6806/15885 -- osd_ping(ping e509 stamp 2018-07-11 08:05:00.933629) v4 -- 0x55f660ea5200 con 0 > -7573> 2018-07-11 08:05:00.933858 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6807/15885 -- osd_ping(ping e509 stamp 2018-07-11 08:05:00.933629) v4 -- 0x55f660ea5400 con 0 > -7572> 2018-07-11 08:05:00.933871 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.108:6802/11967 -- osd_ping(ping e509 stamp 2018-07-11 08:05:00.933629) v4 -- 0x55f660ea5600 con 0 > -7571> 2018-07-11 08:05:00.933883 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.108:6803/11967 -- osd_ping(ping e509 stamp 2018-07-11 08:05:00.933629) v4 -- 0x55f660ea5800 con 0 > -7570> 2018-07-11 08:05:00.933896 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.107:6806/12617 -- osd_ping(ping e509 stamp 2018-07-11 08:05:00.933629) v4 -- 0x55f660ea5a00 con 0 > -7569> 2018-07-11 08:05:00.933908 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.107:6807/12617 -- osd_ping(ping e509 stamp 2018-07-11 08:05:00.933629) v4 -- 0x55f660ea5c00 con 0 > -7568> 2018-07-11 08:05:00.933921 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6802/15919 -- osd_ping(ping e509 stamp 2018-07-11 08:05:00.933629) v4 -- 0x55f660ea5e00 con 0 > -7567> 2018-07-11 08:05:00.933934 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6803/15919 -- osd_ping(ping e509 stamp 2018-07-11 08:05:00.933629) v4 -- 0x55f663808000 con 0 > -7566> 2018-07-11 08:05:00.934071 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6802/15255 conn(0x55f656ad4000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 192 0x55f653732400 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:00.933629) v4 > -7565> 2018-07-11 08:05:00.934093 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.0 10.8.128.113:6802/15255 192 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:00.933629) v4 ==== 2004+0+0 (1375899517 0 0) 0x55f653732400 con 0x55f656ad4000 > -7564> 2018-07-11 08:05:00.934066 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6803/15255 conn(0x55f656ad5800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 192 0x55f66b1cd800 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:00.933629) v4 > -7563> 2018-07-11 08:05:00.934108 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.0 10.8.128.113:6803/15255 192 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:00.933629) v4 ==== 2004+0+0 (1375899517 0 0) 0x55f66b1cd800 con 0x55f656ad5800 > -7562> 2018-07-11 08:05:00.934185 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6807/12677 conn(0x55f64e3d9800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.1 seq 192 0x55f653732400 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:00.933629) v4 > -7561> 2018-07-11 08:05:00.934206 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6811/15287 conn(0x55f64e402800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.4 seq 192 0x55f66b1cd800 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:00.933629) v4 > -7560> 2018-07-11 08:05:00.934207 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6806/15263 conn(0x55f64e3ed000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.5 seq 192 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:00.933629) v4 > -7559> 2018-07-11 08:05:00.934219 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.4 10.8.128.113:6811/15287 192 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:00.933629) v4 ==== 2004+0+0 (1375899517 0 0) 0x55f66b1cd800 con 0x55f64e402800 > -7558> 2018-07-11 08:05:00.934212 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.1 10.8.128.106:6807/12677 192 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:00.933629) v4 ==== 2004+0+0 (1375899517 0 0) 0x55f653732400 con 0x55f64e3d9800 > -7557> 2018-07-11 08:05:00.934230 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.5 10.8.128.113:6806/15263 192 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:00.933629) v4 ==== 2004+0+0 (1375899517 0 0) 0x55f67d1afe00 con 0x55f64e3ed000 > -7556> 2018-07-11 08:05:00.934279 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.107:6806/12617 conn(0x55f656ade000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.9 seq 192 0x55f66b1cd800 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:00.933629) v4 > -7555> 2018-07-11 08:05:00.934294 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.9 10.8.128.107:6806/12617 192 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:00.933629) v4 ==== 2004+0+0 (1375899517 0 0) 0x55f66b1cd800 con 0x55f656ade000 > -7554> 2018-07-11 08:05:00.934288 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6806/12677 conn(0x55f64e3d8000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.1 seq 192 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:00.933629) v4 > -7553> 2018-07-11 08:05:00.934304 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.1 10.8.128.106:6806/12677 192 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:00.933629) v4 ==== 2004+0+0 (1375899517 0 0) 0x55f67d1afe00 con 0x55f64e3d8000 > -7552> 2018-07-11 08:05:00.934303 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6808/12651 conn(0x55f64e3ea000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.2 seq 192 0x55f653732400 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:00.933629) v4 > -7551> 2018-07-11 08:05:00.934325 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6810/12651 conn(0x55f64e3eb800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.2 seq 192 0x55f66b1cd800 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:00.933629) v4 > -7550> 2018-07-11 08:05:00.934326 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.2 10.8.128.106:6808/12651 192 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:00.933629) v4 ==== 2004+0+0 (1375899517 0 0) 0x55f653732400 con 0x55f64e3ea000 > -7549> 2018-07-11 08:05:00.934338 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.2 10.8.128.106:6810/12651 192 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:00.933629) v4 ==== 2004+0+0 (1375899517 0 0) 0x55f66b1cd800 con 0x55f64e3eb800 > -7548> 2018-07-11 08:05:00.934350 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.107:6807/12617 conn(0x55f656adf800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.9 seq 192 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:00.933629) v4 > -7547> 2018-07-11 08:05:00.934369 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.9 10.8.128.107:6807/12617 192 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:00.933629) v4 ==== 2004+0+0 (1375899517 0 0) 0x55f67d1afe00 con 0x55f656adf800 > -7546> 2018-07-11 08:05:00.934375 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.115:6802/13323 conn(0x55f656acd000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.6 seq 192 0x55f66b1cd800 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:00.933629) v4 > -7545> 2018-07-11 08:05:00.934374 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6811/12650 conn(0x55f656ad8800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.3 seq 192 0x55f653732400 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:00.933629) v4 > -7544> 2018-07-11 08:05:00.934388 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.6 10.8.128.115:6802/13323 192 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:00.933629) v4 ==== 2004+0+0 (1375899517 0 0) 0x55f66b1cd800 con 0x55f656acd000 > -7543> 2018-07-11 08:05:00.934389 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.3 10.8.128.106:6811/12650 192 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:00.933629) v4 ==== 2004+0+0 (1375899517 0 0) 0x55f653732400 con 0x55f656ad8800 > -7542> 2018-07-11 08:05:00.934420 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6802/15919 conn(0x55f64e3db000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.11 seq 192 0x55f66b1cd800 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:00.933629) v4 > -7541> 2018-07-11 08:05:00.934416 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6809/12650 conn(0x55f656ad7000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.3 seq 192 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:00.933629) v4 > -7540> 2018-07-11 08:05:00.934430 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6810/15287 conn(0x55f64e401000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.4 seq 192 0x55f653732400 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:00.933629) v4 > -7539> 2018-07-11 08:05:00.934443 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.4 10.8.128.113:6810/15287 192 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:00.933629) v4 ==== 2004+0+0 (1375899517 0 0) 0x55f653732400 con 0x55f64e401000 > -7538> 2018-07-11 08:05:00.934433 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.11 10.8.128.114:6802/15919 192 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:00.933629) v4 ==== 2004+0+0 (1375899517 0 0) 0x55f66b1cd800 con 0x55f64e3db000 > -7537> 2018-07-11 08:05:00.934445 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.3 10.8.128.106:6809/12650 192 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:00.933629) v4 ==== 2004+0+0 (1375899517 0 0) 0x55f67d1afe00 con 0x55f656ad7000 > -7536> 2018-07-11 08:05:00.934501 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6807/15263 conn(0x55f64e3ee800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.5 seq 192 0x55f653732400 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:00.933629) v4 > -7535> 2018-07-11 08:05:00.934514 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.115:6803/13323 conn(0x55f656ace800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.6 seq 192 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:00.933629) v4 > -7534> 2018-07-11 08:05:00.934513 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6806/15885 conn(0x55f64e3fe000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.7 seq 192 0x55f66b1cd800 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:00.933629) v4 > -7533> 2018-07-11 08:05:00.934528 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.6 10.8.128.115:6803/13323 192 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:00.933629) v4 ==== 2004+0+0 (1375899517 0 0) 0x55f67d1afe00 con 0x55f656ace800 > -7532> 2018-07-11 08:05:00.934521 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.5 10.8.128.113:6807/15263 192 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:00.933629) v4 ==== 2004+0+0 (1375899517 0 0) 0x55f653732400 con 0x55f64e3ee800 > -7531> 2018-07-11 08:05:00.934537 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.7 10.8.128.114:6806/15885 192 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:00.933629) v4 ==== 2004+0+0 (1375899517 0 0) 0x55f66b1cd800 con 0x55f64e3fe000 > -7530> 2018-07-11 08:05:00.934581 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.108:6802/11967 conn(0x55f656aca000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.8 seq 192 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:00.933629) v4 > -7529> 2018-07-11 08:05:00.934581 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.108:6803/11967 conn(0x55f656acb800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.8 seq 192 0x55f653732400 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:00.933629) v4 > -7528> 2018-07-11 08:05:00.934602 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.8 10.8.128.108:6802/11967 192 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:00.933629) v4 ==== 2004+0+0 (1375899517 0 0) 0x55f67d1afe00 con 0x55f656aca000 > -7527> 2018-07-11 08:05:00.934612 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.8 10.8.128.108:6803/11967 192 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:00.933629) v4 ==== 2004+0+0 (1375899517 0 0) 0x55f653732400 con 0x55f656acb800 > -7526> 2018-07-11 08:05:00.934649 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6807/15885 conn(0x55f64e3ff800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.7 seq 192 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:00.933629) v4 > -7525> 2018-07-11 08:05:00.934666 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.7 10.8.128.114:6807/15885 192 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:00.933629) v4 ==== 2004+0+0 (1375899517 0 0) 0x55f67d1afe00 con 0x55f64e3ff800 > -7524> 2018-07-11 08:05:00.934700 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6803/15919 conn(0x55f64e3dc800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.11 seq 192 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:00.933629) v4 > -7523> 2018-07-11 08:05:00.934712 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.11 10.8.128.114:6803/15919 192 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:00.933629) v4 ==== 2004+0+0 (1375899517 0 0) 0x55f67d1afe00 con 0x55f64e3dc800 > -7522> 2018-07-11 08:05:01.329299 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15885 conn(0x55f663216000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 194 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:05:01.964716) v4 > -7521> 2018-07-11 08:05:01.329326 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.7 10.8.128.114:0/15885 194 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:01.964716) v4 ==== 2004+0+0 (2589278600 0 0) 0x55f67d1afe00 con 0x55f663216000 > -7520> 2018-07-11 08:05:01.329313 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15885 conn(0x55f663217800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 194 0x55f66b1cd800 osd_ping(ping e509 stamp 2018-07-11 08:05:01.964716) v4 > -7519> 2018-07-11 08:05:01.329339 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:01.964716) v4 -- 0x55f663808000 con 0 > -7518> 2018-07-11 08:05:01.329338 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.7 10.8.128.114:0/15885 194 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:01.964716) v4 ==== 2004+0+0 (2589278600 0 0) 0x55f66b1cd800 con 0x55f663217800 > -7517> 2018-07-11 08:05:01.329367 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:01.964716) v4 -- 0x55f660ea5e00 con 0 > -7516> 2018-07-11 08:05:01.499097 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15287 conn(0x55f66dce2800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.4 seq 197 0x55f653732400 osd_ping(ping e509 stamp 2018-07-11 08:05:02.743257) v4 > -7515> 2018-07-11 08:05:01.499120 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.4 10.8.128.113:0/15287 197 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:02.743257) v4 ==== 2004+0+0 (2368887055 0 0) 0x55f653732400 con 0x55f66dce2800 > -7514> 2018-07-11 08:05:01.499132 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:02.743257) v4 -- 0x55f660ea5800 con 0 > -7513> 2018-07-11 08:05:01.499138 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15287 conn(0x55f66dce1000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.4 seq 197 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:05:02.743257) v4 > -7512> 2018-07-11 08:05:01.499155 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.4 10.8.128.113:0/15287 197 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:02.743257) v4 ==== 2004+0+0 (2368887055 0 0) 0x55f64e20fc00 con 0x55f66dce1000 > -7511> 2018-07-11 08:05:01.499168 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:02.743257) v4 -- 0x55f663808000 con 0 > -7510> 2018-07-11 08:05:01.606223 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.115:0/13323 conn(0x55f663ebf800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.6 seq 180 0x55f6548fbe00 osd_ping(ping e509 stamp 2018-07-11 08:05:01.958642) v4 > -7509> 2018-07-11 08:05:01.606252 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.6 10.8.128.115:0/13323 180 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:01.958642) v4 ==== 2004+0+0 (4195217014 0 0) 0x55f6548fbe00 con 0x55f663ebf800 > -7508> 2018-07-11 08:05:01.606264 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:01.958642) v4 -- 0x55f660ea5800 con 0 > -7507> 2018-07-11 08:05:01.606317 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.115:0/13323 conn(0x55f663ebe000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.6 seq 180 0x55f65d025400 osd_ping(ping e509 stamp 2018-07-11 08:05:01.958642) v4 > -7506> 2018-07-11 08:05:01.606343 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.6 10.8.128.115:0/13323 180 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:01.958642) v4 ==== 2004+0+0 (4195217014 0 0) 0x55f65d025400 con 0x55f663ebe000 > -7505> 2018-07-11 08:05:01.606361 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:01.958642) v4 -- 0x55f660ea5e00 con 0 > -7504> 2018-07-11 08:05:01.659244 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.114:6805/15885 conn(0x55f66315a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.7 seq 2531 0x55f653732400 MOSDScrubReserve(5.4 REQUEST e509) v1 > -7503> 2018-07-11 08:05:01.659270 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.7 10.8.128.114:6805/15885 2531 ==== MOSDScrubReserve(5.4 REQUEST e509) v1 ==== 43+0+0 (3003630469 0 0) 0x55f653732400 con 0x55f66315a000 > -7502> 2018-07-11 08:05:01.659351 7f491983f700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.114:6805/15885 -- MOSDScrubReserve(5.4 REJECT e509) v1 -- 0x55f66d001400 con 0 > -7501> 2018-07-11 08:05:01.659619 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.114:6805/15885 conn(0x55f66315a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.7 seq 2532 0x55f660ea5800 MOSDScrubReserve(5.4 RELEASE e509) v1 > -7500> 2018-07-11 08:05:01.659633 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.7 10.8.128.114:6805/15885 2532 ==== MOSDScrubReserve(5.4 RELEASE e509) v1 ==== 43+0+0 (2082093447 0 0) 0x55f660ea5800 con 0x55f66315a000 > -7499> 2018-07-11 08:05:01.724626 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f656924b40 con 0 > -7498> 2018-07-11 08:05:01.726276 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 197 0x55f65a3f6780 replica scrub(pg: 6.15,from:0'0,to:430'105780,epoch:509/504,start:6:a821f7c7:::benchmark_data_magna108_35617_object1011002:0,end:6:a822ba4a:::benchmark_data_magna108_35617_object780720:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -7497> 2018-07-11 08:05:01.726306 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 197 ==== replica scrub(pg: 6.15,from:0'0,to:430'105780,epoch:509/504,start:6:a821f7c7:::benchmark_data_magna108_35617_object1011002:0,end:6:a822ba4a:::benchmark_data_magna108_35617_object780720:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 219+0+0 (1566678168 0 0) 0x55f65a3f6780 con 0x55f66315e800 > -7496> 2018-07-11 08:05:01.799523 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0a000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 186 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:05:02.434918) v4 > -7495> 2018-07-11 08:05:01.799545 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.11 10.8.128.114:0/15919 186 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:02.434918) v4 ==== 2004+0+0 (1007995564 0 0) 0x55f67d1afe00 con 0x55f663f0a000 > -7494> 2018-07-11 08:05:01.799554 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:02.434918) v4 -- 0x55f663808000 con 0 > -7493> 2018-07-11 08:05:01.799558 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0b800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 186 0x55f66b1cd800 osd_ping(ping e509 stamp 2018-07-11 08:05:02.434918) v4 > -7492> 2018-07-11 08:05:01.799587 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.11 10.8.128.114:0/15919 186 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:02.434918) v4 ==== 2004+0+0 (1007995564 0 0) 0x55f66b1cd800 con 0x55f663f0b800 > -7491> 2018-07-11 08:05:01.799599 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:02.434918) v4 -- 0x55f660ea5e00 con 0 > -7490> 2018-07-11 08:05:01.961431 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 413 0x55f65d025400 MOSDScrubReserve(11.3s5 REQUEST e509) v1 > -7489> 2018-07-11 08:05:01.961464 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 413 ==== MOSDScrubReserve(11.3s5 REQUEST e509) v1 ==== 43+0+0 (433425070 0 0) 0x55f65d025400 con 0x55f668a29000 > -7488> 2018-07-11 08:05:01.961516 7f491a040700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e509) v1 -- 0x55f6743f6a00 con 0 > -7487> 2018-07-11 08:05:01.961854 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 414 0x55f660ea5a00 MOSDScrubReserve(11.3s5 RELEASE e509) v1 > -7486> 2018-07-11 08:05:01.961886 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 414 ==== MOSDScrubReserve(11.3s5 RELEASE e509) v1 ==== 43+0+0 (3603460268 0 0) 0x55f660ea5a00 con 0x55f668a29000 > -7485> 2018-07-11 08:05:02.072634 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15255 conn(0x55f663ec1000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.0 seq 184 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:05:03.316819) v4 > -7484> 2018-07-11 08:05:02.072658 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.0 10.8.128.113:0/15255 184 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:03.316819) v4 ==== 2004+0+0 (1041084787 0 0) 0x55f64e20fc00 con 0x55f663ec1000 > -7483> 2018-07-11 08:05:02.072657 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15255 conn(0x55f663ec2800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.0 seq 184 0x55f66b1cd800 osd_ping(ping e509 stamp 2018-07-11 08:05:03.316819) v4 > -7482> 2018-07-11 08:05:02.072668 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:03.316819) v4 -- 0x55f660ea5c00 con 0 > -7481> 2018-07-11 08:05:02.072680 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.0 10.8.128.113:0/15255 184 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:03.316819) v4 ==== 2004+0+0 (1041084787 0 0) 0x55f66b1cd800 con 0x55f663ec2800 > -7480> 2018-07-11 08:05:02.072699 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:03.316819) v4 -- 0x55f660ea4800 con 0 > -7479> 2018-07-11 08:05:02.487280 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f6565766c0 con 0 > -7478> 2018-07-11 08:05:02.488995 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 198 0x55f65a3f6a00 replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:509/504,start:6:a822ba4a:::benchmark_data_magna108_35617_object780720:0,end:6:a823e31e:::benchmark_data_magna108_35617_object867501:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -7477> 2018-07-11 08:05:02.489028 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 198 ==== replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:509/504,start:6:a822ba4a:::benchmark_data_magna108_35617_object780720:0,end:6:a823e31e:::benchmark_data_magna108_35617_object867501:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (3698646113 0 0) 0x55f65a3f6a00 con 0x55f66315e800 > -7476> 2018-07-11 08:05:02.526280 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12650 conn(0x55f663ed9000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.3 seq 190 0x55f6743f6a00 osd_ping(ping e509 stamp 2018-07-11 08:05:02.582785) v4 > -7475> 2018-07-11 08:05:02.526302 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.3 10.8.128.106:0/12650 190 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:02.582785) v4 ==== 2004+0+0 (3567442207 0 0) 0x55f6743f6a00 con 0x55f663ed9000 > -7474> 2018-07-11 08:05:02.526312 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:02.582785) v4 -- 0x55f660ea4000 con 0 > -7473> 2018-07-11 08:05:02.526306 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12650 conn(0x55f663eda800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.3 seq 190 0x55f6548fbe00 osd_ping(ping e509 stamp 2018-07-11 08:05:02.582785) v4 > -7472> 2018-07-11 08:05:02.526327 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.3 10.8.128.106:0/12650 190 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:02.582785) v4 ==== 2004+0+0 (3567442207 0 0) 0x55f6548fbe00 con 0x55f663eda800 > -7471> 2018-07-11 08:05:02.526345 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:02.582785) v4 -- 0x55f660ea4400 con 0 > -7470> 2018-07-11 08:05:02.961609 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 415 0x55f66b1cd800 MOSDScrubReserve(11.3s5 REQUEST e509) v1 > -7469> 2018-07-11 08:05:02.961633 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 415 ==== MOSDScrubReserve(11.3s5 REQUEST e509) v1 ==== 43+0+0 (433425070 0 0) 0x55f66b1cd800 con 0x55f668a29000 > -7468> 2018-07-11 08:05:02.961700 7f491a040700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e509) v1 -- 0x55f660ea5a00 con 0 > -7467> 2018-07-11 08:05:02.962090 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 416 0x55f660ea5a00 MOSDScrubReserve(11.3s5 RELEASE e509) v1 > -7466> 2018-07-11 08:05:02.962105 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 416 ==== MOSDScrubReserve(11.3s5 RELEASE e509) v1 ==== 43+0+0 (3603460268 0 0) 0x55f660ea5a00 con 0x55f668a29000 > -7465> 2018-07-11 08:05:03.080669 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.107:0/12617 conn(0x55f64e3c3800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.9 seq 190 0x55f6743f6a00 osd_ping(ping e509 stamp 2018-07-11 08:05:03.080160) v4 > -7464> 2018-07-11 08:05:03.080692 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.9 10.8.128.107:0/12617 190 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:03.080160) v4 ==== 2004+0+0 (1127870375 0 0) 0x55f6743f6a00 con 0x55f64e3c3800 > -7463> 2018-07-11 08:05:03.080702 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:03.080160) v4 -- 0x55f66e945a00 con 0 > -7462> 2018-07-11 08:05:03.080690 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.107:0/12617 conn(0x55f64e3c5000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.9 seq 190 0x55f660ea4c00 osd_ping(ping e509 stamp 2018-07-11 08:05:03.080160) v4 > -7461> 2018-07-11 08:05:03.080713 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.9 10.8.128.107:0/12617 190 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:03.080160) v4 ==== 2004+0+0 (1127870375 0 0) 0x55f660ea4c00 con 0x55f64e3c5000 > -7460> 2018-07-11 08:05:03.080731 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:03.080160) v4 -- 0x55f64e12fe00 con 0 > -7459> 2018-07-11 08:05:03.209283 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f65fcfa240 con 0 > -7458> 2018-07-11 08:05:03.210800 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 199 0x55f65a3f6c80 replica scrub(pg: 6.15,from:0'0,to:430'106615,epoch:509/504,start:6:a823e31e:::benchmark_data_magna108_35617_object867501:0,end:6:a824b902:::benchmark_data_magna108_35617_object612676:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -7457> 2018-07-11 08:05:03.210832 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 199 ==== replica scrub(pg: 6.15,from:0'0,to:430'106615,epoch:509/504,start:6:a823e31e:::benchmark_data_magna108_35617_object867501:0,end:6:a824b902:::benchmark_data_magna108_35617_object612676:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (1354364746 0 0) 0x55f65a3f6c80 con 0x55f66315e800 > -7456> 2018-07-11 08:05:03.659544 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.114:6805/15885 conn(0x55f66315a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.7 seq 2533 0x55f6548fbe00 MOSDScrubReserve(5.4 REQUEST e509) v1 > -7455> 2018-07-11 08:05:03.659580 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.7 10.8.128.114:6805/15885 2533 ==== MOSDScrubReserve(5.4 REQUEST e509) v1 ==== 43+0+0 (3003630469 0 0) 0x55f6548fbe00 con 0x55f66315a000 > -7454> 2018-07-11 08:05:03.659666 7f491983f700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.114:6805/15885 -- MOSDScrubReserve(5.4 REJECT e509) v1 -- 0x55f660ea5800 con 0 > -7453> 2018-07-11 08:05:03.659982 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.114:6805/15885 conn(0x55f66315a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.7 seq 2534 0x55f660ea5800 MOSDScrubReserve(5.4 RELEASE e509) v1 > -7452> 2018-07-11 08:05:03.660013 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.7 10.8.128.114:6805/15885 2534 ==== MOSDScrubReserve(5.4 RELEASE e509) v1 ==== 43+0+0 (2082093447 0 0) 0x55f660ea5800 con 0x55f66315a000 > -7451> 2018-07-11 08:05:03.834091 7f491803c700 5 osd.10 509 heartbeat: osd_stat(789 GB used, 142 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,0,0,1]) > -7450> 2018-07-11 08:05:03.834108 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6802/15255 -- osd_ping(ping e509 stamp 2018-07-11 08:05:03.834106) v4 -- 0x55f663808200 con 0 > -7449> 2018-07-11 08:05:03.834146 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6803/15255 -- osd_ping(ping e509 stamp 2018-07-11 08:05:03.834106) v4 -- 0x55f663808400 con 0 > -7448> 2018-07-11 08:05:03.834166 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6806/12677 -- osd_ping(ping e509 stamp 2018-07-11 08:05:03.834106) v4 -- 0x55f663808600 con 0 > -7447> 2018-07-11 08:05:03.834182 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6807/12677 -- osd_ping(ping e509 stamp 2018-07-11 08:05:03.834106) v4 -- 0x55f663808800 con 0 > -7446> 2018-07-11 08:05:03.834195 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6808/12651 -- osd_ping(ping e509 stamp 2018-07-11 08:05:03.834106) v4 -- 0x55f663808a00 con 0 > -7445> 2018-07-11 08:05:03.834204 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6810/12651 -- osd_ping(ping e509 stamp 2018-07-11 08:05:03.834106) v4 -- 0x55f663808c00 con 0 > -7444> 2018-07-11 08:05:03.834214 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6809/12650 -- osd_ping(ping e509 stamp 2018-07-11 08:05:03.834106) v4 -- 0x55f663808e00 con 0 > -7443> 2018-07-11 08:05:03.834226 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6811/12650 -- osd_ping(ping e509 stamp 2018-07-11 08:05:03.834106) v4 -- 0x55f663809000 con 0 > -7442> 2018-07-11 08:05:03.834273 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6810/15287 -- osd_ping(ping e509 stamp 2018-07-11 08:05:03.834106) v4 -- 0x55f663809200 con 0 > -7441> 2018-07-11 08:05:03.834292 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6811/15287 -- osd_ping(ping e509 stamp 2018-07-11 08:05:03.834106) v4 -- 0x55f663809400 con 0 > -7440> 2018-07-11 08:05:03.834306 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6806/15263 -- osd_ping(ping e509 stamp 2018-07-11 08:05:03.834106) v4 -- 0x55f663809600 con 0 > -7439> 2018-07-11 08:05:03.834321 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6807/15263 -- osd_ping(ping e509 stamp 2018-07-11 08:05:03.834106) v4 -- 0x55f663809800 con 0 > -7438> 2018-07-11 08:05:03.834333 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.115:6802/13323 -- osd_ping(ping e509 stamp 2018-07-11 08:05:03.834106) v4 -- 0x55f663809a00 con 0 > -7437> 2018-07-11 08:05:03.834345 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.115:6803/13323 -- osd_ping(ping e509 stamp 2018-07-11 08:05:03.834106) v4 -- 0x55f663809c00 con 0 > -7436> 2018-07-11 08:05:03.834358 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6806/15885 -- osd_ping(ping e509 stamp 2018-07-11 08:05:03.834106) v4 -- 0x55f663809e00 con 0 > -7435> 2018-07-11 08:05:03.834372 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6807/15885 -- osd_ping(ping e509 stamp 2018-07-11 08:05:03.834106) v4 -- 0x55f66c32c000 con 0 > -7434> 2018-07-11 08:05:03.834384 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.108:6802/11967 -- osd_ping(ping e509 stamp 2018-07-11 08:05:03.834106) v4 -- 0x55f66c32c200 con 0 > -7433> 2018-07-11 08:05:03.834396 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.108:6803/11967 -- osd_ping(ping e509 stamp 2018-07-11 08:05:03.834106) v4 -- 0x55f66c32c400 con 0 > -7432> 2018-07-11 08:05:03.834408 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.107:6806/12617 -- osd_ping(ping e509 stamp 2018-07-11 08:05:03.834106) v4 -- 0x55f66c32c600 con 0 > -7431> 2018-07-11 08:05:03.834422 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.107:6807/12617 -- osd_ping(ping e509 stamp 2018-07-11 08:05:03.834106) v4 -- 0x55f66c32c800 con 0 > -7430> 2018-07-11 08:05:03.834436 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6802/15919 -- osd_ping(ping e509 stamp 2018-07-11 08:05:03.834106) v4 -- 0x55f66c32ca00 con 0 > -7429> 2018-07-11 08:05:03.834448 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6803/15919 -- osd_ping(ping e509 stamp 2018-07-11 08:05:03.834106) v4 -- 0x55f66c32cc00 con 0 > -7428> 2018-07-11 08:05:03.834502 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6802/15255 conn(0x55f656ad4000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 193 0x55f660ea4c00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:03.834106) v4 > -7427> 2018-07-11 08:05:03.834523 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.0 10.8.128.113:6802/15255 193 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:03.834106) v4 ==== 2004+0+0 (1697753411 0 0) 0x55f660ea4c00 con 0x55f656ad4000 > -7426> 2018-07-11 08:05:03.834597 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.107:6806/12617 conn(0x55f656ade000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.9 seq 193 0x55f6743f6a00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:03.834106) v4 > -7425> 2018-07-11 08:05:03.834618 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.9 10.8.128.107:6806/12617 193 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:03.834106) v4 ==== 2004+0+0 (1697753411 0 0) 0x55f6743f6a00 con 0x55f656ade000 > -7424> 2018-07-11 08:05:03.834624 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6806/12677 conn(0x55f64e3d8000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.1 seq 193 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:03.834106) v4 > -7423> 2018-07-11 08:05:03.834643 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.1 10.8.128.106:6806/12677 193 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:03.834106) v4 ==== 2004+0+0 (1697753411 0 0) 0x55f67d1afe00 con 0x55f64e3d8000 > -7422> 2018-07-11 08:05:03.834676 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6803/15255 conn(0x55f656ad5800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 193 0x55f6743f6a00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:03.834106) v4 > -7421> 2018-07-11 08:05:03.834685 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6809/12650 conn(0x55f656ad7000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.3 seq 193 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:03.834106) v4 > -7420> 2018-07-11 08:05:03.834687 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6808/12651 conn(0x55f64e3ea000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.2 seq 193 0x55f660ea4c00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:03.834106) v4 > -7419> 2018-07-11 08:05:03.834696 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.3 10.8.128.106:6809/12650 193 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:03.834106) v4 ==== 2004+0+0 (1697753411 0 0) 0x55f67d1afe00 con 0x55f656ad7000 > -7418> 2018-07-11 08:05:03.834694 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.0 10.8.128.113:6803/15255 193 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:03.834106) v4 ==== 2004+0+0 (1697753411 0 0) 0x55f6743f6a00 con 0x55f656ad5800 > -7417> 2018-07-11 08:05:03.834703 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.2 10.8.128.106:6808/12651 193 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:03.834106) v4 ==== 2004+0+0 (1697753411 0 0) 0x55f660ea4c00 con 0x55f64e3ea000 > -7416> 2018-07-11 08:05:03.834740 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6810/12651 conn(0x55f64e3eb800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.2 seq 193 0x55f6743f6a00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:03.834106) v4 > -7415> 2018-07-11 08:05:03.834747 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6806/15263 conn(0x55f64e3ed000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.5 seq 193 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:03.834106) v4 > -7414> 2018-07-11 08:05:03.834753 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.2 10.8.128.106:6810/12651 193 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:03.834106) v4 ==== 2004+0+0 (1697753411 0 0) 0x55f6743f6a00 con 0x55f64e3eb800 > -7413> 2018-07-11 08:05:03.834763 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.5 10.8.128.113:6806/15263 193 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:03.834106) v4 ==== 2004+0+0 (1697753411 0 0) 0x55f67d1afe00 con 0x55f64e3ed000 > -7412> 2018-07-11 08:05:03.834773 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6811/12650 conn(0x55f656ad8800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.3 seq 193 0x55f660ea4c00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:03.834106) v4 > -7411> 2018-07-11 08:05:03.834787 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.3 10.8.128.106:6811/12650 193 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:03.834106) v4 ==== 2004+0+0 (1697753411 0 0) 0x55f660ea4c00 con 0x55f656ad8800 > -7410> 2018-07-11 08:05:03.834791 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.107:6807/12617 conn(0x55f656adf800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.9 seq 193 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:03.834106) v4 > -7409> 2018-07-11 08:05:03.834803 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.9 10.8.128.107:6807/12617 193 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:03.834106) v4 ==== 2004+0+0 (1697753411 0 0) 0x55f67d1afe00 con 0x55f656adf800 > -7408> 2018-07-11 08:05:03.834821 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6811/15287 conn(0x55f64e402800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.4 seq 193 0x55f6743f6a00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:03.834106) v4 > -7407> 2018-07-11 08:05:03.834833 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.115:6803/13323 conn(0x55f656ace800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.6 seq 193 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:03.834106) v4 > -7406> 2018-07-11 08:05:03.834835 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6810/15287 conn(0x55f64e401000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.4 seq 193 0x55f660ea4c00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:03.834106) v4 > -7405> 2018-07-11 08:05:03.834836 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.4 10.8.128.113:6811/15287 193 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:03.834106) v4 ==== 2004+0+0 (1697753411 0 0) 0x55f6743f6a00 con 0x55f64e402800 > -7404> 2018-07-11 08:05:03.834847 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.6 10.8.128.115:6803/13323 193 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:03.834106) v4 ==== 2004+0+0 (1697753411 0 0) 0x55f67d1afe00 con 0x55f656ace800 > -7403> 2018-07-11 08:05:03.834847 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.4 10.8.128.113:6810/15287 193 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:03.834106) v4 ==== 2004+0+0 (1697753411 0 0) 0x55f660ea4c00 con 0x55f64e401000 > -7402> 2018-07-11 08:05:03.834877 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6807/12677 conn(0x55f64e3d9800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.1 seq 193 0x55f660ea4c00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:03.834106) v4 > -7401> 2018-07-11 08:05:03.834889 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.1 10.8.128.106:6807/12677 193 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:03.834106) v4 ==== 2004+0+0 (1697753411 0 0) 0x55f660ea4c00 con 0x55f64e3d9800 > -7400> 2018-07-11 08:05:03.834889 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.115:6802/13323 conn(0x55f656acd000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.6 seq 193 0x55f6743f6a00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:03.834106) v4 > -7399> 2018-07-11 08:05:03.834901 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.6 10.8.128.115:6802/13323 193 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:03.834106) v4 ==== 2004+0+0 (1697753411 0 0) 0x55f6743f6a00 con 0x55f656acd000 > -7398> 2018-07-11 08:05:03.834935 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6807/15263 conn(0x55f64e3ee800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.5 seq 193 0x55f660ea4c00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:03.834106) v4 > -7397> 2018-07-11 08:05:03.834948 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.5 10.8.128.113:6807/15263 193 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:03.834106) v4 ==== 2004+0+0 (1697753411 0 0) 0x55f660ea4c00 con 0x55f64e3ee800 > -7396> 2018-07-11 08:05:03.834946 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6806/15885 conn(0x55f64e3fe000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.7 seq 193 0x55f6743f6a00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:03.834106) v4 > -7395> 2018-07-11 08:05:03.834949 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6803/15919 conn(0x55f64e3dc800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.11 seq 193 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:03.834106) v4 > -7394> 2018-07-11 08:05:03.834958 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.7 10.8.128.114:6806/15885 193 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:03.834106) v4 ==== 2004+0+0 (1697753411 0 0) 0x55f6743f6a00 con 0x55f64e3fe000 > -7393> 2018-07-11 08:05:03.834964 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.11 10.8.128.114:6803/15919 193 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:03.834106) v4 ==== 2004+0+0 (1697753411 0 0) 0x55f67d1afe00 con 0x55f64e3dc800 > -7392> 2018-07-11 08:05:03.834981 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.108:6803/11967 conn(0x55f656acb800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.8 seq 193 0x55f660ea4c00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:03.834106) v4 > -7391> 2018-07-11 08:05:03.834994 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.8 10.8.128.108:6803/11967 193 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:03.834106) v4 ==== 2004+0+0 (1697753411 0 0) 0x55f660ea4c00 con 0x55f656acb800 > -7390> 2018-07-11 08:05:03.834993 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6802/15919 conn(0x55f64e3db000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.11 seq 193 0x55f6743f6a00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:03.834106) v4 > -7389> 2018-07-11 08:05:03.835003 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.11 10.8.128.114:6802/15919 193 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:03.834106) v4 ==== 2004+0+0 (1697753411 0 0) 0x55f6743f6a00 con 0x55f64e3db000 > -7388> 2018-07-11 08:05:03.835001 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6807/15885 conn(0x55f64e3ff800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.7 seq 193 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:03.834106) v4 > -7387> 2018-07-11 08:05:03.835013 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.7 10.8.128.114:6807/15885 193 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:03.834106) v4 ==== 2004+0+0 (1697753411 0 0) 0x55f67d1afe00 con 0x55f64e3ff800 > -7386> 2018-07-11 08:05:03.835045 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.108:6802/11967 conn(0x55f656aca000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.8 seq 193 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:03.834106) v4 > -7385> 2018-07-11 08:05:03.835058 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.8 10.8.128.108:6802/11967 193 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:03.834106) v4 ==== 2004+0+0 (1697753411 0 0) 0x55f67d1afe00 con 0x55f656aca000 > -7384> 2018-07-11 08:05:03.949033 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12651 conn(0x55f663219000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25 cs=1 l=1). rx osd.2 seq 201 0x55f660ea4c00 osd_ping(ping e509 stamp 2018-07-11 08:05:04.005465) v4 > -7383> 2018-07-11 08:05:03.949055 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.2 10.8.128.106:0/12651 201 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:04.005465) v4 ==== 2004+0+0 (724355453 0 0) 0x55f660ea4c00 con 0x55f663219000 > -7382> 2018-07-11 08:05:03.949066 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:04.005465) v4 -- 0x55f64e12f600 con 0 > -7381> 2018-07-11 08:05:03.949105 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12651 conn(0x55f66321a800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25 cs=1 l=1). rx osd.2 seq 201 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:05:04.005465) v4 > -7380> 2018-07-11 08:05:03.949124 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.2 10.8.128.106:0/12651 201 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:04.005465) v4 ==== 2004+0+0 (724355453 0 0) 0x55f67d1afe00 con 0x55f66321a800 > -7379> 2018-07-11 08:05:03.949135 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:04.005465) v4 -- 0x55f663808600 con 0 > -7378> 2018-07-11 08:05:03.961850 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 417 0x55f6743f6a00 MOSDScrubReserve(11.3s5 REQUEST e509) v1 > -7377> 2018-07-11 08:05:03.961869 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 417 ==== MOSDScrubReserve(11.3s5 REQUEST e509) v1 ==== 43+0+0 (433425070 0 0) 0x55f6743f6a00 con 0x55f668a29000 > -7376> 2018-07-11 08:05:03.961950 7f491a040700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e509) v1 -- 0x55f660ea5a00 con 0 > -7375> 2018-07-11 08:05:03.962356 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 418 0x55f660ea5a00 MOSDScrubReserve(11.3s5 RELEASE e509) v1 > -7374> 2018-07-11 08:05:03.962371 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 418 ==== MOSDScrubReserve(11.3s5 RELEASE e509) v1 ==== 43+0+0 (3603460268 0 0) 0x55f660ea5a00 con 0x55f668a29000 > -7373> 2018-07-11 08:05:04.047922 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f67f495440 con 0 > -7372> 2018-07-11 08:05:04.049632 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 200 0x55f65a3f6f00 replica scrub(pg: 6.15,from:0'0,to:430'106387,epoch:509/504,start:6:a824b902:::benchmark_data_magna108_35617_object612676:0,end:6:a8256839:::benchmark_data_magna108_35617_object190264:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -7371> 2018-07-11 08:05:04.049663 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 200 ==== replica scrub(pg: 6.15,from:0'0,to:430'106387,epoch:509/504,start:6:a824b902:::benchmark_data_magna108_35617_object612676:0,end:6:a8256839:::benchmark_data_magna108_35617_object190264:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (531317014 0 0) 0x55f65a3f6f00 con 0x55f66315e800 > -7370> 2018-07-11 08:05:04.826909 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12650 conn(0x55f663eda800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.3 seq 191 0x55f64e12fc00 osd_ping(ping e509 stamp 2018-07-11 08:05:04.883207) v4 > -7369> 2018-07-11 08:05:04.826916 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12650 conn(0x55f663ed9000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.3 seq 191 0x55f653732e00 osd_ping(ping e509 stamp 2018-07-11 08:05:04.883207) v4 > -7368> 2018-07-11 08:05:04.826950 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.3 10.8.128.106:0/12650 191 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:04.883207) v4 ==== 2004+0+0 (2991501593 0 0) 0x55f64e12fc00 con 0x55f663eda800 > -7367> 2018-07-11 08:05:04.826952 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.3 10.8.128.106:0/12650 191 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:04.883207) v4 ==== 2004+0+0 (2991501593 0 0) 0x55f653732e00 con 0x55f663ed9000 > -7366> 2018-07-11 08:05:04.826960 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:04.883207) v4 -- 0x55f64e12f600 con 0 > -7365> 2018-07-11 08:05:04.826988 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:04.883207) v4 -- 0x55f66e945000 con 0 > -7364> 2018-07-11 08:05:04.962126 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 419 0x55f66e945a00 MOSDScrubReserve(11.3s5 REQUEST e509) v1 > -7363> 2018-07-11 08:05:04.962145 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 419 ==== MOSDScrubReserve(11.3s5 REQUEST e509) v1 ==== 43+0+0 (433425070 0 0) 0x55f66e945a00 con 0x55f668a29000 > -7362> 2018-07-11 08:05:04.962228 7f491a040700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e509) v1 -- 0x55f660ea5a00 con 0 > -7361> 2018-07-11 08:05:04.962522 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 420 0x55f66e944800 MOSDScrubReserve(11.3s5 RELEASE e509) v1 > -7360> 2018-07-11 08:05:04.962536 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 420 ==== MOSDScrubReserve(11.3s5 RELEASE e509) v1 ==== 43+0+0 (3603460268 0 0) 0x55f66e944800 con 0x55f668a29000 > -7359> 2018-07-11 08:05:04.999606 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15287 conn(0x55f66dce2800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.4 seq 198 0x55f660ea4c00 osd_ping(ping e509 stamp 2018-07-11 08:05:06.243685) v4 > -7358> 2018-07-11 08:05:04.999608 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15287 conn(0x55f66dce1000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.4 seq 198 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:05:06.243685) v4 > -7357> 2018-07-11 08:05:04.999630 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.4 10.8.128.113:0/15287 198 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:06.243685) v4 ==== 2004+0+0 (2793078526 0 0) 0x55f660ea4c00 con 0x55f66dce2800 > -7356> 2018-07-11 08:05:04.999635 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.4 10.8.128.113:0/15287 198 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:06.243685) v4 ==== 2004+0+0 (2793078526 0 0) 0x55f64e20fc00 con 0x55f66dce1000 > -7355> 2018-07-11 08:05:04.999640 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:06.243685) v4 -- 0x55f65e71c800 con 0 > -7354> 2018-07-11 08:05:04.999664 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:06.243685) v4 -- 0x55f660ea5c00 con 0 > -7353> 2018-07-11 08:05:05.106712 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.115:0/13323 conn(0x55f663ebf800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.6 seq 181 0x55f64e12fc00 osd_ping(ping e509 stamp 2018-07-11 08:05:05.459107) v4 > -7352> 2018-07-11 08:05:05.106738 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.6 10.8.128.115:0/13323 181 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:05.459107) v4 ==== 2004+0+0 (1969389890 0 0) 0x55f64e12fc00 con 0x55f663ebf800 > -7351> 2018-07-11 08:05:05.106748 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:05.459107) v4 -- 0x55f64f769c00 con 0 > -7350> 2018-07-11 08:05:05.106738 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.115:0/13323 conn(0x55f663ebe000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.6 seq 181 0x55f653732e00 osd_ping(ping e509 stamp 2018-07-11 08:05:05.459107) v4 > -7349> 2018-07-11 08:05:05.106759 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.6 10.8.128.115:0/13323 181 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:05.459107) v4 ==== 2004+0+0 (1969389890 0 0) 0x55f653732e00 con 0x55f663ebe000 > -7348> 2018-07-11 08:05:05.106778 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:05.459107) v4 -- 0x55f66d001600 con 0 > -7347> 2018-07-11 08:05:05.195414 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.108:0/11967 conn(0x55f663f6f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=1). rx osd.8 seq 184 0x55f660ea5a00 osd_ping(ping e509 stamp 2018-07-11 08:05:04.955135) v4 > -7346> 2018-07-11 08:05:05.195436 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.8 10.8.128.108:0/11967 184 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:04.955135) v4 ==== 2004+0+0 (1601527442 0 0) 0x55f660ea5a00 con 0x55f663f6f000 > -7345> 2018-07-11 08:05:05.195434 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.108:0/11967 conn(0x55f663f70800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=1). rx osd.8 seq 184 0x55f660ea4c00 osd_ping(ping e509 stamp 2018-07-11 08:05:04.955135) v4 > -7344> 2018-07-11 08:05:05.195448 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:04.955135) v4 -- 0x55f66d000e00 con 0 > -7343> 2018-07-11 08:05:05.195455 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.8 10.8.128.108:0/11967 184 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:04.955135) v4 ==== 2004+0+0 (1601527442 0 0) 0x55f660ea4c00 con 0x55f663f70800 > -7342> 2018-07-11 08:05:05.195466 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:04.955135) v4 -- 0x55f66d001200 con 0 > -7341> 2018-07-11 08:05:05.526671 7f491d046700 1 -- 10.8.128.107:6800/14173 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55f663502580 con 0 > -7340> 2018-07-11 08:05:05.526748 7f491d046700 1 -- 10.8.128.107:6800/14173 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55f6631f18c0 con 0 > -7339> 2018-07-11 08:05:05.535774 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 29] Generated table #37036: 2638241 keys, 74546667 bytes > -7338> 2018-07-11 08:05:05.535821 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296305535806, "cf_name": "default", "job": 29, "event": "table_file_creation", "file_number": 37036, "file_size": 74546667, "table_properties": {"data_size": 67112118, "index_size": 690376, "filter_size": 6743183, "raw_key_size": 76215855, "raw_average_key_size": 28, "raw_value_size": 31365758, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} > -7337> 2018-07-11 08:05:05.659881 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.114:6805/15885 conn(0x55f66315a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.7 seq 2535 0x55f64e12fc00 MOSDScrubReserve(5.4 REQUEST e509) v1 > -7336> 2018-07-11 08:05:05.659904 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.7 10.8.128.114:6805/15885 2535 ==== MOSDScrubReserve(5.4 REQUEST e509) v1 ==== 43+0+0 (3003630469 0 0) 0x55f64e12fc00 con 0x55f66315a000 > -7335> 2018-07-11 08:05:05.659987 7f491983f700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.114:6805/15885 -- MOSDScrubReserve(5.4 REJECT e509) v1 -- 0x55f660ea5800 con 0 > -7334> 2018-07-11 08:05:05.660274 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.114:6805/15885 conn(0x55f66315a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.7 seq 2536 0x55f660ea5800 MOSDScrubReserve(5.4 RELEASE e509) v1 > -7333> 2018-07-11 08:05:05.660290 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.7 10.8.128.114:6805/15885 2536 ==== MOSDScrubReserve(5.4 RELEASE e509) v1 ==== 43+0+0 (2082093447 0 0) 0x55f660ea5800 con 0x55f66315a000 > -7332> 2018-07-11 08:05:05.772955 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed7800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 192 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:05:05.829339) v4 > -7331> 2018-07-11 08:05:05.772977 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.1 10.8.128.106:0/12677 192 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:05.829339) v4 ==== 2004+0+0 (1834050786 0 0) 0x55f67d1afe00 con 0x55f663ed7800 > -7330> 2018-07-11 08:05:05.772989 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:05.829339) v4 -- 0x55f660ea5c00 con 0 > -7329> 2018-07-11 08:05:05.773059 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed6000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 192 0x55f660ea4c00 osd_ping(ping e509 stamp 2018-07-11 08:05:05.829339) v4 > -7328> 2018-07-11 08:05:05.773084 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.1 10.8.128.106:0/12677 192 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:05.829339) v4 ==== 2004+0+0 (1834050786 0 0) 0x55f660ea4c00 con 0x55f663ed6000 > -7327> 2018-07-11 08:05:05.773096 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:05.829339) v4 -- 0x55f6716b3000 con 0 > -7326> 2018-07-11 08:05:06.015433 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f666a978c0 con 0 > -7325> 2018-07-11 08:05:06.017180 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 201 0x55f65a3f7180 replica scrub(pg: 6.15,from:0'0,to:430'106860,epoch:509/504,start:6:a8256839:::benchmark_data_magna108_35617_object190264:0,end:6:a8266728:::benchmark_data_magna108_35617_object562173:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -7324> 2018-07-11 08:05:06.017206 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 201 ==== replica scrub(pg: 6.15,from:0'0,to:430'106860,epoch:509/504,start:6:a8256839:::benchmark_data_magna108_35617_object190264:0,end:6:a8266728:::benchmark_data_magna108_35617_object562173:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (4090353556 0 0) 0x55f65a3f7180 con 0x55f66315e800 > -7323> 2018-07-11 08:05:06.052023 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0d000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 196 0x55f66d001200 osd_ping(ping e509 stamp 2018-07-11 08:05:07.296262) v4 > -7322> 2018-07-11 08:05:06.052021 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0e800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 196 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:05:07.296262) v4 > -7321> 2018-07-11 08:05:06.052052 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.5 10.8.128.113:0/15263 196 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:07.296262) v4 ==== 2004+0+0 (1861773842 0 0) 0x55f66d001200 con 0x55f663f0d000 > -7320> 2018-07-11 08:05:06.052056 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.5 10.8.128.113:0/15263 196 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:07.296262) v4 ==== 2004+0+0 (1861773842 0 0) 0x55f64e20fc00 con 0x55f663f0e800 > -7319> 2018-07-11 08:05:06.052064 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:07.296262) v4 -- 0x55f6716b2e00 con 0 > -7318> 2018-07-11 08:05:06.052097 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:07.296262) v4 -- 0x55f660ea5c00 con 0 > -7317> 2018-07-11 08:05:06.207122 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.115:0/13323 conn(0x55f663ebf800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.6 seq 182 0x55f6716b3000 osd_ping(ping e509 stamp 2018-07-11 08:05:06.559579) v4 > -7316> 2018-07-11 08:05:06.207144 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.6 10.8.128.115:0/13323 182 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:06.559579) v4 ==== 2004+0+0 (1222460091 0 0) 0x55f6716b3000 con 0x55f663ebf800 > -7315> 2018-07-11 08:05:06.207155 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:06.559579) v4 -- 0x55f6716b2e00 con 0 > -7314> 2018-07-11 08:05:06.207295 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.115:0/13323 conn(0x55f663ebe000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.6 seq 182 0x55f653732e00 osd_ping(ping e509 stamp 2018-07-11 08:05:06.559579) v4 > -7313> 2018-07-11 08:05:06.207314 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.6 10.8.128.115:0/13323 182 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:06.559579) v4 ==== 2004+0+0 (1222460091 0 0) 0x55f653732e00 con 0x55f663ebe000 > -7312> 2018-07-11 08:05:06.207327 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:06.559579) v4 -- 0x55f6716b2a00 con 0 > -7311> 2018-07-11 08:05:06.552430 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0d000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 197 0x55f66d001200 osd_ping(ping e509 stamp 2018-07-11 08:05:07.796669) v4 > -7310> 2018-07-11 08:05:06.552452 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.5 10.8.128.113:0/15263 197 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:07.796669) v4 ==== 2004+0+0 (3056076615 0 0) 0x55f66d001200 con 0x55f663f0d000 > -7309> 2018-07-11 08:05:06.552461 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:07.796669) v4 -- 0x55f6716b2800 con 0 > -7308> 2018-07-11 08:05:06.552493 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0e800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 197 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:05:07.796669) v4 > -7307> 2018-07-11 08:05:06.552519 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.5 10.8.128.113:0/15263 197 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:07.796669) v4 ==== 2004+0+0 (3056076615 0 0) 0x55f67d1afe00 con 0x55f663f0e800 > -7306> 2018-07-11 08:05:06.552536 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:07.796669) v4 -- 0x55f660ea5600 con 0 > -7305> 2018-07-11 08:05:06.629756 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15885 conn(0x55f663217800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 195 0x55f660ea5a00 osd_ping(ping e509 stamp 2018-07-11 08:05:07.265160) v4 > -7304> 2018-07-11 08:05:06.629765 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15885 conn(0x55f663216000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 195 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:05:07.265160) v4 > -7303> 2018-07-11 08:05:06.629776 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.7 10.8.128.114:0/15885 195 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:07.265160) v4 ==== 2004+0+0 (373076314 0 0) 0x55f660ea5a00 con 0x55f663217800 > -7302> 2018-07-11 08:05:06.629780 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.7 10.8.128.114:0/15885 195 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:07.265160) v4 ==== 2004+0+0 (373076314 0 0) 0x55f64e20fc00 con 0x55f663216000 > -7301> 2018-07-11 08:05:06.629786 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:07.265160) v4 -- 0x55f64e25d400 con 0 > -7300> 2018-07-11 08:05:06.629812 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:07.265160) v4 -- 0x55f660ea5400 con 0 > -7299> 2018-07-11 08:05:06.773153 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15255 conn(0x55f663ec2800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.0 seq 185 0x55f653732e00 osd_ping(ping e509 stamp 2018-07-11 08:05:08.017297) v4 > -7298> 2018-07-11 08:05:06.773159 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15255 conn(0x55f663ec1000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.0 seq 185 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:05:08.017297) v4 > -7297> 2018-07-11 08:05:06.773174 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.0 10.8.128.113:0/15255 185 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:08.017297) v4 ==== 2004+0+0 (2571444199 0 0) 0x55f653732e00 con 0x55f663ec2800 > -7296> 2018-07-11 08:05:06.773179 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.0 10.8.128.113:0/15255 185 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:08.017297) v4 ==== 2004+0+0 (2571444199 0 0) 0x55f67d1afe00 con 0x55f663ec1000 > -7295> 2018-07-11 08:05:06.773186 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:08.017297) v4 -- 0x55f6548fb000 con 0 > -7294> 2018-07-11 08:05:06.773219 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:08.017297) v4 -- 0x55f660ea5000 con 0 > -7293> 2018-07-11 08:05:06.895781 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.108:0/11967 conn(0x55f663f6f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=1). rx osd.8 seq 185 0x55f660ea5a00 osd_ping(ping e509 stamp 2018-07-11 08:05:06.655602) v4 > -7292> 2018-07-11 08:05:06.895814 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.8 10.8.128.108:0/11967 185 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:06.655602) v4 ==== 2004+0+0 (3849848670 0 0) 0x55f660ea5a00 con 0x55f663f6f000 > -7291> 2018-07-11 08:05:06.895824 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:06.655602) v4 -- 0x55f6548fa000 con 0 > -7290> 2018-07-11 08:05:06.895876 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.108:0/11967 conn(0x55f663f70800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=1). rx osd.8 seq 185 0x55f6716b3000 osd_ping(ping e509 stamp 2018-07-11 08:05:06.655602) v4 > -7289> 2018-07-11 08:05:06.895900 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.8 10.8.128.108:0/11967 185 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:06.655602) v4 ==== 2004+0+0 (3849848670 0 0) 0x55f6716b3000 con 0x55f663f70800 > -7288> 2018-07-11 08:05:06.895912 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:06.655602) v4 -- 0x55f6716b2800 con 0 > -7287> 2018-07-11 08:05:06.989591 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f65bd71b00 con 0 > -7286> 2018-07-11 08:05:06.991353 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 202 0x55f65a3f7400 replica scrub(pg: 6.15,from:0'0,to:430'106525,epoch:509/504,start:6:a8266728:::benchmark_data_magna108_35617_object562173:0,end:6:a827c7f6:::benchmark_data_magna108_35617_object280401:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -7285> 2018-07-11 08:05:06.991385 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 202 ==== replica scrub(pg: 6.15,from:0'0,to:430'106525,epoch:509/504,start:6:a8266728:::benchmark_data_magna108_35617_object562173:0,end:6:a827c7f6:::benchmark_data_magna108_35617_object280401:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (3164964927 0 0) 0x55f65a3f7400 con 0x55f66315e800 > -7284> 2018-07-11 08:05:07.099865 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0a000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 187 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:05:07.735333) v4 > -7283> 2018-07-11 08:05:07.099890 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.11 10.8.128.114:0/15919 187 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:07.735333) v4 ==== 2004+0+0 (1954944186 0 0) 0x55f64e20fc00 con 0x55f663f0a000 > -7282> 2018-07-11 08:05:07.099906 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:07.735333) v4 -- 0x55f660ea5000 con 0 > -7281> 2018-07-11 08:05:07.099921 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0b800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 187 0x55f653732e00 osd_ping(ping e509 stamp 2018-07-11 08:05:07.735333) v4 > -7280> 2018-07-11 08:05:07.099942 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.11 10.8.128.114:0/15919 187 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:07.735333) v4 ==== 2004+0+0 (1954944186 0 0) 0x55f653732e00 con 0x55f663f0b800 > -7279> 2018-07-11 08:05:07.099953 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:07.735333) v4 -- 0x55f6548fa000 con 0 > -7278> 2018-07-11 08:05:07.181197 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.107:0/12617 conn(0x55f64e3c3800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.9 seq 191 0x55f660ea5a00 osd_ping(ping e509 stamp 2018-07-11 08:05:07.180846) v4 > -7277> 2018-07-11 08:05:07.181196 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.107:0/12617 conn(0x55f64e3c5000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.9 seq 191 0x55f66d001200 osd_ping(ping e509 stamp 2018-07-11 08:05:07.180846) v4 > -7276> 2018-07-11 08:05:07.181221 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.9 10.8.128.107:0/12617 191 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:07.180846) v4 ==== 2004+0+0 (2934005343 0 0) 0x55f660ea5a00 con 0x55f64e3c3800 > -7275> 2018-07-11 08:05:07.181242 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:07.180846) v4 -- 0x55f64f769000 con 0 > -7274> 2018-07-11 08:05:07.181236 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.9 10.8.128.107:0/12617 191 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:07.180846) v4 ==== 2004+0+0 (2934005343 0 0) 0x55f66d001200 con 0x55f64e3c5000 > -7273> 2018-07-11 08:05:07.181270 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:07.180846) v4 -- 0x55f65c0f2a00 con 0 > -7272> 2018-07-11 08:05:07.652965 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0d000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 198 0x55f6716b2800 osd_ping(ping e509 stamp 2018-07-11 08:05:08.897185) v4 > -7271> 2018-07-11 08:05:07.653025 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.5 10.8.128.113:0/15263 198 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:08.897185) v4 ==== 2004+0+0 (42851425 0 0) 0x55f6716b2800 con 0x55f663f0d000 > -7270> 2018-07-11 08:05:07.653045 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:08.897185) v4 -- 0x55f65c0f2c00 con 0 > -7269> 2018-07-11 08:05:07.653078 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0e800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 198 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:05:08.897185) v4 > -7268> 2018-07-11 08:05:07.653100 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.5 10.8.128.113:0/15263 198 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:08.897185) v4 ==== 2004+0+0 (42851425 0 0) 0x55f67d1afe00 con 0x55f663f0e800 > -7267> 2018-07-11 08:05:07.653111 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:08.897185) v4 -- 0x55f660ea4a00 con 0 > -7266> 2018-07-11 08:05:07.660003 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.114:6805/15885 conn(0x55f66315a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.7 seq 2537 0x55f66d001200 MOSDScrubReserve(5.4 REQUEST e509) v1 > -7265> 2018-07-11 08:05:07.660020 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.7 10.8.128.114:6805/15885 2537 ==== MOSDScrubReserve(5.4 REQUEST e509) v1 ==== 43+0+0 (3003630469 0 0) 0x55f66d001200 con 0x55f66315a000 > -7264> 2018-07-11 08:05:07.660099 7f491983f700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.114:6805/15885 -- MOSDScrubReserve(5.4 REJECT e509) v1 -- 0x55f660ea5800 con 0 > -7263> 2018-07-11 08:05:07.660413 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.114:6805/15885 conn(0x55f66315a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.7 seq 2538 0x55f660ea5800 MOSDScrubReserve(5.4 RELEASE e509) v1 > -7262> 2018-07-11 08:05:07.660428 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.7 10.8.128.114:6805/15885 2538 ==== MOSDScrubReserve(5.4 RELEASE e509) v1 ==== 43+0+0 (2082093447 0 0) 0x55f660ea5800 con 0x55f66315a000 > -7261> 2018-07-11 08:05:07.727305 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12650 conn(0x55f663eda800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.3 seq 192 0x55f6716b2800 osd_ping(ping e509 stamp 2018-07-11 08:05:07.783770) v4 > -7260> 2018-07-11 08:05:07.727330 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.3 10.8.128.106:0/12650 192 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:07.783770) v4 ==== 2004+0+0 (3360581605 0 0) 0x55f6716b2800 con 0x55f663eda800 > -7259> 2018-07-11 08:05:07.727344 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:07.783770) v4 -- 0x55f65c0f3200 con 0 > -7258> 2018-07-11 08:05:07.727348 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12650 conn(0x55f663ed9000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.3 seq 192 0x55f653732e00 osd_ping(ping e509 stamp 2018-07-11 08:05:07.783770) v4 > -7257> 2018-07-11 08:05:07.727379 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.3 10.8.128.106:0/12650 192 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:07.783770) v4 ==== 2004+0+0 (3360581605 0 0) 0x55f653732e00 con 0x55f663ed9000 > -7256> 2018-07-11 08:05:07.727394 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:07.783770) v4 -- 0x55f64f769000 con 0 > -7255> 2018-07-11 08:05:07.873529 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15255 conn(0x55f663ec1000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.0 seq 186 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:05:09.117846) v4 > -7254> 2018-07-11 08:05:07.873550 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.0 10.8.128.113:0/15255 186 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:09.117846) v4 ==== 2004+0+0 (511178987 0 0) 0x55f64e20fc00 con 0x55f663ec1000 > -7253> 2018-07-11 08:05:07.873562 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:09.117846) v4 -- 0x55f660ea4a00 con 0 > -7252> 2018-07-11 08:05:07.873569 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15255 conn(0x55f663ec2800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.0 seq 186 0x55f660ea5a00 osd_ping(ping e509 stamp 2018-07-11 08:05:09.117846) v4 > -7251> 2018-07-11 08:05:07.873590 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.0 10.8.128.113:0/15255 186 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:09.117846) v4 ==== 2004+0+0 (511178987 0 0) 0x55f660ea5a00 con 0x55f663ec2800 > -7250> 2018-07-11 08:05:07.873602 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:09.117846) v4 -- 0x55f6548fb800 con 0 > -7249> 2018-07-11 08:05:07.962534 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 421 0x55f653732e00 MOSDScrubReserve(11.3s5 REQUEST e509) v1 > -7248> 2018-07-11 08:05:07.962553 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 421 ==== MOSDScrubReserve(11.3s5 REQUEST e509) v1 ==== 43+0+0 (433425070 0 0) 0x55f653732e00 con 0x55f668a29000 > -7247> 2018-07-11 08:05:07.962624 7f491a040700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e509) v1 -- 0x55f66e944800 con 0 > -7246> 2018-07-11 08:05:07.962979 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 422 0x55f66e944800 MOSDScrubReserve(11.3s5 RELEASE e509) v1 > -7245> 2018-07-11 08:05:07.962995 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 422 ==== MOSDScrubReserve(11.3s5 RELEASE e509) v1 ==== 43+0+0 (3603460268 0 0) 0x55f66e944800 con 0x55f668a29000 > -7244> 2018-07-11 08:05:08.073406 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed7800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 193 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:05:08.129929) v4 > -7243> 2018-07-11 08:05:08.073409 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed6000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 193 0x55f65c0f3000 osd_ping(ping e509 stamp 2018-07-11 08:05:08.129929) v4 > -7242> 2018-07-11 08:05:08.073426 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.1 10.8.128.106:0/12677 193 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:08.129929) v4 ==== 2004+0+0 (3906023395 0 0) 0x55f67d1afe00 con 0x55f663ed7800 > -7241> 2018-07-11 08:05:08.073429 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.1 10.8.128.106:0/12677 193 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:08.129929) v4 ==== 2004+0+0 (3906023395 0 0) 0x55f65c0f3000 con 0x55f663ed6000 > -7240> 2018-07-11 08:05:08.073436 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:08.129929) v4 -- 0x55f660ea4200 con 0 > -7239> 2018-07-11 08:05:08.073458 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:08.129929) v4 -- 0x55f65c0f3400 con 0 > -7238> 2018-07-11 08:05:08.188940 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f65fcfbd40 con 0 > -7237> 2018-07-11 08:05:08.190498 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 203 0x55f65a3f7680 replica scrub(pg: 6.15,from:0'0,to:430'106617,epoch:509/504,start:6:a827c7f6:::benchmark_data_magna108_35617_object280401:0,end:6:a8289215:::benchmark_data_magna108.ceph.redhat.com_2478_object371296:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -7236> 2018-07-11 08:05:08.190528 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 203 ==== replica scrub(pg: 6.15,from:0'0,to:430'106617,epoch:509/504,start:6:a827c7f6:::benchmark_data_magna108_35617_object280401:0,end:6:a8289215:::benchmark_data_magna108.ceph.redhat.com_2478_object371296:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (3097379164 0 0) 0x55f65a3f7680 con 0x55f66315e800 > -7235> 2018-07-11 08:05:08.596216 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.108:0/11967 conn(0x55f663f6f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=1). rx osd.8 seq 186 0x55f660ea5a00 osd_ping(ping e509 stamp 2018-07-11 08:05:08.356046) v4 > -7234> 2018-07-11 08:05:08.596269 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.8 10.8.128.108:0/11967 186 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:08.356046) v4 ==== 2004+0+0 (1669481280 0 0) 0x55f660ea5a00 con 0x55f663f6f000 > -7233> 2018-07-11 08:05:08.596287 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:08.356046) v4 -- 0x55f653d4a400 con 0 > -7232> 2018-07-11 08:05:08.596322 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.108:0/11967 conn(0x55f663f70800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=1). rx osd.8 seq 186 0x55f6716b2800 osd_ping(ping e509 stamp 2018-07-11 08:05:08.356046) v4 > -7231> 2018-07-11 08:05:08.596343 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.8 10.8.128.108:0/11967 186 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:08.356046) v4 ==== 2004+0+0 (1669481280 0 0) 0x55f6716b2800 con 0x55f663f70800 > -7230> 2018-07-11 08:05:08.596355 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:08.356046) v4 -- 0x55f65c0f3600 con 0 > -7229> 2018-07-11 08:05:08.800343 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0a000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 188 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:05:09.435838) v4 > -7228> 2018-07-11 08:05:08.800365 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.11 10.8.128.114:0/15919 188 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:09.435838) v4 ==== 2004+0+0 (2554682997 0 0) 0x55f64e20fc00 con 0x55f663f0a000 > -7227> 2018-07-11 08:05:08.800377 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:09.435838) v4 -- 0x55f660ea4200 con 0 > -7226> 2018-07-11 08:05:08.800374 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0b800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 188 0x55f66b1cca00 osd_ping(ping e509 stamp 2018-07-11 08:05:09.435838) v4 > -7225> 2018-07-11 08:05:08.800399 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.11 10.8.128.114:0/15919 188 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:09.435838) v4 ==== 2004+0+0 (2554682997 0 0) 0x55f66b1cca00 con 0x55f663f0b800 > -7224> 2018-07-11 08:05:08.800414 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:09.435838) v4 -- 0x55f653d4a400 con 0 > -7223> 2018-07-11 08:05:09.247756 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f66c7ef680 con 0 > -7222> 2018-07-11 08:05:09.249516 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 204 0x55f65a3f7900 replica scrub(pg: 6.15,from:0'0,to:430'105734,epoch:509/504,start:6:a8289215:::benchmark_data_magna108.ceph.redhat.com_2478_object371296:0,end:6:a82943ac:::benchmark_data_magna108.ceph.redhat.com_2478_object80981:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -7221> 2018-07-11 08:05:09.249552 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 204 ==== replica scrub(pg: 6.15,from:0'0,to:430'105734,epoch:509/504,start:6:a8289215:::benchmark_data_magna108.ceph.redhat.com_2478_object371296:0,end:6:a82943ac:::benchmark_data_magna108.ceph.redhat.com_2478_object80981:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 247+0+0 (3631068549 0 0) 0x55f65a3f7900 con 0x55f66315e800 > -7220> 2018-07-11 08:05:09.300732 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0a000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 189 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:05:09.936215) v4 > -7219> 2018-07-11 08:05:09.300754 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.11 10.8.128.114:0/15919 189 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:09.936215) v4 ==== 2004+0+0 (2657260356 0 0) 0x55f67d1afe00 con 0x55f663f0a000 > -7218> 2018-07-11 08:05:09.300764 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:09.936215) v4 -- 0x55f660ea4200 con 0 > -7217> 2018-07-11 08:05:09.300745 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0b800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 189 0x55f660ea5a00 osd_ping(ping e509 stamp 2018-07-11 08:05:09.936215) v4 > -7216> 2018-07-11 08:05:09.300775 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.11 10.8.128.114:0/15919 189 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:09.936215) v4 ==== 2004+0+0 (2657260356 0 0) 0x55f660ea5a00 con 0x55f663f0b800 > -7215> 2018-07-11 08:05:09.300792 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:09.936215) v4 -- 0x55f653d4a400 con 0 > -7214> 2018-07-11 08:05:09.734606 7f491803c700 5 osd.10 509 heartbeat: osd_stat(789 GB used, 142 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,0,1]) > -7213> 2018-07-11 08:05:09.734628 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6802/15255 -- osd_ping(ping e509 stamp 2018-07-11 08:05:09.734626) v4 -- 0x55f66c32ce00 con 0 > -7212> 2018-07-11 08:05:09.734678 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6803/15255 -- osd_ping(ping e509 stamp 2018-07-11 08:05:09.734626) v4 -- 0x55f66c32d000 con 0 > -7211> 2018-07-11 08:05:09.734695 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6806/12677 -- osd_ping(ping e509 stamp 2018-07-11 08:05:09.734626) v4 -- 0x55f66c32d200 con 0 > -7210> 2018-07-11 08:05:09.734710 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6807/12677 -- osd_ping(ping e509 stamp 2018-07-11 08:05:09.734626) v4 -- 0x55f66c32d400 con 0 > -7209> 2018-07-11 08:05:09.734723 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6808/12651 -- osd_ping(ping e509 stamp 2018-07-11 08:05:09.734626) v4 -- 0x55f66c32d600 con 0 > -7208> 2018-07-11 08:05:09.734735 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6810/12651 -- osd_ping(ping e509 stamp 2018-07-11 08:05:09.734626) v4 -- 0x55f66c32d800 con 0 > -7207> 2018-07-11 08:05:09.734747 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6809/12650 -- osd_ping(ping e509 stamp 2018-07-11 08:05:09.734626) v4 -- 0x55f66c32da00 con 0 > -7206> 2018-07-11 08:05:09.734759 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6811/12650 -- osd_ping(ping e509 stamp 2018-07-11 08:05:09.734626) v4 -- 0x55f66c32dc00 con 0 > -7205> 2018-07-11 08:05:09.734771 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6810/15287 -- osd_ping(ping e509 stamp 2018-07-11 08:05:09.734626) v4 -- 0x55f66c32de00 con 0 > -7204> 2018-07-11 08:05:09.734789 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6811/15287 -- osd_ping(ping e509 stamp 2018-07-11 08:05:09.734626) v4 -- 0x55f67667e000 con 0 > -7203> 2018-07-11 08:05:09.734803 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6806/15263 -- osd_ping(ping e509 stamp 2018-07-11 08:05:09.734626) v4 -- 0x55f67667e200 con 0 > -7202> 2018-07-11 08:05:09.734817 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6807/15263 -- osd_ping(ping e509 stamp 2018-07-11 08:05:09.734626) v4 -- 0x55f67667e400 con 0 > -7201> 2018-07-11 08:05:09.734832 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.115:6802/13323 -- osd_ping(ping e509 stamp 2018-07-11 08:05:09.734626) v4 -- 0x55f67667e600 con 0 > -7200> 2018-07-11 08:05:09.734846 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.115:6803/13323 -- osd_ping(ping e509 stamp 2018-07-11 08:05:09.734626) v4 -- 0x55f67667e800 con 0 > -7199> 2018-07-11 08:05:09.734859 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6806/15885 -- osd_ping(ping e509 stamp 2018-07-11 08:05:09.734626) v4 -- 0x55f67667ea00 con 0 > -7198> 2018-07-11 08:05:09.734871 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6807/15885 -- osd_ping(ping e509 stamp 2018-07-11 08:05:09.734626) v4 -- 0x55f67667ec00 con 0 > -7197> 2018-07-11 08:05:09.734885 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.108:6802/11967 -- osd_ping(ping e509 stamp 2018-07-11 08:05:09.734626) v4 -- 0x55f67667ee00 con 0 > -7196> 2018-07-11 08:05:09.734897 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.108:6803/11967 -- osd_ping(ping e509 stamp 2018-07-11 08:05:09.734626) v4 -- 0x55f67667f000 con 0 > -7195> 2018-07-11 08:05:09.734910 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.107:6806/12617 -- osd_ping(ping e509 stamp 2018-07-11 08:05:09.734626) v4 -- 0x55f67667f200 con 0 > -7194> 2018-07-11 08:05:09.734922 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.107:6807/12617 -- osd_ping(ping e509 stamp 2018-07-11 08:05:09.734626) v4 -- 0x55f67667f400 con 0 > -7193> 2018-07-11 08:05:09.734935 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6802/15919 -- osd_ping(ping e509 stamp 2018-07-11 08:05:09.734626) v4 -- 0x55f67667f600 con 0 > -7192> 2018-07-11 08:05:09.734947 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6803/15919 -- osd_ping(ping e509 stamp 2018-07-11 08:05:09.734626) v4 -- 0x55f67667f800 con 0 > -7191> 2018-07-11 08:05:09.735114 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6802/15255 conn(0x55f656ad4000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 194 0x55f65c0f3600 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:09.734626) v4 > -7190> 2018-07-11 08:05:09.735137 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.0 10.8.128.113:6802/15255 194 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:09.734626) v4 ==== 2004+0+0 (2169298035 0 0) 0x55f65c0f3600 con 0x55f656ad4000 > -7189> 2018-07-11 08:05:09.735199 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6808/12651 conn(0x55f64e3ea000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.2 seq 194 0x55f65c0f3600 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:09.734626) v4 > -7188> 2018-07-11 08:05:09.735179 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6803/15255 conn(0x55f656ad5800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 194 0x55f660ea5a00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:09.734626) v4 > -7187> 2018-07-11 08:05:09.735215 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.2 10.8.128.106:6808/12651 194 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:09.734626) v4 ==== 2004+0+0 (2169298035 0 0) 0x55f65c0f3600 con 0x55f64e3ea000 > -7186> 2018-07-11 08:05:09.735219 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.0 10.8.128.113:6803/15255 194 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:09.734626) v4 ==== 2004+0+0 (2169298035 0 0) 0x55f660ea5a00 con 0x55f656ad5800 > -7185> 2018-07-11 08:05:09.735219 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.107:6807/12617 conn(0x55f656adf800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.9 seq 194 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:09.734626) v4 > -7184> 2018-07-11 08:05:09.735239 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.9 10.8.128.107:6807/12617 194 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:09.734626) v4 ==== 2004+0+0 (2169298035 0 0) 0x55f64e20fc00 con 0x55f656adf800 > -7183> 2018-07-11 08:05:09.735300 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6809/12650 conn(0x55f656ad7000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.3 seq 194 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:09.734626) v4 > -7182> 2018-07-11 08:05:09.735303 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6807/12677 conn(0x55f64e3d9800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.1 seq 194 0x55f65c0f3600 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:09.734626) v4 > -7181> 2018-07-11 08:05:09.735271 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6811/15287 conn(0x55f64e402800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.4 seq 194 0x55f660ea5a00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:09.734626) v4 > -7180> 2018-07-11 08:05:09.735317 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.1 10.8.128.106:6807/12677 194 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:09.734626) v4 ==== 2004+0+0 (2169298035 0 0) 0x55f65c0f3600 con 0x55f64e3d9800 > -7179> 2018-07-11 08:05:09.735318 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.3 10.8.128.106:6809/12650 194 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:09.734626) v4 ==== 2004+0+0 (2169298035 0 0) 0x55f64e20fc00 con 0x55f656ad7000 > -7178> 2018-07-11 08:05:09.735321 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.4 10.8.128.113:6811/15287 194 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:09.734626) v4 ==== 2004+0+0 (2169298035 0 0) 0x55f660ea5a00 con 0x55f64e402800 > -7177> 2018-07-11 08:05:09.735350 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6810/15287 conn(0x55f64e401000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.4 seq 194 0x55f65c0f3600 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:09.734626) v4 > -7176> 2018-07-11 08:05:09.735362 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.4 10.8.128.113:6810/15287 194 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:09.734626) v4 ==== 2004+0+0 (2169298035 0 0) 0x55f65c0f3600 con 0x55f64e401000 > -7175> 2018-07-11 08:05:09.735349 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6806/12677 conn(0x55f64e3d8000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.1 seq 194 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:09.734626) v4 > -7174> 2018-07-11 08:05:09.735371 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.107:6806/12617 conn(0x55f656ade000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.9 seq 194 0x55f660ea5a00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:09.734626) v4 > -7173> 2018-07-11 08:05:09.735375 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.1 10.8.128.106:6806/12677 194 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:09.734626) v4 ==== 2004+0+0 (2169298035 0 0) 0x55f64e20fc00 con 0x55f64e3d8000 > -7172> 2018-07-11 08:05:09.735388 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.9 10.8.128.107:6806/12617 194 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:09.734626) v4 ==== 2004+0+0 (2169298035 0 0) 0x55f660ea5a00 con 0x55f656ade000 > -7171> 2018-07-11 08:05:09.735407 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6806/15263 conn(0x55f64e3ed000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.5 seq 194 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:09.734626) v4 > -7170> 2018-07-11 08:05:09.735418 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.5 10.8.128.113:6806/15263 194 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:09.734626) v4 ==== 2004+0+0 (2169298035 0 0) 0x55f64e20fc00 con 0x55f64e3ed000 > -7169> 2018-07-11 08:05:09.735415 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6811/12650 conn(0x55f656ad8800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.3 seq 194 0x55f65c0f3600 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:09.734626) v4 > -7168> 2018-07-11 08:05:09.735427 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.3 10.8.128.106:6811/12650 194 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:09.734626) v4 ==== 2004+0+0 (2169298035 0 0) 0x55f65c0f3600 con 0x55f656ad8800 > -7167> 2018-07-11 08:05:09.735427 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6810/12651 conn(0x55f64e3eb800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.2 seq 194 0x55f660ea5a00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:09.734626) v4 > -7166> 2018-07-11 08:05:09.735437 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.2 10.8.128.106:6810/12651 194 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:09.734626) v4 ==== 2004+0+0 (2169298035 0 0) 0x55f660ea5a00 con 0x55f64e3eb800 > -7165> 2018-07-11 08:05:09.735458 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6807/15263 conn(0x55f64e3ee800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.5 seq 194 0x55f65c0f3600 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:09.734626) v4 > -7164> 2018-07-11 08:05:09.735461 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.115:6803/13323 conn(0x55f656ace800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.6 seq 194 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:09.734626) v4 > -7163> 2018-07-11 08:05:09.735468 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.115:6802/13323 conn(0x55f656acd000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.6 seq 194 0x55f660ea5a00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:09.734626) v4 > -7162> 2018-07-11 08:05:09.735480 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.5 10.8.128.113:6807/15263 194 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:09.734626) v4 ==== 2004+0+0 (2169298035 0 0) 0x55f65c0f3600 con 0x55f64e3ee800 > -7161> 2018-07-11 08:05:09.735491 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.6 10.8.128.115:6803/13323 194 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:09.734626) v4 ==== 2004+0+0 (2169298035 0 0) 0x55f64e20fc00 con 0x55f656ace800 > -7160> 2018-07-11 08:05:09.735494 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.6 10.8.128.115:6802/13323 194 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:09.734626) v4 ==== 2004+0+0 (2169298035 0 0) 0x55f660ea5a00 con 0x55f656acd000 > -7159> 2018-07-11 08:05:09.735522 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.108:6803/11967 conn(0x55f656acb800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.8 seq 194 0x55f65c0f3600 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:09.734626) v4 > -7158> 2018-07-11 08:05:09.735535 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.8 10.8.128.108:6803/11967 194 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:09.734626) v4 ==== 2004+0+0 (2169298035 0 0) 0x55f65c0f3600 con 0x55f656acb800 > -7157> 2018-07-11 08:05:09.735532 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6807/15885 conn(0x55f64e3ff800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.7 seq 194 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:09.734626) v4 > -7156> 2018-07-11 08:05:09.735542 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6806/15885 conn(0x55f64e3fe000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.7 seq 194 0x55f660ea5a00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:09.734626) v4 > -7155> 2018-07-11 08:05:09.735551 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.7 10.8.128.114:6807/15885 194 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:09.734626) v4 ==== 2004+0+0 (2169298035 0 0) 0x55f64e20fc00 con 0x55f64e3ff800 > -7154> 2018-07-11 08:05:09.735557 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.7 10.8.128.114:6806/15885 194 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:09.734626) v4 ==== 2004+0+0 (2169298035 0 0) 0x55f660ea5a00 con 0x55f64e3fe000 > -7153> 2018-07-11 08:05:09.735580 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.108:6802/11967 conn(0x55f656aca000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.8 seq 194 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:09.734626) v4 > -7152> 2018-07-11 08:05:09.735593 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.8 10.8.128.108:6802/11967 194 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:09.734626) v4 ==== 2004+0+0 (2169298035 0 0) 0x55f64e20fc00 con 0x55f656aca000 > -7151> 2018-07-11 08:05:09.735594 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6802/15919 conn(0x55f64e3db000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.11 seq 194 0x55f660ea5a00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:09.734626) v4 > -7150> 2018-07-11 08:05:09.735606 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.11 10.8.128.114:6802/15919 194 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:09.734626) v4 ==== 2004+0+0 (2169298035 0 0) 0x55f660ea5a00 con 0x55f64e3db000 > -7149> 2018-07-11 08:05:09.735634 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6803/15919 conn(0x55f64e3dc800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.11 seq 194 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:09.734626) v4 > -7148> 2018-07-11 08:05:09.735646 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.11 10.8.128.114:6803/15919 194 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:09.734626) v4 ==== 2004+0+0 (2169298035 0 0) 0x55f64e20fc00 con 0x55f64e3dc800 > -7147> 2018-07-11 08:05:09.849635 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12651 conn(0x55f663219000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25 cs=1 l=1). rx osd.2 seq 202 0x55f65c0f3600 osd_ping(ping e509 stamp 2018-07-11 08:05:09.905988) v4 > -7146> 2018-07-11 08:05:09.849662 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.2 10.8.128.106:0/12651 202 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:09.905988) v4 ==== 2004+0+0 (884869960 0 0) 0x55f65c0f3600 con 0x55f663219000 > -7145> 2018-07-11 08:05:09.849659 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12651 conn(0x55f66321a800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25 cs=1 l=1). rx osd.2 seq 202 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:05:09.905988) v4 > -7144> 2018-07-11 08:05:09.849673 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:09.905988) v4 -- 0x55f66c32d600 con 0 > -7143> 2018-07-11 08:05:09.849684 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.2 10.8.128.106:0/12651 202 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:09.905988) v4 ==== 2004+0+0 (884869960 0 0) 0x55f64e20fc00 con 0x55f66321a800 > -7142> 2018-07-11 08:05:09.849707 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:09.905988) v4 -- 0x55f67667e200 con 0 > -7141> 2018-07-11 08:05:10.059500 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f66b5978c0 con 0 > -7140> 2018-07-11 08:05:10.061122 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 205 0x55f67c238000 replica scrub(pg: 6.15,from:0'0,to:430'106099,epoch:509/504,start:6:a82943ac:::benchmark_data_magna108.ceph.redhat.com_2478_object80981:0,end:6:a82a2dbe:::benchmark_data_magna108_35617_object891108:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -7139> 2018-07-11 08:05:10.061145 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 205 ==== replica scrub(pg: 6.15,from:0'0,to:430'106099,epoch:509/504,start:6:a82943ac:::benchmark_data_magna108.ceph.redhat.com_2478_object80981:0,end:6:a82a2dbe:::benchmark_data_magna108_35617_object891108:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 232+0+0 (2046012182 0 0) 0x55f67c238000 con 0x55f66315e800 > -7138> 2018-07-11 08:05:10.081658 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.107:0/12617 conn(0x55f64e3c5000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.9 seq 192 0x55f65c0f3400 osd_ping(ping e509 stamp 2018-07-11 08:05:10.081298) v4 > -7137> 2018-07-11 08:05:10.081677 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.107:0/12617 conn(0x55f64e3c3800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.9 seq 192 0x55f660ea5a00 osd_ping(ping e509 stamp 2018-07-11 08:05:10.081298) v4 > -7136> 2018-07-11 08:05:10.081687 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.9 10.8.128.107:0/12617 192 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:10.081298) v4 ==== 2004+0+0 (890457214 0 0) 0x55f65c0f3400 con 0x55f64e3c5000 > -7135> 2018-07-11 08:05:10.081698 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.9 10.8.128.107:0/12617 192 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:10.081298) v4 ==== 2004+0+0 (890457214 0 0) 0x55f660ea5a00 con 0x55f64e3c3800 > -7134> 2018-07-11 08:05:10.081703 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:10.081298) v4 -- 0x55f66c32d400 con 0 > -7133> 2018-07-11 08:05:10.081736 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:10.081298) v4 -- 0x55f67667e000 con 0 > -7132> 2018-07-11 08:05:10.299965 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15287 conn(0x55f66dce2800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.4 seq 199 0x55f66c32d600 osd_ping(ping e509 stamp 2018-07-11 08:05:11.544268) v4 > -7131> 2018-07-11 08:05:10.299987 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.4 10.8.128.113:0/15287 199 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:11.544268) v4 ==== 2004+0+0 (2569173494 0 0) 0x55f66c32d600 con 0x55f66dce2800 > -7130> 2018-07-11 08:05:10.299998 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:11.544268) v4 -- 0x55f66c32d400 con 0 > -7129> 2018-07-11 08:05:10.300096 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15287 conn(0x55f66dce1000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.4 seq 199 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:05:11.544268) v4 > -7128> 2018-07-11 08:05:10.300112 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.4 10.8.128.113:0/15287 199 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:11.544268) v4 ==== 2004+0+0 (2569173494 0 0) 0x55f67d1afe00 con 0x55f66dce1000 > -7127> 2018-07-11 08:05:10.300122 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:11.544268) v4 -- 0x55f67667e200 con 0 > -7126> 2018-07-11 08:05:10.401169 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0b800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 190 0x55f66b1cca00 osd_ping(ping e509 stamp 2018-07-11 08:05:11.036641) v4 > -7125> 2018-07-11 08:05:10.401195 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.11 10.8.128.114:0/15919 190 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:11.036641) v4 ==== 2004+0+0 (1269759367 0 0) 0x55f66b1cca00 con 0x55f663f0b800 > -7124> 2018-07-11 08:05:10.401205 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:11.036641) v4 -- 0x55f67667e000 con 0 > -7123> 2018-07-11 08:05:10.401197 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0a000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 190 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:05:11.036641) v4 > -7122> 2018-07-11 08:05:10.401230 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.11 10.8.128.114:0/15919 190 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:11.036641) v4 ==== 2004+0+0 (1269759367 0 0) 0x55f64e20fc00 con 0x55f663f0a000 > -7121> 2018-07-11 08:05:10.401242 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:11.036641) v4 -- 0x55f66c32da00 con 0 > -7120> 2018-07-11 08:05:10.433377 7f491e048700 10 monclient: tick > -7119> 2018-07-11 08:05:10.433390 7f491e048700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2018-07-11 08:04:40.433388) > -7118> 2018-07-11 08:05:10.527028 7f491d046700 1 -- 10.8.128.107:6800/14173 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55f653d5a000 con 0 > -7117> 2018-07-11 08:05:10.527090 7f491d046700 1 -- 10.8.128.107:6800/14173 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55f666f258c0 con 0 > -7116> 2018-07-11 08:05:10.743952 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f64f189b00 con 0 > -7115> 2018-07-11 08:05:10.745529 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 206 0x55f67c238280 replica scrub(pg: 6.15,from:0'0,to:430'105866,epoch:509/504,start:6:a82a2dbe:::benchmark_data_magna108_35617_object891108:0,end:6:a82b0c03:::benchmark_data_magna108_41525_object46065:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -7114> 2018-07-11 08:05:10.745561 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 206 ==== replica scrub(pg: 6.15,from:0'0,to:430'105866,epoch:509/504,start:6:a82a2dbe:::benchmark_data_magna108_35617_object891108:0,end:6:a82b0c03:::benchmark_data_magna108_41525_object46065:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 217+0+0 (870681455 0 0) 0x55f67c238280 con 0x55f66315e800 > -7113> 2018-07-11 08:05:10.950119 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12651 conn(0x55f663219000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25 cs=1 l=1). rx osd.2 seq 203 0x55f66c32d600 osd_ping(ping e509 stamp 2018-07-11 08:05:11.006609) v4 > -7112> 2018-07-11 08:05:10.950147 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.2 10.8.128.106:0/12651 203 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:11.006609) v4 ==== 2004+0+0 (1153097219 0 0) 0x55f66c32d600 con 0x55f663219000 > -7111> 2018-07-11 08:05:10.950163 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:11.006609) v4 -- 0x55f66c32ce00 con 0 > -7110> 2018-07-11 08:05:10.950207 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12651 conn(0x55f66321a800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25 cs=1 l=1). rx osd.2 seq 203 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:05:11.006609) v4 > -7109> 2018-07-11 08:05:10.950226 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.2 10.8.128.106:0/12651 203 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:11.006609) v4 ==== 2004+0+0 (1153097219 0 0) 0x55f67d1afe00 con 0x55f66321a800 > -7108> 2018-07-11 08:05:10.950238 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:11.006609) v4 -- 0x55f66c32d200 con 0 > -7107> 2018-07-11 08:05:11.227840 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12650 conn(0x55f663ed9000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.3 seq 193 0x55f660ea5a00 osd_ping(ping e509 stamp 2018-07-11 08:05:11.284312) v4 > -7106> 2018-07-11 08:05:11.227876 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.3 10.8.128.106:0/12650 193 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:11.284312) v4 ==== 2004+0+0 (2388699930 0 0) 0x55f660ea5a00 con 0x55f663ed9000 > -7105> 2018-07-11 08:05:11.227884 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12650 conn(0x55f663eda800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.3 seq 193 0x55f66c32d400 osd_ping(ping e509 stamp 2018-07-11 08:05:11.284312) v4 > -7104> 2018-07-11 08:05:11.227894 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:11.284312) v4 -- 0x55f66c32d800 con 0 > -7103> 2018-07-11 08:05:11.227910 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.3 10.8.128.106:0/12650 193 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:11.284312) v4 ==== 2004+0+0 (2388699930 0 0) 0x55f66c32d400 con 0x55f663eda800 > -7102> 2018-07-11 08:05:11.227938 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:11.284312) v4 -- 0x55f65c0f3800 con 0 > -7101> 2018-07-11 08:05:11.507588 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.115:0/13323 conn(0x55f663ebf800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.6 seq 183 0x55f66c32d600 osd_ping(ping e509 stamp 2018-07-11 08:05:11.860031) v4 > -7100> 2018-07-11 08:05:11.507611 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.6 10.8.128.115:0/13323 183 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:11.860031) v4 ==== 2004+0+0 (2361989560 0 0) 0x55f66c32d600 con 0x55f663ebf800 > -7099> 2018-07-11 08:05:11.507614 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.115:0/13323 conn(0x55f663ebe000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.6 seq 183 0x55f66b1cca00 osd_ping(ping e509 stamp 2018-07-11 08:05:11.860031) v4 > -7098> 2018-07-11 08:05:11.507628 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:11.860031) v4 -- 0x55f65c0f3a00 con 0 > -7097> 2018-07-11 08:05:11.507631 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.6 10.8.128.115:0/13323 183 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:11.860031) v4 ==== 2004+0+0 (2361989560 0 0) 0x55f66b1cca00 con 0x55f663ebe000 > -7096> 2018-07-11 08:05:11.507654 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:11.860031) v4 -- 0x55f66c32d000 con 0 > -7095> 2018-07-11 08:05:11.521713 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f64e6a98c0 con 0 > -7094> 2018-07-11 08:05:11.523437 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 207 0x55f67c238500 replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:509/504,start:6:a82b0c03:::benchmark_data_magna108_41525_object46065:0,end:6:a82c1f69:::benchmark_data_magna108.ceph.redhat.com_2478_object294167:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -7093> 2018-07-11 08:05:11.523468 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 207 ==== replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:509/504,start:6:a82b0c03:::benchmark_data_magna108_41525_object46065:0,end:6:a82c1f69:::benchmark_data_magna108.ceph.redhat.com_2478_object294167:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 232+0+0 (1615435016 0 0) 0x55f67c238500 con 0x55f66315e800 > -7092> 2018-07-11 08:05:11.660476 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.114:6805/15885 conn(0x55f66315a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.7 seq 2539 0x55f66c32d400 MOSDScrubReserve(5.4 REQUEST e509) v1 > -7091> 2018-07-11 08:05:11.660500 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.7 10.8.128.114:6805/15885 2539 ==== MOSDScrubReserve(5.4 REQUEST e509) v1 ==== 43+0+0 (3003630469 0 0) 0x55f66c32d400 con 0x55f66315a000 > -7090> 2018-07-11 08:05:11.660579 7f491983f700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.114:6805/15885 -- MOSDScrubReserve(5.4 REJECT e509) v1 -- 0x55f660ea5800 con 0 > -7089> 2018-07-11 08:05:11.660955 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.114:6805/15885 conn(0x55f66315a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.7 seq 2540 0x55f660ea5800 MOSDScrubReserve(5.4 RELEASE e509) v1 > -7088> 2018-07-11 08:05:11.660973 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.7 10.8.128.114:6805/15885 2540 ==== MOSDScrubReserve(5.4 RELEASE e509) v1 ==== 43+0+0 (2082093447 0 0) 0x55f660ea5800 con 0x55f66315a000 > -7087> 2018-07-11 08:05:11.782503 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.107:0/12617 conn(0x55f64e3c3800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.9 seq 193 0x55f660ea5a00 osd_ping(ping e509 stamp 2018-07-11 08:05:11.781824) v4 > -7086> 2018-07-11 08:05:11.782532 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.107:0/12617 conn(0x55f64e3c5000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.9 seq 193 0x55f65c0f3a00 osd_ping(ping e509 stamp 2018-07-11 08:05:11.781824) v4 > -7085> 2018-07-11 08:05:11.782545 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.9 10.8.128.107:0/12617 193 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:11.781824) v4 ==== 2004+0+0 (128847311 0 0) 0x55f660ea5a00 con 0x55f64e3c3800 > -7084> 2018-07-11 08:05:11.782557 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.9 10.8.128.107:0/12617 193 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:11.781824) v4 ==== 2004+0+0 (128847311 0 0) 0x55f65c0f3a00 con 0x55f64e3c5000 > -7083> 2018-07-11 08:05:11.782561 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:11.781824) v4 -- 0x55f653d4a400 con 0 > -7082> 2018-07-11 08:05:11.782600 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:11.781824) v4 -- 0x55f663c0a600 con 0 > -7081> 2018-07-11 08:05:12.530376 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15885 conn(0x55f663217800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 196 0x55f66b1cca00 osd_ping(ping e509 stamp 2018-07-11 08:05:13.165750) v4 > -7080> 2018-07-11 08:05:12.530401 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.7 10.8.128.114:0/15885 196 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:13.165750) v4 ==== 2004+0+0 (2933881463 0 0) 0x55f66b1cca00 con 0x55f663217800 > -7079> 2018-07-11 08:05:12.530413 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:13.165750) v4 -- 0x55f65d025c00 con 0 > -7078> 2018-07-11 08:05:12.530428 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15885 conn(0x55f663216000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 196 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:05:13.165750) v4 > -7077> 2018-07-11 08:05:12.530455 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.7 10.8.128.114:0/15885 196 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:13.165750) v4 ==== 2004+0+0 (2933881463 0 0) 0x55f64e20fc00 con 0x55f663216000 > -7076> 2018-07-11 08:05:12.530471 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:13.165750) v4 -- 0x55f660ea4200 con 0 > -7075> 2018-07-11 08:05:12.608020 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.115:0/13323 conn(0x55f663ebe000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.6 seq 184 0x55f660ea5a00 osd_ping(ping e509 stamp 2018-07-11 08:05:12.960503) v4 > -7074> 2018-07-11 08:05:12.608043 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.6 10.8.128.115:0/13323 184 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:12.960503) v4 ==== 2004+0+0 (2386052836 0 0) 0x55f660ea5a00 con 0x55f663ebe000 > -7073> 2018-07-11 08:05:12.608058 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:12.960503) v4 -- 0x55f65299b400 con 0 > -7072> 2018-07-11 08:05:12.608050 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.115:0/13323 conn(0x55f663ebf800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.6 seq 184 0x55f65c0f3c00 osd_ping(ping e509 stamp 2018-07-11 08:05:12.960503) v4 > -7071> 2018-07-11 08:05:12.608067 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.6 10.8.128.115:0/13323 184 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:12.960503) v4 ==== 2004+0+0 (2386052836 0 0) 0x55f65c0f3c00 con 0x55f663ebf800 > -7070> 2018-07-11 08:05:12.608084 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:12.960503) v4 -- 0x55f663c0ae00 con 0 > -7069> 2018-07-11 08:05:12.635101 7f491803c700 5 osd.10 509 heartbeat: osd_stat(789 GB used, 142 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,0,0,0,1]) > -7068> 2018-07-11 08:05:12.635116 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6802/15255 -- osd_ping(ping e509 stamp 2018-07-11 08:05:12.635114) v4 -- 0x55f67667fa00 con 0 > -7067> 2018-07-11 08:05:12.635152 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6803/15255 -- osd_ping(ping e509 stamp 2018-07-11 08:05:12.635114) v4 -- 0x55f67667fc00 con 0 > -7066> 2018-07-11 08:05:12.635170 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6806/12677 -- osd_ping(ping e509 stamp 2018-07-11 08:05:12.635114) v4 -- 0x55f67667fe00 con 0 > -7065> 2018-07-11 08:05:12.635186 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6807/12677 -- osd_ping(ping e509 stamp 2018-07-11 08:05:12.635114) v4 -- 0x55f67141e000 con 0 > -7064> 2018-07-11 08:05:12.635199 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6808/12651 -- osd_ping(ping e509 stamp 2018-07-11 08:05:12.635114) v4 -- 0x55f67141e200 con 0 > -7063> 2018-07-11 08:05:12.635211 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6810/12651 -- osd_ping(ping e509 stamp 2018-07-11 08:05:12.635114) v4 -- 0x55f67141e400 con 0 > -7062> 2018-07-11 08:05:12.635228 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6809/12650 -- osd_ping(ping e509 stamp 2018-07-11 08:05:12.635114) v4 -- 0x55f67141e600 con 0 > -7061> 2018-07-11 08:05:12.635244 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6811/12650 -- osd_ping(ping e509 stamp 2018-07-11 08:05:12.635114) v4 -- 0x55f67141e800 con 0 > -7060> 2018-07-11 08:05:12.635269 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6810/15287 -- osd_ping(ping e509 stamp 2018-07-11 08:05:12.635114) v4 -- 0x55f67141ea00 con 0 > -7059> 2018-07-11 08:05:12.635285 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6811/15287 -- osd_ping(ping e509 stamp 2018-07-11 08:05:12.635114) v4 -- 0x55f67141ec00 con 0 > -7058> 2018-07-11 08:05:12.635298 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6806/15263 -- osd_ping(ping e509 stamp 2018-07-11 08:05:12.635114) v4 -- 0x55f67141ee00 con 0 > -7057> 2018-07-11 08:05:12.635312 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6807/15263 -- osd_ping(ping e509 stamp 2018-07-11 08:05:12.635114) v4 -- 0x55f67141f000 con 0 > -7056> 2018-07-11 08:05:12.635326 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.115:6802/13323 -- osd_ping(ping e509 stamp 2018-07-11 08:05:12.635114) v4 -- 0x55f67141f200 con 0 > -7055> 2018-07-11 08:05:12.635339 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.115:6803/13323 -- osd_ping(ping e509 stamp 2018-07-11 08:05:12.635114) v4 -- 0x55f67141f400 con 0 > -7054> 2018-07-11 08:05:12.635352 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6806/15885 -- osd_ping(ping e509 stamp 2018-07-11 08:05:12.635114) v4 -- 0x55f67141f600 con 0 > -7053> 2018-07-11 08:05:12.635365 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6807/15885 -- osd_ping(ping e509 stamp 2018-07-11 08:05:12.635114) v4 -- 0x55f67141f800 con 0 > -7052> 2018-07-11 08:05:12.635378 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.108:6802/11967 -- osd_ping(ping e509 stamp 2018-07-11 08:05:12.635114) v4 -- 0x55f67141fa00 con 0 > -7051> 2018-07-11 08:05:12.635390 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.108:6803/11967 -- osd_ping(ping e509 stamp 2018-07-11 08:05:12.635114) v4 -- 0x55f67141fc00 con 0 > -7050> 2018-07-11 08:05:12.635402 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.107:6806/12617 -- osd_ping(ping e509 stamp 2018-07-11 08:05:12.635114) v4 -- 0x55f67141fe00 con 0 > -7049> 2018-07-11 08:05:12.635416 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.107:6807/12617 -- osd_ping(ping e509 stamp 2018-07-11 08:05:12.635114) v4 -- 0x55f671426000 con 0 > -7048> 2018-07-11 08:05:12.635430 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6802/15919 -- osd_ping(ping e509 stamp 2018-07-11 08:05:12.635114) v4 -- 0x55f671426200 con 0 > -7047> 2018-07-11 08:05:12.635442 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6803/15919 -- osd_ping(ping e509 stamp 2018-07-11 08:05:12.635114) v4 -- 0x55f671426400 con 0 > -7046> 2018-07-11 08:05:12.635532 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6802/15255 conn(0x55f656ad4000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 195 0x55f65c0f3c00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:12.635114) v4 > -7045> 2018-07-11 08:05:12.635545 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6803/15255 conn(0x55f656ad5800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 195 0x55f660ea5a00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:12.635114) v4 > -7044> 2018-07-11 08:05:12.635552 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.0 10.8.128.113:6802/15255 195 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:12.635114) v4 ==== 2004+0+0 (313360956 0 0) 0x55f65c0f3c00 con 0x55f656ad4000 > -7043> 2018-07-11 08:05:12.635571 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.0 10.8.128.113:6803/15255 195 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:12.635114) v4 ==== 2004+0+0 (313360956 0 0) 0x55f660ea5a00 con 0x55f656ad5800 > -7042> 2018-07-11 08:05:12.635618 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6806/12677 conn(0x55f64e3d8000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.1 seq 195 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:12.635114) v4 > -7041> 2018-07-11 08:05:12.635623 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.107:6806/12617 conn(0x55f656ade000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.9 seq 195 0x55f660ea5a00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:12.635114) v4 > -7040> 2018-07-11 08:05:12.635641 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.1 10.8.128.106:6806/12677 195 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:12.635114) v4 ==== 2004+0+0 (313360956 0 0) 0x55f67d1afe00 con 0x55f64e3d8000 > -7039> 2018-07-11 08:05:12.635644 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.9 10.8.128.107:6806/12617 195 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:12.635114) v4 ==== 2004+0+0 (313360956 0 0) 0x55f660ea5a00 con 0x55f656ade000 > -7038> 2018-07-11 08:05:12.635638 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6807/12677 conn(0x55f64e3d9800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.1 seq 195 0x55f65c0f3c00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:12.635114) v4 > -7037> 2018-07-11 08:05:12.635663 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.1 10.8.128.106:6807/12677 195 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:12.635114) v4 ==== 2004+0+0 (313360956 0 0) 0x55f65c0f3c00 con 0x55f64e3d9800 > -7036> 2018-07-11 08:05:12.635672 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.107:6807/12617 conn(0x55f656adf800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.9 seq 195 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:12.635114) v4 > -7035> 2018-07-11 08:05:12.635698 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.9 10.8.128.107:6807/12617 195 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:12.635114) v4 ==== 2004+0+0 (313360956 0 0) 0x55f67d1afe00 con 0x55f656adf800 > -7034> 2018-07-11 08:05:12.635709 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6810/12651 conn(0x55f64e3eb800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.2 seq 195 0x55f660ea5a00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:12.635114) v4 > -7033> 2018-07-11 08:05:12.635724 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.2 10.8.128.106:6810/12651 195 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:12.635114) v4 ==== 2004+0+0 (313360956 0 0) 0x55f660ea5a00 con 0x55f64e3eb800 > -7032> 2018-07-11 08:05:12.635722 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6810/15287 conn(0x55f64e401000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.4 seq 195 0x55f65c0f3c00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:12.635114) v4 > -7031> 2018-07-11 08:05:12.635745 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6809/12650 conn(0x55f656ad7000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.3 seq 195 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:12.635114) v4 > -7030> 2018-07-11 08:05:12.635759 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.3 10.8.128.106:6809/12650 195 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:12.635114) v4 ==== 2004+0+0 (313360956 0 0) 0x55f67d1afe00 con 0x55f656ad7000 > -7029> 2018-07-11 08:05:12.635771 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6811/15287 conn(0x55f64e402800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.4 seq 195 0x55f660ea5a00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:12.635114) v4 > -7028> 2018-07-11 08:05:12.635786 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.4 10.8.128.113:6811/15287 195 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:12.635114) v4 ==== 2004+0+0 (313360956 0 0) 0x55f660ea5a00 con 0x55f64e402800 > -7027> 2018-07-11 08:05:12.635782 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.4 10.8.128.113:6810/15287 195 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:12.635114) v4 ==== 2004+0+0 (313360956 0 0) 0x55f65c0f3c00 con 0x55f64e401000 > -7026> 2018-07-11 08:05:12.635793 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.115:6803/13323 conn(0x55f656ace800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.6 seq 195 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:12.635114) v4 > -7025> 2018-07-11 08:05:12.635806 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.6 10.8.128.115:6803/13323 195 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:12.635114) v4 ==== 2004+0+0 (313360956 0 0) 0x55f67d1afe00 con 0x55f656ace800 > -7024> 2018-07-11 08:05:12.635822 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.115:6802/13323 conn(0x55f656acd000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.6 seq 195 0x55f660ea5a00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:12.635114) v4 > -7023> 2018-07-11 08:05:12.635835 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.6 10.8.128.115:6802/13323 195 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:12.635114) v4 ==== 2004+0+0 (313360956 0 0) 0x55f660ea5a00 con 0x55f656acd000 > -7022> 2018-07-11 08:05:12.635835 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6808/12651 conn(0x55f64e3ea000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.2 seq 195 0x55f65c0f3c00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:12.635114) v4 > -7021> 2018-07-11 08:05:12.635854 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.2 10.8.128.106:6808/12651 195 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:12.635114) v4 ==== 2004+0+0 (313360956 0 0) 0x55f65c0f3c00 con 0x55f64e3ea000 > -7020> 2018-07-11 08:05:12.635853 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6806/15263 conn(0x55f64e3ed000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.5 seq 195 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:12.635114) v4 > -7019> 2018-07-11 08:05:12.635864 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6806/15885 conn(0x55f64e3fe000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.7 seq 195 0x55f660ea5a00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:12.635114) v4 > -7018> 2018-07-11 08:05:12.635880 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.5 10.8.128.113:6806/15263 195 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:12.635114) v4 ==== 2004+0+0 (313360956 0 0) 0x55f67d1afe00 con 0x55f64e3ed000 > -7017> 2018-07-11 08:05:12.635882 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.7 10.8.128.114:6806/15885 195 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:12.635114) v4 ==== 2004+0+0 (313360956 0 0) 0x55f660ea5a00 con 0x55f64e3fe000 > -7016> 2018-07-11 08:05:12.635907 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6807/15263 conn(0x55f64e3ee800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.5 seq 195 0x55f65c0f3c00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:12.635114) v4 > -7015> 2018-07-11 08:05:12.635917 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6807/15885 conn(0x55f64e3ff800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.7 seq 195 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:12.635114) v4 > -7014> 2018-07-11 08:05:12.635921 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.5 10.8.128.113:6807/15263 195 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:12.635114) v4 ==== 2004+0+0 (313360956 0 0) 0x55f65c0f3c00 con 0x55f64e3ee800 > -7013> 2018-07-11 08:05:12.635925 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6802/15919 conn(0x55f64e3db000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.11 seq 195 0x55f660ea5a00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:12.635114) v4 > -7012> 2018-07-11 08:05:12.635930 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.7 10.8.128.114:6807/15885 195 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:12.635114) v4 ==== 2004+0+0 (313360956 0 0) 0x55f67d1afe00 con 0x55f64e3ff800 > -7011> 2018-07-11 08:05:12.635938 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.11 10.8.128.114:6802/15919 195 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:12.635114) v4 ==== 2004+0+0 (313360956 0 0) 0x55f660ea5a00 con 0x55f64e3db000 > -7010> 2018-07-11 08:05:12.635959 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6803/15919 conn(0x55f64e3dc800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.11 seq 195 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:12.635114) v4 > -7009> 2018-07-11 08:05:12.635963 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6811/12650 conn(0x55f656ad8800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.3 seq 195 0x55f65c0f3c00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:12.635114) v4 > -7008> 2018-07-11 08:05:12.635971 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.11 10.8.128.114:6803/15919 195 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:12.635114) v4 ==== 2004+0+0 (313360956 0 0) 0x55f67d1afe00 con 0x55f64e3dc800 > -7007> 2018-07-11 08:05:12.635976 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.3 10.8.128.106:6811/12650 195 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:12.635114) v4 ==== 2004+0+0 (313360956 0 0) 0x55f65c0f3c00 con 0x55f656ad8800 > -7006> 2018-07-11 08:05:12.636001 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.108:6802/11967 conn(0x55f656aca000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.8 seq 195 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:12.635114) v4 > -7005> 2018-07-11 08:05:12.636012 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.8 10.8.128.108:6802/11967 195 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:12.635114) v4 ==== 2004+0+0 (313360956 0 0) 0x55f67d1afe00 con 0x55f656aca000 > -7004> 2018-07-11 08:05:12.636019 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.108:6803/11967 conn(0x55f656acb800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.8 seq 195 0x55f65c0f3c00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:12.635114) v4 > -7003> 2018-07-11 08:05:12.636036 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.8 10.8.128.108:6803/11967 195 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:12.635114) v4 ==== 2004+0+0 (313360956 0 0) 0x55f65c0f3c00 con 0x55f656acb800 > -7002> 2018-07-11 08:05:12.696622 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.108:0/11967 conn(0x55f663f6f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=1). rx osd.8 seq 187 0x55f660ea5a00 osd_ping(ping e509 stamp 2018-07-11 08:05:12.456454) v4 > -7001> 2018-07-11 08:05:12.696649 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.8 10.8.128.108:0/11967 187 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:12.456454) v4 ==== 2004+0+0 (3117617426 0 0) 0x55f660ea5a00 con 0x55f663f6f000 > -7000> 2018-07-11 08:05:12.696664 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:12.456454) v4 -- 0x55f67667fc00 con 0 > -6999> 2018-07-11 08:05:12.696657 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.108:0/11967 conn(0x55f663f70800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=1). rx osd.8 seq 187 0x55f65c0f3c00 osd_ping(ping e509 stamp 2018-07-11 08:05:12.456454) v4 > -6998> 2018-07-11 08:05:12.696675 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.8 10.8.128.108:0/11967 187 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:12.456454) v4 ==== 2004+0+0 (3117617426 0 0) 0x55f65c0f3c00 con 0x55f663f70800 > -6997> 2018-07-11 08:05:12.696684 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:12.456454) v4 -- 0x55f67141e200 con 0 > -6996> 2018-07-11 08:05:12.766163 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 29] Generated table #37037: 2638241 keys, 74546650 bytes > -6995> 2018-07-11 08:05:12.766211 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296312766195, "cf_name": "default", "job": 29, "event": "table_file_creation", "file_number": 37037, "file_size": 74546650, "table_properties": {"data_size": 67112152, "index_size": 690325, "filter_size": 6743183, "raw_key_size": 76215857, "raw_average_key_size": 28, "raw_value_size": 31365760, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} > -6994> 2018-07-11 08:05:12.953409 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0d000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 199 0x55f65c0f3a00 osd_ping(ping e509 stamp 2018-07-11 08:05:14.197722) v4 > -6993> 2018-07-11 08:05:12.953432 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.5 10.8.128.113:0/15263 199 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:14.197722) v4 ==== 2004+0+0 (2636516373 0 0) 0x55f65c0f3a00 con 0x55f663f0d000 > -6992> 2018-07-11 08:05:12.953443 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:14.197722) v4 -- 0x55f67141e200 con 0 > -6991> 2018-07-11 08:05:12.953523 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0e800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 199 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:05:14.197722) v4 > -6990> 2018-07-11 08:05:12.953555 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.5 10.8.128.113:0/15263 199 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:14.197722) v4 ==== 2004+0+0 (2636516373 0 0) 0x55f67d1afe00 con 0x55f663f0e800 > -6989> 2018-07-11 08:05:12.953573 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:14.197722) v4 -- 0x55f67141e600 con 0 > -6988> 2018-07-11 08:05:12.963295 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 423 0x55f66b1cca00 MOSDScrubReserve(11.3s5 REQUEST e509) v1 > -6987> 2018-07-11 08:05:12.963314 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 423 ==== MOSDScrubReserve(11.3s5 REQUEST e509) v1 ==== 43+0+0 (433425070 0 0) 0x55f66b1cca00 con 0x55f668a29000 > -6986> 2018-07-11 08:05:12.963387 7f491a040700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e509) v1 -- 0x55f66e944800 con 0 > -6985> 2018-07-11 08:05:12.963825 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 424 0x55f66e944800 MOSDScrubReserve(11.3s5 RELEASE e509) v1 > -6984> 2018-07-11 08:05:12.963840 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 424 ==== MOSDScrubReserve(11.3s5 RELEASE e509) v1 ==== 43+0+0 (3603460268 0 0) 0x55f66e944800 con 0x55f668a29000 > -6983> 2018-07-11 08:05:13.151205 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f6696aefc0 con 0 > -6982> 2018-07-11 08:05:13.152909 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 208 0x55f67c238780 replica scrub(pg: 6.15,from:0'0,to:430'105525,epoch:509/504,start:6:a82c1f69:::benchmark_data_magna108.ceph.redhat.com_2478_object294167:0,end:6:a82ce642:::benchmark_data_magna108_35617_object346583:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -6981> 2018-07-11 08:05:13.152941 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 208 ==== replica scrub(pg: 6.15,from:0'0,to:430'105525,epoch:509/504,start:6:a82c1f69:::benchmark_data_magna108.ceph.redhat.com_2478_object294167:0,end:6:a82ce642:::benchmark_data_magna108_35617_object346583:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (2765339273 0 0) 0x55f67c238780 con 0x55f66315e800 > -6980> 2018-07-11 08:05:13.173972 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15255 conn(0x55f663ec2800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.0 seq 187 0x55f660ea5a00 osd_ping(ping e509 stamp 2018-07-11 08:05:14.418310) v4 > -6979> 2018-07-11 08:05:13.173981 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15255 conn(0x55f663ec1000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.0 seq 187 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:05:14.418310) v4 > -6978> 2018-07-11 08:05:13.173995 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.0 10.8.128.113:0/15255 187 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:14.418310) v4 ==== 2004+0+0 (3059306502 0 0) 0x55f660ea5a00 con 0x55f663ec2800 > -6977> 2018-07-11 08:05:13.174006 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:14.418310) v4 -- 0x55f65299b400 con 0 > -6976> 2018-07-11 08:05:13.174002 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.0 10.8.128.113:0/15255 187 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:14.418310) v4 ==== 2004+0+0 (3059306502 0 0) 0x55f64e20fc00 con 0x55f663ec1000 > -6975> 2018-07-11 08:05:13.174031 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:14.418310) v4 -- 0x55f67141e600 con 0 > -6974> 2018-07-11 08:05:13.200499 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15287 conn(0x55f66dce2800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.4 seq 200 0x55f65c0f3c00 osd_ping(ping e509 stamp 2018-07-11 08:05:14.444783) v4 > -6973> 2018-07-11 08:05:13.200520 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.4 10.8.128.113:0/15287 200 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:14.444783) v4 ==== 2004+0+0 (1819390772 0 0) 0x55f65c0f3c00 con 0x55f66dce2800 > -6972> 2018-07-11 08:05:13.200530 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:14.444783) v4 -- 0x55f67141e200 con 0 > -6971> 2018-07-11 08:05:13.200716 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15287 conn(0x55f66dce1000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.4 seq 200 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:05:14.444783) v4 > -6970> 2018-07-11 08:05:13.200734 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.4 10.8.128.113:0/15287 200 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:14.444783) v4 ==== 2004+0+0 (1819390772 0 0) 0x55f67d1afe00 con 0x55f66dce1000 > -6969> 2018-07-11 08:05:13.200745 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:14.444783) v4 -- 0x55f67141e600 con 0 > -6968> 2018-07-11 08:05:13.630965 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15885 conn(0x55f663216000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 197 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:05:14.266361) v4 > -6967> 2018-07-11 08:05:13.630989 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.7 10.8.128.114:0/15885 197 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:14.266361) v4 ==== 2004+0+0 (385256412 0 0) 0x55f64e20fc00 con 0x55f663216000 > -6966> 2018-07-11 08:05:13.630985 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15885 conn(0x55f663217800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 197 0x55f67667fc00 osd_ping(ping e509 stamp 2018-07-11 08:05:14.266361) v4 > -6965> 2018-07-11 08:05:13.631001 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:14.266361) v4 -- 0x55f67141e600 con 0 > -6964> 2018-07-11 08:05:13.631001 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.7 10.8.128.114:0/15885 197 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:14.266361) v4 ==== 2004+0+0 (385256412 0 0) 0x55f67667fc00 con 0x55f663217800 > -6963> 2018-07-11 08:05:13.631027 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:14.266361) v4 -- 0x55f6548fb200 con 0 > -6962> 2018-07-11 08:05:13.837557 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f684163b00 con 0 > -6961> 2018-07-11 08:05:13.839150 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 209 0x55f67c238a00 replica scrub(pg: 6.15,from:0'0,to:430'105802,epoch:509/504,start:6:a82ce642:::benchmark_data_magna108_35617_object346583:0,end:6:a82dedea:::benchmark_data_magna108_35617_object893555:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -6960> 2018-07-11 08:05:13.839181 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 209 ==== replica scrub(pg: 6.15,from:0'0,to:430'105802,epoch:509/504,start:6:a82ce642:::benchmark_data_magna108_35617_object346583:0,end:6:a82dedea:::benchmark_data_magna108_35617_object893555:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (2910290749 0 0) 0x55f67c238a00 con 0x55f66315e800 > -6959> 2018-07-11 08:05:13.963589 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 425 0x55f660ea5a00 MOSDScrubReserve(11.3s5 REQUEST e509) v1 > -6958> 2018-07-11 08:05:13.963613 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 425 ==== MOSDScrubReserve(11.3s5 REQUEST e509) v1 ==== 43+0+0 (433425070 0 0) 0x55f660ea5a00 con 0x55f668a29000 > -6957> 2018-07-11 08:05:13.963675 7f491a040700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e509) v1 -- 0x55f66e944800 con 0 > -6956> 2018-07-11 08:05:13.964044 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 426 0x55f66e944800 MOSDScrubReserve(11.3s5 RELEASE e509) v1 > -6955> 2018-07-11 08:05:13.964063 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 426 ==== MOSDScrubReserve(11.3s5 RELEASE e509) v1 ==== 43+0+0 (3603460268 0 0) 0x55f66e944800 con 0x55f668a29000 > -6954> 2018-07-11 08:05:13.973851 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed6000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 194 0x55f65c0f3a00 osd_ping(ping e509 stamp 2018-07-11 08:05:14.030429) v4 > -6953> 2018-07-11 08:05:13.973852 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed7800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 194 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:05:14.030429) v4 > -6952> 2018-07-11 08:05:13.973877 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.1 10.8.128.106:0/12677 194 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:14.030429) v4 ==== 2004+0+0 (4098422700 0 0) 0x55f65c0f3a00 con 0x55f663ed6000 > -6951> 2018-07-11 08:05:13.973881 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.1 10.8.128.106:0/12677 194 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:14.030429) v4 ==== 2004+0+0 (4098422700 0 0) 0x55f67d1afe00 con 0x55f663ed7800 > -6950> 2018-07-11 08:05:13.973889 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:14.030429) v4 -- 0x55f67667fa00 con 0 > -6949> 2018-07-11 08:05:13.973913 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:14.030429) v4 -- 0x55f67667fe00 con 0 > -6948> 2018-07-11 08:05:14.082935 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.107:0/12617 conn(0x55f64e3c3800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.9 seq 194 0x55f67667fc00 osd_ping(ping e509 stamp 2018-07-11 08:05:14.082569) v4 > -6947> 2018-07-11 08:05:14.082953 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.107:0/12617 conn(0x55f64e3c5000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.9 seq 194 0x55f67141e200 osd_ping(ping e509 stamp 2018-07-11 08:05:14.082569) v4 > -6946> 2018-07-11 08:05:14.082971 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.9 10.8.128.107:0/12617 194 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:14.082569) v4 ==== 2004+0+0 (1063984430 0 0) 0x55f67141e200 con 0x55f64e3c5000 > -6945> 2018-07-11 08:05:14.082965 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.9 10.8.128.107:0/12617 194 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:14.082569) v4 ==== 2004+0+0 (1063984430 0 0) 0x55f67667fc00 con 0x55f64e3c3800 > -6944> 2018-07-11 08:05:14.082986 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:14.082569) v4 -- 0x55f663c0ae00 con 0 > -6943> 2018-07-11 08:05:14.083011 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:14.082569) v4 -- 0x55f669deae00 con 0 > -6942> 2018-07-11 08:05:14.274447 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15255 conn(0x55f663ec2800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.0 seq 188 0x55f6631cce00 osd_ping(ping e509 stamp 2018-07-11 08:05:15.518716) v4 > -6941> 2018-07-11 08:05:14.274468 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.0 10.8.128.113:0/15255 188 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:15.518716) v4 ==== 2004+0+0 (101485981 0 0) 0x55f6631cce00 con 0x55f663ec2800 > -6940> 2018-07-11 08:05:14.274479 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:15.518716) v4 -- 0x55f673970000 con 0 > -6939> 2018-07-11 08:05:14.274551 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15255 conn(0x55f663ec1000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.0 seq 188 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:05:15.518716) v4 > -6938> 2018-07-11 08:05:14.274570 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.0 10.8.128.113:0/15255 188 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:15.518716) v4 ==== 2004+0+0 (101485981 0 0) 0x55f64e20fc00 con 0x55f663ec1000 > -6937> 2018-07-11 08:05:14.274581 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:15.518716) v4 -- 0x55f660ea4200 con 0 > -6936> 2018-07-11 08:05:14.727411 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f6700b86c0 con 0 > -6935> 2018-07-11 08:05:14.729004 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 210 0x55f67c238c80 replica scrub(pg: 6.15,from:0'0,to:430'105851,epoch:509/504,start:6:a82dedea:::benchmark_data_magna108_35617_object893555:0,end:6:a82ebde9:::benchmark_data_magna108_35617_object699149:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -6934> 2018-07-11 08:05:14.729028 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 210 ==== replica scrub(pg: 6.15,from:0'0,to:430'105851,epoch:509/504,start:6:a82dedea:::benchmark_data_magna108_35617_object893555:0,end:6:a82ebde9:::benchmark_data_magna108_35617_object699149:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (1214085262 0 0) 0x55f67c238c80 con 0x55f66315e800 > -6933> 2018-07-11 08:05:14.963786 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 427 0x55f67667fc00 MOSDScrubReserve(11.3s5 REQUEST e509) v1 > -6932> 2018-07-11 08:05:14.963808 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 427 ==== MOSDScrubReserve(11.3s5 REQUEST e509) v1 ==== 43+0+0 (433425070 0 0) 0x55f67667fc00 con 0x55f668a29000 > -6931> 2018-07-11 08:05:14.963871 7f491a040700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e509) v1 -- 0x55f66e944800 con 0 > -6930> 2018-07-11 08:05:14.964339 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 428 0x55f66e944800 MOSDScrubReserve(11.3s5 RELEASE e509) v1 > -6929> 2018-07-11 08:05:14.964357 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 428 ==== MOSDScrubReserve(11.3s5 RELEASE e509) v1 ==== 43+0+0 (3603460268 0 0) 0x55f66e944800 con 0x55f668a29000 > -6928> 2018-07-11 08:05:15.050807 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12651 conn(0x55f663219000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25 cs=1 l=1). rx osd.2 seq 204 0x55f65c0f3a00 osd_ping(ping e509 stamp 2018-07-11 08:05:15.107158) v4 > -6927> 2018-07-11 08:05:15.050831 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.2 10.8.128.106:0/12651 204 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:15.107158) v4 ==== 2004+0+0 (472736078 0 0) 0x55f65c0f3a00 con 0x55f663219000 > -6926> 2018-07-11 08:05:15.050851 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:15.107158) v4 -- 0x55f663c0b000 con 0 > -6925> 2018-07-11 08:05:15.050847 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12651 conn(0x55f66321a800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25 cs=1 l=1). rx osd.2 seq 204 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:05:15.107158) v4 > -6924> 2018-07-11 08:05:15.050865 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.2 10.8.128.106:0/12651 204 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:15.107158) v4 ==== 2004+0+0 (472736078 0 0) 0x55f67d1afe00 con 0x55f66321a800 > -6923> 2018-07-11 08:05:15.050878 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:15.107158) v4 -- 0x55f64e12fa00 con 0 > -6922> 2018-07-11 08:05:15.101543 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0a000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 191 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:05:15.737027) v4 > -6921> 2018-07-11 08:05:15.101543 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0b800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 191 0x55f6631cce00 osd_ping(ping e509 stamp 2018-07-11 08:05:15.737027) v4 > -6920> 2018-07-11 08:05:15.101571 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.11 10.8.128.114:0/15919 191 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:15.737027) v4 ==== 2004+0+0 (1225337130 0 0) 0x55f64e20fc00 con 0x55f663f0a000 > -6919> 2018-07-11 08:05:15.101579 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.11 10.8.128.114:0/15919 191 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:15.737027) v4 ==== 2004+0+0 (1225337130 0 0) 0x55f6631cce00 con 0x55f663f0b800 > -6918> 2018-07-11 08:05:15.101589 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:15.737027) v4 -- 0x55f653733800 con 0 > -6917> 2018-07-11 08:05:15.101617 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:15.737027) v4 -- 0x55f673970600 con 0 > -6916> 2018-07-11 08:05:15.374873 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15255 conn(0x55f663ec1000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.0 seq 189 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:05:16.619269) v4 > -6915> 2018-07-11 08:05:15.374923 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.0 10.8.128.113:0/15255 189 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:16.619269) v4 ==== 2004+0+0 (91097352 0 0) 0x55f67d1afe00 con 0x55f663ec1000 > -6914> 2018-07-11 08:05:15.374942 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:16.619269) v4 -- 0x55f653733000 con 0 > -6913> 2018-07-11 08:05:15.374959 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15255 conn(0x55f663ec2800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.0 seq 189 0x55f673970200 osd_ping(ping e509 stamp 2018-07-11 08:05:16.619269) v4 > -6912> 2018-07-11 08:05:15.374980 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.0 10.8.128.113:0/15255 189 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:16.619269) v4 ==== 2004+0+0 (91097352 0 0) 0x55f673970200 con 0x55f663ec2800 > -6911> 2018-07-11 08:05:15.374997 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:16.619269) v4 -- 0x55f673970800 con 0 > -6910> 2018-07-11 08:05:15.486000 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f684162480 con 0 > -6909> 2018-07-11 08:05:15.487616 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 211 0x55f67c238f00 replica scrub(pg: 6.15,from:0'0,to:430'106747,epoch:509/504,start:6:a82ebde9:::benchmark_data_magna108_35617_object699149:0,end:6:a82f7f0f:::benchmark_data_magna108_35617_object49088:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -6908> 2018-07-11 08:05:15.487647 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 211 ==== replica scrub(pg: 6.15,from:0'0,to:430'106747,epoch:509/504,start:6:a82ebde9:::benchmark_data_magna108_35617_object699149:0,end:6:a82f7f0f:::benchmark_data_magna108_35617_object49088:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 217+0+0 (805823231 0 0) 0x55f67c238f00 con 0x55f66315e800 > -6907> 2018-07-11 08:05:15.527380 7f491d046700 1 -- 10.8.128.107:6800/14173 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55f667d77080 con 0 > -6906> 2018-07-11 08:05:15.527453 7f491d046700 1 -- 10.8.128.107:6800/14173 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55f667d76b00 con 0 > -6905> 2018-07-11 08:05:15.597039 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.108:0/11967 conn(0x55f663f6f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=1). rx osd.8 seq 188 0x55f6631cce00 osd_ping(ping e509 stamp 2018-07-11 08:05:15.356883) v4 > -6904> 2018-07-11 08:05:15.597063 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.8 10.8.128.108:0/11967 188 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:15.356883) v4 ==== 2004+0+0 (378787072 0 0) 0x55f6631cce00 con 0x55f663f6f000 > -6903> 2018-07-11 08:05:15.597072 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:15.356883) v4 -- 0x55f673970800 con 0 > -6902> 2018-07-11 08:05:15.597109 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.108:0/11967 conn(0x55f663f70800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=1). rx osd.8 seq 188 0x55f663c0ae00 osd_ping(ping e509 stamp 2018-07-11 08:05:15.356883) v4 > -6901> 2018-07-11 08:05:15.597140 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.8 10.8.128.108:0/11967 188 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:15.356883) v4 ==== 2004+0+0 (378787072 0 0) 0x55f663c0ae00 con 0x55f663f70800 > -6900> 2018-07-11 08:05:15.597158 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:15.356883) v4 -- 0x55f65d024600 con 0 > -6899> 2018-07-11 08:05:16.254333 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f64f9cb8c0 con 0 > -6898> 2018-07-11 08:05:16.255885 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 212 0x55f67c239180 replica scrub(pg: 6.15,from:0'0,to:461'106886,epoch:509/504,start:6:a82f7f0f:::benchmark_data_magna108_35617_object49088:0,end:6:a83096c1:::benchmark_data_magna108_35617_object343825:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -6897> 2018-07-11 08:05:16.255920 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 212 ==== replica scrub(pg: 6.15,from:0'0,to:461'106886,epoch:509/504,start:6:a82f7f0f:::benchmark_data_magna108_35617_object49088:0,end:6:a83096c1:::benchmark_data_magna108_35617_object343825:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 217+0+0 (3287782407 0 0) 0x55f67c239180 con 0x55f66315e800 > -6896> 2018-07-11 08:05:16.528296 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12650 conn(0x55f663eda800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.3 seq 194 0x55f65c0f3a00 osd_ping(ping e509 stamp 2018-07-11 08:05:16.584874) v4 > -6895> 2018-07-11 08:05:16.528325 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.3 10.8.128.106:0/12650 194 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:16.584874) v4 ==== 2004+0+0 (294140534 0 0) 0x55f65c0f3a00 con 0x55f663eda800 > -6894> 2018-07-11 08:05:16.528322 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12650 conn(0x55f663ed9000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.3 seq 194 0x55f673970200 osd_ping(ping e509 stamp 2018-07-11 08:05:16.584874) v4 > -6893> 2018-07-11 08:05:16.528335 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:16.584874) v4 -- 0x55f64cf24200 con 0 > -6892> 2018-07-11 08:05:16.528341 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.3 10.8.128.106:0/12650 194 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:16.584874) v4 ==== 2004+0+0 (294140534 0 0) 0x55f673970200 con 0x55f663ed9000 > -6891> 2018-07-11 08:05:16.528355 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:16.584874) v4 -- 0x55f673970800 con 0 > -6890> 2018-07-11 08:05:16.531269 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15885 conn(0x55f663216000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 198 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:05:17.166786) v4 > -6889> 2018-07-11 08:05:16.531290 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.7 10.8.128.114:0/15885 198 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:17.166786) v4 ==== 2004+0+0 (3653978154 0 0) 0x55f64e20fc00 con 0x55f663216000 > -6888> 2018-07-11 08:05:16.531302 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:17.166786) v4 -- 0x55f653732800 con 0 > -6887> 2018-07-11 08:05:16.531300 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15885 conn(0x55f663217800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 198 0x55f6631cce00 osd_ping(ping e509 stamp 2018-07-11 08:05:17.166786) v4 > -6886> 2018-07-11 08:05:16.531327 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.7 10.8.128.114:0/15885 198 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:17.166786) v4 ==== 2004+0+0 (3653978154 0 0) 0x55f6631cce00 con 0x55f663217800 > -6885> 2018-07-11 08:05:16.531344 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:17.166786) v4 -- 0x55f673970a00 con 0 > -6884> 2018-07-11 08:05:16.964123 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 429 0x55f673970200 MOSDScrubReserve(11.3s5 REQUEST e509) v1 > -6883> 2018-07-11 08:05:16.964142 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 429 ==== MOSDScrubReserve(11.3s5 REQUEST e509) v1 ==== 43+0+0 (433425070 0 0) 0x55f673970200 con 0x55f668a29000 > -6882> 2018-07-11 08:05:16.964227 7f491a040700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e509) v1 -- 0x55f66e944800 con 0 > -6881> 2018-07-11 08:05:16.964565 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 430 0x55f66e944800 MOSDScrubReserve(11.3s5 RELEASE e509) v1 > -6880> 2018-07-11 08:05:16.964580 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 430 ==== MOSDScrubReserve(11.3s5 RELEASE e509) v1 ==== 43+0+0 (3603460268 0 0) 0x55f66e944800 con 0x55f668a29000 > -6879> 2018-07-11 08:05:17.075384 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15255 conn(0x55f663ec2800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.0 seq 190 0x55f6631cce00 osd_ping(ping e509 stamp 2018-07-11 08:05:18.319786) v4 > -6878> 2018-07-11 08:05:17.075405 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.0 10.8.128.113:0/15255 190 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:18.319786) v4 ==== 2004+0+0 (3445501465 0 0) 0x55f6631cce00 con 0x55f663ec2800 > -6877> 2018-07-11 08:05:17.075421 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:18.319786) v4 -- 0x55f673971200 con 0 > -6876> 2018-07-11 08:05:17.075415 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15255 conn(0x55f663ec1000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.0 seq 190 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:05:18.319786) v4 > -6875> 2018-07-11 08:05:17.075434 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.0 10.8.128.113:0/15255 190 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:18.319786) v4 ==== 2004+0+0 (3445501465 0 0) 0x55f67d1afe00 con 0x55f663ec1000 > -6874> 2018-07-11 08:05:17.075448 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:18.319786) v4 -- 0x55f66e944000 con 0 > -6873> 2018-07-11 08:05:17.308450 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.115:0/13323 conn(0x55f663ebe000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.6 seq 185 0x55f673970e00 osd_ping(ping e509 stamp 2018-07-11 08:05:17.660955) v4 > -6872> 2018-07-11 08:05:17.308471 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.6 10.8.128.115:0/13323 185 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:17.660955) v4 ==== 2004+0+0 (4256910041 0 0) 0x55f673970e00 con 0x55f663ebe000 > -6871> 2018-07-11 08:05:17.308480 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:17.660955) v4 -- 0x55f673971200 con 0 > -6870> 2018-07-11 08:05:17.308473 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.115:0/13323 conn(0x55f663ebf800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.6 seq 185 0x55f65d024600 osd_ping(ping e509 stamp 2018-07-11 08:05:17.660955) v4 > -6869> 2018-07-11 08:05:17.308499 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.6 10.8.128.115:0/13323 185 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:17.660955) v4 ==== 2004+0+0 (4256910041 0 0) 0x55f65d024600 con 0x55f663ebf800 > -6868> 2018-07-11 08:05:17.308516 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:17.660955) v4 -- 0x55f64cf24200 con 0 > -6867> 2018-07-11 08:05:17.335597 7f491803c700 5 osd.10 509 heartbeat: osd_stat(789 GB used, 141 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,0,0,0,1]) > -6866> 2018-07-11 08:05:17.335612 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6802/15255 -- osd_ping(ping e509 stamp 2018-07-11 08:05:17.335610) v4 -- 0x55f671426600 con 0 > -6865> 2018-07-11 08:05:17.335635 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6803/15255 -- osd_ping(ping e509 stamp 2018-07-11 08:05:17.335610) v4 -- 0x55f671426800 con 0 > -6864> 2018-07-11 08:05:17.335662 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6806/12677 -- osd_ping(ping e509 stamp 2018-07-11 08:05:17.335610) v4 -- 0x55f671426a00 con 0 > -6863> 2018-07-11 08:05:17.335677 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6807/12677 -- osd_ping(ping e509 stamp 2018-07-11 08:05:17.335610) v4 -- 0x55f671426c00 con 0 > -6862> 2018-07-11 08:05:17.335690 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6808/12651 -- osd_ping(ping e509 stamp 2018-07-11 08:05:17.335610) v4 -- 0x55f671426e00 con 0 > -6861> 2018-07-11 08:05:17.335701 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6810/12651 -- osd_ping(ping e509 stamp 2018-07-11 08:05:17.335610) v4 -- 0x55f671427000 con 0 > -6860> 2018-07-11 08:05:17.335713 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6809/12650 -- osd_ping(ping e509 stamp 2018-07-11 08:05:17.335610) v4 -- 0x55f671427200 con 0 > -6859> 2018-07-11 08:05:17.335731 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6811/12650 -- osd_ping(ping e509 stamp 2018-07-11 08:05:17.335610) v4 -- 0x55f671427400 con 0 > -6858> 2018-07-11 08:05:17.335745 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6810/15287 -- osd_ping(ping e509 stamp 2018-07-11 08:05:17.335610) v4 -- 0x55f671427600 con 0 > -6857> 2018-07-11 08:05:17.335757 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6811/15287 -- osd_ping(ping e509 stamp 2018-07-11 08:05:17.335610) v4 -- 0x55f671427800 con 0 > -6856> 2018-07-11 08:05:17.335770 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6806/15263 -- osd_ping(ping e509 stamp 2018-07-11 08:05:17.335610) v4 -- 0x55f671427a00 con 0 > -6855> 2018-07-11 08:05:17.335784 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6807/15263 -- osd_ping(ping e509 stamp 2018-07-11 08:05:17.335610) v4 -- 0x55f671427c00 con 0 > -6854> 2018-07-11 08:05:17.335796 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.115:6802/13323 -- osd_ping(ping e509 stamp 2018-07-11 08:05:17.335610) v4 -- 0x55f671427e00 con 0 > -6853> 2018-07-11 08:05:17.335810 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.115:6803/13323 -- osd_ping(ping e509 stamp 2018-07-11 08:05:17.335610) v4 -- 0x55f650806000 con 0 > -6852> 2018-07-11 08:05:17.335826 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6806/15885 -- osd_ping(ping e509 stamp 2018-07-11 08:05:17.335610) v4 -- 0x55f650806200 con 0 > -6851> 2018-07-11 08:05:17.335839 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6807/15885 -- osd_ping(ping e509 stamp 2018-07-11 08:05:17.335610) v4 -- 0x55f650806400 con 0 > -6850> 2018-07-11 08:05:17.335853 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.108:6802/11967 -- osd_ping(ping e509 stamp 2018-07-11 08:05:17.335610) v4 -- 0x55f650806600 con 0 > -6849> 2018-07-11 08:05:17.335865 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.108:6803/11967 -- osd_ping(ping e509 stamp 2018-07-11 08:05:17.335610) v4 -- 0x55f650806800 con 0 > -6848> 2018-07-11 08:05:17.335878 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.107:6806/12617 -- osd_ping(ping e509 stamp 2018-07-11 08:05:17.335610) v4 -- 0x55f650806a00 con 0 > -6847> 2018-07-11 08:05:17.335891 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.107:6807/12617 -- osd_ping(ping e509 stamp 2018-07-11 08:05:17.335610) v4 -- 0x55f650806c00 con 0 > -6846> 2018-07-11 08:05:17.335903 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6802/15919 -- osd_ping(ping e509 stamp 2018-07-11 08:05:17.335610) v4 -- 0x55f650806e00 con 0 > -6845> 2018-07-11 08:05:17.335915 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6803/15919 -- osd_ping(ping e509 stamp 2018-07-11 08:05:17.335610) v4 -- 0x55f650807000 con 0 > -6844> 2018-07-11 08:05:17.336018 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6803/15255 conn(0x55f656ad5800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 196 0x55f673970e00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:17.335610) v4 > -6843> 2018-07-11 08:05:17.336016 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6802/15255 conn(0x55f656ad4000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 196 0x55f65c0f3a00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:17.335610) v4 > -6842> 2018-07-11 08:05:17.336045 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.0 10.8.128.113:6803/15255 196 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:17.335610) v4 ==== 2004+0+0 (3213318654 0 0) 0x55f673970e00 con 0x55f656ad5800 > -6841> 2018-07-11 08:05:17.336047 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.0 10.8.128.113:6802/15255 196 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:17.335610) v4 ==== 2004+0+0 (3213318654 0 0) 0x55f65c0f3a00 con 0x55f656ad4000 > -6840> 2018-07-11 08:05:17.336091 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.107:6806/12617 conn(0x55f656ade000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.9 seq 196 0x55f673970e00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:17.335610) v4 > -6839> 2018-07-11 08:05:17.336112 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.9 10.8.128.107:6806/12617 196 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:17.335610) v4 ==== 2004+0+0 (3213318654 0 0) 0x55f673970e00 con 0x55f656ade000 > -6838> 2018-07-11 08:05:17.336120 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6807/12677 conn(0x55f64e3d9800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.1 seq 196 0x55f65c0f3a00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:17.335610) v4 > -6837> 2018-07-11 08:05:17.336153 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6809/12650 conn(0x55f656ad7000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.3 seq 196 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:17.335610) v4 > -6836> 2018-07-11 08:05:17.336174 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.3 10.8.128.106:6809/12650 196 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:17.335610) v4 ==== 2004+0+0 (3213318654 0 0) 0x55f64e20fc00 con 0x55f656ad7000 > -6835> 2018-07-11 08:05:17.336204 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6810/12651 conn(0x55f64e3eb800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.2 seq 196 0x55f673970e00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:17.335610) v4 > -6834> 2018-07-11 08:05:17.336138 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.1 10.8.128.106:6807/12677 196 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:17.335610) v4 ==== 2004+0+0 (3213318654 0 0) 0x55f65c0f3a00 con 0x55f64e3d9800 > -6833> 2018-07-11 08:05:17.336217 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.2 10.8.128.106:6810/12651 196 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:17.335610) v4 ==== 2004+0+0 (3213318654 0 0) 0x55f673970e00 con 0x55f64e3eb800 > -6832> 2018-07-11 08:05:17.336239 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.107:6807/12617 conn(0x55f656adf800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.9 seq 196 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:17.335610) v4 > -6831> 2018-07-11 08:05:17.336266 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.9 10.8.128.107:6807/12617 196 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:17.335610) v4 ==== 2004+0+0 (3213318654 0 0) 0x55f64e20fc00 con 0x55f656adf800 > -6830> 2018-07-11 08:05:17.336287 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6806/12677 conn(0x55f64e3d8000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.1 seq 196 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:17.335610) v4 > -6829> 2018-07-11 08:05:17.336297 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.1 10.8.128.106:6806/12677 196 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:17.335610) v4 ==== 2004+0+0 (3213318654 0 0) 0x55f64e20fc00 con 0x55f64e3d8000 > -6828> 2018-07-11 08:05:17.336292 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.115:6802/13323 conn(0x55f656acd000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.6 seq 196 0x55f673970e00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:17.335610) v4 > -6827> 2018-07-11 08:05:17.336292 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6808/12651 conn(0x55f64e3ea000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.2 seq 196 0x55f65c0f3a00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:17.335610) v4 > -6826> 2018-07-11 08:05:17.336315 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6806/15263 conn(0x55f64e3ed000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.5 seq 196 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:17.335610) v4 > -6825> 2018-07-11 08:05:17.336318 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.6 10.8.128.115:6802/13323 196 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:17.335610) v4 ==== 2004+0+0 (3213318654 0 0) 0x55f673970e00 con 0x55f656acd000 > -6824> 2018-07-11 08:05:17.336321 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.2 10.8.128.106:6808/12651 196 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:17.335610) v4 ==== 2004+0+0 (3213318654 0 0) 0x55f65c0f3a00 con 0x55f64e3ea000 > -6823> 2018-07-11 08:05:17.336324 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.5 10.8.128.113:6806/15263 196 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:17.335610) v4 ==== 2004+0+0 (3213318654 0 0) 0x55f64e20fc00 con 0x55f64e3ed000 > -6822> 2018-07-11 08:05:17.336375 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6811/12650 conn(0x55f656ad8800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.3 seq 196 0x55f65c0f3a00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:17.335610) v4 > -6821> 2018-07-11 08:05:17.336381 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.115:6803/13323 conn(0x55f656ace800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.6 seq 196 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:17.335610) v4 > -6820> 2018-07-11 08:05:17.336376 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6806/15885 conn(0x55f64e3fe000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.7 seq 196 0x55f673970e00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:17.335610) v4 > -6819> 2018-07-11 08:05:17.336400 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.6 10.8.128.115:6803/13323 196 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:17.335610) v4 ==== 2004+0+0 (3213318654 0 0) 0x55f64e20fc00 con 0x55f656ace800 > -6818> 2018-07-11 08:05:17.336398 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.3 10.8.128.106:6811/12650 196 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:17.335610) v4 ==== 2004+0+0 (3213318654 0 0) 0x55f65c0f3a00 con 0x55f656ad8800 > -6817> 2018-07-11 08:05:17.336407 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.7 10.8.128.114:6806/15885 196 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:17.335610) v4 ==== 2004+0+0 (3213318654 0 0) 0x55f673970e00 con 0x55f64e3fe000 > -6816> 2018-07-11 08:05:17.336443 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6811/15287 conn(0x55f64e402800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.4 seq 196 0x55f673970e00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:17.335610) v4 > -6815> 2018-07-11 08:05:17.336453 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6810/15287 conn(0x55f64e401000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.4 seq 196 0x55f65c0f3a00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:17.335610) v4 > -6814> 2018-07-11 08:05:17.336458 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.108:6802/11967 conn(0x55f656aca000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.8 seq 196 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:17.335610) v4 > -6813> 2018-07-11 08:05:17.336462 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.4 10.8.128.113:6811/15287 196 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:17.335610) v4 ==== 2004+0+0 (3213318654 0 0) 0x55f673970e00 con 0x55f64e402800 > -6812> 2018-07-11 08:05:17.336471 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.8 10.8.128.108:6802/11967 196 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:17.335610) v4 ==== 2004+0+0 (3213318654 0 0) 0x55f64e20fc00 con 0x55f656aca000 > -6811> 2018-07-11 08:05:17.336470 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.4 10.8.128.113:6810/15287 196 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:17.335610) v4 ==== 2004+0+0 (3213318654 0 0) 0x55f65c0f3a00 con 0x55f64e401000 > -6810> 2018-07-11 08:05:17.336506 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6807/15885 conn(0x55f64e3ff800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.7 seq 196 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:17.335610) v4 > -6809> 2018-07-11 08:05:17.336505 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6802/15919 conn(0x55f64e3db000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.11 seq 196 0x55f673970e00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:17.335610) v4 > -6808> 2018-07-11 08:05:17.336519 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.7 10.8.128.114:6807/15885 196 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:17.335610) v4 ==== 2004+0+0 (3213318654 0 0) 0x55f64e20fc00 con 0x55f64e3ff800 > -6807> 2018-07-11 08:05:17.336521 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.11 10.8.128.114:6802/15919 196 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:17.335610) v4 ==== 2004+0+0 (3213318654 0 0) 0x55f673970e00 con 0x55f64e3db000 > -6806> 2018-07-11 08:05:17.336522 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6807/15263 conn(0x55f64e3ee800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.5 seq 196 0x55f65c0f3a00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:17.335610) v4 > -6805> 2018-07-11 08:05:17.336543 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.5 10.8.128.113:6807/15263 196 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:17.335610) v4 ==== 2004+0+0 (3213318654 0 0) 0x55f65c0f3a00 con 0x55f64e3ee800 > -6804> 2018-07-11 08:05:17.336548 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6803/15919 conn(0x55f64e3dc800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.11 seq 196 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:17.335610) v4 > -6803> 2018-07-11 08:05:17.336565 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.11 10.8.128.114:6803/15919 196 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:17.335610) v4 ==== 2004+0+0 (3213318654 0 0) 0x55f64e20fc00 con 0x55f64e3dc800 > -6802> 2018-07-11 08:05:17.336585 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.108:6803/11967 conn(0x55f656acb800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.8 seq 196 0x55f65c0f3a00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:17.335610) v4 > -6801> 2018-07-11 08:05:17.336598 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.8 10.8.128.108:6803/11967 196 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:17.335610) v4 ==== 2004+0+0 (3213318654 0 0) 0x55f65c0f3a00 con 0x55f656acb800 > -6800> 2018-07-11 08:05:17.434897 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 29] Generated table #37038: 2638242 keys, 74546174 bytes > -6799> 2018-07-11 08:05:17.434944 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296317434928, "cf_name": "default", "job": 29, "event": "table_file_creation", "file_number": 37038, "file_size": 74546174, "table_properties": {"data_size": 67112168, "index_size": 689830, "filter_size": 6743186, "raw_key_size": 76215884, "raw_average_key_size": 28, "raw_value_size": 31365770, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638242, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} > -6798> 2018-07-11 08:05:17.474277 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed7800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 195 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:05:17.530865) v4 > -6797> 2018-07-11 08:05:17.474299 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.1 10.8.128.106:0/12677 195 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:17.530865) v4 ==== 2004+0+0 (3195718379 0 0) 0x55f64e20fc00 con 0x55f663ed7800 > -6796> 2018-07-11 08:05:17.474309 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:17.530865) v4 -- 0x55f671427200 con 0 > -6795> 2018-07-11 08:05:17.474316 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed6000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 195 0x55f65c0f3a00 osd_ping(ping e509 stamp 2018-07-11 08:05:17.530865) v4 > -6794> 2018-07-11 08:05:17.474344 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.1 10.8.128.106:0/12677 195 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:17.530865) v4 ==== 2004+0+0 (3195718379 0 0) 0x55f65c0f3a00 con 0x55f663ed6000 > -6793> 2018-07-11 08:05:17.474365 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:17.530865) v4 -- 0x55f671426e00 con 0 > -6792> 2018-07-11 08:05:18.001946 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0a000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 192 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:05:18.637445) v4 > -6791> 2018-07-11 08:05:18.001968 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.11 10.8.128.114:0/15919 192 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:18.637445) v4 ==== 2004+0+0 (3823478640 0 0) 0x55f67d1afe00 con 0x55f663f0a000 > -6790> 2018-07-11 08:05:18.001987 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:18.637445) v4 -- 0x55f671427200 con 0 > -6789> 2018-07-11 08:05:18.001977 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0b800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 192 0x55f673970e00 osd_ping(ping e509 stamp 2018-07-11 08:05:18.637445) v4 > -6788> 2018-07-11 08:05:18.002011 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.11 10.8.128.114:0/15919 192 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:18.637445) v4 ==== 2004+0+0 (3823478640 0 0) 0x55f673970e00 con 0x55f663f0b800 > -6787> 2018-07-11 08:05:18.002024 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:18.637445) v4 -- 0x55f671427000 con 0 > -6786> 2018-07-11 08:05:18.183472 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.107:0/12617 conn(0x55f64e3c3800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.9 seq 195 0x55f6631cce00 osd_ping(ping e509 stamp 2018-07-11 08:05:18.183077) v4 > -6785> 2018-07-11 08:05:18.183485 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.107:0/12617 conn(0x55f64e3c5000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.9 seq 195 0x55f65d024600 osd_ping(ping e509 stamp 2018-07-11 08:05:18.183077) v4 > -6784> 2018-07-11 08:05:18.183497 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.9 10.8.128.107:0/12617 195 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:18.183077) v4 ==== 2004+0+0 (491557340 0 0) 0x55f6631cce00 con 0x55f64e3c3800 > -6783> 2018-07-11 08:05:18.183513 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.9 10.8.128.107:0/12617 195 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:18.183077) v4 ==== 2004+0+0 (491557340 0 0) 0x55f65d024600 con 0x55f64e3c5000 > -6782> 2018-07-11 08:05:18.183516 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:18.183077) v4 -- 0x55f671427000 con 0 > -6781> 2018-07-11 08:05:18.183569 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:18.183077) v4 -- 0x55f671426e00 con 0 > -6780> 2018-07-11 08:05:18.253839 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0e800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 200 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:05:19.498303) v4 > -6779> 2018-07-11 08:05:18.253860 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.5 10.8.128.113:0/15263 200 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:19.498303) v4 ==== 2004+0+0 (2072040561 0 0) 0x55f64e20fc00 con 0x55f663f0e800 > -6778> 2018-07-11 08:05:18.253870 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:19.498303) v4 -- 0x55f671427200 con 0 > -6777> 2018-07-11 08:05:18.253866 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0d000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 200 0x55f65c0f3a00 osd_ping(ping e509 stamp 2018-07-11 08:05:19.498303) v4 > -6776> 2018-07-11 08:05:18.253883 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.5 10.8.128.113:0/15263 200 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:19.498303) v4 ==== 2004+0+0 (2072040561 0 0) 0x55f65c0f3a00 con 0x55f663f0d000 > -6775> 2018-07-11 08:05:18.253892 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:19.498303) v4 -- 0x55f671426e00 con 0 > -6774> 2018-07-11 08:05:18.391281 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f669fa8480 con 0 > -6773> 2018-07-11 08:05:18.392993 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 213 0x55f67c239400 replica scrub(pg: 6.15,from:0'0,to:430'105821,epoch:509/504,start:6:a83096c1:::benchmark_data_magna108_35617_object343825:0,end:6:a83193fd:::benchmark_data_magna108.ceph.redhat.com_2478_object278319:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -6772> 2018-07-11 08:05:18.393016 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 213 ==== replica scrub(pg: 6.15,from:0'0,to:430'105821,epoch:509/504,start:6:a83096c1:::benchmark_data_magna108_35617_object343825:0,end:6:a83193fd:::benchmark_data_magna108.ceph.redhat.com_2478_object278319:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (1393233701 0 0) 0x55f67c239400 con 0x55f66315e800 > -6771> 2018-07-11 08:05:18.497491 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.108:0/11967 conn(0x55f663f70800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=1). rx osd.8 seq 189 0x55f65d024600 osd_ping(ping e509 stamp 2018-07-11 08:05:18.257338) v4 > -6770> 2018-07-11 08:05:18.497489 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.108:0/11967 conn(0x55f663f6f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=1). rx osd.8 seq 189 0x55f673970e00 osd_ping(ping e509 stamp 2018-07-11 08:05:18.257338) v4 > -6769> 2018-07-11 08:05:18.497516 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.8 10.8.128.108:0/11967 189 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:18.257338) v4 ==== 2004+0+0 (244543973 0 0) 0x55f65d024600 con 0x55f663f70800 > -6768> 2018-07-11 08:05:18.497519 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.8 10.8.128.108:0/11967 189 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:18.257338) v4 ==== 2004+0+0 (244543973 0 0) 0x55f673970e00 con 0x55f663f6f000 > -6767> 2018-07-11 08:05:18.497526 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:18.257338) v4 -- 0x55f671426e00 con 0 > -6766> 2018-07-11 08:05:18.497548 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:18.257338) v4 -- 0x55f671427000 con 0 > -6765> 2018-07-11 08:05:18.775814 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15255 conn(0x55f663ec1000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.0 seq 191 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:05:20.020185) v4 > -6764> 2018-07-11 08:05:18.775844 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.0 10.8.128.113:0/15255 191 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:20.020185) v4 ==== 2004+0+0 (462277823 0 0) 0x55f67d1afe00 con 0x55f663ec1000 > -6763> 2018-07-11 08:05:18.775860 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:20.020185) v4 -- 0x55f671427200 con 0 > -6762> 2018-07-11 08:05:18.775852 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15255 conn(0x55f663ec2800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.0 seq 191 0x55f6631cce00 osd_ping(ping e509 stamp 2018-07-11 08:05:20.020185) v4 > -6761> 2018-07-11 08:05:18.775873 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.0 10.8.128.113:0/15255 191 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:20.020185) v4 ==== 2004+0+0 (462277823 0 0) 0x55f6631cce00 con 0x55f663ec2800 > -6760> 2018-07-11 08:05:18.775886 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:20.020185) v4 -- 0x55f671427000 con 0 > -6759> 2018-07-11 08:05:18.831640 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15885 conn(0x55f663217800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 199 0x55f673970e00 osd_ping(ping e509 stamp 2018-07-11 08:05:19.467204) v4 > -6758> 2018-07-11 08:05:18.831646 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15885 conn(0x55f663216000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 199 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:05:19.467204) v4 > -6757> 2018-07-11 08:05:18.831660 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.7 10.8.128.114:0/15885 199 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:19.467204) v4 ==== 2004+0+0 (2485381528 0 0) 0x55f673970e00 con 0x55f663217800 > -6756> 2018-07-11 08:05:18.831670 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:19.467204) v4 -- 0x55f671427000 con 0 > -6755> 2018-07-11 08:05:18.831665 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.7 10.8.128.114:0/15885 199 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:19.467204) v4 ==== 2004+0+0 (2485381528 0 0) 0x55f64e20fc00 con 0x55f663216000 > -6754> 2018-07-11 08:05:18.831692 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:19.467204) v4 -- 0x55f671427200 con 0 > -6753> 2018-07-11 08:05:18.964469 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 431 0x55f6631cce00 MOSDScrubReserve(11.3s5 REQUEST e509) v1 > -6752> 2018-07-11 08:05:18.964488 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 431 ==== MOSDScrubReserve(11.3s5 REQUEST e509) v1 ==== 43+0+0 (433425070 0 0) 0x55f6631cce00 con 0x55f668a29000 > -6751> 2018-07-11 08:05:18.964570 7f491a040700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e509) v1 -- 0x55f66e944800 con 0 > -6750> 2018-07-11 08:05:18.964914 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 432 0x55f66e944800 MOSDScrubReserve(11.3s5 RELEASE e509) v1 > -6749> 2018-07-11 08:05:18.964928 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 432 ==== MOSDScrubReserve(11.3s5 RELEASE e509) v1 ==== 43+0+0 (3603460268 0 0) 0x55f66e944800 con 0x55f668a29000 > -6748> 2018-07-11 08:05:19.100802 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15287 conn(0x55f66dce2800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.4 seq 201 0x55f65c0f3a00 osd_ping(ping e509 stamp 2018-07-11 08:05:20.345312) v4 > -6747> 2018-07-11 08:05:19.100827 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.4 10.8.128.113:0/15287 201 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:20.345312) v4 ==== 2004+0+0 (2042080181 0 0) 0x55f65c0f3a00 con 0x55f66dce2800 > -6746> 2018-07-11 08:05:19.100839 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:20.345312) v4 -- 0x55f671426c00 con 0 > -6745> 2018-07-11 08:05:19.100936 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15287 conn(0x55f66dce1000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.4 seq 201 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:05:20.345312) v4 > -6744> 2018-07-11 08:05:19.100952 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.4 10.8.128.113:0/15287 201 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:20.345312) v4 ==== 2004+0+0 (2042080181 0 0) 0x55f67d1afe00 con 0x55f66dce1000 > -6743> 2018-07-11 08:05:19.100960 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:20.345312) v4 -- 0x55f671426a00 con 0 > -6742> 2018-07-11 08:05:19.424813 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f6637d8900 con 0 > -6741> 2018-07-11 08:05:19.426471 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 214 0x55f67c239680 replica scrub(pg: 6.15,from:0'0,to:430'106784,epoch:509/504,start:6:a83193fd:::benchmark_data_magna108.ceph.redhat.com_2478_object278319:0,end:6:a832e164:::benchmark_data_magna108_35617_object962853:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -6740> 2018-07-11 08:05:19.426495 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 214 ==== replica scrub(pg: 6.15,from:0'0,to:430'106784,epoch:509/504,start:6:a83193fd:::benchmark_data_magna108.ceph.redhat.com_2478_object278319:0,end:6:a832e164:::benchmark_data_magna108_35617_object962853:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (1511984187 0 0) 0x55f67c239680 con 0x55f66315e800 > -6739> 2018-07-11 08:05:19.428910 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12650 conn(0x55f663eda800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.3 seq 195 0x55f65d024600 osd_ping(ping e509 stamp 2018-07-11 08:05:19.485445) v4 > -6738> 2018-07-11 08:05:19.428934 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.3 10.8.128.106:0/12650 195 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:19.485445) v4 ==== 2004+0+0 (2332101708 0 0) 0x55f65d024600 con 0x55f663eda800 > -6737> 2018-07-11 08:05:19.428952 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:19.485445) v4 -- 0x55f671426600 con 0 > -6736> 2018-07-11 08:05:19.428943 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12650 conn(0x55f663ed9000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.3 seq 195 0x55f673970e00 osd_ping(ping e509 stamp 2018-07-11 08:05:19.485445) v4 > -6735> 2018-07-11 08:05:19.428967 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.3 10.8.128.106:0/12650 195 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:19.485445) v4 ==== 2004+0+0 (2332101708 0 0) 0x55f673970e00 con 0x55f663ed9000 > -6734> 2018-07-11 08:05:19.428981 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:19.485445) v4 -- 0x55f673971400 con 0 > -6733> 2018-07-11 08:05:19.661397 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.114:6805/15885 conn(0x55f66315a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.7 seq 2541 0x55f671426c00 MOSDScrubReserve(5.4 REQUEST e509) v1 > -6732> 2018-07-11 08:05:19.661424 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.7 10.8.128.114:6805/15885 2541 ==== MOSDScrubReserve(5.4 REQUEST e509) v1 ==== 43+0+0 (3003630469 0 0) 0x55f671426c00 con 0x55f66315a000 > -6731> 2018-07-11 08:05:19.661462 7f491983f700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.114:6805/15885 -- MOSDScrubReserve(5.4 REJECT e509) v1 -- 0x55f660ea5800 con 0 > -6730> 2018-07-11 08:05:19.661807 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.114:6805/15885 conn(0x55f66315a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.7 seq 2542 0x55f660ea5800 MOSDScrubReserve(5.4 RELEASE e509) v1 > -6729> 2018-07-11 08:05:19.661825 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.7 10.8.128.114:6805/15885 2542 ==== MOSDScrubReserve(5.4 RELEASE e509) v1 ==== 43+0+0 (2082093447 0 0) 0x55f660ea5800 con 0x55f66315a000 > -6728> 2018-07-11 08:05:19.929302 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12650 conn(0x55f663ed9000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.3 seq 196 0x55f673971200 osd_ping(ping e509 stamp 2018-07-11 08:05:19.985970) v4 > -6727> 2018-07-11 08:05:19.929327 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.3 10.8.128.106:0/12650 196 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:19.985970) v4 ==== 2004+0+0 (2438483895 0 0) 0x55f673971200 con 0x55f663ed9000 > -6726> 2018-07-11 08:05:19.929338 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:19.985970) v4 -- 0x55f673971600 con 0 > -6725> 2018-07-11 08:05:19.929329 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12650 conn(0x55f663eda800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.3 seq 196 0x55f65d024600 osd_ping(ping e509 stamp 2018-07-11 08:05:19.985970) v4 > -6724> 2018-07-11 08:05:19.929347 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.3 10.8.128.106:0/12650 196 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:19.985970) v4 ==== 2004+0+0 (2438483895 0 0) 0x55f65d024600 con 0x55f663eda800 > -6723> 2018-07-11 08:05:19.929360 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:19.985970) v4 -- 0x55f653d4aa00 con 0 > -6722> 2018-07-11 08:05:19.964680 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 433 0x55f673970e00 MOSDScrubReserve(11.3s5 REQUEST e509) v1 > -6721> 2018-07-11 08:05:19.964700 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 433 ==== MOSDScrubReserve(11.3s5 REQUEST e509) v1 ==== 43+0+0 (433425070 0 0) 0x55f673970e00 con 0x55f668a29000 > -6720> 2018-07-11 08:05:19.964778 7f491a040700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e509) v1 -- 0x55f66e944800 con 0 > -6719> 2018-07-11 08:05:19.965171 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 434 0x55f66e944800 MOSDScrubReserve(11.3s5 RELEASE e509) v1 > -6718> 2018-07-11 08:05:19.965197 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 434 ==== MOSDScrubReserve(11.3s5 RELEASE e509) v1 ==== 43+0+0 (3603460268 0 0) 0x55f66e944800 con 0x55f668a29000 > -6717> 2018-07-11 08:05:20.130151 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f66663c6c0 con 0 > -6716> 2018-07-11 08:05:20.131724 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 215 0x55f67c239900 replica scrub(pg: 6.15,from:0'0,to:430'106104,epoch:509/504,start:6:a832e164:::benchmark_data_magna108_35617_object962853:0,end:6:a834254b:::benchmark_data_magna108_35617_object905840:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -6715> 2018-07-11 08:05:20.131756 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 215 ==== replica scrub(pg: 6.15,from:0'0,to:430'106104,epoch:509/504,start:6:a832e164:::benchmark_data_magna108_35617_object962853:0,end:6:a834254b:::benchmark_data_magna108_35617_object905840:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (2062799482 0 0) 0x55f67c239900 con 0x55f66315e800 > -6714> 2018-07-11 08:05:20.208804 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.115:0/13323 conn(0x55f663ebf800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.6 seq 186 0x55f671426600 osd_ping(ping e509 stamp 2018-07-11 08:05:20.561383) v4 > -6713> 2018-07-11 08:05:20.208824 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.115:0/13323 conn(0x55f663ebe000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.6 seq 186 0x55f673971200 osd_ping(ping e509 stamp 2018-07-11 08:05:20.561383) v4 > -6712> 2018-07-11 08:05:20.208832 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.6 10.8.128.115:0/13323 186 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:20.561383) v4 ==== 2004+0+0 (2381296121 0 0) 0x55f671426600 con 0x55f663ebf800 > -6711> 2018-07-11 08:05:20.208842 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.6 10.8.128.115:0/13323 186 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:20.561383) v4 ==== 2004+0+0 (2381296121 0 0) 0x55f673971200 con 0x55f663ebe000 > -6710> 2018-07-11 08:05:20.208849 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:20.561383) v4 -- 0x55f6631cda00 con 0 > -6709> 2018-07-11 08:05:20.208881 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:20.561383) v4 -- 0x55f673971c00 con 0 > -6708> 2018-07-11 08:05:20.374727 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed7800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 196 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:05:20.431368) v4 > -6707> 2018-07-11 08:05:20.374763 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.1 10.8.128.106:0/12677 196 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:20.431368) v4 ==== 2004+0+0 (3554075216 0 0) 0x55f64e20fc00 con 0x55f663ed7800 > -6706> 2018-07-11 08:05:20.374775 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:20.431368) v4 -- 0x55f671426a00 con 0 > -6705> 2018-07-11 08:05:20.374852 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed6000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 196 0x55f653d4aa00 osd_ping(ping e509 stamp 2018-07-11 08:05:20.431368) v4 > -6704> 2018-07-11 08:05:20.374876 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.1 10.8.128.106:0/12677 196 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:20.431368) v4 ==== 2004+0+0 (3554075216 0 0) 0x55f653d4aa00 con 0x55f663ed6000 > -6703> 2018-07-11 08:05:20.374892 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:20.431368) v4 -- 0x55f6631cd600 con 0 > -6702> 2018-07-11 08:05:20.433490 7f491e048700 10 monclient: tick > -6701> 2018-07-11 08:05:20.433504 7f491e048700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2018-07-11 08:04:50.433503) > -6700> 2018-07-11 08:05:20.527733 7f491d046700 1 -- 10.8.128.107:6800/14173 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55f66398e580 con 0 > -6699> 2018-07-11 08:05:20.527805 7f491d046700 1 -- 10.8.128.107:6800/14173 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55f64cc76b00 con 0 > -6698> 2018-07-11 08:05:20.858661 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f66affab40 con 0 > -6697> 2018-07-11 08:05:20.860168 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 216 0x55f67c239b80 replica scrub(pg: 6.15,from:0'0,to:430'106478,epoch:509/504,start:6:a834254b:::benchmark_data_magna108_35617_object905840:0,end:6:a8357a54:::benchmark_data_magna108.ceph.redhat.com_2478_object100080:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -6696> 2018-07-11 08:05:20.860199 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 216 ==== replica scrub(pg: 6.15,from:0'0,to:430'106478,epoch:509/504,start:6:a834254b:::benchmark_data_magna108_35617_object905840:0,end:6:a8357a54:::benchmark_data_magna108.ceph.redhat.com_2478_object100080:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (954898923 0 0) 0x55f67c239b80 con 0x55f66315e800 > -6695> 2018-07-11 08:05:20.951136 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12651 conn(0x55f663219000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25 cs=1 l=1). rx osd.2 seq 205 0x55f653d4aa00 osd_ping(ping e509 stamp 2018-07-11 08:05:21.007746) v4 > -6694> 2018-07-11 08:05:20.951165 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.2 10.8.128.106:0/12651 205 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:21.007746) v4 ==== 2004+0+0 (4215072729 0 0) 0x55f653d4aa00 con 0x55f663219000 > -6693> 2018-07-11 08:05:20.951183 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:21.007746) v4 -- 0x55f6631cc600 con 0 > -6692> 2018-07-11 08:05:20.951215 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12651 conn(0x55f66321a800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25 cs=1 l=1). rx osd.2 seq 205 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:05:21.007746) v4 > -6691> 2018-07-11 08:05:20.951239 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.2 10.8.128.106:0/12651 205 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:21.007746) v4 ==== 2004+0+0 (4215072729 0 0) 0x55f67d1afe00 con 0x55f66321a800 > -6690> 2018-07-11 08:05:20.951262 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:21.007746) v4 -- 0x55f671426a00 con 0 > -6689> 2018-07-11 08:05:21.132127 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15885 conn(0x55f663216000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 200 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:05:21.767668) v4 > -6688> 2018-07-11 08:05:21.132152 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.7 10.8.128.114:0/15885 200 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:21.767668) v4 ==== 2004+0+0 (1448153324 0 0) 0x55f64e20fc00 con 0x55f663216000 > -6687> 2018-07-11 08:05:21.132162 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:21.767668) v4 -- 0x55f671426a00 con 0 > -6686> 2018-07-11 08:05:21.132150 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15885 conn(0x55f663217800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 200 0x55f673971a00 osd_ping(ping e509 stamp 2018-07-11 08:05:21.767668) v4 > -6685> 2018-07-11 08:05:21.132169 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.7 10.8.128.114:0/15885 200 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:21.767668) v4 ==== 2004+0+0 (1448153324 0 0) 0x55f673971a00 con 0x55f663217800 > -6684> 2018-07-11 08:05:21.132186 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:21.767668) v4 -- 0x55f673971c00 con 0 > -6683> 2018-07-11 08:05:21.451766 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12651 conn(0x55f663219000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25 cs=1 l=1). rx osd.2 seq 206 0x55f6631cd600 osd_ping(ping e509 stamp 2018-07-11 08:05:21.508252) v4 > -6682> 2018-07-11 08:05:21.451800 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.2 10.8.128.106:0/12651 206 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:21.508252) v4 ==== 2004+0+0 (625912328 0 0) 0x55f6631cd600 con 0x55f663219000 > -6681> 2018-07-11 08:05:21.451818 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:21.508252) v4 -- 0x55f6631cc600 con 0 > -6680> 2018-07-11 08:05:21.451870 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12651 conn(0x55f66321a800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25 cs=1 l=1). rx osd.2 seq 206 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:05:21.508252) v4 > -6679> 2018-07-11 08:05:21.451891 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.2 10.8.128.106:0/12651 206 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:21.508252) v4 ==== 2004+0+0 (625912328 0 0) 0x55f67d1afe00 con 0x55f66321a800 > -6678> 2018-07-11 08:05:21.451906 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:21.508252) v4 -- 0x55f66e944000 con 0 > -6677> 2018-07-11 08:05:21.557822 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f660817b00 con 0 > -6676> 2018-07-11 08:05:21.559555 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 217 0x55f65b1fc000 replica scrub(pg: 6.15,from:0'0,to:430'105850,epoch:509/504,start:6:a8357a54:::benchmark_data_magna108.ceph.redhat.com_2478_object100080:0,end:6:a836798b:::benchmark_data_magna108_35617_object351201:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -6675> 2018-07-11 08:05:21.559586 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 217 ==== replica scrub(pg: 6.15,from:0'0,to:430'105850,epoch:509/504,start:6:a8357a54:::benchmark_data_magna108.ceph.redhat.com_2478_object100080:0,end:6:a836798b:::benchmark_data_magna108_35617_object351201:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (2083643898 0 0) 0x55f65b1fc000 con 0x55f66315e800 > -6674> 2018-07-11 08:05:22.268849 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f652de7d40 con 0 > -6673> 2018-07-11 08:05:22.270396 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 218 0x55f65b1fc280 replica scrub(pg: 6.15,from:0'0,to:430'105414,epoch:509/504,start:6:a836798b:::benchmark_data_magna108_35617_object351201:0,end:6:a83761ce:::benchmark_data_magna108.ceph.redhat.com_2478_object109641:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -6672> 2018-07-11 08:05:22.270427 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 218 ==== replica scrub(pg: 6.15,from:0'0,to:430'105414,epoch:509/504,start:6:a836798b:::benchmark_data_magna108_35617_object351201:0,end:6:a83761ce:::benchmark_data_magna108.ceph.redhat.com_2478_object109641:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (1609673480 0 0) 0x55f65b1fc280 con 0x55f66315e800 > -6671> 2018-07-11 08:05:22.702341 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0a000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 193 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:05:23.337903) v4 > -6670> 2018-07-11 08:05:22.702370 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.11 10.8.128.114:0/15919 193 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:23.337903) v4 ==== 2004+0+0 (3889219160 0 0) 0x55f64e20fc00 con 0x55f663f0a000 > -6669> 2018-07-11 08:05:22.702388 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:23.337903) v4 -- 0x55f66e944000 con 0 > -6668> 2018-07-11 08:05:22.702417 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0b800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 193 0x55f673971200 osd_ping(ping e509 stamp 2018-07-11 08:05:23.337903) v4 > -6667> 2018-07-11 08:05:22.702439 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.11 10.8.128.114:0/15919 193 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:23.337903) v4 ==== 2004+0+0 (3889219160 0 0) 0x55f673971200 con 0x55f663f0b800 > -6666> 2018-07-11 08:05:22.702453 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:23.337903) v4 -- 0x55f673971c00 con 0 > -6665> 2018-07-11 08:05:22.876223 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15255 conn(0x55f663ec2800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.0 seq 192 0x55f673971a00 osd_ping(ping e509 stamp 2018-07-11 08:05:24.120699) v4 > -6664> 2018-07-11 08:05:22.876244 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.0 10.8.128.113:0/15255 192 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:24.120699) v4 ==== 2004+0+0 (630175961 0 0) 0x55f673971a00 con 0x55f663ec2800 > -6663> 2018-07-11 08:05:22.876260 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15255 conn(0x55f663ec1000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.0 seq 192 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:05:24.120699) v4 > -6662> 2018-07-11 08:05:22.876274 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:24.120699) v4 -- 0x55f673971c00 con 0 > -6661> 2018-07-11 08:05:22.876281 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.0 10.8.128.113:0/15255 192 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:24.120699) v4 ==== 2004+0+0 (630175961 0 0) 0x55f67d1afe00 con 0x55f663ec1000 > -6660> 2018-07-11 08:05:22.876304 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:24.120699) v4 -- 0x55f66e944000 con 0 > -6659> 2018-07-11 08:05:22.954213 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0d000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 201 0x55f653d4aa00 osd_ping(ping e509 stamp 2018-07-11 08:05:24.198722) v4 > -6658> 2018-07-11 08:05:22.954233 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0e800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 201 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:05:24.198722) v4 > -6657> 2018-07-11 08:05:22.954237 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.5 10.8.128.113:0/15263 201 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:24.198722) v4 ==== 2004+0+0 (4125066920 0 0) 0x55f653d4aa00 con 0x55f663f0d000 > -6656> 2018-07-11 08:05:22.954259 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.5 10.8.128.113:0/15263 201 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:24.198722) v4 ==== 2004+0+0 (4125066920 0 0) 0x55f64e20fc00 con 0x55f663f0e800 > -6655> 2018-07-11 08:05:22.954265 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:24.198722) v4 -- 0x55f65e71dc00 con 0 > -6654> 2018-07-11 08:05:22.954300 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:24.198722) v4 -- 0x55f66d001800 con 0 > -6653> 2018-07-11 08:05:23.236053 7f491803c700 5 osd.10 509 heartbeat: osd_stat(789 GB used, 141 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,0,0,1]) > -6652> 2018-07-11 08:05:23.236067 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6802/15255 -- osd_ping(ping e509 stamp 2018-07-11 08:05:23.236066) v4 -- 0x55f650807200 con 0 > -6651> 2018-07-11 08:05:23.236104 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6803/15255 -- osd_ping(ping e509 stamp 2018-07-11 08:05:23.236066) v4 -- 0x55f650807400 con 0 > -6650> 2018-07-11 08:05:23.236120 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6806/12677 -- osd_ping(ping e509 stamp 2018-07-11 08:05:23.236066) v4 -- 0x55f650807600 con 0 > -6649> 2018-07-11 08:05:23.236135 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6807/12677 -- osd_ping(ping e509 stamp 2018-07-11 08:05:23.236066) v4 -- 0x55f650807800 con 0 > -6648> 2018-07-11 08:05:23.236148 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6808/12651 -- osd_ping(ping e509 stamp 2018-07-11 08:05:23.236066) v4 -- 0x55f650807a00 con 0 > -6647> 2018-07-11 08:05:23.236158 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6810/12651 -- osd_ping(ping e509 stamp 2018-07-11 08:05:23.236066) v4 -- 0x55f650807c00 con 0 > -6646> 2018-07-11 08:05:23.236170 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6809/12650 -- osd_ping(ping e509 stamp 2018-07-11 08:05:23.236066) v4 -- 0x55f650807e00 con 0 > -6645> 2018-07-11 08:05:23.236183 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6811/12650 -- osd_ping(ping e509 stamp 2018-07-11 08:05:23.236066) v4 -- 0x55f66064a000 con 0 > -6644> 2018-07-11 08:05:23.236201 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6810/15287 -- osd_ping(ping e509 stamp 2018-07-11 08:05:23.236066) v4 -- 0x55f66064a200 con 0 > -6643> 2018-07-11 08:05:23.236212 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6811/15287 -- osd_ping(ping e509 stamp 2018-07-11 08:05:23.236066) v4 -- 0x55f66064a400 con 0 > -6642> 2018-07-11 08:05:23.236226 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6806/15263 -- osd_ping(ping e509 stamp 2018-07-11 08:05:23.236066) v4 -- 0x55f66064a600 con 0 > -6641> 2018-07-11 08:05:23.236238 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6807/15263 -- osd_ping(ping e509 stamp 2018-07-11 08:05:23.236066) v4 -- 0x55f66064a800 con 0 > -6640> 2018-07-11 08:05:23.236268 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.115:6802/13323 -- osd_ping(ping e509 stamp 2018-07-11 08:05:23.236066) v4 -- 0x55f66064aa00 con 0 > -6639> 2018-07-11 08:05:23.236280 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.115:6803/13323 -- osd_ping(ping e509 stamp 2018-07-11 08:05:23.236066) v4 -- 0x55f66064ac00 con 0 > -6638> 2018-07-11 08:05:23.236294 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6806/15885 -- osd_ping(ping e509 stamp 2018-07-11 08:05:23.236066) v4 -- 0x55f66064ae00 con 0 > -6637> 2018-07-11 08:05:23.236307 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6807/15885 -- osd_ping(ping e509 stamp 2018-07-11 08:05:23.236066) v4 -- 0x55f66064b000 con 0 > -6636> 2018-07-11 08:05:23.236319 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.108:6802/11967 -- osd_ping(ping e509 stamp 2018-07-11 08:05:23.236066) v4 -- 0x55f66064b200 con 0 > -6635> 2018-07-11 08:05:23.236333 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.108:6803/11967 -- osd_ping(ping e509 stamp 2018-07-11 08:05:23.236066) v4 -- 0x55f66064b400 con 0 > -6634> 2018-07-11 08:05:23.236346 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.107:6806/12617 -- osd_ping(ping e509 stamp 2018-07-11 08:05:23.236066) v4 -- 0x55f66064b600 con 0 > -6633> 2018-07-11 08:05:23.236359 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.107:6807/12617 -- osd_ping(ping e509 stamp 2018-07-11 08:05:23.236066) v4 -- 0x55f66064b800 con 0 > -6632> 2018-07-11 08:05:23.236372 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6802/15919 -- osd_ping(ping e509 stamp 2018-07-11 08:05:23.236066) v4 -- 0x55f66064ba00 con 0 > -6631> 2018-07-11 08:05:23.236383 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6803/15919 -- osd_ping(ping e509 stamp 2018-07-11 08:05:23.236066) v4 -- 0x55f66064bc00 con 0 > -6630> 2018-07-11 08:05:23.236512 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6807/12677 conn(0x55f64e3d9800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.1 seq 197 0x55f653d4aa00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:23.236066) v4 > -6629> 2018-07-11 08:05:23.236536 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.1 10.8.128.106:6807/12677 197 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:23.236066) v4 ==== 2004+0+0 (1693765230 0 0) 0x55f653d4aa00 con 0x55f64e3d9800 > -6628> 2018-07-11 08:05:23.236534 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.107:6806/12617 conn(0x55f656ade000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.9 seq 197 0x55f673971a00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:23.236066) v4 > -6627> 2018-07-11 08:05:23.236557 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.9 10.8.128.107:6806/12617 197 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:23.236066) v4 ==== 2004+0+0 (1693765230 0 0) 0x55f673971a00 con 0x55f656ade000 > -6626> 2018-07-11 08:05:23.236576 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6802/15255 conn(0x55f656ad4000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 197 0x55f653d4aa00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:23.236066) v4 > -6625> 2018-07-11 08:05:23.236610 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.0 10.8.128.113:6802/15255 197 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:23.236066) v4 ==== 2004+0+0 (1693765230 0 0) 0x55f653d4aa00 con 0x55f656ad4000 > -6624> 2018-07-11 08:05:23.236634 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6803/15255 conn(0x55f656ad5800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 197 0x55f673971a00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:23.236066) v4 > -6623> 2018-07-11 08:05:23.236661 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6808/12651 conn(0x55f64e3ea000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.2 seq 197 0x55f653d4aa00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:23.236066) v4 > -6622> 2018-07-11 08:05:23.236655 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.0 10.8.128.113:6803/15255 197 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:23.236066) v4 ==== 2004+0+0 (1693765230 0 0) 0x55f673971a00 con 0x55f656ad5800 > -6621> 2018-07-11 08:05:23.236676 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6806/12677 conn(0x55f64e3d8000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.1 seq 197 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:23.236066) v4 > -6620> 2018-07-11 08:05:23.236696 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.2 10.8.128.106:6808/12651 197 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:23.236066) v4 ==== 2004+0+0 (1693765230 0 0) 0x55f653d4aa00 con 0x55f64e3ea000 > -6619> 2018-07-11 08:05:23.236719 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.1 10.8.128.106:6806/12677 197 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:23.236066) v4 ==== 2004+0+0 (1693765230 0 0) 0x55f67d1afe00 con 0x55f64e3d8000 > -6618> 2018-07-11 08:05:23.236729 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6811/15287 conn(0x55f64e402800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.4 seq 197 0x55f673971a00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:23.236066) v4 > -6617> 2018-07-11 08:05:23.236748 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6806/15263 conn(0x55f64e3ed000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.5 seq 197 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:23.236066) v4 > -6616> 2018-07-11 08:05:23.236759 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.5 10.8.128.113:6806/15263 197 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:23.236066) v4 ==== 2004+0+0 (1693765230 0 0) 0x55f67d1afe00 con 0x55f64e3ed000 > -6615> 2018-07-11 08:05:23.236755 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6810/15287 conn(0x55f64e401000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.4 seq 197 0x55f653d4aa00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:23.236066) v4 > -6614> 2018-07-11 08:05:23.236749 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.4 10.8.128.113:6811/15287 197 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:23.236066) v4 ==== 2004+0+0 (1693765230 0 0) 0x55f673971a00 con 0x55f64e402800 > -6613> 2018-07-11 08:05:23.236770 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.4 10.8.128.113:6810/15287 197 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:23.236066) v4 ==== 2004+0+0 (1693765230 0 0) 0x55f653d4aa00 con 0x55f64e401000 > -6612> 2018-07-11 08:05:23.236780 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6809/12650 conn(0x55f656ad7000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.3 seq 197 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:23.236066) v4 > -6611> 2018-07-11 08:05:23.236788 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.3 10.8.128.106:6809/12650 197 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:23.236066) v4 ==== 2004+0+0 (1693765230 0 0) 0x55f67d1afe00 con 0x55f656ad7000 > -6610> 2018-07-11 08:05:23.236809 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.107:6807/12617 conn(0x55f656adf800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.9 seq 197 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:23.236066) v4 > -6609> 2018-07-11 08:05:23.236819 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.9 10.8.128.107:6807/12617 197 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:23.236066) v4 ==== 2004+0+0 (1693765230 0 0) 0x55f67d1afe00 con 0x55f656adf800 > -6608> 2018-07-11 08:05:23.236817 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6811/12650 conn(0x55f656ad8800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.3 seq 197 0x55f653d4aa00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:23.236066) v4 > -6607> 2018-07-11 08:05:23.236817 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6810/12651 conn(0x55f64e3eb800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.2 seq 197 0x55f673971a00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:23.236066) v4 > -6606> 2018-07-11 08:05:23.236838 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.3 10.8.128.106:6811/12650 197 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:23.236066) v4 ==== 2004+0+0 (1693765230 0 0) 0x55f653d4aa00 con 0x55f656ad8800 > -6605> 2018-07-11 08:05:23.236844 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.2 10.8.128.106:6810/12651 197 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:23.236066) v4 ==== 2004+0+0 (1693765230 0 0) 0x55f673971a00 con 0x55f64e3eb800 > -6604> 2018-07-11 08:05:23.236852 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.115:6803/13323 conn(0x55f656ace800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.6 seq 197 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:23.236066) v4 > -6603> 2018-07-11 08:05:23.236865 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.6 10.8.128.115:6803/13323 197 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:23.236066) v4 ==== 2004+0+0 (1693765230 0 0) 0x55f67d1afe00 con 0x55f656ace800 > -6602> 2018-07-11 08:05:23.236887 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.115:6802/13323 conn(0x55f656acd000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.6 seq 197 0x55f673971a00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:23.236066) v4 > -6601> 2018-07-11 08:05:23.236887 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6807/15263 conn(0x55f64e3ee800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.5 seq 197 0x55f653d4aa00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:23.236066) v4 > -6600> 2018-07-11 08:05:23.236908 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.6 10.8.128.115:6802/13323 197 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:23.236066) v4 ==== 2004+0+0 (1693765230 0 0) 0x55f673971a00 con 0x55f656acd000 > -6599> 2018-07-11 08:05:23.236914 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.5 10.8.128.113:6807/15263 197 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:23.236066) v4 ==== 2004+0+0 (1693765230 0 0) 0x55f653d4aa00 con 0x55f64e3ee800 > -6598> 2018-07-11 08:05:23.236924 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6803/15919 conn(0x55f64e3dc800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.11 seq 197 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:23.236066) v4 > -6597> 2018-07-11 08:05:23.236938 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.11 10.8.128.114:6803/15919 197 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:23.236066) v4 ==== 2004+0+0 (1693765230 0 0) 0x55f67d1afe00 con 0x55f64e3dc800 > -6596> 2018-07-11 08:05:23.236960 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.108:6803/11967 conn(0x55f656acb800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.8 seq 197 0x55f653d4aa00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:23.236066) v4 > -6595> 2018-07-11 08:05:23.236962 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6802/15919 conn(0x55f64e3db000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.11 seq 197 0x55f673971a00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:23.236066) v4 > -6594> 2018-07-11 08:05:23.236968 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6807/15885 conn(0x55f64e3ff800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.7 seq 197 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:23.236066) v4 > -6593> 2018-07-11 08:05:23.236983 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.8 10.8.128.108:6803/11967 197 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:23.236066) v4 ==== 2004+0+0 (1693765230 0 0) 0x55f653d4aa00 con 0x55f656acb800 > -6592> 2018-07-11 08:05:23.236990 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.7 10.8.128.114:6807/15885 197 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:23.236066) v4 ==== 2004+0+0 (1693765230 0 0) 0x55f67d1afe00 con 0x55f64e3ff800 > -6591> 2018-07-11 08:05:23.236986 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.11 10.8.128.114:6802/15919 197 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:23.236066) v4 ==== 2004+0+0 (1693765230 0 0) 0x55f673971a00 con 0x55f64e3db000 > -6590> 2018-07-11 08:05:23.237015 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.108:6802/11967 conn(0x55f656aca000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.8 seq 197 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:23.236066) v4 > -6589> 2018-07-11 08:05:23.237025 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.8 10.8.128.108:6802/11967 197 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:23.236066) v4 ==== 2004+0+0 (1693765230 0 0) 0x55f67d1afe00 con 0x55f656aca000 > -6588> 2018-07-11 08:05:23.237037 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6806/15885 conn(0x55f64e3fe000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.7 seq 197 0x55f673971a00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:23.236066) v4 > -6587> 2018-07-11 08:05:23.237053 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.7 10.8.128.114:6806/15885 197 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:23.236066) v4 ==== 2004+0+0 (1693765230 0 0) 0x55f673971a00 con 0x55f64e3fe000 > -6586> 2018-07-11 08:05:23.426921 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 29] Generated table #37039: 2638241 keys, 74545999 bytes > -6585> 2018-07-11 08:05:23.426958 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296323426942, "cf_name": "default", "job": 29, "event": "table_file_creation", "file_number": 37039, "file_size": 74545999, "table_properties": {"data_size": 67112153, "index_size": 689673, "filter_size": 6743183, "raw_key_size": 76215854, "raw_average_key_size": 28, "raw_value_size": 31365757, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} > -6584> 2018-07-11 08:05:23.454597 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0d000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 202 0x55f653d4aa00 osd_ping(ping e509 stamp 2018-07-11 08:05:24.699128) v4 > -6583> 2018-07-11 08:05:23.454620 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.5 10.8.128.113:0/15263 202 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:24.699128) v4 ==== 2004+0+0 (2240809331 0 0) 0x55f653d4aa00 con 0x55f663f0d000 > -6582> 2018-07-11 08:05:23.454632 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:24.699128) v4 -- 0x55f65e71dc00 con 0 > -6581> 2018-07-11 08:05:23.454742 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0e800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 202 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:05:24.699128) v4 > -6580> 2018-07-11 08:05:23.454771 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.5 10.8.128.113:0/15263 202 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:24.699128) v4 ==== 2004+0+0 (2240809331 0 0) 0x55f67d1afe00 con 0x55f663f0e800 > -6579> 2018-07-11 08:05:23.454789 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:24.699128) v4 -- 0x55f650807e00 con 0 > -6578> 2018-07-11 08:05:23.709301 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.115:0/13323 conn(0x55f663ebf800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.6 seq 187 0x55f6631cc600 osd_ping(ping e509 stamp 2018-07-11 08:05:24.061853) v4 > -6577> 2018-07-11 08:05:23.709324 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.115:0/13323 conn(0x55f663ebe000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.6 seq 187 0x55f673971a00 osd_ping(ping e509 stamp 2018-07-11 08:05:24.061853) v4 > -6576> 2018-07-11 08:05:23.709352 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.6 10.8.128.115:0/13323 187 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:24.061853) v4 ==== 2004+0+0 (1823172468 0 0) 0x55f673971a00 con 0x55f663ebe000 > -6575> 2018-07-11 08:05:23.709349 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.6 10.8.128.115:0/13323 187 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:24.061853) v4 ==== 2004+0+0 (1823172468 0 0) 0x55f6631cc600 con 0x55f663ebf800 > -6574> 2018-07-11 08:05:23.709362 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:24.061853) v4 -- 0x55f650807400 con 0 > -6573> 2018-07-11 08:05:23.709390 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:24.061853) v4 -- 0x55f65e71dc00 con 0 > -6572> 2018-07-11 08:05:23.954992 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0e800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 203 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:05:25.199544) v4 > -6571> 2018-07-11 08:05:23.955015 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.5 10.8.128.113:0/15263 203 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:25.199544) v4 ==== 2004+0+0 (4294227903 0 0) 0x55f64e20fc00 con 0x55f663f0e800 > -6570> 2018-07-11 08:05:23.955025 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:25.199544) v4 -- 0x55f650807e00 con 0 > -6569> 2018-07-11 08:05:23.955014 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0d000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 203 0x55f653d4aa00 osd_ping(ping e509 stamp 2018-07-11 08:05:25.199544) v4 > -6568> 2018-07-11 08:05:23.955038 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.5 10.8.128.113:0/15263 203 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:25.199544) v4 ==== 2004+0+0 (4294227903 0 0) 0x55f653d4aa00 con 0x55f663f0d000 > -6567> 2018-07-11 08:05:23.955057 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:25.199544) v4 -- 0x55f65e71dc00 con 0 > -6566> 2018-07-11 08:05:23.976660 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15255 conn(0x55f663ec2800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.0 seq 193 0x55f673971200 osd_ping(ping e509 stamp 2018-07-11 08:05:25.221117) v4 > -6565> 2018-07-11 08:05:23.976670 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15255 conn(0x55f663ec1000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.0 seq 193 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:05:25.221117) v4 > -6564> 2018-07-11 08:05:23.976681 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.0 10.8.128.113:0/15255 193 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:25.221117) v4 ==== 2004+0+0 (1086272198 0 0) 0x55f673971200 con 0x55f663ec2800 > -6563> 2018-07-11 08:05:23.976694 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.0 10.8.128.113:0/15255 193 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:25.221117) v4 ==== 2004+0+0 (1086272198 0 0) 0x55f67d1afe00 con 0x55f663ec1000 > -6562> 2018-07-11 08:05:23.976699 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:25.221117) v4 -- 0x55f650807400 con 0 > -6561> 2018-07-11 08:05:23.976723 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:25.221117) v4 -- 0x55f650807e00 con 0 > -6560> 2018-07-11 08:05:24.029814 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12650 conn(0x55f663eda800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.3 seq 197 0x55f6631cc600 osd_ping(ping e509 stamp 2018-07-11 08:05:24.086400) v4 > -6559> 2018-07-11 08:05:24.029840 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.3 10.8.128.106:0/12650 197 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:24.086400) v4 ==== 2004+0+0 (799424019 0 0) 0x55f6631cc600 con 0x55f663eda800 > -6558> 2018-07-11 08:05:24.029841 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12650 conn(0x55f663ed9000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.3 seq 197 0x55f673971a00 osd_ping(ping e509 stamp 2018-07-11 08:05:24.086400) v4 > -6557> 2018-07-11 08:05:24.029854 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:24.086400) v4 -- 0x55f65e71dc00 con 0 > -6556> 2018-07-11 08:05:24.029862 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.3 10.8.128.106:0/12650 197 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:24.086400) v4 ==== 2004+0+0 (799424019 0 0) 0x55f673971a00 con 0x55f663ed9000 > -6555> 2018-07-11 08:05:24.029886 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:24.086400) v4 -- 0x55f650807400 con 0 > -6554> 2018-07-11 08:05:24.035477 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f667bb4480 con 0 > -6553> 2018-07-11 08:05:24.037169 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 219 0x55f65b1fc500 replica scrub(pg: 6.15,from:0'0,to:430'106548,epoch:509/504,start:6:a83761ce:::benchmark_data_magna108.ceph.redhat.com_2478_object109641:0,end:6:a8385cea:::benchmark_data_magna108.ceph.redhat.com_2478_object259588:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -6552> 2018-07-11 08:05:24.037200 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 219 ==== replica scrub(pg: 6.15,from:0'0,to:430'106548,epoch:509/504,start:6:a83761ce:::benchmark_data_magna108.ceph.redhat.com_2478_object109641:0,end:6:a8385cea:::benchmark_data_magna108.ceph.redhat.com_2478_object259588:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 248+0+0 (2858310831 0 0) 0x55f65b1fc500 con 0x55f66315e800 > -6551> 2018-07-11 08:05:24.084176 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.107:0/12617 conn(0x55f64e3c5000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.9 seq 196 0x55f653d4aa00 osd_ping(ping e509 stamp 2018-07-11 08:05:24.083637) v4 > -6550> 2018-07-11 08:05:24.084200 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.9 10.8.128.107:0/12617 196 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:24.083637) v4 ==== 2004+0+0 (2265584732 0 0) 0x55f653d4aa00 con 0x55f64e3c5000 > -6549> 2018-07-11 08:05:24.084256 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:24.083637) v4 -- 0x55f65e71dc00 con 0 > -6548> 2018-07-11 08:05:24.084265 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.107:0/12617 conn(0x55f64e3c3800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.9 seq 196 0x55f673971200 osd_ping(ping e509 stamp 2018-07-11 08:05:24.083637) v4 > -6547> 2018-07-11 08:05:24.084287 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.9 10.8.128.107:0/12617 196 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:24.083637) v4 ==== 2004+0+0 (2265584732 0 0) 0x55f673971200 con 0x55f64e3c3800 > -6546> 2018-07-11 08:05:24.084298 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:24.083637) v4 -- 0x55f650807400 con 0 > -6545> 2018-07-11 08:05:24.397897 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.108:0/11967 conn(0x55f663f6f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=1). rx osd.8 seq 190 0x55f673971a00 osd_ping(ping e509 stamp 2018-07-11 08:05:24.157785) v4 > -6544> 2018-07-11 08:05:24.397928 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.8 10.8.128.108:0/11967 190 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:24.157785) v4 ==== 2004+0+0 (446872889 0 0) 0x55f673971a00 con 0x55f663f6f000 > -6543> 2018-07-11 08:05:24.397929 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.108:0/11967 conn(0x55f663f70800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=1). rx osd.8 seq 190 0x55f6631cc600 osd_ping(ping e509 stamp 2018-07-11 08:05:24.157785) v4 > -6542> 2018-07-11 08:05:24.397942 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:24.157785) v4 -- 0x55f650807400 con 0 > -6541> 2018-07-11 08:05:24.397954 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.8 10.8.128.108:0/11967 190 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:24.157785) v4 ==== 2004+0+0 (446872889 0 0) 0x55f6631cc600 con 0x55f663f70800 > -6540> 2018-07-11 08:05:24.397977 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:24.157785) v4 -- 0x55f65e71dc00 con 0 > -6539> 2018-07-11 08:05:24.401215 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15287 conn(0x55f66dce2800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.4 seq 202 0x55f653d4aa00 osd_ping(ping e509 stamp 2018-07-11 08:05:25.645714) v4 > -6538> 2018-07-11 08:05:24.401239 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.4 10.8.128.113:0/15287 202 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:25.645714) v4 ==== 2004+0+0 (1315452545 0 0) 0x55f653d4aa00 con 0x55f66dce2800 > -6537> 2018-07-11 08:05:24.401256 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:25.645714) v4 -- 0x55f65e71dc00 con 0 > -6536> 2018-07-11 08:05:24.401292 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15287 conn(0x55f66dce1000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.4 seq 202 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:05:25.645714) v4 > -6535> 2018-07-11 08:05:24.401318 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.4 10.8.128.113:0/15287 202 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:25.645714) v4 ==== 2004+0+0 (1315452545 0 0) 0x55f64e20fc00 con 0x55f66dce1000 > -6534> 2018-07-11 08:05:24.401336 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:25.645714) v4 -- 0x55f650807e00 con 0 > -6533> 2018-07-11 08:05:24.455386 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0e800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 204 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:05:25.699953) v4 > -6532> 2018-07-11 08:05:24.455411 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.5 10.8.128.113:0/15263 204 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:25.699953) v4 ==== 2004+0+0 (3891766185 0 0) 0x55f67d1afe00 con 0x55f663f0e800 > -6531> 2018-07-11 08:05:24.455425 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:25.699953) v4 -- 0x55f650807600 con 0 > -6530> 2018-07-11 08:05:24.455439 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0d000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 204 0x55f6631cc600 osd_ping(ping e509 stamp 2018-07-11 08:05:25.699953) v4 > -6529> 2018-07-11 08:05:24.455460 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.5 10.8.128.113:0/15263 204 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:25.699953) v4 ==== 2004+0+0 (3891766185 0 0) 0x55f6631cc600 con 0x55f663f0d000 > -6528> 2018-07-11 08:05:24.455472 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:25.699953) v4 -- 0x55f662489400 con 0 > -6527> 2018-07-11 08:05:24.475162 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed6000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 197 0x55f653d4aa00 osd_ping(ping e509 stamp 2018-07-11 08:05:24.531899) v4 > -6526> 2018-07-11 08:05:24.475168 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed7800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 197 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:05:24.531899) v4 > -6525> 2018-07-11 08:05:24.475185 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.1 10.8.128.106:0/12677 197 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:24.531899) v4 ==== 2004+0+0 (3505410325 0 0) 0x55f64e20fc00 con 0x55f663ed7800 > -6524> 2018-07-11 08:05:24.475194 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:24.531899) v4 -- 0x55f66d001800 con 0 > -6523> 2018-07-11 08:05:24.475185 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.1 10.8.128.106:0/12677 197 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:24.531899) v4 ==== 2004+0+0 (3505410325 0 0) 0x55f653d4aa00 con 0x55f663ed6000 > -6522> 2018-07-11 08:05:24.475213 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:24.531899) v4 -- 0x55f664254800 con 0 > -6521> 2018-07-11 08:05:24.609743 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f664eb1440 con 0 > -6520> 2018-07-11 08:05:24.611293 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 220 0x55f65b1fca00 replica scrub(pg: 6.15,from:0'0,to:430'106205,epoch:509/504,start:6:a8385cea:::benchmark_data_magna108.ceph.redhat.com_2478_object259588:0,end:6:a839631b:::benchmark_data_magna108_35617_object451647:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -6519> 2018-07-11 08:05:24.611322 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 220 ==== replica scrub(pg: 6.15,from:0'0,to:430'106205,epoch:509/504,start:6:a8385cea:::benchmark_data_magna108.ceph.redhat.com_2478_object259588:0,end:6:a839631b:::benchmark_data_magna108_35617_object451647:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (1924280226 0 0) 0x55f65b1fca00 con 0x55f66315e800 > -6518> 2018-07-11 08:05:24.662070 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.114:6805/15885 conn(0x55f66315a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.7 seq 2543 0x55f6631cc600 MOSDScrubReserve(5.4 REQUEST e509) v1 > -6517> 2018-07-11 08:05:24.662094 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.7 10.8.128.114:6805/15885 2543 ==== MOSDScrubReserve(5.4 REQUEST e509) v1 ==== 43+0+0 (3003630469 0 0) 0x55f6631cc600 con 0x55f66315a000 > -6516> 2018-07-11 08:05:24.662134 7f491983f700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.114:6805/15885 -- MOSDScrubReserve(5.4 REJECT e509) v1 -- 0x55f660ea5800 con 0 > -6515> 2018-07-11 08:05:24.662487 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.114:6805/15885 conn(0x55f66315a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.7 seq 2544 0x55f660ea5800 MOSDScrubReserve(5.4 RELEASE e509) v1 > -6514> 2018-07-11 08:05:24.662506 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.7 10.8.128.114:6805/15885 2544 ==== MOSDScrubReserve(5.4 RELEASE e509) v1 ==== 43+0+0 (2082093447 0 0) 0x55f660ea5800 con 0x55f66315a000 > -6513> 2018-07-11 08:05:25.002701 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0a000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 194 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:05:25.638265) v4 > -6512> 2018-07-11 08:05:25.002701 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0b800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 194 0x55f673971200 osd_ping(ping e509 stamp 2018-07-11 08:05:25.638265) v4 > -6511> 2018-07-11 08:05:25.002740 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.11 10.8.128.114:0/15919 194 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:25.638265) v4 ==== 2004+0+0 (3272768449 0 0) 0x55f67d1afe00 con 0x55f663f0a000 > -6510> 2018-07-11 08:05:25.002742 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.11 10.8.128.114:0/15919 194 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:25.638265) v4 ==== 2004+0+0 (3272768449 0 0) 0x55f673971200 con 0x55f663f0b800 > -6509> 2018-07-11 08:05:25.002752 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:25.638265) v4 -- 0x55f66d001800 con 0 > -6508> 2018-07-11 08:05:25.002785 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:25.638265) v4 -- 0x55f650807400 con 0 > -6507> 2018-07-11 08:05:25.394391 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f664e6ad80 con 0 > -6506> 2018-07-11 08:05:25.396014 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 221 0x55f65b1fcc80 replica scrub(pg: 6.15,from:0'0,to:461'106887,epoch:509/504,start:6:a839631b:::benchmark_data_magna108_35617_object451647:0,end:6:a83a6c3a:::benchmark_data_magna108.ceph.redhat.com_2478_object380571:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -6505> 2018-07-11 08:05:25.396046 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 221 ==== replica scrub(pg: 6.15,from:0'0,to:461'106887,epoch:509/504,start:6:a839631b:::benchmark_data_magna108_35617_object451647:0,end:6:a83a6c3a:::benchmark_data_magna108.ceph.redhat.com_2478_object380571:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (358734767 0 0) 0x55f65b1fcc80 con 0x55f66315e800 > -6504> 2018-07-11 08:05:25.528089 7f491d046700 1 -- 10.8.128.107:6800/14173 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55f6631cf8c0 con 0 > -6503> 2018-07-11 08:05:25.528152 7f491d046700 1 -- 10.8.128.107:6800/14173 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55f6631ce840 con 0 > -6502> 2018-07-11 08:05:25.784829 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.107:0/12617 conn(0x55f64e3c3800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.9 seq 197 0x55f673971a00 osd_ping(ping e509 stamp 2018-07-11 08:05:25.784323) v4 > -6501> 2018-07-11 08:05:25.784829 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.107:0/12617 conn(0x55f64e3c5000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.9 seq 197 0x55f664254800 osd_ping(ping e509 stamp 2018-07-11 08:05:25.784323) v4 > -6500> 2018-07-11 08:05:25.784856 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.9 10.8.128.107:0/12617 197 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:25.784323) v4 ==== 2004+0+0 (2984218189 0 0) 0x55f673971a00 con 0x55f64e3c3800 > -6499> 2018-07-11 08:05:25.784870 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:25.784323) v4 -- 0x55f650807400 con 0 > -6498> 2018-07-11 08:05:25.784861 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.9 10.8.128.107:0/12617 197 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:25.784323) v4 ==== 2004+0+0 (2984218189 0 0) 0x55f664254800 con 0x55f64e3c5000 > -6497> 2018-07-11 08:05:25.784913 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:25.784323) v4 -- 0x55f664255e00 con 0 > -6496> 2018-07-11 08:05:26.103159 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0b800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 195 0x55f673971200 osd_ping(ping e509 stamp 2018-07-11 08:05:26.738704) v4 > -6495> 2018-07-11 08:05:26.103182 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.11 10.8.128.114:0/15919 195 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:26.738704) v4 ==== 2004+0+0 (4219522906 0 0) 0x55f673971200 con 0x55f663f0b800 > -6494> 2018-07-11 08:05:26.103197 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:26.738704) v4 -- 0x55f650807400 con 0 > -6493> 2018-07-11 08:05:26.103185 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0a000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 195 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:05:26.738704) v4 > -6492> 2018-07-11 08:05:26.103206 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.11 10.8.128.114:0/15919 195 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:26.738704) v4 ==== 2004+0+0 (4219522906 0 0) 0x55f64e20fc00 con 0x55f663f0a000 > -6491> 2018-07-11 08:05:26.103224 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:26.738704) v4 -- 0x55f66d001800 con 0 > -6490> 2018-07-11 08:05:26.175582 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed7800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 198 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:05:26.232290) v4 > -6489> 2018-07-11 08:05:26.175620 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.1 10.8.128.106:0/12677 198 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:26.232290) v4 ==== 2004+0+0 (996393292 0 0) 0x55f67d1afe00 con 0x55f663ed7800 > -6488> 2018-07-11 08:05:26.175632 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:26.232290) v4 -- 0x55f66d001800 con 0 > -6487> 2018-07-11 08:05:26.175650 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed6000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 198 0x55f664255c00 osd_ping(ping e509 stamp 2018-07-11 08:05:26.232290) v4 > -6486> 2018-07-11 08:05:26.175675 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.1 10.8.128.106:0/12677 198 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:26.232290) v4 ==== 2004+0+0 (996393292 0 0) 0x55f664255c00 con 0x55f663ed6000 > -6485> 2018-07-11 08:05:26.175692 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:26.232290) v4 -- 0x55f664255e00 con 0 > -6484> 2018-07-11 08:05:26.432550 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15885 conn(0x55f663217800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 201 0x55f673971a00 osd_ping(ping e509 stamp 2018-07-11 08:05:27.068118) v4 > -6483> 2018-07-11 08:05:26.432576 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.7 10.8.128.114:0/15885 201 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:27.068118) v4 ==== 2004+0+0 (1773259020 0 0) 0x55f673971a00 con 0x55f663217800 > -6482> 2018-07-11 08:05:26.432587 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:27.068118) v4 -- 0x55f673971c00 con 0 > -6481> 2018-07-11 08:05:26.432640 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15885 conn(0x55f663216000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 201 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:05:27.068118) v4 > -6480> 2018-07-11 08:05:26.432661 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.7 10.8.128.114:0/15885 201 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:27.068118) v4 ==== 2004+0+0 (1773259020 0 0) 0x55f64e20fc00 con 0x55f663216000 > -6479> 2018-07-11 08:05:26.432673 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:27.068118) v4 -- 0x55f66d001000 con 0 > -6478> 2018-07-11 08:05:26.638791 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f668073d40 con 0 > -6477> 2018-07-11 08:05:26.640342 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 222 0x55f65b1fcf00 replica scrub(pg: 6.15,from:0'0,to:430'106605,epoch:509/504,start:6:a83a6c3a:::benchmark_data_magna108.ceph.redhat.com_2478_object380571:0,end:6:a83bab3c:::benchmark_data_magna108_35617_object163440:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -6476> 2018-07-11 08:05:26.640372 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 222 ==== replica scrub(pg: 6.15,from:0'0,to:430'106605,epoch:509/504,start:6:a83a6c3a:::benchmark_data_magna108.ceph.redhat.com_2478_object380571:0,end:6:a83bab3c:::benchmark_data_magna108_35617_object163440:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (2173156278 0 0) 0x55f65b1fcf00 con 0x55f66315e800 > -6475> 2018-07-11 08:05:26.662331 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.114:6805/15885 conn(0x55f66315a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.7 seq 2545 0x55f664254800 MOSDScrubReserve(5.4 REQUEST e509) v1 > -6474> 2018-07-11 08:05:26.662350 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.7 10.8.128.114:6805/15885 2545 ==== MOSDScrubReserve(5.4 REQUEST e509) v1 ==== 43+0+0 (3003630469 0 0) 0x55f664254800 con 0x55f66315a000 > -6473> 2018-07-11 08:05:26.662424 7f491983f700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.114:6805/15885 -- MOSDScrubReserve(5.4 REJECT e509) v1 -- 0x55f660ea5800 con 0 > -6472> 2018-07-11 08:05:26.662731 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.114:6805/15885 conn(0x55f66315a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.7 seq 2546 0x55f660ea5800 MOSDScrubReserve(5.4 RELEASE e509) v1 > -6471> 2018-07-11 08:05:26.662758 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.7 10.8.128.114:6805/15885 2546 ==== MOSDScrubReserve(5.4 RELEASE e509) v1 ==== 43+0+0 (2082093447 0 0) 0x55f660ea5800 con 0x55f66315a000 > -6470> 2018-07-11 08:05:26.698351 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.108:0/11967 conn(0x55f663f6f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=1). rx osd.8 seq 191 0x55f673971200 osd_ping(ping e509 stamp 2018-07-11 08:05:26.458233) v4 > -6469> 2018-07-11 08:05:26.698374 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.8 10.8.128.108:0/11967 191 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:26.458233) v4 ==== 2004+0+0 (1900073412 0 0) 0x55f673971200 con 0x55f663f6f000 > -6468> 2018-07-11 08:05:26.698384 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:26.458233) v4 -- 0x55f673971c00 con 0 > -6467> 2018-07-11 08:05:26.698400 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.108:0/11967 conn(0x55f663f70800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=1). rx osd.8 seq 191 0x55f664255e00 osd_ping(ping e509 stamp 2018-07-11 08:05:26.458233) v4 > -6466> 2018-07-11 08:05:26.698421 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.8 10.8.128.108:0/11967 191 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:26.458233) v4 ==== 2004+0+0 (1900073412 0 0) 0x55f664255e00 con 0x55f663f70800 > -6465> 2018-07-11 08:05:26.698433 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:26.458233) v4 -- 0x55f663193a00 con 0 > -6464> 2018-07-11 08:05:26.752443 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12651 conn(0x55f663219000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25 cs=1 l=1). rx osd.2 seq 207 0x55f6631f5200 osd_ping(ping e509 stamp 2018-07-11 08:05:26.808899) v4 > -6463> 2018-07-11 08:05:26.752464 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.2 10.8.128.106:0/12651 207 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:26.808899) v4 ==== 2004+0+0 (3982738599 0 0) 0x55f6631f5200 con 0x55f663219000 > -6462> 2018-07-11 08:05:26.752475 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:26.808899) v4 -- 0x55f668a30200 con 0 > -6461> 2018-07-11 08:05:26.752511 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12651 conn(0x55f66321a800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25 cs=1 l=1). rx osd.2 seq 207 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:05:26.808899) v4 > -6460> 2018-07-11 08:05:26.752529 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.2 10.8.128.106:0/12651 207 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:26.808899) v4 ==== 2004+0+0 (3982738599 0 0) 0x55f67d1afe00 con 0x55f66321a800 > -6459> 2018-07-11 08:05:26.752541 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:26.808899) v4 -- 0x55f6716b3400 con 0 > -6458> 2018-07-11 08:05:27.276169 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed7800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 199 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:05:27.332787) v4 > -6457> 2018-07-11 08:05:27.276183 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed6000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 199 0x55f664255e00 osd_ping(ping e509 stamp 2018-07-11 08:05:27.332787) v4 > -6456> 2018-07-11 08:05:27.276198 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.1 10.8.128.106:0/12677 199 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:27.332787) v4 ==== 2004+0+0 (2214981027 0 0) 0x55f64e20fc00 con 0x55f663ed7800 > -6455> 2018-07-11 08:05:27.276202 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.1 10.8.128.106:0/12677 199 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:27.332787) v4 ==== 2004+0+0 (2214981027 0 0) 0x55f664255e00 con 0x55f663ed6000 > -6454> 2018-07-11 08:05:27.276209 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:27.332787) v4 -- 0x55f6716b2c00 con 0 > -6453> 2018-07-11 08:05:27.276237 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:27.332787) v4 -- 0x55f64e25d600 con 0 > -6452> 2018-07-11 08:05:27.485368 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.107:0/12617 conn(0x55f64e3c3800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.9 seq 198 0x55f673971a00 osd_ping(ping e509 stamp 2018-07-11 08:05:27.484990) v4 > -6451> 2018-07-11 08:05:27.485389 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.9 10.8.128.107:0/12617 198 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:27.484990) v4 ==== 2004+0+0 (3005285797 0 0) 0x55f673971a00 con 0x55f64e3c3800 > -6450> 2018-07-11 08:05:27.485402 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:27.484990) v4 -- 0x55f673971e00 con 0 > -6449> 2018-07-11 08:05:27.485417 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.107:0/12617 conn(0x55f64e3c5000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.9 seq 198 0x55f6631f5200 osd_ping(ping e509 stamp 2018-07-11 08:05:27.484990) v4 > -6448> 2018-07-11 08:05:27.485449 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.9 10.8.128.107:0/12617 198 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:27.484990) v4 ==== 2004+0+0 (3005285797 0 0) 0x55f6631f5200 con 0x55f64e3c5000 > -6447> 2018-07-11 08:05:27.485467 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:27.484990) v4 -- 0x55f663171c00 con 0 > -6446> 2018-07-11 08:05:27.809666 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.115:0/13323 conn(0x55f663ebf800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.6 seq 188 0x55f664255e00 osd_ping(ping e509 stamp 2018-07-11 08:05:28.162326) v4 > -6445> 2018-07-11 08:05:27.809701 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.6 10.8.128.115:0/13323 188 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:28.162326) v4 ==== 2004+0+0 (2432901000 0 0) 0x55f664255e00 con 0x55f663ebf800 > -6444> 2018-07-11 08:05:27.809716 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:28.162326) v4 -- 0x55f666c46000 con 0 > -6443> 2018-07-11 08:05:27.809750 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.115:0/13323 conn(0x55f663ebe000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.6 seq 188 0x55f673971200 osd_ping(ping e509 stamp 2018-07-11 08:05:28.162326) v4 > -6442> 2018-07-11 08:05:27.809772 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.6 10.8.128.115:0/13323 188 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:28.162326) v4 ==== 2004+0+0 (2432901000 0 0) 0x55f673971200 con 0x55f663ebe000 > -6441> 2018-07-11 08:05:27.809783 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:28.162326) v4 -- 0x55f673971e00 con 0 > -6440> 2018-07-11 08:05:27.853104 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12651 conn(0x55f663219000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25 cs=1 l=1). rx osd.2 seq 208 0x55f6631f5200 osd_ping(ping e509 stamp 2018-07-11 08:05:27.909545) v4 > -6439> 2018-07-11 08:05:27.853131 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.2 10.8.128.106:0/12651 208 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:27.909545) v4 ==== 2004+0+0 (3648350572 0 0) 0x55f6631f5200 con 0x55f663219000 > -6438> 2018-07-11 08:05:27.853140 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:27.909545) v4 -- 0x55f666c47c00 con 0 > -6437> 2018-07-11 08:05:27.853136 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12651 conn(0x55f66321a800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25 cs=1 l=1). rx osd.2 seq 208 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:05:27.909545) v4 > -6436> 2018-07-11 08:05:27.853157 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.2 10.8.128.106:0/12651 208 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:27.909545) v4 ==== 2004+0+0 (3648350572 0 0) 0x55f67d1afe00 con 0x55f66321a800 > -6435> 2018-07-11 08:05:27.853170 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:27.909545) v4 -- 0x55f6548fa200 con 0 > -6434> 2018-07-11 08:05:27.901781 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15287 conn(0x55f66dce1000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.4 seq 203 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:05:29.146294) v4 > -6433> 2018-07-11 08:05:27.901804 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.4 10.8.128.113:0/15287 203 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:29.146294) v4 ==== 2004+0+0 (2200129293 0 0) 0x55f64e20fc00 con 0x55f66dce1000 > -6432> 2018-07-11 08:05:27.901802 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15287 conn(0x55f66dce2800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.4 seq 203 0x55f664255e00 osd_ping(ping e509 stamp 2018-07-11 08:05:29.146294) v4 > -6431> 2018-07-11 08:05:27.901816 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:29.146294) v4 -- 0x55f64cc80a00 con 0 > -6430> 2018-07-11 08:05:27.901827 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.4 10.8.128.113:0/15287 203 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:29.146294) v4 ==== 2004+0+0 (2200129293 0 0) 0x55f664255e00 con 0x55f66dce2800 > -6429> 2018-07-11 08:05:27.901850 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:29.146294) v4 -- 0x55f64c7d3e00 con 0 > -6428> 2018-07-11 08:05:27.936509 7f491803c700 5 osd.10 509 heartbeat: osd_stat(789 GB used, 141 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,0,0,0,1]) > -6427> 2018-07-11 08:05:27.936524 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6802/15255 -- osd_ping(ping e509 stamp 2018-07-11 08:05:27.936522) v4 -- 0x55f66064be00 con 0 > -6426> 2018-07-11 08:05:27.936562 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6803/15255 -- osd_ping(ping e509 stamp 2018-07-11 08:05:27.936522) v4 -- 0x55f67d136000 con 0 > -6425> 2018-07-11 08:05:27.936579 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6806/12677 -- osd_ping(ping e509 stamp 2018-07-11 08:05:27.936522) v4 -- 0x55f67d136200 con 0 > -6424> 2018-07-11 08:05:27.936594 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6807/12677 -- osd_ping(ping e509 stamp 2018-07-11 08:05:27.936522) v4 -- 0x55f67d136400 con 0 > -6423> 2018-07-11 08:05:27.936606 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6808/12651 -- osd_ping(ping e509 stamp 2018-07-11 08:05:27.936522) v4 -- 0x55f67d136600 con 0 > -6422> 2018-07-11 08:05:27.936615 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6810/12651 -- osd_ping(ping e509 stamp 2018-07-11 08:05:27.936522) v4 -- 0x55f67d136800 con 0 > -6421> 2018-07-11 08:05:27.936627 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6809/12650 -- osd_ping(ping e509 stamp 2018-07-11 08:05:27.936522) v4 -- 0x55f67d136a00 con 0 > -6420> 2018-07-11 08:05:27.936638 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6811/12650 -- osd_ping(ping e509 stamp 2018-07-11 08:05:27.936522) v4 -- 0x55f67d136c00 con 0 > -6419> 2018-07-11 08:05:27.936657 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6810/15287 -- osd_ping(ping e509 stamp 2018-07-11 08:05:27.936522) v4 -- 0x55f67d136e00 con 0 > -6418> 2018-07-11 08:05:27.936668 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6811/15287 -- osd_ping(ping e509 stamp 2018-07-11 08:05:27.936522) v4 -- 0x55f67d137000 con 0 > -6417> 2018-07-11 08:05:27.936681 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6806/15263 -- osd_ping(ping e509 stamp 2018-07-11 08:05:27.936522) v4 -- 0x55f67d137200 con 0 > -6416> 2018-07-11 08:05:27.936693 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6807/15263 -- osd_ping(ping e509 stamp 2018-07-11 08:05:27.936522) v4 -- 0x55f67d137400 con 0 > -6415> 2018-07-11 08:05:27.936706 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.115:6802/13323 -- osd_ping(ping e509 stamp 2018-07-11 08:05:27.936522) v4 -- 0x55f67d137600 con 0 > -6414> 2018-07-11 08:05:27.936719 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.115:6803/13323 -- osd_ping(ping e509 stamp 2018-07-11 08:05:27.936522) v4 -- 0x55f67d137800 con 0 > -6413> 2018-07-11 08:05:27.936732 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6806/15885 -- osd_ping(ping e509 stamp 2018-07-11 08:05:27.936522) v4 -- 0x55f67d137a00 con 0 > -6412> 2018-07-11 08:05:27.936748 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6807/15885 -- osd_ping(ping e509 stamp 2018-07-11 08:05:27.936522) v4 -- 0x55f67d137c00 con 0 > -6411> 2018-07-11 08:05:27.936760 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.108:6802/11967 -- osd_ping(ping e509 stamp 2018-07-11 08:05:27.936522) v4 -- 0x55f67d137e00 con 0 > -6410> 2018-07-11 08:05:27.936772 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.108:6803/11967 -- osd_ping(ping e509 stamp 2018-07-11 08:05:27.936522) v4 -- 0x55f67d138000 con 0 > -6409> 2018-07-11 08:05:27.936784 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.107:6806/12617 -- osd_ping(ping e509 stamp 2018-07-11 08:05:27.936522) v4 -- 0x55f67d138200 con 0 > -6408> 2018-07-11 08:05:27.936797 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.107:6807/12617 -- osd_ping(ping e509 stamp 2018-07-11 08:05:27.936522) v4 -- 0x55f67d138400 con 0 > -6407> 2018-07-11 08:05:27.936809 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6802/15919 -- osd_ping(ping e509 stamp 2018-07-11 08:05:27.936522) v4 -- 0x55f67d138600 con 0 > -6406> 2018-07-11 08:05:27.936821 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6803/15919 -- osd_ping(ping e509 stamp 2018-07-11 08:05:27.936522) v4 -- 0x55f67d138800 con 0 > -6405> 2018-07-11 08:05:27.936952 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6802/15255 conn(0x55f656ad4000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 198 0x55f664255e00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:27.936522) v4 > -6404> 2018-07-11 08:05:27.936974 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.0 10.8.128.113:6802/15255 198 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:27.936522) v4 ==== 2004+0+0 (2080955550 0 0) 0x55f664255e00 con 0x55f656ad4000 > -6403> 2018-07-11 08:05:27.936958 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6803/15255 conn(0x55f656ad5800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 198 0x55f673971200 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:27.936522) v4 > -6402> 2018-07-11 08:05:27.936997 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.0 10.8.128.113:6803/15255 198 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:27.936522) v4 ==== 2004+0+0 (2080955550 0 0) 0x55f673971200 con 0x55f656ad5800 > -6401> 2018-07-11 08:05:27.937036 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.107:6806/12617 conn(0x55f656ade000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.9 seq 198 0x55f673971200 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:27.936522) v4 > -6400> 2018-07-11 08:05:27.937042 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6809/12650 conn(0x55f656ad7000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.3 seq 198 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:27.936522) v4 > -6399> 2018-07-11 08:05:27.937062 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.3 10.8.128.106:6809/12650 198 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:27.936522) v4 ==== 2004+0+0 (2080955550 0 0) 0x55f67d1afe00 con 0x55f656ad7000 > -6398> 2018-07-11 08:05:27.937086 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6806/12677 conn(0x55f64e3d8000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.1 seq 198 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:27.936522) v4 > -6397> 2018-07-11 08:05:27.937097 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.1 10.8.128.106:6806/12677 198 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:27.936522) v4 ==== 2004+0+0 (2080955550 0 0) 0x55f67d1afe00 con 0x55f64e3d8000 > -6396> 2018-07-11 08:05:27.937054 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.9 10.8.128.107:6806/12617 198 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:27.936522) v4 ==== 2004+0+0 (2080955550 0 0) 0x55f673971200 con 0x55f656ade000 > -6395> 2018-07-11 08:05:27.937112 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6810/15287 conn(0x55f64e401000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.4 seq 198 0x55f664255e00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:27.936522) v4 > -6394> 2018-07-11 08:05:27.937119 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.107:6807/12617 conn(0x55f656adf800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.9 seq 198 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:27.936522) v4 > -6393> 2018-07-11 08:05:27.937127 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.9 10.8.128.107:6807/12617 198 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:27.936522) v4 ==== 2004+0+0 (2080955550 0 0) 0x55f67d1afe00 con 0x55f656adf800 > -6392> 2018-07-11 08:05:27.937127 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.4 10.8.128.113:6810/15287 198 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:27.936522) v4 ==== 2004+0+0 (2080955550 0 0) 0x55f664255e00 con 0x55f64e401000 > -6391> 2018-07-11 08:05:27.937170 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6806/15263 conn(0x55f64e3ed000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.5 seq 198 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:27.936522) v4 > -6390> 2018-07-11 08:05:27.937183 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.5 10.8.128.113:6806/15263 198 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:27.936522) v4 ==== 2004+0+0 (2080955550 0 0) 0x55f67d1afe00 con 0x55f64e3ed000 > -6389> 2018-07-11 08:05:27.937178 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6810/12651 conn(0x55f64e3eb800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.2 seq 198 0x55f673971200 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:27.936522) v4 > -6388> 2018-07-11 08:05:27.937200 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.2 10.8.128.106:6810/12651 198 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:27.936522) v4 ==== 2004+0+0 (2080955550 0 0) 0x55f673971200 con 0x55f64e3eb800 > -6387> 2018-07-11 08:05:27.937196 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6807/12677 conn(0x55f64e3d9800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.1 seq 198 0x55f664255e00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:27.936522) v4 > -6386> 2018-07-11 08:05:27.937217 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.1 10.8.128.106:6807/12677 198 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:27.936522) v4 ==== 2004+0+0 (2080955550 0 0) 0x55f664255e00 con 0x55f64e3d9800 > -6385> 2018-07-11 08:05:27.937222 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.115:6803/13323 conn(0x55f656ace800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.6 seq 198 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:27.936522) v4 > -6384> 2018-07-11 08:05:27.937227 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6811/15287 conn(0x55f64e402800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.4 seq 198 0x55f673971200 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:27.936522) v4 > -6383> 2018-07-11 08:05:27.937235 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.6 10.8.128.115:6803/13323 198 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:27.936522) v4 ==== 2004+0+0 (2080955550 0 0) 0x55f67d1afe00 con 0x55f656ace800 > -6382> 2018-07-11 08:05:27.937236 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.4 10.8.128.113:6811/15287 198 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:27.936522) v4 ==== 2004+0+0 (2080955550 0 0) 0x55f673971200 con 0x55f64e402800 > -6381> 2018-07-11 08:05:27.937259 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6807/15263 conn(0x55f64e3ee800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.5 seq 198 0x55f664255e00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:27.936522) v4 > -6380> 2018-07-11 08:05:27.937286 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.5 10.8.128.113:6807/15263 198 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:27.936522) v4 ==== 2004+0+0 (2080955550 0 0) 0x55f664255e00 con 0x55f64e3ee800 > -6379> 2018-07-11 08:05:27.937299 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.115:6802/13323 conn(0x55f656acd000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.6 seq 198 0x55f673971200 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:27.936522) v4 > -6378> 2018-07-11 08:05:27.937313 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.6 10.8.128.115:6802/13323 198 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:27.936522) v4 ==== 2004+0+0 (2080955550 0 0) 0x55f673971200 con 0x55f656acd000 > -6377> 2018-07-11 08:05:27.937321 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6811/12650 conn(0x55f656ad8800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.3 seq 198 0x55f664255e00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:27.936522) v4 > -6376> 2018-07-11 08:05:27.937334 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.3 10.8.128.106:6811/12650 198 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:27.936522) v4 ==== 2004+0+0 (2080955550 0 0) 0x55f664255e00 con 0x55f656ad8800 > -6375> 2018-07-11 08:05:27.937350 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6806/15885 conn(0x55f64e3fe000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.7 seq 198 0x55f673971200 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:27.936522) v4 > -6374> 2018-07-11 08:05:27.937363 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.7 10.8.128.114:6806/15885 198 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:27.936522) v4 ==== 2004+0+0 (2080955550 0 0) 0x55f673971200 con 0x55f64e3fe000 > -6373> 2018-07-11 08:05:27.937357 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.108:6802/11967 conn(0x55f656aca000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.8 seq 198 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:27.936522) v4 > -6372> 2018-07-11 08:05:27.937362 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6808/12651 conn(0x55f64e3ea000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.2 seq 198 0x55f664255e00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:27.936522) v4 > -6371> 2018-07-11 08:05:27.937375 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.2 10.8.128.106:6808/12651 198 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:27.936522) v4 ==== 2004+0+0 (2080955550 0 0) 0x55f664255e00 con 0x55f64e3ea000 > -6370> 2018-07-11 08:05:27.937375 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.8 10.8.128.108:6802/11967 198 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:27.936522) v4 ==== 2004+0+0 (2080955550 0 0) 0x55f67d1afe00 con 0x55f656aca000 > -6369> 2018-07-11 08:05:27.937392 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6802/15919 conn(0x55f64e3db000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.11 seq 198 0x55f673971200 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:27.936522) v4 > -6368> 2018-07-11 08:05:27.937405 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.11 10.8.128.114:6802/15919 198 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:27.936522) v4 ==== 2004+0+0 (2080955550 0 0) 0x55f673971200 con 0x55f64e3db000 > -6367> 2018-07-11 08:05:27.937427 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6803/15919 conn(0x55f64e3dc800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.11 seq 198 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:27.936522) v4 > -6366> 2018-07-11 08:05:27.937447 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.11 10.8.128.114:6803/15919 198 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:27.936522) v4 ==== 2004+0+0 (2080955550 0 0) 0x55f67d1afe00 con 0x55f64e3dc800 > -6365> 2018-07-11 08:05:27.937486 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.108:6803/11967 conn(0x55f656acb800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.8 seq 198 0x55f664255e00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:27.936522) v4 > -6364> 2018-07-11 08:05:27.937501 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.8 10.8.128.108:6803/11967 198 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:27.936522) v4 ==== 2004+0+0 (2080955550 0 0) 0x55f664255e00 con 0x55f656acb800 > -6363> 2018-07-11 08:05:27.937496 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6807/15885 conn(0x55f64e3ff800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.7 seq 198 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:27.936522) v4 > -6362> 2018-07-11 08:05:27.937515 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.7 10.8.128.114:6807/15885 198 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:27.936522) v4 ==== 2004+0+0 (2080955550 0 0) 0x55f67d1afe00 con 0x55f64e3ff800 > -6361> 2018-07-11 08:05:28.010463 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 29] Generated table #37040: 2638240 keys, 74546953 bytes > -6360> 2018-07-11 08:05:28.010499 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296328010483, "cf_name": "default", "job": 29, "event": "table_file_creation", "file_number": 37040, "file_size": 74546953, "table_properties": {"data_size": 67112080, "index_size": 690702, "filter_size": 6743181, "raw_key_size": 76215827, "raw_average_key_size": 28, "raw_value_size": 31365747, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} > -6359> 2018-07-11 08:05:28.376588 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed7800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 200 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:05:28.433288) v4 > -6358> 2018-07-11 08:05:28.376619 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.1 10.8.128.106:0/12677 200 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:28.433288) v4 ==== 2004+0+0 (4234310991 0 0) 0x55f67d1afe00 con 0x55f663ed7800 > -6357> 2018-07-11 08:05:28.376642 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:28.433288) v4 -- 0x55f67d137800 con 0 > -6356> 2018-07-11 08:05:28.376737 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed6000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 200 0x55f664255e00 osd_ping(ping e509 stamp 2018-07-11 08:05:28.433288) v4 > -6355> 2018-07-11 08:05:28.376757 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.1 10.8.128.106:0/12677 200 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:28.433288) v4 ==== 2004+0+0 (4234310991 0 0) 0x55f664255e00 con 0x55f663ed6000 > -6354> 2018-07-11 08:05:28.376769 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:28.433288) v4 -- 0x55f67d136c00 con 0 > -6353> 2018-07-11 08:05:28.392009 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f661770000 con 0 > -6352> 2018-07-11 08:05:28.393812 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 223 0x55f65b1fd180 replica scrub(pg: 6.15,from:0'0,to:430'106768,epoch:509/504,start:6:a83bab3c:::benchmark_data_magna108_35617_object163440:0,end:6:a83c5b95:::benchmark_data_magna108_35617_object1003272:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -6351> 2018-07-11 08:05:28.393842 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 223 ==== replica scrub(pg: 6.15,from:0'0,to:430'106768,epoch:509/504,start:6:a83bab3c:::benchmark_data_magna108_35617_object163440:0,end:6:a83c5b95:::benchmark_data_magna108_35617_object1003272:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 219+0+0 (2391493233 0 0) 0x55f65b1fd180 con 0x55f66315e800 > -6350> 2018-07-11 08:05:28.402316 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15287 conn(0x55f66dce2800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.4 seq 204 0x55f6631f5200 osd_ping(ping e509 stamp 2018-07-11 08:05:29.646819) v4 > -6349> 2018-07-11 08:05:28.402342 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.4 10.8.128.113:0/15287 204 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:29.646819) v4 ==== 2004+0+0 (3787281816 0 0) 0x55f6631f5200 con 0x55f66dce2800 > -6348> 2018-07-11 08:05:28.402356 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:29.646819) v4 -- 0x55f67d136c00 con 0 > -6347> 2018-07-11 08:05:28.402451 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15287 conn(0x55f66dce1000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.4 seq 204 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:05:29.646819) v4 > -6346> 2018-07-11 08:05:28.402474 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.4 10.8.128.113:0/15287 204 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:29.646819) v4 ==== 2004+0+0 (3787281816 0 0) 0x55f64e20fc00 con 0x55f66dce1000 > -6345> 2018-07-11 08:05:28.402496 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:29.646819) v4 -- 0x55f67d137800 con 0 > -6344> 2018-07-11 08:05:28.662634 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.114:6805/15885 conn(0x55f66315a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.7 seq 2547 0x55f664255e00 MOSDScrubReserve(5.4 REQUEST e509) v1 > -6343> 2018-07-11 08:05:28.662653 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.7 10.8.128.114:6805/15885 2547 ==== MOSDScrubReserve(5.4 REQUEST e509) v1 ==== 43+0+0 (3003630469 0 0) 0x55f664255e00 con 0x55f66315a000 > -6342> 2018-07-11 08:05:28.662726 7f491983f700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.114:6805/15885 -- MOSDScrubReserve(5.4 REJECT e509) v1 -- 0x55f660ea5800 con 0 > -6341> 2018-07-11 08:05:28.663004 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.114:6805/15885 conn(0x55f66315a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.7 seq 2548 0x55f67d136600 MOSDScrubReserve(5.4 RELEASE e509) v1 > -6340> 2018-07-11 08:05:28.663018 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.7 10.8.128.114:6805/15885 2548 ==== MOSDScrubReserve(5.4 RELEASE e509) v1 ==== 43+0+0 (2082093447 0 0) 0x55f67d136600 con 0x55f66315a000 > -6339> 2018-07-11 08:05:28.730165 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12650 conn(0x55f663ed9000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.3 seq 198 0x55f673971200 osd_ping(ping e509 stamp 2018-07-11 08:05:28.786949) v4 > -6338> 2018-07-11 08:05:28.730190 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.3 10.8.128.106:0/12650 198 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:28.786949) v4 ==== 2004+0+0 (908472752 0 0) 0x55f673971200 con 0x55f663ed9000 > -6337> 2018-07-11 08:05:28.730182 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12650 conn(0x55f663eda800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.3 seq 198 0x55f6631f5200 osd_ping(ping e509 stamp 2018-07-11 08:05:28.786949) v4 > -6336> 2018-07-11 08:05:28.730199 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:28.786949) v4 -- 0x55f67d137000 con 0 > -6335> 2018-07-11 08:05:28.730199 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.3 10.8.128.106:0/12650 198 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:28.786949) v4 ==== 2004+0+0 (908472752 0 0) 0x55f6631f5200 con 0x55f663eda800 > -6334> 2018-07-11 08:05:28.730222 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:28.786949) v4 -- 0x55f64c7d3e00 con 0 > -6333> 2018-07-11 08:05:28.965871 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 435 0x55f673971a00 MOSDScrubReserve(11.3s5 REQUEST e509) v1 > -6332> 2018-07-11 08:05:28.965890 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 435 ==== MOSDScrubReserve(11.3s5 REQUEST e509) v1 ==== 43+0+0 (433425070 0 0) 0x55f673971a00 con 0x55f668a29000 > -6331> 2018-07-11 08:05:28.965963 7f491a040700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e509) v1 -- 0x55f66e944800 con 0 > -6330> 2018-07-11 08:05:28.966341 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 436 0x55f66e944800 MOSDScrubReserve(11.3s5 RELEASE e509) v1 > -6329> 2018-07-11 08:05:28.966356 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 436 ==== MOSDScrubReserve(11.3s5 RELEASE e509) v1 ==== 43+0+0 (3603460268 0 0) 0x55f66e944800 con 0x55f668a29000 > -6328> 2018-07-11 08:05:28.998801 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.108:0/11967 conn(0x55f663f6f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=1). rx osd.8 seq 192 0x55f673971200 osd_ping(ping e509 stamp 2018-07-11 08:05:28.758712) v4 > -6327> 2018-07-11 08:05:28.998822 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.8 10.8.128.108:0/11967 192 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:28.758712) v4 ==== 2004+0+0 (1747603257 0 0) 0x55f673971200 con 0x55f663f6f000 > -6326> 2018-07-11 08:05:28.998838 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:28.758712) v4 -- 0x55f673971e00 con 0 > -6325> 2018-07-11 08:05:28.998872 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.108:0/11967 conn(0x55f663f70800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=1). rx osd.8 seq 192 0x55f660ea5800 osd_ping(ping e509 stamp 2018-07-11 08:05:28.758712) v4 > -6324> 2018-07-11 08:05:28.998892 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.8 10.8.128.108:0/11967 192 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:28.758712) v4 ==== 2004+0+0 (1747603257 0 0) 0x55f660ea5800 con 0x55f663f70800 > -6323> 2018-07-11 08:05:28.998904 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:28.758712) v4 -- 0x55f64c7d3e00 con 0 > -6322> 2018-07-11 08:05:29.155740 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0d000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 205 0x55f6631f5200 osd_ping(ping e509 stamp 2018-07-11 08:05:30.400370) v4 > -6321> 2018-07-11 08:05:29.155761 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.5 10.8.128.113:0/15263 205 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:30.400370) v4 ==== 2004+0+0 (3271516523 0 0) 0x55f6631f5200 con 0x55f663f0d000 > -6320> 2018-07-11 08:05:29.155749 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0e800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 205 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:05:30.400370) v4 > -6319> 2018-07-11 08:05:29.155770 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:30.400370) v4 -- 0x55f667d08200 con 0 > -6318> 2018-07-11 08:05:29.155772 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.5 10.8.128.113:0/15263 205 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:30.400370) v4 ==== 2004+0+0 (3271516523 0 0) 0x55f67d1afe00 con 0x55f663f0e800 > -6317> 2018-07-11 08:05:29.155788 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:30.400370) v4 -- 0x55f67d137200 con 0 > -6316> 2018-07-11 08:05:29.276945 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15255 conn(0x55f663ec1000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.0 seq 194 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:05:30.521549) v4 > -6315> 2018-07-11 08:05:29.276992 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.0 10.8.128.113:0/15255 194 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:30.521549) v4 ==== 2004+0+0 (3743032643 0 0) 0x55f64e20fc00 con 0x55f663ec1000 > -6314> 2018-07-11 08:05:29.277012 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:30.521549) v4 -- 0x55f67d136a00 con 0 > -6313> 2018-07-11 08:05:29.277024 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15255 conn(0x55f663ec2800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.0 seq 194 0x55f67d136000 osd_ping(ping e509 stamp 2018-07-11 08:05:30.521549) v4 > -6312> 2018-07-11 08:05:29.277050 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.0 10.8.128.113:0/15255 194 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:30.521549) v4 ==== 2004+0+0 (3743032643 0 0) 0x55f67d136000 con 0x55f663ec2800 > -6311> 2018-07-11 08:05:29.277068 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:30.521549) v4 -- 0x55f66cb50000 con 0 > -6310> 2018-07-11 08:05:29.514332 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f65d514900 con 0 > -6309> 2018-07-11 08:05:29.515946 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 224 0x55f65b1fd400 replica scrub(pg: 6.15,from:0'0,to:430'106149,epoch:509/504,start:6:a83c5b95:::benchmark_data_magna108_35617_object1003272:0,end:6:a83d4bbf:::benchmark_data_magna108_35617_object519793:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -6308> 2018-07-11 08:05:29.515975 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 224 ==== replica scrub(pg: 6.15,from:0'0,to:430'106149,epoch:509/504,start:6:a83c5b95:::benchmark_data_magna108_35617_object1003272:0,end:6:a83d4bbf:::benchmark_data_magna108_35617_object519793:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 219+0+0 (572528745 0 0) 0x55f65b1fd400 con 0x55f66315e800 > -6307> 2018-07-11 08:05:30.099295 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.108:0/11967 conn(0x55f663f70800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=1). rx osd.8 seq 193 0x55f660ea5800 osd_ping(ping e509 stamp 2018-07-11 08:05:29.859149) v4 > -6306> 2018-07-11 08:05:30.099305 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.108:0/11967 conn(0x55f663f6f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=1). rx osd.8 seq 193 0x55f673971200 osd_ping(ping e509 stamp 2018-07-11 08:05:29.859149) v4 > -6305> 2018-07-11 08:05:30.099332 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.8 10.8.128.108:0/11967 193 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:29.859149) v4 ==== 2004+0+0 (459798593 0 0) 0x55f660ea5800 con 0x55f663f70800 > -6304> 2018-07-11 08:05:30.099336 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.8 10.8.128.108:0/11967 193 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:29.859149) v4 ==== 2004+0+0 (459798593 0 0) 0x55f673971200 con 0x55f663f6f000 > -6303> 2018-07-11 08:05:30.099345 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:29.859149) v4 -- 0x55f667d08600 con 0 > -6302> 2018-07-11 08:05:30.099377 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:29.859149) v4 -- 0x55f66cb50000 con 0 > -6301> 2018-07-11 08:05:30.110134 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.115:0/13323 conn(0x55f663ebf800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.6 seq 189 0x55f667d08200 osd_ping(ping e509 stamp 2018-07-11 08:05:30.462772) v4 > -6300> 2018-07-11 08:05:30.110162 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.6 10.8.128.115:0/13323 189 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:30.462772) v4 ==== 2004+0+0 (1757859322 0 0) 0x55f667d08200 con 0x55f663ebf800 > -6299> 2018-07-11 08:05:30.110180 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:30.462772) v4 -- 0x55f667d08600 con 0 > -6298> 2018-07-11 08:05:30.110233 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.115:0/13323 conn(0x55f663ebe000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.6 seq 189 0x55f67d136000 osd_ping(ping e509 stamp 2018-07-11 08:05:30.462772) v4 > -6297> 2018-07-11 08:05:30.110261 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.6 10.8.128.115:0/13323 189 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:30.462772) v4 ==== 2004+0+0 (1757859322 0 0) 0x55f67d136000 con 0x55f663ebe000 > -6296> 2018-07-11 08:05:30.110273 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:30.462772) v4 -- 0x55f66cb50200 con 0 > -6295> 2018-07-11 08:05:30.219215 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f65e90c240 con 0 > -6294> 2018-07-11 08:05:30.220770 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 225 0x55f65b1fd680 replica scrub(pg: 6.15,from:0'0,to:430'105828,epoch:509/504,start:6:a83d4bbf:::benchmark_data_magna108_35617_object519793:0,end:6:a83e6139:::benchmark_data_magna108.ceph.redhat.com_2478_object368001:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -6293> 2018-07-11 08:05:30.220801 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 225 ==== replica scrub(pg: 6.15,from:0'0,to:430'105828,epoch:509/504,start:6:a83d4bbf:::benchmark_data_magna108_35617_object519793:0,end:6:a83e6139:::benchmark_data_magna108.ceph.redhat.com_2478_object368001:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (1959467921 0 0) 0x55f65b1fd680 con 0x55f66315e800 > -6292> 2018-07-11 08:05:30.433608 7f491e048700 10 monclient: tick > -6291> 2018-07-11 08:05:30.433626 7f491e048700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2018-07-11 08:05:00.433622) > -6290> 2018-07-11 08:05:30.528446 7f491d046700 1 -- 10.8.128.107:6800/14173 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55f667d76dc0 con 0 > -6289> 2018-07-11 08:05:30.528513 7f491d046700 1 -- 10.8.128.107:6800/14173 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55f65e5d4840 con 0 > -6288> 2018-07-11 08:05:30.662985 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.114:6805/15885 conn(0x55f66315a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.7 seq 2549 0x55f667d08600 MOSDScrubReserve(5.4 REQUEST e509) v1 > -6287> 2018-07-11 08:05:30.663007 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.7 10.8.128.114:6805/15885 2549 ==== MOSDScrubReserve(5.4 REQUEST e509) v1 ==== 43+0+0 (3003630469 0 0) 0x55f667d08600 con 0x55f66315a000 > -6286> 2018-07-11 08:05:30.663087 7f491983f700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.114:6805/15885 -- MOSDScrubReserve(5.4 REJECT e509) v1 -- 0x55f67d136600 con 0 > -6285> 2018-07-11 08:05:30.663396 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.114:6805/15885 conn(0x55f66315a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.7 seq 2550 0x55f67d136600 MOSDScrubReserve(5.4 RELEASE e509) v1 > -6284> 2018-07-11 08:05:30.663412 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.7 10.8.128.114:6805/15885 2550 ==== MOSDScrubReserve(5.4 RELEASE e509) v1 ==== 43+0+0 (2082093447 0 0) 0x55f67d136600 con 0x55f66315a000 > -6283> 2018-07-11 08:05:30.985925 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.107:0/12617 conn(0x55f64e3c3800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.9 seq 199 0x55f673971200 osd_ping(ping e509 stamp 2018-07-11 08:05:30.985552) v4 > -6282> 2018-07-11 08:05:30.985952 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.9 10.8.128.107:0/12617 199 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:30.985552) v4 ==== 2004+0+0 (512464299 0 0) 0x55f673971200 con 0x55f64e3c3800 > -6281> 2018-07-11 08:05:30.985937 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.107:0/12617 conn(0x55f64e3c5000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.9 seq 199 0x55f660ea5800 osd_ping(ping e509 stamp 2018-07-11 08:05:30.985552) v4 > -6280> 2018-07-11 08:05:30.985979 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.9 10.8.128.107:0/12617 199 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:30.985552) v4 ==== 2004+0+0 (512464299 0 0) 0x55f660ea5800 con 0x55f64e3c5000 > -6279> 2018-07-11 08:05:30.985965 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:30.985552) v4 -- 0x55f66cb50200 con 0 > -6278> 2018-07-11 08:05:30.986020 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:30.985552) v4 -- 0x55f666182200 con 0 > -6277> 2018-07-11 08:05:31.033595 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f655c83440 con 0 > -6276> 2018-07-11 08:05:31.035203 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 226 0x55f65b1fd900 replica scrub(pg: 6.15,from:0'0,to:430'106389,epoch:509/504,start:6:a83e6139:::benchmark_data_magna108.ceph.redhat.com_2478_object368001:0,end:6:a83f18ce:::benchmark_data_magna108_35617_object572358:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -6275> 2018-07-11 08:05:31.035234 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 226 ==== replica scrub(pg: 6.15,from:0'0,to:430'106389,epoch:509/504,start:6:a83e6139:::benchmark_data_magna108.ceph.redhat.com_2478_object368001:0,end:6:a83f18ce:::benchmark_data_magna108_35617_object572358:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (3480092458 0 0) 0x55f65b1fd900 con 0x55f66315e800 > -6274> 2018-07-11 08:05:31.133046 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15885 conn(0x55f663217800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 202 0x55f67d136000 osd_ping(ping e509 stamp 2018-07-11 08:05:31.768585) v4 > -6273> 2018-07-11 08:05:31.133067 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.7 10.8.128.114:0/15885 202 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:31.768585) v4 ==== 2004+0+0 (346398570 0 0) 0x55f67d136000 con 0x55f663217800 > -6272> 2018-07-11 08:05:31.133080 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:31.768585) v4 -- 0x55f66cb50200 con 0 > -6271> 2018-07-11 08:05:31.133079 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15885 conn(0x55f663216000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 202 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:05:31.768585) v4 > -6270> 2018-07-11 08:05:31.133101 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.7 10.8.128.114:0/15885 202 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:31.768585) v4 ==== 2004+0+0 (346398570 0 0) 0x55f67d1afe00 con 0x55f663216000 > -6269> 2018-07-11 08:05:31.133112 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:31.768585) v4 -- 0x55f67d136a00 con 0 > -6268> 2018-07-11 08:05:31.199701 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.108:0/11967 conn(0x55f663f6f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=1). rx osd.8 seq 194 0x55f673971200 osd_ping(ping e509 stamp 2018-07-11 08:05:30.959625) v4 > -6267> 2018-07-11 08:05:31.199731 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.8 10.8.128.108:0/11967 194 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:30.959625) v4 ==== 2004+0+0 (1558014460 0 0) 0x55f673971200 con 0x55f663f6f000 > -6266> 2018-07-11 08:05:31.199730 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.108:0/11967 conn(0x55f663f70800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=1). rx osd.8 seq 194 0x55f666182000 osd_ping(ping e509 stamp 2018-07-11 08:05:30.959625) v4 > -6265> 2018-07-11 08:05:31.199748 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:30.959625) v4 -- 0x55f66cb50200 con 0 > -6264> 2018-07-11 08:05:31.199752 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.8 10.8.128.108:0/11967 194 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:30.959625) v4 ==== 2004+0+0 (1558014460 0 0) 0x55f666182000 con 0x55f663f70800 > -6263> 2018-07-11 08:05:31.199777 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:30.959625) v4 -- 0x55f666182400 con 0 > -6262> 2018-07-11 08:05:31.403551 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0a000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 196 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:05:32.039126) v4 > -6261> 2018-07-11 08:05:31.403565 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0b800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 196 0x55f67d136000 osd_ping(ping e509 stamp 2018-07-11 08:05:32.039126) v4 > -6260> 2018-07-11 08:05:31.403579 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.11 10.8.128.114:0/15919 196 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:32.039126) v4 ==== 2004+0+0 (2859134685 0 0) 0x55f64e20fc00 con 0x55f663f0a000 > -6259> 2018-07-11 08:05:31.403587 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.11 10.8.128.114:0/15919 196 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:32.039126) v4 ==== 2004+0+0 (2859134685 0 0) 0x55f67d136000 con 0x55f663f0b800 > -6258> 2018-07-11 08:05:31.403597 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:32.039126) v4 -- 0x55f67d136200 con 0 > -6257> 2018-07-11 08:05:31.403628 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:32.039126) v4 -- 0x55f66cb50400 con 0 > -6256> 2018-07-11 08:05:31.630606 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12650 conn(0x55f663eda800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.3 seq 199 0x55f666182200 osd_ping(ping e509 stamp 2018-07-11 08:05:31.687415) v4 > -6255> 2018-07-11 08:05:31.630629 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.3 10.8.128.106:0/12650 199 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:31.687415) v4 ==== 2004+0+0 (2062242331 0 0) 0x55f666182200 con 0x55f663eda800 > -6254> 2018-07-11 08:05:31.630640 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:31.687415) v4 -- 0x55f666182600 con 0 > -6253> 2018-07-11 08:05:31.630632 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12650 conn(0x55f663ed9000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.3 seq 199 0x55f673971200 osd_ping(ping e509 stamp 2018-07-11 08:05:31.687415) v4 > -6252> 2018-07-11 08:05:31.630660 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.3 10.8.128.106:0/12650 199 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:31.687415) v4 ==== 2004+0+0 (2062242331 0 0) 0x55f673971200 con 0x55f663ed9000 > -6251> 2018-07-11 08:05:31.630678 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:31.687415) v4 -- 0x55f66cb50600 con 0 > -6250> 2018-07-11 08:05:31.942945 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f6716c5680 con 0 > -6249> 2018-07-11 08:05:31.944675 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 227 0x55f65b1fdb80 replica scrub(pg: 6.15,from:0'0,to:430'105581,epoch:509/504,start:6:a83f18ce:::benchmark_data_magna108_35617_object572358:0,end:6:a83fd668:::benchmark_data_magna108_35617_object1031222:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -6248> 2018-07-11 08:05:31.944708 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 227 ==== replica scrub(pg: 6.15,from:0'0,to:430'105581,epoch:509/504,start:6:a83f18ce:::benchmark_data_magna108_35617_object572358:0,end:6:a83fd668:::benchmark_data_magna108_35617_object1031222:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 219+0+0 (3742692676 0 0) 0x55f65b1fdb80 con 0x55f66315e800 > -6247> 2018-07-11 08:05:32.056185 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0e800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 206 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:05:33.300840) v4 > -6246> 2018-07-11 08:05:32.056186 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0d000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 206 0x55f666182000 osd_ping(ping e509 stamp 2018-07-11 08:05:33.300840) v4 > -6245> 2018-07-11 08:05:32.056212 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.5 10.8.128.113:0/15263 206 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:33.300840) v4 ==== 2004+0+0 (3703058668 0 0) 0x55f67d1afe00 con 0x55f663f0e800 > -6244> 2018-07-11 08:05:32.056216 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.5 10.8.128.113:0/15263 206 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:33.300840) v4 ==== 2004+0+0 (3703058668 0 0) 0x55f666182000 con 0x55f663f0d000 > -6243> 2018-07-11 08:05:32.056225 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:33.300840) v4 -- 0x55f67d136200 con 0 > -6242> 2018-07-11 08:05:32.056269 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:33.300840) v4 -- 0x55f666182800 con 0 > -6241> 2018-07-11 08:05:32.233591 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15885 conn(0x55f663216000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 203 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:05:32.869187) v4 > -6240> 2018-07-11 08:05:32.233618 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.7 10.8.128.114:0/15885 203 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:32.869187) v4 ==== 2004+0+0 (740010746 0 0) 0x55f64e20fc00 con 0x55f663216000 > -6239> 2018-07-11 08:05:32.233636 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:32.869187) v4 -- 0x55f67d136200 con 0 > -6238> 2018-07-11 08:05:32.233655 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15885 conn(0x55f663217800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 203 0x55f67d136000 osd_ping(ping e509 stamp 2018-07-11 08:05:32.869187) v4 > -6237> 2018-07-11 08:05:32.233676 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.7 10.8.128.114:0/15885 203 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:32.869187) v4 ==== 2004+0+0 (740010746 0 0) 0x55f67d136000 con 0x55f663217800 > -6236> 2018-07-11 08:05:32.233688 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:32.869187) v4 -- 0x55f66cb50800 con 0 > -6235> 2018-07-11 08:05:32.300144 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.108:0/11967 conn(0x55f663f6f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=1). rx osd.8 seq 195 0x55f673971200 osd_ping(ping e509 stamp 2018-07-11 08:05:32.060075) v4 > -6234> 2018-07-11 08:05:32.300165 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.8 10.8.128.108:0/11967 195 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:32.060075) v4 ==== 2004+0+0 (2381014709 0 0) 0x55f673971200 con 0x55f663f6f000 > -6233> 2018-07-11 08:05:32.300174 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:32.060075) v4 -- 0x55f66cb50a00 con 0 > -6232> 2018-07-11 08:05:32.300203 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.108:0/11967 conn(0x55f663f70800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=1). rx osd.8 seq 195 0x55f666182600 osd_ping(ping e509 stamp 2018-07-11 08:05:32.060075) v4 > -6231> 2018-07-11 08:05:32.300227 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.8 10.8.128.108:0/11967 195 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:32.060075) v4 ==== 2004+0+0 (2381014709 0 0) 0x55f666182600 con 0x55f663f70800 > -6230> 2018-07-11 08:05:32.300239 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:32.060075) v4 -- 0x55f666182a00 con 0 > -6229> 2018-07-11 08:05:32.503992 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0b800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 197 0x55f67d136000 osd_ping(ping e509 stamp 2018-07-11 08:05:33.139558) v4 > -6228> 2018-07-11 08:05:32.504017 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.11 10.8.128.114:0/15919 197 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:33.139558) v4 ==== 2004+0+0 (1882515303 0 0) 0x55f67d136000 con 0x55f663f0b800 > -6227> 2018-07-11 08:05:32.504010 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0a000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 197 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:05:33.139558) v4 > -6226> 2018-07-11 08:05:32.504029 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:33.139558) v4 -- 0x55f66cb50c00 con 0 > -6225> 2018-07-11 08:05:32.504038 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.11 10.8.128.114:0/15919 197 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:33.139558) v4 ==== 2004+0+0 (1882515303 0 0) 0x55f67d1afe00 con 0x55f663f0a000 > -6224> 2018-07-11 08:05:32.504053 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:33.139558) v4 -- 0x55f64cc80a00 con 0 > -6223> 2018-07-11 08:05:32.553527 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12651 conn(0x55f663219000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25 cs=1 l=1). rx osd.2 seq 209 0x55f666182000 osd_ping(ping e509 stamp 2018-07-11 08:05:32.610213) v4 > -6222> 2018-07-11 08:05:32.553559 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.2 10.8.128.106:0/12651 209 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:32.610213) v4 ==== 2004+0+0 (4039795676 0 0) 0x55f666182000 con 0x55f663219000 > -6221> 2018-07-11 08:05:32.553574 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:32.610213) v4 -- 0x55f666182c00 con 0 > -6220> 2018-07-11 08:05:32.553566 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12651 conn(0x55f66321a800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25 cs=1 l=1). rx osd.2 seq 209 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:05:32.610213) v4 > -6219> 2018-07-11 08:05:32.553585 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.2 10.8.128.106:0/12651 209 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:32.610213) v4 ==== 2004+0+0 (4039795676 0 0) 0x55f64e20fc00 con 0x55f66321a800 > -6218> 2018-07-11 08:05:32.553600 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:32.610213) v4 -- 0x55f66dcebc00 con 0 > -6217> 2018-07-11 08:05:32.586417 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f65e90d680 con 0 > -6216> 2018-07-11 08:05:32.588086 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 228 0x55f6507c6000 replica scrub(pg: 6.15,from:0'0,to:461'106888,epoch:509/504,start:6:a83fd668:::benchmark_data_magna108_35617_object1031222:0,end:6:a840e4e2:::benchmark_data_magna108.ceph.redhat.com_2478_object7688:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -6215> 2018-07-11 08:05:32.588116 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 228 ==== replica scrub(pg: 6.15,from:0'0,to:461'106888,epoch:509/504,start:6:a83fd668:::benchmark_data_magna108_35617_object1031222:0,end:6:a840e4e2:::benchmark_data_magna108.ceph.redhat.com_2478_object7688:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 232+0+0 (764180220 0 0) 0x55f6507c6000 con 0x55f66315e800 > -6214> 2018-07-11 08:05:32.686415 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.107:0/12617 conn(0x55f64e3c3800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.9 seq 200 0x55f673971200 osd_ping(ping e509 stamp 2018-07-11 08:05:32.686032) v4 > -6213> 2018-07-11 08:05:32.686434 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.9 10.8.128.107:0/12617 200 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:32.686032) v4 ==== 2004+0+0 (2353293510 0 0) 0x55f673971200 con 0x55f64e3c3800 > -6212> 2018-07-11 08:05:32.686408 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.107:0/12617 conn(0x55f64e3c5000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.9 seq 200 0x55f666182600 osd_ping(ping e509 stamp 2018-07-11 08:05:32.686032) v4 > -6211> 2018-07-11 08:05:32.686445 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:32.686032) v4 -- 0x55f66cb50c00 con 0 > -6210> 2018-07-11 08:05:32.686446 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.9 10.8.128.107:0/12617 200 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:32.686032) v4 ==== 2004+0+0 (2353293510 0 0) 0x55f666182600 con 0x55f64e3c5000 > -6209> 2018-07-11 08:05:32.686493 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:32.686032) v4 -- 0x55f666182c00 con 0 > -6208> 2018-07-11 08:05:33.076967 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed7800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 201 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:05:33.133805) v4 > -6207> 2018-07-11 08:05:33.076992 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.1 10.8.128.106:0/12677 201 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:33.133805) v4 ==== 2004+0+0 (460629186 0 0) 0x55f67d1afe00 con 0x55f663ed7800 > -6206> 2018-07-11 08:05:33.077003 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:33.133805) v4 -- 0x55f66dcebc00 con 0 > -6205> 2018-07-11 08:05:33.077014 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed6000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 201 0x55f666182000 osd_ping(ping e509 stamp 2018-07-11 08:05:33.133805) v4 > -6204> 2018-07-11 08:05:33.077037 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.1 10.8.128.106:0/12677 201 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:33.133805) v4 ==== 2004+0+0 (460629186 0 0) 0x55f666182000 con 0x55f663ed6000 > -6203> 2018-07-11 08:05:33.077048 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:33.133805) v4 -- 0x55f666182c00 con 0 > -6202> 2018-07-11 08:05:33.334029 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15885 conn(0x55f663217800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 204 0x55f67d136000 osd_ping(ping e509 stamp 2018-07-11 08:05:33.969613) v4 > -6201> 2018-07-11 08:05:33.334044 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15885 conn(0x55f663216000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 204 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:05:33.969613) v4 > -6200> 2018-07-11 08:05:33.334053 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.7 10.8.128.114:0/15885 204 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:33.969613) v4 ==== 2004+0+0 (285687239 0 0) 0x55f67d136000 con 0x55f663217800 > -6199> 2018-07-11 08:05:33.334068 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.7 10.8.128.114:0/15885 204 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:33.969613) v4 ==== 2004+0+0 (285687239 0 0) 0x55f64e20fc00 con 0x55f663216000 > -6198> 2018-07-11 08:05:33.334075 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:33.969613) v4 -- 0x55f66cb50c00 con 0 > -6197> 2018-07-11 08:05:33.334101 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:33.969613) v4 -- 0x55f66dcebc00 con 0 > -6196> 2018-07-11 08:05:33.604292 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0a000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 198 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:05:34.239942) v4 > -6195> 2018-07-11 08:05:33.604313 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.11 10.8.128.114:0/15919 198 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:34.239942) v4 ==== 2004+0+0 (3473865106 0 0) 0x55f67d1afe00 con 0x55f663f0a000 > -6194> 2018-07-11 08:05:33.604311 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0b800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 198 0x55f673971200 osd_ping(ping e509 stamp 2018-07-11 08:05:34.239942) v4 > -6193> 2018-07-11 08:05:33.604324 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:34.239942) v4 -- 0x55f65d025e00 con 0 > -6192> 2018-07-11 08:05:33.604328 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.11 10.8.128.114:0/15919 198 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:34.239942) v4 ==== 2004+0+0 (3473865106 0 0) 0x55f673971200 con 0x55f663f0b800 > -6191> 2018-07-11 08:05:33.604351 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:34.239942) v4 -- 0x55f66cb50e00 con 0 > -6190> 2018-07-11 08:05:33.663234 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.114:6805/15885 conn(0x55f66315a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.7 seq 2551 0x55f666182600 MOSDScrubReserve(5.4 REQUEST e509) v1 > -6189> 2018-07-11 08:05:33.663263 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.7 10.8.128.114:6805/15885 2551 ==== MOSDScrubReserve(5.4 REQUEST e509) v1 ==== 43+0+0 (3003630469 0 0) 0x55f666182600 con 0x55f66315a000 > -6188> 2018-07-11 08:05:33.663346 7f491983f700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.114:6805/15885 -- MOSDScrubReserve(5.4 REJECT e509) v1 -- 0x55f67d136600 con 0 > -6187> 2018-07-11 08:05:33.663580 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.114:6805/15885 conn(0x55f66315a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.7 seq 2552 0x55f666182e00 MOSDScrubReserve(5.4 RELEASE e509) v1 > -6186> 2018-07-11 08:05:33.663594 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.7 10.8.128.114:6805/15885 2552 ==== MOSDScrubReserve(5.4 RELEASE e509) v1 ==== 43+0+0 (2082093447 0 0) 0x55f666182e00 con 0x55f66315a000 > -6185> 2018-07-11 08:05:33.836964 7f491803c700 5 osd.10 509 heartbeat: osd_stat(789 GB used, 141 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,0,0,0,1]) > -6184> 2018-07-11 08:05:33.836980 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6802/15255 -- osd_ping(ping e509 stamp 2018-07-11 08:05:33.836978) v4 -- 0x55f67d138a00 con 0 > -6183> 2018-07-11 08:05:33.837006 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6803/15255 -- osd_ping(ping e509 stamp 2018-07-11 08:05:33.836978) v4 -- 0x55f67d138c00 con 0 > -6182> 2018-07-11 08:05:33.837024 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6806/12677 -- osd_ping(ping e509 stamp 2018-07-11 08:05:33.836978) v4 -- 0x55f67d138e00 con 0 > -6181> 2018-07-11 08:05:33.837040 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6807/12677 -- osd_ping(ping e509 stamp 2018-07-11 08:05:33.836978) v4 -- 0x55f67d139000 con 0 > -6180> 2018-07-11 08:05:33.837051 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6808/12651 -- osd_ping(ping e509 stamp 2018-07-11 08:05:33.836978) v4 -- 0x55f67d139200 con 0 > -6179> 2018-07-11 08:05:33.837061 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6810/12651 -- osd_ping(ping e509 stamp 2018-07-11 08:05:33.836978) v4 -- 0x55f67d139400 con 0 > -6178> 2018-07-11 08:05:33.837075 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6809/12650 -- osd_ping(ping e509 stamp 2018-07-11 08:05:33.836978) v4 -- 0x55f67d139600 con 0 > -6177> 2018-07-11 08:05:33.837088 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6811/12650 -- osd_ping(ping e509 stamp 2018-07-11 08:05:33.836978) v4 -- 0x55f67d139800 con 0 > -6176> 2018-07-11 08:05:33.837100 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6810/15287 -- osd_ping(ping e509 stamp 2018-07-11 08:05:33.836978) v4 -- 0x55f67d139a00 con 0 > -6175> 2018-07-11 08:05:33.837117 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6811/15287 -- osd_ping(ping e509 stamp 2018-07-11 08:05:33.836978) v4 -- 0x55f67d139c00 con 0 > -6174> 2018-07-11 08:05:33.837131 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6806/15263 -- osd_ping(ping e509 stamp 2018-07-11 08:05:33.836978) v4 -- 0x55f67d139e00 con 0 > -6173> 2018-07-11 08:05:33.837147 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6807/15263 -- osd_ping(ping e509 stamp 2018-07-11 08:05:33.836978) v4 -- 0x55f6507f8000 con 0 > -6172> 2018-07-11 08:05:33.837163 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.115:6802/13323 -- osd_ping(ping e509 stamp 2018-07-11 08:05:33.836978) v4 -- 0x55f6507f8200 con 0 > -6171> 2018-07-11 08:05:33.837176 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.115:6803/13323 -- osd_ping(ping e509 stamp 2018-07-11 08:05:33.836978) v4 -- 0x55f6507f8400 con 0 > -6170> 2018-07-11 08:05:33.837190 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6806/15885 -- osd_ping(ping e509 stamp 2018-07-11 08:05:33.836978) v4 -- 0x55f6507f8600 con 0 > -6169> 2018-07-11 08:05:33.837202 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6807/15885 -- osd_ping(ping e509 stamp 2018-07-11 08:05:33.836978) v4 -- 0x55f6507f8800 con 0 > -6168> 2018-07-11 08:05:33.837215 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.108:6802/11967 -- osd_ping(ping e509 stamp 2018-07-11 08:05:33.836978) v4 -- 0x55f6507f8a00 con 0 > -6167> 2018-07-11 08:05:33.837226 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.108:6803/11967 -- osd_ping(ping e509 stamp 2018-07-11 08:05:33.836978) v4 -- 0x55f6507f8c00 con 0 > -6166> 2018-07-11 08:05:33.837239 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.107:6806/12617 -- osd_ping(ping e509 stamp 2018-07-11 08:05:33.836978) v4 -- 0x55f6507f8e00 con 0 > -6165> 2018-07-11 08:05:33.837269 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.107:6807/12617 -- osd_ping(ping e509 stamp 2018-07-11 08:05:33.836978) v4 -- 0x55f6507f9000 con 0 > -6164> 2018-07-11 08:05:33.837282 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6802/15919 -- osd_ping(ping e509 stamp 2018-07-11 08:05:33.836978) v4 -- 0x55f6507f9200 con 0 > -6163> 2018-07-11 08:05:33.837294 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6803/15919 -- osd_ping(ping e509 stamp 2018-07-11 08:05:33.836978) v4 -- 0x55f6507f9400 con 0 > -6162> 2018-07-11 08:05:33.837440 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6802/15255 conn(0x55f656ad4000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 199 0x55f67d136600 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:33.836978) v4 > -6161> 2018-07-11 08:05:33.837461 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.0 10.8.128.113:6802/15255 199 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:33.836978) v4 ==== 2004+0+0 (3646869738 0 0) 0x55f67d136600 con 0x55f656ad4000 > -6160> 2018-07-11 08:05:33.837472 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6803/15255 conn(0x55f656ad5800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 199 0x55f673971200 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:33.836978) v4 > -6159> 2018-07-11 08:05:33.837500 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.0 10.8.128.113:6803/15255 199 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:33.836978) v4 ==== 2004+0+0 (3646869738 0 0) 0x55f673971200 con 0x55f656ad5800 > -6158> 2018-07-11 08:05:33.837535 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6807/12677 conn(0x55f64e3d9800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.1 seq 199 0x55f67d136600 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:33.836978) v4 > -6157> 2018-07-11 08:05:33.837559 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.1 10.8.128.106:6807/12677 199 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:33.836978) v4 ==== 2004+0+0 (3646869738 0 0) 0x55f67d136600 con 0x55f64e3d9800 > -6156> 2018-07-11 08:05:33.837528 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.107:6807/12617 conn(0x55f656adf800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.9 seq 199 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:33.836978) v4 > -6155> 2018-07-11 08:05:33.837582 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.9 10.8.128.107:6807/12617 199 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:33.836978) v4 ==== 2004+0+0 (3646869738 0 0) 0x55f64e20fc00 con 0x55f656adf800 > -6154> 2018-07-11 08:05:33.837540 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.107:6806/12617 conn(0x55f656ade000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.9 seq 199 0x55f673971200 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:33.836978) v4 > -6153> 2018-07-11 08:05:33.837602 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.9 10.8.128.107:6806/12617 199 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:33.836978) v4 ==== 2004+0+0 (3646869738 0 0) 0x55f673971200 con 0x55f656ade000 > -6152> 2018-07-11 08:05:33.837617 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6806/12677 conn(0x55f64e3d8000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.1 seq 199 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:33.836978) v4 > -6151> 2018-07-11 08:05:33.837636 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.1 10.8.128.106:6806/12677 199 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:33.836978) v4 ==== 2004+0+0 (3646869738 0 0) 0x55f64e20fc00 con 0x55f64e3d8000 > -6150> 2018-07-11 08:05:33.837635 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6808/12651 conn(0x55f64e3ea000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.2 seq 199 0x55f67d136600 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:33.836978) v4 > -6149> 2018-07-11 08:05:33.837657 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.2 10.8.128.106:6808/12651 199 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:33.836978) v4 ==== 2004+0+0 (3646869738 0 0) 0x55f67d136600 con 0x55f64e3ea000 > -6148> 2018-07-11 08:05:33.837677 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6810/12651 conn(0x55f64e3eb800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.2 seq 199 0x55f673971200 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:33.836978) v4 > -6147> 2018-07-11 08:05:33.837679 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6806/15263 conn(0x55f64e3ed000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.5 seq 199 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:33.836978) v4 > -6146> 2018-07-11 08:05:33.837708 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6811/12650 conn(0x55f656ad8800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.3 seq 199 0x55f67d136600 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:33.836978) v4 > -6145> 2018-07-11 08:05:33.837713 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.5 10.8.128.113:6806/15263 199 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:33.836978) v4 ==== 2004+0+0 (3646869738 0 0) 0x55f64e20fc00 con 0x55f64e3ed000 > -6144> 2018-07-11 08:05:33.837720 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.2 10.8.128.106:6810/12651 199 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:33.836978) v4 ==== 2004+0+0 (3646869738 0 0) 0x55f673971200 con 0x55f64e3eb800 > -6143> 2018-07-11 08:05:33.837729 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.3 10.8.128.106:6811/12650 199 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:33.836978) v4 ==== 2004+0+0 (3646869738 0 0) 0x55f67d136600 con 0x55f656ad8800 > -6142> 2018-07-11 08:05:33.837771 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6809/12650 conn(0x55f656ad7000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.3 seq 199 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:33.836978) v4 > -6141> 2018-07-11 08:05:33.837792 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.3 10.8.128.106:6809/12650 199 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:33.836978) v4 ==== 2004+0+0 (3646869738 0 0) 0x55f64e20fc00 con 0x55f656ad7000 > -6140> 2018-07-11 08:05:33.837789 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6811/15287 conn(0x55f64e402800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.4 seq 199 0x55f673971200 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:33.836978) v4 > -6139> 2018-07-11 08:05:33.837785 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6810/15287 conn(0x55f64e401000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.4 seq 199 0x55f67d136600 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:33.836978) v4 > -6138> 2018-07-11 08:05:33.837810 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.4 10.8.128.113:6810/15287 199 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:33.836978) v4 ==== 2004+0+0 (3646869738 0 0) 0x55f67d136600 con 0x55f64e401000 > -6137> 2018-07-11 08:05:33.837812 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.4 10.8.128.113:6811/15287 199 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:33.836978) v4 ==== 2004+0+0 (3646869738 0 0) 0x55f673971200 con 0x55f64e402800 > -6136> 2018-07-11 08:05:33.837839 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.115:6803/13323 conn(0x55f656ace800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.6 seq 199 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:33.836978) v4 > -6135> 2018-07-11 08:05:33.837863 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6807/15263 conn(0x55f64e3ee800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.5 seq 199 0x55f67d136600 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:33.836978) v4 > -6134> 2018-07-11 08:05:33.837859 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.6 10.8.128.115:6803/13323 199 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:33.836978) v4 ==== 2004+0+0 (3646869738 0 0) 0x55f64e20fc00 con 0x55f656ace800 > -6133> 2018-07-11 08:05:33.837875 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.115:6802/13323 conn(0x55f656acd000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.6 seq 199 0x55f673971200 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:33.836978) v4 > -6132> 2018-07-11 08:05:33.837881 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.5 10.8.128.113:6807/15263 199 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:33.836978) v4 ==== 2004+0+0 (3646869738 0 0) 0x55f67d136600 con 0x55f64e3ee800 > -6131> 2018-07-11 08:05:33.837895 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.6 10.8.128.115:6802/13323 199 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:33.836978) v4 ==== 2004+0+0 (3646869738 0 0) 0x55f673971200 con 0x55f656acd000 > -6130> 2018-07-11 08:05:33.837923 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.108:6802/11967 conn(0x55f656aca000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.8 seq 199 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:33.836978) v4 > -6129> 2018-07-11 08:05:33.837923 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.108:6803/11967 conn(0x55f656acb800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.8 seq 199 0x55f67d136600 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:33.836978) v4 > -6128> 2018-07-11 08:05:33.837943 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.8 10.8.128.108:6802/11967 199 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:33.836978) v4 ==== 2004+0+0 (3646869738 0 0) 0x55f64e20fc00 con 0x55f656aca000 > -6127> 2018-07-11 08:05:33.837943 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6802/15919 conn(0x55f64e3db000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.11 seq 199 0x55f673971200 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:33.836978) v4 > -6126> 2018-07-11 08:05:33.837954 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.8 10.8.128.108:6803/11967 199 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:33.836978) v4 ==== 2004+0+0 (3646869738 0 0) 0x55f67d136600 con 0x55f656acb800 > -6125> 2018-07-11 08:05:33.837963 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.11 10.8.128.114:6802/15919 199 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:33.836978) v4 ==== 2004+0+0 (3646869738 0 0) 0x55f673971200 con 0x55f64e3db000 > -6124> 2018-07-11 08:05:33.837986 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6803/15919 conn(0x55f64e3dc800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.11 seq 199 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:33.836978) v4 > -6123> 2018-07-11 08:05:33.838001 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.11 10.8.128.114:6803/15919 199 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:33.836978) v4 ==== 2004+0+0 (3646869738 0 0) 0x55f64e20fc00 con 0x55f64e3dc800 > -6122> 2018-07-11 08:05:33.838011 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6806/15885 conn(0x55f64e3fe000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.7 seq 199 0x55f673971200 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:33.836978) v4 > -6121> 2018-07-11 08:05:33.838031 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.7 10.8.128.114:6806/15885 199 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:33.836978) v4 ==== 2004+0+0 (3646869738 0 0) 0x55f673971200 con 0x55f64e3fe000 > -6120> 2018-07-11 08:05:33.838033 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6807/15885 conn(0x55f64e3ff800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.7 seq 199 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:33.836978) v4 > -6119> 2018-07-11 08:05:33.838047 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.7 10.8.128.114:6807/15885 199 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:33.836978) v4 ==== 2004+0+0 (3646869738 0 0) 0x55f64e20fc00 con 0x55f64e3ff800 > -6118> 2018-07-11 08:05:33.839222 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 29] Generated table #37041: 2638241 keys, 74546309 bytes > -6117> 2018-07-11 08:05:33.839258 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296333839239, "cf_name": "default", "job": 29, "event": "table_file_creation", "file_number": 37041, "file_size": 74546309, "table_properties": {"data_size": 67112174, "index_size": 689962, "filter_size": 6743183, "raw_key_size": 76215851, "raw_average_key_size": 28, "raw_value_size": 31365754, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} > -6116> 2018-07-11 08:05:34.302622 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15287 conn(0x55f66dce1000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.4 seq 205 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:05:35.547314) v4 > -6115> 2018-07-11 08:05:34.302622 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15287 conn(0x55f66dce2800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.4 seq 205 0x55f67d136600 osd_ping(ping e509 stamp 2018-07-11 08:05:35.547314) v4 > -6114> 2018-07-11 08:05:34.302659 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.4 10.8.128.113:0/15287 205 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:35.547314) v4 ==== 2004+0+0 (776352386 0 0) 0x55f64e20fc00 con 0x55f66dce1000 > -6113> 2018-07-11 08:05:34.302663 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.4 10.8.128.113:0/15287 205 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:35.547314) v4 ==== 2004+0+0 (776352386 0 0) 0x55f67d136600 con 0x55f66dce2800 > -6112> 2018-07-11 08:05:34.302676 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:35.547314) v4 -- 0x55f67d139e00 con 0 > -6111> 2018-07-11 08:05:34.302704 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:35.547314) v4 -- 0x55f67d139000 con 0 > -6110> 2018-07-11 08:05:34.512830 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f654917200 con 0 > -6109> 2018-07-11 08:05:34.515114 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 229 0x55f6507c6280 replica scrub(pg: 6.15,from:0'0,to:430'105975,epoch:509/504,start:6:a840e4e2:::benchmark_data_magna108.ceph.redhat.com_2478_object7688:0,end:6:a841f449:::benchmark_data_magna108_35617_object361831:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -6108> 2018-07-11 08:05:34.515146 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 229 ==== replica scrub(pg: 6.15,from:0'0,to:430'105975,epoch:509/504,start:6:a840e4e2:::benchmark_data_magna108.ceph.redhat.com_2478_object7688:0,end:6:a841f449:::benchmark_data_magna108_35617_object361831:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 231+0+0 (2745269098 0 0) 0x55f6507c6280 con 0x55f66315e800 > -6107> 2018-07-11 08:05:34.810524 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.115:0/13323 conn(0x55f663ebf800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.6 seq 190 0x55f666182000 osd_ping(ping e509 stamp 2018-07-11 08:05:35.163218) v4 > -6106> 2018-07-11 08:05:34.810538 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.115:0/13323 conn(0x55f663ebe000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.6 seq 190 0x55f673971200 osd_ping(ping e509 stamp 2018-07-11 08:05:35.163218) v4 > -6105> 2018-07-11 08:05:34.810551 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.6 10.8.128.115:0/13323 190 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:35.163218) v4 ==== 2004+0+0 (3376138858 0 0) 0x55f666182000 con 0x55f663ebf800 > -6104> 2018-07-11 08:05:34.810558 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.6 10.8.128.115:0/13323 190 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:35.163218) v4 ==== 2004+0+0 (3376138858 0 0) 0x55f673971200 con 0x55f663ebe000 > -6103> 2018-07-11 08:05:34.810569 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:35.163218) v4 -- 0x55f67d139000 con 0 > -6102> 2018-07-11 08:05:34.810601 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:35.163218) v4 -- 0x55f67d139400 con 0 > -6101> 2018-07-11 08:05:35.177535 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15255 conn(0x55f663ec1000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.0 seq 195 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:05:36.422138) v4 > -6100> 2018-07-11 08:05:35.177559 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.0 10.8.128.113:0/15255 195 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:36.422138) v4 ==== 2004+0+0 (1182582597 0 0) 0x55f67d1afe00 con 0x55f663ec1000 > -6099> 2018-07-11 08:05:35.177556 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15255 conn(0x55f663ec2800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.0 seq 195 0x55f67d136000 osd_ping(ping e509 stamp 2018-07-11 08:05:36.422138) v4 > -6098> 2018-07-11 08:05:35.177576 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.0 10.8.128.113:0/15255 195 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:36.422138) v4 ==== 2004+0+0 (1182582597 0 0) 0x55f67d136000 con 0x55f663ec2800 > -6097> 2018-07-11 08:05:35.177578 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:36.422138) v4 -- 0x55f67d139e00 con 0 > -6096> 2018-07-11 08:05:35.177611 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:36.422138) v4 -- 0x55f67d139400 con 0 > -6095> 2018-07-11 08:05:35.248974 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f664300240 con 0 > -6094> 2018-07-11 08:05:35.250682 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 230 0x55f6507c6500 replica scrub(pg: 6.15,from:0'0,to:430'106293,epoch:509/504,start:6:a841f449:::benchmark_data_magna108_35617_object361831:0,end:6:a843012c:::benchmark_data_magna108_35617_object83224:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -6093> 2018-07-11 08:05:35.250712 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 230 ==== replica scrub(pg: 6.15,from:0'0,to:430'106293,epoch:509/504,start:6:a841f449:::benchmark_data_magna108_35617_object361831:0,end:6:a843012c:::benchmark_data_magna108_35617_object83224:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 217+0+0 (2608815912 0 0) 0x55f6507c6500 con 0x55f66315e800 > -6092> 2018-07-11 08:05:35.528843 7f491d046700 1 -- 10.8.128.107:6800/14173 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55f663151080 con 0 > -6091> 2018-07-11 08:05:35.528920 7f491d046700 1 -- 10.8.128.107:6800/14173 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55f64cc762c0 con 0 > -6090> 2018-07-11 08:05:35.556534 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0e800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 207 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:05:36.801261) v4 > -6089> 2018-07-11 08:05:35.556534 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0d000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 207 0x55f67d136600 osd_ping(ping e509 stamp 2018-07-11 08:05:36.801261) v4 > -6088> 2018-07-11 08:05:35.556560 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.5 10.8.128.113:0/15263 207 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:36.801261) v4 ==== 2004+0+0 (3779279247 0 0) 0x55f64e20fc00 con 0x55f663f0e800 > -6087> 2018-07-11 08:05:35.556564 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.5 10.8.128.113:0/15263 207 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:36.801261) v4 ==== 2004+0+0 (3779279247 0 0) 0x55f67d136600 con 0x55f663f0d000 > -6086> 2018-07-11 08:05:35.556572 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:36.801261) v4 -- 0x55f67d139e00 con 0 > -6085> 2018-07-11 08:05:35.556598 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:36.801261) v4 -- 0x55f67d139000 con 0 > -6084> 2018-07-11 08:05:35.663436 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.114:6805/15885 conn(0x55f66315a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.7 seq 2553 0x55f666182000 MOSDScrubReserve(5.4 REQUEST e509) v1 > -6083> 2018-07-11 08:05:35.663455 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.7 10.8.128.114:6805/15885 2553 ==== MOSDScrubReserve(5.4 REQUEST e509) v1 ==== 43+0+0 (3003630469 0 0) 0x55f666182000 con 0x55f66315a000 > -6082> 2018-07-11 08:05:35.663534 7f491983f700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.114:6805/15885 -- MOSDScrubReserve(5.4 REJECT e509) v1 -- 0x55f666182e00 con 0 > -6081> 2018-07-11 08:05:35.663869 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.114:6805/15885 conn(0x55f66315a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.7 seq 2554 0x55f666182e00 MOSDScrubReserve(5.4 RELEASE e509) v1 > -6080> 2018-07-11 08:05:35.663883 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.7 10.8.128.114:6805/15885 2554 ==== MOSDScrubReserve(5.4 RELEASE e509) v1 ==== 43+0+0 (2082093447 0 0) 0x55f666182e00 con 0x55f66315a000 > -6079> 2018-07-11 08:05:36.054119 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12651 conn(0x55f663219000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25 cs=1 l=1). rx osd.2 seq 210 0x55f67d136600 osd_ping(ping e509 stamp 2018-07-11 08:05:36.110856) v4 > -6078> 2018-07-11 08:05:36.054145 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.2 10.8.128.106:0/12651 210 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:36.110856) v4 ==== 2004+0+0 (281074631 0 0) 0x55f67d136600 con 0x55f663219000 > -6077> 2018-07-11 08:05:36.054159 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:36.110856) v4 -- 0x55f666183400 con 0 > -6076> 2018-07-11 08:05:36.054158 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12651 conn(0x55f66321a800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25 cs=1 l=1). rx osd.2 seq 210 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:05:36.110856) v4 > -6075> 2018-07-11 08:05:36.054178 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.2 10.8.128.106:0/12651 210 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:36.110856) v4 ==== 2004+0+0 (281074631 0 0) 0x55f67d1afe00 con 0x55f66321a800 > -6074> 2018-07-11 08:05:36.054190 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:36.110856) v4 -- 0x55f67d139600 con 0 > -6073> 2018-07-11 08:05:36.113101 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f6642c0480 con 0 > -6072> 2018-07-11 08:05:36.114697 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 231 0x55f6507c6780 replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:509/504,start:6:a843012c:::benchmark_data_magna108_35617_object83224:0,end:6:a8446b75:::benchmark_data_magna108_35617_object386777:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -6071> 2018-07-11 08:05:36.114720 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 231 ==== replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:509/504,start:6:a843012c:::benchmark_data_magna108_35617_object83224:0,end:6:a8446b75:::benchmark_data_magna108_35617_object386777:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 217+0+0 (1250524399 0 0) 0x55f6507c6780 con 0x55f66315e800 > -6070> 2018-07-11 08:05:36.137443 7f491803c700 5 osd.10 509 heartbeat: osd_stat(789 GB used, 141 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,1]) > -6069> 2018-07-11 08:05:36.137457 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6802/15255 -- osd_ping(ping e509 stamp 2018-07-11 08:05:36.137455) v4 -- 0x55f6507f9600 con 0 > -6068> 2018-07-11 08:05:36.137494 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6803/15255 -- osd_ping(ping e509 stamp 2018-07-11 08:05:36.137455) v4 -- 0x55f6507f9800 con 0 > -6067> 2018-07-11 08:05:36.137511 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6806/12677 -- osd_ping(ping e509 stamp 2018-07-11 08:05:36.137455) v4 -- 0x55f6507f9a00 con 0 > -6066> 2018-07-11 08:05:36.137525 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6807/12677 -- osd_ping(ping e509 stamp 2018-07-11 08:05:36.137455) v4 -- 0x55f6507f9c00 con 0 > -6065> 2018-07-11 08:05:36.137537 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6808/12651 -- osd_ping(ping e509 stamp 2018-07-11 08:05:36.137455) v4 -- 0x55f6507f9e00 con 0 > -6064> 2018-07-11 08:05:36.137549 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6810/12651 -- osd_ping(ping e509 stamp 2018-07-11 08:05:36.137455) v4 -- 0x55f65c79a000 con 0 > -6063> 2018-07-11 08:05:36.137561 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6809/12650 -- osd_ping(ping e509 stamp 2018-07-11 08:05:36.137455) v4 -- 0x55f65c79a200 con 0 > -6062> 2018-07-11 08:05:36.137573 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6811/12650 -- osd_ping(ping e509 stamp 2018-07-11 08:05:36.137455) v4 -- 0x55f65c79a400 con 0 > -6061> 2018-07-11 08:05:36.137592 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6810/15287 -- osd_ping(ping e509 stamp 2018-07-11 08:05:36.137455) v4 -- 0x55f65c79a600 con 0 > -6060> 2018-07-11 08:05:36.137604 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6811/15287 -- osd_ping(ping e509 stamp 2018-07-11 08:05:36.137455) v4 -- 0x55f65c79a800 con 0 > -6059> 2018-07-11 08:05:36.137616 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6806/15263 -- osd_ping(ping e509 stamp 2018-07-11 08:05:36.137455) v4 -- 0x55f65c79aa00 con 0 > -6058> 2018-07-11 08:05:36.137629 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6807/15263 -- osd_ping(ping e509 stamp 2018-07-11 08:05:36.137455) v4 -- 0x55f65c79ac00 con 0 > -6057> 2018-07-11 08:05:36.137640 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.115:6802/13323 -- osd_ping(ping e509 stamp 2018-07-11 08:05:36.137455) v4 -- 0x55f65c79ae00 con 0 > -6056> 2018-07-11 08:05:36.137653 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.115:6803/13323 -- osd_ping(ping e509 stamp 2018-07-11 08:05:36.137455) v4 -- 0x55f65c79b000 con 0 > -6055> 2018-07-11 08:05:36.137666 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6806/15885 -- osd_ping(ping e509 stamp 2018-07-11 08:05:36.137455) v4 -- 0x55f65c79b200 con 0 > -6054> 2018-07-11 08:05:36.137684 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6807/15885 -- osd_ping(ping e509 stamp 2018-07-11 08:05:36.137455) v4 -- 0x55f65c79b400 con 0 > -6053> 2018-07-11 08:05:36.137697 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.108:6802/11967 -- osd_ping(ping e509 stamp 2018-07-11 08:05:36.137455) v4 -- 0x55f65c79b600 con 0 > -6052> 2018-07-11 08:05:36.137709 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.108:6803/11967 -- osd_ping(ping e509 stamp 2018-07-11 08:05:36.137455) v4 -- 0x55f65c79b800 con 0 > -6051> 2018-07-11 08:05:36.137722 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.107:6806/12617 -- osd_ping(ping e509 stamp 2018-07-11 08:05:36.137455) v4 -- 0x55f65c79ba00 con 0 > -6050> 2018-07-11 08:05:36.137734 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.107:6807/12617 -- osd_ping(ping e509 stamp 2018-07-11 08:05:36.137455) v4 -- 0x55f65c79bc00 con 0 > -6049> 2018-07-11 08:05:36.137747 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6802/15919 -- osd_ping(ping e509 stamp 2018-07-11 08:05:36.137455) v4 -- 0x55f65c79be00 con 0 > -6048> 2018-07-11 08:05:36.137760 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6803/15919 -- osd_ping(ping e509 stamp 2018-07-11 08:05:36.137455) v4 -- 0x55f65c79c000 con 0 > -6047> 2018-07-11 08:05:36.137834 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6802/15255 conn(0x55f656ad4000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 200 0x55f67d138a00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:36.137455) v4 > -6046> 2018-07-11 08:05:36.137858 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.0 10.8.128.113:6802/15255 200 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:36.137455) v4 ==== 2004+0+0 (4030812490 0 0) 0x55f67d138a00 con 0x55f656ad4000 > -6045> 2018-07-11 08:05:36.137910 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6803/15255 conn(0x55f656ad5800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 200 0x55f67d136000 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:36.137455) v4 > -6044> 2018-07-11 08:05:36.137913 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6807/12677 conn(0x55f64e3d9800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.1 seq 200 0x55f67d138a00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:36.137455) v4 > -6043> 2018-07-11 08:05:36.137932 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.0 10.8.128.113:6803/15255 200 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:36.137455) v4 ==== 2004+0+0 (4030812490 0 0) 0x55f67d136000 con 0x55f656ad5800 > -6042> 2018-07-11 08:05:36.137933 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.1 10.8.128.106:6807/12677 200 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:36.137455) v4 ==== 2004+0+0 (4030812490 0 0) 0x55f67d138a00 con 0x55f64e3d9800 > -6041> 2018-07-11 08:05:36.137964 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.107:6806/12617 conn(0x55f656ade000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.9 seq 200 0x55f67d136000 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:36.137455) v4 > -6040> 2018-07-11 08:05:36.138001 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.9 10.8.128.107:6806/12617 200 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:36.137455) v4 ==== 2004+0+0 (4030812490 0 0) 0x55f67d136000 con 0x55f656ade000 > -6039> 2018-07-11 08:05:36.138024 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6808/12651 conn(0x55f64e3ea000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.2 seq 200 0x55f67d138a00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:36.137455) v4 > -6038> 2018-07-11 08:05:36.138039 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6810/12651 conn(0x55f64e3eb800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.2 seq 200 0x55f67d136000 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:36.137455) v4 > -6037> 2018-07-11 08:05:36.138051 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.2 10.8.128.106:6810/12651 200 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:36.137455) v4 ==== 2004+0+0 (4030812490 0 0) 0x55f67d136000 con 0x55f64e3eb800 > -6036> 2018-07-11 08:05:36.138046 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.2 10.8.128.106:6808/12651 200 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:36.137455) v4 ==== 2004+0+0 (4030812490 0 0) 0x55f67d138a00 con 0x55f64e3ea000 > -6035> 2018-07-11 08:05:36.138043 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.107:6807/12617 conn(0x55f656adf800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.9 seq 200 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:36.137455) v4 > -6034> 2018-07-11 08:05:36.138082 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.9 10.8.128.107:6807/12617 200 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:36.137455) v4 ==== 2004+0+0 (4030812490 0 0) 0x55f64e20fc00 con 0x55f656adf800 > -6033> 2018-07-11 08:05:36.138083 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6811/15287 conn(0x55f64e402800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.4 seq 200 0x55f67d136000 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:36.137455) v4 > -6032> 2018-07-11 08:05:36.138093 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.4 10.8.128.113:6811/15287 200 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:36.137455) v4 ==== 2004+0+0 (4030812490 0 0) 0x55f67d136000 con 0x55f64e402800 > -6031> 2018-07-11 08:05:36.138108 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6806/12677 conn(0x55f64e3d8000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.1 seq 200 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:36.137455) v4 > -6030> 2018-07-11 08:05:36.138119 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.1 10.8.128.106:6806/12677 200 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:36.137455) v4 ==== 2004+0+0 (4030812490 0 0) 0x55f64e20fc00 con 0x55f64e3d8000 > -6029> 2018-07-11 08:05:36.138126 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.115:6802/13323 conn(0x55f656acd000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.6 seq 200 0x55f67d136000 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:36.137455) v4 > -6028> 2018-07-11 08:05:36.138125 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6810/15287 conn(0x55f64e401000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.4 seq 200 0x55f67d138a00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:36.137455) v4 > -6027> 2018-07-11 08:05:36.138141 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.6 10.8.128.115:6802/13323 200 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:36.137455) v4 ==== 2004+0+0 (4030812490 0 0) 0x55f67d136000 con 0x55f656acd000 > -6026> 2018-07-11 08:05:36.138139 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6806/15263 conn(0x55f64e3ed000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.5 seq 200 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:36.137455) v4 > -6025> 2018-07-11 08:05:36.138148 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.4 10.8.128.113:6810/15287 200 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:36.137455) v4 ==== 2004+0+0 (4030812490 0 0) 0x55f67d138a00 con 0x55f64e401000 > -6024> 2018-07-11 08:05:36.138149 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.5 10.8.128.113:6806/15263 200 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:36.137455) v4 ==== 2004+0+0 (4030812490 0 0) 0x55f64e20fc00 con 0x55f64e3ed000 > -6023> 2018-07-11 08:05:36.138190 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6809/12650 conn(0x55f656ad7000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.3 seq 200 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:36.137455) v4 > -6022> 2018-07-11 08:05:36.138203 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.3 10.8.128.106:6809/12650 200 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:36.137455) v4 ==== 2004+0+0 (4030812490 0 0) 0x55f64e20fc00 con 0x55f656ad7000 > -6021> 2018-07-11 08:05:36.138204 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6811/12650 conn(0x55f656ad8800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.3 seq 200 0x55f67d138a00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:36.137455) v4 > -6020> 2018-07-11 08:05:36.138234 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6806/15885 conn(0x55f64e3fe000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.7 seq 200 0x55f67d136000 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:36.137455) v4 > -6019> 2018-07-11 08:05:36.138235 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.115:6803/13323 conn(0x55f656ace800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.6 seq 200 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:36.137455) v4 > -6018> 2018-07-11 08:05:36.138283 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.7 10.8.128.114:6806/15885 200 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:36.137455) v4 ==== 2004+0+0 (4030812490 0 0) 0x55f67d136000 con 0x55f64e3fe000 > -6017> 2018-07-11 08:05:36.138281 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.3 10.8.128.106:6811/12650 200 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:36.137455) v4 ==== 2004+0+0 (4030812490 0 0) 0x55f67d138a00 con 0x55f656ad8800 > -6016> 2018-07-11 08:05:36.138289 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.6 10.8.128.115:6803/13323 200 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:36.137455) v4 ==== 2004+0+0 (4030812490 0 0) 0x55f64e20fc00 con 0x55f656ace800 > -6015> 2018-07-11 08:05:36.138330 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.108:6802/11967 conn(0x55f656aca000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.8 seq 200 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:36.137455) v4 > -6014> 2018-07-11 08:05:36.138331 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6802/15919 conn(0x55f64e3db000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.11 seq 200 0x55f67d136000 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:36.137455) v4 > -6013> 2018-07-11 08:05:36.138346 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.8 10.8.128.108:6802/11967 200 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:36.137455) v4 ==== 2004+0+0 (4030812490 0 0) 0x55f64e20fc00 con 0x55f656aca000 > -6012> 2018-07-11 08:05:36.138341 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6807/15263 conn(0x55f64e3ee800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.5 seq 200 0x55f67d138a00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:36.137455) v4 > -6011> 2018-07-11 08:05:36.138351 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.11 10.8.128.114:6802/15919 200 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:36.137455) v4 ==== 2004+0+0 (4030812490 0 0) 0x55f67d136000 con 0x55f64e3db000 > -6010> 2018-07-11 08:05:36.138363 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.5 10.8.128.113:6807/15263 200 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:36.137455) v4 ==== 2004+0+0 (4030812490 0 0) 0x55f67d138a00 con 0x55f64e3ee800 > -6009> 2018-07-11 08:05:36.138374 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6807/15885 conn(0x55f64e3ff800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.7 seq 200 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:36.137455) v4 > -6008> 2018-07-11 08:05:36.138386 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.7 10.8.128.114:6807/15885 200 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:36.137455) v4 ==== 2004+0+0 (4030812490 0 0) 0x55f64e20fc00 con 0x55f64e3ff800 > -6007> 2018-07-11 08:05:36.138414 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6803/15919 conn(0x55f64e3dc800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.11 seq 200 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:36.137455) v4 > -6006> 2018-07-11 08:05:36.138411 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.108:6803/11967 conn(0x55f656acb800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.8 seq 200 0x55f67d138a00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:36.137455) v4 > -6005> 2018-07-11 08:05:36.138428 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.11 10.8.128.114:6803/15919 200 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:36.137455) v4 ==== 2004+0+0 (4030812490 0 0) 0x55f64e20fc00 con 0x55f64e3dc800 > -6004> 2018-07-11 08:05:36.138430 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.8 10.8.128.108:6803/11967 200 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:36.137455) v4 ==== 2004+0+0 (4030812490 0 0) 0x55f67d138a00 con 0x55f656acb800 > -6003> 2018-07-11 08:05:36.278073 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15255 conn(0x55f663ec2800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.0 seq 196 0x55f67d136000 osd_ping(ping e509 stamp 2018-07-11 08:05:37.522612) v4 > -6002> 2018-07-11 08:05:36.278095 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.0 10.8.128.113:0/15255 196 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:37.522612) v4 ==== 2004+0+0 (435504377 0 0) 0x55f67d136000 con 0x55f663ec2800 > -6001> 2018-07-11 08:05:36.278092 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15255 conn(0x55f663ec1000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.0 seq 196 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:05:37.522612) v4 > -6000> 2018-07-11 08:05:36.278112 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:37.522612) v4 -- 0x55f67d139400 con 0 > -5999> 2018-07-11 08:05:36.278120 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.0 10.8.128.113:0/15255 196 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:37.522612) v4 ==== 2004+0+0 (435504377 0 0) 0x55f64e20fc00 con 0x55f663ec1000 > -5998> 2018-07-11 08:05:36.278143 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:37.522612) v4 -- 0x55f6507f9a00 con 0 > -5997> 2018-07-11 08:05:36.786906 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.107:0/12617 conn(0x55f64e3c3800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.9 seq 201 0x55f673971200 osd_ping(ping e509 stamp 2018-07-11 08:05:36.786550) v4 > -5996> 2018-07-11 08:05:36.786915 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.107:0/12617 conn(0x55f64e3c5000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.9 seq 201 0x55f67d138a00 osd_ping(ping e509 stamp 2018-07-11 08:05:36.786550) v4 > -5995> 2018-07-11 08:05:36.786932 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.9 10.8.128.107:0/12617 201 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:36.786550) v4 ==== 2004+0+0 (3259845708 0 0) 0x55f673971200 con 0x55f64e3c3800 > -5994> 2018-07-11 08:05:36.786938 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.9 10.8.128.107:0/12617 201 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:36.786550) v4 ==== 2004+0+0 (3259845708 0 0) 0x55f67d138a00 con 0x55f64e3c5000 > -5993> 2018-07-11 08:05:36.786948 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:36.786550) v4 -- 0x55f67d138c00 con 0 > -5992> 2018-07-11 08:05:36.787006 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:36.786550) v4 -- 0x55f6507f9600 con 0 > -5991> 2018-07-11 08:05:36.804511 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f65d553440 con 0 > -5990> 2018-07-11 08:05:36.806074 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 232 0x55f6507c6a00 replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:509/504,start:6:a8446b75:::benchmark_data_magna108_35617_object386777:0,end:6:a8456549:::benchmark_data_magna108.ceph.redhat.com_2478_object357161:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -5989> 2018-07-11 08:05:36.806097 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 232 ==== replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:509/504,start:6:a8446b75:::benchmark_data_magna108_35617_object386777:0,end:6:a8456549:::benchmark_data_magna108.ceph.redhat.com_2478_object357161:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (636878185 0 0) 0x55f6507c6a00 con 0x55f66315e800 > -5988> 2018-07-11 08:05:36.834638 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15885 conn(0x55f663217800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 205 0x55f67d136000 osd_ping(ping e509 stamp 2018-07-11 08:05:37.470224) v4 > -5987> 2018-07-11 08:05:36.834663 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.7 10.8.128.114:0/15885 205 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:37.470224) v4 ==== 2004+0+0 (69031485 0 0) 0x55f67d136000 con 0x55f663217800 > -5986> 2018-07-11 08:05:36.834678 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:37.470224) v4 -- 0x55f67d138c00 con 0 > -5985> 2018-07-11 08:05:36.834789 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15885 conn(0x55f663216000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 205 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:05:37.470224) v4 > -5984> 2018-07-11 08:05:36.834810 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.7 10.8.128.114:0/15885 205 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:37.470224) v4 ==== 2004+0+0 (69031485 0 0) 0x55f67d1afe00 con 0x55f663216000 > -5983> 2018-07-11 08:05:36.834822 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:37.470224) v4 -- 0x55f6507f9a00 con 0 > -5982> 2018-07-11 08:05:36.966921 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 437 0x55f673971200 MOSDScrubReserve(11.3s5 REQUEST e509) v1 > -5981> 2018-07-11 08:05:36.966941 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 437 ==== MOSDScrubReserve(11.3s5 REQUEST e509) v1 ==== 43+0+0 (433425070 0 0) 0x55f673971200 con 0x55f668a29000 > -5980> 2018-07-11 08:05:36.967004 7f491a040700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e509) v1 -- 0x55f66e944800 con 0 > -5979> 2018-07-11 08:05:36.967324 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 438 0x55f67d138c00 MOSDScrubReserve(11.3s5 RELEASE e509) v1 > -5978> 2018-07-11 08:05:36.967352 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 438 ==== MOSDScrubReserve(11.3s5 RELEASE e509) v1 ==== 43+0+0 (3603460268 0 0) 0x55f67d138c00 con 0x55f668a29000 > -5977> 2018-07-11 08:05:37.000591 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.108:0/11967 conn(0x55f663f6f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=1). rx osd.8 seq 196 0x55f67d136000 osd_ping(ping e509 stamp 2018-07-11 08:05:36.760536) v4 > -5976> 2018-07-11 08:05:37.000598 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.108:0/11967 conn(0x55f663f70800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=1). rx osd.8 seq 196 0x55f666183400 osd_ping(ping e509 stamp 2018-07-11 08:05:36.760536) v4 > -5975> 2018-07-11 08:05:37.000615 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.8 10.8.128.108:0/11967 196 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:36.760536) v4 ==== 2004+0+0 (3772152160 0 0) 0x55f666183400 con 0x55f663f70800 > -5974> 2018-07-11 08:05:37.000614 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.8 10.8.128.108:0/11967 196 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:36.760536) v4 ==== 2004+0+0 (3772152160 0 0) 0x55f67d136000 con 0x55f663f6f000 > -5973> 2018-07-11 08:05:37.000624 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:36.760536) v4 -- 0x55f666183800 con 0 > -5972> 2018-07-11 08:05:37.000651 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:36.760536) v4 -- 0x55f66cb51200 con 0 > -5971> 2018-07-11 08:05:37.104672 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0a000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 199 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:05:37.740316) v4 > -5970> 2018-07-11 08:05:37.104689 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0b800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 199 0x55f66e944800 osd_ping(ping e509 stamp 2018-07-11 08:05:37.740316) v4 > -5969> 2018-07-11 08:05:37.104697 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.11 10.8.128.114:0/15919 199 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:37.740316) v4 ==== 2004+0+0 (1847552370 0 0) 0x55f64e20fc00 con 0x55f663f0a000 > -5968> 2018-07-11 08:05:37.104706 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.11 10.8.128.114:0/15919 199 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:37.740316) v4 ==== 2004+0+0 (1847552370 0 0) 0x55f66e944800 con 0x55f663f0b800 > -5967> 2018-07-11 08:05:37.104711 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:37.740316) v4 -- 0x55f67d139600 con 0 > -5966> 2018-07-11 08:05:37.104740 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:37.740316) v4 -- 0x55f66cb51400 con 0 > -5965> 2018-07-11 08:05:37.287425 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.107:0/12617 conn(0x55f64e3c3800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.9 seq 202 0x55f67d136000 osd_ping(ping e509 stamp 2018-07-11 08:05:37.287025) v4 > -5964> 2018-07-11 08:05:37.287473 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.9 10.8.128.107:0/12617 202 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:37.287025) v4 ==== 2004+0+0 (3450437624 0 0) 0x55f67d136000 con 0x55f64e3c3800 > -5963> 2018-07-11 08:05:37.287465 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.107:0/12617 conn(0x55f64e3c5000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.9 seq 202 0x55f6507f9600 osd_ping(ping e509 stamp 2018-07-11 08:05:37.287025) v4 > -5962> 2018-07-11 08:05:37.287488 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:37.287025) v4 -- 0x55f66cb51600 con 0 > -5961> 2018-07-11 08:05:37.287493 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.9 10.8.128.107:0/12617 202 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:37.287025) v4 ==== 2004+0+0 (3450437624 0 0) 0x55f6507f9600 con 0x55f64e3c5000 > -5960> 2018-07-11 08:05:37.287506 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:37.287025) v4 -- 0x55f666183a00 con 0 > -5959> 2018-07-11 08:05:37.531060 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12650 conn(0x55f663ed9000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.3 seq 200 0x55f66e944800 osd_ping(ping e509 stamp 2018-07-11 08:05:37.587907) v4 > -5958> 2018-07-11 08:05:37.531047 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12650 conn(0x55f663eda800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.3 seq 200 0x55f666183400 osd_ping(ping e509 stamp 2018-07-11 08:05:37.587907) v4 > -5957> 2018-07-11 08:05:37.531088 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.3 10.8.128.106:0/12650 200 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:37.587907) v4 ==== 2004+0+0 (723126035 0 0) 0x55f66e944800 con 0x55f663ed9000 > -5956> 2018-07-11 08:05:37.531092 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.3 10.8.128.106:0/12650 200 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:37.587907) v4 ==== 2004+0+0 (723126035 0 0) 0x55f666183400 con 0x55f663eda800 > -5955> 2018-07-11 08:05:37.531099 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:37.587907) v4 -- 0x55f66cb51800 con 0 > -5954> 2018-07-11 08:05:37.531132 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:37.587907) v4 -- 0x55f666183c00 con 0 > -5953> 2018-07-11 08:05:38.456951 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0d000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 208 0x55f6507f9600 osd_ping(ping e509 stamp 2018-07-11 08:05:39.701713) v4 > -5952> 2018-07-11 08:05:38.456984 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.5 10.8.128.113:0/15263 208 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:39.701713) v4 ==== 2004+0+0 (2501258555 0 0) 0x55f6507f9600 con 0x55f663f0d000 > -5951> 2018-07-11 08:05:38.456995 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:39.701713) v4 -- 0x55f666183c00 con 0 > -5950> 2018-07-11 08:05:38.457003 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0e800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 208 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:05:39.701713) v4 > -5949> 2018-07-11 08:05:38.457026 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.5 10.8.128.113:0/15263 208 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:39.701713) v4 ==== 2004+0+0 (2501258555 0 0) 0x55f67d1afe00 con 0x55f663f0e800 > -5948> 2018-07-11 08:05:38.457041 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:39.701713) v4 -- 0x55f67d139600 con 0 > -5947> 2018-07-11 08:05:38.534965 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 29] Generated table #37042: 2638241 keys, 74546437 bytes > -5946> 2018-07-11 08:05:38.535013 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296338534997, "cf_name": "default", "job": 29, "event": "table_file_creation", "file_number": 37042, "file_size": 74546437, "table_properties": {"data_size": 67112133, "index_size": 690131, "filter_size": 6743183, "raw_key_size": 76215851, "raw_average_key_size": 28, "raw_value_size": 31365754, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} > -5945> 2018-07-11 08:05:38.579897 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f65f557200 con 0 > -5944> 2018-07-11 08:05:38.581612 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 233 0x55f6507c6c80 replica scrub(pg: 6.15,from:0'0,to:430'106581,epoch:509/504,start:6:a8456549:::benchmark_data_magna108.ceph.redhat.com_2478_object357161:0,end:6:a8463676:::benchmark_data_magna108.ceph.redhat.com_2478_object167376:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -5943> 2018-07-11 08:05:38.581645 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 233 ==== replica scrub(pg: 6.15,from:0'0,to:430'106581,epoch:509/504,start:6:a8456549:::benchmark_data_magna108.ceph.redhat.com_2478_object357161:0,end:6:a8463676:::benchmark_data_magna108.ceph.redhat.com_2478_object167376:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 248+0+0 (1541483839 0 0) 0x55f6507c6c80 con 0x55f66315e800 > -5942> 2018-07-11 08:05:38.805132 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0a000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 200 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:05:39.440756) v4 > -5941> 2018-07-11 08:05:38.805155 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.11 10.8.128.114:0/15919 200 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:39.440756) v4 ==== 2004+0+0 (556371903 0 0) 0x55f64e20fc00 con 0x55f663f0a000 > -5940> 2018-07-11 08:05:38.805165 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:39.440756) v4 -- 0x55f67d139600 con 0 > -5939> 2018-07-11 08:05:38.805150 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0b800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 200 0x55f67d136000 osd_ping(ping e509 stamp 2018-07-11 08:05:39.440756) v4 > -5938> 2018-07-11 08:05:38.805173 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.11 10.8.128.114:0/15919 200 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:39.440756) v4 ==== 2004+0+0 (556371903 0 0) 0x55f67d136000 con 0x55f663f0b800 > -5937> 2018-07-11 08:05:38.805190 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:39.440756) v4 -- 0x55f66cb51800 con 0 > -5936> 2018-07-11 08:05:38.954783 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12651 conn(0x55f663219000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25 cs=1 l=1). rx osd.2 seq 211 0x55f666183400 osd_ping(ping e509 stamp 2018-07-11 08:05:39.011446) v4 > -5935> 2018-07-11 08:05:38.954815 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.2 10.8.128.106:0/12651 211 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:39.011446) v4 ==== 2004+0+0 (90251212 0 0) 0x55f666183400 con 0x55f663219000 > -5934> 2018-07-11 08:05:38.954830 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:39.011446) v4 -- 0x55f666183c00 con 0 > -5933> 2018-07-11 08:05:38.954910 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12651 conn(0x55f66321a800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25 cs=1 l=1). rx osd.2 seq 211 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:05:39.011446) v4 > -5932> 2018-07-11 08:05:38.954931 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.2 10.8.128.106:0/12651 211 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:39.011446) v4 ==== 2004+0+0 (90251212 0 0) 0x55f67d1afe00 con 0x55f66321a800 > -5931> 2018-07-11 08:05:38.954941 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:39.011446) v4 -- 0x55f67d139600 con 0 > -5930> 2018-07-11 08:05:38.977461 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed7800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 202 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:05:39.034234) v4 > -5929> 2018-07-11 08:05:38.977487 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.1 10.8.128.106:0/12677 202 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:39.034234) v4 ==== 2004+0+0 (2383589939 0 0) 0x55f64e20fc00 con 0x55f663ed7800 > -5928> 2018-07-11 08:05:38.977502 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:39.034234) v4 -- 0x55f67d139600 con 0 > -5927> 2018-07-11 08:05:38.977503 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed6000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 202 0x55f6507f9600 osd_ping(ping e509 stamp 2018-07-11 08:05:39.034234) v4 > -5926> 2018-07-11 08:05:38.977524 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.1 10.8.128.106:0/12677 202 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:39.034234) v4 ==== 2004+0+0 (2383589939 0 0) 0x55f6507f9600 con 0x55f663ed6000 > -5925> 2018-07-11 08:05:38.977535 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:39.034234) v4 -- 0x55f666183c00 con 0 > -5924> 2018-07-11 08:05:39.315843 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f65d552480 con 0 > -5923> 2018-07-11 08:05:39.317758 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 234 0x55f6507c7180 replica scrub(pg: 6.15,from:0'0,to:430'106516,epoch:509/504,start:6:a8463676:::benchmark_data_magna108.ceph.redhat.com_2478_object167376:0,end:6:a847268e:::benchmark_data_magna108_35617_object656320:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -5922> 2018-07-11 08:05:39.317790 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 234 ==== replica scrub(pg: 6.15,from:0'0,to:430'106516,epoch:509/504,start:6:a8463676:::benchmark_data_magna108.ceph.redhat.com_2478_object167376:0,end:6:a847268e:::benchmark_data_magna108_35617_object656320:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (3862008350 0 0) 0x55f6507c7180 con 0x55f66315e800 > -5921> 2018-07-11 08:05:39.602993 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15287 conn(0x55f66dce1000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.4 seq 206 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:05:40.847727) v4 > -5920> 2018-07-11 08:05:39.603018 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.4 10.8.128.113:0/15287 206 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:40.847727) v4 ==== 2004+0+0 (1099717476 0 0) 0x55f67d1afe00 con 0x55f66dce1000 > -5919> 2018-07-11 08:05:39.603028 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:40.847727) v4 -- 0x55f67d139600 con 0 > -5918> 2018-07-11 08:05:39.603044 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15287 conn(0x55f66dce2800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.4 seq 206 0x55f666183400 osd_ping(ping e509 stamp 2018-07-11 08:05:40.847727) v4 > -5917> 2018-07-11 08:05:39.603066 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.4 10.8.128.113:0/15287 206 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:40.847727) v4 ==== 2004+0+0 (1099717476 0 0) 0x55f666183400 con 0x55f66dce2800 > -5916> 2018-07-11 08:05:39.603078 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:40.847727) v4 -- 0x55f666183c00 con 0 > -5915> 2018-07-11 08:05:39.664066 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.114:6805/15885 conn(0x55f66315a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.7 seq 2555 0x55f6507f9600 MOSDScrubReserve(5.4 REQUEST e509) v1 > -5914> 2018-07-11 08:05:39.664089 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.7 10.8.128.114:6805/15885 2555 ==== MOSDScrubReserve(5.4 REQUEST e509) v1 ==== 43+0+0 (3003630469 0 0) 0x55f6507f9600 con 0x55f66315a000 > -5913> 2018-07-11 08:05:39.664129 7f491983f700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.114:6805/15885 -- MOSDScrubReserve(5.4 REJECT e509) v1 -- 0x55f666182e00 con 0 > -5912> 2018-07-11 08:05:39.664482 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.114:6805/15885 conn(0x55f66315a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.7 seq 2556 0x55f666182e00 MOSDScrubReserve(5.4 RELEASE e509) v1 > -5911> 2018-07-11 08:05:39.664497 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.7 10.8.128.114:6805/15885 2556 ==== MOSDScrubReserve(5.4 RELEASE e509) v1 ==== 43+0+0 (2082093447 0 0) 0x55f666182e00 con 0x55f66315a000 > -5910> 2018-07-11 08:05:39.831440 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12650 conn(0x55f663eda800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.3 seq 201 0x55f666183400 osd_ping(ping e509 stamp 2018-07-11 08:05:39.888334) v4 > -5909> 2018-07-11 08:05:39.831465 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.3 10.8.128.106:0/12650 201 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:39.888334) v4 ==== 2004+0+0 (3311744858 0 0) 0x55f666183400 con 0x55f663eda800 > -5908> 2018-07-11 08:05:39.831457 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12650 conn(0x55f663ed9000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.3 seq 201 0x55f66e944800 osd_ping(ping e509 stamp 2018-07-11 08:05:39.888334) v4 > -5907> 2018-07-11 08:05:39.831479 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:39.888334) v4 -- 0x55f667d0ca00 con 0 > -5906> 2018-07-11 08:05:39.831484 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.3 10.8.128.106:0/12650 201 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:39.888334) v4 ==== 2004+0+0 (3311744858 0 0) 0x55f66e944800 con 0x55f663ed9000 > -5905> 2018-07-11 08:05:39.831507 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:39.888334) v4 -- 0x55f66cb51a00 con 0 > -5904> 2018-07-11 08:05:40.014763 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f652468000 con 0 > -5903> 2018-07-11 08:05:40.016366 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 235 0x55f6507c7400 replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:509/504,start:6:a847268e:::benchmark_data_magna108_35617_object656320:0,end:6:a8482c6b:::benchmark_data_magna108_35617_object208862:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -5902> 2018-07-11 08:05:40.016396 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 235 ==== replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:509/504,start:6:a847268e:::benchmark_data_magna108_35617_object656320:0,end:6:a8482c6b:::benchmark_data_magna108_35617_object208862:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (559111186 0 0) 0x55f6507c7400 con 0x55f66315e800 > -5901> 2018-07-11 08:05:40.055414 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12651 conn(0x55f663219000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25 cs=1 l=1). rx osd.2 seq 212 0x55f666183c00 osd_ping(ping e509 stamp 2018-07-11 08:05:40.112097) v4 > -5900> 2018-07-11 08:05:40.055435 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.2 10.8.128.106:0/12651 212 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:40.112097) v4 ==== 2004+0+0 (1614076050 0 0) 0x55f666183c00 con 0x55f663219000 > -5899> 2018-07-11 08:05:40.055444 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:40.112097) v4 -- 0x55f667d0cc00 con 0 > -5898> 2018-07-11 08:05:40.055470 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12651 conn(0x55f66321a800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25 cs=1 l=1). rx osd.2 seq 212 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:05:40.112097) v4 > -5897> 2018-07-11 08:05:40.055489 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.2 10.8.128.106:0/12651 212 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:40.112097) v4 ==== 2004+0+0 (1614076050 0 0) 0x55f64e20fc00 con 0x55f66321a800 > -5896> 2018-07-11 08:05:40.055501 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:40.112097) v4 -- 0x55f67d138e00 con 0 > -5895> 2018-07-11 08:05:40.077868 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed7800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 203 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:05:40.134748) v4 > -5894> 2018-07-11 08:05:40.077889 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.1 10.8.128.106:0/12677 203 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:40.134748) v4 ==== 2004+0+0 (87866328 0 0) 0x55f67d1afe00 con 0x55f663ed7800 > -5893> 2018-07-11 08:05:40.077888 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed6000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 203 0x55f667d0ca00 osd_ping(ping e509 stamp 2018-07-11 08:05:40.134748) v4 > -5892> 2018-07-11 08:05:40.077902 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:40.134748) v4 -- 0x55f65d025e00 con 0 > -5891> 2018-07-11 08:05:40.077913 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.1 10.8.128.106:0/12677 203 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:40.134748) v4 ==== 2004+0+0 (87866328 0 0) 0x55f667d0ca00 con 0x55f663ed6000 > -5890> 2018-07-11 08:05:40.077933 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:40.134748) v4 -- 0x55f667d0d600 con 0 > -5889> 2018-07-11 08:05:40.110958 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.115:0/13323 conn(0x55f663ebf800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.6 seq 191 0x55f666183c00 osd_ping(ping e509 stamp 2018-07-11 08:05:40.463683) v4 > -5888> 2018-07-11 08:05:40.110981 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.6 10.8.128.115:0/13323 191 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:40.463683) v4 ==== 2004+0+0 (1903806094 0 0) 0x55f666183c00 con 0x55f663ebf800 > -5887> 2018-07-11 08:05:40.110991 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:40.463683) v4 -- 0x55f653d4a000 con 0 > -5886> 2018-07-11 08:05:40.111044 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.115:0/13323 conn(0x55f663ebe000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.6 seq 191 0x55f67d136000 osd_ping(ping e509 stamp 2018-07-11 08:05:40.463683) v4 > -5885> 2018-07-11 08:05:40.111065 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.6 10.8.128.115:0/13323 191 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:40.463683) v4 ==== 2004+0+0 (1903806094 0 0) 0x55f67d136000 con 0x55f663ebe000 > -5884> 2018-07-11 08:05:40.111080 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:40.463683) v4 -- 0x55f66cb51a00 con 0 > -5883> 2018-07-11 08:05:40.433740 7f491e048700 10 monclient: tick > -5882> 2018-07-11 08:05:40.433755 7f491e048700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2018-07-11 08:05:10.433753) > -5881> 2018-07-11 08:05:40.529203 7f491d046700 1 -- 10.8.128.107:6800/14173 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55f667d77600 con 0 > -5880> 2018-07-11 08:05:40.529288 7f491d046700 1 -- 10.8.128.107:6800/14173 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55f666f242c0 con 0 > -5879> 2018-07-11 08:05:40.664138 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.114:6805/15885 conn(0x55f66315a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.7 seq 2557 0x55f666183c00 MOSDScrubReserve(5.4 REQUEST e509) v1 > -5878> 2018-07-11 08:05:40.664158 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.7 10.8.128.114:6805/15885 2557 ==== MOSDScrubReserve(5.4 REQUEST e509) v1 ==== 43+0+0 (3003630469 0 0) 0x55f666183c00 con 0x55f66315a000 > -5877> 2018-07-11 08:05:40.664225 7f491983f700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.114:6805/15885 -- MOSDScrubReserve(5.4 REJECT e509) v1 -- 0x55f666182e00 con 0 > -5876> 2018-07-11 08:05:40.664554 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.114:6805/15885 conn(0x55f66315a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.7 seq 2558 0x55f666182e00 MOSDScrubReserve(5.4 RELEASE e509) v1 > -5875> 2018-07-11 08:05:40.664573 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.7 10.8.128.114:6805/15885 2558 ==== MOSDScrubReserve(5.4 RELEASE e509) v1 ==== 43+0+0 (2082093447 0 0) 0x55f666182e00 con 0x55f66315a000 > -5874> 2018-07-11 08:05:40.703373 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15287 conn(0x55f66dce2800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.4 seq 207 0x55f667d0ca00 osd_ping(ping e509 stamp 2018-07-11 08:05:41.948170) v4 > -5873> 2018-07-11 08:05:40.703398 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.4 10.8.128.113:0/15287 207 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:41.948170) v4 ==== 2004+0+0 (4050419346 0 0) 0x55f667d0ca00 con 0x55f66dce2800 > -5872> 2018-07-11 08:05:40.703407 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:41.948170) v4 -- 0x55f64f769800 con 0 > -5871> 2018-07-11 08:05:40.703398 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15287 conn(0x55f66dce1000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.4 seq 207 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:05:41.948170) v4 > -5870> 2018-07-11 08:05:40.703415 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.4 10.8.128.113:0/15287 207 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:41.948170) v4 ==== 2004+0+0 (4050419346 0 0) 0x55f64e20fc00 con 0x55f66dce1000 > -5869> 2018-07-11 08:05:40.703424 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:41.948170) v4 -- 0x55f653d4be00 con 0 > -5868> 2018-07-11 08:05:40.898069 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 29] Generated table #37043: 1067172 keys, 27570702 bytes > -5867> 2018-07-11 08:05:40.898113 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296340898097, "cf_name": "default", "job": 29, "event": "table_file_creation", "file_number": 37043, "file_size": 27570702, "table_properties": {"data_size": 24595035, "index_size": 252308, "filter_size": 2722369, "raw_key_size": 30829425, "raw_average_key_size": 28, "raw_value_size": 10138026, "raw_average_value_size": 9, "num_data_blocks": 6042, "num_entries": 1067172, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "214442", "kMergeOperands": "0"}} > -5866> 2018-07-11 08:05:40.898147 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1173] [default] [JOB 29] Compacted 2@3 + 11@4 files to L4 => 698488122 bytes > -5865> 2018-07-11 08:05:40.907769 7f4922851700 3 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/column_family.cc:744] [default] Increasing compaction threads because of estimated pending compaction bytes 29184525102 > -5864> 2018-07-11 08:05:40.908006 7f4922851700 4 rocksdb: (Original Log Time 2018/07/11-08:05:40.907736) [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:621] [default] compacted to: base level 1 max bytes base 268435456 files[1 7 48 480 3640 796 0] max score 1.12, MB/sec: 18.6 rd, 13.9 wr, level 4, files in(2, 11) out(10) MB in(108.7, 782.0) out(666.1), read-write-amplify(14.3) write-amplify(6.1) OK, records in: 35405602, records dropped: 10594262 > > -5863> 2018-07-11 08:05:40.908015 7f4922851700 4 rocksdb: (Original Log Time 2018/07/11-08:05:40.907761) EVENT_LOG_v1 {"time_micros": 1531296340907750, "job": 29, "event": "compaction_finished", "compaction_time_micros": 50171842, "output_level": 4, "num_output_files": 10, "total_output_size": 698488122, "num_input_records": 35405602, "num_output_records": 24811340, "num_subcompactions": 1, "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 7, 48, 480, 3640, 796, 0]} > -5862> 2018-07-11 08:05:40.908653 7f4922851700 5 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:307] [JOB 29] Delete db.slow/036785.sst type=2 #36785 -- OK > > -5861> 2018-07-11 08:05:40.908678 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296340908662, "job": 29, "event": "table_file_deletion", "file_number": 36785} > -5860> 2018-07-11 08:05:40.908690 7f4922851700 5 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:307] [JOB 29] Delete db.slow/036581.sst type=2 #36581 -- OK > > -5859> 2018-07-11 08:05:40.908698 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296340908695, "job": 29, "event": "table_file_deletion", "file_number": 36581} > -5858> 2018-07-11 08:05:40.908706 7f4922851700 5 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:307] [JOB 29] Delete db.slow/027369.sst type=2 #27369 -- OK > > -5857> 2018-07-11 08:05:40.908712 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296340908709, "job": 29, "event": "table_file_deletion", "file_number": 27369} > -5856> 2018-07-11 08:05:40.908718 7f4922851700 5 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:307] [JOB 29] Delete db.slow/027368.sst type=2 #27368 -- OK > > -5855> 2018-07-11 08:05:40.908725 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296340908721, "job": 29, "event": "table_file_deletion", "file_number": 27368} > -5854> 2018-07-11 08:05:40.908740 7f4922851700 5 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:307] [JOB 29] Delete db.slow/027365.sst type=2 #27365 -- OK > > -5853> 2018-07-11 08:05:40.908746 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296340908743, "job": 29, "event": "table_file_deletion", "file_number": 27365} > -5852> 2018-07-11 08:05:40.908755 7f4922851700 5 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:307] [JOB 29] Delete db.slow/027364.sst type=2 #27364 -- OK > > -5851> 2018-07-11 08:05:40.908760 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296340908757, "job": 29, "event": "table_file_deletion", "file_number": 27364} > -5850> 2018-07-11 08:05:40.908767 7f4922851700 5 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:307] [JOB 29] Delete db.slow/027363.sst type=2 #27363 -- OK > > -5849> 2018-07-11 08:05:40.908772 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296340908770, "job": 29, "event": "table_file_deletion", "file_number": 27363} > -5848> 2018-07-11 08:05:40.908780 7f4922851700 5 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:307] [JOB 29] Delete db.slow/027361.sst type=2 #27361 -- OK > > -5847> 2018-07-11 08:05:40.908785 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296340908783, "job": 29, "event": "table_file_deletion", "file_number": 27361} > -5846> 2018-07-11 08:05:40.908794 7f4922851700 5 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:307] [JOB 29] Delete db.slow/026985.sst type=2 #26985 -- OK > > -5845> 2018-07-11 08:05:40.908800 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296340908797, "job": 29, "event": "table_file_deletion", "file_number": 26985} > -5844> 2018-07-11 08:05:40.908808 7f4922851700 5 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:307] [JOB 29] Delete db.slow/026984.sst type=2 #26984 -- OK > > -5843> 2018-07-11 08:05:40.908813 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296340908811, "job": 29, "event": "table_file_deletion", "file_number": 26984} > -5842> 2018-07-11 08:05:40.908821 7f4922851700 5 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:307] [JOB 29] Delete db.slow/026983.sst type=2 #26983 -- OK > > -5841> 2018-07-11 08:05:40.908826 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296340908824, "job": 29, "event": "table_file_deletion", "file_number": 26983} > -5840> 2018-07-11 08:05:40.908834 7f4922851700 5 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:307] [JOB 29] Delete db.slow/026981.sst type=2 #26981 -- OK > > -5839> 2018-07-11 08:05:40.908839 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296340908837, "job": 29, "event": "table_file_deletion", "file_number": 26981} > -5838> 2018-07-11 08:05:40.908847 7f4922851700 5 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/db_impl_files.cc:307] [JOB 29] Delete db.slow/026980.sst type=2 #26980 -- OK > > -5837> 2018-07-11 08:05:40.908853 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296340908850, "job": 29, "event": "table_file_deletion", "file_number": 26980} > -5836> 2018-07-11 08:05:40.909047 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1403] [default] [JOB 30] Compacting 1@3 + 18@4 files to L4, score 1.12 > -5835> 2018-07-11 08:05:40.909064 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1407] [default] Compaction start summary: Base version 29 Base level 3, inputs: [36637(76MB)], [25818(71MB) 26341(71MB) 26342(71MB) 26343(71MB) 26347(71MB) 26348(71MB) 26349(71MB) 26350(71MB) 26351(71MB) 26352(71MB) 26292(71MB) 29948(71MB) 29949(71MB) 29950(71MB) 26002(71MB) 26003(71MB) 26004(71MB) 33964(71MB)] > > -5834> 2018-07-11 08:05:40.909082 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296340909070, "job": 30, "event": "compaction_started", "files_L3": [36637], "files_L4": [25818, 26341, 26342, 26343, 26347, 26348, 26349, 26350, 26351, 26352, 26292, 29948, 29949, 29950, 26002, 26003, 26004, 33964], "score": 1.12303, "input_data_size": 1422044065} > -5833> 2018-07-11 08:05:40.967643 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 439 0x55f66e944800 MOSDScrubReserve(11.3s5 REQUEST e509) v1 > -5832> 2018-07-11 08:05:40.967664 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 439 ==== MOSDScrubReserve(11.3s5 REQUEST e509) v1 ==== 43+0+0 (433425070 0 0) 0x55f66e944800 con 0x55f668a29000 > -5831> 2018-07-11 08:05:40.967701 7f491a040700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e509) v1 -- 0x55f67d138c00 con 0 > -5830> 2018-07-11 08:05:40.968057 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 440 0x55f66cb51c00 MOSDScrubReserve(11.3s5 RELEASE e509) v1 > -5829> 2018-07-11 08:05:40.968071 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 440 ==== MOSDScrubReserve(11.3s5 RELEASE e509) v1 ==== 43+0+0 (3603460268 0 0) 0x55f66cb51c00 con 0x55f668a29000 > -5828> 2018-07-11 08:05:41.437905 7f491803c700 5 osd.10 509 heartbeat: osd_stat(789 GB used, 141 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,0,0,0,1]) > -5827> 2018-07-11 08:05:41.437929 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6802/15255 -- osd_ping(ping e509 stamp 2018-07-11 08:05:41.437926) v4 -- 0x55f65c79c200 con 0 > -5826> 2018-07-11 08:05:41.437976 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6803/15255 -- osd_ping(ping e509 stamp 2018-07-11 08:05:41.437926) v4 -- 0x55f65c79c400 con 0 > -5825> 2018-07-11 08:05:41.437993 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6806/12677 -- osd_ping(ping e509 stamp 2018-07-11 08:05:41.437926) v4 -- 0x55f65c79c600 con 0 > -5824> 2018-07-11 08:05:41.438006 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6807/12677 -- osd_ping(ping e509 stamp 2018-07-11 08:05:41.437926) v4 -- 0x55f65c79c800 con 0 > -5823> 2018-07-11 08:05:41.438018 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6808/12651 -- osd_ping(ping e509 stamp 2018-07-11 08:05:41.437926) v4 -- 0x55f65c79ca00 con 0 > -5822> 2018-07-11 08:05:41.438028 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6810/12651 -- osd_ping(ping e509 stamp 2018-07-11 08:05:41.437926) v4 -- 0x55f65c79cc00 con 0 > -5821> 2018-07-11 08:05:41.438040 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6809/12650 -- osd_ping(ping e509 stamp 2018-07-11 08:05:41.437926) v4 -- 0x55f65c79ce00 con 0 > -5820> 2018-07-11 08:05:41.438053 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6811/12650 -- osd_ping(ping e509 stamp 2018-07-11 08:05:41.437926) v4 -- 0x55f65c79d000 con 0 > -5819> 2018-07-11 08:05:41.438064 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6810/15287 -- osd_ping(ping e509 stamp 2018-07-11 08:05:41.437926) v4 -- 0x55f65c79d200 con 0 > -5818> 2018-07-11 08:05:41.438084 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6811/15287 -- osd_ping(ping e509 stamp 2018-07-11 08:05:41.437926) v4 -- 0x55f65c79d400 con 0 > -5817> 2018-07-11 08:05:41.438096 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6806/15263 -- osd_ping(ping e509 stamp 2018-07-11 08:05:41.437926) v4 -- 0x55f65c79d600 con 0 > -5816> 2018-07-11 08:05:41.438109 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6807/15263 -- osd_ping(ping e509 stamp 2018-07-11 08:05:41.437926) v4 -- 0x55f65c79d800 con 0 > -5815> 2018-07-11 08:05:41.438125 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.115:6802/13323 -- osd_ping(ping e509 stamp 2018-07-11 08:05:41.437926) v4 -- 0x55f65c79da00 con 0 > -5814> 2018-07-11 08:05:41.438137 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.115:6803/13323 -- osd_ping(ping e509 stamp 2018-07-11 08:05:41.437926) v4 -- 0x55f65c79dc00 con 0 > -5813> 2018-07-11 08:05:41.438150 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6806/15885 -- osd_ping(ping e509 stamp 2018-07-11 08:05:41.437926) v4 -- 0x55f65c79de00 con 0 > -5812> 2018-07-11 08:05:41.438164 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6807/15885 -- osd_ping(ping e509 stamp 2018-07-11 08:05:41.437926) v4 -- 0x55f6739cc000 con 0 > -5811> 2018-07-11 08:05:41.438175 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.108:6802/11967 -- osd_ping(ping e509 stamp 2018-07-11 08:05:41.437926) v4 -- 0x55f6739cc200 con 0 > -5810> 2018-07-11 08:05:41.438186 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.108:6803/11967 -- osd_ping(ping e509 stamp 2018-07-11 08:05:41.437926) v4 -- 0x55f6739cc400 con 0 > -5809> 2018-07-11 08:05:41.438198 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.107:6806/12617 -- osd_ping(ping e509 stamp 2018-07-11 08:05:41.437926) v4 -- 0x55f6739cc600 con 0 > -5808> 2018-07-11 08:05:41.438211 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.107:6807/12617 -- osd_ping(ping e509 stamp 2018-07-11 08:05:41.437926) v4 -- 0x55f6739cc800 con 0 > -5807> 2018-07-11 08:05:41.438223 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6802/15919 -- osd_ping(ping e509 stamp 2018-07-11 08:05:41.437926) v4 -- 0x55f6739cca00 con 0 > -5806> 2018-07-11 08:05:41.438234 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6803/15919 -- osd_ping(ping e509 stamp 2018-07-11 08:05:41.437926) v4 -- 0x55f6739ccc00 con 0 > -5805> 2018-07-11 08:05:41.438416 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6807/12677 conn(0x55f64e3d9800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.1 seq 201 0x55f663193400 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:41.437926) v4 > -5804> 2018-07-11 08:05:41.438418 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.107:6806/12617 conn(0x55f656ade000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.9 seq 201 0x55f67d138c00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:41.437926) v4 > -5803> 2018-07-11 08:05:41.438447 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.9 10.8.128.107:6806/12617 201 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:41.437926) v4 ==== 2004+0+0 (1786302000 0 0) 0x55f67d138c00 con 0x55f656ade000 > -5802> 2018-07-11 08:05:41.438445 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.1 10.8.128.106:6807/12677 201 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:41.437926) v4 ==== 2004+0+0 (1786302000 0 0) 0x55f663193400 con 0x55f64e3d9800 > -5801> 2018-07-11 08:05:41.438503 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6803/15255 conn(0x55f656ad5800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 201 0x55f67d138c00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:41.437926) v4 > -5800> 2018-07-11 08:05:41.438523 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.0 10.8.128.113:6803/15255 201 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:41.437926) v4 ==== 2004+0+0 (1786302000 0 0) 0x55f67d138c00 con 0x55f656ad5800 > -5799> 2018-07-11 08:05:41.438528 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6802/15255 conn(0x55f656ad4000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 201 0x55f663193400 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:41.437926) v4 > -5798> 2018-07-11 08:05:41.438540 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.0 10.8.128.113:6802/15255 201 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:41.437926) v4 ==== 2004+0+0 (1786302000 0 0) 0x55f663193400 con 0x55f656ad4000 > -5797> 2018-07-11 08:05:41.438518 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.107:6807/12617 conn(0x55f656adf800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.9 seq 201 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:41.437926) v4 > -5796> 2018-07-11 08:05:41.438579 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.9 10.8.128.107:6807/12617 201 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:41.437926) v4 ==== 2004+0+0 (1786302000 0 0) 0x55f64e20fc00 con 0x55f656adf800 > -5795> 2018-07-11 08:05:41.438591 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6810/12651 conn(0x55f64e3eb800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.2 seq 201 0x55f67d138c00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:41.437926) v4 > -5794> 2018-07-11 08:05:41.438603 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6811/12650 conn(0x55f656ad8800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.3 seq 201 0x55f663193400 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:41.437926) v4 > -5793> 2018-07-11 08:05:41.438610 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.2 10.8.128.106:6810/12651 201 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:41.437926) v4 ==== 2004+0+0 (1786302000 0 0) 0x55f67d138c00 con 0x55f64e3eb800 > -5792> 2018-07-11 08:05:41.438619 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.3 10.8.128.106:6811/12650 201 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:41.437926) v4 ==== 2004+0+0 (1786302000 0 0) 0x55f663193400 con 0x55f656ad8800 > -5791> 2018-07-11 08:05:41.438630 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6806/12677 conn(0x55f64e3d8000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.1 seq 201 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:41.437926) v4 > -5790> 2018-07-11 08:05:41.438646 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.1 10.8.128.106:6806/12677 201 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:41.437926) v4 ==== 2004+0+0 (1786302000 0 0) 0x55f64e20fc00 con 0x55f64e3d8000 > -5789> 2018-07-11 08:05:41.438663 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6808/12651 conn(0x55f64e3ea000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.2 seq 201 0x55f663193400 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:41.437926) v4 > -5788> 2018-07-11 08:05:41.438669 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6811/15287 conn(0x55f64e402800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.4 seq 201 0x55f67d138c00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:41.437926) v4 > -5787> 2018-07-11 08:05:41.438685 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.2 10.8.128.106:6808/12651 201 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:41.437926) v4 ==== 2004+0+0 (1786302000 0 0) 0x55f663193400 con 0x55f64e3ea000 > -5786> 2018-07-11 08:05:41.438688 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6806/15263 conn(0x55f64e3ed000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.5 seq 201 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:41.437926) v4 > -5785> 2018-07-11 08:05:41.438688 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.4 10.8.128.113:6811/15287 201 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:41.437926) v4 ==== 2004+0+0 (1786302000 0 0) 0x55f67d138c00 con 0x55f64e402800 > -5784> 2018-07-11 08:05:41.438702 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.5 10.8.128.113:6806/15263 201 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:41.437926) v4 ==== 2004+0+0 (1786302000 0 0) 0x55f64e20fc00 con 0x55f64e3ed000 > -5783> 2018-07-11 08:05:41.438729 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6809/12650 conn(0x55f656ad7000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.3 seq 201 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:41.437926) v4 > -5782> 2018-07-11 08:05:41.438741 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.3 10.8.128.106:6809/12650 201 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:41.437926) v4 ==== 2004+0+0 (1786302000 0 0) 0x55f64e20fc00 con 0x55f656ad7000 > -5781> 2018-07-11 08:05:41.438742 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.115:6802/13323 conn(0x55f656acd000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.6 seq 201 0x55f67d138c00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:41.437926) v4 > -5780> 2018-07-11 08:05:41.438739 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6810/15287 conn(0x55f64e401000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.4 seq 201 0x55f663193400 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:41.437926) v4 > -5779> 2018-07-11 08:05:41.438756 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.6 10.8.128.115:6802/13323 201 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:41.437926) v4 ==== 2004+0+0 (1786302000 0 0) 0x55f67d138c00 con 0x55f656acd000 > -5778> 2018-07-11 08:05:41.438757 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.4 10.8.128.113:6810/15287 201 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:41.437926) v4 ==== 2004+0+0 (1786302000 0 0) 0x55f663193400 con 0x55f64e401000 > -5777> 2018-07-11 08:05:41.438801 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.115:6803/13323 conn(0x55f656ace800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.6 seq 201 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:41.437926) v4 > -5776> 2018-07-11 08:05:41.438807 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6807/15263 conn(0x55f64e3ee800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.5 seq 201 0x55f663193400 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:41.437926) v4 > -5775> 2018-07-11 08:05:41.438807 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6806/15885 conn(0x55f64e3fe000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.7 seq 201 0x55f67d138c00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:41.437926) v4 > -5774> 2018-07-11 08:05:41.438817 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.6 10.8.128.115:6803/13323 201 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:41.437926) v4 ==== 2004+0+0 (1786302000 0 0) 0x55f64e20fc00 con 0x55f656ace800 > -5773> 2018-07-11 08:05:41.438828 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.5 10.8.128.113:6807/15263 201 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:41.437926) v4 ==== 2004+0+0 (1786302000 0 0) 0x55f663193400 con 0x55f64e3ee800 > -5772> 2018-07-11 08:05:41.438834 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.7 10.8.128.114:6806/15885 201 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:41.437926) v4 ==== 2004+0+0 (1786302000 0 0) 0x55f67d138c00 con 0x55f64e3fe000 > -5771> 2018-07-11 08:05:41.438871 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.108:6802/11967 conn(0x55f656aca000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.8 seq 201 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:41.437926) v4 > -5770> 2018-07-11 08:05:41.438877 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.108:6803/11967 conn(0x55f656acb800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.8 seq 201 0x55f663193400 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:41.437926) v4 > -5769> 2018-07-11 08:05:41.438885 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.8 10.8.128.108:6802/11967 201 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:41.437926) v4 ==== 2004+0+0 (1786302000 0 0) 0x55f64e20fc00 con 0x55f656aca000 > -5768> 2018-07-11 08:05:41.438879 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6802/15919 conn(0x55f64e3db000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.11 seq 201 0x55f67d138c00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:41.437926) v4 > -5767> 2018-07-11 08:05:41.438901 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.11 10.8.128.114:6802/15919 201 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:41.437926) v4 ==== 2004+0+0 (1786302000 0 0) 0x55f67d138c00 con 0x55f64e3db000 > -5766> 2018-07-11 08:05:41.438899 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.8 10.8.128.108:6803/11967 201 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:41.437926) v4 ==== 2004+0+0 (1786302000 0 0) 0x55f663193400 con 0x55f656acb800 > -5765> 2018-07-11 08:05:41.438928 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6807/15885 conn(0x55f64e3ff800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.7 seq 201 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:41.437926) v4 > -5764> 2018-07-11 08:05:41.438941 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.7 10.8.128.114:6807/15885 201 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:41.437926) v4 ==== 2004+0+0 (1786302000 0 0) 0x55f64e20fc00 con 0x55f64e3ff800 > -5763> 2018-07-11 08:05:41.438978 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6803/15919 conn(0x55f64e3dc800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.11 seq 201 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:41.437926) v4 > -5762> 2018-07-11 08:05:41.438991 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.11 10.8.128.114:6803/15919 201 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:41.437926) v4 ==== 2004+0+0 (1786302000 0 0) 0x55f64e20fc00 con 0x55f64e3dc800 > -5761> 2018-07-11 08:05:41.570995 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f66d402fc0 con 0 > -5760> 2018-07-11 08:05:41.572859 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 236 0x55f6507c7680 replica scrub(pg: 6.15,from:0'0,to:430'105521,epoch:509/504,start:6:a8482c6b:::benchmark_data_magna108_35617_object208862:0,end:6:a849027f:::benchmark_data_magna108_35617_object728366:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -5759> 2018-07-11 08:05:41.572892 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 236 ==== replica scrub(pg: 6.15,from:0'0,to:430'105521,epoch:509/504,start:6:a8482c6b:::benchmark_data_magna108_35617_object208862:0,end:6:a849027f:::benchmark_data_magna108_35617_object728366:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (1923010770 0 0) 0x55f6507c7680 con 0x55f66315e800 > -5758> 2018-07-11 08:05:41.701031 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.108:0/11967 conn(0x55f663f70800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=1). rx osd.8 seq 197 0x55f663193400 osd_ping(ping e509 stamp 2018-07-11 08:05:41.460983) v4 > -5757> 2018-07-11 08:05:41.701056 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.8 10.8.128.108:0/11967 197 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:41.460983) v4 ==== 2004+0+0 (3987654039 0 0) 0x55f663193400 con 0x55f663f70800 > -5756> 2018-07-11 08:05:41.701066 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:41.460983) v4 -- 0x55f65c79d000 con 0 > -5755> 2018-07-11 08:05:41.701084 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.108:0/11967 conn(0x55f663f6f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=1). rx osd.8 seq 197 0x55f67d138c00 osd_ping(ping e509 stamp 2018-07-11 08:05:41.460983) v4 > -5754> 2018-07-11 08:05:41.701107 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.8 10.8.128.108:0/11967 197 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:41.460983) v4 ==== 2004+0+0 (3987654039 0 0) 0x55f67d138c00 con 0x55f663f6f000 > -5753> 2018-07-11 08:05:41.701119 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:41.460983) v4 -- 0x55f65c79da00 con 0 > -5752> 2018-07-11 08:05:41.957303 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0e800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 209 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:05:43.202128) v4 > -5751> 2018-07-11 08:05:41.957319 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0d000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 209 0x55f667d0ca00 osd_ping(ping e509 stamp 2018-07-11 08:05:43.202128) v4 > -5750> 2018-07-11 08:05:41.957336 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.5 10.8.128.113:0/15263 209 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:43.202128) v4 ==== 2004+0+0 (3537272161 0 0) 0x55f667d0ca00 con 0x55f663f0d000 > -5749> 2018-07-11 08:05:41.957336 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.5 10.8.128.113:0/15263 209 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:43.202128) v4 ==== 2004+0+0 (3537272161 0 0) 0x55f64e20fc00 con 0x55f663f0e800 > -5748> 2018-07-11 08:05:41.957345 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:43.202128) v4 -- 0x55f65c79ca00 con 0 > -5747> 2018-07-11 08:05:41.957373 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:43.202128) v4 -- 0x55f6739cc000 con 0 > -5746> 2018-07-11 08:05:41.967838 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 441 0x55f67d136000 MOSDScrubReserve(11.3s5 REQUEST e509) v1 > -5745> 2018-07-11 08:05:41.967856 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 441 ==== MOSDScrubReserve(11.3s5 REQUEST e509) v1 ==== 43+0+0 (433425070 0 0) 0x55f67d136000 con 0x55f668a29000 > -5744> 2018-07-11 08:05:41.967936 7f491a040700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e509) v1 -- 0x55f66cb51c00 con 0 > -5743> 2018-07-11 08:05:41.968213 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 442 0x55f65c79d400 MOSDScrubReserve(11.3s5 RELEASE e509) v1 > -5742> 2018-07-11 08:05:41.968228 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 442 ==== MOSDScrubReserve(11.3s5 RELEASE e509) v1 ==== 43+0+0 (3603460268 0 0) 0x55f65c79d400 con 0x55f668a29000 > -5741> 2018-07-11 08:05:42.178372 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15255 conn(0x55f663ec1000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.0 seq 197 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:05:43.423178) v4 > -5740> 2018-07-11 08:05:42.178398 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.0 10.8.128.113:0/15255 197 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:43.423178) v4 ==== 2004+0+0 (1274797451 0 0) 0x55f67d1afe00 con 0x55f663ec1000 > -5739> 2018-07-11 08:05:42.178405 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15255 conn(0x55f663ec2800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.0 seq 197 0x55f67d138c00 osd_ping(ping e509 stamp 2018-07-11 08:05:43.423178) v4 > -5738> 2018-07-11 08:05:42.178415 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:43.423178) v4 -- 0x55f65c79dc00 con 0 > -5737> 2018-07-11 08:05:42.178426 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.0 10.8.128.113:0/15255 197 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:43.423178) v4 ==== 2004+0+0 (1274797451 0 0) 0x55f67d138c00 con 0x55f663ec2800 > -5736> 2018-07-11 08:05:42.178440 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:43.423178) v4 -- 0x55f67d192000 con 0 > -5735> 2018-07-11 08:05:42.731901 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12650 conn(0x55f663ed9000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.3 seq 202 0x55f66cb51c00 osd_ping(ping e509 stamp 2018-07-11 08:05:42.788818) v4 > -5734> 2018-07-11 08:05:42.731913 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12650 conn(0x55f663eda800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.3 seq 202 0x55f663193400 osd_ping(ping e509 stamp 2018-07-11 08:05:42.788818) v4 > -5733> 2018-07-11 08:05:42.731925 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.3 10.8.128.106:0/12650 202 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:42.788818) v4 ==== 2004+0+0 (3905104698 0 0) 0x55f66cb51c00 con 0x55f663ed9000 > -5732> 2018-07-11 08:05:42.731934 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:42.788818) v4 -- 0x55f67d192000 con 0 > -5731> 2018-07-11 08:05:42.731930 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.3 10.8.128.106:0/12650 202 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:42.788818) v4 ==== 2004+0+0 (3905104698 0 0) 0x55f663193400 con 0x55f663eda800 > -5730> 2018-07-11 08:05:42.731964 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:42.788818) v4 -- 0x55f65c79ca00 con 0 > -5729> 2018-07-11 08:05:42.735165 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15885 conn(0x55f663217800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 206 0x55f67d138c00 osd_ping(ping e509 stamp 2018-07-11 08:05:43.370854) v4 > -5728> 2018-07-11 08:05:42.735185 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.7 10.8.128.114:0/15885 206 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:43.370854) v4 ==== 2004+0+0 (3638973790 0 0) 0x55f67d138c00 con 0x55f663217800 > -5727> 2018-07-11 08:05:42.735197 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:43.370854) v4 -- 0x55f67d192000 con 0 > -5726> 2018-07-11 08:05:42.735224 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15885 conn(0x55f663216000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 206 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:05:43.370854) v4 > -5725> 2018-07-11 08:05:42.735252 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.7 10.8.128.114:0/15885 206 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:43.370854) v4 ==== 2004+0+0 (3638973790 0 0) 0x55f64e20fc00 con 0x55f663216000 > -5724> 2018-07-11 08:05:42.735266 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:43.370854) v4 -- 0x55f65c79d600 con 0 > -5723> 2018-07-11 08:05:42.740513 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f664b86d80 con 0 > -5722> 2018-07-11 08:05:42.742195 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 237 0x55f6507c7900 replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:509/504,start:6:a849027f:::benchmark_data_magna108_35617_object728366:0,end:6:a849b846:::benchmark_data_magna108_35617_object211229:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -5721> 2018-07-11 08:05:42.742225 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 237 ==== replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:509/504,start:6:a849027f:::benchmark_data_magna108_35617_object728366:0,end:6:a849b846:::benchmark_data_magna108_35617_object211229:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (5936194 0 0) 0x55f6507c7900 con 0x55f66315e800 > -5720> 2018-07-11 08:05:42.905429 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0a000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 201 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:05:43.541140) v4 > -5719> 2018-07-11 08:05:42.905451 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.11 10.8.128.114:0/15919 201 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:43.541140) v4 ==== 2004+0+0 (2673195244 0 0) 0x55f67d1afe00 con 0x55f663f0a000 > -5718> 2018-07-11 08:05:42.905444 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0b800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 201 0x55f66cb51c00 osd_ping(ping e509 stamp 2018-07-11 08:05:43.541140) v4 > -5717> 2018-07-11 08:05:42.905463 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:43.541140) v4 -- 0x55f65c79ce00 con 0 > -5716> 2018-07-11 08:05:42.905467 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.11 10.8.128.114:0/15919 201 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:43.541140) v4 ==== 2004+0+0 (2673195244 0 0) 0x55f66cb51c00 con 0x55f663f0b800 > -5715> 2018-07-11 08:05:42.905487 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:43.541140) v4 -- 0x55f67d192200 con 0 > -5714> 2018-07-11 08:05:42.968038 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 443 0x55f67d138c00 MOSDScrubReserve(11.3s5 REQUEST e509) v1 > -5713> 2018-07-11 08:05:42.968058 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 443 ==== MOSDScrubReserve(11.3s5 REQUEST e509) v1 ==== 43+0+0 (433425070 0 0) 0x55f67d138c00 con 0x55f668a29000 > -5712> 2018-07-11 08:05:42.968095 7f491a040700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e509) v1 -- 0x55f65c79d400 con 0 > -5711> 2018-07-11 08:05:42.968489 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 444 0x55f65c79d400 MOSDScrubReserve(11.3s5 RELEASE e509) v1 > -5710> 2018-07-11 08:05:42.968504 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 444 ==== MOSDScrubReserve(11.3s5 RELEASE e509) v1 ==== 43+0+0 (3603460268 0 0) 0x55f65c79d400 con 0x55f668a29000 > -5709> 2018-07-11 08:05:43.188113 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.107:0/12617 conn(0x55f64e3c3800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.9 seq 203 0x55f66cb51c00 osd_ping(ping e509 stamp 2018-07-11 08:05:43.187570) v4 > -5708> 2018-07-11 08:05:43.188136 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.9 10.8.128.107:0/12617 203 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:43.187570) v4 ==== 2004+0+0 (3539866645 0 0) 0x55f66cb51c00 con 0x55f64e3c3800 > -5707> 2018-07-11 08:05:43.188094 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.107:0/12617 conn(0x55f64e3c5000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.9 seq 203 0x55f667d0ca00 osd_ping(ping e509 stamp 2018-07-11 08:05:43.187570) v4 > -5706> 2018-07-11 08:05:43.188152 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.9 10.8.128.107:0/12617 203 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:43.187570) v4 ==== 2004+0+0 (3539866645 0 0) 0x55f667d0ca00 con 0x55f64e3c5000 > -5705> 2018-07-11 08:05:43.188161 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:43.187570) v4 -- 0x55f67d192a00 con 0 > -5704> 2018-07-11 08:05:43.188188 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:43.187570) v4 -- 0x55f65c79c800 con 0 > -5703> 2018-07-11 08:05:43.484404 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f66542efc0 con 0 > -5702> 2018-07-11 08:05:43.486076 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 238 0x55f6507c7b80 replica scrub(pg: 6.15,from:0'0,to:430'105692,epoch:509/504,start:6:a849b846:::benchmark_data_magna108_35617_object211229:0,end:6:a84accd8:::benchmark_data_magna108.ceph.redhat.com_2478_object22073:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -5701> 2018-07-11 08:05:43.486109 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 238 ==== replica scrub(pg: 6.15,from:0'0,to:430'105692,epoch:509/504,start:6:a849b846:::benchmark_data_magna108_35617_object211229:0,end:6:a84accd8:::benchmark_data_magna108.ceph.redhat.com_2478_object22073:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 232+0+0 (2408907535 0 0) 0x55f6507c7b80 con 0x55f66315e800 > -5700> 2018-07-11 08:05:43.611418 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.115:0/13323 conn(0x55f663ebf800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.6 seq 192 0x55f65c79ca00 osd_ping(ping e509 stamp 2018-07-11 08:05:43.964173) v4 > -5699> 2018-07-11 08:05:43.611439 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.6 10.8.128.115:0/13323 192 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:43.964173) v4 ==== 2004+0+0 (446148294 0 0) 0x55f65c79ca00 con 0x55f663ebf800 > -5698> 2018-07-11 08:05:43.611450 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:43.964173) v4 -- 0x55f65c79c800 con 0 > -5697> 2018-07-11 08:05:43.611607 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.115:0/13323 conn(0x55f663ebe000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.6 seq 192 0x55f67d192600 osd_ping(ping e509 stamp 2018-07-11 08:05:43.964173) v4 > -5696> 2018-07-11 08:05:43.611626 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.6 10.8.128.115:0/13323 192 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:43.964173) v4 ==== 2004+0+0 (446148294 0 0) 0x55f67d192600 con 0x55f663ebe000 > -5695> 2018-07-11 08:05:43.611638 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:43.964173) v4 -- 0x55f67d192a00 con 0 > -5694> 2018-07-11 08:05:43.835571 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15885 conn(0x55f663217800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 207 0x55f66cb51c00 osd_ping(ping e509 stamp 2018-07-11 08:05:44.471266) v4 > -5693> 2018-07-11 08:05:43.835602 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.7 10.8.128.114:0/15885 207 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:44.471266) v4 ==== 2004+0+0 (1135616219 0 0) 0x55f66cb51c00 con 0x55f663217800 > -5692> 2018-07-11 08:05:43.835621 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:44.471266) v4 -- 0x55f67d192a00 con 0 > -5691> 2018-07-11 08:05:43.835675 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15885 conn(0x55f663216000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 207 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:05:44.471266) v4 > -5690> 2018-07-11 08:05:43.835697 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.7 10.8.128.114:0/15885 207 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:44.471266) v4 ==== 2004+0+0 (1135616219 0 0) 0x55f64e20fc00 con 0x55f663216000 > -5689> 2018-07-11 08:05:43.835708 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:44.471266) v4 -- 0x55f65c79ce00 con 0 > -5688> 2018-07-11 08:05:44.005894 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0a000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 202 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:05:44.641559) v4 > -5687> 2018-07-11 08:05:44.005894 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0b800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 202 0x55f67d192600 osd_ping(ping e509 stamp 2018-07-11 08:05:44.641559) v4 > -5686> 2018-07-11 08:05:44.005922 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.11 10.8.128.114:0/15919 202 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:44.641559) v4 ==== 2004+0+0 (3712335230 0 0) 0x55f67d1afe00 con 0x55f663f0a000 > -5685> 2018-07-11 08:05:44.005925 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.11 10.8.128.114:0/15919 202 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:44.641559) v4 ==== 2004+0+0 (3712335230 0 0) 0x55f67d192600 con 0x55f663f0b800 > -5684> 2018-07-11 08:05:44.005939 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:44.641559) v4 -- 0x55f65c79c600 con 0 > -5683> 2018-07-11 08:05:44.005965 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:44.641559) v4 -- 0x55f67d192c00 con 0 > -5682> 2018-07-11 08:05:44.203744 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15287 conn(0x55f66dce2800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.4 seq 208 0x55f667d0ca00 osd_ping(ping e509 stamp 2018-07-11 08:05:45.448614) v4 > -5681> 2018-07-11 08:05:44.203765 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.4 10.8.128.113:0/15287 208 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:45.448614) v4 ==== 2004+0+0 (3944169978 0 0) 0x55f667d0ca00 con 0x55f66dce2800 > -5680> 2018-07-11 08:05:44.203777 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:45.448614) v4 -- 0x55f65c79c200 con 0 > -5679> 2018-07-11 08:05:44.203776 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15287 conn(0x55f66dce1000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.4 seq 208 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:05:45.448614) v4 > -5678> 2018-07-11 08:05:44.203797 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.4 10.8.128.113:0/15287 208 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:45.448614) v4 ==== 2004+0+0 (3944169978 0 0) 0x55f64e20fc00 con 0x55f66dce1000 > -5677> 2018-07-11 08:05:44.203809 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:45.448614) v4 -- 0x55f653d4be00 con 0 > -5676> 2018-07-11 08:05:44.257692 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0e800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 210 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:05:45.502539) v4 > -5675> 2018-07-11 08:05:44.257708 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0d000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 210 0x55f65c79ca00 osd_ping(ping e509 stamp 2018-07-11 08:05:45.502539) v4 > -5674> 2018-07-11 08:05:44.257716 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.5 10.8.128.113:0/15263 210 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:45.502539) v4 ==== 2004+0+0 (2732602519 0 0) 0x55f67d1afe00 con 0x55f663f0e800 > -5673> 2018-07-11 08:05:44.257725 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.5 10.8.128.113:0/15263 210 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:45.502539) v4 ==== 2004+0+0 (2732602519 0 0) 0x55f65c79ca00 con 0x55f663f0d000 > -5672> 2018-07-11 08:05:44.257731 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:45.502539) v4 -- 0x55f6548fae00 con 0 > -5671> 2018-07-11 08:05:44.257763 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:45.502539) v4 -- 0x55f64f769800 con 0 > -5670> 2018-07-11 08:05:44.310036 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f664b87b00 con 0 > -5669> 2018-07-11 08:05:44.311700 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 239 0x55f6713c8000 replica scrub(pg: 6.15,from:0'0,to:430'105706,epoch:509/504,start:6:a84accd8:::benchmark_data_magna108.ceph.redhat.com_2478_object22073:0,end:6:a84ba01d:::benchmark_data_magna108_35617_object992145:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -5668> 2018-07-11 08:05:44.311722 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 239 ==== replica scrub(pg: 6.15,from:0'0,to:430'105706,epoch:509/504,start:6:a84accd8:::benchmark_data_magna108.ceph.redhat.com_2478_object22073:0,end:6:a84ba01d:::benchmark_data_magna108_35617_object992145:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 232+0+0 (2256701290 0 0) 0x55f6713c8000 con 0x55f66315e800 > -5667> 2018-07-11 08:05:44.506280 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0a000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 203 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:05:45.141983) v4 > -5666> 2018-07-11 08:05:44.506292 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0b800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 203 0x55f66cb51c00 osd_ping(ping e509 stamp 2018-07-11 08:05:45.141983) v4 > -5665> 2018-07-11 08:05:44.506305 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.11 10.8.128.114:0/15919 203 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:45.141983) v4 ==== 2004+0+0 (3488151912 0 0) 0x55f64e20fc00 con 0x55f663f0a000 > -5664> 2018-07-11 08:05:44.506315 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.11 10.8.128.114:0/15919 203 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:45.141983) v4 ==== 2004+0+0 (3488151912 0 0) 0x55f66cb51c00 con 0x55f663f0b800 > -5663> 2018-07-11 08:05:44.506320 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:45.141983) v4 -- 0x55f6548fae00 con 0 > -5662> 2018-07-11 08:05:44.506348 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:45.141983) v4 -- 0x55f67d192c00 con 0 > -5661> 2018-07-11 08:05:44.758157 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0e800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 211 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:05:46.003034) v4 > -5660> 2018-07-11 08:05:44.758163 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0d000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 211 0x55f667d0ca00 osd_ping(ping e509 stamp 2018-07-11 08:05:46.003034) v4 > -5659> 2018-07-11 08:05:44.758194 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.5 10.8.128.113:0/15263 211 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:46.003034) v4 ==== 2004+0+0 (3137553663 0 0) 0x55f67d1afe00 con 0x55f663f0e800 > -5658> 2018-07-11 08:05:44.758198 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.5 10.8.128.113:0/15263 211 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:46.003034) v4 ==== 2004+0+0 (3137553663 0 0) 0x55f667d0ca00 con 0x55f663f0d000 > -5657> 2018-07-11 08:05:44.758209 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:46.003034) v4 -- 0x55f6548fae00 con 0 > -5656> 2018-07-11 08:05:44.758237 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:46.003034) v4 -- 0x55f66dceb200 con 0 > -5655> 2018-07-11 08:05:44.778227 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed7800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 204 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:05:44.835135) v4 > -5654> 2018-07-11 08:05:44.778262 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.1 10.8.128.106:0/12677 204 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:44.835135) v4 ==== 2004+0+0 (3858863729 0 0) 0x55f64e20fc00 con 0x55f663ed7800 > -5653> 2018-07-11 08:05:44.778272 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:44.835135) v4 -- 0x55f6548fae00 con 0 > -5652> 2018-07-11 08:05:44.778305 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed6000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 204 0x55f64f769800 osd_ping(ping e509 stamp 2018-07-11 08:05:44.835135) v4 > -5651> 2018-07-11 08:05:44.778323 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.1 10.8.128.106:0/12677 204 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:44.835135) v4 ==== 2004+0+0 (3858863729 0 0) 0x55f64f769800 con 0x55f663ed6000 > -5650> 2018-07-11 08:05:44.778351 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:44.835135) v4 -- 0x55f66dceb200 con 0 > -5649> 2018-07-11 08:05:44.968398 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 445 0x55f67d192600 MOSDScrubReserve(11.3s5 REQUEST e509) v1 > -5648> 2018-07-11 08:05:44.968417 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 445 ==== MOSDScrubReserve(11.3s5 REQUEST e509) v1 ==== 43+0+0 (433425070 0 0) 0x55f67d192600 con 0x55f668a29000 > -5647> 2018-07-11 08:05:44.968462 7f491a040700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e509) v1 -- 0x55f65c79d400 con 0 > -5646> 2018-07-11 08:05:44.968858 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 446 0x55f65c79d400 MOSDScrubReserve(11.3s5 RELEASE e509) v1 > -5645> 2018-07-11 08:05:44.968872 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 446 ==== MOSDScrubReserve(11.3s5 RELEASE e509) v1 ==== 43+0+0 (3603460268 0 0) 0x55f65c79d400 con 0x55f668a29000 > -5644> 2018-07-11 08:05:45.354358 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f65bb678c0 con 0 > -5643> 2018-07-11 08:05:45.356030 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12651 conn(0x55f66321a800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25 cs=1 l=1). rx osd.2 seq 213 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:05:45.412720) v4 > -5642> 2018-07-11 08:05:45.356053 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.2 10.8.128.106:0/12651 213 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:45.412720) v4 ==== 2004+0+0 (534926139 0 0) 0x55f67d1afe00 con 0x55f66321a800 > -5641> 2018-07-11 08:05:45.356065 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:45.412720) v4 -- 0x55f66dcea200 con 0 > -5640> 2018-07-11 08:05:45.356054 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 240 0x55f6713c8280 replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:509/504,start:6:a84ba01d:::benchmark_data_magna108_35617_object992145:0,end:6:a84cb4df:::benchmark_data_magna108.ceph.redhat.com_2478_object16502:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -5639> 2018-07-11 08:05:45.356087 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 240 ==== replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:509/504,start:6:a84ba01d:::benchmark_data_magna108_35617_object992145:0,end:6:a84cb4df:::benchmark_data_magna108.ceph.redhat.com_2478_object16502:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 232+0+0 (3059874707 0 0) 0x55f6713c8280 con 0x55f66315e800 > -5638> 2018-07-11 08:05:45.356175 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12651 conn(0x55f663219000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25 cs=1 l=1). rx osd.2 seq 213 0x55f667d0ca00 osd_ping(ping e509 stamp 2018-07-11 08:05:45.412720) v4 > -5637> 2018-07-11 08:05:45.356201 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.2 10.8.128.106:0/12651 213 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:45.412720) v4 ==== 2004+0+0 (534926139 0 0) 0x55f667d0ca00 con 0x55f663219000 > -5636> 2018-07-11 08:05:45.356218 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:45.412720) v4 -- 0x55f66dcea600 con 0 > -5635> 2018-07-11 08:05:45.488647 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.107:0/12617 conn(0x55f64e3c5000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.9 seq 204 0x55f66dceb200 osd_ping(ping e509 stamp 2018-07-11 08:05:45.488292) v4 > -5634> 2018-07-11 08:05:45.488652 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.107:0/12617 conn(0x55f64e3c3800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.9 seq 204 0x55f66cb51c00 osd_ping(ping e509 stamp 2018-07-11 08:05:45.488292) v4 > -5633> 2018-07-11 08:05:45.488669 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.9 10.8.128.107:0/12617 204 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:45.488292) v4 ==== 2004+0+0 (1191945124 0 0) 0x55f66dceb200 con 0x55f64e3c5000 > -5632> 2018-07-11 08:05:45.488679 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:45.488292) v4 -- 0x55f66dcea600 con 0 > -5631> 2018-07-11 08:05:45.488674 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.9 10.8.128.107:0/12617 204 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:45.488292) v4 ==== 2004+0+0 (1191945124 0 0) 0x55f66cb51c00 con 0x55f64e3c3800 > -5630> 2018-07-11 08:05:45.488722 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:45.488292) v4 -- 0x55f67d193200 con 0 > -5629> 2018-07-11 08:05:45.529561 7f491d046700 1 -- 10.8.128.107:6800/14173 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55f663151080 con 0 > -5628> 2018-07-11 08:05:45.529629 7f491d046700 1 -- 10.8.128.107:6800/14173 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55f65e5d4840 con 0 > -5627> 2018-07-11 08:05:45.538357 7f491803c700 5 osd.10 509 heartbeat: osd_stat(789 GB used, 141 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,1]) > -5626> 2018-07-11 08:05:45.538370 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6802/15255 -- osd_ping(ping e509 stamp 2018-07-11 08:05:45.538369) v4 -- 0x55f6739cce00 con 0 > -5625> 2018-07-11 08:05:45.538392 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6803/15255 -- osd_ping(ping e509 stamp 2018-07-11 08:05:45.538369) v4 -- 0x55f6739cd000 con 0 > -5624> 2018-07-11 08:05:45.538405 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6806/12677 -- osd_ping(ping e509 stamp 2018-07-11 08:05:45.538369) v4 -- 0x55f6739cd200 con 0 > -5623> 2018-07-11 08:05:45.538422 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6807/12677 -- osd_ping(ping e509 stamp 2018-07-11 08:05:45.538369) v4 -- 0x55f6739cd400 con 0 > -5622> 2018-07-11 08:05:45.538435 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6808/12651 -- osd_ping(ping e509 stamp 2018-07-11 08:05:45.538369) v4 -- 0x55f6739cd600 con 0 > -5621> 2018-07-11 08:05:45.538446 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6810/12651 -- osd_ping(ping e509 stamp 2018-07-11 08:05:45.538369) v4 -- 0x55f6739cd800 con 0 > -5620> 2018-07-11 08:05:45.538458 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6809/12650 -- osd_ping(ping e509 stamp 2018-07-11 08:05:45.538369) v4 -- 0x55f6739cda00 con 0 > -5619> 2018-07-11 08:05:45.538476 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6811/12650 -- osd_ping(ping e509 stamp 2018-07-11 08:05:45.538369) v4 -- 0x55f6739cdc00 con 0 > -5618> 2018-07-11 08:05:45.538488 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6810/15287 -- osd_ping(ping e509 stamp 2018-07-11 08:05:45.538369) v4 -- 0x55f6739cde00 con 0 > -5617> 2018-07-11 08:05:45.538501 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6811/15287 -- osd_ping(ping e509 stamp 2018-07-11 08:05:45.538369) v4 -- 0x55f67dc4c000 con 0 > -5616> 2018-07-11 08:05:45.538517 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6806/15263 -- osd_ping(ping e509 stamp 2018-07-11 08:05:45.538369) v4 -- 0x55f67dc4c200 con 0 > -5615> 2018-07-11 08:05:45.538531 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6807/15263 -- osd_ping(ping e509 stamp 2018-07-11 08:05:45.538369) v4 -- 0x55f67dc4c400 con 0 > -5614> 2018-07-11 08:05:45.538544 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.115:6802/13323 -- osd_ping(ping e509 stamp 2018-07-11 08:05:45.538369) v4 -- 0x55f67dc4c600 con 0 > -5613> 2018-07-11 08:05:45.538557 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.115:6803/13323 -- osd_ping(ping e509 stamp 2018-07-11 08:05:45.538369) v4 -- 0x55f67dc4c800 con 0 > -5612> 2018-07-11 08:05:45.538569 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6806/15885 -- osd_ping(ping e509 stamp 2018-07-11 08:05:45.538369) v4 -- 0x55f67dc4ca00 con 0 > -5611> 2018-07-11 08:05:45.538581 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6807/15885 -- osd_ping(ping e509 stamp 2018-07-11 08:05:45.538369) v4 -- 0x55f67dc4cc00 con 0 > -5610> 2018-07-11 08:05:45.538592 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.108:6802/11967 -- osd_ping(ping e509 stamp 2018-07-11 08:05:45.538369) v4 -- 0x55f67dc4ce00 con 0 > -5609> 2018-07-11 08:05:45.538604 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.108:6803/11967 -- osd_ping(ping e509 stamp 2018-07-11 08:05:45.538369) v4 -- 0x55f67dc4d000 con 0 > -5608> 2018-07-11 08:05:45.538616 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.107:6806/12617 -- osd_ping(ping e509 stamp 2018-07-11 08:05:45.538369) v4 -- 0x55f67dc4d200 con 0 > -5607> 2018-07-11 08:05:45.538629 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.107:6807/12617 -- osd_ping(ping e509 stamp 2018-07-11 08:05:45.538369) v4 -- 0x55f67dc4d400 con 0 > -5606> 2018-07-11 08:05:45.538641 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6802/15919 -- osd_ping(ping e509 stamp 2018-07-11 08:05:45.538369) v4 -- 0x55f67dc4d600 con 0 > -5605> 2018-07-11 08:05:45.538653 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6803/15919 -- osd_ping(ping e509 stamp 2018-07-11 08:05:45.538369) v4 -- 0x55f67dc4d800 con 0 > -5604> 2018-07-11 08:05:45.538792 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.107:6806/12617 conn(0x55f656ade000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.9 seq 202 0x55f66cb51c00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:45.538369) v4 > -5603> 2018-07-11 08:05:45.538813 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.9 10.8.128.107:6806/12617 202 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:45.538369) v4 ==== 2004+0+0 (4074011720 0 0) 0x55f66cb51c00 con 0x55f656ade000 > -5602> 2018-07-11 08:05:45.538843 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6806/12677 conn(0x55f64e3d8000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.1 seq 202 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:45.538369) v4 > -5601> 2018-07-11 08:05:45.538852 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6803/15255 conn(0x55f656ad5800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 202 0x55f66cb51c00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:45.538369) v4 > -5600> 2018-07-11 08:05:45.538862 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.1 10.8.128.106:6806/12677 202 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:45.538369) v4 ==== 2004+0+0 (4074011720 0 0) 0x55f64e20fc00 con 0x55f64e3d8000 > -5599> 2018-07-11 08:05:45.538863 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.0 10.8.128.113:6803/15255 202 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:45.538369) v4 ==== 2004+0+0 (4074011720 0 0) 0x55f66cb51c00 con 0x55f656ad5800 > -5598> 2018-07-11 08:05:45.538855 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6802/15255 conn(0x55f656ad4000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 202 0x55f667d0ca00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:45.538369) v4 > -5597> 2018-07-11 08:05:45.538900 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.107:6807/12617 conn(0x55f656adf800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.9 seq 202 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:45.538369) v4 > -5596> 2018-07-11 08:05:45.538907 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.0 10.8.128.113:6802/15255 202 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:45.538369) v4 ==== 2004+0+0 (4074011720 0 0) 0x55f667d0ca00 con 0x55f656ad4000 > -5595> 2018-07-11 08:05:45.538917 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.9 10.8.128.107:6807/12617 202 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:45.538369) v4 ==== 2004+0+0 (4074011720 0 0) 0x55f64e20fc00 con 0x55f656adf800 > -5594> 2018-07-11 08:05:45.538922 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6810/12651 conn(0x55f64e3eb800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.2 seq 202 0x55f66cb51c00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:45.538369) v4 > -5593> 2018-07-11 08:05:45.538936 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.2 10.8.128.106:6810/12651 202 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:45.538369) v4 ==== 2004+0+0 (4074011720 0 0) 0x55f66cb51c00 con 0x55f64e3eb800 > -5592> 2018-07-11 08:05:45.538958 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6810/15287 conn(0x55f64e401000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.4 seq 202 0x55f667d0ca00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:45.538369) v4 > -5591> 2018-07-11 08:05:45.538977 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.4 10.8.128.113:6810/15287 202 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:45.538369) v4 ==== 2004+0+0 (4074011720 0 0) 0x55f667d0ca00 con 0x55f64e401000 > -5590> 2018-07-11 08:05:45.538979 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6806/15263 conn(0x55f64e3ed000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.5 seq 202 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:45.538369) v4 > -5589> 2018-07-11 08:05:45.538985 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6811/15287 conn(0x55f64e402800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.4 seq 202 0x55f66cb51c00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:45.538369) v4 > -5588> 2018-07-11 08:05:45.538994 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.5 10.8.128.113:6806/15263 202 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:45.538369) v4 ==== 2004+0+0 (4074011720 0 0) 0x55f64e20fc00 con 0x55f64e3ed000 > -5587> 2018-07-11 08:05:45.538999 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.4 10.8.128.113:6811/15287 202 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:45.538369) v4 ==== 2004+0+0 (4074011720 0 0) 0x55f66cb51c00 con 0x55f64e402800 > -5586> 2018-07-11 08:05:45.539027 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6807/12677 conn(0x55f64e3d9800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.1 seq 202 0x55f667d0ca00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:45.538369) v4 > -5585> 2018-07-11 08:05:45.539039 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6809/12650 conn(0x55f656ad7000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.3 seq 202 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:45.538369) v4 > -5584> 2018-07-11 08:05:45.539043 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6802/15919 conn(0x55f64e3db000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.11 seq 202 0x55f66cb51c00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:45.538369) v4 > -5583> 2018-07-11 08:05:45.539045 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.1 10.8.128.106:6807/12677 202 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:45.538369) v4 ==== 2004+0+0 (4074011720 0 0) 0x55f667d0ca00 con 0x55f64e3d9800 > -5582> 2018-07-11 08:05:45.539058 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.11 10.8.128.114:6802/15919 202 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:45.538369) v4 ==== 2004+0+0 (4074011720 0 0) 0x55f66cb51c00 con 0x55f64e3db000 > -5581> 2018-07-11 08:05:45.539056 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.3 10.8.128.106:6809/12650 202 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:45.538369) v4 ==== 2004+0+0 (4074011720 0 0) 0x55f64e20fc00 con 0x55f656ad7000 > -5580> 2018-07-11 08:05:45.539106 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.115:6803/13323 conn(0x55f656ace800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.6 seq 202 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:45.538369) v4 > -5579> 2018-07-11 08:05:45.539120 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.115:6802/13323 conn(0x55f656acd000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.6 seq 202 0x55f66cb51c00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:45.538369) v4 > -5578> 2018-07-11 08:05:45.539134 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.6 10.8.128.115:6802/13323 202 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:45.538369) v4 ==== 2004+0+0 (4074011720 0 0) 0x55f66cb51c00 con 0x55f656acd000 > -5577> 2018-07-11 08:05:45.539131 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6808/12651 conn(0x55f64e3ea000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.2 seq 202 0x55f667d0ca00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:45.538369) v4 > -5576> 2018-07-11 08:05:45.539120 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.6 10.8.128.115:6803/13323 202 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:45.538369) v4 ==== 2004+0+0 (4074011720 0 0) 0x55f64e20fc00 con 0x55f656ace800 > -5575> 2018-07-11 08:05:45.539152 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.2 10.8.128.106:6808/12651 202 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:45.538369) v4 ==== 2004+0+0 (4074011720 0 0) 0x55f667d0ca00 con 0x55f64e3ea000 > -5574> 2018-07-11 08:05:45.539171 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6806/15885 conn(0x55f64e3fe000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.7 seq 202 0x55f66cb51c00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:45.538369) v4 > -5573> 2018-07-11 08:05:45.539184 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.7 10.8.128.114:6806/15885 202 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:45.538369) v4 ==== 2004+0+0 (4074011720 0 0) 0x55f66cb51c00 con 0x55f64e3fe000 > -5572> 2018-07-11 08:05:45.539196 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6811/12650 conn(0x55f656ad8800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.3 seq 202 0x55f667d0ca00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:45.538369) v4 > -5571> 2018-07-11 08:05:45.539214 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.3 10.8.128.106:6811/12650 202 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:45.538369) v4 ==== 2004+0+0 (4074011720 0 0) 0x55f667d0ca00 con 0x55f656ad8800 > -5570> 2018-07-11 08:05:45.539217 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6807/15885 conn(0x55f64e3ff800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.7 seq 202 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:45.538369) v4 > -5569> 2018-07-11 08:05:45.539233 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.7 10.8.128.114:6807/15885 202 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:45.538369) v4 ==== 2004+0+0 (4074011720 0 0) 0x55f64e20fc00 con 0x55f64e3ff800 > -5568> 2018-07-11 08:05:45.539273 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6807/15263 conn(0x55f64e3ee800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.5 seq 202 0x55f667d0ca00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:45.538369) v4 > -5567> 2018-07-11 08:05:45.539286 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6803/15919 conn(0x55f64e3dc800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.11 seq 202 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:45.538369) v4 > -5566> 2018-07-11 08:05:45.539292 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.5 10.8.128.113:6807/15263 202 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:45.538369) v4 ==== 2004+0+0 (4074011720 0 0) 0x55f667d0ca00 con 0x55f64e3ee800 > -5565> 2018-07-11 08:05:45.539308 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.11 10.8.128.114:6803/15919 202 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:45.538369) v4 ==== 2004+0+0 (4074011720 0 0) 0x55f64e20fc00 con 0x55f64e3dc800 > -5564> 2018-07-11 08:05:45.539339 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.108:6802/11967 conn(0x55f656aca000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.8 seq 202 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:45.538369) v4 > -5563> 2018-07-11 08:05:45.539344 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.108:6803/11967 conn(0x55f656acb800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.8 seq 202 0x55f667d0ca00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:45.538369) v4 > -5562> 2018-07-11 08:05:45.539350 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.8 10.8.128.108:6802/11967 202 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:45.538369) v4 ==== 2004+0+0 (4074011720 0 0) 0x55f64e20fc00 con 0x55f656aca000 > -5561> 2018-07-11 08:05:45.539364 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.8 10.8.128.108:6803/11967 202 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:45.538369) v4 ==== 2004+0+0 (4074011720 0 0) 0x55f667d0ca00 con 0x55f656acb800 > -5560> 2018-07-11 08:05:45.968712 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 447 0x55f66cb51c00 MOSDScrubReserve(11.3s5 REQUEST e509) v1 > -5559> 2018-07-11 08:05:45.968744 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 447 ==== MOSDScrubReserve(11.3s5 REQUEST e509) v1 ==== 43+0+0 (433425070 0 0) 0x55f66cb51c00 con 0x55f668a29000 > -5558> 2018-07-11 08:05:45.968791 7f491a040700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e509) v1 -- 0x55f65c79d400 con 0 > -5557> 2018-07-11 08:05:45.969127 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 448 0x55f6739cd000 MOSDScrubReserve(11.3s5 RELEASE e509) v1 > -5556> 2018-07-11 08:05:45.969141 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 448 ==== MOSDScrubReserve(11.3s5 RELEASE e509) v1 ==== 43+0+0 (3603460268 0 0) 0x55f6739cd000 con 0x55f668a29000 > -5555> 2018-07-11 08:05:46.086280 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f655ad3680 con 0 > -5554> 2018-07-11 08:05:46.087782 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 241 0x55f6713c8500 replica scrub(pg: 6.15,from:0'0,to:461'106889,epoch:509/504,start:6:a84cb4df:::benchmark_data_magna108.ceph.redhat.com_2478_object16502:0,end:6:a84d922d:::benchmark_data_magna108_35617_object1065421:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -5553> 2018-07-11 08:05:46.087811 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 241 ==== replica scrub(pg: 6.15,from:0'0,to:461'106889,epoch:509/504,start:6:a84cb4df:::benchmark_data_magna108.ceph.redhat.com_2478_object16502:0,end:6:a84d922d:::benchmark_data_magna108_35617_object1065421:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (20828182 0 0) 0x55f6713c8500 con 0x55f66315e800 > -5552> 2018-07-11 08:05:46.458579 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0d000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 212 0x55f667d0ca00 osd_ping(ping e509 stamp 2018-07-11 08:05:47.703490) v4 > -5551> 2018-07-11 08:05:46.458571 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0e800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 212 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:05:47.703490) v4 > -5550> 2018-07-11 08:05:46.458607 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.5 10.8.128.113:0/15263 212 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:47.703490) v4 ==== 2004+0+0 (2170422598 0 0) 0x55f667d0ca00 con 0x55f663f0d000 > -5549> 2018-07-11 08:05:46.458610 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.5 10.8.128.113:0/15263 212 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:47.703490) v4 ==== 2004+0+0 (2170422598 0 0) 0x55f64e20fc00 con 0x55f663f0e800 > -5548> 2018-07-11 08:05:46.458625 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:47.703490) v4 -- 0x55f656b55600 con 0 > -5547> 2018-07-11 08:05:46.458655 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:47.703490) v4 -- 0x55f6739cd200 con 0 > -5546> 2018-07-11 08:05:46.867746 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f65a4a6240 con 0 > -5545> 2018-07-11 08:05:46.968943 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 449 0x55f67d193000 MOSDScrubReserve(11.3s5 REQUEST e509) v1 > -5544> 2018-07-11 08:05:46.968965 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 449 ==== MOSDScrubReserve(11.3s5 REQUEST e509) v1 ==== 43+0+0 (433425070 0 0) 0x55f67d193000 con 0x55f668a29000 > -5543> 2018-07-11 08:05:46.969036 7f491a040700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e509) v1 -- 0x55f6739cd000 con 0 > -5542> 2018-07-11 08:05:46.969460 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 450 0x55f6739cd000 MOSDScrubReserve(11.3s5 RELEASE e509) v1 > -5541> 2018-07-11 08:05:46.969479 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 450 ==== MOSDScrubReserve(11.3s5 RELEASE e509) v1 ==== 43+0+0 (3603460268 0 0) 0x55f6739cd000 con 0x55f668a29000 > -5540> 2018-07-11 08:05:47.001484 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.108:0/11967 conn(0x55f663f6f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=1). rx osd.8 seq 198 0x55f65c79d400 osd_ping(ping e509 stamp 2018-07-11 08:05:46.761486) v4 > -5539> 2018-07-11 08:05:47.001508 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.8 10.8.128.108:0/11967 198 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:46.761486) v4 ==== 2004+0+0 (1252758620 0 0) 0x55f65c79d400 con 0x55f663f6f000 > -5538> 2018-07-11 08:05:47.001519 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:46.761486) v4 -- 0x55f67d193600 con 0 > -5537> 2018-07-11 08:05:47.001560 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.108:0/11967 conn(0x55f663f70800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=1). rx osd.8 seq 198 0x55f667d0ca00 osd_ping(ping e509 stamp 2018-07-11 08:05:46.761486) v4 > -5536> 2018-07-11 08:05:47.001581 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.8 10.8.128.108:0/11967 198 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:46.761486) v4 ==== 2004+0+0 (1252758620 0 0) 0x55f667d0ca00 con 0x55f663f70800 > -5535> 2018-07-11 08:05:47.001592 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:46.761486) v4 -- 0x55f667d09e00 con 0 > -5534> 2018-07-11 08:05:47.019997 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 242 0x55f6713c8780 replica scrub(pg: 6.15,from:0'0,to:430'106873,epoch:509/504,start:6:a84d922d:::benchmark_data_magna108_35617_object1065421:0,end:6:a84e7121:::benchmark_data_magna108_35617_object60347:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -5533> 2018-07-11 08:05:47.020019 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 242 ==== replica scrub(pg: 6.15,from:0'0,to:430'106873,epoch:509/504,start:6:a84d922d:::benchmark_data_magna108_35617_object1065421:0,end:6:a84e7121:::benchmark_data_magna108_35617_object60347:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (4293271182 0 0) 0x55f6713c8780 con 0x55f66315e800 > -5532> 2018-07-11 08:05:47.406697 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0a000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 204 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:05:48.042403) v4 > -5531> 2018-07-11 08:05:47.406718 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.11 10.8.128.114:0/15919 204 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:48.042403) v4 ==== 2004+0+0 (2841570834 0 0) 0x55f67d1afe00 con 0x55f663f0a000 > -5530> 2018-07-11 08:05:47.406728 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:48.042403) v4 -- 0x55f6739cd200 con 0 > -5529> 2018-07-11 08:05:47.406731 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0b800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 204 0x55f67d193400 osd_ping(ping e509 stamp 2018-07-11 08:05:48.042403) v4 > -5528> 2018-07-11 08:05:47.406757 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.11 10.8.128.114:0/15919 204 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:48.042403) v4 ==== 2004+0+0 (2841570834 0 0) 0x55f67d193400 con 0x55f663f0b800 > -5527> 2018-07-11 08:05:47.406768 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:48.042403) v4 -- 0x55f67d193600 con 0 > -5526> 2018-07-11 08:05:47.432305 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12650 conn(0x55f663ed9000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.3 seq 203 0x55f65c79d400 osd_ping(ping e509 stamp 2018-07-11 08:05:47.489241) v4 > -5525> 2018-07-11 08:05:47.432331 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.3 10.8.128.106:0/12650 203 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:47.489241) v4 ==== 2004+0+0 (2286292230 0 0) 0x55f65c79d400 con 0x55f663ed9000 > -5524> 2018-07-11 08:05:47.432332 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12650 conn(0x55f663eda800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.3 seq 203 0x55f667d0ca00 osd_ping(ping e509 stamp 2018-07-11 08:05:47.489241) v4 > -5523> 2018-07-11 08:05:47.432347 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:47.489241) v4 -- 0x55f67d193800 con 0 > -5522> 2018-07-11 08:05:47.432352 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.3 10.8.128.106:0/12650 203 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:47.489241) v4 ==== 2004+0+0 (2286292230 0 0) 0x55f667d0ca00 con 0x55f663eda800 > -5521> 2018-07-11 08:05:47.432380 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:47.489241) v4 -- 0x55f667d08800 con 0 > -5520> 2018-07-11 08:05:47.478694 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15255 conn(0x55f663ec1000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.0 seq 198 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:05:48.723622) v4 > -5519> 2018-07-11 08:05:47.478718 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.0 10.8.128.113:0/15255 198 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:48.723622) v4 ==== 2004+0+0 (4293011071 0 0) 0x55f64e20fc00 con 0x55f663ec1000 > -5518> 2018-07-11 08:05:47.478735 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:48.723622) v4 -- 0x55f66dcea200 con 0 > -5517> 2018-07-11 08:05:47.478737 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15255 conn(0x55f663ec2800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.0 seq 198 0x55f67d193400 osd_ping(ping e509 stamp 2018-07-11 08:05:48.723622) v4 > -5516> 2018-07-11 08:05:47.478758 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.0 10.8.128.113:0/15255 198 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:48.723622) v4 ==== 2004+0+0 (4293011071 0 0) 0x55f67d193400 con 0x55f663ec2800 > -5515> 2018-07-11 08:05:47.478769 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:48.723622) v4 -- 0x55f67d193a00 con 0 > -5514> 2018-07-11 08:05:47.656520 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12651 conn(0x55f663219000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25 cs=1 l=1). rx osd.2 seq 214 0x55f667d09e00 osd_ping(ping e509 stamp 2018-07-11 08:05:47.713352) v4 > -5513> 2018-07-11 08:05:47.656541 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.2 10.8.128.106:0/12651 214 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:47.713352) v4 ==== 2004+0+0 (3636852714 0 0) 0x55f667d09e00 con 0x55f663219000 > -5512> 2018-07-11 08:05:47.656558 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:47.713352) v4 -- 0x55f66dc71e00 con 0 > -5511> 2018-07-11 08:05:47.656562 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12651 conn(0x55f66321a800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25 cs=1 l=1). rx osd.2 seq 214 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:05:47.713352) v4 > -5510> 2018-07-11 08:05:47.656579 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.2 10.8.128.106:0/12651 214 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:47.713352) v4 ==== 2004+0+0 (3636852714 0 0) 0x55f67d1afe00 con 0x55f66321a800 > -5509> 2018-07-11 08:05:47.656591 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:47.713352) v4 -- 0x55f64e25c000 con 0 > -5508> 2018-07-11 08:05:47.711923 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.115:0/13323 conn(0x55f663ebf800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.6 seq 193 0x55f667d0ca00 osd_ping(ping e509 stamp 2018-07-11 08:05:48.064666) v4 > -5507> 2018-07-11 08:05:47.711944 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.6 10.8.128.115:0/13323 193 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:48.064666) v4 ==== 2004+0+0 (1028154510 0 0) 0x55f667d0ca00 con 0x55f663ebf800 > -5506> 2018-07-11 08:05:47.711960 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:48.064666) v4 -- 0x55f66dc71e00 con 0 > -5505> 2018-07-11 08:05:47.711981 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.115:0/13323 conn(0x55f663ebe000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.6 seq 193 0x55f65c79d400 osd_ping(ping e509 stamp 2018-07-11 08:05:48.064666) v4 > -5504> 2018-07-11 08:05:47.712002 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.6 10.8.128.115:0/13323 193 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:48.064666) v4 ==== 2004+0+0 (1028154510 0 0) 0x55f65c79d400 con 0x55f663ebe000 > -5503> 2018-07-11 08:05:47.712014 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:48.064666) v4 -- 0x55f67d193a00 con 0 > -5502> 2018-07-11 08:05:47.932740 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12650 conn(0x55f663ed9000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.3 seq 204 0x55f67d193400 osd_ping(ping e509 stamp 2018-07-11 08:05:47.989697) v4 > -5501> 2018-07-11 08:05:47.932765 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.3 10.8.128.106:0/12650 204 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:47.989697) v4 ==== 2004+0+0 (1644895950 0 0) 0x55f67d193400 con 0x55f663ed9000 > -5500> 2018-07-11 08:05:47.932762 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12650 conn(0x55f663eda800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.3 seq 204 0x55f667d09e00 osd_ping(ping e509 stamp 2018-07-11 08:05:47.989697) v4 > -5499> 2018-07-11 08:05:47.932783 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:47.989697) v4 -- 0x55f67d193c00 con 0 > -5498> 2018-07-11 08:05:47.932787 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.3 10.8.128.106:0/12650 204 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:47.989697) v4 ==== 2004+0+0 (1644895950 0 0) 0x55f667d09e00 con 0x55f663eda800 > -5497> 2018-07-11 08:05:47.932817 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:47.989697) v4 -- 0x55f66dc71800 con 0 > -5496> 2018-07-11 08:05:47.969129 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 451 0x55f65c79d400 MOSDScrubReserve(11.3s5 REQUEST e509) v1 > -5495> 2018-07-11 08:05:47.969148 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 451 ==== MOSDScrubReserve(11.3s5 REQUEST e509) v1 ==== 43+0+0 (433425070 0 0) 0x55f65c79d400 con 0x55f668a29000 > -5494> 2018-07-11 08:05:47.969224 7f491a040700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e509) v1 -- 0x55f6739cd000 con 0 > -5493> 2018-07-11 08:05:47.969646 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 452 0x55f6739cd000 MOSDScrubReserve(11.3s5 RELEASE e509) v1 > -5492> 2018-07-11 08:05:47.969675 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 452 ==== MOSDScrubReserve(11.3s5 RELEASE e509) v1 ==== 43+0+0 (3603460268 0 0) 0x55f6739cd000 con 0x55f668a29000 > -5491> 2018-07-11 08:05:47.980094 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 30] Generated table #37044: 2638241 keys, 74546713 bytes > -5490> 2018-07-11 08:05:47.980142 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296347980126, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 37044, "file_size": 74546713, "table_properties": {"data_size": 67112134, "index_size": 690406, "filter_size": 6743183, "raw_key_size": 76215836, "raw_average_key_size": 28, "raw_value_size": 31365739, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} > -5489> 2018-07-11 08:05:48.278639 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed6000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 205 0x55f667d0ca00 osd_ping(ping e509 stamp 2018-07-11 08:05:48.335636) v4 > -5488> 2018-07-11 08:05:48.278642 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed7800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 205 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:05:48.335636) v4 > -5487> 2018-07-11 08:05:48.278666 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.1 10.8.128.106:0/12677 205 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:48.335636) v4 ==== 2004+0+0 (3953616481 0 0) 0x55f667d0ca00 con 0x55f663ed6000 > -5486> 2018-07-11 08:05:48.278670 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.1 10.8.128.106:0/12677 205 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:48.335636) v4 ==== 2004+0+0 (3953616481 0 0) 0x55f64e20fc00 con 0x55f663ed7800 > -5485> 2018-07-11 08:05:48.278677 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:48.335636) v4 -- 0x55f64cf24600 con 0 > -5484> 2018-07-11 08:05:48.278701 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:48.335636) v4 -- 0x55f66dc71400 con 0 > -5483> 2018-07-11 08:05:48.545333 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f65c9f3200 con 0 > -5482> 2018-07-11 08:05:48.547667 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 243 0x55f6713c8a00 replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:509/504,start:6:a84e7121:::benchmark_data_magna108_35617_object60347:0,end:6:a84f5d01:::benchmark_data_magna108.ceph.redhat.com_2478_object94862:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -5481> 2018-07-11 08:05:48.547697 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 243 ==== replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:509/504,start:6:a84e7121:::benchmark_data_magna108_35617_object60347:0,end:6:a84f5d01:::benchmark_data_magna108.ceph.redhat.com_2478_object94862:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 231+0+0 (59956566 0 0) 0x55f6713c8a00 con 0x55f66315e800 > -5480> 2018-07-11 08:05:49.443551 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f64f53c240 con 0 > -5479> 2018-07-11 08:05:49.445201 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 244 0x55f6713c8c80 replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:509/504,start:6:a84f5d01:::benchmark_data_magna108.ceph.redhat.com_2478_object94862:0,end:6:a8502265:::benchmark_data_magna108.ceph.redhat.com_2478_object46673:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -5478> 2018-07-11 08:05:49.445235 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 244 ==== replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:509/504,start:6:a84f5d01:::benchmark_data_magna108.ceph.redhat.com_2478_object94862:0,end:6:a8502265:::benchmark_data_magna108.ceph.redhat.com_2478_object46673:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 246+0+0 (3167946153 0 0) 0x55f6713c8c80 con 0x55f66315e800 > -5477> 2018-07-11 08:05:49.736060 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15885 conn(0x55f663217800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 208 0x55f67d193400 osd_ping(ping e509 stamp 2018-07-11 08:05:50.371793) v4 > -5476> 2018-07-11 08:05:49.736070 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15885 conn(0x55f663216000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 208 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:05:50.371793) v4 > -5475> 2018-07-11 08:05:49.736094 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.7 10.8.128.114:0/15885 208 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:50.371793) v4 ==== 2004+0+0 (3163613333 0 0) 0x55f67d193400 con 0x55f663217800 > -5474> 2018-07-11 08:05:49.736098 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.7 10.8.128.114:0/15885 208 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:50.371793) v4 ==== 2004+0+0 (3163613333 0 0) 0x55f67d1afe00 con 0x55f663216000 > -5473> 2018-07-11 08:05:49.736112 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:50.371793) v4 -- 0x55f65fa96200 con 0 > -5472> 2018-07-11 08:05:49.736146 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:50.371793) v4 -- 0x55f66dc71400 con 0 > -5471> 2018-07-11 08:05:49.969512 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 453 0x55f65fa96000 MOSDScrubReserve(11.3s5 REQUEST e509) v1 > -5470> 2018-07-11 08:05:49.969537 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 453 ==== MOSDScrubReserve(11.3s5 REQUEST e509) v1 ==== 43+0+0 (433425070 0 0) 0x55f65fa96000 con 0x55f668a29000 > -5469> 2018-07-11 08:05:49.969581 7f491a040700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e509) v1 -- 0x55f6739cd000 con 0 > -5468> 2018-07-11 08:05:49.969918 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 454 0x55f65fa96200 MOSDScrubReserve(11.3s5 RELEASE e509) v1 > -5467> 2018-07-11 08:05:49.969949 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 454 ==== MOSDScrubReserve(11.3s5 RELEASE e509) v1 ==== 43+0+0 (3603460268 0 0) 0x55f65fa96200 con 0x55f668a29000 > -5466> 2018-07-11 08:05:50.104154 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15287 conn(0x55f66dce2800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.4 seq 209 0x55f667d09e00 osd_ping(ping e509 stamp 2018-07-11 08:05:51.349112) v4 > -5465> 2018-07-11 08:05:50.104171 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15287 conn(0x55f66dce1000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.4 seq 209 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:05:51.349112) v4 > -5464> 2018-07-11 08:05:50.104180 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.4 10.8.128.113:0/15287 209 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:51.349112) v4 ==== 2004+0+0 (1206279201 0 0) 0x55f667d09e00 con 0x55f66dce2800 > -5463> 2018-07-11 08:05:50.104189 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.4 10.8.128.113:0/15287 209 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:51.349112) v4 ==== 2004+0+0 (1206279201 0 0) 0x55f64e20fc00 con 0x55f66dce1000 > -5462> 2018-07-11 08:05:50.104197 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:51.349112) v4 -- 0x55f64cf25c00 con 0 > -5461> 2018-07-11 08:05:50.104228 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:51.349112) v4 -- 0x55f66dc71400 con 0 > -5460> 2018-07-11 08:05:50.137853 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f65bb4b680 con 0 > -5459> 2018-07-11 08:05:50.139428 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 245 0x55f6713c8f00 replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:509/504,start:6:a8502265:::benchmark_data_magna108.ceph.redhat.com_2478_object46673:0,end:6:a8513ca9:::benchmark_data_magna108.ceph.redhat.com_2478_object251216:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -5458> 2018-07-11 08:05:50.139451 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 245 ==== replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:509/504,start:6:a8502265:::benchmark_data_magna108.ceph.redhat.com_2478_object46673:0,end:6:a8513ca9:::benchmark_data_magna108.ceph.redhat.com_2478_object251216:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 247+0+0 (2469300628 0 0) 0x55f6713c8f00 con 0x55f66315e800 > -5457> 2018-07-11 08:05:50.233114 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12650 conn(0x55f663ed9000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.3 seq 205 0x55f67d193400 osd_ping(ping e509 stamp 2018-07-11 08:05:50.290130) v4 > -5456> 2018-07-11 08:05:50.233135 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12650 conn(0x55f663eda800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.3 seq 205 0x55f64cf24600 osd_ping(ping e509 stamp 2018-07-11 08:05:50.290130) v4 > -5455> 2018-07-11 08:05:50.233146 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.3 10.8.128.106:0/12650 205 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:50.290130) v4 ==== 2004+0+0 (2962452177 0 0) 0x55f67d193400 con 0x55f663ed9000 > -5454> 2018-07-11 08:05:50.233153 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.3 10.8.128.106:0/12650 205 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:50.290130) v4 ==== 2004+0+0 (2962452177 0 0) 0x55f64cf24600 con 0x55f663eda800 > -5453> 2018-07-11 08:05:50.233161 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:50.290130) v4 -- 0x55f65fa96400 con 0 > -5452> 2018-07-11 08:05:50.233191 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:50.290130) v4 -- 0x55f64cf25c00 con 0 > -5451> 2018-07-11 08:05:50.433838 7f491e048700 10 monclient: tick > -5450> 2018-07-11 08:05:50.433853 7f491e048700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2018-07-11 08:05:20.433852) > -5449> 2018-07-11 08:05:50.529903 7f491d046700 1 -- 10.8.128.107:6800/14173 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55f667d76dc0 con 0 > -5448> 2018-07-11 08:05:50.529970 7f491d046700 1 -- 10.8.128.107:6800/14173 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55f6631ce840 con 0 > -5447> 2018-07-11 08:05:50.557009 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12651 conn(0x55f663219000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25 cs=1 l=1). rx osd.2 seq 215 0x55f64cf24600 osd_ping(ping e509 stamp 2018-07-11 08:05:50.613977) v4 > -5446> 2018-07-11 08:05:50.557031 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.2 10.8.128.106:0/12651 215 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:50.613977) v4 ==== 2004+0+0 (1029125380 0 0) 0x55f64cf24600 con 0x55f663219000 > -5445> 2018-07-11 08:05:50.557020 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12651 conn(0x55f66321a800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25 cs=1 l=1). rx osd.2 seq 215 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:05:50.613977) v4 > -5444> 2018-07-11 08:05:50.557040 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:50.613977) v4 -- 0x55f64cf25c00 con 0 > -5443> 2018-07-11 08:05:50.557043 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.2 10.8.128.106:0/12651 215 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:50.613977) v4 ==== 2004+0+0 (1029125380 0 0) 0x55f67d1afe00 con 0x55f66321a800 > -5442> 2018-07-11 08:05:50.557058 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:50.613977) v4 -- 0x55f66dc71400 con 0 > -5441> 2018-07-11 08:05:50.558885 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0e800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 213 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:05:51.803902) v4 > -5440> 2018-07-11 08:05:50.558907 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.5 10.8.128.113:0/15263 213 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:51.803902) v4 ==== 2004+0+0 (2112483518 0 0) 0x55f64e20fc00 con 0x55f663f0e800 > -5439> 2018-07-11 08:05:50.558920 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:51.803902) v4 -- 0x55f66dc71400 con 0 > -5438> 2018-07-11 08:05:50.558923 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0d000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 213 0x55f667d09e00 osd_ping(ping e509 stamp 2018-07-11 08:05:51.803902) v4 > -5437> 2018-07-11 08:05:50.558948 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.5 10.8.128.113:0/15263 213 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:51.803902) v4 ==== 2004+0+0 (2112483518 0 0) 0x55f667d09e00 con 0x55f663f0d000 > -5436> 2018-07-11 08:05:50.558960 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:51.803902) v4 -- 0x55f64cf25c00 con 0 > -5435> 2018-07-11 08:05:50.665183 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.114:6805/15885 conn(0x55f66315a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.7 seq 2559 0x55f64cf24600 MOSDScrubReserve(5.4 REQUEST e509) v1 > -5434> 2018-07-11 08:05:50.665202 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.7 10.8.128.114:6805/15885 2559 ==== MOSDScrubReserve(5.4 REQUEST e509) v1 ==== 43+0+0 (3003630469 0 0) 0x55f64cf24600 con 0x55f66315a000 > -5433> 2018-07-11 08:05:50.665252 7f491983f700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.114:6805/15885 -- MOSDScrubReserve(5.4 REJECT e509) v1 -- 0x55f666182e00 con 0 > -5432> 2018-07-11 08:05:50.665557 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.114:6805/15885 conn(0x55f66315a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.7 seq 2560 0x55f666182e00 MOSDScrubReserve(5.4 RELEASE e509) v1 > -5431> 2018-07-11 08:05:50.665571 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.7 10.8.128.114:6805/15885 2560 ==== MOSDScrubReserve(5.4 RELEASE e509) v1 ==== 43+0+0 (2082093447 0 0) 0x55f666182e00 con 0x55f66315a000 > -5430> 2018-07-11 08:05:50.789319 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.107:0/12617 conn(0x55f64e3c5000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.9 seq 205 0x55f667d09e00 osd_ping(ping e509 stamp 2018-07-11 08:05:50.788781) v4 > -5429> 2018-07-11 08:05:50.789345 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.9 10.8.128.107:0/12617 205 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:50.788781) v4 ==== 2004+0+0 (3042400037 0 0) 0x55f667d09e00 con 0x55f64e3c5000 > -5428> 2018-07-11 08:05:50.789386 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:50.788781) v4 -- 0x55f663c0ac00 con 0 > -5427> 2018-07-11 08:05:50.789485 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.107:0/12617 conn(0x55f64e3c3800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.9 seq 205 0x55f6739cd000 osd_ping(ping e509 stamp 2018-07-11 08:05:50.788781) v4 > -5426> 2018-07-11 08:05:50.789505 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.9 10.8.128.107:0/12617 205 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:50.788781) v4 ==== 2004+0+0 (3042400037 0 0) 0x55f6739cd000 con 0x55f64e3c3800 > -5425> 2018-07-11 08:05:50.789523 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:50.788781) v4 -- 0x55f65fa96600 con 0 > -5424> 2018-07-11 08:05:50.838764 7f491803c700 5 osd.10 509 heartbeat: osd_stat(789 GB used, 141 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,0,0,1]) > -5423> 2018-07-11 08:05:50.838779 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6802/15255 -- osd_ping(ping e509 stamp 2018-07-11 08:05:50.838777) v4 -- 0x55f67dc4da00 con 0 > -5422> 2018-07-11 08:05:50.838802 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6803/15255 -- osd_ping(ping e509 stamp 2018-07-11 08:05:50.838777) v4 -- 0x55f67dc4dc00 con 0 > -5421> 2018-07-11 08:05:50.838819 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6806/12677 -- osd_ping(ping e509 stamp 2018-07-11 08:05:50.838777) v4 -- 0x55f67dc4de00 con 0 > -5420> 2018-07-11 08:05:50.838836 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6807/12677 -- osd_ping(ping e509 stamp 2018-07-11 08:05:50.838777) v4 -- 0x55f652954000 con 0 > -5419> 2018-07-11 08:05:50.838849 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6808/12651 -- osd_ping(ping e509 stamp 2018-07-11 08:05:50.838777) v4 -- 0x55f652954200 con 0 > -5418> 2018-07-11 08:05:50.838859 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6810/12651 -- osd_ping(ping e509 stamp 2018-07-11 08:05:50.838777) v4 -- 0x55f652954400 con 0 > -5417> 2018-07-11 08:05:50.838871 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6809/12650 -- osd_ping(ping e509 stamp 2018-07-11 08:05:50.838777) v4 -- 0x55f652954600 con 0 > -5416> 2018-07-11 08:05:50.838883 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6811/12650 -- osd_ping(ping e509 stamp 2018-07-11 08:05:50.838777) v4 -- 0x55f652954800 con 0 > -5415> 2018-07-11 08:05:50.838902 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6810/15287 -- osd_ping(ping e509 stamp 2018-07-11 08:05:50.838777) v4 -- 0x55f652954a00 con 0 > -5414> 2018-07-11 08:05:50.838914 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6811/15287 -- osd_ping(ping e509 stamp 2018-07-11 08:05:50.838777) v4 -- 0x55f652954c00 con 0 > -5413> 2018-07-11 08:05:50.838927 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6806/15263 -- osd_ping(ping e509 stamp 2018-07-11 08:05:50.838777) v4 -- 0x55f652954e00 con 0 > -5412> 2018-07-11 08:05:50.838942 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6807/15263 -- osd_ping(ping e509 stamp 2018-07-11 08:05:50.838777) v4 -- 0x55f652955000 con 0 > -5411> 2018-07-11 08:05:50.838955 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.115:6802/13323 -- osd_ping(ping e509 stamp 2018-07-11 08:05:50.838777) v4 -- 0x55f652955200 con 0 > -5410> 2018-07-11 08:05:50.838970 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.115:6803/13323 -- osd_ping(ping e509 stamp 2018-07-11 08:05:50.838777) v4 -- 0x55f652955400 con 0 > -5409> 2018-07-11 08:05:50.838983 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6806/15885 -- osd_ping(ping e509 stamp 2018-07-11 08:05:50.838777) v4 -- 0x55f652955600 con 0 > -5408> 2018-07-11 08:05:50.838995 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6807/15885 -- osd_ping(ping e509 stamp 2018-07-11 08:05:50.838777) v4 -- 0x55f652955800 con 0 > -5407> 2018-07-11 08:05:50.839007 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.108:6802/11967 -- osd_ping(ping e509 stamp 2018-07-11 08:05:50.838777) v4 -- 0x55f652955a00 con 0 > -5406> 2018-07-11 08:05:50.839017 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.108:6803/11967 -- osd_ping(ping e509 stamp 2018-07-11 08:05:50.838777) v4 -- 0x55f652955c00 con 0 > -5405> 2018-07-11 08:05:50.839029 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.107:6806/12617 -- osd_ping(ping e509 stamp 2018-07-11 08:05:50.838777) v4 -- 0x55f652955e00 con 0 > -5404> 2018-07-11 08:05:50.839042 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.107:6807/12617 -- osd_ping(ping e509 stamp 2018-07-11 08:05:50.838777) v4 -- 0x55f64fcd4000 con 0 > -5403> 2018-07-11 08:05:50.839054 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6802/15919 -- osd_ping(ping e509 stamp 2018-07-11 08:05:50.838777) v4 -- 0x55f64fcd4200 con 0 > -5402> 2018-07-11 08:05:50.839067 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6803/15919 -- osd_ping(ping e509 stamp 2018-07-11 08:05:50.838777) v4 -- 0x55f64fcd4400 con 0 > -5401> 2018-07-11 08:05:50.839203 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6803/15255 conn(0x55f656ad5800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 203 0x55f6739cd000 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:50.838777) v4 > -5400> 2018-07-11 08:05:50.839223 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.0 10.8.128.113:6803/15255 203 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:50.838777) v4 ==== 2004+0+0 (3494683956 0 0) 0x55f6739cd000 con 0x55f656ad5800 > -5399> 2018-07-11 08:05:50.839214 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6802/15255 conn(0x55f656ad4000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 203 0x55f663c0b600 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:50.838777) v4 > -5398> 2018-07-11 08:05:50.839235 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.0 10.8.128.113:6802/15255 203 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:50.838777) v4 ==== 2004+0+0 (3494683956 0 0) 0x55f663c0b600 con 0x55f656ad4000 > -5397> 2018-07-11 08:05:50.839257 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.107:6806/12617 conn(0x55f656ade000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.9 seq 203 0x55f6739cd000 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:50.838777) v4 > -5396> 2018-07-11 08:05:50.839269 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.9 10.8.128.107:6806/12617 203 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:50.838777) v4 ==== 2004+0+0 (3494683956 0 0) 0x55f6739cd000 con 0x55f656ade000 > -5395> 2018-07-11 08:05:50.839292 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6806/12677 conn(0x55f64e3d8000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.1 seq 203 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:50.838777) v4 > -5394> 2018-07-11 08:05:50.839314 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.1 10.8.128.106:6806/12677 203 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:50.838777) v4 ==== 2004+0+0 (3494683956 0 0) 0x55f64e20fc00 con 0x55f64e3d8000 > -5393> 2018-07-11 08:05:50.839324 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6810/12651 conn(0x55f64e3eb800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.2 seq 203 0x55f6739cd000 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:50.838777) v4 > -5392> 2018-07-11 08:05:50.839347 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.107:6807/12617 conn(0x55f656adf800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.9 seq 203 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:50.838777) v4 > -5391> 2018-07-11 08:05:50.839339 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6807/15263 conn(0x55f64e3ee800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.5 seq 203 0x55f663c0b600 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:50.838777) v4 > -5390> 2018-07-11 08:05:50.839360 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.9 10.8.128.107:6807/12617 203 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:50.838777) v4 ==== 2004+0+0 (3494683956 0 0) 0x55f64e20fc00 con 0x55f656adf800 > -5389> 2018-07-11 08:05:50.839338 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.2 10.8.128.106:6810/12651 203 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:50.838777) v4 ==== 2004+0+0 (3494683956 0 0) 0x55f6739cd000 con 0x55f64e3eb800 > -5388> 2018-07-11 08:05:50.839363 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.5 10.8.128.113:6807/15263 203 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:50.838777) v4 ==== 2004+0+0 (3494683956 0 0) 0x55f663c0b600 con 0x55f64e3ee800 > -5387> 2018-07-11 08:05:50.839411 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6809/12650 conn(0x55f656ad7000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.3 seq 203 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:50.838777) v4 > -5386> 2018-07-11 08:05:50.839425 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.3 10.8.128.106:6809/12650 203 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:50.838777) v4 ==== 2004+0+0 (3494683956 0 0) 0x55f64e20fc00 con 0x55f656ad7000 > -5385> 2018-07-11 08:05:50.839433 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6811/15287 conn(0x55f64e402800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.4 seq 203 0x55f6739cd000 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:50.838777) v4 > -5384> 2018-07-11 08:05:50.839452 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.4 10.8.128.113:6811/15287 203 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:50.838777) v4 ==== 2004+0+0 (3494683956 0 0) 0x55f6739cd000 con 0x55f64e402800 > -5383> 2018-07-11 08:05:50.839461 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.115:6803/13323 conn(0x55f656ace800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.6 seq 203 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:50.838777) v4 > -5382> 2018-07-11 08:05:50.839464 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6811/12650 conn(0x55f656ad8800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.3 seq 203 0x55f663c0b600 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:50.838777) v4 > -5381> 2018-07-11 08:05:50.839475 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.6 10.8.128.115:6803/13323 203 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:50.838777) v4 ==== 2004+0+0 (3494683956 0 0) 0x55f64e20fc00 con 0x55f656ace800 > -5380> 2018-07-11 08:05:50.839489 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.3 10.8.128.106:6811/12650 203 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:50.838777) v4 ==== 2004+0+0 (3494683956 0 0) 0x55f663c0b600 con 0x55f656ad8800 > -5379> 2018-07-11 08:05:50.839519 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6806/15263 conn(0x55f64e3ed000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.5 seq 203 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:50.838777) v4 > -5378> 2018-07-11 08:05:50.839533 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.5 10.8.128.113:6806/15263 203 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:50.838777) v4 ==== 2004+0+0 (3494683956 0 0) 0x55f64e20fc00 con 0x55f64e3ed000 > -5377> 2018-07-11 08:05:50.839561 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6802/15919 conn(0x55f64e3db000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.11 seq 203 0x55f6739cd000 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:50.838777) v4 > -5376> 2018-07-11 08:05:50.839576 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.11 10.8.128.114:6802/15919 203 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:50.838777) v4 ==== 2004+0+0 (3494683956 0 0) 0x55f6739cd000 con 0x55f64e3db000 > -5375> 2018-07-11 08:05:50.839559 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6810/15287 conn(0x55f64e401000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.4 seq 203 0x55f663c0b600 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:50.838777) v4 > -5374> 2018-07-11 08:05:50.839599 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.4 10.8.128.113:6810/15287 203 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:50.838777) v4 ==== 2004+0+0 (3494683956 0 0) 0x55f663c0b600 con 0x55f64e401000 > -5373> 2018-07-11 08:05:50.839617 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.108:6802/11967 conn(0x55f656aca000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.8 seq 203 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:50.838777) v4 > -5372> 2018-07-11 08:05:50.839624 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.115:6802/13323 conn(0x55f656acd000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.6 seq 203 0x55f6739cd000 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:50.838777) v4 > -5371> 2018-07-11 08:05:50.839632 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6807/12677 conn(0x55f64e3d9800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.1 seq 203 0x55f663c0b600 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:50.838777) v4 > -5370> 2018-07-11 08:05:50.839632 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.8 10.8.128.108:6802/11967 203 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:50.838777) v4 ==== 2004+0+0 (3494683956 0 0) 0x55f64e20fc00 con 0x55f656aca000 > -5369> 2018-07-11 08:05:50.839644 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.6 10.8.128.115:6802/13323 203 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:50.838777) v4 ==== 2004+0+0 (3494683956 0 0) 0x55f6739cd000 con 0x55f656acd000 > -5368> 2018-07-11 08:05:50.839652 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.1 10.8.128.106:6807/12677 203 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:50.838777) v4 ==== 2004+0+0 (3494683956 0 0) 0x55f663c0b600 con 0x55f64e3d9800 > -5367> 2018-07-11 08:05:50.839684 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6803/15919 conn(0x55f64e3dc800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.11 seq 203 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:50.838777) v4 > -5366> 2018-07-11 08:05:50.839694 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6806/15885 conn(0x55f64e3fe000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.7 seq 203 0x55f6739cd000 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:50.838777) v4 > -5365> 2018-07-11 08:05:50.839702 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.11 10.8.128.114:6803/15919 203 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:50.838777) v4 ==== 2004+0+0 (3494683956 0 0) 0x55f64e20fc00 con 0x55f64e3dc800 > -5364> 2018-07-11 08:05:50.839706 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6808/12651 conn(0x55f64e3ea000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.2 seq 203 0x55f663c0b600 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:50.838777) v4 > -5363> 2018-07-11 08:05:50.839725 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.2 10.8.128.106:6808/12651 203 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:50.838777) v4 ==== 2004+0+0 (3494683956 0 0) 0x55f663c0b600 con 0x55f64e3ea000 > -5362> 2018-07-11 08:05:50.839731 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6807/15885 conn(0x55f64e3ff800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.7 seq 203 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:50.838777) v4 > -5361> 2018-07-11 08:05:50.839708 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.7 10.8.128.114:6806/15885 203 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:50.838777) v4 ==== 2004+0+0 (3494683956 0 0) 0x55f6739cd000 con 0x55f64e3fe000 > -5360> 2018-07-11 08:05:50.839740 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.7 10.8.128.114:6807/15885 203 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:50.838777) v4 ==== 2004+0+0 (3494683956 0 0) 0x55f64e20fc00 con 0x55f64e3ff800 > -5359> 2018-07-11 08:05:50.839775 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.108:6803/11967 conn(0x55f656acb800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.8 seq 203 0x55f663c0b600 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:50.838777) v4 > -5358> 2018-07-11 08:05:50.839793 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.8 10.8.128.108:6803/11967 203 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:50.838777) v4 ==== 2004+0+0 (3494683956 0 0) 0x55f663c0b600 con 0x55f656acb800 > -5357> 2018-07-11 08:05:50.868496 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f6528a3d40 con 0 > -5356> 2018-07-11 08:05:50.869877 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 246 0x55f6713c9400 replica scrub(pg: 6.15,from:0'0,to:430'106339,epoch:509/504,start:6:a8513ca9:::benchmark_data_magna108.ceph.redhat.com_2478_object251216:0,end:6:a85230fa:::benchmark_data_magna108_35617_object509499:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -5355> 2018-07-11 08:05:50.869906 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 246 ==== replica scrub(pg: 6.15,from:0'0,to:430'106339,epoch:509/504,start:6:a8513ca9:::benchmark_data_magna108.ceph.redhat.com_2478_object251216:0,end:6:a85230fa:::benchmark_data_magna108_35617_object509499:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (3996367359 0 0) 0x55f6713c9400 con 0x55f66315e800 > -5354> 2018-07-11 08:05:51.057586 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12651 conn(0x55f663219000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25 cs=1 l=1). rx osd.2 seq 216 0x55f663c0b600 osd_ping(ping e509 stamp 2018-07-11 08:05:51.114464) v4 > -5353> 2018-07-11 08:05:51.057616 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.2 10.8.128.106:0/12651 216 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:51.114464) v4 ==== 2004+0+0 (4152901786 0 0) 0x55f663c0b600 con 0x55f663219000 > -5352> 2018-07-11 08:05:51.057630 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:51.114464) v4 -- 0x55f67dc4da00 con 0 > -5351> 2018-07-11 08:05:51.057680 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12651 conn(0x55f66321a800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25 cs=1 l=1). rx osd.2 seq 216 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:05:51.114464) v4 > -5350> 2018-07-11 08:05:51.057699 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.2 10.8.128.106:0/12651 216 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:51.114464) v4 ==== 2004+0+0 (4152901786 0 0) 0x55f64e20fc00 con 0x55f66321a800 > -5349> 2018-07-11 08:05:51.057710 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:51.114464) v4 -- 0x55f652954600 con 0 > -5348> 2018-07-11 08:05:51.101886 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.108:0/11967 conn(0x55f663f6f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=1). rx osd.8 seq 199 0x55f6739cd000 osd_ping(ping e509 stamp 2018-07-11 08:05:50.861928) v4 > -5347> 2018-07-11 08:05:51.101907 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.8 10.8.128.108:0/11967 199 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:50.861928) v4 ==== 2004+0+0 (490619662 0 0) 0x55f6739cd000 con 0x55f663f6f000 > -5346> 2018-07-11 08:05:51.101916 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:50.861928) v4 -- 0x55f67dc4dc00 con 0 > -5345> 2018-07-11 08:05:51.101941 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.108:0/11967 conn(0x55f663f70800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=1). rx osd.8 seq 199 0x55f667d09e00 osd_ping(ping e509 stamp 2018-07-11 08:05:50.861928) v4 > -5344> 2018-07-11 08:05:51.101961 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.8 10.8.128.108:0/11967 199 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:50.861928) v4 ==== 2004+0+0 (490619662 0 0) 0x55f667d09e00 con 0x55f663f70800 > -5343> 2018-07-11 08:05:51.101977 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:50.861928) v4 -- 0x55f67dc4da00 con 0 > -5342> 2018-07-11 08:05:51.579199 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15255 conn(0x55f663ec2800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.0 seq 199 0x55f67d193400 osd_ping(ping e509 stamp 2018-07-11 08:05:52.824082) v4 > -5341> 2018-07-11 08:05:51.579225 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.0 10.8.128.113:0/15255 199 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:52.824082) v4 ==== 2004+0+0 (1601404329 0 0) 0x55f67d193400 con 0x55f663ec2800 > -5340> 2018-07-11 08:05:51.579236 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:52.824082) v4 -- 0x55f65fa96600 con 0 > -5339> 2018-07-11 08:05:51.579237 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15255 conn(0x55f663ec1000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.0 seq 199 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:05:52.824082) v4 > -5338> 2018-07-11 08:05:51.579272 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.0 10.8.128.113:0/15255 199 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:52.824082) v4 ==== 2004+0+0 (1601404329 0 0) 0x55f67d1afe00 con 0x55f663ec1000 > -5337> 2018-07-11 08:05:51.579287 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:52.824082) v4 -- 0x55f67dc4de00 con 0 > -5336> 2018-07-11 08:05:52.259357 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0d000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 214 0x55f663c0b600 osd_ping(ping e509 stamp 2018-07-11 08:05:53.504340) v4 > -5335> 2018-07-11 08:05:52.259350 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0e800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 214 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:05:53.504340) v4 > -5334> 2018-07-11 08:05:52.259390 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.5 10.8.128.113:0/15263 214 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:53.504340) v4 ==== 2004+0+0 (824846875 0 0) 0x55f663c0b600 con 0x55f663f0d000 > -5333> 2018-07-11 08:05:52.259392 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.5 10.8.128.113:0/15263 214 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:53.504340) v4 ==== 2004+0+0 (824846875 0 0) 0x55f64e20fc00 con 0x55f663f0e800 > -5332> 2018-07-11 08:05:52.259406 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:53.504340) v4 -- 0x55f663c0ac00 con 0 > -5331> 2018-07-11 08:05:52.259443 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:53.504340) v4 -- 0x55f66dc71400 con 0 > -5330> 2018-07-11 08:05:52.476806 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f65f84afc0 con 0 > -5329> 2018-07-11 08:05:52.477389 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 30] Generated table #37045: 2638241 keys, 74546342 bytes > -5328> 2018-07-11 08:05:52.477424 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296352477408, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 37045, "file_size": 74546342, "table_properties": {"data_size": 67112125, "index_size": 690044, "filter_size": 6743183, "raw_key_size": 76215857, "raw_average_key_size": 28, "raw_value_size": 31365760, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} > -5327> 2018-07-11 08:05:52.478603 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 247 0x55f6713c9680 replica scrub(pg: 6.15,from:0'0,to:430'106078,epoch:509/504,start:6:a85230fa:::benchmark_data_magna108_35617_object509499:0,end:6:a8539e67:::benchmark_data_magna108_35617_object849096:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -5326> 2018-07-11 08:05:52.478645 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 247 ==== replica scrub(pg: 6.15,from:0'0,to:430'106078,epoch:509/504,start:6:a85230fa:::benchmark_data_magna108_35617_object509499:0,end:6:a8539e67:::benchmark_data_magna108_35617_object849096:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (867117425 0 0) 0x55f6713c9680 con 0x55f66315e800 > -5325> 2018-07-11 08:05:52.533520 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12650 conn(0x55f663ed9000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.3 seq 206 0x55f6739cd000 osd_ping(ping e509 stamp 2018-07-11 08:05:52.590591) v4 > -5324> 2018-07-11 08:05:52.533545 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.3 10.8.128.106:0/12650 206 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:52.590591) v4 ==== 2004+0+0 (2739074040 0 0) 0x55f6739cd000 con 0x55f663ed9000 > -5323> 2018-07-11 08:05:52.533544 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12650 conn(0x55f663eda800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.3 seq 206 0x55f667d09e00 osd_ping(ping e509 stamp 2018-07-11 08:05:52.590591) v4 > -5322> 2018-07-11 08:05:52.533555 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:52.590591) v4 -- 0x55f65fa96600 con 0 > -5321> 2018-07-11 08:05:52.533561 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.3 10.8.128.106:0/12650 206 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:52.590591) v4 ==== 2004+0+0 (2739074040 0 0) 0x55f667d09e00 con 0x55f663eda800 > -5320> 2018-07-11 08:05:52.533578 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:52.590591) v4 -- 0x55f6780b3800 con 0 > -5319> 2018-07-11 08:05:52.707144 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0a000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 205 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:05:53.342844) v4 > -5318> 2018-07-11 08:05:52.707145 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0b800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 205 0x55f67d193400 osd_ping(ping e509 stamp 2018-07-11 08:05:53.342844) v4 > -5317> 2018-07-11 08:05:52.707176 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.11 10.8.128.114:0/15919 205 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:53.342844) v4 ==== 2004+0+0 (568230650 0 0) 0x55f67d193400 con 0x55f663f0b800 > -5316> 2018-07-11 08:05:52.707173 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.11 10.8.128.114:0/15919 205 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:53.342844) v4 ==== 2004+0+0 (568230650 0 0) 0x55f67d1afe00 con 0x55f663f0a000 > -5315> 2018-07-11 08:05:52.707186 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:53.342844) v4 -- 0x55f65fa96800 con 0 > -5314> 2018-07-11 08:05:52.707210 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:53.342844) v4 -- 0x55f64cc80800 con 0 > -5313> 2018-07-11 08:05:53.012323 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.115:0/13323 conn(0x55f663ebf800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.6 seq 194 0x55f663c0ac00 osd_ping(ping e509 stamp 2018-07-11 08:05:53.365139) v4 > -5312> 2018-07-11 08:05:53.012350 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.6 10.8.128.115:0/13323 194 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:53.365139) v4 ==== 2004+0+0 (3299958355 0 0) 0x55f663c0ac00 con 0x55f663ebf800 > -5311> 2018-07-11 08:05:53.012365 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:53.365139) v4 -- 0x55f664255400 con 0 > -5310> 2018-07-11 08:05:53.012365 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.115:0/13323 conn(0x55f663ebe000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.6 seq 194 0x55f6739cd000 osd_ping(ping e509 stamp 2018-07-11 08:05:53.365139) v4 > -5309> 2018-07-11 08:05:53.012389 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.6 10.8.128.115:0/13323 194 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:53.365139) v4 ==== 2004+0+0 (3299958355 0 0) 0x55f6739cd000 con 0x55f663ebe000 > -5308> 2018-07-11 08:05:53.012401 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:53.365139) v4 -- 0x55f65fa96a00 con 0 > -5307> 2018-07-11 08:05:53.230489 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f656ebd680 con 0 > -5306> 2018-07-11 08:05:53.232051 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 248 0x55f6713c9900 replica scrub(pg: 6.15,from:0'0,to:430'106733,epoch:509/504,start:6:a8539e67:::benchmark_data_magna108_35617_object849096:0,end:6:a854b470:::benchmark_data_magna108.ceph.redhat.com_2478_object60942:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -5305> 2018-07-11 08:05:53.232083 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 248 ==== replica scrub(pg: 6.15,from:0'0,to:430'106733,epoch:509/504,start:6:a8539e67:::benchmark_data_magna108_35617_object849096:0,end:6:a854b470:::benchmark_data_magna108.ceph.redhat.com_2478_object60942:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 232+0+0 (1185124522 0 0) 0x55f6713c9900 con 0x55f66315e800 > -5304> 2018-07-11 08:05:53.279489 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15255 conn(0x55f663ec1000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.0 seq 200 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:05:54.524533) v4 > -5303> 2018-07-11 08:05:53.279517 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.0 10.8.128.113:0/15255 200 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:54.524533) v4 ==== 2004+0+0 (3364473794 0 0) 0x55f64e20fc00 con 0x55f663ec1000 > -5302> 2018-07-11 08:05:53.279532 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:54.524533) v4 -- 0x55f64cc80800 con 0 > -5301> 2018-07-11 08:05:53.279545 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15255 conn(0x55f663ec2800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.0 seq 200 0x55f67d193400 osd_ping(ping e509 stamp 2018-07-11 08:05:54.524533) v4 > -5300> 2018-07-11 08:05:53.279565 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.0 10.8.128.113:0/15255 200 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:54.524533) v4 ==== 2004+0+0 (3364473794 0 0) 0x55f67d193400 con 0x55f663ec2800 > -5299> 2018-07-11 08:05:53.279580 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:54.524533) v4 -- 0x55f65fa96a00 con 0 > -5298> 2018-07-11 08:05:53.665648 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.114:6805/15885 conn(0x55f66315a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.7 seq 2561 0x55f667d09e00 MOSDScrubReserve(5.4 REQUEST e509) v1 > -5297> 2018-07-11 08:05:53.665685 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.7 10.8.128.114:6805/15885 2561 ==== MOSDScrubReserve(5.4 REQUEST e509) v1 ==== 43+0+0 (3003630469 0 0) 0x55f667d09e00 con 0x55f66315a000 > -5296> 2018-07-11 08:05:53.665772 7f491983f700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.114:6805/15885 -- MOSDScrubReserve(5.4 REJECT e509) v1 -- 0x55f666182e00 con 0 > -5295> 2018-07-11 08:05:53.666105 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.114:6805/15885 conn(0x55f66315a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.7 seq 2562 0x55f666182e00 MOSDScrubReserve(5.4 RELEASE e509) v1 > -5294> 2018-07-11 08:05:53.666123 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.7 10.8.128.114:6805/15885 2562 ==== MOSDScrubReserve(5.4 RELEASE e509) v1 ==== 43+0+0 (2082093447 0 0) 0x55f666182e00 con 0x55f66315a000 > -5293> 2018-07-11 08:05:53.958016 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12651 conn(0x55f663219000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25 cs=1 l=1). rx osd.2 seq 217 0x55f664255400 osd_ping(ping e509 stamp 2018-07-11 08:05:54.015070) v4 > -5292> 2018-07-11 08:05:53.958051 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.2 10.8.128.106:0/12651 217 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:54.015070) v4 ==== 2004+0+0 (3182131758 0 0) 0x55f664255400 con 0x55f663219000 > -5291> 2018-07-11 08:05:53.958068 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:54.015070) v4 -- 0x55f663f5f600 con 0 > -5290> 2018-07-11 08:05:53.958168 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12651 conn(0x55f66321a800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25 cs=1 l=1). rx osd.2 seq 217 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:05:54.015070) v4 > -5289> 2018-07-11 08:05:53.958196 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.2 10.8.128.106:0/12651 217 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:54.015070) v4 ==== 2004+0+0 (3182131758 0 0) 0x55f67d1afe00 con 0x55f66321a800 > -5288> 2018-07-11 08:05:53.958213 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:54.015070) v4 -- 0x55f64c8aaa00 con 0 > -5287> 2018-07-11 08:05:53.959708 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0e800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 215 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:05:55.204727) v4 > -5286> 2018-07-11 08:05:53.959710 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0d000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 215 0x55f65d024000 osd_ping(ping e509 stamp 2018-07-11 08:05:55.204727) v4 > -5285> 2018-07-11 08:05:53.959730 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.5 10.8.128.113:0/15263 215 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:55.204727) v4 ==== 2004+0+0 (3132102860 0 0) 0x55f64e20fc00 con 0x55f663f0e800 > -5284> 2018-07-11 08:05:53.959731 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.5 10.8.128.113:0/15263 215 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:55.204727) v4 ==== 2004+0+0 (3132102860 0 0) 0x55f65d024000 con 0x55f663f0d000 > -5283> 2018-07-11 08:05:53.959750 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:55.204727) v4 -- 0x55f667d09800 con 0 > -5282> 2018-07-11 08:05:53.959781 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:55.204727) v4 -- 0x55f666c46a00 con 0 > -5281> 2018-07-11 08:05:54.002328 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.108:0/11967 conn(0x55f663f6f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=1). rx osd.8 seq 200 0x55f6739cd000 osd_ping(ping e509 stamp 2018-07-11 08:05:53.762381) v4 > -5280> 2018-07-11 08:05:54.002352 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.8 10.8.128.108:0/11967 200 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:53.762381) v4 ==== 2004+0+0 (1960151688 0 0) 0x55f6739cd000 con 0x55f663f6f000 > -5279> 2018-07-11 08:05:54.002365 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:53.762381) v4 -- 0x55f65fa96c00 con 0 > -5278> 2018-07-11 08:05:54.002455 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.108:0/11967 conn(0x55f663f70800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=1). rx osd.8 seq 200 0x55f664255400 osd_ping(ping e509 stamp 2018-07-11 08:05:53.762381) v4 > -5277> 2018-07-11 08:05:54.002472 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.8 10.8.128.108:0/11967 200 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:53.762381) v4 ==== 2004+0+0 (1960151688 0 0) 0x55f664255400 con 0x55f663f70800 > -5276> 2018-07-11 08:05:54.002489 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:53.762381) v4 -- 0x55f67c196000 con 0 > -5275> 2018-07-11 08:05:54.068558 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f6528a2d80 con 0 > -5274> 2018-07-11 08:05:54.070041 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 249 0x55f6713c9b80 replica scrub(pg: 6.15,from:0'0,to:430'106462,epoch:509/504,start:6:a854b470:::benchmark_data_magna108.ceph.redhat.com_2478_object60942:0,end:6:a855e2eb:::benchmark_data_magna108.ceph.redhat.com_2478_object441007:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -5273> 2018-07-11 08:05:54.070070 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 249 ==== replica scrub(pg: 6.15,from:0'0,to:430'106462,epoch:509/504,start:6:a854b470:::benchmark_data_magna108.ceph.redhat.com_2478_object60942:0,end:6:a855e2eb:::benchmark_data_magna108.ceph.redhat.com_2478_object441007:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 247+0+0 (1635440567 0 0) 0x55f6713c9b80 con 0x55f66315e800 > -5272> 2018-07-11 08:05:54.179098 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed7800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 206 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:05:54.236050) v4 > -5271> 2018-07-11 08:05:54.179137 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.1 10.8.128.106:0/12677 206 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:54.236050) v4 ==== 2004+0+0 (1064226857 0 0) 0x55f67d1afe00 con 0x55f663ed7800 > -5270> 2018-07-11 08:05:54.179149 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:54.236050) v4 -- 0x55f667d09800 con 0 > -5269> 2018-07-11 08:05:54.179156 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed6000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 206 0x55f65d024000 osd_ping(ping e509 stamp 2018-07-11 08:05:54.236050) v4 > -5268> 2018-07-11 08:05:54.179176 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.1 10.8.128.106:0/12677 206 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:54.236050) v4 ==== 2004+0+0 (1064226857 0 0) 0x55f65d024000 con 0x55f663ed6000 > -5267> 2018-07-11 08:05:54.179187 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:54.236050) v4 -- 0x55f67c196200 con 0 > -5266> 2018-07-11 08:05:54.204693 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15287 conn(0x55f66dce1000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.4 seq 210 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:05:55.449555) v4 > -5265> 2018-07-11 08:05:54.204715 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.4 10.8.128.113:0/15287 210 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:55.449555) v4 ==== 2004+0+0 (2424167831 0 0) 0x55f64e20fc00 con 0x55f66dce1000 > -5264> 2018-07-11 08:05:54.204724 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:55.449555) v4 -- 0x55f667d09800 con 0 > -5263> 2018-07-11 08:05:54.204737 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15287 conn(0x55f66dce2800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.4 seq 210 0x55f67c196000 osd_ping(ping e509 stamp 2018-07-11 08:05:55.449555) v4 > -5262> 2018-07-11 08:05:54.204758 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.4 10.8.128.113:0/15287 210 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:55.449555) v4 ==== 2004+0+0 (2424167831 0 0) 0x55f67c196000 con 0x55f66dce2800 > -5261> 2018-07-11 08:05:54.204769 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:55.449555) v4 -- 0x55f67c196200 con 0 > -5260> 2018-07-11 08:05:54.289816 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.107:0/12617 conn(0x55f64e3c3800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.9 seq 206 0x55f67d193400 osd_ping(ping e509 stamp 2018-07-11 08:05:54.289495) v4 > -5259> 2018-07-11 08:05:54.289825 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.107:0/12617 conn(0x55f64e3c5000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.9 seq 206 0x55f65d024000 osd_ping(ping e509 stamp 2018-07-11 08:05:54.289495) v4 > -5258> 2018-07-11 08:05:54.289839 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.9 10.8.128.107:0/12617 206 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:54.289495) v4 ==== 2004+0+0 (433185960 0 0) 0x55f67d193400 con 0x55f64e3c3800 > -5257> 2018-07-11 08:05:54.289842 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.9 10.8.128.107:0/12617 206 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:54.289495) v4 ==== 2004+0+0 (433185960 0 0) 0x55f65d024000 con 0x55f64e3c5000 > -5256> 2018-07-11 08:05:54.289855 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:54.289495) v4 -- 0x55f65fa96c00 con 0 > -5255> 2018-07-11 08:05:54.289879 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:54.289495) v4 -- 0x55f67c196200 con 0 > -5254> 2018-07-11 08:05:54.339173 7f491803c700 5 osd.10 509 heartbeat: osd_stat(789 GB used, 141 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,0,1]) > -5253> 2018-07-11 08:05:54.339186 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6802/15255 -- osd_ping(ping e509 stamp 2018-07-11 08:05:54.339184) v4 -- 0x55f64fcd4600 con 0 > -5252> 2018-07-11 08:05:54.339221 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6803/15255 -- osd_ping(ping e509 stamp 2018-07-11 08:05:54.339184) v4 -- 0x55f64fcd4800 con 0 > -5251> 2018-07-11 08:05:54.339238 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6806/12677 -- osd_ping(ping e509 stamp 2018-07-11 08:05:54.339184) v4 -- 0x55f64fcd4a00 con 0 > -5250> 2018-07-11 08:05:54.339263 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6807/12677 -- osd_ping(ping e509 stamp 2018-07-11 08:05:54.339184) v4 -- 0x55f64fcd4c00 con 0 > -5249> 2018-07-11 08:05:54.339277 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6808/12651 -- osd_ping(ping e509 stamp 2018-07-11 08:05:54.339184) v4 -- 0x55f64fcd4e00 con 0 > -5248> 2018-07-11 08:05:54.339290 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6810/12651 -- osd_ping(ping e509 stamp 2018-07-11 08:05:54.339184) v4 -- 0x55f64fcd5000 con 0 > -5247> 2018-07-11 08:05:54.339302 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6809/12650 -- osd_ping(ping e509 stamp 2018-07-11 08:05:54.339184) v4 -- 0x55f64fcd5200 con 0 > -5246> 2018-07-11 08:05:54.339320 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6811/12650 -- osd_ping(ping e509 stamp 2018-07-11 08:05:54.339184) v4 -- 0x55f64fcd5400 con 0 > -5245> 2018-07-11 08:05:54.339333 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6810/15287 -- osd_ping(ping e509 stamp 2018-07-11 08:05:54.339184) v4 -- 0x55f64fcd5600 con 0 > -5244> 2018-07-11 08:05:54.339345 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6811/15287 -- osd_ping(ping e509 stamp 2018-07-11 08:05:54.339184) v4 -- 0x55f64fcd5800 con 0 > -5243> 2018-07-11 08:05:54.339361 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6806/15263 -- osd_ping(ping e509 stamp 2018-07-11 08:05:54.339184) v4 -- 0x55f64fcd5a00 con 0 > -5242> 2018-07-11 08:05:54.339375 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6807/15263 -- osd_ping(ping e509 stamp 2018-07-11 08:05:54.339184) v4 -- 0x55f64fcd5c00 con 0 > -5241> 2018-07-11 08:05:54.339390 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.115:6802/13323 -- osd_ping(ping e509 stamp 2018-07-11 08:05:54.339184) v4 -- 0x55f64fcd5e00 con 0 > -5240> 2018-07-11 08:05:54.339404 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.115:6803/13323 -- osd_ping(ping e509 stamp 2018-07-11 08:05:54.339184) v4 -- 0x55f6841a4000 con 0 > -5239> 2018-07-11 08:05:54.339417 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6806/15885 -- osd_ping(ping e509 stamp 2018-07-11 08:05:54.339184) v4 -- 0x55f6841a4200 con 0 > -5238> 2018-07-11 08:05:54.339430 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6807/15885 -- osd_ping(ping e509 stamp 2018-07-11 08:05:54.339184) v4 -- 0x55f6841a4400 con 0 > -5237> 2018-07-11 08:05:54.339443 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.108:6802/11967 -- osd_ping(ping e509 stamp 2018-07-11 08:05:54.339184) v4 -- 0x55f6841a4600 con 0 > -5236> 2018-07-11 08:05:54.339455 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.108:6803/11967 -- osd_ping(ping e509 stamp 2018-07-11 08:05:54.339184) v4 -- 0x55f6841a4800 con 0 > -5235> 2018-07-11 08:05:54.339467 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.107:6806/12617 -- osd_ping(ping e509 stamp 2018-07-11 08:05:54.339184) v4 -- 0x55f6841a4a00 con 0 > -5234> 2018-07-11 08:05:54.339480 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.107:6807/12617 -- osd_ping(ping e509 stamp 2018-07-11 08:05:54.339184) v4 -- 0x55f6841a4c00 con 0 > -5233> 2018-07-11 08:05:54.339493 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6802/15919 -- osd_ping(ping e509 stamp 2018-07-11 08:05:54.339184) v4 -- 0x55f6841a4e00 con 0 > -5232> 2018-07-11 08:05:54.339505 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6803/15919 -- osd_ping(ping e509 stamp 2018-07-11 08:05:54.339184) v4 -- 0x55f6841a5000 con 0 > -5231> 2018-07-11 08:05:54.339595 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6803/15255 conn(0x55f656ad5800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 204 0x55f67d193400 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:54.339184) v4 > -5230> 2018-07-11 08:05:54.339622 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.0 10.8.128.113:6803/15255 204 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:54.339184) v4 ==== 2004+0+0 (2725296034 0 0) 0x55f67d193400 con 0x55f656ad5800 > -5229> 2018-07-11 08:05:54.339623 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6802/15255 conn(0x55f656ad4000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 204 0x55f65d024000 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:54.339184) v4 > -5228> 2018-07-11 08:05:54.339653 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.0 10.8.128.113:6802/15255 204 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:54.339184) v4 ==== 2004+0+0 (2725296034 0 0) 0x55f65d024000 con 0x55f656ad4000 > -5227> 2018-07-11 08:05:54.339656 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.107:6806/12617 conn(0x55f656ade000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.9 seq 204 0x55f67d193400 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:54.339184) v4 > -5226> 2018-07-11 08:05:54.339686 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.9 10.8.128.107:6806/12617 204 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:54.339184) v4 ==== 2004+0+0 (2725296034 0 0) 0x55f67d193400 con 0x55f656ade000 > -5225> 2018-07-11 08:05:54.339734 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6806/12677 conn(0x55f64e3d8000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.1 seq 204 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:54.339184) v4 > -5224> 2018-07-11 08:05:54.339709 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6807/12677 conn(0x55f64e3d9800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.1 seq 204 0x55f65d024000 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:54.339184) v4 > -5223> 2018-07-11 08:05:54.339756 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.1 10.8.128.106:6806/12677 204 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:54.339184) v4 ==== 2004+0+0 (2725296034 0 0) 0x55f67d1afe00 con 0x55f64e3d8000 > -5222> 2018-07-11 08:05:54.339758 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.1 10.8.128.106:6807/12677 204 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:54.339184) v4 ==== 2004+0+0 (2725296034 0 0) 0x55f65d024000 con 0x55f64e3d9800 > -5221> 2018-07-11 08:05:54.339786 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.107:6807/12617 conn(0x55f656adf800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.9 seq 204 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:54.339184) v4 > -5220> 2018-07-11 08:05:54.339797 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.9 10.8.128.107:6807/12617 204 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:54.339184) v4 ==== 2004+0+0 (2725296034 0 0) 0x55f67d1afe00 con 0x55f656adf800 > -5219> 2018-07-11 08:05:54.339801 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6811/15287 conn(0x55f64e402800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.4 seq 204 0x55f67d193400 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:54.339184) v4 > -5218> 2018-07-11 08:05:54.339821 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.4 10.8.128.113:6811/15287 204 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:54.339184) v4 ==== 2004+0+0 (2725296034 0 0) 0x55f67d193400 con 0x55f64e402800 > -5217> 2018-07-11 08:05:54.339845 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6810/12651 conn(0x55f64e3eb800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.2 seq 204 0x55f67d193400 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:54.339184) v4 > -5216> 2018-07-11 08:05:54.339855 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.2 10.8.128.106:6810/12651 204 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:54.339184) v4 ==== 2004+0+0 (2725296034 0 0) 0x55f67d193400 con 0x55f64e3eb800 > -5215> 2018-07-11 08:05:54.339853 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6808/12651 conn(0x55f64e3ea000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.2 seq 204 0x55f65d024000 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:54.339184) v4 > -5214> 2018-07-11 08:05:54.339866 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.2 10.8.128.106:6808/12651 204 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:54.339184) v4 ==== 2004+0+0 (2725296034 0 0) 0x55f65d024000 con 0x55f64e3ea000 > -5213> 2018-07-11 08:05:54.339863 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6806/15263 conn(0x55f64e3ed000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.5 seq 204 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:54.339184) v4 > -5212> 2018-07-11 08:05:54.339875 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.5 10.8.128.113:6806/15263 204 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:54.339184) v4 ==== 2004+0+0 (2725296034 0 0) 0x55f67d1afe00 con 0x55f64e3ed000 > -5211> 2018-07-11 08:05:54.339896 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6810/15287 conn(0x55f64e401000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.4 seq 204 0x55f65d024000 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:54.339184) v4 > -5210> 2018-07-11 08:05:54.339909 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.4 10.8.128.113:6810/15287 204 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:54.339184) v4 ==== 2004+0+0 (2725296034 0 0) 0x55f65d024000 con 0x55f64e401000 > -5209> 2018-07-11 08:05:54.339923 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6809/12650 conn(0x55f656ad7000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.3 seq 204 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:54.339184) v4 > -5208> 2018-07-11 08:05:54.339937 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.3 10.8.128.106:6809/12650 204 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:54.339184) v4 ==== 2004+0+0 (2725296034 0 0) 0x55f67d1afe00 con 0x55f656ad7000 > -5207> 2018-07-11 08:05:54.339937 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6811/12650 conn(0x55f656ad8800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.3 seq 204 0x55f65d024000 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:54.339184) v4 > -5206> 2018-07-11 08:05:54.339947 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.3 10.8.128.106:6811/12650 204 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:54.339184) v4 ==== 2004+0+0 (2725296034 0 0) 0x55f65d024000 con 0x55f656ad8800 > -5205> 2018-07-11 08:05:54.339975 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6807/15263 conn(0x55f64e3ee800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.5 seq 204 0x55f65d024000 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:54.339184) v4 > -5204> 2018-07-11 08:05:54.339986 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.115:6803/13323 conn(0x55f656ace800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.6 seq 204 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:54.339184) v4 > -5203> 2018-07-11 08:05:54.339999 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.6 10.8.128.115:6803/13323 204 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:54.339184) v4 ==== 2004+0+0 (2725296034 0 0) 0x55f67d1afe00 con 0x55f656ace800 > -5202> 2018-07-11 08:05:54.339987 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.5 10.8.128.113:6807/15263 204 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:54.339184) v4 ==== 2004+0+0 (2725296034 0 0) 0x55f65d024000 con 0x55f64e3ee800 > -5201> 2018-07-11 08:05:54.340034 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.108:6802/11967 conn(0x55f656aca000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.8 seq 204 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:54.339184) v4 > -5200> 2018-07-11 08:05:54.340042 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.108:6803/11967 conn(0x55f656acb800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.8 seq 204 0x55f65d024000 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:54.339184) v4 > -5199> 2018-07-11 08:05:54.340046 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.8 10.8.128.108:6802/11967 204 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:54.339184) v4 ==== 2004+0+0 (2725296034 0 0) 0x55f67d1afe00 con 0x55f656aca000 > -5198> 2018-07-11 08:05:54.340058 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.8 10.8.128.108:6803/11967 204 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:54.339184) v4 ==== 2004+0+0 (2725296034 0 0) 0x55f65d024000 con 0x55f656acb800 > -5197> 2018-07-11 08:05:54.340027 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6806/15885 conn(0x55f64e3fe000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.7 seq 204 0x55f67d193400 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:54.339184) v4 > -5196> 2018-07-11 08:05:54.340075 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.7 10.8.128.114:6806/15885 204 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:54.339184) v4 ==== 2004+0+0 (2725296034 0 0) 0x55f67d193400 con 0x55f64e3fe000 > -5195> 2018-07-11 08:05:54.340085 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6807/15885 conn(0x55f64e3ff800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.7 seq 204 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:54.339184) v4 > -5194> 2018-07-11 08:05:54.340096 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.7 10.8.128.114:6807/15885 204 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:54.339184) v4 ==== 2004+0+0 (2725296034 0 0) 0x55f67d1afe00 con 0x55f64e3ff800 > -5193> 2018-07-11 08:05:54.340128 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6803/15919 conn(0x55f64e3dc800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.11 seq 204 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:54.339184) v4 > -5192> 2018-07-11 08:05:54.340131 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.115:6802/13323 conn(0x55f656acd000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.6 seq 204 0x55f67d193400 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:54.339184) v4 > -5191> 2018-07-11 08:05:54.340141 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.11 10.8.128.114:6803/15919 204 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:54.339184) v4 ==== 2004+0+0 (2725296034 0 0) 0x55f67d1afe00 con 0x55f64e3dc800 > -5190> 2018-07-11 08:05:54.340154 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.6 10.8.128.115:6802/13323 204 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:54.339184) v4 ==== 2004+0+0 (2725296034 0 0) 0x55f67d193400 con 0x55f656acd000 > -5189> 2018-07-11 08:05:54.340185 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6802/15919 conn(0x55f64e3db000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.11 seq 204 0x55f67d193400 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:54.339184) v4 > -5188> 2018-07-11 08:05:54.340202 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.11 10.8.128.114:6802/15919 204 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:54.339184) v4 ==== 2004+0+0 (2725296034 0 0) 0x55f67d193400 con 0x55f64e3db000 > -5187> 2018-07-11 08:05:54.436496 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15885 conn(0x55f663217800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 209 0x55f67d193400 osd_ping(ping e509 stamp 2018-07-11 08:05:55.072247) v4 > -5186> 2018-07-11 08:05:54.436517 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.7 10.8.128.114:0/15885 209 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:55.072247) v4 ==== 2004+0+0 (886129516 0 0) 0x55f67d193400 con 0x55f663217800 > -5185> 2018-07-11 08:05:54.436529 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:55.072247) v4 -- 0x55f65fa96c00 con 0 > -5184> 2018-07-11 08:05:54.436553 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15885 conn(0x55f663216000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 209 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:05:55.072247) v4 > -5183> 2018-07-11 08:05:54.436574 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.7 10.8.128.114:0/15885 209 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:55.072247) v4 ==== 2004+0+0 (886129516 0 0) 0x55f67d1afe00 con 0x55f663216000 > -5182> 2018-07-11 08:05:54.436585 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:55.072247) v4 -- 0x55f64fcd4a00 con 0 > -5181> 2018-07-11 08:05:54.705130 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15287 conn(0x55f66dce2800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.4 seq 211 0x55f65d024000 osd_ping(ping e509 stamp 2018-07-11 08:05:55.950083) v4 > -5180> 2018-07-11 08:05:54.705157 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.4 10.8.128.113:0/15287 211 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:55.950083) v4 ==== 2004+0+0 (4079416778 0 0) 0x55f65d024000 con 0x55f66dce2800 > -5179> 2018-07-11 08:05:54.705175 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:55.950083) v4 -- 0x55f64fcd4600 con 0 > -5178> 2018-07-11 08:05:54.705225 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15287 conn(0x55f66dce1000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.4 seq 211 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:05:55.950083) v4 > -5177> 2018-07-11 08:05:54.705255 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.4 10.8.128.113:0/15287 211 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:55.950083) v4 ==== 2004+0+0 (4079416778 0 0) 0x55f64e20fc00 con 0x55f66dce1000 > -5176> 2018-07-11 08:05:54.705267 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:55.950083) v4 -- 0x55f64fcd4a00 con 0 > -5175> 2018-07-11 08:05:54.711771 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f66387a480 con 0 > -5174> 2018-07-11 08:05:54.713286 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 250 0x55f67c24c280 replica scrub(pg: 6.15,from:0'0,to:430'106830,epoch:509/504,start:6:a855e2eb:::benchmark_data_magna108.ceph.redhat.com_2478_object441007:0,end:6:a856d1e1:::benchmark_data_magna108_35617_object1024733:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -5173> 2018-07-11 08:05:54.713317 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 250 ==== replica scrub(pg: 6.15,from:0'0,to:430'106830,epoch:509/504,start:6:a855e2eb:::benchmark_data_magna108.ceph.redhat.com_2478_object441007:0,end:6:a856d1e1:::benchmark_data_magna108_35617_object1024733:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 234+0+0 (4218807411 0 0) 0x55f67c24c280 con 0x55f66315e800 > -5172> 2018-07-11 08:05:55.330404 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f655a9a000 con 0 > -5171> 2018-07-11 08:05:55.331943 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 251 0x55f67c24c500 replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:509/504,start:6:a856d1e1:::benchmark_data_magna108_35617_object1024733:0,end:6:a857ad8c:::benchmark_data_magna108_35617_object434162:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -5170> 2018-07-11 08:05:55.331969 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 251 ==== replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:509/504,start:6:a856d1e1:::benchmark_data_magna108_35617_object1024733:0,end:6:a857ad8c:::benchmark_data_magna108_35617_object434162:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 219+0+0 (2607776460 0 0) 0x55f67c24c500 con 0x55f66315e800 > -5169> 2018-07-11 08:05:55.530264 7f491d046700 1 -- 10.8.128.107:6800/14173 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55f656a94580 con 0 > -5168> 2018-07-11 08:05:55.530326 7f491d046700 1 -- 10.8.128.107:6800/14173 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55f6631cf8c0 con 0 > -5167> 2018-07-11 08:05:55.658687 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12651 conn(0x55f663219000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25 cs=1 l=1). rx osd.2 seq 218 0x55f67c196000 osd_ping(ping e509 stamp 2018-07-11 08:05:55.715571) v4 > -5166> 2018-07-11 08:05:55.658714 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.2 10.8.128.106:0/12651 218 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:55.715571) v4 ==== 2004+0+0 (678955116 0 0) 0x55f67c196000 con 0x55f663219000 > -5165> 2018-07-11 08:05:55.658724 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:55.715571) v4 -- 0x55f64fcd4600 con 0 > -5164> 2018-07-11 08:05:55.658766 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12651 conn(0x55f66321a800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25 cs=1 l=1). rx osd.2 seq 218 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:05:55.715571) v4 > -5163> 2018-07-11 08:05:55.658787 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.2 10.8.128.106:0/12651 218 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:55.715571) v4 ==== 2004+0+0 (678955116 0 0) 0x55f67d1afe00 con 0x55f66321a800 > -5162> 2018-07-11 08:05:55.658798 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:55.715571) v4 -- 0x55f64fcd4a00 con 0 > -5161> 2018-07-11 08:05:56.033962 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12650 conn(0x55f663eda800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.3 seq 207 0x55f65d024000 osd_ping(ping e509 stamp 2018-07-11 08:05:56.091051) v4 > -5160> 2018-07-11 08:05:56.033989 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.3 10.8.128.106:0/12650 207 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:56.091051) v4 ==== 2004+0+0 (4099595561 0 0) 0x55f65d024000 con 0x55f663eda800 > -5159> 2018-07-11 08:05:56.033985 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12650 conn(0x55f663ed9000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.3 seq 207 0x55f6739cd000 osd_ping(ping e509 stamp 2018-07-11 08:05:56.091051) v4 > -5158> 2018-07-11 08:05:56.034010 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:56.091051) v4 -- 0x55f64fcd4600 con 0 > -5157> 2018-07-11 08:05:56.034018 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.3 10.8.128.106:0/12650 207 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:56.091051) v4 ==== 2004+0+0 (4099595561 0 0) 0x55f6739cd000 con 0x55f663ed9000 > -5156> 2018-07-11 08:05:56.034043 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:56.091051) v4 -- 0x55f65fa96c00 con 0 > -5155> 2018-07-11 08:05:56.065838 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f664077440 con 0 > -5154> 2018-07-11 08:05:56.067481 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 252 0x55f67c24c780 replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:509/504,start:6:a857ad8c:::benchmark_data_magna108_35617_object434162:0,end:6:a858e040:::benchmark_data_magna108_35617_object779382:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -5153> 2018-07-11 08:05:56.067512 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 252 ==== replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:509/504,start:6:a857ad8c:::benchmark_data_magna108_35617_object434162:0,end:6:a858e040:::benchmark_data_magna108_35617_object779382:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (2129236711 0 0) 0x55f67c24c780 con 0x55f66315e800 > -5152> 2018-07-11 08:05:56.405684 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15287 conn(0x55f66dce2800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.4 seq 212 0x55f67c196000 osd_ping(ping e509 stamp 2018-07-11 08:05:57.650656) v4 > -5151> 2018-07-11 08:05:56.405707 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.4 10.8.128.113:0/15287 212 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:57.650656) v4 ==== 2004+0+0 (987917115 0 0) 0x55f67c196000 con 0x55f66dce2800 > -5150> 2018-07-11 08:05:56.405719 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:57.650656) v4 -- 0x55f67c196400 con 0 > -5149> 2018-07-11 08:05:56.405722 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15287 conn(0x55f66dce1000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.4 seq 212 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:05:57.650656) v4 > -5148> 2018-07-11 08:05:56.405745 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.4 10.8.128.113:0/15287 212 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:57.650656) v4 ==== 2004+0+0 (987917115 0 0) 0x55f64e20fc00 con 0x55f66dce1000 > -5147> 2018-07-11 08:05:56.405756 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:57.650656) v4 -- 0x55f667d09800 con 0 > -5146> 2018-07-11 08:05:56.860001 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0e800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 216 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:05:58.105088) v4 > -5145> 2018-07-11 08:05:56.860026 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.5 10.8.128.113:0/15263 216 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:58.105088) v4 ==== 2004+0+0 (243033878 0 0) 0x55f67d1afe00 con 0x55f663f0e800 > -5144> 2018-07-11 08:05:56.860029 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0d000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 216 0x55f64fcd4600 osd_ping(ping e509 stamp 2018-07-11 08:05:58.105088) v4 > -5143> 2018-07-11 08:05:56.860040 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:58.105088) v4 -- 0x55f65d025600 con 0 > -5142> 2018-07-11 08:05:56.860054 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.5 10.8.128.113:0/15263 216 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:58.105088) v4 ==== 2004+0+0 (243033878 0 0) 0x55f64fcd4600 con 0x55f663f0d000 > -5141> 2018-07-11 08:05:56.860073 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:58.105088) v4 -- 0x55f67c196600 con 0 > -5140> 2018-07-11 08:05:57.112726 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.115:0/13323 conn(0x55f663ebf800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.6 seq 195 0x55f67c196000 osd_ping(ping e509 stamp 2018-07-11 08:05:57.465614) v4 > -5139> 2018-07-11 08:05:57.112751 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.6 10.8.128.115:0/13323 195 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:57.465614) v4 ==== 2004+0+0 (3935561609 0 0) 0x55f67c196000 con 0x55f663ebf800 > -5138> 2018-07-11 08:05:57.112765 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:57.465614) v4 -- 0x55f67c196800 con 0 > -5137> 2018-07-11 08:05:57.112805 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.115:0/13323 conn(0x55f663ebe000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.6 seq 195 0x55f67d193400 osd_ping(ping e509 stamp 2018-07-11 08:05:57.465614) v4 > -5136> 2018-07-11 08:05:57.112826 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.6 10.8.128.115:0/13323 195 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:57.465614) v4 ==== 2004+0+0 (3935561609 0 0) 0x55f67d193400 con 0x55f663ebe000 > -5135> 2018-07-11 08:05:57.112843 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:57.465614) v4 -- 0x55f65fa96e00 con 0 > -5134> 2018-07-11 08:05:57.177065 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 30] Generated table #37046: 2638241 keys, 74546279 bytes > -5133> 2018-07-11 08:05:57.177099 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296357177084, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 37046, "file_size": 74546279, "table_properties": {"data_size": 67112151, "index_size": 689955, "filter_size": 6743183, "raw_key_size": 76215853, "raw_average_key_size": 28, "raw_value_size": 31365756, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} > -5132> 2018-07-11 08:05:57.190517 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.107:0/12617 conn(0x55f64e3c5000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.9 seq 207 0x55f64fcd4600 osd_ping(ping e509 stamp 2018-07-11 08:05:57.189981) v4 > -5131> 2018-07-11 08:05:57.190541 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.9 10.8.128.107:0/12617 207 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:57.189981) v4 ==== 2004+0+0 (758385373 0 0) 0x55f64fcd4600 con 0x55f64e3c5000 > -5130> 2018-07-11 08:05:57.190558 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:57.189981) v4 -- 0x55f67c196a00 con 0 > -5129> 2018-07-11 08:05:57.190613 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.107:0/12617 conn(0x55f64e3c3800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.9 seq 207 0x55f6739cd000 osd_ping(ping e509 stamp 2018-07-11 08:05:57.189981) v4 > -5128> 2018-07-11 08:05:57.190632 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.9 10.8.128.107:0/12617 207 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:57.189981) v4 ==== 2004+0+0 (758385373 0 0) 0x55f6739cd000 con 0x55f64e3c3800 > -5127> 2018-07-11 08:05:57.190644 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:57.189981) v4 -- 0x55f65fa96e00 con 0 > -5126> 2018-07-11 08:05:57.407550 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0a000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 206 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:05:58.043296) v4 > -5125> 2018-07-11 08:05:57.407565 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0b800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 206 0x55f67d193400 osd_ping(ping e509 stamp 2018-07-11 08:05:58.043296) v4 > -5124> 2018-07-11 08:05:57.407577 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.11 10.8.128.114:0/15919 206 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:58.043296) v4 ==== 2004+0+0 (3272797398 0 0) 0x55f64e20fc00 con 0x55f663f0a000 > -5123> 2018-07-11 08:05:57.407582 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.11 10.8.128.114:0/15919 206 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:58.043296) v4 ==== 2004+0+0 (3272797398 0 0) 0x55f67d193400 con 0x55f663f0b800 > -5122> 2018-07-11 08:05:57.407592 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:58.043296) v4 -- 0x55f65fa96e00 con 0 > -5121> 2018-07-11 08:05:57.407617 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:58.043296) v4 -- 0x55f65d025600 con 0 > -5120> 2018-07-11 08:05:57.679512 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed7800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 207 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:05:57.736612) v4 > -5119> 2018-07-11 08:05:57.679521 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed6000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 207 0x55f67c196000 osd_ping(ping e509 stamp 2018-07-11 08:05:57.736612) v4 > -5118> 2018-07-11 08:05:57.679535 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.1 10.8.128.106:0/12677 207 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:57.736612) v4 ==== 2004+0+0 (746383868 0 0) 0x55f67d1afe00 con 0x55f663ed7800 > -5117> 2018-07-11 08:05:57.679539 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.1 10.8.128.106:0/12677 207 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:57.736612) v4 ==== 2004+0+0 (746383868 0 0) 0x55f67c196000 con 0x55f663ed6000 > -5116> 2018-07-11 08:05:57.679545 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:57.736612) v4 -- 0x55f653d4ba00 con 0 > -5115> 2018-07-11 08:05:57.679569 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:57.736612) v4 -- 0x55f67c196c00 con 0 > -5114> 2018-07-11 08:05:57.741534 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f65f6738c0 con 0 > -5113> 2018-07-11 08:05:57.743108 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 253 0x55f67c24ca00 replica scrub(pg: 6.15,from:0'0,to:430'106132,epoch:509/504,start:6:a858e040:::benchmark_data_magna108_35617_object779382:0,end:6:a85a463d:::benchmark_data_magna108_35617_object490359:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -5112> 2018-07-11 08:05:57.743139 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 253 ==== replica scrub(pg: 6.15,from:0'0,to:430'106132,epoch:509/504,start:6:a858e040:::benchmark_data_magna108_35617_object779382:0,end:6:a85a463d:::benchmark_data_magna108_35617_object490359:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (709593372 0 0) 0x55f67c24ca00 con 0x55f66315e800 > -5111> 2018-07-11 08:05:57.936902 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15885 conn(0x55f663217800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 210 0x55f6739cd000 osd_ping(ping e509 stamp 2018-07-11 08:05:58.572698) v4 > -5110> 2018-07-11 08:05:57.936913 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15885 conn(0x55f663216000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 210 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:05:58.572698) v4 > -5109> 2018-07-11 08:05:57.936924 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.7 10.8.128.114:0/15885 210 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:58.572698) v4 ==== 2004+0+0 (2769362918 0 0) 0x55f6739cd000 con 0x55f663217800 > -5108> 2018-07-11 08:05:57.936930 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.7 10.8.128.114:0/15885 210 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:58.572698) v4 ==== 2004+0+0 (2769362918 0 0) 0x55f64e20fc00 con 0x55f663216000 > -5107> 2018-07-11 08:05:57.936934 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:58.572698) v4 -- 0x55f65fa96e00 con 0 > -5106> 2018-07-11 08:05:57.936968 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:58.572698) v4 -- 0x55f653d4ba00 con 0 > -5105> 2018-07-11 08:05:57.959219 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12651 conn(0x55f663219000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25 cs=1 l=1). rx osd.2 seq 219 0x55f64fcd4600 osd_ping(ping e509 stamp 2018-07-11 08:05:58.016206) v4 > -5104> 2018-07-11 08:05:57.959244 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.2 10.8.128.106:0/12651 219 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:58.016206) v4 ==== 2004+0+0 (3454058906 0 0) 0x55f64fcd4600 con 0x55f663219000 > -5103> 2018-07-11 08:05:57.959265 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:58.016206) v4 -- 0x55f67c197000 con 0 > -5102> 2018-07-11 08:05:57.959299 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12651 conn(0x55f66321a800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25 cs=1 l=1). rx osd.2 seq 219 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:05:58.016206) v4 > -5101> 2018-07-11 08:05:57.959320 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.2 10.8.128.106:0/12651 219 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:58.016206) v4 ==== 2004+0+0 (3454058906 0 0) 0x55f67d1afe00 con 0x55f66321a800 > -5100> 2018-07-11 08:05:57.959331 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:58.016206) v4 -- 0x55f66dceb400 con 0 > -5099> 2018-07-11 08:05:58.290997 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.107:0/12617 conn(0x55f64e3c3800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.9 seq 208 0x55f67d193400 osd_ping(ping e509 stamp 2018-07-11 08:05:58.290656) v4 > -5098> 2018-07-11 08:05:58.291003 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.107:0/12617 conn(0x55f64e3c5000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.9 seq 208 0x55f67c196c00 osd_ping(ping e509 stamp 2018-07-11 08:05:58.290656) v4 > -5097> 2018-07-11 08:05:58.291019 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.9 10.8.128.107:0/12617 208 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:58.290656) v4 ==== 2004+0+0 (3114290228 0 0) 0x55f67d193400 con 0x55f64e3c3800 > -5096> 2018-07-11 08:05:58.291029 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:58.290656) v4 -- 0x55f65fa97000 con 0 > -5095> 2018-07-11 08:05:58.291052 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.9 10.8.128.107:0/12617 208 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:58.290656) v4 ==== 2004+0+0 (3114290228 0 0) 0x55f67c196c00 con 0x55f64e3c5000 > -5094> 2018-07-11 08:05:58.291070 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:58.290656) v4 -- 0x55f67c197200 con 0 > -5093> 2018-07-11 08:05:58.621725 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f676e246c0 con 0 > -5092> 2018-07-11 08:05:58.623347 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 254 0x55f67c24cc80 replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:509/504,start:6:a85a463d:::benchmark_data_magna108_35617_object490359:0,end:6:a85b6552:::benchmark_data_magna108_35617_object796787:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -5091> 2018-07-11 08:05:58.623381 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 254 ==== replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:509/504,start:6:a85a463d:::benchmark_data_magna108_35617_object490359:0,end:6:a85b6552:::benchmark_data_magna108_35617_object796787:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (3724090137 0 0) 0x55f67c24cc80 con 0x55f66315e800 > -5090> 2018-07-11 08:05:58.706229 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15287 conn(0x55f66dce1000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.4 seq 213 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:05:59.951239) v4 > -5089> 2018-07-11 08:05:58.706264 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.4 10.8.128.113:0/15287 213 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:59.951239) v4 ==== 2004+0+0 (3788579346 0 0) 0x55f64e20fc00 con 0x55f66dce1000 > -5088> 2018-07-11 08:05:58.706284 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:59.951239) v4 -- 0x55f64e20f600 con 0 > -5087> 2018-07-11 08:05:58.706293 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15287 conn(0x55f66dce2800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.4 seq 213 0x55f64fcd4600 osd_ping(ping e509 stamp 2018-07-11 08:05:59.951239) v4 > -5086> 2018-07-11 08:05:58.706315 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.4 10.8.128.113:0/15287 213 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:59.951239) v4 ==== 2004+0+0 (3788579346 0 0) 0x55f64fcd4600 con 0x55f66dce2800 > -5085> 2018-07-11 08:05:58.706326 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:59.951239) v4 -- 0x55f67c197600 con 0 > -5084> 2018-07-11 08:05:59.179898 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15255 conn(0x55f663ec1000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.0 seq 201 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:06:00.424930) v4 > -5083> 2018-07-11 08:05:59.179928 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.0 10.8.128.113:0/15255 201 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:00.424930) v4 ==== 2004+0+0 (2776685507 0 0) 0x55f67d1afe00 con 0x55f663ec1000 > -5082> 2018-07-11 08:05:59.179949 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:00.424930) v4 -- 0x55f663fb8a00 con 0 > -5081> 2018-07-11 08:05:59.179950 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15255 conn(0x55f663ec2800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.0 seq 201 0x55f6739cd000 osd_ping(ping e509 stamp 2018-07-11 08:06:00.424930) v4 > -5080> 2018-07-11 08:05:59.179964 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.0 10.8.128.113:0/15255 201 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:00.424930) v4 ==== 2004+0+0 (2776685507 0 0) 0x55f6739cd000 con 0x55f663ec2800 > -5079> 2018-07-11 08:05:59.179980 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:00.424930) v4 -- 0x55f65fa97200 con 0 > -5078> 2018-07-11 08:05:59.379857 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed7800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 208 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:05:59.437027) v4 > -5077> 2018-07-11 08:05:59.379896 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.1 10.8.128.106:0/12677 208 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:59.437027) v4 ==== 2004+0+0 (1963281306 0 0) 0x55f64e20fc00 con 0x55f663ed7800 > -5076> 2018-07-11 08:05:59.379908 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:59.437027) v4 -- 0x55f6631cc800 con 0 > -5075> 2018-07-11 08:05:59.379943 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed6000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 208 0x55f67c197200 osd_ping(ping e509 stamp 2018-07-11 08:05:59.437027) v4 > -5074> 2018-07-11 08:05:59.379965 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.1 10.8.128.106:0/12677 208 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:59.437027) v4 ==== 2004+0+0 (1963281306 0 0) 0x55f67c197200 con 0x55f663ed6000 > -5073> 2018-07-11 08:05:59.379977 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:59.437027) v4 -- 0x55f67c197800 con 0 > -5072> 2018-07-11 08:05:59.384411 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f6502d66c0 con 0 > -5071> 2018-07-11 08:05:59.386160 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 255 0x55f67c24cf00 replica scrub(pg: 6.15,from:0'0,to:430'106598,epoch:509/504,start:6:a85b6552:::benchmark_data_magna108_35617_object796787:0,end:6:a85c68b0:::benchmark_data_magna108_35617_object692619:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -5070> 2018-07-11 08:05:59.386190 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 255 ==== replica scrub(pg: 6.15,from:0'0,to:430'106598,epoch:509/504,start:6:a85b6552:::benchmark_data_magna108_35617_object796787:0,end:6:a85c68b0:::benchmark_data_magna108_35617_object692619:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (3934441250 0 0) 0x55f67c24cf00 con 0x55f66315e800 > -5069> 2018-07-11 08:05:59.639646 7f491803c700 5 osd.10 509 heartbeat: osd_stat(789 GB used, 141 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,1]) > -5068> 2018-07-11 08:05:59.639669 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6802/15255 -- osd_ping(ping e509 stamp 2018-07-11 08:05:59.639666) v4 -- 0x55f6841a5200 con 0 > -5067> 2018-07-11 08:05:59.639705 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6803/15255 -- osd_ping(ping e509 stamp 2018-07-11 08:05:59.639666) v4 -- 0x55f6841a5400 con 0 > -5066> 2018-07-11 08:05:59.639723 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6806/12677 -- osd_ping(ping e509 stamp 2018-07-11 08:05:59.639666) v4 -- 0x55f6841a5600 con 0 > -5065> 2018-07-11 08:05:59.639738 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6807/12677 -- osd_ping(ping e509 stamp 2018-07-11 08:05:59.639666) v4 -- 0x55f6841a5800 con 0 > -5064> 2018-07-11 08:05:59.639750 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6808/12651 -- osd_ping(ping e509 stamp 2018-07-11 08:05:59.639666) v4 -- 0x55f6841a5a00 con 0 > -5063> 2018-07-11 08:05:59.639761 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6810/12651 -- osd_ping(ping e509 stamp 2018-07-11 08:05:59.639666) v4 -- 0x55f6841a5c00 con 0 > -5062> 2018-07-11 08:05:59.639773 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6809/12650 -- osd_ping(ping e509 stamp 2018-07-11 08:05:59.639666) v4 -- 0x55f6841a5e00 con 0 > -5061> 2018-07-11 08:05:59.639786 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6811/12650 -- osd_ping(ping e509 stamp 2018-07-11 08:05:59.639666) v4 -- 0x55f673ada000 con 0 > -5060> 2018-07-11 08:05:59.639799 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6810/15287 -- osd_ping(ping e509 stamp 2018-07-11 08:05:59.639666) v4 -- 0x55f673ada200 con 0 > -5059> 2018-07-11 08:05:59.639816 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6811/15287 -- osd_ping(ping e509 stamp 2018-07-11 08:05:59.639666) v4 -- 0x55f673ada400 con 0 > -5058> 2018-07-11 08:05:59.639829 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6806/15263 -- osd_ping(ping e509 stamp 2018-07-11 08:05:59.639666) v4 -- 0x55f673ada600 con 0 > -5057> 2018-07-11 08:05:59.639842 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6807/15263 -- osd_ping(ping e509 stamp 2018-07-11 08:05:59.639666) v4 -- 0x55f673ada800 con 0 > -5056> 2018-07-11 08:05:59.639854 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.115:6802/13323 -- osd_ping(ping e509 stamp 2018-07-11 08:05:59.639666) v4 -- 0x55f673adaa00 con 0 > -5055> 2018-07-11 08:05:59.639866 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.115:6803/13323 -- osd_ping(ping e509 stamp 2018-07-11 08:05:59.639666) v4 -- 0x55f673adac00 con 0 > -5054> 2018-07-11 08:05:59.639881 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6806/15885 -- osd_ping(ping e509 stamp 2018-07-11 08:05:59.639666) v4 -- 0x55f673adae00 con 0 > -5053> 2018-07-11 08:05:59.639893 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6807/15885 -- osd_ping(ping e509 stamp 2018-07-11 08:05:59.639666) v4 -- 0x55f673adb000 con 0 > -5052> 2018-07-11 08:05:59.639905 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.108:6802/11967 -- osd_ping(ping e509 stamp 2018-07-11 08:05:59.639666) v4 -- 0x55f673adb200 con 0 > -5051> 2018-07-11 08:05:59.639917 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.108:6803/11967 -- osd_ping(ping e509 stamp 2018-07-11 08:05:59.639666) v4 -- 0x55f673adb400 con 0 > -5050> 2018-07-11 08:05:59.639930 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.107:6806/12617 -- osd_ping(ping e509 stamp 2018-07-11 08:05:59.639666) v4 -- 0x55f673adb600 con 0 > -5049> 2018-07-11 08:05:59.639945 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.107:6807/12617 -- osd_ping(ping e509 stamp 2018-07-11 08:05:59.639666) v4 -- 0x55f673adb800 con 0 > -5048> 2018-07-11 08:05:59.639958 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6802/15919 -- osd_ping(ping e509 stamp 2018-07-11 08:05:59.639666) v4 -- 0x55f673adba00 con 0 > -5047> 2018-07-11 08:05:59.639970 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6803/15919 -- osd_ping(ping e509 stamp 2018-07-11 08:05:59.639666) v4 -- 0x55f673adbc00 con 0 > -5046> 2018-07-11 08:05:59.640198 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6809/12650 conn(0x55f656ad7000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.3 seq 205 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:59.639666) v4 > -5045> 2018-07-11 08:05:59.640237 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.3 10.8.128.106:6809/12650 205 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:59.639666) v4 ==== 2004+0+0 (2614470983 0 0) 0x55f67d1afe00 con 0x55f656ad7000 > -5044> 2018-07-11 08:05:59.640288 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6807/12677 conn(0x55f64e3d9800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.1 seq 205 0x55f67c197200 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:59.639666) v4 > -5043> 2018-07-11 08:05:59.640231 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6810/12651 conn(0x55f64e3eb800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.2 seq 205 0x55f6739cd000 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:59.639666) v4 > -5042> 2018-07-11 08:05:59.640342 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.1 10.8.128.106:6807/12677 205 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:59.639666) v4 ==== 2004+0+0 (2614470983 0 0) 0x55f67c197200 con 0x55f64e3d9800 > -5041> 2018-07-11 08:05:59.640289 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6806/12677 conn(0x55f64e3d8000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.1 seq 205 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:59.639666) v4 > -5040> 2018-07-11 08:05:59.640352 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.2 10.8.128.106:6810/12651 205 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:59.639666) v4 ==== 2004+0+0 (2614470983 0 0) 0x55f6739cd000 con 0x55f64e3eb800 > -5039> 2018-07-11 08:05:59.640361 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.1 10.8.128.106:6806/12677 205 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:59.639666) v4 ==== 2004+0+0 (2614470983 0 0) 0x55f67d1afe00 con 0x55f64e3d8000 > -5038> 2018-07-11 08:05:59.640392 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.107:6807/12617 conn(0x55f656adf800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.9 seq 205 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:59.639666) v4 > -5037> 2018-07-11 08:05:59.640403 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.9 10.8.128.107:6807/12617 205 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:59.639666) v4 ==== 2004+0+0 (2614470983 0 0) 0x55f67d1afe00 con 0x55f656adf800 > -5036> 2018-07-11 08:05:59.640420 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6802/15255 conn(0x55f656ad4000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 205 0x55f67c197200 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:59.639666) v4 > -5035> 2018-07-11 08:05:59.640439 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6806/15263 conn(0x55f64e3ed000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.5 seq 205 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:59.639666) v4 > -5034> 2018-07-11 08:05:59.640436 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6803/15255 conn(0x55f656ad5800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 205 0x55f6739cd000 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:59.639666) v4 > -5033> 2018-07-11 08:05:59.640452 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.5 10.8.128.113:6806/15263 205 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:59.639666) v4 ==== 2004+0+0 (2614470983 0 0) 0x55f67d1afe00 con 0x55f64e3ed000 > -5032> 2018-07-11 08:05:59.640454 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.0 10.8.128.113:6803/15255 205 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:59.639666) v4 ==== 2004+0+0 (2614470983 0 0) 0x55f6739cd000 con 0x55f656ad5800 > -5031> 2018-07-11 08:05:59.640442 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.0 10.8.128.113:6802/15255 205 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:59.639666) v4 ==== 2004+0+0 (2614470983 0 0) 0x55f67c197200 con 0x55f656ad4000 > -5030> 2018-07-11 08:05:59.640482 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.115:6803/13323 conn(0x55f656ace800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.6 seq 205 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:59.639666) v4 > -5029> 2018-07-11 08:05:59.640514 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.6 10.8.128.115:6803/13323 205 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:59.639666) v4 ==== 2004+0+0 (2614470983 0 0) 0x55f67d1afe00 con 0x55f656ace800 > -5028> 2018-07-11 08:05:59.640510 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6811/15287 conn(0x55f64e402800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.4 seq 205 0x55f6739cd000 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:59.639666) v4 > -5027> 2018-07-11 08:05:59.640525 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.4 10.8.128.113:6811/15287 205 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:59.639666) v4 ==== 2004+0+0 (2614470983 0 0) 0x55f6739cd000 con 0x55f64e402800 > -5026> 2018-07-11 08:05:59.640520 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6811/12650 conn(0x55f656ad8800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.3 seq 205 0x55f67c197200 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:59.639666) v4 > -5025> 2018-07-11 08:05:59.640550 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.108:6802/11967 conn(0x55f656aca000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.8 seq 205 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:59.639666) v4 > -5024> 2018-07-11 08:05:59.640549 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.3 10.8.128.106:6811/12650 205 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:59.639666) v4 ==== 2004+0+0 (2614470983 0 0) 0x55f67c197200 con 0x55f656ad8800 > -5023> 2018-07-11 08:05:59.640564 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.8 10.8.128.108:6802/11967 205 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:59.639666) v4 ==== 2004+0+0 (2614470983 0 0) 0x55f67d1afe00 con 0x55f656aca000 > -5022> 2018-07-11 08:05:59.640562 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.107:6806/12617 conn(0x55f656ade000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.9 seq 205 0x55f6739cd000 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:59.639666) v4 > -5021> 2018-07-11 08:05:59.640576 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.9 10.8.128.107:6806/12617 205 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:59.639666) v4 ==== 2004+0+0 (2614470983 0 0) 0x55f6739cd000 con 0x55f656ade000 > -5020> 2018-07-11 08:05:59.640594 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6803/15919 conn(0x55f64e3dc800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.11 seq 205 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:59.639666) v4 > -5019> 2018-07-11 08:05:59.640606 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.11 10.8.128.114:6803/15919 205 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:59.639666) v4 ==== 2004+0+0 (2614470983 0 0) 0x55f67d1afe00 con 0x55f64e3dc800 > -5018> 2018-07-11 08:05:59.640602 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6810/15287 conn(0x55f64e401000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.4 seq 205 0x55f67c197200 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:59.639666) v4 > -5017> 2018-07-11 08:05:59.640622 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.4 10.8.128.113:6810/15287 205 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:59.639666) v4 ==== 2004+0+0 (2614470983 0 0) 0x55f67c197200 con 0x55f64e401000 > -5016> 2018-07-11 08:05:59.640623 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.115:6802/13323 conn(0x55f656acd000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.6 seq 205 0x55f6739cd000 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:59.639666) v4 > -5015> 2018-07-11 08:05:59.640634 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6807/15885 conn(0x55f64e3ff800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.7 seq 205 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:59.639666) v4 > -5014> 2018-07-11 08:05:59.640638 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.6 10.8.128.115:6802/13323 205 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:59.639666) v4 ==== 2004+0+0 (2614470983 0 0) 0x55f6739cd000 con 0x55f656acd000 > -5013> 2018-07-11 08:05:59.640646 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.7 10.8.128.114:6807/15885 205 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:59.639666) v4 ==== 2004+0+0 (2614470983 0 0) 0x55f67d1afe00 con 0x55f64e3ff800 > -5012> 2018-07-11 08:05:59.640666 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6808/12651 conn(0x55f64e3ea000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.2 seq 205 0x55f67c197200 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:59.639666) v4 > -5011> 2018-07-11 08:05:59.640676 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6806/15885 conn(0x55f64e3fe000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.7 seq 205 0x55f6739cd000 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:59.639666) v4 > -5010> 2018-07-11 08:05:59.640685 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.2 10.8.128.106:6808/12651 205 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:59.639666) v4 ==== 2004+0+0 (2614470983 0 0) 0x55f67c197200 con 0x55f64e3ea000 > -5009> 2018-07-11 08:05:59.640694 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.7 10.8.128.114:6806/15885 205 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:59.639666) v4 ==== 2004+0+0 (2614470983 0 0) 0x55f6739cd000 con 0x55f64e3fe000 > -5008> 2018-07-11 08:05:59.640726 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6807/15263 conn(0x55f64e3ee800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.5 seq 205 0x55f67c197200 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:59.639666) v4 > -5007> 2018-07-11 08:05:59.640739 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6802/15919 conn(0x55f64e3db000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.11 seq 205 0x55f6739cd000 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:59.639666) v4 > -5006> 2018-07-11 08:05:59.640744 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.5 10.8.128.113:6807/15263 205 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:59.639666) v4 ==== 2004+0+0 (2614470983 0 0) 0x55f67c197200 con 0x55f64e3ee800 > -5005> 2018-07-11 08:05:59.640760 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.11 10.8.128.114:6802/15919 205 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:59.639666) v4 ==== 2004+0+0 (2614470983 0 0) 0x55f6739cd000 con 0x55f64e3db000 > -5004> 2018-07-11 08:05:59.640801 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.108:6803/11967 conn(0x55f656acb800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.8 seq 205 0x55f67c197200 osd_ping(ping_reply e509 stamp 2018-07-11 08:05:59.639666) v4 > -5003> 2018-07-11 08:05:59.640820 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.8 10.8.128.108:6803/11967 205 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:05:59.639666) v4 ==== 2004+0+0 (2614470983 0 0) 0x55f67c197200 con 0x55f656acb800 > -5002> 2018-07-11 08:05:59.902743 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.108:0/11967 conn(0x55f663f6f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=1). rx osd.8 seq 201 0x55f6739cd000 osd_ping(ping e509 stamp 2018-07-11 08:05:59.662833) v4 > -5001> 2018-07-11 08:05:59.902767 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.8 10.8.128.108:0/11967 201 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:59.662833) v4 ==== 2004+0+0 (2700694438 0 0) 0x55f6739cd000 con 0x55f663f6f000 > -5000> 2018-07-11 08:05:59.902781 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:59.662833) v4 -- 0x55f6841a5400 con 0 > -4999> 2018-07-11 08:05:59.902822 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.108:0/11967 conn(0x55f663f70800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=1). rx osd.8 seq 201 0x55f67c197200 osd_ping(ping e509 stamp 2018-07-11 08:05:59.662833) v4 > -4998> 2018-07-11 08:05:59.902843 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.8 10.8.128.108:0/11967 201 ==== osd_ping(ping e509 stamp 2018-07-11 08:05:59.662833) v4 ==== 2004+0+0 (2700694438 0 0) 0x55f67c197200 con 0x55f663f70800 > -4997> 2018-07-11 08:05:59.902859 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:05:59.662833) v4 -- 0x55f6841a5200 con 0 > -4996> 2018-07-11 08:06:00.140091 7f491803c700 5 osd.10 509 heartbeat: osd_stat(789 GB used, 141 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,0,0,1]) > -4995> 2018-07-11 08:06:00.140109 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6802/15255 -- osd_ping(ping e509 stamp 2018-07-11 08:06:00.140107) v4 -- 0x55f673adbe00 con 0 > -4994> 2018-07-11 08:06:00.140140 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6803/15255 -- osd_ping(ping e509 stamp 2018-07-11 08:06:00.140107) v4 -- 0x55f676628000 con 0 > -4993> 2018-07-11 08:06:00.140157 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6806/12677 -- osd_ping(ping e509 stamp 2018-07-11 08:06:00.140107) v4 -- 0x55f676628200 con 0 > -4992> 2018-07-11 08:06:00.140171 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6807/12677 -- osd_ping(ping e509 stamp 2018-07-11 08:06:00.140107) v4 -- 0x55f676628400 con 0 > -4991> 2018-07-11 08:06:00.140183 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6808/12651 -- osd_ping(ping e509 stamp 2018-07-11 08:06:00.140107) v4 -- 0x55f676628600 con 0 > -4990> 2018-07-11 08:06:00.140194 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6810/12651 -- osd_ping(ping e509 stamp 2018-07-11 08:06:00.140107) v4 -- 0x55f676628800 con 0 > -4989> 2018-07-11 08:06:00.140205 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6809/12650 -- osd_ping(ping e509 stamp 2018-07-11 08:06:00.140107) v4 -- 0x55f676628a00 con 0 > -4988> 2018-07-11 08:06:00.140217 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6811/12650 -- osd_ping(ping e509 stamp 2018-07-11 08:06:00.140107) v4 -- 0x55f676628c00 con 0 > -4987> 2018-07-11 08:06:00.140234 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6810/15287 -- osd_ping(ping e509 stamp 2018-07-11 08:06:00.140107) v4 -- 0x55f676628e00 con 0 > -4986> 2018-07-11 08:06:00.140265 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6811/15287 -- osd_ping(ping e509 stamp 2018-07-11 08:06:00.140107) v4 -- 0x55f676629000 con 0 > -4985> 2018-07-11 08:06:00.140322 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6806/15263 -- osd_ping(ping e509 stamp 2018-07-11 08:06:00.140107) v4 -- 0x55f676629200 con 0 > -4984> 2018-07-11 08:06:00.140342 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6807/15263 -- osd_ping(ping e509 stamp 2018-07-11 08:06:00.140107) v4 -- 0x55f676629400 con 0 > -4983> 2018-07-11 08:06:00.140355 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.115:6802/13323 -- osd_ping(ping e509 stamp 2018-07-11 08:06:00.140107) v4 -- 0x55f676629600 con 0 > -4982> 2018-07-11 08:06:00.140368 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.115:6803/13323 -- osd_ping(ping e509 stamp 2018-07-11 08:06:00.140107) v4 -- 0x55f676629800 con 0 > -4981> 2018-07-11 08:06:00.140381 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6806/15885 -- osd_ping(ping e509 stamp 2018-07-11 08:06:00.140107) v4 -- 0x55f676629a00 con 0 > -4980> 2018-07-11 08:06:00.140397 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6807/15885 -- osd_ping(ping e509 stamp 2018-07-11 08:06:00.140107) v4 -- 0x55f676629c00 con 0 > -4979> 2018-07-11 08:06:00.140409 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.108:6802/11967 -- osd_ping(ping e509 stamp 2018-07-11 08:06:00.140107) v4 -- 0x55f676629e00 con 0 > -4978> 2018-07-11 08:06:00.140420 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.108:6803/11967 -- osd_ping(ping e509 stamp 2018-07-11 08:06:00.140107) v4 -- 0x55f67662a000 con 0 > -4977> 2018-07-11 08:06:00.140432 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.107:6806/12617 -- osd_ping(ping e509 stamp 2018-07-11 08:06:00.140107) v4 -- 0x55f67662a200 con 0 > -4976> 2018-07-11 08:06:00.140445 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.107:6807/12617 -- osd_ping(ping e509 stamp 2018-07-11 08:06:00.140107) v4 -- 0x55f67662a400 con 0 > -4975> 2018-07-11 08:06:00.140458 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6802/15919 -- osd_ping(ping e509 stamp 2018-07-11 08:06:00.140107) v4 -- 0x55f67662a600 con 0 > -4974> 2018-07-11 08:06:00.140469 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6803/15919 -- osd_ping(ping e509 stamp 2018-07-11 08:06:00.140107) v4 -- 0x55f67662a800 con 0 > -4973> 2018-07-11 08:06:00.140567 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6807/12677 conn(0x55f64e3d9800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.1 seq 206 0x55f64fcd4600 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:00.140107) v4 > -4972> 2018-07-11 08:06:00.140588 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.1 10.8.128.106:6807/12677 206 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:00.140107) v4 ==== 2004+0+0 (4087010378 0 0) 0x55f64fcd4600 con 0x55f64e3d9800 > -4971> 2018-07-11 08:06:00.140635 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6811/12650 conn(0x55f656ad8800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.3 seq 206 0x55f64fcd4600 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:00.140107) v4 > -4970> 2018-07-11 08:06:00.140654 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.3 10.8.128.106:6811/12650 206 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:00.140107) v4 ==== 2004+0+0 (4087010378 0 0) 0x55f64fcd4600 con 0x55f656ad8800 > -4969> 2018-07-11 08:06:00.140670 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6806/12677 conn(0x55f64e3d8000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.1 seq 206 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:00.140107) v4 > -4968> 2018-07-11 08:06:00.140641 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6803/15255 conn(0x55f656ad5800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 206 0x55f6739cd000 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:00.140107) v4 > -4967> 2018-07-11 08:06:00.140703 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6802/15255 conn(0x55f656ad4000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 206 0x55f64fcd4600 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:00.140107) v4 > -4966> 2018-07-11 08:06:00.140716 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.0 10.8.128.113:6803/15255 206 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:00.140107) v4 ==== 2004+0+0 (4087010378 0 0) 0x55f6739cd000 con 0x55f656ad5800 > -4965> 2018-07-11 08:06:00.140691 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.1 10.8.128.106:6806/12677 206 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:00.140107) v4 ==== 2004+0+0 (4087010378 0 0) 0x55f67d1afe00 con 0x55f64e3d8000 > -4964> 2018-07-11 08:06:00.140720 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.0 10.8.128.113:6802/15255 206 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:00.140107) v4 ==== 2004+0+0 (4087010378 0 0) 0x55f64fcd4600 con 0x55f656ad4000 > -4963> 2018-07-11 08:06:00.140759 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.107:6806/12617 conn(0x55f656ade000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.9 seq 206 0x55f6739cd000 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:00.140107) v4 > -4962> 2018-07-11 08:06:00.140777 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.9 10.8.128.107:6806/12617 206 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:00.140107) v4 ==== 2004+0+0 (4087010378 0 0) 0x55f6739cd000 con 0x55f656ade000 > -4961> 2018-07-11 08:06:00.140782 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6808/12651 conn(0x55f64e3ea000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.2 seq 206 0x55f64fcd4600 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:00.140107) v4 > -4960> 2018-07-11 08:06:00.140773 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.107:6807/12617 conn(0x55f656adf800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.9 seq 206 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:00.140107) v4 > -4959> 2018-07-11 08:06:00.140796 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.2 10.8.128.106:6808/12651 206 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:00.140107) v4 ==== 2004+0+0 (4087010378 0 0) 0x55f64fcd4600 con 0x55f64e3ea000 > -4958> 2018-07-11 08:06:00.140805 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.9 10.8.128.107:6807/12617 206 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:00.140107) v4 ==== 2004+0+0 (4087010378 0 0) 0x55f67d1afe00 con 0x55f656adf800 > -4957> 2018-07-11 08:06:00.140849 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6809/12650 conn(0x55f656ad7000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.3 seq 206 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:00.140107) v4 > -4956> 2018-07-11 08:06:00.140851 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6810/15287 conn(0x55f64e401000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.4 seq 206 0x55f64fcd4600 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:00.140107) v4 > -4955> 2018-07-11 08:06:00.140861 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6811/15287 conn(0x55f64e402800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.4 seq 206 0x55f6739cd000 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:00.140107) v4 > -4954> 2018-07-11 08:06:00.140874 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.3 10.8.128.106:6809/12650 206 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:00.140107) v4 ==== 2004+0+0 (4087010378 0 0) 0x55f67d1afe00 con 0x55f656ad7000 > -4953> 2018-07-11 08:06:00.140878 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.4 10.8.128.113:6811/15287 206 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:00.140107) v4 ==== 2004+0+0 (4087010378 0 0) 0x55f6739cd000 con 0x55f64e402800 > -4952> 2018-07-11 08:06:00.140878 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.4 10.8.128.113:6810/15287 206 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:00.140107) v4 ==== 2004+0+0 (4087010378 0 0) 0x55f64fcd4600 con 0x55f64e401000 > -4951> 2018-07-11 08:06:00.140924 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6810/12651 conn(0x55f64e3eb800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.2 seq 206 0x55f6739cd000 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:00.140107) v4 > -4950> 2018-07-11 08:06:00.140937 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6807/15263 conn(0x55f64e3ee800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.5 seq 206 0x55f64fcd4600 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:00.140107) v4 > -4949> 2018-07-11 08:06:00.140923 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6806/15263 conn(0x55f64e3ed000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.5 seq 206 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:00.140107) v4 > -4948> 2018-07-11 08:06:00.140948 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.2 10.8.128.106:6810/12651 206 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:00.140107) v4 ==== 2004+0+0 (4087010378 0 0) 0x55f6739cd000 con 0x55f64e3eb800 > -4947> 2018-07-11 08:06:00.140955 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.5 10.8.128.113:6807/15263 206 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:00.140107) v4 ==== 2004+0+0 (4087010378 0 0) 0x55f64fcd4600 con 0x55f64e3ee800 > -4946> 2018-07-11 08:06:00.140959 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.5 10.8.128.113:6806/15263 206 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:00.140107) v4 ==== 2004+0+0 (4087010378 0 0) 0x55f67d1afe00 con 0x55f64e3ed000 > -4945> 2018-07-11 08:06:00.140992 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.115:6802/13323 conn(0x55f656acd000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.6 seq 206 0x55f6739cd000 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:00.140107) v4 > -4944> 2018-07-11 08:06:00.141010 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.6 10.8.128.115:6802/13323 206 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:00.140107) v4 ==== 2004+0+0 (4087010378 0 0) 0x55f6739cd000 con 0x55f656acd000 > -4943> 2018-07-11 08:06:00.141011 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.115:6803/13323 conn(0x55f656ace800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.6 seq 206 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:00.140107) v4 > -4942> 2018-07-11 08:06:00.141025 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.6 10.8.128.115:6803/13323 206 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:00.140107) v4 ==== 2004+0+0 (4087010378 0 0) 0x55f67d1afe00 con 0x55f656ace800 > -4941> 2018-07-11 08:06:00.141054 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6806/15885 conn(0x55f64e3fe000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.7 seq 206 0x55f6739cd000 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:00.140107) v4 > -4940> 2018-07-11 08:06:00.141065 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.108:6802/11967 conn(0x55f656aca000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.8 seq 206 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:00.140107) v4 > -4939> 2018-07-11 08:06:00.141072 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.7 10.8.128.114:6806/15885 206 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:00.140107) v4 ==== 2004+0+0 (4087010378 0 0) 0x55f6739cd000 con 0x55f64e3fe000 > -4938> 2018-07-11 08:06:00.141080 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.8 10.8.128.108:6802/11967 206 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:00.140107) v4 ==== 2004+0+0 (4087010378 0 0) 0x55f67d1afe00 con 0x55f656aca000 > -4937> 2018-07-11 08:06:00.141073 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.108:6803/11967 conn(0x55f656acb800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.8 seq 206 0x55f64fcd4600 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:00.140107) v4 > -4936> 2018-07-11 08:06:00.141091 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.8 10.8.128.108:6803/11967 206 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:00.140107) v4 ==== 2004+0+0 (4087010378 0 0) 0x55f64fcd4600 con 0x55f656acb800 > -4935> 2018-07-11 08:06:00.141119 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6807/15885 conn(0x55f64e3ff800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.7 seq 206 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:00.140107) v4 > -4934> 2018-07-11 08:06:00.141121 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6802/15919 conn(0x55f64e3db000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.11 seq 206 0x55f6739cd000 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:00.140107) v4 > -4933> 2018-07-11 08:06:00.141133 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.7 10.8.128.114:6807/15885 206 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:00.140107) v4 ==== 2004+0+0 (4087010378 0 0) 0x55f67d1afe00 con 0x55f64e3ff800 > -4932> 2018-07-11 08:06:00.141137 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.11 10.8.128.114:6802/15919 206 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:00.140107) v4 ==== 2004+0+0 (4087010378 0 0) 0x55f6739cd000 con 0x55f64e3db000 > -4931> 2018-07-11 08:06:00.141166 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6803/15919 conn(0x55f64e3dc800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.11 seq 206 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:00.140107) v4 > -4930> 2018-07-11 08:06:00.141180 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.11 10.8.128.114:6803/15919 206 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:00.140107) v4 ==== 2004+0+0 (4087010378 0 0) 0x55f67d1afe00 con 0x55f64e3dc800 > -4929> 2018-07-11 08:06:00.155297 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f65ef7f680 con 0 > -4928> 2018-07-11 08:06:00.156828 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 256 0x55f67c24d180 replica scrub(pg: 6.15,from:0'0,to:430'106030,epoch:509/504,start:6:a85c68b0:::benchmark_data_magna108_35617_object692619:0,end:6:a85d3b27:::benchmark_data_magna108_35617_object68312:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -4927> 2018-07-11 08:06:00.156852 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 256 ==== replica scrub(pg: 6.15,from:0'0,to:430'106030,epoch:509/504,start:6:a85c68b0:::benchmark_data_magna108_35617_object692619:0,end:6:a85d3b27:::benchmark_data_magna108_35617_object68312:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 217+0+0 (646557311 0 0) 0x55f67c24d180 con 0x55f66315e800 > -4926> 2018-07-11 08:06:00.237356 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15885 conn(0x55f663216000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 211 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:06:00.873160) v4 > -4925> 2018-07-11 08:06:00.237378 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.7 10.8.128.114:0/15885 211 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:00.873160) v4 ==== 2004+0+0 (213113363 0 0) 0x55f67d1afe00 con 0x55f663216000 > -4924> 2018-07-11 08:06:00.237365 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15885 conn(0x55f663217800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 211 0x55f6739cd000 osd_ping(ping e509 stamp 2018-07-11 08:06:00.873160) v4 > -4923> 2018-07-11 08:06:00.237394 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:00.873160) v4 -- 0x55f67662a400 con 0 > -4922> 2018-07-11 08:06:00.237402 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.7 10.8.128.114:0/15885 211 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:00.873160) v4 ==== 2004+0+0 (213113363 0 0) 0x55f6739cd000 con 0x55f663217800 > -4921> 2018-07-11 08:06:00.237421 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:00.873160) v4 -- 0x55f676629a00 con 0 > -4920> 2018-07-11 08:06:00.307863 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0a000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 207 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:06:00.943705) v4 > -4919> 2018-07-11 08:06:00.307884 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.11 10.8.128.114:0/15919 207 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:00.943705) v4 ==== 2004+0+0 (4032627409 0 0) 0x55f64e20fc00 con 0x55f663f0a000 > -4918> 2018-07-11 08:06:00.307900 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:00.943705) v4 -- 0x55f676629e00 con 0 > -4917> 2018-07-11 08:06:00.307895 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0b800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 207 0x55f67d193400 osd_ping(ping e509 stamp 2018-07-11 08:06:00.943705) v4 > -4916> 2018-07-11 08:06:00.307924 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.11 10.8.128.114:0/15919 207 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:00.943705) v4 ==== 2004+0+0 (4032627409 0 0) 0x55f67d193400 con 0x55f663f0b800 > -4915> 2018-07-11 08:06:00.307939 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:00.943705) v4 -- 0x55f676629600 con 0 > -4914> 2018-07-11 08:06:00.433932 7f491e048700 10 monclient: tick > -4913> 2018-07-11 08:06:00.433946 7f491e048700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2018-07-11 08:05:30.433945) > -4912> 2018-07-11 08:06:00.530607 7f491d046700 1 -- 10.8.128.107:6800/14173 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55f66398fb80 con 0 > -4911> 2018-07-11 08:06:00.530669 7f491d046700 1 -- 10.8.128.107:6800/14173 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55f66398e840 con 0 > -4910> 2018-07-11 08:06:00.591529 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.107:0/12617 conn(0x55f64e3c3800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.9 seq 209 0x55f6739cd000 osd_ping(ping e509 stamp 2018-07-11 08:06:00.591144) v4 > -4909> 2018-07-11 08:06:00.591532 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.107:0/12617 conn(0x55f64e3c5000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.9 seq 209 0x55f67c197200 osd_ping(ping e509 stamp 2018-07-11 08:06:00.591144) v4 > -4908> 2018-07-11 08:06:00.591556 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.9 10.8.128.107:0/12617 209 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:00.591144) v4 ==== 2004+0+0 (3158665578 0 0) 0x55f67c197200 con 0x55f64e3c5000 > -4907> 2018-07-11 08:06:00.591554 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.9 10.8.128.107:0/12617 209 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:00.591144) v4 ==== 2004+0+0 (3158665578 0 0) 0x55f6739cd000 con 0x55f64e3c3800 > -4906> 2018-07-11 08:06:00.591578 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:00.591144) v4 -- 0x55f676628c00 con 0 > -4905> 2018-07-11 08:06:00.591612 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:00.591144) v4 -- 0x55f676629000 con 0 > -4904> 2018-07-11 08:06:00.734305 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12650 conn(0x55f663eda800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.3 seq 208 0x55f64fcd4600 osd_ping(ping e509 stamp 2018-07-11 08:06:00.791507) v4 > -4903> 2018-07-11 08:06:00.734325 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.3 10.8.128.106:0/12650 208 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:00.791507) v4 ==== 2004+0+0 (3569007370 0 0) 0x55f64fcd4600 con 0x55f663eda800 > -4902> 2018-07-11 08:06:00.734339 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:00.791507) v4 -- 0x55f676628600 con 0 > -4901> 2018-07-11 08:06:00.734422 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12650 conn(0x55f663ed9000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.3 seq 208 0x55f67d193400 osd_ping(ping e509 stamp 2018-07-11 08:06:00.791507) v4 > -4900> 2018-07-11 08:06:00.734443 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.3 10.8.128.106:0/12650 208 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:00.791507) v4 ==== 2004+0+0 (3569007370 0 0) 0x55f67d193400 con 0x55f663ed9000 > -4899> 2018-07-11 08:06:00.734456 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:00.791507) v4 -- 0x55f676628800 con 0 > -4898> 2018-07-11 08:06:00.970851 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 455 0x55f6739cd000 MOSDScrubReserve(11.3s5 REQUEST e509) v1 > -4897> 2018-07-11 08:06:00.970882 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 455 ==== MOSDScrubReserve(11.3s5 REQUEST e509) v1 ==== 43+0+0 (433425070 0 0) 0x55f6739cd000 con 0x55f668a29000 > -4896> 2018-07-11 08:06:00.970928 7f491a040700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e509) v1 -- 0x55f65fa96200 con 0 > -4895> 2018-07-11 08:06:00.971343 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 456 0x55f65fa96200 MOSDScrubReserve(11.3s5 RELEASE e509) v1 > -4894> 2018-07-11 08:06:00.971358 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 456 ==== MOSDScrubReserve(11.3s5 RELEASE e509) v1 ==== 43+0+0 (3603460268 0 0) 0x55f65fa96200 con 0x55f668a29000 > -4893> 2018-07-11 08:06:01.213139 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.115:0/13323 conn(0x55f663ebf800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.6 seq 196 0x55f67c197200 osd_ping(ping e509 stamp 2018-07-11 08:06:01.566053) v4 > -4892> 2018-07-11 08:06:01.213161 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.6 10.8.128.115:0/13323 196 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:01.566053) v4 ==== 2004+0+0 (301771911 0 0) 0x55f67c197200 con 0x55f663ebf800 > -4891> 2018-07-11 08:06:01.213172 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:01.566053) v4 -- 0x55f676628600 con 0 > -4890> 2018-07-11 08:06:01.213225 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.115:0/13323 conn(0x55f663ebe000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.6 seq 196 0x55f67d193400 osd_ping(ping e509 stamp 2018-07-11 08:06:01.566053) v4 > -4889> 2018-07-11 08:06:01.213244 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.6 10.8.128.115:0/13323 196 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:01.566053) v4 ==== 2004+0+0 (301771911 0 0) 0x55f67d193400 con 0x55f663ebe000 > -4888> 2018-07-11 08:06:01.213261 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:01.566053) v4 -- 0x55f6841a5400 con 0 > -4887> 2018-07-11 08:06:01.616068 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 30] Generated table #37047: 2638240 keys, 74546053 bytes > -4886> 2018-07-11 08:06:01.616105 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296361616089, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 37047, "file_size": 74546053, "table_properties": {"data_size": 67112078, "index_size": 689804, "filter_size": 6743181, "raw_key_size": 76215827, "raw_average_key_size": 28, "raw_value_size": 31365747, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638240, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} > -4885> 2018-07-11 08:06:01.851018 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f66810d440 con 0 > -4884> 2018-07-11 08:06:01.852688 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 257 0x55f67c24d400 replica scrub(pg: 6.15,from:0'0,to:430'105913,epoch:509/504,start:6:a85d3b27:::benchmark_data_magna108_35617_object68312:0,end:6:a85e3be4:::benchmark_data_magna108_35617_object296818:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -4883> 2018-07-11 08:06:01.852722 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 257 ==== replica scrub(pg: 6.15,from:0'0,to:430'105913,epoch:509/504,start:6:a85d3b27:::benchmark_data_magna108_35617_object68312:0,end:6:a85e3be4:::benchmark_data_magna108_35617_object296818:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 217+0+0 (2468035577 0 0) 0x55f67c24d400 con 0x55f66315e800 > -4882> 2018-07-11 08:06:01.971167 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 457 0x55f676628800 MOSDScrubReserve(11.3s5 REQUEST e509) v1 > -4881> 2018-07-11 08:06:01.971188 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 457 ==== MOSDScrubReserve(11.3s5 REQUEST e509) v1 ==== 43+0+0 (433425070 0 0) 0x55f676628800 con 0x55f668a29000 > -4880> 2018-07-11 08:06:01.971262 7f491a040700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e509) v1 -- 0x55f65fa96200 con 0 > -4879> 2018-07-11 08:06:01.971575 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 458 0x55f6841a5400 MOSDScrubReserve(11.3s5 RELEASE e509) v1 > -4878> 2018-07-11 08:06:01.971606 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 458 ==== MOSDScrubReserve(11.3s5 RELEASE e509) v1 ==== 43+0+0 (3603460268 0 0) 0x55f6841a5400 con 0x55f668a29000 > -4877> 2018-07-11 08:06:02.008213 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0a000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 208 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:06:02.644066) v4 > -4876> 2018-07-11 08:06:02.008243 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.11 10.8.128.114:0/15919 208 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:02.644066) v4 ==== 2004+0+0 (3330153661 0 0) 0x55f67d1afe00 con 0x55f663f0a000 > -4875> 2018-07-11 08:06:02.008260 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:02.644066) v4 -- 0x55f676629c00 con 0 > -4874> 2018-07-11 08:06:02.008270 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0b800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 208 0x55f67d193400 osd_ping(ping e509 stamp 2018-07-11 08:06:02.644066) v4 > -4873> 2018-07-11 08:06:02.008291 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.11 10.8.128.114:0/15919 208 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:02.644066) v4 ==== 2004+0+0 (3330153661 0 0) 0x55f67d193400 con 0x55f663f0b800 > -4872> 2018-07-11 08:06:02.008303 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:02.644066) v4 -- 0x55f65fa97600 con 0 > -4871> 2018-07-11 08:06:02.292113 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.107:0/12617 conn(0x55f64e3c3800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.9 seq 210 0x55f65fa96200 osd_ping(ping e509 stamp 2018-07-11 08:06:02.291661) v4 > -4870> 2018-07-11 08:06:02.292157 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.107:0/12617 conn(0x55f64e3c5000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.9 seq 210 0x55f64fcd4600 osd_ping(ping e509 stamp 2018-07-11 08:06:02.291661) v4 > -4869> 2018-07-11 08:06:02.292164 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.9 10.8.128.107:0/12617 210 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:02.291661) v4 ==== 2004+0+0 (3460437997 0 0) 0x55f65fa96200 con 0x55f64e3c3800 > -4868> 2018-07-11 08:06:02.292185 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.9 10.8.128.107:0/12617 210 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:02.291661) v4 ==== 2004+0+0 (3460437997 0 0) 0x55f64fcd4600 con 0x55f64e3c5000 > -4867> 2018-07-11 08:06:02.292192 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:02.291661) v4 -- 0x55f65fa97800 con 0 > -4866> 2018-07-11 08:06:02.292237 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:02.291661) v4 -- 0x55f673adbe00 con 0 > -4865> 2018-07-11 08:06:02.508665 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0a000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 209 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:06:03.144458) v4 > -4864> 2018-07-11 08:06:02.508675 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0b800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 209 0x55f67d193400 osd_ping(ping e509 stamp 2018-07-11 08:06:03.144458) v4 > -4863> 2018-07-11 08:06:02.508687 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.11 10.8.128.114:0/15919 209 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:03.144458) v4 ==== 2004+0+0 (545255353 0 0) 0x55f64e20fc00 con 0x55f663f0a000 > -4862> 2018-07-11 08:06:02.508703 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.11 10.8.128.114:0/15919 209 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:03.144458) v4 ==== 2004+0+0 (545255353 0 0) 0x55f67d193400 con 0x55f663f0b800 > -4861> 2018-07-11 08:06:02.508711 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:03.144458) v4 -- 0x55f676629800 con 0 > -4860> 2018-07-11 08:06:02.508738 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:03.144458) v4 -- 0x55f65fa97a00 con 0 > -4859> 2018-07-11 08:06:02.650507 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f666947d40 con 0 > -4858> 2018-07-11 08:06:02.652009 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 258 0x55f67c24d680 replica scrub(pg: 6.15,from:0'0,to:430'106210,epoch:509/504,start:6:a85e3be4:::benchmark_data_magna108_35617_object296818:0,end:6:a85f1162:::benchmark_data_magna108_35617_object1107853:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -4857> 2018-07-11 08:06:02.652039 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 258 ==== replica scrub(pg: 6.15,from:0'0,to:430'106210,epoch:509/504,start:6:a85e3be4:::benchmark_data_magna108_35617_object296818:0,end:6:a85f1162:::benchmark_data_magna108_35617_object1107853:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 219+0+0 (3475280427 0 0) 0x55f67c24d680 con 0x55f66315e800 > -4856> 2018-07-11 08:06:02.659701 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12651 conn(0x55f663219000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25 cs=1 l=1). rx osd.2 seq 220 0x55f676628600 osd_ping(ping e509 stamp 2018-07-11 08:06:02.716782) v4 > -4855> 2018-07-11 08:06:02.659722 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.2 10.8.128.106:0/12651 220 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:02.716782) v4 ==== 2004+0+0 (2499032878 0 0) 0x55f676628600 con 0x55f663219000 > -4854> 2018-07-11 08:06:02.659732 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:02.716782) v4 -- 0x55f6841a5200 con 0 > -4853> 2018-07-11 08:06:02.659757 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12651 conn(0x55f66321a800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25 cs=1 l=1). rx osd.2 seq 220 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:06:02.716782) v4 > -4852> 2018-07-11 08:06:02.659780 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.2 10.8.128.106:0/12651 220 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:02.716782) v4 ==== 2004+0+0 (2499032878 0 0) 0x55f67d1afe00 con 0x55f66321a800 > -4851> 2018-07-11 08:06:02.659797 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:02.716782) v4 -- 0x55f676629800 con 0 > -4850> 2018-07-11 08:06:02.760330 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0e800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 217 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:06:04.005513) v4 > -4849> 2018-07-11 08:06:02.760354 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.5 10.8.128.113:0/15263 217 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:04.005513) v4 ==== 2004+0+0 (1470046075 0 0) 0x55f64e20fc00 con 0x55f663f0e800 > -4848> 2018-07-11 08:06:02.760372 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:04.005513) v4 -- 0x55f676629200 con 0 > -4847> 2018-07-11 08:06:02.760370 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0d000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 217 0x55f673adbe00 osd_ping(ping e509 stamp 2018-07-11 08:06:04.005513) v4 > -4846> 2018-07-11 08:06:02.760387 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.5 10.8.128.113:0/15263 217 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:04.005513) v4 ==== 2004+0+0 (1470046075 0 0) 0x55f673adbe00 con 0x55f663f0d000 > -4845> 2018-07-11 08:06:02.760400 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:04.005513) v4 -- 0x55f67c197800 con 0 > -4844> 2018-07-11 08:06:02.806573 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15287 conn(0x55f66dce2800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.4 seq 214 0x55f676628600 osd_ping(ping e509 stamp 2018-07-11 08:06:04.051767) v4 > -4843> 2018-07-11 08:06:02.806596 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.4 10.8.128.113:0/15287 214 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:04.051767) v4 ==== 2004+0+0 (2785513111 0 0) 0x55f676628600 con 0x55f66dce2800 > -4842> 2018-07-11 08:06:02.806606 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:04.051767) v4 -- 0x55f67c197a00 con 0 > -4841> 2018-07-11 08:06:02.806651 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15287 conn(0x55f66dce1000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.4 seq 214 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:06:04.051767) v4 > -4840> 2018-07-11 08:06:02.806673 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.4 10.8.128.113:0/15287 214 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:04.051767) v4 ==== 2004+0+0 (2785513111 0 0) 0x55f67d1afe00 con 0x55f66dce1000 > -4839> 2018-07-11 08:06:02.806685 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:04.051767) v4 -- 0x55f676628a00 con 0 > -4838> 2018-07-11 08:06:02.880279 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed6000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 209 0x55f673adbe00 osd_ping(ping e509 stamp 2018-07-11 08:06:02.937433) v4 > -4837> 2018-07-11 08:06:02.880300 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.1 10.8.128.106:0/12677 209 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:02.937433) v4 ==== 2004+0+0 (3024831741 0 0) 0x55f673adbe00 con 0x55f663ed6000 > -4836> 2018-07-11 08:06:02.880310 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:02.937433) v4 -- 0x55f67c197c00 con 0 > -4835> 2018-07-11 08:06:02.880294 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed7800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 209 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:06:02.937433) v4 > -4834> 2018-07-11 08:06:02.880317 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.1 10.8.128.106:0/12677 209 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:02.937433) v4 ==== 2004+0+0 (3024831741 0 0) 0x55f64e20fc00 con 0x55f663ed7800 > -4833> 2018-07-11 08:06:02.880330 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:02.937433) v4 -- 0x55f676628200 con 0 > -4832> 2018-07-11 08:06:02.971391 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 459 0x55f65fa96200 MOSDScrubReserve(11.3s5 REQUEST e509) v1 > -4831> 2018-07-11 08:06:02.971412 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 459 ==== MOSDScrubReserve(11.3s5 REQUEST e509) v1 ==== 43+0+0 (433425070 0 0) 0x55f65fa96200 con 0x55f668a29000 > -4830> 2018-07-11 08:06:02.971460 7f491a040700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e509) v1 -- 0x55f6841a5400 con 0 > -4829> 2018-07-11 08:06:02.971791 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 460 0x55f65fa97c00 MOSDScrubReserve(11.3s5 RELEASE e509) v1 > -4828> 2018-07-11 08:06:02.971822 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 460 ==== MOSDScrubReserve(11.3s5 RELEASE e509) v1 ==== 43+0+0 (3603460268 0 0) 0x55f65fa97c00 con 0x55f668a29000 > -4827> 2018-07-11 08:06:03.040610 7f491803c700 5 osd.10 509 heartbeat: osd_stat(790 GB used, 141 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,0,1]) > -4826> 2018-07-11 08:06:03.040625 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6802/15255 -- osd_ping(ping e509 stamp 2018-07-11 08:06:03.040623) v4 -- 0x55f67662aa00 con 0 > -4825> 2018-07-11 08:06:03.040649 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6803/15255 -- osd_ping(ping e509 stamp 2018-07-11 08:06:03.040623) v4 -- 0x55f67662ac00 con 0 > -4824> 2018-07-11 08:06:03.040663 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6806/12677 -- osd_ping(ping e509 stamp 2018-07-11 08:06:03.040623) v4 -- 0x55f67662ae00 con 0 > -4823> 2018-07-11 08:06:03.040679 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6807/12677 -- osd_ping(ping e509 stamp 2018-07-11 08:06:03.040623) v4 -- 0x55f67662b000 con 0 > -4822> 2018-07-11 08:06:03.040692 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6808/12651 -- osd_ping(ping e509 stamp 2018-07-11 08:06:03.040623) v4 -- 0x55f67662b200 con 0 > -4821> 2018-07-11 08:06:03.040703 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6810/12651 -- osd_ping(ping e509 stamp 2018-07-11 08:06:03.040623) v4 -- 0x55f67662b400 con 0 > -4820> 2018-07-11 08:06:03.040714 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6809/12650 -- osd_ping(ping e509 stamp 2018-07-11 08:06:03.040623) v4 -- 0x55f67662b600 con 0 > -4819> 2018-07-11 08:06:03.040738 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6811/12650 -- osd_ping(ping e509 stamp 2018-07-11 08:06:03.040623) v4 -- 0x55f67662b800 con 0 > -4818> 2018-07-11 08:06:03.040756 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6810/15287 -- osd_ping(ping e509 stamp 2018-07-11 08:06:03.040623) v4 -- 0x55f67662ba00 con 0 > -4817> 2018-07-11 08:06:03.040768 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6811/15287 -- osd_ping(ping e509 stamp 2018-07-11 08:06:03.040623) v4 -- 0x55f67662bc00 con 0 > -4816> 2018-07-11 08:06:03.040786 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6806/15263 -- osd_ping(ping e509 stamp 2018-07-11 08:06:03.040623) v4 -- 0x55f67662be00 con 0 > -4815> 2018-07-11 08:06:03.040804 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6807/15263 -- osd_ping(ping e509 stamp 2018-07-11 08:06:03.040623) v4 -- 0x55f664254e00 con 0 > -4814> 2018-07-11 08:06:03.040817 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.115:6802/13323 -- osd_ping(ping e509 stamp 2018-07-11 08:06:03.040623) v4 -- 0x55f664255800 con 0 > -4813> 2018-07-11 08:06:03.040830 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.115:6803/13323 -- osd_ping(ping e509 stamp 2018-07-11 08:06:03.040623) v4 -- 0x55f664254a00 con 0 > -4812> 2018-07-11 08:06:03.040843 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6806/15885 -- osd_ping(ping e509 stamp 2018-07-11 08:06:03.040623) v4 -- 0x55f656b55a00 con 0 > -4811> 2018-07-11 08:06:03.040855 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6807/15885 -- osd_ping(ping e509 stamp 2018-07-11 08:06:03.040623) v4 -- 0x55f6631cca00 con 0 > -4810> 2018-07-11 08:06:03.040868 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.108:6802/11967 -- osd_ping(ping e509 stamp 2018-07-11 08:06:03.040623) v4 -- 0x55f6631cd200 con 0 > -4809> 2018-07-11 08:06:03.040880 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.108:6803/11967 -- osd_ping(ping e509 stamp 2018-07-11 08:06:03.040623) v4 -- 0x55f6631cd400 con 0 > -4808> 2018-07-11 08:06:03.040892 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.107:6806/12617 -- osd_ping(ping e509 stamp 2018-07-11 08:06:03.040623) v4 -- 0x55f67d1af600 con 0 > -4807> 2018-07-11 08:06:03.040905 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.107:6807/12617 -- osd_ping(ping e509 stamp 2018-07-11 08:06:03.040623) v4 -- 0x55f67d1aec00 con 0 > -4806> 2018-07-11 08:06:03.040916 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6802/15919 -- osd_ping(ping e509 stamp 2018-07-11 08:06:03.040623) v4 -- 0x55f67d1ae000 con 0 > -4805> 2018-07-11 08:06:03.040927 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6803/15919 -- osd_ping(ping e509 stamp 2018-07-11 08:06:03.040623) v4 -- 0x55f66dceaa00 con 0 > -4804> 2018-07-11 08:06:03.041011 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6803/15255 conn(0x55f656ad5800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 207 0x55f6841a5400 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:03.040623) v4 > -4803> 2018-07-11 08:06:03.041033 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.0 10.8.128.113:6803/15255 207 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:03.040623) v4 ==== 2004+0+0 (626577023 0 0) 0x55f6841a5400 con 0x55f656ad5800 > -4802> 2018-07-11 08:06:03.041044 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6802/15255 conn(0x55f656ad4000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 207 0x55f676628600 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:03.040623) v4 > -4801> 2018-07-11 08:06:03.041099 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.0 10.8.128.113:6802/15255 207 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:03.040623) v4 ==== 2004+0+0 (626577023 0 0) 0x55f676628600 con 0x55f656ad4000 > -4800> 2018-07-11 08:06:03.041118 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6810/12651 conn(0x55f64e3eb800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.2 seq 207 0x55f6841a5400 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:03.040623) v4 > -4799> 2018-07-11 08:06:03.041124 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6806/12677 conn(0x55f64e3d8000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.1 seq 207 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:03.040623) v4 > -4798> 2018-07-11 08:06:03.041135 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.2 10.8.128.106:6810/12651 207 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:03.040623) v4 ==== 2004+0+0 (626577023 0 0) 0x55f6841a5400 con 0x55f64e3eb800 > -4797> 2018-07-11 08:06:03.041145 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.1 10.8.128.106:6806/12677 207 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:03.040623) v4 ==== 2004+0+0 (626577023 0 0) 0x55f67d1afe00 con 0x55f64e3d8000 > -4796> 2018-07-11 08:06:03.041179 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.107:6807/12617 conn(0x55f656adf800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.9 seq 207 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:03.040623) v4 > -4795> 2018-07-11 08:06:03.041192 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.9 10.8.128.107:6807/12617 207 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:03.040623) v4 ==== 2004+0+0 (626577023 0 0) 0x55f67d1afe00 con 0x55f656adf800 > -4794> 2018-07-11 08:06:03.041196 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.107:6806/12617 conn(0x55f656ade000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.9 seq 207 0x55f6841a5400 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:03.040623) v4 > -4793> 2018-07-11 08:06:03.041213 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.9 10.8.128.107:6806/12617 207 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:03.040623) v4 ==== 2004+0+0 (626577023 0 0) 0x55f6841a5400 con 0x55f656ade000 > -4792> 2018-07-11 08:06:03.041231 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6806/15263 conn(0x55f64e3ed000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.5 seq 207 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:03.040623) v4 > -4791> 2018-07-11 08:06:03.041210 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6811/12650 conn(0x55f656ad8800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.3 seq 207 0x55f676628600 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:03.040623) v4 > -4790> 2018-07-11 08:06:03.041244 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.5 10.8.128.113:6806/15263 207 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:03.040623) v4 ==== 2004+0+0 (626577023 0 0) 0x55f67d1afe00 con 0x55f64e3ed000 > -4789> 2018-07-11 08:06:03.041255 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.3 10.8.128.106:6811/12650 207 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:03.040623) v4 ==== 2004+0+0 (626577023 0 0) 0x55f676628600 con 0x55f656ad8800 > -4788> 2018-07-11 08:06:03.041290 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6811/15287 conn(0x55f64e402800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.4 seq 207 0x55f6841a5400 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:03.040623) v4 > -4787> 2018-07-11 08:06:03.041295 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6809/12650 conn(0x55f656ad7000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.3 seq 207 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:03.040623) v4 > -4786> 2018-07-11 08:06:03.041304 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6810/15287 conn(0x55f64e401000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.4 seq 207 0x55f676628600 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:03.040623) v4 > -4785> 2018-07-11 08:06:03.041314 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.4 10.8.128.113:6811/15287 207 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:03.040623) v4 ==== 2004+0+0 (626577023 0 0) 0x55f6841a5400 con 0x55f64e402800 > -4784> 2018-07-11 08:06:03.041318 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.3 10.8.128.106:6809/12650 207 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:03.040623) v4 ==== 2004+0+0 (626577023 0 0) 0x55f67d1afe00 con 0x55f656ad7000 > -4783> 2018-07-11 08:06:03.041321 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.4 10.8.128.113:6810/15287 207 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:03.040623) v4 ==== 2004+0+0 (626577023 0 0) 0x55f676628600 con 0x55f64e401000 > -4782> 2018-07-11 08:06:03.041360 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.115:6803/13323 conn(0x55f656ace800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.6 seq 207 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:03.040623) v4 > -4781> 2018-07-11 08:06:03.041373 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.6 10.8.128.115:6803/13323 207 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:03.040623) v4 ==== 2004+0+0 (626577023 0 0) 0x55f67d1afe00 con 0x55f656ace800 > -4780> 2018-07-11 08:06:03.041364 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6807/12677 conn(0x55f64e3d9800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.1 seq 207 0x55f676628600 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:03.040623) v4 > -4779> 2018-07-11 08:06:03.041410 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.1 10.8.128.106:6807/12677 207 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:03.040623) v4 ==== 2004+0+0 (626577023 0 0) 0x55f676628600 con 0x55f64e3d9800 > -4778> 2018-07-11 08:06:03.041417 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.115:6802/13323 conn(0x55f656acd000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.6 seq 207 0x55f6841a5400 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:03.040623) v4 > -4777> 2018-07-11 08:06:03.041433 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.6 10.8.128.115:6802/13323 207 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:03.040623) v4 ==== 2004+0+0 (626577023 0 0) 0x55f6841a5400 con 0x55f656acd000 > -4776> 2018-07-11 08:06:03.041443 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6803/15919 conn(0x55f64e3dc800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.11 seq 207 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:03.040623) v4 > -4775> 2018-07-11 08:06:03.041457 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.11 10.8.128.114:6803/15919 207 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:03.040623) v4 ==== 2004+0+0 (626577023 0 0) 0x55f67d1afe00 con 0x55f64e3dc800 > -4774> 2018-07-11 08:06:03.041460 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6807/15263 conn(0x55f64e3ee800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.5 seq 207 0x55f676628600 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:03.040623) v4 > -4773> 2018-07-11 08:06:03.041467 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6806/15885 conn(0x55f64e3fe000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.7 seq 207 0x55f6841a5400 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:03.040623) v4 > -4772> 2018-07-11 08:06:03.041480 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.7 10.8.128.114:6806/15885 207 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:03.040623) v4 ==== 2004+0+0 (626577023 0 0) 0x55f6841a5400 con 0x55f64e3fe000 > -4771> 2018-07-11 08:06:03.041479 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.5 10.8.128.113:6807/15263 207 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:03.040623) v4 ==== 2004+0+0 (626577023 0 0) 0x55f676628600 con 0x55f64e3ee800 > -4770> 2018-07-11 08:06:03.041500 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6807/15885 conn(0x55f64e3ff800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.7 seq 207 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:03.040623) v4 > -4769> 2018-07-11 08:06:03.041515 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.7 10.8.128.114:6807/15885 207 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:03.040623) v4 ==== 2004+0+0 (626577023 0 0) 0x55f67d1afe00 con 0x55f64e3ff800 > -4768> 2018-07-11 08:06:03.041514 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6808/12651 conn(0x55f64e3ea000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.2 seq 207 0x55f676628600 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:03.040623) v4 > -4767> 2018-07-11 08:06:03.041513 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6802/15919 conn(0x55f64e3db000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.11 seq 207 0x55f6841a5400 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:03.040623) v4 > -4766> 2018-07-11 08:06:03.041533 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.2 10.8.128.106:6808/12651 207 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:03.040623) v4 ==== 2004+0+0 (626577023 0 0) 0x55f676628600 con 0x55f64e3ea000 > -4765> 2018-07-11 08:06:03.041548 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.11 10.8.128.114:6802/15919 207 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:03.040623) v4 ==== 2004+0+0 (626577023 0 0) 0x55f6841a5400 con 0x55f64e3db000 > -4764> 2018-07-11 08:06:03.041553 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.108:6802/11967 conn(0x55f656aca000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.8 seq 207 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:03.040623) v4 > -4763> 2018-07-11 08:06:03.041566 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.8 10.8.128.108:6802/11967 207 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:03.040623) v4 ==== 2004+0+0 (626577023 0 0) 0x55f67d1afe00 con 0x55f656aca000 > -4762> 2018-07-11 08:06:03.041591 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.108:6803/11967 conn(0x55f656acb800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.8 seq 207 0x55f676628600 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:03.040623) v4 > -4761> 2018-07-11 08:06:03.041610 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.8 10.8.128.108:6803/11967 207 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:03.040623) v4 ==== 2004+0+0 (626577023 0 0) 0x55f676628600 con 0x55f656acb800 > -4760> 2018-07-11 08:06:03.137685 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15885 conn(0x55f663216000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 212 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:06:03.773572) v4 > -4759> 2018-07-11 08:06:03.137704 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.7 10.8.128.114:0/15885 212 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:03.773572) v4 ==== 2004+0+0 (1352852597 0 0) 0x55f67d1afe00 con 0x55f663216000 > -4758> 2018-07-11 08:06:03.137713 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:03.773572) v4 -- 0x55f67662b600 con 0 > -4757> 2018-07-11 08:06:03.137778 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15885 conn(0x55f663217800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 212 0x55f6841a5400 osd_ping(ping e509 stamp 2018-07-11 08:06:03.773572) v4 > -4756> 2018-07-11 08:06:03.137801 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.7 10.8.128.114:0/15885 212 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:03.773572) v4 ==== 2004+0+0 (1352852597 0 0) 0x55f6841a5400 con 0x55f663217800 > -4755> 2018-07-11 08:06:03.137818 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:03.773572) v4 -- 0x55f67662b400 con 0 > -4754> 2018-07-11 08:06:03.567139 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f665a7b440 con 0 > -4753> 2018-07-11 08:06:03.568786 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 259 0x55f67c24d900 replica scrub(pg: 6.15,from:0'0,to:430'106491,epoch:509/504,start:6:a85f1162:::benchmark_data_magna108_35617_object1107853:0,end:6:a85fc785:::benchmark_data_magna108_35617_object890968:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -4752> 2018-07-11 08:06:03.568821 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 259 ==== replica scrub(pg: 6.15,from:0'0,to:430'106491,epoch:509/504,start:6:a85f1162:::benchmark_data_magna108_35617_object1107853:0,end:6:a85fc785:::benchmark_data_magna108_35617_object890968:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 219+0+0 (3352852732 0 0) 0x55f67c24d900 con 0x55f66315e800 > -4751> 2018-07-11 08:06:03.609052 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0a000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 210 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:06:04.244880) v4 > -4750> 2018-07-11 08:06:03.609079 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.11 10.8.128.114:0/15919 210 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:04.244880) v4 ==== 2004+0+0 (224229498 0 0) 0x55f64e20fc00 con 0x55f663f0a000 > -4749> 2018-07-11 08:06:03.609075 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0b800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 210 0x55f67d193400 osd_ping(ping e509 stamp 2018-07-11 08:06:04.244880) v4 > -4748> 2018-07-11 08:06:03.609089 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:04.244880) v4 -- 0x55f67662b600 con 0 > -4747> 2018-07-11 08:06:03.609100 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.11 10.8.128.114:0/15919 210 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:04.244880) v4 ==== 2004+0+0 (224229498 0 0) 0x55f67d193400 con 0x55f663f0b800 > -4746> 2018-07-11 08:06:03.609122 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:04.244880) v4 -- 0x55f67662b400 con 0 > -4745> 2018-07-11 08:06:03.666744 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.114:6805/15885 conn(0x55f66315a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.7 seq 2563 0x55f676628600 MOSDScrubReserve(5.4 REQUEST e509) v1 > -4744> 2018-07-11 08:06:03.666763 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.7 10.8.128.114:6805/15885 2563 ==== MOSDScrubReserve(5.4 REQUEST e509) v1 ==== 43+0+0 (3003630469 0 0) 0x55f676628600 con 0x55f66315a000 > -4743> 2018-07-11 08:06:03.666796 7f491983f700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.114:6805/15885 -- MOSDScrubReserve(5.4 REJECT e509) v1 -- 0x55f666182e00 con 0 > -4742> 2018-07-11 08:06:03.667104 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.114:6805/15885 conn(0x55f66315a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.7 seq 2564 0x55f666182e00 MOSDScrubReserve(5.4 RELEASE e509) v1 > -4741> 2018-07-11 08:06:03.667118 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.7 10.8.128.114:6805/15885 2564 ==== MOSDScrubReserve(5.4 RELEASE e509) v1 ==== 43+0+0 (2082093447 0 0) 0x55f666182e00 con 0x55f66315a000 > -4740> 2018-07-11 08:06:03.760335 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12651 conn(0x55f663219000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25 cs=1 l=1). rx osd.2 seq 221 0x55f673adbe00 osd_ping(ping e509 stamp 2018-07-11 08:06:03.817314) v4 > -4739> 2018-07-11 08:06:03.760365 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.2 10.8.128.106:0/12651 221 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:03.817314) v4 ==== 2004+0+0 (3764581438 0 0) 0x55f673adbe00 con 0x55f663219000 > -4738> 2018-07-11 08:06:03.760382 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:03.817314) v4 -- 0x55f67c197e00 con 0 > -4737> 2018-07-11 08:06:03.760442 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12651 conn(0x55f66321a800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25 cs=1 l=1). rx osd.2 seq 221 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:06:03.817314) v4 > -4736> 2018-07-11 08:06:03.760463 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.2 10.8.128.106:0/12651 221 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:03.817314) v4 ==== 2004+0+0 (3764581438 0 0) 0x55f67d1afe00 con 0x55f66321a800 > -4735> 2018-07-11 08:06:03.760475 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:03.817314) v4 -- 0x55f67662ae00 con 0 > -4734> 2018-07-11 08:06:03.980679 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed6000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 210 0x55f67662aa00 osd_ping(ping e509 stamp 2018-07-11 08:06:04.037853) v4 > -4733> 2018-07-11 08:06:03.980703 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.1 10.8.128.106:0/12677 210 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:04.037853) v4 ==== 2004+0+0 (1320737254 0 0) 0x55f67662aa00 con 0x55f663ed6000 > -4732> 2018-07-11 08:06:03.980716 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:04.037853) v4 -- 0x55f64fc92000 con 0 > -4731> 2018-07-11 08:06:03.980710 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed7800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 210 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:06:04.037853) v4 > -4730> 2018-07-11 08:06:03.980732 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.1 10.8.128.106:0/12677 210 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:04.037853) v4 ==== 2004+0+0 (1320737254 0 0) 0x55f64e20fc00 con 0x55f663ed7800 > -4729> 2018-07-11 08:06:03.980753 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:04.037853) v4 -- 0x55f676628200 con 0 > -4728> 2018-07-11 08:06:03.992750 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.107:0/12617 conn(0x55f64e3c5000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.9 seq 211 0x55f673adbe00 osd_ping(ping e509 stamp 2018-07-11 08:06:03.992263) v4 > -4727> 2018-07-11 08:06:03.992732 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.107:0/12617 conn(0x55f64e3c3800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.9 seq 211 0x55f6841a5400 osd_ping(ping e509 stamp 2018-07-11 08:06:03.992263) v4 > -4726> 2018-07-11 08:06:03.992794 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.9 10.8.128.107:0/12617 211 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:03.992263) v4 ==== 2004+0+0 (2005346522 0 0) 0x55f673adbe00 con 0x55f64e3c5000 > -4725> 2018-07-11 08:06:03.992806 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:03.992263) v4 -- 0x55f64fc92000 con 0 > -4724> 2018-07-11 08:06:03.992804 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.9 10.8.128.107:0/12617 211 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:03.992263) v4 ==== 2004+0+0 (2005346522 0 0) 0x55f6841a5400 con 0x55f64e3c3800 > -4723> 2018-07-11 08:06:03.992867 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:03.992263) v4 -- 0x55f67662b400 con 0 > -4722> 2018-07-11 08:06:04.269024 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f66810c480 con 0 > -4721> 2018-07-11 08:06:04.270690 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 260 0x55f67c24db80 replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:509/504,start:6:a85fc785:::benchmark_data_magna108_35617_object890968:0,end:6:a860bf40:::benchmark_data_magna108_35617_object683399:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -4720> 2018-07-11 08:06:04.270722 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 260 ==== replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:509/504,start:6:a85fc785:::benchmark_data_magna108_35617_object890968:0,end:6:a860bf40:::benchmark_data_magna108_35617_object683399:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (3439011698 0 0) 0x55f67c24db80 con 0x55f66315e800 > -4719> 2018-07-11 08:06:04.480392 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15255 conn(0x55f663ec1000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.0 seq 202 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:06:05.725479) v4 > -4718> 2018-07-11 08:06:04.480415 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.0 10.8.128.113:0/15255 202 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:05.725479) v4 ==== 2004+0+0 (2605697238 0 0) 0x55f67d1afe00 con 0x55f663ec1000 > -4717> 2018-07-11 08:06:04.480430 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:05.725479) v4 -- 0x55f673ada600 con 0 > -4716> 2018-07-11 08:06:04.480424 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15255 conn(0x55f663ec2800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.0 seq 202 0x55f67d193400 osd_ping(ping e509 stamp 2018-07-11 08:06:05.725479) v4 > -4715> 2018-07-11 08:06:04.480441 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.0 10.8.128.113:0/15255 202 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:05.725479) v4 ==== 2004+0+0 (2605697238 0 0) 0x55f67d193400 con 0x55f663ec2800 > -4714> 2018-07-11 08:06:04.480455 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:05.725479) v4 -- 0x55f67662ac00 con 0 > -4713> 2018-07-11 08:06:04.838122 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15885 conn(0x55f663217800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 213 0x55f6841a5400 osd_ping(ping e509 stamp 2018-07-11 08:06:05.473983) v4 > -4712> 2018-07-11 08:06:04.838137 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15885 conn(0x55f663216000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 213 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:06:05.473983) v4 > -4711> 2018-07-11 08:06:04.838155 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.7 10.8.128.114:0/15885 213 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:05.473983) v4 ==== 2004+0+0 (4210736228 0 0) 0x55f64e20fc00 con 0x55f663216000 > -4710> 2018-07-11 08:06:04.838154 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.7 10.8.128.114:0/15885 213 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:05.473983) v4 ==== 2004+0+0 (4210736228 0 0) 0x55f6841a5400 con 0x55f663217800 > -4709> 2018-07-11 08:06:04.838166 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:05.473983) v4 -- 0x55f6841a5e00 con 0 > -4708> 2018-07-11 08:06:04.838196 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:05.473983) v4 -- 0x55f65fa97e00 con 0 > -4707> 2018-07-11 08:06:05.203270 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.108:0/11967 conn(0x55f663f6f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=1). rx osd.8 seq 202 0x55f67d193400 osd_ping(ping e509 stamp 2018-07-11 08:06:04.963289) v4 > -4706> 2018-07-11 08:06:05.203270 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.108:0/11967 conn(0x55f663f70800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=1). rx osd.8 seq 202 0x55f67662aa00 osd_ping(ping e509 stamp 2018-07-11 08:06:04.963289) v4 > -4705> 2018-07-11 08:06:05.203308 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.8 10.8.128.108:0/11967 202 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:04.963289) v4 ==== 2004+0+0 (2626643504 0 0) 0x55f67662aa00 con 0x55f663f70800 > -4704> 2018-07-11 08:06:05.203307 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.8 10.8.128.108:0/11967 202 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:04.963289) v4 ==== 2004+0+0 (2626643504 0 0) 0x55f67d193400 con 0x55f663f6f000 > -4703> 2018-07-11 08:06:05.203319 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:04.963289) v4 -- 0x55f64fc92200 con 0 > -4702> 2018-07-11 08:06:05.203349 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:04.963289) v4 -- 0x55f67db36000 con 0 > -4701> 2018-07-11 08:06:05.240463 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f64f3806c0 con 0 > -4700> 2018-07-11 08:06:05.242104 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 261 0x55f683ee4000 replica scrub(pg: 6.15,from:0'0,to:430'106170,epoch:509/504,start:6:a860bf40:::benchmark_data_magna108_35617_object683399:0,end:6:a8617ff7:::benchmark_data_magna108.ceph.redhat.com_2478_object151432:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -4699> 2018-07-11 08:06:05.242137 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 261 ==== replica scrub(pg: 6.15,from:0'0,to:430'106170,epoch:509/504,start:6:a860bf40:::benchmark_data_magna108_35617_object683399:0,end:6:a8617ff7:::benchmark_data_magna108.ceph.redhat.com_2478_object151432:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (3367794926 0 0) 0x55f683ee4000 con 0x55f66315e800 > -4698> 2018-07-11 08:06:05.530948 7f491d046700 1 -- 10.8.128.107:6800/14173 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55f666f258c0 con 0 > -4697> 2018-07-11 08:06:05.531025 7f491d046700 1 -- 10.8.128.107:6800/14173 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55f64cc76000 con 0 > -4696> 2018-07-11 08:06:05.826344 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 30] Generated table #37048: 2638241 keys, 74547069 bytes > -4695> 2018-07-11 08:06:05.826394 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296365826378, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 37048, "file_size": 74547069, "table_properties": {"data_size": 67112177, "index_size": 690719, "filter_size": 6743183, "raw_key_size": 76215860, "raw_average_key_size": 28, "raw_value_size": 31365763, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} > -4694> 2018-07-11 08:06:05.909518 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0b800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 211 0x55f6841a5400 osd_ping(ping e509 stamp 2018-07-11 08:06:06.545306) v4 > -4693> 2018-07-11 08:06:05.909525 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0a000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 211 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:06:06.545306) v4 > -4692> 2018-07-11 08:06:05.909542 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.11 10.8.128.114:0/15919 211 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:06.545306) v4 ==== 2004+0+0 (2513806822 0 0) 0x55f6841a5400 con 0x55f663f0b800 > -4691> 2018-07-11 08:06:05.909544 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.11 10.8.128.114:0/15919 211 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:06.545306) v4 ==== 2004+0+0 (2513806822 0 0) 0x55f67d1afe00 con 0x55f663f0a000 > -4690> 2018-07-11 08:06:05.909554 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:06.545306) v4 -- 0x55f67db36000 con 0 > -4689> 2018-07-11 08:06:05.909581 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:06.545306) v4 -- 0x55f6841a5e00 con 0 > -4688> 2018-07-11 08:06:06.293418 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.107:0/12617 conn(0x55f64e3c3800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.9 seq 212 0x55f67d193400 osd_ping(ping e509 stamp 2018-07-11 08:06:06.292886) v4 > -4687> 2018-07-11 08:06:06.293443 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.9 10.8.128.107:0/12617 212 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:06.292886) v4 ==== 2004+0+0 (1918099512 0 0) 0x55f67d193400 con 0x55f64e3c3800 > -4686> 2018-07-11 08:06:06.293427 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.107:0/12617 conn(0x55f64e3c5000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.9 seq 212 0x55f673adbe00 osd_ping(ping e509 stamp 2018-07-11 08:06:06.292886) v4 > -4685> 2018-07-11 08:06:06.293455 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:06.292886) v4 -- 0x55f67db36000 con 0 > -4684> 2018-07-11 08:06:06.293455 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.9 10.8.128.107:0/12617 212 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:06.292886) v4 ==== 2004+0+0 (1918099512 0 0) 0x55f673adbe00 con 0x55f64e3c5000 > -4683> 2018-07-11 08:06:06.293500 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:06.292886) v4 -- 0x55f64fc92400 con 0 > -4682> 2018-07-11 08:06:06.457717 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f6562c8480 con 0 > -4681> 2018-07-11 08:06:06.459284 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 262 0x55f683ee4280 replica scrub(pg: 6.15,from:0'0,to:430'106674,epoch:509/504,start:6:a8617ff7:::benchmark_data_magna108.ceph.redhat.com_2478_object151432:0,end:6:a8629138:::benchmark_data_magna108_35617_object1024165:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -4680> 2018-07-11 08:06:06.459315 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 262 ==== replica scrub(pg: 6.15,from:0'0,to:430'106674,epoch:509/504,start:6:a8617ff7:::benchmark_data_magna108.ceph.redhat.com_2478_object151432:0,end:6:a8629138:::benchmark_data_magna108_35617_object1024165:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 234+0+0 (2729473906 0 0) 0x55f683ee4280 con 0x55f66315e800 > -4679> 2018-07-11 08:06:06.634718 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12650 conn(0x55f663ed9000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.3 seq 209 0x55f6841a5400 osd_ping(ping e509 stamp 2018-07-11 08:06:06.691946) v4 > -4678> 2018-07-11 08:06:06.634751 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.3 10.8.128.106:0/12650 209 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:06.691946) v4 ==== 2004+0+0 (1700074700 0 0) 0x55f6841a5400 con 0x55f663ed9000 > -4677> 2018-07-11 08:06:06.634769 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:06.691946) v4 -- 0x55f67db36000 con 0 > -4676> 2018-07-11 08:06:06.634816 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12650 conn(0x55f663eda800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.3 seq 209 0x55f64fc92200 osd_ping(ping e509 stamp 2018-07-11 08:06:06.691946) v4 > -4675> 2018-07-11 08:06:06.634840 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.3 10.8.128.106:0/12650 209 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:06.691946) v4 ==== 2004+0+0 (1700074700 0 0) 0x55f64fc92200 con 0x55f663eda800 > -4674> 2018-07-11 08:06:06.634857 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:06.691946) v4 -- 0x55f64fc92400 con 0 > -4673> 2018-07-11 08:06:06.660778 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12651 conn(0x55f663219000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25 cs=1 l=1). rx osd.2 seq 222 0x55f673adbe00 osd_ping(ping e509 stamp 2018-07-11 08:06:06.717915) v4 > -4672> 2018-07-11 08:06:06.660803 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.2 10.8.128.106:0/12651 222 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:06.717915) v4 ==== 2004+0+0 (114349182 0 0) 0x55f673adbe00 con 0x55f663219000 > -4671> 2018-07-11 08:06:06.660843 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:06.717915) v4 -- 0x55f64fc92400 con 0 > -4670> 2018-07-11 08:06:06.660839 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12651 conn(0x55f66321a800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25 cs=1 l=1). rx osd.2 seq 222 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:06:06.717915) v4 > -4669> 2018-07-11 08:06:06.660855 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.2 10.8.128.106:0/12651 222 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:06.717915) v4 ==== 2004+0+0 (114349182 0 0) 0x55f64e20fc00 con 0x55f66321a800 > -4668> 2018-07-11 08:06:06.660868 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:06.717915) v4 -- 0x55f6841a5e00 con 0 > -4667> 2018-07-11 08:06:07.113567 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.115:0/13323 conn(0x55f663ebf800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.6 seq 197 0x55f64fc92200 osd_ping(ping e509 stamp 2018-07-11 08:06:07.466534) v4 > -4666> 2018-07-11 08:06:07.113608 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.6 10.8.128.115:0/13323 197 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:07.466534) v4 ==== 2004+0+0 (1396605215 0 0) 0x55f64fc92200 con 0x55f663ebf800 > -4665> 2018-07-11 08:06:07.113628 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:07.466534) v4 -- 0x55f64fc92600 con 0 > -4664> 2018-07-11 08:06:07.113623 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.115:0/13323 conn(0x55f663ebe000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.6 seq 197 0x55f67d193400 osd_ping(ping e509 stamp 2018-07-11 08:06:07.466534) v4 > -4663> 2018-07-11 08:06:07.113642 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.6 10.8.128.115:0/13323 197 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:07.466534) v4 ==== 2004+0+0 (1396605215 0 0) 0x55f67d193400 con 0x55f663ebe000 > -4662> 2018-07-11 08:06:07.113663 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:07.466534) v4 -- 0x55f67db36200 con 0 > -4661> 2018-07-11 08:06:07.161228 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12651 conn(0x55f663219000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25 cs=1 l=1). rx osd.2 seq 223 0x55f673adbe00 osd_ping(ping e509 stamp 2018-07-11 08:06:07.218415) v4 > -4660> 2018-07-11 08:06:07.161266 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.2 10.8.128.106:0/12651 223 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:07.218415) v4 ==== 2004+0+0 (562033256 0 0) 0x55f673adbe00 con 0x55f663219000 > -4659> 2018-07-11 08:06:07.161282 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:07.218415) v4 -- 0x55f64fc92800 con 0 > -4658> 2018-07-11 08:06:07.161295 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12651 conn(0x55f66321a800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25 cs=1 l=1). rx osd.2 seq 223 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:06:07.218415) v4 > -4657> 2018-07-11 08:06:07.161320 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.2 10.8.128.106:0/12651 223 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:07.218415) v4 ==== 2004+0+0 (562033256 0 0) 0x55f67d1afe00 con 0x55f66321a800 > -4656> 2018-07-11 08:06:07.161331 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:07.218415) v4 -- 0x55f6841a5600 con 0 > -4655> 2018-07-11 08:06:07.304996 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f652bd66c0 con 0 > -4654> 2018-07-11 08:06:07.306760 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 263 0x55f683ee4500 replica scrub(pg: 6.15,from:0'0,to:430'105468,epoch:509/504,start:6:a8629138:::benchmark_data_magna108_35617_object1024165:0,end:6:a8636782:::benchmark_data_magna108_35617_object413047:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -4653> 2018-07-11 08:06:07.306788 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 263 ==== replica scrub(pg: 6.15,from:0'0,to:430'105468,epoch:509/504,start:6:a8629138:::benchmark_data_magna108_35617_object1024165:0,end:6:a8636782:::benchmark_data_magna108_35617_object413047:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 219+0+0 (2628427355 0 0) 0x55f683ee4500 con 0x55f66315e800 > -4652> 2018-07-11 08:06:07.380870 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15255 conn(0x55f663ec1000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.0 seq 203 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:06:08.625993) v4 > -4651> 2018-07-11 08:06:07.380890 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15255 conn(0x55f663ec2800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.0 seq 203 0x55f6841a5400 osd_ping(ping e509 stamp 2018-07-11 08:06:08.625993) v4 > -4650> 2018-07-11 08:06:07.380898 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.0 10.8.128.113:0/15255 203 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:08.625993) v4 ==== 2004+0+0 (3195234502 0 0) 0x55f64e20fc00 con 0x55f663ec1000 > -4649> 2018-07-11 08:06:07.380916 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.0 10.8.128.113:0/15255 203 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:08.625993) v4 ==== 2004+0+0 (3195234502 0 0) 0x55f6841a5400 con 0x55f663ec2800 > -4648> 2018-07-11 08:06:07.380922 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:08.625993) v4 -- 0x55f6841a5600 con 0 > -4647> 2018-07-11 08:06:07.380953 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:08.625993) v4 -- 0x55f67db36200 con 0 > -4646> 2018-07-11 08:06:07.460671 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0d000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 218 0x55f64fc92200 osd_ping(ping e509 stamp 2018-07-11 08:06:08.705961) v4 > -4645> 2018-07-11 08:06:07.460690 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.5 10.8.128.113:0/15263 218 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:08.705961) v4 ==== 2004+0+0 (244714108 0 0) 0x55f64fc92200 con 0x55f663f0d000 > -4644> 2018-07-11 08:06:07.460699 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:08.705961) v4 -- 0x55f64fc92a00 con 0 > -4643> 2018-07-11 08:06:07.460752 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0e800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 218 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:06:08.705961) v4 > -4642> 2018-07-11 08:06:07.460775 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.5 10.8.128.113:0/15263 218 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:08.705961) v4 ==== 2004+0+0 (244714108 0 0) 0x55f67d1afe00 con 0x55f663f0e800 > -4641> 2018-07-11 08:06:07.460788 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:08.705961) v4 -- 0x55f6631cc800 con 0 > -4640> 2018-07-11 08:06:07.503593 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.108:0/11967 conn(0x55f663f6f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=1). rx osd.8 seq 203 0x55f67d193400 osd_ping(ping e509 stamp 2018-07-11 08:06:07.263724) v4 > -4639> 2018-07-11 08:06:07.503615 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.8 10.8.128.108:0/11967 203 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:07.263724) v4 ==== 2004+0+0 (3963542206 0 0) 0x55f67d193400 con 0x55f663f6f000 > -4638> 2018-07-11 08:06:07.503627 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:07.263724) v4 -- 0x55f67db36400 con 0 > -4637> 2018-07-11 08:06:07.503690 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.108:0/11967 conn(0x55f663f70800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=1). rx osd.8 seq 203 0x55f64fc92800 osd_ping(ping e509 stamp 2018-07-11 08:06:07.263724) v4 > -4636> 2018-07-11 08:06:07.503710 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.8 10.8.128.108:0/11967 203 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:07.263724) v4 ==== 2004+0+0 (3963542206 0 0) 0x55f64fc92800 con 0x55f663f70800 > -4635> 2018-07-11 08:06:07.503722 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:07.263724) v4 -- 0x55f64fc92c00 con 0 > -4634> 2018-07-11 08:06:07.741072 7f491803c700 5 osd.10 509 heartbeat: osd_stat(790 GB used, 141 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,0,1]) > -4633> 2018-07-11 08:06:07.741090 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6802/15255 -- osd_ping(ping e509 stamp 2018-07-11 08:06:07.741088) v4 -- 0x55f66dcebe00 con 0 > -4632> 2018-07-11 08:06:07.741119 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6803/15255 -- osd_ping(ping e509 stamp 2018-07-11 08:06:07.741088) v4 -- 0x55f66dcea000 con 0 > -4631> 2018-07-11 08:06:07.741136 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6806/12677 -- osd_ping(ping e509 stamp 2018-07-11 08:06:07.741088) v4 -- 0x55f65d025200 con 0 > -4630> 2018-07-11 08:06:07.741151 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6807/12677 -- osd_ping(ping e509 stamp 2018-07-11 08:06:07.741088) v4 -- 0x55f653d4a800 con 0 > -4629> 2018-07-11 08:06:07.741164 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6808/12651 -- osd_ping(ping e509 stamp 2018-07-11 08:06:07.741088) v4 -- 0x55f669dea600 con 0 > -4628> 2018-07-11 08:06:07.741174 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6810/12651 -- osd_ping(ping e509 stamp 2018-07-11 08:06:07.741088) v4 -- 0x55f669dea000 con 0 > -4627> 2018-07-11 08:06:07.741186 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6809/12650 -- osd_ping(ping e509 stamp 2018-07-11 08:06:07.741088) v4 -- 0x55f664256800 con 0 > -4626> 2018-07-11 08:06:07.741209 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6811/12650 -- osd_ping(ping e509 stamp 2018-07-11 08:06:07.741088) v4 -- 0x55f64f769a00 con 0 > -4625> 2018-07-11 08:06:07.741222 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6810/15287 -- osd_ping(ping e509 stamp 2018-07-11 08:06:07.741088) v4 -- 0x55f6780b2e00 con 0 > -4624> 2018-07-11 08:06:07.741233 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6811/15287 -- osd_ping(ping e509 stamp 2018-07-11 08:06:07.741088) v4 -- 0x55f65e71c200 con 0 > -4623> 2018-07-11 08:06:07.741260 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6806/15263 -- osd_ping(ping e509 stamp 2018-07-11 08:06:07.741088) v4 -- 0x55f660949600 con 0 > -4622> 2018-07-11 08:06:07.741274 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6807/15263 -- osd_ping(ping e509 stamp 2018-07-11 08:06:07.741088) v4 -- 0x55f660949e00 con 0 > -4621> 2018-07-11 08:06:07.741286 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.115:6802/13323 -- osd_ping(ping e509 stamp 2018-07-11 08:06:07.741088) v4 -- 0x55f660949a00 con 0 > -4620> 2018-07-11 08:06:07.741302 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.115:6803/13323 -- osd_ping(ping e509 stamp 2018-07-11 08:06:07.741088) v4 -- 0x55f667d0dc00 con 0 > -4619> 2018-07-11 08:06:07.741316 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6806/15885 -- osd_ping(ping e509 stamp 2018-07-11 08:06:07.741088) v4 -- 0x55f667d0d200 con 0 > -4618> 2018-07-11 08:06:07.741328 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6807/15885 -- osd_ping(ping e509 stamp 2018-07-11 08:06:07.741088) v4 -- 0x55f667d0d000 con 0 > -4617> 2018-07-11 08:06:07.741340 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.108:6802/11967 -- osd_ping(ping e509 stamp 2018-07-11 08:06:07.741088) v4 -- 0x55f666c46400 con 0 > -4616> 2018-07-11 08:06:07.741355 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.108:6803/11967 -- osd_ping(ping e509 stamp 2018-07-11 08:06:07.741088) v4 -- 0x55f66bf10000 con 0 > -4615> 2018-07-11 08:06:07.741368 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.107:6806/12617 -- osd_ping(ping e509 stamp 2018-07-11 08:06:07.741088) v4 -- 0x55f66bf10200 con 0 > -4614> 2018-07-11 08:06:07.741381 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.107:6807/12617 -- osd_ping(ping e509 stamp 2018-07-11 08:06:07.741088) v4 -- 0x55f66bf10400 con 0 > -4613> 2018-07-11 08:06:07.741394 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6802/15919 -- osd_ping(ping e509 stamp 2018-07-11 08:06:07.741088) v4 -- 0x55f66bf10600 con 0 > -4612> 2018-07-11 08:06:07.741405 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6803/15919 -- osd_ping(ping e509 stamp 2018-07-11 08:06:07.741088) v4 -- 0x55f66bf10800 con 0 > -4611> 2018-07-11 08:06:07.741483 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6802/15255 conn(0x55f656ad4000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 208 0x55f64fc92800 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:07.741088) v4 > -4610> 2018-07-11 08:06:07.741534 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.0 10.8.128.113:6802/15255 208 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:07.741088) v4 ==== 2004+0+0 (2450665529 0 0) 0x55f64fc92800 con 0x55f656ad4000 > -4609> 2018-07-11 08:06:07.741494 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6803/15255 conn(0x55f656ad5800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 208 0x55f67d193400 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:07.741088) v4 > -4608> 2018-07-11 08:06:07.741564 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.0 10.8.128.113:6803/15255 208 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:07.741088) v4 ==== 2004+0+0 (2450665529 0 0) 0x55f67d193400 con 0x55f656ad5800 > -4607> 2018-07-11 08:06:07.741574 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6809/12650 conn(0x55f656ad7000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.3 seq 208 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:07.741088) v4 > -4606> 2018-07-11 08:06:07.741607 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.3 10.8.128.106:6809/12650 208 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:07.741088) v4 ==== 2004+0+0 (2450665529 0 0) 0x55f64e20fc00 con 0x55f656ad7000 > -4605> 2018-07-11 08:06:07.741614 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6810/12651 conn(0x55f64e3eb800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.2 seq 208 0x55f67d193400 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:07.741088) v4 > -4604> 2018-07-11 08:06:07.741637 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6806/12677 conn(0x55f64e3d8000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.1 seq 208 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:07.741088) v4 > -4603> 2018-07-11 08:06:07.741647 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.1 10.8.128.106:6806/12677 208 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:07.741088) v4 ==== 2004+0+0 (2450665529 0 0) 0x55f64e20fc00 con 0x55f64e3d8000 > -4602> 2018-07-11 08:06:07.741645 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.2 10.8.128.106:6810/12651 208 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:07.741088) v4 ==== 2004+0+0 (2450665529 0 0) 0x55f67d193400 con 0x55f64e3eb800 > -4601> 2018-07-11 08:06:07.741654 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6808/12651 conn(0x55f64e3ea000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.2 seq 208 0x55f64fc92800 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:07.741088) v4 > -4600> 2018-07-11 08:06:07.741675 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.2 10.8.128.106:6808/12651 208 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:07.741088) v4 ==== 2004+0+0 (2450665529 0 0) 0x55f64fc92800 con 0x55f64e3ea000 > -4599> 2018-07-11 08:06:07.741709 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6806/15263 conn(0x55f64e3ed000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.5 seq 208 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:07.741088) v4 > -4598> 2018-07-11 08:06:07.741724 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.5 10.8.128.113:6806/15263 208 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:07.741088) v4 ==== 2004+0+0 (2450665529 0 0) 0x55f64e20fc00 con 0x55f64e3ed000 > -4597> 2018-07-11 08:06:07.741748 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6811/15287 conn(0x55f64e402800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.4 seq 208 0x55f67d193400 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:07.741088) v4 > -4596> 2018-07-11 08:06:07.741755 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6807/12677 conn(0x55f64e3d9800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.1 seq 208 0x55f64fc92800 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:07.741088) v4 > -4595> 2018-07-11 08:06:07.741767 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.4 10.8.128.113:6811/15287 208 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:07.741088) v4 ==== 2004+0+0 (2450665529 0 0) 0x55f67d193400 con 0x55f64e402800 > -4594> 2018-07-11 08:06:07.741776 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.1 10.8.128.106:6807/12677 208 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:07.741088) v4 ==== 2004+0+0 (2450665529 0 0) 0x55f64fc92800 con 0x55f64e3d9800 > -4593> 2018-07-11 08:06:07.741793 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.107:6806/12617 conn(0x55f656ade000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.9 seq 208 0x55f67d193400 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:07.741088) v4 > -4592> 2018-07-11 08:06:07.741804 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.9 10.8.128.107:6806/12617 208 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:07.741088) v4 ==== 2004+0+0 (2450665529 0 0) 0x55f67d193400 con 0x55f656ade000 > -4591> 2018-07-11 08:06:07.741820 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6811/12650 conn(0x55f656ad8800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.3 seq 208 0x55f64fc92800 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:07.741088) v4 > -4590> 2018-07-11 08:06:07.741816 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.115:6803/13323 conn(0x55f656ace800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.6 seq 208 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:07.741088) v4 > -4589> 2018-07-11 08:06:07.741842 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.6 10.8.128.115:6803/13323 208 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:07.741088) v4 ==== 2004+0+0 (2450665529 0 0) 0x55f64e20fc00 con 0x55f656ace800 > -4588> 2018-07-11 08:06:07.741839 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.115:6802/13323 conn(0x55f656acd000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.6 seq 208 0x55f67d193400 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:07.741088) v4 > -4587> 2018-07-11 08:06:07.741839 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.3 10.8.128.106:6811/12650 208 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:07.741088) v4 ==== 2004+0+0 (2450665529 0 0) 0x55f64fc92800 con 0x55f656ad8800 > -4586> 2018-07-11 08:06:07.741857 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.6 10.8.128.115:6802/13323 208 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:07.741088) v4 ==== 2004+0+0 (2450665529 0 0) 0x55f67d193400 con 0x55f656acd000 > -4585> 2018-07-11 08:06:07.741869 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.107:6807/12617 conn(0x55f656adf800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.9 seq 208 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:07.741088) v4 > -4584> 2018-07-11 08:06:07.741892 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6802/15919 conn(0x55f64e3db000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.11 seq 208 0x55f67d193400 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:07.741088) v4 > -4583> 2018-07-11 08:06:07.741896 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.9 10.8.128.107:6807/12617 208 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:07.741088) v4 ==== 2004+0+0 (2450665529 0 0) 0x55f64e20fc00 con 0x55f656adf800 > -4582> 2018-07-11 08:06:07.741908 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.11 10.8.128.114:6802/15919 208 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:07.741088) v4 ==== 2004+0+0 (2450665529 0 0) 0x55f67d193400 con 0x55f64e3db000 > -4581> 2018-07-11 08:06:07.741934 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6810/15287 conn(0x55f64e401000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.4 seq 208 0x55f64fc92800 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:07.741088) v4 > -4580> 2018-07-11 08:06:07.741941 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.108:6802/11967 conn(0x55f656aca000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.8 seq 208 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:07.741088) v4 > -4579> 2018-07-11 08:06:07.741949 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6806/15885 conn(0x55f64e3fe000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.7 seq 208 0x55f67d193400 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:07.741088) v4 > -4578> 2018-07-11 08:06:07.741956 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.8 10.8.128.108:6802/11967 208 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:07.741088) v4 ==== 2004+0+0 (2450665529 0 0) 0x55f64e20fc00 con 0x55f656aca000 > -4577> 2018-07-11 08:06:07.741955 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.4 10.8.128.113:6810/15287 208 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:07.741088) v4 ==== 2004+0+0 (2450665529 0 0) 0x55f64fc92800 con 0x55f64e401000 > -4576> 2018-07-11 08:06:07.741964 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.7 10.8.128.114:6806/15885 208 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:07.741088) v4 ==== 2004+0+0 (2450665529 0 0) 0x55f67d193400 con 0x55f64e3fe000 > -4575> 2018-07-11 08:06:07.741985 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6807/15885 conn(0x55f64e3ff800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.7 seq 208 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:07.741088) v4 > -4574> 2018-07-11 08:06:07.741999 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.7 10.8.128.114:6807/15885 208 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:07.741088) v4 ==== 2004+0+0 (2450665529 0 0) 0x55f64e20fc00 con 0x55f64e3ff800 > -4573> 2018-07-11 08:06:07.742003 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6807/15263 conn(0x55f64e3ee800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.5 seq 208 0x55f64fc92800 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:07.741088) v4 > -4572> 2018-07-11 08:06:07.742022 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.5 10.8.128.113:6807/15263 208 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:07.741088) v4 ==== 2004+0+0 (2450665529 0 0) 0x55f64fc92800 con 0x55f64e3ee800 > -4571> 2018-07-11 08:06:07.742028 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6803/15919 conn(0x55f64e3dc800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.11 seq 208 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:07.741088) v4 > -4570> 2018-07-11 08:06:07.742041 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.11 10.8.128.114:6803/15919 208 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:07.741088) v4 ==== 2004+0+0 (2450665529 0 0) 0x55f64e20fc00 con 0x55f64e3dc800 > -4569> 2018-07-11 08:06:07.742068 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.108:6803/11967 conn(0x55f656acb800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.8 seq 208 0x55f64fc92800 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:07.741088) v4 > -4568> 2018-07-11 08:06:07.742082 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.8 10.8.128.108:6803/11967 208 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:07.741088) v4 ==== 2004+0+0 (2450665529 0 0) 0x55f64fc92800 con 0x55f656acb800 > -4567> 2018-07-11 08:06:07.967306 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f6513b2900 con 0 > -4566> 2018-07-11 08:06:07.969119 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 264 0x55f683ee4780 replica scrub(pg: 6.15,from:0'0,to:430'105428,epoch:509/504,start:6:a8636782:::benchmark_data_magna108_35617_object413047:0,end:6:a8646650:::benchmark_data_magna108_41525_object46805:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -4565> 2018-07-11 08:06:07.969148 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 264 ==== replica scrub(pg: 6.15,from:0'0,to:430'105428,epoch:509/504,start:6:a8636782:::benchmark_data_magna108_35617_object413047:0,end:6:a8646650:::benchmark_data_magna108_41525_object46805:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 217+0+0 (133990568 0 0) 0x55f683ee4780 con 0x55f66315e800 > -4564> 2018-07-11 08:06:07.972090 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 461 0x55f67d193400 MOSDScrubReserve(11.3s5 REQUEST e509) v1 > -4563> 2018-07-11 08:06:07.972109 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 461 ==== MOSDScrubReserve(11.3s5 REQUEST e509) v1 ==== 43+0+0 (433425070 0 0) 0x55f67d193400 con 0x55f668a29000 > -4562> 2018-07-11 08:06:07.972169 7f491a040700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e509) v1 -- 0x55f65fa97c00 con 0 > -4561> 2018-07-11 08:06:07.972562 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 462 0x55f65fa97c00 MOSDScrubReserve(11.3s5 RELEASE e509) v1 > -4560> 2018-07-11 08:06:07.972577 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 462 ==== MOSDScrubReserve(11.3s5 RELEASE e509) v1 ==== 43+0+0 (3603460268 0 0) 0x55f65fa97c00 con 0x55f668a29000 > -4559> 2018-07-11 08:06:07.993904 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.107:0/12617 conn(0x55f64e3c5000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.9 seq 213 0x55f64fc92800 osd_ping(ping e509 stamp 2018-07-11 08:06:07.993567) v4 > -4558> 2018-07-11 08:06:07.993905 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.107:0/12617 conn(0x55f64e3c3800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.9 seq 213 0x55f6841a5400 osd_ping(ping e509 stamp 2018-07-11 08:06:07.993567) v4 > -4557> 2018-07-11 08:06:07.993923 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.9 10.8.128.107:0/12617 213 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:07.993567) v4 ==== 2004+0+0 (1346261794 0 0) 0x55f64fc92800 con 0x55f64e3c5000 > -4556> 2018-07-11 08:06:07.993924 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.9 10.8.128.107:0/12617 213 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:07.993567) v4 ==== 2004+0+0 (1346261794 0 0) 0x55f6841a5400 con 0x55f64e3c3800 > -4555> 2018-07-11 08:06:07.993936 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:07.993567) v4 -- 0x55f66dcebe00 con 0 > -4554> 2018-07-11 08:06:07.993962 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:07.993567) v4 -- 0x55f67db36a00 con 0 > -4553> 2018-07-11 08:06:08.209822 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0a000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 212 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:06:08.845700) v4 > -4552> 2018-07-11 08:06:08.209833 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0b800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 212 0x55f67db36600 osd_ping(ping e509 stamp 2018-07-11 08:06:08.845700) v4 > -4551> 2018-07-11 08:06:08.209847 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.11 10.8.128.114:0/15919 212 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:08.845700) v4 ==== 2004+0+0 (709757507 0 0) 0x55f64e20fc00 con 0x55f663f0a000 > -4550> 2018-07-11 08:06:08.209855 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.11 10.8.128.114:0/15919 212 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:08.845700) v4 ==== 2004+0+0 (709757507 0 0) 0x55f67db36600 con 0x55f663f0b800 > -4549> 2018-07-11 08:06:08.209871 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:08.845700) v4 -- 0x55f6631cc800 con 0 > -4548> 2018-07-11 08:06:08.209903 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:08.845700) v4 -- 0x55f67db36c00 con 0 > -4547> 2018-07-11 08:06:08.338611 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15885 conn(0x55f663216000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 214 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:06:08.974436) v4 > -4546> 2018-07-11 08:06:08.338634 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.7 10.8.128.114:0/15885 214 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:08.974436) v4 ==== 2004+0+0 (2391943083 0 0) 0x55f67d1afe00 con 0x55f663216000 > -4545> 2018-07-11 08:06:08.338657 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:08.974436) v4 -- 0x55f667d08c00 con 0 > -4544> 2018-07-11 08:06:08.338653 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15885 conn(0x55f663217800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 214 0x55f6841a5400 osd_ping(ping e509 stamp 2018-07-11 08:06:08.974436) v4 > -4543> 2018-07-11 08:06:08.338682 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.7 10.8.128.114:0/15885 214 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:08.974436) v4 ==== 2004+0+0 (2391943083 0 0) 0x55f6841a5400 con 0x55f663217800 > -4542> 2018-07-11 08:06:08.338694 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:08.974436) v4 -- 0x55f67db36e00 con 0 > -4541> 2018-07-11 08:06:08.664601 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f6615aa900 con 0 > -4540> 2018-07-11 08:06:08.666157 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 265 0x55f683ee4a00 replica scrub(pg: 6.15,from:0'0,to:461'106890,epoch:509/504,start:6:a8646650:::benchmark_data_magna108_41525_object46805:0,end:6:a8654658:::benchmark_data_magna108_35617_object849773:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -4539> 2018-07-11 08:06:08.666186 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 265 ==== replica scrub(pg: 6.15,from:0'0,to:461'106890,epoch:509/504,start:6:a8646650:::benchmark_data_magna108_41525_object46805:0,end:6:a8654658:::benchmark_data_magna108_35617_object849773:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 217+0+0 (3328002661 0 0) 0x55f683ee4a00 con 0x55f66315e800 > -4538> 2018-07-11 08:06:08.681155 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed7800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 211 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:06:08.738304) v4 > -4537> 2018-07-11 08:06:08.681180 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.1 10.8.128.106:0/12677 211 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:08.738304) v4 ==== 2004+0+0 (3448444104 0 0) 0x55f64e20fc00 con 0x55f663ed7800 > -4536> 2018-07-11 08:06:08.681203 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:08.738304) v4 -- 0x55f667d08c00 con 0 > -4535> 2018-07-11 08:06:08.681244 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed6000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 211 0x55f64fc92200 osd_ping(ping e509 stamp 2018-07-11 08:06:08.738304) v4 > -4534> 2018-07-11 08:06:08.681277 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.1 10.8.128.106:0/12677 211 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:08.738304) v4 ==== 2004+0+0 (3448444104 0 0) 0x55f64fc92200 con 0x55f663ed6000 > -4533> 2018-07-11 08:06:08.681289 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:08.738304) v4 -- 0x55f64fc92e00 con 0 > -4532> 2018-07-11 08:06:08.706955 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15287 conn(0x55f66dce2800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.4 seq 215 0x55f66dcebe00 osd_ping(ping e509 stamp 2018-07-11 08:06:09.952204) v4 > -4531> 2018-07-11 08:06:08.706976 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.4 10.8.128.113:0/15287 215 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:09.952204) v4 ==== 2004+0+0 (774245183 0 0) 0x55f66dcebe00 con 0x55f66dce2800 > -4530> 2018-07-11 08:06:08.706985 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:09.952204) v4 -- 0x55f64fc93000 con 0 > -4529> 2018-07-11 08:06:08.707008 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15287 conn(0x55f66dce1000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.4 seq 215 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:06:09.952204) v4 > -4528> 2018-07-11 08:06:08.707030 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.4 10.8.128.113:0/15287 215 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:09.952204) v4 ==== 2004+0+0 (774245183 0 0) 0x55f67d1afe00 con 0x55f66dce1000 > -4527> 2018-07-11 08:06:08.707041 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:09.952204) v4 -- 0x55f653d4a600 con 0 > -4526> 2018-07-11 08:06:08.839162 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15885 conn(0x55f663217800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 215 0x55f67db36600 osd_ping(ping e509 stamp 2018-07-11 08:06:09.475019) v4 > -4525> 2018-07-11 08:06:08.839162 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15885 conn(0x55f663216000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 215 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:06:09.475019) v4 > -4524> 2018-07-11 08:06:08.839190 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.7 10.8.128.114:0/15885 215 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:09.475019) v4 ==== 2004+0+0 (1353654234 0 0) 0x55f67db36600 con 0x55f663217800 > -4523> 2018-07-11 08:06:08.839193 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.7 10.8.128.114:0/15885 215 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:09.475019) v4 ==== 2004+0+0 (1353654234 0 0) 0x55f64e20fc00 con 0x55f663216000 > -4522> 2018-07-11 08:06:08.839205 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:09.475019) v4 -- 0x55f64e25c200 con 0 > -4521> 2018-07-11 08:06:08.839232 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:09.475019) v4 -- 0x55f67db37000 con 0 > -4520> 2018-07-11 08:06:09.181618 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed6000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 212 0x55f64fc92200 osd_ping(ping e509 stamp 2018-07-11 08:06:09.238858) v4 > -4519> 2018-07-11 08:06:09.181620 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed7800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 212 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:06:09.238858) v4 > -4518> 2018-07-11 08:06:09.181643 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.1 10.8.128.106:0/12677 212 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:09.238858) v4 ==== 2004+0+0 (3760365832 0 0) 0x55f64fc92200 con 0x55f663ed6000 > -4517> 2018-07-11 08:06:09.181645 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.1 10.8.128.106:0/12677 212 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:09.238858) v4 ==== 2004+0+0 (3760365832 0 0) 0x55f67d1afe00 con 0x55f663ed7800 > -4516> 2018-07-11 08:06:09.181663 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:09.238858) v4 -- 0x55f64fc93200 con 0 > -4515> 2018-07-11 08:06:09.181697 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:09.238858) v4 -- 0x55f66dc71a00 con 0 > -4514> 2018-07-11 08:06:09.310127 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0a000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 213 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:06:09.946066) v4 > -4513> 2018-07-11 08:06:09.310147 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.11 10.8.128.114:0/15919 213 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:09.946066) v4 ==== 2004+0+0 (3608383486 0 0) 0x55f64e20fc00 con 0x55f663f0a000 > -4512> 2018-07-11 08:06:09.310159 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:09.946066) v4 -- 0x55f6548fbc00 con 0 > -4511> 2018-07-11 08:06:09.310181 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0b800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 213 0x55f6841a5400 osd_ping(ping e509 stamp 2018-07-11 08:06:09.946066) v4 > -4510> 2018-07-11 08:06:09.310206 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.11 10.8.128.114:0/15919 213 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:09.946066) v4 ==== 2004+0+0 (3608383486 0 0) 0x55f6841a5400 con 0x55f663f0b800 > -4509> 2018-07-11 08:06:09.310218 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:09.946066) v4 -- 0x55f67db37000 con 0 > -4508> 2018-07-11 08:06:09.667486 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.114:6805/15885 conn(0x55f66315a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.7 seq 2565 0x55f66dcebe00 MOSDScrubReserve(5.4 REQUEST e509) v1 > -4507> 2018-07-11 08:06:09.667519 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.7 10.8.128.114:6805/15885 2565 ==== MOSDScrubReserve(5.4 REQUEST e509) v1 ==== 43+0+0 (3003630469 0 0) 0x55f66dcebe00 con 0x55f66315a000 > -4506> 2018-07-11 08:06:09.667570 7f491983f700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.114:6805/15885 -- MOSDScrubReserve(5.4 REJECT e509) v1 -- 0x55f666182e00 con 0 > -4505> 2018-07-11 08:06:09.667845 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.114:6805/15885 conn(0x55f66315a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.7 seq 2566 0x55f64fc93200 MOSDScrubReserve(5.4 RELEASE e509) v1 > -4504> 2018-07-11 08:06:09.667864 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.7 10.8.128.114:6805/15885 2566 ==== MOSDScrubReserve(5.4 RELEASE e509) v1 ==== 43+0+0 (2082093447 0 0) 0x55f64fc93200 con 0x55f66315a000 > -4503> 2018-07-11 08:06:09.694562 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.107:0/12617 conn(0x55f64e3c5000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.9 seq 214 0x55f64fc92200 osd_ping(ping e509 stamp 2018-07-11 08:06:09.694067) v4 > -4502> 2018-07-11 08:06:09.694561 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.107:0/12617 conn(0x55f64e3c3800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.9 seq 214 0x55f67db36600 osd_ping(ping e509 stamp 2018-07-11 08:06:09.694067) v4 > -4501> 2018-07-11 08:06:09.694590 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.9 10.8.128.107:0/12617 214 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:09.694067) v4 ==== 2004+0+0 (1236812769 0 0) 0x55f64fc92200 con 0x55f64e3c5000 > -4500> 2018-07-11 08:06:09.694602 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.9 10.8.128.107:0/12617 214 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:09.694067) v4 ==== 2004+0+0 (1236812769 0 0) 0x55f67db36600 con 0x55f64e3c3800 > -4499> 2018-07-11 08:06:09.694613 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:09.694067) v4 -- 0x55f64fc93800 con 0 > -4498> 2018-07-11 08:06:09.694648 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:09.694067) v4 -- 0x55f67db37200 con 0 > -4497> 2018-07-11 08:06:10.061773 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12651 conn(0x55f663219000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25 cs=1 l=1). rx osd.2 seq 224 0x55f666182e00 osd_ping(ping e509 stamp 2018-07-11 08:06:10.118887) v4 > -4496> 2018-07-11 08:06:10.061794 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.2 10.8.128.106:0/12651 224 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:10.118887) v4 ==== 2004+0+0 (3126651691 0 0) 0x55f666182e00 con 0x55f663219000 > -4495> 2018-07-11 08:06:10.061808 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:10.118887) v4 -- 0x55f64fc93a00 con 0 > -4494> 2018-07-11 08:06:10.061854 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12651 conn(0x55f66321a800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25 cs=1 l=1). rx osd.2 seq 224 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:06:10.118887) v4 > -4493> 2018-07-11 08:06:10.061873 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.2 10.8.128.106:0/12651 224 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:10.118887) v4 ==== 2004+0+0 (3126651691 0 0) 0x55f67d1afe00 con 0x55f66321a800 > -4492> 2018-07-11 08:06:10.061884 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:10.118887) v4 -- 0x55f67d1aee00 con 0 > -4491> 2018-07-11 08:06:10.281212 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15255 conn(0x55f663ec2800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.0 seq 204 0x55f6841a5400 osd_ping(ping e509 stamp 2018-07-11 08:06:11.526516) v4 > -4490> 2018-07-11 08:06:10.281236 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.0 10.8.128.113:0/15255 204 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:11.526516) v4 ==== 2004+0+0 (3318819096 0 0) 0x55f6841a5400 con 0x55f663ec2800 > -4489> 2018-07-11 08:06:10.281257 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:11.526516) v4 -- 0x55f67db37400 con 0 > -4488> 2018-07-11 08:06:10.281257 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15255 conn(0x55f663ec1000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.0 seq 204 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:06:11.526516) v4 > -4487> 2018-07-11 08:06:10.281280 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.0 10.8.128.113:0/15255 204 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:11.526516) v4 ==== 2004+0+0 (3318819096 0 0) 0x55f64e20fc00 con 0x55f663ec1000 > -4486> 2018-07-11 08:06:10.281292 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:11.526516) v4 -- 0x55f669deb800 con 0 > -4485> 2018-07-11 08:06:10.301201 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f655a11d40 con 0 > -4484> 2018-07-11 08:06:10.302720 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 266 0x55f683ee4c80 replica scrub(pg: 6.15,from:0'0,to:430'106310,epoch:509/504,start:6:a8654658:::benchmark_data_magna108_35617_object849773:0,end:6:a86625d5:::benchmark_data_magna108_35617_object727138:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -4483> 2018-07-11 08:06:10.302744 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 266 ==== replica scrub(pg: 6.15,from:0'0,to:430'106310,epoch:509/504,start:6:a8654658:::benchmark_data_magna108_35617_object849773:0,end:6:a86625d5:::benchmark_data_magna108_35617_object727138:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (2954235018 0 0) 0x55f683ee4c80 con 0x55f66315e800 > -4482> 2018-07-11 08:06:10.368183 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 30] Generated table #37049: 2638241 keys, 74546099 bytes > -4481> 2018-07-11 08:06:10.368238 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296370368221, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 37049, "file_size": 74546099, "table_properties": {"data_size": 67112109, "index_size": 689817, "filter_size": 6743183, "raw_key_size": 76215847, "raw_average_key_size": 28, "raw_value_size": 31365750, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} > -4480> 2018-07-11 08:06:10.434031 7f491e048700 10 monclient: tick > -4479> 2018-07-11 08:06:10.434045 7f491e048700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2018-07-11 08:05:40.434043) > -4478> 2018-07-11 08:06:10.531299 7f491d046700 1 -- 10.8.128.107:6800/14173 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55f667d77600 con 0 > -4477> 2018-07-11 08:06:10.531386 7f491d046700 1 -- 10.8.128.107:6800/14173 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55f667d76840 con 0 > -4476> 2018-07-11 08:06:11.010579 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0a000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 214 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:06:11.646458) v4 > -4475> 2018-07-11 08:06:11.010599 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0b800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 214 0x55f67db36600 osd_ping(ping e509 stamp 2018-07-11 08:06:11.646458) v4 > -4474> 2018-07-11 08:06:11.010610 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.11 10.8.128.114:0/15919 214 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:11.646458) v4 ==== 2004+0+0 (1234061352 0 0) 0x55f67d1afe00 con 0x55f663f0a000 > -4473> 2018-07-11 08:06:11.010620 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.11 10.8.128.114:0/15919 214 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:11.646458) v4 ==== 2004+0+0 (1234061352 0 0) 0x55f67db36600 con 0x55f663f0b800 > -4472> 2018-07-11 08:06:11.010625 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:11.646458) v4 -- 0x55f669deb800 con 0 > -4471> 2018-07-11 08:06:11.010657 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:11.646458) v4 -- 0x55f67db37400 con 0 > -4470> 2018-07-11 08:06:11.041663 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f655a10000 con 0 > -4469> 2018-07-11 08:06:11.043198 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 267 0x55f683ee4f00 replica scrub(pg: 6.15,from:0'0,to:430'106472,epoch:509/504,start:6:a86625d5:::benchmark_data_magna108_35617_object727138:0,end:6:a8670bba:::benchmark_data_magna108.ceph.redhat.com_2478_object355037:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -4468> 2018-07-11 08:06:11.043230 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 267 ==== replica scrub(pg: 6.15,from:0'0,to:430'106472,epoch:509/504,start:6:a86625d5:::benchmark_data_magna108_35617_object727138:0,end:6:a8670bba:::benchmark_data_magna108.ceph.redhat.com_2478_object355037:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (307080686 0 0) 0x55f683ee4f00 con 0x55f66315e800 > -4467> 2018-07-11 08:06:11.482102 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed7800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 213 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:06:11.539261) v4 > -4466> 2018-07-11 08:06:11.482126 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.1 10.8.128.106:0/12677 213 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:11.539261) v4 ==== 2004+0+0 (344837959 0 0) 0x55f64e20fc00 con 0x55f663ed7800 > -4465> 2018-07-11 08:06:11.482138 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:11.539261) v4 -- 0x55f669deb800 con 0 > -4464> 2018-07-11 08:06:11.482148 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed6000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 213 0x55f64fc93a00 osd_ping(ping e509 stamp 2018-07-11 08:06:11.539261) v4 > -4463> 2018-07-11 08:06:11.482177 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.1 10.8.128.106:0/12677 213 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:11.539261) v4 ==== 2004+0+0 (344837959 0 0) 0x55f64fc93a00 con 0x55f663ed6000 > -4462> 2018-07-11 08:06:11.482195 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:11.539261) v4 -- 0x55f64fc93c00 con 0 > -4461> 2018-07-11 08:06:11.667780 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.114:6805/15885 conn(0x55f66315a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.7 seq 2567 0x55f64fc92200 MOSDScrubReserve(5.4 REQUEST e509) v1 > -4460> 2018-07-11 08:06:11.667815 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.7 10.8.128.114:6805/15885 2567 ==== MOSDScrubReserve(5.4 REQUEST e509) v1 ==== 43+0+0 (3003630469 0 0) 0x55f64fc92200 con 0x55f66315a000 > -4459> 2018-07-11 08:06:11.667903 7f491983f700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.114:6805/15885 -- MOSDScrubReserve(5.4 REJECT e509) v1 -- 0x55f64fc93200 con 0 > -4458> 2018-07-11 08:06:11.668220 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.114:6805/15885 conn(0x55f66315a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.7 seq 2568 0x55f64fc93c00 MOSDScrubReserve(5.4 RELEASE e509) v1 > -4457> 2018-07-11 08:06:11.668264 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.7 10.8.128.114:6805/15885 2568 ==== MOSDScrubReserve(5.4 RELEASE e509) v1 ==== 43+0+0 (2082093447 0 0) 0x55f64fc93c00 con 0x55f66315a000 > -4456> 2018-07-11 08:06:11.813970 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.115:0/13323 conn(0x55f663ebf800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.6 seq 198 0x55f64fc93a00 osd_ping(ping e509 stamp 2018-07-11 08:06:12.166990) v4 > -4455> 2018-07-11 08:06:11.814002 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.6 10.8.128.115:0/13323 198 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:12.166990) v4 ==== 2004+0+0 (1574352515 0 0) 0x55f64fc93a00 con 0x55f663ebf800 > -4454> 2018-07-11 08:06:11.814013 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.115:0/13323 conn(0x55f663ebe000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.6 seq 198 0x55f6841a5400 osd_ping(ping e509 stamp 2018-07-11 08:06:12.166990) v4 > -4453> 2018-07-11 08:06:11.814024 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:12.166990) v4 -- 0x55f673908000 con 0 > -4452> 2018-07-11 08:06:11.814034 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.6 10.8.128.115:0/13323 198 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:12.166990) v4 ==== 2004+0+0 (1574352515 0 0) 0x55f6841a5400 con 0x55f663ebe000 > -4451> 2018-07-11 08:06:11.814055 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:12.166990) v4 -- 0x55f67db37600 con 0 > -4450> 2018-07-11 08:06:11.918647 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f66dbd2480 con 0 > -4449> 2018-07-11 08:06:11.920244 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 268 0x55f683ee5180 replica scrub(pg: 6.15,from:0'0,to:430'106752,epoch:509/504,start:6:a8670bba:::benchmark_data_magna108.ceph.redhat.com_2478_object355037:0,end:6:a8682302:::benchmark_data_magna108_35617_object529077:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -4448> 2018-07-11 08:06:11.920281 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 268 ==== replica scrub(pg: 6.15,from:0'0,to:430'106752,epoch:509/504,start:6:a8670bba:::benchmark_data_magna108.ceph.redhat.com_2478_object355037:0,end:6:a8682302:::benchmark_data_magna108_35617_object529077:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (2293622898 0 0) 0x55f683ee5180 con 0x55f66315e800 > -4447> 2018-07-11 08:06:11.935231 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12650 conn(0x55f663ed9000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.3 seq 210 0x55f67db36600 osd_ping(ping e509 stamp 2018-07-11 08:06:11.992436) v4 > -4446> 2018-07-11 08:06:11.935268 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.3 10.8.128.106:0/12650 210 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:11.992436) v4 ==== 2004+0+0 (3744662512 0 0) 0x55f67db36600 con 0x55f663ed9000 > -4445> 2018-07-11 08:06:11.935283 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:11.992436) v4 -- 0x55f67db37600 con 0 > -4444> 2018-07-11 08:06:11.935279 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12650 conn(0x55f663eda800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.3 seq 210 0x55f64fc93200 osd_ping(ping e509 stamp 2018-07-11 08:06:11.992436) v4 > -4443> 2018-07-11 08:06:11.935297 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.3 10.8.128.106:0/12650 210 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:11.992436) v4 ==== 2004+0+0 (3744662512 0 0) 0x55f64fc93200 con 0x55f663eda800 > -4442> 2018-07-11 08:06:11.935309 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:11.992436) v4 -- 0x55f673908200 con 0 > -4441> 2018-07-11 08:06:11.972583 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 463 0x55f6841a5400 MOSDScrubReserve(11.3s5 REQUEST e509) v1 > -4440> 2018-07-11 08:06:11.972601 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 463 ==== MOSDScrubReserve(11.3s5 REQUEST e509) v1 ==== 43+0+0 (433425070 0 0) 0x55f6841a5400 con 0x55f668a29000 > -4439> 2018-07-11 08:06:11.972676 7f491a040700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e509) v1 -- 0x55f65fa97c00 con 0 > -4438> 2018-07-11 08:06:11.972983 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 464 0x55f67db37600 MOSDScrubReserve(11.3s5 RELEASE e509) v1 > -4437> 2018-07-11 08:06:11.972997 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 464 ==== MOSDScrubReserve(11.3s5 RELEASE e509) v1 ==== 43+0+0 (3603460268 0 0) 0x55f67db37600 con 0x55f668a29000 > -4436> 2018-07-11 08:06:12.204038 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.108:0/11967 conn(0x55f663f6f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=1). rx osd.8 seq 204 0x55f67db36600 osd_ping(ping e509 stamp 2018-07-11 08:06:11.964175) v4 > -4435> 2018-07-11 08:06:12.204062 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.8 10.8.128.108:0/11967 204 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:11.964175) v4 ==== 2004+0+0 (1303210061 0 0) 0x55f67db36600 con 0x55f663f6f000 > -4434> 2018-07-11 08:06:12.204072 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:11.964175) v4 -- 0x55f67db37a00 con 0 > -4433> 2018-07-11 08:06:12.204106 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.108:0/11967 conn(0x55f663f70800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=1). rx osd.8 seq 204 0x55f673908000 osd_ping(ping e509 stamp 2018-07-11 08:06:11.964175) v4 > -4432> 2018-07-11 08:06:12.204127 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.8 10.8.128.108:0/11967 204 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:11.964175) v4 ==== 2004+0+0 (1303210061 0 0) 0x55f673908000 con 0x55f663f70800 > -4431> 2018-07-11 08:06:12.204143 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:11.964175) v4 -- 0x55f673908400 con 0 > -4430> 2018-07-11 08:06:12.207357 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15287 conn(0x55f66dce2800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.4 seq 216 0x55f64fc93200 osd_ping(ping e509 stamp 2018-07-11 08:06:13.452652) v4 > -4429> 2018-07-11 08:06:12.207377 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.4 10.8.128.113:0/15287 216 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:13.452652) v4 ==== 2004+0+0 (3126902964 0 0) 0x55f64fc93200 con 0x55f66dce2800 > -4428> 2018-07-11 08:06:12.207398 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:13.452652) v4 -- 0x55f673908600 con 0 > -4427> 2018-07-11 08:06:12.207411 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15287 conn(0x55f66dce1000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.4 seq 216 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:06:13.452652) v4 > -4426> 2018-07-11 08:06:12.207438 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.4 10.8.128.113:0/15287 216 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:13.452652) v4 ==== 2004+0+0 (3126902964 0 0) 0x55f67d1afe00 con 0x55f66dce1000 > -4425> 2018-07-11 08:06:12.207455 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:13.452652) v4 -- 0x55f6548fa400 con 0 > -4424> 2018-07-11 08:06:12.339559 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15885 conn(0x55f663216000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 216 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:06:12.975434) v4 > -4423> 2018-07-11 08:06:12.339580 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.7 10.8.128.114:0/15885 216 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:12.975434) v4 ==== 2004+0+0 (684451475 0 0) 0x55f64e20fc00 con 0x55f663216000 > -4422> 2018-07-11 08:06:12.339590 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:12.975434) v4 -- 0x55f64cc81600 con 0 > -4421> 2018-07-11 08:06:12.339598 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15885 conn(0x55f663217800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 216 0x55f65fa97c00 osd_ping(ping e509 stamp 2018-07-11 08:06:12.975434) v4 > -4420> 2018-07-11 08:06:12.339624 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.7 10.8.128.114:0/15885 216 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:12.975434) v4 ==== 2004+0+0 (684451475 0 0) 0x55f65fa97c00 con 0x55f663217800 > -4419> 2018-07-11 08:06:12.339640 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:12.975434) v4 -- 0x55f67db37c00 con 0 > -4418> 2018-07-11 08:06:12.576121 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f650b0a6c0 con 0 > -4417> 2018-07-11 08:06:12.577900 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 269 0x55f683ee5400 replica scrub(pg: 6.15,from:0'0,to:430'105384,epoch:509/504,start:6:a8682302:::benchmark_data_magna108_35617_object529077:0,end:6:a86927fc:::benchmark_data_magna108.ceph.redhat.com_2478_object305030:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -4416> 2018-07-11 08:06:12.577931 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 269 ==== replica scrub(pg: 6.15,from:0'0,to:430'105384,epoch:509/504,start:6:a8682302:::benchmark_data_magna108_35617_object529077:0,end:6:a86927fc:::benchmark_data_magna108.ceph.redhat.com_2478_object305030:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (1348552581 0 0) 0x55f683ee5400 con 0x55f66315e800 > -4415> 2018-07-11 08:06:12.710998 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0b800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 215 0x55f67db36600 osd_ping(ping e509 stamp 2018-07-11 08:06:13.346881) v4 > -4414> 2018-07-11 08:06:12.710997 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0a000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 215 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:06:13.346881) v4 > -4413> 2018-07-11 08:06:12.711027 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.11 10.8.128.114:0/15919 215 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:13.346881) v4 ==== 2004+0+0 (185145050 0 0) 0x55f67db36600 con 0x55f663f0b800 > -4412> 2018-07-11 08:06:12.711030 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.11 10.8.128.114:0/15919 215 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:13.346881) v4 ==== 2004+0+0 (185145050 0 0) 0x55f67d1afe00 con 0x55f663f0a000 > -4411> 2018-07-11 08:06:12.711042 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:13.346881) v4 -- 0x55f67db37c00 con 0 > -4410> 2018-07-11 08:06:12.711071 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:13.346881) v4 -- 0x55f64cc81600 con 0 > -4409> 2018-07-11 08:06:12.761048 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0e800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 219 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:06:14.006409) v4 > -4408> 2018-07-11 08:06:12.761075 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.5 10.8.128.113:0/15263 219 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:14.006409) v4 ==== 2004+0+0 (2735252945 0 0) 0x55f64e20fc00 con 0x55f663f0e800 > -4407> 2018-07-11 08:06:12.761094 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:14.006409) v4 -- 0x55f64cc81600 con 0 > -4406> 2018-07-11 08:06:12.761088 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0d000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 219 0x55f673908000 osd_ping(ping e509 stamp 2018-07-11 08:06:14.006409) v4 > -4405> 2018-07-11 08:06:12.761109 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.5 10.8.128.113:0/15263 219 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:14.006409) v4 ==== 2004+0+0 (2735252945 0 0) 0x55f673908000 con 0x55f663f0d000 > -4404> 2018-07-11 08:06:12.761126 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:14.006409) v4 -- 0x55f673908600 con 0 > -4403> 2018-07-11 08:06:12.962428 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12651 conn(0x55f663219000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25 cs=1 l=1). rx osd.2 seq 225 0x55f64fc93200 osd_ping(ping e509 stamp 2018-07-11 08:06:13.019467) v4 > -4402> 2018-07-11 08:06:12.962450 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.2 10.8.128.106:0/12651 225 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:13.019467) v4 ==== 2004+0+0 (4224095561 0 0) 0x55f64fc93200 con 0x55f663219000 > -4401> 2018-07-11 08:06:12.962466 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:13.019467) v4 -- 0x55f673908600 con 0 > -4400> 2018-07-11 08:06:12.962573 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12651 conn(0x55f66321a800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25 cs=1 l=1). rx osd.2 seq 225 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:06:13.019467) v4 > -4399> 2018-07-11 08:06:12.962598 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.2 10.8.128.106:0/12651 225 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:13.019467) v4 ==== 2004+0+0 (4224095561 0 0) 0x55f67d1afe00 con 0x55f66321a800 > -4398> 2018-07-11 08:06:12.962615 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:13.019467) v4 -- 0x55f65e71cc00 con 0 > -4397> 2018-07-11 08:06:12.972865 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 465 0x55f65fa97c00 MOSDScrubReserve(11.3s5 REQUEST e509) v1 > -4396> 2018-07-11 08:06:12.972884 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 465 ==== MOSDScrubReserve(11.3s5 REQUEST e509) v1 ==== 43+0+0 (433425070 0 0) 0x55f65fa97c00 con 0x55f668a29000 > -4395> 2018-07-11 08:06:12.972951 7f491a040700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e509) v1 -- 0x55f67db37600 con 0 > -4394> 2018-07-11 08:06:12.973328 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 466 0x55f67db37600 MOSDScrubReserve(11.3s5 RELEASE e509) v1 > -4393> 2018-07-11 08:06:12.973343 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 466 ==== MOSDScrubReserve(11.3s5 RELEASE e509) v1 ==== 43+0+0 (3603460268 0 0) 0x55f67db37600 con 0x55f668a29000 > -4392> 2018-07-11 08:06:13.041547 7f491803c700 5 osd.10 509 heartbeat: osd_stat(790 GB used, 141 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,0,1]) > -4391> 2018-07-11 08:06:13.041562 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6802/15255 -- osd_ping(ping e509 stamp 2018-07-11 08:06:13.041560) v4 -- 0x55f66bf10a00 con 0 > -4390> 2018-07-11 08:06:13.041586 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6803/15255 -- osd_ping(ping e509 stamp 2018-07-11 08:06:13.041560) v4 -- 0x55f66bf10c00 con 0 > -4389> 2018-07-11 08:06:13.041601 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6806/12677 -- osd_ping(ping e509 stamp 2018-07-11 08:06:13.041560) v4 -- 0x55f66bf10e00 con 0 > -4388> 2018-07-11 08:06:13.041616 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6807/12677 -- osd_ping(ping e509 stamp 2018-07-11 08:06:13.041560) v4 -- 0x55f66bf11000 con 0 > -4387> 2018-07-11 08:06:13.041629 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6808/12651 -- osd_ping(ping e509 stamp 2018-07-11 08:06:13.041560) v4 -- 0x55f66bf11200 con 0 > -4386> 2018-07-11 08:06:13.041639 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6810/12651 -- osd_ping(ping e509 stamp 2018-07-11 08:06:13.041560) v4 -- 0x55f66bf11400 con 0 > -4385> 2018-07-11 08:06:13.041651 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6809/12650 -- osd_ping(ping e509 stamp 2018-07-11 08:06:13.041560) v4 -- 0x55f66bf11600 con 0 > -4384> 2018-07-11 08:06:13.041663 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6811/12650 -- osd_ping(ping e509 stamp 2018-07-11 08:06:13.041560) v4 -- 0x55f66bf11800 con 0 > -4383> 2018-07-11 08:06:13.041681 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6810/15287 -- osd_ping(ping e509 stamp 2018-07-11 08:06:13.041560) v4 -- 0x55f66bf11a00 con 0 > -4382> 2018-07-11 08:06:13.041692 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6811/15287 -- osd_ping(ping e509 stamp 2018-07-11 08:06:13.041560) v4 -- 0x55f66bf11c00 con 0 > -4381> 2018-07-11 08:06:13.041705 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6806/15263 -- osd_ping(ping e509 stamp 2018-07-11 08:06:13.041560) v4 -- 0x55f66bf11e00 con 0 > -4380> 2018-07-11 08:06:13.041725 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6807/15263 -- osd_ping(ping e509 stamp 2018-07-11 08:06:13.041560) v4 -- 0x55f67d150000 con 0 > -4379> 2018-07-11 08:06:13.041738 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.115:6802/13323 -- osd_ping(ping e509 stamp 2018-07-11 08:06:13.041560) v4 -- 0x55f67d150200 con 0 > -4378> 2018-07-11 08:06:13.041751 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.115:6803/13323 -- osd_ping(ping e509 stamp 2018-07-11 08:06:13.041560) v4 -- 0x55f67d150400 con 0 > -4377> 2018-07-11 08:06:13.041762 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6806/15885 -- osd_ping(ping e509 stamp 2018-07-11 08:06:13.041560) v4 -- 0x55f67d150600 con 0 > -4376> 2018-07-11 08:06:13.041774 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6807/15885 -- osd_ping(ping e509 stamp 2018-07-11 08:06:13.041560) v4 -- 0x55f67d150800 con 0 > -4375> 2018-07-11 08:06:13.041785 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.108:6802/11967 -- osd_ping(ping e509 stamp 2018-07-11 08:06:13.041560) v4 -- 0x55f67d150a00 con 0 > -4374> 2018-07-11 08:06:13.041797 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.108:6803/11967 -- osd_ping(ping e509 stamp 2018-07-11 08:06:13.041560) v4 -- 0x55f67d150c00 con 0 > -4373> 2018-07-11 08:06:13.041809 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.107:6806/12617 -- osd_ping(ping e509 stamp 2018-07-11 08:06:13.041560) v4 -- 0x55f67d150e00 con 0 > -4372> 2018-07-11 08:06:13.041821 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.107:6807/12617 -- osd_ping(ping e509 stamp 2018-07-11 08:06:13.041560) v4 -- 0x55f67d151000 con 0 > -4371> 2018-07-11 08:06:13.041833 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6802/15919 -- osd_ping(ping e509 stamp 2018-07-11 08:06:13.041560) v4 -- 0x55f67d151200 con 0 > -4370> 2018-07-11 08:06:13.041845 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6803/15919 -- osd_ping(ping e509 stamp 2018-07-11 08:06:13.041560) v4 -- 0x55f67d151400 con 0 > -4369> 2018-07-11 08:06:13.041958 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6803/15255 conn(0x55f656ad5800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 209 0x55f67d14e000 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:13.041560) v4 > -4368> 2018-07-11 08:06:13.041982 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.0 10.8.128.113:6803/15255 209 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:13.041560) v4 ==== 2004+0+0 (2360672137 0 0) 0x55f67d14e000 con 0x55f656ad5800 > -4367> 2018-07-11 08:06:13.041987 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6802/15255 conn(0x55f656ad4000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 209 0x55f64fc93200 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:13.041560) v4 > -4366> 2018-07-11 08:06:13.042017 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.0 10.8.128.113:6802/15255 209 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:13.041560) v4 ==== 2004+0+0 (2360672137 0 0) 0x55f64fc93200 con 0x55f656ad4000 > -4365> 2018-07-11 08:06:13.042075 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6807/12677 conn(0x55f64e3d9800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.1 seq 209 0x55f64fc93200 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:13.041560) v4 > -4364> 2018-07-11 08:06:13.042127 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6806/15263 conn(0x55f64e3ed000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.5 seq 209 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:13.041560) v4 > -4363> 2018-07-11 08:06:13.042143 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.107:6806/12617 conn(0x55f656ade000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.9 seq 209 0x55f67d14e000 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:13.041560) v4 > -4362> 2018-07-11 08:06:13.042133 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.1 10.8.128.106:6807/12677 209 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:13.041560) v4 ==== 2004+0+0 (2360672137 0 0) 0x55f64fc93200 con 0x55f64e3d9800 > -4361> 2018-07-11 08:06:13.042166 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.9 10.8.128.107:6806/12617 209 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:13.041560) v4 ==== 2004+0+0 (2360672137 0 0) 0x55f67d14e000 con 0x55f656ade000 > -4360> 2018-07-11 08:06:13.042147 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.5 10.8.128.113:6806/15263 209 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:13.041560) v4 ==== 2004+0+0 (2360672137 0 0) 0x55f64e20fc00 con 0x55f64e3ed000 > -4359> 2018-07-11 08:06:13.042214 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6810/12651 conn(0x55f64e3eb800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.2 seq 209 0x55f67d14e000 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:13.041560) v4 > -4358> 2018-07-11 08:06:13.042229 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.2 10.8.128.106:6810/12651 209 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:13.041560) v4 ==== 2004+0+0 (2360672137 0 0) 0x55f67d14e000 con 0x55f64e3eb800 > -4357> 2018-07-11 08:06:13.042229 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6808/12651 conn(0x55f64e3ea000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.2 seq 209 0x55f64fc93200 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:13.041560) v4 > -4356> 2018-07-11 08:06:13.042241 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6809/12650 conn(0x55f656ad7000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.3 seq 209 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:13.041560) v4 > -4355> 2018-07-11 08:06:13.042256 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.2 10.8.128.106:6808/12651 209 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:13.041560) v4 ==== 2004+0+0 (2360672137 0 0) 0x55f64fc93200 con 0x55f64e3ea000 > -4354> 2018-07-11 08:06:13.042264 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6811/15287 conn(0x55f64e402800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.4 seq 209 0x55f67d14e000 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:13.041560) v4 > -4353> 2018-07-11 08:06:13.042271 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.3 10.8.128.106:6809/12650 209 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:13.041560) v4 ==== 2004+0+0 (2360672137 0 0) 0x55f64e20fc00 con 0x55f656ad7000 > -4352> 2018-07-11 08:06:13.042277 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.4 10.8.128.113:6811/15287 209 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:13.041560) v4 ==== 2004+0+0 (2360672137 0 0) 0x55f67d14e000 con 0x55f64e402800 > -4351> 2018-07-11 08:06:13.042295 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.107:6807/12617 conn(0x55f656adf800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.9 seq 209 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:13.041560) v4 > -4350> 2018-07-11 08:06:13.042305 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.9 10.8.128.107:6807/12617 209 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:13.041560) v4 ==== 2004+0+0 (2360672137 0 0) 0x55f64e20fc00 con 0x55f656adf800 > -4349> 2018-07-11 08:06:13.042308 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.115:6802/13323 conn(0x55f656acd000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.6 seq 209 0x55f67d14e000 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:13.041560) v4 > -4348> 2018-07-11 08:06:13.042300 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6810/15287 conn(0x55f64e401000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.4 seq 209 0x55f64fc93200 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:13.041560) v4 > -4347> 2018-07-11 08:06:13.042323 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.6 10.8.128.115:6802/13323 209 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:13.041560) v4 ==== 2004+0+0 (2360672137 0 0) 0x55f67d14e000 con 0x55f656acd000 > -4346> 2018-07-11 08:06:13.042323 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6806/12677 conn(0x55f64e3d8000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.1 seq 209 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:13.041560) v4 > -4345> 2018-07-11 08:06:13.042331 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.1 10.8.128.106:6806/12677 209 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:13.041560) v4 ==== 2004+0+0 (2360672137 0 0) 0x55f64e20fc00 con 0x55f64e3d8000 > -4344> 2018-07-11 08:06:13.042331 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.4 10.8.128.113:6810/15287 209 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:13.041560) v4 ==== 2004+0+0 (2360672137 0 0) 0x55f64fc93200 con 0x55f64e401000 > -4343> 2018-07-11 08:06:13.042351 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6806/15885 conn(0x55f64e3fe000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.7 seq 209 0x55f67d14e000 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:13.041560) v4 > -4342> 2018-07-11 08:06:13.042363 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.7 10.8.128.114:6806/15885 209 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:13.041560) v4 ==== 2004+0+0 (2360672137 0 0) 0x55f67d14e000 con 0x55f64e3fe000 > -4341> 2018-07-11 08:06:13.042364 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.115:6803/13323 conn(0x55f656ace800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.6 seq 209 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:13.041560) v4 > -4340> 2018-07-11 08:06:13.042373 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.6 10.8.128.115:6803/13323 209 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:13.041560) v4 ==== 2004+0+0 (2360672137 0 0) 0x55f64e20fc00 con 0x55f656ace800 > -4339> 2018-07-11 08:06:13.042377 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6811/12650 conn(0x55f656ad8800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.3 seq 209 0x55f64fc93200 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:13.041560) v4 > -4338> 2018-07-11 08:06:13.042395 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.3 10.8.128.106:6811/12650 209 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:13.041560) v4 ==== 2004+0+0 (2360672137 0 0) 0x55f64fc93200 con 0x55f656ad8800 > -4337> 2018-07-11 08:06:13.042400 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6802/15919 conn(0x55f64e3db000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.11 seq 209 0x55f67d14e000 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:13.041560) v4 > -4336> 2018-07-11 08:06:13.042417 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.11 10.8.128.114:6802/15919 209 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:13.041560) v4 ==== 2004+0+0 (2360672137 0 0) 0x55f67d14e000 con 0x55f64e3db000 > -4335> 2018-07-11 08:06:13.042425 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.108:6802/11967 conn(0x55f656aca000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.8 seq 209 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:13.041560) v4 > -4334> 2018-07-11 08:06:13.042439 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.8 10.8.128.108:6802/11967 209 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:13.041560) v4 ==== 2004+0+0 (2360672137 0 0) 0x55f64e20fc00 con 0x55f656aca000 > -4333> 2018-07-11 08:06:13.042442 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6807/15263 conn(0x55f64e3ee800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.5 seq 209 0x55f64fc93200 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:13.041560) v4 > -4332> 2018-07-11 08:06:13.042460 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.5 10.8.128.113:6807/15263 209 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:13.041560) v4 ==== 2004+0+0 (2360672137 0 0) 0x55f64fc93200 con 0x55f64e3ee800 > -4331> 2018-07-11 08:06:13.042473 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6807/15885 conn(0x55f64e3ff800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.7 seq 209 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:13.041560) v4 > -4330> 2018-07-11 08:06:13.042485 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.7 10.8.128.114:6807/15885 209 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:13.041560) v4 ==== 2004+0+0 (2360672137 0 0) 0x55f64e20fc00 con 0x55f64e3ff800 > -4329> 2018-07-11 08:06:13.042505 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.108:6803/11967 conn(0x55f656acb800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.8 seq 209 0x55f64fc93200 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:13.041560) v4 > -4328> 2018-07-11 08:06:13.042513 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6803/15919 conn(0x55f64e3dc800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.11 seq 209 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:13.041560) v4 > -4327> 2018-07-11 08:06:13.042523 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.11 10.8.128.114:6803/15919 209 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:13.041560) v4 ==== 2004+0+0 (2360672137 0 0) 0x55f64e20fc00 con 0x55f64e3dc800 > -4326> 2018-07-11 08:06:13.042523 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.8 10.8.128.108:6803/11967 209 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:13.041560) v4 ==== 2004+0+0 (2360672137 0 0) 0x55f64fc93200 con 0x55f656acb800 > -4325> 2018-07-11 08:06:13.194985 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.107:0/12617 conn(0x55f64e3c3800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.9 seq 215 0x55f67d14e000 osd_ping(ping e509 stamp 2018-07-11 08:06:13.194651) v4 > -4324> 2018-07-11 08:06:13.194991 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.107:0/12617 conn(0x55f64e3c5000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.9 seq 215 0x55f64fc93200 osd_ping(ping e509 stamp 2018-07-11 08:06:13.194651) v4 > -4323> 2018-07-11 08:06:13.195005 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.9 10.8.128.107:0/12617 215 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:13.194651) v4 ==== 2004+0+0 (2697218355 0 0) 0x55f67d14e000 con 0x55f64e3c3800 > -4322> 2018-07-11 08:06:13.195007 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.9 10.8.128.107:0/12617 215 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:13.194651) v4 ==== 2004+0+0 (2697218355 0 0) 0x55f64fc93200 con 0x55f64e3c5000 > -4321> 2018-07-11 08:06:13.195014 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:13.194651) v4 -- 0x55f67d14e200 con 0 > -4320> 2018-07-11 08:06:13.195038 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:13.194651) v4 -- 0x55f66bf11000 con 0 > -4319> 2018-07-11 08:06:13.292774 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f65393e900 con 0 > -4318> 2018-07-11 08:06:13.294348 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 270 0x55f683ee5680 replica scrub(pg: 6.15,from:0'0,to:430'105744,epoch:509/504,start:6:a86927fc:::benchmark_data_magna108.ceph.redhat.com_2478_object305030:0,end:6:a869f45d:::benchmark_data_magna108.ceph.redhat.com_2478_object401868:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -4317> 2018-07-11 08:06:13.294373 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 270 ==== replica scrub(pg: 6.15,from:0'0,to:430'105744,epoch:509/504,start:6:a86927fc:::benchmark_data_magna108.ceph.redhat.com_2478_object305030:0,end:6:a869f45d:::benchmark_data_magna108.ceph.redhat.com_2478_object401868:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 248+0+0 (2856673910 0 0) 0x55f683ee5680 con 0x55f66315e800 > -4316> 2018-07-11 08:06:13.635722 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12650 conn(0x55f663eda800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.3 seq 211 0x55f673908000 osd_ping(ping e509 stamp 2018-07-11 08:06:13.692891) v4 > -4315> 2018-07-11 08:06:13.635746 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.3 10.8.128.106:0/12650 211 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:13.692891) v4 ==== 2004+0+0 (2080096004 0 0) 0x55f673908000 con 0x55f663eda800 > -4314> 2018-07-11 08:06:13.635756 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:13.692891) v4 -- 0x55f66bf11000 con 0 > -4313> 2018-07-11 08:06:13.635768 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12650 conn(0x55f663ed9000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.3 seq 211 0x55f67db36600 osd_ping(ping e509 stamp 2018-07-11 08:06:13.692891) v4 > -4312> 2018-07-11 08:06:13.635793 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.3 10.8.128.106:0/12650 211 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:13.692891) v4 ==== 2004+0+0 (2080096004 0 0) 0x55f67db36600 con 0x55f663ed9000 > -4311> 2018-07-11 08:06:13.635805 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:13.692891) v4 -- 0x55f67d14e200 con 0 > -4310> 2018-07-11 08:06:13.973139 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 467 0x55f67d14e000 MOSDScrubReserve(11.3s5 REQUEST e509) v1 > -4309> 2018-07-11 08:06:13.973173 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 467 ==== MOSDScrubReserve(11.3s5 REQUEST e509) v1 ==== 43+0+0 (433425070 0 0) 0x55f67d14e000 con 0x55f668a29000 > -4308> 2018-07-11 08:06:13.973261 7f491a040700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e509) v1 -- 0x55f67db37600 con 0 > -4307> 2018-07-11 08:06:13.973635 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 468 0x55f67d14e200 MOSDScrubReserve(11.3s5 RELEASE e509) v1 > -4306> 2018-07-11 08:06:13.973667 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 468 ==== MOSDScrubReserve(11.3s5 RELEASE e509) v1 ==== 43+0+0 (3603460268 0 0) 0x55f67d14e200 con 0x55f668a29000 > -4305> 2018-07-11 08:06:14.114513 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.115:0/13323 conn(0x55f663ebe000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.6 seq 199 0x55f67db36600 osd_ping(ping e509 stamp 2018-07-11 08:06:14.467470) v4 > -4304> 2018-07-11 08:06:14.114533 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.115:0/13323 conn(0x55f663ebf800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.6 seq 199 0x55f64fc93200 osd_ping(ping e509 stamp 2018-07-11 08:06:14.467470) v4 > -4303> 2018-07-11 08:06:14.114546 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.6 10.8.128.115:0/13323 199 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:14.467470) v4 ==== 2004+0+0 (1480490848 0 0) 0x55f67db36600 con 0x55f663ebe000 > -4302> 2018-07-11 08:06:14.114554 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.6 10.8.128.115:0/13323 199 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:14.467470) v4 ==== 2004+0+0 (1480490848 0 0) 0x55f64fc93200 con 0x55f663ebf800 > -4301> 2018-07-11 08:06:14.114570 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:14.467470) v4 -- 0x55f67d14e400 con 0 > -4300> 2018-07-11 08:06:14.114602 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:14.467470) v4 -- 0x55f66bf11000 con 0 > -4299> 2018-07-11 08:06:14.382600 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed7800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 214 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:06:14.439787) v4 > -4298> 2018-07-11 08:06:14.382624 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.1 10.8.128.106:0/12677 214 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:14.439787) v4 ==== 2004+0+0 (1801722733 0 0) 0x55f64e20fc00 con 0x55f663ed7800 > -4297> 2018-07-11 08:06:14.382636 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:14.439787) v4 -- 0x55f66bf11e00 con 0 > -4296> 2018-07-11 08:06:14.382713 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed6000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 214 0x55f673908000 osd_ping(ping e509 stamp 2018-07-11 08:06:14.439787) v4 > -4295> 2018-07-11 08:06:14.382736 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.1 10.8.128.106:0/12677 214 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:14.439787) v4 ==== 2004+0+0 (1801722733 0 0) 0x55f673908000 con 0x55f663ed6000 > -4294> 2018-07-11 08:06:14.382751 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:14.439787) v4 -- 0x55f66bf10a00 con 0 > -4293> 2018-07-11 08:06:14.411397 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0a000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 216 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:06:15.047280) v4 > -4292> 2018-07-11 08:06:14.411423 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.11 10.8.128.114:0/15919 216 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:15.047280) v4 ==== 2004+0+0 (2955764084 0 0) 0x55f67d1afe00 con 0x55f663f0a000 > -4291> 2018-07-11 08:06:14.411425 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0b800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 216 0x55f67db37600 osd_ping(ping e509 stamp 2018-07-11 08:06:15.047280) v4 > -4290> 2018-07-11 08:06:14.411438 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:15.047280) v4 -- 0x55f66bf11600 con 0 > -4289> 2018-07-11 08:06:14.411438 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.11 10.8.128.114:0/15919 216 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:15.047280) v4 ==== 2004+0+0 (2955764084 0 0) 0x55f67db37600 con 0x55f663f0b800 > -4288> 2018-07-11 08:06:14.411469 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:15.047280) v4 -- 0x55f67d14e600 con 0 > -4287> 2018-07-11 08:06:14.656439 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f661e058c0 con 0 > -4286> 2018-07-11 08:06:14.658046 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 271 0x55f683ee5b80 replica scrub(pg: 6.15,from:0'0,to:430'106224,epoch:509/504,start:6:a869f45d:::benchmark_data_magna108.ceph.redhat.com_2478_object401868:0,end:6:a86b1cd6:::benchmark_data_magna108_35617_object696192:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -4285> 2018-07-11 08:06:14.658075 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 271 ==== replica scrub(pg: 6.15,from:0'0,to:430'106224,epoch:509/504,start:6:a869f45d:::benchmark_data_magna108.ceph.redhat.com_2478_object401868:0,end:6:a86b1cd6:::benchmark_data_magna108_35617_object696192:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (2154651100 0 0) 0x55f683ee5b80 con 0x55f66315e800 > -4284> 2018-07-11 08:06:14.668153 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.114:6805/15885 conn(0x55f66315a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.7 seq 2569 0x55f64fc93200 MOSDScrubReserve(5.4 REQUEST e509) v1 > -4283> 2018-07-11 08:06:14.668171 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.7 10.8.128.114:6805/15885 2569 ==== MOSDScrubReserve(5.4 REQUEST e509) v1 ==== 43+0+0 (3003630469 0 0) 0x55f64fc93200 con 0x55f66315a000 > -4282> 2018-07-11 08:06:14.668244 7f491983f700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.114:6805/15885 -- MOSDScrubReserve(5.4 REJECT e509) v1 -- 0x55f64fc93c00 con 0 > -4281> 2018-07-11 08:06:14.668531 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.114:6805/15885 conn(0x55f66315a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.7 seq 2570 0x55f64fc93c00 MOSDScrubReserve(5.4 RELEASE e509) v1 > -4280> 2018-07-11 08:06:14.668546 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.7 10.8.128.114:6805/15885 2570 ==== MOSDScrubReserve(5.4 RELEASE e509) v1 ==== 43+0+0 (2082093447 0 0) 0x55f64fc93c00 con 0x55f66315a000 > -4279> 2018-07-11 08:06:14.769019 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 30] Generated table #37050: 2638241 keys, 74546380 bytes > -4278> 2018-07-11 08:06:14.769066 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296374769051, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 37050, "file_size": 74546380, "table_properties": {"data_size": 67112167, "index_size": 690040, "filter_size": 6743183, "raw_key_size": 76215859, "raw_average_key_size": 28, "raw_value_size": 31365762, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} > -4277> 2018-07-11 08:06:14.981775 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15255 conn(0x55f663ec2800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.0 seq 205 0x55f67db36600 osd_ping(ping e509 stamp 2018-07-11 08:06:16.226984) v4 > -4276> 2018-07-11 08:06:14.981771 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15255 conn(0x55f663ec1000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.0 seq 205 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:06:16.226984) v4 > -4275> 2018-07-11 08:06:14.981807 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.0 10.8.128.113:0/15255 205 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:16.226984) v4 ==== 2004+0+0 (3505493462 0 0) 0x55f64e20fc00 con 0x55f663ec1000 > -4274> 2018-07-11 08:06:14.981804 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.0 10.8.128.113:0/15255 205 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:16.226984) v4 ==== 2004+0+0 (3505493462 0 0) 0x55f67db36600 con 0x55f663ec2800 > -4273> 2018-07-11 08:06:14.981825 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:16.226984) v4 -- 0x55f66bf11600 con 0 > -4272> 2018-07-11 08:06:14.981860 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:16.226984) v4 -- 0x55f67d14e600 con 0 > -4271> 2018-07-11 08:06:15.483118 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed6000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 215 0x55f66bf10a00 osd_ping(ping e509 stamp 2018-07-11 08:06:15.540301) v4 > -4270> 2018-07-11 08:06:15.483152 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.1 10.8.128.106:0/12677 215 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:15.540301) v4 ==== 2004+0+0 (2091023981 0 0) 0x55f66bf10a00 con 0x55f663ed6000 > -4269> 2018-07-11 08:06:15.483163 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:15.540301) v4 -- 0x55f673908800 con 0 > -4268> 2018-07-11 08:06:15.483140 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed7800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 215 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:06:15.540301) v4 > -4267> 2018-07-11 08:06:15.483174 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.1 10.8.128.106:0/12677 215 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:15.540301) v4 ==== 2004+0+0 (2091023981 0 0) 0x55f67d1afe00 con 0x55f663ed7800 > -4266> 2018-07-11 08:06:15.483189 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:15.540301) v4 -- 0x55f66bf10e00 con 0 > -4265> 2018-07-11 08:06:15.531656 7f491d046700 1 -- 10.8.128.107:6800/14173 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55f6635038c0 con 0 > -4264> 2018-07-11 08:06:15.531721 7f491d046700 1 -- 10.8.128.107:6800/14173 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55f6631cf340 con 0 > -4263> 2018-07-11 08:06:15.541967 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f661e1c900 con 0 > -4262> 2018-07-11 08:06:15.543633 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 272 0x55f67be44000 replica scrub(pg: 6.15,from:0'0,to:430'105956,epoch:509/504,start:6:a86b1cd6:::benchmark_data_magna108_35617_object696192:0,end:6:a86c2286:::benchmark_data_magna108_35617_object966283:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -4261> 2018-07-11 08:06:15.543657 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 272 ==== replica scrub(pg: 6.15,from:0'0,to:430'105956,epoch:509/504,start:6:a86b1cd6:::benchmark_data_magna108_35617_object696192:0,end:6:a86c2286:::benchmark_data_magna108_35617_object966283:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (3567438420 0 0) 0x55f67be44000 con 0x55f66315e800 > -4260> 2018-07-11 08:06:15.941985 7f491803c700 5 osd.10 509 heartbeat: osd_stat(790 GB used, 141 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,0,1]) > -4259> 2018-07-11 08:06:15.942003 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6802/15255 -- osd_ping(ping e509 stamp 2018-07-11 08:06:15.942001) v4 -- 0x55f67d151600 con 0 > -4258> 2018-07-11 08:06:15.942034 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6803/15255 -- osd_ping(ping e509 stamp 2018-07-11 08:06:15.942001) v4 -- 0x55f67d151800 con 0 > -4257> 2018-07-11 08:06:15.942049 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6806/12677 -- osd_ping(ping e509 stamp 2018-07-11 08:06:15.942001) v4 -- 0x55f67d151a00 con 0 > -4256> 2018-07-11 08:06:15.942064 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6807/12677 -- osd_ping(ping e509 stamp 2018-07-11 08:06:15.942001) v4 -- 0x55f67d151c00 con 0 > -4255> 2018-07-11 08:06:15.942078 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6808/12651 -- osd_ping(ping e509 stamp 2018-07-11 08:06:15.942001) v4 -- 0x55f67d151e00 con 0 > -4254> 2018-07-11 08:06:15.942091 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6810/12651 -- osd_ping(ping e509 stamp 2018-07-11 08:06:15.942001) v4 -- 0x55f66caf0000 con 0 > -4253> 2018-07-11 08:06:15.942103 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6809/12650 -- osd_ping(ping e509 stamp 2018-07-11 08:06:15.942001) v4 -- 0x55f66caf0200 con 0 > -4252> 2018-07-11 08:06:15.942115 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6811/12650 -- osd_ping(ping e509 stamp 2018-07-11 08:06:15.942001) v4 -- 0x55f66caf0400 con 0 > -4251> 2018-07-11 08:06:15.942127 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6810/15287 -- osd_ping(ping e509 stamp 2018-07-11 08:06:15.942001) v4 -- 0x55f66caf0600 con 0 > -4250> 2018-07-11 08:06:15.942144 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6811/15287 -- osd_ping(ping e509 stamp 2018-07-11 08:06:15.942001) v4 -- 0x55f66caf0800 con 0 > -4249> 2018-07-11 08:06:15.942156 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6806/15263 -- osd_ping(ping e509 stamp 2018-07-11 08:06:15.942001) v4 -- 0x55f66caf0a00 con 0 > -4248> 2018-07-11 08:06:15.942168 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6807/15263 -- osd_ping(ping e509 stamp 2018-07-11 08:06:15.942001) v4 -- 0x55f66caf0c00 con 0 > -4247> 2018-07-11 08:06:15.942180 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.115:6802/13323 -- osd_ping(ping e509 stamp 2018-07-11 08:06:15.942001) v4 -- 0x55f66caf0e00 con 0 > -4246> 2018-07-11 08:06:15.942193 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.115:6803/13323 -- osd_ping(ping e509 stamp 2018-07-11 08:06:15.942001) v4 -- 0x55f66caf1000 con 0 > -4245> 2018-07-11 08:06:15.942206 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6806/15885 -- osd_ping(ping e509 stamp 2018-07-11 08:06:15.942001) v4 -- 0x55f66caf1200 con 0 > -4244> 2018-07-11 08:06:15.942218 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6807/15885 -- osd_ping(ping e509 stamp 2018-07-11 08:06:15.942001) v4 -- 0x55f66caf1400 con 0 > -4243> 2018-07-11 08:06:15.942232 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.108:6802/11967 -- osd_ping(ping e509 stamp 2018-07-11 08:06:15.942001) v4 -- 0x55f66caf1600 con 0 > -4242> 2018-07-11 08:06:15.942244 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.108:6803/11967 -- osd_ping(ping e509 stamp 2018-07-11 08:06:15.942001) v4 -- 0x55f66caf1800 con 0 > -4241> 2018-07-11 08:06:15.942274 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.107:6806/12617 -- osd_ping(ping e509 stamp 2018-07-11 08:06:15.942001) v4 -- 0x55f66caf1a00 con 0 > -4240> 2018-07-11 08:06:15.942287 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.107:6807/12617 -- osd_ping(ping e509 stamp 2018-07-11 08:06:15.942001) v4 -- 0x55f66caf1c00 con 0 > -4239> 2018-07-11 08:06:15.942300 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6802/15919 -- osd_ping(ping e509 stamp 2018-07-11 08:06:15.942001) v4 -- 0x55f66caf1e00 con 0 > -4238> 2018-07-11 08:06:15.942313 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6803/15919 -- osd_ping(ping e509 stamp 2018-07-11 08:06:15.942001) v4 -- 0x55f66cb34000 con 0 > -4237> 2018-07-11 08:06:15.942473 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6803/15255 conn(0x55f656ad5800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 210 0x55f67db36600 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:15.942001) v4 > -4236> 2018-07-11 08:06:15.942480 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6802/15255 conn(0x55f656ad4000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 210 0x55f673908800 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:15.942001) v4 > -4235> 2018-07-11 08:06:15.942494 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.0 10.8.128.113:6803/15255 210 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:15.942001) v4 ==== 2004+0+0 (3842653429 0 0) 0x55f67db36600 con 0x55f656ad5800 > -4234> 2018-07-11 08:06:15.942507 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.0 10.8.128.113:6802/15255 210 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:15.942001) v4 ==== 2004+0+0 (3842653429 0 0) 0x55f673908800 con 0x55f656ad4000 > -4233> 2018-07-11 08:06:15.942547 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.107:6806/12617 conn(0x55f656ade000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.9 seq 210 0x55f67db36600 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:15.942001) v4 > -4232> 2018-07-11 08:06:15.942562 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.9 10.8.128.107:6806/12617 210 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:15.942001) v4 ==== 2004+0+0 (3842653429 0 0) 0x55f67db36600 con 0x55f656ade000 > -4231> 2018-07-11 08:06:15.942603 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6810/12651 conn(0x55f64e3eb800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.2 seq 210 0x55f67db36600 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:15.942001) v4 > -4230> 2018-07-11 08:06:15.942616 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.2 10.8.128.106:6810/12651 210 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:15.942001) v4 ==== 2004+0+0 (3842653429 0 0) 0x55f67db36600 con 0x55f64e3eb800 > -4229> 2018-07-11 08:06:15.942619 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.107:6807/12617 conn(0x55f656adf800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.9 seq 210 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:15.942001) v4 > -4228> 2018-07-11 08:06:15.942612 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6811/12650 conn(0x55f656ad8800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.3 seq 210 0x55f673908800 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:15.942001) v4 > -4227> 2018-07-11 08:06:15.942658 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.9 10.8.128.107:6807/12617 210 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:15.942001) v4 ==== 2004+0+0 (3842653429 0 0) 0x55f64e20fc00 con 0x55f656adf800 > -4226> 2018-07-11 08:06:15.942660 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.3 10.8.128.106:6811/12650 210 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:15.942001) v4 ==== 2004+0+0 (3842653429 0 0) 0x55f673908800 con 0x55f656ad8800 > -4225> 2018-07-11 08:06:15.942664 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6811/15287 conn(0x55f64e402800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.4 seq 210 0x55f67db36600 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:15.942001) v4 > -4224> 2018-07-11 08:06:15.942675 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.4 10.8.128.113:6811/15287 210 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:15.942001) v4 ==== 2004+0+0 (3842653429 0 0) 0x55f67db36600 con 0x55f64e402800 > -4223> 2018-07-11 08:06:15.942710 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6809/12650 conn(0x55f656ad7000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.3 seq 210 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:15.942001) v4 > -4222> 2018-07-11 08:06:15.942728 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.3 10.8.128.106:6809/12650 210 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:15.942001) v4 ==== 2004+0+0 (3842653429 0 0) 0x55f64e20fc00 con 0x55f656ad7000 > -4221> 2018-07-11 08:06:15.942711 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6807/12677 conn(0x55f64e3d9800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.1 seq 210 0x55f673908800 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:15.942001) v4 > -4220> 2018-07-11 08:06:15.942748 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.1 10.8.128.106:6807/12677 210 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:15.942001) v4 ==== 2004+0+0 (3842653429 0 0) 0x55f673908800 con 0x55f64e3d9800 > -4219> 2018-07-11 08:06:15.942754 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6806/15885 conn(0x55f64e3fe000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.7 seq 210 0x55f67db36600 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:15.942001) v4 > -4218> 2018-07-11 08:06:15.942766 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6806/12677 conn(0x55f64e3d8000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.1 seq 210 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:15.942001) v4 > -4217> 2018-07-11 08:06:15.942779 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.1 10.8.128.106:6806/12677 210 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:15.942001) v4 ==== 2004+0+0 (3842653429 0 0) 0x55f64e20fc00 con 0x55f64e3d8000 > -4216> 2018-07-11 08:06:15.942783 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6810/15287 conn(0x55f64e401000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.4 seq 210 0x55f673908800 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:15.942001) v4 > -4215> 2018-07-11 08:06:15.942769 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.7 10.8.128.114:6806/15885 210 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:15.942001) v4 ==== 2004+0+0 (3842653429 0 0) 0x55f67db36600 con 0x55f64e3fe000 > -4214> 2018-07-11 08:06:15.942793 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.4 10.8.128.113:6810/15287 210 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:15.942001) v4 ==== 2004+0+0 (3842653429 0 0) 0x55f673908800 con 0x55f64e401000 > -4213> 2018-07-11 08:06:15.942815 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.115:6803/13323 conn(0x55f656ace800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.6 seq 210 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:15.942001) v4 > -4212> 2018-07-11 08:06:15.942824 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6808/12651 conn(0x55f64e3ea000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.2 seq 210 0x55f673908800 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:15.942001) v4 > -4211> 2018-07-11 08:06:15.942828 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.6 10.8.128.115:6803/13323 210 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:15.942001) v4 ==== 2004+0+0 (3842653429 0 0) 0x55f64e20fc00 con 0x55f656ace800 > -4210> 2018-07-11 08:06:15.942836 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.2 10.8.128.106:6808/12651 210 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:15.942001) v4 ==== 2004+0+0 (3842653429 0 0) 0x55f673908800 con 0x55f64e3ea000 > -4209> 2018-07-11 08:06:15.942833 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.115:6802/13323 conn(0x55f656acd000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.6 seq 210 0x55f67db36600 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:15.942001) v4 > -4208> 2018-07-11 08:06:15.942846 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.6 10.8.128.115:6802/13323 210 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:15.942001) v4 ==== 2004+0+0 (3842653429 0 0) 0x55f67db36600 con 0x55f656acd000 > -4207> 2018-07-11 08:06:15.942861 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6806/15263 conn(0x55f64e3ed000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.5 seq 210 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:15.942001) v4 > -4206> 2018-07-11 08:06:15.942869 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6807/15263 conn(0x55f64e3ee800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.5 seq 210 0x55f673908800 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:15.942001) v4 > -4205> 2018-07-11 08:06:15.942878 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.5 10.8.128.113:6806/15263 210 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:15.942001) v4 ==== 2004+0+0 (3842653429 0 0) 0x55f64e20fc00 con 0x55f64e3ed000 > -4204> 2018-07-11 08:06:15.942879 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.5 10.8.128.113:6807/15263 210 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:15.942001) v4 ==== 2004+0+0 (3842653429 0 0) 0x55f673908800 con 0x55f64e3ee800 > -4203> 2018-07-11 08:06:15.942880 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6802/15919 conn(0x55f64e3db000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.11 seq 210 0x55f67db36600 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:15.942001) v4 > -4202> 2018-07-11 08:06:15.942893 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.11 10.8.128.114:6802/15919 210 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:15.942001) v4 ==== 2004+0+0 (3842653429 0 0) 0x55f67db36600 con 0x55f64e3db000 > -4201> 2018-07-11 08:06:15.942910 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6803/15919 conn(0x55f64e3dc800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.11 seq 210 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:15.942001) v4 > -4200> 2018-07-11 08:06:15.942925 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.11 10.8.128.114:6803/15919 210 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:15.942001) v4 ==== 2004+0+0 (3842653429 0 0) 0x55f64e20fc00 con 0x55f64e3dc800 > -4199> 2018-07-11 08:06:15.942955 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6807/15885 conn(0x55f64e3ff800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.7 seq 210 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:15.942001) v4 > -4198> 2018-07-11 08:06:15.942971 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.7 10.8.128.114:6807/15885 210 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:15.942001) v4 ==== 2004+0+0 (3842653429 0 0) 0x55f64e20fc00 con 0x55f64e3ff800 > -4197> 2018-07-11 08:06:15.943004 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.108:6802/11967 conn(0x55f656aca000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.8 seq 210 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:15.942001) v4 > -4196> 2018-07-11 08:06:15.943019 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.8 10.8.128.108:6802/11967 210 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:15.942001) v4 ==== 2004+0+0 (3842653429 0 0) 0x55f64e20fc00 con 0x55f656aca000 > -4195> 2018-07-11 08:06:15.943012 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.108:6803/11967 conn(0x55f656acb800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.8 seq 210 0x55f673908800 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:15.942001) v4 > -4194> 2018-07-11 08:06:15.943042 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.8 10.8.128.108:6803/11967 210 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:15.942001) v4 ==== 2004+0+0 (3842653429 0 0) 0x55f673908800 con 0x55f656acb800 > -4193> 2018-07-11 08:06:16.095502 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.107:0/12617 conn(0x55f64e3c3800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.9 seq 216 0x55f67db36600 osd_ping(ping e509 stamp 2018-07-11 08:06:16.095128) v4 > -4192> 2018-07-11 08:06:16.095528 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.9 10.8.128.107:0/12617 216 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:16.095128) v4 ==== 2004+0+0 (1979995322 0 0) 0x55f67db36600 con 0x55f64e3c3800 > -4191> 2018-07-11 08:06:16.095539 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:16.095128) v4 -- 0x55f66caf0000 con 0 > -4190> 2018-07-11 08:06:16.095529 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.107:0/12617 conn(0x55f64e3c5000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.9 seq 216 0x55f673908800 osd_ping(ping e509 stamp 2018-07-11 08:06:16.095128) v4 > -4189> 2018-07-11 08:06:16.095554 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.9 10.8.128.107:0/12617 216 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:16.095128) v4 ==== 2004+0+0 (1979995322 0 0) 0x55f673908800 con 0x55f64e3c5000 > -4188> 2018-07-11 08:06:16.095588 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:16.095128) v4 -- 0x55f67d151600 con 0 > -4187> 2018-07-11 08:06:16.111740 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0a000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 217 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:06:16.747679) v4 > -4186> 2018-07-11 08:06:16.111759 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.11 10.8.128.114:0/15919 217 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:16.747679) v4 ==== 2004+0+0 (518962870 0 0) 0x55f64e20fc00 con 0x55f663f0a000 > -4185> 2018-07-11 08:06:16.111758 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0b800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 217 0x55f67db37600 osd_ping(ping e509 stamp 2018-07-11 08:06:16.747679) v4 > -4184> 2018-07-11 08:06:16.111775 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:16.747679) v4 -- 0x55f66bf10e00 con 0 > -4183> 2018-07-11 08:06:16.111786 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.11 10.8.128.114:0/15919 217 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:16.747679) v4 ==== 2004+0+0 (518962870 0 0) 0x55f67db37600 con 0x55f663f0b800 > -4182> 2018-07-11 08:06:16.111802 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:16.747679) v4 -- 0x55f67d151800 con 0 > -4181> 2018-07-11 08:06:16.307802 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15287 conn(0x55f66dce2800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.4 seq 217 0x55f673908600 osd_ping(ping e509 stamp 2018-07-11 08:06:17.553097) v4 > -4180> 2018-07-11 08:06:16.307824 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.4 10.8.128.113:0/15287 217 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:17.553097) v4 ==== 2004+0+0 (842258835 0 0) 0x55f673908600 con 0x55f66dce2800 > -4179> 2018-07-11 08:06:16.307840 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:17.553097) v4 -- 0x55f673908a00 con 0 > -4178> 2018-07-11 08:06:16.307943 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15287 conn(0x55f66dce1000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.4 seq 217 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:06:17.553097) v4 > -4177> 2018-07-11 08:06:16.307962 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.4 10.8.128.113:0/15287 217 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:17.553097) v4 ==== 2004+0+0 (842258835 0 0) 0x55f67d1afe00 con 0x55f66dce1000 > -4176> 2018-07-11 08:06:16.307976 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:17.553097) v4 -- 0x55f66bf10e00 con 0 > -4175> 2018-07-11 08:06:16.474637 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f65649cb40 con 0 > -4174> 2018-07-11 08:06:16.476272 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 273 0x55f67be44280 replica scrub(pg: 6.15,from:0'0,to:430'106220,epoch:509/504,start:6:a86c2286:::benchmark_data_magna108_35617_object966283:0,end:6:a86ce870:::benchmark_data_magna108_35617_object513200:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -4173> 2018-07-11 08:06:16.476296 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 273 ==== replica scrub(pg: 6.15,from:0'0,to:430'106220,epoch:509/504,start:6:a86c2286:::benchmark_data_magna108_35617_object966283:0,end:6:a86ce870:::benchmark_data_magna108_35617_object513200:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (3769506843 0 0) 0x55f67be44280 con 0x55f66315e800 > -4172> 2018-07-11 08:06:16.904497 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.108:0/11967 conn(0x55f663f6f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=1). rx osd.8 seq 205 0x55f67db36600 osd_ping(ping e509 stamp 2018-07-11 08:06:16.664645) v4 > -4171> 2018-07-11 08:06:16.904514 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.108:0/11967 conn(0x55f663f70800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=1). rx osd.8 seq 205 0x55f673908800 osd_ping(ping e509 stamp 2018-07-11 08:06:16.664645) v4 > -4170> 2018-07-11 08:06:16.904539 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.8 10.8.128.108:0/11967 205 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:16.664645) v4 ==== 2004+0+0 (3619992591 0 0) 0x55f67db36600 con 0x55f663f6f000 > -4169> 2018-07-11 08:06:16.904542 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.8 10.8.128.108:0/11967 205 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:16.664645) v4 ==== 2004+0+0 (3619992591 0 0) 0x55f673908800 con 0x55f663f70800 > -4168> 2018-07-11 08:06:16.904553 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:16.664645) v4 -- 0x55f67d151800 con 0 > -4167> 2018-07-11 08:06:16.904581 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:16.664645) v4 -- 0x55f673908a00 con 0 > -4166> 2018-07-11 08:06:16.973574 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 469 0x55f67db37600 MOSDScrubReserve(11.3s5 REQUEST e509) v1 > -4165> 2018-07-11 08:06:16.973592 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 469 ==== MOSDScrubReserve(11.3s5 REQUEST e509) v1 ==== 43+0+0 (433425070 0 0) 0x55f67db37600 con 0x55f668a29000 > -4164> 2018-07-11 08:06:16.973675 7f491a040700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e509) v1 -- 0x55f67d14e200 con 0 > -4163> 2018-07-11 08:06:16.973959 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 470 0x55f67d151800 MOSDScrubReserve(11.3s5 RELEASE e509) v1 > -4162> 2018-07-11 08:06:16.973973 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 470 ==== MOSDScrubReserve(11.3s5 RELEASE e509) v1 ==== 43+0+0 (3603460268 0 0) 0x55f67d151800 con 0x55f668a29000 > -4161> 2018-07-11 08:06:17.196003 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.107:0/12617 conn(0x55f64e3c5000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.9 seq 217 0x55f673908600 osd_ping(ping e509 stamp 2018-07-11 08:06:17.195659) v4 > -4160> 2018-07-11 08:06:17.196013 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.107:0/12617 conn(0x55f64e3c3800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.9 seq 217 0x55f67db36600 osd_ping(ping e509 stamp 2018-07-11 08:06:17.195659) v4 > -4159> 2018-07-11 08:06:17.196026 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.9 10.8.128.107:0/12617 217 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:17.195659) v4 ==== 2004+0+0 (2295970447 0 0) 0x55f673908600 con 0x55f64e3c5000 > -4158> 2018-07-11 08:06:17.196034 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.9 10.8.128.107:0/12617 217 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:17.195659) v4 ==== 2004+0+0 (2295970447 0 0) 0x55f67db36600 con 0x55f64e3c3800 > -4157> 2018-07-11 08:06:17.196045 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:17.195659) v4 -- 0x55f673908c00 con 0 > -4156> 2018-07-11 08:06:17.196094 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:17.195659) v4 -- 0x55f67d14ea00 con 0 > -4155> 2018-07-11 08:06:17.256430 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f654b76d80 con 0 > -4154> 2018-07-11 08:06:17.258266 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 274 0x55f67be44500 replica scrub(pg: 6.15,from:0'0,to:430'106467,epoch:509/504,start:6:a86ce870:::benchmark_data_magna108_35617_object513200:0,end:6:a86d8d4c:::benchmark_data_magna108.ceph.redhat.com_2478_object315419:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -4153> 2018-07-11 08:06:17.258298 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 274 ==== replica scrub(pg: 6.15,from:0'0,to:430'106467,epoch:509/504,start:6:a86ce870:::benchmark_data_magna108_35617_object513200:0,end:6:a86d8d4c:::benchmark_data_magna108.ceph.redhat.com_2478_object315419:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (1310365645 0 0) 0x55f67be44500 con 0x55f66315e800 > -4152> 2018-07-11 08:06:17.461454 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0e800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 220 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:06:18.706859) v4 > -4151> 2018-07-11 08:06:17.461480 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.5 10.8.128.113:0/15263 220 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:18.706859) v4 ==== 2004+0+0 (3288774524 0 0) 0x55f64e20fc00 con 0x55f663f0e800 > -4150> 2018-07-11 08:06:17.461498 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:18.706859) v4 -- 0x55f66bf10e00 con 0 > -4149> 2018-07-11 08:06:17.461501 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0d000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 220 0x55f673908800 osd_ping(ping e509 stamp 2018-07-11 08:06:18.706859) v4 > -4148> 2018-07-11 08:06:17.461526 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.5 10.8.128.113:0/15263 220 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:18.706859) v4 ==== 2004+0+0 (3288774524 0 0) 0x55f673908800 con 0x55f663f0d000 > -4147> 2018-07-11 08:06:17.461542 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:18.706859) v4 -- 0x55f673908e00 con 0 > -4146> 2018-07-11 08:06:17.640053 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15885 conn(0x55f663217800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 217 0x55f67d14e200 osd_ping(ping e509 stamp 2018-07-11 08:06:18.275896) v4 > -4145> 2018-07-11 08:06:17.640063 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15885 conn(0x55f663216000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 217 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:06:18.275896) v4 > -4144> 2018-07-11 08:06:17.640090 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.7 10.8.128.114:0/15885 217 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:18.275896) v4 ==== 2004+0+0 (720461307 0 0) 0x55f67d1afe00 con 0x55f663216000 > -4143> 2018-07-11 08:06:17.640087 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.7 10.8.128.114:0/15885 217 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:18.275896) v4 ==== 2004+0+0 (720461307 0 0) 0x55f67d14e200 con 0x55f663217800 > -4142> 2018-07-11 08:06:17.640109 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:18.275896) v4 -- 0x55f66bf10e00 con 0 > -4141> 2018-07-11 08:06:17.640134 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:18.275896) v4 -- 0x55f67d14ea00 con 0 > -4140> 2018-07-11 08:06:17.882169 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15255 conn(0x55f663ec1000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.0 seq 206 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:06:19.127518) v4 > -4139> 2018-07-11 08:06:17.882202 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.0 10.8.128.113:0/15255 206 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:19.127518) v4 ==== 2004+0+0 (1698249017 0 0) 0x55f64e20fc00 con 0x55f663ec1000 > -4138> 2018-07-11 08:06:17.882201 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15255 conn(0x55f663ec2800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.0 seq 206 0x55f67db36600 osd_ping(ping e509 stamp 2018-07-11 08:06:19.127518) v4 > -4137> 2018-07-11 08:06:17.882219 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.0 10.8.128.113:0/15255 206 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:19.127518) v4 ==== 2004+0+0 (1698249017 0 0) 0x55f67db36600 con 0x55f663ec2800 > -4136> 2018-07-11 08:06:17.882217 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:19.127518) v4 -- 0x55f66bf10e00 con 0 > -4135> 2018-07-11 08:06:17.882256 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:19.127518) v4 -- 0x55f67d14ea00 con 0 > -4134> 2018-07-11 08:06:17.970809 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f65f94afc0 con 0 > -4133> 2018-07-11 08:06:17.972491 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 275 0x55f67be44780 replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:509/504,start:6:a86d8d4c:::benchmark_data_magna108.ceph.redhat.com_2478_object315419:0,end:6:a86ef6cf:::benchmark_data_magna108_35617_object340683:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -4132> 2018-07-11 08:06:17.972520 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 275 ==== replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:509/504,start:6:a86d8d4c:::benchmark_data_magna108.ceph.redhat.com_2478_object315419:0,end:6:a86ef6cf:::benchmark_data_magna108_35617_object340683:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (4250238406 0 0) 0x55f67be44780 con 0x55f66315e800 > -4131> 2018-07-11 08:06:18.214874 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.115:0/13323 conn(0x55f663ebf800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.6 seq 200 0x55f673908c00 osd_ping(ping e509 stamp 2018-07-11 08:06:18.567940) v4 > -4130> 2018-07-11 08:06:18.214900 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.6 10.8.128.115:0/13323 200 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:18.567940) v4 ==== 2004+0+0 (2531411272 0 0) 0x55f673908c00 con 0x55f663ebf800 > -4129> 2018-07-11 08:06:18.214917 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:18.567940) v4 -- 0x55f673909000 con 0 > -4128> 2018-07-11 08:06:18.214919 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.115:0/13323 conn(0x55f663ebe000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.6 seq 200 0x55f67d14e200 osd_ping(ping e509 stamp 2018-07-11 08:06:18.567940) v4 > -4127> 2018-07-11 08:06:18.214943 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.6 10.8.128.115:0/13323 200 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:18.567940) v4 ==== 2004+0+0 (2531411272 0 0) 0x55f67d14e200 con 0x55f663ebe000 > -4126> 2018-07-11 08:06:18.214958 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:18.567940) v4 -- 0x55f67d14ea00 con 0 > -4125> 2018-07-11 08:06:18.263031 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12651 conn(0x55f66321a800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25 cs=1 l=1). rx osd.2 seq 226 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:06:18.320062) v4 > -4124> 2018-07-11 08:06:18.263056 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.2 10.8.128.106:0/12651 226 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:18.320062) v4 ==== 2004+0+0 (1187552 0 0) 0x55f67d1afe00 con 0x55f66321a800 > -4123> 2018-07-11 08:06:18.263059 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12651 conn(0x55f663219000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25 cs=1 l=1). rx osd.2 seq 226 0x55f673908e00 osd_ping(ping e509 stamp 2018-07-11 08:06:18.320062) v4 > -4122> 2018-07-11 08:06:18.263074 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:18.320062) v4 -- 0x55f66bf10e00 con 0 > -4121> 2018-07-11 08:06:18.263084 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.2 10.8.128.106:0/12651 226 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:18.320062) v4 ==== 2004+0+0 (1187552 0 0) 0x55f673908e00 con 0x55f663219000 > -4120> 2018-07-11 08:06:18.263101 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:18.320062) v4 -- 0x55f673909200 con 0 > -4119> 2018-07-11 08:06:18.608266 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15287 conn(0x55f66dce1000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.4 seq 218 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:06:19.853678) v4 > -4118> 2018-07-11 08:06:18.608300 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.4 10.8.128.113:0/15287 218 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:19.853678) v4 ==== 2004+0+0 (782071813 0 0) 0x55f64e20fc00 con 0x55f66dce1000 > -4117> 2018-07-11 08:06:18.608291 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15287 conn(0x55f66dce2800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.4 seq 218 0x55f673908c00 osd_ping(ping e509 stamp 2018-07-11 08:06:19.853678) v4 > -4116> 2018-07-11 08:06:18.608311 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:19.853678) v4 -- 0x55f65e71cc00 con 0 > -4115> 2018-07-11 08:06:18.608315 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.4 10.8.128.113:0/15287 218 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:19.853678) v4 ==== 2004+0+0 (782071813 0 0) 0x55f673908c00 con 0x55f66dce2800 > -4114> 2018-07-11 08:06:18.608344 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:19.853678) v4 -- 0x55f673909400 con 0 > -4113> 2018-07-11 08:06:18.740415 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15885 conn(0x55f663217800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 218 0x55f67db36600 osd_ping(ping e509 stamp 2018-07-11 08:06:19.376323) v4 > -4112> 2018-07-11 08:06:18.740415 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15885 conn(0x55f663216000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 218 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:06:19.376323) v4 > -4111> 2018-07-11 08:06:18.740446 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.7 10.8.128.114:0/15885 218 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:19.376323) v4 ==== 2004+0+0 (105050589 0 0) 0x55f67d1afe00 con 0x55f663216000 > -4110> 2018-07-11 08:06:18.740443 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.7 10.8.128.114:0/15885 218 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:19.376323) v4 ==== 2004+0+0 (105050589 0 0) 0x55f67db36600 con 0x55f663217800 > -4109> 2018-07-11 08:06:18.740463 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:19.376323) v4 -- 0x55f64e20ec00 con 0 > -4108> 2018-07-11 08:06:18.740493 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:19.376323) v4 -- 0x55f67d14ec00 con 0 > -4107> 2018-07-11 08:06:18.973992 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 471 0x55f67d14e200 MOSDScrubReserve(11.3s5 REQUEST e509) v1 > -4106> 2018-07-11 08:06:18.974011 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 471 ==== MOSDScrubReserve(11.3s5 REQUEST e509) v1 ==== 43+0+0 (433425070 0 0) 0x55f67d14e200 con 0x55f668a29000 > -4105> 2018-07-11 08:06:18.974085 7f491a040700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e509) v1 -- 0x55f67d151800 con 0 > -4104> 2018-07-11 08:06:18.974390 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 472 0x55f67d14ee00 MOSDScrubReserve(11.3s5 RELEASE e509) v1 > -4103> 2018-07-11 08:06:18.974421 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 472 ==== MOSDScrubReserve(11.3s5 RELEASE e509) v1 ==== 43+0+0 (3603460268 0 0) 0x55f67d14ee00 con 0x55f668a29000 > -4102> 2018-07-11 08:06:19.363535 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12651 conn(0x55f663219000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25 cs=1 l=1). rx osd.2 seq 227 0x55f673908e00 osd_ping(ping e509 stamp 2018-07-11 08:06:19.420765) v4 > -4101> 2018-07-11 08:06:19.363561 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.2 10.8.128.106:0/12651 227 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:19.420765) v4 ==== 2004+0+0 (2059438751 0 0) 0x55f673908e00 con 0x55f663219000 > -4100> 2018-07-11 08:06:19.363571 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:19.420765) v4 -- 0x55f673909400 con 0 > -4099> 2018-07-11 08:06:19.363613 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12651 conn(0x55f66321a800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25 cs=1 l=1). rx osd.2 seq 227 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:06:19.420765) v4 > -4098> 2018-07-11 08:06:19.363633 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.2 10.8.128.106:0/12651 227 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:19.420765) v4 ==== 2004+0+0 (2059438751 0 0) 0x55f64e20fc00 con 0x55f66321a800 > -4097> 2018-07-11 08:06:19.363645 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:19.420765) v4 -- 0x55f64e20ec00 con 0 > -4096> 2018-07-11 08:06:19.396372 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 30] Generated table #37051: 2638241 keys, 74545767 bytes > -4095> 2018-07-11 08:06:19.396412 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296379396392, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 37051, "file_size": 74545767, "table_properties": {"data_size": 67112127, "index_size": 689467, "filter_size": 6743183, "raw_key_size": 76215846, "raw_average_key_size": 28, "raw_value_size": 31365749, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} > -4094> 2018-07-11 08:06:19.536129 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12650 conn(0x55f663eda800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.3 seq 212 0x55f673908c00 osd_ping(ping e509 stamp 2018-07-11 08:06:19.593447) v4 > -4093> 2018-07-11 08:06:19.536158 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.3 10.8.128.106:0/12650 212 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:19.593447) v4 ==== 2004+0+0 (741046734 0 0) 0x55f673908c00 con 0x55f663eda800 > -4092> 2018-07-11 08:06:19.536160 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12650 conn(0x55f663ed9000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.3 seq 212 0x55f67db36600 osd_ping(ping e509 stamp 2018-07-11 08:06:19.593447) v4 > -4091> 2018-07-11 08:06:19.536180 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:19.593447) v4 -- 0x55f673909400 con 0 > -4090> 2018-07-11 08:06:19.536186 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.3 10.8.128.106:0/12650 212 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:19.593447) v4 ==== 2004+0+0 (741046734 0 0) 0x55f67db36600 con 0x55f663ed9000 > -4089> 2018-07-11 08:06:19.536213 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:19.593447) v4 -- 0x55f67d14f200 con 0 > -4088> 2018-07-11 08:06:19.612152 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0b800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 218 0x55f67d151800 osd_ping(ping e509 stamp 2018-07-11 08:06:20.248067) v4 > -4087> 2018-07-11 08:06:19.612172 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0a000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 218 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:06:20.248067) v4 > -4086> 2018-07-11 08:06:19.612185 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.11 10.8.128.114:0/15919 218 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:20.248067) v4 ==== 2004+0+0 (1856003311 0 0) 0x55f67d151800 con 0x55f663f0b800 > -4085> 2018-07-11 08:06:19.612187 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.11 10.8.128.114:0/15919 218 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:20.248067) v4 ==== 2004+0+0 (1856003311 0 0) 0x55f67d1afe00 con 0x55f663f0a000 > -4084> 2018-07-11 08:06:19.612199 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:20.248067) v4 -- 0x55f67d14f400 con 0 > -4083> 2018-07-11 08:06:19.612227 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:20.248067) v4 -- 0x55f666c47800 con 0 > -4082> 2018-07-11 08:06:19.654284 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f666d9afc0 con 0 > -4081> 2018-07-11 08:06:19.655908 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 276 0x55f67be44a00 replica scrub(pg: 6.15,from:0'0,to:430'106715,epoch:509/504,start:6:a86ef6cf:::benchmark_data_magna108_35617_object340683:0,end:6:a8700a5a:::benchmark_data_magna108.ceph.redhat.com_2478_object369366:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -4080> 2018-07-11 08:06:19.655941 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 276 ==== replica scrub(pg: 6.15,from:0'0,to:430'106715,epoch:509/504,start:6:a86ef6cf:::benchmark_data_magna108_35617_object340683:0,end:6:a8700a5a:::benchmark_data_magna108.ceph.redhat.com_2478_object369366:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (2632908770 0 0) 0x55f67be44a00 con 0x55f66315e800 > -4079> 2018-07-11 08:06:19.761796 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0e800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 221 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:06:21.007270) v4 > -4078> 2018-07-11 08:06:19.761822 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.5 10.8.128.113:0/15263 221 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:21.007270) v4 ==== 2004+0+0 (970557265 0 0) 0x55f64e20fc00 con 0x55f663f0e800 > -4077> 2018-07-11 08:06:19.761818 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0d000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 221 0x55f673908e00 osd_ping(ping e509 stamp 2018-07-11 08:06:21.007270) v4 > -4076> 2018-07-11 08:06:19.761839 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.5 10.8.128.113:0/15263 221 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:21.007270) v4 ==== 2004+0+0 (970557265 0 0) 0x55f673908e00 con 0x55f663f0d000 > -4075> 2018-07-11 08:06:19.761840 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:21.007270) v4 -- 0x55f666c47800 con 0 > -4074> 2018-07-11 08:06:19.761885 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:21.007270) v4 -- 0x55f673909600 con 0 > -4073> 2018-07-11 08:06:20.323129 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f667d986c0 con 0 > -4072> 2018-07-11 08:06:20.324901 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 277 0x55f67be44c80 replica scrub(pg: 6.15,from:0'0,to:430'105701,epoch:509/504,start:6:a8700a5a:::benchmark_data_magna108.ceph.redhat.com_2478_object369366:0,end:6:a87103d2:::benchmark_data_magna108.ceph.redhat.com_2478_object252855:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -4071> 2018-07-11 08:06:20.324934 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 277 ==== replica scrub(pg: 6.15,from:0'0,to:430'105701,epoch:509/504,start:6:a8700a5a:::benchmark_data_magna108.ceph.redhat.com_2478_object369366:0,end:6:a87103d2:::benchmark_data_magna108.ceph.redhat.com_2478_object252855:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 248+0+0 (186822441 0 0) 0x55f67be44c80 con 0x55f66315e800 > -4070> 2018-07-11 08:06:20.434151 7f491e048700 10 monclient: tick > -4069> 2018-07-11 08:06:20.434177 7f491e048700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2018-07-11 08:05:50.434176) > -4068> 2018-07-11 08:06:20.464293 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12651 conn(0x55f66321a800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25 cs=1 l=1). rx osd.2 seq 228 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:06:20.521325) v4 > -4067> 2018-07-11 08:06:20.464313 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.2 10.8.128.106:0/12651 228 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:20.521325) v4 ==== 2004+0+0 (1373875061 0 0) 0x55f67d1afe00 con 0x55f66321a800 > -4066> 2018-07-11 08:06:20.464323 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:20.521325) v4 -- 0x55f666c47800 con 0 > -4065> 2018-07-11 08:06:20.464333 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12651 conn(0x55f663219000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25 cs=1 l=1). rx osd.2 seq 228 0x55f673908e00 osd_ping(ping e509 stamp 2018-07-11 08:06:20.521325) v4 > -4064> 2018-07-11 08:06:20.464366 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.2 10.8.128.106:0/12651 228 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:20.521325) v4 ==== 2004+0+0 (1373875061 0 0) 0x55f673908e00 con 0x55f663219000 > -4063> 2018-07-11 08:06:20.464384 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:20.521325) v4 -- 0x55f673909600 con 0 > -4062> 2018-07-11 08:06:20.532004 7f491d046700 1 -- 10.8.128.107:6800/14173 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55f663157b80 con 0 > -4061> 2018-07-11 08:06:20.532070 7f491d046700 1 -- 10.8.128.107:6800/14173 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55f663157340 con 0 > -4060> 2018-07-11 08:06:20.782695 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15255 conn(0x55f663ec1000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.0 seq 207 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:06:22.027992) v4 > -4059> 2018-07-11 08:06:20.782697 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15255 conn(0x55f663ec2800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.0 seq 207 0x55f67db36600 osd_ping(ping e509 stamp 2018-07-11 08:06:22.027992) v4 > -4058> 2018-07-11 08:06:20.782724 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.0 10.8.128.113:0/15255 207 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:22.027992) v4 ==== 2004+0+0 (3937852891 0 0) 0x55f67db36600 con 0x55f663ec2800 > -4057> 2018-07-11 08:06:20.782722 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.0 10.8.128.113:0/15255 207 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:22.027992) v4 ==== 2004+0+0 (3937852891 0 0) 0x55f64e20fc00 con 0x55f663ec1000 > -4056> 2018-07-11 08:06:20.782743 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:22.027992) v4 -- 0x55f67d14f400 con 0 > -4055> 2018-07-11 08:06:20.782766 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:22.027992) v4 -- 0x55f666c47800 con 0 > -4054> 2018-07-11 08:06:21.154376 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f65dc07d40 con 0 > -4053> 2018-07-11 08:06:21.156003 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 278 0x55f67be45180 replica scrub(pg: 6.15,from:0'0,to:430'106415,epoch:509/504,start:6:a87103d2:::benchmark_data_magna108.ceph.redhat.com_2478_object252855:0,end:6:a8721343:::benchmark_data_magna108_35617_object441441:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -4052> 2018-07-11 08:06:21.156038 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 278 ==== replica scrub(pg: 6.15,from:0'0,to:430'106415,epoch:509/504,start:6:a87103d2:::benchmark_data_magna108.ceph.redhat.com_2478_object252855:0,end:6:a8721343:::benchmark_data_magna108_35617_object441441:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (2989465210 0 0) 0x55f67be45180 con 0x55f66315e800 > -4051> 2018-07-11 08:06:21.242455 7f491803c700 5 osd.10 509 heartbeat: osd_stat(790 GB used, 141 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,1]) > -4050> 2018-07-11 08:06:21.242471 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6802/15255 -- osd_ping(ping e509 stamp 2018-07-11 08:06:21.242469) v4 -- 0x55f66cb34200 con 0 > -4049> 2018-07-11 08:06:21.242500 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6803/15255 -- osd_ping(ping e509 stamp 2018-07-11 08:06:21.242469) v4 -- 0x55f66cb34400 con 0 > -4048> 2018-07-11 08:06:21.242516 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6806/12677 -- osd_ping(ping e509 stamp 2018-07-11 08:06:21.242469) v4 -- 0x55f66cb34600 con 0 > -4047> 2018-07-11 08:06:21.242531 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6807/12677 -- osd_ping(ping e509 stamp 2018-07-11 08:06:21.242469) v4 -- 0x55f66cb34800 con 0 > -4046> 2018-07-11 08:06:21.242543 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6808/12651 -- osd_ping(ping e509 stamp 2018-07-11 08:06:21.242469) v4 -- 0x55f66cb34a00 con 0 > -4045> 2018-07-11 08:06:21.242554 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6810/12651 -- osd_ping(ping e509 stamp 2018-07-11 08:06:21.242469) v4 -- 0x55f66cb34c00 con 0 > -4044> 2018-07-11 08:06:21.242566 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6809/12650 -- osd_ping(ping e509 stamp 2018-07-11 08:06:21.242469) v4 -- 0x55f66cb34e00 con 0 > -4043> 2018-07-11 08:06:21.242578 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6811/12650 -- osd_ping(ping e509 stamp 2018-07-11 08:06:21.242469) v4 -- 0x55f66cb35000 con 0 > -4042> 2018-07-11 08:06:21.242597 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6810/15287 -- osd_ping(ping e509 stamp 2018-07-11 08:06:21.242469) v4 -- 0x55f66cb35200 con 0 > -4041> 2018-07-11 08:06:21.242609 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6811/15287 -- osd_ping(ping e509 stamp 2018-07-11 08:06:21.242469) v4 -- 0x55f66cb35400 con 0 > -4040> 2018-07-11 08:06:21.242622 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6806/15263 -- osd_ping(ping e509 stamp 2018-07-11 08:06:21.242469) v4 -- 0x55f66cb35600 con 0 > -4039> 2018-07-11 08:06:21.242639 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6807/15263 -- osd_ping(ping e509 stamp 2018-07-11 08:06:21.242469) v4 -- 0x55f66cb35800 con 0 > -4038> 2018-07-11 08:06:21.242652 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.115:6802/13323 -- osd_ping(ping e509 stamp 2018-07-11 08:06:21.242469) v4 -- 0x55f66cb35a00 con 0 > -4037> 2018-07-11 08:06:21.242665 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.115:6803/13323 -- osd_ping(ping e509 stamp 2018-07-11 08:06:21.242469) v4 -- 0x55f66cb35c00 con 0 > -4036> 2018-07-11 08:06:21.242679 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6806/15885 -- osd_ping(ping e509 stamp 2018-07-11 08:06:21.242469) v4 -- 0x55f66cb35e00 con 0 > -4035> 2018-07-11 08:06:21.242692 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6807/15885 -- osd_ping(ping e509 stamp 2018-07-11 08:06:21.242469) v4 -- 0x55f65fa3c000 con 0 > -4034> 2018-07-11 08:06:21.242704 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.108:6802/11967 -- osd_ping(ping e509 stamp 2018-07-11 08:06:21.242469) v4 -- 0x55f65fa3c200 con 0 > -4033> 2018-07-11 08:06:21.242715 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.108:6803/11967 -- osd_ping(ping e509 stamp 2018-07-11 08:06:21.242469) v4 -- 0x55f65fa3c400 con 0 > -4032> 2018-07-11 08:06:21.242727 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.107:6806/12617 -- osd_ping(ping e509 stamp 2018-07-11 08:06:21.242469) v4 -- 0x55f65fa3c600 con 0 > -4031> 2018-07-11 08:06:21.242739 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.107:6807/12617 -- osd_ping(ping e509 stamp 2018-07-11 08:06:21.242469) v4 -- 0x55f65fa3c800 con 0 > -4030> 2018-07-11 08:06:21.242751 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6802/15919 -- osd_ping(ping e509 stamp 2018-07-11 08:06:21.242469) v4 -- 0x55f65fa3ca00 con 0 > -4029> 2018-07-11 08:06:21.242763 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6803/15919 -- osd_ping(ping e509 stamp 2018-07-11 08:06:21.242469) v4 -- 0x55f65fa3cc00 con 0 > -4028> 2018-07-11 08:06:21.242924 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6807/12677 conn(0x55f64e3d9800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.1 seq 211 0x55f673909400 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:21.242469) v4 > -4027> 2018-07-11 08:06:21.242951 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.1 10.8.128.106:6807/12677 211 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:21.242469) v4 ==== 2004+0+0 (1121759407 0 0) 0x55f673909400 con 0x55f64e3d9800 > -4026> 2018-07-11 08:06:21.242974 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6803/15255 conn(0x55f656ad5800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 211 0x55f67db36600 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:21.242469) v4 > -4025> 2018-07-11 08:06:21.242995 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6802/15255 conn(0x55f656ad4000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 211 0x55f673909400 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:21.242469) v4 > -4024> 2018-07-11 08:06:21.242994 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.0 10.8.128.113:6803/15255 211 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:21.242469) v4 ==== 2004+0+0 (1121759407 0 0) 0x55f67db36600 con 0x55f656ad5800 > -4023> 2018-07-11 08:06:21.243010 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.0 10.8.128.113:6802/15255 211 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:21.242469) v4 ==== 2004+0+0 (1121759407 0 0) 0x55f673909400 con 0x55f656ad4000 > -4022> 2018-07-11 08:06:21.243005 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6806/15263 conn(0x55f64e3ed000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.5 seq 211 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:21.242469) v4 > -4021> 2018-07-11 08:06:21.243045 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.5 10.8.128.113:6806/15263 211 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:21.242469) v4 ==== 2004+0+0 (1121759407 0 0) 0x55f64e20fc00 con 0x55f64e3ed000 > -4020> 2018-07-11 08:06:21.243056 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6810/12651 conn(0x55f64e3eb800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.2 seq 211 0x55f67db36600 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:21.242469) v4 > -4019> 2018-07-11 08:06:21.243056 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6811/12650 conn(0x55f656ad8800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.3 seq 211 0x55f673909400 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:21.242469) v4 > -4018> 2018-07-11 08:06:21.243092 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.3 10.8.128.106:6811/12650 211 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:21.242469) v4 ==== 2004+0+0 (1121759407 0 0) 0x55f673909400 con 0x55f656ad8800 > -4017> 2018-07-11 08:06:21.243077 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.2 10.8.128.106:6810/12651 211 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:21.242469) v4 ==== 2004+0+0 (1121759407 0 0) 0x55f67db36600 con 0x55f64e3eb800 > -4016> 2018-07-11 08:06:21.243107 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6809/12650 conn(0x55f656ad7000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.3 seq 211 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:21.242469) v4 > -4015> 2018-07-11 08:06:21.243141 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.107:6806/12617 conn(0x55f656ade000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.9 seq 211 0x55f67db36600 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:21.242469) v4 > -4014> 2018-07-11 08:06:21.243141 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6808/12651 conn(0x55f64e3ea000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.2 seq 211 0x55f673909400 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:21.242469) v4 > -4013> 2018-07-11 08:06:21.243146 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.3 10.8.128.106:6809/12650 211 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:21.242469) v4 ==== 2004+0+0 (1121759407 0 0) 0x55f64e20fc00 con 0x55f656ad7000 > -4012> 2018-07-11 08:06:21.243163 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.9 10.8.128.107:6806/12617 211 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:21.242469) v4 ==== 2004+0+0 (1121759407 0 0) 0x55f67db36600 con 0x55f656ade000 > -4011> 2018-07-11 08:06:21.243163 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.2 10.8.128.106:6808/12651 211 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:21.242469) v4 ==== 2004+0+0 (1121759407 0 0) 0x55f673909400 con 0x55f64e3ea000 > -4010> 2018-07-11 08:06:21.243188 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6806/12677 conn(0x55f64e3d8000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.1 seq 211 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:21.242469) v4 > -4009> 2018-07-11 08:06:21.243202 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.1 10.8.128.106:6806/12677 211 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:21.242469) v4 ==== 2004+0+0 (1121759407 0 0) 0x55f64e20fc00 con 0x55f64e3d8000 > -4008> 2018-07-11 08:06:21.243219 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6811/15287 conn(0x55f64e402800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.4 seq 211 0x55f67db36600 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:21.242469) v4 > -4007> 2018-07-11 08:06:21.243216 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6810/15287 conn(0x55f64e401000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.4 seq 211 0x55f673909400 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:21.242469) v4 > -4006> 2018-07-11 08:06:21.243238 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.4 10.8.128.113:6811/15287 211 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:21.242469) v4 ==== 2004+0+0 (1121759407 0 0) 0x55f67db36600 con 0x55f64e402800 > -4005> 2018-07-11 08:06:21.243240 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.4 10.8.128.113:6810/15287 211 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:21.242469) v4 ==== 2004+0+0 (1121759407 0 0) 0x55f673909400 con 0x55f64e401000 > -4004> 2018-07-11 08:06:21.243254 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.115:6803/13323 conn(0x55f656ace800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.6 seq 211 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:21.242469) v4 > -4003> 2018-07-11 08:06:21.243268 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.6 10.8.128.115:6803/13323 211 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:21.242469) v4 ==== 2004+0+0 (1121759407 0 0) 0x55f64e20fc00 con 0x55f656ace800 > -4002> 2018-07-11 08:06:21.243307 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6807/15885 conn(0x55f64e3ff800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.7 seq 211 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:21.242469) v4 > -4001> 2018-07-11 08:06:21.243305 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.115:6802/13323 conn(0x55f656acd000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.6 seq 211 0x55f67db36600 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:21.242469) v4 > -4000> 2018-07-11 08:06:21.243309 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6807/15263 conn(0x55f64e3ee800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.5 seq 211 0x55f673909400 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:21.242469) v4 > -3999> 2018-07-11 08:06:21.243321 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.7 10.8.128.114:6807/15885 211 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:21.242469) v4 ==== 2004+0+0 (1121759407 0 0) 0x55f64e20fc00 con 0x55f64e3ff800 > -3998> 2018-07-11 08:06:21.243323 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.6 10.8.128.115:6802/13323 211 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:21.242469) v4 ==== 2004+0+0 (1121759407 0 0) 0x55f67db36600 con 0x55f656acd000 > -3997> 2018-07-11 08:06:21.243323 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.5 10.8.128.113:6807/15263 211 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:21.242469) v4 ==== 2004+0+0 (1121759407 0 0) 0x55f673909400 con 0x55f64e3ee800 > -3996> 2018-07-11 08:06:21.243352 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.107:6807/12617 conn(0x55f656adf800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.9 seq 211 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:21.242469) v4 > -3995> 2018-07-11 08:06:21.243363 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.9 10.8.128.107:6807/12617 211 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:21.242469) v4 ==== 2004+0+0 (1121759407 0 0) 0x55f64e20fc00 con 0x55f656adf800 > -3994> 2018-07-11 08:06:21.243376 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6806/15885 conn(0x55f64e3fe000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.7 seq 211 0x55f67db36600 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:21.242469) v4 > -3993> 2018-07-11 08:06:21.243391 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.7 10.8.128.114:6806/15885 211 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:21.242469) v4 ==== 2004+0+0 (1121759407 0 0) 0x55f67db36600 con 0x55f64e3fe000 > -3992> 2018-07-11 08:06:21.243426 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.108:6802/11967 conn(0x55f656aca000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.8 seq 211 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:21.242469) v4 > -3991> 2018-07-11 08:06:21.243443 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.8 10.8.128.108:6802/11967 211 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:21.242469) v4 ==== 2004+0+0 (1121759407 0 0) 0x55f64e20fc00 con 0x55f656aca000 > -3990> 2018-07-11 08:06:21.243446 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6802/15919 conn(0x55f64e3db000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.11 seq 211 0x55f67db36600 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:21.242469) v4 > -3989> 2018-07-11 08:06:21.243451 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.108:6803/11967 conn(0x55f656acb800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.8 seq 211 0x55f673909400 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:21.242469) v4 > -3988> 2018-07-11 08:06:21.243463 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.11 10.8.128.114:6802/15919 211 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:21.242469) v4 ==== 2004+0+0 (1121759407 0 0) 0x55f67db36600 con 0x55f64e3db000 > -3987> 2018-07-11 08:06:21.243465 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.8 10.8.128.108:6803/11967 211 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:21.242469) v4 ==== 2004+0+0 (1121759407 0 0) 0x55f673909400 con 0x55f656acb800 > -3986> 2018-07-11 08:06:21.243490 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6803/15919 conn(0x55f64e3dc800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.11 seq 211 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:21.242469) v4 > -3985> 2018-07-11 08:06:21.243505 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.11 10.8.128.114:6803/15919 211 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:21.242469) v4 ==== 2004+0+0 (1121759407 0 0) 0x55f64e20fc00 con 0x55f64e3dc800 > -3984> 2018-07-11 08:06:21.296431 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.107:0/12617 conn(0x55f64e3c3800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.9 seq 218 0x55f67db36600 osd_ping(ping e509 stamp 2018-07-11 08:06:21.296090) v4 > -3983> 2018-07-11 08:06:21.296450 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.9 10.8.128.107:0/12617 218 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:21.296090) v4 ==== 2004+0+0 (2214230342 0 0) 0x55f67db36600 con 0x55f64e3c3800 > -3982> 2018-07-11 08:06:21.296445 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.107:0/12617 conn(0x55f64e3c5000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.9 seq 218 0x55f673909400 osd_ping(ping e509 stamp 2018-07-11 08:06:21.296090) v4 > -3981> 2018-07-11 08:06:21.296462 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:21.296090) v4 -- 0x55f66cb34400 con 0 > -3980> 2018-07-11 08:06:21.296474 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.9 10.8.128.107:0/12617 218 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:21.296090) v4 ==== 2004+0+0 (2214230342 0 0) 0x55f673909400 con 0x55f64e3c5000 > -3979> 2018-07-11 08:06:21.296500 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.107:0/12617 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:21.296090) v4 -- 0x55f66cb34200 con 0 > -3978> 2018-07-11 08:06:21.383550 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed7800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 216 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:06:21.440814) v4 > -3977> 2018-07-11 08:06:21.383587 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.1 10.8.128.106:0/12677 216 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:21.440814) v4 ==== 2004+0+0 (2103252416 0 0) 0x55f64e20fc00 con 0x55f663ed7800 > -3976> 2018-07-11 08:06:21.383599 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:21.440814) v4 -- 0x55f66cb34e00 con 0 > -3975> 2018-07-11 08:06:21.383616 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed6000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 216 0x55f673908e00 osd_ping(ping e509 stamp 2018-07-11 08:06:21.440814) v4 > -3974> 2018-07-11 08:06:21.383637 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.1 10.8.128.106:0/12677 216 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:21.440814) v4 ==== 2004+0+0 (2103252416 0 0) 0x55f673908e00 con 0x55f663ed6000 > -3973> 2018-07-11 08:06:21.383649 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:21.440814) v4 -- 0x55f673909600 con 0 > -3972> 2018-07-11 08:06:21.640796 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15885 conn(0x55f663216000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 219 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:06:22.276752) v4 > -3971> 2018-07-11 08:06:21.640799 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15885 conn(0x55f663217800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 219 0x55f67d151800 osd_ping(ping e509 stamp 2018-07-11 08:06:22.276752) v4 > -3970> 2018-07-11 08:06:21.640829 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.7 10.8.128.114:0/15885 219 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:22.276752) v4 ==== 2004+0+0 (1333886599 0 0) 0x55f67d1afe00 con 0x55f663216000 > -3969> 2018-07-11 08:06:21.640839 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:22.276752) v4 -- 0x55f66cb34600 con 0 > -3968> 2018-07-11 08:06:21.640833 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.7 10.8.128.114:0/15885 219 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:22.276752) v4 ==== 2004+0+0 (1333886599 0 0) 0x55f67d151800 con 0x55f663217800 > -3967> 2018-07-11 08:06:21.640868 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:22.276752) v4 -- 0x55f67d14f400 con 0 > -3966> 2018-07-11 08:06:21.859160 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f66522f8c0 con 0 > -3965> 2018-07-11 08:06:21.860715 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 279 0x55f67be45400 replica scrub(pg: 6.15,from:0'0,to:430'106549,epoch:509/504,start:6:a8721343:::benchmark_data_magna108_35617_object441441:0,end:6:a872fbc1:::benchmark_data_magna108.ceph.redhat.com_2478_object383255:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -3964> 2018-07-11 08:06:21.860747 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 279 ==== replica scrub(pg: 6.15,from:0'0,to:430'106549,epoch:509/504,start:6:a8721343:::benchmark_data_magna108_35617_object441441:0,end:6:a872fbc1:::benchmark_data_magna108.ceph.redhat.com_2478_object383255:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (565474254 0 0) 0x55f67be45400 con 0x55f66315e800 > -3963> 2018-07-11 08:06:22.315326 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.115:0/13323 conn(0x55f663ebf800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.6 seq 201 0x55f673909400 osd_ping(ping e509 stamp 2018-07-11 08:06:22.668414) v4 > -3962> 2018-07-11 08:06:22.315366 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.6 10.8.128.115:0/13323 201 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:22.668414) v4 ==== 2004+0+0 (3732243513 0 0) 0x55f673909400 con 0x55f663ebf800 > -3961> 2018-07-11 08:06:22.315369 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.115:0/13323 conn(0x55f663ebe000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.6 seq 201 0x55f67db36600 osd_ping(ping e509 stamp 2018-07-11 08:06:22.668414) v4 > -3960> 2018-07-11 08:06:22.315385 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:22.668414) v4 -- 0x55f673909800 con 0 > -3959> 2018-07-11 08:06:22.315396 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.6 10.8.128.115:0/13323 201 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:22.668414) v4 ==== 2004+0+0 (3732243513 0 0) 0x55f67db36600 con 0x55f663ebe000 > -3958> 2018-07-11 08:06:22.315423 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:22.668414) v4 -- 0x55f67d14f400 con 0 > -3957> 2018-07-11 08:06:22.519799 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f6619a26c0 con 0 > -3956> 2018-07-11 08:06:22.521358 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 280 0x55f67be45680 replica scrub(pg: 6.15,from:0'0,to:430'105918,epoch:509/504,start:6:a872fbc1:::benchmark_data_magna108.ceph.redhat.com_2478_object383255:0,end:6:a873e866:::benchmark_data_magna108_35617_object212207:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -3955> 2018-07-11 08:06:22.521388 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 280 ==== replica scrub(pg: 6.15,from:0'0,to:430'105918,epoch:509/504,start:6:a872fbc1:::benchmark_data_magna108.ceph.redhat.com_2478_object383255:0,end:6:a873e866:::benchmark_data_magna108_35617_object212207:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 233+0+0 (3198369869 0 0) 0x55f67be45680 con 0x55f66315e800 > -3954> 2018-07-11 08:06:22.662172 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0e800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 222 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:06:23.907678) v4 > -3953> 2018-07-11 08:06:22.662183 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0d000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 222 0x55f673909600 osd_ping(ping e509 stamp 2018-07-11 08:06:23.907678) v4 > -3952> 2018-07-11 08:06:22.662196 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.5 10.8.128.113:0/15263 222 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:23.907678) v4 ==== 2004+0+0 (240507472 0 0) 0x55f64e20fc00 con 0x55f663f0e800 > -3951> 2018-07-11 08:06:22.662202 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.5 10.8.128.113:0/15263 222 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:23.907678) v4 ==== 2004+0+0 (240507472 0 0) 0x55f673909600 con 0x55f663f0d000 > -3950> 2018-07-11 08:06:22.662215 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:23.907678) v4 -- 0x55f66cb34600 con 0 > -3949> 2018-07-11 08:06:22.662252 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:23.907678) v4 -- 0x55f673909800 con 0 > -3948> 2018-07-11 08:06:22.708765 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15287 conn(0x55f66dce1000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.4 seq 219 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:06:23.954127) v4 > -3947> 2018-07-11 08:06:22.708776 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15287 conn(0x55f66dce2800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.4 seq 219 0x55f673909400 osd_ping(ping e509 stamp 2018-07-11 08:06:23.954127) v4 > -3946> 2018-07-11 08:06:22.708790 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.4 10.8.128.113:0/15287 219 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:23.954127) v4 ==== 2004+0+0 (3554883581 0 0) 0x55f67d1afe00 con 0x55f66dce1000 > -3945> 2018-07-11 08:06:22.708802 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:23.954127) v4 -- 0x55f66cb34600 con 0 > -3944> 2018-07-11 08:06:22.708804 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.4 10.8.128.113:0/15287 219 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:23.954127) v4 ==== 2004+0+0 (3554883581 0 0) 0x55f673909400 con 0x55f66dce2800 > -3943> 2018-07-11 08:06:22.708828 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:23.954127) v4 -- 0x55f673909800 con 0 > -3942> 2018-07-11 08:06:22.804950 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.108:0/11967 conn(0x55f663f6f000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=1). rx osd.8 seq 206 0x55f67d151800 osd_ping(ping e509 stamp 2018-07-11 08:06:22.565084) v4 > -3941> 2018-07-11 08:06:22.804971 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.8 10.8.128.108:0/11967 206 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:22.565084) v4 ==== 2004+0+0 (1985267843 0 0) 0x55f67d151800 con 0x55f663f6f000 > -3940> 2018-07-11 08:06:22.804981 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:22.565084) v4 -- 0x55f67d14f400 con 0 > -3939> 2018-07-11 08:06:22.804969 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.108:0/11967 conn(0x55f663f70800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=22 cs=1 l=1). rx osd.8 seq 206 0x55f673909600 osd_ping(ping e509 stamp 2018-07-11 08:06:22.565084) v4 > -3938> 2018-07-11 08:06:22.804991 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.8 10.8.128.108:0/11967 206 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:22.565084) v4 ==== 2004+0+0 (1985267843 0 0) 0x55f673909600 con 0x55f663f70800 > -3937> 2018-07-11 08:06:22.805008 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.108:0/11967 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:22.565084) v4 -- 0x55f673909800 con 0 > -3936> 2018-07-11 08:06:23.084100 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed7800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 217 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:06:23.141366) v4 > -3935> 2018-07-11 08:06:23.084102 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12677 conn(0x55f663ed6000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.1 seq 217 0x55f673909400 osd_ping(ping e509 stamp 2018-07-11 08:06:23.141366) v4 > -3934> 2018-07-11 08:06:23.084123 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.1 10.8.128.106:0/12677 217 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:23.141366) v4 ==== 2004+0+0 (3649075452 0 0) 0x55f64e20fc00 con 0x55f663ed7800 > -3933> 2018-07-11 08:06:23.084126 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.1 10.8.128.106:0/12677 217 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:23.141366) v4 ==== 2004+0+0 (3649075452 0 0) 0x55f673909400 con 0x55f663ed6000 > -3932> 2018-07-11 08:06:23.084133 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:23.141366) v4 -- 0x55f666c47800 con 0 > -3931> 2018-07-11 08:06:23.084154 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12677 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:23.141366) v4 -- 0x55f673909a00 con 0 > -3930> 2018-07-11 08:06:23.209208 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15287 conn(0x55f66dce2800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.4 seq 220 0x55f673909600 osd_ping(ping e509 stamp 2018-07-11 08:06:24.454696) v4 > -3929> 2018-07-11 08:06:23.209229 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.4 10.8.128.113:0/15287 220 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:24.454696) v4 ==== 2004+0+0 (3891817085 0 0) 0x55f673909600 con 0x55f66dce2800 > -3928> 2018-07-11 08:06:23.209244 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:24.454696) v4 -- 0x55f673909c00 con 0 > -3927> 2018-07-11 08:06:23.209269 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15287 conn(0x55f66dce1000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.4 seq 220 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:06:24.454696) v4 > -3926> 2018-07-11 08:06:23.209291 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.4 10.8.128.113:0/15287 220 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:24.454696) v4 ==== 2004+0+0 (3891817085 0 0) 0x55f67d1afe00 con 0x55f66dce1000 > -3925> 2018-07-11 08:06:23.209300 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15287 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:24.454696) v4 -- 0x55f653d4b800 con 0 > -3924> 2018-07-11 08:06:23.762736 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0d000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 223 0x55f673909400 osd_ping(ping e509 stamp 2018-07-11 08:06:25.008112) v4 > -3923> 2018-07-11 08:06:23.762762 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.5 10.8.128.113:0/15263 223 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:25.008112) v4 ==== 2004+0+0 (2315514792 0 0) 0x55f673909400 con 0x55f663f0d000 > -3922> 2018-07-11 08:06:23.762771 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:25.008112) v4 -- 0x55f673909e00 con 0 > -3921> 2018-07-11 08:06:23.762803 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0e800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 223 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:06:25.008112) v4 > -3920> 2018-07-11 08:06:23.762830 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.5 10.8.128.113:0/15263 223 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:25.008112) v4 ==== 2004+0+0 (2315514792 0 0) 0x55f64e20fc00 con 0x55f663f0e800 > -3919> 2018-07-11 08:06:23.762850 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:25.008112) v4 -- 0x55f65c072000 con 0 > -3918> 2018-07-11 08:06:23.806568 7f4922851700 4 rocksdb: [/builddir/build/BUILD/ceph-12.2.5/src/rocksdb/db/compaction_job.cc:1116] [default] [JOB 30] Generated table #37052: 2638241 keys, 74546882 bytes > -3917> 2018-07-11 08:06:23.806617 7f4922851700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1531296383806601, "cf_name": "default", "job": 30, "event": "table_file_creation", "file_number": 37052, "file_size": 74546882, "table_properties": {"data_size": 67112148, "index_size": 690561, "filter_size": 6743183, "raw_key_size": 76215859, "raw_average_key_size": 28, "raw_value_size": 31365762, "raw_average_value_size": 11, "num_data_blocks": 16490, "num_entries": 2638241, "filter_policy_name": "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "0", "kMergeOperands": "0"}} > -3916> 2018-07-11 08:06:24.142916 7f491803c700 5 osd.10 509 heartbeat: osd_stat(790 GB used, 141 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,0,0,0,1]) > -3915> 2018-07-11 08:06:24.142936 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6802/15255 -- osd_ping(ping e509 stamp 2018-07-11 08:06:24.142933) v4 -- 0x55f65fa3ce00 con 0 > -3914> 2018-07-11 08:06:24.142985 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6803/15255 -- osd_ping(ping e509 stamp 2018-07-11 08:06:24.142933) v4 -- 0x55f65fa3d000 con 0 > -3913> 2018-07-11 08:06:24.143002 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6806/12677 -- osd_ping(ping e509 stamp 2018-07-11 08:06:24.142933) v4 -- 0x55f65fa3d200 con 0 > -3912> 2018-07-11 08:06:24.143017 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6807/12677 -- osd_ping(ping e509 stamp 2018-07-11 08:06:24.142933) v4 -- 0x55f65fa3d400 con 0 > -3911> 2018-07-11 08:06:24.143030 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6808/12651 -- osd_ping(ping e509 stamp 2018-07-11 08:06:24.142933) v4 -- 0x55f65fa3d600 con 0 > -3910> 2018-07-11 08:06:24.143041 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6810/12651 -- osd_ping(ping e509 stamp 2018-07-11 08:06:24.142933) v4 -- 0x55f65fa3d800 con 0 > -3909> 2018-07-11 08:06:24.143052 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6809/12650 -- osd_ping(ping e509 stamp 2018-07-11 08:06:24.142933) v4 -- 0x55f65fa3da00 con 0 > -3908> 2018-07-11 08:06:24.143064 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6811/12650 -- osd_ping(ping e509 stamp 2018-07-11 08:06:24.142933) v4 -- 0x55f65fa3dc00 con 0 > -3907> 2018-07-11 08:06:24.143074 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6810/15287 -- osd_ping(ping e509 stamp 2018-07-11 08:06:24.142933) v4 -- 0x55f65fa3de00 con 0 > -3906> 2018-07-11 08:06:24.143095 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6811/15287 -- osd_ping(ping e509 stamp 2018-07-11 08:06:24.142933) v4 -- 0x55f67c1fa000 con 0 > -3905> 2018-07-11 08:06:24.143110 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6806/15263 -- osd_ping(ping e509 stamp 2018-07-11 08:06:24.142933) v4 -- 0x55f67c1fa200 con 0 > -3904> 2018-07-11 08:06:24.143123 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6807/15263 -- osd_ping(ping e509 stamp 2018-07-11 08:06:24.142933) v4 -- 0x55f67c1fa400 con 0 > -3903> 2018-07-11 08:06:24.143135 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.115:6802/13323 -- osd_ping(ping e509 stamp 2018-07-11 08:06:24.142933) v4 -- 0x55f67c1fa600 con 0 > -3902> 2018-07-11 08:06:24.143147 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.115:6803/13323 -- osd_ping(ping e509 stamp 2018-07-11 08:06:24.142933) v4 -- 0x55f67c1fa800 con 0 > -3901> 2018-07-11 08:06:24.143163 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6806/15885 -- osd_ping(ping e509 stamp 2018-07-11 08:06:24.142933) v4 -- 0x55f67c1faa00 con 0 > -3900> 2018-07-11 08:06:24.143176 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6807/15885 -- osd_ping(ping e509 stamp 2018-07-11 08:06:24.142933) v4 -- 0x55f67c1fac00 con 0 > -3899> 2018-07-11 08:06:24.143191 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.108:6802/11967 -- osd_ping(ping e509 stamp 2018-07-11 08:06:24.142933) v4 -- 0x55f67c1fae00 con 0 > -3898> 2018-07-11 08:06:24.143202 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.108:6803/11967 -- osd_ping(ping e509 stamp 2018-07-11 08:06:24.142933) v4 -- 0x55f67c1fb000 con 0 > -3897> 2018-07-11 08:06:24.143215 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.107:6806/12617 -- osd_ping(ping e509 stamp 2018-07-11 08:06:24.142933) v4 -- 0x55f67c1fb200 con 0 > -3896> 2018-07-11 08:06:24.143228 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.107:6807/12617 -- osd_ping(ping e509 stamp 2018-07-11 08:06:24.142933) v4 -- 0x55f67c1fb400 con 0 > -3895> 2018-07-11 08:06:24.143240 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6802/15919 -- osd_ping(ping e509 stamp 2018-07-11 08:06:24.142933) v4 -- 0x55f67c1fb600 con 0 > -3894> 2018-07-11 08:06:24.143264 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6803/15919 -- osd_ping(ping e509 stamp 2018-07-11 08:06:24.142933) v4 -- 0x55f67c1fb800 con 0 > -3893> 2018-07-11 08:06:24.143452 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.107:6806/12617 conn(0x55f656ade000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.9 seq 212 0x55f67d151800 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:24.142933) v4 > -3892> 2018-07-11 08:06:24.143472 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.9 10.8.128.107:6806/12617 212 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:24.142933) v4 ==== 2004+0+0 (2826565768 0 0) 0x55f67d151800 con 0x55f656ade000 > -3891> 2018-07-11 08:06:24.143537 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6806/12677 conn(0x55f64e3d8000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.1 seq 212 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:24.142933) v4 > -3890> 2018-07-11 08:06:24.143557 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.1 10.8.128.106:6806/12677 212 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:24.142933) v4 ==== 2004+0+0 (2826565768 0 0) 0x55f64e20fc00 con 0x55f64e3d8000 > -3889> 2018-07-11 08:06:24.143580 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6802/15255 conn(0x55f656ad4000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 212 0x55f673909600 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:24.142933) v4 > -3888> 2018-07-11 08:06:24.143585 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6810/12651 conn(0x55f64e3eb800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.2 seq 212 0x55f67d151800 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:24.142933) v4 > -3887> 2018-07-11 08:06:24.143596 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.0 10.8.128.113:6802/15255 212 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:24.142933) v4 ==== 2004+0+0 (2826565768 0 0) 0x55f673909600 con 0x55f656ad4000 > -3886> 2018-07-11 08:06:24.143598 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.2 10.8.128.106:6810/12651 212 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:24.142933) v4 ==== 2004+0+0 (2826565768 0 0) 0x55f67d151800 con 0x55f64e3eb800 > -3885> 2018-07-11 08:06:24.143586 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.107:6807/12617 conn(0x55f656adf800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.9 seq 212 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:24.142933) v4 > -3884> 2018-07-11 08:06:24.143619 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.9 10.8.128.107:6807/12617 212 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:24.142933) v4 ==== 2004+0+0 (2826565768 0 0) 0x55f64e20fc00 con 0x55f656adf800 > -3883> 2018-07-11 08:06:24.143641 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6811/12650 conn(0x55f656ad8800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.3 seq 212 0x55f673909600 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:24.142933) v4 > -3882> 2018-07-11 08:06:24.143655 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.3 10.8.128.106:6811/12650 212 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:24.142933) v4 ==== 2004+0+0 (2826565768 0 0) 0x55f673909600 con 0x55f656ad8800 > -3881> 2018-07-11 08:06:24.143638 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6803/15255 conn(0x55f656ad5800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 212 0x55f67d151800 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:24.142933) v4 > -3880> 2018-07-11 08:06:24.143657 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6809/12650 conn(0x55f656ad7000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.3 seq 212 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:24.142933) v4 > -3879> 2018-07-11 08:06:24.143674 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.0 10.8.128.113:6803/15255 212 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:24.142933) v4 ==== 2004+0+0 (2826565768 0 0) 0x55f67d151800 con 0x55f656ad5800 > -3878> 2018-07-11 08:06:24.143675 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6808/12651 conn(0x55f64e3ea000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.2 seq 212 0x55f673909600 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:24.142933) v4 > -3877> 2018-07-11 08:06:24.143678 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.3 10.8.128.106:6809/12650 212 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:24.142933) v4 ==== 2004+0+0 (2826565768 0 0) 0x55f64e20fc00 con 0x55f656ad7000 > -3876> 2018-07-11 08:06:24.143684 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.2 10.8.128.106:6808/12651 212 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:24.142933) v4 ==== 2004+0+0 (2826565768 0 0) 0x55f673909600 con 0x55f64e3ea000 > -3875> 2018-07-11 08:06:24.143710 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6807/12677 conn(0x55f64e3d9800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.1 seq 212 0x55f673909600 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:24.142933) v4 > -3874> 2018-07-11 08:06:24.143720 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.1 10.8.128.106:6807/12677 212 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:24.142933) v4 ==== 2004+0+0 (2826565768 0 0) 0x55f673909600 con 0x55f64e3d9800 > -3873> 2018-07-11 08:06:24.143719 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6806/15263 conn(0x55f64e3ed000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.5 seq 212 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:24.142933) v4 > -3872> 2018-07-11 08:06:24.143730 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.5 10.8.128.113:6806/15263 212 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:24.142933) v4 ==== 2004+0+0 (2826565768 0 0) 0x55f64e20fc00 con 0x55f64e3ed000 > -3871> 2018-07-11 08:06:24.143725 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6811/15287 conn(0x55f64e402800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.4 seq 212 0x55f67d151800 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:24.142933) v4 > -3870> 2018-07-11 08:06:24.143745 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.4 10.8.128.113:6811/15287 212 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:24.142933) v4 ==== 2004+0+0 (2826565768 0 0) 0x55f67d151800 con 0x55f64e402800 > -3869> 2018-07-11 08:06:24.143753 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6807/15263 conn(0x55f64e3ee800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.5 seq 212 0x55f673909600 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:24.142933) v4 > -3868> 2018-07-11 08:06:24.143766 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.5 10.8.128.113:6807/15263 212 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:24.142933) v4 ==== 2004+0+0 (2826565768 0 0) 0x55f673909600 con 0x55f64e3ee800 > -3867> 2018-07-11 08:06:24.143765 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.115:6803/13323 conn(0x55f656ace800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.6 seq 212 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:24.142933) v4 > -3866> 2018-07-11 08:06:24.143776 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.6 10.8.128.115:6803/13323 212 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:24.142933) v4 ==== 2004+0+0 (2826565768 0 0) 0x55f64e20fc00 con 0x55f656ace800 > -3865> 2018-07-11 08:06:24.143778 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6806/15885 conn(0x55f64e3fe000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.7 seq 212 0x55f67d151800 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:24.142933) v4 > -3864> 2018-07-11 08:06:24.143795 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.7 10.8.128.114:6806/15885 212 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:24.142933) v4 ==== 2004+0+0 (2826565768 0 0) 0x55f67d151800 con 0x55f64e3fe000 > -3863> 2018-07-11 08:06:24.143795 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6810/15287 conn(0x55f64e401000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.4 seq 212 0x55f673909600 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:24.142933) v4 > -3862> 2018-07-11 08:06:24.143804 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.4 10.8.128.113:6810/15287 212 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:24.142933) v4 ==== 2004+0+0 (2826565768 0 0) 0x55f673909600 con 0x55f64e401000 > -3861> 2018-07-11 08:06:24.143836 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.108:6802/11967 conn(0x55f656aca000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.8 seq 212 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:24.142933) v4 > -3860> 2018-07-11 08:06:24.143843 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.115:6802/13323 conn(0x55f656acd000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.6 seq 212 0x55f67d151800 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:24.142933) v4 > -3859> 2018-07-11 08:06:24.143849 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.8 10.8.128.108:6802/11967 212 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:24.142933) v4 ==== 2004+0+0 (2826565768 0 0) 0x55f64e20fc00 con 0x55f656aca000 > -3858> 2018-07-11 08:06:24.143859 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.6 10.8.128.115:6802/13323 212 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:24.142933) v4 ==== 2004+0+0 (2826565768 0 0) 0x55f67d151800 con 0x55f656acd000 > -3857> 2018-07-11 08:06:24.143887 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6807/15885 conn(0x55f64e3ff800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.7 seq 212 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:24.142933) v4 > -3856> 2018-07-11 08:06:24.143884 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.108:6803/11967 conn(0x55f656acb800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.8 seq 212 0x55f673909600 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:24.142933) v4 > -3855> 2018-07-11 08:06:24.143892 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6802/15919 conn(0x55f64e3db000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.11 seq 212 0x55f67d151800 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:24.142933) v4 > -3854> 2018-07-11 08:06:24.143901 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.7 10.8.128.114:6807/15885 212 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:24.142933) v4 ==== 2004+0+0 (2826565768 0 0) 0x55f64e20fc00 con 0x55f64e3ff800 > -3853> 2018-07-11 08:06:24.143902 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.8 10.8.128.108:6803/11967 212 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:24.142933) v4 ==== 2004+0+0 (2826565768 0 0) 0x55f673909600 con 0x55f656acb800 > -3852> 2018-07-11 08:06:24.143904 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.11 10.8.128.114:6802/15919 212 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:24.142933) v4 ==== 2004+0+0 (2826565768 0 0) 0x55f67d151800 con 0x55f64e3db000 > -3851> 2018-07-11 08:06:24.143932 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6803/15919 conn(0x55f64e3dc800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.11 seq 212 0x55f64e20fc00 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:24.142933) v4 > -3850> 2018-07-11 08:06:24.143944 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.11 10.8.128.114:6803/15919 212 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:24.142933) v4 ==== 2004+0+0 (2826565768 0 0) 0x55f64e20fc00 con 0x55f64e3dc800 > -3849> 2018-07-11 08:06:24.236495 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12650 conn(0x55f663ed9000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.3 seq 213 0x55f67d151800 osd_ping(ping e509 stamp 2018-07-11 08:06:24.293910) v4 > -3848> 2018-07-11 08:06:24.236519 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.3 10.8.128.106:0/12650 213 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:24.293910) v4 ==== 2004+0+0 (3265911132 0 0) 0x55f67d151800 con 0x55f663ed9000 > -3847> 2018-07-11 08:06:24.236515 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12650 conn(0x55f663eda800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=18 cs=1 l=1). rx osd.3 seq 213 0x55f673909600 osd_ping(ping e509 stamp 2018-07-11 08:06:24.293910) v4 > -3846> 2018-07-11 08:06:24.236530 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:24.293910) v4 -- 0x55f65fa3d000 con 0 > -3845> 2018-07-11 08:06:24.236538 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.3 10.8.128.106:0/12650 213 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:24.293910) v4 ==== 2004+0+0 (3265911132 0 0) 0x55f673909600 con 0x55f663eda800 > -3844> 2018-07-11 08:06:24.236559 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12650 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:24.293910) v4 -- 0x55f65fa3ce00 con 0 > -3843> 2018-07-11 08:06:24.283054 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15255 conn(0x55f663ec1000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.0 seq 208 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:06:25.528566) v4 > -3842> 2018-07-11 08:06:24.283075 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.0 10.8.128.113:0/15255 208 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:25.528566) v4 ==== 2004+0+0 (3243881031 0 0) 0x55f64e20fc00 con 0x55f663ec1000 > -3841> 2018-07-11 08:06:24.283087 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:25.528566) v4 -- 0x55f67c1fa200 con 0 > -3840> 2018-07-11 08:06:24.283118 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15255 conn(0x55f663ec2800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.0 seq 208 0x55f67db36600 osd_ping(ping e509 stamp 2018-07-11 08:06:25.528566) v4 > -3839> 2018-07-11 08:06:24.283139 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.0 10.8.128.113:0/15255 208 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:25.528566) v4 ==== 2004+0+0 (3243881031 0 0) 0x55f67db36600 con 0x55f663ec2800 > -3838> 2018-07-11 08:06:24.283150 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:25.528566) v4 -- 0x55f65fa3d000 con 0 > -3837> 2018-07-11 08:06:24.326474 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f6553ec240 con 0 > -3836> 2018-07-11 08:06:24.328326 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 281 0x55f67be45900 replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:509/504,start:6:a873e866:::benchmark_data_magna108_35617_object212207:0,end:6:a875032b:::benchmark_data_magna108_35617_object942618:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -3835> 2018-07-11 08:06:24.328355 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 281 ==== replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:509/504,start:6:a873e866:::benchmark_data_magna108_35617_object212207:0,end:6:a875032b:::benchmark_data_magna108_35617_object942618:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 218+0+0 (2074630643 0 0) 0x55f67be45900 con 0x55f66315e800 > -3834> 2018-07-11 08:06:24.783659 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15255 conn(0x55f663ec2800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.0 seq 209 0x55f67d151800 osd_ping(ping e509 stamp 2018-07-11 08:06:26.029025) v4 > -3833> 2018-07-11 08:06:24.783670 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15255 conn(0x55f663ec1000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.0 seq 209 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:06:26.029025) v4 > -3832> 2018-07-11 08:06:24.783686 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.0 10.8.128.113:0/15255 209 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:26.029025) v4 ==== 2004+0+0 (3163327321 0 0) 0x55f67d151800 con 0x55f663ec2800 > -3831> 2018-07-11 08:06:24.783697 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:26.029025) v4 -- 0x55f67d14f400 con 0 > -3830> 2018-07-11 08:06:24.783698 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.0 10.8.128.113:0/15255 209 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:26.029025) v4 ==== 2004+0+0 (3163327321 0 0) 0x55f67d1afe00 con 0x55f663ec1000 > -3829> 2018-07-11 08:06:24.783724 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15255 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:26.029025) v4 -- 0x55f65fa3da00 con 0 > -3828> 2018-07-11 08:06:24.974809 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 473 0x55f67db36600 MOSDScrubReserve(11.3s5 REQUEST e509) v1 > -3827> 2018-07-11 08:06:24.974836 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 473 ==== MOSDScrubReserve(11.3s5 REQUEST e509) v1 ==== 43+0+0 (433425070 0 0) 0x55f67db36600 con 0x55f668a29000 > -3826> 2018-07-11 08:06:24.974878 7f491a040700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e509) v1 -- 0x55f67d14ee00 con 0 > -3825> 2018-07-11 08:06:24.975175 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 474 0x55f67d14f600 MOSDScrubReserve(11.3s5 RELEASE e509) v1 > -3824> 2018-07-11 08:06:24.975189 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 474 ==== MOSDScrubReserve(11.3s5 RELEASE e509) v1 ==== 43+0+0 (3603460268 0 0) 0x55f67d14f600 con 0x55f668a29000 > -3823> 2018-07-11 08:06:25.164771 7f4931830700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.106:0/12651 conn(0x55f663219000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25 cs=1 l=1). rx osd.2 seq 229 0x55f673909400 osd_ping(ping e509 stamp 2018-07-11 08:06:25.221980) v4 > -3822> 2018-07-11 08:06:25.164794 7f4931830700 1 -- 10.8.128.107:6802/14173 <== osd.2 10.8.128.106:0/12651 229 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:25.221980) v4 ==== 2004+0+0 (3616830858 0 0) 0x55f673909400 con 0x55f663219000 > -3821> 2018-07-11 08:06:25.164819 7f4931830700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:25.221980) v4 -- 0x55f673909e00 con 0 > -3820> 2018-07-11 08:06:25.164865 7f4932031700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.106:0/12651 conn(0x55f66321a800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=25 cs=1 l=1). rx osd.2 seq 229 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:06:25.221980) v4 > -3819> 2018-07-11 08:06:25.164884 7f4932031700 1 -- 10.8.128.107:6803/14173 <== osd.2 10.8.128.106:0/12651 229 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:25.221980) v4 ==== 2004+0+0 (3616830858 0 0) 0x55f64e20fc00 con 0x55f66321a800 > -3818> 2018-07-11 08:06:25.164896 7f4932031700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.106:0/12651 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:25.221980) v4 -- 0x55f65fa3d200 con 0 > -3817> 2018-07-11 08:06:25.243404 7f491803c700 5 osd.10 509 heartbeat: osd_stat(790 GB used, 141 GB avail, 931 GB total, peers [0,1,2,3,4,5,6,7,8,9,11] op hist [0,0,0,0,0,0,0,0,0,0,1]) > -3816> 2018-07-11 08:06:25.243419 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6802/15255 -- osd_ping(ping e509 stamp 2018-07-11 08:06:25.243417) v4 -- 0x55f67c1fba00 con 0 > -3815> 2018-07-11 08:06:25.243443 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6803/15255 -- osd_ping(ping e509 stamp 2018-07-11 08:06:25.243417) v4 -- 0x55f67c1fbc00 con 0 > -3814> 2018-07-11 08:06:25.243460 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6806/12677 -- osd_ping(ping e509 stamp 2018-07-11 08:06:25.243417) v4 -- 0x55f67c1fbe00 con 0 > -3813> 2018-07-11 08:06:25.243476 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6807/12677 -- osd_ping(ping e509 stamp 2018-07-11 08:06:25.243417) v4 -- 0x55f67665a000 con 0 > -3812> 2018-07-11 08:06:25.243490 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6808/12651 -- osd_ping(ping e509 stamp 2018-07-11 08:06:25.243417) v4 -- 0x55f67665a200 con 0 > -3811> 2018-07-11 08:06:25.243500 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6810/12651 -- osd_ping(ping e509 stamp 2018-07-11 08:06:25.243417) v4 -- 0x55f67665a400 con 0 > -3810> 2018-07-11 08:06:25.243512 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6809/12650 -- osd_ping(ping e509 stamp 2018-07-11 08:06:25.243417) v4 -- 0x55f67665a600 con 0 > -3809> 2018-07-11 08:06:25.243531 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.106:6811/12650 -- osd_ping(ping e509 stamp 2018-07-11 08:06:25.243417) v4 -- 0x55f67665a800 con 0 > -3808> 2018-07-11 08:06:25.243547 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6810/15287 -- osd_ping(ping e509 stamp 2018-07-11 08:06:25.243417) v4 -- 0x55f67665aa00 con 0 > -3807> 2018-07-11 08:06:25.243560 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6811/15287 -- osd_ping(ping e509 stamp 2018-07-11 08:06:25.243417) v4 -- 0x55f67665ac00 con 0 > -3806> 2018-07-11 08:06:25.243573 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6806/15263 -- osd_ping(ping e509 stamp 2018-07-11 08:06:25.243417) v4 -- 0x55f67665ae00 con 0 > -3805> 2018-07-11 08:06:25.243586 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.113:6807/15263 -- osd_ping(ping e509 stamp 2018-07-11 08:06:25.243417) v4 -- 0x55f67665b000 con 0 > -3804> 2018-07-11 08:06:25.243598 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.115:6802/13323 -- osd_ping(ping e509 stamp 2018-07-11 08:06:25.243417) v4 -- 0x55f67665b200 con 0 > -3803> 2018-07-11 08:06:25.243610 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.115:6803/13323 -- osd_ping(ping e509 stamp 2018-07-11 08:06:25.243417) v4 -- 0x55f67665b400 con 0 > -3802> 2018-07-11 08:06:25.243626 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6806/15885 -- osd_ping(ping e509 stamp 2018-07-11 08:06:25.243417) v4 -- 0x55f67665b600 con 0 > -3801> 2018-07-11 08:06:25.243639 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6807/15885 -- osd_ping(ping e509 stamp 2018-07-11 08:06:25.243417) v4 -- 0x55f67665b800 con 0 > -3800> 2018-07-11 08:06:25.243651 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.108:6802/11967 -- osd_ping(ping e509 stamp 2018-07-11 08:06:25.243417) v4 -- 0x55f67665ba00 con 0 > -3799> 2018-07-11 08:06:25.243663 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.108:6803/11967 -- osd_ping(ping e509 stamp 2018-07-11 08:06:25.243417) v4 -- 0x55f67665bc00 con 0 > -3798> 2018-07-11 08:06:25.243675 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.107:6806/12617 -- osd_ping(ping e509 stamp 2018-07-11 08:06:25.243417) v4 -- 0x55f67665be00 con 0 > -3797> 2018-07-11 08:06:25.243690 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.107:6807/12617 -- osd_ping(ping e509 stamp 2018-07-11 08:06:25.243417) v4 -- 0x55f676664000 con 0 > -3796> 2018-07-11 08:06:25.243703 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6802/15919 -- osd_ping(ping e509 stamp 2018-07-11 08:06:25.243417) v4 -- 0x55f676664200 con 0 > -3795> 2018-07-11 08:06:25.243713 7f491803c700 1 -- 10.8.128.107:0/14173 --> 10.8.128.114:6803/15919 -- osd_ping(ping e509 stamp 2018-07-11 08:06:25.243417) v4 -- 0x55f676664400 con 0 > -3794> 2018-07-11 08:06:25.243805 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6802/15255 conn(0x55f656ad4000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 213 0x55f673909600 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:25.243417) v4 > -3793> 2018-07-11 08:06:25.243830 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.0 10.8.128.113:6802/15255 213 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:25.243417) v4 ==== 2004+0+0 (2526677030 0 0) 0x55f673909600 con 0x55f656ad4000 > -3792> 2018-07-11 08:06:25.243849 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.107:6806/12617 conn(0x55f656ade000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.9 seq 213 0x55f67d14fa00 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:25.243417) v4 > -3791> 2018-07-11 08:06:25.243870 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.9 10.8.128.107:6806/12617 213 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:25.243417) v4 ==== 2004+0+0 (2526677030 0 0) 0x55f67d14fa00 con 0x55f656ade000 > -3790> 2018-07-11 08:06:25.243879 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6806/12677 conn(0x55f64e3d8000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.1 seq 213 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:25.243417) v4 > -3789> 2018-07-11 08:06:25.243900 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.1 10.8.128.106:6806/12677 213 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:25.243417) v4 ==== 2004+0+0 (2526677030 0 0) 0x55f67d1afe00 con 0x55f64e3d8000 > -3788> 2018-07-11 08:06:25.243941 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6803/15255 conn(0x55f656ad5800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=17 cs=1 l=1). rx osd.0 seq 213 0x55f67d14fa00 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:25.243417) v4 > -3787> 2018-07-11 08:06:25.243957 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.0 10.8.128.113:6803/15255 213 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:25.243417) v4 ==== 2004+0+0 (2526677030 0 0) 0x55f67d14fa00 con 0x55f656ad5800 > -3786> 2018-07-11 08:06:25.243933 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6808/12651 conn(0x55f64e3ea000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.2 seq 213 0x55f673909600 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:25.243417) v4 > -3785> 2018-07-11 08:06:25.243994 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.2 10.8.128.106:6808/12651 213 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:25.243417) v4 ==== 2004+0+0 (2526677030 0 0) 0x55f673909600 con 0x55f64e3ea000 > -3784> 2018-07-11 08:06:25.243995 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.107:6807/12617 conn(0x55f656adf800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.9 seq 213 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:25.243417) v4 > -3783> 2018-07-11 08:06:25.243989 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6811/15287 conn(0x55f64e402800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.4 seq 213 0x55f67d14fa00 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:25.243417) v4 > -3782> 2018-07-11 08:06:25.244017 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.9 10.8.128.107:6807/12617 213 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:25.243417) v4 ==== 2004+0+0 (2526677030 0 0) 0x55f67d1afe00 con 0x55f656adf800 > -3781> 2018-07-11 08:06:25.244026 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.4 10.8.128.113:6811/15287 213 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:25.243417) v4 ==== 2004+0+0 (2526677030 0 0) 0x55f67d14fa00 con 0x55f64e402800 > -3780> 2018-07-11 08:06:25.244068 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6810/12651 conn(0x55f64e3eb800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.2 seq 213 0x55f67d14fa00 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:25.243417) v4 > -3779> 2018-07-11 08:06:25.244071 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6809/12650 conn(0x55f656ad7000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.3 seq 213 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:25.243417) v4 > -3778> 2018-07-11 08:06:25.244069 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6807/12677 conn(0x55f64e3d9800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.1 seq 213 0x55f673909600 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:25.243417) v4 > -3777> 2018-07-11 08:06:25.244082 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.3 10.8.128.106:6809/12650 213 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:25.243417) v4 ==== 2004+0+0 (2526677030 0 0) 0x55f67d1afe00 con 0x55f656ad7000 > -3776> 2018-07-11 08:06:25.244082 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.2 10.8.128.106:6810/12651 213 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:25.243417) v4 ==== 2004+0+0 (2526677030 0 0) 0x55f67d14fa00 con 0x55f64e3eb800 > -3775> 2018-07-11 08:06:25.244099 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.1 10.8.128.106:6807/12677 213 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:25.243417) v4 ==== 2004+0+0 (2526677030 0 0) 0x55f673909600 con 0x55f64e3d9800 > -3774> 2018-07-11 08:06:25.244105 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6806/15263 conn(0x55f64e3ed000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.5 seq 213 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:25.243417) v4 > -3773> 2018-07-11 08:06:25.244123 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.5 10.8.128.113:6806/15263 213 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:25.243417) v4 ==== 2004+0+0 (2526677030 0 0) 0x55f67d1afe00 con 0x55f64e3ed000 > -3772> 2018-07-11 08:06:25.244127 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.115:6802/13323 conn(0x55f656acd000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.6 seq 213 0x55f67d14fa00 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:25.243417) v4 > -3771> 2018-07-11 08:06:25.244139 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.6 10.8.128.115:6802/13323 213 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:25.243417) v4 ==== 2004+0+0 (2526677030 0 0) 0x55f67d14fa00 con 0x55f656acd000 > -3770> 2018-07-11 08:06:25.244151 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.106:6811/12650 conn(0x55f656ad8800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.3 seq 213 0x55f673909600 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:25.243417) v4 > -3769> 2018-07-11 08:06:25.244159 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.115:6803/13323 conn(0x55f656ace800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.6 seq 213 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:25.243417) v4 > -3768> 2018-07-11 08:06:25.244195 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.6 10.8.128.115:6803/13323 213 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:25.243417) v4 ==== 2004+0+0 (2526677030 0 0) 0x55f67d1afe00 con 0x55f656ace800 > -3767> 2018-07-11 08:06:25.244191 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.3 10.8.128.106:6811/12650 213 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:25.243417) v4 ==== 2004+0+0 (2526677030 0 0) 0x55f673909600 con 0x55f656ad8800 > -3766> 2018-07-11 08:06:25.244204 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6806/15885 conn(0x55f64e3fe000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.7 seq 213 0x55f67d14fa00 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:25.243417) v4 > -3765> 2018-07-11 08:06:25.244216 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.7 10.8.128.114:6806/15885 213 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:25.243417) v4 ==== 2004+0+0 (2526677030 0 0) 0x55f67d14fa00 con 0x55f64e3fe000 > -3764> 2018-07-11 08:06:25.244251 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6810/15287 conn(0x55f64e401000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.4 seq 213 0x55f673909600 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:25.243417) v4 > -3763> 2018-07-11 08:06:25.244260 7f493102f700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6802/15919 conn(0x55f64e3db000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.11 seq 213 0x55f67d14fa00 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:25.243417) v4 > -3762> 2018-07-11 08:06:25.244271 7f493102f700 1 -- 10.8.128.107:0/14173 <== osd.11 10.8.128.114:6802/15919 213 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:25.243417) v4 ==== 2004+0+0 (2526677030 0 0) 0x55f67d14fa00 con 0x55f64e3db000 > -3761> 2018-07-11 08:06:25.244271 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.4 10.8.128.113:6810/15287 213 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:25.243417) v4 ==== 2004+0+0 (2526677030 0 0) 0x55f673909600 con 0x55f64e401000 > -3760> 2018-07-11 08:06:25.244271 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6803/15919 conn(0x55f64e3dc800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.11 seq 213 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:25.243417) v4 > -3759> 2018-07-11 08:06:25.244284 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.11 10.8.128.114:6803/15919 213 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:25.243417) v4 ==== 2004+0+0 (2526677030 0 0) 0x55f67d1afe00 con 0x55f64e3dc800 > -3758> 2018-07-11 08:06:25.244316 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.114:6807/15885 conn(0x55f64e3ff800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=15 cs=1 l=1). rx osd.7 seq 213 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:25.243417) v4 > -3757> 2018-07-11 08:06:25.244317 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.113:6807/15263 conn(0x55f64e3ee800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.5 seq 213 0x55f673909600 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:25.243417) v4 > -3756> 2018-07-11 08:06:25.244328 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.7 10.8.128.114:6807/15885 213 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:25.243417) v4 ==== 2004+0+0 (2526677030 0 0) 0x55f67d1afe00 con 0x55f64e3ff800 > -3755> 2018-07-11 08:06:25.244336 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.5 10.8.128.113:6807/15263 213 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:25.243417) v4 ==== 2004+0+0 (2526677030 0 0) 0x55f673909600 con 0x55f64e3ee800 > -3754> 2018-07-11 08:06:25.244359 7f4932031700 5 -- 10.8.128.107:0/14173 >> 10.8.128.108:6802/11967 conn(0x55f656aca000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.8 seq 213 0x55f67d1afe00 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:25.243417) v4 > -3753> 2018-07-11 08:06:25.244371 7f4932031700 1 -- 10.8.128.107:0/14173 <== osd.8 10.8.128.108:6802/11967 213 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:25.243417) v4 ==== 2004+0+0 (2526677030 0 0) 0x55f67d1afe00 con 0x55f656aca000 > -3752> 2018-07-11 08:06:25.244380 7f4931830700 5 -- 10.8.128.107:0/14173 >> 10.8.128.108:6803/11967 conn(0x55f656acb800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=13 cs=1 l=1). rx osd.8 seq 213 0x55f673909600 osd_ping(ping_reply e509 stamp 2018-07-11 08:06:25.243417) v4 > -3751> 2018-07-11 08:06:25.244397 7f4931830700 1 -- 10.8.128.107:0/14173 <== osd.8 10.8.128.108:6803/11967 213 ==== osd_ping(ping_reply e509 stamp 2018-07-11 08:06:25.243417) v4 ==== 2004+0+0 (2526677030 0 0) 0x55f673909600 con 0x55f656acb800 > -3750> 2018-07-11 08:06:25.276158 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f65d0c3d40 con 0 > -3749> 2018-07-11 08:06:25.277681 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 282 0x55f67be45b80 replica scrub(pg: 6.15,from:0'0,to:461'106891,epoch:509/504,start:6:a875032b:::benchmark_data_magna108_35617_object942618:0,end:6:a876165d:::benchmark_data_magna108_35617_object1098261:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -3748> 2018-07-11 08:06:25.277712 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 282 ==== replica scrub(pg: 6.15,from:0'0,to:461'106891,epoch:509/504,start:6:a875032b:::benchmark_data_magna108_35617_object942618:0,end:6:a876165d:::benchmark_data_magna108_35617_object1098261:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 219+0+0 (217361742 0 0) 0x55f67be45b80 con 0x55f66315e800 > -3747> 2018-07-11 08:06:25.512538 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0b800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 219 0x55f67d14fa00 osd_ping(ping e509 stamp 2018-07-11 08:06:26.148459) v4 > -3746> 2018-07-11 08:06:25.512538 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15919 conn(0x55f663f0a000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.11 seq 219 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:06:26.148459) v4 > -3745> 2018-07-11 08:06:25.512565 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.11 10.8.128.114:0/15919 219 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:26.148459) v4 ==== 2004+0+0 (598175015 0 0) 0x55f67d14fa00 con 0x55f663f0b800 > -3744> 2018-07-11 08:06:25.512568 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.11 10.8.128.114:0/15919 219 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:26.148459) v4 ==== 2004+0+0 (598175015 0 0) 0x55f67d1afe00 con 0x55f663f0a000 > -3743> 2018-07-11 08:06:25.512575 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:26.148459) v4 -- 0x55f67c1fbc00 con 0 > -3742> 2018-07-11 08:06:25.512601 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15919 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:26.148459) v4 -- 0x55f67665ae00 con 0 > -3741> 2018-07-11 08:06:25.532356 7f491d046700 1 -- 10.8.128.107:6800/14173 --> 10.8.128.102:6800/14211 -- mgrreport(unknown.10 +0-0 packed 742 osd_metrics=1) v5 -- 0x55f6635038c0 con 0 > -3740> 2018-07-11 08:06:25.532427 7f491d046700 1 -- 10.8.128.107:6800/14173 --> 10.8.128.102:6800/14211 -- pg_stats(18 pgs tid 0 v 0) v1 -- 0x55f667d76840 con 0 > -3739> 2018-07-11 08:06:25.741189 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.114:0/15885 conn(0x55f663216000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 220 0x55f64e20fc00 osd_ping(ping e509 stamp 2018-07-11 08:06:26.377174) v4 > -3738> 2018-07-11 08:06:25.741211 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.7 10.8.128.114:0/15885 220 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:26.377174) v4 ==== 2004+0+0 (2546013912 0 0) 0x55f64e20fc00 con 0x55f663216000 > -3737> 2018-07-11 08:06:25.741221 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:26.377174) v4 -- 0x55f67665ae00 con 0 > -3736> 2018-07-11 08:06:25.741234 7f493102f700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.114:0/15885 conn(0x55f663217800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.7 seq 220 0x55f67d14ee00 osd_ping(ping e509 stamp 2018-07-11 08:06:26.377174) v4 > -3735> 2018-07-11 08:06:25.741266 7f493102f700 1 -- 10.8.128.107:6803/14173 <== osd.7 10.8.128.114:0/15885 220 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:26.377174) v4 ==== 2004+0+0 (2546013912 0 0) 0x55f67d14ee00 con 0x55f663217800 > -3734> 2018-07-11 08:06:25.741279 7f493102f700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.114:0/15885 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:26.377174) v4 -- 0x55f67d151800 con 0 > -3733> 2018-07-11 08:06:25.815765 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.115:0/13323 conn(0x55f663ebf800 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.6 seq 202 0x55f673909600 osd_ping(ping e509 stamp 2018-07-11 08:06:26.168866) v4 > -3732> 2018-07-11 08:06:25.815791 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.6 10.8.128.115:0/13323 202 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:26.168866) v4 ==== 2004+0+0 (4029098488 0 0) 0x55f673909600 con 0x55f663ebf800 > -3731> 2018-07-11 08:06:25.815808 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:26.168866) v4 -- 0x55f67665a000 con 0 > -3730> 2018-07-11 08:06:25.815903 7f493102f700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.115:0/13323 conn(0x55f663ebe000 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=16 cs=1 l=1). rx osd.6 seq 202 0x55f67d14fa00 osd_ping(ping e509 stamp 2018-07-11 08:06:26.168866) v4 > -3729> 2018-07-11 08:06:25.815922 7f493102f700 1 -- 10.8.128.107:6802/14173 <== osd.6 10.8.128.115:0/13323 202 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:26.168866) v4 ==== 2004+0+0 (4029098488 0 0) 0x55f67d14fa00 con 0x55f663ebe000 > -3728> 2018-07-11 08:06:25.815933 7f493102f700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.115:0/13323 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:26.168866) v4 -- 0x55f67d14fc00 con 0 > -3727> 2018-07-11 08:06:25.974985 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 475 0x55f67d14ee00 MOSDScrubReserve(11.3s5 REQUEST e509) v1 > -3726> 2018-07-11 08:06:25.975006 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 475 ==== MOSDScrubReserve(11.3s5 REQUEST e509) v1 ==== 43+0+0 (433425070 0 0) 0x55f67d14ee00 con 0x55f668a29000 > -3725> 2018-07-11 08:06:25.975072 7f491a040700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.115:6801/13323 -- MOSDScrubReserve(11.3s0 REJECT e509) v1 -- 0x55f67d14f600 con 0 > -3724> 2018-07-11 08:06:25.975395 7f493102f700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.115:6801/13323 conn(0x55f668a29000 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=23 cs=1 l=0). rx osd.6 seq 476 0x55f67d14fe00 MOSDScrubReserve(11.3s5 RELEASE e509) v1 > -3723> 2018-07-11 08:06:25.975423 7f493102f700 1 -- 10.8.128.107:6801/14173 <== osd.6 10.8.128.115:6801/13323 476 ==== MOSDScrubReserve(11.3s5 RELEASE e509) v1 ==== 43+0+0 (3603460268 0 0) 0x55f67d14fe00 con 0x55f668a29000 > -3722> 2018-07-11 08:06:26.063240 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0d000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 224 0x55f673909400 osd_ping(ping e509 stamp 2018-07-11 08:06:27.308765) v4 > -3721> 2018-07-11 08:06:26.063272 7f4931830700 1 -- 10.8.128.107:6803/14173 <== osd.5 10.8.128.113:0/15263 224 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:27.308765) v4 ==== 2004+0+0 (256779917 0 0) 0x55f673909400 con 0x55f663f0d000 > -3720> 2018-07-11 08:06:26.063264 7f4932031700 5 -- 10.8.128.107:6802/14173 >> 10.8.128.113:0/15263 conn(0x55f663f0e800 :6802 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=24 cs=1 l=1). rx osd.5 seq 224 0x55f67d1afe00 osd_ping(ping e509 stamp 2018-07-11 08:06:27.308765) v4 > -3719> 2018-07-11 08:06:26.063294 7f4931830700 1 -- 10.8.128.107:6803/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:27.308765) v4 -- 0x55f67665a000 con 0 > -3718> 2018-07-11 08:06:26.063300 7f4932031700 1 -- 10.8.128.107:6802/14173 <== osd.5 10.8.128.113:0/15263 224 ==== osd_ping(ping e509 stamp 2018-07-11 08:06:27.308765) v4 ==== 2004+0+0 (256779917 0 0) 0x55f67d1afe00 con 0x55f663f0e800 > -3717> 2018-07-11 08:06:26.063325 7f4932031700 1 -- 10.8.128.107:6802/14173 --> 10.8.128.113:0/15263 -- osd_ping(ping_reply e509 stamp 2018-07-11 08:06:27.308765) v4 -- 0x55f67665ae00 con 0 > -3716> 2018-07-11 08:06:26.307911 7f491b042700 1 -- 10.8.128.107:6801/14173 --> 10.8.128.113:6805/15263 -- rep_scrubmap(6.15 e509 from shard 10) v1 -- 0x55f655c94b40 con 0 > -3715> 2018-07-11 08:06:26.309662 7f4931830700 5 -- 10.8.128.107:6801/14173 >> 10.8.128.113:6805/15263 conn(0x55f66315e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=37 cs=1 l=0). rx osd.5 seq 283 0x55f6712ea000 replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:509/504,start:6:a876165d:::benchmark_data_magna108_35617_object1098261:0,end:6:a876befa:::benchmark_data_magna108_35617_object955894:0,chunky:1,deep:1,seed:4294967295,version:7) v7 > -3714> 2018-07-11 08:06:26.309694 7f4931830700 1 -- 10.8.128.107:6801/14173 <== osd.5 10.8.128.113:6805/15263 283 ==== replica scrub(pg: 6.15,from:0'0,to:0'0,epoch:509/504,start:6:a876165d:::benchmark_data_magna108_35617_object1098261:0,end:6:a876befa:::benchmark_data_magna108_35617_object955894:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 219+0+0 (1033471138 0 0) 0x55f6712ea000 con 0x55f66315e800 > -3713> 2018-07-11 08:06:26.597135 7f4931830700 5 -- 10.8.128.107:6803/14173 >> 10.8.128.107:0/12617 conn(0x55f64e3c5000 :6803 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=21 cs=1 l=1). rx osd.9 seq 219 0x55f673909600 osd_ping(ping e509 stamp 2018-07-11 08:06:26.596568) v4 > -3712> 2018-07-11 08:06:26.597160 7f4931830700 1 -- 10.8.128.107:6803/14173 <